DevOps SRE Blog

This post is more of a personal note than a perfect analysis. I have applied lot of workarounds and still not able to identify exactly what caused issue. Writing this blog so it will act as future reference if similar issue happens and add more details about the issue. We were adding monitoring for Kafka consumer lag using kafka-consumer-groups.sh which is located under Kafka installation bin directory. We were able to get lag in all environment except one and script execution failed with below error ( $KAFKA_BIN/kafka-consumer-groups.sh –describe –all-groups –bootstrap-server localhost:9092 ): java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: metadata at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165) at kafka.admin...

  We may experience performance problems due to stuck threads i.e., when the thread is not making any progress and just stuck. Typically this may be due to contention with other threads, waiting for an event or waiting for data to be arrived on a network connection. This blog post is about finding other end of the connection which got stuck. If the application is simple and it is connecting to couple of hosts, it is very easy to identify other end of connection. If the application connects multiple boundary systems, databases, it becomes hard to identify the slow connection especially if the code path for all these end points is same. Middleware is one such application. Some applications have built in mechanism to identify this problem. For example, Oracle SOA and OSB generates diagnostic dumps under $DOMAIN_HOME/servers/<serverName>/adr/diag/ofm/<domainName>/<serverName>/incident. This directory contains readme.txt with details like composite, osb service that th...