HOW TO FIND THE ENDPOINT A JAVA THREAD IS READING FROM

 We may experience performance problems due to stuck threads i.e., when the thread is not making any progress and just stuck. Typically this may be due to contention with other threads, waiting for an event or waiting for data to be arrived on a network connection. This blog post is about finding other end of the connection which got stuck.

If the application is simple and it is connecting to couple of hosts, it is very easy to identify other end of connection. If the application connects multiple boundary systems, databases, it becomes hard to identify the slow connection especially if the code path for all these end points is same. Middleware is one such application. Some applications have built in mechanism to identify this problem. For example, Oracle SOA and OSB generates diagnostic dumps under $DOMAIN_HOME/servers/<serverName>/adr/diag/ofm/<domainName>/<serverName>/incident. This directory contains readme.txt with details like composite, osb service that the thread was handling when thread is stuck. By looking at composite/service code we can easily identify the endpoint. The other source of information is logs. Usually logs write thread id and composite name while starting execution which can help identifying the service.

If the logs are rotated or diagnostic dump doesn’t have service details, it will be challenging to identify the network endpoint. It is possible to get endpoint with the help of tools like jstack, lsof and strace. Let us see what these tools can do.

jstack: It prints snapshot of current state of Java threads. It also print operating system’s native thread id as shown below. To take thread dumps: jstack <pid> > td.txt

lsof: lsof prints list of all currently opened files. As everything is a file in Unix based systems, we can see the network connections as well. Sample command to list network connections of a process: lsof -a -i4 -i6 -itcp -iudp -P -p 12319 > lsof.txt

strace: Strace prints all system calls that a process calls. We can use below command to print system calls made by threads of a process. Each thread’s system calls will be written to a separate file. Thread id is appended to file name provided with -o option. Here, thread id is OS native thread id and not Java thread id.
strace -s 400 -fftTy -e ‘trace=!futex,getdents,clock_gettime,gettimeofday’ -p 3096 -o trace.out

Steps to find endpoint details:
1. Run jstack, lsof and strace commands as mentioned above. Strace is very resource intensive and slows down the process. Be cautious while using it in prod. Run strace command for a minute and hit ctrl + c. Need to run jstack, lsof and starace in quick succession so that they capture same snapshot.

2. Go to jstack output and find the thread that got stuck and copy the nid which is hex format. Convert it decimal format.

3. Now, go to directory where strace output is present. Name of the file corresponding to the thread of concern will be like trace.out.<nid>. For example, if nid is 0x319c, file name will be trace.out.12700

4. Find if any of the following system calls present in the file: recvfrom, recv, read, epoll, epoll_ctl, poll.

5. Find the File Descriptor of those calls and search them in lsof output which will give hostname and port of the remote host.

Comments

Post a Comment

Popular posts from this blog

HOW WE REDUCED SOA OSB PROVISIONING FROM 4 DAYS TO 4 HOURS

 In our organization, we’ve undertaken the task of upgrading both SOA and OSB from version 12.2.1.3 to 12.2.1.4. This involves an out-of-place migration approach, wherein we create a new environment and switch traffic during the cutover. As you know, this process typically requires several days we opt for manual provisioing. A very significant amount of automation work was previously accomplished for 12.2.1.3 by our predecessors. Despite these efforts, the process still used to take lot of time due to following factors: Bugs in existing automation. For example, SOA composite deployments is not stable which warrants lot of manual work Import of 10s of TLS Certificates into KSS Manual retiring and shutting down of services Monitoring setup Miscellaneous: Increasing Log file count, enabling GC logging, changing heap parameters, bash profile changes While our current automation initiatives have yielded substantial savings, we aimed to capitalize on this upgrade activity to enable end-t...
Read more

NOT ABLE TO START RABBITMQ CLUSTER: CANNOT DECLARE A QUEUE ‘~S’ ON NODE ‘~S’: ~255P

We have 7 node rabbitmq cluster and we are using sharding queues. We have shutdown all non prod rabbitmq nodes as part of VM patching. As it was non prod, we thought of shutting down all together instead of rolling patching. We couldn’t start the any of the rabbitmq post patching. All of nodes were failing to start with the below error: 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> “Cannot declare a queue ‘~s’ on node ‘~s’: ~255p”, 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> [“queue ‘sharding: shard.my.queue – rabbit@rabbit1’ in vhost ‘my_vhost'” This is a known issue in some old versions of rabbitmq. rabbitmq node fails to come up with above error. Work around for this is to disable sharding policy for the problematic sharding queue. It can be done either from command line or from rabbitmq admin page. It requires to connect a running node. But as all the nodes were down, we could not disable sharding policy. Luckily, rabbitmq allows disabling sharding pulgi...
Read more

SOA SUITE 12.2.1.4 INSTALLATION: GOT EXCEPTION WHEN AUTO CONFIGURING THE SCHEMA COMPONENT(S) WITH DATA OBTAINED FROM SHADOW TABLE

  We planned to do out of place upgrade from soa suite 12.2.1.3 to 12.2.1.4. As part of this, we planned to automate end to end provisioning. While executing domain configuration using ansible and python scripts, we got following error in domain creation step. Got exception when auto configuring the schema component(s) with data obtained from shadow table. Failed to build JDBC Connection object: com.oracle.cie.domain.script.jython.CommandExceptionHandler. exception. Quick internet search gave few solutions like deleting domain-registry.xml file, verifying dehydration store password supplied to script etc. But none of them helped. After trying several options, I thought I might have done some stupid error prior to domain creation step. To track down the issue, though of checking what getDatabaseDefaults actually doing. I tried to grep for the getDatabaseDefaults function call inside oracle home. But couldn’t find the source code for the python function. As a final resort, thought of...
Read more