How to take Java Thread and Heap Dumps

Weblogic is nothing but a java process. So like any other java process we can take Thread and Heap dumps to analyze issues. Let us look at them one by one.

Thread Dump: Thread dump gives a snapshot of threads at the time of taking dumps. It is primarily used to analyze performance and lock/resource contention issues. It is recommended to take 5-6 thread dumps in intervals of 10-15 seconds to understand if there is any issue. With single thread dump we won’t be able judge if it contention is momentarily or of long duration. There are multiple ways of taking thread dumps but my personal choice is jstack as we can take dumps even during most of the jvm hangs.

1) Using WLST: This method is not recommended as weblogic doesn’t respond during high loads. Also, wlst uses lot of resources. Following code snippet is taken from Oracle documentation:

Place following code in a file code thread_dump.py. Please make changes according to your setup.
serverName = ‘soa_server1’
counter = 0
sleepTime = 10000
connect(‘weblogic’,’welcome1′,’t3://abc.com:8001′)
for counter in range(10):
    java.lang.Thread.sleep(sleepTime)
    fileName = ‘dump’ + serverName + ‘_’ +
    str((java.util.Calendar.getInstance().getTimeInMillis())) + ‘.dmp’
    threadDump(‘true’, fileName, serverName)
$ wlst.sh thread_dump.py

2) kill -3  

Kill -3 sends SIGQUIT signal to the process in unix systems. JVM handles this signal and sends thread dumps to STDERR. So, thread dumps will be written to server.out/nohup.out depending on the way process is started. 

3) jrcmd print_threads >> threaddump.txt

jrcmd is versatile tool that comes with jdk and can do lot of other tasks as well.

4) jstack

To take thread dumps 5 times with a delay of 10 seconds:

for i in {1..5}
do
   pid=24234
   timestamp=$(date “+%Y.%m.%d-%H.%M.%S”)
   jstack  $pid > threadDump_$pid_$timestamp.txt
   echo “sleeping …”
   sleep 10s
done

Heap Dump: Heap dump gives snapshot of memory of a java process. Heap dumps can generate huge files depending on size of the max heap setting. Say, if max heap setting is 2 GB, a heap file of more than 2 GB will be generated. Following methods can be used to generate heap dumps:

To take heap dumps:

1) jrcmd hprofdump filename=<dumpfileName>

2) jmap -dump:format=b,file=<dumpfileName> <pid>

e.g., jmap -dump:format=b,file=dump.hprof 35632

3) Java heap dump is required to analyze Out of Memory errors. So, it will be handy to generate heap dumps automatically when OOM occurs. Following java command line option can be used. This option can be added to server startup scripts.

java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<dumpFileLoc>

Apart from above options, thread and heap dumps can be generated using GUI tools like visualVM, jConsole, JMC

Comments

Post a Comment

Popular posts from this blog

HOW WE REDUCED SOA OSB PROVISIONING FROM 4 DAYS TO 4 HOURS

 In our organization, we’ve undertaken the task of upgrading both SOA and OSB from version 12.2.1.3 to 12.2.1.4. This involves an out-of-place migration approach, wherein we create a new environment and switch traffic during the cutover. As you know, this process typically requires several days we opt for manual provisioing. A very significant amount of automation work was previously accomplished for 12.2.1.3 by our predecessors. Despite these efforts, the process still used to take lot of time due to following factors: Bugs in existing automation. For example, SOA composite deployments is not stable which warrants lot of manual work Import of 10s of TLS Certificates into KSS Manual retiring and shutting down of services Monitoring setup Miscellaneous: Increasing Log file count, enabling GC logging, changing heap parameters, bash profile changes While our current automation initiatives have yielded substantial savings, we aimed to capitalize on this upgrade activity to enable end-t...
Read more

NOT ABLE TO START RABBITMQ CLUSTER: CANNOT DECLARE A QUEUE ‘~S’ ON NODE ‘~S’: ~255P

We have 7 node rabbitmq cluster and we are using sharding queues. We have shutdown all non prod rabbitmq nodes as part of VM patching. As it was non prod, we thought of shutting down all together instead of rolling patching. We couldn’t start the any of the rabbitmq post patching. All of nodes were failing to start with the below error: 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> “Cannot declare a queue ‘~s’ on node ‘~s’: ~255p”, 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> [“queue ‘sharding: shard.my.queue – rabbit@rabbit1’ in vhost ‘my_vhost'” This is a known issue in some old versions of rabbitmq. rabbitmq node fails to come up with above error. Work around for this is to disable sharding policy for the problematic sharding queue. It can be done either from command line or from rabbitmq admin page. It requires to connect a running node. But as all the nodes were down, we could not disable sharding policy. Luckily, rabbitmq allows disabling sharding pulgi...
Read more

SOA SUITE 12.2.1.4 INSTALLATION: GOT EXCEPTION WHEN AUTO CONFIGURING THE SCHEMA COMPONENT(S) WITH DATA OBTAINED FROM SHADOW TABLE

  We planned to do out of place upgrade from soa suite 12.2.1.3 to 12.2.1.4. As part of this, we planned to automate end to end provisioning. While executing domain configuration using ansible and python scripts, we got following error in domain creation step. Got exception when auto configuring the schema component(s) with data obtained from shadow table. Failed to build JDBC Connection object: com.oracle.cie.domain.script.jython.CommandExceptionHandler. exception. Quick internet search gave few solutions like deleting domain-registry.xml file, verifying dehydration store password supplied to script etc. But none of them helped. After trying several options, I thought I might have done some stupid error prior to domain creation step. To track down the issue, though of checking what getDatabaseDefaults actually doing. I tried to grep for the getDatabaseDefaults function call inside oracle home. But couldn’t find the source code for the python function. As a final resort, thought of...
Read more