HIGH LOAD AVERAGE BUT LOW CPU ULITIZATION IN LINUX

We have two node Oracle SOA cluster. Both nodes get almost same number of requests but we observed that on one node load average was way higher than second node though cpu utilization was almost same.

1) One reason could be for high load average is if system has lot of processes that are uninterruptible state. 

2) Checked if there were any processes in  ‘D’ (Disk wait) state.

top -b -n 1 | awk ‘{if (NR <=7) print; else if ($8 == “D”) print $1 ” ” $8 ” ”  $12 }’

Output:

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
14688 D df
16092 D df
17324 D df
18610 D df
19192 D df
19735 D df
19759 D df
20619 D df

3) There are lot df processes waiting for disk. Find the time since they have been waiting for disk.

ps -eo pid,cmd,lstart | grep df
14688 df -h                       Tue May 16 22:45:01 2017
16092 df -h                       Tue May 16 22:50:01 2017
17324 df -h                       Tue May 16 22:55:01 2017
18610 df -h                       Tue May 16 23:00:01 2017
19192 df -kh                      Tue May 16 23:01:46 2017
19735 df -kh                      Tue May 16 23:04:54 2017
19759 df -h                       Tue May 16 23:05:01 2017

From above, we can conclude that df commands started on May 16th were hanging (possibly due to NFS issue.) Killing the processed reduced load average gradually.

Comments

Post a Comment

Popular posts from this blog

HOW WE REDUCED SOA OSB PROVISIONING FROM 4 DAYS TO 4 HOURS

 In our organization, we’ve undertaken the task of upgrading both SOA and OSB from version 12.2.1.3 to 12.2.1.4. This involves an out-of-place migration approach, wherein we create a new environment and switch traffic during the cutover. As you know, this process typically requires several days we opt for manual provisioing. A very significant amount of automation work was previously accomplished for 12.2.1.3 by our predecessors. Despite these efforts, the process still used to take lot of time due to following factors: Bugs in existing automation. For example, SOA composite deployments is not stable which warrants lot of manual work Import of 10s of TLS Certificates into KSS Manual retiring and shutting down of services Monitoring setup Miscellaneous: Increasing Log file count, enabling GC logging, changing heap parameters, bash profile changes While our current automation initiatives have yielded substantial savings, we aimed to capitalize on this upgrade activity to enable end-t...
Read more

RABBITMQ CONNECTION ERROR: JAVAX.NET.SSL.SSLHANDSHAKEEXCEPTION: INVALID ECDH SERVERKEYEXCHANGE SIGNATURE

We have a 7 node test rabbitmq cluster. One of the rabbitmq users reported that they were getting “ javax.net.ssl.SSLHandshakeException: Invalid ECDH ServerKeyExchange signature ” error while connecting to Rabbitmq. We checked logs and also checked application up time. Rabbitmq servers hadn’t been restarted for the past 25 days. Also, we didn’t observe any errors in logs related to vhost used by client. We also checked if the certificate is expired and it was not. We got reasonably confident that it was not our issue and asked to check from client side. As day progressed, we got complaints from two other client application teams. One of them uses php and other uses java. At this point, we started suspecting rabbitmq. Also, one team with multiple consumers told us that they are facing issue only with few of the consumers. With this information, we felt that issue could be with few of the nodes and not complete cluster. One of the blogs on the internet suggested that this error could occ...
Read more

NOT ABLE TO START RABBITMQ CLUSTER: CANNOT DECLARE A QUEUE ‘~S’ ON NODE ‘~S’: ~255P

We have 7 node rabbitmq cluster and we are using sharding queues. We have shutdown all non prod rabbitmq nodes as part of VM patching. As it was non prod, we thought of shutting down all together instead of rolling patching. We couldn’t start the any of the rabbitmq post patching. All of nodes were failing to start with the below error: 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> “Cannot declare a queue ‘~s’ on node ‘~s’: ~255p”, 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> [“queue ‘sharding: shard.my.queue – rabbit@rabbit1’ in vhost ‘my_vhost'” This is a known issue in some old versions of rabbitmq. rabbitmq node fails to come up with above error. Work around for this is to disable sharding policy for the problematic sharding queue. It can be done either from command line or from rabbitmq admin page. It requires to connect a running node. But as all the nodes were down, we could not disable sharding policy. Luckily, rabbitmq allows disabling sharding pulgi...
Read more