DevOps SRE Blog

We were trying to deploy a composite from em console and it got failed with below error. We checked server logs as em console didn’t through enough information. We found following error message in logs: Oct 6, 2020 4:43:10,904 AM UTC Error HTTP abc.xyz wls_soa1 ExecuteThread: ‘0’ for queue: ‘weblogic.socket.Muxer’ WLS Kernel 4ff02afa-c6f1-45cd-8bc6-85d8b0d5da72-00003489 1601959390904 [severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] BEA-101096 POST size exceeded the MaxPostSize=327,679. The jar we were trying to deploy was in MBs and was higher than http Max Post Size. We increased (admin console -> Servers -> ServerName -> Protocols -> http) its value to resolve the issue.

Our MW applications are hosted in cloud. We received a incident saying development em console is extremely slow and developers are not able to open instances from em console. As with any performance issue, we ran top command on VMs. We didn’t see any high cpu utilization on node 1 but node 2 utilization is almost 100% as shown below: ps command showed that this process was running journalctl: We requested our sysadmin team to investigate further. There were some Linux bugs that cause high cpu utilization. So, initial investigation went in that direction. Later one of colleagues, found that cron job scheduled on this particular node was consuming all the cpu. He disabled the cronjob and cpu utilization came down. The cron job was as below: [or acle@abc123 ~]$ crontab -l #* * * * * /home/oracle/.local/cpu/bin/update >/dev/null 2>&1 It looked suspicious to me as I don’t remember of setting up such cron job even though these are the new applications we are managing. So, we though...

  We recently faced a performance issue in one of the weblogic applications we are supporting. We tried to debug the issue using OS tools and application logging. Unfortunately, we were not succeeded. So, we thought of using Java profiling tool Java Mission Control (JMC) and flight recorder. When we tried to connect to JVM using JMC from our laptops, connection was failing. It was due to firewall restrictions. You might already aware that in such scenarios we can use port forwarding to make remote port appear as local port. So, we have enabled port forwarding by changing putty session settings. Still, we are not able to connect to JVM from JMC. Came across this nice  article  after searching internet. In nutshell, if we want connect to remote JMX port using port forwarding technique, we need to do below two tasks: Enable jmxremote.rmi.port in addition to regular jmxremote.port. Forward jmxremote.port in addition to jmxremote.rmi.port. As this is weblogic based application...

  In 12.2.1.4.0, oracle came up with reference configuration. What it means is that by default soa/osb comes up with some tuning parameters e.g., JTA timeout values, datasource connections, Java heap settings Xms and Xmx. The default Xms and Xmx values are 4 GB for Admin server and 8 GB for managed server. These values are specified in $FMW_HOME/Oracle_Home/soa/common/bin/setSOARefConfigEnv.sh. These values may not be suitable for your workload. You can set desired heap values by modifying this file. But this is not best practice for the following reasons: When product is upgraded these values will be lost and need to redo again. If there are multiple clusters in you domain, say soa and bam, same setting applies to both clusters. It may not be desired. It is better to place custom heap settings by creating file called setUserOverrides.sh in $DOMAIN_HOME/bin. Bonus advantage is values present in this file are honored during domain upgrades too. Sample setUserOverrides.sh file for re...