DevOps SRE Blog

We have two node Oracle SOA cluster. Both nodes get almost same number of requests but we observed that on one node load average was way higher than second node though cpu utilization was almost same. 1) One reason could be for high load average is if system has lot of processes that are uninterruptible state.  2) Checked if there were any processes in  ‘D’ (Disk wait) state. top -b -n 1 | awk ‘{if (NR <=7) print; else if ($8 == “D”) print $1 ” ” $8 ” ”  $12 }’ Output: PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND 14688 D df 16092 D df 17324 D df 18610 D df 19192 D df 19735 D df 19759 D df 20619 D df 3) There are lot df processes waiting for disk. Find the time since they have been waiting for disk. ps -eo pid,cmd,lstart | grep df 14688 df -h                       Tue May 16 22:45:01 2017 16092 df -h           ...

We have 100+ Oracle middleware servers that are running on Azure cloud. Most of these servers were not used during weekend. So, guys in infra team thought of saving cost by shutting down MW servers. They asked us  to provide a single script to shutdown all machines and another script to start them. They would invoke these scripts through Ansible and shutdown VMs once all processes on the VM are stopped. We came up with a wrapper script that will invoke individual component startup scripts. We have two node cluster and wrapper script is placed on node 1. Node 1 will connect to node 2 using ssh and execute individual component scripts on that node. The order is start managed server node manager -> OHS NM ->  Start OHS etc with delay of 30 seconds. But ssh session that starts managed server node manager never terminates. Found this answer on stackoverflow helped.  We added nohup and routed output to /dev/null. Snippet of the code: echo “`ssh $userName@$secondNodeIP /bin/bash ...

One of our soa servers health went into WARNING state. Also, in application health tab of admin console, we could see ‘MDB application soa-infra is NOT connected to messaging system’ error message. While analyzing server logs, found below error messages: <27-Jun-2019 15:01:03 o’clock BST> <[ACTIVE] ExecuteThread: ’54’ for queue: ‘weblogic.kernel.Default (self-tuning)’> <> <> <23aac39c-aac6-4725-9f14-6f7fd98a1a7c-000001a5> <1561644063760> <[severity-value: 16] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <JMS Module “SISJMSModule” deployed to WebLogic domain sod_domain defines an entity of type Uniform Distributed Topic with name “SISJMSModule!CSFNotificationTopic” with default targeting enabled. There is no JMS server having a persistent store with distribution policy “Distributed” available in WebLogic domain sod_domain to host the entity. The destination “SISJMSModule!CSFNotificationTopic” will be hosted when a JMS server having a...

We all aware that Weblogic uses internal LDAP server to store user credential information. We won’t be able to start servers if LDAP server gets corrupted. Weblogic takes backup of LDAP configuration and usually keeps 7 days old backups. Following procedure can be followed to restore LDAP configruation from backups.   1) Shutdown complete domain.   2) Take backup of data directory.       mv <domain_home>/servers/AdminServer/data to <domain_home>/servers/AdminServer/data.bkp   3) cd <domain_home>/servers/AdminServer/data.bkp/ldap/backup   4) Restore ldap files from backup: unzip -j EmbeddedLDAPBackup.1.zip -d <domain_home>/servers/AdminServer/data/ldap/ldapfiles

Following error occurred in our test B2B environment after accidentally changing keystore password from B2B console. Error is gone after re-entering correct keystore password. 2017-10-24T09:29:22.458-07:00] [soa_server2] [ERROR] [] [oracle.soa.b2b.engine] [tid: DaemonWorkThread: ’15’ of WorkManager: ‘wm/SOAWorkManager’] [userId: ] [ecid: 676f960c-bafc-428e-bfc3-ce699d0cd75d-00422c87,0] [APP: soa-infra] java.security.UnrecoverableKeyException: Cannot recover key[[ at sun.security.provider.KeyProtector.recover(KeyProtector.java:328) at sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:138) at sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:55) at java.security.KeyStore.getKey(KeyStore.java:792) at sun.security.ssl.SunX509KeyManagerImpl.(SunX509KeyManagerImpl.java:131) at sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:68) at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:259) at oracle.tip.b2b.securit...

One of our Oracle SOA servers were crashing lately and Oracle requested for core dumps to analyze issue further. In hotspot crash report, following message is written: Not able to take core dump as core file size is set to zero. Below are steps followed to generate core dumps on JVM crash: Step 1: Increased ulimit using following command: ulimit -c unlimited > /dev/null 2>&1 Step 2: Restarted server. But no luck, still no core dumps are generated. It was not effective as it is applicable to only for the current shell session and not to new shell session. Step 3: Updated the value in .bashrc file ‘ulimit -c unlimited > /dev/null 2>&1′ . This made all new sessions’ ulimit to new value. Again started soa server, still ulimit value for core dump file size is zero. Step 4: This was not expected. On analysis found that, soa server is taking its ulimit values from its parent process which is node manager process as server was started via node manager. Restarted node manage...

Java and python both have time modules. So when we tried to use strftime function in time, wlst picked up time module present in Java and not that one in python. This is due to name collision. To avoid this error, import time module with an alias name as below and prefix strftime call with alias name. import time as pytime