DevOps SRE Blog

  As sysadmins, we need to daily deal with logs. It is overwhelming to check huge log files especially for application servers like weblogic which has numerous log files like diagnostic, server log, out log, OHS logs etc. It is time consuming and quickly lead to fatigue if log files are huge like weblogic. I have developed 2 scripts which extract and summarize logs. I have been using these scripts in both production and test environments. These scripts helped in quickly reacting to a situation and saved time. If your organization already has tools like Splunk, ELK, these scripts may not much use. But for those who couldn’t afford, these scripts could help. Please find brief description about each script as below: Log Extractor : This script takes time in mins or time period as input and extracts logs produced between that time period. We can also filter log files based on log severity as shown below screenshot. Script Location:  https://github.com/RameshPoonati/oracleMiddlewar...

  Recently, we had an issue at VM level which forced us to restart application servers. Most of the weblogic applications started without any issue except one application. When we tried to start admin server, it was hung for a long time. I faced similar issues earlier ( Case 1 ,   Case 2 ,   Case 3 ). So, verified if I ran into one these issues. But luck didn’t favor me and got to investigate it on a Friday evening. I tried restarting again (Friday inertia to deep dive and was looking for a short cut). But it didn’t help so I really got to get my hands dirty. I checked logs but that was not useful much as server.out/nohup.out was stuck after writing below line. <Apr 8, 2021 10:15:45,040 PM PDT> <Warning> <Coherence> <BEA-000000> <2021-04-08 22:15:45.040/85.650 Oracle Coherence GE 12.2.1.3.0 <Warning> (thread=[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): The cluster name has not been configur...