DevOps SRE Blog

Let us learn about SSL/TLS and how it applies to Weblogic and Fusion Middleware. 1) What is SSL/TLS? Transport Layer Secuity (TLS) and Secure Socket Layer (SSL) are cryptographic protocols used to securely communicate over a computer network. SSL was deprecated in 2014 after finding a vulnerability. TLS became new standard for secured http. TLS 1.3 is latest TLS version. 2) Why SSL/TLS? It is easy to intercept and read data transmitted in plain text over a network. Obviously we don’t like bank details, email or personal information fall in wrong hands. TLS protects from this by encrypting data between client and server.   3) How TLS works? TLS uses public key infrastructure to encrypt data exchange. TLS handshake at high level:      i)    Agree on version of TLS to use.      ii)   Agree on Cipher suites to use.      iii) Validate identity of the server using server certificate by client. In case of two way ...

There was an interesting issue that involved EBS, SOA and ODI with a client that I worked earlier. Customer is using Product Data Hub (PDH) application along with Integrated SOA Gateway (ISG) for publishing product details from master repository (EBS) to downstream systems.  Oracle SOA is used as integration layer and ODI is used to extract bulk data from PDH. All applications are deployed in Azure and Azure Application Gateway is used as Loadbalancer (LB). High level flow is as below: EBS Business Event -> SOA -> ODI PIM Webservice -> ODI Scenario -> EBS DB. Lately the customer has been observing stuck threads in SOA and ODI and response never comes back to SOA even after waiting for hours. I got interested in this problem, will let you know the reason at the end. We followed below procedure to find the root cause: The issue was replicated at will in lower envs. We observed stuck threads in both SOA and ODI when issue occurred. We took the thread dumps. We could see th...

We had an interesting problem over weekend and thought of sharing with all. One of my colleagues called me up and told that admin server in OSB cluster was not starting up and process was hanging. Here are the steps we followed to troubleshoot the issue. 1) He told that server had been up more than a week and he added JMX port before restarting the server. He was very confident that it was not causing issue. But I insisted to rollback this recent change and start again. Situation was not improved, server startup was hanging again. So, we rule out that this change was not causing issue. 2) We ran top command and checked if there was any pressure on resources. cpu, memory and load average were very low. 3) Checked log files for potential errors. Nothing useful information was found in logs too. 4) The symptoms were similar to recent issue I have worked on. So, we ran lsof command to find if server was waiting for any network connection ( lsof -a -i4 -i6 -itcp -p <pid> ).  lsof...

Recently I started working on a new client engagement. This setup was new to us. I was troubleshooting some issue and thought of checking log files. I was able to find server.log and diagnostic log files but surprisingly server.out files were missing all together. Here are the steps I have followed to troubleshoot the issue: 1) Checked if the managed server was started from console (node manager) or using startup scripts. As you know that stdout logs will be written to nohup if started from commandline. Server was started from node manager, so this possibility was ruled out. 2) Compared with logging config of an env where server.out logs are present. Didn’t find any difference. 3) As it is dev env, thought of checking it quickly restarting server. Restarted server and I could see server.out file and it was getting updated. 4) Returned to my desk after lunch, surprisingly the log was gone again. 5) Checked lsof output for this process and found interesting thing as below: [myid@abc...

This blog covers basics of Vim/vi that are useful in day to day activities of a weblogic/fmw admin. Navigation Commands : h – Move cursor left j – Move cursor down k – Move cursor up l – Move cursor right 0 – Move to first character of real line gj/gk – Move down or up by display lines instead of real lines g0 – To first character of display line (when line spreads multiple lines, useful in navigating log files) gg – Go to beginning of the file G – Go to last line of the file 8G/8gg – Go to 8th line w – Move to start of the word e – Move to end of the word b – Move backward to start of the word Movement by finding characte r: f{char} – Move to the next occurrence of {char} F{char} – Move backward to the previous occurrence of {char} t{char} – Move to the character before the next occurrence of {char} T{char} – Move backward to the character after the previous occurrence of {char} *...

Patching is a confusing topic for beginners. In this post, I will try to explain different terminologies. CPU – Short for Critical Patch Update. It comprises of security patches.   PSU – Short for Patch Set Updates. It includes security and priority fixes. Both patches are released quarterly on the Tuesday closest to the 17th day of the months of January, April, July and October. Both the patches are cumulative that means every patch includes fixes of the early patches. So, there is no need to apply earlier patches. Once a PSU has been applied on the system, the recommended method to apply all future CPU program security content is to apply future PSUs. In other words, once a PSU has been applied, it is not recommended to switch back to traditional N-apply patches. Reverting to CPU from PSU is complex and time consuming task and so it is not recommended. One-Off : If Oracle finds critical vulnerability that can’t wait till next patch release, it rel...

As you all know Oracle SOA Suite uses dehydration tables to store the state of instances. We can query these tables to troubleshoot issues. Here are few sqls I find useful: 1) Count of bpel instances created between two timestamps. Count doesn’t include mediator instances: select count(1) from cube_instance  where creation_date between to_timestamp('2018-04-23 09:00', 'YYYY-MM-DD HH24:MI') and to_timestamp('2018-04-24 17:00', 'YYYY-MM-DD HH24:MI') 2) Count of bpel instances created between two timestamps group by hour: select to_char(creation_date, 'YYYY-MM-DD HH24'), count(1) from cube_instance where creation_date between to_timestamp('2018-04-23 09:00', 'YYYY-MM-DD HH24:MI') and to_timestamp('2018-04-24 17:00', 'YYYY-MM-DD HH24:MI') group by to_char(creation_date, 'YYYY-MM-DD HH24') order by to_char(creation_date, 'YYYY-MM-DD HH24'); 3) Average and max response times: select TO_CHAR(created_time, ...