DevOps SRE Blog

Oracle weblogic managed servers (SOA/OSB/ESS) were failing to restart after making few changes. The error written into logs was not straight forward and we spent couple of hours debugging the issue. Sharing this so that others can benefit from it. We are using Exadata as dehydration store. Network domain for applications we are managing got changed. As part of this change we changed datasource, ftp adapter config to refer to new domain. We made all required changes and tried to restart servers. We were able to bring up admin server but all managed servers failed to restart. Following error was written logs. Caused by: java.io.IOException: [DeploymentService:290066]Error occurred while downloading files from Administration Server for deployment request “42,336,427,916,066,804”. Underlying error is: “[DeploymentService:290065]Deployment service servlet encountered an Exception while handling the deployment datatransfer message for request id “42,336,427,916,066,804...

Kubernetes port forwarding feature can be used to access a pod port from local machine without accessing pod. This can be achieved by other means like creating service. But port forwarding is somewhat quick. Port forwarding can be enabled by running following command: kubectl port-forward <pod Name> <local port>: <pod port> e.g., kubectl port-forward my-pod 9000:80 I have a test  k8s cluster with single master node and two worker nodes. I have enabled port forward by using above command and invoked curl by opening another ssh session from master node. vagrant@master-1:~$ curl http://localhost:9000 curl: (52) Empty reply from server I was expecting some output but got “Empty reply from server”. Also observed, in ssh session where port forwarding is enabled, below error message. E0811 13:51:54.597132   13384 portforward.go:391] an error occurred forwarding 9000 -> 80: error forwarding port 80 to pod ce0a72c043baa73a773b0803c9b3dc365229e...

Let us learn about SSL/TLS and how it applies to Weblogic and Fusion Middleware. 1) What is SSL/TLS? Transport Layer Secuity (TLS) and Secure Socket Layer (SSL) are cryptographic protocols used to securely communicate over a computer network. SSL was deprecated in 2014 after finding a vulnerability. TLS became new standard for secured http. TLS 1.3 is latest TLS version. 2) Why SSL/TLS? It is easy to intercept and read data transmitted in plain text over a network. Obviously we don’t like bank details, email or personal information fall in wrong hands. TLS protects from this by encrypting data between client and server.   3) How TLS works? TLS uses public key infrastructure to encrypt data exchange. TLS handshake at high level:      i)    Agree on version of TLS to use.      ii)   Agree on Cipher suites to use.      iii) Validate identity of the server using server certificate by client. In case of two way ...

There was an interesting issue that involved EBS, SOA and ODI with a client that I worked earlier. Customer is using Product Data Hub (PDH) application along with Integrated SOA Gateway (ISG) for publishing product details from master repository (EBS) to downstream systems.  Oracle SOA is used as integration layer and ODI is used to extract bulk data from PDH. All applications are deployed in Azure and Azure Application Gateway is used as Loadbalancer (LB). High level flow is as below: EBS Business Event -> SOA -> ODI PIM Webservice -> ODI Scenario -> EBS DB. Lately the customer has been observing stuck threads in SOA and ODI and response never comes back to SOA even after waiting for hours. I got interested in this problem, will let you know the reason at the end. We followed below procedure to find the root cause: The issue was replicated at will in lower envs. We observed stuck threads in both SOA and ODI when issue occurred. We took the thread dumps. We could see th...

We had an interesting problem over weekend and thought of sharing with all. One of my colleagues called me up and told that admin server in OSB cluster was not starting up and process was hanging. Here are the steps we followed to troubleshoot the issue. 1) He told that server had been up more than a week and he added JMX port before restarting the server. He was very confident that it was not causing issue. But I insisted to rollback this recent change and start again. Situation was not improved, server startup was hanging again. So, we rule out that this change was not causing issue. 2) We ran top command and checked if there was any pressure on resources. cpu, memory and load average were very low. 3) Checked log files for potential errors. Nothing useful information was found in logs too. 4) The symptoms were similar to recent issue I have worked on. So, we ran lsof command to find if server was waiting for any network connection ( lsof -a -i4 -i6 -itcp -p <pid> ).  lsof...

Recently I started working on a new client engagement. This setup was new to us. I was troubleshooting some issue and thought of checking log files. I was able to find server.log and diagnostic log files but surprisingly server.out files were missing all together. Here are the steps I have followed to troubleshoot the issue: 1) Checked if the managed server was started from console (node manager) or using startup scripts. As you know that stdout logs will be written to nohup if started from commandline. Server was started from node manager, so this possibility was ruled out. 2) Compared with logging config of an env where server.out logs are present. Didn’t find any difference. 3) As it is dev env, thought of checking it quickly restarting server. Restarted server and I could see server.out file and it was getting updated. 4) Returned to my desk after lunch, surprisingly the log was gone again. 5) Checked lsof output for this process and found interesting thing as below: [myid@abc...

This blog covers basics of Vim/vi that are useful in day to day activities of a weblogic/fmw admin. Navigation Commands : h – Move cursor left j – Move cursor down k – Move cursor up l – Move cursor right 0 – Move to first character of real line gj/gk – Move down or up by display lines instead of real lines g0 – To first character of display line (when line spreads multiple lines, useful in navigating log files) gg – Go to beginning of the file G – Go to last line of the file 8G/8gg – Go to 8th line w – Move to start of the word e – Move to end of the word b – Move backward to start of the word Movement by finding characte r: f{char} – Move to the next occurrence of {char} F{char} – Move backward to the previous occurrence of {char} t{char} – Move to the character before the next occurrence of {char} T{char} – Move backward to the character after the previous occurrence of {char} *...