DevOps SRE Blog

I was configuring new datasource from admin console and got below error while activating datasource. Error An error occurred during activation of changes, please see the log for details. Error oracle.net.ns.NetException: Unknown host specified Error Substituted for the exception oracle.net.ns.NetException which lacks a String contructor, original message – Unknown host specified I have double checked datasource configuration and everything looked fine. My colleague, Vijay, suggested that there could be a problem with datasource other than current one. I went to jdbc config directory ($DOMAIN_HOME/config/jdbc directory and retrieved all hostnames using combination grep and awk commands. The suspicion came true, there was another hostname that was not getting resolved. We have untargeted datasoruces from all servers and activated changes and issue got resolved.

We were building a domain with both SOA and ESS in same cluster. We got below error in domain configuration wizard during domain creation. CFGFWK-64254: Error occurred in “Artifacts Generation” phase execution Encountered Error: CFGFWK-64038: The app-svc name “ESSAPP” must have target. CFGFWK-64038: The app-svc name “ESSAPP” must have target. CFGFWK-64038: Provide a valid target. During ConfigMapping Reason for this error was that we selected  Oracle Enterprise Scheduler (ESS)  service in  Templates  screen of domain configuration wizard but it was not selected to deploy on any server. Fix to the problem is to select  ESS-MGD-SVRS  server group in  managed servers  screen. With this selection, ESS will be deployed on that particular managed server.

Oracle weblogic managed servers (SOA/OSB/ESS) were failing to restart after making few changes. The error written into logs was not straight forward and we spent couple of hours debugging the issue. Sharing this so that others can benefit from it. We are using Exadata as dehydration store. Network domain for applications we are managing got changed. As part of this change we changed datasource, ftp adapter config to refer to new domain. We made all required changes and tried to restart servers. We were able to bring up admin server but all managed servers failed to restart. Following error was written logs. Caused by: java.io.IOException: [DeploymentService:290066]Error occurred while downloading files from Administration Server for deployment request “42,336,427,916,066,804”. Underlying error is: “[DeploymentService:290065]Deployment service servlet encountered an Exception while handling the deployment datatransfer message for request id “42,336,427,916,066,804...

Kubernetes port forwarding feature can be used to access a pod port from local machine without accessing pod. This can be achieved by other means like creating service. But port forwarding is somewhat quick. Port forwarding can be enabled by running following command: kubectl port-forward <pod Name> <local port>: <pod port> e.g., kubectl port-forward my-pod 9000:80 I have a test  k8s cluster with single master node and two worker nodes. I have enabled port forward by using above command and invoked curl by opening another ssh session from master node. vagrant@master-1:~$ curl http://localhost:9000 curl: (52) Empty reply from server I was expecting some output but got “Empty reply from server”. Also observed, in ssh session where port forwarding is enabled, below error message. E0811 13:51:54.597132   13384 portforward.go:391] an error occurred forwarding 9000 -> 80: error forwarding port 80 to pod ce0a72c043baa73a773b0803c9b3dc365229e...

Let us learn about SSL/TLS and how it applies to Weblogic and Fusion Middleware. 1) What is SSL/TLS? Transport Layer Secuity (TLS) and Secure Socket Layer (SSL) are cryptographic protocols used to securely communicate over a computer network. SSL was deprecated in 2014 after finding a vulnerability. TLS became new standard for secured http. TLS 1.3 is latest TLS version. 2) Why SSL/TLS? It is easy to intercept and read data transmitted in plain text over a network. Obviously we don’t like bank details, email or personal information fall in wrong hands. TLS protects from this by encrypting data between client and server.   3) How TLS works? TLS uses public key infrastructure to encrypt data exchange. TLS handshake at high level:      i)    Agree on version of TLS to use.      ii)   Agree on Cipher suites to use.      iii) Validate identity of the server using server certificate by client. In case of two way ...

There was an interesting issue that involved EBS, SOA and ODI with a client that I worked earlier. Customer is using Product Data Hub (PDH) application along with Integrated SOA Gateway (ISG) for publishing product details from master repository (EBS) to downstream systems.  Oracle SOA is used as integration layer and ODI is used to extract bulk data from PDH. All applications are deployed in Azure and Azure Application Gateway is used as Loadbalancer (LB). High level flow is as below: EBS Business Event -> SOA -> ODI PIM Webservice -> ODI Scenario -> EBS DB. Lately the customer has been observing stuck threads in SOA and ODI and response never comes back to SOA even after waiting for hours. I got interested in this problem, will let you know the reason at the end. We followed below procedure to find the root cause: The issue was replicated at will in lower envs. We observed stuck threads in both SOA and ODI when issue occurred. We took the thread dumps. We could see th...

We had an interesting problem over weekend and thought of sharing with all. One of my colleagues called me up and told that admin server in OSB cluster was not starting up and process was hanging. Here are the steps we followed to troubleshoot the issue. 1) He told that server had been up more than a week and he added JMX port before restarting the server. He was very confident that it was not causing issue. But I insisted to rollback this recent change and start again. Situation was not improved, server startup was hanging again. So, we rule out that this change was not causing issue. 2) We ran top command and checked if there was any pressure on resources. cpu, memory and load average were very low. 3) Checked log files for potential errors. Nothing useful information was found in logs too. 4) The symptoms were similar to recent issue I have worked on. So, we ran lsof command to find if server was waiting for any network connection ( lsof -a -i4 -i6 -itcp -p <pid> ).  lsof...