DevOps SRE Blog

We were trying to deploy a composite from em console and it got failed with below error. We checked server logs as em console didn’t through enough information. We found following error message in logs: Oct 6, 2020 4:43:10,904 AM UTC Error HTTP abc.xyz wls_soa1 ExecuteThread: ‘0’ for queue: ‘weblogic.socket.Muxer’ WLS Kernel 4ff02afa-c6f1-45cd-8bc6-85d8b0d5da72-00003489 1601959390904 [severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] BEA-101096 POST size exceeded the MaxPostSize=327,679. The jar we were trying to deploy was in MBs and was higher than http Max Post Size. We increased (admin console -> Servers -> ServerName -> Protocols -> http) its value to resolve the issue.

Our MW applications are hosted in cloud. We received a incident saying development em console is extremely slow and developers are not able to open instances from em console. As with any performance issue, we ran top command on VMs. We didn’t see any high cpu utilization on node 1 but node 2 utilization is almost 100% as shown below: ps command showed that this process was running journalctl: We requested our sysadmin team to investigate further. There were some Linux bugs that cause high cpu utilization. So, initial investigation went in that direction. Later one of colleagues, found that cron job scheduled on this particular node was consuming all the cpu. He disabled the cronjob and cpu utilization came down. The cron job was as below: [or acle@abc123 ~]$ crontab -l #* * * * * /home/oracle/.local/cpu/bin/update >/dev/null 2>&1 It looked suspicious to me as I don’t remember of setting up such cron job even though these are the new applications we are managing. So, we though...

  We recently faced a performance issue in one of the weblogic applications we are supporting. We tried to debug the issue using OS tools and application logging. Unfortunately, we were not succeeded. So, we thought of using Java profiling tool Java Mission Control (JMC) and flight recorder. When we tried to connect to JVM using JMC from our laptops, connection was failing. It was due to firewall restrictions. You might already aware that in such scenarios we can use port forwarding to make remote port appear as local port. So, we have enabled port forwarding by changing putty session settings. Still, we are not able to connect to JVM from JMC. Came across this nice  article  after searching internet. In nutshell, if we want connect to remote JMX port using port forwarding technique, we need to do below two tasks: Enable jmxremote.rmi.port in addition to regular jmxremote.port. Forward jmxremote.port in addition to jmxremote.rmi.port. As this is weblogic based application...

  In 12.2.1.4.0, oracle came up with reference configuration. What it means is that by default soa/osb comes up with some tuning parameters e.g., JTA timeout values, datasource connections, Java heap settings Xms and Xmx. The default Xms and Xmx values are 4 GB for Admin server and 8 GB for managed server. These values are specified in $FMW_HOME/Oracle_Home/soa/common/bin/setSOARefConfigEnv.sh. These values may not be suitable for your workload. You can set desired heap values by modifying this file. But this is not best practice for the following reasons: When product is upgraded these values will be lost and need to redo again. If there are multiple clusters in you domain, say soa and bam, same setting applies to both clusters. It may not be desired. It is better to place custom heap settings by creating file called setUserOverrides.sh in $DOMAIN_HOME/bin. Bonus advantage is values present in this file are honored during domain upgrades too. Sample setUserOverrides.sh file for re...

I was configuring new datasource from admin console and got below error while activating datasource. Error An error occurred during activation of changes, please see the log for details. Error oracle.net.ns.NetException: Unknown host specified Error Substituted for the exception oracle.net.ns.NetException which lacks a String contructor, original message – Unknown host specified I have double checked datasource configuration and everything looked fine. My colleague, Vijay, suggested that there could be a problem with datasource other than current one. I went to jdbc config directory ($DOMAIN_HOME/config/jdbc directory and retrieved all hostnames using combination grep and awk commands. The suspicion came true, there was another hostname that was not getting resolved. We have untargeted datasoruces from all servers and activated changes and issue got resolved.

We were building a domain with both SOA and ESS in same cluster. We got below error in domain configuration wizard during domain creation. CFGFWK-64254: Error occurred in “Artifacts Generation” phase execution Encountered Error: CFGFWK-64038: The app-svc name “ESSAPP” must have target. CFGFWK-64038: The app-svc name “ESSAPP” must have target. CFGFWK-64038: Provide a valid target. During ConfigMapping Reason for this error was that we selected  Oracle Enterprise Scheduler (ESS)  service in  Templates  screen of domain configuration wizard but it was not selected to deploy on any server. Fix to the problem is to select  ESS-MGD-SVRS  server group in  managed servers  screen. With this selection, ESS will be deployed on that particular managed server.

Oracle weblogic managed servers (SOA/OSB/ESS) were failing to restart after making few changes. The error written into logs was not straight forward and we spent couple of hours debugging the issue. Sharing this so that others can benefit from it. We are using Exadata as dehydration store. Network domain for applications we are managing got changed. As part of this change we changed datasource, ftp adapter config to refer to new domain. We made all required changes and tried to restart servers. We were able to bring up admin server but all managed servers failed to restart. Following error was written logs. Caused by: java.io.IOException: [DeploymentService:290066]Error occurred while downloading files from Administration Server for deployment request “42,336,427,916,066,804”. Underlying error is: “[DeploymentService:290065]Deployment service servlet encountered an Exception while handling the deployment datatransfer message for request id “42,336,427,916,066,804...