DevOps SRE Blog

Oracle weblogic managed servers (SOA/OSB/ESS) were failing to restart after making few changes. The error written into logs was not straight forward and we spent couple of hours debugging the issue. Sharing this so that others can benefit from it. We are using Exadata as dehydration store. Network domain for applications we are managing got changed. As part of this change we changed datasource, ftp adapter config to refer to new domain. We made all required changes and tried to restart servers. We were able to bring up admin server but all managed servers failed to restart. Following error was written logs. Caused by: java.io.IOException: [DeploymentService:290066]Error occurred while downloading files from Administration Server for deployment request “42,336,427,916,066,804”. Underlying error is: “[DeploymentService:290065]Deployment service servlet encountered an Exception while handling the deployment datatransfer message for request id “42,336,427,916,066,804...

Let us learn about SSL/TLS and how it applies to Weblogic and Fusion Middleware. 1) What is SSL/TLS? Transport Layer Secuity (TLS) and Secure Socket Layer (SSL) are cryptographic protocols used to securely communicate over a computer network. SSL was deprecated in 2014 after finding a vulnerability. TLS became new standard for secured http. TLS 1.3 is latest TLS version. 2) Why SSL/TLS? It is easy to intercept and read data transmitted in plain text over a network. Obviously we don’t like bank details, email or personal information fall in wrong hands. TLS protects from this by encrypting data between client and server.   3) How TLS works? TLS uses public key infrastructure to encrypt data exchange. TLS handshake at high level:      i)    Agree on version of TLS to use.      ii)   Agree on Cipher suites to use.      iii) Validate identity of the server using server certificate by client. In case of two way ...

We had an interesting problem over weekend and thought of sharing with all. One of my colleagues called me up and told that admin server in OSB cluster was not starting up and process was hanging. Here are the steps we followed to troubleshoot the issue. 1) He told that server had been up more than a week and he added JMX port before restarting the server. He was very confident that it was not causing issue. But I insisted to rollback this recent change and start again. Situation was not improved, server startup was hanging again. So, we rule out that this change was not causing issue. 2) We ran top command and checked if there was any pressure on resources. cpu, memory and load average were very low. 3) Checked log files for potential errors. Nothing useful information was found in logs too. 4) The symptoms were similar to recent issue I have worked on. So, we ran lsof command to find if server was waiting for any network connection ( lsof -a -i4 -i6 -itcp -p <pid> ).  lsof...

This is in continuation to the first  part in the series. Let us discuss grep and lsof in part 2: grep : 1) Case insensitive search: grep -i "string" FILE   2) Checking for full words, not for sub-strings using grep -w grep -w "serachword" file 3) Displaying lines before/after/around the match using grep -A, -B and -C. This is very useful if you want to see lines around search string: i) grep -A <N> "string" FILENAME # display N lines after match: ii) grep -B <N> "string" FILENAME # display N lines before match iii) grep -C 2 "Example" demo_text # display N lines around match 4) Searching in all files recursively using grep -r grep -r "ramesh" * 5) Invert match using grep -v grep -v -e "pattern" -e "pattern" 6) Counting the number of matches using grep -c. There is no need to pipe the output to wc. grep -c "pattern" filename grep -v -c this demo_file (Count of lines not matching patter...

In this series of blog posts, I will provide useful Unix commands and tips that are useful in everyday sysadmin life in general and weblogic/oracle middleware admins in particular. I am writing these blogs based on personal notes gathered over years mostly from web. Unfortunately, I didn’t maintain source of the articles/websites. Extremely sorry for not giving due credits. Let us start the series with find command. I am dedicating whole blog post to find as it is versatile and useful. find command is used to search files/directories in a directory. Using find exec option e.g., renamme, copy. 1) To find all files and directories in a given directory and sub directories. find . # Finds all the files and directories in current directory.It searches current dir if none is specified find /abc/xyz # Finds all the files and directories in directory named /abc/xyz 2) We can limit search results only to either files or directories. find . -type f # limits the results to only files. f here...

Scripts to monitor and managed Oracle Fusion Middleware applications are available at following github location:  https://github.com/RameshPoonati/oracleMiddlewareScripts Weblogic : Disable Diagnostic Modules Error Report Generation from logs Application/Deployment Monitoring http log monitoring JMS Bridge Monitoring JMS Consumer Monitoring Log Archiving JMS Queue Monitoring Server Monitoring Domain Startup/Shutdown Thread Dump Generation Thread Monitoring SOA Suite : Monitor Adapter Status Reference instance response time from audit trail ODI : Duplicate Execution Monitoring Oracle Service Bus (OSB) : OSB Proxy Status Change Shell Scripts : Disk Usage Monitoring

We had DB based partition for dehydration store and we used to drop old partitions every month to purge old data. We dropped partition as usual but forgot to rebuild indexes. Due to this indexes became unusable and connection pool size of db adapter reached maximum value (1000 in this case). Below error was written to log files.  [2017-07-01T05:49:45.884-07:00] [osb_server1] [ERROR] [OSB-381967] [oracle.osb.transports.jca.jcatransport] [tid: [ACTIVE].ExecuteThread: ’96’ for queue: ‘weblogic.kernel.Default (self-tuning)’] [userId: ] [ecid: 79e678b0-d24f-4750-938a-8fb9e6c2fab2-01e97c36,1:5426731] [APP: Service Bus JCA Transport Provider] [FlowId: 0000LnyCmqJ6YNBrVWYBU81PAaVR00fxSh] Invoke JCA outbound service failed with application error, exception: com.bea.wli.sb.transports.jca.JCATransportException: oracle.tip.adapter.sa.api.JCABindingException: oracle.tip.adapter.sa.impl.fw.ext.org.collaxa.thirdparty.apache.wsif.WSIFException: servicebus:/CommonServices/Ar...

Once one of my teammates called me up and told me that OSB server in test environment is not coming up. He told me that he tried restarting servers multiple times and there is no luck. The previous day, I have setup a monitoring script in that environment. So initially, I was misguided by a statistics related error and initial efforts gone in that direction. From my experience, initial startup errors could be due to network or db. So I have ran lsof command and found that there was a connection to remote jms and socket status is SYN_SENT (lsof -p | grep TCP | grep -v ESTABLISHED). This gave me enough clue and revisited logs and found below error. <11-Apr-2019 07:10:29 o’clock BST> << eis/jms/XXX_RemoteTopic > ResourceAllocationException generated by resource adapter on call to ManagedConnectionFactory.createManagedConnection(): “BINDING.JCA-12141 ERRJMS_CONN_FAC_NOT_FOUND. ERRJMS_CONN_FAC_NOT_FOUND. Unable to instantiate connection factory. JMS adapter was unable to look...

It was observed “EM” application was not active under deployments section also have seen LDAP errors in Admin log. So I have brought down Admin server and took backup of changelog.data & changelog.index files under $DomainHome/servers/AdminServer/data/ldap/ldapfiles and removed them. Restarted the admin server successfully, After this activity EM application is active and able to access the “EM” console. Error from Server Logs : <Error> <EmbeddedLDAP> <hostname.abc.com> <AdminServer> <VDE Replication Thread> <<anonymous>> <> <bc4be7ee-e9 70-4d24-b7e5-42a89a9f80a1-000000bf> <1556530851514> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <Agreement ‘osb_server1’: In valid Changetype: 64in change#167764> ####<29-Apr-2019 10:40:51 o’clock BST> <Error> <EmbeddedLDAP> <hostname.abc.com> <AdminServer> <VDE Replication Threa...

We had Oracle Service Bus 12.1.3 cluster. Admin server happened to restart for some reason. But admin server startup process used to stuck and not move at all. Logs when admin server was stuck listed below: AdminServer.log ####<May 22, 2017 4:23:10 AM PDT> <Info> <Deployer> <serverabc.xyz> <AdminServer> <[STANDBY] ExecuteThread: ‘3’ for queue: ‘weblogic.kernel.Default (self-tuning)’> <<WLS Kernel>> <> <0d1240df-6563-4d52-be74-200ed896969e-00000004> <1495452190002> <BEA-149060> <Module /httpTransport of application Service Bus Kernel successfully transitioned from STATE_PREPARED to STATE_ADMIN on server AdminServer.> ####<May 22, 2017 4:23:14 AM PDT> <Info> <Server> <serverabc.xyz> <AdminServer> <[ACTIVE] ExecuteThread: ‘1’ for queue: ‘weblogic.kernel.Default (self-tuning)’> <<WLS Kernel>> <> <> <14954...