DevOps SRE Blog

As you all know Oracle SOA Suite uses dehydration tables to store the state of instances. We can query these tables to troubleshoot issues. Here are few sqls I find useful: 1) Count of bpel instances created between two timestamps. Count doesn’t include mediator instances: select count(1) from cube_instance  where creation_date between to_timestamp('2018-04-23 09:00', 'YYYY-MM-DD HH24:MI') and to_timestamp('2018-04-24 17:00', 'YYYY-MM-DD HH24:MI') 2) Count of bpel instances created between two timestamps group by hour: select to_char(creation_date, 'YYYY-MM-DD HH24'), count(1) from cube_instance where creation_date between to_timestamp('2018-04-23 09:00', 'YYYY-MM-DD HH24:MI') and to_timestamp('2018-04-24 17:00', 'YYYY-MM-DD HH24:MI') group by to_char(creation_date, 'YYYY-MM-DD HH24') order by to_char(creation_date, 'YYYY-MM-DD HH24'); 3) Average and max response times: select TO_CHAR(created_time, ...

We can setup email notifications whenever an error occurs in MFT like failed connection to source or target etc. This is very useful feature of MFT. We had setup this monitoring for our MFT servers. We used to get below error frequently in production server while connecting to a 3rd party application.   Error occurred while polling the listening source XYZ Cause: Unexpected error occurred while polling the JCA source.                  Action: Review the diagnostic log for the exact error, which begins with the string 'ERROR:'. This error line, along with the preceding lines, indicates the reason for this exception. Contact Oracle Support Services if the problem persists. The endpoint here was the target system to which MFT transfers file. We had setup retry at target so transfers were retried whenever this error occurs. So, it didn’t have direct impact on business. But we used to get around 100 alerts mails with these connection errors. It was tiring experience to read and chec...

Scripts to monitor and managed Oracle Fusion Middleware applications are available at following github location:  https://github.com/RameshPoonati/oracleMiddlewareScripts Weblogic : Disable Diagnostic Modules Error Report Generation from logs Application/Deployment Monitoring http log monitoring JMS Bridge Monitoring JMS Consumer Monitoring Log Archiving JMS Queue Monitoring Server Monitoring Domain Startup/Shutdown Thread Dump Generation Thread Monitoring SOA Suite : Monitor Adapter Status Reference instance response time from audit trail ODI : Duplicate Execution Monitoring Oracle Service Bus (OSB) : OSB Proxy Status Change Shell Scripts : Disk Usage Monitoring

MFT supports 3 ways of transferring files: 1) Polling 2) Schedule 3) Events Each one has different use case. Event based is used if you want to transfer a file based on an event. e.g., Trigger file transfer after completion of an extract process. Following error occurred in our production environment when an external application invoked MFT: Caused By: javax.xml.ws.soap.SOAPFaultException: MFT Event processing failed with exception code: MFT_WS_EVENT_SERVICE_EVENT_ALREADY_ACTIVE_ON_SOURCE           at com.sun.xml.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:197)                at  MFT doesn’t allow parallel processing of events. So, calling application has to call MFT sequentially. In our case, there was no pending file transfer and previous file was successfully transferred. It seems a bug in MFT and it was not able to mark the transfer as complete. There are two ways to mark the previous tra...

There is a known bug in Oracle Managed File Transfer (MFT) which causes leaking of connections. This used to cause OOM error once in a month. We have an alert – which sends mail if heap usage is greater than 95% and 3 consecutive garbage collections not able to free much heap space. As we have two node cluster, we restart the server which reached heap utilization of more than 95%. On that particular day, the person on shift missed the alert. He checked alter after sometime and server health became ‘Not Reachable’. He tried to restart from console but he was not able to stop it and server status was showing as ‘RUNNING’. He tried to restart from command line even it didn’t work. He gave me a call. I faced similar issue when I was working for a different customer. So I was quickly able to identify issue.   Checked if there was any zombie process running (ps aux | grep Z). Yes, there was a zombie process. It was server process. Looks like serve...

We have patched our production Oracle Managed File Transfer (MFT) and restarted server but it was failing to start. Server logs didn’t through any apparent clue. But that morning while logging into MFT console, I have observed an error similar to “Not able to find core.java”. Luckily, it struck to me and searched for the jar file in prod but not able to find. Later found the location of the file in lower env.  Copied the jar file from lower env to prod env and able to restart the server. Location of the file is:   $MFT_HOME/modules/oracle.mft   Not sure how this file alone got deleted from production server. It is still a mystery 🙂