Not Able to Start Weblogic/MFT Server After Out of Memory Error

There is a known bug in Oracle Managed File Transfer (MFT) which causes leaking of connections. This used to cause OOM error once in a month. We have an alert – which sends mail if heap usage is greater than 95% and 3 consecutive garbage collections not able to free much heap space. As we have two node cluster, we restart the server which reached heap utilization of more than 95%. On that particular day, the person on shift missed the alert. He checked alter after sometime and server health became ‘Not Reachable’.

He tried to restart from console but he was not able to stop it and server status was showing as ‘RUNNING’. He tried to restart from command line even it didn’t work. He gave me a call. I faced similar issue when I was working for a different customer. So I was quickly able to identify issue.  

Checked if there was any zombie process running (ps aux | grep Z). Yes, there was a zombie process. It was server process. Looks like server shutdown didn’t cleanly shutdown the server. Weblogic will not allow to run another process to run unless this zombie process is killed

Parent id of this zombie is 1 which is init. So, we have rebooted VM and able start the MFT server post VM reboot.

Comments

Post a Comment

Popular posts from this blog

SOA SUITE 12.2.1.4 INSTALLATION: GOT EXCEPTION WHEN AUTO CONFIGURING THE SCHEMA COMPONENT(S) WITH DATA OBTAINED FROM SHADOW TABLE

  We planned to do out of place upgrade from soa suite 12.2.1.3 to 12.2.1.4. As part of this, we planned to automate end to end provisioning. While executing domain configuration using ansible and python scripts, we got following error in domain creation step. Got exception when auto configuring the schema component(s) with data obtained from shadow table. Failed to build JDBC Connection object: com.oracle.cie.domain.script.jython.CommandExceptionHandler. exception. Quick internet search gave few solutions like deleting domain-registry.xml file, verifying dehydration store password supplied to script etc. But none of them helped. After trying several options, I thought I might have done some stupid error prior to domain creation step. To track down the issue, though of checking what getDatabaseDefaults actually doing. I tried to grep for the getDatabaseDefaults function call inside oracle home. But couldn’t find the source code for the python function. As a final resort, thought of...
Read more

HOW WE REDUCED SOA OSB PROVISIONING FROM 4 DAYS TO 4 HOURS

 In our organization, we’ve undertaken the task of upgrading both SOA and OSB from version 12.2.1.3 to 12.2.1.4. This involves an out-of-place migration approach, wherein we create a new environment and switch traffic during the cutover. As you know, this process typically requires several days we opt for manual provisioing. A very significant amount of automation work was previously accomplished for 12.2.1.3 by our predecessors. Despite these efforts, the process still used to take lot of time due to following factors: Bugs in existing automation. For example, SOA composite deployments is not stable which warrants lot of manual work Import of 10s of TLS Certificates into KSS Manual retiring and shutting down of services Monitoring setup Miscellaneous: Increasing Log file count, enabling GC logging, changing heap parameters, bash profile changes While our current automation initiatives have yielded substantial savings, we aimed to capitalize on this upgrade activity to enable end-t...
Read more

RABBITMQ CONNECTION ERROR: JAVAX.NET.SSL.SSLHANDSHAKEEXCEPTION: INVALID ECDH SERVERKEYEXCHANGE SIGNATURE

We have a 7 node test rabbitmq cluster. One of the rabbitmq users reported that they were getting “ javax.net.ssl.SSLHandshakeException: Invalid ECDH ServerKeyExchange signature ” error while connecting to Rabbitmq. We checked logs and also checked application up time. Rabbitmq servers hadn’t been restarted for the past 25 days. Also, we didn’t observe any errors in logs related to vhost used by client. We also checked if the certificate is expired and it was not. We got reasonably confident that it was not our issue and asked to check from client side. As day progressed, we got complaints from two other client application teams. One of them uses php and other uses java. At this point, we started suspecting rabbitmq. Also, one team with multiple consumers told us that they are facing issue only with few of the consumers. With this information, we felt that issue could be with few of the nodes and not complete cluster. One of the blogs on the internet suggested that this error could occ...
Read more