WEBLOGIC ADMIN SERVER STARTUP FAILING WITH FORCE_SHUTTING_DOWN

 We shutdown our test weblogic admin server to apply recent security patches. Our colleague applied the path and tried to start Admin server. Server didn’t come-up and was failing with below error:

  1. A MultiException has 6 exceptions. They are:
  2. weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid. The user name or password or both from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.
  3. java.lang.IllegalStateException: Unable to perform operation: post construct on weblogic.security.SecurityService
  4. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.jndi.internal.RemoteNamingService errors were found.
    Server state changed to FAILED. A critical service failed. The server will shut itself down. Server state changed to FORCE_SHUTTING_DOWN.

He tried multiple options, like reentering credentials in boot.properties etc. He asked me if I could help in resolving issue. I started my analysis by executing top command. Observed 1 zombie java process. Thought that this process might be causing issue and rebooted VM. It didn’t help. Looked at the log messages again. Ideally we should not get this error as the password was not changed. So, source of the credential which is boot.properties doesn’t have problem. Other possibility was that credential store with which weblogic validates the credentials might have corrupted. The credentials store in this case was weblogic local LDAP store.

Took backup LDAP store files located at $DOMAIN_HOME/servers/AdminServer/data/ldap/ldapfiles and copied LDAP files from managed server’s LDAP store. Admin server was started without a problem this time.

After enquiring my colleague what actions he had done during patching activity, I am thinking following sequence of events caused the problem:

  1. Stopped managed servers and admin server from weblogic console.
  2. Managed server processes were stopped but Admin server process was not getting stopped.
  3. He killed the admin server process using kill command. It was not killed properly and the process became a zombie process.
  4. Looks like this LDAP store got corrupted during above step.

Comments

Post a Comment

Popular posts from this blog

SOA SUITE 12.2.1.4 INSTALLATION: GOT EXCEPTION WHEN AUTO CONFIGURING THE SCHEMA COMPONENT(S) WITH DATA OBTAINED FROM SHADOW TABLE

  We planned to do out of place upgrade from soa suite 12.2.1.3 to 12.2.1.4. As part of this, we planned to automate end to end provisioning. While executing domain configuration using ansible and python scripts, we got following error in domain creation step. Got exception when auto configuring the schema component(s) with data obtained from shadow table. Failed to build JDBC Connection object: com.oracle.cie.domain.script.jython.CommandExceptionHandler. exception. Quick internet search gave few solutions like deleting domain-registry.xml file, verifying dehydration store password supplied to script etc. But none of them helped. After trying several options, I thought I might have done some stupid error prior to domain creation step. To track down the issue, though of checking what getDatabaseDefaults actually doing. I tried to grep for the getDatabaseDefaults function call inside oracle home. But couldn’t find the source code for the python function. As a final resort, thought of...
Read more

HOW WE REDUCED SOA OSB PROVISIONING FROM 4 DAYS TO 4 HOURS

 In our organization, we’ve undertaken the task of upgrading both SOA and OSB from version 12.2.1.3 to 12.2.1.4. This involves an out-of-place migration approach, wherein we create a new environment and switch traffic during the cutover. As you know, this process typically requires several days we opt for manual provisioing. A very significant amount of automation work was previously accomplished for 12.2.1.3 by our predecessors. Despite these efforts, the process still used to take lot of time due to following factors: Bugs in existing automation. For example, SOA composite deployments is not stable which warrants lot of manual work Import of 10s of TLS Certificates into KSS Manual retiring and shutting down of services Monitoring setup Miscellaneous: Increasing Log file count, enabling GC logging, changing heap parameters, bash profile changes While our current automation initiatives have yielded substantial savings, we aimed to capitalize on this upgrade activity to enable end-t...
Read more

RABBITMQ CONNECTION ERROR: JAVAX.NET.SSL.SSLHANDSHAKEEXCEPTION: INVALID ECDH SERVERKEYEXCHANGE SIGNATURE

We have a 7 node test rabbitmq cluster. One of the rabbitmq users reported that they were getting “ javax.net.ssl.SSLHandshakeException: Invalid ECDH ServerKeyExchange signature ” error while connecting to Rabbitmq. We checked logs and also checked application up time. Rabbitmq servers hadn’t been restarted for the past 25 days. Also, we didn’t observe any errors in logs related to vhost used by client. We also checked if the certificate is expired and it was not. We got reasonably confident that it was not our issue and asked to check from client side. As day progressed, we got complaints from two other client application teams. One of them uses php and other uses java. At this point, we started suspecting rabbitmq. Also, one team with multiple consumers told us that they are facing issue only with few of the consumers. With this information, we felt that issue could be with few of the nodes and not complete cluster. One of the blogs on the internet suggested that this error could occ...
Read more