WEBLOGIC ADMIN SERVER STUCK DURING STARTUP

 Recently, we had an issue at VM level which forced us to restart application servers. Most of the weblogic applications started without any issue except one application. When we tried to start admin server, it was hung for a long time. I faced similar issues earlier (Case 1, Case 2, Case 3). So, verified if I ran into one these issues. But luck didn’t favor me and got to investigate it on a Friday evening.

I tried restarting again (Friday inertia to deep dive and was looking for a short cut). But it didn’t help so I really got to get my hands dirty. I checked logs but that was not useful much as server.out/nohup.out was stuck after writing below line.

<Apr 8, 2021 10:15:45,040 PM PDT> <Warning> <Coherence> <BEA-000000> <2021-04-08 22:15:45.040/85.650 Oracle Coherence GE 12.2.1.3.0 <Warning> (thread=[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): The cluster name has not been configured, a value of "oracle's cluster" has been automatically generated>
<Apr 8, 2021 10:15:55,101 PM PDT> <Warning> <oracle.security.opss.internal.runtime.ServiceContextManagerImpl> <BEA-000000> <Bootstrap services are used by OPSS internally and clients should never need to directly read/write bootstrap credentials. If required, use Wlst or configuration management interfaces.>
<Apr 8, 2021 10:15:55,730 PM PDT> <Warning> <oracle.dms.instrument> <DMS-50763> <Attempt to create pre-existing noun /soa_domain/AdminServer/em, of type wls_jaxrsapp_resources, with a conflicting type wls_ear.>
<Apr 8, 2021 10:15:56,067 PM PDT> <Notice> <Log Management> <BEA-170027> <The server has successfully established a connection with the Domain level Diagnostic Service.>

Next, took multiple thread dumps of the admin server process and noticed that below weblogic file store thread was not progressing.

"[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" #29 daemon prio=5 os_prio=0 tid=0x00007f8158003000 nid=0x623d in Object.wait() [0x00007f81a0823000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at weblogic.common.CompletionRequest.getResult(CompletionRequest.java:115)
        - locked <0x00000006d4998f70> (a weblogic.common.CompletionRequest)
        at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.readRecord(PersistentStoreDataArchive.java:708)
        - locked <0x00000006faa39810> (a weblogic.diagnostics.archive.wlstore.HarvestedPersistentStoreDataArchive)
        at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.recreateUnfinishedIndexPage(PersistentStoreDataArchive.java:641)
        at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.readIndexInfo(PersistentStoreDataArchive.java:560)
        at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.<init>(PersistentStoreDataArchive.java:385)
        at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.<init>(PersistentStoreDataArchive.java:343)
        at weblogic.diagnostics.archive.wlstore.HarvestedPersistentStoreDataArchive.<init>(HarvestedPersistentStoreDataArchive.java:45)
        at weblogic.diagnostics.accessor.DiagnosticDataAccessServiceFactory.createHarvestedDataAccessService(DiagnosticDataAccessServiceFactory.java:212)
        at weblogic.diagnostics.accessor.DiagnosticDataAccessServiceFactory.createDiagnosticDataAccessService_internal(DiagnosticDataAccessServiceFactory.java:80)
        at weblogic.diagnostics.accessor.DiagnosticDataAccessServiceFactory.createDiagnosticDataAccessService(DiagnosticDataAccessServiceFactory.java:56)
        at weblogic.diagnostics.accessor.WLSAccessorMBeanFactoryImpl.createDiagnosticDataAccessService(WLSAccessorMBeanFactoryImpl.java:260)
        at weblogic.diagnostics.accessor.DataAccessRuntime.createDiagnosticDataAccessService(DataAccessRuntime.java:129)
        at weblogic.diagnostics.accessor.WLSAccessorMBeanFactoryImpl.createDataAccessRuntime(WLSAccessorMBeanFactoryImpl.java:226)
        at weblogic.diagnostics.accessor.WLSAccessorMBeanFactoryImpl.access$000(WLSAccessorMBeanFactoryImpl.java:60)
        at weblogic.diagnostics.accessor.WLSAccessorMBeanFactoryImpl$1.run(WLSAccessorMBeanFactoryImpl.java:211)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:370)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:163)
        at weblogic.diagnostics.accessor.WLSAccessorMBeanFactoryImpl.createDiagnosticDataAccessRuntime(WLSAccessorMBeanFactoryImpl.java:207)

From above thread stack, we have concluded that process was stuck while reading diagnostic file store data. As it is a non-prod application and data is related to weblogic diagnostic, we took backup of filestore files and removed original files. Admin server started happily this time.

Comments

Post a Comment

Popular posts from this blog

HOW WE REDUCED SOA OSB PROVISIONING FROM 4 DAYS TO 4 HOURS

 In our organization, we’ve undertaken the task of upgrading both SOA and OSB from version 12.2.1.3 to 12.2.1.4. This involves an out-of-place migration approach, wherein we create a new environment and switch traffic during the cutover. As you know, this process typically requires several days we opt for manual provisioing. A very significant amount of automation work was previously accomplished for 12.2.1.3 by our predecessors. Despite these efforts, the process still used to take lot of time due to following factors: Bugs in existing automation. For example, SOA composite deployments is not stable which warrants lot of manual work Import of 10s of TLS Certificates into KSS Manual retiring and shutting down of services Monitoring setup Miscellaneous: Increasing Log file count, enabling GC logging, changing heap parameters, bash profile changes While our current automation initiatives have yielded substantial savings, we aimed to capitalize on this upgrade activity to enable end-t...
Read more

NOT ABLE TO START RABBITMQ CLUSTER: CANNOT DECLARE A QUEUE ‘~S’ ON NODE ‘~S’: ~255P

We have 7 node rabbitmq cluster and we are using sharding queues. We have shutdown all non prod rabbitmq nodes as part of VM patching. As it was non prod, we thought of shutting down all together instead of rolling patching. We couldn’t start the any of the rabbitmq post patching. All of nodes were failing to start with the below error: 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> “Cannot declare a queue ‘~s’ on node ‘~s’: ~255p”, 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> [“queue ‘sharding: shard.my.queue – rabbit@rabbit1’ in vhost ‘my_vhost'” This is a known issue in some old versions of rabbitmq. rabbitmq node fails to come up with above error. Work around for this is to disable sharding policy for the problematic sharding queue. It can be done either from command line or from rabbitmq admin page. It requires to connect a running node. But as all the nodes were down, we could not disable sharding policy. Luckily, rabbitmq allows disabling sharding pulgi...
Read more

SOA SUITE 12.2.1.4 INSTALLATION: GOT EXCEPTION WHEN AUTO CONFIGURING THE SCHEMA COMPONENT(S) WITH DATA OBTAINED FROM SHADOW TABLE

  We planned to do out of place upgrade from soa suite 12.2.1.3 to 12.2.1.4. As part of this, we planned to automate end to end provisioning. While executing domain configuration using ansible and python scripts, we got following error in domain creation step. Got exception when auto configuring the schema component(s) with data obtained from shadow table. Failed to build JDBC Connection object: com.oracle.cie.domain.script.jython.CommandExceptionHandler. exception. Quick internet search gave few solutions like deleting domain-registry.xml file, verifying dehydration store password supplied to script etc. But none of them helped. After trying several options, I thought I might have done some stupid error prior to domain creation step. To track down the issue, though of checking what getDatabaseDefaults actually doing. I tried to grep for the getDatabaseDefaults function call inside oracle home. But couldn’t find the source code for the python function. As a final resort, thought of...
Read more