Server state changed to FAILED - Error occurred while downloading files from Administration Server for deployment request

Oracle weblogic managed servers (SOA/OSB/ESS) were failing to restart after making few changes. The error written into logs was not straight forward and we spent couple of hours debugging the issue. Sharing this so that others can benefit from it.

We are using Exadata as dehydration store. Network domain for applications we are managing got changed. As part of this change we changed datasource, ftp adapter config to refer to new domain. We made all required changes and tried to restart servers. We were able to bring up admin server but all managed servers failed to restart. Following error was written logs.

Caused by: java.io.IOException: [DeploymentService:290066]Error occurred while downloading files from Administration Server for deployment request “42,336,427,916,066,804”. Underlying error is: “[DeploymentService:290065]Deployment service servlet encountered an Exception while handling the deployment datatransfer message for request id “42,336,427,916,066,804” from server “wls_soa2”. Exception is: “files list is empty”.”        at weblogic.deploy.service.datatransferhandlers.HttpDataTransferHandler.getDataAsStream(HttpDataTransferHandler.java:92)        at weblogic.deploy.service.datatransferhandlers.DataHandlerManager$RemoteDataTransferHandler.getDataAsStream(DataHandlerManager.java:175)        at weblogic.deploy.internal.targetserver.datamanagement.ConfigDataUpdate.doDownload(ConfigDataUpdate.java:153)

<Aug 19, 2020 7:07:26 AM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED.>

<Aug 19, 2020 7:07:26 AM UTC> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down.>

<Aug 19, 2020 7:07:26 AM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN.>

<Aug 19, 2020 7:07:26 AM UTC> <Info> <WebLogicServer> <BEA-000238> <Shutdown has completed.>

We tried to do clean restart by clearing tmp and cache files located in manged server folders, $DOMAIN_HOME/servers/$MSERVER_NAME. It didn’t help. We searched oracle metalink. We found few docs but those were not relevant to our issue. We tried to test datasources that got updated to check if any datasources are causing issue. When we went to datasource testing page, we didn’t find any datasource instances to test. It was blank. At this point of time, we suspected that issue was with datasources.

We further analyzed admin server logs and found following error message:

<Aug 19, 2020 7:51:25,353 AM UTC> <Error> <Deployer> <abc.xyz.oraclevcn.com> <wls_soa2> <[ACTIVE] ExecuteThread: ‘6’ for queue: ‘weblogic.kernel.Default (self-tuning)’> <<WLS Kernel>> <> <29ddb107-83d3-49f7-94b8-06a137622ca6-00000010> <1597823485353> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-149205> <Failed to initialize the application  SOADataSource” due to error weblogic.application.ModuleException: java.lang.IllegalArgumentException: Invalid host:port abc-db.xyz.oraclevcn.com

It turned out that while updating datasource configuration in ONS tab, we forgot to add port numbers, we just added host names. 
When tried to correct this from admin console and activate changes, we got below error:
Message icon – Error An error occurred during activation of changes, please see the log for details.
Message icon – Error weblogic.common.ResourceException: Data Source mds-soa already exists
Message icon – Error Data Source mds-soa already exists
So, took backup of jdbc files location at $DOMAIN_HOME/config/jdbc and manually added port numbers. We were able to restart the servers after adding port number.

 

Comments

Post a Comment

Popular posts from this blog

HOW WE REDUCED SOA OSB PROVISIONING FROM 4 DAYS TO 4 HOURS

 In our organization, we’ve undertaken the task of upgrading both SOA and OSB from version 12.2.1.3 to 12.2.1.4. This involves an out-of-place migration approach, wherein we create a new environment and switch traffic during the cutover. As you know, this process typically requires several days we opt for manual provisioing. A very significant amount of automation work was previously accomplished for 12.2.1.3 by our predecessors. Despite these efforts, the process still used to take lot of time due to following factors: Bugs in existing automation. For example, SOA composite deployments is not stable which warrants lot of manual work Import of 10s of TLS Certificates into KSS Manual retiring and shutting down of services Monitoring setup Miscellaneous: Increasing Log file count, enabling GC logging, changing heap parameters, bash profile changes While our current automation initiatives have yielded substantial savings, we aimed to capitalize on this upgrade activity to enable end-t...
Read more

NOT ABLE TO START RABBITMQ CLUSTER: CANNOT DECLARE A QUEUE ‘~S’ ON NODE ‘~S’: ~255P

We have 7 node rabbitmq cluster and we are using sharding queues. We have shutdown all non prod rabbitmq nodes as part of VM patching. As it was non prod, we thought of shutting down all together instead of rolling patching. We couldn’t start the any of the rabbitmq post patching. All of nodes were failing to start with the below error: 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> “Cannot declare a queue ‘~s’ on node ‘~s’: ~255p”, 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> [“queue ‘sharding: shard.my.queue – rabbit@rabbit1’ in vhost ‘my_vhost'” This is a known issue in some old versions of rabbitmq. rabbitmq node fails to come up with above error. Work around for this is to disable sharding policy for the problematic sharding queue. It can be done either from command line or from rabbitmq admin page. It requires to connect a running node. But as all the nodes were down, we could not disable sharding policy. Luckily, rabbitmq allows disabling sharding pulgi...
Read more

SOA SUITE 12.2.1.4 INSTALLATION: GOT EXCEPTION WHEN AUTO CONFIGURING THE SCHEMA COMPONENT(S) WITH DATA OBTAINED FROM SHADOW TABLE

  We planned to do out of place upgrade from soa suite 12.2.1.3 to 12.2.1.4. As part of this, we planned to automate end to end provisioning. While executing domain configuration using ansible and python scripts, we got following error in domain creation step. Got exception when auto configuring the schema component(s) with data obtained from shadow table. Failed to build JDBC Connection object: com.oracle.cie.domain.script.jython.CommandExceptionHandler. exception. Quick internet search gave few solutions like deleting domain-registry.xml file, verifying dehydration store password supplied to script etc. But none of them helped. After trying several options, I thought I might have done some stupid error prior to domain creation step. To track down the issue, though of checking what getDatabaseDefaults actually doing. I tried to grep for the getDatabaseDefaults function call inside oracle home. But couldn’t find the source code for the python function. As a final resort, thought of...
Read more