Struck with Stuck Threads Running Across EBS, SOA and ODI

There was an interesting issue that involved EBS, SOA and ODI with a client that I worked earlier. Customer is using Product Data Hub (PDH) application along with Integrated SOA Gateway (ISG) for publishing product details from master repository (EBS) to downstream systems.  Oracle SOA is used as integration layer and ODI is used to extract bulk data from PDH. All applications are deployed in Azure and Azure Application Gateway is used as Loadbalancer (LB). High level flow is as below:

EBS Business Event -> SOA -> ODI PIM Webservice -> ODI Scenario -> EBS DB.

Lately the customer has been observing stuck threads in SOA and ODI and response never comes back to SOA even after waiting for hours. I got interested in this problem, will let you know the reason at the end. We followed below procedure to find the root cause:

The issue was replicated at will in lower envs. We observed stuck threads in both SOA and ODI when issue occurred. We took the thread dumps. We could see that in SOA and ODI, threads were waiting on network connection. These threads were reading from network at two ends of the webservice call i.e., between SOA and ODI. We also observed that ODI scenario got completed successfully but no response was sent back to SOA. We checked SOA and ODI logs but no useful information was found.

We checked network connections opened by SOA and ODI servers using lsof. Both servers have had a network connection in ESTABLISHED state with other server (SOA to ODI and vice versa). This gave us clue that connectivity might be timing out somewhere between SOA and ODI and LB is one of the suspects. My colleague, Siva, also told that this issue was occurring when ODI scenario takes more than 4 mins to complete.

Couple of days passed. I was going through old incidents and CRs to better understand a new client system (I have recently been moved to a different client assignment). Customer is on Oracle Cloud Infrastructure (OCI). It was a CR to increase read timeout value of OCI LB. It was Eureka moment !!!. I immediately opened google and searched for “Azure loadbalancer 4 minutes read timeout. There were lot of search results, which explained about our issue. Here is brief explanation, Azure App Gateway has default read timeout setting of 4 minutes after which it discards connection. Unfortunately, it doesn’t do it gracefully and do not let know both client and backend servers about closing connection. So, in our case, both SOA and ODI always keep the sockets open unaware of this behavior.

We also wanted to double check our understanding with a controlled test. Developed a simple two-way synchronous SOA composite that will take time as parameter and wait for those many minutes and give response. Both SOA and ODI use same LB, so it can validate our understanding. We first ran with 3 minutes and then with 5 minutes in soapui. We got response for 3 mins request but soapui request was hanging when tested with 5 minutes. Later, we replaced LB url with ip in soapui and tested with 5 minutes input. This time we got response. This confirmed that the issue was with LB’s 4 minute read timeout.

Our colleague raised issue with Microsoft. Microsoft told that read timeout cannot be changed for Application Gateway when used with private ip. So, practical solution for now is to tune EBS DB to provide response within 4 mins.

We got this issue almost a year ago, at that time we were not able to diagnose the problem. In fact, we were misguided to apply a patch in EBS. This is the reason for my interest. I reflected on why we are able to solve this problem this time. Here are my thoughts:

First, we are able to replicate the issue at will now in lower env and found a pattern that issue occurs when ODI scenario takes more than 4 mins. Credits to my colleague Siva. Second, purely luck. Came across LB read timeout setting in my current assignment by chance. Third, we were quickly able to develop sample application to demonstrate our understanding. Thanks to my colleague Vijay who helped in learning bit of developing soa composites.

Thanks for reading long post.

Comments

Post a Comment

Popular posts from this blog

HOW WE REDUCED SOA OSB PROVISIONING FROM 4 DAYS TO 4 HOURS

 In our organization, we’ve undertaken the task of upgrading both SOA and OSB from version 12.2.1.3 to 12.2.1.4. This involves an out-of-place migration approach, wherein we create a new environment and switch traffic during the cutover. As you know, this process typically requires several days we opt for manual provisioing. A very significant amount of automation work was previously accomplished for 12.2.1.3 by our predecessors. Despite these efforts, the process still used to take lot of time due to following factors: Bugs in existing automation. For example, SOA composite deployments is not stable which warrants lot of manual work Import of 10s of TLS Certificates into KSS Manual retiring and shutting down of services Monitoring setup Miscellaneous: Increasing Log file count, enabling GC logging, changing heap parameters, bash profile changes While our current automation initiatives have yielded substantial savings, we aimed to capitalize on this upgrade activity to enable end-t...
Read more

NOT ABLE TO START RABBITMQ CLUSTER: CANNOT DECLARE A QUEUE ‘~S’ ON NODE ‘~S’: ~255P

We have 7 node rabbitmq cluster and we are using sharding queues. We have shutdown all non prod rabbitmq nodes as part of VM patching. As it was non prod, we thought of shutting down all together instead of rolling patching. We couldn’t start the any of the rabbitmq post patching. All of nodes were failing to start with the below error: 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> “Cannot declare a queue ‘~s’ on node ‘~s’: ~255p”, 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> [“queue ‘sharding: shard.my.queue – rabbit@rabbit1’ in vhost ‘my_vhost'” This is a known issue in some old versions of rabbitmq. rabbitmq node fails to come up with above error. Work around for this is to disable sharding policy for the problematic sharding queue. It can be done either from command line or from rabbitmq admin page. It requires to connect a running node. But as all the nodes were down, we could not disable sharding policy. Luckily, rabbitmq allows disabling sharding pulgi...
Read more

SOA SUITE 12.2.1.4 INSTALLATION: GOT EXCEPTION WHEN AUTO CONFIGURING THE SCHEMA COMPONENT(S) WITH DATA OBTAINED FROM SHADOW TABLE

  We planned to do out of place upgrade from soa suite 12.2.1.3 to 12.2.1.4. As part of this, we planned to automate end to end provisioning. While executing domain configuration using ansible and python scripts, we got following error in domain creation step. Got exception when auto configuring the schema component(s) with data obtained from shadow table. Failed to build JDBC Connection object: com.oracle.cie.domain.script.jython.CommandExceptionHandler. exception. Quick internet search gave few solutions like deleting domain-registry.xml file, verifying dehydration store password supplied to script etc. But none of them helped. After trying several options, I thought I might have done some stupid error prior to domain creation step. To track down the issue, though of checking what getDatabaseDefaults actually doing. I tried to grep for the getDatabaseDefaults function call inside oracle home. But couldn’t find the source code for the python function. As a final resort, thought of...
Read more