INTRIGUING 502 BAD GATEWAY ERROR WHILE INVOKING KONG THAT PROXIES BOOMI SERVICE

 Our dev team has exposed a Boomi service through Kong Gateway to be consumed by other teams. The service was working for most of the time but throwing 502 errors randomly. Initially, we thought it was network glitch and ignored those errors. As the testing progressed to higher environments, the error rate increased. Though it was below 1 %, due to limitations at both client and upstream we were tasked to analyze and fix the issue. The high level request flow is like this: Client -> Kong Gateway -> Boomi VIP -> Boomi Atoms -> Upstream/backend service.

We started analysis with Kong and Boomi logs. Found below entries:

Kong:

{“date”:1.664432446599347E9,”log”:”2022-09-29T06:20:46.599260909Z stderr F 2022/09/29 06:20:46 [error] 38#0: *1693441 upstream prematurely closed connection while reading response header from upstream, client: 10.200.63.6, server: kong, request: \”POST /test/outbound/consumer HTTP/1.0\”, upstream:

Boomi:

2022_09_28.shared_http_server.: [28/Sep/2022:23:00:34 -0700] “POST /ws/rest/party/consumer HTTP/1.1″ 499 0 “-” “Java/11.0.5” “7d09643c-f584-4435-8d40-31cbbb820f82” “execution-f18712ff-5524-4e47-8c30-d19d43e6c88f-2022.09.28” 58043

So, Kong which is client got premature closing of connection error. At the same time, Boomi which is server got “client closed connection” (http 499) error. So, we thought someone sitting in between client and server closed connection as both Kong and Boomi thought other side of the connection closed it. As network loadbalaner is connecting both client and server, we checked with our network team, if there was any issue with loadbalancer. Dev teams, network teams and other infrastructure teams huddled into a call. Network team verified LB and confirmed that there was no issue with LB. We also changed Kong upstream endpoint to one of the Boomi backend servers instead of LB. Still we got 502 error. So, we confirmed that it was not due to LB.

As network is out of question, Dev team sent some artificial load to the service, while we were monitoring systems. Someone from Dev team noticed that service was failing when Boomi was taking more than 30 seconds to send response back to Kong. With this data, we searched internet and found this link. As we are not sure if this could resolve our problem, someone suggested to decrease timeout of value to 10 seconds to replicate issue easily. So we changed com.boomi.container.sharedServer.http.maxIdleTime value in container.properties file to 10 seconds. That suggestion worked and we could see much higher failures when timeout was set to 10 seconds. At this point, we were confident about the change and increase timeout to much higher value. We monitored for failures for next couple of days and issue didn’t occur again. If Boomi service handler had honored timeout setout at webserver Jetty layer, this issue would have been identified much quickly.

Without the team effort, the issue would have taken much more time to resolve. Thanks for Aditya, Minu, Parvathy, Kalyan and Abhilash in helping me in learning new things.

Comments

Post a Comment

Popular posts from this blog

HOW WE REDUCED SOA OSB PROVISIONING FROM 4 DAYS TO 4 HOURS

 In our organization, we’ve undertaken the task of upgrading both SOA and OSB from version 12.2.1.3 to 12.2.1.4. This involves an out-of-place migration approach, wherein we create a new environment and switch traffic during the cutover. As you know, this process typically requires several days we opt for manual provisioing. A very significant amount of automation work was previously accomplished for 12.2.1.3 by our predecessors. Despite these efforts, the process still used to take lot of time due to following factors: Bugs in existing automation. For example, SOA composite deployments is not stable which warrants lot of manual work Import of 10s of TLS Certificates into KSS Manual retiring and shutting down of services Monitoring setup Miscellaneous: Increasing Log file count, enabling GC logging, changing heap parameters, bash profile changes While our current automation initiatives have yielded substantial savings, we aimed to capitalize on this upgrade activity to enable end-t...
Read more

NOT ABLE TO START RABBITMQ CLUSTER: CANNOT DECLARE A QUEUE ‘~S’ ON NODE ‘~S’: ~255P

We have 7 node rabbitmq cluster and we are using sharding queues. We have shutdown all non prod rabbitmq nodes as part of VM patching. As it was non prod, we thought of shutting down all together instead of rolling patching. We couldn’t start the any of the rabbitmq post patching. All of nodes were failing to start with the below error: 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> “Cannot declare a queue ‘~s’ on node ‘~s’: ~255p”, 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> [“queue ‘sharding: shard.my.queue – rabbit@rabbit1’ in vhost ‘my_vhost'” This is a known issue in some old versions of rabbitmq. rabbitmq node fails to come up with above error. Work around for this is to disable sharding policy for the problematic sharding queue. It can be done either from command line or from rabbitmq admin page. It requires to connect a running node. But as all the nodes were down, we could not disable sharding policy. Luckily, rabbitmq allows disabling sharding pulgi...
Read more

SOA SUITE 12.2.1.4 INSTALLATION: GOT EXCEPTION WHEN AUTO CONFIGURING THE SCHEMA COMPONENT(S) WITH DATA OBTAINED FROM SHADOW TABLE

  We planned to do out of place upgrade from soa suite 12.2.1.3 to 12.2.1.4. As part of this, we planned to automate end to end provisioning. While executing domain configuration using ansible and python scripts, we got following error in domain creation step. Got exception when auto configuring the schema component(s) with data obtained from shadow table. Failed to build JDBC Connection object: com.oracle.cie.domain.script.jython.CommandExceptionHandler. exception. Quick internet search gave few solutions like deleting domain-registry.xml file, verifying dehydration store password supplied to script etc. But none of them helped. After trying several options, I thought I might have done some stupid error prior to domain creation step. To track down the issue, though of checking what getDatabaseDefaults actually doing. I tried to grep for the getDatabaseDefaults function call inside oracle home. But couldn’t find the source code for the python function. As a final resort, thought of...
Read more