TLS Certificate that lead to team wars

This happened some time around Apr’2019. This story has all features of a suspense movie with a chaotic first half, surprise intermission/interval and fast paced unraveling second half.
 
 
Our client has Oracle EBS/ERP and it is integrated with other applications using Oracle middleware (java based) in Azure cloud. ERP admins clone ERP test environment from prod once in a month. Post cloning they do bunch of config updates to test environment. They also do a service re-deployment in our middleware application from ERP. This service is like a gateway to ERP application.
 
Architecture diagram is as shown below. ERP application has two nodes and is behind loadbalancer (LB). SSL is offloaded at LB. While servicing requests, middleware contacts ERP application via LB. During deployment, ERP send executable jar file to middleware, middleware does lot of validations including validation of ERP endpoint which is exposed through LB (https). If all validations are passed, middleware deploys jar into its application server.
 
 
ERP team got usual cloning request and when they tried to deploy service they got unable to find valid certification path to requested target error. They have verified their checklist and didn’t find anything missing. They asked to verify from our end.
 
The error was very clear, application was not able to validate certification chain. We have verified our trust store for root CA certificate. It was there and also checked if there were any changes done to application during clone window. No changes were done. At this point, I was very confident that issue
didn’t lie in middleware. Same was communicated to ERP team.
 
This triggered chain of arguments and counter arguments. Both teams invented new weapons to prove their point. ERP lead advised me import complete certificate chain, ERP server certificate and intermediate certificate as well. As you know, server sends both server and intermediate certificates. I resisted that as adding root CA certificate would suffice for validating server certificate chain.
 
This discussion went on for a while. ERP lead called me next evening and again suggested to import complete certificate chain. Thought of giving it a try even though I felt it wouldn’t help. Our teammate imported certificate chain. BOOM !!! It worked and I was humbled.
 
I thought over it through the night but was not able to figure it out. We have this routine of discussing important issues and do a postmortem of major incidents. One of the engineers said something and it triggered chain of thoughts. I immediately opened LB url and inspected certificate chain in chrome, Firefox and IE. There lies the culprit !!! In one of the browsers (don’t remember which one), it showed only server certificate. This also verified using openssl.
 
That means LB was sending only server certificate without intermediate certificate. Looks like some of the browsers cache certificates for better performance and display the complete certificate chain even though website doesn’t send intermediate one. This somewhat caused misdirection of troubleshooting.
 
It was found that someone changed certificate configuration on LB during the cloning window. We contacted concerned team and they corrected it. We have removed intermediate and server certificate from trust store, it started working again without those certificates.
 
Lessons Learnt: First one, sometimes error will be so misleading and shouldn’t look down at options. Second one, conflict between teams may be triggered by unknown third party. So, involve as many as teams in the war ?.
 
Side Note – Our team used to import server certificate instead of root CA cert. That was good enough as trust chain completes with server certificate itself. But that was not best practice for two reasons. 1. We need to import too many server certificates. 2. Server certificates expire every two years. It is lot of maintenance and there is good chance of forgetting. So, we started importing only root CA.

Comments

Post a Comment

Popular posts from this blog

HOW WE REDUCED SOA OSB PROVISIONING FROM 4 DAYS TO 4 HOURS

 In our organization, we’ve undertaken the task of upgrading both SOA and OSB from version 12.2.1.3 to 12.2.1.4. This involves an out-of-place migration approach, wherein we create a new environment and switch traffic during the cutover. As you know, this process typically requires several days we opt for manual provisioing. A very significant amount of automation work was previously accomplished for 12.2.1.3 by our predecessors. Despite these efforts, the process still used to take lot of time due to following factors: Bugs in existing automation. For example, SOA composite deployments is not stable which warrants lot of manual work Import of 10s of TLS Certificates into KSS Manual retiring and shutting down of services Monitoring setup Miscellaneous: Increasing Log file count, enabling GC logging, changing heap parameters, bash profile changes While our current automation initiatives have yielded substantial savings, we aimed to capitalize on this upgrade activity to enable end-t...
Read more

NOT ABLE TO START RABBITMQ CLUSTER: CANNOT DECLARE A QUEUE ‘~S’ ON NODE ‘~S’: ~255P

We have 7 node rabbitmq cluster and we are using sharding queues. We have shutdown all non prod rabbitmq nodes as part of VM patching. As it was non prod, we thought of shutting down all together instead of rolling patching. We couldn’t start the any of the rabbitmq post patching. All of nodes were failing to start with the below error: 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> “Cannot declare a queue ‘~s’ on node ‘~s’: ~255p”, 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> [“queue ‘sharding: shard.my.queue – rabbit@rabbit1’ in vhost ‘my_vhost'” This is a known issue in some old versions of rabbitmq. rabbitmq node fails to come up with above error. Work around for this is to disable sharding policy for the problematic sharding queue. It can be done either from command line or from rabbitmq admin page. It requires to connect a running node. But as all the nodes were down, we could not disable sharding policy. Luckily, rabbitmq allows disabling sharding pulgi...
Read more

SOA SUITE 12.2.1.4 INSTALLATION: GOT EXCEPTION WHEN AUTO CONFIGURING THE SCHEMA COMPONENT(S) WITH DATA OBTAINED FROM SHADOW TABLE

  We planned to do out of place upgrade from soa suite 12.2.1.3 to 12.2.1.4. As part of this, we planned to automate end to end provisioning. While executing domain configuration using ansible and python scripts, we got following error in domain creation step. Got exception when auto configuring the schema component(s) with data obtained from shadow table. Failed to build JDBC Connection object: com.oracle.cie.domain.script.jython.CommandExceptionHandler. exception. Quick internet search gave few solutions like deleting domain-registry.xml file, verifying dehydration store password supplied to script etc. But none of them helped. After trying several options, I thought I might have done some stupid error prior to domain creation step. To track down the issue, though of checking what getDatabaseDefaults actually doing. I tried to grep for the getDatabaseDefaults function call inside oracle home. But couldn’t find the source code for the python function. As a final resort, thought of...
Read more