DevOps SRE Blog

  We planned to do out of place upgrade from soa suite 12.2.1.3 to 12.2.1.4. As part of this, we planned to automate end to end provisioning. While executing domain configuration using ansible and python scripts, we got following error in domain creation step. Got exception when auto configuring the schema component(s) with data obtained from shadow table. Failed to build JDBC Connection object: com.oracle.cie.domain.script.jython.CommandExceptionHandler. exception. Quick internet search gave few solutions like deleting domain-registry.xml file, verifying dehydration store password supplied to script etc. But none of them helped. After trying several options, I thought I might have done some stupid error prior to domain creation step. To track down the issue, though of checking what getDatabaseDefaults actually doing. I tried to grep for the getDatabaseDefaults function call inside oracle home. But couldn’t find the source code for the python function. As a final resort, thought of...

We have a 7 node test rabbitmq cluster. One of the rabbitmq users reported that they were getting “ javax.net.ssl.SSLHandshakeException: Invalid ECDH ServerKeyExchange signature ” error while connecting to Rabbitmq. We checked logs and also checked application up time. Rabbitmq servers hadn’t been restarted for the past 25 days. Also, we didn’t observe any errors in logs related to vhost used by client. We also checked if the certificate is expired and it was not. We got reasonably confident that it was not our issue and asked to check from client side. As day progressed, we got complaints from two other client application teams. One of them uses php and other uses java. At this point, we started suspecting rabbitmq. Also, one team with multiple consumers told us that they are facing issue only with few of the consumers. With this information, we felt that issue could be with few of the nodes and not complete cluster. One of the blogs on the internet suggested that this error could occ...

We have 7 node rabbitmq cluster and we are using sharding queues. We have shutdown all non prod rabbitmq nodes as part of VM patching. As it was non prod, we thought of shutting down all together instead of rolling patching. We couldn’t start the any of the rabbitmq post patching. All of nodes were failing to start with the below error: 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> “Cannot declare a queue ‘~s’ on node ‘~s’: ~255p”, 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> [“queue ‘sharding: shard.my.queue – rabbit@rabbit1’ in vhost ‘my_vhost'” This is a known issue in some old versions of rabbitmq. rabbitmq node fails to come up with above error. Work around for this is to disable sharding policy for the problematic sharding queue. It can be done either from command line or from rabbitmq admin page. It requires to connect a running node. But as all the nodes were down, we could not disable sharding policy. Luckily, rabbitmq allows disabling sharding pulgi...

We use VMWare Wavefront for monitoring and visualization. We use telegraf agent as metrics collector. As part of Kafka monitoring, we have created alerts for Zookeeper availability. It was working fine initially but stopped working after a Kafka upgrade. We checked telegraf agent and zookeeper logs but could not find anything suspicious. We checked telegraf zookeeper github repository. It mentioned that it uses zookeeper mntr command to get monitoring data. We hadn’t whitelisted any zookeeper 4lw commands in earlier versions of Kafka too. It turned out that from Zookeeper version 3.5.3 onwards, we need to explicitly whilelist commands. As mntr is disabled by default, telegraf was not able to collect metrics. Issue got resolved after whitelisting mntr command in zookeeper.properties file and restart of zookeeper.

  In the   previous blog post , we discussed about how to monitor weblogic based applications including SOA and OSB using Vmware Wavefront. In this blog post, let us explore open source alternative with Prometheus and Grafana. We will use Oracle   Weblogic Monitoring Exporter   in place of jolokia agent to export metric to Prometheus and Grafana for visualizing metrics. High level steps are: 1. Download and install Weblogic Monitoring Exporter. 2. Install Prometheus & Grafana 3. Configure Prometheus to scrape metrics from wls-exporter 4. Configure Grafana dashboards using Prometheus datasource. Below are detailed steps: Install Weblogic Monitoring Exporter : Go to  Weblogic Monitoring Exporter github releases  page and download latest  get_v<version>.sh  script. Copy the exporter configuration file  located here . and pass that as parameter (e.g., ./get_v2.1.2.sh exporter_config.yaml) to get_v<version>.sh. This script download...

  Traditionally sysadmins used bash, WLST/Jython scripts to monitor weblogic based applications including Oracle SOA Suite and Oracle Service Bus. There are few disadvantages with this approach: Need to maintain multiple scripts to monitor a single domain. Sysadmins needs to be proficient in 1 or 2 scripting language. Developing script may take few hours to few days. Will not have access to historical monitoring data for doing trend analysis. With the advent of modern monitoring tools like Prometheus and Grafana, above challenges are addressed. We can setup beautiful graphs and dashboards quickly and tools like Grafana also support alerting mechanism. Wavefront is an integrated solution that can store time series metrics, supports great visualizations, alerting, tracing and more. This blog post describes steps to integrate Weblogic monitoring with Wavefront. Weblogic doesn’t expose time series metrics data directly. So, we need to install Jolokia on Weblogic for exposing JMX metric...

We recently built a new Oracle SOA suite environment with version 12.2.1.3. A particular service which is exposed through Gateway was throwing 401 Unauthorized error. There were no instances for this service in em console. So, we initially thought that issue might be with Gateway and analysis was directed towards it. From Gateway logs we found that back end (SOA service) was throwing 401 error. When we checked soa access logs (usually located under server logs directory) we observed 401 errors for this service. But there were no instances in em console. When we tail the logs and hit the service multiple times from postman, we observed below error in logs. <Jan 26, 2023 8:32:05,927 PM PST> <Error> <oracle.wsm.resources.security> <WSM-00008> <Login Exception: [Security:090938]Authentication failure: The specified user failed to log in. javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User specified user denied.> <Jan...