DevOps SRE Blog

When we get errors related to datasource in weblogic, first thing we do is go to datasource Monitoring -> Testing tab and test datasource. Sometimes, targets might be missing from this tab. During server startup, server tries to load datasource and create connections to database. If there is any underlying issue with database like password change, password expiry, unavailability of database, it doesn’t load the datasource and we see missing targets in testing tab. Some of the errors, we can see are: ####<Apr 21, 2022 3:44:12,251 AM PDT> <Info> <JDBC> <orasoa-test12-w2> <WLS_SOA2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <d47e58ce-a023-4de2-acb7-24391c8056df-0000000a> <1650537852251> <[severity-value: 64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-001508> <Destroying data source connection pool TestDS.> ####<A...

  We wanted to monitor Rabbitmq through Wavefront. So, we followed below steps to configure the integration: Install telegraf agent. Enable Rabbitmq management plugin. Configure telegraf agent to use Rabbitmq input plugin. Restart telegraf Even after completing above steps we were not able to see complete data in Wavefront dashboards. We were able to see basic node level data but not able to see queue level data. We checked /var/log/messages where telegraf logs are written and found following messages. Apr 10 03:48:36 test-app1 telegraf: 2022-04-10T10:48:36Z E! [inputs.rabbitmq] Error in plugin: getting “/api/overview” failed: 401 Unauthorized Apr 10 03:48:36 test-app1 telegraf: 2022-04-10T10:48:36Z E! [inputs.rabbitmq] Error in plugin: getting “/api/nodes” failed: 401 Unauthorized Apr 10 03:48:36 test-app1 telegraf: 2022-04-10T10:48:36Z E! [inputs.rabbitmq] Error in plugin: getting “/api/exchanges” failed: 401 Unauthorized Apr 10 03:48:36 test-app1 telegraf: 2022-04-10T10:48:36Z E...

Recently, we have setup monitoring of Kubernetes using wavefront. Here are some of the useful alerts and their queries: POD Memory Utilization: ts(“kubernetes.pod.memory.working_set”, namespace_name=”xyz”)/ts(“kubernetes.pod.memory.limit”, namespace_name=”xyz”) * 100 Kong POD CPU Utilization: ts(“kubernetes.pod.cpu.usage_rate”, namespace_name=”xyz”)/ts(“kubernetes.pod.cpu.limit”, namespace_name=”xyz”) * 100 Kong Replica Count Mismatch: ts(“kubernetes.deployment.desired_replicas”, namespace_name=”xyz”) – ts(“kubernetes.deployment.available_replicas” and namespace_name=”xyz”) New Pod Created/Pod Deleted: highpass(0, ts(“kubernetes.pod.uptime”, namespace_name=”xyz”) < 630000) Container Restart: mdiff(10m, ts(“kubernetes.pod.restart_count”, namespace_name=”xyz”))

  I was writing small bash script to take backup of a application configuration. It was tested from command line and everything looks fine. Later, I had setup a cronjob to run it at a scheduled time. But there was no backup. One of the common reasons for this is when we use relative paths in the script. crontab default working directory is user’s home directory. If scripts have relative paths, it will not be able to find right files and directories. I checked if there were any such relative paths but nothing was found. To troubleshoot the issue, placed echo almost after every single statement. After analyzing these echo outputs, I suspected that crontab is not able to find one of the commands used in the script. But this utility was already installed and got confirmation by using “which” command. That means that PATH variable used by crontab is different from regular login shell. Got it confirmed by printing PATH variable. To fix the issue, modified PATH variable in the script to i...

We recently updated Boomi Java version to 11.0.14 through Boomi UI. When we were trying to restart atoms, one of the atoms refused to start. Unfortunately, no error message was written to logs. Luckily, there is a run option for atom script which usually throws error when there is an issue. I usually use this option for diagnosing startup issues. When executed ./atom run from command line, got below message: Error: Password file not found: $molecule_home/jre/lib/management/jmxremote.password Apparently, we forgot to backup and copy jmxremote.password file even though it was there in Boomi documentation. We copied jmxremote.password from earlier backups and able to start atom.

  There could be multiple reasons for higher utilization of disk space in Boomi. Some of the reasons are too many files in following directories: logs – Stores atom logs execution – Logs of each process execution data – Data processed by atom processes processes – Deployed processes Some time ago, we received higher disk usage alert. We cleared some disk space by deleting files from logs and execution directories. Issue reoccurred after couple of days. We concluded that it was not regular disk fill-up and wanted to investigate further. We logged into terminal and issued following command from boomi molecule home directory to find out which directory was occupying more space. Found that execution directory was occupying more space. du -sh * | sort -rh Next, we wanted to check from which date this started occurring by using below command: cd $molecule_home/execution find . | awk -F'/' ' { print $2 } ' | uniq -c From output of the above command, we confirmed that issue wa...

We were updating Java version using Atmosphere UI and tried to restart the atoms from terminal. All atoms were successfully restarted except one atom. We were getting timeout error while running ./atom stop command. We could see following error message on cluster status page of atmosphere UI. EW_ID_MISMATCH There are differing view IDs in the various view snapshot files. Severe These nodes are reporting the issue: Node 1 10_x_x_x Found different view id [10_x_x_x|90] in "node.10_x_x_x.dat" Node 2 10_x_x_x Found different view id [10_x_x_x|101] in "node.10_x_x_x.dat" We analyzed atom logs but could not find the reason for timeout. We killed the process and removed above dat files from $MOLECULE_HOME/bin/views directory. But we got again same problem, we couldn’t stop the atom. So, to troubleshoot further, captured system calls using strace by attaching to running process. Following lines were repeating in the trace log. open(" /tmp/i4jdaemon__data_boomi_Boomi_At...