DevOps SRE Blog

We had a Kubernetes POC cluster with version 1.18. This cluster got corrupted during experimentation by our team. Instead of starting with clean slate, which is comparatively easy, we tried to rebuild cluster with existing kubelet, etcd, kubeadm. This post is basically my own reference/notes of the issues faced and their fixes. Issue 1 :  kubeadm join  failing with below error. This happens only while join control plane nodes not worker nodes. failure loading certificate for CA: couldn’t load the certificate file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory Solution : Copied following files from node 1 to other control plane nodes /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/ca.key /etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.pub /etc/kubernetes/pki/front-proxy-ca.crt /etc/kubernetes/pki/front-proxy-ca.key Issue 2 : Flannel pods are crashlooping. Found following error continuously in kube-proxy logs.node.go:125] Failed to retrie...

  As part of regular maintenance, we shutdown Kafka. When we tried to start it, it failed with below error: [2021-11-15 05:01:03,824] INFO Socket connection established to mykafka-a1.abc.com/10.0.95.100:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2021-11-15 05:01:03,824] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [2021-11-15 05:01:03,889] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient) [2021-11-15 05:01:03,927] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper) [2021-11-15 05:01:03,927] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn) [2021-11-15 05:01:03,930] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient) [2021-11-15 05:01:03,933] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) kafka.zooke...

We as sysadmin spend lot of time at command line and most of the time we use few regular commands like going to logs directory, searching history. I have prepared bash aliases that can be used to save few keystrokes for various applications like weblogic, kubernetes, Boomi etc. And these can be downloaded from   github . Please feel free to use and add your own commands/scripts.

  We have a 3 node Boomi cluster that we were not able to start. When we start it and check status status is shown as ‘ Atom is Running’ but after some time atom stops. When we checked container logs we found below error: Nov 11, 2021 4:15:48 AM PST FINE [com.boomi.container.plugin.BasePluginManager createPluginClassLoader] Creating QUEUE_SERVER Plugin ClassLoader with overrides: [] Nov 11, 2021 4:15:48 AM PST SEVERE [com.boomi.container.core.BaseContainer start] Atom startup failed, shutting down Nov 11, 2021 4:15:48 AM PST INFO [com.boomi.util.LogUtil doLog] Atom stop requested. Current status is INITIALIZING Nov 11, 2021 4:15:48 AM PST INFO [com.boomi.container.config.ContainerConfig setStatus] Container status changed from INITIALIZING to STOPPING: Atom startup failed, shutting down Nov 11, 2021 4:15:48 AM PST FINE [com.boomi.schedule.Scheduler removeScheduleGroup] Removing schedule for groupId: internal-dead-attachment-group Nov 11, 2021 4:15:48 AM PST INFO [com.boomi.containe...

There are multiple ways to abort recoverable instances in SOA suites: From em console Updating dehydration tables Programmatically First option, em console, is useful when there are few instances to be aborted. Though we can update dehydration tables to change instance status, it is not a Oracle recommended approach. I found main tables that are updated when will abort instances by tracing using JMC. But this list is not exhaustive. I am providing sql statements only for reference. Don’t execute these statement unless you understand implications. Aborting instances through SOA management API is a cleaner and quick approach. Please refer to this  git repo  for the script based approach. DELETE FROM MEDIATOR_CALLBACK WHERE FLOW_ID = 27594227; DELETE FROM MEDIATOR_CORRELATION WHERE FLOW_ID = 27594227; UPDATE CUBE_INSTANCE SET SCOPE_REVISION = SCOPE_REVISION + 1 WHERE FLOW_ID = 27594227; UPDATE SCA_COMMON_FAULT SET STATE=256 WHERE FLOW_ID = 27594227 AND STATE <> 2816 AND ST...

We have a two node Kubernetes cluster with one master and one worker node. Kong api is installed as containers on this cluster. Log files in our Kong API server were located in root directory of the host machine. As the number of logs were getting increased, there was a danger of filling up root directory. So, we decided to move it to separate mount keeping directory path same. Our plan was as below: 1. Cordon and drain the node. 2. Stop kubelet and docker. 3. Move logs to separate mount. 4. Start docker and kubelet. 5. Uncordon the node. We followed the plan but Kong pod was stuck in Init state. We have verified Kong access, error logs, Init (wait-for-db container) container logs and kubelet logs (/var/log/messages in our case). There was nothing suspicious written. We also checked status of all kube-system pods and all were running fine. We verified Control Plane logs as well. We tried restarting docker, kubelet and Cassandra pods (Kong’s persistent store). But nothing worked. Sooner...

  As sysadmins, we need to daily deal with logs. It is overwhelming to check huge log files especially for application servers like weblogic which has numerous log files like diagnostic, server log, out log, OHS logs etc. It is time consuming and quickly lead to fatigue if log files are huge like weblogic. I have developed 2 scripts which extract and summarize logs. I have been using these scripts in both production and test environments. These scripts helped in quickly reacting to a situation and saved time. If your organization already has tools like Splunk, ELK, these scripts may not much use. But for those who couldn’t afford, these scripts could help. Please find brief description about each script as below: Log Extractor : This script takes time in mins or time period as input and extracts logs produced between that time period. We can also filter log files based on log severity as shown below screenshot. Script Location:  https://github.com/RameshPoonati/oracleMiddlewar...