DevOps SRE Blog

We have faced weird issue while starting Weblogic managed server from admin console. We spent good amount of time in analyzing and fixing the issue. Sharing this if it can help others. We made a configuration change in weblogic domain that required to restart all servers including admin server. We did rolling restart of servers. All servers were started except one of the managed servers. While starting the server from console, we got the error: “ the Node Manager associated with machine SOAHOST1 is not reachable “ We logged in into manged server VM and noticed that NM was running fine. Somehow, admin server showing status as unreachable in admin console (Environment -> Machines -> Machine Name -> Monitoring -> Node Manager Status). To confirm that it was not network issue, we tested network connectivity using telnet ( telnet managedServerHost:5556 ). Connection was successful. We checked the Admin server log and found below error: <Nov 16, 2022 11:41:07,357 AM PST> ...

We had an issue with shared storage server which is used to store weblogic admin server configuration. So, we restarted the admin server once the storage issue was resolved. But restart failed with below errors: <Nov 13, 2022 10:06:15,079 PM PST> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: A MultiException has 20 exceptions. They are: 1. java.lang.NullPointerException 2. java.lang.IllegalStateException: Unable to perform operation: post construct on weblogic.store.admin.DefaultStoreService 3. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.transaction.internal.TransactionService errors were found 4. java.lang.IllegalStateException: Unable to perform operation: resolve on weblogic.transaction.internal.TransactionService 5. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.jdbc.common.internal.JDBCService errors were found 20. java.lang.Illeg...

We were trying to integrate Spring Cloud Gateway running on K8S with HashiCorp Vault. Wanted to share info how we resolved these issues. Issue 1 : [ERROR] auth.kubernetes.auth_kubernetes: login unauthorized due to: Post “https://10.0.0.:6443/apis/authentication.k8s.io/v1/tokenreviews”: x509: certificate signed by unknown authority Solution : We used following kubernetes auth config to authenticate client to vault: vault write auth/kubernetes/config token_reviewer_jwt=”$SA_JWT_TOKEN” kubernetes_host=”$K8S_HOST” kubernetes_ca_cert=”$SA_CA_CRT” issuer=”https://kubernetes.default.svc.cluster.local” disable_iss_validation=”true” We extracted certificate info using below command: export SA_CA_CRT=$(kubectl config view –raw –minify –flatten –output ‘jsonpath={.clusters[].cluster.certificate-authority-data}’ | base64 –decode) While copying the certificate info to vault container we used  echo $SA_CA_CRT  instead of  echo “$SA_CA_CRT” . Due to this new line characters from certifi...

  Our dev team has exposed a Boomi service through Kong Gateway to be consumed by other teams. The service was working for most of the time but throwing 502 errors randomly. Initially, we thought it was network glitch and ignored those errors. As the testing progressed to higher environments, the error rate increased. Though it was below 1 %, due to limitations at both client and upstream we were tasked to analyze and fix the issue. The high level request flow is like this:   Client -> Kong Gateway -> Boomi VIP -> Boomi Atoms -> Upstream/backend service. We started analysis with Kong and Boomi logs. Found below entries: Kong : {“date”:1.664432446599347E9,”log”:”2022-09-29T06:20:46.599260909Z stderr F 2022/09/29 06:20:46 [error] 38#0: *1693441  upstream prematurely closed connection  while reading response header from upstream, client: 10.200.63.6, server: kong, request: \”POST /test/outbound/consumer HTTP/1.0\”, upstream: Boomi : 2022_09_28.shared_http_se...

  We were trying to integrate Spring Cloud Gateway (SCG) on Kubernetes with HashiCorp. We followed the steps mentioned in vault   documentation . We were able to bring up vault and vault injector but SCG pods were stuck in init state. Found following error in SCG application pod vault-agent-init container logs: NAME READY STATUS RESTARTS AGE scg-0 0/2 Init:0/1 0 9h vault-0 1/1 Running 0 10h vault-agent-injector-5c89c7dfc5-n2v6v 1/1 Running 0 20h Cgo: disabled Log Level: info Version: Vault v1.8.4 Version Sha: 925bc650ad1d997e84fbb832f302a6bfe0105bbb 2022-09-30T16:24:28.007Z [INFO] sink.server: starting sink server 2022-09-30T16:24:28.007Z INFO creating watcher 2022-09-30T16:25:28.008Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1s To verify that vault injector was connecting to right vault instance, we verified init agent config using below kubectl command: kubectl exec -it scg-0 -c va...

  This blog is about connection reset error we received while connecting from Boomi atom to remote sftp. This issue can occur for any client connecting to a remote sftp server. The sftp server we were trying to connect was old sftp server and it has been used in many integrations. We haven’t faced reset issue for other integrations flow except this new one. So, we first suspected that there was something wrong with the process itself. We looked closely at the process. The process has multiple processes – one main process and multiple child processes. Main process checks for availability of files and child processes transfer files to Boomi atoms for further processing. The first main process and few child processes succeed always but last few process would fail always (usually after processing 7 or 8 files). We tweaked the process to drill down the issue. Here are few thing we tried: Disabled all other processes that connect to the same sftp server. Issue still occurred. We checked ...

There are couple of ways to monitor Kafka consumer Lag. There is an open source project called   Apache Burrow   which has lot of features and easy to set up. It filters out lot of false positives too. If there are an limitations of using it in your project, we can write bash scripts using Kafka consumer utility which ships with Kafka. We have written one such script. If you are interested,   please check it out here .