DevOps SRE Blog

We recently built a new Oracle SOA suite environment with version 12.2.1.3. A particular service which is exposed through Gateway was throwing 401 Unauthorized error. There were no instances for this service in em console. So, we initially thought that issue might be with Gateway and analysis was directed towards it. From Gateway logs we found that back end (SOA service) was throwing 401 error. When we checked soa access logs (usually located under server logs directory) we observed 401 errors for this service. But there were no instances in em console. When we tail the logs and hit the service multiple times from postman, we observed below error in logs. <Jan 26, 2023 8:32:05,927 PM PST> <Error> <oracle.wsm.resources.security> <WSM-00008> <Login Exception: [Security:090938]Authentication failure: The specified user failed to log in. javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User specified user denied.> <Jan...

Datasource monitoring is one of the key monitoring topics in weblogic based applications. We have been using   simple WLST script   to alert us when a datasource is not working properly. Most of the times, root cause will be a intermittent network/DB issue. To recover from the issue, we will reset/restart the datasource manually. Even though our team is very responsive, sometimes delays in response are inevitable. As one of the tenets of SRE is to automate as much as possible to reduce human toil and manual errors, we have done further automation. We have improved on the earlier version to recover datasource automatically. If the issue can’t be resolved with restart, script will send an email with error message and datasource details. This will save time in gathering error and db information. DBA team, which is part of alert targets, can also quickly act on the alert. New script is available at   this location .

We have faced weird issue while starting Weblogic managed server from admin console. We spent good amount of time in analyzing and fixing the issue. Sharing this if it can help others. We made a configuration change in weblogic domain that required to restart all servers including admin server. We did rolling restart of servers. All servers were started except one of the managed servers. While starting the server from console, we got the error: “ the Node Manager associated with machine SOAHOST1 is not reachable “ We logged in into manged server VM and noticed that NM was running fine. Somehow, admin server showing status as unreachable in admin console (Environment -> Machines -> Machine Name -> Monitoring -> Node Manager Status). To confirm that it was not network issue, we tested network connectivity using telnet ( telnet managedServerHost:5556 ). Connection was successful. We checked the Admin server log and found below error: <Nov 16, 2022 11:41:07,357 AM PST> ...

We had an issue with shared storage server which is used to store weblogic admin server configuration. So, we restarted the admin server once the storage issue was resolved. But restart failed with below errors: <Nov 13, 2022 10:06:15,079 PM PST> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: A MultiException has 20 exceptions. They are: 1. java.lang.NullPointerException 2. java.lang.IllegalStateException: Unable to perform operation: post construct on weblogic.store.admin.DefaultStoreService 3. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.transaction.internal.TransactionService errors were found 4. java.lang.IllegalStateException: Unable to perform operation: resolve on weblogic.transaction.internal.TransactionService 5. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.jdbc.common.internal.JDBCService errors were found 20. java.lang.Illeg...

We were trying to integrate Spring Cloud Gateway running on K8S with HashiCorp Vault. Wanted to share info how we resolved these issues. Issue 1 : [ERROR] auth.kubernetes.auth_kubernetes: login unauthorized due to: Post “https://10.0.0.:6443/apis/authentication.k8s.io/v1/tokenreviews”: x509: certificate signed by unknown authority Solution : We used following kubernetes auth config to authenticate client to vault: vault write auth/kubernetes/config token_reviewer_jwt=”$SA_JWT_TOKEN” kubernetes_host=”$K8S_HOST” kubernetes_ca_cert=”$SA_CA_CRT” issuer=”https://kubernetes.default.svc.cluster.local” disable_iss_validation=”true” We extracted certificate info using below command: export SA_CA_CRT=$(kubectl config view –raw –minify –flatten –output ‘jsonpath={.clusters[].cluster.certificate-authority-data}’ | base64 –decode) While copying the certificate info to vault container we used  echo $SA_CA_CRT  instead of  echo “$SA_CA_CRT” . Due to this new line characters from certifi...

  Our dev team has exposed a Boomi service through Kong Gateway to be consumed by other teams. The service was working for most of the time but throwing 502 errors randomly. Initially, we thought it was network glitch and ignored those errors. As the testing progressed to higher environments, the error rate increased. Though it was below 1 %, due to limitations at both client and upstream we were tasked to analyze and fix the issue. The high level request flow is like this:   Client -> Kong Gateway -> Boomi VIP -> Boomi Atoms -> Upstream/backend service. We started analysis with Kong and Boomi logs. Found below entries: Kong : {“date”:1.664432446599347E9,”log”:”2022-09-29T06:20:46.599260909Z stderr F 2022/09/29 06:20:46 [error] 38#0: *1693441  upstream prematurely closed connection  while reading response header from upstream, client: 10.200.63.6, server: kong, request: \”POST /test/outbound/consumer HTTP/1.0\”, upstream: Boomi : 2022_09_28.shared_http_se...

  We were trying to integrate Spring Cloud Gateway (SCG) on Kubernetes with HashiCorp. We followed the steps mentioned in vault   documentation . We were able to bring up vault and vault injector but SCG pods were stuck in init state. Found following error in SCG application pod vault-agent-init container logs: NAME READY STATUS RESTARTS AGE scg-0 0/2 Init:0/1 0 9h vault-0 1/1 Running 0 10h vault-agent-injector-5c89c7dfc5-n2v6v 1/1 Running 0 20h Cgo: disabled Log Level: info Version: Vault v1.8.4 Version Sha: 925bc650ad1d997e84fbb832f302a6bfe0105bbb 2022-09-30T16:24:28.007Z [INFO] sink.server: starting sink server 2022-09-30T16:24:28.007Z INFO creating watcher 2022-09-30T16:25:28.008Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1s To verify that vault injector was connecting to right vault instance, we verified init agent config using below kubectl command: kubectl exec -it scg-0 -c va...