SCG AND VAULT INTEGRATION: LOGIN UNAUTHORIZED DUE TO: X509: CERTIFICATE SIGNED BY UNKNOWN AUTHORITY

We were trying to integrate Spring Cloud Gateway running on K8S with HashiCorp Vault. Wanted to share info how we resolved these issues.

Issue 1: [ERROR] auth.kubernetes.auth_kubernetes: login unauthorized due to: Post “https://10.0.0.:6443/apis/authentication.k8s.io/v1/tokenreviews”: x509: certificate signed by unknown authority

Solution: We used following kubernetes auth config to authenticate client to vault:
vault write auth/kubernetes/config token_reviewer_jwt=”$SA_JWT_TOKEN” kubernetes_host=”$K8S_HOST” kubernetes_ca_cert=”$SA_CA_CRT” issuer=”https://kubernetes.default.svc.cluster.local” disable_iss_validation=”true”

We extracted certificate info using below command:
export SA_CA_CRT=$(kubectl config view –raw –minify –flatten –output ‘jsonpath={.clusters[].cluster.certificate-authority-data}’ | base64 –decode)

While copying the certificate info to vault container we used echo $SA_CA_CRT instead of echo “$SA_CA_CRT”. Due to this new line characters from certificate were removed due to which certificate format became non compliant with pem format. After extracting certificate info using double quote, SCG was able to authenticate with Vault.

Issue 2: Authentication is failing with: {“kind”:”Status”,”apiVersion”:”v1″,”metadata”:{},”status”:”Failure”,”message”:”tokenreviews.authentication.k8s.io is forbidden: User \”system:serviceaccount:dev-scg:service-account-scg\” cannot create resource \”tokenreviews\” in API group \”authentication.k8s.io\” at the cluster scope”,”reason”:”Forbidden”,”details”:{“group”:”authentication.k8s.io”,”kind”:”tokenreviews”},”code”:403}

Solution: As per the above error message, service account doesn’t have right cluster role to create resource tokenreviews. Issue got resolved after creating cluster role binding as below:

  apiVersion: rbac.authorization.k8s.io/v1beta1
  kind: ClusterRoleBinding
  metadata:
    name: dev-tokenreview-binding
    namespace: dev-scg
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: system:auth-delegator
  subjects:
  - kind: ServiceAccount
    name: dev-scg-user
    namespace: dev-scg

Issue 3: SCG container’s state was not changing to ready state and was getting restarted repeatedly.

Solution: We didn’t observe much activity in gateway logs and after writing below few lines, SCG container was getting restarted.

2022-11-07 15:10:07.106 INFO 1 — [ main] trationDelegate$BeanPostProcessorChecker : Bean ‘reactorDeferringLoadBalancerExchangeFilterFunction’ of type [org.springframework.cloud.client.loadbalancer.reactive.DeferringLoadBalancerExchangeFilterFunction] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2022-11-07 15:10:30.516 INFO 1 — [ main] i.p.s.c.g.session.SessionConfiguration : hazelcastServiceName = jwt-gateway-headless

We took thread dump of the SCG and analyzed. Classloader thread had spent more than 90 seconds and still could not load all the classes. Meanwhile, readiness probe had been killing the container before it was fully started. All this happened due to very low cpu limits applied on the pod. SCG container got into ready state after increasing cpu limits.

Comments

Post a Comment

Popular posts from this blog

SOA SUITE 12.2.1.4 INSTALLATION: GOT EXCEPTION WHEN AUTO CONFIGURING THE SCHEMA COMPONENT(S) WITH DATA OBTAINED FROM SHADOW TABLE

  We planned to do out of place upgrade from soa suite 12.2.1.3 to 12.2.1.4. As part of this, we planned to automate end to end provisioning. While executing domain configuration using ansible and python scripts, we got following error in domain creation step. Got exception when auto configuring the schema component(s) with data obtained from shadow table. Failed to build JDBC Connection object: com.oracle.cie.domain.script.jython.CommandExceptionHandler. exception. Quick internet search gave few solutions like deleting domain-registry.xml file, verifying dehydration store password supplied to script etc. But none of them helped. After trying several options, I thought I might have done some stupid error prior to domain creation step. To track down the issue, though of checking what getDatabaseDefaults actually doing. I tried to grep for the getDatabaseDefaults function call inside oracle home. But couldn’t find the source code for the python function. As a final resort, thought of...
Read more

HOW WE REDUCED SOA OSB PROVISIONING FROM 4 DAYS TO 4 HOURS

 In our organization, we’ve undertaken the task of upgrading both SOA and OSB from version 12.2.1.3 to 12.2.1.4. This involves an out-of-place migration approach, wherein we create a new environment and switch traffic during the cutover. As you know, this process typically requires several days we opt for manual provisioing. A very significant amount of automation work was previously accomplished for 12.2.1.3 by our predecessors. Despite these efforts, the process still used to take lot of time due to following factors: Bugs in existing automation. For example, SOA composite deployments is not stable which warrants lot of manual work Import of 10s of TLS Certificates into KSS Manual retiring and shutting down of services Monitoring setup Miscellaneous: Increasing Log file count, enabling GC logging, changing heap parameters, bash profile changes While our current automation initiatives have yielded substantial savings, we aimed to capitalize on this upgrade activity to enable end-t...
Read more

RABBITMQ CONNECTION ERROR: JAVAX.NET.SSL.SSLHANDSHAKEEXCEPTION: INVALID ECDH SERVERKEYEXCHANGE SIGNATURE

We have a 7 node test rabbitmq cluster. One of the rabbitmq users reported that they were getting “ javax.net.ssl.SSLHandshakeException: Invalid ECDH ServerKeyExchange signature ” error while connecting to Rabbitmq. We checked logs and also checked application up time. Rabbitmq servers hadn’t been restarted for the past 25 days. Also, we didn’t observe any errors in logs related to vhost used by client. We also checked if the certificate is expired and it was not. We got reasonably confident that it was not our issue and asked to check from client side. As day progressed, we got complaints from two other client application teams. One of them uses php and other uses java. At this point, we started suspecting rabbitmq. Also, one team with multiple consumers told us that they are facing issue only with few of the consumers. With this information, we felt that issue could be with few of the nodes and not complete cluster. One of the blogs on the internet suggested that this error could occ...
Read more