CyberArk secret provider for k8s not updating the secret

Secrets Management - Conjur, Secrets Hub & CP Conjur Enterprise
,
1

We are in the process of testing integration between CyberArk Conjur enterprise 12.4 and OpenShift 4.8. We are running into an odd issue with Secrets Provider for Kubernetes > Init container > Kubernetes Secrets mode.
The init container finishes executing successfully with logs indicating that the secrets were retrieved from Conjur and K8s secret was updated. But the actual k8s secret was not updated. Based on the conjur logs, we confirmed that the init container was able to fetch the secrets.

Did anyone experience this issue before? Please suggest any tips that could help troubleshoot this.

Init Container Log

DEBUG: 2022/03/01 19:39:27.901561 authenticator.go:258: CAKC044 Buffer time:  30s
DEBUG: 2022/03/01 19:39:27.901835 requests.go:46: CAKC046 Authn request to: https://t-cdap-follower-svc.test.acme.net/authn-k8s/test-cluster/Tst1/host%2Ftest-app-csp/authenticate
DEBUG: 2022/03/01 19:39:27.901862 authenticator.go:318: CAKC069 Sending authn-k8s request...
INFO:  2022/03/01 19:39:28.148447 authenticator.go:116: CAKC035 Successfully authenticated
INFO:  2022/03/01 19:39:28.148525 conjur_secrets_retriever.go:71: CSPFK003I Retrieving following secrets from DAP/Conjur: [Tst1:variable:ConjurSync/TestApp/APS_0_TST_WAU_SL_ACME/Operating System-Win_Dom_Acc_Test-1.1.1.1-testusername/password Tst1:variable:ConjurSync/TestApp/APS_0_TST_WAU_SL_ACME/Operating System-Win_Dom_Acc_Test-1.1.1.1-testusername/username]
INFO:  2022/03/01 19:39:28.148537 conjur_client.go:21: CSPFK002I Creating DAP/Conjur client
INFO:  2022/03/01 19:39:28.185514 provide_conjur_secrets.go:153: CSPFK009I DAP/Conjur Secrets updated in Kubernetes successfully

k8s secret

---
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: test-app-namespace
type: Opaque
stringData:
  conjur-map: |-
    DB_USERNAME: Tst1:variable:ConjurSync/TestApp/APS_0_TST_WAU_SL_ACME/Operating System-Win_Dom_Acc_Test-1.1.1.1-testusername/username
    DB_PASSWORD: Tst1:variable:ConjurSync/TestApp/APS_0_TST_WAU_SL_ACME/Operating System-Win_Dom_Acc_Test-1.1.1.1-testusername/password

Init Container Env

  - image: image-registry.openshift-image-registry.svc:5000/test-app-namespace/secrets-provider-for-k8s:1.4
    imagePullPolicy: Always
    name: cyberark-secrets-provider-for-k8s
    env:
      - name: CONJUR_AUTHN_LOGIN
        value: host/test-app-csp
      - name: CONTAINER_MODE
        value: init
      - name: DEBUG
        value: "true"
      - name: SECRETS_DESTINATION
        value: k8s_secrets
      - name: K8S_SECRETS
        value: db-credentials
      - name: MY_POD_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.name
      - name: MY_POD_NAMESPACE
        valueFrom:
          fieldRef:
            fieldPath: metadata.namespace
    envFrom:
      - configMapRef:
          name: conjur-connect
2

Hi @supp123,

I can’t say that I’ve heard of this before. When you say the K8s secret is not updated, do you the value is blank, or the value is not changing to a new password value after a password change?

Nate

3

Hello @nathan.whipple Thank you for responding! Based on the status of the K8s secret, there were absolutely no updates to it. I am assuming that the init container would update my original k8s secret as shown below based on the values from Conjur. Did I get it wrong?

---
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: test-app-namespace
type: Opaque
data:
  DB_USERNAME: JDJKDKEKDID=
  DB_PASSWORD: SHSJENDKEKL= 
stringData:
  conjur-map: |-
    DB_USERNAME: Tst1:variable:ConjurSync/TestApp/APS_0_TST_WAU_SL_ACME/Operating System-Win_Dom_Acc_Test-1.1.1.1-testusername/username
    DB_PASSWORD: Tst1:variable:ConjurSync/TestApp/APS_0_TST_WAU_SL_ACME/Operating System-Win_Dom_Acc_Test-1.1.1.1-testusername/password
4

@supp123, I think the issue here is that you used the same key name twice in that manifest. Once in the data: block and again in the stringData: block. In the sample from the documentation the key name in the data: block is different than the key names in the stringData: block.

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  DBName:   bXlhcHBEQg==
stringData:
  conjur-map: |-   
    username: secrets/username
    password: secrets/password

According the Kubernetes documentation around using stringData:

When you create these types of Secrets using a manifest, the API server checks whether the expected key does exists in the data field,

My expectation here is that the manifest sets the key:value pair based on the data: block then when it attempts to set the same key:value pair for the stringData: block it attempts to verify that the key already exists, sees that it does and does not update the value. I hope that helps!

Nate

5

We worked with CyberArk engineering team to resolve the issue. It appears that the k8s secrets provider and push to file modes does not handle the Tst1:variable: part of our secret references. After we removed it, there are no issues using both k8s secret provider and push to file mode.

Thank you @nathan.whipple for your inputs.

Satheesh

1 Like
6

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.