Kubernetes CSI Provider JWT Authentication Issues

Secrets Management - Conjur, Secrets Hub & CP
,
1

I’ve installed conjur via helm:

export DATA_KEY=<data> && export VERSION="2.0.7" \
helm install \
   --set dataKey="$DATA_KEY" \
   --set account.create=true \
   --set account.name=dev-cluster \
   --set postgres.persistentVolume.create=false \
   --set image.repository=csm.artifactory.cec.lab.emc.com/csm-users/tyea1/conjur \
   --set image.tag=latest \
   --set nginx.image.repository=csm.artifactory.cec.lab.emc.com/csm-users/tyea1/nginx \
   --set nginx.image.tag=1.15 \
   --set postgres.image.repository=csm.artifactory.cec.lab.emc.com/csm-users/tyea1/postgres \
   --set postgres.image.tag=15.4 \
   conjur \
   -f conjur-values.yaml \
   https://github.com/cyberark/conjur-oss-helm-chart/releases/download/v$VERSION/conjur-oss-$VERSION.tgz

with values like this:

authenticators="authn,authn-jwt/dev-cluster"

I initialize conjur-cli:

conjur init --self-signed -a dev-cluster -u https://conjur-conjur-oss.default.svc.cluster.local:<port>

I setup authn-jwt:

- !policy
  id: authn-jwt/dev-cluster
  body:
    - !webservice

    - !variable issuer
    - !variable audience
    - !variable token-app-property
    - !variable public-keys
    - !variable jwks-uri
    - !variable identity-path

    - !group apps

    - !permit
      role: !group apps
      privilege: [ read, authenticate ]
      resource: !webservice

I configure the above variables:

conjur variable set -i authn-jwt/dev-cluster/jwks-uri -v "https://<master-node-ip>:6443/openid/v1/jwks"

kubectl get --raw /openid/v1/jwks > jwks.json
conjur variable set -i authn-jwt/dev-cluster/public-keys -v "{\"type\":\"jwks\", \"value\":$(cat jwks.json | jq -c .)}"

conjur variable set -i authn-jwt/dev-cluster/issuer -v "https://kubernetes.default.svc.cluster.local"

conjur variable set -i authn-jwt/dev-cluster/token-app-property -v "sub"

conjur variable set -i authn-jwt/dev-cluster/identity-path -v csm-authorization

conjur variable set -i authn-jwt/dev-cluster/audience -v "conjur"

I load the app policy:

- !policy
  id: csm-authorization
  body:
    - !host
      id: system:serviceaccount:authorization:storage-service
      annotations:
        authn-jwt/dev-cluster/kubernetes.io/namespace: "authorization"
        authn-jwt/dev-cluster/kubernetes.io/serviceaccount/name: "storage-service"

    - !group apps

    - !grant
      roles:
        - !group apps
      members:
        - !host system:serviceaccount:authorization:storage-service

I load secret variables:

- !policy
  id: secrets
  body:
    - !group apps
    - &variables
      - !variable username
      - !variable password
    - !permit
      role: !group apps
      privilege: [ read ]
      resource: *variables

I install csi-secrets-store:

helm install csi-secrets-store \
  secrets-store-csi-driver/secrets-store-csi-driver \
  --wait \
  --namespace kube-system \
  --set syncSecret.enabled="false" \
  --set enableSecretRotation="false" \
  --set 'tokenRequests[0].audience=conjur'

I install the conjur-csi-provider:

helm install conjur-csi-provider \
  cyberark/conjur-k8s-csi-provider \
  --wait \
  --set daemonSet.image.tag="0.2.0" \
  --set provider.name="conjur" \
  --set provider.healthPort="8080" \
  --set provider.socketDir="/var/run/secrets-store-csi-providers"

My application pod fails to mount the volume:

MountVolume.SetUp failed for volume "secrets-store-inline-credentials-from-conjur" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod authorization/storage-service-dcb6b44fc-j84jx, err: rpc error: code = Unknown desc = CKCP016 Failed to get Conjur secrets: CKCP029 Failed to authenticate: CKCP015 Request failed with status code 401

conjur-oss logs:

[origin=<host>] [request_id=d3310575-79e9-47f0-8690-8c49f2fa8a0c] [tid=34] Started POST "/authn-jwt/dev-cluster/dev-cluster/system:serviceaccount:authorization:storage-service/authenticate" for <host> at 2025-07-14 17:43:33 +0000
[origin=<host>] [request_id=d3310575-79e9-47f0-8690-8c49f2fa8a0c] [tid=34] Processing by AuthenticateController#authenticate_jwt as HTML
[origin=<host>] [request_id=d3310575-79e9-47f0-8690-8c49f2fa8a0c] [tid=34]   Parameters: {:controller=>"authenticate", :action=>"authenticate_jwt", :service_id=>"dev-cluster", :account=>"dev-cluster", :id=>"system:serviceaccount:authorization:storage-service"}
[origin=<host>] [request_id=d3310575-79e9-47f0-8690-8c49f2fa8a0c] [tid=34] CONJ00057I Started authentication flow for authenticator 'authn-jwt'
[origin=<host>] [request_id=d3310575-79e9-47f0-8690-8c49f2fa8a0c] [tid=34] dev-cluster:user:USERNAME_MISSING failed to authenticate with authenticator authn-jwt service dev-cluster:webservice:conjur/authn-jwt/dev-cluster: CONJ00122E Invalid signing key settings: One of the following must be defined: jwks-uri, public-keys, or provider-uri
[origin=<host>] [request_id=d3310575-79e9-47f0-8690-8c49f2fa8a0c] [tid=34] dev-cluster:user:system:serviceaccount:authorization:storage-service failed to authenticate with authenticator authn-jwt service dev-cluster:webservice:conjur/authn-jwt/dev-cluster: CONJ00122E Invalid signing key settings: One of the following must be defined: jwks-uri, public-keys, or provider-uri
[origin=<host>] [request_id=d3310575-79e9-47f0-8690-8c49f2fa8a0c] [tid=34] CONJ00048I Authentication Error: #<Errors::Authentication::AuthnJwt::InvalidSigningKeySettings: CONJ00122E Invalid signing key settings: One of the following must be defined: jwks-uri, public-keys, or provider-uri>

What am I missing?

2

I think the problem here may be that you have both jwks-uri and public-keys defined. Try with just one of the two. public-keys is there to support scenarios where jwks-uri can’t be reached by the Conjur server.