CAKC029E - Kubernetes Authenticator declining CSR

hikkoiri · June 23, 2020, 2:04pm

CAKC029E - Kubernetes Authenticator declining CSR

Hello,

I do have a problem right now, when trying to submit a CSR with the conjur-authn-k8s-client to a kubernetes authenticator within Openshift.
I retrieve an unexpected error CAKC029E Received invalid response to certificate signing request. Reason: status code 401 and I hope you can help to find the root of this problem.

The setup

I am running the open-source version of Conjur within Openshift in the namespace conjur-ns.
The Conjur already loaded the following policies. I got inspired by the policy templates provided in your kubenetes demo and stripped the configuration down to a minimum:

# file-name: app-identity-def.yml
- !policy
  id: test-app
  annotations:
    description: This policy connects authn identities to an application identity. It defines a layer named for an application that contains the whitelisted identities that can authenticate to the authn-k8s endpoint. Any permissions granted to the application layer will be inherited by the whitelisted authn identities, thereby granting access to the authenticated identity.
  body:
  - !layer

  # add authn identities to application layer so authn roles inherit app's permissions
  - !grant
    role: !layer 
    members:
    - !layer /conjur/authn-k8s/test-app-authenticator/apps

---
# file-name: authn-any-policy-branch.yml
### This policy is defined to verify that hosts can authenticate with Conjur
### from anywhere in the policy branch, to retrieve secrets for k8s

# Define a policy and add a host to it
- !policy
  id: some-apps
  annotations:
    description: Identities permitted to authenticate
  body:
  - !layer
    annotations:
      description: Layer of authenticator identities permitted to call authn svc
  - &hosts
    - !host
      id: test-app-ns/*/*
      annotations:
        kubernetes/authentication-container-name: authenticator
        openshift: "true"

  - !grant
    role: !layer
    members: *hosts

# Inherit test-app's permissions
- !grant
  role: !layer test-app
  members: !layer some-apps

# Allow the host to authenticate with the authn-k8s authenticator
- !grant
  role: !layer conjur/authn-k8s/test-app-authenticator/users
  members: !layer some-apps

---
# file-name: cluster-authn-svc-def.yml
# This policy defines an authn-k8s endpoint, CA creds and a layer for whitelisted identities permitted to authenticate to it
- !policy
  id: conjur/authn-k8s/test-app-authenticator
  annotations:
    description: Namespace defs for the Conjur cluster in dev
  body:
  - !webservice
    annotations:
      description: authn service for cluster

  - !policy
    id: ca
    body:
    - !variable
      id: cert
      annotations:
        description: CA cert for Kubernetes Pods.
    - !variable
      id: key
      annotations:
        description: CA key for Kubernetes Pods.

  # define layer of whitelisted authn ids permitted to call authn service
  - !layer users

  - !permit
    resource: !webservice
    privilege: [ read, authenticate ]
    role: !layer users

- !grant
  role: !layer conjur/authn-k8s/test-app-authenticator/users
  members:
    - !layer conjur/authn-k8s/test-app-authenticator/apps

---
# file-name: project-authn-def.yml  
# This policy defines a layer of whitelisted identities permitted to authenticate to the authn-k8s endpoint.
- !policy
  id: conjur/authn-k8s/test-app-authenticator/apps
  annotations:
    description: Identities permitted to authenticate 
  body:
  - !layer
    annotations:
      description: Layer of authenticator identities permitted to call authn svc
  - &hosts
    - !host
      id: test-app-ns/*/*
      annotations:
        kubernetes/authentication-container-name: test-app-conjur-authn-k8s-client
        openshift: "true"

    - !host
      id: test-app-ns/deployment/test-app
      annotations:
        kubernetes/authentication-container-name: test-app-conjur-authn-k8s-client
        openshift: "true"


  - !grant
    role: !layer
    members: *hosts

The conclude the enormous configuration file, the authenticator_id of the kubernetes authenticator is test-app-authenticator and it should give authentication clients with the name test-app-conjur-authn-k8s-client from the test-app-ns namespace the possibility to fetch an access code.

The policies were loaded into Conjur over an admin user:

conjur list
[
  "adminAccount:policy:root",
  "adminAccount:policy:conjur/authn-k8s/test-app-authenticator/apps",
  "adminAccount:layer:conjur/authn-k8s/test-app-authenticator/apps",
  "adminAccount:host:conjur/authn-k8s/test-app-authenticator/apps/test-app-ns/*/*",
  "adminAccount:host:conjur/authn-k8s/test-app-authenticator/apps/test-app-ns/deployment/test-app",
  "adminAccount:policy:conjur/authn-k8s/test-app-authenticator",
  "adminAccount:webservice:conjur/authn-k8s/test-app-authenticator",
  "adminAccount:policy:conjur/authn-k8s/test-app-authenticator/ca",
  "adminAccount:variable:conjur/authn-k8s/test-app-authenticator/ca/cert",
  "adminAccount:variable:conjur/authn-k8s/test-app-authenticator/ca/key",
  "adminAccount:layer:conjur/authn-k8s/test-app-authenticator/users",
  "adminAccount:policy:test-app",
  "adminAccount:layer:test-app",
  "adminAccount:policy:some-apps",
  "adminAccount:layer:some-apps",
  "adminAccount:host:some-apps/test-app-ns/*/*"
]

and the ca/cert & ca/key values were populated.
The authenticator got enabled by the following command:

oc patch secret/conjur-conjur-authenticators --patch \
'{"data":{"key": "YXV0aG4sYXV0aG4tazhzL3Rlc3QtYXBwLWF1dGhlbnRpY2F0b3IK"}}'
#secret value is the base64 encoded version of: authn,authn-k8s/test-app-authenticator

Expected behavior

When starting a test application in the configured test-app-ns namespace with the following Deployment configuration, the auth-client should be able to pull an access code and store it under /run/conjur.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test-app
  name: test-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-app
  template:
    metadata:
      labels:
        app: test-app
    spec:
      containers:
      - image: busybox
        command: ["sleep"]
        args: ["infinity"]
        imagePullPolicy: Always
        name: test-app
        ports:
        - containerPort: 8080
        volumeMounts:
          - mountPath: /run/conjur
            name: conjur-access-token
            readOnly: true
      initContainers:
      - image: cyberark/conjur-authn-k8s-client
        imagePullPolicy: Always
        name: test-app-conjur-authn-k8s-client
        env:
          - name: CONTAINER_MODE
            value: init
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: MY_POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: CONJUR_AUTHN_URL
            value: https://conjur-conjur-oss.conjur-ns/authn-k8s/test-app-authenticator
          - name: CONJUR_ACCOUNT
            value: default
          - name: CONJUR_AUTHN_LOGIN
            value: host/conjur/authn-k8s/test-app-authenticator/apps/test-app
          - name: CONJUR_SSL_CERTIFICATE
            valueFrom:
              configMapKeyRef:
                name: conjur-cert
                key: ssl-certificate
        volumeMounts:
          - mountPath: /run/conjur
            name: conjur-access-token
      volumes:
        - name: conjur-access-token
          emptyDir:
            medium: Memory

Actual behavior

Conjur won’t recognize the client as authorized entity and returns a 401 error.

[origin=10.128.2.48] [request_id=b0fb76ed-afff-4835-8b59-57f70ffb44cb] [tid=34] Started POST "/authn-k8s/test-app-authenticator/inject_client_cert" for 127.0.0.1 at 2020-06-23 13:51:01 +0000
[origin=10.128.2.48] [request_id=b0fb76ed-afff-4835-8b59-57f70ffb44cb] [tid=34] Processing by AuthenticateController#k8s_inject_client_cert as HTML
[origin=10.128.2.48] [request_id=b0fb76ed-afff-4835-8b59-57f70ffb44cb] [tid=34]   Parameters: {:controller=>"authenticate", :action=>"k8s_inject_client_cert", :service_id=>"test-app-authenticator"}
[origin=10.128.2.48] [request_id=b0fb76ed-afff-4835-8b59-57f70ffb44cb] [tid=34] Completed 401 Unauthorized in 2ms

The client then complains about a failed login:

INFO: 2020/06/23 12:59:29 main.go:20: CAKC014I Kubernetes Authenticator Client v0.18.0-3383197 starting up...
INFO: 2020/06/23 12:59:29 main.go:45: CAKC006I Authenticating as user '&{host/conjur/authn-k8s/test-app-authenticator/apps/test-app host.conjur.authn-k8s test-app-authenticator.apps.test-app}'
INFO: 2020/06/23 12:59:29 authenticator.go:181: CAKC005I Trying to login Conjur...
INFO: 2020/06/23 12:59:29 authenticator.go:113: CAKC007I Logging in as user &{host/conjur/authn-k8s/test-app-authenticator/apps/test-app host.conjur.authn-k8s test-app-authenticator.apps.test-app}.
INFO: 2020/06/23 12:59:29 requests.go:23: CAKC011I Login request to: https://conjur-conjur-oss.conjur-ns/authn-k8s/test-app-authenticator/inject_client_cert
ERROR: 2020/06/23 12:59:29 authenticator.go:133: CAKC029E Received invalid response to certificate signing request. Reason: status code 401, 
ERROR: 2020/06/23 12:59:29 authenticator.go:184: CAKC015E Login failed
ERROR: 2020/06/23 12:59:29 main.go:48: CAKC016E Failed to authenticate

Am I missing something or did I mess up in the configuration? Tbh as a first time Conjur user the whole setup is pretty overwhelming and hard to grasp.
Already thanks in advance for any help!

nathan.whipple · June 23, 2020, 2:37pm

Are you doing any SSL offloading between the client app and the Conjur server?

hikkoiri · June 23, 2020, 2:50pm

No, the Conjur server does that.

github.com

cyberark/conjur-oss-helm-chart/blob/master/conjur-oss/files/conjur.conf#L25


  proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;
  proxy_set_header X-SSL-Client-Certificate $ssl_client_escaped_cert;

  # Container health status
  location /status {
    access_log off;
    return 200 'OK';
  }

  location / {
    proxy_pass http://127.0.0.1:80;
  }
}

server {
  # redirect outside unencrypted access to https
  listen 9000;
  listen [::]:9000;
  return 301 https://$host$request_uri;
}

I used this Helm chart to deploy Conjur.

hikkoiri · June 24, 2020, 7:48am

Is there any chance to change the log level to DEBUG, to receive more information?

hikkoiri · June 24, 2020, 11:00am

Nevermind, I found a way to increase the log level through the CONJUR_LOG_LEVEL env variable.

Then in the logs I found this message:

Authentication Error: #<Errors::Authentication::Security::AccountNotDefined: CONJ00008E Account 'default' is not defined in Conjur>

I was wondering, because the account was set to default by the Helm chart. (Compare with https://github.com/cyberark/conjur-oss-helm-chart/blob/master/conjur-oss/values.yaml#L14), but the I realized, that I created all policies through the admin account with the name adminAccount. I was not aware that the names should match.
Following this scheme, I was able to resolve some misconfigurations in the yaml configs above.
To conclude, it works now.

But still, it would be really nice, if that log level configuration would appear in the corresponding documentation.

system · July 1, 2020, 11:00am

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Authenticator pod failing to authenticate Secrets Management - Conjur, Secrets Hub & CP	21	7070	November 18, 2019
Conjur-OSS authentication fails in OpenShift Automation General opensource , openshift , conjur	1	163	October 5, 2023
NoMethodError error thrown on Conjur OSS kubernetes authenticator request Secrets Management - Conjur, Secrets Hub & CP opensource , kubernetes	7	644	September 12, 2020
Secrets Provider authenticator pod failing to authenticate Secrets Management - Conjur, Secrets Hub & CP kubernetes	7	1327	October 2, 2020
K8s Authenticator failing while trying to inject client cert Conjur Enterprise kubernetes , dap , conjur	4	747	October 5, 2021

CAKC029E - Kubernetes Authenticator declining CSR

CAKC029E - Kubernetes Authenticator declining CSR

The setup

Expected behavior

Actual behavior

Related Topics