Integration with Kubernetes Authenticator Client (sidecar)

Hi all,

I’m working on a Kubernetes integration to deploy a simple MySQL client application with Kubernetes Authenticator Client configured as a sidecar. However, when I deploy the application, my Init cyberark/conjur-authn-k8s-client container fails to ‘error’.

I am not sure if I’m missing any steps, or something is not properly configured.

Here is my environment:

  • Kubernetes cluster setup in an on-prem lab
  • Volumes provisioned by NFS
  • MariaDB database - master-slave

And here are the steps I’ve taken so far:

  • Created a ‘conjur-demo’ namespace
  • Installed the Conjur OSS v2.0.0 Helm Chart following the Simple Install guide
  • As my cluster already had a load balancer, I manually changed the service/conjur-helm-v2-conjur-oss-ingress from Type:LoadBalancer to Type:NodePort
  • As the storageClass for my Conjur PVC is not set by default, I changed it to nfs. This allowed me to connect to Conjur via my load balancer’s external IP + Conjur Ingress port: https://172.16.220.56:32149
  • In my local host file, I then pointed the custom domain to the IP address of my cluster load balancer IP: 172.16.220.56 conjur.myorg.com
  • Exec into my conjur-oss pod to created a default account
  • Initialised with -u https://conjur.myorg.com:32149 -a "default"
  • Successfully able to login with the admin user
  • Loaded policies into Conjur:
--- 
# project-authn-def.yml
# this policy defines a layer of whitelisted identities permitted to authenticate to the authn-k8s endpoint
- !policy
  id: conjur/authn-k8s/sidecar/apps
  annotations:
    description: identities permitted to authenticate
  body:
  - !layer
    annotations:
      description: Layer of authenticator identities permitted to call authn svc
      
  - &hosts
    - !host
      id: sidecar-demo/*/*
      annotations:
        kubernetes/authentication-container-name: authenticator
        
    - !host
      id: sidecar-demo/sidecar-demo-app
      annotations:
        kubernetes/authentication-container-name: authenticator

  - !grant
    role: !layer
    members: *hosts

---
- !policy
  id: conjur/authn-k8s/sidecar/users
  
  body:
    - !layer

---
# cluster-authn-svc-def.yml
# this policy defines an authn-k8s endpoint, CA creds and a layer for whitelisted identities permitted to authenticate to it

- !policy
  id: conjur/authn-k8s/sidecar
  
  body:
  - !webservice
    annotations:
      description: authn service for cluster
      
  - !policy
    id: ca
    body:
    - !variable
      id: cert
      annotations:
        description: CA cert for Kubernetes Pods.
    - !variable
      id: key
      annotations:
        description: CA key for Kubernetes Pods.
    
  # define layer of whitelisted authn ids permitted to call authn service
  - !layer users
  
  - !permit
    resource: !webservice
    privilege: [ read, authenticate ]
    role: !layer users
    
- !grant
  role: !layer conjur/authn-k8s/sidecar/users
  members:
    - !layer conjur/authn-k8s/sidecar/apps

---
# from authn-any-policy-branch.yml
### This policy is defined to verify that hosts can authenticate with Conjur
### from anywhere in the policy branch, to retireve secrets for k8s

- !policy
  id: sidecar-apps
  annotations:
    description: Identities permitted to authenticate
  body:
  - !layer
    annotations:
      description: Layer of authenticator identities permitted to call authn svc
  - &hosts
    - !host
      id: sidecar-demo/*/*
      annotations:
        authn-k8s/authentication-container-name: authenticator
        
  - !grant
    role: !layer
    members: *hosts
    
# inherit sidecar-demo's permissions
- !grant
  role: !layer sidecar-demo
  members: !layer sidecar-apps
  
# allow the host to authenticate with the uathn-k8s authenticator
- !grant
  role: !layer conjur/authn-k8s/sidecar/users
  members: !layer sidecar-apps

---
# from app-identity-def.yml
# this policy connects authn identities to an application identity
- !policy
  id: sidecar-demo
  
  body:
  - !layer
 # add authn identities to application layer so authn roles inherit app's permissions
  - !grant
    role: !layer
    - !layer conjur/authn-k8s/sidecar/apps
  • Added role binding to sidecar-demo namespace where I intend to deploy my application:
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: conjur-server-authn-role-binding
  namespace: sidecar-demo
subjects:
- kind: ServiceAccount
  name: conjur-helm-v2-conjur-oss
  namespace: conjur-demo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: conjur-helm-v2-conjur-authenticator
  • Initialized the CA with AUTHENTICATOR_ID='sidecar' and CONJUR_ACCOUNT='default':
#!/bin/bash
set -e
AUTHENTICATOR_ID='sidecar'
CONJUR_ACCOUNT='default'

# Generate OpenSSL private key
openssl genrsa -out ca.key 2048

CONFIG="
[ req ]
distinguished_name = dn
x509_extensions = v3_ca
[ dn ]
[ v3_ca ]
basicConstraints = critical,CA:TRUE
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always,issuer:always
"

# Generate root CA certificate
openssl req -x509 -new -nodes -key ca.key -sha1 -days 3650 -set_serial 0x0 -out ca.cert \
  -subj "/CN=conjur.authn-k8s.$AUTHENTICATOR_ID/OU=Conjur Kubernetes CA/O=$CONJUR_ACCOUNT" \
  -config <(echo "$CONFIG")
 
# Verify cert
openssl x509 -in ca.cert -text -noout

# Load variable values
conjur variable values add conjur/authn-k8s/$AUTHENTICATOR_ID/ca/key "$(cat ca.key)"
conjur variable values add conjur/authn-k8s/$AUTHENTICATOR_ID/ca/cert "$(cat ca.cert)"
  • And finally deployed the app:
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: sidecar-demo-app
  name: sidecar-demo-app
  namespace: sidecar-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sidecar-demo-app
  template:
    metadata:
      labels:
        app: sidecar-demo-app
    spec:
      serviceAccountName: sidecar-service-account
      containers:
      - image: mysql:5.6
        imagePullPolicy: Always
        name: sidecar-demo-app
        args:
          - mysql
            --host=vm-mariadb.default.svc.cluster.local
            --user=root
            --password=5jqJL2b8hqn3
        ports:
        - containerPort: 8080
        env:
          - name: CONJUR_APPLIANCE_URL
            value: "https://conjur.myorg.com:32149"
          - name: CONJUR_ACCOUNT
            value: default
          - name: CONJUR_AUTHN_TOKEN_FILE
            value: /run/conjur/access-token
          - name: CONJUR_SSL_CERTIFICATE
            valueFrom:
              configMapKeyRef:
                name: conjur-cert
                key: ssl-certificate
        volumeMounts:
        - mountPath: /run/conjur
          name: conjur-access-token
          readOnly: true
      initContainers:
      - image: cyberark/conjur-authn-k8s-client
        imagePullPolicy: Always
        name: authenticator
        env:
          - name: CONTAINER_MODE
            value: sidecar
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: MY_POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: CONJUR_AUTHN_URL
            value: "https://conjur.myorg.com:32149/authn-k8s/sidecar"
          - name: CONJUR_ACCOUNT
            value: default
          - name: CONJUR_AUTHN_LOGIN
            value: "host/conjur/authn-k8s/sidecar/apps/sidecar-demo/sidecar-demo-app"
          - name: CONJUR_SSL_CERTIFICATE
            valueFrom:
              configMapKeyRef:
                name: conjur-cert
                key: ssl-certificate
        volumeMounts:
          - mountPath: /run/conjur
            name: conjur-access-token
      imagePullSecrets:
        - name: dockerpullsecret
      volumes:
        - name: conjur-access-token
          emptyDir:
            medium: Memory

This is the state of my sidecar-demo-app pods:

Name:           sidecar-demo-app-75ff75fd49-xfz46
Namespace:      sidecar-demo
Priority:       0
Node:           k8s-kubespray-worker-0/172.16.220.56
Start Time:     Thu, 10 Sep 2020 13:53:14 +0200
Labels:         app=sidecar-demo-app
                pod-template-hash=75ff75fd49
Annotations:    <none>
Status:         Pending
IP:             10.233.65.137
IPs:            <none>
Controlled By:  ReplicaSet/sidecar-demo-app-75ff75fd49
Init Containers:
  authenticator:
    Container ID:   docker://bd6d0ba1a411da3e176de28f539e65c8d64bb1f92acf4390741154262e9ad947
    Image:          cyberark/conjur-authn-k8s-client
    Image ID:       docker-pullable://cyberark/conjur-authn-k8s-client@sha256:3e8e9e8df1bb5d739dc8ab905f9547eec43019563313f6a2dee4ee3455d0ab2c
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 10 Sep 2020 14:03:20 +0200
      Finished:     Thu, 10 Sep 2020 14:05:29 +0200
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 10 Sep 2020 14:00:27 +0200
      Finished:     Thu, 10 Sep 2020 14:02:36 +0200
    Ready:          False
    Restart Count:  4
    Environment:
      CONTAINER_MODE:          sidecar
      MY_POD_NAME:             sidecar-demo-app-75ff75fd49-xfz46 (v1:metadata.name)
      MY_POD_NAMESPACE:        sidecar-demo (v1:metadata.namespace)
      MY_POD_IP:                (v1:status.podIP)
      CONJUR_AUTHN_URL:        https://conjur.myorg.com:32149/authn-k8s/sidecar
      CONJUR_ACCOUNT:          default
      CONJUR_AUTHN_LOGIN:      host/conjur/authn-k8s/sidecar/apps/sidecar-demo/sidecar-demo-app
      CONJUR_SSL_CERTIFICATE:  <set to the key 'ssl-certificate' of config map 'conjur-cert'>  Optional: false
    Mounts:
      /run/conjur from conjur-access-token (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from sidecar-service-account-token-s8k6r (ro)
Containers:
  sidecar-demo-app:
    Container ID:
    Image:         mysql:5.6
    Image ID:
    Port:          8080/TCP
    Host Port:     0/TCP
    Args:
      mysql --host=vm-mariadb.default.svc.cluster.local --user=root --password=5jqJL2b8hqn3
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      CONJUR_APPLIANCE_URL:     https://conjur.myorg.com:32149
      CONJUR_ACCOUNT:           default
      CONJUR_AUTHN_TOKEN_FILE:  /run/conjur/access-token
      CONJUR_SSL_CERTIFICATE:   <set to the key 'ssl-certificate' of config map 'conjur-cert'>  Optional: false
    Mounts:
      /run/conjur from conjur-access-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from sidecar-service-account-token-s8k6r (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  conjur-access-token:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  sidecar-service-account-token-s8k6r:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  sidecar-service-account-token-s8k6r
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                  From                             Message
  ----     ------     ----                 ----                             -------
  Normal   Scheduled  12m                  default-scheduler                Successfully assigned sidecar-demo/sidecar-demo-app-75ff75fd49-xfz46 to k8s-kubespray-worker-0
  Normal   Pulling    2m13s (x5 over 12m)  kubelet, k8s-kubespray-worker-0  Pulling image "cyberark/conjur-authn-k8s-client"
  Normal   Pulled     2m12s (x5 over 12m)  kubelet, k8s-kubespray-worker-0  Successfully pulled image "cyberark/conjur-authn-k8s-client"
  Normal   Created    2m12s (x5 over 12m)  kubelet, k8s-kubespray-worker-0  Created container authenticator
  Normal   Started    2m12s (x5 over 12m)  kubelet, k8s-kubespray-worker-0  Started container authenticator
  Warning  BackOff    3s (x7 over 7m54s)   kubelet, k8s-kubespray-worker-0  Back-off restarting failed container

Also, I am unsure if I correctly configured the environment variables:

  • CONJUR_APPLIANCE_URL=https://conjur.myorg.com:32149
  • CONJUR_AUTHN_URL=https://conjur.myorg.com:32149/authn-k8s/sidecar
  • CONJUR_AUTHN_LOGIN=host/conjur/authn-k8s/sidecar/apps/sidecar-demo/sidecar-demo-app

And for CONJUR_AUTHENTICATORS, how can I make sure auth-k8s/sidecar has been added? If it hasn’t, how can I add it?

Any help will be greatly appreciated!
Jivana

I also was wondering: " how can I make sure auth-k8s/sidecar has been added? If it hasn’t, how can I add it?" which if I understand correctly the helm chart needs to be upgraded? or patch the current conjur-authenticator secret/configmap ?

Anyone from conjur able to help with the question?

Hi @jhunt , @amanz,

Here is a Conjur-OSS authn-k8s
Troubleshooting Guide that might help.

There are a couple of ways to check whether the authn-k8s authenticator has been enabled with an authenticator ID of sidecar:

  1. Read the enabled authenticators from the Conjur authenticators Kubernetes Secret. Typically Helm install will name this Secret {HELM_RELEASE}-conjur-authenticators. The commands to do this are in the Step-by-Step Troubleshooting Guide for Conjur OSS Kubernetes Authentication, but I’ll add them here:

    # Set environment. Modify as necessary to match your setup.
    $ CONJUR_NAMESPACE=conjur-demo
    $ HELM_RELEASE=conjur-helm-v2
    
    $ authenticators="$(kubectl get secret \
              -n "$CONJUR_NAMESPACE" \
              "$HELM_RELEASE-conjur-authenticators" \
              --template={{.data.key}} \
              | base64 -d \
              | sed 's/,/\n  /g')"
    $ echo "Enabled Conjur authenticators:"; echo "  $authenticators"
    Enabled Conjur authenticators:
      authn
      authn-k8s/sidecar
    $
    
  2. …OR…Read the current authenticators Helm chart value for your Helm Release:

    $ helm list --namespace conjur-demo
    NAME          	NAMESPACE 	REVISION	UPDATED                                	STATUS  	CHART           	APP VERSION
    conjur-helm-v2	conjur-demo	4       	2021-02-22 09:37:33.730289392 -0500 EST	deployed	conjur-oss-2.0.3	           
    $ helm get values --namespace conjur-demo conjur-helm-v2 | grep authenticators
    authenticators: authn,authn-k8s/sidecar
    $
    
    

In your case, if you don’t see authenticators set to authn,authn-k8s/sidecar, you can add authn-k8s/sidecar as follows (also from the Troubleshooting Guide) (This assumes that you have a local git clone of the Conjur OSS helm chart at ./conjur-oss):

```
# Set environment. Modify as necessary to match your setup.
$ CONJUR_NAMESPACE=conjur-demo
$ HELM_RELEASE=conjur-helm-v2
$ AUTHENTICATOR_ID="sidecar"

$ helm upgrade \
    -n "$CONJUR_NAMESPACE" \
    --reuse-values \
    --set authenticators="authn\,authn-k8s/$AUTHENTICATOR_ID" \
    --wait \
    --timeout 300s \
    "$HELM_RELEASE" \
    ./conjur-oss
```

There are some other things I can suggest to try with your setup:

  • Your deployment includes a mysql:5.6 container that’s using predefined --host, --user, and --password on the command line. Is the intent to replace these predefined values with values read from Conjur? In order to do that, I think the easiest method will be to incorporate Summon along with mysql:5.6 into a custom Docker container, and use Summon to read the secrets from Conjur.

To see how to do this, please check out this
hands-on demonstration of Conjur authn-k8s.
This will give you a quick reference “playground” with several examples of integration with Conjur on your own, local Kubernetes-in-Docker setup.

  • Your deployment includes the conjur-authn-k8s-client within the initContainers section of the application manifest, but the CONTAINER_MODE is set to sidecar. Try switching CONTAINER_MODE to init, or move the conjur-authn-k8s-client section to the regular containers section of the manifest.

  • Your values for CONJUR_AUTHN_URL and CONJUR_AUTHN_LOGIN appear correct to me.
    For CONJUR_APPLIANCE_URL, you can use the internal Kubernetes service address of your Conjur server, since both the Conjur server and the application pod are within the same Kubernetes cluster. The format would be:

    https://{CONJUR_SERVICE_NAME}.{CONJUR_NAMESPACE}.svc.cluster.local
    

which in your case, I believe would translate to:

```
CONJUR_APPLIANCE_URL=https://conjur-oss.conjur-demo.svc.cluster.local
```

Again, it might help to check out the
hands-on demonstration of Conjur authn-k8s.
This will give you a quick reference “playground” with several examples of integration with Conjur on your own, local Kubernetes-in-Docker setup, as a reference.

Also, the
Step-by-Step Troubleshooting Guide for Conjur OSS Kubernetes Authentication may help.

Hope this helps, let me know if you have more questions.

  • Dane
1 Like

Thank you @dane, that helped