Integration with Kubernetes Authenticator Client (sidecar)

Hi all,

I’m working on a Kubernetes integration to deploy a simple MySQL client application with Kubernetes Authenticator Client configured as a sidecar. However, when I deploy the application, my Init cyberark/conjur-authn-k8s-client container fails to ‘error’.

I am not sure if I’m missing any steps, or something is not properly configured.

Here is my environment:

  • Kubernetes cluster setup in an on-prem lab
  • Volumes provisioned by NFS
  • MariaDB database - master-slave

And here are the steps I’ve taken so far:

  • Created a ‘conjur-demo’ namespace
  • Installed the Conjur OSS v2.0.0 Helm Chart following the Simple Install guide
  • As my cluster already had a load balancer, I manually changed the service/conjur-helm-v2-conjur-oss-ingress from Type:LoadBalancer to Type:NodePort
  • As the storageClass for my Conjur PVC is not set by default, I changed it to nfs. This allowed me to connect to Conjur via my load balancer’s external IP + Conjur Ingress port: https://172.16.220.56:32149
  • In my local host file, I then pointed the custom domain to the IP address of my cluster load balancer IP: 172.16.220.56 conjur.myorg.com
  • Exec into my conjur-oss pod to created a default account
  • Initialised with -u https://conjur.myorg.com:32149 -a "default"
  • Successfully able to login with the admin user
  • Loaded policies into Conjur:
--- 
# project-authn-def.yml
# this policy defines a layer of whitelisted identities permitted to authenticate to the authn-k8s endpoint
- !policy
  id: conjur/authn-k8s/sidecar/apps
  annotations:
    description: identities permitted to authenticate
  body:
  - !layer
    annotations:
      description: Layer of authenticator identities permitted to call authn svc
      
  - &hosts
    - !host
      id: sidecar-demo/*/*
      annotations:
        kubernetes/authentication-container-name: authenticator
        
    - !host
      id: sidecar-demo/sidecar-demo-app
      annotations:
        kubernetes/authentication-container-name: authenticator

  - !grant
    role: !layer
    members: *hosts

---
- !policy
  id: conjur/authn-k8s/sidecar/users
  
  body:
    - !layer

---
# cluster-authn-svc-def.yml
# this policy defines an authn-k8s endpoint, CA creds and a layer for whitelisted identities permitted to authenticate to it

- !policy
  id: conjur/authn-k8s/sidecar
  
  body:
  - !webservice
    annotations:
      description: authn service for cluster
      
  - !policy
    id: ca
    body:
    - !variable
      id: cert
      annotations:
        description: CA cert for Kubernetes Pods.
    - !variable
      id: key
      annotations:
        description: CA key for Kubernetes Pods.
    
  # define layer of whitelisted authn ids permitted to call authn service
  - !layer users
  
  - !permit
    resource: !webservice
    privilege: [ read, authenticate ]
    role: !layer users
    
- !grant
  role: !layer conjur/authn-k8s/sidecar/users
  members:
    - !layer conjur/authn-k8s/sidecar/apps

---
# from authn-any-policy-branch.yml
### This policy is defined to verify that hosts can authenticate with Conjur
### from anywhere in the policy branch, to retireve secrets for k8s

- !policy
  id: sidecar-apps
  annotations:
    description: Identities permitted to authenticate
  body:
  - !layer
    annotations:
      description: Layer of authenticator identities permitted to call authn svc
  - &hosts
    - !host
      id: sidecar-demo/*/*
      annotations:
        authn-k8s/authentication-container-name: authenticator
        
  - !grant
    role: !layer
    members: *hosts
    
# inherit sidecar-demo's permissions
- !grant
  role: !layer sidecar-demo
  members: !layer sidecar-apps
  
# allow the host to authenticate with the uathn-k8s authenticator
- !grant
  role: !layer conjur/authn-k8s/sidecar/users
  members: !layer sidecar-apps

---
# from app-identity-def.yml
# this policy connects authn identities to an application identity
- !policy
  id: sidecar-demo
  
  body:
  - !layer
 # add authn identities to application layer so authn roles inherit app's permissions
  - !grant
    role: !layer
    - !layer conjur/authn-k8s/sidecar/apps
  • Added role binding to sidecar-demo namespace where I intend to deploy my application:
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: conjur-server-authn-role-binding
  namespace: sidecar-demo
subjects:
- kind: ServiceAccount
  name: conjur-helm-v2-conjur-oss
  namespace: conjur-demo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: conjur-helm-v2-conjur-authenticator
  • Initialized the CA with AUTHENTICATOR_ID='sidecar' and CONJUR_ACCOUNT='default':
#!/bin/bash
set -e
AUTHENTICATOR_ID='sidecar'
CONJUR_ACCOUNT='default'

# Generate OpenSSL private key
openssl genrsa -out ca.key 2048

CONFIG="
[ req ]
distinguished_name = dn
x509_extensions = v3_ca
[ dn ]
[ v3_ca ]
basicConstraints = critical,CA:TRUE
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always,issuer:always
"

# Generate root CA certificate
openssl req -x509 -new -nodes -key ca.key -sha1 -days 3650 -set_serial 0x0 -out ca.cert \
  -subj "/CN=conjur.authn-k8s.$AUTHENTICATOR_ID/OU=Conjur Kubernetes CA/O=$CONJUR_ACCOUNT" \
  -config <(echo "$CONFIG")
 
# Verify cert
openssl x509 -in ca.cert -text -noout

# Load variable values
conjur variable values add conjur/authn-k8s/$AUTHENTICATOR_ID/ca/key "$(cat ca.key)"
conjur variable values add conjur/authn-k8s/$AUTHENTICATOR_ID/ca/cert "$(cat ca.cert)"
  • And finally deployed the app:
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: sidecar-demo-app
  name: sidecar-demo-app
  namespace: sidecar-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sidecar-demo-app
  template:
    metadata:
      labels:
        app: sidecar-demo-app
    spec:
      serviceAccountName: sidecar-service-account
      containers:
      - image: mysql:5.6
        imagePullPolicy: Always
        name: sidecar-demo-app
        args:
          - mysql
            --host=vm-mariadb.default.svc.cluster.local
            --user=root
            --password=5jqJL2b8hqn3
        ports:
        - containerPort: 8080
        env:
          - name: CONJUR_APPLIANCE_URL
            value: "https://conjur.myorg.com:32149"
          - name: CONJUR_ACCOUNT
            value: default
          - name: CONJUR_AUTHN_TOKEN_FILE
            value: /run/conjur/access-token
          - name: CONJUR_SSL_CERTIFICATE
            valueFrom:
              configMapKeyRef:
                name: conjur-cert
                key: ssl-certificate
        volumeMounts:
        - mountPath: /run/conjur
          name: conjur-access-token
          readOnly: true
      initContainers:
      - image: cyberark/conjur-authn-k8s-client
        imagePullPolicy: Always
        name: authenticator
        env:
          - name: CONTAINER_MODE
            value: sidecar
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: MY_POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: CONJUR_AUTHN_URL
            value: "https://conjur.myorg.com:32149/authn-k8s/sidecar"
          - name: CONJUR_ACCOUNT
            value: default
          - name: CONJUR_AUTHN_LOGIN
            value: "host/conjur/authn-k8s/sidecar/apps/sidecar-demo/sidecar-demo-app"
          - name: CONJUR_SSL_CERTIFICATE
            valueFrom:
              configMapKeyRef:
                name: conjur-cert
                key: ssl-certificate
        volumeMounts:
          - mountPath: /run/conjur
            name: conjur-access-token
      imagePullSecrets:
        - name: dockerpullsecret
      volumes:
        - name: conjur-access-token
          emptyDir:
            medium: Memory

This is the state of my sidecar-demo-app pods:

Name:           sidecar-demo-app-75ff75fd49-xfz46
Namespace:      sidecar-demo
Priority:       0
Node:           k8s-kubespray-worker-0/172.16.220.56
Start Time:     Thu, 10 Sep 2020 13:53:14 +0200
Labels:         app=sidecar-demo-app
                pod-template-hash=75ff75fd49
Annotations:    <none>
Status:         Pending
IP:             10.233.65.137
IPs:            <none>
Controlled By:  ReplicaSet/sidecar-demo-app-75ff75fd49
Init Containers:
  authenticator:
    Container ID:   docker://bd6d0ba1a411da3e176de28f539e65c8d64bb1f92acf4390741154262e9ad947
    Image:          cyberark/conjur-authn-k8s-client
    Image ID:       docker-pullable://cyberark/conjur-authn-k8s-client@sha256:3e8e9e8df1bb5d739dc8ab905f9547eec43019563313f6a2dee4ee3455d0ab2c
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 10 Sep 2020 14:03:20 +0200
      Finished:     Thu, 10 Sep 2020 14:05:29 +0200
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 10 Sep 2020 14:00:27 +0200
      Finished:     Thu, 10 Sep 2020 14:02:36 +0200
    Ready:          False
    Restart Count:  4
    Environment:
      CONTAINER_MODE:          sidecar
      MY_POD_NAME:             sidecar-demo-app-75ff75fd49-xfz46 (v1:metadata.name)
      MY_POD_NAMESPACE:        sidecar-demo (v1:metadata.namespace)
      MY_POD_IP:                (v1:status.podIP)
      CONJUR_AUTHN_URL:        https://conjur.myorg.com:32149/authn-k8s/sidecar
      CONJUR_ACCOUNT:          default
      CONJUR_AUTHN_LOGIN:      host/conjur/authn-k8s/sidecar/apps/sidecar-demo/sidecar-demo-app
      CONJUR_SSL_CERTIFICATE:  <set to the key 'ssl-certificate' of config map 'conjur-cert'>  Optional: false
    Mounts:
      /run/conjur from conjur-access-token (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from sidecar-service-account-token-s8k6r (ro)
Containers:
  sidecar-demo-app:
    Container ID:
    Image:         mysql:5.6
    Image ID:
    Port:          8080/TCP
    Host Port:     0/TCP
    Args:
      mysql --host=vm-mariadb.default.svc.cluster.local --user=root --password=5jqJL2b8hqn3
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      CONJUR_APPLIANCE_URL:     https://conjur.myorg.com:32149
      CONJUR_ACCOUNT:           default
      CONJUR_AUTHN_TOKEN_FILE:  /run/conjur/access-token
      CONJUR_SSL_CERTIFICATE:   <set to the key 'ssl-certificate' of config map 'conjur-cert'>  Optional: false
    Mounts:
      /run/conjur from conjur-access-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from sidecar-service-account-token-s8k6r (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  conjur-access-token:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  sidecar-service-account-token-s8k6r:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  sidecar-service-account-token-s8k6r
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                  From                             Message
  ----     ------     ----                 ----                             -------
  Normal   Scheduled  12m                  default-scheduler                Successfully assigned sidecar-demo/sidecar-demo-app-75ff75fd49-xfz46 to k8s-kubespray-worker-0
  Normal   Pulling    2m13s (x5 over 12m)  kubelet, k8s-kubespray-worker-0  Pulling image "cyberark/conjur-authn-k8s-client"
  Normal   Pulled     2m12s (x5 over 12m)  kubelet, k8s-kubespray-worker-0  Successfully pulled image "cyberark/conjur-authn-k8s-client"
  Normal   Created    2m12s (x5 over 12m)  kubelet, k8s-kubespray-worker-0  Created container authenticator
  Normal   Started    2m12s (x5 over 12m)  kubelet, k8s-kubespray-worker-0  Started container authenticator
  Warning  BackOff    3s (x7 over 7m54s)   kubelet, k8s-kubespray-worker-0  Back-off restarting failed container

Also, I am unsure if I correctly configured the environment variables:

  • CONJUR_APPLIANCE_URL=https://conjur.myorg.com:32149
  • CONJUR_AUTHN_URL=https://conjur.myorg.com:32149/authn-k8s/sidecar
  • CONJUR_AUTHN_LOGIN=host/conjur/authn-k8s/sidecar/apps/sidecar-demo/sidecar-demo-app

And for CONJUR_AUTHENTICATORS, how can I make sure auth-k8s/sidecar has been added? If it hasn’t, how can I add it?

Any help will be greatly appreciated!
Jivana