Hi all,
I’m working on a Kubernetes integration to deploy a simple MySQL client application with Kubernetes Authenticator Client configured as a sidecar. However, when I deploy the application, my Init cyberark/conjur-authn-k8s-client container fails to ‘error’.
I am not sure if I’m missing any steps, or something is not properly configured.
Here is my environment:
- Kubernetes cluster setup in an on-prem lab
- Volumes provisioned by NFS
- MariaDB database - master-slave
And here are the steps I’ve taken so far:
- Created a ‘conjur-demo’ namespace
- Installed the Conjur OSS v2.0.0 Helm Chart following the Simple Install guide
- As my cluster already had a load balancer, I manually changed the
service/conjur-helm-v2-conjur-oss-ingressfromType:LoadBalancertoType:NodePort - As the
storageClassfor my ConjurPVCis not set by default, I changed it tonfs. This allowed me to connect to Conjur via my load balancer’s external IP + Conjur Ingress port:https://172.16.220.56:32149 - In my local host file, I then pointed the custom domain to the IP address of my cluster load balancer IP:
172.16.220.56 conjur.myorg.com - Exec into my
conjur-osspod to created adefaultaccount - Initialised with
-u https://conjur.myorg.com:32149 -a "default" - Successfully able to login with the
adminuser - Loaded policies into Conjur:
--- # project-authn-def.yml # this policy defines a layer of whitelisted identities permitted to authenticate to the authn-k8s endpoint - !policy id: conjur/authn-k8s/sidecar/apps annotations: description: identities permitted to authenticate body: - !layer annotations: description: Layer of authenticator identities permitted to call authn svc - &hosts - !host id: sidecar-demo/*/* annotations: kubernetes/authentication-container-name: authenticator - !host id: sidecar-demo/sidecar-demo-app annotations: kubernetes/authentication-container-name: authenticator - !grant role: !layer members: *hosts --- - !policy id: conjur/authn-k8s/sidecar/users body: - !layer --- # cluster-authn-svc-def.yml # this policy defines an authn-k8s endpoint, CA creds and a layer for whitelisted identities permitted to authenticate to it - !policy id: conjur/authn-k8s/sidecar body: - !webservice annotations: description: authn service for cluster - !policy id: ca body: - !variable id: cert annotations: description: CA cert for Kubernetes Pods. - !variable id: key annotations: description: CA key for Kubernetes Pods. # define layer of whitelisted authn ids permitted to call authn service - !layer users - !permit resource: !webservice privilege: [ read, authenticate ] role: !layer users - !grant role: !layer conjur/authn-k8s/sidecar/users members: - !layer conjur/authn-k8s/sidecar/apps --- # from authn-any-policy-branch.yml ### This policy is defined to verify that hosts can authenticate with Conjur ### from anywhere in the policy branch, to retireve secrets for k8s - !policy id: sidecar-apps annotations: description: Identities permitted to authenticate body: - !layer annotations: description: Layer of authenticator identities permitted to call authn svc - &hosts - !host id: sidecar-demo/*/* annotations: authn-k8s/authentication-container-name: authenticator - !grant role: !layer members: *hosts # inherit sidecar-demo's permissions - !grant role: !layer sidecar-demo members: !layer sidecar-apps # allow the host to authenticate with the uathn-k8s authenticator - !grant role: !layer conjur/authn-k8s/sidecar/users members: !layer sidecar-apps --- # from app-identity-def.yml # this policy connects authn identities to an application identity - !policy id: sidecar-demo body: - !layer # add authn identities to application layer so authn roles inherit app's permissions - !grant role: !layer - !layer conjur/authn-k8s/sidecar/apps
- Added role binding to
sidecar-demonamespace where I intend to deploy my application:
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: conjur-server-authn-role-binding namespace: sidecar-demo subjects: - kind: ServiceAccount name: conjur-helm-v2-conjur-oss namespace: conjur-demo roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: conjur-helm-v2-conjur-authenticator
- Initialized the CA with
AUTHENTICATOR_ID='sidecar'andCONJUR_ACCOUNT='default':
#!/bin/bash set -e AUTHENTICATOR_ID='sidecar' CONJUR_ACCOUNT='default' # Generate OpenSSL private key openssl genrsa -out ca.key 2048 CONFIG=" [ req ] distinguished_name = dn x509_extensions = v3_ca [ dn ] [ v3_ca ] basicConstraints = critical,CA:TRUE subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer:always " # Generate root CA certificate openssl req -x509 -new -nodes -key ca.key -sha1 -days 3650 -set_serial 0x0 -out ca.cert \ -subj "/CN=conjur.authn-k8s.$AUTHENTICATOR_ID/OU=Conjur Kubernetes CA/O=$CONJUR_ACCOUNT" \ -config <(echo "$CONFIG") # Verify cert openssl x509 -in ca.cert -text -noout # Load variable values conjur variable values add conjur/authn-k8s/$AUTHENTICATOR_ID/ca/key "$(cat ca.key)" conjur variable values add conjur/authn-k8s/$AUTHENTICATOR_ID/ca/cert "$(cat ca.cert)"
- And finally deployed the app:
apiVersion: apps/v1 kind: Deployment metadata: labels: app: sidecar-demo-app name: sidecar-demo-app namespace: sidecar-demo spec: replicas: 1 selector: matchLabels: app: sidecar-demo-app template: metadata: labels: app: sidecar-demo-app spec: serviceAccountName: sidecar-service-account containers: - image: mysql:5.6 imagePullPolicy: Always name: sidecar-demo-app args: - mysql --host=vm-mariadb.default.svc.cluster.local --user=root --password=5jqJL2b8hqn3 ports: - containerPort: 8080 env: - name: CONJUR_APPLIANCE_URL value: "https://conjur.myorg.com:32149" - name: CONJUR_ACCOUNT value: default - name: CONJUR_AUTHN_TOKEN_FILE value: /run/conjur/access-token - name: CONJUR_SSL_CERTIFICATE valueFrom: configMapKeyRef: name: conjur-cert key: ssl-certificate volumeMounts: - mountPath: /run/conjur name: conjur-access-token readOnly: true initContainers: - image: cyberark/conjur-authn-k8s-client imagePullPolicy: Always name: authenticator env: - name: CONTAINER_MODE value: sidecar - name: MY_POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: MY_POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: MY_POD_IP valueFrom: fieldRef: fieldPath: status.podIP - name: CONJUR_AUTHN_URL value: "https://conjur.myorg.com:32149/authn-k8s/sidecar" - name: CONJUR_ACCOUNT value: default - name: CONJUR_AUTHN_LOGIN value: "host/conjur/authn-k8s/sidecar/apps/sidecar-demo/sidecar-demo-app" - name: CONJUR_SSL_CERTIFICATE valueFrom: configMapKeyRef: name: conjur-cert key: ssl-certificate volumeMounts: - mountPath: /run/conjur name: conjur-access-token imagePullSecrets: - name: dockerpullsecret volumes: - name: conjur-access-token emptyDir: medium: Memory
This is the state of my sidecar-demo-app pods:
Name: sidecar-demo-app-75ff75fd49-xfz46 Namespace: sidecar-demo Priority: 0 Node: k8s-kubespray-worker-0/172.16.220.56 Start Time: Thu, 10 Sep 2020 13:53:14 +0200 Labels: app=sidecar-demo-app pod-template-hash=75ff75fd49 Annotations: <none> Status: Pending IP: 10.233.65.137 IPs: <none> Controlled By: ReplicaSet/sidecar-demo-app-75ff75fd49 Init Containers: authenticator: Container ID: docker://bd6d0ba1a411da3e176de28f539e65c8d64bb1f92acf4390741154262e9ad947 Image: cyberark/conjur-authn-k8s-client Image ID: docker-pullable://cyberark/conjur-authn-k8s-client@sha256:3e8e9e8df1bb5d739dc8ab905f9547eec43019563313f6a2dee4ee3455d0ab2c Port: <none> Host Port: <none> State: Terminated Reason: Error Exit Code: 1 Started: Thu, 10 Sep 2020 14:03:20 +0200 Finished: Thu, 10 Sep 2020 14:05:29 +0200 Last State: Terminated Reason: Error Exit Code: 1 Started: Thu, 10 Sep 2020 14:00:27 +0200 Finished: Thu, 10 Sep 2020 14:02:36 +0200 Ready: False Restart Count: 4 Environment: CONTAINER_MODE: sidecar MY_POD_NAME: sidecar-demo-app-75ff75fd49-xfz46 (v1:metadata.name) MY_POD_NAMESPACE: sidecar-demo (v1:metadata.namespace) MY_POD_IP: (v1:status.podIP) CONJUR_AUTHN_URL: https://conjur.myorg.com:32149/authn-k8s/sidecar CONJUR_ACCOUNT: default CONJUR_AUTHN_LOGIN: host/conjur/authn-k8s/sidecar/apps/sidecar-demo/sidecar-demo-app CONJUR_SSL_CERTIFICATE: <set to the key 'ssl-certificate' of config map 'conjur-cert'> Optional: false Mounts: /run/conjur from conjur-access-token (rw) /var/run/secrets/kubernetes.io/serviceaccount from sidecar-service-account-token-s8k6r (ro) Containers: sidecar-demo-app: Container ID: Image: mysql:5.6 Image ID: Port: 8080/TCP Host Port: 0/TCP Args: mysql --host=vm-mariadb.default.svc.cluster.local --user=root --password=5jqJL2b8hqn3 State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Environment: CONJUR_APPLIANCE_URL: https://conjur.myorg.com:32149 CONJUR_ACCOUNT: default CONJUR_AUTHN_TOKEN_FILE: /run/conjur/access-token CONJUR_SSL_CERTIFICATE: <set to the key 'ssl-certificate' of config map 'conjur-cert'> Optional: false Mounts: /run/conjur from conjur-access-token (ro) /var/run/secrets/kubernetes.io/serviceaccount from sidecar-service-account-token-s8k6r (ro) Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: conjur-access-token: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: Memory SizeLimit: <unset> sidecar-service-account-token-s8k6r: Type: Secret (a volume populated by a Secret) SecretName: sidecar-service-account-token-s8k6r Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 12m default-scheduler Successfully assigned sidecar-demo/sidecar-demo-app-75ff75fd49-xfz46 to k8s-kubespray-worker-0 Normal Pulling 2m13s (x5 over 12m) kubelet, k8s-kubespray-worker-0 Pulling image "cyberark/conjur-authn-k8s-client" Normal Pulled 2m12s (x5 over 12m) kubelet, k8s-kubespray-worker-0 Successfully pulled image "cyberark/conjur-authn-k8s-client" Normal Created 2m12s (x5 over 12m) kubelet, k8s-kubespray-worker-0 Created container authenticator Normal Started 2m12s (x5 over 12m) kubelet, k8s-kubespray-worker-0 Started container authenticator Warning BackOff 3s (x7 over 7m54s) kubelet, k8s-kubespray-worker-0 Back-off restarting failed container
Also, I am unsure if I correctly configured the environment variables:
CONJUR_APPLIANCE_URL=https://conjur.myorg.com:32149CONJUR_AUTHN_URL=https://conjur.myorg.com:32149/authn-k8s/sidecarCONJUR_AUTHN_LOGIN=host/conjur/authn-k8s/sidecar/apps/sidecar-demo/sidecar-demo-app
And for CONJUR_AUTHENTICATORS, how can I make sure auth-k8s/sidecar has been added? If it hasn’t, how can I add it?
Any help will be greatly appreciated!
Jivana