Conjur follower installation in minikube cluster

Secrets Management - Conjur, Secrets Hub & CP
1

Hi Team,

I am trying to install conjur followers in minikube cluster environment.

NAME READY STATUS RESTARTS AGE
conjur-follower-586f47c89-w6wb8 0/1 Running 1 22h
conjur-follower-586f47c89-xjxqr 0/1 Running 1 22h

I am getting readiness probe failed :
Events:
Type Reason Age From Message

Warning Unhealthy 4m41s (x3590 over 20h) kubelet, minikube Readiness probe failed: Get https://172.17.0.6:443/health: dial tcp 172.17.0.6:443: connect: connection refused.

Any help or suggestions on this would be helpful

Thanks,
Shubham

2

Hello Shubham,

It would be helpful to get a little more information about your deployment. Could you post your deployment manifest, and the output of the container logs for the authenticator container? It may be called authenticator, or seed-fetcher, or something like that.

kubectl logs -n <namespace> conjur-follower-586f47c89-w6wb8 -c authenticator

Thanks,

Ben Floyd

3

Hello @benfloyd

Thanks for your reply,

Below are the logs:
shubham@minikube:~$ kubectl describe deployments
Name: conjur-follower
Namespace: conjur
CreationTimestamp: Mon, 27 Jul 2020 21:32:20 -0700
Labels: app=conjur-follower
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=conjur-follower
Replicas: 2 desired | 2 updated | 2 total | 0 available | 2 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=conjur-follower
name=conjur-follower
role=follower
Service Account: conjur-cluster
Init Containers:
authenticator:
Image: docker.io/shubhamindia/seed-fetcher:conjur
Port:
Host Port:
Environment:
CONJUR_SEED_FILE_URL: /tmp/follower-seed.tar
SEEDFILE_DIR: /tmp/seedfile
FOLLOWER_HOSTNAME: conjur-follower
AUTHENTICATOR_ID: staging
CONJUR_ACCOUNT: CAU
CONJUR_SSL_CERTIFICATE: <set to the key ‘ssl-certificate’ of config map ‘server-certificate’> Optional: false
MY_POD_NAME: (v1:metadata.name)
MY_POD_NAMESPACE: (v1:metadata.namespace)
MY_POD_IP: (v1:status.podIP)
CONJUR_AUTHN_LOGIN: host/conjur/authn-k8s/staging/apps/conjur/service_account/conjur-cluster
Mounts:
/run/conjur from conjur-token (rw)
/tmp/seedfile from seedfile (rw)
Containers:
conjur-appliance:
Image: docker.io/shubhamindia/conjur-appliance:conjur
Ports: 443/TCP, 636/TCP, 5432/TCP, 5433/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
Command:
/tmp/seedfile/start-follower.sh
Readiness: http-get https://:443/health delay=15s timeout=5s period=10s #success=1 #failure=3
Environment:
CONJUR_AUTHENTICATORS: authn-k8s/staging
SEEDFILE_DIR: /tmp/seedfile
Mounts:
/tmp/seedfile from seedfile (ro)
Volumes:
seedfile:
Type: EmptyDir (a temporary directory that shares a pod’s lifetime)
Medium: Memory
SizeLimit:
conjur-token:
Type: EmptyDir (a temporary directory that shares a pod’s lifetime)
Medium: Memory
SizeLimit:
Conditions:
Type Status Reason

Available False MinimumReplicasUnavailable
Progressing False ProgressDeadlineExceeded
OldReplicaSets: conjur-follower-586f47c89 (2/2 replicas created)
NewReplicaSet:
Events:

Also, attached are the follower logs:
shubham@minikube:~$ kubectl describe pod conjur-follower-586f47c89-w6wb8
Name: conjur-follower-586f47c89-w6wb8
Namespace: conjur
Priority: 0
Node: minikube/192.168.99.100
Start Time: Mon, 27 Jul 2020 21:32:21 -0700
Labels: app=conjur-follower
name=conjur-follower
pod-template-hash=586f47c89
role=follower
Annotations:
Status: Running
IP: 172.17.0.6
IPs:
IP: 172.17.0.6
Controlled By: ReplicaSet/conjur-follower-586f47c89
Init Containers:
authenticator:
Container ID: docker://36a3449c1d8539ab731483f9fbc5f3ce57aa4787ce9a199ae24bbb5d9a65148b
Image: docker.io/shubhamindia/seed-fetcher:conjur
Image ID: docker-pullable://shubhamindia/seed-fetcher@sha256:d6e1bc12c3dfc2b592213fd2c7d8f0cf250e9c2deee92d2f5e398677facc37ff
Port:
Host Port:
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 29 Jul 2020 20:32:58 -0700
Finished: Wed, 29 Jul 2020 20:33:00 -0700
Ready: True
Restart Count: 2
Environment:
CONJUR_SEED_FILE_URL: /tmp/follower-seed.tar
SEEDFILE_DIR: /tmp/seedfile
FOLLOWER_HOSTNAME: conjur-follower
AUTHENTICATOR_ID: staging
CONJUR_ACCOUNT: CAU
CONJUR_SSL_CERTIFICATE: <set to the key ‘ssl-certificate’ of config map ‘server-certificate’> Optional: false
MY_POD_NAME: conjur-follower-586f47c89-w6wb8 (v1:metadata.name)
MY_POD_NAMESPACE: conjur (v1:metadata.namespace)
MY_POD_IP: (v1:status.podIP)
CONJUR_AUTHN_LOGIN: host/conjur/authn-k8s/staging/apps/conjur/service_account/conjur-cluster
Mounts:
/run/conjur from conjur-token (rw)
/tmp/seedfile from seedfile (rw)
/var/run/secrets/kubernetes.io/serviceaccount from conjur-cluster-token-q6grp (ro)
Containers:
conjur-appliance:
Container ID: docker://e8365551cc7cae8737deb4881671ba16b0038d7a1efe41e5735cf8e3bf4d0a2d
Image: docker.io/shubhamindia/conjur-appliance:conjur
Image ID: docker-pullable://shubhamindia/conjur-appliance@sha256:f40fe85d690e0ac88d0e3979d569a28d4ce474991558c73eee06e5ea6149fc15
Ports: 443/TCP, 636/TCP, 5432/TCP, 5433/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
Command:
/tmp/seedfile/start-follower.sh
State: Running
Started: Wed, 29 Jul 2020 20:33:14 -0700
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Tue, 28 Jul 2020 00:11:43 -0700
Finished: Wed, 29 Jul 2020 20:30:34 -0700
Ready: False
Restart Count: 2
Readiness: http-get https://:443/health delay=15s timeout=5s period=10s #success=1 #failure=3
Environment:
CONJUR_AUTHENTICATORS: authn-k8s/staging
SEEDFILE_DIR: /tmp/seedfile
Mounts:
/tmp/seedfile from seedfile (ro)
/var/run/secrets/kubernetes.io/serviceaccount from conjur-cluster-token-q6grp (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
seedfile:
Type: EmptyDir (a temporary directory that shares a pod’s lifetime)
Medium: Memory
SizeLimit:
conjur-token:
Type: EmptyDir (a temporary directory that shares a pod’s lifetime)
Medium: Memory
SizeLimit:
conjur-cluster-token-q6grp:
Type: Secret (a volume populated by a Secret)
SecretName: conjur-cluster-token-q6grp
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message

Warning Unhealthy 4m44s (x682 over 118m) kubelet, minikube Readiness probe failed: Get https://172.17.0.6:443/health: dial tcp 172.17.0.6:443: connect: connection refused

Thanks,
Shubham

4

Hey Shubham,

I see a few issues with this manifest, and I’m trying to figure out the best way to give you a working manifest. I’m going to note some of them here and hopefully it fixes your issues. Without looking at both the Conjur and kubernetes environments and all the variable settings that tie it together, it’s hard to know if things are set up correctly. Anyway, see below and try it out and let me know!

  • CONJUR_SEED_FILE_URL
    • This is a URL: https://{{ DAP_HOSTNAME }}/configuration/{{ DAP_ACCT }}/seed/follower
  • The host ports look strange… did you scrub those?
  • What version of the appliance image is this? It is tagged as :conjur so I can’t see it.

HTH,

Ben Floyd

5

Hi Ben,

Thanks for your reply,
Hope you are doing good

  • I had followed the steps from the docs and used “https://github.com/cyberark/kubernetes-conjur-deploy” to deploy follower.

  • The seed file that is generated on the conjur-master server is copied locally on the server where minikube cluster is running and it is under /tmp/ folder.

  • Ran the ./start script from github to deploy the followers automatically.

  • Did nothing with the ports just reran the ./start script once as after restarting vm pods were not showing up and after running the script again it showed the same pods which were created earlier with the same issue.

  • Version of conjur appliance image is 5.2.7.

Regards,
Shubham