Authenticator Client Error

somitnirmata · July 31, 2022, 4:38pm

I have deployed conjur open source and Kubernetes Authenticator as well. While deploying below yaml in cluster i get Authenticator Client Error Logs (mentioned below).

Test yaml Deployed:

apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: test-app
name: test-app
namespace: test-app-namespace
spec:
selector:
matchLabels:
app: test-app
replicas: 1
template:
metadata:
labels:
app: test-app
spec:
serviceAccountName: test-app-sa
containers:
- name: test-app
image: nginx:latest
ports:
- containerPort: 80
envFrom:
- configMapRef:
name: conjur-connect
volumeMounts:
- mountPath: /run/conjur
name: conjur-access-token
readOnly: true
initContainers:
- image: cyberark/conjur-authn-k8s-client
imagePullPolicy: Always
name: authenticator
env:
- name: CONJUR_AUTHN_LOGIN
value: host/test-app
- name: CONTAINER_MODE
value: init
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
envFrom:
- configMapRef:
name: conjur-connect
volumeMounts:
- mountPath: /run/conjur
name: conjur-access-token
- mountPath: /etc/conjur/ssl
name: conjur-client-ssl
volumes:
- name: conjur-access-token
emptyDir:
medium: Memory
- name: conjur-client-ssl
emptyDir:
medium: Memory

Authenticator Client Error Logs:

INFO: 2022/07/31 14:53:28.150378 main.go:19: CAKC048 Kubernetes Authenticator Client v0.23.1-dev starting up…
INFO: 2022/07/31 14:53:28.150541 configuration_factory.go:79: CAKC070 Chosen “authn-k8s” configuration
INFO: 2022/07/31 14:53:28.150732 authenticator_factory.go:31: CAKC075 Chosen “authn-k8s” flow
ERROR: 2022/07/31 14:53:28.327146 client.go:17: CAKC014 Failed to append Conjur CA cert
ERROR: 2022/07/31 14:53:28.327295 main.go:72: CAKC019 Failed to instantiate Authenticator object

https://docs.conjur.org/Latest/en/Content/Integrations/k8s-ocp/k8s-set-up-apps.htm

While setting up the Authenticator using below helm install command, the the CM conjur-configmap data “conjurSslCertificate and conjurSslCertificateBase64” shows empty. I injected the conjurSslCertificate manually so

helm install cluster-prep cyberark/conjur-config-cluster-prep
–namespace cyberark-conjur
–create-namespace
–set conjur.account=“myorg”
–set conjur.applianceUrl=“https://conjur-follower.cyberark-conjur.svc.cluster.local”
–set conjur.certificateFilePath=“/fullchain.pem”
–set authnK8s.authenticatorID=“dev-cluster”
–set authnK8s.serviceAccount.name=“authn-k8s-sa”

I injected the conjurSslCertificate manually so that while setting apps (Set up applications (cert-based authentication)) below steps won’t get fail.

helm install namespace-prep cyberark/conjur-config-namespace-prep
–namespace test-apps-namespace
–set authnK8s.goldenConfigMap=“conjur-configmap”
–set authnK8s.namespace=“cyberark-conjur”

This looks with certification issue. Is there anyway to know we are giving the right conjur.certificateFilePath one while helm install cluster-prep or anything other steps to do for configurations.

Thanks in advance

Regards,
Somit

rpothier · August 1, 2022, 3:11pm

Hi @somitnirmata
Are you seeing any Helm errors? Were you seeing Helm errors to begin with that required setting theconjurSslCertificate manually? When Helm installing with the remote helm repo cyberark/conjur-config-cluster-prep the cert needs to be B64 encoded, and I would expect Helm to give this error - “If you are using helm install with a (remote) chart reference, please use conjur.certficateBase64 instead of conjur.certificateFilePath.”
Are you seeing that error?

Hope this help - Rob

somitnirmata · August 2, 2022, 12:48pm

Hi Rob,

Thanks for the reply.

Yes. I see error when i do below helm install, when conjur-configmap has no conjurSslCertificate data. So inorder to avoid this we manullay injected ssl into this

helm install namespace-prep cyberark/conjur-config-namespace-prep
–namespace test-apps-namespace
–set authnK8s.goldenConfigMap=“conjur-configmap”
–set authnK8s.namespace=“cyberark-conjur”

Is it possible for you tell why the conjurSslCertificate: kept empty?
apiVersion: v1
data:
authnK8sAuthenticatorID: dev-cluster
authnK8sClusterRole: conjur-clusterrole
authnK8sNamespace: cyberark-conjur
authnK8sServiceAccount: authn-k8s-sa
conjurAccount: myConjurAccount
conjurApplianceUrl: https://conjur123.lab.nirmata.co
conjurSslCertificate: “”
conjurSslCertificateBase64: “”

Thanks,
Somit

rpothier · August 2, 2022, 3:35pm

Hi Somit,
Can you try creating the conjur-configmap with base64 encoding ?
So

helm install my-conjur-release . \
     --set conjur.certificateBase64="$(base64 -w0 path/to/conjur.pem)" \
     ...

Rob

somitnirmata · August 3, 2022, 1:43pm

Hi Rob,

Tried the above, but gets a syntax error.

helm install cluster-prep cyberark/conjur-config-cluster-prep
–namespace cyberark-conjur
–create-namespace
–set conjur.account=“myConjurAccount”
–set conjur.applianceUrl=“https://url”
–set conjur.certificateFilePath=“/Users/somitsebastian/conjur/authenticator-setup-2/pem/conjur.pem”
–set conjur.certificateBase64=“$(base64 -w0 /Users/somitsebastian/conjur/authenticator-setup-2/pem/conjur.pem)”
–set authnK8s.authenticatorID=“dev-cluster”
–set authnK8s.serviceAccount.name=“authn-k8s-sa”
base64: invalid option – w
Usage: base64 [-hvDd] [-b num] [-i in_file] [-o out_file]
-h, --help display this message
-Dd, --decode decodes input
-b, --break break encoded string into num character lines
-i, --input input file (default: “-” for stdin)
-o, --output output file (default: “-” for stdout)
Error: INSTALLATION FAILED: values don’t meet the specifications of the schema(s) in the following chart(s):
conjur-config-cluster-prep:

conjur.certificateBase64: String length must be greater than or equal to 1

Its saying base64: invalid option – w and specification error. By any chance you meant base64 -d. In that case I have tried that also and getting specification error:

helm install cluster-prep cyberark/conjur-config-cluster-prep
–namespace cyberark-conjur
–create-namespace
–set conjur.account=“myConjurAccount”
–set conjur.applianceUrl=“https://url”
–set conjur.certificateFilePath=“/Users/somitsebastian/conjur/authenticator-setup-2/pem/conjur.pem”
–set conjur.certificateBase64=“$(base64 -d /Users/somitsebastian/conjur/authenticator-setup-2/pem/conjur.pem)”
–set authnK8s.authenticatorID=“dev-cluster”
–set authnK8s.serviceAccount.name=“authn-k8s-sa”
Error: INSTALLATION FAILED: values don’t meet the specifications of the schema(s) in the following chart(s):
conjur-config-cluster-prep:

conjur.certificateBase64: Does not match pattern ‘^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$’
Somit

rpothier · August 8, 2022, 4:58pm

Hi Somit,
are you on a Mac? On Linux I have the -w0 option, but I don’t see the -w0 on a mac.

Rob

rpothier · August 10, 2022, 3:05pm

Also if you download the Helm charts you can use the cert directly, you won’t have to base64 encode if that is an issue.

Rob

somitnirmata · September 16, 2022, 11:20am

Hi Rob,

I was able to resolve the issue using below:

helm install cluster-prep8 cyberark/conjur-config-cluster-prep
–namespace cyberark-conjur
–create-namespace
*–set conjur.certificateBase64=“$(cat fullchain.pem | base64)” *
–set conjur.account=“myConjurAccount”
–set conjur.applianceUrl=“https://conjur123.lab.nirmata.co”
–set authnK8s.authenticatorID=“dev-cluster”
–set authnK8s.serviceAccount.name=“authn-k8s-sa”

Since I was using Mac I used --set conjur.certificateBase64=“$(cat fullchain.pem | base64)”.

Now I have data populated in conjurSslCertificate & conjurSslCertificateBase64. Also created a ConfigMap and a RoleBinding in the test-app-namespace namespace using details from the Golden ConfigMap, conjur-configmap which is located in the cyberark-conjur namespace
helm install namespace-prep cyberark/conjur-config-namespace-prep
–namespace test-apps-namespace
–create-namespace
–set authnK8s.goldenConfigMap=“conjur-configmap”
–set authnK8s.namespace=“cyberark-conjur”

As the next time I tried deploying the deployment file where I get auth error. Before the deploy I have setup policies and secrets as mentioned in the doc.

https://docs.conjur.org/Latest/en/Content/Integrations/k8s-ocp/cjr-k8s-authn-client-authk8s.htm?tocpath=Integrations|OpenShift%252FKubernetes|App%20owner%253A%20Set%20up%20workloads%20in%20Kubernetes|Set%20up%20workloads%20(cert-based%20authn)|_____1#Step5

Auth Error shows in authenticator container:
ERROR: 2022/09/16 11:19:21.047049 main.go:49: CAKC016 Failed to authenticate
INFO: 2022/09/16 11:19:29.549622 authenticator.go:84: CAKC040 Authenticating as user ‘host/test-app’
ERROR: 2022/09/16 11:19:29.698850 authenticator.go:185: CAKC029 Received invalid response to certificate signing request. Reason: status code 401,
ERROR: 2022/09/16 11:19:29.698937 authenticator.go:271: CAKC015 Login failed

Manifest File:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: test-app
name: test-app
namespace: test-apps-namespace
spec:
selector:
matchLabels:
app: test-app
replicas: 1
template:
metadata:
labels:
app: test-app
spec:
serviceAccountName: test-app-sa
containers:
- name: test-app
image: nginx:latest
ports:
- containerPort: 8080
envFrom:
- configMapRef:
name: conjur-connect
volumeMounts:
- mountPath: /run/conjur
name: conjur-access-token
readOnly: true
initContainers:
- image: cyberark/conjur-authn-k8s-client
imagePullPolicy: Always
name: authenticator
env:
- name: CONJUR_AUTHN_LOGIN
value: host/test-app
- name: CONTAINER_MODE
value: init
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
envFrom:
- configMapRef:
name: conjur-connect
volumeMounts:
- mountPath: /run/conjur
name: conjur-access-token
- mountPath: /etc/conjur/ssl
name: conjur-client-ssl
volumes:
- name: conjur-access-token
emptyDir:
medium: Memory
- name: conjur-client-ssl
emptyDir:
medium: Memory

The input for conjur.certificateBase64 is same as my-conjur-oss-conjur-ssl-cert tls.crt

somitnirmata · September 22, 2022, 7:53am

Hi Team,

Can anyone help with the above query?

Thanks in advance

Somit

rpothier · September 22, 2022, 8:31pm

Hi Somit, The manifest looks give like the example. Did you try setting the CONJUR_LOG_LEVEL env variable to debug? If you used the conjur-oss helm chart you can set the env variable in conjur-oss/values.yaml. That might get a better hint of why it is failing.

somitnirmata · September 27, 2022, 10:20am

Hi Rob,
Please see the logs file of conjur-oss after setting the CONJUR_LOG_LEVEL to debug:
/var/lib/ruby/lib/ruby/gems/2.5.0/gems/railties-5.2.4.4/lib/rails/rack/logger.rb:26:in block in call' [origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/activesupport-5.2.4.4/lib/active_support/tagged_logging.rb:71:in block in tagged’
e[1me[35mSequel::Postgres::Database (2.7ms)e[0m SELECT pg_try_advisory_lock(767003715) AS “v” LIMIT 1
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/activesupport-5.2.4.4/lib/active_support/tagged_logging.rb:28:in `tagged’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/activesupport-5.2.4.4/lib/active_support/tagged_logging.rb:71:in `tagged’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/railties-5.2.4.4/lib/rails/rack/logger.rb:26:in `call’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.4/lib/action_dispatch/middleware/request_id.rb:27:in `call’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/activesupport-5.2.4.4/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.4/lib/action_dispatch/middleware/executor.rb:14:in `call’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.4/lib/action_dispatch/middleware/static.rb:127:in `call’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/conjur-rack-heartbeat-2.2.0/lib/rack/heartbeat.rb:20:in `call’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /opt/conjur-server/lib/rack/remove_request_parameters.rb:26:in `call’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /opt/conjur-server/lib/rack/default_content_type.rb:68:in `call’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/railties-5.2.4.4/lib/rails/engine.rb:524:in `call’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/puma-3.12.6/lib/puma/configuration.rb:227:in `call’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/puma-3.12.6/lib/puma/server.rb:706:in `handle_request’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/puma-3.12.6/lib/puma/server.rb:476:in `process_client’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/puma-3.12.6/lib/puma/server.rb:334:in `block in run’
[origin=192.168.236.147] [request_id=ba7dece8-aca4-4e4e-832d-2e4498435a2f] [tid=40] /var/lib/ruby/lib/ruby/gems/2.5.0/gems/puma-3.12.6/lib/puma/thread_pool.rb:135:in `block in spawn_thread’
e[1me[36mSequel::Postgres::Database (45.8ms)e[0m e[1m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl