Failed to produce seed file deploying followers in k8s cluster Cert-based auth

Secrets Management - Conjur, Secrets Hub & CP Conjur Enterprise
,
1

hi
we have a problem with deploying followers in k8s cluster Cert-based auth.
we see the problem from the cluster side:
<134>1 2023-03-07T17:56:00.000+00:00 b1f73893dca0 evoke-seed - - [meta sequenceId=“180”] E, [2023-03-07T17:56:00.289535 #81165] ERROR – : Failed to produce seed file: root key not present
but there is nothing about “failed” status on the conjur master side.
we added a certificate to the master referring to the instruction (Certificate requirements) and where: CN=service.namespace.svc.cluster.local

no seed was generated therefore followers.

KR

2

Hello

Has anyone had such a problem. what could this error be related to?
<15>1 2023-03-09T20:34:54.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“174”] [origin=10.10.10.249] [request_id=018d9cf5-67ba-476e-ba34-157acd26cf44] [tid=291277] CONJ00027D Host id cainteger:host:conjur-follower extracted from CSR common name
<15>1 2023-03-09T20:34:54.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“175”] [origin=10.10.10.249] [request_id=018d9cf5-67ba-476e-ba34-157acd26cf44] [tid=291277] Sequel::Postgres::Database (0.3ms) SELECT * FROM “resources” WHERE “resource_id” = ‘cainteger:host:conjur-follower’
<15>1 2023-03-09T20:34:54.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“176”] [origin=10.10.10.249] [request_id=018d9cf5-67ba-476e-ba34-157acd26cf44] [tid=291277] Sequel::Postgres::Database (0.3ms) SELECT * FROM “annotations” WHERE (“annotations”.“resource_id” = ‘cainteger:host:conjur-follower’)
<15>1 2023-03-09T20:34:54.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“177”] [origin=10.10.10.249] [request_id=018d9cf5-67ba-476e-ba34-157acd26cf44] [tid=291277] CONJ00024I Retrieved value of annotation ‘authn-k8s/authentication-container-name’
<15>1 2023-03-09T20:34:54.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“178”] [origin=10.10.10.249] [request_id=018d9cf5-67ba-476e-ba34-157acd26cf44] [tid=291277] CONJ00024I Retrieved value of annotation ‘authn-k8s/authentication-container-name’
<15>1 2023-03-09T20:34:54.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“179”] [origin=10.10.10.249] [request_id=018d9cf5-67ba-476e-ba34-157acd26cf44] [tid=291277] CONJ00024I Retrieved value of annotation ‘authn-k8s/authentication-container-name’
<15>1 2023-03-09T20:34:54.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“180”] [origin=10.10.10.249] [request_id=018d9cf5-67ba-476e-ba34-157acd26cf44] [tid=291277] CONJ00024I Retrieved value of annotation ‘authn-k8s/authentication-container-name’
<15>1 2023-03-09T20:34:54.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“181”] [origin=10.10.10.249] [request_id=018d9cf5-67ba-476e-ba34-157acd26cf44] [tid=291277] CONJ00024I Retrieved value of annotation ‘authn-k8s/authentication-container-name’
<15>1 2023-03-09T20:34:54.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“182”] [origin=10.10.10.249] [request_id=018d9cf5-67ba-476e-ba34-157acd26cf44] [tid=291277] CONJ00024I Retrieved value of annotation ‘authn-k8s/authentication-container-name’
<15>1 2023-03-09T20:34:54.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“183”] [origin=10.10.10.249] [request_id=018d9cf5-67ba-476e-ba34-157acd26cf44] [tid=291277] Sequel::Postgres::Database (0.2ms) SELECT * FROM “resources” WHERE “resource_id” = ‘cainteger:variable:conjur/authn-k8s/dev-cluster/ca/cert’
<15>1 2023-03-09T20:34:54.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“184”] [origin=10.10.10.249] [request_id=018d9cf5-67ba-476e-ba34-157acd26cf44] [tid=291277] Sequel::Postgres::Database (0.2ms) SELECT * FROM “resources” WHERE “resource_id” = ‘cainteger:variable:conjur/authn-k8s/dev-cluster/ca/key’
<15>1 2023-03-09T20:34:54.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“185”] [origin=10.10.10.249] [request_id=018d9cf5-67ba-476e-ba34-157acd26cf44] [tid=291277] Sequel::Postgres::Database (0.2ms) SELECT * FROM “secrets” WHERE (“secrets”.“resource_id” = ‘cainteger:variable:conjur/authn-k8s/dev-cluster/ca/cert’) ORDER BY “version” DESC LIMIT 1
<15>1 2023-03-09T20:34:54.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“186”] [origin=10.10.10.249] [request_id=018d9cf5-67ba-476e-ba34-157acd26cf44] [tid=291277] Sequel::Postgres::Database (0.2ms) SELECT * FROM “secrets” WHERE (“secrets”.“resource_id” = ‘cainteger:variable:conjur/authn-k8s/dev-cluster/ca/key’) ORDER BY “version” DESC LIMIT 1
<15>1 2023-03-09T20:34:54.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“187”] [origin=10.10.10.249] [request_id=018d9cf5-67ba-476e-ba34-157acd26cf44] [tid=291277] Sequel::Postgres::Database (0.2ms) SELECT * FROM “slosilo_keystore” WHERE “id” = ‘authn:cainteger’
<15>1 2023-03-09T20:34:54.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“188”] [origin=10.10.10.249] [request_id=018d9cf5-67ba-476e-ba34-157acd26cf44] [tid=291277] Sequel::Postgres::Database (0.2ms) SELECT * FROM “roles” WHERE (“role_id” = ‘cainteger:host:conjur-follower’) LIMIT 1
<15>1 2023-03-09T20:34:54.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“189”] [origin=10.10.10.249] [request_id=018d9cf5-67ba-476e-ba34-157acd26cf44] [tid=291277] Sequel::Postgres::Database (0.1ms) COMMIT
<14>1 2023-03-09T20:34:54.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“190”] [origin=10.10.10.249] [request_id=018d9cf5-67ba-476e-ba34-157acd26cf44] [tid=291277] Completed 200 OK in 126ms (Views: 0.2ms | Allocations: 21011)
<14>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“191”] [origin=127.0.0.1] [request_id=930cf066-8613-44fb-a2c6-9fcca07d0830] [tid=291277] Started GET “/resources/cainteger/webservice/conjur%2Fseed-generation/?check=true&privilege=execute” for 127.0.0.1 at 2023-03-09 20:34:55 +0000
<15>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“192”] [origin=127.0.0.1] [request_id=930cf066-8613-44fb-a2c6-9fcca07d0830] [tid=291277] Sequel::Postgres::Database (0.9ms) SELECT * FROM “slosilo_keystore” WHERE (“fingerprint” = ‘b63d49dbf658719e93742d4bd7baf08d10abfca8c9de688c9c45900a2f394818’) LIMIT 1
<14>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“193”] [origin=127.0.0.1] [request_id=930cf066-8613-44fb-a2c6-9fcca07d0830] [tid=291277] Processing by ResourcesController#check_permission as /
<14>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“194”] [origin=127.0.0.1] [request_id=930cf066-8613-44fb-a2c6-9fcca07d0830] [tid=291277] Parameters: {“check”=>“true”, “privilege”=>“execute”, :controller=>“resources”, :action=>“check_permission”, :account=>“cainteger”, :kind=>“webservice”, :identifier=>“conjur/seed-generation”}
<15>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“195”] [origin=127.0.0.1] [request_id=930cf066-8613-44fb-a2c6-9fcca07d0830] [tid=291277] Sequel::Postgres::Database (0.2ms) BEGIN
<15>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“196”] [origin=127.0.0.1] [request_id=930cf066-8613-44fb-a2c6-9fcca07d0830] [tid=291277] Sequel::Postgres::Database (0.5ms) SELECT * FROM “roles” WHERE “role_id” = ‘cainteger:host:conjur-follower’
<15>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“197”] [origin=127.0.0.1] [request_id=930cf066-8613-44fb-a2c6-9fcca07d0830] [tid=291277] Sequel::Postgres::Database (0.4ms) SELECT * FROM “resources” WHERE “resource_id” = ‘cainteger:webservice:conjur/seed-generation’
<15>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“198”] [origin=127.0.0.1] [request_id=930cf066-8613-44fb-a2c6-9fcca07d0830] [tid=291277] Sequel::Postgres::Database (1.1ms) SELECT is_resource_visible(‘cainteger:webservice:conjur/seed-generation’, ‘cainteger:host:conjur-follower’) LIMIT 1
<15>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“199”] [origin=127.0.0.1] [request_id=930cf066-8613-44fb-a2c6-9fcca07d0830] [tid=291277] Sequel::Postgres::Database (1.1ms) SELECT * FROM is_role_allowed_to(‘cainteger:host:conjur-follower’, ‘execute’, ‘cainteger:webservice:conjur/seed-generation’) LIMIT 1
<15>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“200”] [origin=127.0.0.1] [request_id=930cf066-8613-44fb-a2c6-9fcca07d0830] [tid=291277] Sequel::Postgres::Database (0.1ms) COMMIT
<14>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 conjur-possum 214754 - [meta sequenceId=“201”] [origin=127.0.0.1] [request_id=930cf066-8613-44fb-a2c6-9fcca07d0830] [tid=291277] Completed 204 No Content in 10ms (Allocations: 794)
<134>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 evoke-seed - - [meta sequenceId=“202”] E, [2023-03-09T20:34:55.051271 #81165] ERROR – : Failed to produce seed file: root key not present
<134>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 evoke-seed - - [meta sequenceId=“203”] 2023-03-09 20:34:55 - Evoke::CertStore::NoRootKeyError - root key not present:
<134>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 evoke-seed - - [meta sequenceId=“204”] /opt/conjur/evoke/lib/evoke/cert_store.rb:214:in create_cert' <134>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 evoke-seed - - [meta sequenceId="205"] /opt/conjur/evoke/lib/evoke/cert_store.rb:110:in x509_cert_for’
<134>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 evoke-seed - - [meta sequenceId=“206”] /opt/conjur/evoke/lib/evoke/cert_store.rb:102:in cert_for' <134>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 evoke-seed - - [meta sequenceId="207"] /opt/conjur/evoke/lib/evoke/action/seed/base.rb:79:in cert’
<134>1 2023-03-09T20:34:55.000+00:00 b1f73893dca0 evoke-seed - - [meta sequenceId=“208”] /opt/conjur/evoke/lib/evoke/action/seed/follower.rb:17:in `block in class:Follower’

3

Hi @Jakub ,

we see the problem from the cluster side:
<134>1 2023-03-07T17:56:00.000+00:00 b1f73893dca0 evoke-seed - - [meta sequenceId=“180”] E, [2023-03-07T17:56:00.289535 #81165] ERROR – : Failed to produce seed file: root key not present
but there is nothing about “failed” status on the conjur master side.

Can you clarify what you mean by “from the cluster side”? Is this not from the Leader’s log?

If this is, in fact, from the Leader logs. This suggests that there is a mismatch between the hostname requested of the seed service and the certificates imported into the Leader.

One troubleshooting step you can try is to list the certificates currently imported on the Conjur leader with:

docker exec <conjur-container-name> ls -la /opt/conjur/etc/ssl

For example:

$ docker exec <conjur-container-name> ls -la /opt/conjur/etc/ssl
total 40
drwxr-xr-x 1 root   ssl-cert 4096 Mar 13 13:20 .
drwxr-xr-x 1 conjur root     4096 Mar 13 13:17 ..
lrwxrwxrwx 1 root   root        6 Mar 13 13:19 c493b61a.0 -> ca.pem
-rw-r--r-- 1 root   root     2920 Mar 13 13:19 ca.pem
-rw-r----- 1 root   ssl-cert 1708 Mar 13 13:20 conjur-follower.mycompany.local.key
-rw-r--r-- 1 root   root     4486 Mar 13 13:20 conjur-follower.mycompany.local.pem
-rw-r----- 1 root   ssl-cert 1704 Mar 13 13:20 conjur-master.mycompany.local.key
-rw-r--r-- 1 root   root     4616 Mar 13 13:20 conjur-master.mycompany.local.pem
lrwxrwxrwx 1 root   root       53 Mar 13 13:20 conjur.key -> /opt/conjur/etc/ssl/conjur-master.mycompany.local.key
lrwxrwxrwx 1 root   root       53 Mar 13 13:20 conjur.pem -> /opt/conjur/etc/ssl/conjur-master.mycompany.local.pem
lrwxrwxrwx 1 root   root       26 Mar 13 13:19 master.pem -> /opt/conjur/etc/ssl/ca.pem

One of these must match the common name (CN) of the Follower certificate specified in the seed request.

I also recommend opening a support case for Conjur Enterprise to share more of the specific details from your manifests and logs to help troubleshoot further.

Thanks!
Micah

4

Hi @Jakub,

As Micah noted, your certificate file is not being found when the seed request is received. I’ve written about this issue previously. Please take a look in this thread for more details.

Regards,
Nate

5

Hi,
Thank you for your help. Yes, it was a problem with the wrong CN in the certificate

KR
Jakub