I followed your steps and re-deployed the Secrets Provider via helm charts and it worked! I’m really confused why though… as to what changed commpared to the other times I tried.
NAME READY STATUS RESTARTS AGE
mysql-client-778657c698-rmz2g 1/1 Running 0 3d5h
nfs-nfs-server-provisioner-0 1/1 Running 3 188d
secrets-provider-rfvm7 0/1 Completed 0 17m
vm-mariadb-master-0 1/1 Running 0 188d
vm-mariadb-slave-0 1/1 Running 0 188d
Here’s the log for secrets-provider-rfvm7 which looks successful:
INFO: 2020/10/02 16:44:43 main.go:23: CSPFK008I CyberArk Secrets Provider for Kubernetes v1.1.0-dev starting up
DEBUG: 2020/10/02 16:44:43 main.go:115: CSPFK001D Debug mode is enabled
INFO: 2020/10/02 16:44:43 main.go:81: CSPFK001I Authenticating as user 'host/conjur/authn-k8s/provider/apps/secrets-provider-host'
INFO: 2020/10/02 16:44:43 authenticator.go:197: CAKC005I Trying to login Conjur...
INFO: 2020/10/02 16:44:43 authenticator.go:116: CAKC007I Logging in as user 'host/conjur/authn-k8s/provider/apps/secrets-provider-host'
INFO: 2020/10/02 16:44:43 requests.go:23: CAKC011I Login request to: https://conjur-helm-v2-conjur-oss.conjur-demo.svc.cluster.local/authn-k8s/provider/inject_client_cert
INFO: 2020/10/02 16:44:44 file.go:35: CAKC017I Waiting for file /etc/conjur/ssl/client.pem to become available...
INFO: 2020/10/02 16:44:44 authenticator.go:159: CAKC015I Loaded client certificate successfully from /etc/conjur/ssl/client.pem
INFO: 2020/10/02 16:44:44 authenticator.go:171: CAKC016I Deleted client certificate from memory
INFO: 2020/10/02 16:44:44 authenticator.go:203: CAKC002I Logged in
INFO: 2020/10/02 16:44:44 authenticator.go:186: CAKC008I Cert expires: 2020-10-05 16:44:43 +0000 UTC
INFO: 2020/10/02 16:44:44 authenticator.go:187: CAKC009I Current date: 2020-10-02 16:44:44.522679508 +0000 UTC
INFO: 2020/10/02 16:44:44 authenticator.go:188: CAKC010I Buffer time: 30s
INFO: 2020/10/02 16:44:44 requests.go:47: CAKC012I Authn request to: https://conjur-helm-v2-conjur-oss.conjur-demo.svc.cluster.local/authn-k8s/provider/default/host%2Fconjur%2Fauthn-k8s%2Fprovider%2Fapps%2Fsecrets-provider-host/authenticate
INFO: 2020/10/02 16:44:44 authenticator.go:266: CAKC001I Successfully authenticated
INFO: 2020/10/02 16:44:44 k8s_secrets_client.go:54: CSPFK004I Creating Kubernetes client
INFO: 2020/10/02 16:44:44 k8s_secrets_client.go:19: CSPFK005I Retrieving Kubernetes secret 'db-credentials-demo' from namespace 'default'
DEBUG: 2020/10/02 16:44:44 provide_conjur_secrets.go:127: CSPFK009D Processing 'conjur-map' data entry value of Kubernetes Secret 'db-credentials-demo'
INFO: 2020/10/02 16:44:44 conjur_secrets_retriever.go:11: CSPFK003I Retrieving following secrets from DAP/Conjur: [db_credentials/username db_credentials/password]
INFO: 2020/10/02 16:44:44 conjur_client.go:21: CSPFK002I Creating DAP/Conjur client
INFO: 2020/10/02 16:44:44 k8s_secrets_client.go:54: CSPFK004I Creating Kubernetes client
INFO: 2020/10/02 16:44:44 k8s_secrets_client.go:38: CSPFK006I Updating Kubernetes secret 'db-credentials-demo' in namespace 'default'
INFO: 2020/10/02 16:44:44 main.go:102: CSPFK009I DAP/Conjur Secrets updated in Kubernetes successfully
And here is what I get for the Conjur logs, looks like it’s showing what the database is doing - is this what it should look like? The file is huge, so this is just an excerpt from the end of the file:
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[0m
e[1me[35mSequel::Postgres::Database (0.3ms)e[0m SELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (0.6ms)e[0m e[1mSELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1e[0m
e[1me[35mSequel::Postgres::Database (1.2ms)e[0m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[1me[36mSequel::Postgres::Database (0.3ms)e[0m e[1mSELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1e[0m
[origin=172.16.220.56] [request_id=2a6f39df-7830-4f38-bb51-14062aec33aa] [tid=40] Started GET "/" for 172.16.220.56 at 2020-10-02 16:55:25 +0000
[origin=172.16.220.56] [request_id=2a6f39df-7830-4f38-bb51-14062aec33aa] [tid=40] Processing by StatusController#index as HTML
[origin=172.16.220.56] [request_id=2a6f39df-7830-4f38-bb51-14062aec33aa] [tid=40] e[1me[35mSequel::Postgres::Database (0.6ms)e[0m BEGIN
[origin=172.16.220.56] [request_id=2a6f39df-7830-4f38-bb51-14062aec33aa] [tid=40] Rendered status/index.html.erb within layouts/application (0.1ms)
[origin=172.16.220.56] [request_id=2a6f39df-7830-4f38-bb51-14062aec33aa] [tid=40] Rendered shared/_navigation.html.erb (0.2ms)
[origin=172.16.220.56] [request_id=2a6f39df-7830-4f38-bb51-14062aec33aa] [tid=40] Rendered shared/_footer.html.erb (0.2ms)
[origin=172.16.220.56] [request_id=2a6f39df-7830-4f38-bb51-14062aec33aa] [tid=40] e[1me[36mSequel::Postgres::Database (0.3ms)e[0m e[1mCOMMITe[0m
[origin=172.16.220.56] [request_id=2a6f39df-7830-4f38-bb51-14062aec33aa] [tid=40] Completed 200 OK in 3ms (Views: 1.5ms)
e[1me[35mSequel::Postgres::Database (0.7ms)e[0m SELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (1.2ms)e[0m e[1m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[0m
e[1me[35mSequel::Postgres::Database (0.3ms)e[0m SELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1
[origin=172.16.220.56] [request_id=7eae7547-e527-4284-95b2-60187f2f3221] [tid=37] Started GET "/" for 172.16.220.56 at 2020-10-02 16:55:27 +0000
[origin=172.16.220.56] [request_id=7eae7547-e527-4284-95b2-60187f2f3221] [tid=37] Processing by StatusController#index as HTML
[origin=172.16.220.56] [request_id=7eae7547-e527-4284-95b2-60187f2f3221] [tid=37] e[1me[35mSequel::Postgres::Database (0.3ms)e[0m BEGIN
[origin=172.16.220.56] [request_id=7eae7547-e527-4284-95b2-60187f2f3221] [tid=37] Rendered status/index.html.erb within layouts/application (0.1ms)
[origin=172.16.220.56] [request_id=7eae7547-e527-4284-95b2-60187f2f3221] [tid=37] Rendered shared/_navigation.html.erb (0.2ms)
[origin=172.16.220.56] [request_id=7eae7547-e527-4284-95b2-60187f2f3221] [tid=37] Rendered shared/_footer.html.erb (0.2ms)
[origin=172.16.220.56] [request_id=7eae7547-e527-4284-95b2-60187f2f3221] [tid=37] e[1me[36mSequel::Postgres::Database (0.3ms)e[0m e[1mCOMMITe[0m
[origin=172.16.220.56] [request_id=7eae7547-e527-4284-95b2-60187f2f3221] [tid=37] Completed 200 OK in 3ms (Views: 1.6ms)
e[1me[36mSequel::Postgres::Database (0.6ms)e[0m e[1mSELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1e[0m
e[1me[35mSequel::Postgres::Database (1.3ms)e[0m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[1me[36mSequel::Postgres::Database (0.3ms)e[0m e[1mSELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1e[0m
e[1me[35mSequel::Postgres::Database (0.6ms)e[0m SELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (1.2ms)e[0m e[1m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[0m
e[1me[35mSequel::Postgres::Database (0.3ms)e[0m SELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (0.7ms)e[0m e[1mSELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1e[0m
e[1me[35mSequel::Postgres::Database (1.4ms)e[0m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[1me[36mSequel::Postgres::Database (0.3ms)e[0m e[1mSELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1e[0m
e[1me[35mSequel::Postgres::Database (0.7ms)e[0m SELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (1.3ms)e[0m e[1m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[0m
e[1me[35mSequel::Postgres::Database (0.4ms)e[0m SELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (0.6ms)e[0m e[1mSELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1e[0m
e[1me[35mSequel::Postgres::Database (1.3ms)e[0m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[1me[36mSequel::Postgres::Database (0.3ms)e[0m e[1mSELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1e[0m
[origin=172.16.220.56] [request_id=52e3182b-49e3-4b4a-9c0a-0cd644d5da9c] [tid=34] Started GET "/" for 172.16.220.56 at 2020-10-02 16:55:32 +0000
[origin=172.16.220.56] [request_id=52e3182b-49e3-4b4a-9c0a-0cd644d5da9c] [tid=34] Processing by StatusController#index as HTML
[origin=172.16.220.56] [request_id=52e3182b-49e3-4b4a-9c0a-0cd644d5da9c] [tid=34] e[1me[35mSequel::Postgres::Database (0.4ms)e[0m BEGIN
[origin=172.16.220.56] [request_id=52e3182b-49e3-4b4a-9c0a-0cd644d5da9c] [tid=34] Rendered status/index.html.erb within layouts/application (0.1ms)
[origin=172.16.220.56] [request_id=52e3182b-49e3-4b4a-9c0a-0cd644d5da9c] [tid=34] Rendered shared/_navigation.html.erb (0.2ms)
[origin=172.16.220.56] [request_id=52e3182b-49e3-4b4a-9c0a-0cd644d5da9c] [tid=34] Rendered shared/_footer.html.erb (0.2ms)
[origin=172.16.220.56] [request_id=52e3182b-49e3-4b4a-9c0a-0cd644d5da9c] [tid=34] e[1me[36mSequel::Postgres::Database (0.3ms)e[0m e[1mCOMMITe[0m
[origin=172.16.220.56] [request_id=52e3182b-49e3-4b4a-9c0a-0cd644d5da9c] [tid=34] Completed 200 OK in 3ms (Views: 1.7ms)
e[1me[35mSequel::Postgres::Database (0.7ms)e[0m SELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (1.2ms)e[0m e[1m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[0m
e[1me[35mSequel::Postgres::Database (0.3ms)e[0m SELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (0.7ms)e[0m e[1mSELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1e[0m
e[1me[35mSequel::Postgres::Database (1.2ms)e[0m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[1me[36mSequel::Postgres::Database (0.3ms)e[0m e[1mSELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1e[0m
e[1me[35mSequel::Postgres::Database (0.6ms)e[0m SELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (1.2ms)e[0m e[1m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[0m
e[1me[35mSequel::Postgres::Database (0.3ms)e[0m SELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (0.7ms)e[0m e[1mSELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1e[0m
e[1me[35mSequel::Postgres::Database (1.2ms)e[0m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[1me[36mSequel::Postgres::Database (0.3ms)e[0m e[1mSELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1e[0m
e[1me[35mSequel::Postgres::Database (0.7ms)e[0m SELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (1.2ms)e[0m e[1m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[0m
e[1me[35mSequel::Postgres::Database (0.4ms)e[0m SELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1
[origin=172.16.220.56] [request_id=e9f4c971-b47b-4b62-bebe-6a9acdad4558] [tid=39] Started GET "/" for 172.16.220.56 at 2020-10-02 16:55:37 +0000
[origin=172.16.220.56] [request_id=e9f4c971-b47b-4b62-bebe-6a9acdad4558] [tid=39] Processing by StatusController#index as HTML
[origin=172.16.220.56] [request_id=e9f4c971-b47b-4b62-bebe-6a9acdad4558] [tid=39] e[1me[35mSequel::Postgres::Database (0.4ms)e[0m BEGIN
[origin=172.16.220.56] [request_id=e9f4c971-b47b-4b62-bebe-6a9acdad4558] [tid=39] Rendered status/index.html.erb within layouts/application (0.1ms)
[origin=172.16.220.56] [request_id=e9f4c971-b47b-4b62-bebe-6a9acdad4558] [tid=39] Rendered shared/_navigation.html.erb (0.2ms)
[origin=172.16.220.56] [request_id=e9f4c971-b47b-4b62-bebe-6a9acdad4558] [tid=39] Rendered shared/_footer.html.erb (0.2ms)
[origin=172.16.220.56] [request_id=e9f4c971-b47b-4b62-bebe-6a9acdad4558] [tid=39] e[1me[36mSequel::Postgres::Database (0.3ms)e[0m e[1mCOMMITe[0m
[origin=172.16.220.56] [request_id=e9f4c971-b47b-4b62-bebe-6a9acdad4558] [tid=39] Completed 200 OK in 3ms (Views: 1.6ms)
e[1me[36mSequel::Postgres::Database (0.6ms)e[0m e[1mSELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1e[0m
e[1me[35mSequel::Postgres::Database (1.2ms)e[0m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[1me[36mSequel::Postgres::Database (0.3ms)e[0m e[1mSELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1e[0m
e[1me[35mSequel::Postgres::Database (0.7ms)e[0m SELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (1.3ms)e[0m e[1m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[0m
e[1me[35mSequel::Postgres::Database (0.3ms)e[0m SELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (0.7ms)e[0m e[1mSELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1e[0m
e[1me[35mSequel::Postgres::Database (1.3ms)e[0m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[1me[36mSequel::Postgres::Database (0.3ms)e[0m e[1mSELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1e[0m
e[1me[35mSequel::Postgres::Database (0.7ms)e[0m SELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (1.7ms)e[0m e[1m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[0m
e[1me[35mSequel::Postgres::Database (0.4ms)e[0m SELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (0.7ms)e[0m e[1mSELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1e[0m
e[1me[35mSequel::Postgres::Database (1.3ms)e[0m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[1me[36mSequel::Postgres::Database (0.3ms)e[0m e[1mSELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1e[0m
[origin=172.16.220.56] [request_id=b382aba9-7080-43dd-afa4-ef4dda2b1cbd] [tid=42] Started GET "/" for 172.16.220.56 at 2020-10-02 16:55:42 +0000
[origin=172.16.220.56] [request_id=b382aba9-7080-43dd-afa4-ef4dda2b1cbd] [tid=42] Processing by StatusController#index as HTML
[origin=172.16.220.56] [request_id=b382aba9-7080-43dd-afa4-ef4dda2b1cbd] [tid=42] e[1me[35mSequel::Postgres::Database (0.4ms)e[0m BEGIN
[origin=172.16.220.56] [request_id=b382aba9-7080-43dd-afa4-ef4dda2b1cbd] [tid=42] Rendered status/index.html.erb within layouts/application (0.1ms)
[origin=172.16.220.56] [request_id=b382aba9-7080-43dd-afa4-ef4dda2b1cbd] [tid=42] Rendered shared/_navigation.html.erb (0.2ms)
[origin=172.16.220.56] [request_id=b382aba9-7080-43dd-afa4-ef4dda2b1cbd] [tid=42] Rendered shared/_footer.html.erb (0.2ms)
[origin=172.16.220.56] [request_id=b382aba9-7080-43dd-afa4-ef4dda2b1cbd] [tid=42] e[1me[36mSequel::Postgres::Database (0.3ms)e[0m e[1mCOMMITe[0m
[origin=172.16.220.56] [request_id=b382aba9-7080-43dd-afa4-ef4dda2b1cbd] [tid=42] Completed 200 OK in 3ms (Views: 1.5ms)
e[1me[35mSequel::Postgres::Database (0.6ms)e[0m SELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (1.5ms)e[0m e[1m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[0m
e[1me[35mSequel::Postgres::Database (0.3ms)e[0m SELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (0.6ms)e[0m e[1mSELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1e[0m
e[1me[35mSequel::Postgres::Database (1.2ms)e[0m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[1me[36mSequel::Postgres::Database (0.3ms)e[0m e[1mSELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1e[0m
e[1me[35mSequel::Postgres::Database (0.6ms)e[0m SELECT pg_try_advisory_lock(767003715) AS "v" LIMIT 1
e[1me[36mSequel::Postgres::Database (1.3ms)e[0m e[1m SELECT ttl.resource_id, ttl.value AS ttl, rotators.value AS rotator_name
FROM annotations ttl
-- This ensures we get only entries with both
-- a ttl and a rotator specified
JOIN annotations rotators ON (
rotators.resource_id = ttl.resource_id
AND rotators.name = 'rotation/rotator'
)
LEFT JOIN secrets ON ttl.resource_id = secrets.resource_id
LEFT JOIN (
SELECT resource_id, MAX(version) AS version
FROM secrets
GROUP BY resource_id
) max_version ON max_version.resource_id = ttl.resource_id
WHERE ttl.name = 'rotation/ttl'
AND secrets.version = max_version.version
AND (
secrets.expires_at < NOW() OR secrets.expires_at IS NULL
)
e[0m
e[1me[35mSequel::Postgres::Database (0.3ms)e[0m SELECT pg_advisory_unlock(767003715) AS "v" LIMIT 1
Many thanks for your help, this is my first successful Kubernetes integration! 
I would be really curious though to understand what changed in my cofiguration. The only thing I can think of is having edited the deployment as you suggested and deleting the old replicaset.