CyberArk DAP version upgrade from 12.0 to 12.7

Hi Team,

Our current DAP version is 12.0 and we would like to upgrade to latest version 12.7.
In our current setup below are installed

  1. Master layer (DAP master & standby’s)
  2. Kubernetes followers
  3. Vault synchronizer.

Please help me on below query’s to upgrade our current setup.

  1. Where to get the conjur-appliance to upgrade the master layer (DAP master & standby’s) ?
  2. Below are the images used in manifest file of kubernetes followers , do I need to upgrade both the images ?

Where can I get those docker images ?

initContainers:

============

  1. Do I need to upgrade the vault synchronizer as well ?

  2. Currently our Vault version is 12.0.0 , will that CyberArk DAP version 12.7 is compatible with the current Vault version 12.0 ?

  3. Any other steps need to taken care other than above to upgrade DAP ?

Regards,
Phani.G

Hi,

All the upgrade processes are documented in the CyberArk Docs

Master/ Standbys
https://docs.cyberark.com/Product-Doc/OnlineHelp/AAM-DAP/Latest/en/Content/Deployment/dap-upgrade.htm

Vault Synchronizer
[Upgrade Vault Synchronizer]

Latest software is available in the market place, vault synchronizer can work with vault version starting 10.5 onwards, so no compatibility issues.

For K8S images, you can download and upload to your repos.

Good luck :slight_smile:

Thanks gautam for reply !

For upgradation on K8s , do I need to upgrade both seedfetcher & conjur-appliance images ?

Regards,
Phani.G

Hi Phani,

I would suggest yes to update all to the latest.
We have performed and found no issues.

Kr,
Gautam

Hi gautam,

we are upgrading the DAP conjur appliance from 12.0.0 to 12.7.0.1, during the restoration process through backup it is failing on the master.

We have encrypted the master key using the KMS.

Below is the output of exceptions, please review and suggest to us any insight on resolving the issue .

root@b4a73efec8ac:/opt/conjur/etc# evoke restore --accept-eula

Error reading PKCS11 configuration file [Errno::ENOENT, No such file or directory @ rb_sysopen - /opt/conjur/etc/pkcs11.yml]

error: No valid KMS key found at /opt/conjur/etc/kms_master_key.aws.us-east-1

root@b4a73efec8ac:/opt/conjur/etc# exit

root@11a8b2d251d6:/opt/conjur/etc# evoke keys show-master-key

Error reading PKCS11 configuration file [Errno::ENOENT, No such file or directory @ rb_sysopen - /opt/conjur/etc/pkcs11.yml]

error: No valid KMS key found at /opt/conjur/etc/kms_master_key.aws.us-east-1

root@11a8b2d251d6:/opt/conjur/etc#

root@11a8b2d251d6:/opt/conjur/etc# evoke keys kms validate --region us-east-1

error: No valid KMS key

root@11a8b2d251d6:/opt/conjur/etc#

Regards,

Phani.G

Hi @phanimngr,

You can safely ignore the PKCS11 error message in this case as that is just a result of it failing to unlock the keys with KMS. It sounds like when the backup was unpacked the key blob used for KMS was not present or couldn’t be found. Can you step into the container and list the contents of /opt/conjur/etc please?

docker exec <container_name> ls -la /opt/conjur/etc

Regards,
Nate

Hi @phanimngr,

@micahlee found that in 12.2 we changed the file path format for the cipher text blob stored in /opt/conjur/etc. It changed from:

/opt/conjur/etc/kms_master_key.#{ec2_region}.enc

to

/opt/conjur/etc/kms_master_key.aws.#{ec2_region}.enc

This means you should be able to correct this issue by renaming the existing key blob to the new format:

docker exec <container_name> cp /opt/conjur/etc/kms_master_key.us-east-1.enc /opt/conjur/etc/kms_master_key.aws.us-east-1.enc

Please let me know how you make out!

Regards,
Nate

Thanks, Nate for your reply!

As suggested, below changes are working

docker exec <container_name> cp /opt/conjur/etc/kms_master_key.us-east-1.enc /opt/conjur/etc/kms_master_key.aws.us-east-1

Regards,
Phani.G

Hi Nate,

Today, I tried to upgrade the Kubernetes DAP followers from 12.0 to 12.7 after the upgradation of the master layer, during this process we encountered the below issue. Please review the below logs and suggest to us any insight on resolving this issue.

As a part of the upgrade, we have modified only this manifest file 006-follower_manifest.yml file and applied the changes.
Also, I don’t find the CyberArk document for upgrading the Kubernetes followers.

Only I find this link, but it is covering only upgradation of the OpenShift Kubernetes followers “Upgrade Conjur Kubernetes Follower

Error logs:

DAP Seed Fetcher v0.6.0-912
Trying to fetch seedfile from https://qadapmr.mhf.mhc/configuration/spglobal/seed/follower
Hostname is — conjur-follower —
Using master ssl cert from /tmp/master.crt
Calculated vars:

Running authenticator…
INFO: 2023/01/18 07:44:15.096215 main.go:19: CAKC048 Kubernetes Authenticator Client v0.23.1-dev starting up…
INFO: 2023/01/18 07:44:15.096260 configuration_factory.go:79: CAKC070 Chosen “authn-k8s” configuration
INFO: 2023/01/18 07:44:15.096295 authenticator_factory.go:31: CAKC075 Chosen “authn-k8s” flow
INFO: 2023/01/18 07:44:15.182183 authenticator.go:84: CAKC040 Authenticating as user ‘host/conjur/authn-k8s/NP-West/apps/seed-fetcher-app’
IP Address of master: 10.164.140.76
ERROR: 2023/01/18 07:44:33.690296 authenticator.go:179: CAKC028 Failed to send https login request or response. Reason: Post https://qadapmr.mhf.mhc/authn-k8s/NP-West/inject_client_cert: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
ERROR: 2023/01/18 07:44:33.690320 authenticator.go:271: CAKC015 Login failed
ERROR: 2023/01/18 07:44:33.690326 main.go:49: CAKC016 Failed to authenticate
INFO: 2023/01/18 07:44:35.900573 authenticator.go:84: CAKC040 Authenticating as user ‘host/conjur/authn-k8s/NP-West/apps/seed-fetcher-app’

ERROR: 2023/01/18 07:46:38.155231 authenticator.go:323: CAKC027 Failed to send https authenticate request or receive response. Reason: Post https://qadapmr.mhf.mhc/authn-k8s/NP-West/spglobal/host%2Fconjur%2Fauthn-k8s%2FNP-West%2Fapps%2Fseed-fetcher-app/authenticate: context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Events logs:

kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
conjur-follower-6f65497899-qckqg 0/1 Init:Error 431 (7m28s ago) 45h 10.165.152.185 ip-10-165-152-132.us-west-2.compute.internal
conjur-follower-d95c975fb-8n5w4 0/1 Running 616 (5m31s ago) 2d18h 10.165.152.183 ip-10-165-152-132.us-west-2.compute.internal
conjur-follower-d95c975fb-p7zqz 0/1 CrashLoopBackOff 615 (110s ago) 2d18h 10.165.152.139 ip-10-165-152-132.us-west-2.compute.internal

kubectl get events -n dapfollower
LAST SEEN TYPE REASON OBJECT MESSAGE
3m53s Warning BackOff pod/conjur-follower-6f65497899-qckqg Back-off restarting failed container
17m Warning Unhealthy pod/conjur-follower-d95c975fb-8n5w4 Readiness probe failed: Get https://10.165.152.183:443/health: dial tcp 10.165.152.183:443: connect: connection refused
2m29s Warning BackOff pod/conjur-follower-d95c975fb-8n5w4 Back-off restarting failed container
27m Normal Pulled pod/conjur-follower-d95c975fb-p7zqz Container image “130312249203.dkr.ecr.us-west-2.amazonaws.com/cyberarkdapnp:conjur-appliance” already present on machine
7m13s Warning Unhealthy pod/conjur-follower-d95c975fb-p7zqz Readiness probe failed: Get https://10.165.152.139:443/health: dial tcp 10.165.152.139:443: connect: connection refused
2m59s Warning BackOff pod/conjur-follower-d95c975fb-p7zqz Back-off restarting failed container
48s Warning FailedGetResourceMetric horizontalpodautoscaler/conjur-follower failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

Manifest file:- I have modified the seedfetcher & conjur-appliance image names in the manifest file as created in the docker images.

Regards,
Phani.G