Using AWS KMS for encrypting DAP Master

Conjur AAM/DAP
#1

Hello Everyone,

I am looking for some guidance on using the KMS to encrypt. Any suggestions on how to achieve this would be very helpful.
I will explain the sequence of steps that I have followed.

Before I started the process Conjur Master is configured and it is running and the database is not encrypted (default settings).
I assigned a EC2 role to the instance to make sure it has permissions to create the the IAM roles and the keys in KMS.
I ran the command “evoke keys kms” it created a KMS key and also it created a role named conjur-appliance . It also create a encrypted key with the name kms_master_key.us-east-2.enc in /opt/conjur/etc.

From the documentation what I am not clear is that should I create master key myself using the the process explained in this bookmark

and then use the Amazon CLI to encrypt the master Key that is being created or should the evoke KMS keys will do that.

Also I am not very sure on how the Conjur Appliance role is being used. Should that be assigned to all the instances where we have Conjur HA Cluster nodes are runnning?

I am using the Docker CE and deployed Conjur appliance on Amazon Linux 2 AMI. Thank you

#2

I wrote this up a while back for my own notes. I haven’t gone back and looked at it for some time, but it should get you unblocked.

Create IAM policy with the contents of https://gist.github.com/whip113/a2da9aa66ecbcd63d59edfe7d6046dc2 (Note: AWS has changed the IAM policies a few times since I wrote this policy, and due to AWS “eventual consistency” I’ve had a difficult time A/B testing to ensure it works reliably. This is an optional, least privileges IAM role used to create the key, key policy, conjur-appliance IAM role, and link them. IAM-FullPermissions would avoid any headaches, but may not be applicable for all customers)

  • Deploy a disposable instance of the Conjur appliance and assign the IAM role created above to it
  • Create a new key by running evoke keys kms <region>
  • Deploy a new prod Conjur instance and assign the newly created conjur-appliance IAM role to it. This role grants the instance the permissions necessary to retrieve the key created in step 3.
  • Configure the prod instance with evoke configure master
  • Copy the file /opt/conjur/etc/kms_master_key.<region>.enc from the disposable instance to the same location in the container of your prod instance
  • Terminate the disposable instance, you don’t need it anymore
  • Run sv stop conjur pg nginx on the prod instance
  • Run evoke keys encrypt on the prod instance. This encrypts the service keys and certificate private keys that are on disk.
  • Run evoke keys unlock. This decrypts the service keys and certificate private keys with the KMS key and loads them into the kernel keyring.
  • Run sv start conjur pg nginx

At this point you can create your seeds and deploy your standbys and followers. Note that these instances will require the same conjur-appliance role to be assigned to them so they can access the key encryption key stored in KMS. Also note, if you wish to import 3rd party signed certificates, you’ll want to do that prior to running evoke keys encrypt. I hope that helps!

Regards,
Nate

#3

Thank you Nathan. I will follow the steps and update this post.

#4

@nathan.whipple I was able to get that working. I was trying generate the seed files for followers and I was unable to do it. I am getting an error that I need to decrypt the keys but even after I runt he command i seem to get the same message. Can you please provide some guidance on this. Screenshot of the error message below

image
image1397×112 29.4 KB

#5

Take a look at /opt/conjur/etc/ssl in your master container. After running evoke keys decrypt-all, there shouldn’t be any files in that directory with an enc extension. There is a combination of events that can happen where the file ui.key is created while the keys are encrypted. Since that file is already present and unencrypted, the evoke keys decrypt-all command skips decrypting ui.key.enc (you can actually see that ui is already decrypted message in the log output). The subsequent command to generate a seed file then checks if there are any files with an enc extension and throws an error assuming the files are all encrypted. The fix is as simple as deleting the file ui.key.enc. I’ll double check, but I’m quite certain I had a bug opened for this issue already, but if you could submit a support case anyhow just so we can associate the bug with a customer it would be greatly appreciated.

Regards,
Nate

#6

@nathan.whipple I tried the follow the instructions but I did not see any .enc file in the /opt/conjur/etc/ssl directory. I did find the the ui.key.enc in /opt/conjur/etc folder. I tried to remove it and then tried to generate see file and it gave me an error message as error: Can’t find group for 1000. I tried to encrypt and decrypt-all again but that does not make any difference. I am attaching the screenshot with all the details. Your help is much appreciated.

image
image1920×638 45.6 KB

#7

Ah, yes, you are correct on the path, my mistake. I’m glad you were able to find it though.

As for the group 1000 error, I’ve run into this before but I can’t seem to reproduce it right now. I suspect its a file ownership issue on the key blob. Can you post the output of ls -al in the /opt/conjur/etc directory? We probably just need to chown the file to fix the permissions.

Regards,
Nate

#8

The screenshot of the directory. I seem to get a different error now which seems strange as this is the only master. I can recreate the whole thing am testing this out in my environment before implementing for the client

image
image750×302 53.4 KB

image
image1444×215 15.6 KB

#9

As I suspected, the permissions on the file kms_master_key.us-east-2.enc are incorrect. To fix that, you should be able to do chown root:root kms_master_key.us-east-2.enc.

The second error is odd and not one I’ve seen before. What version are you using and what else has been done on this server? (e.g., auto-failover cluster, failover testing, etc) Also, if you change the follower name to something else, does it work then?

#10

@nathan.whipple the error message I got was because of some docker server issue. Once I restarted the error disappeared and I was able to change the permissions on the kms_master_key.us-east-2.enc and create the seed files for followers. I used the seed the files for configuring followers.

I have few follow-up questions.

We generated the seed file after executing the evoke keys decrypt-all command. Once the seed files are generated. What actions needs to taken on the master to encrypt the keys again and when that needs to be done? is that needs to ne done after configuring the followers and standbys or it dos not matter. I am assuming it can be done using the following commands on the master

sv stop conjur pg nginx
evoke keys encrypt
evoke keys unlock
sv start conjur pg nginx

For testing I used the seed file to configure two followers one EC2 instance was assigned the conjur-appliance role and other was not. I was able to configure both followers and both were able to connect to the master and I did not have to use the master key while unpacking the seedfile or configuring the followers. If you could explain how we need to secure the keys present on the followers and standby as well it will be very helpful.

#11

Great questions. The reason we need to decrypt the keys prior to generating a follower seed file is that the command evoke seed follower myfollower.nate.lab is going to check for a certificate named /opt/conjur/etc/ssl/myfollower.nate.lab.pem. If that certificate file is not found, and you are using self-signed certificates, then evoke will automatically issue a new certificate, then generate the seed. However, to generate that certificate the keys had to be decypted and the seed file is thus created with unencrypted contents. That’s why you were able to deploy followers successfully even without the IAM role assigned.

Now typically we would be expecting to use certificates signed by an external CA, so those certificates would be present already after you did the evoke ca import steps. Once you’ve imported 3rd party certificates, you would then run evoke keys encrypt and your seed generation would work as expected (because the certificate is already present, evoke isn’t going to attempt to issue one). To accomplish a similar workflow with self signed certificates, the process should go something like this:

  • run evoke keys decrypt-all
  • run evoke ca issue myfollower.nate.lab, this will create the certificate for the follower
  • run evoke keys encrypt
  • run evoke seed follower myfollower.nate.lab to generate your seed

I’ll point our tech writer team to this thread, as clearly we have some gaps in the documented steps and processes necessary to achieve the desired outcome. Many apologies for that, and kudos for your patience and determination.

Regards,
Nate

#12

@nathan.whipple Thank you I really appreciate your help and patience in explaining this. I think I have figured this out and will update if I run into anything while implementing this in production instances.
I certainly appreciate you involving the document writing team as that would help everyone.

#13

@nathan.whipple I have one follow-up question . Appreciate your patience. I know we have protection through the KMS when we setup the follower/standby for the first time. Does the keys in the follower/standby remain in the encrypted after the configuration process or we have to manually take any action like we did for the master. Also once the initial process is complete and if for some reason the standby takes over as a master in HA cluster auto fail over scenarios is there any additional action that need to taken or considered. Thanks

#14
  • The keys are encrypted on disk, no further action is necessary. You can verify by doing something like docker exec conjur-appliance ls /opt/conjur/etc and verifying all the key files end with an .enc extension
  • No further action is necessary. You’d repair the cluster per your usual process.

Side note: You can run the evoke keys deycrpt-all command on standbys or followers if necessary. Likewise, you can run evoke keys encrypt if the keys are in a decrypted state.