Using AWS KMS for encrypting DAP Master

Conjur AAM/DAP
#1

Hello Everyone,

I am looking for some guidance on using the KMS to encrypt. Any suggestions on how to achieve this would be very helpful.
I will explain the sequence of steps that I have followed.

Before I started the process Conjur Master is configured and it is running and the database is not encrypted (default settings).
I assigned a EC2 role to the instance to make sure it has permissions to create the the IAM roles and the keys in KMS.
I ran the command “evoke keys kms” it created a KMS key and also it created a role named conjur-appliance . It also create a encrypted key with the name kms_master_key.us-east-2.enc in /opt/conjur/etc.

From the documentation what I am not clear is that should I create master key myself using the the process explained in this bookmark

and then use the Amazon CLI to encrypt the master Key that is being created or should the evoke KMS keys will do that.

Also I am not very sure on how the Conjur Appliance role is being used. Should that be assigned to all the instances where we have Conjur HA Cluster nodes are runnning?

I am using the Docker CE and deployed Conjur appliance on Amazon Linux 2 AMI. Thank you

#2

I wrote this up a while back for my own notes. I haven’t gone back and looked at it for some time, but it should get you unblocked.

Create IAM policy with the contents of https://gist.github.com/whip113/a2da9aa66ecbcd63d59edfe7d6046dc2 (Note: AWS has changed the IAM policies a few times since I wrote this policy, and due to AWS “eventual consistency” I’ve had a difficult time A/B testing to ensure it works reliably. This is an optional, least privileges IAM role used to create the key, key policy, conjur-appliance IAM role, and link them. IAM-FullPermissions would avoid any headaches, but may not be applicable for all customers)

  • Deploy a disposable instance of the Conjur appliance and assign the IAM role created above to it
  • Create a new key by running evoke keys kms <region>
  • Deploy a new prod Conjur instance and assign the newly created conjur-appliance IAM role to it. This role grants the instance the permissions necessary to retrieve the key created in step 3.
  • Configure the prod instance with evoke configure master
  • Copy the file /opt/conjur/etc/kms_master_key.<region>.enc from the disposable instance to the same location in the container of your prod instance
  • Terminate the disposable instance, you don’t need it anymore
  • Run sv stop conjur pg nginx on the prod instance
  • Run evoke keys encrypt on the prod instance. This encrypts the service keys and certificate private keys that are on disk.
  • Run evoke keys unlock. This decrypts the service keys and certificate private keys with the KMS key and loads them into the kernel keyring.
  • Run sv start conjur pg nginx

At this point you can create your seeds and deploy your standbys and followers. Note that these instances will require the same conjur-appliance role to be assigned to them so they can access the key encryption key stored in KMS. Also note, if you wish to import 3rd party signed certificates, you’ll want to do that prior to running evoke keys encrypt. I hope that helps!

Regards,
Nate

#3

Thank you Nathan. I will follow the steps and update this post.

#4

@nathan.whipple I was able to get that working. I was trying generate the seed files for followers and I was unable to do it. I am getting an error that I need to decrypt the keys but even after I runt he command i seem to get the same message. Can you please provide some guidance on this. Screenshot of the error message below

image
image1397×112 29.4 KB

#5

Take a look at /opt/conjur/etc/ssl in your master container. After running evoke keys decrypt-all, there shouldn’t be any files in that directory with an enc extension. There is a combination of events that can happen where the file ui.key is created while the keys are encrypted. Since that file is already present and unencrypted, the evoke keys decrypt-all command skips decrypting ui.key.enc (you can actually see that ui is already decrypted message in the log output). The subsequent command to generate a seed file then checks if there are any files with an enc extension and throws an error assuming the files are all encrypted. The fix is as simple as deleting the file ui.key.enc. I’ll double check, but I’m quite certain I had a bug opened for this issue already, but if you could submit a support case anyhow just so we can associate the bug with a customer it would be greatly appreciated.

Regards,
Nate

#6

@nathan.whipple I tried the follow the instructions but I did not see any .enc file in the /opt/conjur/etc/ssl directory. I did find the the ui.key.enc in /opt/conjur/etc folder. I tried to remove it and then tried to generate see file and it gave me an error message as error: Can’t find group for 1000. I tried to encrypt and decrypt-all again but that does not make any difference. I am attaching the screenshot with all the details. Your help is much appreciated.

image
image1920×638 45.6 KB

#7

Ah, yes, you are correct on the path, my mistake. I’m glad you were able to find it though.

As for the group 1000 error, I’ve run into this before but I can’t seem to reproduce it right now. I suspect its a file ownership issue on the key blob. Can you post the output of ls -al in the /opt/conjur/etc directory? We probably just need to chown the file to fix the permissions.

Regards,
Nate

#8

The screenshot of the directory. I seem to get a different error now which seems strange as this is the only master. I can recreate the whole thing am testing this out in my environment before implementing for the client

image
image750×302 53.4 KB

image
image1444×215 15.6 KB

#9

As I suspected, the permissions on the file kms_master_key.us-east-2.enc are incorrect. To fix that, you should be able to do chown root:root kms_master_key.us-east-2.enc.

The second error is odd and not one I’ve seen before. What version are you using and what else has been done on this server? (e.g., auto-failover cluster, failover testing, etc) Also, if you change the follower name to something else, does it work then?

#10

@nathan.whipple the error message I got was because of some docker server issue. Once I restarted the error disappeared and I was able to change the permissions on the kms_master_key.us-east-2.enc and create the seed files for followers. I used the seed the files for configuring followers.

I have few follow-up questions.

We generated the seed file after executing the evoke keys decrypt-all command. Once the seed files are generated. What actions needs to taken on the master to encrypt the keys again and when that needs to be done? is that needs to ne done after configuring the followers and standbys or it dos not matter. I am assuming it can be done using the following commands on the master

sv stop conjur pg nginx
evoke keys encrypt
evoke keys unlock
sv start conjur pg nginx

For testing I used the seed file to configure two followers one EC2 instance was assigned the conjur-appliance role and other was not. I was able to configure both followers and both were able to connect to the master and I did not have to use the master key while unpacking the seedfile or configuring the followers. If you could explain how we need to secure the keys present on the followers and standby as well it will be very helpful.

#11

Great questions. The reason we need to decrypt the keys prior to generating a follower seed file is that the command evoke seed follower myfollower.nate.lab is going to check for a certificate named /opt/conjur/etc/ssl/myfollower.nate.lab.pem. If that certificate file is not found, and you are using self-signed certificates, then evoke will automatically issue a new certificate, then generate the seed. However, to generate that certificate the keys had to be decypted and the seed file is thus created with unencrypted contents. That’s why you were able to deploy followers successfully even without the IAM role assigned.

Now typically we would be expecting to use certificates signed by an external CA, so those certificates would be present already after you did the evoke ca import steps. Once you’ve imported 3rd party certificates, you would then run evoke keys encrypt and your seed generation would work as expected (because the certificate is already present, evoke isn’t going to attempt to issue one). To accomplish a similar workflow with self signed certificates, the process should go something like this:

  • run evoke keys decrypt-all
  • run evoke ca issue myfollower.nate.lab, this will create the certificate for the follower
  • run evoke keys encrypt
  • run evoke seed follower myfollower.nate.lab to generate your seed

I’ll point our tech writer team to this thread, as clearly we have some gaps in the documented steps and processes necessary to achieve the desired outcome. Many apologies for that, and kudos for your patience and determination.

Regards,
Nate

#12

@nathan.whipple Thank you I really appreciate your help and patience in explaining this. I think I have figured this out and will update if I run into anything while implementing this in production instances.
I certainly appreciate you involving the document writing team as that would help everyone.

#13

@nathan.whipple I have one follow-up question . Appreciate your patience. I know we have protection through the KMS when we setup the follower/standby for the first time. Does the keys in the follower/standby remain in the encrypted after the configuration process or we have to manually take any action like we did for the master. Also once the initial process is complete and if for some reason the standby takes over as a master in HA cluster auto fail over scenarios is there any additional action that need to taken or considered. Thanks

#14
  • The keys are encrypted on disk, no further action is necessary. You can verify by doing something like docker exec conjur-appliance ls /opt/conjur/etc and verifying all the key files end with an .enc extension
  • No further action is necessary. You’d repair the cluster per your usual process.

Side note: You can run the evoke keys deycrpt-all command on standbys or followers if necessary. Likewise, you can run evoke keys encrypt if the keys are in a decrypted state.

#15

@nathan.whipple . Appreciate your help with this.

I wanted to get your thoughts on the addressing a scenario if we need to deploy conjur in two different AWS regions and also if we need to have followers on prem as well as in AWS.

In the process that is being explained above we are able to to use the one IAM role on the master, standby and followers which has permissions to decrypt the keys using the key which is generated for the specific region.

Since KMS is specific to a region lets assume that we are deploying the secondary standbys which do not have participate in the auto failover in secondary AWS region. It will be great if you can explain how should the key generation and encrypting the standby would need to be approached? Also would that have any impact if we need to promote the secondary standby as the master in an event that we loose one AWS region?

If we need to deploy followers on Prem when the master key is encrypted using the Amazon, how should we deploy the followers? I am assuming that we need to decrypt the keys before generating the seedfile for the follower and once the follower is configured we need to probably use random key to encrypt the keys present on the follower as KMS will be unavailable. I would appreciate if you could provide some clarity on this as well. Thank you

#16

@joe.garcia @izgerij @jody.hunt Any ideas on approaching this will be very helpful.

#17

Hi @sjohnkennedy ,

It is possible to create seed files for a single DAP environment that utilize different encryption strategies. We don’t talk about it much because it’s not a great UX, but the over view goes something like this:

  1. Deploy your master and setup your first key encryption strategy
  2. Encrypt your keys with that strategy
  3. Generate your seed files
  4. Decrypt the keys, switch encryption strategies
  5. Encrypt keys with the new strategy
  6. Generate your seed files
  7. Repeat the steps above until you have seed files for all desired encryption strategies

You can see how this starts to get unwieldy rather quickly. It’s also not documented, so there is that problem as well. All that said, for two AWS regions it shouldn’t be too bad. You’d use a throw-away instance in both regions to configure the KMS integration in each, then copy the key blobs from each to your master. The key blob tells the master which region/key to use when encrypting the keys. Encrypt the keys with region 1, create seeds, decrypt keys and re-encrypt with region 2, create seeds. The node in that region will use the key in that region to unlock the other keys.

As for your last question with the Followers on prem, you could follower the steps you mentioned and decrypt the keys, create your seed, deploy on-prem followers, then wrap their keys with a file based random string key. However, the file based approach introduces challenges with how you deliver the key securely when it’s needed. Some customers have done that by using CCP or CP. You could also consider an HSM integration, or even using the AWS client to assume the role in AWS and use KMS from on-prem.

Hopefully I’ve helped here and not muddied the waters too much. I was hesitant to respond at first as I know this can turn into a rather deep rabbit hole.

Regards,
Nate

1 Like
#18

@sjohnkennedy, I misled you on x-region KMS setup. Its a bit of good news though as it’s actually quite easy. You only need to run the evoke keys kms <region> command multiple times, none of the encrypted/decrypting steps I mentioned before are needed. The evoke keys kms <region> command creates the key blob for each region. When the key is needed, the evoke utility checks the well known local address to discover details about the ec2 instance, then uses the key blob that matches the region the instance is deployed in to get the correct key from KMS. I had to build a lab of this to test for real rather than trust how I understood the code, and I’m glad I did. This is what I did in my lab:

  • Deployed a disposable instance in us-east-1
  • Ran evoke keys kms us-east-1
  • Ran evoke keys kms us-west-2
  • Copied the key blobs to my real master in us-east-1
  • Assigned the conjur-appliance role to my master
  • Ran evoke keys encrypt on my master and confirmed the keys were encrypted
  • Deployed a standby (follower would be the same) in us-west-2
  • Created a seed file and unpacked in on my standby, then confirmed the keys were encrypted (to make sure I didn’t mess up my test)
  • Unpacked seed and ran evoke keys exec evoke configure standby to confirm that I could not configure the standby (no IAM role applied yet, again to make sure I didn’t make a mistake in my test)
  • Applied the conjur-appliance IAM role to my standby
  • Ran evoke keys exec evoke configure standby again and successfully configured the standby

Sorry for the confusion, and hopefully the above clarifies how to work with x-region support with KMS. Let me know how your own testing goes!

Regards,
Nate

1 Like
#19

Thanks Nate for following up. I will test this out when I get sometime and update you.