Best practice advice for apps that use both secretless broker and init container for auth

In apps that have both a secretless broker and an init authentication container (for injecting secrets that are impossible to use in a broker scenario), the containers must have a unique name (Kubernetes requirement), but DAP requires that you specify the name of the authenticator container as an annotation in the policy. I haven’t experimented with this yet, but I need to see if I can add multiple annotations to a host grant or if I can add multiple, identical grants to a host with only the annotation being different. Or maybe there is a better practice?

Hey @jgarabedian -

I would suggest in this case using separate application identities for the authentication init container and for secretless. They can both be members of the same layer that inherit access to all of the secrets for the application, if desired - or you can limit each to the set of secrets that each will require (eg Secretless only has access to the secrets it needs to open connections to the target services it will be configured to connect to).

Hope this helps - please follow up here if you have any additional questions.

Geri

Hi @izgerij -

Thanks for your reply. @jgarabedian kindly posted this question on my behalf. Please let me know if I have understood your suggestion correctly, because this approach – while very sensible – seems to run afoul of a hidden DAP requirement for host IDs and is causing a 401 when the secretless sidecar attempts to authenticate to a DAP follower in the cluster. From what I can tell, because the authentication request is coming from a pod running in the “ut-dev” cluster within the “fx” namespace using the “address” deployment, the host ID is required to be “host/conjur/authn-k8s/ut-dev/apps/fx/deployment/address”. I thought the host ID was an arbitrary string, but I suppose there is some underlying security checks. Here are the logs from secretless showing that a request to “inject_client_cert” fails with a 401:

INFO: 2020/02/28 23:05:56 authenticator.go:181: CAKC005I Trying to login Conjur...
INFO: 2020/02/28 23:05:56 authenticator.go:113: CAKC007I Logging in as user &{host/conjur/authn-k8s/ut-dev/apps/fx/deployment/address/secretless host.conjur.authn-k8s.ut-dev.apps.fx deployment.address.secretless}.
2020/02/28 23:05:56 Info: Conjur provider is authenticating as &{host/conjur/authn-k8s/ut-dev/apps/fx/deployment/address/secretless host.conjur.authn-k8s.ut-dev.apps.fx deployment.address.secretless} ...
INFO: 2020/02/28 23:05:56 requests.go:23: CAKC011I Login request to: https://dap-follower.dap.svc.cluster.local/authn-k8s/ut-dev/inject_client_cert
ERROR: 2020/02/28 23:05:56 authenticator.go:133: CAKC029E Received invalid response to certificate signing request. Reason: status code 401, 
ERROR: 2020/02/28 23:05:56 authenticator.go:184: CAKC015E Login failed
2020/02/28 23:05:56 Info: Conjur provider received an error on authenticate: CAKC015E Login failed

Here are the relevant resources in the policy. As you can see, a host for the “authenticator” (which is used by summon) container and a host for “secretless” have been created with the appropriate annotation and have been permitted to authenticate, but this presumably conflicts with what DAP expects the origination host to be:

- !host
  id: conjur/authn-k8s/ut-dev/apps/fx/deployment/address/authenticator
  annotations:
    kubernetes/authentication-container-name: authenticator

- !permit
  role: !host conjur/authn-k8s/ut-dev/apps/fx/deployment/address/authenticator
  privilege: [ authenticate ]
  resource: !webservice conjur/authn-k8s/ut-dev

- !host
  id: conjur/authn-k8s/ut-dev/apps/fx/deployment/address/secretless
  annotations:
    kubernetes/authentication-container-name: secretless

- !permit
  role: !host conjur/authn-k8s/ut-dev/apps/fx/deployment/address/secretless
  privilege: [ authenticate ]
  resource: !webservice conjur/authn-k8s/ut-dev

And here are the debug logs on the DAP follower in the cluster showing a CSRNamespaceMismatch error:

<14>1 2020-02-28T23:04:52.000+00:00 dap-follower-7f985798f5-2cv47 conjur-possum 15514 - [meta sequenceId="1"] [origin=10.233.103.11] [request_id=a20f3c76-ab64-4bcf-805b-bb1dbb5a3ca6] [tid=15571] Started POST "/authn-k8s/ut-dev/inject_client_cert" for 10.233.103.11 at 2020-02-28 23:04:52 +0000
<14>1 2020-02-28T23:04:52.000+00:00 dap-follower-7f985798f5-2cv47 conjur-possum 15514 - [meta sequenceId="2"] [origin=10.233.103.11] [request_id=a20f3c76-ab64-4bcf-805b-bb1dbb5a3ca6] [tid=15571] Processing by AuthenticateController#k8s_inject_client_cert as HTML
<14>1 2020-02-28T23:04:52.000+00:00 dap-follower-7f985798f5-2cv47 conjur-possum 15514 - [meta sequenceId="3"] [origin=10.233.103.11] [request_id=a20f3c76-ab64-4bcf-805b-bb1dbb5a3ca6] [tid=15571]   Parameters: {"service_id"=>"ut-dev"}
<15>1 2020-02-28T23:04:52.000+00:00 dap-follower-7f985798f5-2cv47 conjur-possum 15514 - [meta sequenceId="4"] [origin=10.233.103.11] [request_id=a20f3c76-ab64-4bcf-805b-bb1dbb5a3ca6] [tid=15571]   Sequel::Postgres::Database (0.3ms)  BEGIN
<15>1 2020-02-28T23:04:52.000+00:00 dap-follower-7f985798f5-2cv47 conjur-possum 15514 - [meta sequenceId="5"] [origin=10.233.103.11] [request_id=a20f3c76-ab64-4bcf-805b-bb1dbb5a3ca6] [tid=15571] Authentication Error: #<Errors::Authentication::AuthnK8s::CSRNamespaceMismatch: CONJ00023E Namespace in SPIFFE ID 'deployment' must match namespace implied by common name 'fx'>

Thank you in advance for any clarification on this topic.

Hey @jburns -

In the policy sample you gave, are the deployment names for Secretless and the authenticator container address/secretless and address/authenticator, respectively? The form of the deployment-based application identity is usually something like conjur/authn-k8s/{authenticator/service ID}/apps/{namespace}/deployment/{deployment_name} (based on the application identity docs here). I would expect the Secretless and authenticator containers to be defined in the same deployment as the application - without seeing your manifest, it’s hard to tell if this is the case or not.

If you are using the authenticator client and Secretless together, I would suggest mounting the Conjur access token retrieved by the authenticator into the Secretless container in the same way that you mount the access token into the application container (there are some example manifests that do this in the docs here).

Once you mount the access token into the Secretless container, you can configure it to communicate with DAP using the access token the same as you would configure your application - by specifying:

  • CONJUR_AUTHN_TOKEN_FILE - eg /run/conjur/access-token
  • CONJUR_APPLIANCE_URL - eg https://conjur-follower.<CONJUR_NAMESPACE_NAME>.svc.cluster.local/api
  • CONJUR_ACCOUNT
  • CONJUR_SSL_CERTIFICATE

in the Secretless container environment.

This does mean that you have to give Secretless and the authenticator the same secret access in DAP, but you do get the added benefit of not having to worry about your app logging the secrets to connect to the resources that Secretless is managing.

We are hoping to add many new connectors for Secretless this year - which ones are we missing to enable you to go full-Secretless in this workflow?

If you are interested, we also released a new SDK for contributing Secretless connectors. In addition, we have an open issue for enabling container-based identity here - if this workflow is appealing to you and is something you would use, I encourage you to leave your feedback on that issue.

I look forward to hearing more from you - but if following this post we find there are still questions about your deployment configuration, it might be a good time to open up a Salesforce case to dig a little deeper into your specific problem.

Thanks, and I hope this is helpful -

Geri

1 Like

Thank you @izgerij - this has been an edifying discussion.

The deployment name is just “address”, but because I wanted to use summon and secretless, I was running secretless and conjur-appliance containers as sidecars. Because I couldn’t name the containers the same, it was causing issues when the host was trying to authenticate – the kubernetes annotation would only ever match one container name for the host. This could potentially be alleviated by running a single secretless container that mounts the API token on the application container. On that subject…

It appears that secretless will not mount the API token until it is required to broker a connection – is it possible to change this behavior? It is causing applications that require summon to hang for lack of a token. For example:

Deployment “address”, which uses only secretless, mounts the token in both containers after a brokered connection:

$ k exec address-b855d89c-92bx9 -c address -- ls /var/run/conjur
access-token
$ k exec address-b855d89c-92bx9 -c authenticator -- ls /var/run/conjur
access-token

Deployment “fx-holiday”, which uses summon to inject secrets on startup, forever hangs while waiting in vain for secretless to create the access-token file:

$ k exec fx-holiday-754fd8c7d7-6mdk6 -c authenticator -- ls /var/run/conjur
$ k exec fx-holiday-754fd8c7d7-6mdk6 -c fx-holiday -- ls /var/run/conjur

Can I pass a configuration to secretless to fetch the token on startup?

MSSQL and HTTP connectors cover the majority of our authenticated connections, so I’m hoping to use secretless for everything! I find the container-based identity interesting and useful for certain workflows, but as of now that is not something we would need.

Thanks again for your detailed, helpful reply.

Josh

Re: updating Secretless to start the Conjur authentication process on startup and inject the Conjur auth token into the app container -

If this is a workflow that you’re interested in, I’d encourage you to submit a feature request to Secretless with this suggestion.

Given that we design Secretless to be a secure enclave that is intended to take away the secrets management responsibilities from the app, my sense is that this is not a feature we would want to build - but if you start the conversation there, we may find that others feel differently :slight_smile:

I’ve also raised with our product team that we should consider building some better tools to enable users to leverage the Kubernetes authenticator for direct-to-DAP app access together with Secretless, when the app requires access to target services that Secretless doesn’t support. We do hope that over time Secretless supports the vast majority of target services you’d need (and it sounds like it nearly does, for your use case!), but there could still be occasions where it doesn’t - and it’d be nice if there were still an easy way to connect to those services (ie perhaps by leveraging Summon).

In summary:

  • If you’d like us to explore (perhaps optionally?) enabling Secretless to kick off DAP authentication on startup and to facilitate injecting the DAP access token into the application container, please submit a feature request to Secretless
  • It sounds like you’d prefer not to run two sidecars if you can avoid it, so that the following options aren’t appealing:
    • Running the authenticator client and configuring it to inject the DAP access token into the app container AND the Secretless container
    • Pushing for the container-based authentication so that the authenticator client and Secretless can run their own DAP authentication workflows, with their own DAP identities and privileges
  • For now, the best possible workflow available to you is running the authenticator client and injecting the access token it retrieves from DAP into the Secretless container and the app container.

I hope that’s a good summary! I’ll continue to give some thought to how we can make this easier, and I’d be glad to consider any additional suggestions you might have.

Thanks!
Geri

The container-based authentication would indeed be a solution here, but it seems like overkill for what we’re needing – I don’t actually need to limit access to secrets between containers in a deployment, it would just be a mechanism for running multiple authenticators under a single deployment.

I’m happy to report that running the authenticator and having it inject the access token into both secretless and the application container on startup has been a success! I eventually want to move 100% of secret management to secretless, so this is a stop gap measure that I’m willing to entertain during the interstitial migration period.

Thank you so much for your help!

1 Like