Need help with AWS ECS Fargate tasks metadata endpoint in the Conjur_iam_client (assume role to fetch secrets)

Hi,

I am using the conjur_iam_client for iam role authentication to conjur from an app (api) deployed on AWS ECS fargate environment. I am getting hung up at the metadata url to use. Andrew’s code for EC2 and other instances uses http://169.254.169.254/latest/meta-data/iam/security-credentials/ and recommended to use create_conjur_iam_client_from_env function (known iam role name, access key, secret key etc).
My challenge is I do not know the access key, secret key and the token for the fargate task that is running on a container spun up by ECS.
Here is the error log I believe just saying the endpoint is not working ( I switched the client to use http://169.254.170.2/v2/metadata for metadata endpoint)

23:06:46
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

23:06:46
INFO: Started reloader process [6]

23:06:51
Process SpawnProcess-1:

23:06:51
Traceback (most recent call last):

23:06:51
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap

23:06:51
self.run()

23:06:51
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run

23:06:51
self._target(*self._args, **self._kwargs)

23:06:51
File "/usr/local/lib/python3.7/site-packages/uvicorn/supervisors/statreload.py", line 29, in handle_fds

23:06:51
target(**kwargs)

23:06:51
File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 307, in run

23:06:51
loop.run_until_complete(self.serve(sockets=sockets))

23:06:51
File "uvloop/loop.pyx", line 1417, in uvloop.loop.Loop.run_until_complete

23:06:51
File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 314, in serve

23:06:51
config.load()

23:06:51
File "/usr/local/lib/python3.7/site-packages/uvicorn/config.py", line 186, in load

23:06:51
self.loaded_app = import_from_string(self.app)

23:06:51
File "/usr/local/lib/python3.7/site-packages/uvicorn/importer.py", line 20, in import_from_string

23:06:51
module = importlib.import_module(module_str)

23:06:51
File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module

23:06:51
return _bootstrap._gcd_import(name[level:], package, level)

23:06:51
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import

23:06:51
File "<frozen importlib._bootstrap>", line 983, in _find_and_load

23:06:51
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked

23:06:51
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked

23:06:51
File "<frozen importlib._bootstrap_external>", line 728, in exec_module

23:06:51
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

23:06:51
File "/rainmaker-api/main.py", line 9, in <module>

23:06:51
api_token= conjur_iam_auth.conjur_hydra_test.fetchsecrets()

23:06:51
File "/rainmaker-api/conjur_iam_auth.py", line 22, in fetchsecrets

23:06:51
conjur_iam_session_token = conjur_iam_client.get_conjur_iam_session_token(appliance_url,account,service_id,username,cert_file)

23:06:51
File "/usr/local/lib/python3.7/site-packages/conjur_iam_client.py", line 159, in get_conjur_iam_session_token

23:06:51
iam_api_key = create_conjur_iam_api_key(iam_role_name, access_key, secret_key, token)

23:06:51
File "/usr/local/lib/python3.7/site-packages/conjur_iam_client.py", line 101, in create_conjur_iam_api_key

23:06:51
access_key, secret_key, token = get_iam_role_metadata(iam_role_name)

23:06:51
File "/usr/local/lib/python3.7/site-packages/conjur_iam_client.py", line 49, in get_iam_role_metadata

23:06:51
raise IAMRoleNotAvailableException()

23:06:51
conjur_iam_client.IAMRoleNotAvailableException: Most likely the ec2 instance is configured with no or an incorrect iam role

23:06:51
INFO: Stopping reloader process [6]

Hi VAMSI,

It looks you are on the right path with changing the metadata url.
I plan on testing this out with ECS within the next couple of days.

This link has some valuable information on how ECS handles IAM roles: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html

From the link above:

From inside the container, you can query the credentials with the following command:
curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

Output:
{
“AccessKeyId”: “ACCESS_KEY_ID”,
“Expiration”: “EXPIRATION_DATE”,
“RoleArn”: “TASK_ROLE_ARN”,
“SecretAccessKey”: “SECRET_ACCESS_KEY”,
“Token”: “SECURITY_TOKEN_STRING”
}

So I would double check the metadata url. And I will take a look at the code and get back to you.

Thanks,
Andrew

I think I found it:

The code is concatenating the iam role name to the metadata url before sending it out.

You could remove + role_name and see if it works as a quick fix.

Moving forward:
Do you know if there is an environment variable present in ECS containers that implies it is an ECS instance?

Regards,
Andrew

Hi Andrew,

Thanks for the quick response. I have tried your suggestion but could not get the access id etc credentials.

File “/rainmaker-api/main.py”, line 9, in

api_token= conjur_iam_auth.conjur_hydra_test.fetchsecrets()

File “/rainmaker-api/conjur_iam_auth.py”, line 22, in fetchsecrets

iamrolename=conjur_iam_client.get_iam_role_metadata(iam_role_name)

File “/usr/local/lib/python3.7/site-packages/conjur_iam_client.py”, line 56, in get_iam_role_metadata

access_key_id = json_dict[“AccessKeyId”]

KeyError: ‘AccessKeyId’

INFO: Stopping reloader process [6]

As for this - Do you know if there is an environment variable present in ECS containers that implies it is an ECS instance – there are quite a few as you can see in the example metadata response json on this page

https://docs.aws.amazon.com/AmazonECS/latest/userguide/task-metadata-endpoint-fargate.html#task-metadata-endpoint-fargate-response
including the LaunchType (EC2 vs FARGATE), but cannot think of anything that distinguishes a container running on ECS vs a
container running on a EC2 instance (VM). The container running on an EC2 instance is working with conjur using your original code as expected.

I have also tried using this metadata endpoint
http://169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
but getting a connection error with this one.

Thanks.

image002.png

image002.jpg

I tried to replicate this in my lab but I am running into some issues with setting up a ECS cluster.

I think $AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is an environment variable within the container that points to the endpoint that will return the access key, secret key, etc.

Could you try to change:

To:
r = requests.get(AWS_METADATA_URL + os.environ['AWS_CONTAINER_CREDENTIALS_RELATIVE_URI'])

Where:
AWS_METADATA_URL=“http://169.254.170.2

I will need more time to test this on my side.

Thanks for your patience,
Andrew

Tried that, with not much luck on metadata url handshake going through.

api_token= conjur_iam_auth.conjur_hydra_test.fetchsecrets()

File “/rainmaker-api/conjur_iam_auth.py”, line 22, in fetchsecrets

conjur_iam_session_token = conjur_iam_client.get_conjur_iam_session_token(appliance_url,account,service_id,username,cert_file)

File “/usr/local/lib/python3.7/site-packages/conjur_iam_client.py”, line 160, in get_conjur_iam_session_token

iam_api_key = create_conjur_iam_api_key(iam_role_name, access_key, secret_key, token)

File “/usr/local/lib/python3.7/site-packages/conjur_iam_client.py”, line 99, in create_conjur_iam_api_key

iam_role_name = get_iam_role_name()

File “/usr/local/lib/python3.7/site-packages/conjur_iam_client.py”, line 44, in get_iam_role_name

File “/usr/local/lib/python3.7/site-packages/conjur_iam_client.py”, line 44, in get_iam_role_name

r = requests.get(AWS_METADATA_URL + os.environ[‘AWS_CONTAINER_CREDENTIALS_RELATIVE_URI’])

r = requests.get(AWS_METADATA_URL + os.environ[‘AWS_CONTAINER_CREDENTIALS_RELATIVE_URI’])

File “/usr/local/lib/python3.7/os.py”, line 678, in getitem

File “/usr/local/lib/python3.7/os.py”, line 678, in getitem

raise KeyError(key) from None

KeyError: ‘AWS_CONTAINER_CREDENTIALS_RELATIVE_URI’

INFO: Stopping reloader process [6]

image002.png

image002.jpg

The issue might be resolved with: https://stackoverflow.com/a/55466354
It seems like AWS_CONTAINER_CREDENTIALS_RELATIVE_URI should be set.

Summary of the stackoverflow:

In summary; make sure to both have a role for 1) executionRoleArn for access to run the task and 2) taskRoleArn for access to make API requests to authorized AWS services set in your task definition.

Regards,
Andrew

I ran this command in docker file and got an output printed, it doesn’t have the expected environment variable

AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

RUN echo ‘export $(strings /proc/1/environ | grep AWS_CONTAINER_CREDENTIALS_RELATIVE_URI)’ >> /root/.profile

Ensured the ecsTaskExecutionRole has this in trust relationship

{

“Version”: “2012-10-17”,

“Statement”: [

{

“Sid”: “”,

“Effect”: “Allow”,

“Principal”: {

“Service”: [

ecs.amazonaws.com”,

ecs-tasks.amazonaws.com

]

},

“Action”: “sts:AssumeRole”

}

]

}

image002.png

image002.jpg

Hmm this is odd. I have setup a fargate instance within my lab and I am able to retrieve the access, secret and token from the following endpoint:

169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI. 

This environment variable is only available to only 1 specific process within the fargate container and this process being the process that is executed on the last CMD line of the Dockerfile.

Thanks,
Andrew

Hi Andrew, sorry for the late reply, we got reprioritized and that project is on hold, so couldn’t get the ECS going to verify.
On another subject, can you recommend good resources (docs, code sample - python would be great) for machine identity conjur access implementation?
Thank you
Vamsi