Need help with AWS ECS Fargate tasks metadata endpoint in the Conjur_iam_client (assume role to fetch secrets)

Hi,

I am using the conjur_iam_client for iam role authentication to conjur from an app (api) deployed on AWS ECS fargate environment. I am getting hung up at the metadata url to use. Andrew’s code for EC2 and other instances uses http://169.254.169.254/latest/meta-data/iam/security-credentials/ and recommended to use create_conjur_iam_client_from_env function (known iam role name, access key, secret key etc).
My challenge is I do not know the access key, secret key and the token for the fargate task that is running on a container spun up by ECS.
Here is the error log I believe just saying the endpoint is not working ( I switched the client to use http://169.254.170.2/v2/metadata for metadata endpoint)

23:06:46
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

23:06:46
INFO: Started reloader process [6]

23:06:51
Process SpawnProcess-1:

23:06:51
Traceback (most recent call last):

23:06:51
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap

23:06:51
self.run()

23:06:51
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run

23:06:51
self._target(*self._args, **self._kwargs)

23:06:51
File "/usr/local/lib/python3.7/site-packages/uvicorn/supervisors/statreload.py", line 29, in handle_fds

23:06:51
target(**kwargs)

23:06:51
File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 307, in run

23:06:51
loop.run_until_complete(self.serve(sockets=sockets))

23:06:51
File "uvloop/loop.pyx", line 1417, in uvloop.loop.Loop.run_until_complete

23:06:51
File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 314, in serve

23:06:51
config.load()

23:06:51
File "/usr/local/lib/python3.7/site-packages/uvicorn/config.py", line 186, in load

23:06:51
self.loaded_app = import_from_string(self.app)

23:06:51
File "/usr/local/lib/python3.7/site-packages/uvicorn/importer.py", line 20, in import_from_string

23:06:51
module = importlib.import_module(module_str)

23:06:51
File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module

23:06:51
return _bootstrap._gcd_import(name[level:], package, level)

23:06:51
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import

23:06:51
File "<frozen importlib._bootstrap>", line 983, in _find_and_load

23:06:51
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked

23:06:51
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked

23:06:51
File "<frozen importlib._bootstrap_external>", line 728, in exec_module

23:06:51
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

23:06:51
File "/rainmaker-api/main.py", line 9, in <module>

23:06:51
api_token= conjur_iam_auth.conjur_hydra_test.fetchsecrets()

23:06:51
File "/rainmaker-api/conjur_iam_auth.py", line 22, in fetchsecrets

23:06:51
conjur_iam_session_token = conjur_iam_client.get_conjur_iam_session_token(appliance_url,account,service_id,username,cert_file)

23:06:51
File "/usr/local/lib/python3.7/site-packages/conjur_iam_client.py", line 159, in get_conjur_iam_session_token

23:06:51
iam_api_key = create_conjur_iam_api_key(iam_role_name, access_key, secret_key, token)

23:06:51
File "/usr/local/lib/python3.7/site-packages/conjur_iam_client.py", line 101, in create_conjur_iam_api_key

23:06:51
access_key, secret_key, token = get_iam_role_metadata(iam_role_name)

23:06:51
File "/usr/local/lib/python3.7/site-packages/conjur_iam_client.py", line 49, in get_iam_role_metadata

23:06:51
raise IAMRoleNotAvailableException()

23:06:51
conjur_iam_client.IAMRoleNotAvailableException: Most likely the ec2 instance is configured with no or an incorrect iam role

23:06:51
INFO: Stopping reloader process [6]

Hi VAMSI,

It looks you are on the right path with changing the metadata url.
I plan on testing this out with ECS within the next couple of days.

This link has some valuable information on how ECS handles IAM roles: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html

From the link above:

From inside the container, you can query the credentials with the following command:
curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

Output:
{
“AccessKeyId”: “ACCESS_KEY_ID”,
“Expiration”: “EXPIRATION_DATE”,
“RoleArn”: “TASK_ROLE_ARN”,
“SecretAccessKey”: “SECRET_ACCESS_KEY”,
“Token”: “SECURITY_TOKEN_STRING”
}

So I would double check the metadata url. And I will take a look at the code and get back to you.

Thanks,
Andrew

I think I found it:

The code is concatenating the iam role name to the metadata url before sending it out.

You could remove + role_name and see if it works as a quick fix.

Moving forward:
Do you know if there is an environment variable present in ECS containers that implies it is an ECS instance?

Regards,
Andrew

Hi Andrew,

Thanks for the quick response. I have tried your suggestion but could not get the access id etc credentials.

File “/rainmaker-api/main.py”, line 9, in

api_token= conjur_iam_auth.conjur_hydra_test.fetchsecrets()

File “/rainmaker-api/conjur_iam_auth.py”, line 22, in fetchsecrets

iamrolename=conjur_iam_client.get_iam_role_metadata(iam_role_name)

File “/usr/local/lib/python3.7/site-packages/conjur_iam_client.py”, line 56, in get_iam_role_metadata

access_key_id = json_dict[“AccessKeyId”]

KeyError: ‘AccessKeyId’

INFO: Stopping reloader process [6]

As for this - Do you know if there is an environment variable present in ECS containers that implies it is an ECS instance – there are quite a few as you can see in the example metadata response json on this page

https://docs.aws.amazon.com/AmazonECS/latest/userguide/task-metadata-endpoint-fargate.html#task-metadata-endpoint-fargate-response
including the LaunchType (EC2 vs FARGATE), but cannot think of anything that distinguishes a container running on ECS vs a
container running on a EC2 instance (VM). The container running on an EC2 instance is working with conjur using your original code as expected.

I have also tried using this metadata endpoint
http://169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
but getting a connection error with this one.

Thanks.

image002.png

image002.jpg

I tried to replicate this in my lab but I am running into some issues with setting up a ECS cluster.

I think $AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is an environment variable within the container that points to the endpoint that will return the access key, secret key, etc.

Could you try to change:

To:
r = requests.get(AWS_METADATA_URL + os.environ['AWS_CONTAINER_CREDENTIALS_RELATIVE_URI'])

Where:
AWS_METADATA_URL=“http://169.254.170.2

I will need more time to test this on my side.

Thanks for your patience,
Andrew

Tried that, with not much luck on metadata url handshake going through.

api_token= conjur_iam_auth.conjur_hydra_test.fetchsecrets()

File “/rainmaker-api/conjur_iam_auth.py”, line 22, in fetchsecrets

conjur_iam_session_token = conjur_iam_client.get_conjur_iam_session_token(appliance_url,account,service_id,username,cert_file)

File “/usr/local/lib/python3.7/site-packages/conjur_iam_client.py”, line 160, in get_conjur_iam_session_token

iam_api_key = create_conjur_iam_api_key(iam_role_name, access_key, secret_key, token)

File “/usr/local/lib/python3.7/site-packages/conjur_iam_client.py”, line 99, in create_conjur_iam_api_key

iam_role_name = get_iam_role_name()

File “/usr/local/lib/python3.7/site-packages/conjur_iam_client.py”, line 44, in get_iam_role_name

File “/usr/local/lib/python3.7/site-packages/conjur_iam_client.py”, line 44, in get_iam_role_name

r = requests.get(AWS_METADATA_URL + os.environ[‘AWS_CONTAINER_CREDENTIALS_RELATIVE_URI’])

r = requests.get(AWS_METADATA_URL + os.environ[‘AWS_CONTAINER_CREDENTIALS_RELATIVE_URI’])

File “/usr/local/lib/python3.7/os.py”, line 678, in getitem

File “/usr/local/lib/python3.7/os.py”, line 678, in getitem

raise KeyError(key) from None

KeyError: ‘AWS_CONTAINER_CREDENTIALS_RELATIVE_URI’

INFO: Stopping reloader process [6]

image002.png

image002.jpg

The issue might be resolved with: https://stackoverflow.com/a/55466354
It seems like AWS_CONTAINER_CREDENTIALS_RELATIVE_URI should be set.

Summary of the stackoverflow:

In summary; make sure to both have a role for 1) executionRoleArn for access to run the task and 2) taskRoleArn for access to make API requests to authorized AWS services set in your task definition.

Regards,
Andrew

I ran this command in docker file and got an output printed, it doesn’t have the expected environment variable

AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

RUN echo ‘export $(strings /proc/1/environ | grep AWS_CONTAINER_CREDENTIALS_RELATIVE_URI)’ >> /root/.profile

Ensured the ecsTaskExecutionRole has this in trust relationship

{

“Version”: “2012-10-17”,

“Statement”: [

{

“Sid”: “”,

“Effect”: “Allow”,

“Principal”: {

“Service”: [

ecs.amazonaws.com”,

ecs-tasks.amazonaws.com

]

},

“Action”: “sts:AssumeRole”

}

]

}

image002.png

image002.jpg