Need help with AWS ECS Fargate tasks metadata endpoint in the Conjur_iam_client (assume role to fetch secrets)

vmaddirala · January 31, 2020, 11:37pm

Hi,

I am using the conjur_iam_client for iam role authentication to conjur from an app (api) deployed on AWS ECS fargate environment. I am getting hung up at the metadata url to use. Andrew’s code for EC2 and other instances uses http://169.254.169.254/latest/meta-data/iam/security-credentials/ and recommended to use create_conjur_iam_client_from_env function (known iam role name, access key, secret key etc).
My challenge is I do not know the access key, secret key and the token for the fargate task that is running on a container spun up by ECS.
Here is the error log I believe just saying the endpoint is not working ( I switched the client to use http://169.254.170.2/v2/metadata for metadata endpoint)

23:06:46
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

23:06:46
INFO: Started reloader process [6]

23:06:51
Process SpawnProcess-1:

23:06:51
Traceback (most recent call last):

23:06:51
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap

23:06:51
self.run()

23:06:51
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run

23:06:51
self._target(*self._args, **self._kwargs)

23:06:51
File "/usr/local/lib/python3.7/site-packages/uvicorn/supervisors/statreload.py", line 29, in handle_fds

23:06:51
target(**kwargs)

23:06:51
File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 307, in run

23:06:51
loop.run_until_complete(self.serve(sockets=sockets))

23:06:51
File "uvloop/loop.pyx", line 1417, in uvloop.loop.Loop.run_until_complete

23:06:51
File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 314, in serve

23:06:51
config.load()

23:06:51
File "/usr/local/lib/python3.7/site-packages/uvicorn/config.py", line 186, in load

23:06:51
self.loaded_app = import_from_string(self.app)

23:06:51
File "/usr/local/lib/python3.7/site-packages/uvicorn/importer.py", line 20, in import_from_string

23:06:51
module = importlib.import_module(module_str)

23:06:51
File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module

23:06:51
return _bootstrap._gcd_import(name[level:], package, level)

23:06:51
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import

23:06:51
File "<frozen importlib._bootstrap>", line 983, in _find_and_load

23:06:51
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked

23:06:51
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked

23:06:51
File "<frozen importlib._bootstrap_external>", line 728, in exec_module

23:06:51
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

23:06:51
File "/rainmaker-api/main.py", line 9, in <module>

23:06:51
api_token= conjur_iam_auth.conjur_hydra_test.fetchsecrets()

23:06:51
File "/rainmaker-api/conjur_iam_auth.py", line 22, in fetchsecrets

23:06:51
conjur_iam_session_token = conjur_iam_client.get_conjur_iam_session_token(appliance_url,account,service_id,username,cert_file)

23:06:51
File "/usr/local/lib/python3.7/site-packages/conjur_iam_client.py", line 159, in get_conjur_iam_session_token

23:06:51
iam_api_key = create_conjur_iam_api_key(iam_role_name, access_key, secret_key, token)

23:06:51
File "/usr/local/lib/python3.7/site-packages/conjur_iam_client.py", line 101, in create_conjur_iam_api_key

23:06:51
access_key, secret_key, token = get_iam_role_metadata(iam_role_name)

23:06:51
File "/usr/local/lib/python3.7/site-packages/conjur_iam_client.py", line 49, in get_iam_role_metadata

23:06:51
raise IAMRoleNotAvailableException()

23:06:51
conjur_iam_client.IAMRoleNotAvailableException: Most likely the ec2 instance is configured with no or an incorrect iam role

23:06:51
INFO: Stopping reloader process [6]

AndrewCopeland · February 1, 2020, 12:14am

Hi VAMSI,

It looks you are on the right path with changing the metadata url.
I plan on testing this out with ECS within the next couple of days.

This link has some valuable information on how ECS handles IAM roles: Task IAM role - Amazon Elastic Container Service

From the link above:

From inside the container, you can query the credentials with the following command:
curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

Output:
{
“AccessKeyId”: “ACCESS_KEY_ID”,
“Expiration”: “EXPIRATION_DATE”,
“RoleArn”: “TASK_ROLE_ARN”,
“SecretAccessKey”: “SECRET_ACCESS_KEY”,
“Token”: “SECURITY_TOKEN_STRING”
}

So I would double check the metadata url. And I will take a look at the code and get back to you.

Thanks,
Andrew

AndrewCopeland · February 1, 2020, 12:21am

I think I found it:

github.com

AndrewCopeland/conjur-iam-api-key/blob/master/conjur_iam_client.py#L47




def get_aws_region():

    # return requests.get(AWS_AVAILABILITY_ZONE).text[:-1]

    return "us-east-1"



def get_iam_role_name():

    r = requests.get(AWS_METADATA_URL)

    return r.text



def get_iam_role_metadata(role_name):

    r = requests.get(AWS_METADATA_URL + role_name)

    if r.status_code == 404:

        raise IAMRoleNotAvailableException()



    json_dict = json.loads(r.text)



    access_key_id = json_dict["AccessKeyId"]

    secret_access_key = json_dict["SecretAccessKey"]

    token = json_dict["Token"]



    return access_key_id, secret_access_key, token

The code is concatenating the iam role name to the metadata url before sending it out.

You could remove + role_name and see if it works as a quick fix.

Moving forward:
Do you know if there is an environment variable present in ECS containers that implies it is an ECS instance?

Regards,
Andrew

vmaddirala · February 1, 2020, 3:42am

Hi Andrew,

Thanks for the quick response. I have tried your suggestion but could not get the access id etc credentials.

File “/rainmaker-api/main.py”, line 9, in

api_token= conjur_iam_auth.conjur_hydra_test.fetchsecrets()

File “/rainmaker-api/conjur_iam_auth.py”, line 22, in fetchsecrets

iamrolename=conjur_iam_client.get_iam_role_metadata(iam_role_name)

File “/usr/local/lib/python3.7/site-packages/conjur_iam_client.py”, line 56, in get_iam_role_metadata

access_key_id = json_dict[“AccessKeyId”]

KeyError: ‘AccessKeyId’

INFO: Stopping reloader process [6]

As for this - Do you know if there is an environment variable present in ECS containers that implies it is an ECS instance – there are quite a few as you can see in the example metadata response json on this page

https://docs.aws.amazon.com/AmazonECS/latest/userguide/task-metadata-endpoint-fargate.html#task-metadata-endpoint-fargate-response including the LaunchType (EC2 vs FARGATE), but cannot think of anything that distinguishes a container running on ECS vs a
container running on a EC2 instance (VM). The container running on an EC2 instance is working with conjur using your original code as expected.

I have also tried using this metadata endpoint
http://169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI but getting a connection error with this one.

Thanks.

AndrewCopeland · February 3, 2020, 8:32pm

I tried to replicate this in my lab but I am running into some issues with setting up a ECS cluster.

I think $AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is an environment variable within the container that points to the endpoint that will return the access key, secret key, etc.

Could you try to change:

github.com

AndrewCopeland/conjur-iam-api-key/blob/master/conjur_iam_client.py#L47




def get_aws_region():

    # return requests.get(AWS_AVAILABILITY_ZONE).text[:-1]

    return "us-east-1"



def get_iam_role_name():

    r = requests.get(AWS_METADATA_URL)

    return r.text



def get_iam_role_metadata(role_name):

    r = requests.get(AWS_METADATA_URL + role_name)

    if r.status_code == 404:

        raise IAMRoleNotAvailableException()



    json_dict = json.loads(r.text)



    access_key_id = json_dict["AccessKeyId"]

    secret_access_key = json_dict["SecretAccessKey"]

    token = json_dict["Token"]



    return access_key_id, secret_access_key, token

To:
r = requests.get(AWS_METADATA_URL + os.environ['AWS_CONTAINER_CREDENTIALS_RELATIVE_URI'])

Where:
AWS_METADATA_URL=“http://169.254.170.2”

I will need more time to test this on my side.

Thanks for your patience,
Andrew

vmaddirala · February 3, 2020, 9:15pm

Tried that, with not much luck on metadata url handshake going through.

api_token= conjur_iam_auth.conjur_hydra_test.fetchsecrets()

File “/rainmaker-api/conjur_iam_auth.py”, line 22, in fetchsecrets

conjur_iam_session_token = conjur_iam_client.get_conjur_iam_session_token(appliance_url,account,service_id,username,cert_file)

File “/usr/local/lib/python3.7/site-packages/conjur_iam_client.py”, line 160, in get_conjur_iam_session_token

iam_api_key = create_conjur_iam_api_key(iam_role_name, access_key, secret_key, token)

File “/usr/local/lib/python3.7/site-packages/conjur_iam_client.py”, line 99, in create_conjur_iam_api_key

iam_role_name = get_iam_role_name()

File “/usr/local/lib/python3.7/site-packages/conjur_iam_client.py”, line 44, in get_iam_role_name

r = requests.get(AWS_METADATA_URL + os.environ[‘AWS_CONTAINER_CREDENTIALS_RELATIVE_URI’])

File “/usr/local/lib/python3.7/os.py”, line 678, in getitem

raise KeyError(key) from None

KeyError: ‘AWS_CONTAINER_CREDENTIALS_RELATIVE_URI’

INFO: Stopping reloader process [6]

AndrewCopeland · February 3, 2020, 9:47pm

The issue might be resolved with: python 3.x - Can't access S3 bucket from within Fargate container (Bad Request and unable to locate credentials) - Stack Overflow
It seems like AWS_CONTAINER_CREDENTIALS_RELATIVE_URI should be set.

Summary of the stackoverflow:

In summary; make sure to both have a role for 1) executionRoleArn for access to run the task and 2) taskRoleArn for access to make API requests to authorized AWS services set in your task definition.

Regards,
Andrew

vmaddirala · February 4, 2020, 5:58pm

I ran this command in docker file and got an output printed, it doesn’t have the expected environment variable

AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

RUN echo ‘export $(strings /proc/1/environ | grep AWS_CONTAINER_CREDENTIALS_RELATIVE_URI)’ >> /root/.profile

Ensured the ecsTaskExecutionRole has this in trust relationship

{

“Version”: “2012-10-17”,

“Statement”: [

{

“Sid”: “”,

“Effect”: “Allow”,

“Principal”: {

“Service”: [

“ecs.amazonaws.com”,

“ecs-tasks.amazonaws.com”

]

},

“Action”: “sts:AssumeRole”

}

]

}

AndrewCopeland · April 1, 2020, 6:07pm

Hmm this is odd. I have setup a fargate instance within my lab and I am able to retrieve the access, secret and token from the following endpoint:

169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI.

This environment variable is only available to only 1 specific process within the fargate container and this process being the process that is executed on the last CMD line of the Dockerfile.

Thanks,
Andrew

vmaddirala · April 29, 2020, 7:34pm

Hi Andrew, sorry for the late reply, we got reprioritized and that project is on hold, so couldn’t get the ECS going to verify.
On another subject, can you recommend good resources (docs, code sample - python would be great) for machine identity conjur access implementation?
Thank you
Vamsi

system · August 8, 2020, 4:38am

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

joe.garcia · May 4, 2022, 12:37am

I know this is an old post, but came across it while Googling for Fargate’s metadata service info. Just an FYI, if you use boto3 in your Python script, it’ll figure out the proper metadata service URI to use whether it’s Fargate, EC2, or whatever. Check out my code here: GitHub - infamousjoeg/authn-iam-ec2-demo: A demonstration using @CyberArk Conjur's authn-iam integration with AWS EC2 – I’m using this in both EC2 and ECS Fargate unchanged with the exception of the IAM Role ARN.

Topic		Replies	Views
How do we integrate AWS ECS with CyberArk Conjur Secrets Management - Conjur, Secrets Hub & CP	4	774	June 2, 2022
Assume IAM role by non AWS machine and authenticate with Conjur Conjur Enterprise	12	1261	October 22, 2019
NEW Conjur authn-iam Python Client Secrets Management - Conjur, Secrets Hub & CP opensource , community-code , community , summon , aws	7	1182	July 16, 2020
Pulling Secrets in Java on EC2 with IAM Secrets Management - Conjur, Secrets Hub & CP cloud , aws	3	1061	January 30, 2021
Installing Conjur in an EKS Kubernetes Cluster with Helm Secrets Management - Conjur, Secrets Hub & CP	0	150	November 16, 2022

Need help with AWS ECS Fargate tasks metadata endpoint in the Conjur_iam_client (assume role to fetch secrets)

Related Topics