S3 Pre-signed URL generation from ECS using Task role - amazon-web-services

I want to create a s3 presigned url for reading an object in S3 to my clients. My application is running in ECS.
I want to use the ECS Task Role to create the S3 Pre-signed URL using python sdk like this
s3_client.generate_presigned_url('get_object',
Params={'Bucket': bucket_name,
'Key': object_name},
ExpiresIn=expiration)
Question:
If a client receives a presigned url right at the boundary of task role credential rotation wont it stop working ?
This article mentions to use permanent credentials - https://aws.amazon.com/premiumsupport/knowledge-center/presigned-url-s3-bucket-expiration/
If you created a presigned URL using a temporary token, then the URL expires when the token expires, even if the URL was created with a later expiration time.
Is there a way to make sure the presigned url is valid around the credential rotation boundary. I would like to provide atleast 10 mins of validity for the presigned url.
Note: This answer also recommends using IAM user credentials - Avoid pre-signed URL expiry when IAM role key rotates
I am thinking if there is any way ECS can take advantage of the Task Role ?

By using the ECS task role alone you are limited to whenever it expires for your signed URL. The credentials by default last 6 hours but you would need to validate the meta-data endpoint to understand how long is left.
An example response from the meta-data endpoint is below, as you can see there's a attribute containing the Expiration value.
{
"AccessKeyId": "ACCESS_KEY_ID",
"Expiration": "EXPIRATION_DATE",
"RoleArn": "TASK_ROLE_ARN",
"SecretAccessKey": "SECRET_ACCESS_KEY",
"Token": "SECURITY_TOKEN_STRING"
}
If it must be at least 10 minutes you can do this by creating another role (one that has the permissions) then using STS with assume-role. One of the argument you can pass is duration-seconds which provides upto 12 hours to be specified.
If you do this you can then assume the role and generate the presigned URL, which can be used for the length of the duration-seconds you specified. Your task role would have permissions to assume the role, which would mean you do not require an IAM user.
This only works if you require the link for shorter than 12 hours, otherwise you would have been limited to IAM user.

If a client receives a presigned url right at the boundary of task role credential rotation wont it stop working
Yes. Pre-signed urls are linked to the IAM entities that created them. Thus, in your case, if you generate the url just before IAM role expires, the url will expire as well. This happens regardless of expiry time of the url itself.
To avoid that, IAM user should be used to generated the pre-signed urls, since IAM user's credentials are permanent, unlike those of IAM roles.
You can also reduce the impact of the role's credentials expire time by increasing it to 12h for example:

Related

AWS IAM and java sdk service clients - How does credentials are generated using role associated with a service?

I am running java app on ECS fargate and have attached a role with the fargate task to perform s3 operations. From my java process i have started using DefaultCredentialsProviderChain.java from aws sdk to get the credentials and create s3 client. I am facing issues to understand below mentioned questions:
Which class in the chain (in DefaultCredentialsProviderChain.java) gets credentials (temporary key and secret access key) from the role associated ? And For how long these temp credentials are valid ?
Can i cache the s3 client created with credentials from DefaultCredentialsProviderChain.java so that on java service startup s3 client will be built once or s3 client needs to be created each and every time any s3 operation needs to be performed?
I followed below mentioned link but did not get my answers, Can anyone please help me to understand these queries.
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html#credentials-chain
Which class in the chain (in DefaultCredentialsProviderChain.java) gets credentials
As you have attached a runtime role, the AWS will provide a metadata service. The credential provider will use the service to fetch the runtime credentials. By default the runtime credentials are valid for one hour (as far I recall).
https://docs.aws.amazon.com/AmazonS3/latest/userguide/AuthUsingTempSessionToken.html
Can i cache the s3 client created with credentials
Yes you can. The credential provider keeps track of the session lifetime and refresh the token some time before expiration (as well - as far I remember).
This actually places some limits when creating a presigned url. The presigned url is valid only until the credentials are valid. So it may happen that you create a presigned url and the url will be valid shorter time than expected

Create temporary credentials to upload file to S3 bucket

I need to validate client and then generate temporary credentials (valid for few seconds) using which client can upload a file on my S3 bucket. I cannot create a user for the client. First I validate the client using OAuth and if the client is valid, I need to enable it to upload the file to S3. I know about presigned URL way, but am wondering is there another way.
using which client can upload a file on my S3 bucket.
..
know about presigned URL way, but am wondering is there another way
As already answered, I see two ways. The presigned url or assumed IAM role (e. g. though cognito or own/custom identity broker)
There is a significant difference.
While using the presign url, it allows the client to upload/update a very specific object defined by the url in S3. I like this approach because of its simplicity, more control over expiration and I imho more secure (less work around managing permissions)
When using the assumed credentials, you may give the user more privileges (e. g. upload any object with specific prefix, tag the object, ,..). However you may have more work to manage the permission and control the expiration (default lifetime of the assumed role credentials is 15min and can be prolonged to 12h).
First I validate the client using OAuth
Still you may create a presigned url using the assumed (temporary) identity
You could use Amazon Cognito with OAuth. This will assume an IAM role and generate temporary credentials for you.

Cannot download from AWS S3: Access Denied

So I'm hosting an web app on EC2 which let users upload/download photos from S3. EC2 and S3 are in the same region.
Credentials: Following the official docs, credentials should be automatically supplied by the IAM role(with s3fullAccess permission) I created and associated to EC2.
Problem: User (from browser) can't download photos, but can successfully upload them. Download link was generated by aws-sdk's getSignedUrl() method.
Error Message:
InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
Firstly, access key seems correct as users can successfully upload stuffs.
My guess is that, since the user clicks link in the browser to download, there's no interaction between EC2 and S3, thus IAM role is not used when downloading. But isn't the signedDownloadUrl generated with EC2 credentials? And there's access key pairs supplied altogether in the query string. So my guess might be wrong..
Any suggestion/idea is appreciated.
Edit
I didn't create an IAM user, only have an IAM role. Don't know if this matters.
ASIA... is a temporary access key id from a set of temporary credentials for an IAM role. (Role credentials are always temporary.) AKIA... is an IAM user.
The problem here is that the x-amz-security-token accompanying the credentials was not used in the signing process that generated the S3 signed URL, so it doesn't appear in the signed URL... but it needs to be there for this to work.
If you are signing your request using temporary security credentials [...] you must include the corresponding security token in your request by adding the x-amz-security-token header.
http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html#UsingTemporarySecurityCredentials
If you don't include x-amz-security-token, the system assumes it should look up the access-key-id in the IAM ussr database, and it won't be found there, which causes the InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records error.
Note that you can't simply add it to the already-generated URL. It needs to be incorporated into the signing process.

How do you increase the token expiration time for a boto3.s3.transfer download?

I am using boto3's s3.transfer to download several 4GB+ files from s3. All but one were able to download, but the one that failed gave the following error:
ERROR: An error occurred (ExpiredToken) when calling the GetObject operation: The provided token has expired.
I am using it the same way it is document at http://boto3.readthedocs.org/en/latest/_modules/boto3/s3/transfer.html
s3_client = session.client('s3')
transfer = S3Transfer(s3_client)
# Download s3://bucket/key to /tmp/myfile
transfer.download_file('bucket', 'key', '/tmp/myfile')
Is there a way to increase the expiration time of the signed url used inside boto3?
In case it is relevant, I am using Cognito to get the credentials, and with them, a session
client = boto3.client('cognito-identity', AWS_REGION)
# credentials[] contains the IdentityId and Token I get from my server
# which I get using client.get_open_id_token_for_developer_identity
# with TokenDuration=86400
resp = client.get_credentials_for_identity(IdentityId=credentials['IdentityId'],
Logins={'cognito-identity.amazonaws.com': credentials['Token']})
# The resp contains the actual temporary AWS secret/access codes and a session token, to be
# used with the rest of the AWS APIs
secretKey = resp['Credentials']['SecretKey']
accessKey = resp['Credentials']['AccessKeyId']
sessionToken = resp['Credentials']['SessionToken']
session = Session(aws_access_key_id=accessKey,
aws_secret_access_key=secretKey,
aws_session_token=sessionToken,
region_name=AWS_REGION)
s3_client = session.client('s3')
The problem you are experiencing is not linked to S3 signed URL as you are supposing.
Cognito is build on top of an IAM service called Security Token Service (STS). This service allows to generate temporary credentials (access key and secret key) by assuming a role (IAM user, EC2 instance, Lambda function etc ...) or by providing Web Identity token, for federated identities scenarios, using Google, Facebook, Amazon.
These credentials are limited in scope (to whatever IAM Role you have defined) and in time, between 15 secs and several hours, depending on the use case.
The credentials you are obtaining through Cognito are generated by STS. At low level, STS API allows to specify how long you want these credentials to stay valid (see http://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html) . However, I can not find an equivalent in Cognito API (https://docs.aws.amazon.com/cognitoidentity/latest/APIReference/API_GetCredentialsForIdentity.html). I would love to be proven wrong on that point.
Why do you see the error ?
My guess, based on the elements you provided, is that your download code is running longer than the life of the temporary credentials you received. First download works, but later does not.
How to workaround it ?
At low level, a clean solution would be to use STS instead of Cognito, but that
would require a huge amount of work and your will loose all the
benefits using Cognito (stable user ID across login providers,
multiple login providers, unauthenticated users ...)
Another solution, assuming you have multiple file transfers, in a loop, would be to check credentials expiration time, and renew them in between file transfer. Check resp['Credentials']['Expiration'] for the expiration time. You can renew Cognito provided credentials by calling get_credentials_for_identity again.
You can also consider downloading your file through one of the 50+ edge locations provided by AWS. This is a new S3 capability, just released this week, that considerably speed up the upload or download of large files. See http://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html for more details. There is price tag associated to that usage, see http://aws.amazon.com/s3/pricing/

S3: IAM vs Temporary credentials

I am planning to use S3 for my application. I want to grant permission for users of my application to access data stored in S3 for sometime and they will use GET on the objects stored.
Can somebody explain which one is preferred and why for my requirement ??
Thanks in advance.
UPDATE
Both IAM and temporary credentials can generate the signature for URL. Is there any advantage in using one than other ?? Is my understanding correct w.r.t temporary credentials ??
question is a bit misleading.
you have 3 things here:
1) IAM credentials
2) Temporary credentials via IAM STS
3) Presigned URL
If you do the presigned URL you can just provide the URL and the user can consume it before it expires.
It really does not matter if you are using permananent or temporary IAM credentials to generate the presigned URL as far as the URL is concerned.
If you are on an EC2 instance I would avocate for using an instance role and giving that the permissions needed to do the signing and let the sdk/cli handle it for you (it will under the hood use temporary credentials via IAM STS)
In my opinion, if you need to access to a S3 bucket from S3 Client for AWS SDK, I will use Temoporary credentials.
If you need to access to S3 bucket from a link of the webview, I will use Signed URLs.