So, I'm inside an AWS lambda function. I need to return temp creds from an API endpoint, so that an upload can be performed from browser directly to S3.
The files can be very large, hundreds of gigabytes, so the creds need to last a long time. What's the easiest way to get these type of creds inside a lambda?
The short answer is that you need to assume a role, as I describe in this blog post. A key part of that post is using a session policy to scope the assumed role to a single key on S3.
However, if it takes more than an hour to upload the file, that solution won't work as written, because a Lambda can't assume another role with a requested duration > one hour (see role chaining), a limit that can't be increased.
This means that you need to create a user that can assume the role, and make that user's long-term credentials available to the Lambda (typically via Secrets Manager). Once you've retrieved those credentials, use them to create an STS client (you don't say what language you're using, and I typically use Python, so that's what's shown):
sts_client = boto3.client(
'sts',
aws_access_key_id=stored_access_key,
aws_secret_access_key=stored_secret_key)
Then with those credentials you can assume a role that can write to the file. In following the blog post, the base role has permissions to write to any file on S3, and the session policy limits the assumed role to the specific file:
session_policy = json.dumps({
'Version': '2012-10-17',
'Statement': [
{
'Effect': 'Allow',
'Action': 's3:PutObject',
'Resource': f"arn:aws:s3:::{BUCKET}/{KEY}"
}
]
})
response = sts_client.assume_role(
RoleArn=ASSUMABLE_ROLE_ARN,
RoleSessionName="example",
Policy=session_policy,
DurationSeconds=12 * 3600
)
# these are the credentials that you'd pass to the client application
limited_access_key = response['Credentials']['AccessKeyId']
limited_secret_key = response['Credentials']['SecretAccessKey']
limited_session_token = response['Credentials']['SessionToken']
12 hours is enough time to transfer 500 GB over a 100 mbps connection. If you need to more time than that, then you'll have to create an actual user and return its credentials. You can attach an inline policy to this user to limit its access to the single file (serving the same purpose as the session policy in this example). But since you're limited to 5,000 IAM users in an account, this is not something that you want to do on a regular basis.
Related
I have a usecase to use AWS Lambda to copy files/objects from one S3 bucket to another. In this usecase Source S3 bucket is in a separate AWS account(say Account 1) where the provider has only given us AccessKey & SecretAccess Key. Our Lambda runs in Account 2 and the destination bucket can be either in Account 2 or some other account 3 altogether which can be accessed using IAM role. The setup is like this due to multiple partner sharing data files
Usually, I used to use the following boto3 command to copy the contents between two buckets when everything is in the same account but want to know how this can be modified for the new usecase
copy_source_object = {'Bucket': source_bucket_name, 'Key': source_file_key}
s3_client.copy_object(CopySource=copy_source_object, Bucket=destination_bucket_name, Key=destination_file_key)
How can the above code be modified to fit my usecase of having accesskey based connection to source bucket and roles for destination bucket(which can be cross-account role as well)? Please let me know if any clarification is required
There's multiple options here. Easiest is by providing credentials to boto3 (docs). I would suggest retrieving the keys from the SSM parameter store or secrets manager so they're not stored hardcoded.
Edit: I realize the problem now, you can't use the same session for both buckets, makes sense. The exact thing you want is not possible (ie. use copy_object). The trick is to use 2 separate session so you don't mix the credentials. You would need to get_object from the first account and put_object to the second objects. You should be able to simply put the resp['Body'] from the get in the put request but I haven't tested this.
import boto3
acc1_session = boto3.session.Session(
aws_access_key_id=ACCESS_KEY_acc1,
aws_secret_access_key=SECRET_KEY_acc1
)
acc2_session = boto3.session.Session(
aws_access_key_id=ACCESS_KEY_acc2,
aws_secret_access_key=SECRET_KEY_acc2
)
acc1_client = acc1_session.client('s3')
acc2_client = acc2_session.client('s3')
copy_source_object = {'Bucket': source_bucket_name, 'Key': source_file_key}
resp = acc1_client.get_object(Bucket=source_bucket_name, Key=source_file_key)
acc2_client.put_object(Bucket=destination_bucket_name, Key=destination_file_key, Body=resp['Body'])
Your situation appears to be:
Account-1:
Amazon S3 bucket containing files you wish to copy
You have an Access Key + Secret Key from Account-1 that can read these objects
Account-2:
AWS Lambda function that has an IAM Role that can write to a destination bucket
When using the CopyObject() command, the credentials used must have read permission on the source bucket and write permission on the destination bucket. There are normally two ways to do this:
Use credentials from Account-1 to 'push' the file to Account-2. This requires a Bucket Policy on the destination bucket that permits PutObject for the Account-1 credentials. Also, you should set ACL= bucket-owner-full-control to handover control to Account-2. (This sounds similar to your situation.) OR
Use credentials from Account-2 to 'pull' the file from Account-1. This requires a Bucket Policy on the source bucket that permits GetObject for the Account-2 credentials.
If you can't ask for a change to the Bucket Policy on the source bucket that permits Account-2 to read the contents, then **you'll need a Bucket Policy on the Destination bucket that permits write access by the credentials from Account-1`.
This is made more complex by the fact that you are potentially copying the object to a bucket in "some other account". There is no easy answer if you are starting to use 3 accounts in the process.
Bottom line: If possible, ask them for a change to the source bucket's Bucket Policy so that your Lambda function can read the files without having to change credentials. It can then copy objects to any bucket that the function's IAM Role can access.
I have a scenario in which I want to have access to resources within one account from another one in AWS (cross-account access) in code. And I want to implement this access using NodeJs, implemented as lambda function and also as a long-running code on EC2.
Reading how to do this online, I know I need temporary credentials generated by aws.STS, like this:
const AWS = require('aws-sdk');
const sts = new AWS.STS();
const stsResults = await sts.assumeRole({
RoleArn: 'arn:aws:iam::111111111111:role/role_name',
RoleSessionName: 'STRING_VALUE',
}).promise();
const dynamodb = new AWS.DynamoDB({
region: 'us-east-1',
accessKeyId: stsResults.Credentials.AccessKeyId,
secretAccessKey:stsResults.Credentials.SecretAccessKey,
sessionToken: stsResults.Credentials.SessionToken
});
My question is about the RoleSessionName attribute which is a required one. I'm having a hard time understanding what it does and how I should use it. This is what the AWS documentation has to say about it:
RoleSessionName — (String) An identifier for the assumed role session.
Use the role session name to uniquely identify a session when the same
role is assumed by different principals or for different reasons. In
cross-account scenarios, the role session name is visible to, and can
be logged by the account that owns the role. The role session name is
also used in the ARN of the assumed role principal. This means that
subsequent cross-account API requests that use the temporary security
credentials will expose the role session name to the external account
in their AWS CloudTrail logs.
The regex used to validate this parameter is a string of characters
consisting of upper- and lower-case alphanumeric characters with no
spaces. You can also include underscores or any of the following
characters: =,.#-
Personally, I'm not concerned about security since both accounts are owned by the same company and the only reason to have multiple accounts is to logically separate resources. What I would like to know is the impact of this attribute on the performance of the assumeRole function call. Should I use the same RoleSessionName for all my lambda functions? Should I create a random ID each time I create a new session?
As per the documentation you quoted:
Use the role session name to uniquely identify a session when the same role is assumed by different principals or for different reasons.
Let's say you have an IAM Role and it is assumed by a program. This will return a set of temporary credentials that can be used to access AWS services.
In an audit trail, anything done by the Role will be tracked as having been done by the Role (not by the entity that assumed the Role). This makes it difficult to trace back the source of these API calls, since the role could be assumed by "different principals or for different reasons". For example, multiple programs might use the role.
To assist in tracing the 'origin' of such requests, the RoleSessionName is provided to identify the particular assumption. It's there to help you identify which app is using the credentials.
I am trying to create a route53 recordset from Account1 in Account2.
From reading other posts and online search I am thinking of doing something like this:
from boto3 import Session
session = Session(aws_access_key_id=*****,aws_secret_access_key=****,region_name='us-east-1')
r53_client = session.client('route53')
r53_resource = session.resource('route53')
Want to know from someone experienced if this is the right way to do this? Or is there a better way to achieve above?
Here is updated code:
def lambda_handler(event, context):
sts = boto3.client('sts')
response = sts.assume_role(
RoleArn='arn:aws:iam::***123:role/lambda',
RoleSessionName='my-random-session-name',
DurationSeconds= 900 # how many seconds these credentials will work
)
tempAccessKeyId = response['Credentials']['AccessKeyId']
tempSecretAccessKey = response['Credentials']['SecretAccessKey']
tempSessionToken = response['Credentials']['SessionToken']
client = boto3.client('route53',
region_name = 'us-west-2',
aws_access_key_id=tempAccessKeyId,
aws_secret_access_key=tempSecretAccessKey,
aws_session_token=tempSessionToken)
response = client.list_resource_record_sets(
HostedZoneId='***',
StartRecordName='test.example.com.',
StartRecordType='A'
)
print(response)
Based on the fact that you are doing this from an AWS Lambda function, the most secure way to do it would be:
In Account 1:
Create an IAM Role (Role 1) that will be used by the Lambda function
Assign permissions to the role that allows it to assume Role-2
Also assign any other permissions the Lambda function requires (you would normally add the AWSLambdaBasicExecutionRole managed policy to allow logging)
Assign Role 1 to the Lambda function
In Account 2:
Create an IAM Role (Role 2) with trust permissions that allows Role 1 in Account 1 to assume it
Grant Role 2 appropriate permissions to use Amazon Route 53
In your Lambda code, you would call AssumeRole() on Role 2. This will provide a set of temporary credentials that can be used to access Account 2 (as per your code, above).
See: Switching to an IAM Role (AWS API) - AWS Identity and Access Management
To make an API call to an AWS account, you either need credentials from that AWS account (eg credentials associated with an IAM User), or you need the ability to assume an IAM Role in that account.
So, in your example, if the credentials being provided belong to Account2, then you will be able to make API calls to Account2 (if that IAM User has been granted the necessary Route 53 permissions).
If you are frequently moving between accounts, you can instead specify a profile, which retrieves a different set of credential from the credentials file.
See: python - How to choose an AWS profile when using boto3 to connect to CloudFront - Stack Overflow
I have a lambda that consumes data from another AWS account's Athena. So I have a role that my lambda assumes which have cross-account access. I had used STS client in my lambda. sometimes lambda is running perfectly fine and sometimes it breaks and gives me this error.
"errorMessage": "An error occurred (ExpiredTokenException) when calling the StartQueryExecution operation: The security token included in the request is expired",
"errorType": "ClientError",
STS client i used in my code is :
def assume_role_to_session(role_arn, session_name):
client = boto3.client('sts')
response = client.assume_role(RoleArn=role_arn, RoleSessionName=session_name, DurationSeconds=900)
return boto3.Session(
aws_access_key_id=response['Credentials']['AccessKeyId'],
aws_secret_access_key=response['Credentials']['SecretAccessKey'],
aws_session_token=response['Credentials']['SessionToken'],
region_name='us-east-1')
assume_role_athena_session = assume_role_to_session(role_arn='arn:aws:iam::XXXXXXXXXXX:role/role-name',
session_name='AthenaLambdaSession')
How does this work? I want my lambda to run anytime not just sometimes. What could be the problem?
Look at this document
RoleSessionName (string) -- [REQUIRED]
An identifier for the assumed role session.
Use the role session name to uniquely identify a session when the same
role is assumed by different principals or for different reasons. In
cross-account scenarios, the role session name is visible to, and can
be logged by the account that owns the role. The role session name is
also used in the ARN of the assumed role principal. This means that
subsequent cross-account API requests using the temporary security
credentials will expose the role session name to the external account
in their CloudTrail logs.
So try to replace AthenaLambdaSession with unique RoleSessionName. Or if you want to reuse existing session, try to check session timeout to make sure that enough time to execute your lambda task (max 5 minutes). If < 5 minutes, re-new it.
Conflicting documentation
The documentation here, pertaining to AssumeRole, seems to contradict itself in one continuous block:
You must call this API using existing IAM user credentials. For more
information, see Creating a Role to Delegate Permissions to an IAM
User and Configuring MFA-Protected API Access.
This is an unsigned call, meaning that the app does not need to have
access to any AWS security credentials in order to make the call.
The contradictions are given bold emphasis.
Code sample
The code sample provided here certainly seems to require credentials:
AmazonSecurityTokenServiceClient securityTokenServiceClient = new AmazonSecurityTokenServiceClient(
Config.AccessKey,
secretKeyAsString,
securityTokenServiceConfig);
…
AssumeRoleRequest assumeRoleRequest = new AssumeRoleRequest
{
DurationSeconds = sessionDurationSec,
RoleArn = roleArn,
RoleSessionName = awsUsername,
ExternalId = groupSid
};
…
assumeRoleResponse = securityTokenServiceClient.AssumeRole(assumeRoleRequest);
In conclusion
Which is true? Are the requests in the code sample truly redundant?
Thank you!
The AssumeRole API call does require existing AWS credentials.
In order to assume an IAM role, an existing set of credentials must be used so that AWS knows who is assuming the role. This is so that AWS can verify that the assuming party is allowed to assume the role.
In the documentation:
This is an unsigned call, meaning that the app does not need to have access to any AWS security credentials in order to make the call.
This does appear to be incorrect information.
This is indeed an error in the docs, which is in the process of being corrected. AssumeRole does require existing long-term (IAM User) or temp credentials credentials to call. It is the two federation equivalents, AssumeRoleWithSAML and AssumeRoleWithWebIdentity that can be called without credentials. Sorry for the confusion!