I have data that arrives in S3 Account A that i want to automatically copy to S3 Account B but do not understand how i can reference the files in Account A in my Lambda in Account B to do the copy.
Completed Steps so far:
1 Account B Inline policy added to Execution Role referencing Account A S3 bucket
2 Account B Permission given to Account A to invoke Lambda
3 Account A Bucket policy allowing S3 access to role execution Role Account B
4 Account A Event Notification to Account B Lambda (All ObjectCredte events)
Am i missing some steps or is here and if not how can my Lambda directly reference the individual files captured by the event?
Update due to comments:
From the question above, I'm not sure I understand the setup, but here's how I would approach this from an architectural perspective:
A Lambda function inside account A gets triggered by the S3 event when an object is uploaded.
The Lambda function retrieves the uploaded object from the source bucket
The Lambda function assumes a role in account B, which grants permission to write into the target bucket.
The Lambda function writes the object into the target bucket.
The permissions you need are:
An execution role for the Lambda function in account A that (a) grants permission to read from the source bucket and (b) grants permission to assume the role in account B (see next item below)
A cross-account role in account B, (a) trusting the above execution role and (b) granting permission to write into the target bucket
Note: Make sure to save the object granting bucket-owner-full-control so that account B has permissions to use the copied object.
If you want to replicate the objects to a bucket in a different AWS account and don't care about the fact that it can take up to 15 minutes for the replication to be done, you don’t need to build anything yourself. Simply use the Amazon S3 Replication feature.
Replication enables automatic, asynchronous copying of objects across
Amazon S3 buckets. Buckets that are configured for object replication
can be owned by the same AWS account or by different accounts. You can
copy objects between different AWS Regions or within the same Region.
Related
I have 3 accounts: account A, account B and account C
Objects from account A source bucket are replicated to account B target bucket. Now I want to achieve that replicated object from account b target bucket should be replicated to bucket in account C.
Ofcourse account A has permission only to access account B and account B has permission to access only account C. I cannot change permissions.
so is it possible to replicate replicated objects in account B bucket to account C bucket
Normal S3 replication can't be used to replicate, replicated objects. You have to use S3 Batch Replication instead:
S3 Batch Replication provides you a way to replicate objects that existed before a replication configuration was in place, objects that have previously been replicated, and objects that have failed replication.
I have a lambda function in Account A which will retrieve data from a source. This lambda will then need to trigger off a lambda in Account B in order to pass this data to it which then will be uploaded to DynamoDB.
I understand there will be some sort of cross-account-permissions required but am a little unsure if i need to allow these permission in Account A or Account B. I can see on AWS examples of triggering a lambda from an S3 bucket but that's not helping with what I want to do.
I could potentially have an API Gateaway in the middle for Lambda A to interact with Lambda B but that's just adding an extra resource that's not really required.
Your AWS Lambda function in account A would call the Lambda.invoke() method in the AWS SDK for whatever programming language you are writing the Lambda function with.
I understand there will be some sort of cross-account-permissions
required but am a little unsure if I need to allow these permission in
Account A or Account B
Account B Lambda is the one being called, so Account B has to give permission to Account A to make that call.
Here you got an example of cross-account permissions with lambda function: https://yogeshnile.cloud/configure-a-lambda-function-to-assume-an-iam-role-in-another-aws-account-e005e7533a71
Trigger Lambda in Account B from Lambda in Account A -> Lambda.invoke()
Btw, you don't need the lambda function in account B - you can add permissions to your DynamoDB table to assumed role, so your lambda from account A will be able to write data directly into DynamoDB on account B.
I have one AWS S# and Redshift question:
A company uses two AWS accounts for accessing various AWS services. The analytics team has just configured an Amazon S3 bucket in AWS account A for writing data from the Amazon Redshift cluster provisioned in AWS account B. The team has noticed that the files created in the S3 bucket using UNLOAD command from the Redshift cluster are not accessible to the bucket owner user of the AWS account A that created the S3 bucket.
What could be the reason for this denial of permission for resources belonging to the same AWS account?
I tried to reproduce the scenario for the question, but I can't.
I don't get the S3 Object Ownership and Bucket Ownership.
You are not the only person confused by Amazon S3 object ownership. When writing files from one AWS Account to a bucket owned by a different AWS Account, is possible for the 'ownership' of objects to remain with the 'sending' account. This causes all types of problems.
Fortunately, AWS has introduced a new feature into S3 called Edit Object Ownership that overrides all these issues:
By setting "ACLs disabled" for an S3 Bucket, objects will always be owned by the AWS Account that owns the bucket.
So, you should configure this option for the S3 bucket in your AWS account B and it should all work nicely.
The problem is that the bucket owner in account A does not have access to files that were uploaded by the account B, usually that is solved by specifying acl parameter when uploading files --acl bucket-owner-full-control. Since the upload is done via Redshift you need to tell Redshift to assume a role in the account A for UNLOAD command so files don't change the ownership and continue to belong to account A. Check the following page for more examples on configuring cross account LOAD/UNLOAD https://aws.amazon.com/premiumsupport/knowledge-center/redshift-s3-cross-account/
I’m trying to sync one aws bucket to an another bucket across different iam accounts.
How can I do it periodically so any file written to the source bucket will automatically transforms to the destination? Do I need to use lambdas to execute aws cli sync command?
Thanks
Option 1: AWS CLI Sync
You could run aws s3 sync on a regular basis, which will only copy new/changed files. This makes it very efficient. However, if there is a large number of files (10,000+) then it will take a long time trying to determine which files need to be copied. You will also need to schedule the command to run somewhere (eg a cron job).
Option 2: AWS Lambda function
You could create an AWS Lambda function that is triggered by Amazon S3 whenever a new object is created. The Lambda function will be passed details of the Bucket & Object via the event parameter. The Lambda function could then call CopyObject() to copy the object immediately. The advantage of this method is that the objects are copied as soon as they are created.
(Do not use an AWS Lambda function to call the AWS CLI. The above function would be called for each file individually.)
Option 3: Amazon S3 Replication
You can configure Amazon S3 Replication to automatically replicate newly-created objects between the buckets (including buckets between different AWS Accounts). This is the simplest option since it does not require any coding.
Permissions
When copying S3 objects between accounts, you will need to use a single set of credentials that has both Read permission on the source bucket and Write permission on the target bucket. This can be done in two ways:
Use credentials (IAM User or IAM Role) from the source account that have permission to read the source bucket. Create a bucket policy on the target bucket that permits those credentials to PutObject into the bucket. When copying, specify ACL=public-read to grant object ownership to the destination account.
OR
Use credentials from the target account that have permission to write to the target bucket. Create a bucket policy on the source bucket that permits those credentials to GetObject from the bucket.
In the Documentation for Resource-Based Policies for Lambda, it mentions that it's best practice to include the source-account incase for example you specified a source-arn which referred to an s3 bucket which does not have the account id in the arn, so if you were unlucky and somebody deleted your bucket, and another account created a bucket with the same name they could indirectly access your Lambda function.
But then you also have the notation of a Principal, as in one of the examples they have:
"Principal":{"AWS":"arn:aws:iam::210987654321:root"}
What is the difference between Principal & source-account. Do you use the Principal in the case when you want to refine the permissions down to a particular role or user within an account? And if this isn't your situation and you only want to grant access to your Lambda from an entire account you would use source-account?
One reason of using the aws:SourceAccount is the mitigation of The Confused Deputy Problem.
Specifically, in the context of S3, it is used so that S3 is not considered as the confused deputy.
The principal is what has the permission to trigger the resource, for example in this case the principal is actually the S3 service. This is because S3 is not configured to assume IAM roles, the service is actually the caller of the Lambda function.
The conditions underneath then scope the permissions to only allow the S3 service to call when it is coming from the source account/bucket. Without this it would be an open scope to Amazon S3.
You're correct that principals can be used to reference IAM users/roles and in your example the entire AWS account (assuming the caller is actually an IAM user/role). You would use this method if the caller was an IAM entity vs another AWS service.