How to sync multiple S3 buckets using multiple AWS accounts? - amazon-web-services

Question

I am having trouble syncing two S3 buckets that are attached to two separate AWS accounts.
There are two AWS accounts - Account A which is managed by a third party and Account B, which I manage. I am looking to pull files from an S3 bucket in Account A to an S3 bucket in Account B.
Account A provided me the following instructions:
In Account B, create a new IAM user called LogsUser. Attach the following policy to the user:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::ACCOUNTID:role/12345-LogAccess-role"
}
]
}
Configure the AWS CLI to update the config and credentials files. Specifically, the ~/.aws/config file to look like:
[profile LogsUser]
role_arn = arn:aws:iam::ACCOUNTID:role/12345-LogAccess-role
source_profile = LogsUser
And the ~/.aws/credentials file to look like
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
From here, I am successfully able to query the log files in Account A's bucket using $ aws s3 ls --profile LogsUser s3://bucket-a.
I have set up bucket-b in Account B, however, I am unable to query any files in bucket-b. For example, $ aws s3 ls --profile LogsUser s3://bucket-b returns An error occurred (AccessDenied) when calling the AssumeRole operation: Access denied.
Is there something additional I can add to the config file or my IAM policy to allow access to bucket-b using --profile LogsUser option? I can access bucket-b using other --profile settings, but am not looking to sync to the local file system and then to another bucket.
The desired results is to run a command like aws s3 sync s3://bucket-a s3://bucket-b --profile UserLogs.

Answer 1

For example, if you want to copy “Account A” S3 bucket objects to “Account B” S3 bucket, follow below.
Create a policy for the S3 bucket in “account A” like the below policy. For that, you need “Account B” number, to find the B account number go to Support → Support center and copy the account number from there.
Setup “account A” bucket policy :
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DelegateS3Access",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNT_B_NUMBER:root"
},
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::ACCOUNT_A_BUCKET_NAME/*",
"arn:aws:s3:::ACCOUNT_A_BUCKET_NAME"
]
}
]
}
Log into “Account B” and create a new IAM user or attach the below policy for the existing user.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::ACCOUNT_A_BUCKET_NAME",
"arn:aws:s3:::ACCOUNT_A_BUCKET_NAME/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::ACCOUNT_B_BUCKET_NAME",
"arn:aws:s3:::ACCOUNT_B_BUCKET_NAME/*"
]
}
]
}
Configure AWS CLI with “Account B” IAM user(Which you have created IAM with the above user policy)
aws s3 sync s3://ACCOUNT_A_BUCKET_NAME s3://ACCOUNT_B_BUCKET_NAME --source-region ACCOUNT_A_REGION-NAME --region ACCOUNT_B_REGION-NAME
This way we can copy S3 bucket objects over different AWS accounts.
If you have multiple awscli profiles, use --profile end of the command with profile name.

Answer 2

Your situation is:
You wish to copy from Bucket-A in Account-A
The files need to be copied to Bucket-B in Account-B
Account-A has provided you with the ability to assume LogAccess-role in Account-A, which has access to Bucket-A
When copying files between buckets using the CopyObject() command (which is used by the AWS CLI sync command), it requires:
Read Access on the source bucket (Bucket-A)
Write Access on the destination bucket (Bucket-B)
When you assume LogAccess-role, you receive credentials that have Read Access on Bucket-A. That is great! However, those credentials do not have permission to write to Bucket-B because it is in a separate account.
To overcome this, you should create a Bucket Policy on Bucket-A that grants Write Access to LogAccess-role from Account-B. The Bucket Policy would look something like:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNT-A:role/12345-LogAccess-role"
},
"Action": [
"s3:ListBucket",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::bucket-a",
"arn:aws:s3:::bucket-a/*"
]
}
]
}
(You might need other permissions. Check any error messages for hints.)
That way, LogAccess-role will be able to read from Bucket-A and write to Bucket-B.

Answer 3

I would suggest you to consider you to use AWS S3 bucket replication:
https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html

Answer 4

If you just want to list objects in bucket-b, do this.
First make sure the LogsUser IAM user has got proper permission to access the bucket-b s3 bucket in Account B. You can add this policy to the user if not
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::bucket-b/*"
]
}
]
}
If there is permissions attached to the user, and if the Access keys and Secret Key stored in ~/.aws/credentials stored as [default] belongs to LogsUser IAM user, you can simply list objects inside bucket-b with following command.
aws s3 ls
If you want to run the command aws s3 sync s3://bucket-a s3://bucket-b --profile UserLogs, do this.
Remember, we will be using temporary credentials created by STS after assuming the role with permanent credentials of LogsUser. That means the role in Account A should have proper access to both buckets to perform the action and the bucket(bucket-b) in another account (Account B) should have proper bucket policy to allow the role to perform S3 operations.
To provide permissions to the role to access bucket-b, attach following bucket policy to bucket-b.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNTID:role/12345-LogAccess-role"
},
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::bucket-b/*"
]
}
]
}
Also in Account A, attach a policy to the role like below to allow access to S3 buckets in both the accounts.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::bucket-b/*",
"arn:aws:s3:::bucket-a/*"
]
}
]
}