Is it possible to copy from one AWS accounts S3 bucket into another AWS accounts Redshift cluster? The way I tried to do it was to log in using SQL Workbench to my AWS Account (Account1) and used a IAM User of (Account2) to copy the file over like this:
copy my_table (town,name,number)
from 's3://other-s3-account-bucket/fileToCopy.tsv'
credentials 'aws_access_key_id=<other_accounts_aws_access_key_id>;aws_secret_access_key=<other_accounts_aws_secret_access_key>'
delimiter '\t';
I know the other account's user has s3 permissions after double checking. Do I have share IAM users or setup different permissions in order to do this?
You will need to "pull" the data from the other account's S3 bucket.
AWS Account A has an S3 bucket called source-bucket-account-a.
AWS Account B has a Redshift cluser called TargetCluster.
On bucket source-bucket-account-a, add a bucket policy allowing AWS Account B to read files.
A sample policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DelegateS3Access",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<account-b-number>:root"
},
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::source-bucket-account-a",
"arn:aws:s3:::source-bucket-account-a/*"
]
}
]
}
It's very similar to the following:
http://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example2.html
or the following:
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_policy-examples.html
Once the bucket policy is in place, you use the credentials for AWS Account B to run the copy command because it owns the Redshift cluster. In the copy command, you specify the bucket by it's name source-bucket-account-a.
The bucket policy has granted read access to AWS Account B so it can "pull" the data into Redshift.
Related
I have 100 million small csv files that I have to copy from one aws account into another.
I tried to do parallel S3 copy using boto3 and also tried using aws sync. But due to the larger amount of files I could not get it done in reasonable amount time of time.
Is there any way to copy this large number of files from one account to another account S3 bucket.
You can:
Generate a list of objects by using Amazon S3 Inventory, which can provide a daily or weekly CSV file listing all objects
Pass the list to S3 Batch Operations and configure it to perform a Copy operation
See: Cross-account bulk transfer of files using Amazon S3 Batch Operations | AWS Storage Blog
Imagine you want to transfer files between accounts (A & B).
Attach a bucket policy to the source bucket in Account A
1 Get the Amazon Resource Name (ARN) of the IAM identity (user or role) in Account B (destination account).
2 From Account A, attach a bucket policy to the source bucket that allows the IAM identity in Account B to get objects
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DelegateS3Access",
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::222222222222:user/Jane"},
"Action": ["s3:ListBucket","s3:GetObject"],
"Resource": [
"arn:aws:s3:::awsexamplesourcebucket/*",
"arn:aws:s3:::awsexamplesourcebucket"
]
}
]
}
Attach an IAM policy to a user or role in Account B
From Account B, create an IAM customer managed policy that allows an IAM user or role to copy objects from the source bucket in Account A to the destination bucket in Account B.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::awsexamplesourcebucket",
"arn:aws:s3:::awsexamplesourcebucket/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::awsexampledestinationbucket",
"arn:aws:s3:::awsexampledestinationbucket/*"
]
}
]
}
Attach the customer managed policy to the IAM user or role that you want to use to copy objects between accounts.
Use the IAM user or role in Account B to perform the cross-account copy
After you set up the bucket policy and IAM policy, the IAM user or role in Account B can perform the copy from Account A to Account B. Then, Account B owns the copied objects.
To synchronize all content from a source bucket in Account A to a destination bucket in Account B, the IAM user or role in Account B can run the sync command using the AWS Command Line Interface (AWS CLI):
aws s3 sync s3://awsexamplesourcebucket s3://awsexampledestinationbucket
AWS Refrence
I created two profiles (one for source and one for target bucket) and using below command to copy:
aws s3 cp --profile source_profile s3://source_bucket/file.txt --profile target_profile s3://target_profile/
But it throws below error.
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
Looks like we can't use multiple profiles with aws commands.
The simplest method is to grant permissions via a bucket policy.
Say you have:
Account-A with IAM User-A
Account-B with Bucket-B
Add a bucket policy on Bucket-B:
{
"Id": "CopyBuckets",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GrantAccessToUser-A",
"Action": "s3:*",
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::bucket-b",
"arn:aws:s3:::bucket-b/*"
],
"Principal": {
"AWS": [
"arn:aws:iam::<account-a-id>:user/user-a"
]
}
}
]
}
Then just copy the files as User-A.
See also: aws sync between S3 buckets on different AWS accounts
No, you can't use multiple profiles in one AWS CLI command. Possible solutions:
1) Download files to local disk, then upload them to the target bucket with a separate command.
2) Allow first account access to the target bucket. For this, you will have to create a cross-account role in the source account and assign it the appropriate permissions in the target account. That way you will be using one role/one profile, but this role will be granted permissions in the second account. See https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html
I am trying to copy data in a bucket in one account, in which I have access to an IAM but not admin, to a bucket in another account, in which I am an admin, and failing. I can't even ls the source bucket.
I've followed the directions from AWS and various sources online to give myself list/read/get permissions on the source bucket, with no success. I can provide the details (e.g., the bucket policy json), but it is what is in the AWS docs and other places. What I've done works between two accounts I have admin access to.
This is "multi-region", in the sense that I'm in the US (mainly us-west-2) but the bucket is in eu-central-1. I am specifying the region in the aws cli, and set up a destination bucket in eu-central-1, but can't even list anyway.
I have done this couple of times with my AWS accounts. I am guessing you have setup cross account Access to your S3 bucket, but just double check, here is what I do for granting an S3 bucket cross account access.
Account (A):
S3 bucket (testbucket)
Account (B):
IAM User (testuser) needs access to the S3 bucket testbucket in Account (A)
Here are things that need to happen:
Create a bucket policy on testbucket (A) to grant read/list etc access to to your test bucket.
example:
{
"Version": "2012-10-17",
"Id": "BUCKETPOLICY",
"Statement": [
{
"Sid": "AllowS3ReadObject28",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::900000000:user/testuser"
]
},
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::testbucket",
"arn:aws:s3:::testbucket/*"
]
}
]
}
Create an IAM policy on testuser that also grants read, write, list etc access to the bucket.
It appears that your situation is:
Account A: Bucket A and User A (with limited access rights)
Account B: Bucket B and User B (with admin rights)
You can either push the data from Account A to Bucket B, or you can pull the data from Bucket A using Account B.
Pushing from Account A to Bucket B
Let's assume User A has access to Bucket A. All that's needed is to give User A permission to write to Bucket B. This can be done with a bucket policy on Bucket B:
{
"Id": "PolicyB",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GrantAccessToUserA",
"Action": "s3:*",
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::BUCKET-B",
"arn:aws:s3:::BUCKET-B/*"
],
"Principal": "arn:aws:iam::ACCOUNT-A:user/USER-A"
}
]
}
This grants all s3 permissions to User A on Bucket B. That's excessive, but presumably this is only temporary.
User A would then copy the files from Bucket A to Bucket B. For example:
aws s3 sync s3://BUCKET-A s3://BUCKET-B \
--acl bucket-owner-full-control \
--source-region SOURCE-REGION \
--region DESTINATION-REGION
Important: When copying the files, be sure to use the Access Control List that grants bucket-owner-full-control. This means that the files become owned by the owner of Bucket B. If you don't do this, the files are still owned by User A and can't be deleted by User B, even with admin rights!
Pulling from Bucket A using Account B
To do this, User B must be granted access to Bucket A. You will need enough access rights in Account A to add a bucket policy on Bucket A:
{
"Id": "PolicyA",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GrantAccessToUserB",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::BUCKET-A",
"arn:aws:s3:::BUCKET-A/*"
],
"Principal": "arn:aws:iam::ACCOUNT-B:user/USER-B"
}
]
}
Then, User B can copy the files across:
aws s3 sync s3://BUCKET-A s3://BUCKET-B \
--source-region SOURCE-REGION \
--region DESTINATION-REGION
(You might need to grant some more access rights, I didn't test the above policy.)
The fact that buckets are in different regions does not impact the permissions, but it does impact where you send the command. The command is sent to the destination region, which then pulls from the source region.
See: AWS CLI s3 sync command
I am trying to unload data from Redshift to S3 using iam_role. The unload command works fine as long as I am unloading data to a S3 bucket owned by the same account as the Redshift cluster.
However, if I try to unload data into a S3 bucket owned by another account it doesn't work. I have tried the approach mentioned in these tutorials:
Tutorial: Delegate Access Across AWS Accounts Using IAM Roles
Example: Bucket Owner Granting Cross-Account Bucket Permissions
However, I always get S3ServiceException:Access Denied,Status 403,Error AccessDenied,Rid
Has anyone done this before?
I got it to work. Here's what I did:
Created an IAM Role in Account A that has AmazonS3FullAccess policy (for testing)
Launched an Amazon Redshift cluster in Account A
Loaded data into the Redshift cluster
Test 1: Unload to a bucket in Account A -- success
Test 2: Unload to a bucket in Account B -- fail
Added a bucket policy to the bucket in Account B (see below)
Test 3: Unload to a bucket in Account B -- success!
This is the bucket policy I used:
{
"Id": "Policy11",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PermitRoleAccess",
"Action": "s3:*",
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::my-bucket",
"arn:aws:s3:::my-bucket/*"
],
"Principal": {
"AWS": [
"arn:aws:iam::123456789012:role/Redshift-loader"
]
}
}
]
}
The Redshift-loader role was already associated with my Redshift cluster. This policy grants the role (that lives in a different AWS account) access to this S3 bucket.
I solved it using access_key_id and secret_access_key instead iam_rol
I have two S3 buckets with two different set of access and secret keys. Here is the set:
Bucket1, Key1, Secret1
And
Bucket2, Key2, Secret2, Token
I am trying to trigger S3-S3 copy via aws cli like this:
aws s3 cp s3://key1:secret1#Bucket1 s3://key2:secret2#Bucket2
I have few questions:
I am not sure how to specify token for providing write permission to bucket2
Does aws cli allows specifying key and secret as part of S3 Url ?
What would be the best approach to achieve this use case ?
To copy a file between Amazon S3 buckets, you must use credentials that have permission to access both buckets, or apply a bucket policy to the destination bucket that permits the access.
It is not possible to specify two sets of credentials because the AWS Command-Line Interface (CLI) is only calling a single AWS API, which performs the copy 'from' the source bucket directly to the destination bucket. The AWS CLI does not download the object -- it simply tells S3 to copy the object to another bucket (which can even be in a different region).
Therefore, you should create a bucket policy on the destination bucket that permits the credentials being used (User or Role) to PutObject into the destination bucket.
The policy would be similar to:
{
"Version": "2012-10-17",
"Id": "Policy1",
"Statement": [
{
"Sid": "Stmt1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNT-NUMBER:role/ROLE-NAME"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::DESTINATION-BUCKET/*"
}
]
}
The above is assuming the command is being called from an Amazon EC2 instance with an assigned role. To call from User credentials, use:
"AWS": "arn:aws:iam::ACCOUNT-NUMBER:user/USER-NAME"