We have a service in AWS-Account-A which will copy some files with ACL: 'bucket-owner-full-control' to a s3 bucket in AWS-Account-B . Now there is a AWS-Account-C which already have a assumeRule ( which a S3 Read access policy is attached to it ) from AWS-Account-B, and S3 bucket policy already gave read access to AWS-Account-C rules, So the problem is, AWS-Account-C : Can't read those files which uploaded from AWS-Account-A and only CAN read files which uploaded using AWS-Account-B itself.
I know it's a reallay compliated secnario, but as far as I understand, it's a ownership problem. The bucket policy applies only to objects that are owned by the bucket owner, So it's like , X own some files, and he copy it to Y, now Z can't get it from Y, because it's not owned by Y.
If anyone faced to this kind of sencarios before and have solution, I really appreciate it to give some guidance.
Your problem is that you used Account-A to copy files to a bucket owned by Account-B but now the copied files are owned by Account-A. This is why Account-C cannot access them. Account-C does not have the required permission.
The correct procedure is to create a role in Account-B to be assumed by Account-A. Then before Account-A copies file to the bucket in Account-B, it assumes the Account-B role. Now files copied to the bucket will be owned by Account-B.
For the files currently in the Account-B bucket, while using Account-B's credentials do an inplace copy. This will switch the ownership to Account-B.
Here is an example inplace copy. Note: No data is transferred over the internet just within S3 so it executes quickly.
aws s3 cp s3://mybucket/mykey s3://mybucket/mykey --storage-class STANDARD
The '--recursive' argument to apply to an entire folder of keys.
Warnings:
1) All custom metadata and existing permissions will be lost.
2) Ensure you have backups of your data prior to executing a command such as this.
Related
I need to copy some S3 objects from a client. The client sent us the key and secret and I can list the object using the following command.
AWS_ACCESS_KEY_ID=.... AWS_SECRET_ACCESS_KEY=.... aws s3 ls s3://bucket/company4/
I will need to copy/sync s3://bucket/company4/ (very large) of our client's S3. In this question Copy content from one S3 bucket to another S3 bucket with different keys, it mentioned that it can be done by creating a bucket policy on the destination bucket. However, we probably don't have permission to create the bucket policy because we have limited AWS permissions in our company.
I know we can finish the job by copying the external files to local file system first and then upload them to our S3 bucket. Is there a more efficient way to do the work?
I have read-only access to a source S3 bucket. I cannot change permissions or anything of the sort on this source account and bucket. I do not own this account.
I would like to sync all files from the source bucket to my destination bucket. I own the account that contains the destination bucket.
I have a separate sets of credentials for the source bucket that I do not own and the destination bucket that I do own.
Is there a way to use the AWS CLI to sync between buckets using two sets of credentials?
aws s3 sync s3://source-bucket/ --profile source-profile s3://destination-bucket --profile default
If not, how can I setup permissions on my owned destination bucket to that I can sync with the CLI?
The built-in S3 copy mechanism, at the API level, requires the request be submitted to the target bucket, identifying the source bucket and object inside the request, and using a single set of credentials that has both authorization to read from the source and write to the target.
This is the only supported way to copy from one bucket to another without downloading and uploading the files.
The standard solution is found at http://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example2.html.
You can grant their user access to write your bucket or they can grant your user access to their bucket... but copying from one bucket to another without downloading and re-uploading the files is impossible without the complicity of both account owners to establish a single set of credentials with both privileges.
Use rclone for this. It's convenient but it does download and upload the files I believe which makes it slow for large data volumes.
rclone --config=creds.cfg copy source:bucket-name1/path/ destination:bucket-name2/path/
creds.cfg:
[source]
type = s3
provider = AWS
access_key_id = AAA
secret_access_key = bbb
[target]
type = s3
provider = AWS
access_key_id = CCC
secret_access_key = ddd
For this use case, I would consider Cross-Region Replication Where Source and Destination Buckets Are Owned by Different AWS Accounts
... you set up cross-region replication on the source
bucket owned by one account to replicate objects in a destination
bucket owned by another account.
The process is the same as setting up cross-region replication when
both buckets are owned by the same account, except that you do one
extra step—the destination bucket owner must create a bucket policy
granting the source bucket owner permission for replication actions.
Using AWS CLI; we can copy or sync files directly from one bucket to other. Using SDK; I can see api for download and upload. But can we directly copy files from one bucket to other bucket ( in different aws account) using sdk !
Yes. The CopyObject API call can copy an object between Amazon S3 buckets, including bucket in different regions and different accounts.
To copy objects between accounts, the one set of credentials requires sufficient permission to Read from the source bucket and Write to the destination bucket. You can either:
Use credentials from the destination account, and use a Bucket Policy on the source bucket to grant Read access, or
Use credentials from the source account, and use a Bucket policy on the destination bucket to grant Write access. Make sure you set ACL=public-read to pass ownership of the object to the destination Account.
Please note that it only copies one object at a time, so you would need to loop through a list of objects and call CopyObject for each one individually if you wish to copy multiple objects.
It's easy, see all the CLI commands with the help:
aws s3 --help
Upload a file:
aws s3 cp <path-to-file-from-local> s3://<S3_BUCKET_NAME>/<file-name>
Download a file:
aws s3 cp s3://<S3_BUCKET_NAME>/<file-name> <path-to-file-from-local>
Move a file:
aws s3 mv s3://<S3_BUCKET_NAME>/<file-name> s3://<S3_BUCKET_NAME>/<file-name>
You can use . to specify the current directory, eg:
aws s3 cp s3://MyBucket/Test.txt .
I have a set of video files that were copied from one AWS Bucket from another account to my account in my own bucket.
I'm running into a problem now with all of the files where i am receiving Access Denied errors when I try to make all of the files public.
Specifically, I login to my AWS account, go into S3, drill down through the folder structures to locate one of the videos files.
When I look at this specificfile, the permissions tab on the files does not show any permissions assigned to anyone. No users, groups, or system permissions have been assigned.
At the bottom of the Permissions tab, I see a small box that says "Error: Access Denied". I can't change anything about the file. I can't add meta-data. I can't add a user to the file. I cannot make the file Public.
Is there a way i can gain control of these files so that I can make them public? There are over 15,000 files / around 60GBs of files. I'd like to avoid downloading and reuploading all of the files.
With some assistance and suggestions from the folks here I have tried the following. I made a new folder in my bucket called "media".
I tried this command:
aws s3 cp s3://mybucket/2014/09/17/thumb.jpg s3://mybucket/media --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers full=emailaddress=my_aws_account_email_address
I receive a fatal error 403 when calling the HeadObject operation: Forbidden.
A very interesting conundrum! Fortunately, there is a solution.
First, a recap:
Bucket A in Account A
Bucket B in Account B
User in Account A copies objects to Bucket B (having been granted appropriate permissions to do so)
Objects in Bucket B still belong to Account A and cannot be accessed by Account B
I managed to reproduce this and can confirm that users in Account B cannot access the file -- not even the root user in Account B!
Fortunately, things can be fixed. The aws s3 cp command in the AWS Command-Line Interface (CLI) can update permissions on a file when copied to the same name. However, to trigger this, you also have to update something else otherwise you get this error:
This copy request is illegal because it is trying to copy an object to itself without changing the object's metadata, storage class, website redirect location or encryption attributes.
Therefore, the permissions can be updated with this command:
aws s3 cp s3://my-bucket/ s3://my-bucket/ --recursive --acl bucket-owner-full-control --metadata "One=Two"
Must be run by an Account A user that has access permissions to the objects (eg the user who originally copied the objects to Bucket B)
The metadata content is unimportant, but needed to force the update
--acl bucket-owner-full-control will grant permission to Account B so you'll be able to use the objects as normal
End result: A bucket you can use!
aws s3 cp s3://account1/ s3://accountb/ --recursive --acl bucket-owner-full-control
To correctly set the appropriate permissions for newly added files, add this bucket policy:
[...]
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012::user/their-user"
},
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::my-bucket/*"
}
And set ACL for newly created files in code. Python example:
import boto3
client = boto3.client('s3')
local_file_path = '/home/me/data.csv'
bucket_name = 'my-bucket'
bucket_file_path = 'exports/data.csv'
client.upload_file(
local_file_path,
bucket_name,
bucket_file_path,
ExtraArgs={'ACL':'bucket-owner-full-control'}
)
source: https://medium.com/artificial-industry/how-to-download-files-that-others-put-in-your-aws-s3-bucket-2269e20ed041 (disclaimer: written by me)
In case anyone trying to do the same but using Hadoop/Spark job instead of AWS CLI.
Step 1: Grant user in Account A appropriate permissions to copy
objects to Bucket B. (mentioned in above answer)
Step 2: Set the fs.s3a.acl.default configuration option using Hadoop Configuration. This can be set in conf file or in program:
Conf File:
<property>
<name>fs.s3a.acl.default</name>
<description>Set a canned ACL for newly created and copied objects. Value may be Private,
PublicRead, PublicReadWrite, AuthenticatedRead, LogDeliveryWrite, BucketOwnerRead,
or BucketOwnerFullControl.</description>
<value>$chooseOneFromDescription</value>
</property>
Programmatically:
spark.sparkContext.hadoopConfiguration.set("fs.s3a.acl.default", "BucketOwnerFullControl")
by putting
--acl bucket-owner-full-control
made it to work.
I'm afraid you won't be able to transfer ownership as you wish. Here's what you did:
Old account copies objects into new account.
The "right" way of doing it (assuming you wanted to assume ownership on the new account) would be:
New account copies objects from old account.
See the small but important difference? S3 docs kind of explain it.
I think you might get away with it without needing to download the whole thing by just copying all of the files within the same bucket, and then deleting the old files. Make sure you can change the permissions after doing the copy. This should save you some money too, as you won't have to pay for the data transfer costs of downloading everything.
boto3 "copy_object" solution :
Providing Grant control to the destination bucket owner
client.copy_object(CopySource=copy_source, Bucket=target_bucket, Key=key, GrantFullControl='id=<bucket owner Canonical ID>')
Get for console
Select bucket, Permission tab, "Access Control List" tab
Let's say i have 2 aws accounts: Account1 and AccountZ
I installed and configured s3cmd to have access to Account1.
I created a bucket in AccountZ and made it publicly read/write
I performed an s3cmd put of a text.txt from Account1 to s3://AccountZ/test.txt
Then, after it uploaded, I tried to copy paste AccountZ/test.txt to a different bucket, and it says that there was an error ("The following objects were not copied due to errors from: <AccountZ folder>"). So, I tried to change the permissions to the file, and it says I dont have permissions to do that.
If "upload" a file using the S3 console into AccountZ target directory, that resulting file IS copy/paste-able. So there seems to be an issue with the uploaded file due to the PUT
If i change the permissions config of s3cmd to be the key/secret of AccountZ, then uploaded file's permissions work just fine and the copy/paste command is successful.
How do I upload/PUT a file to S3 so that I can then copy/paste the resulting file in the S3 console?
When an object is uploaded to S3, the owner of the object is the account that created it. In this case, the owner of the object is Account1, even though the bucket exists in AccountZ. The default permissions on the object only allow it to be modified by the owner of the object (Account1). The only thing that AccountZ will be able to do with the object is delete it.
When you create a bucket policy, that policy will automatically apply to any objects in the bucket that are 'owned' by the same account that owns the bucket. Since AccountZ owns the bucket and Account1 owns the object, the bucket policy of public read/write isn't going to apply here.
Try specifying an ACL (eg 'public-read-write') when the object is uploaded. If you need to modify an object that has already been uploaded, try the PutObjectAcl call from the S3 API using Account1's credentials. (http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUTacl.html)
In a similar strategy to what #ScottWolf proposed, I had to do the following to solve my problem:
The solution was that I had to go add a bucket policy in the source data bucket (Account1) that gave permissions to the target. then i had to re-configure my s3 api to use AccountZ's credentials and then just do a copy from Account1 to AccountZ