I'm trying to use an S3 bucket to upload files to as part of a build, it is configured to provide files as a static site and the content is protected using a Lambda and CloudFront. When I manually create files in the bucket they are all visible and everything is happy, but when the files are uploaded what is created are not available, resulting in an access denied response.
The user that's pushing to the bucket does not belong in the same AWS environment, but it has been set up with an ACL that allows it to push to the bucket, and the bucket with a policy that allows it to be pushed to by that user.
The command that I'm using is:
aws s3 sync --no-progress --delete docs/_build/html "s3://my-bucket" --acl bucket-owner-full-control
Is there something else that I can try that basically uses the bucket permissions for anything that's created?
According to OP's feedback in the comment section, setting Object Ownership to Bucket owner preferred fixed the issue.
Related
I need to copy some S3 objects from a client. The client sent us the key and secret and I can list the object using the following command.
AWS_ACCESS_KEY_ID=.... AWS_SECRET_ACCESS_KEY=.... aws s3 ls s3://bucket/company4/
I will need to copy/sync s3://bucket/company4/ (very large) of our client's S3. In this question Copy content from one S3 bucket to another S3 bucket with different keys, it mentioned that it can be done by creating a bucket policy on the destination bucket. However, we probably don't have permission to create the bucket policy because we have limited AWS permissions in our company.
I know we can finish the job by copying the external files to local file system first and then upload them to our S3 bucket. Is there a more efficient way to do the work?
I have read-only access to a source S3 bucket. I cannot change permissions or anything of the sort on this source account and bucket. I do not own this account.
I would like to sync all files from the source bucket to my destination bucket. I own the account that contains the destination bucket.
I have a separate sets of credentials for the source bucket that I do not own and the destination bucket that I do own.
Is there a way to use the AWS CLI to sync between buckets using two sets of credentials?
aws s3 sync s3://source-bucket/ --profile source-profile s3://destination-bucket --profile default
If not, how can I setup permissions on my owned destination bucket to that I can sync with the CLI?
The built-in S3 copy mechanism, at the API level, requires the request be submitted to the target bucket, identifying the source bucket and object inside the request, and using a single set of credentials that has both authorization to read from the source and write to the target.
This is the only supported way to copy from one bucket to another without downloading and uploading the files.
The standard solution is found at http://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example2.html.
You can grant their user access to write your bucket or they can grant your user access to their bucket... but copying from one bucket to another without downloading and re-uploading the files is impossible without the complicity of both account owners to establish a single set of credentials with both privileges.
Use rclone for this. It's convenient but it does download and upload the files I believe which makes it slow for large data volumes.
rclone --config=creds.cfg copy source:bucket-name1/path/ destination:bucket-name2/path/
creds.cfg:
[source]
type = s3
provider = AWS
access_key_id = AAA
secret_access_key = bbb
[target]
type = s3
provider = AWS
access_key_id = CCC
secret_access_key = ddd
For this use case, I would consider Cross-Region Replication Where Source and Destination Buckets Are Owned by Different AWS Accounts
... you set up cross-region replication on the source
bucket owned by one account to replicate objects in a destination
bucket owned by another account.
The process is the same as setting up cross-region replication when
both buckets are owned by the same account, except that you do one
extra step—the destination bucket owner must create a bucket policy
granting the source bucket owner permission for replication actions.
Using AWS CLI; we can copy or sync files directly from one bucket to other. Using SDK; I can see api for download and upload. But can we directly copy files from one bucket to other bucket ( in different aws account) using sdk !
Yes. The CopyObject API call can copy an object between Amazon S3 buckets, including bucket in different regions and different accounts.
To copy objects between accounts, the one set of credentials requires sufficient permission to Read from the source bucket and Write to the destination bucket. You can either:
Use credentials from the destination account, and use a Bucket Policy on the source bucket to grant Read access, or
Use credentials from the source account, and use a Bucket policy on the destination bucket to grant Write access. Make sure you set ACL=public-read to pass ownership of the object to the destination Account.
Please note that it only copies one object at a time, so you would need to loop through a list of objects and call CopyObject for each one individually if you wish to copy multiple objects.
It's easy, see all the CLI commands with the help:
aws s3 --help
Upload a file:
aws s3 cp <path-to-file-from-local> s3://<S3_BUCKET_NAME>/<file-name>
Download a file:
aws s3 cp s3://<S3_BUCKET_NAME>/<file-name> <path-to-file-from-local>
Move a file:
aws s3 mv s3://<S3_BUCKET_NAME>/<file-name> s3://<S3_BUCKET_NAME>/<file-name>
You can use . to specify the current directory, eg:
aws s3 cp s3://MyBucket/Test.txt .
I have suspended object versioning on my S3 Bucket, but I don't want to have to select the check box that says "I acknowledge that existing objects with the same name will be overwritten" every time I upload photos to the s3 bucket.
I successfully set up an S3 Bucket Policy to make it so I don't have to specify that I want the uploads to be publicly viewable on every upload. Is there also an S3 Bucket Policy I can set to bypass the checkmark as well?
Thank you
Is there also an S3 Bucket Policy I can set to bypass the checkmark as well?
Sadly, this is new S3 console future. It is not related to the bucket policies.
You can use AWS CLI or SDK to upload objects "peacefully", without any distractions if you want to skip all the S3 console steps.
I have read-only access to a source S3 bucket. I cannot change permissions or anything of the sort on this source account and bucket. I do not own this account.
I would like to sync all files from the source bucket to my destination bucket. I own the account that contains the destination bucket.
I have a separate sets of credentials for the source bucket that I do not own and the destination bucket that I do own.
Is there a way to use the AWS CLI to sync between buckets using two sets of credentials?
aws s3 sync s3://source-bucket/ --profile source-profile s3://destination-bucket --profile default
If not, how can I setup permissions on my owned destination bucket to that I can sync with the CLI?
The built-in S3 copy mechanism, at the API level, requires the request be submitted to the target bucket, identifying the source bucket and object inside the request, and using a single set of credentials that has both authorization to read from the source and write to the target.
This is the only supported way to copy from one bucket to another without downloading and uploading the files.
The standard solution is found at http://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example2.html.
You can grant their user access to write your bucket or they can grant your user access to their bucket... but copying from one bucket to another without downloading and re-uploading the files is impossible without the complicity of both account owners to establish a single set of credentials with both privileges.
Use rclone for this. It's convenient but it does download and upload the files I believe which makes it slow for large data volumes.
rclone --config=creds.cfg copy source:bucket-name1/path/ destination:bucket-name2/path/
creds.cfg:
[source]
type = s3
provider = AWS
access_key_id = AAA
secret_access_key = bbb
[target]
type = s3
provider = AWS
access_key_id = CCC
secret_access_key = ddd
For this use case, I would consider Cross-Region Replication Where Source and Destination Buckets Are Owned by Different AWS Accounts
... you set up cross-region replication on the source
bucket owned by one account to replicate objects in a destination
bucket owned by another account.
The process is the same as setting up cross-region replication when
both buckets are owned by the same account, except that you do one
extra step—the destination bucket owner must create a bucket policy
granting the source bucket owner permission for replication actions.