I have two S3 buckets in two different regions on two different accounts. I want to use a S3 replication rule to replicate all files (including existing ones) from bucket-a to bucket-b.
bucket-a is an existing bucket with objects in it already, bucket-b is a new, empty bucket.
I created a replication rule and ran the batch operation job to replicate existing objects. After the job finished, 63% of objects failed to replicate, with the errors DstPutObjectNotPermitted or DstMultipartUploadNotPermitted and no further information. Nothing comes up on Google for these errors. (these are coming from the csv file that gets generated after job completion). The remaining objects got replicated as expected.
Here's my configuration:
bucket-a has versioning enabled and it is encrypted with a default aws-managed KMS key. ACL's are enabled, and this is the bucket policy:
{
"Version": "2008-10-17",
"Id": "NoBucketDelete",
"Statement": [
{
"Sid": "NoBucketDeleteStatement",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:DeleteBucket",
"Resource": "arn:aws:s3:::bucket-a"
},
{
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::bucket-a/*",
"arn:aws:s3:::bucket-a"
],
"Condition": {
"Bool": {
"aws:SecureTransport": "false"
}
}
}
]
}
bucket-b also has versioning and ACL's enabled, and is encrypted with a customer-managed key.
The bucket policy is:
{
"Version": "2012-10-17",
"Id": "Policy1644945280205",
"Statement": [
{
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::bucket-b/*",
"arn:aws:s3:::bucket-b"
],
"Condition": {
"Bool": {
"aws:SecureTransport": "false"
}
}
},
{
"Sid": "Stmt1644945277847",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::12345:role/bucket-replication-role"
},
"Action": [
"s3:ReplicateObject",
"s3:ReplicateTags",
"s3:ObjectOwnerOverrideToBucketOwner",
"s3:ReplicateDelete"
],
"Resource": "arn:aws:s3:::bucket-b/*"
}
]
}
...and the KMS key policy is
{
"Version": "2012-10-17",
"Id": "key-consolepolicy-3",
"Statement": [
{
"Sid": "Allow access through S3 for all principals in the account that are authorized to use S3",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"kms:CallerAccount": "12345",
"kms:ViaService": "s3.us-west-2.amazonaws.com"
}
}
},
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::12345:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Allow access for Key Administrators",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::12345:user/root",
"arn:aws:iam::12345:user/user"
]
},
"Action": [
"kms:Create*",
"kms:Describe*",
"kms:Enable*",
"kms:List*",
"kms:Put*",
"kms:Update*",
"kms:Revoke*",
"kms:Disable*",
"kms:Get*",
"kms:Delete*",
"kms:TagResource",
"kms:UntagResource",
"kms:ScheduleKeyDeletion",
"kms:CancelKeyDeletion"
],
"Resource": "*"
},
{
"Sid": "Allow use of the key",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::12345:user/user"
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "*"
},
{
"Sid": "Allow attachment of persistent resources",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::12345:user/user"
},
"Action": [
"kms:CreateGrant",
"kms:ListGrants",
"kms:RevokeGrant"
],
"Resource": "*",
"Condition": {
"Bool": {
"kms:GrantIsForAWSResource": "true"
}
}
}
]
}
I have a role in account-a, bucket-replication-role, with a trust relationship allowing S3 assume role and an attached policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
and an attached policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ReplicateDelete"
],
"Resource": "arn:aws:s3:::bucket-b/*"
},
{
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey"
],
"Resource": [
"arn:aws:kms:us-east-1:12345:key/[account-a-kms-key-id]"
]
},
{
"Effect": "Allow",
"Action": [
"kms:GenerateDataKey",
"kms:Encrypt"
],
"Resource": [
"arn:aws:kms:us-west-2:12345:key/[account-b-kms-key-id]"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ObjectOwnerOverrideToBucketOwner"
],
"Resource": "arn:aws:s3:::bucket-b/*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetReplicationConfiguration",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::bucket-a"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetObjectVersionForReplication",
"s3:GetObjectVersionAcl",
"s3:GetObjectVersionTagging"
],
"Resource": [
"arn:aws:s3:::bucket-a/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ReplicateObject",
"s3:ReplicateTags"
],
"Resource": "arn:aws:s3:::bucket-b/*"
}
]
}
Here is my replication rule, on bucket-a
The above role is attached as well, during creation.
and the batch operation is the default one that gets prompted on the replication rule creation.
The files are just small png's, jsons, html files, etc- nothing weird in there. You can see the replication status FAILED in the object information
Most of my policy rules came from this AWS support page: https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-replication/
Update
I added the following policy to account-b KMS key:
{
"Sid": "AllowS3ReplicationSourceRoleToUseTheKey",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::12345:role/bucket-replication-role"
},
"Action": ["kms:GenerateDataKey", "kms:Encrypt"],
"Resource": "*"
}
and the DstPutObjectNotPermitted errors have gone away, now its just the DstMultipartUploadNotPermitted errors I'm seeing.
Update 2
I tried to recreate the issue with two new buckets, and can not reproduce the issue, so I assume it's something to do with how some of the older files in bucket-a are stored.
This required some help from AWS Support, this was the relevant points of their response:
"DstMultipartUploadNotPermitted" status code indicates that the source objects are multipart uploads and the permissions required for their replication haven't been granted in the resource policies. Note that if a source object is uploaded using multipart upload to the source bucket, then the IAM replication role will also upload the replica object to destination bucket using multipart upload.
I would like to inform you that some extra permissions are to be granted for allowing multipart uploads in an S3 bucket. The list of permissions required for the IAM replication role to perform multipart uploads when KMS encryption is involved are listed below.
s3:PutObject on resource "arn:aws:s3:::DESTINATION-BUCKET/*"
kms:Decrypt and kms:GenerateDataKey on resource "arn:aws:kms:REGION:DESTINATION-ACCOUNT-ID:key/KEY-ID"
...as well as
ensure that the destination bucket policy is granting the "s3:PutObject" permission on resource "arn:aws:s3:::bucket-b/*" to the IAM replication role "arn:aws:iam::12345:role/bucket-replication-role".
...and finally
I would also request you to please grant "kms:Decrypt", and "kms:GenerateDataKey" permissions on the destination KMS key to the IAM replication role "arn:aws:iam::12345:role/bucket-replication-role" in the destination KMS key policy.
After adding all these additional permissions, everything worked as expected.
I would like to create the AWS Config access grant to the Amazon S3 Bucket and the policy is provided below that I write according to the link https://docs.aws.amazon.com/config/latest/developerguide/s3-bucket-policy.html:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AWSConfigBucketPermissionsCheck",
"Effect": "Allow",
"Principal": {
"Service": [
"config.amazonaws.com"
]
},
"Action": "s3:GetBucketAcl",
"Resource": "arn:aws:s3:::targetBucketName"
},
{
"Sid": "AWSConfigBucketExistenceCheck",
"Effect": "Allow",
"Principal": {
"Service": [
"config.amazonaws.com"
]
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::targetBucketName"
},
{
"Sid": "AWSConfigBucketDelivery",
"Effect": "Allow",
"Principal": {
"Service": [
"config.amazonaws.com"
]
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::targetBucketName/[optional] prefix/AWSLogs/sourceAccountID-WithoutHyphens/Config/*",
"Condition": {
"StringEquals": {
"s3:x-amz-acl": "bucket-owner-full-control"
}
}
}
]
}
I want to know about the part provided below:
"Resource": "arn:aws:s3:::targetBucketName/[optional] prefix/AWSLogs/sourceAccountID-WithoutHyphens/Config/*",
Whats the meaning of the prefix and how do I fill the prefix/AWSLogs/sourceAccountID-WithoutHyphens/Config/* part of the policy?
Thanks.
The prefix is what you define when you configure logging to S3. This is optional. Config writes the logs to S3 bucket using a standard path/key format which is "prefix/AWSLogs/sourceAccountID-WithoutHyphens/Config/*".
If you configure Config logging to S3 from console, you won't have to worry about the bucket policy as it will be created automatically. You simply give the bucket name and optional prefix.
I am the owner of AWS AccountC and need List and Get Permissions to BucketName owned by another person/team.
The bucket policy created is attached below. Policy for AccountA and AccountB were already existing and I added the policy for AccountC as given below
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AccessA",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::AccountA:root",
"arn:aws:iam::AccountA:user/ABC-Prod"
]
},
"Action": [
"s3:GetObject",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::BucketName/*",
"arn:aws:s3:::BucketName"
]
},
{
"Sid": "AccessB",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::AccountB:user/service-user",
"arn:aws:iam::AccountB:role/BatchUserRole"
]
},
"Action": "*",
"Resource": [
"arn:aws:s3:::BucketName/*",
"arn:aws:s3:::BucketName"
]
},
{
"Sid": "AccessC",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::AccountC:root"
},
"Action": "s3:List*",
"Resource": "arn:aws:s3:::BucketName"
},
{
"Sid": "AccessD",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::AccountC:root"
},
"Action": "s3:Get*",
"Resource": "arn:aws:s3:::BucketName/*"
}
]
}
I am able to list contents of BucketName using
aws s3 ls BucketName.
However, when I try
aws s3 cp --recursive BucketName/folderName/ ., it gives me an Access Denied error
An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
Block public access is enabled on the bucket, however I believe it should not affect since the Bucket policy is added
Tried multiple way to write the policy but the error persists. Can someone please help me understand what I might be missing here? Would be really grateful
Is it possible to create an S3 bucket policy that allows read and write access from aws machine learning ? I tried below bucket policy, but not work.
bucket policy:
{
"Version": "2008-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "machinelearning.amazonaws.com"
},
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::cxra/*"
},
{
"Effect": "Allow",
"Principal": {
"Service": "machinelearning.amazonaws.com"
},
"Action": "s3:PutObjectAcl",
"Resource": "arn:aws:s3:::cxra/*",
"Condition": {
"StringEquals": {
"s3:x-amz-acl": "bucket-owner-full-control"
}
}
},
{
"Effect": "Allow",
"Principal": {
"Service": "machinelearning.amazonaws.com"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::cxra"
},
{
"Effect": "Allow",
"Principal": {
"Service": "machinelearning.amazonaws.com"
},
"Action": [
"s3:GetBucketLocation"
],
"Resource": "arn:aws:s3:::cxra"
}
]
}
Error screen
Double check your policy against Granting Amazon ML Permissions to Output Predictions to Amazon S3 and Granting Amazon ML Permissions to Read Your Data from Amazon S3
As they are documented as two separate policies, I suggest you follow that, rather than joining the two policies in one single document. It will be easier to troubleshoot.
I've 2 AWS accounts. Account A has S3 bucket 'BUCKET' in which I've put file using Java api. I've configured my 'BUCKET' policy to allow cross-account file publishing.
But, when I try to open this file from Account A, it says AccessDeniedAccess Denied with hostId and requestId.
This file is published through Account B using java api, and this file has same size as that published through api. I tried to change file sizes and the new sizes were shown on AWS S3 console.
Here is my bucket policy:
{
"Version": "2008-10-17",
"Id": "Policy1357935677554",
"Statement": [
{
"Sid": "Stmt1357935647218",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNTB:user/accountb-user"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::bucket-name"
},
{
"Sid": "Stmt1357935676138",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNTB:user/accountb-user"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::bucket-name/*"
},
{
"Sid": "Stmt1357935676138",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNTB:user/accountb-user"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::bucket-name/*"
},
{
"Sid": "Stmt1357935647218",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNTA:user/accounta-user"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::bucket-name"
},
{
"Sid": "Stmt1357935676138",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNTA:user/accounta-user"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::bucket-name/*"
},
{
"Sid": "Stmt1357935676138",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNTA:root"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::bucket-name/*"
}
]
}
The problem is when I try to download/open this file from Account A, I'm not able to open it.
The problem is that by default, when AWS (cli or SDK) upload a file it grants access to the uploader only through s3 ACLs.
In that case, to allow the owner to read the uploaded file, the uploader has to explicitly grant access to the owner of the bucket during the upload. Ex:
with the aws CLI (documentation here): aws s3api put-object --bucket <bucketname> --key <filename> --acl bucket-owner-full-control
with the nodejs API (documentation here): you have to set the params.ACL property of the AWS.S3.upload method to "bucket-owner-full-control"
In parallel, you can also ensure that the Bucket Owner Has Full Control with the bucket policy, (additional documentation here):
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Grant Owner Full control dev",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNTB:root"
},
"Action": [
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::bucket-name/*"
],
"Condition": {
"StringEquals": {
"s3:x-amz-acl": "bucket-owner-full-control"
}
}
}
]
}