I am pushing a s3 file from accountA to accountB but the pushed file is not accessible from accountB. I checked the pushed file and the Owner of the pushed file appears to be accountA.
Here is what I have done.
The IAM role in accountA has this policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*",
"s3-object-lambda:*"
],
"Resource": "*"
}
]
}
The bucket policy in accountB looks like this:
{
"Sid": "S3AllowPutFromDataLake",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::accountId:role/roleNameAccountA"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::bucketName/*"
}
How to fix this?
This is a common problem when copying S3 objects between AWS Accounts. Here are several options to avoid it happening. Pick whichever one you prefer:
Pull instead of Push
The problem occurs when Account A copies an object to Account B. Ownership stays with Account A.
This can be avoided by having Account B trigger the copy. It is, in effect, 'pulling' the object into Account B rather than 'pushing' the object. Ownership will stay with Account B, since Account B requested the copy.
Disable ACLs
The concept of object-level ACLs pre-dates Bucket Policies and causes many problems like the one you are experiencing.
Amazon S3 has now introduced the ability to disable ACLs on a bucket and this is the recommended option when creating new buckets. Disabling the ACLs will also remove this 'ownership' concept that is causing problems. In your situation, it is the Target bucket in Account B that should have ACLs disabled.
See: Disabling ACLs for all new buckets and enforcing Object Ownership - Amazon Simple Storage Service
Specify ownership while copying
When copying the file, it is possible to specify that ownership should be transferred by setting the ACL to bucket-owner-full-control.
Using the AWS CLI:
aws s3 cp s3://bucket-a/foo.txt s3://bucket-b/foo.txt --acl bucket-owner-full-control
Using boto3:
s3_client.copy_object(
ACL = 'bucket-owner-full-control',
Bucket = DESTINATION_BUCKET,
Key = KEY,
CopySource = {'Bucket':SOURCE_BUCKET, 'Key':KEY}
)
Was able to fix this by modifying the bucket policy as below:
{
"Sid": "S3AllowPutFromDataLake",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::accountId:role/roleNameAccountA"
},
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
]
"Resource": "arn:aws:s3:::bucketName/*",
"Condition": {
"StringEquals": {
"s3:x-amz-acl": "bucket-owner-full-control"
}
}
}
And adding this parameter while pushing the file:
'ACL': 'bucket-owner-full-control'
The owner is still accountA but now I am able to access the file from accountB.
Related
we are trying to implement the lambda function which will copy the object from one S3 to another S3 bucket in cross account based on the source S3 bucket events. Currently we are able to copy the file between source and target within same SAG . But when we tried to implement the same logic with cross account , getting the CopyObject operation: Access Denied issue . I have given following bucket policy. Can you please help me to get the correct IAM and bucket policy to resolve this issue .
{
"Version": "2012-10-17",
"Id": "Policy1603404813917",
"Statement": [
{
"Sid": "Stmt1603404812651",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::6888889898:role/Staff"
},
"Action": [
"s3:GetObject",
"s3:GetObjectAcl",
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::source-bucktet-testing-lambda/*",
"arn:aws:s3:::source-bucktet-testing-lambda"
]
}
]
}
based on the https://www.lixu.ca/2016/09/aws-lambda-and-s3-how-to-do-cross_83.html link , Yes, we can implement the same logic with help of access ID and access secret keys for source and dest. But am trying to implement same logic instead of access ID and access secret keys for source and dest, granting access for both source and target buckets with appropriate policy and make it work as like same account .
To reproduce your situation, I did the following:
In Account-A:
Created an Amazon S3 bucket (Bucket-A)
Created an IAM Role (Role-A)
Created an AWS Lambda function (Lambda-A) and assigned Role-A to the function
Configured an Amazon S3 Event on Bucket-A to trigger Lambda-A for "All object create events"
In Account-B:
Created an Amazon S3 bucket (Bucket-B) with a bucket policy (see below)
IAM Role
Role-A has the AWSLambdaBasicExecutionRole managed policy, and also this Inline Policy that assigns the Lambda function permission to read from Bucket-A and write to Bucket-B:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::bucket-a/*"
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::bucket-b/*"
}
]
}
Bucket Policy on destination bucket
The Bucket Policy on Bucket-B permits access from the Role-A IAM Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNT-A:role/role-a"
},
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::bucket-b/*"
}
]
}
Lambda Function
Lambda-A is triggered when an object is created in Bucket-A, and copies it to Bucket-B:
import boto3
import urllib
TARGET_BUCKET = 'bucket-b'
def lambda_handler(event, context):
# Get incoming bucket and key
source_bucket = event['Records'][0]['s3']['bucket']['name']
source_key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'])
# Copy object to different bucket
s3_resource = boto3.resource('s3')
copy_source = {
'Bucket': source_bucket,
'Key': source_key
}
target_key = source_key # Change if desired
s3_resource.Bucket(TARGET_BUCKET).Object(target_key).copy(copy_source, ExtraArgs={'ACL': 'bucket-owner-full-control'})
I grant ACL=bucket-owner-full-control because copying objects to buckets owned by different accounts can sometimes cause the objects to still be 'owned' by the original account. Using this ACL grants ownership to the account that owns the destination bucket.
Testing
I uploaded a file to Bucket-A in Account-A.
The file was correctly copied to Bucket-B in Account-B.
Comments
The solution does NOT require:
A bucket policy on Bucket-A, since Role-A grants the necessary permissions
Turning off S3 Block Public Access, since the permissions assigned do not grant 'public' access
Assuming the following
Above mentioned policy is for the source bucket
6888889898 is the Destination AWS account
Lambda for copying the file is located in the destination AWS account and has Staff role attached to it.
Even after setting all these correctly, the copy operation may fail. This is because the Policy allows you to get/put s3 objects, but not the tags associated with those s3 objects.
You will need to ALLOW the following actions as well "s3:GetObjectTagging" and "s3:PutObjectTagging"
Here is what I am trying to do:
I have access logs in account A which are encrypted default by AWS and I have lambda and s3 bucket in account B. I want to trigger the lambda when a new object lands on the account A s3 bucket and lambda in account B downloads the data and writes it to account B s3 bucket. Below are the blocks I am facing.
First approach:
I was able to get the trigger from account A s3 new object to lambda in account B however, the lambda in account B is not able to download the object - Access Denied error. After looking for a couple of days, I figured that it is because the Access logs are encrypted by default and there is no way I can add lambda role to the encryption role policy so that it can encrypt/decrypt the log files. So moved on to the second approach.
Second approach:
I have moved my lambda to Account A. Now the source s3 bucket and lambda are in Account A and destination s3 bucket is in Account B. Now I can process the Access logs in the Account A via Lambda in Account A but when it writes the file in the Account B s3 bucket I get Access denied error while downloaded/reading the file.
Lambda role policy:
In addition to full s3 access and full lambda access.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1574387531641",
"Effect": "Allow",
"Action": "s3:*",
"Resource": "*"
},
{
"Sid": "Stmt1574387531642",
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::Account-B-bucket",
"arn:aws:s3:::Account-B-bucket/*"
]
}
]
}
Trust relationship
{ "Version": "2012-10-17", "Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com",
"AWS": "arn:aws:iam::Account-B-ID:root"
},
"Action": "sts:AssumeRole"
} ] }
Destination - Account B s3 bucket policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::Account-A-ID:role/service-role/lambda-role"
]
},
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::Account-B-Bucket",
"arn:aws:s3:::Account-B-Bucket/*"
]
},
{
"Sid": "Stmt11111111111111",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::Account-A-ID:root"
},
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::Account-B-Bucket",
"arn:aws:s3:::Account-B-Bucket/*"
]
}
] }
I am stuck here. I want lambda to be able to decrypt the access logs and read/process the data and write it to different account s3 bucket. Am I missing something? Help is much appreciated!
Adding file metadata:
File property screenshot
Lambda Code:
s3 = boto3.client('s3')
# reading access logs from account A. Lambda is also running in account A.
response = s3.get_object(Bucket=access_log_bucket, Key=access_log_key)
body = response['Body']
content = io.BytesIO(body.read())
# processing access logs
processed_content = process_it(content)
# writting to account B s3 bucket
s3.put_object(Body=processed_content,
Bucket=processed_bucket,
Key=processed_key)
Rather than downloading and then uploading the object, I would recommend that you use the copy_object() command.
The benefit of using copy_object() is that the object will be copied directly by Amazon S3, without the need to first download the object.
When doing so, the credentials you use must have read permissions on the source bucket and write permissions on the destination bucket. (However, if you are 'processing' the data, this of course won't apply.)
As part of this command, you can specify an ACL:
ACL='bucket-owner-full-control'
This is required because the object is being written from credentials in Account A to a bucket owned by Account B. Using bucket-owner-full-control will pass control of the object to Account B. (It is not required if using credentials from Account B and 'pulling' an object from Account A.)
Thanks John Rotenstein for the direction. I found the solution. I only needed to add ACL='bucket-owner-full-control' in the put_object. Below is the complete boto3 cmd.
s3.put_object(
ACL='bucket-owner-full-control'
Body=processed_content,
Bucket=processed_bucket,
Key=processed_key)
I'm writing an IAM policy for my Kubernetes workers. I want them to have read/write access to a certain S3 bucket, no access to other S3 buckets in my account, but read access to any bucket that is publicly readable (such as the s3://1000genomes/ bucket, and other buckets where Amazon or other people have put up public data).
(Whether access is granted or denied to public buckets in my account doesn't matter. If I need to grant access to one I can do that explicitly. But I can't go granting access to every public bucket in the world explicitly.)
I can write a stanza that looks like this to give read access to all buckets in the world:
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::*",
"arn:aws:s3:::*/*"
]
}
I can also write one that grants access to the particular named bucket on my account that I want to allow access to, filling the name in for *.
But how do I write a policy that denies access to buckets on my account that aren't the named bucket? Or if I take a different approach, how do I write a grant for all buckets not on my account? Is there something I can put between those extra colons that will do this? I've tried using Resource and NotResource together, but AWS rejects that.
For reference, my policy looks like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:ListAllMyBuckets"
],
"Resource": "arn:aws:s3:::*"
},
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::vg-k8s",
"arn:aws:s3:::vg-k8s/*",
]
}
]
}
With that policy in place on my nodes' group, I have no access to s3://1000genomes from the nodes. To get access to that bucket specifically, I can change it to:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:ListAllMyBuckets"
],
"Resource": "arn:aws:s3:::*"
},
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::vg-k8s",
"arn:aws:s3:::vg-k8s/*",
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::1000genomes",
"arn:aws:s3:::1000genomes/*"
]
}
]
}
But I need a way to do this so I cover all public buckets, not just those I list, but so I don't cover non-public buckets in my account.
How do I write a policy that denies access to buckets on my account that aren't the named bucket?
You don't. IAM policies are deny by default.
How do I give read access to any bucket that is publicly readable?
You don't. No additional permission or policy is required to access public buckets.
You will potentially run into an issue, however, with public bucket access and should make your requests to public buckets unsigned.
To interact with the public bucket from the awscli, use:
aws s3 ls s3://1000genomes/ --no-sign-request
To do the same from boto3, use:
import boto3
from botocore import UNSIGNED
from botocore.config import Config
s3 = boto3.client('s3', config=Config(signature_version=UNSIGNED))
I have 3 accounts I will refer to as account AA, BB and CC.
AA does a putObject to a s3 bucket in BB, and CC has a lambda that is triggered when an object is created in BB
When I create an object in the s3 bucket from account BB, the lambda works as expected. I can do this through the console, or the s3 api.
When the object is put in there from account AA, I am able to read the event in the lambda, but get Access Denied when trying to do an s3:GetObject
At one point I had the lambda in BB, and it was able to perform the s3:GetObject on objects created by AA. It is only when I moved the lambda to CC, did i start experiencing issues with objects created by AA.
here is my s3 bucket policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::AA:root"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::BUCKET_NAME/*"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::CC:role/LAMBDA_ARN"
},
"Action": "s3:Get*",
"Resource": "arn:aws:s3::BUCKET_NAME/*"
}
]
}
And here is my statement from CC lambda that allows access to the s3
{
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::BUCKET_NAME*",
"arn:aws:s3:::BUCKET_NAME*/*"
],
"Effect": "Allow",
}
The lambda execution role has full permissions to s3:get* on account BB.
The fact that it was written by another account should not affect reading that object, As I can take that same object that was written in there from AA, write it into BB again, and CC lambda will be able to read it just fine.
When writing an object to Amazon S3 from a different account (that is, from an account that does not own the S3 bucket), it is recommended to set the ACL to bucket-owner-full-control. This allows the bucket owner to control the object.
As strange as it seems, it is possible to create objects in a bucket that the owning account cannot access!
See: Access Control List (ACL) Overview - Amazon Simple Storage Service
I've tried pretty much every possible bucket policy. Also tried adding a policy to the user, but I get Access Denied every time I try to download an object from s3 bucket using the AWS Console.
Bucket Policy:
{
"Version": "2012-10-17",
"Id": "MyPolicy",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::12345678901011:user/my-username"
},
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::my-bucket",
"arn:aws:s3:::my-bucket/*"
]
},
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::my-bucket/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"XX.XXX.XXX.XXX/24",
"XXX.XXX.XXX.XXX/24"
]
}
}
}
]
}
That doesn't work so I tried adding a policy to my-username:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "StmtXXXXXXXXXX",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::my-bucket",
"arn:aws:s3:::my-bucket/*"
]
}
]
}
As strange as it sounds, it is possible to upload an object to Amazon S3 that the account owning the bucket cannot access.
When an object is uploaded to Amazon S3 (PutObject), it is possible to specify an Access Control List (ACL). Possible values are:
private
public-read
public-read-write
authenticated-read
aws-exec-read
bucket-owner-read
bucket-owner-full-control
You should normally upload objects with the bucket-owner-full-control ACL. This allows the owner of the bucket access to the object and permission to control the object (eg delete it).
If this permission is not supplied, then they cannot access nor modify the object.
I know that it contradicts the way you'd think buckets should work, but it's true!
How to fix it:
Re-upload the objects with bucket-owner-full-control ACL, or
The original uploader can loop through the objects and do an in-place CopyObject with a new ACL. This changes the permissions without having to re-upload.
UPDATE: In November 2021, a new feature was released: Amazon S3 Object Ownership can now disable access control lists to simplify access management for data in S3. This avoids the need to specify object ownership and fixes most problems with object ownership.
You can solve it by using : http://docs.aws.amazon.com/cli/latest/reference/s3api/put-object-acl.html
put-object-acl : This has to be done by original uploader.
But is definitely faster than copying data again.
I had TB's of data to deal with.
aws s3api put-bucket-acl --bucket $foldername --key $i --grant-full-control uri=http://acs.amazonaws.com/groups/global/AllUsers