I've been reading multiple posts like this one about how to transfer data with aws cli from one S3 bucket to another using different accounts but I am still unable to do so. I'm sure it's because I haven't fully grasp the concepts of account + permission settings in AWS yet (e.g. iam account vs access key).
I have a vendor that gave me a user called "Foo" and account number "123456789012" with 2 access keys to access their S3 bucket "SourceBucket" in eu-central-1. I created a profile on my machine with the access key provided by the vendor called "sourceProfile". I have my S3 called "DestinationBucket" in us-east-1 and I set the bucket policy to the following.
{
"Version": "2012-10-17",
"Id": "Policy12345678901234",
"Statement": [
{
"Sid": "Stmt1487222222222",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:user/Foo"
},
"Action": [
"s3:PutObject",
"s3:PutObjectAcl",
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::DestinationBucket/",
"arn:aws:s3:::DestinationBucket/*"
]
}
]
}
Here comes the weird part. I am able to list the files and even download files from the "DestinationBucket" using the following command lines.
aws s3 ls s3://DestinationBucket --profile sourceProfile
aws s3 cp s3://DestinationBucket/test ./ --profile sourceProfile
But when I try to put copy anything to the "DestinationBucket" using the profile, I got Access Denied error.
aws s3 cp test s3://DestinationBucket --profile sourceProfile --region us-east-1
upload failed: ./test to s3://DestinationBucket/test An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
Did I set up the bucket policy especially the list of action right? How could ls and cp from destination to local work but cp from local to destination bucket doesn't work?
Because AWS make it a way that parent account holder must do the delegation.
Actually, beside delegates access on to that particular access key user, you can choose to do replication on the bucket as stated here.
Related
I created two profiles (one for source and one for target bucket) and using below command to copy:
aws s3 cp --profile source_profile s3://source_bucket/file.txt --profile target_profile s3://target_profile/
But it throws below error.
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
Looks like we can't use multiple profiles with aws commands.
The simplest method is to grant permissions via a bucket policy.
Say you have:
Account-A with IAM User-A
Account-B with Bucket-B
Add a bucket policy on Bucket-B:
{
"Id": "CopyBuckets",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GrantAccessToUser-A",
"Action": "s3:*",
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::bucket-b",
"arn:aws:s3:::bucket-b/*"
],
"Principal": {
"AWS": [
"arn:aws:iam::<account-a-id>:user/user-a"
]
}
}
]
}
Then just copy the files as User-A.
See also: aws sync between S3 buckets on different AWS accounts
No, you can't use multiple profiles in one AWS CLI command. Possible solutions:
1) Download files to local disk, then upload them to the target bucket with a separate command.
2) Allow first account access to the target bucket. For this, you will have to create a cross-account role in the source account and assign it the appropriate permissions in the target account. That way you will be using one role/one profile, but this role will be granted permissions in the second account. See https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html
I am trying to create a CloudFormation Stack using the AWS CLI by running the following command:
aws cloudformation create-stack --debug --stack-name ${stackName} --template-url ${s3TemplatePath} --parameters '${parameters}' --region eu-west-1
The template resides in an S3 bucket in the another account, lets call this account 456. The bucket policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Example permissions",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::123:root"
]
},
"Action": [
"s3:*"
],
"Resource": "arn:aws:s3:::cloudformation.template.eberry.digital/*"
}
]
}
("Action: * " is for debugging).
Now for a twist. I am logged into account 456 and I run
aws sts assume-role --role-arn arn:aws:iam::123:role/delegate-access-to-infrastructure-account-role --role-session-name jenkins
and the set the correct environment variables to access 123. The policy attached to the role that I assume allow the user Administrator access while I debug - which still doesn't work.
aws s3api list-buckets
then display the buckets in account 123.
To summarize:
Specifying a template in an S3 bucket owned by account 456, into CloufFormation in the console, while logged into account 123 works.
Specifying a template in an S3 bucket owned by account 123, using the CLI, works.
Specifying a template in an S3 bucket owned by account 456, using the CLI, doesn't work.
The error:
An error occurred (ValidationError) when calling the CreateStack operation: S3 error: Access Denied
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
I don't understand what I am doing wrong and would by thankful for any ideas. In the meantime I will upload the template to all accounts that will use it.
Amazon S3 provides cross-account access through the use of bucket policies. These are IAM resource policies (which are applied to resources—in this case an S3 bucket—rather than IAM principals: users, groups, or roles). You can read more about how Amazon S3 authorises access in the Amazon S3 Developer Guide.
I was a little confused about which account is which, so instead I'll just say that you need this bucket policy when you want to deploy a template in a bucket owned by one AWS account as a stack in a different AWS account. For example, the template is in a bucket owned by AWS account 111111111111 and you want to use that template to deploy a stack in AWS account 222222222222. In this case, you'll need to be logged in to account 222222222222 and specify that account as the principal in the bucket policy.
The following is an example bucket policy that provides access to another AWS account; I use this on my own CloudFormation templates bucket.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::AWS_ACCOUNT_ID_WITHOUT_HYPHENS:root"
},
"Action": [
"s3:GetBucketLocation",
"s3:GetObject",
"s3:GetObjectTagging",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::S3_BUCKET_NAME",
"arn:aws:s3:::S3_BUCKET_NAME/*"
]
}
]
}
You'll need to use the 12-digit account identifier for the AWS account you want to provide access to, and the name of the S3 bucket (you can probably use "Resource": "*", but I haven't tested this).
This is the bucket policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "statement1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::xxxxxxxxxxxx:user/userName"
},
"Action": "*",
"Resource": "arn:aws:s3:::my-super-awesome-bucket-name-test/*"
}
}
Using AWS CLI I am able to list the contents of the bucket:
aws s3 ls s3://my-super-awesome-bucket-name-test
2017-06-28 19:50:42 97 testFile.csv
However, I can't upload files:
aws s3 cp csv_sum.js s3://my-super-awesome-bucket-name-test/
upload failed: ./csv_sum.js to s3://my-super-awesome-bucket-name-test/csv_sum.js An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
Is there something else I need to do to grant my IAM user access? I added the required information via aws configure, is there something else needed?
This doesn't answer your specific question, but...
If you wish to grant Amazon S3 access to a specific IAM User, it is much better to assign a policy directly to the IAM User rather than adding them as a special-case on the S3 bucket policy.
You can similarly assign permissions to IAM Groups, and then any User who is assigned to that Group will inherit the permissions. You can even assign permissions for multiple S3 buckets this way, rather than having to modify several bucket policies.
I got text file to access S3 bucket like following:
arn:aws:iam::############:user/aaaaaaaa-aaaaaaaaa-aaa
User
aaaaaaaa-aaaaaaaaa-aaa
Access key ID
AAAAAAAAAAAAAAAAAAAA
Secret access key
AAAAAAAAAAA/AAAAAAAAAAAAAAAAAAAAAAAAAAAA
{
"Statement": [
{
"Effect":"Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:ListAllMyBuckets",
"s3:GetBucketLocation",
"s3:ListBucket"
],
"Resource":"arn:aws:s3:::bbbbbbbb-bbbbbbbbb-bbbbbbb/*"
}
]
}
I have AWS account and can create my own buckets, but see no UI to acquire such files.
UPDATE
I issued
>aws s3 ls
Unable to locate credentials. You can configure credentials by running "aws configure".
then I did
>aws configure
AWS Access Key ID [None]: AAAAAAAAAAAAAAAAAAAA
AWS Secret Access Key [None]: AAAAAAAAAAA/AAAAAAAAAAAAAAAAAAAAAAAAAAAA
Default region name [None]:
Default output format [None]:
and now
>aws s3 ls
An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied
Why? Why didn't I use neither User nor Resource values from my text file or how to use this data?
UPDATE 2
I tried
>aws s3 ls
>aws s3 ls s3://bbbbbbbb-bbbbbbbbb-bbbbbbb
>aws s3 ls bbbbbbbb-bbbbbbbbb-bbbbbbb
>aws s3 ls bbbbbbbb-bbbbbbbbb-bbbbbbb/*
>aws s3 ls s3:/bbbbbbbb-bbbbbbbbb-bbbbbbb
And got Access denied in all cases.
It appears that your System Administrators have created some configurations in AWS and they wanted to let you know what they have done. The file is a dump of information from various locations -- it is for your reference and is not for 'use' somewhere.
The first line is the Amazon Resource Name (ARN) that uniquely identifies you as a user. It can be used in security policies to grant you access to resources:
arn:aws:iam::############:user/aaaaaaaa-aaaaaaaaa-aaa
They are also telling you your Username:
User
aaaaaaaa-aaaaaaaaa-aaa
The Access Key and Secret Key can be used to identify yourself, as you have done with the AWS Command-Line Interface (CLI):
Access key ID
AAAAAAAAAAAAAAAAAAAA
Secret access key
AAAAAAAAAAA/AAAAAAAAAAAAAAAAAAAAAAAAAAAA
The next part is an IAM Policy:
{
"Statement": [
{
"Effect":"Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:ListAllMyBuckets",
"s3:GetBucketLocation",
"s3:ListBucket"
],
"Resource":"arn:aws:s3:::bbbbbbbb-bbbbbbbbb-bbbbbbb/*"
}
]
}
This policy states that you can perform the listed actions against the specified Amazon S3 bucket.
It's not a great policy however, because the last 3 actions actually apply to a bucket (or no bucket), so should be not used with a Resource statement that specifies bucket/*.
If you are trying to access information in Amazon S3 but receive Access Denied, then contact your System Administrator to update the policy to grant you access.
You have configured your credentials properly based on your update
in the question.
But you haven't specified a default region in the configuration.
Check with your admins what is the region for this S3 bucket. It
could be something like us-east-1 or us-west-2.
Once you have your bucket's region, you can issue a command as
below:
aws s3 ls <name of your bucket> --region us-east-1
The reason you are receiving access denied is you do not have access to other buckets, but only one of the buckets on S3. This is suggested by this line:
"Resource":"arn:aws:s3:::bbbbbbbb-bbbbbbbbb-bbbbbbb/*"
Where bbbbbbbb-bbbbbbbbb-bbbbbbb is name of you bucket.
You need to go to IAM to create a policy for your bucket. Than you need to add this policy to your users account and than you can access this bucket using your users AccessKey and SecretAccessKey
I have a bucket on s3, and a user given full access to that bucket.
I can perform an ls command and see the files in the bucket, but downloading them fails with:
A client error (403) occurred when calling the HeadObject operation: Forbidden
I also attempted this with a user granted full S3 permissions through the IAM console. Same problem.
For reference, here is the IAM policy I have:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "arn:aws:s3:::*"
},
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::mybucket",
"arn:aws:s3:::mybucket/*"
]
}
]
}
I also tried adding a bucket policy, even making the bucket public, and still no go...also, from the console, I tried to set individual permissions on the files in the bucket, and got an error saying I cannot view the bucket, which is strange, since I was viewing it from the console when the message appeared, and can ls anything in the bucket.
EDIT the files in my bucket were copied there from another bucket belonging to a different account, using credentials from my account. May or may not be relevant...
2nd EDIT just tried to upload, download and copy my own files to and from this bucket from other buckets, and it works fine. The issue is specifically with the files placed there from another account's bucket.
Thanks!
I think you need to make sure that the permissions are applied to objects when moving/copying them between buckets with the "bucket-owner-full-control" acl.
Here are the details about how to do this when moving or copying files as well as retroactively:
https://aws.amazon.com/premiumsupport/knowledge-center/s3-bucket-owner-access/
Also, you can read about the various predefined grants here:
http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
The problem here stems from how you get the files into the bucket. Specifically the credentials you have and/or privileges you grant at the time of upload. I ran into a similar permissions issue issue when I had multiple AWS accounts, even though my bucket policy was quite open (as yours is here). I had accidentally used credentials from one account (call it A1) when uploading to a bucket owned by a different account (A2). Because of this A1 kept the permissions on the object and the bucket owner did not get them. There are at least 3 possible ways to fix this in this scenario at time of upload:
Switch accounts. Run $export AWS_DEFAULT_PROFILE=A2 or, for a more permanent change, go modify ~/.aws/credentials and ~/.aws/config to move the correct credentials and configuration under [default]. Then re-upload.
Specify the other profile at time of upload: aws s3 cp foo s3://mybucket --profile A2
Open up the permissions to bucket owner (doesn't require changing profiles): aws s3 cp foo s3://mybucket --acl bucket-owner-full-control
Note that the first two ways involve having a separate AWS profile. If you want to keep two sets of account credentials available to you, this is the way to go. You can set up a profile with your keys, region etc by doing aws configure --profile Foo. See here for more info on Named Profiles.
There are also slightly more involved ways to do this retroactively (post upload) which you can read about here.
To correctly set the appropriate permissions for newly added files, add this bucket policy:
[...]
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012::user/their-user"
},
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::my-bucket/*"
}
Your bucket policy is even more open, so that's not what's blocking you.
However, the uploader needs to set the ACL for newly created files. Python example:
import boto3
client = boto3.client('s3')
local_file_path = '/home/me/data.csv'
bucket_name = 'my-bucket'
bucket_file_path = 'exports/data.csv'
client.upload_file(
local_file_path,
bucket_name,
bucket_file_path,
ExtraArgs={'ACL':'bucket-owner-full-control'}
)
source: https://medium.com/artificial-industry/how-to-download-files-that-others-put-in-your-aws-s3-bucket-2269e20ed041 (disclaimer: written by me)