I have an AWS role with s3 Read only permissions. I have already configured aws cli for an AWS user. So I want to use the same user to browse s3 files in aws cli.
what I did is,
Added trust relationship for root user to the role arn:aws:iam::<1234...>:role/test-role so that i can get this to all my iam users
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<1234..>:root",
"Service": "s3.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
then, I added a policy to user to assume above role.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt12345",
"Effect": "Allow",
"Action": [
"sts:AssumeRole"
],
"Resource": "arn:aws:iam::<1234...>:role/test-role"
}
]
}
When i try to list , I get permission denied error.
aws s3 ls
An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied
I made sure that role has full s3 read permission as follows.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": "*"
}
]
}
Can someone please guide where the problem is ?
If you're using the CLI you need a profile with the correct credentials.
You should have your credentials in an .aws/credentials file, e.g.:
[myprofile]
aws_access_key_id = ... access key ...
aws_secret_access_key = ... secret access key …
Then you can add a profile for the assumed role to the .aws/config file, e.g.:
[profile test-role]
source_profile=myprofile
role_arn = arn:aws:iam::<1234...>:role/test-role
Finally you set AWS_PROFILE to test-role before running the CLI command
SET AWS_PROFILE=test-role
aws s3 ls
I would have just posted a link to the AWS documentation but this site disapproves of link-only answers.
Related
I am trying to allow a user to assume a role on AWS. I attached an assume role policy to a group where the IAM user belongs so that they can assume a particular role. The problem is that the user now uses SSO to login and and is no longer allowed to login into through console with the IAM user credentials, therefore the user is unable to assume the role. How can I configure a user with SSO login to assume an existing IAM role? When i created the Assume role policy I chose both AssumeRole and AssumeRoleWithSaml. But it's still not working.
This is what the AssumeRole policy looks like
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"sts:AssumeRole",
"sts:AssumeRoleWithSAML"
],
"Resource": "arn:aws:iam::xxxxxxxxxxxx:role/service-role/KinesisFirehoseServiceRole--us-east-1-xxxxxxxxxxxxx"
}
]
The Trust relationship for the role looks like this
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "firehose.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
You need to specifically allow that user / role to be able to assume that role. Right now, the Principal is set to only allow the "firehose service" to assume that role.
The 2nd problem is that you need to specifically allow a SSO account to be able to access it. You'll need to get the ARN of your current SSO user session. To get this you should run aws sts get-caller-identity
You should get something like this
{
"UserId": "BROA5DAM2TACHAA38V9J1:daryl.teo#appetiser.com.au",
"Account": "1234567890",
"Arn": "arn:aws:sts::1234567890:assumed-role/AWSReservedSSO_AWSAdministratorAccess_abe68abec87ew/something.username"
}
Or a 1 liner aws sts get-caller-identity --output text --query Arn
Then take that value and add it to your policy as an additional policy statement.
{
"Version": "2012-10-17",
"Statement":
[
{
"Effect": "Allow",
"Principal":
{
"Service": "firehose.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Effect": "Allow",
"Principal":
{
"AWS": "arn:aws:sts::1234567890:assumed-role/AWSReservedSSO_AWSAdministratorAccess_abe68abec87ew/something.username"
},
"Action": "sts:AssumeRole"
},
]
}
And now you can use:
aws sts assume-role --role-arn=arn:aws:iam::123456823432:role/NameOfYourRole --role-session-name=role-session-name
I managed to enable SSO users to assume a role in the account they were authenticated to by using the following. Note that you'll need to replace ${ACCOUNT_ID}, ${SSO_ROLE_NAME}, and ${ASSUMABLE_ROLE_NAME}. You may, of course, need to repackage the bits.
aws iam create-role --role-name ${ASSUMABLE_ROLE_NAME} --assume-role-policy-document file://policy.json --profile $PROFILE
policy.json:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::${ACCOUNT_ID}:root"
},
"Action": "sts:AssumeRole",
"Condition": {
"ArnLike": {
"aws:PrincipalArn": [
"arn:aws:iam::${ACCOUNT_ID}:role/aws-reserved/sso.amazonaws.com/*/AWSReservedSSO_${SSO_ROLE_NAME}_*",
"arn:aws:iam::${ACCOUNT_ID}:role/aws-reserved/sso.amazonaws.com/AWSReservedSSO_${SSO_ROLE_NAME}_*"
]
}
}
}
]
}
Thanks to everyone that responded. I was able to complete the task using this instructions. google saml sso with AWS
I have access to one of two AWS environments and I've created a protected S3 bucket in it to upload files to from an account in the one that I do not. The environment and the account that I don't have access to are what a project's CI uses.
environment I have access to: env1
environment I do not have access to: env2
account I do not have access to: user/ci
bucket name: content
S3 bucket policy:
{
"Version": "2008-10-17",
"Id": "PolicyForCloudFrontPrivateContent",
"Statement": [
{
...
},
{
"Sid": "Allow access to bucket from profile in env1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:user/ci"
},
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket*"
],
"Resource": "arn:aws:s3:::content"
},
{
"Sid": "Allow access to bucket items from profile in env1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:user/ci"
},
"Action": [
"s3:Get*",
"s3:PutObject",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::content",
"arn:aws:s3:::content/*"
]
}
]
}
From inside a container that's configured for env1 and user/ci I'm testing with the command
aws s3 sync content/ s3://content/
and I get the error:
fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
I have two questions:
Am I even using the correct aws command to upload the data to the bucket?
Am I missing something from my bucket policy?
For the latter, I've basically followed what a load of examples and answers online have suggested.
To test your policy, I did the following:
Created an IAM User with no policies
Created an Amazon S3 bucket
Attached your Bucket Policy to the bucket, and updated the ARN and bucket name
Tested access to the bucket with:
aws s3 ls s3://bucketname
aws s3 sync folder/ s3://bucketname/folder/
It worked fine.
Therefore, the policy you display appears to be giving all necessary permissions. It is possible that you have something else that is Denying access on the bucket.
The solution was to given the ACL
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::content",
"arn:aws:s3:::content/*"
]
}
]
}
to user/ci in env1.
I am having trouble syncing two S3 buckets that are attached to two separate AWS accounts.
There are two AWS accounts - Account A which is managed by a third party and Account B, which I manage. I am looking to pull files from an S3 bucket in Account A to an S3 bucket in Account B.
Account A provided me the following instructions:
In Account B, create a new IAM user called LogsUser. Attach the following policy to the user:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::ACCOUNTID:role/12345-LogAccess-role"
}
]
}
Configure the AWS CLI to update the config and credentials files. Specifically, the ~/.aws/config file to look like:
[profile LogsUser]
role_arn = arn:aws:iam::ACCOUNTID:role/12345-LogAccess-role
source_profile = LogsUser
And the ~/.aws/credentials file to look like
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
From here, I am successfully able to query the log files in Account A's bucket using $ aws s3 ls --profile LogsUser s3://bucket-a.
I have set up bucket-b in Account B, however, I am unable to query any files in bucket-b. For example, $ aws s3 ls --profile LogsUser s3://bucket-b returns An error occurred (AccessDenied) when calling the AssumeRole operation: Access denied.
Is there something additional I can add to the config file or my IAM policy to allow access to bucket-b using --profile LogsUser option? I can access bucket-b using other --profile settings, but am not looking to sync to the local file system and then to another bucket.
The desired results is to run a command like aws s3 sync s3://bucket-a s3://bucket-b --profile UserLogs.
For example, if you want to copy “Account A” S3 bucket objects to “Account B” S3 bucket, follow below.
Create a policy for the S3 bucket in “account A” like the below policy. For that, you need “Account B” number, to find the B account number go to Support → Support center and copy the account number from there.
Setup “account A” bucket policy :
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DelegateS3Access",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNT_B_NUMBER:root"
},
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::ACCOUNT_A_BUCKET_NAME/*",
"arn:aws:s3:::ACCOUNT_A_BUCKET_NAME"
]
}
]
}
Log into “Account B” and create a new IAM user or attach the below policy for the existing user.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::ACCOUNT_A_BUCKET_NAME",
"arn:aws:s3:::ACCOUNT_A_BUCKET_NAME/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::ACCOUNT_B_BUCKET_NAME",
"arn:aws:s3:::ACCOUNT_B_BUCKET_NAME/*"
]
}
]
}
Configure AWS CLI with “Account B” IAM user(Which you have created IAM with the above user policy)
aws s3 sync s3://ACCOUNT_A_BUCKET_NAME s3://ACCOUNT_B_BUCKET_NAME --source-region ACCOUNT_A_REGION-NAME --region ACCOUNT_B_REGION-NAME
This way we can copy S3 bucket objects over different AWS accounts.
If you have multiple awscli profiles, use --profile end of the command with profile name.
Your situation is:
You wish to copy from Bucket-A in Account-A
The files need to be copied to Bucket-B in Account-B
Account-A has provided you with the ability to assume LogAccess-role in Account-A, which has access to Bucket-A
When copying files between buckets using the CopyObject() command (which is used by the AWS CLI sync command), it requires:
Read Access on the source bucket (Bucket-A)
Write Access on the destination bucket (Bucket-B)
When you assume LogAccess-role, you receive credentials that have Read Access on Bucket-A. That is great! However, those credentials do not have permission to write to Bucket-B because it is in a separate account.
To overcome this, you should create a Bucket Policy on Bucket-A that grants Write Access to LogAccess-role from Account-B. The Bucket Policy would look something like:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNT-A:role/12345-LogAccess-role"
},
"Action": [
"s3:ListBucket",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::bucket-a",
"arn:aws:s3:::bucket-a/*"
]
}
]
}
(You might need other permissions. Check any error messages for hints.)
That way, LogAccess-role will be able to read from Bucket-A and write to Bucket-B.
I would suggest you to consider you to use AWS S3 bucket replication:
https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html
If you just want to list objects in bucket-b, do this.
First make sure the LogsUser IAM user has got proper permission to access the bucket-b s3 bucket in Account B. You can add this policy to the user if not
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::bucket-b/*"
]
}
]
}
If there is permissions attached to the user, and if the Access keys and Secret Key stored in ~/.aws/credentials stored as [default] belongs to LogsUser IAM user, you can simply list objects inside bucket-b with following command.
aws s3 ls
If you want to run the command aws s3 sync s3://bucket-a s3://bucket-b --profile UserLogs, do this.
Remember, we will be using temporary credentials created by STS after assuming the role with permanent credentials of LogsUser. That means the role in Account A should have proper access to both buckets to perform the action and the bucket(bucket-b) in another account (Account B) should have proper bucket policy to allow the role to perform S3 operations.
To provide permissions to the role to access bucket-b, attach following bucket policy to bucket-b.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNTID:role/12345-LogAccess-role"
},
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::bucket-b/*"
]
}
]
}
Also in Account A, attach a policy to the role like below to allow access to S3 buckets in both the accounts.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::bucket-b/*",
"arn:aws:s3:::bucket-a/*"
]
}
]
}
I have two accounts, account ACCOUNTAAAA, and ACCOUNTBBBB. A bucket (BUCKETAAAA) exists in account ACCOUNTAAAA and an instance with a role (ROLEBBBB) in ACCOUNTBBBB needs to be able to read from it.
I have adding the following permissions to the bucket:
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNTBBBB:role/ROLEBBBB"
},
"Action": [
"s3:*"
],
"Resource": "arn:aws:s3:::BUCKETAAAA/*"
}
I have the following permissions on the role in ACOUNTBBBB:
"Sid": "",
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::BUCKETAAAA/",
"arn:aws:s3:::BUCKETAAAA/*"
]
}
My assumption is I should be able to run aws s3 ls s3://BUCKETAAAA on the EC2 instance with the IAM role attached, and see the contants of BUCKETAAAA. When I try this, I get An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied.
What am I missing here?
To reproduce your situation, I did the following:
Created Role-B in Account-B with EC2 as the Trusted Entity ("Allows EC2 instances to call AWS services on your behalf") and a policy granting access to Bucket-A
Created Bucket-A in Account-A
Added a Bucket Policy to Bucket-A, which grants access to Role-B
Assigned Role-B to an Amazon EC2 instance
The Bucket Policy on Bucket-A was:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AddPerm",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<account>:role/role-b"
},
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::bucket-a",
"arn:aws:s3:::bucket-a/*"
]
}
]
}
The permissions on Role-B were:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BucketA",
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::bucket-a",
"arn:aws:s3:::bucket-a/*"
]
}
]
}
I was able to successfully use aws s3 ls s3://bucket-a and was able to copy files to Bucket-A.
The main difference I see with your attempt was that your Bucket Policy only granted permissions for:
"Resource": "arn:aws:s3:::BUCKETAAAA/*"
This means "anything within Bucket-A" but does not include Bucket-A itself. The command aws s3 ls s3://BUCKETAAAA operates on the bucket, for which the role has no permissions.
Thus, you should also add permissions for the bucket itself:
"Resource": [
"arn:aws:s3:::BUCKETAAAA",
"arn:aws:s3:::BUCKETAAAA/*"
In JupyterHub, installed in an EC2 instance with an IAM role which allows access to a specific S3 bucket when I try to access a file in that bucket with this code:
s3nRdd = spark.sparkContext.textFile("s3n://bucket/file")
I get this error:
IllegalArgumentException: u'AWS Access Key ID and Secret Access Key
must be specified as the username or password (respectively) of a s3n
URL, or by setting the fs.s3n.awsAccessKeyId or
fs.s3n.awsSecretAccessKey properties (respectively).'
However, when I export the AWS access key id and secret access key in the kernel configuration having the same policy as that role, the read for that file succeeds.
As the best practice is to use IAM roles, why doesn't the EC2 role work in this situation?
--update--
The EC2 IAM role has these 2 policies:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1488892557621",
"Action": "s3:*",
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::<bucket_name>",
"arn:aws:s3:::<bucket_name>/*"
]
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "ec2:*",
"Effect": "Allow",
"Resource": "*"
},
{
"Sid": "Stmt1480684159000",
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": [
"*"
]
}
]
}
Also, I am using hadoop version 2.4.0 which doesn't support s3a protocol and updating is not an option.
S3n doesn't support IAM roles, and 2.4 is a very outdated version anyway. Not as buggy as 2.5 when it comes to s3n, but still less than perfect.
If you want to use IAM roles, you are going to have to switch to S3a, and yes, for you, that does mean upgrading Hadoop. sorry.
You must create a bucket policy to allow access from particular IAM roles. Since S3 doesn't trust the roles, the API just fallback and ask for access key.
Just add soemthing like this in your bucket policy, replace all the custom <> parameter with your own values.
{
"Version": "2012-10-17",
"Id": "EC2IAMaccesss",
"Statement": [{
"Sid": "MyAppIAMRolesAccess",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::<acc_id>:role/<yourIAMroleName>"
]
},
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::<yourbucket>/*",
"arn:aws:s3:::<yourbucket>"
]
}
]
}
(updates)
Make sure you give proper policy to the EC2 IAM Roles, because IAM roles is very powerful, no Policy is attach to it out of the box. You must assign a policy, e.g. for minimal S3 access, add AWSS3ReadOnly policy to the roles.
You may encounter issues of spark problematic interaction with IAM roles. Please check the documentation on spark access through s3n:// schema. Otherwise, use s3a://