I am stuck with provisioning end-user access into a cross account shared bucket, and need help figuring out if there are specific policy requirements for using clients to access the bucket, vs straight CLI.
IAM User Accounts are managed in our "Core" AWS Account.
S3 Bucket is provisioned in our "Dev" AWS Account.
S3 Bucket in Dev account is encrypted with KMS key in Dev Account.
We have configured our Bucket Policy to permit the user access.
We have configured user policies to permit access to the S3 bucket.
We have configured user policies to permit use of the KMS key.
When using the CLI our user account can succesfully access and use the S3 bucket. When attempting to connect with a GUI Client (Win-SCP, CyberDuck, MAC ForkLift) we receive permission denied errors.
BUCKET POLICY
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListObjectsInBucket",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::[DEVACCOUNT#]:role/EC2-ROLE-FOR-APP-ACCESS",
"arn:aws:iam::[COREACCOUNT#]:user/end.user"
]
},
"Action": "s3:List*",
"Resource": [
"arn:aws:s3:::dev-mybucket",
"arn:aws:s3:::dev-mybucket/*"
]
},
{
"Sid": "AllObjectActions",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::[DEVACCOUNT#]:role/EC2-ROLE-FOR-APP-ACCESS",
"arn:aws:iam::[COREACCOUNT#]:user/end.user"
]
},
"Action": [
"s3:GetObject",
"s3:Put*"
],
"Resource": "arn:aws:s3:::dev-mybucket/*"
}
]
}
User Policy - access KMS
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowUseOfDevAPPSKey",
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:Describe*"
],
"Resource": [
"arn:aws:kms:ca-central-1:[DEVACCOUNT#]:key/[redacted-key-number]"
]
},
{
"Sid": "AllowAttachmentOfPersistentResources",
"Effect": "Allow",
"Action": [
"kms:CreateGrant",
"kms:List*",
"kms:RevokeGrant"
],
"Resource": [
"arn:aws:kms:ca-central-1:[DEVACCOUNT#]:key/[redacted-key-number]"
],
"Condition": {
"Bool": {
"kms:GrantIsForAWSResource": true
}
}
}
]
}
User policy - Access S3 Bucket
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowAccessToMyBucket",
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::dev-mybucket/",
"arn:aws:s3:::dev-mybucket/*"
]
}
]
}
From aws s3 commands we can 'ls' content and 'cp' content from local to remote and from remote to local.
When configuring access with the GUI Clients we always receive somewhat generic 'permission denied' or 'access denied' type errors.
The GUI client is probably making a call that is not List*, Put* or GetObject.
For example, it might be calling GetObjectVersion, GetObjectAcl or GetBucketAcl.
Try adding Get* permissions in addition to List*.
You might also be able to look at the events in your AWS CloudTrail trail to see what specific API calls were denied.
For details, see: Specifying Permissions in a Policy - Amazon Simple Storage Service
Access to an S3 bucket via a GUI such as the AWS web console or SFTP clients with s3 functionality(FileZilla, Cyberduck, ForkLift, etc.) requires the s3:ListAllMyBuckets action in a policy attached to that IAM user. This is very unfortunate as the user will now have access to see ALL your bucket names in that account even if they just have read, write, and or List access to just one bucket in that account.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations.html
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html
One other option is to go to the bucket URL directly. The user/role will require access via that bucket's Bucket policy.
https://s3.console.aws.amazon.com/s3/buckets/dev-mybucket
Related
I have created an IAM user in my AWS account. IAM user requires permission to access Amazon data Lifecycle Manager. I had given the following permissions to the IAM user
AmazonEC2FullAccess,
AWSDataLifecycleManagerServiceRole
and AWSDataLifecycleManagerServiceRoleForAMIManagement.
But when I tried to access Amazon Data Lifecycle Manager with this IAM user account, I get this following statement on the lifecycle manager page
It is taking a bit longer than usual to fetch your data.
(The page keepy on loading for a longer period of time)
This message doesn't appear when I tried to access the same page with the same IAM user but this time with Administrator-Access.
Can somebody please let me know what's going wrong here, because I want to grant limited permission for my IAM user to manage my AWS resources.
The policies that you mencioned does not include permissions to access Data Lifecycle Manager.
This is another service that is not included on EC2 (this is why AmazonEC2FullAccess does not give you permissions). Additionally, AWSDataLifecycleManagerServiceRole and AWSDataLifecycleManagerServiceRoleForAMIManagement are managed policies to allow AWS Data Lifecycle Manager itself to take actions on AWS resources. So these policies should not be applied to IAM Users.
You need to create a custom IAM Policy with the proper permissions. In case of read only you can use this:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DataLifecycleManagerRead",
"Effect": "Allow",
"Action": [
"dlm:Get*",
"dlm:List*"
],
"Resource": "*"
}
]
}
UPDATE
To create policies through web console, some additional permissions are required because the web shows more information to help during creation process. So in order to have enough permissions to create policies via web use this (some of these are referenced on documentation but seems to be incomplete):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dlm:*",
"iam:GetRole",
"ec2:DescribeTags",
"iam:ListRoles",
"iam:PassRole",
"iam:CreateRole",
"iam:AttachRolePolicy"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:CreateSnapshot",
"ec2:CreateSnapshots",
"ec2:DeleteSnapshot",
"ec2:DescribeInstances",
"ec2:DescribeVolumes",
"ec2:DescribeSnapshots",
"ec2:EnableFastSnapshotRestores",
"ec2:DescribeFastSnapshotRestores",
"ec2:DisableFastSnapshotRestores",
"ec2:CopySnapshot",
"ec2:ModifySnapshotAttribute",
"ec2:DescribeSnapshotAttribute"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:CreateTags"
],
"Resource": "arn:aws:ec2:*::snapshot/*"
},
{
"Effect": "Allow",
"Action": [
"events:PutRule",
"events:DeleteRule",
"events:DescribeRule",
"events:EnableRule",
"events:DisableRule",
"events:ListTargetsByRule",
"events:PutTargets",
"events:RemoveTargets"
],
"Resource": "arn:aws:events:*:*:rule/AwsDataLifecycleRule.managed-cwe.*"
}
]
}
I have access to one of two AWS environments and I've created a protected S3 bucket in it to upload files to from an account in the one that I do not. The environment and the account that I don't have access to are what a project's CI uses.
environment I have access to: env1
environment I do not have access to: env2
account I do not have access to: user/ci
bucket name: content
S3 bucket policy:
{
"Version": "2008-10-17",
"Id": "PolicyForCloudFrontPrivateContent",
"Statement": [
{
...
},
{
"Sid": "Allow access to bucket from profile in env1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:user/ci"
},
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket*"
],
"Resource": "arn:aws:s3:::content"
},
{
"Sid": "Allow access to bucket items from profile in env1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:user/ci"
},
"Action": [
"s3:Get*",
"s3:PutObject",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::content",
"arn:aws:s3:::content/*"
]
}
]
}
From inside a container that's configured for env1 and user/ci I'm testing with the command
aws s3 sync content/ s3://content/
and I get the error:
fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
I have two questions:
Am I even using the correct aws command to upload the data to the bucket?
Am I missing something from my bucket policy?
For the latter, I've basically followed what a load of examples and answers online have suggested.
To test your policy, I did the following:
Created an IAM User with no policies
Created an Amazon S3 bucket
Attached your Bucket Policy to the bucket, and updated the ARN and bucket name
Tested access to the bucket with:
aws s3 ls s3://bucketname
aws s3 sync folder/ s3://bucketname/folder/
It worked fine.
Therefore, the policy you display appears to be giving all necessary permissions. It is possible that you have something else that is Denying access on the bucket.
The solution was to given the ACL
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::content",
"arn:aws:s3:::content/*"
]
}
]
}
to user/ci in env1.
I am the root user of my account and i created one new user and trying to give access to s3 via s3 bucket policy:
Here is my policy details :-
{ "Id": "Policy1542998309644", "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1542998308012", "Action": [ "s3:ListBucket" ], "Effect": "Allow", "Resource": "arn:aws:s3:::aws-bucket-demo-1", "Principal": { "AWS": [ "arn:aws:iam::213171387512:user/Dave" ] } } ]}
in IAM i have not given any access to the new user. I want to provide him access to s3 via s3 bucket policy. Actually i would like to achieve this : https://aws.amazon.com/premiumsupport/knowledge-center/s3-console-access-certain-bucket/ But not from IAM , I want to use only s3 bucket policy.
Based on the following AWS blog post (the blog shows IAM policy, but it can be adapted to a bucket policy):
How can I grant a user Amazon S3 console access to only a certain bucket?
you can make the following bucket policy:
{
"Id": "Policy1589632516440",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1589632482887",
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::aws-bucket-demo-1",
"Principal": {
"AWS": [
"arn:aws:iam::213171387512:user/Dave"
]
}
},
{
"Sid": "Stmt1589632515136",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::aws-bucket-demo-1/*",
"Principal": {
"AWS": [
"arn:aws:iam::213171387512:user/Dave"
]
}
}
]
}
This will require user to url directly to the bucket:
https://s3.console.aws.amazon.com/s3/buckets/aws-bucket-demo-1/
The reason is that the user does not have permissions to list all buckets available. Thus he/she has to go directly to the one you specify.
Obviously the IAM user needs to have AWS Management Console access enabled when you create him/her in the IAM service. With Programmatic access only, IAM users can't use console and no bucket policy can change that.
You will need to use ListBuckets.
It seems like you want this user to only be able to see your bucket but not access anything in it.
I have an AWS account with read/write permissions as shown below:
I'd like to make it so that an IAM user can download files from an S3 bucket but I'm getting access denied when executing aws s3 sync s3://<bucket_name> . I have tried various things, but not to avail. Some steps that I did:
Created a user called s3-full-access
Executed aws configure in my CLI and entered the generated access key id and secret access key for the above user
Created a bucket policy (shown below) that I'd hoped grants access for my user created in first step.
My bucket has a folder name AffectivaLogs in which files were being added anonymously by various users, and it seems like though the bucket is public, the folder inside it is not and I am not even able to make it public, and it leads to following error.
Following are the public access settings:
Update: I updated the bucket policy as follows, but it doesn't work.
To test the situation, I did the following:
Created an IAM User with no attached policies
Created an Amazon S3 bucket
Turned off S3 block public access settings:
Block new public bucket policies
Block public and cross-account access if bucket has public policies
Added a Bucket Policy granting s3:* access to the contents of the bucket for the IAM User
I then ran aws s3 sync and got Access Denied.
I then modified the policy to also permit access to the bucket itself:
{
"Id": "Policy",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "statement",
"Action": "s3:*",
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::my-bucket/*",
"arn:aws:s3:::my-bucket"
],
"Principal": {
"AWS": [
"arn:aws:iam::123456789012:user/stack-user"
]
}
}
]
}
This worked.
Bottom line: Also add permissions to access the bucket, in addition to the contents of the bucket. (I suspect it is because aws s3 sync requires listing of bucket contents, in addition to accessing the objects themselves.)
If you use KMS encryption enabled on bucket you should also add policy that allows you to decrypt data using KMS key.
You can configure the S3 policy with the required principal
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListBucket",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::accountId:user/*
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::bucket"
},
{
"Sid": "GetObjects",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::accountId:user/*
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::bucket/*"
}
]
}
Or you can create IAM policy and attached it to the role
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListBucket",
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::bucket"
},
{
"Sid": "GetObject",
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": "arn:aws:s3:::bucket/*"
}
]
}
In JupyterHub, installed in an EC2 instance with an IAM role which allows access to a specific S3 bucket when I try to access a file in that bucket with this code:
s3nRdd = spark.sparkContext.textFile("s3n://bucket/file")
I get this error:
IllegalArgumentException: u'AWS Access Key ID and Secret Access Key
must be specified as the username or password (respectively) of a s3n
URL, or by setting the fs.s3n.awsAccessKeyId or
fs.s3n.awsSecretAccessKey properties (respectively).'
However, when I export the AWS access key id and secret access key in the kernel configuration having the same policy as that role, the read for that file succeeds.
As the best practice is to use IAM roles, why doesn't the EC2 role work in this situation?
--update--
The EC2 IAM role has these 2 policies:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1488892557621",
"Action": "s3:*",
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::<bucket_name>",
"arn:aws:s3:::<bucket_name>/*"
]
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "ec2:*",
"Effect": "Allow",
"Resource": "*"
},
{
"Sid": "Stmt1480684159000",
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": [
"*"
]
}
]
}
Also, I am using hadoop version 2.4.0 which doesn't support s3a protocol and updating is not an option.
S3n doesn't support IAM roles, and 2.4 is a very outdated version anyway. Not as buggy as 2.5 when it comes to s3n, but still less than perfect.
If you want to use IAM roles, you are going to have to switch to S3a, and yes, for you, that does mean upgrading Hadoop. sorry.
You must create a bucket policy to allow access from particular IAM roles. Since S3 doesn't trust the roles, the API just fallback and ask for access key.
Just add soemthing like this in your bucket policy, replace all the custom <> parameter with your own values.
{
"Version": "2012-10-17",
"Id": "EC2IAMaccesss",
"Statement": [{
"Sid": "MyAppIAMRolesAccess",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::<acc_id>:role/<yourIAMroleName>"
]
},
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::<yourbucket>/*",
"arn:aws:s3:::<yourbucket>"
]
}
]
}
(updates)
Make sure you give proper policy to the EC2 IAM Roles, because IAM roles is very powerful, no Policy is attach to it out of the box. You must assign a policy, e.g. for minimal S3 access, add AWSS3ReadOnly policy to the roles.
You may encounter issues of spark problematic interaction with IAM roles. Please check the documentation on spark access through s3n:// schema. Otherwise, use s3a://