I have an application that uploads videos to an S3 bucket, and then creates a custom policy to allow another user (for the Zencoder service) to grab the files, and upload the transcoded files back into the bucket.
Below is the current custom policy I give to the user during transcoding. Basically I give full read permission to the entire bucket, but I only allow the user to PUT files into a specific nested folder.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowUserToListContentsOfBucket",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::MY-BUCKET"
]
},
{
"Sid": "AllowUserToListContentsOfBucketFolders",
"Effect": "Allow",
"Action": [
"s3:ListBucketMultipartUploads",
"s3:GetObjectAcl",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::MY-BUCKET/*"
]
},
{
"Sid": "AllowUserS3ActionsOfSpecificFolder",
"Effect": "Allow",
"Action": [
"s3:PutObjectAcl",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::MY-BUCKET/some/nested/folder/*"
]
}
]
}
This works for the most part, but in the ~1,000 files transferred over by Zencoder, there's usually one or two that fail with a 403 Forbidden error. I'm not sure why, since files were correctly transferred both before and after the error.
Is there any reason Amazon AWS S3 / IAM would send a 403 Access Denied when such a permission is provided?
Related
I'm attempting to set up permissions for a user account on AWS Transfer Service with SFTP protocol. I have a use case where a user should be able to add a file to a directory but not list the files in it.
When I tweak the IAM role to deny 's3:ListBucket' for a specific subdirectory the put operation fails as well. Theoretically s3 does allow to Put object without having the ability to list the prefixes. AWS transfer service however seems to be implicitly using the list bucket operation before put. Has anyone managed to deny listing ability while still being able to upload.
IAM policy :
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:ListBucket",
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::<my-bucket>"
],
"Sid": "AllowListDirectories",
"Condition": {
"StringLike": {
"s3:prefix": [
"data/partner_2/*"
]
}
}
},
{
"Sid": "DenyMkdir",
"Action": [
"s3:PutObject"
],
"Effect": "Deny",
"Resource": "arn:aws:s3:::<my-bucket>/*/"
},
{
"Sid": "DenyListFilesInSubDirectory",
"Action": [
"s3:ListBucket"
],
"Effect": "Deny",
"Resource": "arn:aws:s3:::<my-bucket>",
"Condition": {
"StringLike": {
"s3:prefix": [
"data/partner_2/data/incoming/*"
]
}
}
},
{
"Effect": "AllowReadWirteInSubDirectory",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:PutObjectTagging",
"s3:PutObjectVersionAcl",
"s3:PutObjectVersionTagging"
],
"Resource": "arn:aws:s3:::<my-bucket>/data/partner_2/data/incoming/*"
},
{
"Effect": "AllowOnlyReadInADifferentDirectory",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion"
],
"Resource": "arn:aws:s3:::<my-bucket>/data/partner_2/data/outgoing/*"
}
]
}
The output from SFTP client:
sftp> cd data/incoming
sftp> ls
Couldn't read directory: Permission denied
sftp> put /Users/foo/Downloads/test.log
Uploading /Users/foo/Downloads/test.log to /data/incoming/test.log
remote open("/data/incoming/test.log"): Permission denied
sftp> get test-one.txt
Fetching /data/incoming/test-one.txt to test-one.txt
sftp> exit
Since you will have to allow the upload to your s3 bucket through SFTP, this answer doesn't quite meet your requirements. If the SFTP requirement wasn't there, you may be able to provide pre-signed urls to the client to upload files securely.
I couldn't manage to find an exact solution, however, a workaround could be allowing list+upload permission to a directory in your bucket that is specific to the client/user. Got a helpful video here to share, and corresponding medium article.
Basically, the IAM policy the user is attached to will have the following permissions to a specific folder while you block all public access to your bucket.
{
"Version": "2023-02-16",
"Statement": [
{
"Sid": "AllowListingOfUserFolder",
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::<my-bucket>"
],
"Condition": {
"StringLike": {
"s3:prefix": [
"restricted-folder-user-1/*",
"restricted-folder-user-1"
]
}
}
},
{
"Sid": "HomeDirObjectAccess",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:GetObjectVersion"
],
"Resource": "arn:aws:s3:::<arn of restricted-folder-user-1>*"
}
]
}
There is a snapshot in S3 of a Postgres DB, but the download button is grayed out... if i navigate to each file in each table, i am able to download the .gz.parquet files individually, but that is crazy.
So I installed the aws cli, configured a default user, tried to run aws s3 cp s3://<your-bucket-name>/<your-snapshot-name> <local-path> but always get:
fatal error: An error occurred (404) when calling the HeadObject operation: Key <your-snapshot-name> does not exist
But it does exist, and I can see it in the aws website and see the root folder if i run aws s3 ls.
So i tried aws s3 cp --recursive s3://<your-bucket-name>/<your-snapshot-name> <local-path> and it goes through all the folders, copies them to my computer, but theyre all empty folders, and i get the following error for every folder its going through:
An error occurred (AccessDenied) when calling the GetObject operation: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
The permissions I'm using are a generic (what i thought was) all access to S3:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*",
"s3-object-lambda:*"
],
"Resource": "*"
}
]
}
Plus two from here:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:ListAllMyBuckets"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::snapshots"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::snapshots/*"
]
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:ListAllMyBuckets"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::snapshots"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::snapshots/*"
]
}
]
}
What am I missing here?
The first error you are experiencing is probably because the aws s3 cp command works on objects, not directories. A good way to copy a whole directory (including subdirectories) is to use aws s3 sync.
The second error mentions "customer master key". This is probably referring to a KMS key that was used to encrypt the file when it was created by Amazon RDS. Try giving yourself kms:* permissions (although you probably only need kms:Decrypt) and it should be able to read the file.
We have created a Bucket in AWS S3 and a IAM-User with a specific policy to restrict the access to this bucket as follows:
Bucket: testbucket
Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectAcl",
"s3:GetObjectVersion",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:PutObjectVersionAcl"
],
"Resource": [
"arn:aws:s3:::testbucket/*",
]
}
]
}
Then we've created a IAM-User and assigned this policy to the permissions of the user. We got an Access-Key and a Secret-Access-Key and can successfully upload files to the bucket and also download them with a given, known URL to the resources.
Now we want be able to also List all objects in the bucket.
Therefor i have changed the policy this way:
New Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectAcl",
"s3:GetObjectVersion",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:PutObjectVersionAcl"
],
"Resource": [
"arn:aws:s3:::testbucket/*",
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::testbucket/*"
]
}
]
}
The new item here is s3:ListBucket.
I have changed this policy successfully and can reload the page on AWS to ensure that these changes are still existing.
However, i've already waited for a hour but i am still unable to list the objects of the bucket.
For testing i use the app CyberDuck. It can list the bucket itself after authentication but it still can't list the objects in the bucket.
Do i need to do something else?
s3:ListBucket applies to the bucket, not to the objects within it. Therefore, the second statement must not have the trailing slash and wildcard and so should be:
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::testbucket"
]
}
I created a bucket which host some web small web page and a few docs which should only be read accessible by users which have a certain login in IAM. These users should only have (read) access to this specific bucket and no other bucket. Ideally these users shouldn't even know that there are other buckets out there.
For this I create a "test" user in IAM, added the user to a group and assigned a group policy as below:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowGroupToSeeBucketListAndAlsoAllowGetBucketLocationRequiredForListBucket",
"Action": [
"s3:ListAllMyBuckets",
"s3:GetBucketLocation"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::*"
]
},
{
"Sid": "AllowS3GetActionsInPrivateFolder",
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::my.web.page/*"
]
}
]
}
When I login with the test user and navigate to S3 I can see all my other buckets and when I click on another bucket I get a "Sorry, no permission" error. This kinda works but ideally the user shouldn't even be able to even list any other buckets.
When I go to https://s3.amazonaws.com/my.web.page/index.html I get a AccessDenied XML message. How can should I modify the policy to be able to open a html page in this bucket with a browser.
The user still has write access to the bucket. How can I only grant read access?
Your help is much appreciated.
Use this policy it will work. Where it says example bucket put you bucket name
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowGroupToSeeBucketListAndAlsoAllowGetBucketLocationRequiredForListBucket",
"Action": [
"s3:GetObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::examplebucket/*"
]
},
{
"Sid": "AllowS3GetActionsInPrivateFolder",
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::my.web.page/*"
]
}
]
}
The following AWS Policy is meant to be bound to an IAM group and then added to users. This will grant every user in the group access to their own folder on Amazon S3.
Now the problem is that with this Policy users still get Access denied in their own folder, they can not list the buckets or perform any other operations.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets",
"s3:GetBucketLocation"
],
"Resource": "arn:aws:s3:::*"
},
{
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::bucketname",
"Condition": {
"StringLike": {
"s3:prefix": [
"",
"home/",
"home/${aws:username}/"
]
}
}
},
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::bucketname/home/${aws:username}",
"arn:aws:s3:::bucketname/home/${aws:username}/*"
]
}
]
}
What I eventually would like is that the user is able to put and get files from their own folder, but not see any of the other folders or buckets, but that doesn't seem possible with this policy.
Ideas?
Apparantly it takes up to a few minutes for the policy to apply, policy validates fine now.