I have been trying to move all the objects in a folder bucketA/product/pic/ up one level within the same bucket bucketA/pic/
I can sync files between local host and the s3 server with
s3cmd sync /script/ s3://bucketA/
as well as put an object:
s3cmd put zip.sh s3://bucketA/
But I'm getting Access Denied error when syncing files within the same bucket:
[root]s3cmd sync s3://bucketA/product/pic s3://bucketA/pic/
WARNING: Empty object name on S3 found, ignoring.
Summary: 441 source files to copy, 0 files at destination to delete
ERROR: S3 error: Access Denied
Is it possible to change the locations of the objects in a folder within the same bucket?
Here's my IAM policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt123456",
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets"
],
"Resource": [
"arn:aws:s3:::*"
]
},
{
"Sid": "Stmt123457",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::bucketA",
"arn:aws:s3:::bucketA/*"
]
}
]
}
Here's my bucket policy which is set to prevent hotlinking:
{
"Version": "2012-10-17",
"Id": "HTTP referrer policy",
"Statement": [
{
"Sid": "Allow in my domains",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::bucketA/*",
"Condition": {
"StringLike": {
"aws:Referer": [
"https://mylocalhostip/*",
"http://mylocalhostip/*"
]
}
}
},
{
"Sid": "Deny access if referer is not my sites",
"Effect": "Deny",
"Principal": {
"AWS": "*"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::bucketA/*",
"Condition": {
"StringNotLike": {
"aws:Referer": [
"http://mylocalhostip/*",
"https://mylocalhostip/*"
]
}
}
}
]
}
These days, it is recommended to use the AWS Command-Line Interface (CLI).
The AWS CLI includes a sync command. See: https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
Related
I'm attempting to set up permissions for a user account on AWS Transfer Service with SFTP protocol. I have a use case where a user should be able to add a file to a directory but not list the files in it.
When I tweak the IAM role to deny 's3:ListBucket' for a specific subdirectory the put operation fails as well. Theoretically s3 does allow to Put object without having the ability to list the prefixes. AWS transfer service however seems to be implicitly using the list bucket operation before put. Has anyone managed to deny listing ability while still being able to upload.
IAM policy :
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:ListBucket",
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::<my-bucket>"
],
"Sid": "AllowListDirectories",
"Condition": {
"StringLike": {
"s3:prefix": [
"data/partner_2/*"
]
}
}
},
{
"Sid": "DenyMkdir",
"Action": [
"s3:PutObject"
],
"Effect": "Deny",
"Resource": "arn:aws:s3:::<my-bucket>/*/"
},
{
"Sid": "DenyListFilesInSubDirectory",
"Action": [
"s3:ListBucket"
],
"Effect": "Deny",
"Resource": "arn:aws:s3:::<my-bucket>",
"Condition": {
"StringLike": {
"s3:prefix": [
"data/partner_2/data/incoming/*"
]
}
}
},
{
"Effect": "AllowReadWirteInSubDirectory",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:PutObjectTagging",
"s3:PutObjectVersionAcl",
"s3:PutObjectVersionTagging"
],
"Resource": "arn:aws:s3:::<my-bucket>/data/partner_2/data/incoming/*"
},
{
"Effect": "AllowOnlyReadInADifferentDirectory",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion"
],
"Resource": "arn:aws:s3:::<my-bucket>/data/partner_2/data/outgoing/*"
}
]
}
The output from SFTP client:
sftp> cd data/incoming
sftp> ls
Couldn't read directory: Permission denied
sftp> put /Users/foo/Downloads/test.log
Uploading /Users/foo/Downloads/test.log to /data/incoming/test.log
remote open("/data/incoming/test.log"): Permission denied
sftp> get test-one.txt
Fetching /data/incoming/test-one.txt to test-one.txt
sftp> exit
Since you will have to allow the upload to your s3 bucket through SFTP, this answer doesn't quite meet your requirements. If the SFTP requirement wasn't there, you may be able to provide pre-signed urls to the client to upload files securely.
I couldn't manage to find an exact solution, however, a workaround could be allowing list+upload permission to a directory in your bucket that is specific to the client/user. Got a helpful video here to share, and corresponding medium article.
Basically, the IAM policy the user is attached to will have the following permissions to a specific folder while you block all public access to your bucket.
{
"Version": "2023-02-16",
"Statement": [
{
"Sid": "AllowListingOfUserFolder",
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::<my-bucket>"
],
"Condition": {
"StringLike": {
"s3:prefix": [
"restricted-folder-user-1/*",
"restricted-folder-user-1"
]
}
}
},
{
"Sid": "HomeDirObjectAccess",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:GetObjectVersion"
],
"Resource": "arn:aws:s3:::<arn of restricted-folder-user-1>*"
}
]
}
I have an S3 bucket called my-bucket... I have several sub-folders inside this bucket, let's call them: sub-folder1, sub-folder2, etc
I want my-bucket-user to have read permission to the bucket and full control permission to all of the sub-folders (so this user cannot write at the root level).
I have tried the following bucket policy, which specifically grants permission on each of the sub folders... but this policy would become too long if I have a lot of sub-folders.
{
"Id": "MyBucketPolicy",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListPermissionOnMyBucket",
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-bucket/*",
"Principal": {
"AWS": [
"arn:aws:iam::773643377756:user/my-bucket-user"
]
}
},
{
"Sid": "FullControlAccessOnSubFolder1",
"Action": "s3:*",
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-bucket/sub-folder1/*",
"Principal": {
"AWS": [
"arn:aws:iam::773643377756:user/my-bucket-user"
]
}
},
{
"Sid": "FullControlAccessOnSubFolder2",
"Action": "s3:*",
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-bucket/sub-folder2/*",
"Principal": {
"AWS": [
"arn:aws:iam::773643377756:user/my-bucket-user"
]
}
}
]
}
Is there a better way of writing this policy?
This policy permits listing the entire contents of the bucket, but only uploading/downloading to a sub-folder (not the root of the bucket):
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:ListBucket",
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-bucket"
},
{
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-bucket/*/*"
}
]
}
Basically, the expression my-bucket/*/* forces the requirement of a / in the Key of the object, meaning it is in a sub-folder.
Here is my policy which grants read/write access still not able to write into S3 bucket
Problem
Still getting below error:
Failed to upload /tmp/test.txt to bucketname/Automation_Result_2019-07-09 04:20:32_.csv: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ConsoleAccess",
"Effect": "Allow",
"Action": [
"s3:GetAccountPublicAccessBlock",
"s3:GetBucketAcl",
"s3:GetBucketLocation",
"s3:GetBucketPolicyStatus",
"s3:GetBucketPublicAccessBlock",
"s3:ListAllMyBuckets"
],
"Resource": "*"
},
{
"Sid": "ListObjectsInBucket",
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": [
"arn:aws:s3:::bucketname"
]
},
{
"Sid": "AllObjectActions",
"Effect": "Allow",
"Action": "s3:*Object",
"Resource": [
"arn:aws:s3:::bucketname/*"
]
}
]
}
Bucket policy
{
"Version": "2012-10-17",
"Id": "MYBUCKETPOLICY",
"Statement": [
{
"Sid": "DenyIncorrectEncryptionHeader",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::bucket-name/*",
"Condition": {
"StringNotEquals": {
"s3:x-amz-server-side-encryption": "aws:kms"
}
}
},
{
"Sid": "DenyUnEncryptedObjectUploads",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::bucket-name/*",
"Condition": {
"Null": {
"s3:x-amz-server-side-encryption": "true"
}
}
}
]
}
Python code (within Lambda function)
Relevant part of code
s3 = boto3.resource('s3', config=Config(signature_version='s3v4'))
target_bucket = 'bucket-name'
target_file = "Output/Automation_Result_"+EST+"_.txt"
s3.meta.client.upload_file('/tmp/test.txt', target_bucket, target_file, ExtraArgs={"ServerSideEncryption": "aws:kms", "SSEKMSKeyId":"XXXXXXX-XXXX-XXXX" })
This is how my bucket public access looks like!
It works fine for me!
I took your policy, renamed the bucket and attached it to a user as their only policy.
I was then able to successfully copy an object to and from the bucket.
If it is not working for you, then either you are not using the credentials that are associated with this policy, or there is another policy that is preventing the access, such as a Deny policy or a scope-limiting policy.
I want to restrict the access to a single folder in S3 bucket.
I have written a IAM role for the same. Somehow I am not upload/sync the files to this folder. Here, bucket is the bucket name and folder is the folder where I want to give access.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowUserToSeeBucketListInTheConsole",
"Action": [
"s3:ListAllMyBuckets",
"s3:GetBucketLocation"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::*"
]
},
{
"Sid": "AllowRootAndHomeListingOfBucket",
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::bucket"
],
"Condition": {
"StringEquals": {
"s3:prefix": [
""
],
"s3:delimiter": [
"/"
]
}
}
},
{
"Sid": "AllowListingOfUserFolder",
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:GetObject",
"s3:GetObjectAcl",
"s3:HeadObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::bucket"
],
"Condition": {
"StringLike": {
"s3:prefix": [
"folder/*"
]
}
}
}
]
}
Please suggest where I am wrong.
This restrictive IAM policy grants only list and upload access to a particular prefix in a particular bucket. It also intends to allow multipart uploads.
References:
https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations.html
https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuAndPermissions.html
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListBucket",
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::mybucket",
"Condition": {
"StringLike": {
"s3:prefix": "my/prefix/is/this/*"
}
}
},
{
"Sid": "UploadObject",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::mybucket/my/prefix/is/this/*",
]
}
]
}
Note that specifying the s3:ListBucket resource compactly as "arn:aws:s3:::mybucket/my/prefix/is/this/*" didn't work.
Since you have requested to suggest, where you are wrong:
1> In AllowListingOfUserFolder, you have used the object as resource but you have used bucket level operations and "s3:prefix" will not work with object level APIs.
Please refer to the sample policies listed here:
http://docs.aws.amazon.com/AmazonS3/latest/dev/example-policies-s3.html#iam-policy-ex1
https://aws.amazon.com/blogs/security/writing-iam-policies-grant-access-to-user-specific-folders-in-an-amazon-s3-bucket/
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListObjectsInBucket",
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": ["arn:aws:s3:::bucket-name"]
},
{
"Sid": "AllObjectActions",
"Effect": "Allow",
"Action": "s3:*Object",
"Resource": ["arn:aws:s3:::bucket-name/*"]
}
]
}
https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_s3_rw-bucket.html
I believe for the scenario you're describing AWS recommends Bucket Policies. AWS IAM should be used to secure AWS resources such as S3 itself, whereas Bucket policies can be used to secure S3 buckets and documents.
Check out this AWS blog post on the subject:
https://aws.amazon.com/blogs/security/iam-policies-and-bucket-policies-and-acls-oh-my-controlling-access-to-s3-resources/
I want to download the whole bucket to a local directory. I tried:
aws s3 sync s3://my-bucket-name . --profile default
I got an authentication error:
download failed: s3://my-bucket-name/thumbnail.jpg to path to
local/thumbnail.jpg A client error (Unknown) occurred when calling the
GetObject operation: Unknown
I believe my IAM is configured correctly as it gives full access to S3 buckets. it works when I try another command, such as:
aws s3 ls
My inline policy for the IAM user is:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": ["arn:aws:s3:::*"]
}
]
}
Did I miss something in this setup?
You could use the following policy if you want to access via cli as well as web console and restrict the bucket to the user and some basic actions on it:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets"
],
"Resource": "arn:aws:s3:::*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::YOURBUCKET"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:ListObjects"
],
"Resource": [
"arn:aws:s3:::YOURBUCKET/*"
]
}
]
}
There is issue in the policy. Change the policy to below.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "arn:aws:s3:::*"
}
]
}
If you want user to access just one bucket then use below policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "arn:aws:s3:::name-of-bucket/*"
}
]
}