Accessing S3 bucket from a dedicated user ( policy failure ? ) - amazon-web-services

I'm trying to create an S3 bucket with a dedicated user for upload/download using terraform.
For some reason the user that is being created is unable to access the bucket:
$ aws iam list-attached-user-policies --user-name csgoserver
{
"AttachedPolicies": [
{
"PolicyName": "AllowUsercsgoserverAccessTocsgofiles",
"PolicyArn": "arn:aws:iam::370179080679:policy/AllowUsercsgoserverAccessTocsgofiles"
}
]
}
$ aws s3 cp bucket.tf s3://csgofiles --profile test
upload failed: .\bucket.tf to s3://csgofiles/bucket.tf An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
$ aws iam get-user-policy --user-name csgoserver --policy-name AllowUsercsgoserverAccessTocsgofiles
An error occurred (NoSuchEntity) when calling the GetUserPolicy operation: The user policy with name AllowUsercsgoserverAccessTocsgofiles cannot be found.
user.tf:
resource "aws_iam_user" "s3user" {
name = var.user_name
force_destroy = true
}
data "aws_iam_policy_document" "default" {
statement {
sid = "AllowUser${var.user_name}AccessTo${var.bucket_name}"
actions = ["s3:*"]
resources = ["arn:aws:s3:::${var.bucket_name}"]
effect = "Allow"
}
}
resource "aws_iam_user_policy" "s3user_policy" {
name = aws_iam_user.s3user.name
user = aws_iam_user.s3user.name
policy = join("", data.aws_iam_policy_document.default.*.json)
}
resource "aws_iam_access_key" "s3user_ak" {
user = aws_iam_user.s3user.name
}
There is one more thing I do not understand. The aws iam get-policy doesn't work on that policy:
$ aws iam list-policies --max-items 2
{
"Policies": [
{
"PolicyName": "AllowUsercsgoserverAccessTocsgofiles",
"PolicyId": "ANPAVMMDEQHTRE4NG3N2E",
"Arn": "arn:aws:iam::370179080679:policy/AllowUsercsgoserverAccessTocsgofiles",
"Path": "/",
"DefaultVersionId": "v1",
"AttachmentCount": 1,
"PermissionsBoundaryUsageCount": 0,
"IsAttachable": true,
"CreateDate": "2021-11-26T22:17:52+00:00",
"UpdateDate": "2021-11-26T22:17:52+00:00"
},
{
"PolicyName": "eks-full-access-policy",
"PolicyId": "ANPAVMMDEQHTR4C65WFT6",
"Arn": "arn:aws:iam::370179080679:policy/eks-full-access-policy",
"Path": "/",
"DefaultVersionId": "v1",
"AttachmentCount": 0,
"PermissionsBoundaryUsageCount": 0,
"IsAttachable": true,
"CreateDate": "2021-03-30T09:57:04+00:00",
"UpdateDate": "2021-03-30T09:57:04+00:00"
}
],
"NextToken": "eyJNYXJrZXIiOiBudWxsLCAiYm90b190cnVuY2F0ZV9hbW91bnQiOiAyfQ=="
}
$ aws iam get-policy --policy-arn arn:aws:iam::370179080679:policy/AllowUsercsgoserverAccessTocsgofile
An error occurred (NoSuchEntity) when calling the GetPolicy operation: Policy arn:aws:iam::370179080679:policy/AllowUsercsgoserverAccessTocsgofile was not found.
$ aws iam get-policy --policy-arn arn:aws:iam::370179080679:policy/eks-full-access-policy
{
"Policy": {
"PolicyName": "eks-full-access-policy",
"PolicyId": "ANPAVMMDEQHTR4C65WFT6",
"Arn": "arn:aws:iam::370179080679:policy/eks-full-access-policy",
"Path": "/",
"DefaultVersionId": "v1",
"AttachmentCount": 0,
"PermissionsBoundaryUsageCount": 0,
"IsAttachable": true,
"CreateDate": "2021-03-30T09:57:04+00:00",
"UpdateDate": "2021-03-30T09:57:04+00:00"
}
}

I don't think your IAM policy is valid.
You could use something similar to the below:
{
"Id": "Policy1638106306386",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1638106302079",
"Action": "s3:*",
"Effect": "Allow",
"Resource": "arn:aws:s3:::examplebucket",
"Principal": {
"AWS": [
"370179080679"
]
}
}
]
}
I would also suggest putting the policy inline in your aws_iam_user_policy resource rather than using a data source.
And looking at the documentation it does seem policy is a required field within the aws_iam_user_policy resource.
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_user_policy
Maybe like the below:
resource "aws_iam_user_policy" "s3user_policy" {
name = aws_iam_user.s3user.name
user = aws_iam_user.s3user.name
policy = jsonencode ({
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1638106302079",
"Action": "s3:*",
"Effect": "Allow",
"Resource": "arn:aws:s3:::examplebucket/*",
}
]
})

I was missing two characters in the Resource key at the end /* ...
data "aws_iam_policy_document" "default" {
statement {
sid = "AllowUser${var.user_name}AccessTo${var.bucket_name}"
actions = ["s3:*"]
resources = ["arn:aws:s3:::${var.bucket_name}/*"]
effect = "Allow"
}
}

Related

Why can't I access my bucket from an assumed role?

I have an S3 bucket with no attached ACLs or policies. It was created by terraform like so:
resource "aws_s3_bucket" "runners_cache" {
bucket = var.runners_cache.bucket
}
I created a role and attached a policy to it; see the following console log for details
$ aws iam get-role --role-name bootstrap-test-bootstrapper
{
"Role": {
{
"Role": {
"Path": "/bootstrap-test/",
"RoleName": "bootstrap-test-bootstrapper",
"RoleId": "#SNIP",
"Arn": "arn:aws:iam::#SNIP:role/bootstrap-test/bootstrap-test-bootstrapper",
... #SNIP
$ aws iam list-attached-role-policies --role-name bootstrap-test-bootstrapper
{
"AttachedPolicies": [
{
"PolicyName": "bootstrap-test-bootstrapper",
"PolicyArn": "arn:aws:iam::#SNIP:policy/bootstrap-test/bootstrap-test-bootstrapper"
},
... #SNIP
$ aws iam get-policy --policy-arn arn:aws:iam::#SNIP:policy/bootstrap-test/bootstrap-test-runner
{
"Policy": {
"PolicyName": "bootstrap-test-runner",
"PolicyId": "#SNIP",
"Arn": "arn:aws:iam::#SNIP:policy/bootstrap-test/bootstrap-test-runner",
"Path": "/bootstrap-test/",
"DefaultVersionId": "v7",
... #SNIP
$ aws iam get-policy-version --policy-arn arn:aws:iam::#SNIP:policy/bootstrap-test/bootstrap-test-runner --version-id v7
{
"PolicyVersion": {
"Document": {
"Statement": [
{
"Action": [
"s3:AbortMultipartUpload",
"s3:CompleteMultipartUpload",
"s3:ListBucket",
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:PutObjectAcl"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::#SNIP-runners-cache/*",
"arn:aws:s3:::#SNIP-cloud-infrastructure-terraform-states/*"
]
},
{
"Action": [
"s3:*"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::*"
]
}
],
"Version": "2012-10-17"
},
"VersionId": "v7",
"IsDefaultVersion": true,
"CreateDate": "2022-08-18T14:16:33+00:00"
}
}
tl;dr this role has an attached policy that allows full access to s3 within the account.
I can successfully assume this role:
$ aws sts assume-role --role-arn arn:aws:iam::#SNIP:role/bootstrap-test/bootstrap-test-bootstrapper --role-session-name test123
{ ... #REDACTED }
$ export AWS_ACCESS_KEY_ID=ASIA2 #REDACTED
$ export AWS_SECRET_ACCESS_KEY=8 #REDACTED
$ export AWS_SESSION_TOKEN=IQoJb #REDACTED
$ aws sts get-caller-identity
{
"UserId": "#SNIP",
"Account": "#SNIP",
"Arn": "arn:aws:sts::#SNIP:assumed-role/bootstrap-test-bootstrapper/test123"
}
However, once I do this, I no longer have access to S3:
$ aws s3 ls #SNIP-runners-cache
An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
$ aws s3 ls
An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied
What am I missing? Is there some default behavior that prevents access to S3? How should I go about debugging these 403 errors?
It is easy to get over-obsessed with the details of the policy and forget about the role itself. In this case the permissions boundary went unnoticed in the CLI, but it is quite easy to see in the web console:
Indeed, #luk2302 was right, the limiting factor was a permissions boundary. After removing it from the role, access to S3 was restored.

AWS Redshift Serverless: `ERROR: Not authorized to get credentials of role`

I've created a serverless Redshift instance, and I'm trying to import a CSV file from an S3 bucket.
I've made an IAM role with full Redshift + Redshift serverless access and S3 Read access, and added this role as a Default Role under the Permissions settings of the Serverless Configuration. Basically, I've tried to do anything that I thought should be necessary according to the documentation.
However, there docs are only targeted at the normal EC2 hosted Redshift for now, and not for the Serverless edition, so there might be something that I've overlooked.
But when I try running a COPY command (generated by the UI), I get this error:
ERROR: Not authorized to get credentials of role arn:aws:iam::0000000000:role/RedshiftFull Detail: ----------------------------------------------- error: Not authorized to get credentials of role arn:aws:iam::00000000:role/RedshiftFull code: 30000 context: query: 18139 location: xen_aws_credentials_mgr.cpp:402 process: padbmaster [pid=8791] ----------------------------------------------- [ErrorId: 1-61dc479b-570a4e96449b228552f2c911]
Here's the command I'm trying to run:
COPY dev."test-schema"."transactions" FROM 's3://bucket-name/something-1_2021-11-01T00_00_00.000Z_2022-01-03.csv' IAM_ROLE 'arn:aws:iam::0000000:role/RedshiftFull' FORMAT AS CSV DELIMITER ',' QUOTE '"' REGION AS 'eu-central-1'
Here's the Role
{
"Role": {
"Path": "/",
"RoleName": "RedshiftFull",
"RoleId": "AROA2PAMxxxxxxx",
"Arn": "arn:aws:iam::000000000:role/RedshiftFull",
"CreateDate": "2022-01-10T13:55:03+00:00",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"redshift.amazonaws.com",
"sagemaker.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
},
"Description": "Allows Redshift clusters to call AWS services on your behalf.",
"MaxSessionDuration": 3600,
"RoleLastUsed": {}
}
}
{
"AttachedPolicies": [
{
"PolicyName": "redshift-serverless",
"PolicyArn": "arn:aws:iam::719432241830:policy/redshift-serverless"
},
{
"PolicyName": "AmazonRedshiftFullAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonRedshiftFullAccess"
},
{
"PolicyName": "AmazonS3ReadOnlyAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
}
]
}
The redshift-serverless policy is here:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "redshift-serverless:*",
"Resource": "*"
}
]
}

"Invalid operation: Not authorized to get credentials of role" trying to load json from S3 to Redshift

I always get the error
Invalid operation: Not authorized to get credentials of role arn:aws:iam::xxxxx:role/default_glue_role
I simply want to load from a json from S3 into a Redshift cluster. It is not clear to me what role I have to attach (to Redshift ?).
I have tried attaching the following IAM policy to Redshift
{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::xxxxx:role/default_glue_role"
}
}
and also tried with "Resource": "*" but I always get same error.
Thanks for help!
I had a long chat with AWS support about this same issues. A few things to check:
Your s3 bucket region is the same as your redshift cluster region
You are not signed in as the root aws user, you need to create a user with the correct permissions and sign in as this user to run your queries
You should add the following permissions to your user and redshift policies:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*",
"redshift:*",
"sqlworkbench:*",
"sts:*",
"secretsmanager:*",
"s3-object-lambda:*",
"ec2:*",
"sns:*",
"cloudwatch:*",
"tag:*",
"redshift-data:*",
"sqlworkbench:*",
"redshift-serverless:*"
],
"Resource": "*"
}
]
}
You should have the following trust relationships in your redshift and user role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"s3.amazonaws.com",
"redshift.amazonaws.com",
"iam.amazonaws.com",
"redshift-serverless.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
The actual set of permissions you need might be less but this is what worked for me. Took me a long time to figure this out! I hope it helps.
It looks like you might also need to add permissions for glue.
The redshift-serverless permission might tell you it's causing an error but you should be able to save it anyway (AWS told me to do this)
For everyone using Terraform:
What fixed for me it was the (4) suggestion from #patrick-ward:
data "aws_iam_policy_document" "dms_assume_role" {
statement {
actions = ["sts:AssumeRole"]
principals {
identifiers = [
"s3.amazonaws.com",
"redshift.amazonaws.com",
"iam.amazonaws.com",
"redshift-serverless.amazonaws.com",
"dms.amazonaws.com"
]
type = "Service"
}
}
}
resource "aws_iam_role" "dms-access-for-endpoint" {
assume_role_policy = data.aws_iam_policy_document.dms_assume_role.json
name = "dms-access-for-endpoint"
}
resource "aws_iam_role_policy_attachment" "dms-access-for-endpoint-AmazonDMSRedshiftS3Role" {
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role"
role = aws_iam_role.dms-access-for-endpoint.name
}
resource "aws_iam_role" "dms-cloudwatch-logs-role" {
assume_role_policy = data.aws_iam_policy_document.dms_assume_role.json
name = "dms-cloudwatch-logs-role"
}
resource "aws_iam_role_policy_attachment" "dms-cloudwatch-logs-role-AmazonDMSCloudWatchLogsRole" {
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonDMSCloudWatchLogsRole"
role = aws_iam_role.dms-cloudwatch-logs-role.name
}
resource "aws_iam_role" "dms-vpc-role" {
assume_role_policy = data.aws_iam_policy_document.dms_assume_role.json
name = "dms-vpc-role"
}
resource "aws_iam_role_policy_attachment" "dms-vpc-role-AmazonDMSVPCManagementRole" {
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonDMSVPCManagementRole"
role = aws_iam_role.dms-vpc-role.name
}

Access denied when calling the AssumeRole operation

I have a cloudformation template that creates a lambda funciton as well as a role for that lambda function. I try assuming the role in the lambda function but keep getting the error :
An error occurred (AccessDenied) when calling the AssumeRole operation: Access denied
Is there a step I'm missing? Not sure why I don't have permission to assume the role. I'm assuming I'm missing some sort of permission if the error I'm getting is access denied as opposed to some execution error.
Cloudformation Snippet :
"LambdaRoleCustomResource": {
"Type": "AWS::IAM::Role",
"Condition": "CreateWebACL",
"DependsOn": "WAFWebACL",
"Properties": {
"RoleName": {
"Fn::Join": ["-", [{
"Ref": "AWS::StackName"
}, "Custom-Resource"]]
},
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Service": ["lambda.amazonaws.com"]
},
"Action": ["sts:AssumeRole"]
}]
},
"Path": "/",
"Policies": [{
"PolicyName": "S3Access",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"s3:CreateBucket",
"s3:GetBucketLocation",
"s3:GetBucketNotification",
"s3:GetObject",
"s3:ListBucket",
"s3:PutBucketNotification"
],
"Resource": {
"Fn::Join": ["", ["arn:aws:s3:::", {
"Ref": "AccessLogBucket"
}]]
}
}]
}
}
Lambda Function Snippet:
sts_client = boto3.client('sts')
sts_credentials = sts_client.assume_role(RoleArn='arn:aws:iam::XXXXXXXXX:role/portal-cloudfront-waf-Custom-Resource', RoleSessionName='custom-resource-cf-session')
sts_credentials = sts_credentials['Credentials']
cf = boto3.client('cloudformation', aws_access_key_id=sts_credentials['AccessKeyId'], aws_secret_access_key=sts_credentials['SecretAccessKey'], aws_session_token=sts_credentials['SessionToken'])
stack_name = event['ResourceProperties']['StackName']
cf_desc = cf.describe_stacks(StackName=stack_name)
global waf
sts_client = boto3.client('sts')
sts_credentials = sts_client.assume_role(RoleArn='arn:aws:iam::XXXXXXXX:role/portal-cloudfront-waf-Custom-Resource', RoleSessionName='custom-resource-waf-session')
sts_credentials = sts_credentials['Credentials']
s3 = boto3.client('waf', aws_access_key_id=sts_credentials['AccessKeyId'], aws_secret_access_key=sts_credentials['SecretAccessKey'], aws_session_token=sts_credentials['SessionToken'])
waf = boto3.client('waf')
Your Lambda function will automatically use the permissions associated with the Role attached to the function. There is no need to create credentials.
So, just use:
cf = boto3.client('cloudformation')
s3 = boto3.client('waf')

AWS put-bucket-notification-configuration for SQS throws "Unable to validate the following destination configurations"

I want to send s3:CreateObject:* events to a SQS Queue. But setting up the notification configuration results in A client error (InvalidArgument) occurred when calling the PutBucketNotificationConfiguration operation: Unable to validate the following destination configurations
This is the how I created the bucket:
aws s3api create-bucket --profile default --bucket my-bucket --create-bucket-configuration LocationConstraint=eu-west-1
This is the how I created the SQS Queue
aws sqs create-queue --profile default --queue-name my-queue --attributes file://attributes.json
with the attributes.json file
{
"DelaySeconds":"0",
"MessageRetentionPeriod":"3600",
"Policy":"{\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":[\"sqs:SendMessage\",\"sqs:ReceiveMessage\"],\"Condition\":{\"ArnLike\": {\"aws:SourceArn\": \"arn:aws:s3:*:*:my-bucket\"}}}]}"
}
And finally the try to set up the notification which throws the error message I listed above:
aws s3api put-bucket-notification-configuration --profile default --bucket my-bucket --notification-configuration file://notification.json`
with the notification.json file
{
"TopicConfigurations": [
],
"QueueConfigurations": [
{
"QueueArn": "arn:aws:sqs:eu-west-1:123456789012:my-queue",
"Events": [
"s3:ObjectCreated:*"
],
"Filter": {
"Key": {
"FilterRules": [
{
"Name": "prefix",
"Value": "my-filter"
}
]
}
}
}
],
"LambdaFunctionConfigurations": [
]
}
I really have no clue where the error could be.
Thanks for any help!
It looks like your SQS policy is not working. Try to add Id to your policy and Resource to your statement. Something like this:
{
"DelaySeconds":"0",
"MessageRetentionPeriod":"3600",
"Policy":"{\"Id\":\"someid\",\"Statement\":[{\"Effect\":\"Allow\",\"Resource\": \"arn:aws:sqs:eu-west-1:123456789012:my-queue\",\"Principal\":\"*\",\"Action\":[\"sqs:SendMessage\",\"sqs:ReceiveMessage\"],\"Condition\":{\"ArnLike\": {\"aws:SourceArn\": \"arn:aws:s3:*:*:my-bucket\"}}}]}"
}
Here is more information:
http://docs.aws.amazon.com/AmazonS3/latest/dev/ways-to-add-notification-config-to-bucket.html#step1-create-sqs-queue-for-notification
Also when calling API from command line, you can use --debug parameter. You would see full error message:
aws --debug s3api ...
I got a script that works. I post it here for whomever else might be puzzling over this :-)
#!/usr/bin/env python
import boto3
import json
bucket_name='spike-bucket-000'
queue_name='spike_queue_000'
region='eu-west-1'
s3 = boto3.client('s3', region)
sqs = boto3.client('sqs', region)
def check_if_bucket_exists(name):
s3.head_bucket(Bucket=bucket_name)
try:
check_if_bucket_exists(bucket_name)
print('Bucket {} exists'.format(bucket_name))
except botocore.exceptions.ClientError:
print('Creating bucket {}'.format(bucket_name))
s3.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={'LocationConstraint': region})
print('Ensuring queue {} exists'.format(queue_name))
response = sqs.create_queue(QueueName=queue_name)
queue_url = response['QueueUrl']
response = sqs.get_queue_attributes(QueueUrl=queue_url, AttributeNames=['QueueArn'])
queue_arn = response['Attributes']['QueueArn']
print('Granting bucket permission to post messages to queue')
queue_policy={
"Version": "2008-10-17",
"Id": "example-ID",
"Statement": [
{
"Sid": "example-statement-ID",
"Effect": "Allow",
"Principal": {
"AWS":"*"
},
"Action": [
"SQS:SendMessage"
],
"Resource": queue_arn,
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:s3:*:*:" + bucket_name
}
}
}
]
}
sqs.set_queue_attributes(QueueUrl=queue_url, Attributes={'Policy': json.dumps(queue_policy)})
print('Configuring bucket to notify object creation to queue')
response = s3.put_bucket_notification_configuration(
Bucket=bucket_name,
NotificationConfiguration={
'QueueConfigurations': [
{
'Id': 'Notify-ObjectCreated-To-Queue',
'QueueArn': queue_arn,
'Events': [
's3:ObjectCreated:*',
]
# ,
# 'Filter': {
# 'Key': {
# 'FilterRules': [
# {
# 'Name': 'prefix'|'suffix',
# 'Value': 'string'
# },
# ]
# }
#}
},
]
}
)
Thanks to Lubo Sach answer, I could get this working using this Policy in SQS:
{
"Version": "2008-10-17",
"Id": "27097a52-cae3-49fe-84ce-0020893e394c",
"Statement": [
{
"Sid": "__owner_statement",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::9970XXXX4660:root"
},
"Action": "SQS:*",
"Resource": "arn:aws:sqs:us-east-1:9970XXXX4660:bucketname"
},
{
"Effect": "Allow",
"Principal": "*",
"Action": [
"sqs:SendMessage",
"sqs:ReceiveMessage"
],
"Resource": "arn:aws:sqs:us-east-1:9970XXXX4660:bucketname",
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:s3:*:*:bucketname"
}
}
}
]
}
there needs to be a corresponding Lambda:Invoke permission; ensure that the Lambda has the permission over the bucket (it breaks if some Invoke:Permissions are given to bucket and keys)