When create an EMR cluster, got this error
The user has the ec2: DescribeSubnets permission in a policy as below. What would be root cause?
{
“Sid”: “AllowOperateClusters”,
“Effect”: “Allow”,
“Action”: [
“elasticmapreduce:RunJobFlow”,
“elasticmapreduce:DescribeCluster”,
“elasticmapreduce:ListClusters”,
“elasticmapreduce:ListBootstrapActions”,
“elasticmapreduce:ListSteps”,
“elasticmapreduce:ListInstanceGroups”,
“elasticmapreduce:OpenEditorInConsole”,
“elasticmapreduce:ListEditors”,
“ec2:DescribeAccountAttributes”,
“ec2:DescribeVpcs”,
“ec2:DescribeAvailabilityZones”,
“ec2:DescribeSubnets”
],
“Resource”: “*”
}
It works after granting the below permissions.
ec2:DescribeNetworkAcls
ec2:DescribeRouteTables
Related
When executing this command,I get this error:
C:\WINDOWS\system32>eksctl create cluster --name eksctl-demo --profile myAdmin2
Error: checking AWS STS access – cannot get role ARN for current session: operation error STS: GetCallerIdentity, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, request send failed, Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: i/o timeout
myAdmin2 IAM users credientials are set up as follows:
Credentials file:
[myAdmin2]
aws_access_key_id = ******************
aws_secret_access_key = ********************
config file:
[profile myAdmin2]
region = us-east-2
output = json
myAdmin2 has access to the console:
C:\WINDOWS\system32>aws iam list-users --profile myAdmin2
{
"Users": [
{
"Path": "/",
"UserName": "myAdmin",
"UserId": "AIDAYYPFV776ELVEJ5ZVQ",
"Arn": "arn:aws:iam::602313981948:user/myAdmin",
"CreateDate": "2022-09-30T19:08:08+00:00"
},
{
"Path": "/",
"UserName": "myAdmin2",
"UserId": "AIDAYYPFV776LEDK2PCCI",
"Arn": "arn:aws:iam::602313981948:user/myAdmin2",
"CreateDate": "2022-09-30T21:39:33+00:00"
}
]
}
I had problems working with myAdmin that's why I created a new IAM user called myAdmin2.
myAdmin2 is granted AdministratorAccess permission:
As shown in this image
aws cli version installed:
C:\WINDOWS\system32>aws --version
aws-cli/2.7.35 Python/3.9.11 Windows/10 exe/AMD64 prompt/off
My Env variables:
C:\WINDOWS\system32>set
AWS_ACCESS_KEY_ID= ***********the same as I have in credentials file
AWS_CONFIG_FILE=~/.aws/config
AWS_DEFAULT_PROFILE=myAdmin2
AWS_DEFAULT_REGION=us-east-2
AWS_PROFILE=myAdmin2
AWS_SECRET_ACCESS_KEY=****************the same as I have in credentials file
AWS_SHARED_CREDENTIALS_FILE=~/.aws/credentials
I think those are all the necessary things I have to mention. If someone can help, please. I can't move on with this error!!
It worked finally! everything was well configured, I just had to reboot my laptop and it resolved the issue!
We have deployed a DMS replication task to replicate our entire Postgres database to Redshift. The tables are getting created with the correct schemas, but the data isn't coming through to Redshift and getting held up in the S3 bucket DMS uses as an intermediary step. This is all deployed via Terraform.
We've configured the IAM roles as described in the replication instance Terraform docs with all three of dms-access-for-endpoint, dms-cloudwatch-logs-role, and dms-vpc-role IAM roles created. The IAM roles are deployed via a different stack to where DMS is deployed from as the roles are used by another, successfully deployed, DMS instance running a different task.
data "aws_iam_policy_document" "dms_assume_role_document" {
statement {
actions = ["sts:AssumeRole"]
principals {
identifiers = [
"s3.amazonaws.com",
"iam.amazonaws.com",
"redshift.amazonaws.com",
"dms.amazonaws.com",
"redshift-serverless.amazonaws.com"
]
type = "Service"
}
}
}
# Database Migration Service requires the below IAM Roles to be created before
# replication instances can be created. See the DMS Documentation for
# additional information: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Security.html#CHAP_Security.APIRole
# * dms-vpc-role
# * dms-cloudwatch-logs-role
# * dms-access-for-endpoint
resource "aws_iam_role" "dms_access_for_endpoint" {
name = "dms-access-for-endpoint"
assume_role_policy = data.aws_iam_policy_document.dms_assume_role_document.json
managed_policy_arns = ["arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role"]
force_detach_policies = true
}
resource "aws_iam_role" "dms_cloudwatch_logs_role" {
name = "dms-cloudwatch-logs-role"
description = "Allow DMS to manage CloudWatch logs."
assume_role_policy = data.aws_iam_policy_document.dms_assume_role_document.json
managed_policy_arns = ["arn:aws:iam::aws:policy/service-role/AmazonDMSCloudWatchLogsRole"]
force_detach_policies = true
}
resource "aws_iam_role" "dms_vpc_role" {
name = "dms-vpc-role"
description = "DMS IAM role for VPC permissions"
assume_role_policy = data.aws_iam_policy_document.dms_assume_role_document.json
managed_policy_arns = ["arn:aws:iam::aws:policy/service-role/AmazonDMSVPCManagementRole"]
force_detach_policies = true
}
However, on runtime, we're seeing the following logs in CloudWatch:
2022-09-01T16:51:38 [SOURCE_UNLOAD ]E: Not retriable error: <AccessDenied> Access Denied [1001705] (anw_retry_strategy.cpp:118)
2022-09-01T16:51:38 [SOURCE_UNLOAD ]E: Failed to list bucket 'dms-sandbox-redshift-intermediate-storage': error code <AccessDenied>: Access Denied [1001713] (s3_dir_actions.cpp:105)
2022-09-01T16:51:38 [SOURCE_UNLOAD ]E: Failed to list bucket 'dms-sandbox-redshift-intermediate-storage' [1001713] (s3_dir_actions.cpp:209)
We also enabled S3 server access logs on the bucket itself to see whether this would give us more information. This is what we're seeing (anonymised):
<id> dms-sandbox-redshift-intermediate-storage [01/Sep/2022:15:43:32 +0000] 10.128.69.80 arn:aws:sts::<account>:assumed-role/dms-access-for-endpoint/dms-session-for-replication-engine <code> REST.GET.BUCKET - "GET /dms-sandbox-redshift-intermediate-storage?delimiter=%2F&max-keys=1000 HTTP/1.1" 403 AccessDenied 243 - 30 - "-" "aws-sdk-cpp/1.8.80/S3/Linux/4.14.276-211.499.amzn2.x86_64 x86_64 GCC/4.9.3" - <code> SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader s3.eu-west-2.amazonaws.com TLSv1.2 -
The above suggests that a service dms-session-for-replication is the service in question that is receiving the AccessDenied responses, but we're unable to pinpoint what this is and how we can fix it.
We attempted to add a bucket policy to the S3 bucket itself but this did not work (this also includes the S3 server access logs bucket):
resource "aws_s3_bucket" "dms_redshift_intermediate" {
# Prefixed with `dms-` as that's what the AmazonDMSRedshiftS3Role policy filters on
bucket = "dms-sandbox-redshift-intermediate-storage"
}
resource "aws_s3_bucket_logging" "log_bucket" {
bucket = aws_s3_bucket.dms_redshift_intermediate.id
target_bucket = aws_s3_bucket.log_bucket.id
target_prefix = "log/"
}
resource "aws_s3_bucket" "log_bucket" {
bucket = "${aws_s3_bucket.dms_redshift_intermediate.id}-logs"
}
resource "aws_s3_bucket_acl" "log_bucket" {
bucket = aws_s3_bucket.log_bucket.id
acl = "log-delivery-write"
}
resource "aws_s3_bucket_policy" "dms_redshift_intermediate_policy" {
bucket = aws_s3_bucket.dms_redshift_intermediate.id
policy = data.aws_iam_policy_document.dms_redshift_intermediate_policy_document.json
}
data "aws_iam_policy_document" "dms_redshift_intermediate_policy_document" {
statement {
actions = [
"s3:*"
]
principals {
identifiers = [
"dms.amazonaws.com",
"redshift.amazonaws.com"
]
type = "Service"
}
resources = [
aws_s3_bucket.dms_redshift_intermediate.arn,
"${aws_s3_bucket.dms_redshift_intermediate.arn}/*"
]
}
}
How do we fix the <AccessDenied> issues that we're seeing on CloudWatch and enable data loading to Redshift? DMS is able to PUT items in the S3 bucket as we're seeing encrypted CSVs appearing in there (the server access logs also confirm this), but DMS is unable to then GET the files back out of it for Redshift. The AccessDenied responses also suggest that it's an IAM role issue not a security group issue but our IAM roles are configured as per the docs so we're confused as to what could be causing this issue.
What we thought was an IAM issue, was actually a security group issue. The COPY command for Redshift was struggling to access S3. By adding a 443 egress rule for HTTPS to the Redshift security group, we were able to pull data through again
resource "aws_security_group_rule" "https_443_egress" {
type = "egress"
description = "Allow HTTP egress from DMS SG"
protocol = "tcp"
to_port = 443
from_port = 443
security_group_id = aws_security_group.redshift.id
cidr_blocks = ["0.0.0.0/0"]
}
So if you're experiencing the same issue as the question, check whether Redshift has access to S3 via HTTPS.
You are right this is an IAM role issue, make sure the role in questions has the following statements added to the policy document,
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource":"arn:aws:s3:::<yourbucketnamehere>/*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource":"arn:aws:s3:::<yourbucketnamehere>"
},
{
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets",
"s3:GetBucketLocation"
],
"Resource": "arn:aws:s3:::*"
}
When I'm trying to create Glue Crawler, I get this error, even though I have full administration access on IAM
{"service":"AWSGlue","statusCode":400,"errorCode":"AccessDeniedException","requestId":"c1a564e7-d012-4e96-946f-a32be287e8ba","errorMessage":"Account 1234567890 is denied access.","type":"AwsServiceError"}
Open IAM
Policy Name: GlueActions (Type :Customer Inline)[
--- "Statement":[
"Resource":
...
"arn:aws:glue:*xxx:catalog"
...
],
"Effect":"Allow"
]
Ensure the above "catalog" is present else create the whole Customer Inline JSON
I am running a terraform script to automate the folder creation in GCP under Organization. I am using a service account to execute the terraform script. While running terraform apply, I get the following error message:
Error: Error creating folder 'prod' in 'organizations/xxxxxxxxxx': googleapi: Error 403: Request had insufficient authentication scopes.
Details:
[
{
"#type": "type.googleapis.com/google.rpc.ErrorInfo",
"domain": "googleapis.com",
"metadata": {
"method": "google.cloud.resourcemanager.v3.Folders.CreateFolder",
"service": "cloudresourcemanager.googleapis.com"
},
"reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT"
}
]
More details:
Reason: insufficientPermissions, Message: Insufficient Permission
on main.tf line 10, in resource "google_folder" "prod":
10: resource "google_folder" "prod" {
enter image description here
I am seeking assistance to know how could I apply CreateFolder permission to my service account in GCP. Screenshot of the error message is also attached as a link in this post.
Thank You.
I'm using this terraform module to create eks cluster: https://github.com/terraform-aws-modules/terraform-aws-eks
Then I create an additional role and added to map_roles input similar to the example in the repo
(my role is to use CloudWatch)
{
rolearn = "arn:aws:iam::66666666666:role/role1"
username = "role1"
groups = ["system:masters"]
}
I can verify that the role is added to the aws-auth config map together with a role created by the module.
I got this error when the app trying to use CloudWatch:
User: arn:aws:sts::xxx:assumed-role/yyy/zzz is not authorized to perform: logs:DescribeLogGroups on resource: arn:aws:logs:xxx:yyy:log-group::log-stream
the User arn in the error message has the yyy part match the role arn created by the module. So I thought I'm using the wrong role? if so how can I choose the correct credential? (I'm using .NETcore, create AmazonCloudWatchLogsClient without specify any credential)
When I manually edit that role and add the log's permission, the app works. Not sure if it's the right way, if so how can I add the permission in terraforming?
I ended up pulling the eks module to local and add more policies to the existing role:
resource "aws_iam_policy" "my_new_policy" {
name_prefix = "eks-worker-my_new_policy-${aws_eks_cluster.this.name}"
description = "EKS worker node my_new_policy policy for cluster ${aws_eks_cluster.this.name}"
policy = data.aws_iam_policy_document.my_new_policy.json
path = var.iam_path
}
data "aws_iam_policy_document" "my_new_policy" {
statement {
sid = "my_new_policy"
effect = "Allow"
actions = [
"logs:DescribeLogGroups"
]
resources = ["*"]
}
}