CodePipeline deployment to Beanstalk fails despite IAM properly set up - amazon-web-services

Question

Context
This was a CodeStar project initially, and then it grew into something bigger. We reused the Beanstalk application to create the stage and prod environments and kept the initially-created dev environment as-is.
We updated the CodePipeline to deploy to our new environments using "Elastic Beanstalk" as the Provider. (While CodeStar had setup a deployment using CloudFormation for the environment it automatically provisioned in the Beanstalk application.)
The problem
The deployment fails due to an error which mentions autoscaling:DescribeAutoScalingGroups as not being authorized to be executed by the CodePipeline's IAM Role.
Here is the whole error message displayed in CodePipeline:
Insufficient permissions
Deployment failed.
The provided role does not have sufficient permissions: User:
arn:aws:sts::xxx:assumed-role/CodeStarWorker-xxx-on-cod-ToolChain/yyy
is not authorized to perform: autoscaling:DescribeAutoScalingGroups
(Service: AmazonAutoScaling; Status Code: 403; Error Code:
AccessDenied; Request ID: 905ee6ef-d75d-4cf8-b5f3-e6b16a5f6477)
Service:AmazonAutoScaling, Message:User:
arn:aws:sts::xxx:assumed-role/CodeStarWorker-xxx-on-cod-ToolChain/yyy
is not authorized to perform: autoscaling:DescribeAutoScalingGroups
Failed to deploy application.
Service:AmazonAutoScaling, Message:User:
arn:aws:sts::xxx:assumed-role/CodeStarWorker-xxx-on-cod-ToolChain/yyy
is not authorized to perform: autoscaling:DescribeAutoScalingGroups
IAM
Here is the CodePipeline Role's content (aka CodeStarWorker-xxx-on-cod-ToolChain):
And here is the associated Permission Boundary (originally generated by CodeStar, and eventually updated by us to try to get this whole thing to work):
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "ssm:GetParameters",
"Resource": "*",
"Condition": {
"StringEquals": {
"ssm:ResourceTag/awscodestar:projectArn": "arn:aws:codestar:yyy:xxx:project/xxx-on-cod"
}
}
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:CreateBucket",
"iam:PassRole",
"secretsmanager:GetSecretValue"
],
"Resource": [
"arn:aws:s3:::aws-codestar-yyy-xxx/xxx-on-cod/ssh/*",
"arn:aws:s3:::elasticbeanstalk-yyy-xxx/*",
"arn:aws:s3:::elasticbeanstalk-yyy-xxx",
"arn:aws:s3:::awscodestar-remote-access-yyy/*",
"arn:aws:s3:::awscodestar-remote-access-signatures-yyy/*",
"arn:aws:iam::xxx:role/CodeStarWorker-xxx-on-cod-CloudFormation",
"arn:aws:secretsmanager:yyy:xxx:secret:xxx"
]
},
{
"Sid": "VisualEditor4",
"Effect": "Allow",
"Action": [
"s3:*",
"codebuild:*",
"ec2:Describe*",
"ec2:*SecurityGroup*",
"iam:PassRole"
],
"Resource": [
"*"
]
},
{
"Sid": "VisualEditor14",
"Effect": "Allow",
"Action": [
"logs:*"
],
"Resource": [
"arn:aws:logs:yyy:xxx:log-group:/aws/elasticbeanstalk/*"
]
},
{
"Sid": "VisualEditor6",
"Effect": "Allow",
"Action": [
"elasticbeanstalk:CreateApplicationVersion",
"elasticbeanstalk:UpdateEnvironment"
],
"Resource": [
"*"
]
},
{
"Sid": "VisualEditor5",
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:SuspendProcesses",
"autoscaling:ResumeProcesses",
"autoscaling:DescribeScalingActivities"
],
"Resource": [
"arn:aws:autoscaling:yyy:xxx:autoScalingGroup:*"
]
},
{
"Sid": "VisualEditor2",
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"sns:Get*",
"sns:Publish",
"logs:DescribeLogGroups",
"cloudtrail:StartLogging",
"lambda:ListFunctions",
"cloudtrail:CreateTrail",
"sns:Subscribe",
"xray:Put*",
"logs:CreateLogGroup",
"logs:PutLogEvents",
"sns:List*"
],
"Resource": "*"
},
{
"Sid": "VisualEditor3",
"Effect": "Allow",
"Action": "*",
"Resource": [
"arn:aws:cloudformation:yyy:xxx:stack/awseb-e-mjdwv9ptcz-stack/2d588c80-5284-11ea-a1d4-068f4db663b8",
"arn:aws:cloudformation:yyy:xxx:stack/awseb-e-mjdwv9ptcz-stack/2d588c80-5284-11ea-a1d4-068f4db663b8/*",
"arn:aws:cloudformation:yyy:xxx:stack/awscodestar-xxx-on-cod-*",
"arn:aws:codebuild:yyy:xxx:project/xxx-on-cod",
"arn:aws:codecommit:yyy:xxx:xxx-on-codecommit",
"arn:aws:codepipeline:yyy:xxx:xxx-on-cod-Pipeline",
"arn:aws:elasticbeanstalk:yyy:xxx:*/xxx-on-cod*",
"arn:aws:s3:::aws-codestar-yyy-xxx-xxx-on-cod-pipe",
"arn:aws:s3:::aws-codestar-yyy-xxx-xxx-on-cod-pipe/*",
"arn:aws:s3:::elasticbeanstalk-yyy-xxx/resources/environments/e-fp3mwptx9q",
"arn:aws:s3:::elasticbeanstalk-yyy-xxx/resources/environments/e-fp3mwptx9q/*",
"arn:aws:s3:::elasticbeanstalk-yyy-xxx/resources/environments/e-mjdwv9ptcz",
"arn:aws:s3:::elasticbeanstalk-yyy-xxx/resources/environments/e-mjdwv9ptcz/*"
]
}
]
}
Pipeline
As you can see, we have two CodeBuild because the first one is the one set up by CodeStar, and the second one is the one that slightly modifies the output artefact so that it is in the right format for a direct upload into Beanstalk.
The succeeded deployment is the one from CodeStar (using CloudFormation Provider), the next one is the failed one (using Beanstalk Provider).
CodeStar CodeBuild (buildspec.yml)
The output artefact is used by the CloudFormation deployment:
version: 0.2
phases:
install:
runtime-versions:
java: openjdk8
commands:
# Upgrade AWS CLI to the latest version
- pip install --upgrade awscli
pre_build:
commands:
- cd $CODEBUILD_SRC_DIR
- mvn clean compile test
build:
commands:
- mvn war:exploded
post_build:
commands:
- cp -r .ebextensions/ target/ROOT/
- aws cloudformation package --template template.yml --s3-bucket $S3_BUCKET --output-template-file template-export.yml
# Do not remove this statement. This command is required for AWS CodeStar projects.
# Update the AWS Partition, AWS Region, account ID and project ID in the project ARN on template-configuration.json file so AWS CloudFormation can tag project resources.
- sed -i.bak 's/\$PARTITION\$/'${PARTITION}'/g;s/\$AWS_REGION\$/'${AWS_REGION}'/g;s/\$ACCOUNT_ID\$/'${ACCOUNT_ID}'/g;s/\$PROJECT_ID\$/'${PROJECT_ID}'/g' template-configuration.json
artifacts:
type: zip
files:
- target/ROOT/**/*
- .ebextensions/**/*
- 'template-export.yml'
- 'template-configuration.json'
Our CodeBuild (buildspec-two.yml)
The output artefact is used by the (failing) Beanstalk deployment:
# Everything up to that point is the very same as the code from above
artifacts:
type: zip
base-directory: 'target/ROOT'
files:
- ./**/*
- .ebextensions/**/*
Conclusion
I've no idea how the deployment could fail since both the Permission Boundary and the base IAM Role mention that autoscaling:DescribeAutoScalingGroups.
Moreover, the deployment to the CodeStar environment is running fine, yet that particular environment which fails the deployment comes from an exact replicate (in terms of configuration).
Any ideas?
(Moreover, the initial dev environment, just as much as the newly-created stage environment, don't even have an AutoScalingGroup associated to them... so I have no idea why the deployment is even trying to do that.)
(And I've looked in S3 to make sure both Artefacts being deployed have the same structure.)

Answer 1

This is a tough one to troubleshoot, but from what I can see there are a couple of potential issues. One is that the 'DescribeAutoScalingGroups' action does not support a resource-level permission, so it must be an asterisk as the resource, and not the resource arn. You could try just removing the:
"Resource": [
"arn:aws:autoscaling:yyy:xxx:autoScalingGroup:*"
]
in the permissions boundary, and replace it with
"Resource": [
"*"
]
and see if that solves the issue.
Second, the 'AWSCodeDeployFullAccess' role does not contain the 'DescribeAutoScalingGroups' action in the policy. You may need to replace and/or add the 'AWSCodeDeployRole' to be able to use that action. That might solve it.
CodeStar projects are pretty locked down when it comes to permissions, so it can get pretty complex expanding the project. Check here:
https://docs.aws.amazon.com/codestar/latest/userguide/add-iam-role.html
and here:
https://docs.aws.amazon.com/codestar/latest/userguide/adh-policy-examples.html