I am getting the following error while deploying Managed Workflow using Apache Airflow using Cloudformation. The error doesnt give out much information and hence difficult to debug
2022-09-16 15:25:42 UTC+0530 MwaaEnvironment CREATE_FAILED Resource
handler returned message: "null (Service: Mwaa, Status Code: 403,
Request ID: de70d88f-383a-43bb-91a7-4ebdee31223e)" (RequestToken:
fe034e51-e5e2-73e9-3616-0bd928d64a1f, HandlerErrorCode: AccessDenied)
It shows access denied but not much information about which resource
Snippet of Cloudformation yaml for the mwaa execution policy
#####################################################################################################################
# CREATE MWAA
#####################################################################################################################
MwaaEnvironment:
Type: AWS::MWAA::Environment
DependsOn: MwaaExecutionPolicy
Properties:
Name: !Sub "${AWS::StackName}-MwaaEnvironment"
SourceBucketArn: !GetAtt EnvironmentBucket.Arn
ExecutionRoleArn: !GetAtt MwaaExecutionRole.Arn
DagS3Path: dags
NetworkConfiguration:
SecurityGroupIds:
- !GetAtt SecurityGroup.GroupId
SubnetIds:
- !Ref PrivateSubnet1
- !Ref PrivateSubnet2
WebserverAccessMode: PUBLIC_ONLY
MaxWorkers: !Ref MaxWorkerNodes
LoggingConfiguration:
DagProcessingLogs:
LogLevel: !Ref DagProcessingLogs
Enabled: true
SchedulerLogs:
LogLevel: !Ref SchedulerLogsLevel
Enabled: true
TaskLogs:
LogLevel: !Ref TaskLogsLevel
Enabled: true
WorkerLogs:
LogLevel: !Ref WorkerLogsLevel
Enabled: true
WebserverLogs:
LogLevel: !Ref WebserverLogsLevel
Enabled: true
SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
VpcId: !Ref VPC
GroupDescription: !Sub "Security Group for Amazon MWAA Environment ${AWS::StackName}-MwaaEnvironment"
GroupName: !Sub "airflow-security-group-${AWS::StackName}-MwaaEnvironment"
SecurityGroupIngress:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !Ref SecurityGroup
IpProtocol: "-1"
SourceSecurityGroupId: !Ref SecurityGroup
SecurityGroupEgress:
Type: AWS::EC2::SecurityGroupEgress
Properties:
GroupId: !Ref SecurityGroup
IpProtocol: "-1"
CidrIp: "0.0.0.0/0"
MwaaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- airflow-env.amazonaws.com
- airflow.amazonaws.com
Action:
- "sts:AssumeRole"
Path: "/service-role/"
MwaaExecutionPolicy:
DependsOn: EnvironmentBucket
Type: AWS::IAM::ManagedPolicy
Properties:
Roles:
- !Ref MwaaExecutionRole
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: airflow:PublishMetrics
Resource:
- !Sub "arn:aws:airflow:${AWS::Region}:${AWS::AccountId}:environment/${EnvironmentName}"
- Effect: Deny
Action: s3:ListAllMyBuckets
Resource:
- !Sub "${EnvironmentBucket.Arn}"
- !Sub "${EnvironmentBucket.Arn}/*"
- Effect: Allow
Action:
- "s3:GetObject*"
- "s3:GetBucket*"
- "s3:List*"
Resource:
- !Sub "${EnvironmentBucket.Arn}"
- !Sub "${EnvironmentBucket.Arn}/*"
- Effect: Allow
Action:
- logs:DescribeLogGroups
Resource: "*"
- Effect: Allow
Action:
- logs:CreateLogStream
- logs:CreateLogGroup
- logs:PutLogEvents
- logs:GetLogEvents
- logs:GetLogRecord
- logs:GetLogGroupFields
- logs:GetQueryResults
- logs:DescribeLogGroups
Resource:
- !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:airflow-${AWS::StackName}*"
- Effect: Allow
Action: cloudwatch:PutMetricData
Resource: "*"
- Effect: Allow
Action:
- sqs:ChangeMessageVisibility
- sqs:DeleteMessage
- sqs:GetQueueAttributes
- sqs:GetQueueUrl
- sqs:ReceiveMessage
- sqs:SendMessage
Resource:
- !Sub "arn:aws:sqs:${AWS::Region}:*:airflow-celery-*"
- Effect: Allow
Action:
- kms:Decrypt
- kms:DescribeKey
- "kms:GenerateDataKey*"
- kms:Encrypt
NotResource: !Sub "arn:aws:kms:*:${AWS::AccountId}:key/*"
Condition:
StringLike:
"kms:ViaService":
- !Sub "sqs.${AWS::Region}.amazonaws.com"
- Action:
- cloudwatch:*
- cloudformation:CreateStack
- cloudformation:DescribeStackEvents
- ec2:AuthorizeSecurityGroupIngress
- ec2:AuthorizeSecurityGroupEgress
- ec2:CancelSpotInstanceRequests
- ec2:CreateRoute
- ec2:CreateSecurityGroup
- ec2:CreateTags
- ec2:DeleteRoute
- ec2:DeleteTags
- ec2:DeleteSecurityGroup
- ec2:DescribeAvailabilityZones
- ec2:DescribeAccountAttributes
- ec2:DescribeInstances
- ec2:DescribeKeyPairs
- ec2:DescribeRouteTables
- ec2:DescribeSecurityGroups
- ec2:DescribeSpotInstanceRequests
- ec2:DescribeSpotPriceHistory
- ec2:DescribeSubnets
- ec2:DescribeVpcAttribute
- ec2:DescribeVpcs
- ec2:DescribeRouteTables
- ec2:DescribeNetworkAcls
- ec2:CreateVpcEndpoint
- ec2:ModifyImageAttribute
- ec2:ModifyInstanceAttribute
- ec2:RequestSpotInstances
- ec2:RevokeSecurityGroupEgress
- ec2:RunInstances
- ec2:TerminateInstances
- elasticmapreduce:*
- iam:GetPolicy
- iam:GetPolicyVersion
- iam:ListRoles
- iam:PassRole
- kms:List*
- s3:*
- sdb:*
Effect: Allow
Resource: "*"
- Effect: Allow
Action: iam:CreateServiceLinkedRole
Resource: "*"
Condition:
StringLike:
iam:AWSServiceName:
- elasticmapreduce.amazonaws.com
- elasticmapreduce.amazonaws.com.cn
Related
I am trying to replace a bunch of custom AWS IAM policies with managed versions. Currently, the template has:
Resources:
...
AutoScalingRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Join ['', [!Ref ServiceName, AutoScalingRole]]
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: ecs-tasks.amazonaws.com
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceAutoscaleRole'
ContainerSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: !Join ['', [!Ref ServiceName, ContainerSecurityGroup]]
VpcId: !Ref VPC
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: !Ref ContainerPort
ToPort: !Ref ContainerPort
SourceSecurityGroupId: !Ref LoadBalancerSecurityGroup
LoadBalancerSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription:
!Join ['', [!Ref ServiceName, LoadBalancerSecurityGroup]]
VpcId: !Ref VPC
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: !Ref LoadBalancerPort
ToPort: !Ref LoadBalancerPort
CidrIp: 0.0.0.0/0
- IpProtocol: tcp
FromPort: 80
ToPort: 80
CidrIp: 0.0.0.0/0
...
CubeJsGroup:
Type: AWS::IAM::Group
Properties:
GroupName: staging-cubejs-group
Policies:
- PolicyName: staging-cubejs-s3-policy
PolicyDocument:
Statement:
- Effect: Allow
Action:
- s3:*
- s3-object-lambda:*
Resource: "*"
- PolicyName: staging-cubejs-athena-policy
PolicyDocument:
Statement:
- Effect: Allow
Action:
- athena:*
Resource: "*"
- Effect: Allow
Action:
- glue:CreateDatabase
- glue:DeleteDatabase
- glue:GetDatabase
- glue:GetDatabases
- glue:UpdateDatabase
- glue:CreateTable
- glue:DeleteTable
- glue:BatchDeleteTable
- glue:UpdateTable
- glue:GetTable
- glue:GetTables
- glue:BatchCreatePartition
- glue:CreatePartition
- glue:DeletePartition
- glue:BatchDeletePartition
- glue:UpdatePartition
- glue:GetPartition
- glue:GetPartitions
- glue:BatchGetPartition
Resource: "*"
- Effect: Allow
Action:
- s3:GetBucketLocation
- s3:GetObject
- s3:ListBucket
- s3:ListBucketMultipartUploads
- s3:ListMultipartUploadParts
- s3:AbortMultipartUpload
- s3:CreateBucket
- s3:PutObject
- s3:PutBucketPublicAccessBloc
Resource: "arn:aws:s3:::aws-athena-query-results-*"
- Effect: Allow
Action:
- cloudwatch:PutMetricAlarm
- cloudwatch:DescribeAlarms
- cloudwatch:DeleteAlarms
Resource: "*"
- Effect: Allow
Action:
- lakeformation:GetDataAccess
Resource: "*"
- Effect: Allow
Action:
- sns:ListTopics
- sns:GetTopicAttributes
Resource: "*"
CubeJsUser:
Type: AWS::IAM::User
Properties:
UserName: 'cubejsUser-staging'
Groups:
- !Ref CubeJsGroup
DependsOn:
- CubeJsGroup
CubeJsUserAccessKey:
Type: AWS::IAM::AccessKey
Properties:
Status: Active
UserName: !Ref CubeJsUser
DependsOn:
- CubeJsUser
Outputs:
Endpoint:
Description: Endpoint
Value: !Join ['', ['https://', !Ref DNSRecord]]
CubeJsUserAccessKey:
Description: "CubeJS user stagin access key id"
Value: !Ref CubeJsUserAccessKey
Export:
Name:
Fn::Sub: "${AWS::StackName}-cubejs-access-id"
CubeJsUserSecretAccessKey:
Description: "CubeJS user access key id"
Value: !GetAtt
- CubeJsUserAccessKey
- SecretAccessKey
Export:
Name:
Fn::Sub: "${AWS::StackName}-cubejs-access-secret"
I like how AutoScalingRole is defined using ManagedPolicyArns etc. To me, it seems like S3FullAccess and such could be used to replace at least some custom permissions. Plus, I would like to maybe drop CubeJsGroup altogether. The template defines all of that CubeJS -related things so that CubeJS had access to Athena and to export keys for CubeJS instance to access AWS. Seems like athena:* could be replaced with AmazonAthenaFullAccess managed policy. Right?
How could I re-write this with as few custom policies as possibly, without breaking anything?
I have used the below cloudformation template code. The issue is, if I did not use 'RoleArn' under 'Targets' the script is running and there is no error but role is not attached under targets and snapshot of the ebs volume is also not created. If I include 'RoleArn' under 'Targets' Iam getting the same error as mentioned above. I have created an volume,EBSsnapshot role and Snapshot rule.
The following is the respective code:
EBSVolume:
Type: 'AWS::EC2::Volume'
Properties:
Size: 1
AvailabilityZone: ap-south-1a
EBSSnapshotRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- events.amazonaws.com
- ec2.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: root
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'ec2:CreateSnapshot'
Resource: '*'
EBSSnapshotRule:
Type: 'AWS::Events::Rule'
Properties:
Description: creates a daily snapshot of EBS volume (1 a.m.)
ScheduleExpression: cron(0 1 * * ? *)
State: ENABLED
RoleArn: !GetAtt
- EBSSnapshotRole
- Arn
Name:
Ref: 'AWS::StackName'
Targets:
- Arn:
'Fn::Join':
- ''
- - 'arn:aws:automation:'
- Ref: 'AWS::Region'
- ':'
- Ref: 'AWS::AccountId'
- ':action/'
- EBSCreateSnapshot/EBSCreateSnapshot_
- Ref: 'AWS::StackName'
Input:
'Fn::Join':
- ''
- - '"'
- !Ref EBSVolume
- '"'
RoleArn: !GetAtt
- EBSSnapshotRole
- Arn
Id: EBSVolume
I have a vpc endpoint created with:
ExecuteApiVPCEndpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal: "*"
Action:
- "execute-api:*"
Resource:
- "*"
ServiceName: !Sub com.amazonaws.${AWS::Region}.execute-api
VpcId: !Ref MyVPC
VpcEndpointType: Interface
SubnetIds:
- !Ref privateSubnet01
- !Ref privateSubnet02
SecurityGroupIds:
- !Ref EndpointSG # has to allow traffic from your VPC
PrivateDnsEnabled: true
and a private API created with:
APIDefinition:
Type: 'AWS::Serverless::Api'
Properties:
StageName: {Ref: Stage}
EndpointConfiguration: PRIVATE
Auth:
ResourcePolicy:
CustomStatements:
- Action: ['execute-api:Invoke']
Effect: Allow
Principal: '*'
Resource: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:/*/*/*"
- Action: ['execute-api:Invoke']
Effect: Deny
Principal: '*'
Resource: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:/*/*/*"
Condition:
ForAllValues:StringNotEquals:
aws:sourceVpce:
- !Ref ExecuteApiVPCEndpoint
The stack was deployed with no error, but the vpce in the resource policy of the API doesn't match with the vpce being created. What's going wrong here? Thanks!
I created a serverless resource for elasticsearch with all necessary policies. We are using seed to build and deploy to cloudformation stacks. The policy is created, elasticsearch domain creation is in process and the policy is also attached to the domain. But it takes very long time and even waiting an hour or so, the status is still “CREATE_IN_PROGRESS”. The process doesn't complete.
I have tried without policies, which creates the elasticsearch domain without taking so much time.
Any suggestions?
Update: Serverless Resource
Resources:
ElasticSearchInstance:
Type: AWS::Elasticsearch::Domain
Properties:
ElasticsearchVersion: 7.4
DomainName: ${self:custom.stage}-domainName
EBSOptions:
EBSEnabled: true
VolumeType: gp2
VolumeSize: 10
ElasticsearchClusterConfig:
InstanceType: "t2.small.elasticsearch"
InstanceCount: 1
DedicatedMasterEnabled: false
ZoneAwarenessEnabled: false
CognitoOptions:
Enabled: true
IdentityPoolId:
Ref: CognitoIdentityPool
RoleArn:
Fn::GetAtt: [ElasticSearchLambdaServiceRole, Arn]
UserPoolId:
Ref: CognitoUserPool
ElasticSearchLambdaServiceRole:
Type: AWS::IAM::Role
Properties:
RoleName: ElasticSearchLambdaServiceRole
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- es.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: ElasticSearchAccessPolicy
PolicyDocument:
Version: "2012-10-17"
Statement:
# # Policy same as "AmazonESCognitoAccess"
- Effect: "Allow"
Action:
- "cognito-idp:DescribeUserPool"
- "cognito-idp:DescribeUserPoolClient"
- "cognito-idp:AdminInitiateAuth"
- "cognito-idp:AdminUserGlobalSignOut"
- "cognito-idp:ListUserPoolClients"
- "cognito-identity:DescribeIdentityPool"
- "cognito-identity:SetIdentityPoolRoles"
- "cognito-identity:GetIdentityPoolRoles"
Resource: "*"
- Effect: "Allow"
Action: "iam:PassRole"
Resource: "*"
Condition:
StringLike:
"iam:PassedToService": "cognito-identity.amazonaws.com"
- Effect: Allow
Action: "es:ESHttpGet"
Resource: "*"
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource:
- 'Fn::Join':
- ':'
-
- 'arn:aws:logs'
- Ref: 'AWS::Region'
- Ref: 'AWS::AccountId'
- 'log-group~:/aws/lambda/*:*:*'
- Effect: "Allow"
Action:
- "es:*"
Resource:
- 'Fn::Join':
- ''
-
- 'arn:aws:es:'
- Ref: 'AWS::Region'
- ':'
- Ref: 'AWS::AccountId'
- ':domain/'
- "${self:custom.stage}-domainName"
- '/*'
I have 2 policies each for S3 and Kinesis stream which includes DescribeStream. The S3 policy works well but I am getting this error with KinesisPolicy.
Resources:
S3
KinesisStream
Firehose
Role:
FirehoseRole
Policies:
S3 policy with the following permissions:
- 's3:AbortMultipartUpload'
- 's3:GetBucketLocation'
- 's3:GetObject'
- 's3:ListBucket'
- 's3:ListBucketMultipartUploads'
- 's3:PutObject'
Kinesis Policy with the following permissions:
- 'kinesis:PutRecord'
- 'kinesis:DescribeStreamSummary'
- 'kinesis:PutRecords'
- 'kinesis:GetShardIterator'
- 'kinesis:GetRecords'
- 'kinesis:DescribeStream'
Error:
The role (firehoseRole) is not authorized to perform DescribeStream on MyKinesisStream.
Cloud formation template
Resources:
S3Bucket:
Type: AWS::S3::Bucket
Properties:
VersioningConfiguration:
Status: Enabled
firehoseRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: firehose.amazonaws.com
Action: 'sts:AssumeRole'
Condition:
StringEquals:
'sts:ExternalId': !Ref 'AWS::AccountId'
DeliveryPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: firehose_delivery_policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 's3:AbortMultipartUpload'
- 's3:GetBucketLocation'
- 's3:GetObject'
- 's3:ListBucket'
- 's3:ListBucketMultipartUploads'
- 's3:PutObject'
Resource:
- !Sub 'arn:aws:s3:::${S3Bucket}'
- !Sub 'arn:aws:s3:::${S3Bucket}*'
Roles:
- !Ref firehoseRole
KinesisPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: kinesis_policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'kinesis:PutRecord'
- 'kinesis:DescribeStreamSummary'
- 'kinesis:PutRecords'
- 'kinesis:GetShardIterator'
- 'kinesis:GetRecords'
- 'kinesis:DescribeStream'
Resource:
- !GetAtt MyKinesisStream.Arn
Roles:
- !Ref firehoseRole
MyKinesisStream:
Type: AWS::Kinesis::Stream
Properties:
ShardCount: 1
DeliveryStream:
Type: AWS::KinesisFirehose::DeliveryStream
Properties:
DeliveryStreamType: KinesisStreamAsSource
KinesisStreamSourceConfiguration:
KinesisStreamARN: !GetAtt MyKinesisStream.Arn
RoleARN: !GetAtt firehoseRole.Arn
S3DestinationConfiguration:
BucketARN: !GetAtt S3Bucket.Arn
BufferingHints:
IntervalInSeconds: 60
SizeInMBs: 50
CompressionFormat: UNCOMPRESSED
Prefix: firehose/
RoleARN: !GetAtt firehoseRole.Arn
I was able to resolve the error. I had to add DependsOn To DeliveryStream and include both the policies.