ECS Failed to create service due to assume role - amazon-web-services

Question

I got the following error when attempting to create an ECS service (Fargate) using Cloud Formation.
Invalid request provided: CreateService error: Unable to assume role and validate the specified targetGroupArn. Please verify that the ECS service role being passed has the proper permissions. (Service: Ecs, Status Code: 400, Request ID: 32dc55bc-3b69-46dd-bf95-f3fff77c2508, Extended Request ID: null)
Things that tried/related:
Updating the role to include even AdministratorAccess (just for troubleshooting).
Allowing several services (ecs, elb, ec2, cloudformation) to assume role (was only ecs-tasks originally).
Create ECS service in web console successfully (same config). (But Cloud Formation doesn't work).
The ECS role has not been updated, the last successful ECS service creation was 21 Nov 2020 (/w Cloud Formation)
The following is the ECS role and Cloud Trail event of the above error. Has anyone faced similar issues or know what is happening?
Edit 1:
ECS template is included, IAM role and the ECS service belongs to different root stack such that it is not possible to use DependsOn attribute. We have CI/CD that ensures the IAM stack is updated before the ECS stack.
ECS Task role used:
EcsTaskRole:
Type: 'AWS::IAM::Role'
Properties:
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AdministratorAccess'
- 'arn:aws:iam::aws:policy/AmazonSQSFullAccess'
- 'arn:aws:iam::aws:policy/AmazonS3FullAccess'
- 'arn:aws:iam::aws:policy/AmazonSNSFullAccess'
- 'arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess'
- 'arn:aws:iam::aws:policy/AmazonRDSFullAccess'
- 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
- 'arn:aws:iam::aws:policy/AmazonSSMReadOnlyAccess'
- 'arn:aws:iam::aws:policy/AWSXrayFullAccess'
- 'arn:aws:iam::aws:policy/AWSBatchFullAccess'
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- ecs-tasks.amazonaws.com
- ecs.amazonaws.com
- cloudformation.amazonaws.com
- elasticloadbalancing.amazonaws.com
- ec2.amazonaws.com
Action:
- 'sts:AssumeRole'
Outputs:
EcsTaskRoleArn:
Description: EcsTaskRoleArn
Value: !GetAtt EcsTaskRole.Arn
Export:
Name: !Sub "${AWS::StackName}-EcsTaskRoleArn"
Event from Cloud Trail: (Masked some info)
{
"eventVersion":"1.08",
"userIdentity":{
"type":"IAMUser",
"principalId":"********",
"arn":"arn:aws:iam::*****:user/****",
"accountId":"*********",
"accessKeyId":"********",
"userName":"********",
"sessionContext":{
"sessionIssuer":{
},
"webIdFederationData":{
},
"attributes":{
"mfaAuthenticated":"false",
"creationDate":"2021-01-01T20:48:02Z"
}
},
"invokedBy":"cloudformation.amazonaws.com"
},
"eventTime":"2021-01-01T20:48:14Z",
"eventSource":"ecs.amazonaws.com",
"eventName":"CreateService",
"awsRegion":"ap-east-1",
"sourceIPAddress":"cloudformation.amazonaws.com",
"userAgent":"cloudformation.amazonaws.com",
"errorCode":"InvalidParameterException",
"errorMessage":"Unable to assume role and validate the specified targetGroupArn. Please verify that the ECS service role being passed has the proper permissions.",
"requestParameters":{
"clientToken":"75e4c412-a82c-b01a-1909-cfdbe788f1f1",
"cluster":"********",
"desiredCount":1,
"enableECSManagedTags":true,
"enableExecuteCommand":false,
"healthCheckGracePeriodSeconds":300,
"launchType":"FARGATE",
"loadBalancers":[
{
"targetGroupArn":"arn:aws:elasticloadbalancing:ap-east-1:********:listener-rule/app/********/e6a62b4cc4d13aaa/098a6759b6062f3f/f374eba8a4fb66e5",
"containerName":"********",
"containerPort":8080
}
],
"networkConfiguration":{
"awsvpcConfiguration":{
"assignPublicIp":"ENABLED",
"securityGroups":[
"sg-025cd908f664b25fe"
],
"subnets":[
"subnet-067502309b0359486",
"subnet-018893d9e397ecac5",
"subnet-0bfb736aefb90f05a"
]
}
},
"propagateTags":"SERVICE",
"serviceName":"********",
"taskDefinition":"arn:aws:ecs:ap-east-1:********:task-definition/********"
},
"responseElements":null,
"requestID":"32dc55bc-3b69-46dd-bf95-f3fff77c2508",
"eventID":"3f872d94-72a7-4ced-96a6-028a6ceeacba",
"readOnly":false,
"eventType":"AwsApiCall",
"managementEvent":true,
"eventCategory":"Management",
"recipientAccountId":"904822583864"
}
Cloud formation template of ECS service
MyServiceLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: my-service-log
RetentionInDays: 365
MyServiceTargetGroup:
Type: 'AWS::ElasticLoadBalancingV2::TargetGroup'
Properties:
HealthCheckPath: /my-service/health
HealthCheckIntervalSeconds: 300
HealthCheckTimeoutSeconds: 10
Name: my-service-target-group
TargetType: ip
Port: 8080
Protocol: HTTP
VpcId: !Ref VpcId
MyServiceListenerRule:
Type: 'AWS::ElasticLoadBalancingV2::ListenerRule'
Properties:
Actions:
- Type: forward
TargetGroupArn: !Ref MyServiceTargetGroup
Conditions:
- Field: path-pattern
Values:
- /my-service/*
ListenerArn: !Ref AppAlbListenerArn
Priority: 164
MyServiceTaskDef:
Type: 'AWS::ECS::TaskDefinition'
Properties:
ContainerDefinitions:
- Name: my-service-container
Image: !Join
- ''
- - !Ref 'AWS::AccountId'
- .dkr.ecr.
- !Ref 'AWS::Region'
- .amazonaws.com/
- 'Fn::ImportValue': !Sub '${RepositoryStackName}-MyServiceECR'
- ':'
- !Ref MyServiceVersion
Essential: true
PortMappings:
- ContainerPort: 8080
Protocol: tcp
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref MyServiceLogGroup
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: my-service
RequiresCompatibilities:
- FARGATE
Cpu: 256
Memory: 512
Family: my-service-taskdef
NetworkMode: awsvpc
ExecutionRoleArn:
'Fn::ImportValue': !Sub '${IamStackName}-EcsTaskRoleArn'
TaskRoleArn:
'Fn::ImportValue': !Sub '${IamStackName}-EcsTaskRoleArn'
Volumes: []
MyServiceECS:
Type: 'AWS::ECS::Service'
Properties:
DesiredCount: 1
Cluster: !Ref EcsCluster
TaskDefinition: !Ref MyServiceTaskDef
LaunchType: FARGATE
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- !Ref SecurityGroupECS
Subnets:
- !Ref DmzSubnet1
- !Ref DmzSubnet2
- !Ref DmzSubnet3
LoadBalancers:
- ContainerName: my-service-container
ContainerPort: '8080'
TargetGroupArn: !Ref MyServiceListenerRule
EnableECSManagedTags: true
PropagateTags: SERVICE
HealthCheckGracePeriodSeconds: 300
DependsOn:
- MyServiceListenerRule

Answer 1

Use the DependsOn attribute to specify the dependency of the AWS::ECS::Service resource on AWS::IAM::Policy.

Answer 2

There are mistakes in your templates. The first apparent one is:
TargetGroupArn: !Ref MyServiceListenerRule
This should be:
TargetGroupArn: !Ref MyServiceTargetGroup
Large chunks of your templates are missing (ALB definition, listener), so can't comment on them.
p.s.
The IAM role is fine, in a sense that it is not the source of the issue. But giving full privileges to a number of services in one role is not a good practice.