AWS CloudFormation cannot create Stack using CodeDeploy blue/green deployments - amazon-web-services

I am trying to deploy a new stack using CloudFormation with an ECS service using the CodeDeploy launch type to enable blue/green deployments.
In the User Guide for performing blue/green deployments via CloudFormation, they state the following:
To enable CloudFormation to perform blue/green deployments on a Stack, include the following information in its stack template:
At least one of the ECS resources that will trigger a blue/green deployment if replaced during a stack update. Currently, those resources are AWS::ECS::TaskDefinition and AWS::ECS::TaskSet.
When I exclude a AWS::ECS::TaskSet, the Stack creation fails and I receive the following error:
Transform AWS::CodeDeployBlueGreen failed with: Failed to transform template.
ECSAttributes must include TaskSets in AWS::CodeDeploy::BlueGreen Hook
If I add a AWS::ECS::TaskSet, the stack fails to create with the following error instead:
Resource handler returned message:
"Invalid request provided: Amazon ECS does not support task set management on services where deployments
are controlled by AWS CodeDeploy.
(Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException;
Request ID: 61b8c146-3ae9-4bc2-ac5c-08a11e194f06; Proxy: null)"
(RequestToken: 86a8a3a5-fe89-9939-15c2-45b08b28c3f3, HandlerErrorCode: InvalidRequest)
These are the relevant parts of my stack template:
Transform:
- AWS::CodeDeployBlueGreen
Hooks:
CodeDeployBlueGreenHook:
Type: AWS::CodeDeploy::BlueGreen
Properties:
ServiceRole: BlueGreenDeploymentRole
Applications:
- Target:
Type: AWS::ECS::Service
LogicalID: EcsService
ECSAttributes:
TaskDefinitions:
- TaskDefinitionBlue
- TaskDefinitionGreen
TaskSets:
- TaskSetBlue
- TaskSetGreen
TrafficRouting:
ProdTrafficRoute:
Type: AWS::ElasticLoadBalancingV2::Listener
LogicalID: LoadBalancerListener
TargetGroups:
- TargetGroupBlue
- TargetGroupGreen
TrafficRoutingConfig:
Type: TimeBasedLinear
TimeBasedLinear:
StepPercentage: 20
BakeTimeMins: 10
AdditionalOptions:
TerminationWaitTimeInMinutes: 60
Resources:
# IAM Role for blue/green deployments
BlueGreenDeploymentRole:
Type: AWS::IAM::Role
Properties:
RoleName: blue-green-deployment-role
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: codedeploy.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: BlueGreenDeploymentPolicy
PolicyDocument:
Statement:
- Effect: Allow
Action:
- codedeploy:Get*
- codedeploy:CreateCloudFormationDeployment
Resource: '*'
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AWSCodeDeployRoleForECS
#########################
# Load Balancer
#########################
# Application Load Balancer
LoadBalancer:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
IpAddressType: ipv4
Name: service-alb
Scheme: internal
Subnets:
- !Ref SubnetOne
- !Ref SubnetTwo
SecurityGroups:
- !Ref SecurityGroup
# Load Balancer Listener
LoadBalancerListener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
DefaultActions:
- Type: forward
ForwardConfig:
TargetGroups:
- TargetGroupArn: !Ref TargetGroupBlue
Weight: 1
- TargetGroupArn: !Ref TargetGroupGreen
Weight: 1
LoadBalancerArn: !Ref LoadBalancer
Port: 8080
Protocol: HTTP
# Load Balancer Target Groups
TargetGroupBlue:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: !Sub ${ServiceName}-blue
TargetType: ip
VpcId: !Ref Vpc
Port: 8080
Protocol: HTTP
HealthCheckPort: 8080
HealthCheckPath: /actuator/health
TargetGroupGreen:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: !Sub ${ServiceName}-green
TargetType: ip
VpcId: !Ref Vpc
Port: 8080
Protocol: HTTP
HealthCheckPort: 8080
HealthCheckPath: /actuator/health
#########################
# ECS
#########################
# ECS Cluster
Cluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: !Ref ServiceName
# ECS Service
EcsService:
Type: AWS::ECS::Service
DependsOn: LoadBalancerListener
Properties:
ServiceName: !Ref ServiceName
Cluster: !Ref Cluster
TaskDefinition: !Ref TaskDefinitionBlue
DeploymentController:
Type: CODE_DEPLOY
DesiredCount: 0
LaunchType: FARGATE
LoadBalancers:
- ContainerName: !Sub ${ServiceName}-container
ContainerPort: 8080
TargetGroupArn: !Ref TargetGroupBlue
- ContainerName: !Sub ${ServiceName}-container
ContainerPort: 8080
TargetGroupArn: !Ref TargetGroupGreen
NetworkConfiguration:
AwsvpcConfiguration:
Subnets:
- !Ref SubnetOne
- !Ref SubnetTwo
SecurityGroups:
- !Ref SecurityGroup
SchedulingStrategy: REPLICA
# Task Definitions
TaskDefinitionBlue:
Type: AWS::ECS::TaskDefinition
Properties:
ContainerDefinitions:
- Name: !Sub ${ServiceName}-container
Image: !Sub ${ImageRepository.RepositoryUri}:latest
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref LogGroup
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: payments
PortMappings:
- ContainerPort: 8080
Cpu: 256
Memory: 512
NetworkMode: awsvpc
Family: !Sub ${ServiceName}
ExecutionRoleArn: !Ref TaskExecutionRole
RequiresCompatibilities:
- FARGATE
TaskDefinitionGreen:
Type: AWS::ECS::TaskDefinition
Properties:
ContainerDefinitions:
- Name: !Sub ${ServiceName}-container
Image: !Sub ${ImageRepository.RepositoryUri}:latest
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref LogGroup
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: payments
PortMappings:
- ContainerPort: 8080
Cpu: 256
Memory: 512
NetworkMode: awsvpc
Family: !Sub ${ServiceName}
ExecutionRoleArn: !Ref TaskExecutionRole
RequiresCompatibilities:
- FARGATE
# Image Repository
ImageRepository:
Type: AWS::ECR::Repository
Properties:
RepositoryName: !Sub ${ServiceName}
LifecyclePolicy:
LifecyclePolicyText: |
{
"rules": [
{
"rulePriority": 1,
"description": "Maintain at most 25 images",
"selection": {
"tagStatus": "untagged",
"countType": "imageCountMoreThan",
"countNumber": 25
},
"action": {
"type": "expire"
}
}
]
}
TaskSetBlue:
Type: AWS::ECS::TaskSet
Properties:
Cluster: !Ref Cluster
LaunchType: FARGATE
Service: !Ref EcsService
TaskDefinition: !Ref TaskDefinitionBlue
NetworkConfiguration:
AwsVpcConfiguration:
Subnets:
- !Ref SubnetOne
- !Ref SubnetTwo
SecurityGroups:
- !Ref SecurityGroup
LoadBalancers:
- ContainerName: !Sub ${ServiceName}-container
ContainerPort: 8080
TargetGroupArn: !Ref TargetGroupBlue
How can I update my template to allow for the blue/green deployment strategy via CodeDeploy?

G/B deployment using CFN is EXTERNAL, not CODE_DEPLOY. There could be many other issues with your template, but your current error relates to using wrong DeploymentController. Please study AWS docs and example:
Perform ECS blue/green deployments through CodeDeploy using AWS CloudFormation

Exactly, AWS doesn't support Blue Green deployment on EC2/ASG...!
The easiest way that I just figure out is to create the deployment group with In-Place method && Then change the deployment group configuration manually from In-place to Blue Green
Or you can use the lambda function, but is a litte bit hard to customize it
Wish this will help you

Related

Deploy CloudFormation Template resulting in error

I want to test the deploy of an ECS stack using AWS CLI from a GitLab pipeline.
My test project's core is a variation of the Docker Compose Flask app.
The file app.py:
import time
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello():
return 'Hello World!'
with its requirements.txt:
flask
The Dockerfile is:
FROM python:3.7-alpine
WORKDIR /code
ENV FLASK_APP=app.py
ENV FLASK_RUN_HOST=0.0.0.0
RUN apk add --no-cache gcc musl-dev linux-headers
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
EXPOSE 5000
COPY . .
CMD ["flask", "run"]
and the Docker Compose file is:
version: "3.9"
services:
web:
image: registry.gitlab.com/<MYNAME>/<MYPROJECT>
x-aws-pull_credentials: "<CREDENTIALS>"
ports:
- "5000:5000"
I generate a CloudFormationTemplate.yml file using an ECS Docker Context (using an ecs Docker Context myecscontext, not the default one) and the command
docker compose convert > CloudFormationTemplate.yml
When I try to deploy on AWS from my local workstation (Win10):
aws cloudformation deploy --template-file CloudFormationTemplate.yml --stack-name test-stack
I get the error
unacceptable character #x0000: special characters are not allowed
in "<unicode string>", position 3
What's wrong? Thanks.
===========
ADDED
Here the CloudFormationTemplate.yml:
AWSTemplateFormatVersion: 2010-09-09
Resources:
CloudMap:
Properties:
Description: Service Map for Docker Compose project cloudformation
Name: cloudformation.local
Vpc: vpc-XXXXXXXX
Type: AWS::ServiceDiscovery::PrivateDnsNamespace
Cluster:
Properties:
ClusterName: cloudformation
Tags:
- Key: com.docker.compose.project
Value: cloudformation
Type: AWS::ECS::Cluster
Default5000Ingress:
Properties:
CidrIp: 0.0.0.0/0
Description: web:5000/tcp on default network
FromPort: 5000
GroupId:
Ref: DefaultNetwork
IpProtocol: TCP
ToPort: 5000
Type: AWS::EC2::SecurityGroupIngress
DefaultNetwork:
Properties:
GroupDescription: cloudformation Security Group for default network
Tags:
- Key: com.docker.compose.project
Value: cloudformation
- Key: com.docker.compose.network
Value: default
VpcId: vpc-XXXXXXXX
Type: AWS::EC2::SecurityGroup
DefaultNetworkIngress:
Properties:
Description: Allow communication within network default
GroupId:
Ref: DefaultNetwork
IpProtocol: "-1"
SourceSecurityGroupId:
Ref: DefaultNetwork
Type: AWS::EC2::SecurityGroupIngress
LoadBalancer:
Properties:
LoadBalancerAttributes:
- Key: load_balancing.cross_zone.enabled
Value: "true"
Scheme: internet-facing
Subnets:
- subnet-XXXXXXXX
- subnet-XXXXXXXX
- subnet-XXXXXXXX
Tags:
- Key: com.docker.compose.project
Value: cloudformation
Type: network
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
LogGroup:
Properties:
LogGroupName: /docker-compose/cloudformation
Type: AWS::Logs::LogGroup
WebService:
DependsOn:
- WebTCP5000Listener
Properties:
Cluster:
Fn::GetAtt:
- Cluster
- Arn
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 100
DeploymentController:
Type: ECS
DesiredCount: 1
LaunchType: FARGATE
LoadBalancers:
- ContainerName: web
ContainerPort: 5000
TargetGroupArn:
Ref: WebTCP5000TargetGroup
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- Ref: DefaultNetwork
Subnets:
- subnet-XXXXXXXX
- subnet-XXXXXXXX
- subnet-XXXXXXXX
PlatformVersion: 1.4.0
PropagateTags: SERVICE
SchedulingStrategy: REPLICA
ServiceRegistries:
- RegistryArn:
Fn::GetAtt:
- WebServiceDiscoveryEntry
- Arn
Tags:
- Key: com.docker.compose.project
Value: cloudformation
- Key: com.docker.compose.service
Value: web
TaskDefinition:
Ref: WebTaskDefinition
Type: AWS::ECS::Service
WebServiceDiscoveryEntry:
Properties:
Description: '"web" service discovery entry in Cloud Map'
DnsConfig:
DnsRecords:
- TTL: 60
Type: A
RoutingPolicy: MULTIVALUE
HealthCheckCustomConfig:
FailureThreshold: 1
Name: web
NamespaceId:
Ref: CloudMap
Type: AWS::ServiceDiscovery::Service
WebTCP5000Listener:
Properties:
DefaultActions:
- ForwardConfig:
TargetGroups:
- TargetGroupArn:
Ref: WebTCP5000TargetGroup
Type: forward
LoadBalancerArn:
Ref: LoadBalancer
Port: 5000
Protocol: TCP
Type: AWS::ElasticLoadBalancingV2::Listener
WebTCP5000TargetGroup:
Properties:
Port: 5000
Protocol: TCP
Tags:
- Key: com.docker.compose.project
Value: cloudformation
TargetType: ip
VpcId: vpc-XXXXXXXX
Type: AWS::ElasticLoadBalancingV2::TargetGroup
WebTaskDefinition:
Properties:
ContainerDefinitions:
- Command:
- XXXXXXXX.compute.internal
- cloudformation.local
Essential: false
Image: docker/ecs-searchdomain-sidecar:1.0
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group:
Ref: LogGroup
awslogs-region:
Ref: AWS::Region
awslogs-stream-prefix: cloudformation
Name: Web_ResolvConf_InitContainer
- DependsOn:
- Condition: SUCCESS
ContainerName: Web_ResolvConf_InitContainer
Essential: true
Image: registry.gitlab.com/MYUSER/cloudformation
LinuxParameters: {}
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group:
Ref: LogGroup
awslogs-region:
Ref: AWS::Region
awslogs-stream-prefix: cloudformation
Name: web
PortMappings:
- ContainerPort: 5000
HostPort: 5000
Protocol: tcp
RepositoryCredentials:
CredentialsParameter: arn:aws:secretsmanager:XXXXXXXXXXXXXXXXXXXXXXXX
Cpu: "256"
ExecutionRoleArn:
Ref: WebTaskExecutionRole
Family: cloudformation-web
Memory: "512"
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
Type: AWS::ECS::TaskDefinition
WebTaskExecutionRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action:
- sts:AssumeRole
Condition: {}
Effect: Allow
Principal:
Service: ecs-tasks.amazonaws.com
Version: 2012-10-17
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
- arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
Policies:
- PolicyDocument:
Statement:
- Action:
- secretsmanager:GetSecretValue
- ssm:GetParameters
- kms:Decrypt
Condition: {}
Effect: Allow
Principal: {}
Resource:
- arn:aws:secretsmanager:XXXXXXXXXXXXXXXXXXXXXXXX
PolicyName: webGrantAccessToSecrets
Tags:
- Key: com.docker.compose.project
Value: cloudformation
- Key: com.docker.compose.service
Value: web
Type: AWS::IAM::Role
docker compose convert does not create valid CloudFormation (CFN) template in default context. Before you attempt to generate it, you have to create ECS context:
docker context create ecs myecscontext
Then you have to switch from default context to myecscontext:
docker context use myecscontext
Use docker context ls to confirm that you are in the correct context (i.e., myecscontext). Then you can use your convert command
docker compose convert
to generate actual CFN template.

ECS Failed to create service due to assume role

I got the following error when attempting to create an ECS service (Fargate) using Cloud Formation.
Invalid request provided: CreateService error: Unable to assume role and validate the specified targetGroupArn. Please verify that the ECS service role being passed has the proper permissions. (Service: Ecs, Status Code: 400, Request ID: 32dc55bc-3b69-46dd-bf95-f3fff77c2508, Extended Request ID: null)
Things that tried/related:
Updating the role to include even AdministratorAccess (just for troubleshooting).
Allowing several services (ecs, elb, ec2, cloudformation) to assume role (was only ecs-tasks originally).
Create ECS service in web console successfully (same config). (But Cloud Formation doesn't work).
The ECS role has not been updated, the last successful ECS service creation was 21 Nov 2020 (/w Cloud Formation)
The following is the ECS role and Cloud Trail event of the above error. Has anyone faced similar issues or know what is happening?
Edit 1:
ECS template is included, IAM role and the ECS service belongs to different root stack such that it is not possible to use DependsOn attribute. We have CI/CD that ensures the IAM stack is updated before the ECS stack.
ECS Task role used:
EcsTaskRole:
Type: 'AWS::IAM::Role'
Properties:
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AdministratorAccess'
- 'arn:aws:iam::aws:policy/AmazonSQSFullAccess'
- 'arn:aws:iam::aws:policy/AmazonS3FullAccess'
- 'arn:aws:iam::aws:policy/AmazonSNSFullAccess'
- 'arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess'
- 'arn:aws:iam::aws:policy/AmazonRDSFullAccess'
- 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
- 'arn:aws:iam::aws:policy/AmazonSSMReadOnlyAccess'
- 'arn:aws:iam::aws:policy/AWSXrayFullAccess'
- 'arn:aws:iam::aws:policy/AWSBatchFullAccess'
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- ecs-tasks.amazonaws.com
- ecs.amazonaws.com
- cloudformation.amazonaws.com
- elasticloadbalancing.amazonaws.com
- ec2.amazonaws.com
Action:
- 'sts:AssumeRole'
Outputs:
EcsTaskRoleArn:
Description: EcsTaskRoleArn
Value: !GetAtt EcsTaskRole.Arn
Export:
Name: !Sub "${AWS::StackName}-EcsTaskRoleArn"
Event from Cloud Trail: (Masked some info)
{
"eventVersion":"1.08",
"userIdentity":{
"type":"IAMUser",
"principalId":"********",
"arn":"arn:aws:iam::*****:user/****",
"accountId":"*********",
"accessKeyId":"********",
"userName":"********",
"sessionContext":{
"sessionIssuer":{
},
"webIdFederationData":{
},
"attributes":{
"mfaAuthenticated":"false",
"creationDate":"2021-01-01T20:48:02Z"
}
},
"invokedBy":"cloudformation.amazonaws.com"
},
"eventTime":"2021-01-01T20:48:14Z",
"eventSource":"ecs.amazonaws.com",
"eventName":"CreateService",
"awsRegion":"ap-east-1",
"sourceIPAddress":"cloudformation.amazonaws.com",
"userAgent":"cloudformation.amazonaws.com",
"errorCode":"InvalidParameterException",
"errorMessage":"Unable to assume role and validate the specified targetGroupArn. Please verify that the ECS service role being passed has the proper permissions.",
"requestParameters":{
"clientToken":"75e4c412-a82c-b01a-1909-cfdbe788f1f1",
"cluster":"********",
"desiredCount":1,
"enableECSManagedTags":true,
"enableExecuteCommand":false,
"healthCheckGracePeriodSeconds":300,
"launchType":"FARGATE",
"loadBalancers":[
{
"targetGroupArn":"arn:aws:elasticloadbalancing:ap-east-1:********:listener-rule/app/********/e6a62b4cc4d13aaa/098a6759b6062f3f/f374eba8a4fb66e5",
"containerName":"********",
"containerPort":8080
}
],
"networkConfiguration":{
"awsvpcConfiguration":{
"assignPublicIp":"ENABLED",
"securityGroups":[
"sg-025cd908f664b25fe"
],
"subnets":[
"subnet-067502309b0359486",
"subnet-018893d9e397ecac5",
"subnet-0bfb736aefb90f05a"
]
}
},
"propagateTags":"SERVICE",
"serviceName":"********",
"taskDefinition":"arn:aws:ecs:ap-east-1:********:task-definition/********"
},
"responseElements":null,
"requestID":"32dc55bc-3b69-46dd-bf95-f3fff77c2508",
"eventID":"3f872d94-72a7-4ced-96a6-028a6ceeacba",
"readOnly":false,
"eventType":"AwsApiCall",
"managementEvent":true,
"eventCategory":"Management",
"recipientAccountId":"904822583864"
}
Cloud formation template of ECS service
MyServiceLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: my-service-log
RetentionInDays: 365
MyServiceTargetGroup:
Type: 'AWS::ElasticLoadBalancingV2::TargetGroup'
Properties:
HealthCheckPath: /my-service/health
HealthCheckIntervalSeconds: 300
HealthCheckTimeoutSeconds: 10
Name: my-service-target-group
TargetType: ip
Port: 8080
Protocol: HTTP
VpcId: !Ref VpcId
MyServiceListenerRule:
Type: 'AWS::ElasticLoadBalancingV2::ListenerRule'
Properties:
Actions:
- Type: forward
TargetGroupArn: !Ref MyServiceTargetGroup
Conditions:
- Field: path-pattern
Values:
- /my-service/*
ListenerArn: !Ref AppAlbListenerArn
Priority: 164
MyServiceTaskDef:
Type: 'AWS::ECS::TaskDefinition'
Properties:
ContainerDefinitions:
- Name: my-service-container
Image: !Join
- ''
- - !Ref 'AWS::AccountId'
- .dkr.ecr.
- !Ref 'AWS::Region'
- .amazonaws.com/
- 'Fn::ImportValue': !Sub '${RepositoryStackName}-MyServiceECR'
- ':'
- !Ref MyServiceVersion
Essential: true
PortMappings:
- ContainerPort: 8080
Protocol: tcp
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref MyServiceLogGroup
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: my-service
RequiresCompatibilities:
- FARGATE
Cpu: 256
Memory: 512
Family: my-service-taskdef
NetworkMode: awsvpc
ExecutionRoleArn:
'Fn::ImportValue': !Sub '${IamStackName}-EcsTaskRoleArn'
TaskRoleArn:
'Fn::ImportValue': !Sub '${IamStackName}-EcsTaskRoleArn'
Volumes: []
MyServiceECS:
Type: 'AWS::ECS::Service'
Properties:
DesiredCount: 1
Cluster: !Ref EcsCluster
TaskDefinition: !Ref MyServiceTaskDef
LaunchType: FARGATE
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- !Ref SecurityGroupECS
Subnets:
- !Ref DmzSubnet1
- !Ref DmzSubnet2
- !Ref DmzSubnet3
LoadBalancers:
- ContainerName: my-service-container
ContainerPort: '8080'
TargetGroupArn: !Ref MyServiceListenerRule
EnableECSManagedTags: true
PropagateTags: SERVICE
HealthCheckGracePeriodSeconds: 300
DependsOn:
- MyServiceListenerRule
Use the DependsOn attribute to specify the dependency of the AWS::ECS::Service resource on AWS::IAM::Policy.
There are mistakes in your templates. The first apparent one is:
TargetGroupArn: !Ref MyServiceListenerRule
This should be:
TargetGroupArn: !Ref MyServiceTargetGroup
Large chunks of your templates are missing (ALB definition, listener), so can't comment on them.
p.s.
The IAM role is fine, in a sense that it is not the source of the issue. But giving full privileges to a number of services in one role is not a good practice.

AWS ELB Listener creation fails with Validation exception

My CloudFormation stack creation fails with a very generic error and I can't seem to figure out why.
I'm creating a single-container ECS service task with ALB.
Here's my stack template:
AWSTemplateFormatVersion: '2010-09-09'
Description: Services Containers
Parameters:
VpcId:
Type: String
Default: vpc-4796bd23
SubnetId:
Type: String
Default: subnet-f4701ff8
ELBSecondarySubnetId:
Type: String
Default: subnet-8a453cef
ECSCluster:
Type: String
Default: dev-ecs
EcsSecurityGroup:
Type: String
Default: sg-74cb7b0c
Color:
Type: String
AllowedValues: ['blue', 'green']
Description: The deployment color
Default: 'blue'
BuildVersion:
Type: String
Description: The build version to deploy
ComPublic:
Type: String
Description: Hosted Zone ID
Default: Z00669325SSURKTK4ZPA
MQPort:
Type: Number
Description: MQ Connectivity port
Default: 5672
Resources:
ApiLogsGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Join ['-', [ path-services-api, !Ref Color ]]
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
ContainerDefinitions:
- Essential: True
Image: ***.dkr.ecr.us-east-1.amazonaws.com/path-services/path-services-api
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref ApiLogsGroup
awslogs-region: us-east-1
awslogs-stream-prefix: !Ref BuildVersion
Name: path-services-api
PortMappings:
- ContainerPort: !Ref MQPort
Protocol: tcp
ExecutionRoleArn: arn:aws:iam::***:role/ecs-task-execution-role
Family: path-services-api
NetworkMode: awsvpc
Cpu: 4096
Memory: 8192 # max
RequiresCompatibilities:
- FARGATE
TaskRoleArn: !Ref ServiceTaskRole
ServiceTaskRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: ['ecs-tasks.amazonaws.com', 'ecs.amazonaws.com']
Action: ['sts:AssumeRole']
Path: /
Policies:
- PolicyName: !Join ['-', [path-services, !Ref Color, read-secrets]]
PolicyDocument:
Statement:
- Effect: Allow
Action:
- 'secretsmanager:ListSecrets'
- 'secretsmanager:DescribeSecret'
- 'secretsmanager:GetRandomPassword'
- 'secretsmanager:GetResourcePolicy'
- 'secretsmanager:GetSecretValue'
- 'secretsmanager:ListSecretVersionIds'
Resource: [ 'arn:aws:secretsmanager:us-east-1:***:secret:prod/path-services*' ]
PathService:
Type: AWS::ECS::Service
DependsOn: LoadBalancerListener
Properties:
Cluster: !Ref ECSCluster
DesiredCount: 1
LaunchType: FARGATE
LoadBalancers:
- ContainerName: path-services-api
ContainerPort: !Ref MQPort
TargetGroupArn: !Ref TargetGroup
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- !Ref EcsSecurityGroup
Subnets:
- !Ref SubnetId
PropagateTags: SERVICE
ServiceName: !Join ['-', [ path-services-api, !Ref Color ] ]
TaskDefinition: !Ref TaskDefinition
TargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
HealthCheckPort: !Ref MQPort
HealthCheckProtocol: TCP
Port: !Ref MQPort
Protocol: TCP
TargetType: ip
VpcId: !Ref VpcId
ApplicationLoadBalancer:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
IpAddressType: ipv4
LoadBalancerAttributes:
- Key: routing.http2.enabled
Value: false
Scheme: internet-facing
SecurityGroups:
- !Ref EcsSecurityGroup
Subnets:
- !Ref SubnetId
- !Ref ELBSecondarySubnetId
Type: application
LoadBalancerListener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
DefaultActions:
- Type: forward
TargetGroupArn: !Ref TargetGroup
LoadBalancerArn: !Ref ApplicationLoadBalancer
Port: !Ref MQPort
Protocol: TCP
ServiceScalingPolicy:
Type: AWS::ApplicationAutoScaling::ScalingPolicy
DependsOn: PathService
Properties:
PolicyName: !Join ['-', [ path-services-api, !Ref Color, scaling-policy ] ]
PolicyType: TargetTrackingScaling
ResourceId: !Join [ '/', [ service, !Ref ECSCluster, !Join ['-', [ path-services-api, !Ref Color ] ] ] ]
ScalableDimension: ecs:service:DesiredCount
ServiceNamespace: ecs
TargetTrackingScalingPolicyConfiguration:
PredefinedMetricSpecification:
PredefinedMetricType: ECSServiceAverageCPUUtilization
ScaleInCooldown: 30
ScaleOutCooldown: 30
TargetValue: 70.0
# Route53Record:
# Type: AWS::Route53::RecordSet
# Properties:
# HostedZoneId: !Ref ComPublic
# Name: ***
# AliasTarget:
# DNSName: ApplicationLoadBalancer.DNSName
# HostedZoneId: !GetAtt ApplicationLoadBalancer.CanonicalHostedZoneI
# TTL: 600
# Type: A
ServiceCpuAlarm:
Type: AWS::CloudWatch::Alarm
DependsOn: PathService
Properties:
AlarmActions:
- !Ref ServiceAlarmTopic
OKActions:
- !Ref ServiceAlarmTopic
ComparisonOperator: GreaterThanOrEqualToThreshold
DatapointsToAlarm: 1
Dimensions:
- Name: Service
Value: !GetAtt PathService.Name
EvaluationPeriods: 1
MetricName: CPUUtilization
Namespace: AWS/ECS
Period: 60
Statistic: Maximum
Threshold: 90
TreatMissingData: notBreaching
ServiceMemoryAlarm:
Type: AWS::CloudWatch::Alarm
DependsOn: PathService
Properties:
AlarmActions:
- !Ref ServiceAlarmTopic
OKActions:
- !Ref ServiceAlarmTopic
ComparisonOperator: GreaterThanOrEqualToThreshold
DatapointsToAlarm: 1
Dimensions:
- Name: Service
Value: !GetAtt PathService.Name
EvaluationPeriods: 1
MetricName: MemoryUtilization
Namespace: AWS/ECS
Period: 60
Statistic: Maximum
Threshold: 90
TreatMissingData: notBreaching
ServiceAlarmTopic:
Type: AWS::SNS::Topic
DependsOn: PathService
Properties:
TopicName: path-services-api-alarm-topic
Subscription:
- Endpoint: ***
Protocol: email
- Endpoint: ***
Protocol: email
CloudFormation stack creation fails on resource LoadBalancerListener with status reason:
Invalid request provided: AWS::ElasticLoadBalancingV2::Listener Validation exception
I've found no similar issue so far and don't know where to get more details about what's wrong. Thank you.
While the answer above is 100% correct, I just want to elaborate a bit more for those who are running into this same issue. #marcin states that the load balancer "type" is "application", but the "protocol" is TCP. He is referencing two separate CF template objects here.
When creating a load balancer, you actually need to define 3 different objects in the CF template. A "load balancer", a "load balancer listener" and a "target group" (the place the load balancer will be sending traffic).
So, if a load balancer's "type" is "application" the associated "TargetGroup" must have a protocol of either "HTTP", or "HTTPS". A type of "TCP" will give this error.
This happens probably because you are using application load balancer:
Type: application
However, you specify that your connection protocol is TCP. This is obviously incorrect, as ALB only supports HTTP and HTTPS protocols. For TCP you need network load balancer (NLB).
I had the same problem, but thanks to cloudformation, one error message can always mean multiple things.. In my case it was a permission error on the load balancer. Try to add this permission to your policy:
Statement:
- Effect: Allow
Action:
- elasticloadbalancing:*
- elasticloadbalancingv2:*
Resource: *
If this fixes your problem, you know for sure it is a permission issue on elastic load balancing. Try to narrow it down by reducing permissions step by step. Hint: some loadbalancing permissions are NOT resource specific, for those you'll need the Resource: * to execute certain actions on it. From the AWS docs:
The Resource types column indicates whether each action supports resource-level permissions. If there is no value for this column, you must specify all resources ("*") in the Resource element of your policy statement. If the column includes a resource type, then you can specify an ARN of that type in a statement with that action.

CloudFormation template stuck at CREATE_IN_PROGRESS when creating ECS service

I'm creating a ECS service in CloudFormation.
I receive no error, it just will sit at the CREATE_IN_PROGRESS on the logical ID = Service phase..
Here's my CF template (ECS cluster & some other stuff above but cut out due to relevance).
TaskDefinition:
Type: 'AWS::ECS::TaskDefinition'
Properties:
Family: flink
Memory: 2048
Cpu: 512
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
ContainerDefinitions:
- Name: flink-jobmanager
Image: ACCOUNT_ID.dkr.ecr.us-west-1.amazonaws.com/teststack-flink:latest
Essential: true
PortMappings:
- ContainerPort: 8081
HostPort: 8081
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: ecs/flink-stream
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: ecs
Command:
- jobmanager
- Name: flink-taskmanager
Image: ACCOUNT_ID.dkr.ecr.us-west-1.amazonaws.com/teststack-flink:latest
Essential: true
Command:
- taskmanager
ExecutionRoleArn: !Sub arn:aws:iam::${AWS::AccountId}:role/ecsTaskExecutionRole
Volumes: []
TaskRoleArn: !Sub arn:aws:iam::${AWS::AccountId}:role/ecsTaskExecutionRole
Tags:
-
Key: EnvironmentStage
Value: !Ref EnvironmentStage
Service:
Type: 'AWS::ECS::Service'
Properties:
ServiceName: !Join ['', [!Ref EnvironmentStage, '-', !Ref 'AWS::StackName']]
Cluster: !Join ['', ['arn:aws:ecs:', !Ref 'AWS::Region', ':', !Ref 'AWS::AccountId', ':cluster/', !Ref ECSCluster]]
LaunchType: FARGATE
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 75
TaskDefinition: !Join ['', ['arn:aws:ecs:', !Ref 'AWS::Region', ':', !Ref 'AWS::AccountId', ':task-definition/', !Ref TaskDefinition]]
# TaskDefinition: !Ref TaskDefinition
DesiredCount: 1
DeploymentController:
Type: ECS
EnableECSManagedTags: true
PropagateTags: TASK_DEFINITION
SchedulingStrategy: REPLICA
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- !Ref FlinkSecurityGroup
Subnets:
- subnet-466da11c
- subnet-6fe65509
Tags:
-
Key: EnvironmentStage
Value: !Ref EnvironmentStage
The containers both deploy to the cluster when I set it up manually
After checking clusters -> CLUSTER_NAME -> tasks -> stopped I saw the following:
Status reason CannotStartContainerError: Error response from daemon:
failed to initialize logging driver: failed to create Cloudwatch log stream:
ResourceNotFoundException: The specified log group does not exist.
The issue was simply that I forgot to add the creating of a log group to to my CF template.. So I added this:
LogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Sub ${EnvironmentStage}-service-flink
Then modified the LogConfiguration in TaskDefinition to this:
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref LogGroup
awslogs-region: !Ref 'AWS::Region'
awslogs-stream-prefix: flink
Now the CF template works like a charm :)

How to integrate/link fargate cluster service with task defintion using cloudformation template

I have following cloudformation template snippet. The entire template creates the ECS fargate cluster along with all resources. but now, I am facing issues with fargate service and task definitions.
The service section of template is as follows :
Service:
Type: AWS::ECS::Service
# This dependency is needed so that the load balancer is setup correctly in time
Properties:
ServiceName: !Ref ServiceName
Cluster: !Ref Cluster
TaskDefinition: !Ref TaskDefinition
DeploymentConfiguration:
MinimumHealthyPercent: 100
MaximumPercent: 200
DesiredCount: 2
# This may need to be adjusted if the container takes a while to start up
HealthCheckGracePeriodSeconds: 30
LaunchType: FARGATE
NetworkConfiguration:
AwsvpcConfiguration:
# change to DISABLED if you're using private subnets that have access to a NAT gateway
AssignPublicIp: ENABLED
Subnets:
- !Ref abcvmnSubnetA
- !Ref abcvmnSubnetB
SecurityGroups:
- !Ref ContainerSecurityGroup
LoadBalancers:
- ContainerName: !Ref ServiceName
ContainerPort: !Ref ContainerPort
TargetGroupArn: !Ref TargetGroup
and the task definition is as follows :
TaskDefinition:
Type: AWS::ECS::TaskDefinition
# Makes sure the log group is created before it is used.
DependsOn: LogGroup
Properties:
# Name of the task definition. Subsequent versions of the task definition are grouped together under this name.
Family: abc-taskdef-dev
# awsvpc is required for Fargate
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
Cpu: 512
Memory: 1GB
# A role needed by ECS.
# "The ARN of the task execution role that containers in this task can assume. All containers in this task are granted the permissions that are specified in this role."
# "There is an optional task execution IAM role that you can specify with Fargate to allow your Fargate tasks to make API calls to Amazon ECR."
ExecutionRoleArn: arn:aws:iam::890543041640:role/ecsTaskExecutionRole
# "The Amazon Resource Name (ARN) of an AWS Identity and Access Management (IAM) role that grants containers in the task permission to call AWS APIs on your behalf."
TaskRoleArn: arn:aws:iam::890543041640:role/ecsTaskExecutionRole
ContainerDefinitions:
- Name: abc-sampleappcontainer-dev
Image: 890543041640.dkr.ecr.eu-central-1.amazonaws.com/abc:latest
PortMappings:
- ContainerPort: 8080
# Send logs to CloudWatch Logs
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-region: eu-central-1
awslogs-group: /ecs/abc-taskdef-dev
awslogs-stream-prefix: ecs
I know that , fargate service and task definitions are related to each other in the cluster. but the issue is, how to make that relationship using template.
I am getting following failure event :
The container abc-service-dev does not exist in the task definition.
(Service: AmazonECS; Status Code: 400; Error Code:
InvalidParameterException; Request ID:
008417e7-126e-11e9-98cb-ef191beeddae)
not sure, where I am doing wrong.
Your line 154
- Name: abc-sampleappcontainer-dev
change to
- Name: !Ref ServiceName
instead. because you have on line 272
- ContainerName: !Ref ServiceName
The two needs to match.
Here's an example that works:
note the name 'jaeger-query'
QueryTaskDef:
Type: 'AWS::ECS::TaskDefinition'
Properties:
ContainerDefinitions:
- Command: !Ref 'AWS::NoValue'
Name: jaeger-query
Cpu: !Ref CpuReservation
Essential: 'true'
Image: !Ref QueryImageName
Memory: !Ref MemoryReservation
Environment:
- Name: SPAN_STORAGE_TYPE
Value: elasticsearch
- Name: ES_SERVER_URLS
Value: !Sub 'http://${EsHost}:9200/'
PortMappings:
- ContainerPort: 16686
- ContainerPort: 16687
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref LxDockerLog
awslogs-region: !Ref 'AWS::Region'
awslogs-stream-prefix: !Ref 'AWS::StackName'
QueryService:
Type: 'AWS::ECS::Service'
DependsOn: AlbListenerRule
Properties:
Cluster: !Ref EcsCluster
Role: !Ref ServiceSchedulingRole
LoadBalancers:
- ContainerName: jaeger-query
ContainerPort: 16686
TargetGroupArn: !Ref AlbTargetGroup
DesiredCount: 2
TaskDefinition: !Ref QueryTaskDef
AlbListenerRule:
Type: 'AWS::ElasticLoadBalancingV2::ListenerRule'
Properties:
Actions:
- Type: forward
TargetGroupArn: !Ref AlbTargetGroup
Conditions:
- Field: host-header
Values: [!Sub '${Subdomain}.${HostedZoneName}']
ListenerArn: !Ref HttpListener
Priority: !Ref ListenerPriority
AlbTargetGroup:
Type: 'AWS::ElasticLoadBalancingV2::TargetGroup'
Properties:
HealthCheckIntervalSeconds: '60'
HealthCheckPath: '/'
HealthCheckProtocol: HTTP
HealthCheckTimeoutSeconds: '30'
HealthyThresholdCount: 10
Port: 16686
Protocol: HTTP
UnhealthyThresholdCount: 10
VpcId: !Ref VpcId
TargetGroupAttributes:
- Key: deregistration_delay.timeout_seconds
Value: !Ref DeregistrationDelay
Refer to here for the full template
https://github.com/Bit-Clouded/Glenlivet/blob/master/analytics/jaeger.template