AWS ECS TaskDefinition autoscaling - amazon-web-services

I'm have an issue hopping you can help.
I wanted to make aws ecs task but I cant find a way to auto-scale it
its stuck with 256CPU and 512Memory, and I dont want to hard code it (2G memory) because I don't need it all the time
Parameters:
SubnetIds:
Type: "List<AWS::EC2::Subnet::Id>"
Description: Select at least two subnets in your selected VPC.
Resources:
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
Family: name
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
ContainerDefinitions:
-
Name: "name"
Image: "**/latest"
Essential: true
logConfiguration:
logDriver: awslogs
options:
awslogs-group: /log/scheduledtasks
awslogs-region: us-east-2
awslogs-stream-prefix: log
Environment:
- Name: OP_MODE
Value: CRONJOB
Secrets:**
ExecutionRoleArn: !ImportValue TaskExecutionRole
Cpu: 256
Memory: 512
TaskSchedule1:
Type: AWS::Events::Rule
Properties:
Description: Description
Name: name
ScheduleExpression: cron(5 10 * * ? *)
State: ENABLED
Targets:
- Id: id-name
RoleArn: !ImportValue SchedulerRole
EcsParameters:
TaskDefinitionArn:
Ref: TaskDefinition
TaskCount: 1
LaunchType: FARGATE
NetworkConfiguration:
AwsVpcConfiguration:
AssignPublicIp: ENABLED
Subnets: !Split [',', !Join [',', !Ref SubnetIds]]
Input: '{"containerOverrides":[{"name":"a","environment":[{"name":"TARGET_JOB","value":"definition"}]}]}'
Arn: !ImportValue Cluster
any tip or resource will be so helpful
I'm still learning, so thank you for your help

You can't do this. This is not how autoscaling of ECS services/tasks work. You autoscale your service by launching multiple instances of the same task, not by automatically increasing/decreasing the amount of ram and cpu you have allocated.

Related

AWS CloudFormation cannot create Stack using CodeDeploy blue/green deployments

I am trying to deploy a new stack using CloudFormation with an ECS service using the CodeDeploy launch type to enable blue/green deployments.
In the User Guide for performing blue/green deployments via CloudFormation, they state the following:
To enable CloudFormation to perform blue/green deployments on a Stack, include the following information in its stack template:
At least one of the ECS resources that will trigger a blue/green deployment if replaced during a stack update. Currently, those resources are AWS::ECS::TaskDefinition and AWS::ECS::TaskSet.
When I exclude a AWS::ECS::TaskSet, the Stack creation fails and I receive the following error:
Transform AWS::CodeDeployBlueGreen failed with: Failed to transform template.
ECSAttributes must include TaskSets in AWS::CodeDeploy::BlueGreen Hook
If I add a AWS::ECS::TaskSet, the stack fails to create with the following error instead:
Resource handler returned message:
"Invalid request provided: Amazon ECS does not support task set management on services where deployments
are controlled by AWS CodeDeploy.
(Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException;
Request ID: 61b8c146-3ae9-4bc2-ac5c-08a11e194f06; Proxy: null)"
(RequestToken: 86a8a3a5-fe89-9939-15c2-45b08b28c3f3, HandlerErrorCode: InvalidRequest)
These are the relevant parts of my stack template:
Transform:
- AWS::CodeDeployBlueGreen
Hooks:
CodeDeployBlueGreenHook:
Type: AWS::CodeDeploy::BlueGreen
Properties:
ServiceRole: BlueGreenDeploymentRole
Applications:
- Target:
Type: AWS::ECS::Service
LogicalID: EcsService
ECSAttributes:
TaskDefinitions:
- TaskDefinitionBlue
- TaskDefinitionGreen
TaskSets:
- TaskSetBlue
- TaskSetGreen
TrafficRouting:
ProdTrafficRoute:
Type: AWS::ElasticLoadBalancingV2::Listener
LogicalID: LoadBalancerListener
TargetGroups:
- TargetGroupBlue
- TargetGroupGreen
TrafficRoutingConfig:
Type: TimeBasedLinear
TimeBasedLinear:
StepPercentage: 20
BakeTimeMins: 10
AdditionalOptions:
TerminationWaitTimeInMinutes: 60
Resources:
# IAM Role for blue/green deployments
BlueGreenDeploymentRole:
Type: AWS::IAM::Role
Properties:
RoleName: blue-green-deployment-role
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: codedeploy.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: BlueGreenDeploymentPolicy
PolicyDocument:
Statement:
- Effect: Allow
Action:
- codedeploy:Get*
- codedeploy:CreateCloudFormationDeployment
Resource: '*'
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AWSCodeDeployRoleForECS
#########################
# Load Balancer
#########################
# Application Load Balancer
LoadBalancer:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
IpAddressType: ipv4
Name: service-alb
Scheme: internal
Subnets:
- !Ref SubnetOne
- !Ref SubnetTwo
SecurityGroups:
- !Ref SecurityGroup
# Load Balancer Listener
LoadBalancerListener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
DefaultActions:
- Type: forward
ForwardConfig:
TargetGroups:
- TargetGroupArn: !Ref TargetGroupBlue
Weight: 1
- TargetGroupArn: !Ref TargetGroupGreen
Weight: 1
LoadBalancerArn: !Ref LoadBalancer
Port: 8080
Protocol: HTTP
# Load Balancer Target Groups
TargetGroupBlue:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: !Sub ${ServiceName}-blue
TargetType: ip
VpcId: !Ref Vpc
Port: 8080
Protocol: HTTP
HealthCheckPort: 8080
HealthCheckPath: /actuator/health
TargetGroupGreen:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: !Sub ${ServiceName}-green
TargetType: ip
VpcId: !Ref Vpc
Port: 8080
Protocol: HTTP
HealthCheckPort: 8080
HealthCheckPath: /actuator/health
#########################
# ECS
#########################
# ECS Cluster
Cluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: !Ref ServiceName
# ECS Service
EcsService:
Type: AWS::ECS::Service
DependsOn: LoadBalancerListener
Properties:
ServiceName: !Ref ServiceName
Cluster: !Ref Cluster
TaskDefinition: !Ref TaskDefinitionBlue
DeploymentController:
Type: CODE_DEPLOY
DesiredCount: 0
LaunchType: FARGATE
LoadBalancers:
- ContainerName: !Sub ${ServiceName}-container
ContainerPort: 8080
TargetGroupArn: !Ref TargetGroupBlue
- ContainerName: !Sub ${ServiceName}-container
ContainerPort: 8080
TargetGroupArn: !Ref TargetGroupGreen
NetworkConfiguration:
AwsvpcConfiguration:
Subnets:
- !Ref SubnetOne
- !Ref SubnetTwo
SecurityGroups:
- !Ref SecurityGroup
SchedulingStrategy: REPLICA
# Task Definitions
TaskDefinitionBlue:
Type: AWS::ECS::TaskDefinition
Properties:
ContainerDefinitions:
- Name: !Sub ${ServiceName}-container
Image: !Sub ${ImageRepository.RepositoryUri}:latest
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref LogGroup
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: payments
PortMappings:
- ContainerPort: 8080
Cpu: 256
Memory: 512
NetworkMode: awsvpc
Family: !Sub ${ServiceName}
ExecutionRoleArn: !Ref TaskExecutionRole
RequiresCompatibilities:
- FARGATE
TaskDefinitionGreen:
Type: AWS::ECS::TaskDefinition
Properties:
ContainerDefinitions:
- Name: !Sub ${ServiceName}-container
Image: !Sub ${ImageRepository.RepositoryUri}:latest
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref LogGroup
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: payments
PortMappings:
- ContainerPort: 8080
Cpu: 256
Memory: 512
NetworkMode: awsvpc
Family: !Sub ${ServiceName}
ExecutionRoleArn: !Ref TaskExecutionRole
RequiresCompatibilities:
- FARGATE
# Image Repository
ImageRepository:
Type: AWS::ECR::Repository
Properties:
RepositoryName: !Sub ${ServiceName}
LifecyclePolicy:
LifecyclePolicyText: |
{
"rules": [
{
"rulePriority": 1,
"description": "Maintain at most 25 images",
"selection": {
"tagStatus": "untagged",
"countType": "imageCountMoreThan",
"countNumber": 25
},
"action": {
"type": "expire"
}
}
]
}
TaskSetBlue:
Type: AWS::ECS::TaskSet
Properties:
Cluster: !Ref Cluster
LaunchType: FARGATE
Service: !Ref EcsService
TaskDefinition: !Ref TaskDefinitionBlue
NetworkConfiguration:
AwsVpcConfiguration:
Subnets:
- !Ref SubnetOne
- !Ref SubnetTwo
SecurityGroups:
- !Ref SecurityGroup
LoadBalancers:
- ContainerName: !Sub ${ServiceName}-container
ContainerPort: 8080
TargetGroupArn: !Ref TargetGroupBlue
How can I update my template to allow for the blue/green deployment strategy via CodeDeploy?
G/B deployment using CFN is EXTERNAL, not CODE_DEPLOY. There could be many other issues with your template, but your current error relates to using wrong DeploymentController. Please study AWS docs and example:
Perform ECS blue/green deployments through CodeDeploy using AWS CloudFormation
Exactly, AWS doesn't support Blue Green deployment on EC2/ASG...!
The easiest way that I just figure out is to create the deployment group with In-Place method && Then change the deployment group configuration manually from In-place to Blue Green
Or you can use the lambda function, but is a litte bit hard to customize it
Wish this will help you

Cloudformation: ECS TaskDefinition CloudWatch logs retention policy

Is it possible to specify a CloudWatch logs retention policy in an ECS Task definition?
Couldn't find any documentation about it.
ServiceTaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
ExecutionRoleArn: !GetAtt EcsTaskExecutionRole.Arn
TaskRoleArn: !GetAtt EcsTaskRole.Arn
Cpu: !Ref TaskDefinitionCpu
Memory: !Ref TaskDefinitionMemory
NetworkMode: awsvpc
ContainerDefinitions:
- Name: !Join ['-', ['container', !Ref AWS::StackName]]
Image: !Ref EcrImage
PortMappings:
- ContainerPort: !Ref Port
HostPort: !Ref Port
Protocol: tcp
Essential: true
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Join ['', ['/ecs/', !Ref AWS::StackName]]
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: ecs
awslogs-create-group: true
# Retention policy ??
Agree with other answer that there is no option to specify log retention in awslogs options
We need to create it and pass it along:
CloudwatchLogsGroup:
Type: 'AWS::Logs::LogGroup'
Properties:
LogGroupName: !Sub '${AWS::StackName}-ECSLogGroup'
RetentionInDays: 14
Container Definition:
ContainerTaskdefinition:
Type: 'AWS::ECS::TaskDefinition'
Properties:
Family: !Ref 'AWS::StackName'
ExecutionRoleArn: !Ref ECSTaskExecutionRole
TaskRoleArn: !Ref ECSTaskExecutionRole
Cpu: '256'
Memory: 1GB
NetworkMode: awsvpc
RequiresCompatibilities:
- EC2
- FARGATE
ContainerDefinitions:
- Name: !Ref 'AWS::StackName'
Cpu: 256
Essential: 'true'
Image: !Ref Image
Memory: '1024'
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref CloudwatchLogsGroup <-- refer to log group
awslogs-region: !Ref 'AWS::Region'
awslogs-stream-prefix: ecs
There appears to be no support for specifying the retention policy when you "auto create" the log group in the Task Definition. You could however create your log group out of band and let your Task Definition consume it (instead of auto creating it). The retention policy could be defined when you create the log group explicitly (that is, in the AWS::Logs::LogGroup definition). See here.
As with all other services that support logging to CloudWatch logs, if you want to set things like KMS encryption and log retention on the log group you have to create the log group first, with the settings you want, then configure the services to log to that log group.

PGAdmin4 on ECS Fargate

I am trying to run PGAdmin4 as a docker container on AWS Fargate but for some reason the container is exiting. Cloudwatch logs were not so helpful either.
Here is my Task definiton and Service. Indentation may not be accurate after copying it here.
Task:
Type: AWS::ECS::TaskDefinition
Properties:
Family: pgadmin
Cpu: 2048
Memory: 4096
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
ExecutionRoleArn: !Ref ECSTaskExecutionRole
ContainerDefinitions:
- Name: pgadmin
Image: dpage/pgadmin4 # From dockerhub
Cpu: 2048
Memory: 4096
PortMappings:
- ContainerPort: 80
Protocol: tcp
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: pgadmin
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: pgadmin
Environment:
- Name: PGADMIN_DEFAULT_EMAIL
Value: s-polimetla#s-polimetla.de
- Name: PGADMIN_DEFAULT_PASSWORD
Value: test1234
Service:
Type: AWS::ECS::Service
DependsOn: ListenerRule
Properties:
ServiceName: pgadmin-service
TaskDefinition: !Ref Task
Cluster: !Ref ECSCluster
LaunchType: FARGATE
DesiredCount: 1
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 100
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
Subnets:
- Fn::ImportValue:
!Sub ${NetworkStackName}-subnet1
- Fn::ImportValue:
!Sub ${NetworkStackName}-subnet2
SecurityGroups:
- Fn::ImportValue:
!Sub ${NetworkStackName}-container-SG
LoadBalancers:
- ContainerName: pgadmin
ContainerPort: 80
TargetGroupArn: !Ref TargetGroup
My cloudwatch logs show the following
You can find the complete template here. Please feel free to correct it if you find anything wrong with it.
I cannot figure out what went wrong from the logs. Any pointers here?

How to integrate/link fargate cluster service with task defintion using cloudformation template

I have following cloudformation template snippet. The entire template creates the ECS fargate cluster along with all resources. but now, I am facing issues with fargate service and task definitions.
The service section of template is as follows :
Service:
Type: AWS::ECS::Service
# This dependency is needed so that the load balancer is setup correctly in time
Properties:
ServiceName: !Ref ServiceName
Cluster: !Ref Cluster
TaskDefinition: !Ref TaskDefinition
DeploymentConfiguration:
MinimumHealthyPercent: 100
MaximumPercent: 200
DesiredCount: 2
# This may need to be adjusted if the container takes a while to start up
HealthCheckGracePeriodSeconds: 30
LaunchType: FARGATE
NetworkConfiguration:
AwsvpcConfiguration:
# change to DISABLED if you're using private subnets that have access to a NAT gateway
AssignPublicIp: ENABLED
Subnets:
- !Ref abcvmnSubnetA
- !Ref abcvmnSubnetB
SecurityGroups:
- !Ref ContainerSecurityGroup
LoadBalancers:
- ContainerName: !Ref ServiceName
ContainerPort: !Ref ContainerPort
TargetGroupArn: !Ref TargetGroup
and the task definition is as follows :
TaskDefinition:
Type: AWS::ECS::TaskDefinition
# Makes sure the log group is created before it is used.
DependsOn: LogGroup
Properties:
# Name of the task definition. Subsequent versions of the task definition are grouped together under this name.
Family: abc-taskdef-dev
# awsvpc is required for Fargate
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
Cpu: 512
Memory: 1GB
# A role needed by ECS.
# "The ARN of the task execution role that containers in this task can assume. All containers in this task are granted the permissions that are specified in this role."
# "There is an optional task execution IAM role that you can specify with Fargate to allow your Fargate tasks to make API calls to Amazon ECR."
ExecutionRoleArn: arn:aws:iam::890543041640:role/ecsTaskExecutionRole
# "The Amazon Resource Name (ARN) of an AWS Identity and Access Management (IAM) role that grants containers in the task permission to call AWS APIs on your behalf."
TaskRoleArn: arn:aws:iam::890543041640:role/ecsTaskExecutionRole
ContainerDefinitions:
- Name: abc-sampleappcontainer-dev
Image: 890543041640.dkr.ecr.eu-central-1.amazonaws.com/abc:latest
PortMappings:
- ContainerPort: 8080
# Send logs to CloudWatch Logs
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-region: eu-central-1
awslogs-group: /ecs/abc-taskdef-dev
awslogs-stream-prefix: ecs
I know that , fargate service and task definitions are related to each other in the cluster. but the issue is, how to make that relationship using template.
I am getting following failure event :
The container abc-service-dev does not exist in the task definition.
(Service: AmazonECS; Status Code: 400; Error Code:
InvalidParameterException; Request ID:
008417e7-126e-11e9-98cb-ef191beeddae)
not sure, where I am doing wrong.
Your line 154
- Name: abc-sampleappcontainer-dev
change to
- Name: !Ref ServiceName
instead. because you have on line 272
- ContainerName: !Ref ServiceName
The two needs to match.
Here's an example that works:
note the name 'jaeger-query'
QueryTaskDef:
Type: 'AWS::ECS::TaskDefinition'
Properties:
ContainerDefinitions:
- Command: !Ref 'AWS::NoValue'
Name: jaeger-query
Cpu: !Ref CpuReservation
Essential: 'true'
Image: !Ref QueryImageName
Memory: !Ref MemoryReservation
Environment:
- Name: SPAN_STORAGE_TYPE
Value: elasticsearch
- Name: ES_SERVER_URLS
Value: !Sub 'http://${EsHost}:9200/'
PortMappings:
- ContainerPort: 16686
- ContainerPort: 16687
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref LxDockerLog
awslogs-region: !Ref 'AWS::Region'
awslogs-stream-prefix: !Ref 'AWS::StackName'
QueryService:
Type: 'AWS::ECS::Service'
DependsOn: AlbListenerRule
Properties:
Cluster: !Ref EcsCluster
Role: !Ref ServiceSchedulingRole
LoadBalancers:
- ContainerName: jaeger-query
ContainerPort: 16686
TargetGroupArn: !Ref AlbTargetGroup
DesiredCount: 2
TaskDefinition: !Ref QueryTaskDef
AlbListenerRule:
Type: 'AWS::ElasticLoadBalancingV2::ListenerRule'
Properties:
Actions:
- Type: forward
TargetGroupArn: !Ref AlbTargetGroup
Conditions:
- Field: host-header
Values: [!Sub '${Subdomain}.${HostedZoneName}']
ListenerArn: !Ref HttpListener
Priority: !Ref ListenerPriority
AlbTargetGroup:
Type: 'AWS::ElasticLoadBalancingV2::TargetGroup'
Properties:
HealthCheckIntervalSeconds: '60'
HealthCheckPath: '/'
HealthCheckProtocol: HTTP
HealthCheckTimeoutSeconds: '30'
HealthyThresholdCount: 10
Port: 16686
Protocol: HTTP
UnhealthyThresholdCount: 10
VpcId: !Ref VpcId
TargetGroupAttributes:
- Key: deregistration_delay.timeout_seconds
Value: !Ref DeregistrationDelay
Refer to here for the full template
https://github.com/Bit-Clouded/Glenlivet/blob/master/analytics/jaeger.template

How to perform AWS CloudFormation autoscaling for ECS instance when cluster has insufficient memory available

I have created CloudFormation template that creates ECS service and task and has autoscaling for tasks. It is pretty basic - if MemoruUtilization for tasks reaches certain value then add 1 task and vice verse. Here are some of the most relevant parts form template.
EcsTd:
Type: AWS::ECS::TaskDefinition
DependsOn: LogGroup
Properties:
Family: !Sub ${EnvironmentName}-${PlatformName}-${Type}
ContainerDefinitions:
- Name: !Sub ${EnvironmentName}-${PlatformName}-${Type}
Image: !Sub ${AWS::AccountId}.dkr.ecr.{AWS::Region}.amazonaws.com/${PlatformName}:${ImageVersion}
Environment:
- Name: APP_ENV
Value: !If [isProd, "production", "staging"]
- Name: APP_DEBUG
Value: "false"
...
PortMappings:
- ContainerPort: 80
HostPort: 0
Memory: !Ref Memory
Essential: true
EcsService:
Type: AWS::ECS::Service
DependsOn: WaitForLoadBalancerListenerRulesCondition
Properties:
ServiceName: !Sub ${EnvironmentName}-${PlatformName}-${Type}
Cluster:
Fn::ImportValue: !Sub ${EnvironmentName}-ECS-${Type}
DesiredCount: !Sub ${DesiredCount}
TaskDefinition: !Ref EcsTd
Role: "learningEcsServiceRole"
LoadBalancers:
- !If
- isWeb
- ContainerPort: 80
ContainerName: !Sub ${EnvironmentName}-${PlatformName}-${Type}
TargetGroupArn: !Ref AlbTargetGroup
- !Ref AWS::NoValue
ServiceScalableTarget:
Type: "AWS::ApplicationAutoScaling::ScalableTarget"
Properties:
MaxCapacity: !Sub ${MaxCount}
MinCapacity: !Sub ${MinCount}
ResourceId: !Join
- /
- - service
- !Sub ${EnvironmentName}-${Type}
- !GetAtt EcsService.Name
RoleARN: arn:aws:iam::645618565575:role/learningEcsServiceRole
ScalableDimension: ecs:service:DesiredCount
ServiceNamespace: ecs
ServiceScaleOutPolicy:
Type : "AWS::ApplicationAutoScaling::ScalingPolicy"
Properties:
PolicyName: !Sub ${EnvironmentName}-${PlatformName}-${Type}- ScaleOutPolicy
PolicyType: StepScaling
ScalingTargetId: !Ref ServiceScalableTarget
StepScalingPolicyConfiguration:
AdjustmentType: ChangeInCapacity
Cooldown: 1800
MetricAggregationType: Average
StepAdjustments:
- MetricIntervalLowerBound: 0
ScalingAdjustment: 1
MemoryScaleOutAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: !Sub ${EnvironmentName}-${PlatformName}-${Type}-MemoryOver70PercentAlarm
AlarmDescription: Alarm if memory utilization greater than 70% of reserved memory
Namespace: AWS/ECS
MetricName: MemoryUtilization
Dimensions:
- Name: ClusterName
Value: !Sub ${EnvironmentName}-${Type}
- Name: ServiceName
Value: !GetAtt EcsService.Name
Statistic: Maximum
Period: '60'
EvaluationPeriods: '1'
Threshold: '70'
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- !Ref ServiceScaleOutPolicy
- !Ref EmailNotification
...
So when ever task starts to run out of memory we'll add new task. However at some point we'll reach the limit how much memory are available in out cluster.
So for example is Cluster consists of one t2.small instance then we have 2Gb RAM. A small amount of that is used by ECS task running in instance so we have less then 2GB RAM. If we set the value of Task's memory to 512Mb then we can put only 3 tasks in that cluster unless we scale up the cluster.
By default ECS service has MemoryReservation metrics that can be used for autoscaling cluster. We would tell that when MemoryReservation in more then 75% then add 1 instance to cluster. That's relatively easy.
EcsCluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: !Sub ${EnvironmentName}-${Type}
SgEcsHost:
...
ECSLaunchConfiguration:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
ImageId: !FindInMap [AWSRegionToAMI, !Ref 'AWS::Region', AMIID]
InstanceType: !Ref InstanceType
SecurityGroups: [ !Ref SgEcsHost ]
AssociatePublicIpAddress: true
IamInstanceProfile: "ecsInstanceRole"
KeyName: !Ref KeyName
UserData:
Fn::Base64: !Sub |
#!/bin/bash
echo ECS_CLUSTER=${EnvironmentName}-${Type} >> /etc/ecs/ecs.config
ECSAutoScalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
VPCZoneIdentifier:
- Fn::ImportValue: !Sub ${EnvironmentName}-SubnetEC2AZ1
- Fn::ImportValue: !Sub ${EnvironmentName}-SubnetEC2AZ2
LaunchConfigurationName: !Ref ECSLaunchConfiguration
MinSize: !Ref AsgMinSize
MaxSize: !Ref AsgMaxSize
DesiredCapacity: !Ref AsgDesiredSize
Tags:
- Key: Name
Value: !Sub ${EnvironmentName}-ECS
PropagateAtLaunch: true
ScalePolicyUp:
Type: AWS::AutoScaling::ScalingPolicy
Properties:
AdjustmentType: ChangeInCapacity
AutoScalingGroupName:
Ref: ECSAutoScalingGroup
Cooldown: '1'
ScalingAdjustment: '1'
MemoryReservationAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
EvaluationPeriods: '1'
Statistic: Average
Threshold: '75'
AlarmDescription: Alarm if MemoryReservation is more then 75%
Period: '60'
AlarmActions:
- Ref: ScalePolicyUp
- Ref: EmailNotification
Namespace: AWS/EC2
Dimensions:
- Name: AutoScalingGroupName
Value:
Ref: ECSAutoScalingGroup
ComparisonOperator: GreaterThanThreshold
MetricName: MemoryReservation
However it does not make sense because that would happen when the third task is added so the new instance will be empty until 4th tasks is scaled. That means we'll be paying for instance that we don't use.
I have noticed that when ECS service tries to add task to cluster where there is not enough free Memory I get
service Production-admin-worker was unable to place a task because no
container instance met all of its requirements. The closest matching
container-instance ################### has
insufficient memory available.
In this example the template's parameters are:
EnvironmentName=Production
PlatformName=Admin
Type=worker
Is it possible to create AWS::CloudWatch::Alarm that looks at ECS cluster events and looks for that particular pattern? The idea would be to scale up instance count in cluster using AWS::AutoScaling::AutoScalingGroup only when AWS::ApplicationAutoScaling::ScalingPolicy adds tasks that does not have space in cluster. And scale down the cluster when MemoryReservation is less then 25% (meaning that there are no tasks running there - AWS::ApplicationAutoScaling::ScalingPolicy has removed them).
That means we'll be paying for instance that we don't use.
Either you pay for the extra/backup capacity in advance, or implement logic to retry the ones that failed due to low capacity.
Couple of ways I can think of:
You could create a custom script/lambda (https://forums.aws.amazon.com/thread.jspa?threadID=94984) that reports a metric say load_factor calculated as number of tasks / number of instances and then base an your auto scaling policy on that. Lambda can be triggered by a CW Rule.
You could also report this from your task implementation instead of a new custom lambda/script.
Create a metric filter that looks for a specific pattern in a log file/group and reports a metric. Then of course use this metric for scaling.
From docs:
When a metric filter finds one of the terms, phrases, or values in your log events, you can increment the value of a CloudWatch metric.