I'm using FARGATE_SPOT as a provider and the desired count is 1. I would like to know how much time ECS Service need to provide a new task after AWS send the SIGKILL.
Service:
Type: "AWS::ECS::Service"
Properties:
ServiceName: service-fargate-spot
Cluster: !Ref EcsClusterName
DeploymentConfiguration:
MinimumHealthyPercent: 100
MaximumPercent: 200
DesiredCount: 1
HealthCheckGracePeriodSeconds: 40
CapacityProviderStrategy:
- CapacityProvider: FARGATE_SPOT
Weight: 1
One running task is enough, but I'd like to not have to wait for the first task to stop before the next one runs.
I have added a Capacity Provider to an ECS cluster. While scale-out events work as expected due to changes in CapacityProviderReservation metric, scale-in events do not work.
In my case, the TargetCapacity property is set to 90, but looking at CloudWatch the average for the CapacityProviderReservation metric currently sits at 50%. This has been the case for the last 16 hours.
According to AWS's own documentation, scale-in events occur -
When using dynamic scaling policies and the size of the group decreases as a result of changes in a metric's value
So it seems like the Capacity Provider is not changing the desired size of the ASG as expected.
Am I missing something here, or do capacity providers tied to ASG's simply not work both ways?
ASG and Capacity Provider resources in CloudFormation
Resources:
AutoScalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
AutoScalingGroupName: !Sub ${ResourceNamePrefix}-asg
VPCZoneIdentifier:
- !Ref PrivateSubnetAId
LaunchTemplate:
LaunchTemplateId: !Ref Ec2LaunchTemplate
Version: !GetAtt Ec2LaunchTemplate.LatestVersionNumber
MinSize: 0
MaxSize: 3
DesiredCapacity: 1
EcsCapacityProvider:
Type: AWS::ECS::CapacityProvider
Properties:
Name: !Sub ${ResourceNamePrefix}-ecs-capacity-provider
AutoScalingGroupProvider:
AutoScalingGroupArn: !Ref AutoScalingGroup
ManagedScaling:
Status: ENABLED
TargetCapacity: 90
ManagedTerminationProtection: DISABLED
Dynamic scaling policy for ASG
Current status of the CapacityProviderReservation metric
The CapacityProviderReservation metric has been at 50% for well over 12 hours.
Current status of the Capacity Provider
As you can see, the desired size is still 2, while it is expected that this should have dropped back to 1.
Update
After deleting and recreating the cluster, I notice that the Capacity Provider changes the DesiredCapacity to 2 instantly, even though there are no tasks running.
I am trying to add a CPUCreditBalance AWS::CloudWatch::Alarm to the EBN application using cloudformation. it is similar to the picture but using cloudformation
EC2 instances and the autoscalinggroup is created by the cloudformation as well. so i dont know how to get either InscanceId or AutoScalingGroupName to place it in this code
CPUCreditBalanceAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: Warning alarm when EC2 rans out of credit
MetricName: CPUCreditBalance
Namespace: AWS/EC2
Period: 300
Statistic: Average
ComparisonOperator: LessThanThreshold
Threshold: 1
EvaluationPeriods: 2
DatapointsToAlarm: 2
TreatMissingData: breaching
Dimensions:
- Name: AutoScalingGroupName
Value: XXXXXXXX
AlarmActions:
- !Ref SnsAlarmWarning
If you have your AWS::AutoScaling::AutoScalingGroup defined in the same template as the alarm, then you can just use Ref to get ASG name:
Dimensions:
- Name: AutoScalingGroupName
Value: !Ref AWSEBAutoScalingGroup
AlarmActions:
- !Ref SnsAlarmWarning
Names of resources created by EB are listed in:
Modifying the resources that Elastic Beanstalk creates for your environment
I'm trying to set up an AWS Auto Scaling Group (ASG) that auto-scales based on average group CPU load.
I have a scale up policy that is supposed to scale the group up by 1 instance once the average CPU usage is higher than 70%. However when the alarm is triggered, the ASG launches several instances at the same time, which it shouldn'd do.
The relevant bits of CloudFormation configuration:
ECSScaleUpPolicy:
Type: AWS::AutoScaling::ScalingPolicy
Properties:
AdjustmentType: "ChangeInCapacity"
AutoScalingGroupName: !Ref ECSAutoScalingGroup
PolicyType: "StepScaling"
MetricAggregationType: "Average"
EstimatedInstanceWarmup: 600
StepAdjustments:
-
MetricIntervalLowerBound: "0"
ScalingAdjustment: "1"
ECSScaleUpAlarm:
Type: "AWS::CloudWatch::Alarm"
Properties:
AlarmDescription: "CPU more than 70% during the last minute."
AlarmName: "ECSScaleUpAlarm"
AlarmActions:
-
!Ref ECSScaleUpPolicy
Dimensions:
-
Name: "ClusterName"
Value: !Ref ECSCluster
MetricName: "CPUReservation"
Namespace: "AWS/ECS"
ComparisonOperator: "GreaterThanOrEqualToThreshold"
Statistic: "Average"
Threshold: 70
Period: 60
EvaluationPeriods: 1
TreatMissingData: "notBreaching"
As you can see, the scaling adjustment is just 1 and the instance warmup is quite long, it should wait for more time before launching the second instance :(
According to the documentation Policy type of Step scaling causes the group capacity to increase or decrease based on the size of the alarm breach. You need to change that to Simple scaling so that the capacity can be set based on a single adjustment.
I recently started using ECS. I was able to deploy a container image in ECR and create task definition for my container with CPU/Memory limits. My use case is that each container will be a long running app (no webserver, no port mapping needed). The containers will be spawned on demand 1 at a time and deleted on demand 1 at a time.
I am able to create a cluster with N server instances. But I'd like to be able for the server instances to automatically scale up/down. For example if there isn't enough CPU/Memory in the cluster, I'd like a new instance to be created.
And if there is an instance with no containers running in it, I'd like that specific instance to be scaled down / deleted. This is to avoid auto scale down termination of a server instance that has running tasks in it.
What steps are needed to be able to achieve this?
Considering that you already have an ECS Cluster created, AWS provides instructions on Scaling cluster instances with CloudWatch Alarms.
Assuming that you want to scale the cluster based on the memory reservation, at a high level, you would need to do the following:
Create an Launch Configuration for your Auto Scaling Group. This
Create an Auto Scaling Group, so that the size of the cluster can be scaled up and down.
Create a CloudWatch Alarm to scale the cluster up if the memory reservation is over 70%
Create a CloudWatch Alarm to scale the cluster down if the memory reservation is under 30%
Because it's more of my specialty I wrote up an example CloudFormation template that should get you started for most of this:
Parameters:
MinInstances:
Type: Number
MaxInstances:
Type: Number
InstanceType:
Type: String
AllowedValues:
- t2.nano
- t2.micro
- t2.small
- t2.medium
- t2.large
VpcSubnetIds:
Type: String
Mappings:
EcsInstanceAmis:
us-east-2:
Ami: ami-1c002379
us-east-1:
Ami: ami-9eb4b1e5
us-west-2:
Ami: ami-1d668865
us-west-1:
Ami: ami-4a2c192a
eu-west-2:
Ami: ami-cb1101af
eu-west-1:
Ami: ami-8fcc32f6
eu-central-1:
Ami: ami-0460cb6b
ap-northeast-1:
Ami: ami-b743bed1
ap-southeast-2:
Ami: ami-c1a6bda2
ap-southeast-1:
Ami: ami-9d1f7efe
ca-central-1:
Ami: ami-b677c9d2
Resources:
Cluster:
Type: AWS::ECS::Cluster
Role:
Type: AWS::IAM::Role
Properties:
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
-
Effect: Allow
Action:
- sts:AssumeRole
Principal:
Service:
- ec2.amazonaws.com
InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles:
- !Ref Role
LaunchConfiguration:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
ImageId: !FindInMap [EcsInstanceAmis, !Ref "AWS::Region", Ami]
InstanceType: !Ref InstanceType
IamInstanceProfile: !Ref InstanceProfile
UserData:
Fn::Base64: !Sub |
#!/bin/bash
echo ECS_CLUSTER=${Cluster} >> /etc/ecs/ecs.config
AutoScalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
MinSize: !Ref MinInstances
MaxSize: !Ref MaxInstances
LaunchConfigurationName: !Ref LaunchConfiguration
HealthCheckGracePeriod: 300
HealthCheckType: EC2
VPCZoneIdentifier: !Split [",", !Ref VpcSubnetIds]
ScaleUpPolicy:
Type: AWS::AutoScaling::ScalingPolicy
Properties:
AdjustmentType: ChangeInCapacity
AutoScalingGroupName: !Ref AutoScalingGroup
Cooldown: '1'
ScalingAdjustment: '1'
MemoryReservationAlarmHigh:
Type: AWS::CloudWatch::Alarm
Properties:
EvaluationPeriods: '2'
Statistic: Average
Threshold: '70'
AlarmDescription: Alarm if Cluster Memory Reservation is to high
Period: '60'
AlarmActions:
- Ref: ScaleUpPolicy
Namespace: AWS/ECS
Dimensions:
- Name: ClusterName
Value: !Ref Cluster
ComparisonOperator: GreaterThanThreshold
MetricName: MemoryReservation
ScaleDownPolicy:
Type: AWS::AutoScaling::ScalingPolicy
Properties:
AdjustmentType: ChangeInCapacity
AutoScalingGroupName: !Ref AutoScalingGroup
Cooldown: '1'
ScalingAdjustment: '-1'
MemoryReservationAlarmLow:
Type: AWS::CloudWatch::Alarm
Properties:
EvaluationPeriods: '2'
Statistic: Average
Threshold: '30'
AlarmDescription: Alarm if Cluster Memory Reservation is to Low
Period: '60'
AlarmActions:
- Ref: ScaleDownPolicy
Namespace: AWS/ECS
Dimensions:
- Name: ClusterName
Value: !Ref Cluster
ComparisonOperator: LessThanThreshold
MetricName: MemoryReservation
This creates an ECS Cluster, a Launch Configuration, An AutoScaling Group, As well as the Alarms based on the ECS Memory Reservation.
Now we can get to the interesting discussions.
Why can't we scale up based on the CPU Utilization And Memory Reservation?
The short answer is you totally can But you're likely to pay a lot for it. EC2 has a known property that when you create an instance, you pay for a minimum of 1 hour, because partial instance hours are charged as full hours. Why that's relevant is, imagine you have multiple alarms. Say you have a bunch of services that are currently running idle, and you fill the cluster. Either the CPU Alarm scales down the cluster, or the Memory Alarm scales up the cluster. One of these will likely scale the cluster to the point that it's alarm is no longer triggered. After the cooldown, period, the other alarm will undo it's last action, After the next cooldown, the action will likely be redone. Thus instances are created then destroyed repeatedly on every other cooldown.
After giving a bunch of thought to this, the strategy that I came up with was to use Application Autoscaling for ECS Services based on CPU Utilization, and Memory Reservation based on the cluster. So if one service is running hot, an extra task will be added to share the load. This will slowly fill the cluster memory reservation capacity. When the memory gets full, the cluster scales up. When a service is cooling down, the services will start shutting down tasks. As the memory reservation on the cluster drops, the cluster will be scaled down.
The thresholds for the CloudWatch Alarms might need to be experimented with, based on your task definitions. The reason for this is that if you put the scale up threshold too high, it may not scale up as the memory gets consumed, and then when autoscaling goes to place another task, it will find that there isn't enough memory available on any instance in the cluster, and therefore be unable to place another task.
As part of this year's re:Invent conference, AWS announced cluster auto scaling for Amazon ECS. Clusters configured with auto scaling can now add more capacity when needed and remove capacity that is not necessary. You can find more information about this in the documentation.
However, depending on what you're trying to run, AWS Fargate could be a better option. Fargate allows you to run containers without provisioning and managing the underlying infrastructure; i.e., you don't have to deal with any EC2 instances. With Fargate, you can make an API call to run your container, the container can run, and then there's nothing to clean up once the container stops running. Fargate is billed per-second (with a 1-minute minimum) and is priced based on the amount of CPU and memory allocated (see here for details).