I am deploying a queue processing ECS service using EC2 deployment using CDK. Here is my stack
from aws_cdk import core, aws_ecs_patterns, aws_ec2, aws_ecs
class EcsTestStack(core.Stack):
def __init__(self, scope: core.Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
_container_image = aws_ecs.ContainerImage.from_asset(
directory=".",
file='Dockerfile.ECS_Test',
exclude=["cdk.out"]
)
_scaling_steps = [{"upper": 0, "change": -1}, {"lower": 50, "change": +1}, {"lower": 100, "change": +2}]
_vpc = aws_ec2.Vpc(self, "ecs-test-vpc-v4", max_azs=3)
_cluster = aws_ecs.Cluster(self, "ecs-test-cluster-v4", vpc=_vpc)
_cluster.add_capacity("ecs-autoscaling-capacity-v4",
instance_type=aws_ec2.InstanceType("t2.small"),
min_capacity=1,
max_capacity=3)
self.ecs_test = aws_ecs_patterns.QueueProcessingEc2Service(
self,
"ECS_Test_Pattern_v4",
cluster=_cluster,
cpu=512,
scaling_steps=_scaling_steps,
memory_limit_mib=256,
image=_container_image,
min_scaling_capacity=1,
max_scaling_capacity=5,
)
The stack starts out with 1 task in the service and 1 EC2 instance. Based on my _scaling_steps, a new task should be added to the service when the number of messages in the queue > 50 and 2 new tasks should be added to the service. The stack starts out with 1 task in the service and 1 EC2 instance.
But when I add 200 new messages to the queue, I can see 1 new task added to my service and then I get this error message in the even.
service
EcsTestStackV4-ECSTestPatternv4QueueProcessingService5C84D200-c00N6R56hB0p
was unable to place a task because no container instance met all of
its requirements. The closest matching container-instance
81e5380c718c4b558567dc6cc1fb1926 has insufficient CPU units available.
I also notice that no new EC2 instances were added.
Question: how do I get more EC2 instances added to my cluster when the service scales up?
I can see 1 new task added to my service and then I get this error message in the even.
This is because t2.small has 1000 CPU units. So your two tasks take all of them, and there is no other instances to place your extra task on.
I also notice that no new EC2 instances were added.
You set min_capacity=1 so you have only instance. The _scaling_steps are for the tasks only, not for your instances in autoscaling group. If you want more instance you have to set min_capacity=2 or whatever value you want.
I guess you thought that QueueProcessingEc2Service scales both instances and tasks. Sadly this is not the case.
Related
I am using CDK to run standalone containerized ECS tasks on EC2.
I'm not using ECS service, because my task should be triggered using a request (in this a lambda), so they are not long running (or aways on). I use the following code, which works for one instance of the task:
this.autoScalingGroup = new autoscaling.AutoScalingGroup(this, 'ASG', {
vpc: this.vpc,
instanceType: ec2.InstanceType.of(
ec2.InstanceClass.C5,
ec2.InstanceSize.XLARGE,
),
machineImage: ecs.EcsOptimizedImage.amazonLinux2(),
minCapacity: 1,
maxCapacity: 10,
updatePolicy: UpdatePolicy.rollingUpdate(),
});
const capacityProvider = new ecs.AsgCapacityProvider(
this,
'AsgCapacityProvider',
{
autoScalingGroup: this.autoScalingGroup,
},
);
this.cluster.addAsgCapacityProvider(capacityProvider);
The problem is that I cannot run concurrent instances. So, it doesn't scale up to more instances. It only runs one instance (my container takes up ~80% of the EC2 instance resources). How would I trigger autoscaling such that when multiple tasks are run more instances get added?
p.s. obviosuly if I set minCapacity to x, x number of concurrent tasks can run, but I'd like to only keep 1 instance up and scale up and down.
The CDK-created capacity provider will auto-scale the instances based on standalone task activity. I suspect scaling is not working for you because your run-task invocations are missing a capacity provider strategy that tells ECS what capacity provider to use. There are a couple of ways to fix this:
Option 1: Set a strategy in the run-task call:
aws ecs run-task \
--cluster <cluster-arn> \
--task-definition <task-definition-name> \
--capacity-provider-strategy capacityProvider=<capacity-provider-name>,weight=1,base=1 \
--count 3 \
Option 2: Set the cluster's *default* capacity provider strategy
The CDK's L2 Cluster construct does not set a default capacity provider strategy on your cluster. See this github issue for context and workarounds. The default can also be set in the console.
I am working with aws boto3 lib and trying to retrieve certain values.
I first retrieve all cluster list, then fetch specific services, then call describe-service for them.
But I am unable to retrieve two fields Minimum tasks and Maximum tasks for services which get displayed on AWS ECS console page under Auto Scaling tab.
Anybody has any idea how to get these values from?
The ECS console hides this fact, but those are actually in the Application AutoScaling configuration, not the ECS configuration. I believe you would need to call describe_scalable_targets in ApplicationAutoScaling to get those values.
Thanks Mark B for help.
You are right and I understand that aws ecs service has to register with autoscaling service which is a separate service. I am providing sample cli and python code to retrieve these values for other now.
aws ecs describe-services --cluster MAGIC-Bonus-Wrappers --services service-name
aws application-autoscaling describe-scalable-targets --service-namespace ecs --resource-ids service/cluster-name/service-name
Python Code:
client = session.client('application-autoscaling')
response = client.describe_scalable_targets(
ServiceNamespace='ecs',
ResourceIds=[serviceId])
def_val = -1, -1
if "ScalableTargets" in response and len(response['ScalableTargets']) > 0 :
target = response['ScalableTargets'][0]
if 'MinCapacity' in target and 'MaxCapacity' in target:
return target['MinCapacity'], target['MaxCapacity']
else:
return def_val
I am trying to deploy an ECS cluster with EC2 deployment using CDK. I am using the QueueProcessingEc2Service ECS pattern.
Here is the stack
from aws_cdk import core, aws_ecs_patterns, aws_ec2, aws_ecs
class EcsTestStack(core.Stack):
def __init__(self, scope: core.Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
_container_image = aws_ecs.ContainerImage.from_asset(
directory=".",
file='Dockerfile.ECS_Test',
exclude=["cdk.out"]
)
vpc = aws_ec2.Vpc(self, "ecs-test-vpc-v4", max_azs=3)
cluster = aws_ecs.Cluster(self, "ecs-test-cluster-v4", vpc=vpc)
cluster.add_capacity("ecs-autoscaling-capacity-v4",
instance_type=aws_ec2.InstanceType("t2.small"),
min_capacity=0,
max_capacity=3)
self.ecs_test = aws_ecs_patterns.QueueProcessingEc2Service(
self,
"ECS_Test_Pattern_v4",
cluster=cluster,
cpu=512,
memory_limit_mib=512,
image=_container_image,
min_scaling_capacity=0,
max_scaling_capacity=5,
)
I would like to stack to be 'cold' when the queue is empty and spin up an EC2 instance and the ECS service and task only when messages get added to the queue.
Setting the min_scaling_capacity to 0 on the QueueProcessingEc2Service and cluster.add_capacity makes the stack not deploy. i,e. the deployment freezes at step 48/56(attaching screenshot). I've deleted the stack from CloudFormation and redeployed and get the same result. When I add the -v flag to the CDK deploy command I get a bunch of Stack EcsTestStackV4 has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS) and nothing else. (attaching screenshot)
It seems to deploy fine when I have the min_scaling_capacity set to 1.
Question: Is setting the min_scaling_capacity to 0 valid? and if it is, why doesn't my stack deeply when it is set to 0?
Minimum size for min_scaling_capacity is 1. So even if you set to it 0, it will be set to 1.
I guess it didn't work when you set min_capacity=0. In this case there are no instances in your cluster and no task will be able to deploy. QueueProcessingEc2Service does not scale out your instances, so you end up with empty auto scaling group.
QueueProcessingEc2Service is only about scaling your tasks, not the instances on which these task run.
UPDATED
Following the AWS instance scheduler I've been able to setup a scheduler that starts and stops at the beginning and end of the day.
However, the instances keep being terminated and reinstalled.
I have an Amazon Elastic Kubernetes Service (EKS) that returns the following CloudWatch log:
discovered the following log in my CloudWatch
13:05:30
2019-11-21 - 13:05:30.251 - INFO : Handler SchedulerRequestHandler scheduling request for service(s) rds, account(s) 612681954602, region(s) eu-central-1 at 2019-11-21 13:05:30.251936
13:05:30
2019-11-21 - 13:05:30.433 - INFO : Running RDS scheduler for account 612681954602 in region(s) eu-central-1
13:05:31
2019-11-21 - 13:05:31.128 - INFO : Fetching rds Instances for account 612681954602 in region eu-central-1
13:05:31
2019-11-21 - 13:05:31.553 - INFO : Number of fetched rds Instances is 2, number of schedulable resources is 0
13:05:31
2019-11-21 - 13:05:31.553 - INFO : Scheduler result {'612681954602': {'started': {}, 'stopped': {}}}
I don't know if it is my EKS that keeps rebooting my instances, but I really would love to keep them stopped until the next day.
How can I prevent my EC2 instances from automatically rebooting every time one has stopped? Or, even better, how can I deactivate my EKS stack automatically?
Update:
I discovered that EKS has a Cluster Autoscaler. Maybe this could be where the problem lies?
https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html
EKS node group would create an auto scaling group to manage the worker nodes. You need specify the minimum, maximum and desired size of worker nodes. Once any instance is stopped, the auto scaling group would create new instance to match the desired instance size.
Check below doc for details,
https://docs.aws.amazon.com/eks/latest/userguide/launch-workers.html
I'm starting a bunch of EC2 Instances using the following code
def start_ec2_instances(self, instanceids):
ec2client = boto3.resource('ec2')
response = ec2client.start_instances(InstanceIds=instanceids)
return
Now it starts successfully. However I want to use wait_until_running method to check the status of the instances and wait until all the instances are started.
wait_until_running method can be issued on single instances only? How do I wait for list of instances that has been started using boto3
This is what I'm doing currently. But wanted to know if there are any other ways to do it in one shot
def wait_until_instance_running(self, instanceids):
ec2 = boto3.resource('ec2')
for instanceid in instanceids:
instance = ec2.Instance(instanceid)
logger.info("Check the state of instance: %s",instanceid)
instance.wait_until_running()
return
Use
Wait until instance is running or
Wait until a successful state is reached
Try this:
ec2 = boto3.client('ec2')
start_ec2_instances(instanceids)
waiter = ec2.get_waiter('instance_running')
waiter.wait(InstanceIds=instanceids)
The waiter function polls every 15 seconds until a successful state is reached. An error is returned after 40 failed checks.
You can use EC2 client's Waiter.wait call to pass an array of EC2 instance Ids.
http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.Waiter.InstanceRunning