Let's suppose that i have an ECS cluster running a service, this service has some autoscaling policies applied to it. When this Service Scales Out, a new task is created, an alarm is then triggered for the AutoScaling group to Launch a new EC2 instance for the Task to Run. I have no troubles on that side.
On the other side, when this service scales In, a task is stoped and deleted. and alarm for Scaling In is triggered for the Autoscaling Group to Terminate an EC2 Instance.
To my knowledge, an AWS Autoscaling Group terminates an EC2 instance (within the group) Randomly in the case of DownScale. Is that so ?
If that's Correct ! is it possible to target only the empty Container Instances to be terminated in the case of DownScale ?
Related
I am making an AWS ECS cluster using EC2 and trying to use capacity providers. I don't really understand why do I need to enable instance scale-in protection inside my AWS Auto Scaling group.
Isn't the point of Auto Scaling the termination of needless EC2 instances?
why do I need to enable instance scale-in protection
This is only needed when you chose to use managed scaling:
When managed scaling is enabled, Amazon ECS manages the scale-in and scale-out actions of the Auto Scaling group used when creating the capacity provider. On your behalf, Amazon ECS creates an AWS Auto Scaling scaling plan with a target tracking scaling policy based on the target capacity value you specify.
The managed scaling ensures that ECS controlled when instances are removed. By doing this it protects any instances that have some tasks running on it from being terminated:
When managed termination protection is enabled, Amazon ECS prevents Amazon EC2 instances that contain tasks and that are in an Auto Scaling group from being terminated during a scale-in action.
The entire idea is that you enable instance scale-in protection on your ASG so that ECS has control over which instances to terminate based on tasks they run. Without this, your ASG could terminate instances based on other criteria, not nervelessly related to "needless EC2 instances". For example, ASG can choose to terminate instances based due to AZRebalance process. This could lead to ASG terminating instances with running tasks, which may not be what you want.
I think you have 2 scaling
1 for ec2 instance
1 for ecs task
So I think flow goes like
You get more requests
Your tasks need more resources (cpu/memory)
Your ecs task autoscaling kicks in and creates more task
your ec2 autoscaling also kicks in and creates more ec2
Questions
your ecs needs to know newly launched ec2 (from ec2 autoscaling group) is the host that it can run on (how?)
I want the number of ec2 instance starts from one.
I think one instance is already running in the ecs cluster
Do I make ec2 autoscaling group start from 0?
ec2 autoscaling group has pricing option to use spot instance, but I didn't see such option when I created ecs cluster instances. To be clear, can I make ecs cluster with 0 ecs-instance, and let ec2 autoscaling group starts with 1 instance?
Your ecs task autoscaling kicks in and creates more task
your ec2 autoscaling also kicks in and creates more ec2
Your ECS task scale if there is a memory so fine it will scale and launch new container, if not enough memory then it will not launch an instance, you need to configure cloud watch alarm to scale ECS cluster based on the task scaling.
Tutorial: Scaling Container Instances with CloudWatch Alarms
your ecs needs to know newly launched ec2 (from ec2 autoscaling group)
is the host that it can run on (how?)
ECS agent registers the new instance with the same cluster, and the new task can be placed in any instance that have available resources to handle and launch the container.
I want the number of ec2 instance starts from one. I think one
instance is already running in the ecs cluster Do I make ec2
autoscaling group start from 0?
You should start from 1 otherwise no container will launch and will throw error no instance found in the cluster. if you do not want run container for now then you can start from 0.
ec2 autoscaling group has pricing option to use spot instance, but I
didn't see such option when I created ecs cluster instances. To be
clear, can I make ecs cluster with 0 ecs-instance, and let ec2
autoscaling group starts with 1 instance?
You can read about spot ECS cluster instance here.
Amazon ECS is a highly scalable, high performance, container
management service that supports Docker containers and allows you to
run applications on a managed cluster of Amazon EC2 instances easily.
ECS already handles the placement and scheduling of containers on EC2
instances. When combined with Spot fleet, ECS can deliver significant
savings over EC2 on-demand pricing.
Currently I am struggling with AWS auto scaling.
Do I need to create instance Auto Scaling Group in parallel to ECS Service Auto Scaling to provision necessary resources for services/tasks, or it is managed automatically by the latter one?
edit:
I am using api/cloudformation only, no web interface.
ECS service scaling and And EC2 instance scaling are independent entities. EC2 instance scaling is not handled if you configure ECS service scaling. It is required(Mandotory) to configure EC2 Instance scaling and maintain enough resources in the cluster if you are opting ECS Service scaling, otherwise ECS service fails to place new Tasks during service scaling activities. In general, it is recommended to scale EC2 Instances based on CPUReservation and MemoryReservation cloudwatch alarms.
EC2 Container Instance scaling tutorial: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch_alarm_autoscaling.html
I have an auto-scaling group with EC2 that implement a custom health.
From time to time, the health check fails and instances are terminated and replaced.
The health check itself is implemented as a shell script that runs on the instances. If it detects problems, it will inform the auto scaling group via the AWS API:
aws autoscaling set-instance-health --instance-id $instance --health-status Unhealthy
The problem is only that I have no information about what check failed, beside the notification:
Cause: At 2017-06-13T09:11:47Z an instance was taken out of service in response to a user health-check
What is the recommended way to debug these type of problems. Is there a way to make AWS only stop instances and not terminate them, so their disk state could be inspected?
(First I thought about "enable termination protection", but from my understanding this will not make a difference, here. Autoscaling group will still terminate the instances when the shutdown was requested by a failing custom health check.)
Using the set-instance-health command tells Auto Scaling that the instance is unhealthy and needs to be replaced. Auto Scaling will then terminate the unhealthy instance and launch a new instance to replace it.
If you wish to perform forensic analysis on an unhealthy instance, remove it from the Auto Scaling group with the aws autoscaling detach-instances command:
Removes one or more instances from the specified Auto Scaling group. After the instances are detached, you can manage them independent of the Auto Scaling group.
If you do not specify the option to decrement the desired capacity, Auto Scaling launches instances to replace the ones that are detached.
If there is a Classic Load Balancer attached to the Auto Scaling group, the instances are deregistered from the load balancer. If there are target groups attached to the Auto Scaling group, the instances are deregistered from the target groups.
So, instead of calling set-instance-health, call detach-instances (and optionally replace it). You can then debug the instance. If you wish to send it back into service, use aws autoscaling attach-instances.
Newbie to AWS, I have a t2.micro EC2 instance supporting my Elastic Beanstalk application. It was created while I deployed my WAR file to Elastic Beanstalk.
My EB application does not need to run 24 hrs a day so it only needs to work during a specific period of time a day. When I tried to stop it from the AWS EC2 management console by selecting "Actions -> Instance state -> Stop", the instance was terminated instead but a new instance was created and initialized and then running automatically later.
How can I stop an EC2 instance properly to suit my need? Thanks.
Your EB application is utilizing an Auto Scaling group to ensure your EC2 instance stays running.
When you stop your EC2 instance, Auto Scaling sees that as an unhealthy instance and terminates it, then replaces it with a new, healthy instance.
You cannot stop EC2 instances that are part of an Auto Scaling group. The best you can do is to edit your Auto Scaling group policy to set min=desired=0. This will terminate the instance and keep it terminated. When you want to get your instance running again, edit your Auto Scaling group and reset min=desired=1.