Is it possible to have exactly one task running in AWS ECS at all time? I don't want to have Blue/Green kind of deployment.
My Requirement:
Min/Desired/Max task = 1;
When I redeploy ECS service, then it should first stop old task and then spin new task. Currently it does opposite.
Any reference would be helpful.
Yes it is possible.
You can create an ECS Service with Number of Tasks as 1 that will set the desired count to 1.
Since you want only 1 task and that should stop and then a new one should come, you can modify the Deployment Configuration with below values:
Minimum Healthy Percent - 0
Maximum Percent - 100
With Desired Count as 1, Minimum Healthy Percent as 0 and Maximum Percent as 100, ECS Service will kill the already running task and then create a new task.
Note: Service will de down during this time.
To explain the behavior you noticed, The default values are
Minimum Healthy Percent - 100
Maximum Percent - 200
and with Desired Count as 1, in this case ECS Service will maintain one running task at all times since Minimum Healthy Percent is 100 i.e. 100% of 1 is 1. However, Maximum Percent as 200 allows ECS Service to create another task as 200% of 1 is 2. So a new task is started first and once this task is stable the old task is stopped.
Related
I created a capacity provider for a particular ecs cluster(t2.small) and it got attached to auto-scaling-group also.
Now I am running 2 tasks with similar resource which utilizes t2.small completely. This is a batch job which runs for max 5 seconds. Now when I set desired count of tasks in service to 4 or 6 my cluster never scales out. According to https://aws.amazon.com/blogs/containers/deep-dive-on-amazon-ecs-cluster-auto-scaling/ , my CapacityProviderReservation should go 200% when desired count increases to 4. But average and maximum CapacityProviderReservation is not going beyond 100%.
I have already set autoscaling policy for service, but still I am not able to scale instances using capacity providers. Can anyone explain me how to implement it
In that deep dive article, it explains:
CapacityProviderReservation = M / N * 100
Where:
M = the number of instances the cluster should have
N = the number of instances currently in the cluster
one of the factors to calculate "M", is the current number of tasks plus the tasks in the "PROVISIONING" state.
In order for a service to go into the "PROVISIONING" state, the service has to also be configured to use a capacity provider.
Once I converted a bunch of services over and scaled them up, there was no capacity for them and instead of failing instantly, they went into the "PROVISIONING" state.
At this point, CapacityProviderReservation went above 100% and the cluster scaled to meet the needs.
If a service is using the rolling update (ECS) deployment type, the
minimum healthy percent represents a lower limit on the number of
tasks in a service that must remain in the RUNNING state during a
deployment, as a percentage of the desired number of tasks (rounded up
to the nearest integer). The parameter also applies while any
container instances are in the DRAINING state if the service contains
tasks using the EC2 launch type. This parameter enables you to deploy
without using additional cluster capacity. For example, if your
service has a desired number of four tasks and a minimum healthy
percent of 50%, the scheduler may stop two existing tasks to free up
cluster capacity before starting two new tasks. Tasks for services
that do not use a load balancer are considered healthy if they are in
the RUNNING state. Tasks for services that do use a load balancer are
considered healthy if they are in the RUNNING state and they are
reported as healthy by the load balancer. The default value for
minimum healthy percent is 100%.
(https://stackoverflow.com/a/40741816/433570, https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-service.html)
So If I have single instance running,
If I have minimum percentage below 50%, the running task will be killed before creating new task.
If I have minimum percentage above or equal to 50% , new deployment won't be deployed, because it is stuck at having 1 instance.
I have to temporarily increase the desired instance count to 2 and do a deployement if I want to deploy a task without service downtime. correct?
I have an ECS cluster with a service in it that is running a task I have defined. It's just a simple flask server as I'm learning how to use ECS. Now I'm trying to understand how to update my app and have it seamlessly deploy.
I start with the flask server returning Hello, World! (rev=1).
I modify my app.py locally to say Hello, World! (rev=2)
I rebuild the docker image, and push to ECR
Since my image is still named image_name:latest, I can simply update the service and force a new deployment with: aws ecs update-service --force-new-deployment --cluster hello-cluster --service hello-service
My minimum percent is set to 100 and my maximum is set to 200% (using rolling updates), so I'm assuming that a new EC2 instance should be set up while the old one is being shutdown. What I observe (continually refreshing the ELB HTTP endpoint) is that that the rev=? in the message alternates back and forth: (rev=1) then (rev=2) without fail (round robin, not randomly).
Then after a little bit (maybe 30 secs?) the flipping stops and the new message appears: Hello, World! (rev=2)
Throughout this process I've noticed that no more EC2 instances have been started. So all this must have been happening on the same instance.
What is going on here? Is this the correct way to update an application in ECS?
This is the normal behavior and it's linked to how you configured your minimum and maximum healthy percent.
A minimum healthy percent of 100% means that at every moment there must be at least 1 task running (for a service that should run 1 instance of your task). A maximum healthy percent of 200% means that you don't allow more than 2 tasks running at the same time (again for a service that should run 1 instance of your task). This means that during a service update ECS will first launch a new task (reaching the maximum of 200% and avoiding to go below 100%) and when this new task is considered healthy it will remove the old one (back to 100%). This explains why both tasks are running at the same time for a short period of time (and are load balanced).
This kind of configuration ensures maximum availability. If you want to avoid this, and can allow a small downtime, you can configure your minimum to 0% and maximum to 100%.
About your EC2 instances: they represent your "cluster" = the hardware that your service use to launch tasks. The process described above happens on this "fixed" hardware.
Let's say, I have a ECS service running. It is configured to run at minimum 2 tasks, desired count is 2 and maximum tasks count is 10. Minimum healthy percent is set at 100, max percent set to 150.
The ECS service is managed using CloudFormation template.
If I update my ECS service CloudFormation template by changing the docker image tag I want to use and apply those changes to the stack - a new deployment is started.
Assume, that before the deployment, the application was under a high load and tasks count was increased to the maximum of 10 running tasks by auto-scaling.
What I expect from the ECS service deployment in such conditions is the following:
10 old tasks are running.
5 new tasks are spawned.
5 old tasks are set to draining state.
5 old tasks are stopped.
5 new tasks are spawned.
Remaining 5 old tasks are set to draining state.
5 old tasks stopped.
What I get is:
8 old tasks are set to draining state.
Desired count of new deployment set to 10 by auto-scaling.
5 new tasks are spawned.
8 old tasks are stopped.
5 new tasks are spawned.
2 old tasks set to draining state.
2 old tasks stopped.
The issue is, that ECS does not take into the account the number of old tasks running before the deployment. It tries to deploy with the desired count of 2, which is set originally in the CloudFormation template. This leads to unexpected termination of 8 tasks before the auto-scaling kicks in.
Is there a way to achieve something like the "expected" scenario during the new version deployment if the service is updated over CloudFormation? I know, I could manually change the desired count in the template before the deployment to the number of tasks, that is currently running. Maybe there is a better way?
It appears, that it is possible to not provide the desired count. Then, then the new deployment uses the old desired count.
If your desired count is 10 at a particular moment and you set minimum health percentage = 150 and the maximum percentage is 200. Then It will spawn 5 more task and initiate the draining of 5 old tasks.
Also, you don't have to change the desired count and take care of the task which is in draining state. It will be drained once the connection will be closed.
I already have the experience in Docker and EC2. But I'm new to ECS. Can someone help me to understand what these two parameters actually does, their difference and usage.
Official Docs says:
The minimum healthy percent represents a lower limit on the number of your service's tasks that must remain in the RUNNING state during a deployment, as a percentage of the desired number of tasks (rounded up to the nearest integer). This parameter enables you to deploy without using additional cluster capacity. For example, if your service has a desired number of four tasks and a minimum healthy percent of 50%, the scheduler may stop two existing tasks to free up cluster capacity before starting two new tasks. Tasks for services that do not use a load balancer are considered healthy if they are in the RUNNING state; tasks for services that do use a load balancer are considered healthy if they are in the RUNNING state and the container instance it is hosted on is reported as healthy by the load balancer. The default value for minimum healthy percent is 50% in the console and 100% for the AWS CLI, the AWS SDKs, and the APIs.
The maximum percent parameter represents an upper limit on the number of your service's tasks that are allowed in the RUNNING or PENDING state during a deployment, as a percentage of the desired number of tasks (rounded down to the nearest integer). This parameter enables you to define the deployment batch size. For example, if your service has a desired number of four tasks and a maximum percent value of 200%, the scheduler may start four new tasks before stopping the four older tasks (provided that the cluster resources required to do this are available). The default value for maximum percent is 200%.
I still didn't get a clear idea about these two parameters.
With minimum healthy percent, if my service has a desired number of 4 tasks and a minimum healthy percent of 25%, how many existing/new tasks will the scheduler start/stop?
With maximum percent, if my service has a desired number of 4 tasks and a maximum percent value of 50%, how many existing/new tasks will the scheduler start/stop?
If my service has only one task running, how to set these parameters to get existing task stopped and new task running.
With desired tasks at 4 and minimum health at 25% then during deployment ECS is allowed to stop all tasks except 1 before it starts new tasks
Maximum should be 100+ so that configuration does not make sense. But if you have 4 desired tasks and maximum at 150% then ECS is allowed to start 2 new tasks before it stops other tasks.
If you want to make sure there is never more than 1 task running during redeployment then you need to set desired to 1, minimum to 0% and maximum to 100%.
If you want to make sure there is always at least 1 task running you need to set desired to 1, minimum to 100% and maximum to 200%.
Other example. If you have desired at 4, minimum at 50% and maximum of 150%. Then ECS can decide what it will do during deployment.
If the cluster does not have more resources it can decide to stop 2 tasks, start 2 new tasks in the new version, wait for the new task to be healthy. Then stop the two remaining tasks and start two new ones.
If the cluster has more resources then ECS can decide to start two new tasks before stopping of existing tasks.
Or you could look at it this way. During redeployment ECS needs to run between MinimumPercent/100*desired and MaximumPercent/100*desired tasks. In this case between 2-6 tasks.