I'm trying to understand from Tutorials and AWS Documentation how the ECS Rolling update works. I'm really confused because I can't solve a small problem.
I am using Gitlab CI/CD and deploying in ECS. I do not use Load Balancer yet, but I think I have to because I'm having a downtime problem. I am using only 1 task for our application, I've set the min and max healthy percent to 0/200 (I think it's false) and when I deploy using:
- aws ecs update-service --region "${REGION}" --cluster "${CLUSTER_NAME}" --service "${SERVICE_NAME}" --task-definition "$TASK_DEFINTION_NAME":${REV} --force-new-deployment
It firsts stops the running task and then starts a new task. Until the service reaches a steady state I can't access my website! Its like 30-40 sec Downtime.
How can I solve this? Should I use Blue/Green Deployment or I am doing something wrong?
Related
I am currently using ECS to deploy a web application. This Application is pushed on my ECR with the latest tag but putting new images doesn't seem to change the code at deployment.
I tried to use the answer found here How do I deploy updated Docker images to Amazon ECS tasks? using aws ecs update-service --cluster <cluster name> --service <service name> --force-new-deployment.
I also put ECS_IMAGE_PULL_BEHAVIOR=always in my ecs config and deployment_minimum_healthy_percent = 0 and deployment_maximum_percent = 200 in my deployment settings.
I notice that the image digest is matching the latest image, but by logging on the container, the code is not different from the previous version. But by executing docker pull <my_image:latest> the changes are there.
You need to update task definition first (create new one) and then deploy it.
ECS task definition
CLI for register task
I have ESC service with EC2 task running on an EC2 instance. The image in the EC2 task is from the ECR uri: for example: 688523422345.dkr.ecr.us-west-1.amazonaws.com/image, I noticed that when I load this image into my EC2 task I just directly using the uri:688523422345.dkr.ecr.us-west-1.amazonaws.com/image:latest, because the image uri never changes and I just keep push image to update it.
However, when the image did updated on ECR, the task and service running on EC2 instance doesn't updating. I wondering why, and search on stack overflow, people told me that using aws ecs update-service --cluster <cluster name> --service <service name> --force-new-deployment to force the service to re-deploy. However, I just got error on not enough memory left on the instance(seems the re-deployment will create new task and it keep taking more memories, not a good solution).
How can I solve this?
This could be because of your Deployment configuration and the parameters:
maximumPercent
minimumHealthyPercent
By default minimumHealthyPercent is 100% which means that replacement operation will first attempt to run new tasks, before terminating old ones, potentially resulting in out of memory error. You can set it up to minimumHealthyPercent to 0 and maximumPercent to 100 as to force termination of existing tasks first, before creating new ones.
It's not working, after tried a lot. I find out is because the EC2 instance already stored all the information from a task(even if deleted task, the instance is still running with the image). The right way to do it is to re-start the instance.
I used aws-cli: aws ec2 reboot-instances --instance-ids <instance_id>
It worked!
I have this bash script:
#!/bin/bash
myClusterId="myCluster"
for service in $(aws ecs list-services --cluster $myClusterId --query "serviceArns[*]" | jq -r 'to_entries[] | .value | sub(".*/";"")'); do
for task in $( aws ecs list-tasks --cluster $myClusterId --service-name $service --desired-status 'RUNNING' --no-paginate --output text --query 'taskArns[*]' ) ; do
aws ecs stop-task --cluster $myClusterId --task $task --reason "Restarted using bash script" > /dev/null 2>&1
done
done
In short, it will restart all my ECS Fargate tasks under myCluster (excluding scheduled tasks triggered by CloudWatch Rules). It's working fine so far.
All my services have minHealthyPercent set to 100 and maxHealthyPercent set to 200. But, I noticed that it didn't keep any healthy tasks during the restart process. All tasks get killed immedietly and my load balancer throws 503 Service Temporarily Unavailable error when new tasks are in pending/provisioning state.
Am I missing something in my script? How do I correctly perform no-downtime services restart process using AWS CLI?
The parameters maximumPercent and minimumHealthyPercent are only used during rolling updates of your ECS service:
The number of tasks that Amazon ECS adds or removes from the service during a rolling update is controlled by the deployment configuration. A deployment configuration consists of the minimum and maximum number of tasks allowed during a service deployment.
Restarting a task is not considered as a new deployment.
To rectify the issue, there are few choices:
include a sleep in your for loop. Its the crudest way, but fastest to implement for testing.
use describe-tasks in the for loop to pull the state of the task just terminated. Proceed with restarting next task only when the state of the most recently restarted one will be RUNNING.
I think your best option would be to do a blue/green deployment through CodeDeploy, assuming you use an Elastic Load Balancer. The blue/green deployment will automatically detect any error and stop the deployment if required.
https://aws.amazon.com/blogs/devops/use-aws-codedeploy-to-implement-blue-green-deployments-for-aws-fargate-and-amazon-ecs/
Haven't been able to find this in docs. Can I just pause an ECS service so it stopped creating new tasks? Or do I have to delete it to stop that behavior?
I just want to temporarily suspend it from creating new tasks on the cluster
It is enough to set the desired number of tasks for a service to 0.
ECS will automatically remove all running tasks.
aws ecs update-service --desired-count 0 --cluster "ecs-my-ClusterName" --service "service-my-ServiceName-117U7OHVC5NJP"
You can accomplish a "pause" by adjusting your service configuration to match your current number of running tasks. For example, if you currently have 3 running tasks in your service, you'd configure the service as below:
This tells the service:
The number of tasks I want is [current-count]
I want you to maintain at least [current-count]
I don't want more than [current-count
These combined effectively halt your service from making any changes.
The accepted answer is incorrect.
If you set both "Minimum healthy percent" and "Maximum healthy percent" to 100, AWS will give you an error similar to following:
To stop service from creating new tasks, you have to update service by updating task definition and setting desired number of tasks to 0. After that you can use AWS CLI (fastest option) to stop existing running tasks , for example:
aws ecs list-services --cluster "ecs-my-ClusterName"
aws ecs list-tasks --cluster "ecs-my-ClusterName" --service "service-my-ServiceName-117U7OHVC5NJP"
After that you will get the list of the running tasks for the service, such as:
{
"taskArns": [
"arn:aws:ecs:us-east-1:XXXXXXXXXXX:task/12e13d93-1e75-4088-a7ab-08546d69dc2c",
"arn:aws:ecs:us-east-1:XXXXXXXXXXX:task/35ed484a-cc8f-4b5f-8400-71e40a185806"
]
}
Finally use below to stop each task:
aws ecs stop-task --cluster "ecs-my-ClusterName" --task 12e13d93-1e75-4088-a7ab-08546d69dc2c
aws ecs stop-task --cluster "ecs-my-ClusterName" --task 35ed484a-cc8f-4b5f-8400-71e40a185806
UPDATE: By setting the desired number of running tasks to 0, ECS will stop and drain all running tasks in that service. There is no need to stop them individually afterwards using CLI commands originally posted above.
I am trying to restart an AWS service (basically stop and start all tasks within the service) without making any changes to the task definition.
The reason for this is because the image has the latest tag attached with every build.
I have tried stopping all tasks and having the services recreate them but this means that there is some temporarily unavailable error when the services are restarting in my instances (2).
What is the best way to handle this? Say, A blue-green deployment strategy so that there is no downtime?
This is what I have currently. It'shortcomings is that my app will be down for a couple of seconds as the service's tasks are being rebuilt after deleting them.
configure_aws_cli(){
aws --version
aws configure set default.region us-east-1
aws configure set default.output json
}
start_tasks() {
start_task=$(aws ecs start-task --cluster $CLUSTER --task-definition $DEFINITION --container-instances $EC2_INSTANCE --group $SERVICE_GROUP --started-by $SERVICE_ID)
echo "$start_task"
}
stop_running_tasks() {
tasks=$(aws ecs list-tasks --cluster $CLUSTER --service $SERVICE | $JQ ".taskArns | . []");
tasks=( $tasks )
for task in "${tasks[#]}"
do
[[ ! -z "$task" ]] && stop_task=$(aws ecs stop-task --cluster $CLUSTER --task "$task")
done
}
push_ecr_image(){
echo "Push built image to ECR"
eval $(aws ecr get-login --region us-east-1)
docker push $AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/repository:$TAG
}
configure_aws_cli
push_ecr_image
stop_running_tasks
start_tasks
Use update-service and the --force-new-deployment flag:
aws ecs update-service --force-new-deployment --service my-service --cluster cluster-name
Hold on a sec.
If I understood you usecase correctly, this is addressed in the official docs:
If your updated Docker image uses the same tag as what is in the existing task definition for your service (for example, my_image:latest), you do not need to create a new revision of your task definition. You can update the service using the procedure below, keep the current settings for your service, and select Force new deployment....
To avoid downtime, you should manipulate 2 parameters: minimum healthy percent and maximum percent:
For example, if your service has a desired number of four tasks and a maximum percent value of 200%, the scheduler may start four new tasks before stopping the four older tasks (provided that the cluster resources required to do this are available). The default value for maximum percent is 200%.
This basically mean, that regardless of whether your task definition changed and to what extent, there can be an "overlap" between the old and the new ones, and this is the way to achieve resilience and reliability.
UPDATE:
Amazon has just introduced External Deployment Controllers for ECS(both EC2 and Fargate). It includes a new level of abstraction called TaskSet. I haven't tried it myself yet, but such fine grain control over service and task management(both APIs are supported) can potentially solve the problem akin this one.
After you push your new image to your Docker repository, you can create a new revision of your task definition (it can be identical to the existing task definition) and update your service to use the new task definition revision. This will trigger a service deployment, and your service will pull the new image from your repository.
This way your task definition stays the same (although updating the service to a new task definition revision is required to trigger the image pull), and still uses the "latest" tag of your image, but you can take advantage of the ECS service deployment functionality to avoid downtime.
The fact that I have to create a new revision of my task definition every time even when there is no change in the task definition itself is not right.
There are a bunch of crude bash implementations on this which means that AWS should have the ECS service scheduler listen for changes/updates in the image, especially for an automated build process.
My crude work-around to this was have two identical task definitions and switch between them for every build. That way I don't have redundant revisions.
Here is the specific script snippet that does that.
update_service() {
echo "change task definition and update service"
taskDefinition=$(aws ecs describe-services --cluster $CLUSTER --services $SERVICE | $JQ ".services | . [].taskDefinition")
if [ "$taskDefinition" = "$TASK_DEF_1" ]; then
newDefinition="$TASK_DEF_2"
else
newDefinition="$TASK_DEF_1"
fi
rollUpdate=$(aws ecs update-service --cluster $CLUSTER --service $SERVICE --task-definition $newDefinition)
}
Did you have this question solved? Perhaps this will work for you.
With a new release image pushed to ECR with a version tag, i.e. v1.05, and the latest tag, the image locator in my task definition needed to be explicitly updated to have this version tag postfixed like :v1.05.
With :latest, this new image did not get pulled by the new container after aws ecs update-service --force-new-deployment --service my-service.
I was doing tagging and pushing like this:
docker tag ${imageId} ${ecrRepoUri}:v1.05
docker tag ${imageId} ${ecrRepoUri}:latest
docker push ${ecrRepoUri}
...where as this is the proper way of pushing multiple tags:
docker tag ${imageId} ${ecrRepoUri}
docker push ${ecrRepoUri}:v1.05
docker push ${ecrRepoUri}:latest
This was briefly mentioned in the official docs without a proper example.
Works great https://github.com/fdfk/ecsServiceRestart
python ecsServiceRestart.py restart --services="app app2" --cluster=test
The quick and dirty way:
login to EC2 instance running the task
find your container with docker container list
use docker restart [container]