I am currently using ECS to deploy a web application. This Application is pushed on my ECR with the latest tag but putting new images doesn't seem to change the code at deployment.
I tried to use the answer found here How do I deploy updated Docker images to Amazon ECS tasks? using aws ecs update-service --cluster <cluster name> --service <service name> --force-new-deployment.
I also put ECS_IMAGE_PULL_BEHAVIOR=always in my ecs config and deployment_minimum_healthy_percent = 0 and deployment_maximum_percent = 200 in my deployment settings.
I notice that the image digest is matching the latest image, but by logging on the container, the code is not different from the previous version. But by executing docker pull <my_image:latest> the changes are there.
You need to update task definition first (create new one) and then deploy it.
ECS task definition
CLI for register task
Related
I want to make sure that the task is running the latest image.
Within the container, I can get the docker image ID (such as 183f9552f0a5) by calling http://169.254.170.2/v2/metadata, however I am looking for a way to get it on my laptop.
Is this possible with AWS CLI or SDK?
You first need to get the Task Definition ARN for the Task using describe_tasks. You can skip this step if you already know the ARN.
aws ecs describe-tasks --tasks TASK_ARN
Then you can use describe_task_definition to get the image name.
aws ecs describe-task-definition --task-definition TASKDEF_ARN
I have pushed the initial docker image into repository and created AWS Fargate using the image, is there any way to update the image as there are certain changes in my docker image.
As simple as:
aws ecs update-service --cluster <cluster> --service <service> --force-new-deployment
See AWS documentation:
If you have updated the Docker image of your application, you can create a new task definition with that image and deploy it to your service.
Note
If your updated Docker image uses the same tag as what is in the existing task definition for your service (for example, my_image:latest), you do not need to create a new revision of your task definition. You can update the service using the procedure below, keep the current settings for your service, and select Force new deployment. The new tasks launched by the deployment pull the current image/tag combination from your repository when they start. The Force new deployment option is also used when updating a Fargate task to use a more current platform version when you specify LATEST. For example, if you specified LATEST and your running tasks are using the 1.0.0 platform version and you want them to relaunch using a newer platform version.
https://docs.aws.amazon.com/AmazonECS/latest/userguide/update-service.html#update-service
Create a new version of the task definition and update the container with latest labels and update the service.
You can write the configuration file once you created a cluster using the default-launch-type as FARGATE for your application and define the respective parameters in your task definition i.e. ecs-params.yaml
Here is one file for the nginx:latest image which is stored in Amazon ECR.
version: '2'
services:
web:
image: account-id.dkr.ecr.ap-southeast-1.amazonaws.com/nginx:latest
ports:
- "80:80"
logging:
driver: awslogs
options:
awslogs-group: awslogs-web
awslogs-region: ap-southeast-1
awslogs-stream-prefix: web-nginx
You simply change the image and you could get the updated image into your deployment as you update the service inside your cluster.
If you have updated the Docker image of your application, you can create a new task definition with that image and deploy it to your service. The service scheduler uses the minimum healthy percent and maximum percent parameters (in the service's deployment configuration) to determine the deployment strategy.
Note: The Execution Role in task definition gives permissions to pull the images from container registry.
You could find the doc guide here, AWS ECS Update Service
I am trying to restart an AWS service (basically stop and start all tasks within the service) without making any changes to the task definition.
The reason for this is because the image has the latest tag attached with every build.
I have tried stopping all tasks and having the services recreate them but this means that there is some temporarily unavailable error when the services are restarting in my instances (2).
What is the best way to handle this? Say, A blue-green deployment strategy so that there is no downtime?
This is what I have currently. It'shortcomings is that my app will be down for a couple of seconds as the service's tasks are being rebuilt after deleting them.
configure_aws_cli(){
aws --version
aws configure set default.region us-east-1
aws configure set default.output json
}
start_tasks() {
start_task=$(aws ecs start-task --cluster $CLUSTER --task-definition $DEFINITION --container-instances $EC2_INSTANCE --group $SERVICE_GROUP --started-by $SERVICE_ID)
echo "$start_task"
}
stop_running_tasks() {
tasks=$(aws ecs list-tasks --cluster $CLUSTER --service $SERVICE | $JQ ".taskArns | . []");
tasks=( $tasks )
for task in "${tasks[#]}"
do
[[ ! -z "$task" ]] && stop_task=$(aws ecs stop-task --cluster $CLUSTER --task "$task")
done
}
push_ecr_image(){
echo "Push built image to ECR"
eval $(aws ecr get-login --region us-east-1)
docker push $AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/repository:$TAG
}
configure_aws_cli
push_ecr_image
stop_running_tasks
start_tasks
Use update-service and the --force-new-deployment flag:
aws ecs update-service --force-new-deployment --service my-service --cluster cluster-name
Hold on a sec.
If I understood you usecase correctly, this is addressed in the official docs:
If your updated Docker image uses the same tag as what is in the existing task definition for your service (for example, my_image:latest), you do not need to create a new revision of your task definition. You can update the service using the procedure below, keep the current settings for your service, and select Force new deployment....
To avoid downtime, you should manipulate 2 parameters: minimum healthy percent and maximum percent:
For example, if your service has a desired number of four tasks and a maximum percent value of 200%, the scheduler may start four new tasks before stopping the four older tasks (provided that the cluster resources required to do this are available). The default value for maximum percent is 200%.
This basically mean, that regardless of whether your task definition changed and to what extent, there can be an "overlap" between the old and the new ones, and this is the way to achieve resilience and reliability.
UPDATE:
Amazon has just introduced External Deployment Controllers for ECS(both EC2 and Fargate). It includes a new level of abstraction called TaskSet. I haven't tried it myself yet, but such fine grain control over service and task management(both APIs are supported) can potentially solve the problem akin this one.
After you push your new image to your Docker repository, you can create a new revision of your task definition (it can be identical to the existing task definition) and update your service to use the new task definition revision. This will trigger a service deployment, and your service will pull the new image from your repository.
This way your task definition stays the same (although updating the service to a new task definition revision is required to trigger the image pull), and still uses the "latest" tag of your image, but you can take advantage of the ECS service deployment functionality to avoid downtime.
The fact that I have to create a new revision of my task definition every time even when there is no change in the task definition itself is not right.
There are a bunch of crude bash implementations on this which means that AWS should have the ECS service scheduler listen for changes/updates in the image, especially for an automated build process.
My crude work-around to this was have two identical task definitions and switch between them for every build. That way I don't have redundant revisions.
Here is the specific script snippet that does that.
update_service() {
echo "change task definition and update service"
taskDefinition=$(aws ecs describe-services --cluster $CLUSTER --services $SERVICE | $JQ ".services | . [].taskDefinition")
if [ "$taskDefinition" = "$TASK_DEF_1" ]; then
newDefinition="$TASK_DEF_2"
else
newDefinition="$TASK_DEF_1"
fi
rollUpdate=$(aws ecs update-service --cluster $CLUSTER --service $SERVICE --task-definition $newDefinition)
}
Did you have this question solved? Perhaps this will work for you.
With a new release image pushed to ECR with a version tag, i.e. v1.05, and the latest tag, the image locator in my task definition needed to be explicitly updated to have this version tag postfixed like :v1.05.
With :latest, this new image did not get pulled by the new container after aws ecs update-service --force-new-deployment --service my-service.
I was doing tagging and pushing like this:
docker tag ${imageId} ${ecrRepoUri}:v1.05
docker tag ${imageId} ${ecrRepoUri}:latest
docker push ${ecrRepoUri}
...where as this is the proper way of pushing multiple tags:
docker tag ${imageId} ${ecrRepoUri}
docker push ${ecrRepoUri}:v1.05
docker push ${ecrRepoUri}:latest
This was briefly mentioned in the official docs without a proper example.
Works great https://github.com/fdfk/ecsServiceRestart
python ecsServiceRestart.py restart --services="app app2" --cluster=test
The quick and dirty way:
login to EC2 instance running the task
find your container with docker container list
use docker restart [container]
The use case is like - developer makes some code changes and the below things happen automatically -
build runs, application artifact created, docker image generated with the artifact, image pushed to Docker registry, AWS ECS tasks and ECS services updated.
I want to know what are the ways to achieve the above automation of update of AWS ECS services. Till now I have implemented AWS ECS update from Jenkins build using -
1>run post build AWS CLi scripts from Jenkins to update ECS
2>post build action or pipeline step to invoke AWS Lambda function. I have created one Lambda function in Java to implement that.
Please let me the other ways we can achieve the above. Thanks.
I'm continuously deploying Docker containers from CircleCI to AWS ECS.
The outline of the deployment flow is as follows:
Build and tag a new Docker image
Login to AWS ECR and push the image
Update task definitions and services of ECS with ecs-deploy
ecs-deploy is a useful script that updates Docker images in ECS.
https://github.com/silinternational/ecs-deploy
You could use a shell script that calls aws cli commands to create cloudformation stacks or directly call the create commands in the aws cli for the ECR repository, Task Definition, Events rule and target(for scheduling).
then you just call this script on your terminal using this command: ./setup.sh and it should execute all your commands at once.
aws ecr create-repository \
--repository-name tasks-${TASK_NAME}-${TASK_ENV} \
;
or if you want to set up your resources via cloudformation templates, you can launch them using this command as long as the template exists at file://name.yml:
aws cloudformation create-stack \
--stack-name stack-name \
--capabilities CAPABILITY_IAM \
--template-body file://name.yml \
--parameters
ParameterKey=ParamName,ParameterValue=${PARAM_NAME} \
;
Take a look at Codefresh - https://docs.codefresh.io/docs/amazon-ecs
You can build your pipeline
Build Step
Push to Registry
Deply to ECS
That easy
While there are a ton of CI/CD tools out there, since I am early in my rollout, I decided to write a small script instead of having CI/CD pipelines do it.
Here is a one-click deploy script I wrote using the ecs-deploy script as a dependency to achieve a rolling deploy of a docker image to ECS.
You can run this locally from your dev or build/deployment box or use Jenkins or some local build tool.
#!/bin/bash
# automatically login to AWS
eval $(aws ecr get-login)
# build local docker image and push repo to AWS
docker build -t <yourlocaldockerimagetag> .
docker tag <yourlocaldockerimagetag>:latest <yourECSRepoURL>:latest
docker -D -l debug push <yourECSRepoURL>:latest
# deploy to ECS
ecs-deploy/ecs-deploy -m 50 -k <access-key> -s <secret-key> -r <aws-region> -c <cluster-name> -n <service-name> -i <yourECSRepoURL>:latest
Parameters:
cluster-name: Your cluster name in ECS
service-name: Your service name that you had created in ECS
yourECSRepoURL: ECS Repository URL
yourlocaldockerimagetag: Any local image tag name
access-key: your AWS access key for deployments
secret-key: your AWS secret key
Make sure you install ecs-deploy before this script.
The -m 50 tells it that it can deploy even if the number of nodes drops to 50%. Ideally you would have an extra node to do deployments, but if you can't afford that setting this would ensure that deployments continue to happen.
If you are also using an ELB (load balancer), then the default deregistration delay for target groups is 5 minutes which is a bit excessive. The deregistration delay is the time to wait for existing requests to complete BEFORE ECS sends a SIGTERM or SIGINT to your docker container. You should lower this by going to the Target Groups in EC2 dashboard and click the Edit Attributes to edit it. Otherwise your deployments may take forever.
I think nobody has mentioned CodePipeline from AWS, it really integrates easilly with many AWS Services including ECS and CodeCommit:
Push commit to CodeCommit Repo, triggering the pipeline execution.
(Optional) Configure a Manual Approval step that needs you to take an action before Build.
Run a CodeBuild Project that builds your Dockerfile and push the image to an ECR Repo.
Run a "Deploy" step that deploys to a specific ECS Service. It updates the services with a new Task Definition that points to the new ECR Image.
I have used this flow with BitBucket also, just configure a BitBucket pipeline that pushes all new code to a CodeCommit Repo as a previous step.
Exactly as #minamiyojo and #astav answers, we ended up glueing ecs-deploy with a template engine to power up our CD pipeline with some reusable component, we just open-sourced as well:
https://github.com/GuccioGucci/yoke
Please refer to Motivation section in README, hope this would help your scenario too.
I’m new to both Amazon’s ECS and docker, and I don’t know how to deploy new images.
I currently create a new image in ECR with
NAME_TAG=my-image-name:my-tag-v1
ECR=my-acct-number.dkr.ecr.us-east-1.amazonaws.com
docker build -t $NAME_TAG .
docker tag -f $NAME_TAG $ECR/$NAME_TAG
$(aws ecr get-login --region us-east-1) #log in
docker push $ECR/$NAME_TAG
At this point I don't know how to deploy the new container from ECR to my cluster.
I created the cluster, task and service using a Cloud Formation template, but updating the TaskDefinition image to $ECR/$NAME_TAG and running a stack update eventually times out and fails with a “service did not stabilize” error.
If I push to my-image-name:latest, my cluster instances do pull down the new image, but they don’t run it, and in any case I want to avoid using the mysterious latest tag.
How am I supposed to deploy new images to ECS?
You should be able to deploy your image using a new task definition every time you deploy.
The task definition lets you set the image version using the attribute "image"
"image":"my-acct-number.dkr.ecr.us-east-1.amazonaws.com/my-image-name:my-tag-v1"
In case you want to use only one task definition, you will have to build you image and tag it with whatever is defined in the the definition.