I am an issue that either I do not understand well or something weird happening with aws ecs service.
I update my code, create a new docker image and push it to be deployed using ECS. The issue is, when the task definition does not changes, the code does not get deployed, even though the image in ECR got updated. How can get my code deployed then? I am assuming that when the image has changed, the services is run the already registered tasks which should pull the image right?
Example of commands I run
aws ecs register-task-definition --cli-input-json file:///deploy/tasks/my-task-definition.json
aws ecs update-service --service my-service --cluster my-clusdter --task-definition my-task-defintion
The first time I run these commands, the code is deployed, if I update my code, push the new image to the Registry, then run these commands, my code does not get deployed.
The updates are pulled every time your ECR gets updated.
Double check the way you're confirming the updated version.
In order to update your container with the updated images, you have to revised your task definition with the latest image from the repository, then your service should be updated with the new task definition you've defined.
It looks like you are on the right track, but I assume that json file is revising your task definition with the same image. If this is the case, you can just change the tag for the image to :latest so that you can run the same commands with the same json every single time.
Related
I am very new to working with AWS but am trying to set up a EC2 service, connected to a github action which deploys my python app to my service.
I am currently creating a ECS cluster [as described by github][1].
During the creation of said cluster the setup asks me for an Image (`repository-url/image:tag`).
What does that mean exactly? I've been looking online for multiple hours but dont understand where I can find said image.
Filling in `12345.dkr.ecr.us-east-2.amazonaws.com/My-Repo:latest` returns a `CannotPullContainerError: inspect image has been retried 1 time(s): failed to resolve ref, not found`.
Could someone help me understand?
Edit: I am completely new to AWS so I apologise if any info is missing and can add whatever is needed to the post.
That would be the docker image (docker image repository and image tag) to deploy to your ECS service. You can't just make that up, it has to be a repository, and image that already exists. You should be creating a docker image that contains your Python app, and pushing that image to an image repository somewhere, such as AWS ECR. You need to be doing that before you look into deploying anything on AWS ECS.
Also, you may be overcomplicating things a lot by using EC2 instead of Fargate.
I'm setting up automated deployments using Azure DevOps to deploy to AWS ECS running services on an EC2 instance (not Fargate). Docker images are loaded into ECR and ECS is fully setup using CloudFormation templates.
After deployments are done (new image loaded and/or any CloudFormation changes applied) I need to restart the service only if it's not running the latest version of the image that the Task Definition uses. If CloudFormation updated the service or task definition then it will automatically restart and I don't want to restart it again. Also, if the docker image was not deployed then I don't want to restart it. So, is there a way to check what version of an image a service task is currently running?
I can't check if the version of a task definition changed because most of the time it would just be the image that changes and not the task definition. I am loading the image using the Latest tag so I can just call ForceNewDeployment on the service to have it deploy the new version, but if it's possible I don't want to do that if nothing has changed.
I initially thought of trying to keep track of it in the deployment process but due to how our release pipeline is setup and some of the limits in Azure DevOps that's not something I'll be able to do.
I think the best solution, if it's possible, would be to check to see what version of an Image is currently running for each task in a service and if an image has an updated version then do a ForceNewDeployment on the service. But...is there a way to accomplish that? Or would there be a better way to do what I'm trying to do?
Edit: When I load the Docker image to ECR I'm adding two tags: "Latest" and the DevOps release number ("1234"). I was initially hoping to just query the ECR images at the end of a release and if one has a tag with the current release number then I know the image has been updated. The issue is that I don't know if the service was already restarted due to a CloudFormation change.
I am trying to migrate a set of microservices from Docker Swarm, to AWS ECS using Fargate.
I have created an ECS cluster. Moreover, I have initialized repositories using the ECR, each of which contains an image of a microservice.
I have successfully came up with a way to create new images, and push them into the ECR. In fact, with each change in the code, a new docker image is built, tagged, and pushed.
Moreover, I have created a task definition that is linked to a service. This task definition contains one container, and all the necessary information. Moreover, its service defines that the task will run in a VPC, and is linked to a load balancer, and has a target group. I am assuming that every new deployment uses the image with the "latest" tag.
So far with what I have explained, everything is clear and is working well.
Below is the part that is confusing me. After every new build, I would like to update the service in order for new tasks with the update image get deployed. I am using the cli to do so with the following command:
aws ecs update-service --cluster <cluster-name> --service <service-name>
Typically, after performing the command, I am monitoring the deployment logs, under the event tab, and checking the state of the service using the following command:
aws ecs describe-services --cluster <cluster-name> --service <service-name>
Finally, I tried to simulate a case where the newly created image contains a bad code. Thus, the new tasks will not be able to get deployed. What I have witnessed is that Fargate will keep trying (without stopping) to deploy the new tasks. Moreover, aside the event logs, the describe-services command does not contain relevant information, other than what Fargate is doing (e.g., registering/deregistering tasks). I am surprised that I could not find any mechanism that instructs Fargate, or the service to stop the deployment and rollback to the already existing one.
I found this article (https://aws.amazon.com/blogs/compute/automating-rollback-of-failed-amazon-ecs-deployments/ ), which provides a solution. However, it is a fairly complicated one, and assumes that each new deployment is triggered by a new task definition, which is not what I want.
Therefore, considering what I have described above, I hope you can answer the following questions:
1) Using CLI commands (For automation purposes) Is there a way to instruct Fargate to automatically stop the current deployment, after failing to deploy new tasks after a few tries?
2) Using the CLI commands, is there a way to monitor the current status of the deployment? For instance, when performing a service update on a service on Docker swarm, the terminal generates live logs on the update process
3) After a failed deployment, is there a way for Fargate to signal an error code, or flag, or message?
At the moment, ECS does not offer deployment status directly. Once you issue a deployment, there is no way to determine its status other than to continually poll for updates until you have enough information to infer from them. Plus unexpected container exits are not logged anywhere. You have to search through failed tasks. The way I get them is by cloudwatch rule that triggers a lambda upon task state change.
I recommend you read: https://medium.com/#aaron.kaz.music/monitoring-the-health-of-ecs-service-deployments-baeea41ae737
As of now, you have a way to do this:
aws ecs wait services-stable --cluster MyCluster --services MyService
The previous example pauses and continues only after it can confirm that the service running on the cluster is stable. Will return 255 exit code after 40 failed checks.
To cancel a deployment, enable ECS Circuit Breaker when creating your service:
aws ecs create-service \
--service-name MyService \
--deployment-configuration "deploymentCircuitBreaker={enable=true,rollback=true}" \
{...}
References:
Service deployment check.
Circuit Breaker
As a newbie to AWS, I have been updating the task definition file every time when I have to update the service.
Say if I have a Task Definition 1 that has a Docker Image, and if I updated the Docker image, refreshing the service will get the latest docker image?
or do I need to update the Task Definition file to let the service pull the latest docker image?
In the AWS ECS, after updating the Docker image, it is necessary to update the task definition and update the relation with the service.
A tool such as ecs-deploy is useful to simplify this type of work.
https://github.com/silinternational/ecs-deploy
If you want to update Docker image frequently, you can further automate such as deploying from CI service.
You could try to use a label within the docker repository like "latest" or "dev" referenced within the task definition.
However, you would not be able to use the update-service deployment orchestration because the task definition version does not change.
With ECS, the task definition version is key to the orchestration of deployments.
I use Docker Hub to store a private Docker image, the repository has a webhook that once the image is updated it calls a service I built to:
update the ECS task definition
update the ECS service
deregister the old ECS task definition
The service is running accordingly. After it runs ECS creates a new task with the new task definition, stops the task with the old task definition and the service come back with the new definition.
The point is that the Docker Image is not updated, once the service starts in the new task definition it remains with the old image.
Am I doing something wrong? How o ensure the docker image is updated?
After analysing the AWS ECS logs I found out that the problem was in the ECS Docker authentication.
To solve that I've added the following data to the file /etc/ecs/ecs.config
ECS_CLUSTER=default
ECS_ENGINE_AUTH_TYPE=dockercfg
ECS_ENGINE_AUTH_DATA={"https://index.docker.io/v1/":{"auth":"YOUR_DOCKER_HUB_AUTH","email":"YOUR_DOCKER_HUB_EMAIL"}}
Just replace the YOUR_DOCKER_HUB_AUTH and YOUR_DOCKER_HUB_EMAIL by your own information and it shall work properly.
To find this information you can execute docker login on your own computer and then look for the data in the file ~/.docker/config.json
For more information on the Private Registry Authentication topic please look at http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html