I'm setting up automated deployments using Azure DevOps to deploy to AWS ECS running services on an EC2 instance (not Fargate). Docker images are loaded into ECR and ECS is fully setup using CloudFormation templates.
After deployments are done (new image loaded and/or any CloudFormation changes applied) I need to restart the service only if it's not running the latest version of the image that the Task Definition uses. If CloudFormation updated the service or task definition then it will automatically restart and I don't want to restart it again. Also, if the docker image was not deployed then I don't want to restart it. So, is there a way to check what version of an image a service task is currently running?
I can't check if the version of a task definition changed because most of the time it would just be the image that changes and not the task definition. I am loading the image using the Latest tag so I can just call ForceNewDeployment on the service to have it deploy the new version, but if it's possible I don't want to do that if nothing has changed.
I initially thought of trying to keep track of it in the deployment process but due to how our release pipeline is setup and some of the limits in Azure DevOps that's not something I'll be able to do.
I think the best solution, if it's possible, would be to check to see what version of an Image is currently running for each task in a service and if an image has an updated version then do a ForceNewDeployment on the service. But...is there a way to accomplish that? Or would there be a better way to do what I'm trying to do?
Edit: When I load the Docker image to ECR I'm adding two tags: "Latest" and the DevOps release number ("1234"). I was initially hoping to just query the ECR images at the end of a release and if one has a tag with the current release number then I know the image has been updated. The issue is that I don't know if the service was already restarted due to a CloudFormation change.
Related
I am very new to working with AWS but am trying to set up a EC2 service, connected to a github action which deploys my python app to my service.
I am currently creating a ECS cluster [as described by github][1].
During the creation of said cluster the setup asks me for an Image (`repository-url/image:tag`).
What does that mean exactly? I've been looking online for multiple hours but dont understand where I can find said image.
Filling in `12345.dkr.ecr.us-east-2.amazonaws.com/My-Repo:latest` returns a `CannotPullContainerError: inspect image has been retried 1 time(s): failed to resolve ref, not found`.
Could someone help me understand?
Edit: I am completely new to AWS so I apologise if any info is missing and can add whatever is needed to the post.
That would be the docker image (docker image repository and image tag) to deploy to your ECS service. You can't just make that up, it has to be a repository, and image that already exists. You should be creating a docker image that contains your Python app, and pushing that image to an image repository somewhere, such as AWS ECR. You need to be doing that before you look into deploying anything on AWS ECS.
Also, you may be overcomplicating things a lot by using EC2 instead of Fargate.
We have pushed our docker images (built on .net core) to AWS ECR repository. These docker images are pulled by AWS ECS (we updated task definitons, task service) and new tasks are created. The initial deployment process worked fine. But the following day after we pushed new updated docker images into the ECR repository, and created revisions of the task definitions and updated them on the service, our changes are not reflecting.
We deleted the ECR repository, ECS service and task definition and re-created all of these anew, but still the issue persists.
(we have killed the old tasks and new tasks have been started with the help of ECS service)
Requesting assistance on this.
It's hard to imagine what's going on with this background. In general if you update your task definition with a new revision and make the new revision point to the new image tag, if you deploy the new revision you should see the changes included in the code that belongs to the image with the new tag. If you see the old code is because either the new revision point to an old tag and/or your new revision has the proper new tag but you are deploying an old revision.
If you are new to ECS perhaps a good way to setup all these plumbing would be to use Copilot. You don't have to use it to deploy to production (even though you can) but it could be a good learning exercise to explore how it sets up things. Basically it's a guided way to setup your ECS cluster as well as pipelines.
I have running my cluster and task is running.
My need is want to update container image in running task in cluster how to do?
My Image is with latest tag and every time any new changes come will push to ecr on latest tag.
Deploying with the tag latest isn't a best practice because you loose a lot of visibility into what you are doing (e.g. scale out events where you deploy more tasks as part of a service will all end up using LATEST but will be effectively running different versions of the code, etc.).
This pontificating aside, you didn't say if you started your task(s) as standalone using the run-task API or if you started your task(s) as part of a service.
If the former, you need to stop your task and run it again. If the latter, you need to redeploy your service using the --force-new-deployment flag.
I'm using GitHub, Jenkins, AWS ECR, AWS ECS.
I want to deploy automatically when GitHub has a new commit.
When GitHub, have new commit, GitHub, sent webhook to Jenkins, Jenkins build images and push to ECR with tag 'latest'.
I wonder how can I make my ECS service restart task and redeploy images automatically when ECR image changed?
Don't use latest in this setup. Have Jenkins pick a tag for the image (maybe based off a source control commit ID, a source control tag name, or a timestamp). Give it the ability to update the ECS tasks, and then (once a build has happened and gone through appropriate pre-launch testing) have Jenkins change the image tag in the task to what it's just built. ECS will see that the image has changed, pull the new image, and launch containers accordingly.
Two other good reasons to do things this way: if you have explicit versions, you can have a pre-production cluster, deploy things there, run tests, and then deploy the same version to production; and if a deploy goes bad, you can straightforwardly roll back by manually setting the tag back to yesterday's build, which is impossible if the only version you have is latest.
Hu guys,
I have ec2 cluster with service and instance. Task is based on latest version of docker file which is allocated in ecr. Now I'm looking for simplest way to finish my pipeline with auto "refresh" service when latest image has been deployed. I can't find any feature from aws to resolve this problem, but I found this: https://github.com/fdfk/ecsServiceRestart but unfortunately it doesn't work (can't communicate with my service). But this case inspired me very much because according to author's approach this solution make a duplicate service before update so it provide something like HA without any downtime. Guys is it possible to go throughout these steps without any downtime at all?
deploy new version of image,
service detect new version of image,
auto refresh with implementation new version
Finally I found the best way to achieve my goal. So it was very easy - I just have used ecs-deploy https://github.com/fabfuel/ecs-deploy which I have adopted to my pipeline. I set up longer timeout with no warning flag and this script do for me everything what exactly need. In my example I have one cluster with 3 instances and 1 service witch two running tasks (two the same nodes behind load balancer). When I update my docker image in ECR, ecs-deploy runs auto update first instance, and according to blue-green deployment it updates next instances one by one with load balancer links too. So in this way I achieved full automated deployment after accepting merge request (of course I skipped few steps in this describe). I hope that this will be helpful for somebody. Cheers!