We have pushed our docker images (built on .net core) to AWS ECR repository. These docker images are pulled by AWS ECS (we updated task definitons, task service) and new tasks are created. The initial deployment process worked fine. But the following day after we pushed new updated docker images into the ECR repository, and created revisions of the task definitions and updated them on the service, our changes are not reflecting.
We deleted the ECR repository, ECS service and task definition and re-created all of these anew, but still the issue persists.
(we have killed the old tasks and new tasks have been started with the help of ECS service)
Requesting assistance on this.
It's hard to imagine what's going on with this background. In general if you update your task definition with a new revision and make the new revision point to the new image tag, if you deploy the new revision you should see the changes included in the code that belongs to the image with the new tag. If you see the old code is because either the new revision point to an old tag and/or your new revision has the proper new tag but you are deploying an old revision.
If you are new to ECS perhaps a good way to setup all these plumbing would be to use Copilot. You don't have to use it to deploy to production (even though you can) but it could be a good learning exercise to explore how it sets up things. Basically it's a guided way to setup your ECS cluster as well as pipelines.
Related
I am trying to have run the lastest task definition image built from GitHub deployment (CD). Seems like on AWS it creates a task definition for example "task-api:1", "task-api:2", on was my cluster is still running task-api: 1 even though there is the latest task as a new image has been built. So far I have to manually stop the old one and start a new one . How can I have it automated?
You must wrap your tasks in a service and use rolling updates for automated deployments.
When the rolling update (ECS) deployment type is used for your service, when a new service deployment is started the Amazon ECS service scheduler replaces the currently running tasks with new tasks.
Read: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-ecs.html
This is DevOps, so you need a CI/CD pipeline that will do the rolling updates for you. Look at CodeBuild, CodeDeploy and CodePipeline (and CodeCommit if you integrate your code repository in AWS with your CI/CD)
Read: https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-ecs-ecr-codedeploy.html
This is a complex topic, but it pays off in the end.
Judging from what you have said in the comments:
I created my task via the AWS console, I am running just the task definition on its own without service plus service with task definition launched via the EC2 not target both of them, so in the task definition JSON file on my Github both repositories they are tied to a revision of a task (could that be a problem?).
It's difficult to understand exactly how you have this set up and it'd probably be a good idea for you to go back and understand the services you are using a little better using the guide you are following or AWS documentation. Pushing a new task definition does not automatically update services to use the new definition.
That said, my guess is that you need to update the service in ECS to use the latest task definition. You can do that in many ways:
Through the console (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-service-console-v2.html).
Through the CLI (https://docs.aws.amazon.com/cli/latest/reference/ecs/update-service.html).
Through the IaC like the CDK (https://docs.aws.amazon.com/cdk/api/latest/docs/aws-ecs-readme.html).
This can be automated but you would need to set up a process to automate it.
I would recommend reading some guides on how you could automate deployment and updates using the CDK. Amazon provide a good guide to get you started https://docs.aws.amazon.com/cdk/latest/guide/ecs_example.html.
I wanted to run my docker image in AWS Fargate, so I followed this simple tutorial. Basically, I created an image on my computer and pushed it to the ECR. The tutorial (and AWS "Push commands" cheat sheet as well) suggests to tag the image as latest.
So I did, I tagged it as "latest", I pushed it to ECR, I created the Cluster and Task and successfully run this image in Fargate. But now I'm confused, because I fixed something in my image and would like to deploy new version. How should I tag the new image?
Should I rename the previous image, and the new one should be "latest" again? All tutorial I found a teaching how to deploy first image to the ECR. But I can't find anything that would show how to deploy new versions?
Thanks,
You don't have to rename the previous image, though you should if you want to keep it. Tagging something as latest when there is another image that is latest will replace the old image with the new. Any new instances of you task in ECS will pick up the new image, but ECS will not automatically replace existing tasks. You have to manually do that (obviously you can do it programmatically).
Please keep in mind that tutorials like this are not always designed for production workloads. They are designed to get you working. This is not the best way to manage ECS task updates.
I'm setting up automated deployments using Azure DevOps to deploy to AWS ECS running services on an EC2 instance (not Fargate). Docker images are loaded into ECR and ECS is fully setup using CloudFormation templates.
After deployments are done (new image loaded and/or any CloudFormation changes applied) I need to restart the service only if it's not running the latest version of the image that the Task Definition uses. If CloudFormation updated the service or task definition then it will automatically restart and I don't want to restart it again. Also, if the docker image was not deployed then I don't want to restart it. So, is there a way to check what version of an image a service task is currently running?
I can't check if the version of a task definition changed because most of the time it would just be the image that changes and not the task definition. I am loading the image using the Latest tag so I can just call ForceNewDeployment on the service to have it deploy the new version, but if it's possible I don't want to do that if nothing has changed.
I initially thought of trying to keep track of it in the deployment process but due to how our release pipeline is setup and some of the limits in Azure DevOps that's not something I'll be able to do.
I think the best solution, if it's possible, would be to check to see what version of an Image is currently running for each task in a service and if an image has an updated version then do a ForceNewDeployment on the service. But...is there a way to accomplish that? Or would there be a better way to do what I'm trying to do?
Edit: When I load the Docker image to ECR I'm adding two tags: "Latest" and the DevOps release number ("1234"). I was initially hoping to just query the ECR images at the end of a release and if one has a tag with the current release number then I know the image has been updated. The issue is that I don't know if the service was already restarted due to a CloudFormation change.
So we are launching an ecommerce store built on magento. We are looking to deploy it on Amazon EC2 instance using RDS as database service and using amazon auto-scaling and elastic load balancer to scale the application when needed.
What I don't understand is this:
I have installed and configured my production magento enviorment on an EC2 instance (database is in RDS). This much is working fine. But now when I want to dynamically scale the number of instances
how will I deploy the code on the dynamically generated instances each time?
Will aws copy the whole instance assign it a new ip and spawn it as a
new instance or will I have to write some code to automate this
process?
Plus will it not be an overhead to pull code from git and deploy every time a new instance is spawned?
A detailed explanation or direction towards some resources on the topic will be greatly appreciated.
You do this in the AutoScalingGroup Launch Configuration. There is a UserData section in the LaunchConfiguration in CloudFormation where you would write a script that is ran when ever the ASG scales up and deploys a new instance.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-launchconfig.html#cfn-as-launchconfig-userdata
This is the same as the UserData section in an EC2 Instance. You can use LifeCycle hooks that will tell the ASG not to put the EC2 instance into load until everything you want to have configured it set up.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-as-lifecyclehook.html
I linked all CloudFormation pages, but you may be using some other CI/CD tool for deploying your infrastructure, but hopefully that gets you started.
To start, do check AWS CloudFormation. You will be creating templates to design how the infrastructure of your application works ~ infrastructure as code. With these templates in place, you can rollout an update to your infrastructure by pushing changes to your templates and/or to your application code.
In my current project, we have a github repository dedicated for these infrastructure templates and a separate repository for our application code. Create a pipeline for creating AWS resources that would rollout an updated to AWS every time you push to the repository on a specific branch.
Create an infrastructure pipeline
have your first stage of the pipeline to trigger build whenever there's code changes to your infrastructure templates. See AWS CodePipeline and also see AWS CodeBuild. These aren't the only AWS resources you'll be needing but those are probably the main ones, of course aside from this being done in cloudformation template as mentioned earlier.
how will I deploy the code on the dynamically generated instances each time?
Check how containers work, it would be better and will greatly supplement on your learning on how launching new version of application work. To begin, see docker, but feel free to check any resources at your disposal
Continuation with my current project: We do have a separate pipeline dedicated for our application, but will also get triggered after our infrastructure pipeline update. Our application pipeline is designed to build a new version of our application via AWS Codebuild, this will create an image that will become a container ~ from the docker documentation.
we have two triggers or two sources that will trigger an update rollout to our application pipeline, one is when there's changes to infrastructure pipeline and it successfully built and second when there's code changes on our github repository connected via AWS CodeBuild.
Check AWS AutoScaling , this areas covers the dynamic launching of new instances, shutting down instances when needed, replacing unhealthy instances when needed. See also AWS CloudWatch, you can design criteria with it to trigger scaling down/up and/or in/out.
Will aws copy the whole instance assign it a new ip and spawn it as a new instance or will I have to write some code to automate this process?
See AWS ElasticLoadBalancing and also check out more on AWS AutoScaling. On the automation process, if ever you'll push through with CloudFormation, instance and/or containers(depending on your design) will be managed gracefully.
Plus will it not be an overhead to pull code from git and deploy every time a new instance is spawned?
As mentioned, earlier having a pipeline for rolling out new versions of your application via CodeBuild, this will create an image with the new code changes and when everything is ready, it will be deployed ~ becomes a container. The old EC2 instance or the old container( depending on how you want your application be deployed) will be gracefully shut down after a new version of your application is up and running. This will give you zero downtime.
I'm using GitHub, Jenkins, AWS ECR, AWS ECS.
I want to deploy automatically when GitHub has a new commit.
When GitHub, have new commit, GitHub, sent webhook to Jenkins, Jenkins build images and push to ECR with tag 'latest'.
I wonder how can I make my ECS service restart task and redeploy images automatically when ECR image changed?
Don't use latest in this setup. Have Jenkins pick a tag for the image (maybe based off a source control commit ID, a source control tag name, or a timestamp). Give it the ability to update the ECS tasks, and then (once a build has happened and gone through appropriate pre-launch testing) have Jenkins change the image tag in the task to what it's just built. ECS will see that the image has changed, pull the new image, and launch containers accordingly.
Two other good reasons to do things this way: if you have explicit versions, you can have a pre-production cluster, deploy things there, run tests, and then deploy the same version to production; and if a deploy goes bad, you can straightforwardly roll back by manually setting the tag back to yesterday's build, which is impossible if the only version you have is latest.