How do I kill a long running deployment in Amazon Opsworks?
We run deployments to an integration environment everytime we commit to our code repo. Our current deployments are taking a long time, which causes deployments to stack on top of each other in Opsworks. We're working on making our deployment process for the application more efficient, but until we get that sorted out, is there an easy way to kill a deployment so we can just run the latest one in the queue?
Unfortunately there is no easy way.
There is no API call to cancel them.
So the only possible approach would be checking on the instance if it's necessary to run the deployment or skip it. You can achieve this with a custom cookbook.
Related
I have an ECS fargate task defined in aws. I would like to run it occasionally as needed.
Is there an easy way to do this?
I have terraform code that defines it as a scheduled task that is disabled. Whenever I want to run it, my procedure is:
Modify the terraform file to enable the task and set the scheduled execution time for five minutes from now.
Deploy the terraform and wait for the task to run.
Undo the terraform changes and redeploy.
This procedure works, but is quite inconvenient. Surely there is a better way to run one-off tasks? I've tried going through the aws web console but it's even worse.
If you want to stick with using the scheduler to run the task, then something like your current process is the only way to achieve that. However it sounds like you don't really want to have the task run on a set schedule at all, instead you only want to run it when needed.
The most direct way to trigger an ECS task to run, is via the RunTask API, which you can trigger from the AWS CLI (which you could wrap in a shell script), or one of the AWS SDKs.
You can try with Lambda. There is a project I wrote Python code with boto3 to run a lot of different tasks in AWS and I'm pretty sure Lambda can solve your problem.
I have running my cluster and task is running.
My need is want to update container image in running task in cluster how to do?
My Image is with latest tag and every time any new changes come will push to ecr on latest tag.
Deploying with the tag latest isn't a best practice because you loose a lot of visibility into what you are doing (e.g. scale out events where you deploy more tasks as part of a service will all end up using LATEST but will be effectively running different versions of the code, etc.).
This pontificating aside, you didn't say if you started your task(s) as standalone using the run-task API or if you started your task(s) as part of a service.
If the former, you need to stop your task and run it again. If the latter, you need to redeploy your service using the --force-new-deployment flag.
I am using AWS CodeDeploy to deploy a binary to an EC2 instance.
When I deploy a new version of the binary, do I need to tell CodeDeploy to kill the old running binary, and if so, what's the best way to do this.
Should I save the pid of the old process to a file & then kill it?
Or, does CodeDeploy automatically kill the old process?
does CodeDeploy automatically kill the old process?
It does not. You have to do it yourself. CD provides special hook for that where you can perform such operation:
ApplicationStop - This deployment lifecycle event occurs even before the application revision is downloaded. You can specify scripts for this event to gracefully stop the application or remove currently installed packages in preparation for a deployment. The AppSpec file and scripts used for this deployment lifecycle event are from the previous successfully deployed application revision.
AWS also provides example of how one can use ApplicationStop to stop wordpress. The other example is SampleApp_Linux.zip which you can download and inspect to check how ApplicationStop is implemented.
I am trying to have run the lastest task definition image built from GitHub deployment (CD). Seems like on AWS it creates a task definition for example "task-api:1", "task-api:2", on was my cluster is still running task-api: 1 even though there is the latest task as a new image has been built. So far I have to manually stop the old one and start a new one . How can I have it automated?
You must wrap your tasks in a service and use rolling updates for automated deployments.
When the rolling update (ECS) deployment type is used for your service, when a new service deployment is started the Amazon ECS service scheduler replaces the currently running tasks with new tasks.
Read: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-ecs.html
This is DevOps, so you need a CI/CD pipeline that will do the rolling updates for you. Look at CodeBuild, CodeDeploy and CodePipeline (and CodeCommit if you integrate your code repository in AWS with your CI/CD)
Read: https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-ecs-ecr-codedeploy.html
This is a complex topic, but it pays off in the end.
Judging from what you have said in the comments:
I created my task via the AWS console, I am running just the task definition on its own without service plus service with task definition launched via the EC2 not target both of them, so in the task definition JSON file on my Github both repositories they are tied to a revision of a task (could that be a problem?).
It's difficult to understand exactly how you have this set up and it'd probably be a good idea for you to go back and understand the services you are using a little better using the guide you are following or AWS documentation. Pushing a new task definition does not automatically update services to use the new definition.
That said, my guess is that you need to update the service in ECS to use the latest task definition. You can do that in many ways:
Through the console (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-service-console-v2.html).
Through the CLI (https://docs.aws.amazon.com/cli/latest/reference/ecs/update-service.html).
Through the IaC like the CDK (https://docs.aws.amazon.com/cdk/api/latest/docs/aws-ecs-readme.html).
This can be automated but you would need to set up a process to automate it.
I would recommend reading some guides on how you could automate deployment and updates using the CDK. Amazon provide a good guide to get you started https://docs.aws.amazon.com/cdk/latest/guide/ecs_example.html.
I am programatically updating the task definition in ECS Service (service has one task) using the update-service api. What is the best way to track the status of this? The API returns immediately and does not wait for the update to complete. One way would be to continuously describe the tasks. Is there a better way?
Is there a better way?
It depends. But an alternative way could by creating dedicated CI/CD pipeline to update your service, especially you this happens regularly. This could be done with AWS CodePipeline with an example of:
Tutorial: Amazon ECS Standard Deployment with CodePipeline
Initial setup can be slow, especially if done for the first time, but once done, you can have a lot of benefits. They include:
automated deployments
notifications if deployment fails or succeeds
clear history of all deployments, and more.