Is there a way to see the history of task definitions for an ECS service? It is easy to see the current task definition for the service as well as the current task definition for the tasks. But I can't see a way to see what the previous task definition registered with the service was. I can easily see all the task definitions I have, but I don't know which ones were registered to which service.
We can usually see the version because of the history of jobs in our Jenkins box. But we recently had a situation where Jenkins didn't have the history for the one we wanted to roll back to. We ended up guessing right because incrementing numbers are easy to guess. But I don't like that we had to guess. I couldn't see it in the CloudWatch logs. I could see the auto-scaling events there, but not task definition changes.
Related
I saw the post How to manage versions of docker image in AWS ECS? and didn’t get a good answer for the question.
In case of updating container image version (for example , from alpine:1.0.0 to alpine 1.0.1)
What is the best practice to update the container image in the task definition? I’m using only one container per task definition.
As far as I understand there are two alternatives:
Create new revision of task definition
Create new task definition that its name contains the version of the image.
The pros of the first option are that I’m creating only one task definition, but the cons are that in case that I want to create new revision only if the definition was changed, then I need to describe the task, get the image from the container list, and then compare the version with the new image version.
Regarding the second option, is that I can see exactly if I created there is a task definition that contains my image or not. The cons are that I will create new task definition for every image version.
In both options, how should I handle the deregister logic?
Probably I missed something so would appreciate your answer.
Thanks!
I've only ever seen the first alternative (Create new revision of task definition) used. If you are using Infrastructure as Code, such as CloudFormation or Terraform, then all the cons you have listed for that are no longer present.
"In both options, how should I handle the deregister logic?"
Just update the ECS service to use the latest version of the task definition. ECS will then deploy a new task, migrate the traffic to that task, and shut down the old task. You don't need to do anything else at that point. There is no special logic you need to implement yourself to deregister anything.
We deploy a Docker image that runs a simple Sinatra API to an ECS Fargate service. Right now our task definition defines the image using a :production tag. We want to use CodeDeploy for a blue/green deployment.
When code is changed - should we push a new image with the :production tag and force a new deployment on our service or instead use specific tags in our task definition (e.g. :97b9d390d869874c35c325632af0fc1c08e013cd) and create a new task revision then update our service to use this new task revision?
Our concern with the second approach is that we don't see any lifecycle rules around task revisions so will they just build up until we have tens/hundreds of thousands?
If we use the first approach, will CodeDeploy be able to roll back a failed deployment in the case there is an issue?
Short answer
In both cases there are no definition roll back if somehow your new image crashed but your current old task should still be alive. But if you are using health check and the current running task is below the required amount (might be due to overflow of user traffic,...etc), Fargate would start up new task with the latest task definition revision which contained the bad image.
Long answer
Since you are just asking CodeDeploy to start up task based on your image, it would create a new task definition that have your image's URI to pull the correct image. And that new task definition would always be used to start up new Fargate task.
So when Fargate found that it needs to create task, it would always try to use the latest revision which would always be the one with bad image.
The good thing is that your old image task if works correctly, it should still be alive, since the minimum running task would be 1 and since the other task is failing continuously, your old image task would not be decommissioned.
You can however overcome this by adding a CloudWatch event to trigger a lambda that either update new task revision with the good image tag or running current Fargate with the previous task definition revision. Here is an article from AWS about this: https://aws.amazon.com/blogs/compute/automating-rollback-of-failed-amazon-ecs-deployments/
A bit more on how Fargate deployment work here and help your old task running when new deployment fail, it would first provision the new task, when all the new tasks is running good, it would decommission old task. So in case the new tasks does not run properly, old task should still be alive.
I have a use case where I schedule a task 24h into the future after an event occurs. This task represents some sort of "deadline" for other things to happen.
The scheduled task triggers a creation of a report. If not all of the above mentioned "other things" have completed by this time, then the triggered report creation process creates it anyways with the information it has at the time.
If, on the other hand, all other things do complete before these 24h, then ideally I'd like to re-use the same Google Cloud Task to trigger the same process (as it's identical as the previous case but will contain all of the information possible).
I would imagine the easiest way to achieve the above is to:
schedule a task 24h into the future
if all information arrives: run the task early before it's scheduled time
However, reading through the Google Cloud Tasks documentation I don't see the option to run the task early. However, that feature does exist on the Cloud Tasks console, so I was wondering if it is available in the documentation and client libraries.
Thanks!
This is probably what you're looking for
https://cloud.google.com/tasks/docs/reference/rest/v2/projects.locations.queues.tasks/run
NOTE: It does say however that "This command is meant to be used for manual debugging"
If I wanted to validate that that an ECS service is running the latest image for a tag, how would I do that?
I can:
describe_services to get the task definition
describe_task_definition to get the image associated
But that image is in whatever form that's in the task definition. If the task definition says service:1.1, that's a good start, but what if a new image has been pushed that is tagged service:1.1 since deployment? There's no way to tell from looking at the image in the task definition.
Maybe that makes sense because it is, after all, the definition, not the task itself. So what about describe_tasks? Looks promising. Except describe_tasks doesn't talk about the image at all. It does have a container ARN, but what good is that? I can't find any API call that uses container ARNs at all -- am I missing something?
Basically -- is there any way to identify the specific image down to the digest level that is running for each task on an ECS service so that you can tell if you should force a new deployment?
Confirmed by Amazon Support, there isn't currently a good way to validate that the image deployed on a given task is the same as the latest image pushed with the tag specified in the task definition.
It's not ideal, but I could compare the deployment's updatedAt and the task definition's image with the image's pushedAt, I suppose. That won't give me an explicit "which image am I using", but it will tell me "has the image tag been pushed since the service was updated?"
This is the only doc i have found for Task Group and it doesn't explain how or where to create one.
I can't find any docs that adequately explain what a Task Group actually is with an example of how to create and use one. It sound like its a way for a service to run multiple different Task Definitions which would be useful to me.
For example, I added a container to a task definition and the service is balancing multiple instances of it on the cluster. But I have another container I want to deploy along with the first one, but I only want a single instance of it to run. So I can't add it to the same task definition because I'd be creating multiple instances of it and consuming unnecessary resources. Seems like this is what Task Groups are for.
You are indeed correct, there exists no proper documentation on this (I opened a support case with our AWS team to verify!).
However, all is not lost. A solution to your conundrum does indeed exist, and is a solution we use every day. You don't have to use the task group, whatever that is (since we don't actually know yet (AWS engineer is writing up some docs for me, will post them here when I get them)).
All you need though are placement constraints (your same doc), which are easy enough to setup. If you have a launch configuration, you can add something like this to the Advanced > User Data section, so that it gets run during boot (or just add it when launching your instance manually (or if you're feeling exceptionally hacky, you can logon to your instance and run the commands manually.. for science and stuff)):
echo ECS_INSTANCE_ATTRIBUTES={\"env\": \"prod\",\"primary\": \"app1\",\"secondary\": \"app2\"} >> /etc/ecs/ecs.config
Everything in quotes is arbitrarily defined by you, so use whatever tags and values make sense for your use case. If you go this route, make sure you add the following line to your docker launch command: --env-file=/etc/ecs/ecs.config
So now that you have an instance that's properly tagged (and make sure it's only the single instance you want (which means you probably need a dedicated launch configuration for this specific type of instance)), you can go ahead and create your ECS service like you were wanting to do. However, make sure you setup your Task Placement correctly, to match the roles that are now configured for your instances:
So for the example above, this service is configured to only launch this task on instances that are configured for both env==prod and secondary==app2 -- since your other two instances aren't configured for secondary==app2, they're not allowed to host this task.
It can be confusing at first, and took us a while to get right, but I hope this helps!
Response from AWS Support
I looked into the procedure how to use the Task Groups and here were my findings: - The assumption is that you already have a task group named "databases" if you had existing tasks launched from RunTask/StartTask API.
When you launch a task using the RunTask or StartTask action, you can specify the name of the task group for the task. If you don't specify a task group for the task, the default name is the family name of the task definition (for example, family:my-task-definition) - So to create a Task Group, either you define a TaskGroup (say webserver) while creating a Task on Task Console or use following command : $ aws ecs run-task --cluster <ecs-cluster> --task-definition taskGroup-td:1 --group webserver
Once created you will notice a Task running with a group: webserver.
Now you can use following placement constraints with the Task Definition to place your tasks only on the containers that are running tasks with this Task Group.
"placementConstraints":
[
{
"expression": "task:group == webserver", "type": "memberOf"
}
]
If you try to run a task with above placementConstraint, but you do not have any task running with taskGroup : webserver, you will receive following error: Run tasks failed Reasons : ["memberOf constraint unsatisfied"].
References: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-placement-constraints.html