Terraforming a AWS ECS Service depends on this configuration:
load_balancer {
target_group_arn = aws_lb_target_group.foo.arn
container_name = "mongo"
container_port = 8080
}
‘container_name’ here needs to be the same value as what’s defined in the task definition. The problem here is that I don’t want to terraform the Task Definition, as that’s an application-level configuration I’d like to keep separate from the terraform / terragrunt which is obviously infrastructure-level configuration. What is the best approach here? Am I forced to define the task definition in terraform?
Additional Background: I have PCF experience where a simple manifest file was tied with the source code. Cloud Foundry was very 12-factor sensitive. Here it seems to be a violation of those principles.
that’s an application-level configuration
Not in AWS. Task definition is an AWS resource, and as any resource it can be managed through IaC tools, such as CloudFormation or Terraform. Thus if you want to use IaC to manage your application at AWS, you have to accept the fact that task definition is part of the AWS infrastructure, not application.
Related
I have declared my ECS modules with the required IAM roles, application load balancer, auto scaling, and codedeploy with blue/green deployment.
Every time I update my task definition, or anything related to the ECS service, I get the error:
Error: updating ECS Service (arn:aws:ecs...): InvalidParameterException: Unable to update task definition on services with a CODE_DEPLOY deployment controller. Use AWS CodeDeploy to trigger a new deployment.
I get it, I should use CodeDeploy to deploy, but it was declared there when I created the service, and set up everything, so I guess changing the code to declare it as a data source instead of a resource would be just a dirty quick fix.
What's the proper way of updating task definitions to prevent this error from happening?
When I update the task definition for one of my ECS services, the service does not re-deploy with the new task definition until I run aws ecs update-service ... --force-new-deployment. Is there some kind of configuration option that controls this behaviour? I've noticed that some of my services do re-deploy when I update the Task Definition, but others don't and I'm not sure why that is.
I've provisioned my infrastructure with terraform.
You need to set the force_new_deployment setting in the aws_ecs_service resource in your Terraform code.
Introduction
I am deploying a Django app using Celery for background tasks on Amazon ECS, and we're using CodePipeline for CI/CD. I would like to be able to split this up into three ECS Services, each running only one task - this is so they can be scaled independently. It is proving hard to do while still meeting two key design goals:
Continuous delivery of changes - must be automated
Infrastructure changes must be managed as code
So, fundamentally, updates to ECS Task Definitions need to be versioned in git and updated as part of the automated release process and when they change, the services using them need to be updated.
For the service that accepts the traffic, this all works fine. The issue is with those services on ECS that are performing background tasks. There, I'm hitting a roadblock in that:
CodeDeploy Deployment Groups insist on being associated with a Load Balancer, and
Any Deployment provider that deals with updating the Task Definition requires a Deployment Group.
I think this is limited to the "CodeDeploy" and "ECS Blue/Green" providers.
Neither my "scheduler" or "worker" service accept traffic
So, it comes down to this: what kind of deployment can I do that doesn't require a load balancer, but will still allow me to update the task definition as part of the deployment?
Details
Now, to give you more specifics, the list of service I want is:
"web" service - runs Django, exposed to ALB on port 8000
"scheduler" service - runs Celery "beat", no exposed port
"worker" service - runs Celery worker, no exposed port
For the "web" service, CI/CD is straightforward, we have a CodeDeploy Application, with a Deployment Group that is associated with the Application Load Balancer and has the correct target groups and this does a "Blue/Green" deployment.
We have built some custom tooling that generates a replacement taskdef.json and appspec.yml for each of the services. These tools are invoked during the Build phase of our pipeline and (for the "web" service) applied at deployment time; this is so that updates to the application environments and resources are also managed in code.
So the flow goes:
Build new docker container
Generate new taskdef.json from source templates - filling in resource IDs (secrets etc.) by querying the CloudFormation stack
Generate new appspec.yml with the revision number of the task definition incremented by 1
CodeDeploy creates a new revision of the application based on the new AppSpec and TaskDef (Build artifacts from previous step) and deploys the updated service on the cluster.
This works well for the "web" service, and I would like something similar for the other two services, but I cannot find a way to either: not have a Deployment Group, but still update the Task Definition; or have a Deployment Group but not have a Load Balancer (because there's no traffic to load balance).
Is there a trick to this? Or a deployment type I've missed that is aimed at background services?
I would appreciate any advice you have to offer. Thanks!
For posterity, the answer I came up with in the end was to create a lambda dedicated to re-deploying the ECS Services for celery beat and the workers. Then have CodePipeline deploy the Web service using a Blue/Green deployment and then call the lambda twice (in parallel): once for the scheduler service, once for the worker service.
None of the built-in deployment types were of any help at all in getting this going.
I recently started getting into the devops side of things and am currently working with Terraform and AWS ECS to setup a simple web server host my web applications.
Using my current Terraform config I can see my cluster being created with a service that has my task definition. I can't figure out how to run the tasks required to launch the webserver from Terraform. I can only see the capability to create the task definitions and services, but not run them. I am very new to both these technologies so I fear I might be missing something simple.
The setup I used is from an example I found online that I tried to follow.
TL;DR: I can create services using Terraform but can't seem to figure out how to run them.
You need define an "aws_ecs_service" resource in Terraform and in there define how many instances of your task you want running. In the example you link, that is done in the main.tf file here.
There is no way you can run the task from terraform except running the external script or aws-cli from the local-exec provisioner.
I have an existing ECS cluster (EC2) created using Terraform. I want to install some software on those EC2 instances using Terraform. One of our business requirements is that we will not be able to destroy and re-create the instances and we have to do it on existing instances.
How should I approach this?
It sounds like your organization is experimenting with running its services in docker and ECS. I also assume you are using AWS ECR to host your docker images (although technically doesn't matter).
When you create an ECS cluster it is initially empty. If you were to re-run your terraform template again it should show you that there are no updates to apply. In order to take the next step you will need to define a ecs-service and a ecs-task-definition. This can either be done in your existing terraform template, in a brand new template, or you can do it manually (aws web console or awscli). Since you are already using terraform I would assume you would continue to use it. Personally I would keep everything in 1 template but again it is up to you.
An ecs-service is essentially the runtime configuration for your ecs-tasks
An ecs-task-definition is a set of docker containers to run. In the simplest case it is 1 single docker container. Here is where you will specify the docker image(s) you will use, how much CPU+RAM for the docker container, etc...
In order for your running ecs service(s) to be updated without your EC2 nodes ever going down you would just need to update your docker image within the ecs-task-definition portion of your terraform template (an ofcourse run terraform).
with all this background info now you can add a Terraform ecs-service Terraform ecs-task-definition to your terraform template.
Since you did not provide your template I cannot say exactly how this should be setup but an example terraform template of a complete ECS cluster running nginx can be found below
Complete Terraform ECS example
more examples can be found at
Official terraform ECS github examples
You could run a provisioner attached to an always triggered null_resource to always run some process against things but I'd strongly recommend you rethink your processes.
Your ECS cluster should be considered completely ephemeral, as with the containers running on them. When you want to update the ECS instances then destroying and replacing the instances (ideally in an autoscaling group) is what you want to do as it greatly simplifies things. You can read more about the benefits of immutable infrastructure elsewhere.
If you absolutely couldn't do this then you'd most likely be best off using another tool, such as Ansible, entirely. You could choose to launch this via Terraform using a null_resource provisioner as mentioned above which would look something like the following:
resource "null_resource" "on_demand_provisioning" {
triggers {
always = "${uuid()}"
}
provisioner "local-exec" {
command = "ansible-playbook -i inventory.yml playbook.yml --ssh-common-args='-o StrictHostKeyChecking=no'"
}
}