I have an ECS service that will be running many instances of one task. All these instances use the exact same task definition but all need to be provided a specific environment variable INDEX that is unique to that instance. I would like the service to monitor each instance and restart it with the same environment variable value if it fails (I.E. If a task with INDEX=555 fails, I would like the service to spin up a new task with INDEX=555 to replace it). Currently I only see the option to set environment variables in the task definition itself, which would require me to create many versions of the exact same task (and the corresponding service as well) with the only difference being the environment variable (this seems wasteful and would clutter the task definition list and service list in that cluster).
In short I want to configure my ECS service such that I can provide a list of values for a specific environment variable along with one task definition and have it create a 1:1 map of a task instance to environment variable. Is it possible to do so? If so, how can I accomplish this?
Important Note: This service is running on Fargate and not EC2 Instances
Sadly, you can't do this for ECS Service. The ECS services are meant to run exact copies of same tasks which are disposable, exchangeable and scalable.
However, you can provide overwrites for the varaibles for tasks outside of a service, which you can do using run-task or start-task with --overrides of AWS CLI or equivalent in any AWS SDK.
Related
CONTEXT:
We have a platform where users can create their own projects - multiple projects per user. We need to provide them with a browser-based IDE to edit those projects.
We decided to go with coder-server. For this we need to configure an auto-scalable cluster on AWS. When the user clicks "Edit Project" we will bring up a new container each time.
https://hub.docker.com/r/codercom/code-server
QUESTION:
How to pass parameters from the url query (my-site.com/edit?project=1234) into a startup script to pre-configure the workspace in a docker container when it starts?
Let's say the stack is AWS + ECS + Fargate. We could use kubernetes instead of ECS if it helps.
I don't have any experience in cluster configuration. Will appreciate any help or at least a direction where to dig further.
The above can be achieved using multiple ways in AWS ECS. The basic requirements for such systems are to launch and terminate containers on the fly while persisting the changes in the files. (I will focus on launching the containers)
Using AWS SDK's:
The task can be easily achieved using AWS SDKs, Using a base task definition. AWS SDK allows starting tasks with overrides on the base task definition.
E.G. If task definition has a memory of 2GB then the SDK can override the memory to parameterised value while launching a task from task def.
Refer to the boto3 (AWS SDK for Python) docs.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task
Overall Solution
Now that we know how to run custom tasks with python SDK (on demand). The overall flow for your application is your API calling AWS lambda function whit parameters to spin up and wait to keep checking task status and update and rout traffic to it once the status is healthy.
API calls AWS lambda functions with parameters
Lambda function using AWS SDK create a new task with overrides from base task definition. (assuming the base task definition already exists)
Keep checking the status of the new task in the same function call and set a flag in your database for your front end to be able to react to it.
Once the status is healthy you can add a rule in the application load balancer using AWS SDK to route traffic to the IP without exposing the IP address to the end client (AWS application load balancer can get expensive, I'll advise using Nginx or HAProxy on ec2 to manage dynamic routing)
Note:
Ensure your Image is lightweight, and the startup times are less than 15 mins as lambda cannot execute beyond that. If that's the case create a microservice for launching ad-hoc containers and hosting them on EC2
Using Terraform:
If you looking for infrastructure provisioning terraform is the way to go. It has a learning curve so recommend it as a secondary option.
Terraform is popular for parametrising using variables and it can be plugged in easily as a backend for an API. The flow of your application still remains the same from step 1, but instead of AWS Lambda API will be calling your ad-hoc container microservice, which in turn calls terraform script and passing variables to it.
Refer to the Terrafrom docs for AWS
https://registry.terraform.io/providers/hashicorp/aws/latest
I have an ECS service and within that ECS service, I want to boot up 3 tasks all from the same task definition. I need each of these tasks to be on a separate EC2 instance, this seems simple enough however I want to pass a different command to each one of the running tasks to specify where their config can be found and some other options via the CLI within my running application.
For example for task 1 I want to pass run-node CONFIG_PATH="/tmp/nodes/node_0 and task 2 run-node CONFIG_PATH="/tmp/nodes/node_1" --bootnode true and task 3 run-node CONFIG_PATH="/tmp/nodes/node_0 --http true"
I'm struggling to see how I can manage individual task instances like this within a single service using Terraform, it seems really easy to manage multiple instances that are all completely equal but I can't find a way to pass custom overrides to each task that are all running off the same task definition.
I am thinking this may be a job for a different dev-ops automation tool but would love to carry on doing it in Terraform if possible.
This is not a limitation of a terraform. This is how ECS service works - runs exact copies of same task definition. Thus, you can't customize individual tasks in an ECS service as all these tasks are meant to be identical, interchangeable and disposable.
To provide overwrites you have to run the tasks outside of a service, which you can do using run-task or start-task with --overrides of AWS CLI or equivalent in any AWS SDK. Sadly there is no equivalent for that in terraform, except running local-exec with AWS CLI.
Requirement: We have n no.of client(region specific), we need to do some calculations daily on client data and we need to store the data in DynamoDB and this process should be dynamic.
Problem: We couldn't find any details on how we can trigger the multiple container dynamically using AWS Fargate which should take some input such as client ids.
We planned to use AWS Lambda + Fargate as the solution.
We can start containers for a task definition using AWS ECS run-task api
Define a task definition with container and default environment variables.
ECS Run Task with launchType Fargate(requires no management of EC2), can be triggered either from Lambda or from cloud watch event rule.
Pass in ContainerOverrides with additional attributes and environment variables when running the task with additional information for that particular run.
Trying to create a service in ECS and unbelievably looks like it is not possible to specify any environmental variables...
Is it possible to do that without updating my task definition and recreate the service? Or a task override option?
This looks a bit cumbersome
The environmental variables are provided through task definition. Thus you have to update the definition to add/change the variables.
You don't have to re-create the service from scratch. You can update your service to use the new version of your task definition. For updating existing service you can use update-service AWS CLI call. The cli also provides --force-new-deployment if you want to force the deployment (but changing task should be enough and forcing would not be required).
You can't define environment variables at service creation time as explained by the other answer but you can define tags: thus one workaround is to
Create a service with a set of tags and propagateTags: SERVICE
On container startup read the cluster and task ARNs from the task metadata endpoint
Read service tags with the DescribeTasks API (note the include: ['TAGS'] parameter)
Configure the environment using the tag values
On AWS ECS you can run a task, or a service.
If you run a task with run_task(**kwargs), you have the option to override some task options, for example the container environment variables, this way you can configure the thing inside the container for example. That's great.
Now, I can't find a way how to do the same with create_service(**kwargs). You can only specify a task, so the created container runs with configuration as specified in the task definition. No way to configure it.
Is there a way how to modify task in a service, or this is not possible with the AWS ECS service?
This is not possible. If you think how services work, they create X number of replicas of the task. All instances of the task have the same parameters, because the purpose is scaling out the task - they should do the same job. Often the traffic is load-balanced (part of service configuration), so it is undesirable that a user will get different response next time than the previous request due to ending up on a task which is configured differently. So bottom line is - that's by design.
Because parameters are shared, if you need to change a parameter, you create a new definition of the task, and then launch that as a service (or update an existing service).
If you want the tasks to be aware of other tasks (and thus behave differently), for example to write data to different shards of a sharded store, you have to implement that in the task's logic.