I have below data with me -
minio:
image: minio/minio:latest
#ports:
# - '9000:9000'
volumes:
- ./data/storage:/data
environment:
MINIO_ACCESS_KEY: minio
MINIO_SECRET_KEY: minio123
command: server /data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
restart: always
I want to manually create task definition in FARGATE ECS and then add containers in it.[No Coding]
Where can I specify volumes specified above inside containers ?
To answer your query specific to volumes, you would have to specify the volumes in a task definition which is used to run a task in AWS Fargate. You can have a look at this documentation. This also lists the limitation when it comes to storage in AWS Fargate. AWS Fargate does not support any way to have persistent storage except EFS which was launched recently.
If your use case allows EFS check out this blog which demonstrates Amazon Elastic Container Service & AWS Fargate, now support Amazon Elastic File System
Related
I have an ECS cluster where I place a container that runs as a daemon to monitor all other processes. However, I'm seeing this containers being killed by OOM from time to time without leaving a trace. I just happened to spot one of them being killed. This is causing some log duplication but I wonder if there is a way to trace these restarts because when I look on the ECS Cluster events, there are no information about this tasks being restarted by any means.
I know more from kubernetes so I would say an analogy here. When this happens on kubernetes you would see a RESTARTS counter when you get information from all pods (kubectl get pods) is there any way to find this information on AWS ECS tasks? I'm struggling to find on documentation
I identified the tasks, and also I identified the status of each tasks to gain more information, but I'm unable to find any hint that the process was restarted or killed before.
this is a task detail example
- attachments: []
attributes:
- name: ecs.cpu-architecture
value: x86_64
availabilityZone: us-east-2c
clusterArn: arn:aws:ecs:us-west-2:99999999999:cluster/dev
connectivity: CONNECTED
connectivityAt: '2023-01-24T23:03:23.315000-05:00'
containerInstanceArn: arn:aws:ecs:us-east-2:99999999999:container-instance/dev/eb8875fhfghghghfjyjk88c8f96433b8
containers:
- containerArn: arn:aws:ecs:us-east-2:99999999999:container/dev/05d4a402ee274a3ca90a86e46292a63a/e54af51f-2420-47ab-bff6-dcd4f976ad2e
cpu: '500'
healthStatus: HEALTHY
image: public.ecr.aws/datadog/agent:7.36.1
lastStatus: RUNNING
memory: '750'
name: datadog-agent
networkBindings:
- bindIP: 0.0.0.0
containerPort: 8125
hostPort: 8125
protocol: udp
- bindIP: 0.0.0.0
containerPort: 8126
hostPort: 8126
protocol: tcp
networkInterfaces: []
runtimeId: 75559b7327258d69fe61cac2dfe58b12d292bdb7b3a720c457231ee9e3e4190a
taskArn: arn:aws:ecs:us-east-2:99999999999:task/dev/05d4a402ee274a3ca90a86e46292a63a
cpu: '500'
createdAt: '2023-01-24T23:03:22.841000-05:00'
desiredStatus: RUNNING
enableExecuteCommand: false
group: service:datadog-agent
healthStatus: HEALTHY
lastStatus: RUNNING
launchType: EC2
memory: '750'
overrides:
containerOverrides:
- name: datadog-agent
inferenceAcceleratorOverrides: []
pullStartedAt: '2023-01-24T23:03:25.471000-05:00'
pullStoppedAt: '2023-01-24T23:03:39.790000-05:00'
startedAt: '2023-01-24T23:03:47.514000-05:00'
startedBy: ecs-svc/1726924224402147943
tags: []
taskArn: arn:aws:ecs:us-west-2:99999999999:task/dev/05d4a402ee274a3ca90a86e46292a63a
taskDefinitionArn: arn:aws:ecs:us-west-2:99999999999:task-definition/datadog-agent-task:5
version: 2
I don't think ECS tracks or exposes a restart counter for tasks. If you want to be notified of tasks restarting you can create an Event Bridge subscription.
You can use ECS Event with EventBridge and add any action like logging when such event happen.
So, after debugging a lot within the little information AWS provides for this use case, I ended up doing a process to find the answer:
List all tasks ids of a given service with aws-cli with flag --desired-status STOPPED and dump all to a json file
aws ecs list-tasks --cluster dev --service-name datadog-agent
--desired-status STOPPED --output json > ecs_tasks.json
using jq and aws-cli, describe all previously found tasks ids to get further information on each one of them
aws ecs describe-tasks --cluster dev --tasks $(jq -j '.taskArns[] |
(.|" ",.)' ./ecs_tasks.json) --output yaml > ecs_tasks_describe.log
I could came up with a script to group and summarize the information but, since I only had to watch over 20 stopped tasks I ended up dumping the information in yaml format for easiness. I found two key properties on the output:
For each task object, there is a reason for why it was stopped that told me nothing more than it was stopped because a container within the task exited (doesn't say the exit code to help though)
stoppedReason: Essential container in task exited
* For each task object, there is an array of containers objects under **containers** property. There you'll sometimes find **reason** property which can explain a bit more of why the container stopped
reason: 'OutOfMemoryError: Container killed due to memory usage'
Note: This information would give you all events for a given service for at least the last hour. In my case it gave me 8 hours of events but AWS documentation only promises 1 hour https://docs.aws.amazon.com/AmazonECS/latest/developerguide/stopped-task-errors.html
Stopped tasks only appear in the Amazon ECS console, AWS CLI, and AWS SDKs for at least 1 hour after the task stops. After that, the details of the stopped task expire and aren't available in Amazon ECS.
I am trying to run the Telegraf as a docker container on AWS fargate.
I have created the Telegraf image file using Dockerfile and built the image and pushed it to ECR.
Now, I am trying to run this image on AWS fargate.
The main challenge I facing is how to mount the configuration (telegraf.conf) file to the container
which required by container to run it.
I tried following this https://kichik.com/2020/09/10/mounting-configuration-files-in-fargate/ blog by spinning two containers but I have more files that I am passing to the telegraf.conf file.
Fargate provides two options to mount files using the Bind mount and EFS. I am trying to use Bind Mount but I am not sure how to provide the configuration files or mount them.
I am showing below how I run the telegraf container using docker-compose.
telegraf1:
image: telegraf:1.20.0
container_name: telegraf
restart: always
depends_on:
- influxdb
networks:
- analytics
volumes:
- /mnt/telegraf/:/var/lib/telegraf
- ./etc/telegraf/:/etc/telegraf/
env_file:
- secrets.env
environment:
INFLUXDB_URL: http://influxdb:8086
command:
--config-directory /etc/telegraf/telegraf.d
--config /etc/telegraf/telegraf.conf
links:
- influxdb
Now I want to achieve same using AWS fargate but not sure how to provide the volume mount on AWS fargate.
Bind mount on Fargate is good for sharing a folder between multiple containers in a single task, but I'm not aware of any way to load external configuration files in Fargate bind mounts, other than running a sidecar container to download those from S3 on task startup.
I generally see EFS used for mounting a folder with configuration files in Fargate.
I'm using ECS through ecs-cli to deploy my API.
I start by launching a cluster of spot instances using this command :
sudo ecs-cli up --region MY REGION --keypair MY KEY PAIR --instance-type t2.micro --capability-iam --size 1 --cluster MY CLUSTER NAME --spot-price 0.01
Then, using the following docker-compose.yml and ecs-params.yml files :
version: '3'
services:
selenium:
image: selenium/standalone-chrome
...etc
api:
image: myapithatusesselenium/myapithatusesselenium
ports:
- 3000:3000
links:
- selenium
...etc
version: 1
task_definition:
task_execution_role: ROLE ID
services:
selenium:
cpu_shares: 600
mem_limit: 700000000
api:
repository_credentials:
credentials_parameter: REPO CREDENTIALS
cpu_shares: 400
mem_limit: 300000000
I'm deploying a service with a load balancer using this command :
sudo ecs-cli compose --file docker-compose.yml --ecs-params ecs-params.yml --project-name MY PROJECT NAME service up --cluster MY CLUSTER NAME --target-group-arn LOAD BALANCER RESSOURCE ID --container-name api --container-port 3000
So, When my API is under a lot of load (When it starts notifying me that the API is going down) I add additional instances by scaling using these commands:
# 1 - scale the number of ec2 instances in the cluster
sudo ecs-cli scale --size 3 --capability-iam
# 2 - scale the number of tasks
sudo ecs-cli compose --file docker-compose.yml --project-name MY PROJECT NAME service scale 3
As you can see the number of tasks and ec2 instances is the same because each container can handle a single task.
When there isn't a lot of load I reduce the size again.
What I need right now is a way to make this automatic (Auto scaling in and out). I can't figure out how to do that.
Thank you !
ECS doesn't nativly autoscaling. You have to use the application autoscaling service for that. You'll need to use the regular aws CLI and call register-scalable-target, and then create a scaling policy
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html
https://docs.aws.amazon.com/cli/latest/reference/application-autoscaling/register-scalable-target.html
https://docs.aws.amazon.com/cli/latest/reference/application-autoscaling/put-scaling-policy.html
I am deploying report portal on AWS FARGATE ECS containers. I want to use below settings.
minio:
image: minio/minio:latest
#ports:
# - '9000:9000'
volumes:
- ./data/storage:/data
environment:
MINIO_ACCESS_KEY: minio
MINIO_SECRET_KEY: minio123
command: server /data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
restart: always
Login to AWS. Select ECS. Create Cluster. Manually Create Task Definition. Add container inside Task Definition. Provide Image Name, Health Check, Commands as specified above inside ECS Task Definition. Save Task Definition. Start the Task.
Getting error as 'server /data' is a not valid command, existing container.
The command has to be comma delimited. Try server, /data
We have an application that uses docker compose that contains links.
I'm trying to deploy this using aws-cli on Amazon Fargate using this command:
ecs-cli compose --project-name myApp --file docker-compose-aws.yml --ecs-params fargate-ecs-params.yml --cluster myCluster --region us-east-1 up --launch-type FARGATE
When my fargate-ecs-params.yml has ecs_network_mode: awsvpc I get the error:
Links are not supported when networkMode=awsvpc
So I've tried changing to ecs_network_mode: awsvpc, however I then get the error:
Fargate only supports network mode ‘awsvpc’
My question is how do I create a task definition for Fargate with a compose file that contains links? Or is this not possible (and in that case then what are my alternatives?)
You can place both container in same task definitons they will automatically linked with each other.
After reading your final comment on the boot sequence and answering that question instead, I solved this (even in non-AWS) using the docker-compose depends.
Simple e.g.
services:
web:
depends_on:
- "web_db"
web_db:
image: mongo:3.6
container_name: my_mongodb
You should be able to remove the deprecated links and just use the hostnames that docker creates from the service container names. e.g. above the website would connect to the hostname: "my_mongodb".