I am trying to run the Telegraf as a docker container on AWS fargate.
I have created the Telegraf image file using Dockerfile and built the image and pushed it to ECR.
Now, I am trying to run this image on AWS fargate.
The main challenge I facing is how to mount the configuration (telegraf.conf) file to the container
which required by container to run it.
I tried following this https://kichik.com/2020/09/10/mounting-configuration-files-in-fargate/ blog by spinning two containers but I have more files that I am passing to the telegraf.conf file.
Fargate provides two options to mount files using the Bind mount and EFS. I am trying to use Bind Mount but I am not sure how to provide the configuration files or mount them.
I am showing below how I run the telegraf container using docker-compose.
telegraf1:
image: telegraf:1.20.0
container_name: telegraf
restart: always
depends_on:
- influxdb
networks:
- analytics
volumes:
- /mnt/telegraf/:/var/lib/telegraf
- ./etc/telegraf/:/etc/telegraf/
env_file:
- secrets.env
environment:
INFLUXDB_URL: http://influxdb:8086
command:
--config-directory /etc/telegraf/telegraf.d
--config /etc/telegraf/telegraf.conf
links:
- influxdb
Now I want to achieve same using AWS fargate but not sure how to provide the volume mount on AWS fargate.
Bind mount on Fargate is good for sharing a folder between multiple containers in a single task, but I'm not aware of any way to load external configuration files in Fargate bind mounts, other than running a sidecar container to download those from S3 on task startup.
I generally see EFS used for mounting a folder with configuration files in Fargate.
Related
So I have a a few operations I need to perform on a running instance - kind of a one-off thing. That involves uploading fixtures to it. The structure is an aws ec2 instance, on which runs a few docker container, including a django container. This the docker container in which I need to upload fixtures. This is all part of a ci/cd pipeline, with a postgres RDS database, nginx proxy, and the django container.
I can scp the fixtures into the AWS EC2 instance. Then I can SSH into the EC2 instance, and from there log into the running container (using docker exec).
That's all fine, but....
How can I access the fixtures that I uploaded to the ec2 instance (the docker host) from within the docker-container?
Note: I'm aware I could commit the fixtures to git, then they would find their way into the docker image and be readily available there. I possible I'd rather scp them directly to ec2 & manage them in command lines (because small changes might be needed to the fixtures, which then means I have to run the whole pipeline and wait a few minutes each time...)
You would want to attach a volume to your Docker container. By attaching a volume essentially you are sharing a folder from the host machine with the running Docker container.
You can attach a volume when you start the container:
docker run -d \
--name=my-container-name \
-v /host/path:/container/path \
myimage:latest
In case of docker compose, you can add a volume like this:
version: "3.9"
services:
myservice:
image: myimage
volumes:
- "/host/path:/container/path"
I have a docker-compose yml file. It contains 11 services/sections. I am able to successfully deploy it on Ubuntu EC2 instance.
Now, I need to host each service/ section from Docker Compose inside AWS ECS. As per my understanding, I need to create a task definition from AWS ECS UI. I can look at my docker-compose file for image details, environment variables, labels, and just add it inside the task definition. Then, start the task. Now, my ECS tasks should work without any additional settings.
Is this a correct understanding that any service/section docker-compose yml file is ECS compatible?
To my understanding, ECS still doesn't allow the build command in docker-compose.yml so you will run into some headaches.
You can build separate images on ECR and have your docker-compose file to run those when you are creating your services. Don't forget to rebuild those images when you are deploying new code.
Sample docker-compose file
version: '3'
services:
web:
container_name: web
image: some_repository/web:latest
db:
container_name: db
image: some_other_repo/db:latest
Hope this helps.
I have below data with me -
minio:
image: minio/minio:latest
#ports:
# - '9000:9000'
volumes:
- ./data/storage:/data
environment:
MINIO_ACCESS_KEY: minio
MINIO_SECRET_KEY: minio123
command: server /data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
restart: always
I want to manually create task definition in FARGATE ECS and then add containers in it.[No Coding]
Where can I specify volumes specified above inside containers ?
To answer your query specific to volumes, you would have to specify the volumes in a task definition which is used to run a task in AWS Fargate. You can have a look at this documentation. This also lists the limitation when it comes to storage in AWS Fargate. AWS Fargate does not support any way to have persistent storage except EFS which was launched recently.
If your use case allows EFS check out this blog which demonstrates Amazon Elastic Container Service & AWS Fargate, now support Amazon Elastic File System
We have an application that uses docker compose that contains links.
I'm trying to deploy this using aws-cli on Amazon Fargate using this command:
ecs-cli compose --project-name myApp --file docker-compose-aws.yml --ecs-params fargate-ecs-params.yml --cluster myCluster --region us-east-1 up --launch-type FARGATE
When my fargate-ecs-params.yml has ecs_network_mode: awsvpc I get the error:
Links are not supported when networkMode=awsvpc
So I've tried changing to ecs_network_mode: awsvpc, however I then get the error:
Fargate only supports network mode ‘awsvpc’
My question is how do I create a task definition for Fargate with a compose file that contains links? Or is this not possible (and in that case then what are my alternatives?)
You can place both container in same task definitons they will automatically linked with each other.
After reading your final comment on the boot sequence and answering that question instead, I solved this (even in non-AWS) using the docker-compose depends.
Simple e.g.
services:
web:
depends_on:
- "web_db"
web_db:
image: mongo:3.6
container_name: my_mongodb
You should be able to remove the deprecated links and just use the hostnames that docker creates from the service container names. e.g. above the website would connect to the hostname: "my_mongodb".
I have been following a tutorial regarding using docker compose with the AWS ECS service located here.
I am using Ubuntu 16.04 and the installation of the ecs-cli was successful.
However, I have been receiving the following error when I try to run sudo ecs-cli compose up in the directory containing the docker-compose.yml file
ERRO[0000] Unable to open ECS Compose Project error="cannot create a task definition with no containers; invalid service config"
FATA[0000] Unable to create and read ECS Compose Project
error="cannot create a task definition with no containers; invalid
service config"
How are docker-compose.yml files and by extension multi-container docker applications supposed to be used with AWS ECS?
I had the same problem and found it was because of yml file formatting. If you did a straight copy and paste of the compose yml, it's missing the correct indentations. Fix those and it'll work.
version: '2'
services:
web:
image: amazon/amazon-ecs-sample
ports:
- "80:80"