I am using AWS ECS to run Docker-based tasks. Since ECS agent task definitions don't support all Docker options, I'm looking for some workaround to pass these options to the docker run command.
Is it possible to pass Docker options like --memory-swappiness, --memory-swap, etc. using Dockerfiles?
You should use something like docker compose for that. Docker compose allows to specify parameters for you containers. In contrast Dockerfile configures your image. For example to limit resources you just create resources section in you compose.yml. Note that swappiness is an obsolete parameter.
See link: https://docs.docker.com/compose/compose-file/#resources
version: '3'
services:
redis:
image: redis:alpine
deploy:
resources:
limits:
cpus: '0.50'
memory: 50M
reservations:
cpus: '0.25'
memory: 20M
Related
I am trying to run the Telegraf as a docker container on AWS fargate.
I have created the Telegraf image file using Dockerfile and built the image and pushed it to ECR.
Now, I am trying to run this image on AWS fargate.
The main challenge I facing is how to mount the configuration (telegraf.conf) file to the container
which required by container to run it.
I tried following this https://kichik.com/2020/09/10/mounting-configuration-files-in-fargate/ blog by spinning two containers but I have more files that I am passing to the telegraf.conf file.
Fargate provides two options to mount files using the Bind mount and EFS. I am trying to use Bind Mount but I am not sure how to provide the configuration files or mount them.
I am showing below how I run the telegraf container using docker-compose.
telegraf1:
image: telegraf:1.20.0
container_name: telegraf
restart: always
depends_on:
- influxdb
networks:
- analytics
volumes:
- /mnt/telegraf/:/var/lib/telegraf
- ./etc/telegraf/:/etc/telegraf/
env_file:
- secrets.env
environment:
INFLUXDB_URL: http://influxdb:8086
command:
--config-directory /etc/telegraf/telegraf.d
--config /etc/telegraf/telegraf.conf
links:
- influxdb
Now I want to achieve same using AWS fargate but not sure how to provide the volume mount on AWS fargate.
Bind mount on Fargate is good for sharing a folder between multiple containers in a single task, but I'm not aware of any way to load external configuration files in Fargate bind mounts, other than running a sidecar container to download those from S3 on task startup.
I generally see EFS used for mounting a folder with configuration files in Fargate.
I'm getting the following error when I try to run docker compose up to deploy my infrastructure to AWS using Docker's ECS integration. Note that I'm running this on Pop!_OS 21.10, which is based on Ubuntu.
NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Things I've tried, based on an exhaustive search of SO and other sites:
Verified the proper format of my ~/.aws/config and ~/.aws/credentials files are formatted correctly, are in the proper place, and have the correct permissions
Verified that the aws cli works fine
Verify that AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION are all set correctly
Tried copying the config and credentials to /root/.aws
Tried setting AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION in the root user's environment
Created /etc/systemd/system/docker.service.d/aws-credentials.conf and populated it with:
[Service]
Environment="AWS_ACCESS_KEY_ID=********************"
Environment="AWS_SECRET_ACCESS_KEY=****************************************"
Ran docker -l debug compose up (Only extra information it provides is DEBUG deploying on AWS with region="us-east-1"
I'm running out of options. If anyone has any other ideas to try, I'd love to hear it. Thanks!
Update: I've also now tried the following, with no luck:
Tried setting Environment="AWS_SHARED_CREDENTIALS_FILE=/home/kespan/.aws/credentials
Tried setting Environment="AWS_SHARED_CREDENTIALS_FILE=/home/kespan/.aws/credentials in /etc/systemd/system/docker.service.d/override.conf
After remembering my IAM account has MFA enabled, generated a token and added Environment="AWS_SESSION_TOKEN=..." to override.conf
Also to note - each time after I've added/modified files under /etc/systemd/system/docker.service.d/ I've run:
sudo systemctl daemon-reload
sudo systemctl restart docker
Edit:
Here's one of the Dockerfiles (both the scraper and scheduler use an identical Dockerfile):
FROM denoland/deno:alpine
WORKDIR /app
USER deno
COPY deps.ts .
RUN deno cache --unstable --no-check deps.ts
COPY . .
RUN deno cache --unstable --no-check mod.ts
RUN mkdir -p /var/tmp/log
CMD ["run", "--unstable", "--allow-all", "--no-check", "mod.ts"]
Here's my docker-compose (some bits redacted):
version: '3'
services:
grafana:
container_name: grafana
image: grafana/grafana
ports:
- "3000:3000"
volumes:
- grafana:/var/lib/grafana
deploy:
replicas: 1
scheduler:
image: scheduler
x-aws-pull-credentials: "arn..."
container_name: scheduler
environment:
DB_CONNECTION_STRING: "postgres://..."
SQS_URL: "..."
SQS_REGION: "us-east-1"
AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
deploy:
replicas: 1
scraper:
image: scraper
x-aws-pull-credentials: "arn..."
container_name: scraper
environment:
DB_CONNECTION_STRING: "postgres://..."
SQS_URL: "..."
SQS_REGION: "us-east-1"
AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
deploy:
replicas: 1
volumes:
grafana:
Have you attempted to use the Amazon ECS Local Container Endpoints tool that AWS Labs provides? It allows you to create an override file for you docker-compose configurations, and it will simulate the ECS endpoints and IAM roles you would be using in AWS.
This is done using the local AWS credentials you have on your workstation. More information is available on the AWS Blog.
I'm trying to deploy XRAY as a sidecar-container of my main container in AWS ECS Fargate using docker-compose; but it creates 2 tasks (Service and Xray) instead of 1 task containing both, the service and the xray daemon.
I have done this in the past without issues using cfn but I cannot make it work with docker-compose.
This is my docker-compose file:
version: "3.9"
services:
web:
image: link-to-private-repo/web
ports: ["80:80"]
xray:
image: amazon/aws-xray-daemon
ports:
- 2000:2000/udp
Thanks.
This is not possible today with the current Docker Compose out of the box experience. This need is tracked in this GH issue. Please weigh in in the issue with your use case.
I have a docker-compose yml file. It contains 11 services/sections. I am able to successfully deploy it on Ubuntu EC2 instance.
Now, I need to host each service/ section from Docker Compose inside AWS ECS. As per my understanding, I need to create a task definition from AWS ECS UI. I can look at my docker-compose file for image details, environment variables, labels, and just add it inside the task definition. Then, start the task. Now, my ECS tasks should work without any additional settings.
Is this a correct understanding that any service/section docker-compose yml file is ECS compatible?
To my understanding, ECS still doesn't allow the build command in docker-compose.yml so you will run into some headaches.
You can build separate images on ECR and have your docker-compose file to run those when you are creating your services. Don't forget to rebuild those images when you are deploying new code.
Sample docker-compose file
version: '3'
services:
web:
container_name: web
image: some_repository/web:latest
db:
container_name: db
image: some_other_repo/db:latest
Hope this helps.
We have an application that uses docker compose that contains links.
I'm trying to deploy this using aws-cli on Amazon Fargate using this command:
ecs-cli compose --project-name myApp --file docker-compose-aws.yml --ecs-params fargate-ecs-params.yml --cluster myCluster --region us-east-1 up --launch-type FARGATE
When my fargate-ecs-params.yml has ecs_network_mode: awsvpc I get the error:
Links are not supported when networkMode=awsvpc
So I've tried changing to ecs_network_mode: awsvpc, however I then get the error:
Fargate only supports network mode ‘awsvpc’
My question is how do I create a task definition for Fargate with a compose file that contains links? Or is this not possible (and in that case then what are my alternatives?)
You can place both container in same task definitons they will automatically linked with each other.
After reading your final comment on the boot sequence and answering that question instead, I solved this (even in non-AWS) using the docker-compose depends.
Simple e.g.
services:
web:
depends_on:
- "web_db"
web_db:
image: mongo:3.6
container_name: my_mongodb
You should be able to remove the deprecated links and just use the hostnames that docker creates from the service container names. e.g. above the website would connect to the hostname: "my_mongodb".