Info:
I created a flask app and on my Dockerfile last command CMD gunicorn -b 0.0.0.0:5000 --access-logfile - "app:create_app()"
I build,tag and upload image on ECR
I used this docker image to create an ECS Fargate instance with the following configs (just posting the one needed for the question):
ECSTaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
Cpu: "256"
Memory: "1024"
RequiresCompatibilities:
- FARGATE
ContainerDefinitions:
- Name: contained_above
.
.
.
ECSService:
Type: AWS::ECS::Service
DependsOn: ListenerRule
Properties:
Cluster: !Sub "${EnvName}-ECScluster"
DesiredCount: 1
LaunchType: FARGATE
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 50
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
Subnets:
- Fn::ImportValue: !Sub "${EnvName}-PUBLIC-SUBNET-1"
- Fn::ImportValue: !Sub "${EnvName}-PUBLIC-SUBNET-2"
SecurityGroups:
- Fn::ImportValue: !Sub "${EnvName}-CONTAINER-SECURITY-GROUP"
ServiceName: !Sub "${EnvName}-ECS-SERVICE"
TaskDefinition: !Ref ECSTaskDefinition
LoadBalancers:
- ContainerName: contained_above
ContainerPort: 5000
TargetGroupArn: !Ref TargetGroup
(App is working normally)
Question
Now my question is what number should be the workers on gunicorn command (my last command in dockerfile)?
On gunicorn design it is stated to use Generally we recommend (2 x $num_cores) + 1 as the number of workers to start off with.
So whats the number of cores on a fargate? Does actually make sense to combine gunicorn with Fargate like the above process? Is there 'compatibility' between loadbalancers and gunicorn workers? What is the connection between DesiredCount of ECS Service and the gunicorn -w workers value? Am I missing or miss-understanding something?
Possible solution(?)
One way that I could call it is the following:
CMD gunicorn -b 0.0.0.0:5000 -w $(( 2 * `cat /proc/cpuinfo | grep 'core id' | wc -l` + 1 )) --access-logfile - "app:create_app()"
But I am not sure if that would be a good solution.
Any insights? Thanks
EDIT: I'm using a configuration file for gunicorn to use when starting:
gunicorn.conf.py
import multiprocessing
bind = "0.0.0.0:8080"
workers = multiprocessing.cpu_count() * 2 + 1
worker_class = "uvicorn.workers.UvicornH11Worker"
keepalive = 0
you can tell gunicorn which config file to use with the --config flag.
Sadly I can't find the source anymore, but I've read that 4-12 workers should be enough to handle hundreds if not thousands of simultaneous requests - depending on your application structure, worker class and payload size.
Do take this with a grain of salt tho, since I can't find the source anymore, but it was in an accepted SO answer from a well-reputated person if I remember correctly.
Offical gunicorn docs state somthing in the 2-4 x $(NUM_CORES)range.
Another option would be as gunicorn docs state at another point:
Generally we recommend (2 x $num_cores) + 1 as the number of workers
to start off with. While not overly scientific, the formula is based
on the assumption that for a given core, one worker will be reading or
writing from the socket while the other worker is processing a
request.
Obviously, your particular hardware and application are going to
affect the optimal number of workers. Our recommendation is to start
with the above guess and tune using TTIN and TTOU signals while the
application is under load.
So far I've been running well with holding true to the 4-12 worker recommendation. My company runs several APIs, which connect to other APIs out there, which results in mostly 1-2seconds request time, with the longest taking up to a whole minute (a lot of external API calls here).
Another colleague I talked to mentioned, they are using 1 worker per 5 simultaneous requests they expect - with similar APIs to ours. Works fine for them as well.
Related
I am creating a tekton project which will spawn docker images which in turn will run few kubectl commands. This I have accomplished by using sidecars in tekton docker:dind image and setting
securityContext:
privileged: true
env:
However, one of the task is failing, since it needs to have an equivalent of --net=host in docker run example.
I have tried to set a podtemplate with hostnetwork: True, but then the task with the sidecar fails to start the docker
Any idea if I could implement --net=host in the task yaml file. It would be really helpful.
Snippet of my task with the sidecar:
sidecars:
- image: mypvtreg:exv1
name: mgmtserver
args:
- --storage-driver=vfs
- --userland-proxy=false
# - --net=host
securityContext:
privileged: true
env:
# Write generated certs to the path shared with the client.
- name: DOCKER_TLS_CERTDIR
value: /certs
volumeMounts:
- mountPath: /certs
As commented by #SYN, Using docker:dind as a sidecar, your builder container, executing in your Task steps, should connect to 127.0.0.1. That's how you would talk to your dind sidecar.
Currently ec2 instance is running and inside that docker container is running.
I have 2 containers running in EC2 instance.
1 - Aplication container.
2 - Data Base(CouchDb) container.
I need to store data which is in database to EBS volumes.
I have a docker-compose file and I'm bringing up containers using 'docker-compose up' command.
version: '2.1'
services:
web:
build: .
ports:
- "4200:4200"
- "7020:7020"
depends_on:
couchdb:
condition: "service_healthy"
volumes:
- ./app:/usr/src/app/app
couchdb:
image: "couchdb:1.7.1"
ports:
- "5984:5984"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:5984/"]
interval: 10s
timeout: 90s
retries: 9
You need to bind the EC2 instance directory with CouchDB container.
Where to Store Data
Important note: There are several ways to store data used by
applications that run in Docker containers. We encourage users of the
couchdb images to familiarize themselves with the options available
Create a data directory on a suitable volume on your host system, e.g.
/home/ec2-user/data. Start your couchdb container like this:
$ docker run --name mycouchdb -v /home/ec2-user/data:/opt/couchdb/data -d couchdb:tag
The -v /home/ec2-user/data:/opt/couchdb/data part of the command mounts the /home/ec2-user/data directory from the
underlying host system as /opt/couchdb/data inside the container,
where CouchDB by default will write its data files.
CouchDB Store Data
Update:
In the case of docker-compose
couchdb:
image: "couchdb:1.7.1"
volumes:
- /home/ec2-user/data:/opt/couchdb/data
ports:
- "5984:5984"
Amazon provides ecs-cli compose command which can set up task definition from docker-compose.yaml
But I am not able to declare memory limits (especially soft one) for such task. Deploy option is not supported.
Skipping unsupported YAML option for service... option name=deploy
Is there way how to achieve this with compose? Or is using compose bad idea and it's better to use native task definitions.
update
My compose file was requested, here is it
version: '3'
services:
worker:
image: 880289074637.dkr.ecr.us-east-1.amazonaws.com/negative-keywords:latest
env_file: .env
command: ["celery", "-A", "negmatch", "worker", "-l", "info"]
deploy:
resources:
limits:
cpus: '0.50'
memory: 256M
reservations:
cpus: '0.25'
memory: 128M
web:
image: 880289074637.dkr.ecr.us-east-1.amazonaws.com/negative-keywords:latest
env_file: .env
ports:
- "80:8000"
depends_on:
- "worker"
deploy:
resources:
limits:
cpus: '0.50'
memory: 256M
reservations:
cpus: '0.25'
memory: 128M
You will need to use v2 of docker compose to set the values.
As of today, according docker docs, deploy is meant for swarm mode deployment only.
Looking for options to set resources on non swarm mode containers?
The options described here are specific to the deploy key and swarm
mode. If you want to set resource constraints on non swarm
deployments, use Compose file format version 2 CPU, memory, and other
resource options. If you have further questions, refer to the
discussion on the GitHub issue docker/compose/4513.
More info on using v2 vs v3.
https://github.com/docker/compose/issues/4513#issuecomment-377311337
Here is sample docker-compose(v2) which sets the soft and hard memory limits on container definition of task. mem_limit is hard limit and mem_reservation is soft limit.
Command -
ecs-cli compose --project-name nginx --file docker-compose.yaml create
Compose File -
version: '2'
services:
nginx:
image: "nginx:latest"
mem_limit: 512m
mem_reservation: 128m
cpu_shares: 0
ports:
- 80
Okay, so spent a day on my EC2 with Traefik and Docker set up, but doesn't seem to be working as described in the docs. I can get the Whoami example running but that doesn't really illustrate what I'm looking for?
For my example I have three AWS API Gateway endpoints and I need to point them to my EC2 IP address that gets routed by my Traefik frontend set up and then uses some backend? Which I'm still uncertain of what kind of backend to use.
I can't seem to find a good YAML example that clearly illustrates something to suit my purpose and needs.
Can anyone point me in the right direction? Any good example Docker YAML examples, configuration set up for my example below? Thanks!
I had taken this article as a guide to provision docker installation with traefik.
EDIT: Before this, create a docker network called proxy.
$ docker network create proxy
version: '3'
networks:
proxy:
external: true
internal:
external: false
services:
reverse-proxy:
image: traefik:latest # The official Traefik docker image
command: --api --docker --acme.email="your-email" # Enables the web UI and tells Træfik to listen to docker
restart: always
labels:
- traefik.frontend.rule=Host:traefik.your-server.net
- traefik.port=8080
networks:
- proxy
ports:
- "80:80" # The HTTP port
- "8080:8080" # The Web UI (enabled by --api)
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- $PWD/traefik.toml:/etc/traefik/traefik.toml
- $PWD/acme.json:/acme.json
db:
image: mariadb:10.3
restart: always
environment:
MYSQL_ROOT_PASSWORD: r00tPassw0rd
volumes:
- vol-db:/var/lib/mysql
networks:
- internal # since you do not need to expose this via traefik, so just set it to internal network
labels:
- traefik.enable=false
api-1:
image: your-api-image
restart: always
networks:
- internal
- proxy
labels:
- "traefik.docker.network=proxy"
- "traefik.enable=true"
- "traefik.frontend.rule=Host:api1.yourdomain.com"
- "traefik.port=80"
- "traefik.protocol=http"
api-2:
image: your-api-2-image
restart: always
networks:
- internal
- proxy
labels:
- "traefik.docker.network=proxy"
- "traefik.enable=true"
- "traefik.frontend.rule=Host:api2.yourdomain.com"
- "traefik.port=80"
- "traefik.protocol=http"
Note: Use this if you want to enable SSL as well. Please note that, this might not work in local server as letsencrypt cannot complete the challenge for SSL setup.
create a blank file acme.json and set its permission to 0600
touch acme.json
chmod 0600 acme.json
After setting up everything,
docker-compose config # this is optional though.
and then,
docker-compose up
I have posted my traefik.toml here
I hope this helps.
Let me know if you face any issues.
Regards,
Kushal.
First of all, the issue is not specific to ElasticSearch, I think (so as not to discourage some potential answers).
I'm using a docker service with dnsrr (DNS round-robin) to allow for the discovery of every node in the cluster: they always try the hostname 'elastic' and (should) get different IPs.
This works perfectly fine when I create 3 VMs on my local machine, but I can't figure out why when I run it on 3 EC2 machines, the one configured as swarm leader only tries its own IP, while the two workers discover each other without issue.
I'm fairly new to AWS, so I guess it must be some kind of misconfiguration somewhere, but I can't figure out what to check.
Thanks in advance if you have any idea as to what may cause this, and even better if you come up with a solution!
The docker compose file as use is as below, simplified to the max to isolate the problem.
version: "3.3"
services:
elastic:
image: docker.elastic.co/elasticsearch/elasticsearch:5.5.2
environment:
- ES_JAVA_OPTS=-Xms1g -Xmx1g
- discovery.zen.ping.unicast.hosts=elastic
- discovery.zen.minimum_master_nodes=2
volumes:
- elastic_data:/usr/share/elasticsearch/data
networks:
- overnet
logging:
driver: "json-file"
options:
max-size: "20m"
max-file: "10"
deploy:
mode: global
endpoint_mode: dnsrr
networks:
overnet:
driver: overlay
driver_opts:
encrypted: "true"
volumes:
elastic_data:
external: true
Try re-creating without encryption enabled to see if that works.
Also be sure you have a Security group between the three nodes with all the proper ports open between them.