Docker compose depedency pass to AWS Elastic Beanstalk - amazon-web-services

In Docker Compose to communicate between images is used name of service. For example:
In docker-compose.yml file should be defined
depends_on:
- database
This dependency can be used in:
"server=database;uid=root;pwd=root;database=database"
Mainly name of defined services in docker-compose.yml file indicate hostname. I use AWS Elastic Beanstalk to deploy my microservices architecture to the cloud and when I run local run by Dockerrun.aws.json generated by container-transform this dependency is not available.
My question is. Do I do some wrong?
Does dependency like in Docker Compose available from WS Elastic Beanstalk?
In my real examples. Parts of docker-compose.yml
version: '3'
services:
rabbitmq: # login guest:guest
image: rabbitmq:3-management
hostname: "rabbitmq"
labels:
NAME: "rabbitmq"
ports:
- "4369:4369"
- "5671:5671"
- "5672:5672"
- "25672:25672"
- "15671:15671"
- "15672:15672"
xms.accounts:
image: ditrikss/accounts
build: ./Microservices/Account/Xms
restart: always
ports:
- 6001:80
depends_on:
- xdb.accounts
- rabbitmq
environment:
- ASPNETCORE_ENVIRONMENT=Production
xdb.accounts:
image: mysql/mysql-server
restart: always
environment:
MYSQL_DATABASE: 'xdb_accounts'
MYSQL_USER: 'root'
MYSQL_PASSWORD: 'root'
MYSQL_ROOT_PASSWORD: 'root'
ports:
- '6002:3306'
volumes:
- "./Databases/Scripts/xdb_Accounts/Create/1_accounts.sql:/docker-entrypoint-initdb.d/1.sql"
- "./Databases/Scripts/xdb_Accounts/Create/2_passwords.sql:/docker-entrypoint-initdb.d/2.sql"
- "./Databases/Scripts/xdb_Accounts/Create/3_channel_features.sql:/docker-entrypoint-initdb.d/3.sql"
- "./Databases/Scripts/xdb_Accounts/Create/4_streaming_features.sql:/docker-entrypoint-initdb.d/4.sql"
And reflecting code of Dockerrun.aws.json file
{
"AWSEBDockerrunVersion": "2",
"containerDefinitions": [
{
"dockerLabels": {
"NAME": "rabbitmq"
},
"essential": true,
"image": "rabbitmq:3-management",
"name": "rabbitmq",
"portMappings": [
{
"containerPort": 4369,
"hostPort": 4369
},
{
"containerPort": 5671,
"hostPort": 5671
},
{
"containerPort": 5672,
"hostPort": 5672
},
{
"containerPort": 25672,
"hostPort": 25672
},
{
"containerPort": 15671,
"hostPort": 15671
},
{
"containerPort": 15672,
"hostPort": 15672
}
]
}
{
"environment": [
{
"name": "MYSQL_DATABASE",
"value": "xdb_accounts"
},
{
"name": "MYSQL_USER",
"value": "root"
},
{
"name": "MYSQL_PASSWORD",
"value": "root"
},
{
"name": "MYSQL_ROOT_PASSWORD",
"value": "root"
}
],
"essential": true,
"image": "mysql/mysql-server",
"mountPoints": [
{
"containerPath": "/docker-entrypoint-initdb.d/1.sql",
"sourceVolume": "_DatabasesScriptsXdb_AccountsCreate1_Accounts_Sql"
},
{
"containerPath": "/docker-entrypoint-initdb.d/2.sql",
"sourceVolume": "_DatabasesScriptsXdb_AccountsCreate2_Passwords_Sql"
},
{
"containerPath": "/docker-entrypoint-initdb.d/3.sql",
"sourceVolume": "_DatabasesScriptsXdb_AccountsCreate3_Channel_Features_Sql"
},
{
"containerPath": "/docker-entrypoint-initdb.d/4.sql",
"sourceVolume": "_DatabasesScriptsXdb_AccountsCreate4_Streaming_Features_Sql"
}
],
"name": "xdb.accounts",
"portMappings": [
{
"containerPort": 3306,
"hostPort": 6002
}
]
},
{
"environment": [
{
"name": "ASPNETCORE_ENVIRONMENT",
"value": "Production"
}
],
"essential": true,
"image": "ditrikss/accounts",
"name": "xms.accounts",
"portMappings": [
{
"containerPort": 80,
"hostPort": 6001
}
]
}
]
}
Thanks in advance!

According to Dockerrun.aws.json v2 reference, you should add links section in your Dockerrun.aws.json file:
Definition of links:
List of containers to link to. Linked containers can discover
each other and communicate securely.
Example usage:
{
"name": "nginx-proxy",
"image": "nginx",
"essential": true,
"memory": 128,
"portMappings": [{
"hostPort": 80,
"containerPort": 80
}],
"links": [
"php-app"
],
"mountPoints": [
{
"sourceVolume": "php-app",
"containerPath": "/var/www/html",
"readOnly": true
}
]
}

Related

Elastic Beanstalk: `eb local run` starts containers out of order, causing startup to fail

I'm using eb local run to locally test out a multicontainer application that I plan on deploying to Elastic Beanstalk. Before deciding to move to EB, I was using docker-compose to spin up the containers. Here's what my docker-compose.yml looked like.
docker-compose.yml:
version: '3.7'
services:
web:
build:
context: ./app
dockerfile: Dockerfile.prod
image: <ECR-Repo>:web
command: gunicorn oddstracker_admin.wsgi:application --bind 0.0.0.0:8000
expose:
- 8000
env_file:
- .env.staging
nginx-proxy:
container_name: nginx-proxy
build: nginx
image: <ECR-Repo>:nginx-proxy
restart: always
ports:
- 443:443
- 80:80
volumes:
- certs:/etc/nginx/certs
- html:/usr/share/nginx/html
- vhost:/etc/nginx/vhost.d
- /var/run/docker.sock:/tmp/docker.sock:ro
depends_on:
- web
nginx-proxy-letsencrypt:
image: jrcs/letsencrypt-nginx-proxy-companion
env_file:
- .env.staging.proxy-companion
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- certs:/etc/nginx/certs
- html:/usr/share/nginx/html
- vhost:/etc/nginx/vhost.d
depends_on:
- nginx-proxy
volumes:
certs:
html:
vhost:
An important aspect (at least what I think is important) is that the nginx-proxy service depends on the web and the nginx-proxy-letsencrypt service depends on nginx-proxy. This ensured that each container would not spin up until the previous one was ready.
When I moved to EB, I was forced to write a Dockerrun.aws.json file in order to run eb local run. Here it is.
Dockerrun.aws.json:
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"command": [
"gunicorn",
"oddstracker_admin.wsgi:application",
"--bind",
"0.0.0.0:8000"
],
"essential": true,
"image": "<ECR-Repo>:web",
"name": "web"
},
{
"essential": true,
"image": "<ECR-Repo>:nginx-proxy",
"mountPoints": [
{
"containerPath": "/etc/nginx/certs",
"sourceVolume": "Certs"
},
{
"containerPath": "/usr/share/nginx/html",
"sourceVolume": "Html"
},
{
"containerPath": "/etc/nginx/vhost.d",
"sourceVolume": "Vhost"
},
{
"containerPath": "/tmp/docker.sock",
"sourceVolume": "VarRunDocker_Sock"
}
],
"name": "nginx-proxy",
"portMappings": [
{
"containerPort": 443,
"hostPort": 443
},
{
"containerPort": 80,
"hostPort": 80
}
],
"links": [
"web"
]
},
{
"essential": true,
"image": "jrcs/letsencrypt-nginx-proxy-companion",
"mountPoints": [
{
"containerPath": "/var/run/docker.sock",
"sourceVolume": "VarRunDocker_Sock"
},
{
"containerPath": "/etc/nginx/certs",
"sourceVolume": "Certs"
},
{
"containerPath": "/usr/share/nginx/html",
"sourceVolume": "Html"
},
{
"containerPath": "/etc/nginx/vhost.d",
"sourceVolume": "Vhost"
}
],
"name": "nginx-proxy-letsencrypt",
"links": [
"nginx-proxy"
]
}
],
"family": "",
"volumes": [
{
"host": {
"sourcePath": "certs"
},
"name": "Certs"
},
{
"host": {
"sourcePath": "html"
},
"name": "Html"
},
{
"host": {
"sourcePath": "vhost"
},
"name": "Vhost"
},
{
"host": {
"sourcePath": "/var/run/docker.sock"
},
"name": "VarRunDocker_Sock"
}
]
}
When it autogenerates a docker-compose.yml to run the application, there is no way to provide a depends_on flag for the various services. Specifically, when I run eb local run, nginx-proxy-letsencrypt runs before nginx-proxy is able to finish and fails the entire process.
So my question: Has anyone found a way to solve this issue, possibly with an additional set of commands within their Dockerrun.aws.json?
When I moved to EB, I was forced to write a Dockerrun.aws.json ?
I don't see a reason why not to use docker-compose.yml. EB supports docker compose:
Docker Compose features. This platform will allow you to leverage the features provided by the Docker Compose tool to define and run multiple containers. You can include the docker-compose.yml file to deploy to Elastic Beanstalk.
So if you have working docker-compose file, my recommendation would be to try to use it.

How to properly diagnose failed multicontainer docker app deployment to elastic beanstalk?

I'm trying to deploy a simple node and mysql server/database with elastic beanstalk. I've successfully containerized the app so I know my docker-compose.yml file is working. Now I'm trying to convert that file to a Dockerrun.aws.json file and deploy using elastic beanstalk. I get the same error everytime I use eb create which is:
ERROR No ecs task definition (or empty definition file) found in environment
So first I tried writing this dockerrun.aws.json file manually, but to no avail. Then I found the container-transform package which supposedly converts these files to different formats. I had a coworker run it for me since I have issues with certain packages referring to python2.7 default on mac instead of 3. Anyways, I then used this file that container-transform gave me and modified a few values to match other examples of dockerrun.aws.json files I've seen, such as memory allotment, and the correct images.
This is my dockerrun.aws.json file:
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"environment": [
{
"name": "NODE_ENV",
"value": "${NODE_ENV}"
},
{
"name": "MYSQL_URI",
"value": "${mysqlcontainer}"
}
],
"essential": true,
"image": "node:latest",
"links": [
"mysqlcontainer"
],
"memory": 512,
"name": "fec-api",
"mountPoints": [
{
"containerPath": "/var/lib/mysql",
"sourceVolume": "My-Db"
}
],
"portMappings": [
{
"containerPort": 3000,
"hostPort": 3000
}
]
},
{
"environment": [
{
"name": "MYSQL_DATABASE",
"value": "fec2"
},
{
"name": "MYSQL_USER",
"value": "fec"
},
{
"name": "MYSQL_PASSWORD",
"value": "fecpassword"
},
{
"name": "MYSQL_ROOT_PASSWORD",
"value": "rootpassword"
}
],
"essential": true,
"image": "mysql:5.7",
"memory": 512,
"mountPoints": [
{
"containerPath": "/var/lib/mysql",
"sourceVolume": "My-Db"
}
],
"name": "mysqlcontainer",
"portMappings": [
{
"containerPort": 3306,
"hostPort": 4306
}
]
}
],
"family": "",
"volumes": [
{
"host": {
"sourcePath": "my-db"
},
"name": "My-Db"
}
]
}
This is my docker-compose.yml file, which works locally:
version: '3'
services:
mysqlcontainer:
container_name: mysqlcontainer
build: ./db/
# image: mysql:5.7
restart: always
environment:
MYSQL_DATABASE: 'fec2'
MYSQL_USER: 'fec'
MYSQL_PASSWORD: 'fecpassword'
MYSQL_ROOT_PASSWORD: 'rootpassword'
ports:
- '4306:3306'
# expose:
# - '3306'
# Where our data will be persisted
volumes:
- my-db:/var/lib/mysql
# - /usr/src/init:/docker-entrypoint-initdb.d/
fec-api:
container_name: fec-api
build: ./server/
ports:
- '3000:3000' # expose ports - HOST:CONTAINER
environment:
- NODE_ENV=${NODE_ENV}
- MYSQL_URI=${mysqlcontainer}
depends_on:
- "mysqlcontainer"
links:
- "mysqlcontainer"
volumes:
my-db:
I expected to get a green checkmark on the elastic beanstalk dashboard after running eb init and eb create. When I run eb init, it does recognize that I have docker configurations so it must know I have the dockerrun.aws.json file in the root directory, but when I eb create I get the error. Any idea how to diagnose this issue, or how to properly compose the docker-compose.yml equivalent for the dockerrun.aws.json file?

Prisma error when trying to run with elastic beanstalk

I have a prisma project that works fine locally when I run $ docker-compose up. I converted the docker-compose.yml file to Dockerrun.aws.json. But now when i try to run the project locally via $ eb local run I get an error
mysql_1 | Version: '5.7.24' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL)
prisma_1 | Exception in thread "main" java.sql.SQLTransientConnectionException: database - Connection is not available, request timed out after 5001ms.
Below is my Dockerrun.aws.json file:
{
"AWSEBDockerrunVersion": "2",
"containerDefinitions": [
{
"environment": [
{
"name": "MYSQL_ROOT_PASSWORD",
"value": "prisma"
}
],
"essential": true,
"memory": 128,
"image": "mysql:5.7",
"mountPoints": [
{
"containerPath": "/var/lib/mysql",
"sourceVolume": "Mysql"
}
],
"name": "mysql",
"portMappings": [
{
"containerPort": 3306,
"hostPort": 3306
}
]
},
{
"environment": [
{
"name": "PRISMA_CONFIG",
"value": "port: 4466\ndatabases:\n default:\n connector: mysql\n host: mysql\n port: 3306\n user: root\n password: prisma\n migrations: true\n"
}
],
"essential": true,
"memory": 128,
"image": "prismagraphql/prisma:1.21",
"name": "prisma",
"portMappings": [
{
"containerPort": 4466,
"hostPort": 4466
}
]
}
],
"family": "",
"volumes": [
{
"host": {
"sourcePath": "mysql"
},
"name": "Mysql"
}
]
}
The error message leads me to believe that there's an issue connecting the prisma container to the mysql instance. If i had to guess it's the PRISMA_CONFIG value but not I'm not 100% sure. Can someone tell me what I'm doing wrong here?
You can not have those /n in there. YAML cares about real carriage and spaces.

How to translate docker-compose.yml to Dockerrun.aws.json for Django

I am following the instructions at https://docs.docker.com/compose/django/ to get a basic dockerized django app going. I am able to run it locally without a problem but I am having trouble to deploy it to AWS using Elastic Beanstalk. After reading here, I figured that I need to translate docker-compose.yml into Dockerrun.aws.json for it to work.
The original docker-compose.yml is
version: '2'
services:
db:
image: postgres
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
depends_on:
- db
and here is what I translated so far
{
"AWSEBDockerrunVersion": 2,
"volumes": [
{
"name": "db"
},
{
"name": "web"
}
],
"containerDefinitions": [
{
"name": "db",
"image": "postgres",
"essential": true,
"memory": 256,
"mountPoints": [
{
"sourceVolume": "db"
"containerPath": "/var/app/current/db"
}
]
},
{
"name": "web",
"image": "web",
"essential": true,
"memory": 256,
"mountPoints": [
{
"sourceVolume": "web"
"containerPath": "/var/app/current/web"
}
],
"portMappings": [
{
"hostPort": 8000,
"containerPort": 8000
}
],
"links": [
"db"
],
"command": "python manage.py runserver 0.0.0.0:8000"
}
]
}
but it's not working. What am I doing wrong?
I was struggling to get the ins and outs of the Dockerrun format. Check out Container Transform: "Transforms docker-compose, ECS, and Marathon configurations"... it's a life-saver. Here is what it outputs for your example:
{
"containerDefinitions": [
{
"essential": true,
"image": "postgres",
"name": "db"
},
{
"command": [
"python",
"manage.py",
"runserver",
"0.0.0.0:8000"
],
"essential": true,
"mountPoints": [
{
"containerPath": "/code",
"sourceVolume": "_"
}
],
"name": "web",
"portMappings": [
{
"containerPort": 8000,
"hostPort": 8000
}
]
}
],
"family": "",
"volumes": [
{
"host": {
"sourcePath": "."
},
"name": "_"
}
]
}
Container web is missing required parameter "image".
Container web is missing required parameter "memory".
Container db is missing required parameter "memory".
That is, in this new format, you must tell it how much memory to allot each container. Also, you need to provide an image - there is no option to build. As is mentioned in the comments, you want to build and push to DockerHub or ECR, then give it that location: eg [org name]/[repo]:latest on Dockerhub, or the URL for ECR. But container-transform does the mountPoints and volumes for you - it's amazing.
You have a few issues.
1) 'web' doesn't appear to be an 'image', you define it as 'build . ' in your docker-compose.. Remember, the Dockerrun.aws.json will have to pull the image from somewhere (easiest is to use ECS's Repositories)
2) I think 'command' is an array. So you'd have:
"command": ["python" "manage.py" "runserver" "0.0.0.0:8000"]
3) your mountPoints are correct, but the volume definition at the top is wrong.
{
"name": "web",
"host": {
"sourcePath": "/var/app/current/db"
}
Im not 100% certain, but the path works for me.
if you have the Dockerrun.aws.json file, next to is a directory called /db .. then that will be the mount location.

Elastic Beanstalk Multicontainer Docker environment: no entries in etc/hosts for lined containers

I have an environment with a few containers. Some of them are linked. When I run the environment with "docker-compose up -d", it creates entries in etc/hosts for linked containers. When I run it with "eb local run", no entries are created. Why is that?
My Dockerrun.aws.json
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"name": "api",
"image": "php7",
"essential": true,
"memory": 128,
"portMappings": [
{
"hostPort": 8080,
"containerPort": 80
}
],
"mountPoints": [
{
"sourceVolume": "api",
"containerPath": "/var/www/html/"
}
]
},
{
"name": "nodeapi",
"image": "nodejs",
"essential": true,
"memory": 256,
"portMappings": [
{
"hostPort": 5000,
"containerPort": 5000
}
],
"mountPoints": [
{
"sourceVolume": "nodeapi",
"containerPath": "/var/www/app/"
}
],
"Logging": "/var/eb_log"
},
{
"name": "proxy",
"image": "nginx",
"essential": true,
"memory": 128,
"links": [
"api",
"nodeapi"
],
"portMappings": [
{
"hostPort": 8443,
"containerPort": 80
}
]
}
]
}
This generates docker-compose.yml:
api:
image: php7
ports:
- 8080:80
nodeapi:
image: nodejs
ports:
- 5000:5000
proxy:
image: nginx
links:
- api:api
- nodeapi:nodeapi
ports:
- 8443:80
Docker switched to DNS based lookups a while back instead of adding entries to /etc/hosts. Linking is also discouraged in favor of using a common network for the containers.
Ok, this is was a local issue. I upgraded Docker and EB cli to the latest versions and this solved the issue. I'm not sure why EB cli failed to add aliases to etc/hosts previously, but after upgrade it does. Now I get same results either by using "docker-compose up" or "eb local run". All linked container are linked now and work as expected.