I'm trying to deploy a simple node and mysql server/database with elastic beanstalk. I've successfully containerized the app so I know my docker-compose.yml file is working. Now I'm trying to convert that file to a Dockerrun.aws.json file and deploy using elastic beanstalk. I get the same error everytime I use eb create which is:
ERROR No ecs task definition (or empty definition file) found in environment
So first I tried writing this dockerrun.aws.json file manually, but to no avail. Then I found the container-transform package which supposedly converts these files to different formats. I had a coworker run it for me since I have issues with certain packages referring to python2.7 default on mac instead of 3. Anyways, I then used this file that container-transform gave me and modified a few values to match other examples of dockerrun.aws.json files I've seen, such as memory allotment, and the correct images.
This is my dockerrun.aws.json file:
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"environment": [
{
"name": "NODE_ENV",
"value": "${NODE_ENV}"
},
{
"name": "MYSQL_URI",
"value": "${mysqlcontainer}"
}
],
"essential": true,
"image": "node:latest",
"links": [
"mysqlcontainer"
],
"memory": 512,
"name": "fec-api",
"mountPoints": [
{
"containerPath": "/var/lib/mysql",
"sourceVolume": "My-Db"
}
],
"portMappings": [
{
"containerPort": 3000,
"hostPort": 3000
}
]
},
{
"environment": [
{
"name": "MYSQL_DATABASE",
"value": "fec2"
},
{
"name": "MYSQL_USER",
"value": "fec"
},
{
"name": "MYSQL_PASSWORD",
"value": "fecpassword"
},
{
"name": "MYSQL_ROOT_PASSWORD",
"value": "rootpassword"
}
],
"essential": true,
"image": "mysql:5.7",
"memory": 512,
"mountPoints": [
{
"containerPath": "/var/lib/mysql",
"sourceVolume": "My-Db"
}
],
"name": "mysqlcontainer",
"portMappings": [
{
"containerPort": 3306,
"hostPort": 4306
}
]
}
],
"family": "",
"volumes": [
{
"host": {
"sourcePath": "my-db"
},
"name": "My-Db"
}
]
}
This is my docker-compose.yml file, which works locally:
version: '3'
services:
mysqlcontainer:
container_name: mysqlcontainer
build: ./db/
# image: mysql:5.7
restart: always
environment:
MYSQL_DATABASE: 'fec2'
MYSQL_USER: 'fec'
MYSQL_PASSWORD: 'fecpassword'
MYSQL_ROOT_PASSWORD: 'rootpassword'
ports:
- '4306:3306'
# expose:
# - '3306'
# Where our data will be persisted
volumes:
- my-db:/var/lib/mysql
# - /usr/src/init:/docker-entrypoint-initdb.d/
fec-api:
container_name: fec-api
build: ./server/
ports:
- '3000:3000' # expose ports - HOST:CONTAINER
environment:
- NODE_ENV=${NODE_ENV}
- MYSQL_URI=${mysqlcontainer}
depends_on:
- "mysqlcontainer"
links:
- "mysqlcontainer"
volumes:
my-db:
I expected to get a green checkmark on the elastic beanstalk dashboard after running eb init and eb create. When I run eb init, it does recognize that I have docker configurations so it must know I have the dockerrun.aws.json file in the root directory, but when I eb create I get the error. Any idea how to diagnose this issue, or how to properly compose the docker-compose.yml equivalent for the dockerrun.aws.json file?
Related
The trouble is that I have an issue when I am trying to deploy the Mailhog and run it with the auth file via AWS EB in the Docker platform. The issue is:
[HTTP] Error reading auth-file: open /app/bin/auth.txt: no such file or directory. Check snapshot logs for details. (ElasticBeanstalk::ExternalInvocationError)
caused by: dcedd77ced7be4bab370671cfe92488db824bd705302ee903ca9aac50232a577
The zip file that I am packaging contains two files:
auth.txt
Dockerrun.aws.json
Below you can find the Dockerrun.aws.json content:
{
"AWSEBDockerrunVersion": "1",
"Image": {
"Name": "mailhog/mailhog",
"Update": "true"
},
"Ports": [
{
"HostPort": 1025,
"ContainerPort": 1025
},
{
"HostPort": 80,
"ContainerPort": 8025
}
],
"Volumes": [
{
"HostDirectory": "/var/app/mailhog",
"ContainerDirectory": "/var/app/mailhog"
}
],
"Entrypoint": "MailHog",
"Command": "-auth-file=/var/app/auth.txt"
}
So the question is how can I specify the path to my auth file that is stored in a ZIP package?
In Docker Compose to communicate between images is used name of service. For example:
In docker-compose.yml file should be defined
depends_on:
- database
This dependency can be used in:
"server=database;uid=root;pwd=root;database=database"
Mainly name of defined services in docker-compose.yml file indicate hostname. I use AWS Elastic Beanstalk to deploy my microservices architecture to the cloud and when I run local run by Dockerrun.aws.json generated by container-transform this dependency is not available.
My question is. Do I do some wrong?
Does dependency like in Docker Compose available from WS Elastic Beanstalk?
In my real examples. Parts of docker-compose.yml
version: '3'
services:
rabbitmq: # login guest:guest
image: rabbitmq:3-management
hostname: "rabbitmq"
labels:
NAME: "rabbitmq"
ports:
- "4369:4369"
- "5671:5671"
- "5672:5672"
- "25672:25672"
- "15671:15671"
- "15672:15672"
xms.accounts:
image: ditrikss/accounts
build: ./Microservices/Account/Xms
restart: always
ports:
- 6001:80
depends_on:
- xdb.accounts
- rabbitmq
environment:
- ASPNETCORE_ENVIRONMENT=Production
xdb.accounts:
image: mysql/mysql-server
restart: always
environment:
MYSQL_DATABASE: 'xdb_accounts'
MYSQL_USER: 'root'
MYSQL_PASSWORD: 'root'
MYSQL_ROOT_PASSWORD: 'root'
ports:
- '6002:3306'
volumes:
- "./Databases/Scripts/xdb_Accounts/Create/1_accounts.sql:/docker-entrypoint-initdb.d/1.sql"
- "./Databases/Scripts/xdb_Accounts/Create/2_passwords.sql:/docker-entrypoint-initdb.d/2.sql"
- "./Databases/Scripts/xdb_Accounts/Create/3_channel_features.sql:/docker-entrypoint-initdb.d/3.sql"
- "./Databases/Scripts/xdb_Accounts/Create/4_streaming_features.sql:/docker-entrypoint-initdb.d/4.sql"
And reflecting code of Dockerrun.aws.json file
{
"AWSEBDockerrunVersion": "2",
"containerDefinitions": [
{
"dockerLabels": {
"NAME": "rabbitmq"
},
"essential": true,
"image": "rabbitmq:3-management",
"name": "rabbitmq",
"portMappings": [
{
"containerPort": 4369,
"hostPort": 4369
},
{
"containerPort": 5671,
"hostPort": 5671
},
{
"containerPort": 5672,
"hostPort": 5672
},
{
"containerPort": 25672,
"hostPort": 25672
},
{
"containerPort": 15671,
"hostPort": 15671
},
{
"containerPort": 15672,
"hostPort": 15672
}
]
}
{
"environment": [
{
"name": "MYSQL_DATABASE",
"value": "xdb_accounts"
},
{
"name": "MYSQL_USER",
"value": "root"
},
{
"name": "MYSQL_PASSWORD",
"value": "root"
},
{
"name": "MYSQL_ROOT_PASSWORD",
"value": "root"
}
],
"essential": true,
"image": "mysql/mysql-server",
"mountPoints": [
{
"containerPath": "/docker-entrypoint-initdb.d/1.sql",
"sourceVolume": "_DatabasesScriptsXdb_AccountsCreate1_Accounts_Sql"
},
{
"containerPath": "/docker-entrypoint-initdb.d/2.sql",
"sourceVolume": "_DatabasesScriptsXdb_AccountsCreate2_Passwords_Sql"
},
{
"containerPath": "/docker-entrypoint-initdb.d/3.sql",
"sourceVolume": "_DatabasesScriptsXdb_AccountsCreate3_Channel_Features_Sql"
},
{
"containerPath": "/docker-entrypoint-initdb.d/4.sql",
"sourceVolume": "_DatabasesScriptsXdb_AccountsCreate4_Streaming_Features_Sql"
}
],
"name": "xdb.accounts",
"portMappings": [
{
"containerPort": 3306,
"hostPort": 6002
}
]
},
{
"environment": [
{
"name": "ASPNETCORE_ENVIRONMENT",
"value": "Production"
}
],
"essential": true,
"image": "ditrikss/accounts",
"name": "xms.accounts",
"portMappings": [
{
"containerPort": 80,
"hostPort": 6001
}
]
}
]
}
Thanks in advance!
According to Dockerrun.aws.json v2 reference, you should add links section in your Dockerrun.aws.json file:
Definition of links:
List of containers to link to. Linked containers can discover
each other and communicate securely.
Example usage:
{
"name": "nginx-proxy",
"image": "nginx",
"essential": true,
"memory": 128,
"portMappings": [{
"hostPort": 80,
"containerPort": 80
}],
"links": [
"php-app"
],
"mountPoints": [
{
"sourceVolume": "php-app",
"containerPath": "/var/www/html",
"readOnly": true
}
]
}
I'm using eb local run to locally test out a multicontainer application that I plan on deploying to Elastic Beanstalk. Before deciding to move to EB, I was using docker-compose to spin up the containers. Here's what my docker-compose.yml looked like.
docker-compose.yml:
version: '3.7'
services:
web:
build:
context: ./app
dockerfile: Dockerfile.prod
image: <ECR-Repo>:web
command: gunicorn oddstracker_admin.wsgi:application --bind 0.0.0.0:8000
expose:
- 8000
env_file:
- .env.staging
nginx-proxy:
container_name: nginx-proxy
build: nginx
image: <ECR-Repo>:nginx-proxy
restart: always
ports:
- 443:443
- 80:80
volumes:
- certs:/etc/nginx/certs
- html:/usr/share/nginx/html
- vhost:/etc/nginx/vhost.d
- /var/run/docker.sock:/tmp/docker.sock:ro
depends_on:
- web
nginx-proxy-letsencrypt:
image: jrcs/letsencrypt-nginx-proxy-companion
env_file:
- .env.staging.proxy-companion
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- certs:/etc/nginx/certs
- html:/usr/share/nginx/html
- vhost:/etc/nginx/vhost.d
depends_on:
- nginx-proxy
volumes:
certs:
html:
vhost:
An important aspect (at least what I think is important) is that the nginx-proxy service depends on the web and the nginx-proxy-letsencrypt service depends on nginx-proxy. This ensured that each container would not spin up until the previous one was ready.
When I moved to EB, I was forced to write a Dockerrun.aws.json file in order to run eb local run. Here it is.
Dockerrun.aws.json:
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"command": [
"gunicorn",
"oddstracker_admin.wsgi:application",
"--bind",
"0.0.0.0:8000"
],
"essential": true,
"image": "<ECR-Repo>:web",
"name": "web"
},
{
"essential": true,
"image": "<ECR-Repo>:nginx-proxy",
"mountPoints": [
{
"containerPath": "/etc/nginx/certs",
"sourceVolume": "Certs"
},
{
"containerPath": "/usr/share/nginx/html",
"sourceVolume": "Html"
},
{
"containerPath": "/etc/nginx/vhost.d",
"sourceVolume": "Vhost"
},
{
"containerPath": "/tmp/docker.sock",
"sourceVolume": "VarRunDocker_Sock"
}
],
"name": "nginx-proxy",
"portMappings": [
{
"containerPort": 443,
"hostPort": 443
},
{
"containerPort": 80,
"hostPort": 80
}
],
"links": [
"web"
]
},
{
"essential": true,
"image": "jrcs/letsencrypt-nginx-proxy-companion",
"mountPoints": [
{
"containerPath": "/var/run/docker.sock",
"sourceVolume": "VarRunDocker_Sock"
},
{
"containerPath": "/etc/nginx/certs",
"sourceVolume": "Certs"
},
{
"containerPath": "/usr/share/nginx/html",
"sourceVolume": "Html"
},
{
"containerPath": "/etc/nginx/vhost.d",
"sourceVolume": "Vhost"
}
],
"name": "nginx-proxy-letsencrypt",
"links": [
"nginx-proxy"
]
}
],
"family": "",
"volumes": [
{
"host": {
"sourcePath": "certs"
},
"name": "Certs"
},
{
"host": {
"sourcePath": "html"
},
"name": "Html"
},
{
"host": {
"sourcePath": "vhost"
},
"name": "Vhost"
},
{
"host": {
"sourcePath": "/var/run/docker.sock"
},
"name": "VarRunDocker_Sock"
}
]
}
When it autogenerates a docker-compose.yml to run the application, there is no way to provide a depends_on flag for the various services. Specifically, when I run eb local run, nginx-proxy-letsencrypt runs before nginx-proxy is able to finish and fails the entire process.
So my question: Has anyone found a way to solve this issue, possibly with an additional set of commands within their Dockerrun.aws.json?
When I moved to EB, I was forced to write a Dockerrun.aws.json ?
I don't see a reason why not to use docker-compose.yml. EB supports docker compose:
Docker Compose features. This platform will allow you to leverage the features provided by the Docker Compose tool to define and run multiple containers. You can include the docker-compose.yml file to deploy to Elastic Beanstalk.
So if you have working docker-compose file, my recommendation would be to try to use it.
I am following the instructions at https://docs.docker.com/compose/django/ to get a basic dockerized django app going. I am able to run it locally without a problem but I am having trouble to deploy it to AWS using Elastic Beanstalk. After reading here, I figured that I need to translate docker-compose.yml into Dockerrun.aws.json for it to work.
The original docker-compose.yml is
version: '2'
services:
db:
image: postgres
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
depends_on:
- db
and here is what I translated so far
{
"AWSEBDockerrunVersion": 2,
"volumes": [
{
"name": "db"
},
{
"name": "web"
}
],
"containerDefinitions": [
{
"name": "db",
"image": "postgres",
"essential": true,
"memory": 256,
"mountPoints": [
{
"sourceVolume": "db"
"containerPath": "/var/app/current/db"
}
]
},
{
"name": "web",
"image": "web",
"essential": true,
"memory": 256,
"mountPoints": [
{
"sourceVolume": "web"
"containerPath": "/var/app/current/web"
}
],
"portMappings": [
{
"hostPort": 8000,
"containerPort": 8000
}
],
"links": [
"db"
],
"command": "python manage.py runserver 0.0.0.0:8000"
}
]
}
but it's not working. What am I doing wrong?
I was struggling to get the ins and outs of the Dockerrun format. Check out Container Transform: "Transforms docker-compose, ECS, and Marathon configurations"... it's a life-saver. Here is what it outputs for your example:
{
"containerDefinitions": [
{
"essential": true,
"image": "postgres",
"name": "db"
},
{
"command": [
"python",
"manage.py",
"runserver",
"0.0.0.0:8000"
],
"essential": true,
"mountPoints": [
{
"containerPath": "/code",
"sourceVolume": "_"
}
],
"name": "web",
"portMappings": [
{
"containerPort": 8000,
"hostPort": 8000
}
]
}
],
"family": "",
"volumes": [
{
"host": {
"sourcePath": "."
},
"name": "_"
}
]
}
Container web is missing required parameter "image".
Container web is missing required parameter "memory".
Container db is missing required parameter "memory".
That is, in this new format, you must tell it how much memory to allot each container. Also, you need to provide an image - there is no option to build. As is mentioned in the comments, you want to build and push to DockerHub or ECR, then give it that location: eg [org name]/[repo]:latest on Dockerhub, or the URL for ECR. But container-transform does the mountPoints and volumes for you - it's amazing.
You have a few issues.
1) 'web' doesn't appear to be an 'image', you define it as 'build . ' in your docker-compose.. Remember, the Dockerrun.aws.json will have to pull the image from somewhere (easiest is to use ECS's Repositories)
2) I think 'command' is an array. So you'd have:
"command": ["python" "manage.py" "runserver" "0.0.0.0:8000"]
3) your mountPoints are correct, but the volume definition at the top is wrong.
{
"name": "web",
"host": {
"sourcePath": "/var/app/current/db"
}
Im not 100% certain, but the path works for me.
if you have the Dockerrun.aws.json file, next to is a directory called /db .. then that will be the mount location.
I have an environment with a few containers. Some of them are linked. When I run the environment with "docker-compose up -d", it creates entries in etc/hosts for linked containers. When I run it with "eb local run", no entries are created. Why is that?
My Dockerrun.aws.json
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"name": "api",
"image": "php7",
"essential": true,
"memory": 128,
"portMappings": [
{
"hostPort": 8080,
"containerPort": 80
}
],
"mountPoints": [
{
"sourceVolume": "api",
"containerPath": "/var/www/html/"
}
]
},
{
"name": "nodeapi",
"image": "nodejs",
"essential": true,
"memory": 256,
"portMappings": [
{
"hostPort": 5000,
"containerPort": 5000
}
],
"mountPoints": [
{
"sourceVolume": "nodeapi",
"containerPath": "/var/www/app/"
}
],
"Logging": "/var/eb_log"
},
{
"name": "proxy",
"image": "nginx",
"essential": true,
"memory": 128,
"links": [
"api",
"nodeapi"
],
"portMappings": [
{
"hostPort": 8443,
"containerPort": 80
}
]
}
]
}
This generates docker-compose.yml:
api:
image: php7
ports:
- 8080:80
nodeapi:
image: nodejs
ports:
- 5000:5000
proxy:
image: nginx
links:
- api:api
- nodeapi:nodeapi
ports:
- 8443:80
Docker switched to DNS based lookups a while back instead of adding entries to /etc/hosts. Linking is also discouraged in favor of using a common network for the containers.
Ok, this is was a local issue. I upgraded Docker and EB cli to the latest versions and this solved the issue. I'm not sure why EB cli failed to add aliases to etc/hosts previously, but after upgrade it does. Now I get same results either by using "docker-compose up" or "eb local run". All linked container are linked now and work as expected.