I have the following minimal docker-compose.yml:
worker:
working_dir: /app
image: <my-repo>.dkr.ecr.us-east-1.amazonaws.com/ocean-boiler:latest
cpu_shares: 4096
mem_limit: 524288000
command: /bin/bash -c "bin/delayed-job --pool=*:1"
When I run it locally it using docker-compose, love and happiness.
When I request ECS runs it remotely, I get the following:
ecs-cli up
=>
time="2016-05-03T11:40:00-07:00" level=info msg="Stopped container..." container="<cid-redacted>/worker" desiredStatus=STOPPED lastStatus=STOPPED taskDefinition="ecscompose-spud:73"
then; we check the fallout using ps:
ecs-cli ps
=>
<cid-redacted>/worker STOPPED Reason: DockerStateError: [8] System error: exec: "/bin/bash -c \"bin/delayed-job --pool=*:1\"": stat /bin/bash -c "bin/delayed-job --pool=*:1": no such file or directory ecscompose-spud:73
I've been down the rabbit hole of not referring to any files without complete paths. My docker instance functions as intended whether I run it locally or on a remote machine, however ordering it about with ecs-cli seems to be sad-panda business.
Just running it locally with docker-compose up functions as intended... Any halp would be appreciated!
EDIT: Finally fixed. Using command: categorically worked super bad for me - my docker containers now contain the command needed to run; and my advice is to avoid the use of command unless you really need it.
Related
Very similar to this question, I cannot connect to my local docker-compose container from my browser (Firefox) on Windows 10 and have been troubleshooting for some time, but I cannot seem to find the issue.
Here is my docker-compose.yml:
version: "3"
services:
frontend:
container_name: frontend
build: ./frontend
ports:
- "3000:3000"
working_dir: /home/node/app/
environment:
DEVELOPMENT: "yes"
stdin_open: true
volumes:
- ./frontend:/home/node/app/
command: bash -c "npm start & npm run build"
my_app_django:
container_name: my_app_django
build: ./backend/
environment:
SECRET_KEY: "... not included ..."
command: ["./rundjango.sh"]
volumes:
- ./backend:/code
- media_volume:/code/media
- static_volume:/code/static
expose:
- "443"
my_app_nginx:
container_name: my_app_nginx
image: nginx:1.17.2-alpine
volumes:
- ./nginx/nginx.dev.conf:/etc/nginx/conf.d/default.conf
- static_volume:/home/app/web/staticfiles
- media_volume:/home/app/web/mediafiles
- ./frontend:/home/app/frontend/
ports:
- "80:80"
depends_on:
- my_app_django
volumes:
static_volume:
media_volume:
I can start the containers with docker-compose -f docker-compose.yml up -d and there are no errors when I check the logs with docker logs my_app_django or docker logs my_app_nginx. Additionally, doing docker ps shows all the containers running as they should.
The odd part about this issue is that on Linux, everything runs without issue and I can find my app on localhost at port 80. The only thing I do differently when I am on Windows is that I run a dos2unix on my .sh files to ensure that they run properly. If I omit this step, then I get many errors which leads me to believe that I have to do this.
If anyone could give guidance/advice as to what may I be doing incorrectly or missing altogether, I would be truly grateful. I am also happy to provide more details, just let me know. Thank you!
EDIT #1: As timur suggested, I did a docker run -p 80:80 -d nginx and here was the output:
Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
bf5952930446: Pull complete
ba755a256dfe: Pull complete
c57dd87d0b93: Pull complete
d7fbf29df889: Pull complete
1f1070938ccd: Pull complete
Digest: sha256:36b74457bccb56fbf8b05f79c85569501b721d4db813b684391d63e02287c0b2
Status: Downloaded newer image for nginx:latest
19b56a66955145e4f59eefff57340b4affe5f7e0d82ad013742a60b479687c40
C:\Program Files\Docker Toolbox\docker.exe: Error response from daemon: driver failed programming external connectivity on endpoint naughty_hoover (8c7b2fa4aef964899c366e1897e38727bb7e4c38431875c5cb8456567005f368): Bind for 0.0.0.0:80 failed: port is already allocated.
This might be the cause of the error but I don't really understand what needs to be done at this point.
EDIT #2: As requested, here are my Dockerfiles (one for backend, one for frontend)
Backend Dockerfile:
FROM python:3
ENV PYTHONUNBUFFERED 1
RUN apt-get update && apt-get install -y imagemagick libxmlsec1-dev pkg-config
RUN mkdir /code
WORKDIR /code
COPY requirements.txt /code/
RUN pip install -r requirements.txt
COPY . /code
Frontend Dockerfile:
FROM node
WORKDIR /home/node/app/
COPY . /home/node/app/
RUN npm install -g react-scripts
RUN npm install
EDIT #3: When I do docker ps, this is what I get:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0da02ad8d746 nginx:1.17.2-alpine "nginx -g 'daemon of…" About an hour ago Up About an hour 0.0.0.0:80->80/tcp my_app_nginx
070291de8362 my_app_frontend "docker-entrypoint.s…" About an hour ago Up About an hour 0.0.0.0:3000->3000/tcp frontend
2fcf551ce3fa my_app_django "./rundjango.sh" 12 days ago Up About an hour 443/tcp my_app_django
As we established you use Docker Toolbox that is backed by VirtualBox rather than default Hyper-V Docker for Windows. In this case you might think of it as a VBox VM that actually runs Docker - so all volume mounts and port mappings apply to docker machine VM, not your host. And management tools (i.e. Docker terminal and docker-compose) actually run on your host OS through MinGW.
Due to this, you don't get binding ports on localhost by default (but you can achieve this by editing VM properties in VirtualBox manually if you so desire - I just googled the second link for some picture tutorials). Suprisingly, the official documentation on this particular topic is pretty scarce - you can get a hint by looking at their examples though.
So in your case, the correct url should be http://192.168.99.100
Another thing that is different between these two solutions is volume mounts. And again, documentation sorta hints at what it should be but I can't point you a more explicit source. As you have probably noticed the terminal you use for all your docker interactions encodes paths a bit differently (I presume because of that MinGW layer) and converted paths get sent off to docker-machine - because it's Linux and would not handle windows-style paths anyway.
From here I see a couple of avenues for you to explore:
Run your project from C:\Users\...\MyProject
As the documentation states, you get c:\Users mounted into /c/Users by default. So theoretically, if you run your docker-compose from your user home folder - paths should automagically align - but since you are having this issue - you are probably running it from somewhere else.
Create another share
You also can create your own mounting mount in Virtual Box. Run pwd in your terminal and note where project root is. Then use Virtual Vox UI and create a path that would make it align with your directory tree (for example, D:\MyProject\ should become /d/MyProject.
Hopefully this will not require you to change your docker-compose.yml either
Alternatively, switch to Hyper-V Docker Desktop - and these particular issues will go away.
Bear in mind, that Hyper-V will not coexist with VirtualBox. So this option might not be available to you if you need VBox for something else.
i have a django app that also uses cerlery and celery also creates a .pid file but if i restart my app with docker-compose up celery fails to start because the old .pid file within the container does not get deleted,
what can i do to solve this isse?
Is there maybe a way on linux-side (debian 9) to remove the file at restart or shutdown of the container?
snippet of my docker config:
celery:
image: echo_echo
command: bash -c "celery worker -A echo -l INFO --pidfile="/tmp/celery/celery.pid" "
volumes:
- echo:/echo
...
Thanks and br
Got the issue fixed, i simply added the (pid) file like so:
--pidfile= --schedule=/var/run/celerybeat-schedule
and also for the worker(s):
--pidfile= --schedule=/var/run/celery-schedule
I'm quite new to Docker but have started thinking about production set-ups, hence needing to crack the challenge of data persistence when using Docker Swarm. I decided to start by creating my deployment infrastructure (TeamCity for builds and NuGet plus the "registry" [https://hub.docker.com/_/registry/] for storing images).
I've started with TeamCity. Obvious this needs data persistence in order to work. I am able to run TeamCity in a container with an EBS drive and everything looks like it is working just fine - TeamCity is working through the set-up steps and my TeamCity drives appear in AWS EBS, but then the worker node TeamCity gets allocated to shuts down and the install process stops.
Here are all the steps I'm following:
Phase 1 - Machine Setup:
Create one AWS instance for master
Create two AWS instances for workers
All are 64-bit Ubuntu t2.mircro instances
Create three elastic IPs for convenience and assign them to the above machines.
Install Docker on all nodes using this: https://docs.docker.com/install/linux/docker-ce/ubuntu/
Install Docker Machine on all nodes using this: https://docs.docker.com/machine/install-machine/
Install Docker Compose on all nodes using this: https://docs.docker.com/compose/install/
Phase 2 - Configure Docker Remote on the Master:
$ sudo docker run -p 2375:2375 --rm -d -v /var/run/docker.sock:/var/run/docker.sock jarkt/docker-remote-api
Phase 3 - install the rexray/ebs plugin on all machines:
$ sudo docker plugin install --grant-all-permissions rexray/ebs REXRAY_PREEMPT=true EBS_ACCESSKEY=XXX EBS_SECRETKEY=YYY
[I lifted the correct values from AWS for XXX and YYY]
I test this using:
$ sudo docker volume create --driver=rexray/ebs --name=delete --opt=size=2
$ sudo docker volume rm delete
All three nodes are able to create and delete drives in AWS EBS with no issue.
Phase 4 - Setup the swarm:
Run this on the master:
$ sudo docker swarm init --advertise-addr eth0:2377
This gives the command to run on each of the workers, which looks like this:
$ sudo docker swarm join --token XXX 1.2.3.4:2377
These execute fine on the worker machines.
Phase 5 - Set up visualisation using Remote Powershell on my local machine:
$ $env:DOCKER_HOST="{master IP address}:2375"
$ docker stack deploy --with-registry-auth -c viz.yml viz
viz.yml looks like this:
version: '3.1'
services:
viz:
image: dockersamples/visualizer
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
ports:
- "8080:8080"
deploy:
placement:
constraints:
- node.role==manager
This works fine and allows me to visualise my swarm.
Phase 6 - Install TeamCity using Remote Powershell on my local machine:
$ docker stack deploy --with-registry-auth -c docker-compose.yml infra
docker-compose.yml looks like this:
version: '3'
services:
teamcity:
image: jetbrains/teamcity-server:2017.1.2
volumes:
- teamcity-server-datadir:/data/teamcity_server/datadir
- teamcity-server-logs:/opt/teamcity/logs
ports:
- "80:8111"
volumes:
teamcity-server-datadir:
driver: rexray/ebs
teamcity-server-logs:
driver: rexray/ebs
[Incorporating NGINX as a proxy is a later step on my to do list.]
I can see both the required drives appear in AWS EBS and the container appear in my swarm visualisation.
However, after a while of seeing the progress screen in TeamCity the worker machine containing the TeamCity instance shuts down and the process abruptly ends.
I'm at a loss as to what to do next. I'm not even sure where to look for logs.
Any help gratefully received!
Cheers,
Steve.
I found a way to get logs for my service. First do this to list the services the stack creates:
$ sudo docker service ls
Then do this to see logs for the service:
$ sudo docker service logs --details {service name}
Now I just need to wade through the logs and see what went wrong...
Update
I found the following error in the logs:
infra_teamcity.1.bhiwz74gnuio#ip-172-31-18-103 | [2018-05-14 17:38:56,849] ERROR - r.configs.dsl.DslPluginManager - DSL plugin compilation failed
infra_teamcity.1.bhiwz74gnuio#ip-172-31-18-103 | exit code: 1
infra_teamcity.1.bhiwz74gnuio#ip-172-31-18-103 | stdout: #
infra_teamcity.1.bhiwz74gnuio#ip-172-31-18-103 | # There is insufficient memory for the Java Runtime Environment to continue.
infra_teamcity.1.bhiwz74gnuio#ip-172-31-18-103 | # Native memory allocation (mmap) failed to map 42012672 bytes for committing reserved memory.
infra_teamcity.1.bhiwz74gnuio#ip-172-31-18-103 | # An error report file with more information is saved as:
infra_teamcity.1.bhiwz74gnuio#ip-172-31-18-103 | # /opt/teamcity/bin/hs_err_pid125.log
infra_teamcity.1.bhiwz74gnuio#ip-172-31-18-103 |
infra_teamcity.1.bhiwz74gnuio#ip-172-31-18-103 | stderr: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000e2dfe000, 42012672, 0) failed; error='Cannot allocate memory' (errno=12)
Which is making me think this is a memory problem. I'm going to try this again with a better AWS instance and see how I get on.
Update 2
Using a larger AWS instance solved the issue. :)
I then discovered that rexray/ebs doesn't like it when a container switches between hosts in my swarm - it duplicates the EBS volumes so that it keeps one per machine. My solution to this was to use an EFS drive in AWS and mount it to each possible host. I then updated the fstab file so that the drive is remounted on every reboot. Job done. Now to look into using a reverse proxy...
Here's the thing, I need to tell Docker to not containerize the container’s networking, because it needs to connect to a MongoDB that is inside a VPN (enterprise private DB).
There is a Docker command that let's me do exactly that: --net=host. Reference here.
So, for example, when running the container on my local machine, I will do something like:
docker run --rm -it --net=host [image-name]:[version] bash -il
And that command will do the trick. Thanks to that, I can connect to the "private" MongoDB.
So, my question is: Is there a way customize the docker run command of a Single Docker Environment on Elastic Beanstalk so I can add the --net=host?
I have tried using the container_commands into the config.yml file to add that instruction there, but I don't think that does what I need, here is a snippet:
container_commands:
00-test_command:
command: bundle exec thin --net=host
01-networking-fix:
command: "docker run --rm -it --net=host [image-name]:[version] bash -il"
I ended up fixing it with two container commands
container_commands:
00_fix_networking:
command: sed -i 's/docker run -d/docker run --net=host -d/' /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh
01_fix_docker_ip:
command: sed -i 's/server $EB_CONFIG_NGINX_UPSTREAM_IP/server localhost/' /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh
Update:
I also had to fix the Upstart script. Unfortunately, I didn't write down what I did because I didn't end up needing to alter the docker run command. You would do a files directive for (I think) /etc/init/docker. AWS edits the Nginx configuration in the same manner as in 01flip.sh in that file as well.
Explanation:
In the 64bit Amazon Linux 2015.03 v2.0.2 running Docker 1.7.1 platform version, the file you need to edit is /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh. This file is now far more complex than Samar's version so I didn't want to put the actual contents in there. However, the change is basically the same. There's the line that starts with
docker run -d
I fixed it with a container command:
container_commands:
00_fix_networking:
command: sed -i 's/docker run -d/docker run --net=host -d/' /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh
This successfully adds the --net=host argument but now there's another problem. The system ends up with an invalid Nginx directive. Using --net=host means that when you run docker inspect <container id> there is no IP address in the NetworkSettings. AWS uses this to create the server directive for Nginx and ends up generating server :<some port you chose> (before adding --net=host it would look like server <ip>:<port>). I needed to patch that file, too. It's generated in /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh.
01_fix_docker_ip:
command: sed -i 's/server $EB_CONFIG_NGINX_UPSTREAM_IP/server localhost/' /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh
While elastic beanstalk is generally well suited for applications that work with standard set of configurations, its difficult to customize and keep things updated along with the updates AWS provides to EB stacks. Having said that, I've done something like below which is a bit hacky but works fine.
files:
"/opt/elasticbeanstalk/hooks/appdeploy/pre/04run.sh":
mode: "000755"
owner: root
group: root
encoding: plain
content: |
#script content of original 04run.sh along with modification on docker run cmd
# eg. I injected multi-ports here
docker run -d \
"${EB_CONFIG_DOCKER_ENV_ARGS[#]}" \
"${EB_CONFIG_DOCKER_VOLUME_MOUNTS[#]}" \
"${EB_CONFIG_DOCKER_ENTRYPOINT_ARGS[#]}" \
"${PORT_ARGS[#]}" \
$EB_CONFIG_DOCKER_IMAGE_STAGING \
"${EB_CONFIG_DOCKER_COMMAND_ARGS[#]}" 2>&1 | tee /tmp/docker_run.log | tee $EB_CONFIG_DOCKER_STAGING_APP_FILE
This is not very neat, at least I have to make sure that it does not break with updates on elastic beanstalk. The above one is for docker 1.5 stack but you can do something similar with the version you're running.
Note that the latest version of the AWS stack (with Docker 1.7.1) has a slightly different pre-deploy setup. You'll need to update the file at the location: /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh
commands:
00001_add_privileged:
cwd: /tmp
command: 'sed -i "s/docker run -d/docker run --privileged -d/" /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh'
or, for example, if you want to pass args to your Docker image:
commands:
00001_modify_docker_run:
cwd: /tmp
command: 'sed -i "s/\$EB_CONFIG_DOCKER_IMAGE_STAGING/\$EB_CONFIG_DOCKER_IMAGE_STAGING -gzip -enable-url-source/" /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh'
I am running a Play 2.2.3 web application on AWS Elastic Beanstalk, using SBTs ability to generate Docker images. Uploading the image from the EB administration interface usually works, but sometimes it gets into a state where I consistently get the following error:
Docker container quit unexpectedly on Thu Nov 27 10:05:37 UTC 2014:
Play server process ID is 1 This application is already running (Or
delete /opt/docker/RUNNING_PID file).
And deployment fails. I cannot get out of this by doing anything else than terminating the environment and setting it up again. How can I avoid that the environment gets into this state?
Sounds like you may be running into the infamous Pid 1 issue. Docker uses a new pid namespace for each container, which means first process gets PID 1. PID 1 is a special ID which should be used only by processes designed to use it. Could you try using Supervisord instead of having playframework running as the primary processes and see if that resolves your issue? Hopefully, supervisord handles Amazon's termination commands better than the play framework.
#dkm was having the same issue with my dockerized play app. I package my apps as standalone for production using '$ sbt clean dist` commands. This produces a .zip file that you can deploy to some folder in your docker container like /var/www/xxxx.
Get a bash shell into your container: $ docker run -it <your image name> /bin/bash
Example: docker run -it centos/myapp /bin/bash
Once the app is there you'll have to create an executable bash script I called mine startapp and the contents should be something like this:
Create the script file in the docker container:
$ touch startapp && chmod +x startapp
$ vi startapp
Add the execute command & any required configurations:
#!/bin/bash
/var/www/<your app name>/bin/<your app name> -Dhttp.port=80 -Dconfig.file=/var/www/pointflow/conf/<your app conf. file>
Save the startapp script then from a new terminal and then you must commit your changes to your container's image so it will be available from here on out:
Get the running container's current ID:
$ docker ps
Commit/Save the changes
$ docker commit <your running containerID> <your image's name>
Example: docker commit 1bce234 centos/myappsname
Now for the grand finale you can docker stop or exit out of the running container's bash. Next start the play app using the following docker command:
$ docker run -d -p 80:80 <your image's name> /bin/sh startapp
Example: docker run -d -p 80:80 centos/myapp /bin/sh startapp
Run docker ps to see if your app is running. You see something similar to this:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
19eae9bc8371 centos/myapp:latest "/bin/sh startapp" 13 seconds ago Up 11 seconds 0.0.0.0:80->80/tcp suspicious_heisenberg
Open a browser and visit your new dockerized app
Hope this helps...