I have problem with docker-compose (1) and docker compose (2) pull. When I try to pull images from direct repository (I use ECR), both commands (sudo docker-compose -f docker-compose.prod.yml pull and sudo docker compose -f docker-compose.prod.yml pull) give me following output:
$ export DOCKER_REGISTRY= {{ secret path to AWS ECP }}
$ $DOCKER_REGISTRY
{{ secret path to AWS ECP }}
$ sudo docker compose -f docker-compose.prod.yml pull
WARN[0000] The "DOCKER_REGISTRY" variable is not set. Defaulting to a blank string.
WARN[0000] The "DOCKER_REGISTRY" variable is not set. Defaulting to a blank string.
WARN[0000] The "DOCKER_REGISTRY" variable is not set. Defaulting to a blank string.
[+] Running 0/0
⠋ cron Pulling 0.0s
⠿ db Error 0.0s
⠋ traefik Pulling 0.0s
⠋ web Pulling 0.0s
WARNING: Some service image(s) must be built from source by running:
docker compose build cron traefik web
invalid reference format
$ sudo docker-compose -f docker-compose.prod.yml pull
WARNING: The DOCKER_REGISTRY variable is not set. Defaulting to a blank string.
Pulling db ... done
In line 1 I export DOCKER_REGISTRY variable, that is using in docker-compose.prod.yml. In line 2 I check this variable and then run both of above commands. (2) sees all needed images in yml file, but can't pull them, because it doesn't see DOCKER_REGISTRY variable. (1) sees only db.
Part of prod.yml file:
version: '3.7'
services:
web:
container_name: web
image: ${DOCKER_REGISTRY}/me-project_web:latest
build:
context: ./MoreEnergy
dockerfile: Dockerfile.prod
restart: always
env_file: ./.env.prod
entrypoint: sh ./entrypoint.sh
command: gunicorn MoreEnergy.wsgi:application --bind 0.0.0.0:8000
expose:
- 8000
volumes:
- static_volume:/home/app/web/staticfiles
- media_volume:/home/app/web/mediafiles
depends_on:
- db
labels:
- "traefik.enable=true"
- "traefik.http.routers.web.rule=Host(`lunev.dmitrium.com`)"
- "traefik.http.routers.web.tls=true"
- "traefik.http.routers.web.tls.certresolver=letsencrypt"
I have .env.prod file with all needed variables too, but (1) and (2) don't see DOCKER_REGISTRY variable anyway.
What should I do? I'm trying to implement CI/CD using GitHub actions. In current state, docker-compose can build and push all my images to AWS ECR, but can't pull them back.
The answer was utterly stupid and simple. 10 minutes before asking the question, I've successfully started the containers, pulled by docker pull command before, but forgot it, because detail of success was not conspicuous. I've done this by NOT using this command with sudo. Yes, it is. sudo docker compose -f docker-compose.prod.yml up and docker compose -f docker-compose.prod.yml up (and pull) commands run correct only if you do it without root access.
Maybe it's not a common solution, and in other cases sudo is required, but not in mine.
Related
I'm trying to get a Django application running on the latest version of Lightsail which supports deploying docker containers as of Nov 2020 (AWS Lightsail Container Announcement).
I've created a very small Django application to test this out. However, my container deployment continues to get stuck and fail.
Here are the only logs I'm able to see:
This is my Dockerfile:
FROM python:3
ENV PYTHONUNBUFFERED=1
WORKDIR /code
COPY requirements.txt /code/
RUN pip install -r requirements.txt
COPY . /code/
And this is my docker-compose.yml:
version: "3.9"
services:
db:
image: postgres
environment:
- POSTGRES_DB=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
web:
build: .
image: argylehacker/app-stats:latest
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
depends_on:
- db
I'm wondering a few things:
Right now I'm only uploading the web container to Lightsail. Should I also be uploading the db container?
Should I create a postgres database in Lightsail and connect to it first?
Do I need to tell Django to run the db migrations before the application starts?
Is there a way to enable more logs from the containers? Or does the lack of logs mean that the containers aren't even able to start.
Thanks for the help!
Docker
This problem stemmed from a bad understanding of Docker. I was previously trying to include image: argylehacker/app-stats:latest in my docker-compose.yml to upload the web container to DockerHub. This is the wrong way of going about things. From what I understand now, docker-compose is most helpful for orchestrating your local environment rather than creating docker images that can be run in containers.
The most important thing is to upload a container to Lightsail that can start your server. When you're using Docker this can be specified using the CMD and the end of your Dockerfile. In my case I needed to add this line to my Dockerfile:
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
So now it looks like this:
FROM python:3
ENV PYTHONUNBUFFERED=1
WORKDIR /code
COPY requirements.txt /code/
RUN pip install -r requirements.txt
COPY . /code/
CMD ["python", "manage.py", "runserver", "2.0.0.0:8000"]
Finally, I removed the image: argylehacker/app-stats:latest line from my docker-compose.yml file.
At this point you should be able to:
Build your container docker build -t argylehacker/app-stats:latest .
Upload it to DockerHub docker push argylehacker/app-stats:latest
Deploy it in AWS Lightsail pointing to argylehacker/app-stats:latest
Troubleshooting
I got stuck on this because I couldn't see any meaningful logs in the Lightsail log terminal. This was because my container wasn't actually running anything.
In order to get debug this locally I took the following steps
Build the image docker build -t argylehacker/app-stats:latest .
Run the container docker run -it --rm -p 8000:8000 argylehacker/app-stats:latest.
At this point docker should be running the container and you can view the logs. This is exactly what Lightsail is going to do when it runs your container.
Answers to my Original Questions
The Dockerfil is very different than a docker-compose file used to compose services. The purpose of docker-compose is to coordinate containers, vs a Dockerfile will define how an image is built. All you need to do for Lightsail is build the image docker build <container>:<tag>
Yes, you'll need to create a Postgres database in AWS Lightsail so that Django can connect to a database and run. You'll modify the settings.py file to include the database credentails once it is available in Lightsail.
Still tracking down the best way to run the db migrations
The lack of logs was because the Dockerfile wasn't starting Django
I've followed the this short guide to create a django app with docker
https://docs.docker.com/compose/django/
and then following copilot instructional to push up the container to ECS:
https://aws.amazon.com/blogs/containers/introducing-aws-copilot/
I've also used this sample to test everything -- which works out fine:
https://github.com/aws-samples/aws-copilot-sample-service
The deploy completes and outputs and URL endpoint.
In my case, the everything is successfully built, but once the test environment is being deployed it just continuously builds at this:
72ff4719 size: 3055
⠏ Deploying load-bal:7158348 to test.
and never finishes. I've even downsized my requirements.txt to a bare minimum.
My Dockerfile
FROM python:3.7.4
ENV PYTHONUNBUFFERED=1
RUN mkdir /code
WORKDIR /code
COPY requirements.txt /code/
RUN pip install -r requirements.txt
EXPOSE 80
COPY . /code/
docker-compose.yml
version: "3.8"
services:
db:
image: postgres
environment:
- POSTGRES_DB=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
depends_on:
- db
requirements.txt
Django==3.0.8
djangorestframework==3.11.0
gunicorn==20.0.4
pipenv==2020.6.2
psycopg2-binary==2.8.5
virtualenv==16.7.6
Instructions I follow:
sudo docker-compose run web django-admin startproject composeexample .
Successfully creates the Django App
copilot init
Setup naming for app and load balancer
Choose to create test environment
Everything builds successfully and then just sits here. I've tried a number of variations, but the only one that works is just doing the copilot instructional without django involved.
6f3494a64128: Pushed
cfe650cc4def: Pushed
a477d6671cc7: Pushed
90df760355a7: Pushed
574ea6c52bdd: Pushed
d1573fad78d1: Pushed
14c1ff636882: Pushed
48ebd1638acd: Pushed
31f78d833a92: Pushed
2ea751c0f96c: Pushed
7a435d49206f: Pushed
9674e3075904: Pushed
831b66a484dc: Pushed
ini: digest: sha256:b7460876bc84b1a26e7513fa6d17b5bffd5560ae958a933984376ed2c9fe53f3 size: 3052
⠏ Deploying aiinterview-lb:ini to test.
tl;dr the Dockerfile that's being used by this tutorial is incomplete for Copilot's purposes. It needs an extra line containing
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
and the EXPOSE directive should be updated to 8000. Because Copilot doesn't recognize Docker Compose syntax and there's no command or entrypoint specified in the Dockerfile, the image will never start with Copilot's configuration settings.
Details
AWS Copilot is designed around "services" consisting of an image, possible sidecars, and additional storage resources. That means that its basic unit of config is the Docker image and the service manifest. It doesn't natively read Docker Compose syntax, so all the config that Copilot knows about is that which is specified in the Dockerfile or image and each service's manifest.yml and addons directory.
In this example, designed for use with Docker Compose, the Dockerfile doesn't have any kind of CMD or ENTRYPOINT directive, so the built image which gets pushed to Amazon ECR by Copilot won't ever start. The tutorial specifies the image's command (python manage.py runserver 0.0.0.0:8000) as an override in docker-compose.yml, so you'll want to update your Dockerfile to the following:
FROM python:3.7.4
ENV PYTHONUNBUFFERED=1
RUN mkdir /code
WORKDIR /code
COPY requirements.txt /code/
RUN pip install -r requirements.txt
EXPOSE 8000
COPY . /code/
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
Note here that I've changed the EXPOSE directive to 8000 to match the command from docker-compose.yml and added the command specified in the web section to the Dockerfile as a CMD directive.
You'll also want to run
copilot init --image postgres --name db --port 5432 --type "Backend Service" --deploy
This will create the db service specified in your docker-compose.yml. You may need to run this first so that your web container doesn't fail to start while searching for credentials.
Some other notes:
You can specify your database credentials by adding variables and secrets in the manifest file for db which is created in your workspace at ./copilot/db/manifest.yml. For more on how to add a secret to SSM and make it accessible to your Copilot services, check out our documentation
variables:
POSTGRES_DB: postgres
POSTGRES_USER: postgres
secrets:
POSTGRES_PASSWORD: POSTGRES_PASSWORD
Your database endpoint is accessible over service discovery at db.$COPILOT_SERVICE_DISCOVERY_ENDPOINT--you may need to update your service code which connects to the database to reflect this endpoint instead of localhost or 0.0.0.0.
I have a docker-compose.yml defined as follows with two services (the database and the app):
version: '3'
services:
db:
build: .
image: postgres
environment:
- POSTGRES_DB=postgres
- POSTGRES_USER=(adminname)
- POSTGRES_PASSWORD=(adminpassword)
- CLOUDINARY_URL=(cloudinarykey)
app:
build: .
ports:
- "8000:8000"
depends_on:
- db
The reason I have build: . in both services is due to how you can't do docker-compose push unless you have a build in all services. However, this means that both services are referring to the same Dockerfile, which builds the entire app. So after I run docker-compose build and look at the images available I see this:
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
mellon_app latest XXXXXXXXXXXX 27 seconds ago 1.14GB
postgres latest XXXXXXXXXXXX 27 seconds ago 1.14GB
The IMAGE_ID is the exact same for both images, the size is exactly the same for both images. This makes me think I've definitely done some unnecessary duplication as they're both just running the same Dockerfile. I don't want to take up any unnecessary space, how do I do this properly?
This is my Dockerfile:
FROM (MY FRIENDS ACCOUNT)/django-npm:latest
RUN mkdir usr/src/mprova
WORKDIR /usr/src/mprova
COPY frontend ./frontend
COPY backend ./backend
WORKDIR /usr/src/mprova/frontend
RUN npm install
RUN npm run build
WORKDIR /usr/src/mprova/backend
ENV DJANGO_PRODUCTION=True
RUN pip3 install -r requirements.txt
EXPOSE 8000
CMD python3 manage.py collectstatic && \
python3 manage.py makemigrations && \
python3 manage.py migrate && \
gunicorn mellon.wsgi --bind 0.0.0.0:8000
What is the proper way to push the images to my Docker hub registry without this duplication?
Proper way is to do
docker build -f {path-to-dockerfile} -t {desired-docker-image-name}.
docker tag {desired-docker-image-name}:latest {desired-remote-image-name}:latest or not latest but what you want, like datetime in int format
docker push {desired-remote-image-name}:latest
and cleanup:
4. docker rmi {desired-docker-image-name}:latest {desired-remote-image-name}:latest
Whole purpose of docker-compose is to help your local development, so it's easier to start several pods and combine them in local docker-compose network etc...
I have deployed django app with docker container. Source code of the app is on the Bitbucket repository. Now I want to setup pipeline for master branch which is intended to make deployment automatic on merge. Problematic part of the pipeline script is below:
docker-compose up --build -d
Above line resuts with error that says:
Permission denied: '/path/to/docker/volume/pgdb'
My docker-compose file section related to postgres is below:
postgres:
container_name: arw-postgres
image: postgres:11
ports:
- 5432:5432
volumes:
- ./pgdb:/var/lib/postgresql/data
env_file: .env
The above specified docker-compose command runs normally with sudo privilege. Actually, I can connect to remote host with root user and can run this command. But I don't want to expose my root user's credentials. How can I recreate my docker container without sudo privileges?
The problem here was non-existence of .dockerignore file. I simply added .dockerignore file with the content pgdb at the same level with Dockerfile. This file avoids copying of protected pgdb folder to docker container, so no permission error occurs.
I am totally a newbie when it comes to Docker. And I am trying to understand it with a dummy project.
I have a django project and my Dockerfile is inside the Django project's root folder. And my docker-compose.yml file is under the top root folder which contains django project folder and other config files.
my docker-compose.yml
version: '3'
services:
db:
image: postgres
container_name: dummy_project_postgres
volumes:
- ./data/db:/var/lib/postgresql/data
event_planner:
build: ./dummy_project
container_name: dummy_project
volumes:
- .:/web
ports:
- "8000:8000"
depends_on:
- db
links:
- db:postgres
and my Dockerfile
FROM python:3
ENV PYTHONUNBUFFERED 1
RUN mkdir /web
WORKDIR /web
ADD requirements.txt /web/
RUN pip install -r requirements.txt
ADD . /web/
I am trying to run the following commands
# stop and remove the existing containers
docker-compose stop
docker-compose rm -f
# up and run the container
docker-compose build
docker-compose up -d
docker-compose exec dummy_project bash
When I do docker-compose up -d, I see this error.
docker-compose up -d
dummy_project_postgres is up-to-date
Starting dummy_project ... done
warning: could not open directory 'data/db/': Permission denied
I know this question asked before, but I didn't quite get the solution I need and I am stuck for hours now.
EDIT: I have all the permissions for all the folders under the top folder
EDIT2: sudo docker-compose up -d also results the same error.
I solved by adding ":z" to end of volume defintion
version: '3'
services:
db:
image: postgres
container_name: dummy_project_postgres
volumes:
- ./data/db:/var/lib/postgresql/data:z
event_planner:
build: ./dummy_project
container_name: dummy_project
volumes:
- .:/web
ports:
- "8000:8000"
depends_on:
- db
links:
- db:postgres
What ":z" means
Labeling systems like SELinux require that proper labels are placed on
volume content mounted into a container. Without a label, the security
system might prevent the processes running inside the container from
using the content. By default, Docker does not change the labels set
by the OS.
To change the label in the container context, you can add either of
two suffixes :z or :Z to the volume mount. These suffixes tell Docker
to relabel file objects on the shared volumes. The z option tells
Docker that two containers share the volume content. As a result,
Docker labels the content with a shared content label. Shared volume
labels allow all containers to read/write content. The Z option tells
Docker to label the content with a private unshared label. Only the
current container can use a private volume.
https://docs.docker.com/engine/reference/commandline/run/#mount-volumes-from-container---volumes-from
what is 'z' flag in docker container's volumes-from option?
You're trying to mount ./data/db in /var/lib/postgresql/data and you're executing docker-compose with a non-privileged user.
So, we can have two possibilities:
Problem with ./data/db permissions.
Problem with /var/lib/postgresql/data
The simpiest solution is execute docker-compose with a privileged user (root), but if you don't want to do that, you can try this:
Give permissions to ./data/db (I see your EDIT that you've already done it).
Give permissions to /var/lib/postgresql/data
How can you give /var/lib/postgresql/data permissions? Read the following lines:
First, note that /var/lib/postgresql/data is auto-generated by postgre
docker, so, you need to define a new Dockerfile which modifies these
permissions. After that, you need also modify docker-compose to use
this new Dockerfile.
./docker-compose.yml
version: '3'
services:
db:
build:
context: ./mypostgres
dockerfile: Dockerfile_mypostgres
container_name: dummy_project_postgres
volumes:
- ./data/db:/var/lib/postgresql/data
event_planner:
build: ./dumy_project
container_name: dummy_project
volumes:
- .:/web
ports:
- "8000:8000"
depends_on:
- db
links:
- db:postgres
./dumy_project/Dockerfile --> Without changes
./mypostgres/Dockerfile_mypostgres
FROM postgres
RUN mkdir -p /var/lib/postgresql/data
RUN chmod -R 777 /var/lib/postresql/data
ENTRYPOINT docker-entrypoint.sh
This solution is for case that your user is not present in docker group.
First check if your user is in docker group:
grep 'docker' /etc/group
Add user to docker group:
If the command return is empty, then create docker group:
sudo groupadd docker
Else if your user is not present in command return then add him to the group:
sudo usermod -aG docker $USER
Reboot your system
Test it again:
docker run hello-world
Tip: Remember to have the docker service started
If it works, try your docker-compose command again.