.dockerignore: some files are ignored, some are not - django

The Problem
Hi I'm new to Docker. I want to ignore some files and directories using .dockerignore in my Django project. In the beginning, no files were ignored , then I searched in stackoverflow and found that its because of the volumes in docker-compose.yml, so I commented it out. But now some of the files and directories are getting ignored but some are not( pycache , db.sqlite3 ). I went through a lot of questions but couldn't find any solution.
Project structure
-src
--coreapp
---migrations
---__init__.py
---__pycache__
---admin.py
---apps.py
---models.py
---tests.py
---views.py
---tests.py
--admin.json
--db.sqlite3
--manage.py
-.dockerignore
-.gitignore
-docker-compose.yml
-Dockerfile
-Procfile
-README.md
-requirements.txt
-runtime.txt
Dockerfile
FROM python:3.7
ENV PYTHONUNBUFFERED 1
COPY ./requirements.txt /code/requirements.txt
RUN pip install -r /code/requirements.txt
COPY . /code/
WORKDIR /code/
EXPOSE 8000
docker-compose.yml
version: '3'
services:
db:
image: postgres
web:
build: .
command: bash -c "python src/manage.py runserver 0.0.0.0:8000"
# volumes:
# - .:/code
ports:
- "8000:8000"
depends_on:
- db
.dockerignore
# Byte-compiled / optimized / DLL files
__pycache__/
**/migrations
src/media
src/db.sqlite3
Procfile
.git
Commands
# build image
sudo docker-compose up --build
# to enter container
sudo docker exec -it [container id] bash
# to check ignored files inside the container
ls
Expected output
# Byte-compiled / optimized / DLL files
__pycache__/ # ignored
**/migrations # ignored
src/media # ignored
src/db.sqlite3 # ignored
Procfile # ignored
.git # ignored
Original Output
# Byte-compiled / optimized / DLL files
__pycache__/ # NOT ignored
**/migrations # ignored
src/media # ignored
src/db.sqlite3 # NOT ignored
Procfile # ignored
.git # ignored
Attempts
__pycache__/
**/__pycache__
**/*__pycache__
**/*__pycache__*
**/*__pycache__/
**/__pycache__/
*/db.sqlite3
db.sqlite3

The .dockerignore file only affects what files are copied into the image in the Dockerfile COPY line (technically, what files are included in the build context). It doesn't mean those files will never exist in the image or in a container, just that they're not included in the initial copy.
You should be able to verify this by looking at the docker build output. After each step there will be a line like ---> 0123456789ab; those hex numbers are valid Docker image IDs. Find the image created immediately after the COPY step and run
docker run --rm 0123456789ab ls
If you explore this way a little bit, you should see that the __pycache__ directory in the container is either absent entirely or different from the host.
Of the specific files you mention, the db.sqlite3 file is your actual application's database, and it will be created when you start the application; that's why you see it if you docker exec into a running container, but not when you docker run a clean container from the image. What is __pycache__? clarifies that the Python interpreter creates that directory on its own whenever it executes an import statement, so it's not surprising that that directory will also reappear on its own.

What exactly do you have in requirements.txt?
Is there some package in this file that created this directory? Because docker CLI can only ignore this before sending context for the build, once the build starts(docker base image, pip install, etc as written in dockerfile) then dockerignore might not be able to ignore it from docker image.
If not then you can try
*/db* -> eliminate files starting with db one level below the root,
*sqlite3
As per https://docs.docker.com/engine/reference/builder/ Matching is done using Go’s filepath.Match rules. A preprocessing step removes leading and trailing whitespace and eliminates . and .. elements using Go’s filepath.Clean. Lines that are blank after preprocessing are ignored.
In your attempts */db.sqlite3 db.sqlite3, maybe the . is getting eliminated as mentioned above and hence unable to remove the requested file from build.

Related

Setting USER in Dockerfile prevents saving file fields (eg. ImageField) in Django

I am trying to containerize Django with Dockerfile and docker-compose.yml as defined below. I built the Dockerfile as (fiifidev/postgres:test) for the compose file. Everything works fine. However anytime I try to save a model with a file field (eg. ImageField or FileField), I get Permission Error PermissionError: [Errno 13] Permission denied docker.
I suspect I am not adding the appropriate permission of user creation (useradd) in the Dockerfile (not sure). But when I remove the USER everything works fine.
How can I fix this any help will be much appreciated.
FROM python:3.10-slim-bullseye as base
# Setup env
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONFAULTHANDLER 1
FROM base AS python-deps
# Install pipenv and compilation dependencies
RUN pip install pipenv
RUN apt-get update && apt-get install -y --no-install-recommends gcc
# Install python dependencies in /.venv
COPY Pipfile .
COPY Pipfile.lock .
RUN PIPENV_VENV_IN_PROJECT=1 pipenv install --deploy
FROM base AS runtime
# Copy virtual env from python-deps stage
COPY --from=python-deps /.venv /.venv
ENV PATH="/.venv/bin:$PATH"
# Create and switch to a new user
RUN useradd --create-home appuser
WORKDIR /home/appuser/src
USER appuser
# Install application into container
COPY --chown=appuser:appuser . .
version: "3.9"
services:
web:
image: fiifidev/postgres:test
command: sh -c "python manage.py makemigrations &&
python manage.py migrate &&
python manage.py initiate_admin &&
python manage.py runserver 0.0.0.0:8000"
volumes:
- .:/home/appuser/src
networks:
postgres-network:
env_file:
- .env
ports:
- ${APP_PORT}:8000
restart: "on-failure"
networks:
postgres-network:
external: true
From Dockerfile:
WORKDIR /home/appuser/src
USER appuser
# Install application into container
COPY --chown=appuser:appuser . .
Here you are creating a src directory and copying your code into it. This is baked into the resulting image.
From docker-compose.yml:
volumes:
- .:/home/appuser/src
Here you are mounting the current directory on your host on top of the the src directory. A mount will take precedence over the image's idea of what a directory contains, so this effectively means your COPY and chown have no effect. (Those files are still there in your image, but the mount hides them; they are not available.)
The behavior as far as permissions here will vary by platform. On Windows, I don't know what would happen. Using Docker Desktop on Mac it would "just work", because Docker Desktop more or less ignores the permissions in this case and makes everything happy.
On Linux, however, the file ownership inside the container and outside the container must match for you to be able to write to it. What's probably happening is that the files in your mounted directory are owned by a different uid (yours) than the 'appuser' in the container has. Therefore, you get permission errors because you don't have permission to write to the files/directory.
There are two solutions you could try:
Chown the files in the host directory (outside the container) to have the same uid that 'appuser' has inside the container.
Or you could specify the uid of 'appuser' when creating it (during the image build). In that case, specify 'appuser' has the same UID as your user on the Linux host. It wouldn't matter if the Linux user has a different name; if the uid number is the same, then you will have permissions.
EDIT: Including from comments. The OP ended up solving this in docker-compose.yml by adding:
user: "${UID}:${GID}"
This would set the user/group for the service to the UID and GID from the environment of the user calling docker-compose. That is, assuming that the shell has set the UID and GID environment variables; not all shells or OSes will behave in the exact same manner.

Django hot reload only working on certain files

I have this weird issue where not all file edits are being picked up by the hot reloader for Django.
I have this structure:
/
app/ ... apps in here.
config/settings.py
manage.py
Now, any changes to config/settings.py or manage.py will result in the django runserver reloading.
But any changes to files inside app/... don't trigger a reload - I have to go and add a newline to manage.py and save (quite irritating).
Any ideas why this might be?
At first I thought it was a docker thing, and it was only picking up files in the base dir, but then changes to config/settings.py also trigger the reload, so clearly it can see deeper.
EDIT: Addition info
Django 3.2, PyCharm, MacOS - yes, all apps are in INSTALLED_APPS
I have another project that has the EXACT same structure and for some reason it works... I'm really stumped.
EDIT adding dc and dockerfile
FROM python:3.8-slim
ENV PYTHONUNBUFFERED 1
WORKDIR /app
COPY ./requirements /requirements
ARG pip_requirement_file
RUN apt-get update && apt-get install -y libjpeg62-turbo-dev zlib1g-dev gcc ca-certificates gcc postgresql-client sed xmlsec1 pax-utils && apt-get clean
RUN pip install --no-cache-dir -r /requirements/$pip_requirement_file \
&& find /usr/local \
\( -type d -a -name test -o -name tests \) \
-o \( -type f -a -name '*.pyc' -o -name '*.pyo' \) \
-exec rm -rf '{}' +
# Copy requirements and install local one
RUN rm -rf /requirements
COPY ./compose/django/entrypoint.sh /entrypoint.sh
RUN sed -i 's/\r//' /entrypoint.sh
RUN chmod +x /entrypoint.sh
COPY ./compose/django/start-server.sh /start-server.sh
RUN sed -i 's/\r//' /start-server.sh
RUN chmod +x /start-server.sh
# Very specifically copy the files we want to avoid bloating the image.
COPY ./manage.py /app/
COPY ./app/ /app/app/
COPY ./admin_static/ /app/admin_static/
COPY ./config/ /app/config/
COPY ./database/ /app/database/
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/start-server.sh"]
docker compose
services:
django:
container_name: django
build:
context: .
dockerfile: ./compose/django/Dockerfile
args:
pip_requirement_file: local.txt
depends_on:
- postgres
ports:
- "8000:8000"
links:
- postgres:postgres
volumes:
- .:/app
env_file: .env
I'd like to thank everyone for trying to help me solve this riddle that has been doing my head in.
I decided to strip both projects (old and new) back to see why the old worked and the new doesn't.
This is what was different.
Inside config/init.py on the old project, this line exists:
from __future__ import absolute_import, unicode_literals
Inside the new project, this was missing.
When I add this in, hot reloading starts working for all files inside app/* ?????
It's working, but honestly, I have no idea why that would make a difference.
I am not 100% sure what the issue is, but offering this answer with a couple of things you can try:
There are reports that Pycharm will sometimes cause problems if it has been configured not to update the timestamps of files when saving them (there is a Preserve files timestamps setting to control this) which you could try toggling. An easy way to verify if this is the issue is to try editing a file with a different editor (or touch the file) and see if that triggers a reload - if it does then the issue is PyCharm.
Note that the default StatReloader will not work if file timestamps don't change.
Try installing pywatchman and the Watchman service, as mention in the documentation, which provides a much more efficient way to watch for changes than StatReloader which simply polls all files for changes every second - if it's a large project it may be that the StatReloader is just taking too long to spot changes.

docker-compose how to reference files in other directories

Having this dockerfile:
FROM python:3.8.3-alpine
ENV MICRO_SERVICE=/home/app/microservice
# RUN addgroup -S $APP_USER && adduser -S $APP_USER -G $APP_USER
# set work directory
RUN mkdir -p $MICRO_SERVICE
RUN mkdir -p $MICRO_SERVICE/static
# where the code lives
WORKDIR $MICRO_SERVICE
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install psycopg2 dependencies
RUN apk update \
&& apk add --virtual build-deps gcc python3-dev musl-dev \
&& apk add postgresql-dev gcc python3-dev musl-dev \
&& apk del build-deps \
&& apk --no-cache add musl-dev linux-headers g++
# install dependencies
RUN pip install --upgrade pip
# copy project
COPY . $MICRO_SERVICE
RUN pip install -r requirements.txt
COPY ./entrypoint.sh $MICRO_SERVICE
CMD ["/bin/bash", "/home/app/microservice/entrypoint.sh"]
and the following docker-compose.yml file:
version: "3.7"
services:
nginx:
build: ./nginx
ports:
- 1300:80
volumes:
- static_volume:/home/app/microservice/static
depends_on:
- web
restart: "on-failure"
web:
build: . #build the image for the web service from the dockerfile in parent directory
command: sh -c "python manage.py collectstatic --no-input &&
gunicorn djsr.wsgi:application --bind 0.0.0.0:${APP_PORT}"
volumes:
- .:/microservice:rw # map data and files from parent directory in host to microservice directory in docker containe
- static_volume:/home/app/microservice/static
env_file:
- .env
image: wevbapp
expose:
- ${APP_PORT}
restart: "on-failure"
volumes:
static_volume:
I need to reference the following files (in the docker-compose.yml file) being in other directories rather than the .devcontainer:
manage.py
requirements.txt
.env
This is my folder structure:
An easy solution would be to move the dockerfile, docker-compose.yml, and .env in the django directory djsr, but I am trying to keep the files structured like this. How can I do reference those files in docker-compose.yml?
It is fairly common to put the couple of Docker-related files in the project root directory, and that can potentially save you some trouble; I'd recommend that as a first choice.
If you do want to keep it all in a subdirectory, it's possible, though. When you run docker-compose, you can specify the location of the configuration file. It will consider all paths as relative to this file's directory.
# Either:
docker-compose -f .devcontainer/docker-compose.yml up
cd .devcontainer && docker-compose up
When you go to build the image, the build reads in a context directory, and COPY statements are always interpreted relative to this directory. For your setup, you need the context directory to be the top of your source tree, and then specify an alternate Dockerfile in a subdirectory.
services:
web:
build:
context: ..
dockerfile: .dockerenv/Dockerfile
For the most part the Dockerfile itself is fine, but where the entrypoint script is in a subdirectory, the COPY command needs to reflect that too. Since you're copying the entire source directory, you could also rearrange things inside the image to be the layout you want.
COPY requirements.txt ./
RUN pip install -r requirements.txt
COPY . ./
# Either:
COPY .dockerenv/entrypoint.sh ./
# Or:
RUN mv .dockerenv/entrypoint.sh .
# Or:
CMD ["./dockerenv/entrypoint.sh"]
I don't recommend the volume structure you have, but if you want to keep it, you also need to change the source path of the bind mount to be the parent directory. (Note particularly, in the previous Dockerfile fragment, a couple of the options involve moving files inside the image, and a bind mount will hide that change.)
services:
web:
volumes:
# Ignore the application built into the container, and use
# whatever's checked out on the host system instead.
- ..:/home/app/microservice
# Further ignore the static assets on the host system and
# use the content in a named volume instead.
- static_volume:/home/app/microservice/static
why don't you mount the same as you did with the folders only for these files?
The source of the mount. For bind mounts, this is the path to the file or directory on the
Docker daemon host. May be specified as source or src.

How do I let my docker volume sync to my filesystem and allow writing from docker build

I'm building a django app using docker. The issue I am having is that my local filesystem is not synced to the docker environment so making local changes have no effect until I rebuild.
I added a volume
- ".:/app:rw"
which is syncing to my local filesystem but does my bundles that get built via webpack during the build don't get inserted (because they aren't in my filesystem)
My dockerfile has this
... setup stuff...
ENV NODE_PATH=$NVM_DIR/versions/node/v$NODE_VERSION/lib/node_modules \
PATH=$NVM_DIR/versions/node/v$NODE_VERSION/bin:$PATH
ENV PATH=/node_modules/.bin:$PATH
COPY package*.json /
RUN (cd / && npm install && rm -rf /tmp/*)
...pip install stuff...
COPY . /app
WORKDIR /app
RUN npm run build
RUN DJANGO_MODE=build python manage.py collectstatic --noinput
So I want to sync to my local filesystem so I can make changes and have them show up immediately AND have my bundles and static assets present. The way I've been developing so far is to just comment out the app:rw line in my docker-compose.yml which allows all the assets and bundles to be present.
The solution that ended up working for me was to assign a volume to the directory I wanted to not sync to my local environment.
volumes:
- ".:/app/:rw"
- "/app/project_folder/static_source/bundles/"
- "/app/project_folder/bundle_tracker/"
- "/app/project_folder/static_source/static/"
Arguably there's probably a better way to do this, but this solution does work. The dockerfile compiles webpack and collect static does it's job both within the container and the last 3 lines above keep my local machine from overwriting them. The downside is that I still have to figure out a better solution for live recompile of scss or javascript, but that's a job for another day.
You can mount a local folder into your Docker image. Just use the --mount option at the docker run command. In the following example the current directory will be available in your Docker image at /app.
docker run -d \
-it \
--name devtest \
--mount type=bind,source="$(pwd)"/target,target=/app \
nginx:latest
Reference: https://docs.docker.com/storage/bind-mounts/

Docker compose with multiple Dockfiles and images (Django+Angular)

I'm building an app that uses Django and Angular that is split up into two different repositories and docker images.
My file structure is:
.
docker-compose.yml
djangoapp/
Dockerfile
manage.py
...
angularapp/
Dockerfile
...
I've been unable to get it to work properly, all documentation I found on this seem to expect you to have the docker-compose.yml file together with the DockerFile.
I've tried multiple different variations but my current (not working) docker-compose.yml file looks like this:
version : '3'
services:
web:
build: ./djangoapp
command: python manage.py runserver 0.0.0.0:8000
ports:
- "8000:8000"
This gives me error can't open file 'manage.py': [Errno 2] No such file or directory.
If I go into the djangoapp/ directory and create a docker-compose.yml file there according to the offical Docker docs, it works fine. So it's nothing wrong with the actual build, the issue is accessing it from outside like I'm trying to do.
Update: I also decided to add my Dockerfile located at ./djangoapp/Dockerfile.
FROM python:3
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
ADD requirements.txt /code/
RUN pip install -r requirements.txt
ADD . /code/
From what I can see it seems just to be a simple typo
version : '3'
services:
web:
build: ./djangoapp
command: python manage.py runserver 0.0.0.0:8000
ports:
- "8000:8000"
That ./ is all your missing I think. I'm pretty sure without it the docker compose file doesn't go into the directory specified.
Update
Now that I saw your Dockerfile, I've noticed that you havent added the manage.py to your container. The same way you added your requirements.txt is what you need to do with the manage.py. When a Docker container is built, it only has what you give it plus any other file structure it has from the base Image.
You can either add it in the Dockerfile or you can have a shared volume between a local directory and a container directory. I think adding it in the Dockerfile would be easiest though.
This issue was solved by rebuilding my Docker image with docker-compose build web command. Seems like at the stage when I was having the error, that build didn't include the files needed. So in the code that I show above in the question post, nothing is wrong, it just needed to be rebuild.