Avoid coping the code twice in dockerfile - dockerfile

I am trying to dockerize the application that I found here...
https://github.com/mmz-001/knowledge_gpt
The following script is working as expected. But I do not want to copy the code twice. How do I fine-tune this dockerfile?
# cat Dockerfile
FROM python:3.10
RUN mkdir /app
RUN mkdir /app/knowledge_gpt/
COPY * /app/knowledge_gpt/
COPY * /app/
WORKDIR /app/
ENV PYTHONPATH=${PYTHONPATH}:${PWD}
RUN pip3 install poetry
RUN poetry config virtualenvs.create false
RUN poetry install --no-dev
WORKDIR /app/knowledge_gpt/
CMD ["streamlit", "run", "main.py"]

The issue here is within the pyproject.toml.
It has the following dependency:
[tool.poetry]
name = "knowledge-gpt"
version = "0.1.0"
...
packages = [{include = "knowledge_gpt"}]
So cmd poetry install will try and build from the root of the project instead of building from the packages defined. Thus you will need to define the --no-root option to your build.
By copying it twice, poetry has built from the root of the project /app
Update the Dockerfile to the following:
FROM python:3.10
RUN mkdir /app
COPY * /app/
WORKDIR /app/
ENV PYTHONPATH=${PYTHONPATH}:${PWD}
RUN pip3 install poetry
RUN poetry config virtualenvs.create false
RUN poetry install --only main --no-root
CMD ["streamlit", "run", "main.py"]
I also created a docker-compose.yml to make building it easier:
version: "3.8"
services:
kgpt:
image: knowledge-gpt
container_name: knowledge-gpt
build:
context: .
dockerfile: ./Dockerfile
ports:
- 8501:8501
networks:
default:

Just a wild guess
change this
COPY * /app/knowledge_gpt/
COPY * /app/
WORKDIR /app/
to this
COPY * /app/knowledge_gpt/
WORKDIR /app/knowledge_gpt/
This will copy the content and set the working directory to current

Related

Django Docker ElasticBeanstalk fails

I'm new to docker and eb deployment, I want to deploy django with docker on eb
here's what I did so far
created a Dockerfile
# Pull base image
FROM python:3.9.16-slim-buster
# Set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
RUN apt-get update &&\
apt-get install -y binutils libproj-dev gdal-bin python-gdal python3-gdal libpq-dev python-dev libcurl4-openssl-dev libssl-dev gcc
# install dependencies
COPY . /code
WORKDIR /code/
RUN pip install -r requirements.txt
# set work directory
WORKDIR /code/app
then in docker-compose.yml
version: '3.7'
services:
web:
build: .
command: python /code/hike/manage.py runserver 0.0.0.0:8000
ports:
- 8000:8000
volumes:
- .:/code
it runs locally, but on deployment it fails and when I get to logs, it says
pg_config is required to build psycopg2 from source.
like it's not using the Dockerfile, I read somewhere I should set Dockerrunder.aws.json but I've no idea what to write in it!

How can I install Django in my Docker file?

I am a newbie to Docker. I have created one Django project and can run it in Docker. However, I have started a second project and have encountered a problem.
I created a virtual env and entered it
pipenv install django~=3.1.0 && pipenv shell
I created a Django project
django-admin startproject config .
I ran it within the virtualenv
python manage.py runserver
and could see the Django spaceship
I then exited the virtualenv and created a Dockerfile
Dockerfile
# Pull base image
FROM python:3.8
# Set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# Set work directory
WORKDIR /code
# Install dependencies
COPY Pipfile Pipfile.lock /code/
RUN pip install pipenv && pipenv install --system
# Copy project
COPY . /code/
I ran
docker build .
and it reported a successful build
I created a docker-compose.yml file
docker-compose.yml
version: '3.8'
services:
web:
build: .
command: python /code/manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- 8000:8000
When I run
docker-compose up
it complains
ImportError: Couldn't import Django. Are you sure it's installed and available on your PYTHONPATH environment variable? Did you forget to activate a virtual environment?
I have read in the comments to this question that virtual envs should not be used in docker files, so I replaced
RUN pip install pipenv && pipenv install --system
with
RUN pip install django~=3.1.0
but I still get the same error.
What is wrong?
Have you tried installing your list of requirements from a separate file, something like this?
COPY requirements.txt /code/requirements.txt
WORKDIR /code
RUN pip install -r requirements.txt
Once you have it installed you can run docker-compose run web /bin/sh to start a shell and then use django-admin startproject to create a django project. You may need to change the path in the docker-compose file so that it reflects where your manage.py file ended up (I moved mine to the root). I was able to get it working with the following:
requirements.txt
django==3.1.0
docker-compose.yml
version: '3.8'
services:
web:
build:
context: .
dockerfile: Dockerfile
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- 8000:8000
Dockerfile
# Pull base image
FROM python:3.8
# Set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# Set work directory
WORKDIR /code
# Install dependencies
COPY requirements.txt /code/requirements.txt
WORKDIR /code
RUN pip install -r requirements.txt
# Copy project
COPY . /code/
File tree looks like this:

Docker compose is not copying any files

Quite simple: the docker-compose configuration below does not allow any files to persist after running. So when I do docker exec -i -t aas-job-hunter_web_1 ls /app -alt, I see nothing.
Here is the (non-)working minimal example: https://github.com/philastrophist/test-docker
I'm on Windows 10, I've allowed mounted drives and enabled the TLS connection. I'm not sure what else to do. The thing that most confuses me is that requirements.txt is clearly copied over (since it installs it all) but it isn't there when I have a look docker exec.
My directory structure is:
parent/
website/
manage.py
...
Dockerfile
docker-compose.yml
...
Dockerfile:
FROM python:3.6
#WORKDIR /app
# By copying over requirements first, we make sure that Docker will cache
# our installed requirements rather than reinstall them on every build
COPY requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt
# Now copy in our code, and run it
COPY . /app
EXPOSE 8000
CMD python website/manage.py runserver 0.0.0.0:8000
# CMD tail -f /dev/null # use when testing
docker-compose.yml:
version: '2'
services:
web:
build: .
ports:
- "8000:8000"
volumes:
- .:/app
links:
- db
db:
image: "postgres:9.6"
ports:
- "5432:5432"
environment:
POSTGRES_PASSWORD: hunter2
Traceback:
> docker-compose -f docker-compose.yml up --build
Building web
Step 1/6 : FROM python:3.6
---> 0668df180a32
Step 2/6 : COPY requirements.txt /app/requirements.txt
---> Using cache
---> 3073d0bef876
Step 3/6 : RUN pip install -r /app/requirements.txt
---> Using cache
---> 8ad63bbb3de5
Step 4/6 : COPY . /app
---> 16390cdd6c2c
Step 5/6 : EXPOSE 8000
---> Running in f628000e8961
Removing intermediate container f628000e8961
---> 80e6994cfbd2
Step 6/6 : CMD python website/manage.py runserver 0.0.0.0:8000
---> Running in acb6b25eb558
Removing intermediate container acb6b25eb558
---> da8876d78103
Successfully built da8876d78103
Successfully tagged aas-job-hunter_web:latest
Starting aas-job-hunter_db_1 ... done
Recreating aas-job-hunter_web_1 ... done
Attaching to aas-job-hunter_db_1, aas-job-hunter_web_1
db_1 | LOG: database system was shut down at 2019-05-24 21:23:31 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
web_1 | python: can't open file 'website/manage.py': [Errno 2] No such file or directory
aas-job-hunter_web_1 exited with code 2
Actually it copies files.
Solution 1
Change CMD to :
CMD python /app/website/manage.py runserver 0.0.0.0:8000
Solution 2
You call WORKDIR before the /app folder is created. So change your Dockerfile to :
FROM python:3.6.2
# By copying over requirements first, we make sure that Docker will cache
# our installed requirements rather than reinstall them on every build
COPY requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt
# Now copy in our code, and run it
COPY . /app
WORKDIR /app
#EXPOSE 8000
CMD python ./website/manage.py runserver 0.0.0.0:8000
# CMD tail -f /dev/null # use when testing
And call it after.
Moreover remember that in your current docker-compose file you are using bind mounts, not volumes, so the context - . will replace entirely the content of /app in your container.
Uncomment #WORKDIR /app.
I also cleaned the other parts a bit up to utilize the WORKDIR more.
FROM python:3.6
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
EXPOSE 8000
CMD python website/manage.py runserver 0.0.0.0:8000
I think there is nothing wrong with COPY. But, you need to set the work directory to /app as your manage.py file is inside /app/website, not in /website inside Docker.
So, I think your Dockerfile should be like this:
FROM python:3.6
RUN mkdir /app
COPY requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt
COPY . /app
WORKDIR /app
EXPOSE 8000
CMD python website/manage.py runserver 0.0.0.0:8000

Can we use RUN cd /app instead of using WORKDIR /app in dockerfile

I want to use RUN cd /app instead of WORKDIR /app in my dockerfile.
Is there any way to do this?
Yes, you can mkdir /app and then cd /app. WORKDIR is used to specify the directory whenever you exec bash to connect to container or the working directory when container start.

Docker error with pipenv on django app: Warning: --system is intended to be used for pre-existing Pipfile

When I was trying to dockerize my django app, I followed a tutorial telling me to structure my Dockerfile like this
FROM python:3.6
ENV PYTHONUNBUFFERED 1
COPY . /code/
WORKDIR /code/
RUN pip install pipenv
RUN pipenv install --system
EXPOSE 8000
After I saved that and run docker build .
the system threw me this error
Warning: --system is intended to be used for pre-existing Pipfile
installation,not installation of specific packages. Aborting.
I think it is complaining about the --system suffix above but the tutorial says it's crucial to have it so that my packages are applied in the entire docker container. I'm new to docker and even pipenv because I took over a previous person's code and isn't sure where their pipfile is or even if they have a pipfile. If you have any insights on how to fix this error thank you in advance.
pipenv --rm
This helped me! I was starting the "Django for beginners" and at the very beginning, got this error (accidently deleted Pipfile & Pipfile.lock)
Your warning is saying you that there is no Pipfile in your project dir.
--system is intended to be used for pre-existing Pipfile.
So before running
docker build .
run
pipenv install
in your project folder
Above solution didn't work for me.
After installing in the virtual env I also had to explicitly include Pipfile and Pipfile.lock into my dockerfile:
COPY Pipfile* .
# Install dependencies
RUN pip install pipenv && pipenv install --system
Then rebuild with docker compose:
docker-compose build
You can find more info in this thread
It has Error in pipenv
It is 👇 ERROR:: --system is intended to be used for pre-existing Pipfile installation, not installation of specific packages. Aborting.
try it
pipenv check or python3 -m pipenv check
Be careful when using Docker bind mounts!
Summary: In my case, I was using bind mounts in my dev environment, and mounting a docker bind mount on a non-empty directory would overwrite the contents of the container's directory, removing the Pipfile and Pipfile.lock, which showed the error mentioned when running the container.
Explanation
Directory structure on the host
> ls project/
docker-compose.yml Dockerfile Pipfile Pipfile.lock app/
Dockerfile
My Dockerfile would copy the contents of the project and then install the dependencies with pipenv, like this:
FROM python:3.8
# ...
COPY Pipfile Pipfile.lock /app/
RUN pipenv install --deploy --ignore-pipfile
COPY ./app /app/
CMD ["pipenv", "run", "uvicorn", "etc..", "--reload"]
Pipfile, Pipfile.lock and the code of ./app would all be in the same /app directory inside the container.
docker-compose.yml
I wanted uvicorn to hot-reload, so I mounted the code in /app inside the container's /app directory.
service:
app:
#...
volumes:
- type: bind
source: ./app
target: /app
This meant that when I changed the code in /app, the code in the container's /app directory would also change.
Effects
The side effect of this bind mount is that the content mounted on /app "obscured" the content previously copied in there.
Container's content with the bind mount:
> ls app/
code1.py code2.py
Container's content without the bind mount:
> ls app/
Pipfile Pipfile.lock code1.py code2.py
Solution
Either make sure that you include the Pipfile and Pipfile.lock as well when mounting the bind mount, or make sure that you COPY these 2 files to a directory that won't get overwritten by a bind mount.