Configuring Django with Nginx, uWSGI and Postgres on Docker - django

I'm trying to setup a Django app on Docker with Nginx, uWSGI and Postgres. I found this great guide on setting up Compose for Django and Postgres: https://docs.docker.com/v1.5/compose/django/
However, now I need to add Nginx and uWSGI. I've tried using files of this repo (https://github.com/baxeico/django-uwsgi-nginx) with the Compose setup of the Docker docs but without succes, sadly.
This is what happens when I enter docker-compose run web:
Step 17 : RUN pip install -r /home/docker/code/app/requirements.txt
---> Running in e1ec89e80d9c
Collecting Django==1.9.1 (from -r /home/docker/code/app/requirements.txt (line 1))
/usr/local/lib/python2.7/dist-packages/pip-7.1.2-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Downloading Django-1.9.1-py2.py3-none-any.whl (6.6MB)
Collecting psycopg2 (from -r /home/docker/code/app/requirements.txt (line 2))
Downloading psycopg2-2.6.1.tar.gz (371kB)
Complete output from command python setup.py egg_info:
running egg_info
creating pip-egg-info/psycopg2.egg-info
writing pip-egg-info/psycopg2.egg-info/PKG-INFO
writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt
writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt
writing manifest file 'pip-egg-info/psycopg2.egg-info/SOURCES.txt'
warning: manifest_maker: standard file '-c' not found
Error: pg_config executable not found.
Please add the directory containing pg_config to the PATH
or specify the full executable path with the option:
python setup.py build_ext --pg-config /path/to/pg_config build ...
or with the pg_config option in 'setup.cfg'.
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-XRgbSA/psycopg2
ERROR: Service 'web' failed to build: The command '/bin/sh -c pip install -r /home/docker/code/app/requirements.txt' returned a non-zero code: 1
This is my Dockerfile:
from ubuntu:precise
run echo "deb http://us.archive.ubuntu.com/ubuntu/ precise-updates main restricted" | tee -a /etc/apt/sources.list.d/precise-updates.list
# update packages
run apt-get update
# install required packages
run apt-get install -y python python-dev python-setuptools python-software-properties
run apt-get install -y sqlite3
run apt-get install -y supervisor
# add nginx stable ppa
run add-apt-repository -y ppa:nginx/stable
# update packages after adding nginx repository
run apt-get update
# install latest stable nginx
run apt-get install -y nginx
# install pip
run easy_install pip
# install uwsgi now because it takes a little while
run pip install uwsgi
# install our code
add . /home/docker/code/
# setup all the configfiles
run echo "daemon off;" >> /etc/nginx/nginx.conf
run rm /etc/nginx/sites-enabled/default
run ln -s /home/docker/code/nginx-app.conf /etc/nginx/sites-enabled/
run ln -s /home/docker/code/supervisor-app.conf /etc/supervisor/conf.d/
# run pip install
run pip install -r /home/docker/code/app/requirements.txt
run cd /home/docker/code/app && ./manage.py syncdb --noinput
expose 80
cmd ["supervisord", "-n"]
And the docker-compose.yml:
db:
image: postgres
web:
build: .
command: python vms/manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
links:
- db
There are also files named nginx-app.conf, supervisor-app.conf, uwsgi_params and uwsgi.ini. These are all from the aforementioned repo. Requirements.txt contains Django 1.9.1, psycopg2 and requests.
If there is a better alternative to this Frankenstein project, I'd love to hear it.

On Ubuntu, make sure that python-dev and libpq-dev have been installed using apt-get, before trying to install psycopg2 using pip.
See the installation docs for more info.

Related

Django Docker ElasticBeanstalk fails

I'm new to docker and eb deployment, I want to deploy django with docker on eb
here's what I did so far
created a Dockerfile
# Pull base image
FROM python:3.9.16-slim-buster
# Set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
RUN apt-get update &&\
apt-get install -y binutils libproj-dev gdal-bin python-gdal python3-gdal libpq-dev python-dev libcurl4-openssl-dev libssl-dev gcc
# install dependencies
COPY . /code
WORKDIR /code/
RUN pip install -r requirements.txt
# set work directory
WORKDIR /code/app
then in docker-compose.yml
version: '3.7'
services:
web:
build: .
command: python /code/hike/manage.py runserver 0.0.0.0:8000
ports:
- 8000:8000
volumes:
- .:/code
it runs locally, but on deployment it fails and when I get to logs, it says
pg_config is required to build psycopg2 from source.
like it's not using the Dockerfile, I read somewhere I should set Dockerrunder.aws.json but I've no idea what to write in it!

Running "/usr/local/bin/gunicorn" in a docker build says " stat /usr/local/bin/gunicorn: no such file or directory"

From the toplevel maps directory, I'm able to install the gunicorn extension ...
(venv) localhost:maps davea$ pip3 install gunicorn
Collecting gunicorn
Downloading gunicorn-20.0.4-py2.py3-none-any.whl (77 kB)
|████████████████████████████████| 77 kB 1.2 MB/s
Requirement already satisfied: setuptools>=3.0 in ./web/venv/lib/python3.7/site-packages (from gunicorn) (45.1.0)
Installing collected packages: gunicorn
Successfully installed gunicorn-20.0.4
Below is my docker-compose.yml file
version: '3'
services:
web:
restart: always
build: ./web
ports: # to access the container from outside
- "8000:8000"
environment:
DEBUG: 'true'
command: /usr/local/bin/gunicorn maps.wsgi:application -w 2 -b :8000
apache:
restart: always
build: ./apache/
ports:
- "80:80"
#volumes:
# - web-static:/www/static
links:
- web:web
mysql:
restart: always
image: mysql:5.7
environment:
MYSQL_DATABASE: 'maps_data'
# So you don't have to use root, but you can if you like
MYSQL_USER: 'chicommons'
# You can use whatever password you like
MYSQL_PASSWORD: 'password'
# Password for root access
MYSQL_ROOT_PASSWORD: 'password'
ports:
- "3406:3406"
volumes:
- my-db:/var/lib/mysql
volumes:
my-db:
And then I have web/Dockerfile as follows ...
FROM python:3.7-slim
RUN apt-get update && apt-get install
RUN apt-get install -y libmariadb-dev-compat libmariadb-dev
RUN apt-get update \
&& apt-get install -y --no-install-recommends gcc \
&& rm -rf /var/lib/apt/lists/*
RUN python -m pip install --upgrade pip
RUN mkdir -p /app/
WORKDIR /app/
RUN pip3 freeze > requirements.txt
COPY requirements.txt requirements.txt
RUN python -m pip install -r requirements.txt
COPY . /app/
However, when I build/start my docker instance, I'm told it can't find my "gunicorn" command ...
(venv) localhost:maps davea$ docker-compose up
Starting maps_web_1 ...
Starting maps_web_1 ... error
ERROR: for maps_web_1 Cannot start service web: OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \"/usr/local/bin/gunicorn\": stat /usr/local/bin/gunicorn: no such file or directory": unknown
ERROR: for web Cannot start service web: OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \"/usr/local/bin/gunicorn\": stat /usr/local/bin/gunicorn: no such file or directory": unknown
ERROR: Encountered errors while bringing up the project.
Your Docker container is a totally isolated environment. Nothing you install on the host is visible inside the container; nothing that happens inside the container is accessible on the host. There's ways to bridge this boundary (with docker run -v bind mounts) but that's not possible during the docker build phase.
In this example your local source tree has a requirements.txt file that lists out the packages that need to be installed when the container is created. (The RUN pip freeze line has no effect; the COPY on the line after it copies it from your local source tree.) It's enough to add the dependency to the requirements.txt file
gunicorn
In your development environment, you can re-run pip install -r requirements.txt to update the packages installed in your virtual environment. When you re-run docker build, having this line in the requirements.txt file will cause it to be installed when the package is built.
You can clean up the Dockerfile a little bit. The resulting Dockerfile would be a pretty typical one for Python packages with C dependencies:
# Start from a totally clean environment with Python installed,
# but no non-system libraries and nothing from your host system.
FROM python:3.7-slim
# Install C dependencies.
# It's important to do apt-get update and install in the
# same command. It's more efficient to only do it once.
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
gcc \
libmariadb-dev \
libmariadb-dev-compat
# Update pip
RUN python -m pip install --upgrade pip
# Create the application directory and point there
# (WORKDIR will implicitly create it)
WORKDIR /app/
# Install all of the Python dependencies. These are
# listed, one to a line, in the requirements.txt file,
# possibly with version constraints. Having this as
# a separate block allows Docker to not repeat it if
# only your application code changes.
COPY requirements.txt .
RUN python -m pip install -r requirements.txt
# Copy in the rest of the application.
COPY . .
# Specify what port your application uses, and the
# default command to use when launching the container.
EXPOSE 8000
CMD /usr/local/bin/gunicorn maps.wsgi:application -w 2 -b :8000

Error installing/initiating pgadmin4 SQLAlchemy 1.1.0 or greater is required

i'm installing pgadmin4 in Mint 19, following this instructions:
https://linuxhint.com/install-pgadmin4-ubuntu/
When i come to run:
https://linuxhint.com/install-pgadmin4-ubuntu/
I have an error:
File "/home/nico/pgadamin4/pgAdmin4/local/lib/python2.7/site-packages/alembic/util/__init__.py", line 31, in <module>
raise CommandError("SQLAlchemy 1.1.0 or greater is required. ")
alembic.util.exc.CommandError: SQLAlchemy 1.1.0 or greater is required
at this point, i don't know how to proceed. I don't really understand the whole installing process, using the virtual environment, so i can't add more information. Anyone can help? Thanks!
Why are you doing it manually?
pgAdmin4 now comes as native linux package.
Goto postgres site, Select your Ubuntu version as "Ubuntu Bionic" (which your Linux mint is based on).
https://www.postgresql.org/download/linux/ubuntu/
Add postgres repo as show on above link, and then do
sudo apt-get update && sudo apt-get install pgadmin4
For pgAdmin 4 v4.21 on Ubuntu, according to the download page:
Install dependencies, create a virtual environment, download, install & configure
Using Python2.x
sudo apt-get install virtualenv python-pip libpq-dev python-dev
cd
virtualenv pgadmin4
cd pgadmin4
source bin/activate
pip install https://ftp.postgresql.org/pub/pgadmin/pgadmin4/v4.21/pip/pgadmin4-4.21-py2.py3-none-any.whl
Using Python3.6 (Preferred to avoid encoding related issues)
sudo apt-get install virtualenv python3-pip libpq-dev python3-dev
cd
virtualenv -p python3 pgadmin4
cd pgadmin4
source bin/activate
pip3 install https://ftp.postgresql.org/pub/pgadmin/pgadmin4/v4.21/pip/pgadmin4-4.21-py2.py3-none-any.whl
Configure
Override default paths and set it to single-user mode in the local configuration file:
nano lib/python2.7/site-packages/pgadmin4/config_local.py
For Python3.x:
nano lib/python3.6/site-packages/pgadmin4/config_local.py
Write:
import os
DATA_DIR = os.path.realpath(os.path.expanduser(u'~/.pgadmin/'))
LOG_FILE = os.path.join(DATA_DIR, 'pgadmin4.log')
SQLITE_PATH = os.path.join(DATA_DIR, 'pgadmin4.db')
SESSION_DB_PATH = os.path.join(DATA_DIR, 'sessions')
STORAGE_DIR = os.path.join(DATA_DIR, 'storage')
SERVER_MODE = False
Run
python lib/python2.7/site-packages/pgadmin4/pgAdmin4.py
For Python3.x:
python3 lib/python3.6/site-packages/pgadmin4/pgAdmin4.py
Access
Access at http://localhost:5050
Exit
Exit with Ctrl-C
Run again
cd ~/pgadmin4
source bin/activate
python lib/python2.7/site-packages/pgadmin4/pgAdmin4.py
For Python3.6
#!/bin/bash
cd ~/pgadmin4
source bin/activate
python3 lib/python3.6/site-packages/pgadmin4/pgAdmin4.py
Make a shortcut
touch ~/pgadmin4/pgadmin4
chmod +x ~/pgadmin4/pgadmin4
nano ~/pgadmin4/pgadmin4
Write:
#!/bin/bash
cd ~/pgadmin4
source bin/activate
python lib/python2.7/site-packages/pgadmin4/pgAdmin4.py
For Python3.6
#!/bin/bash
cd ~/pgadmin4
source bin/activate
python3 lib/python3.6/site-packages/pgadmin4/pgAdmin4.py
Now you can just run it with a simpler command:
~/pgadmin4/pgadmin4
Python3 users
Replace [x] in Python3.6 with your respective version.
Conflict with pgAdmin 3 configuration
pgAdmin 4 will not start in the environment where pgAdmin 3 was previously installed and used because of incompatible configuration in the .pgadmin directory. The simplest solution is to either clear that directory or tweak config_local.py to point to a clean new .pgadmin4 directory.

gunicorn: command not found django

Using Docker to install gunicorn, I am unable to to use the gunicorn command.
To start Django, I have this line in my docker-compose.yaml:
command: bash -c "python manage.py makemigrations && python manage.py migrate && gunicorn myproject.wsgi -b 0.0.0.0:8000"
This results in bash: gunicorn: command not found
When I build the Docker images it says gunicorn has been successfully installed.
My Dockerfile looks like:
FROM python:3.5
ENV PYTHONUNBUFFERED 1
RUN mkdir /config
ADD requirements.txt /config/
RUN pip install -r /config/requirements.txt
RUN mkdir /src;
WORKDIR /src
I've been using this http://ruddra.com/2016/08/14/docker-django-nginx-postgres/ as a guide.
If you are finding that gunicorn doesn't exist it could be because
docker image may use a cached layer of the requirements.txt which doesn't have gunicorn in it as a dependency.
Therefore it will result in not installing gunicorn meanwhile specifying pip install gunicorn in a seperate RUN command will work.
Solution:
Build docker image without caching when edits have been made to requirements.txt
docker build --no-cache .

Amazon S3 + Docker - "403 Forbidden: The difference between the request time and the current time is too large"

I am trying to run my django application in a docker container with static files served from Amazon S3. When I run RUN $(which python3.4) /home/docker/code/vitru/manage.py collectstatic --noinput in my Dockerfile, I get a 403 Forbidden error from Amazon S3 with the following response XML
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>RequestTimeTooSkewed</Code>
<Message>The difference between the request time and the current time is too large.</Message>
<RequestTime>Sat, 27 Dec 2014 11:47:05 GMT</RequestTime>
<ServerTime>2014-12-28T08:45:09Z</ServerTime>
<MaxAllowedSkewMilliseconds>900000</MaxAllowedSkewMilliseconds>
<RequestId>4189D5DAF2FA6649</RequestId>
<HostId>lBAhbNfeV4C7lHdjLwcTpVVH2snd/BW18hsZEQFkxqfgrmdD5pgAJJbAP6ULArRo</HostId>
</Error>
My docker container is running Ubuntu 14.04... if that makes any difference.
I also am running the application using uWSGI, without nginx or apache or any other kind of reverse-proxy server.
I also get the error at run-time, when the files are being served to the site.
Attempted Solution
Other stackoverflow questions have reported a similar error using S3 (none specifically in conjunction with Docker) and they have said that this error is caused when your system clock is out of sync, and can be fixed by running
sudo service ntp stop
sudo ntpd -gq
sudo service ntp start
so I added the following to my Dockerfile, but it didn't fix the problem.
RUN apt-get install -y ntp
RUN ntpd -gq
RUN service ntp start
I also attempted to sync the time on my local machine before building the docker image, using sudo ntpd -gq, but that did not work either.
Dockerfile
FROM ubuntu:14.04
# Get most recent apt-get
RUN apt-get -y update
# Install python and other tools
RUN apt-get install -y tar git curl nano wget dialog net-tools build-essential
RUN apt-get install -y python3 python3-dev python-distribute
RUN apt-get install -y nginx supervisor
# Get Python3 version of pip
RUN apt-get -y install python3-setuptools
RUN easy_install3 pip
# Update system clock so S3 does not get 403 Error
# NOT WORKING
#RUN apt-get install -y ntp
#RUN ntpd -gq
#RUN service ntp start
RUN pip install uwsgi
RUN apt-get -y install libxml2-dev libxslt1-dev
RUN apt-get install -y python-software-properties uwsgi-plugin-python3
# Install GEOS
RUN apt-get -y install binutils libproj-dev gdal-bin
# Install node.js
RUN apt-get install -y nodejs npm
# Install postgresql dependencies
RUN apt-get update && \
apt-get install -y postgresql libpq-dev && \
rm -rf /var/lib/apt/lists
# Install pylibmc dependencies
RUN apt-get update
RUN apt-get install -y libmemcached-dev zlib1g-dev libssl-dev
ADD . /home/docker/code
# Setup config files
RUN ln -s /home/docker/code/supervisor-app.conf /etc/supervisor/conf.d/
RUN pip install -r /home/docker/code/vitru/requirements.txt
# Create directory for logs
RUN mkdir -p /var/logs
# Set environment as staging
ENV env staging
# Run django commands
# python3.4 is at /usr/bin/python3.4, but which works too
RUN $(which python3.4) /home/docker/code/vitru/manage.py collectstatic --noinput
RUN $(which python3.4) /home/docker/code/vitru/manage.py syncdb --noinput
RUN $(which python3.4) /home/docker/code/vitru/manage.py makemigrations --noinput
RUN $(which python3.4) /home/docker/code/vitru/manage.py migrate --noinput
EXPOSE 8000
CMD ["supervisord", "-c", "/home/docker/code/supervisor-app.conf"]
Noted in the comments but for others who come here:
If using boot2docker (i.e. If on windows or Mac), the boot2docker vm has a known time issue when you sleep your machine--see here. Since the host for your docker container is the boot2docker vm, that's where it syncs its time.
I've had success restarting the boot2docker vm. This may cause problems with losing some state, i.e. If you had some data volumes.
Docker containers share clock with the host machine, so syncing your host machine clock should solve the problem. To force the timezone of your container is the same as your host machine you can add -v /etc/localtime:/etc/localtime:ro in docker run.
Anyway, you should not start a service in a Dockerfile. This file contains the steps and commands to build the image for your containers, and any process you run inside a Dockerfile will end after the building process. To start any service you should add a run script or a process control daemon (as supervisord) which will run each time you run a new container.
Restarting Docker for Mac fixes the error on my machine.