I am trying to deploy my Docker container with Django server on Google APP Engine Custom environment,
although it gets deployed but it doesn't start working the way it should work i.e it seems django runserver is not working .
app.yaml:
runtime: custom
env: flex
service: jobs
resources:
cpu: 4
memory_gb: 8
disk_size_gb: 30
Dockerfile:
FROM hsingh1993/jobs:fourth
EXPOSE 8000/tcp
RUN id
USER jovyan
WORKDIR /home/jovyan/trustdjobs
CMD ["python","./manage.py","runserver","0.0.0.0:8000"]
Update 1:
KeyError: 'scrappy'
at call_command (/opt/conda/lib/python3.7/site-packages/django/core/management/__init__.py:103)
It seems your django application is not configured properly, Check urls.py under project to see path defined. Your Django is working properly but when you go on to the app engine URL .
Related
There is a really old thread on stackoverflow here Getting 'DatabaseOperations' object has no attribute 'geo_db_type' error when doing a syncdb
but the difference that I have with their issue is that my containers have the POSTGIS and POSTGRES installed in. Specifically I used QGIS and the image is like so
db:
image: kartoza/postgis:13.0
volumes:
- postgis-data:/var/lib/postgresql
So locally I have two docker images - one is web and the other is the kartoza/postgis
I also have this as well in the settings.py file
import dj_database_url
db_from_env = dj_database_url.config(conn_max_age=500)
DATABASES['default'].update(db_from_env)
which should support the GIS data. I see all my packages gis, geolocation packages installed with no issues. But I am getting the above error when I run heroku run python manage.py migrate
The website runs with very limited functionality as the geo variables are needed to get you past the landing page.
The steps I have taken to deploy is
heroku create appname
heroku stack:set container -a appname
heroku addons:create heroku-postgresql:hobby-dev -a appname
heroku git:remote -a appname
git push heroku main
EDIT The db url on heroku is postgres://foobar:3242q34rq2rq32rf3q2rfq2q2r3vq23rvq23vr#er3-13-234-91-69.compute-
I have also ran the below command and it shows that the db now takes GIS, but I still get the error
$ heroku pg:psql
create extension postgis;
try replacing db with localhost
heroku config:add DATABASE_URL=postgis://foo:bar#localhost:5432/gis
Ok so after 14 or so hours of reading. The issue here is that
Heroku database does NOT have qgis capabilities enabled as a default - you have to use heroku pg:psql then run CREATE EXTENSION postgis;
Having QGIS installed within the docker container results it not being able to perform functions such as hero pg:push localdatabase -postgresqlremoteherokudb and heroku config:add DATABASE_URL=postgis://foo:bar#localhost:5432/gis So if anyone is following the python nearbyshops tutorial, you might end up here as well. I got fixated here thinking it's a db issue, but it was more so to do with heroku configuration settings.
If you are coming here from docker, which might mean you need to install django_heroku package to modify your settings. There are recent 2019 and 2018 guides on how to deploy Geospatial apps to heroku, but again, you don't need those. (at least for me I didn't)
GIS now is available as an extension, you no longer need to add buildpacks.
There are still some configuration issues if you are using gunicorn for staticfiles, so the website at first won't launch properly. Highly suggest that you track your heroku log errors (manual settings.py coding or heroku addons).
Good afternoon,
I am looking for some guidance on where to concentrate my efforts. I keep getting off down these rabbit holes and cannot seem to find the path I am looking for.
I have developed a couple small internal django apps but wish to integrate VueJS into the mix for more dynamic front end.
My goals are:
I want to use Django-restframework for the backend calls
I want to use VueJS for the front end and make calls back to the REST API.
I want all of this to live in Docker container(s) that I can sync using Jenkins.
My questions / concerns:
I keep trying to build a single docker container for both VueJS and Django but starting with either Node or Python, I seem to end up in dependency hell. Does anyone have a good reference link?
I can't decide if I want it completely decoupled or to try to preserve some of the Django templates. The reason for the latter is that I don't want to lose the built in Django authentication. I am not skilled enough to write the whole authentication piece so I would rather not lose that already being done.
If I am complete decoupled and django is strictly the API, I could also have a single docker container for the django, and a second docker container for the front end. Thoughts?
Finally, these webapps are all the same risk level and exist on the same web app server with a separate postgres database server. Should nginx be on the server, then gunicorn in the docker container with django? Where do most devs draw the line on what is native on the server and what is being served from a docker container? These are all pretty low volume apps targeted for specific purposes.
Thanks for all your guidance as I continue to venture into new territory.
Kevin
*Update December 2020
If you are interested in SSR, here's an outline for an approach I have been working on:
Update December 2020
Here's another way to do Django with Vue in containers on DigitalOcean with docker swarm. It a lot simpler than the approach with AWS shown below, but not as scalable:
Update June 2020
I have made some changes to the project that make use of the Cloud Development Kit (CDK) and AWS Fargate instead of ECS with container instances to take advantage of serverless infrastructure.
Here's an overview of the project architecture: https://verbose-equals-true.gitlab.io/django-postgres-vue-gitlab-ecs/start/overview/
and here's an updated architecture diagram:
Edit May 2019
Here is a setup for Django and Vue using ECS, the AWS container orchestration tool and GitLab for CI/CD. The repo is here.
I have been working on a project that demonstrates how to set up a Django + Vue project with docker. It is an ope source project called verbose-equals-true (https://verbose-equals-true.tk). The source code for this project is available here: https://gitlab.com/briancaffey/verbose-equals-true
Here is an overview of how I'm structuring the project for production. The project uses docker compose to orchestrate the different containers for production and also for development.
Let me know if you have any questions any questions about how I'm using Django/Vue/docker. I have documentation with detailed descriptions at https://verbose-equals-true.tk/docs.
Here are some thoughts on your questions/concerns:
I started with the official recommendations from VueJS for how to dockerize a Vue app, and an official example from Docker about how to dockerize a postgres + Django app. You can probably put everything in the same container, but I like separating things out as a way of keeping things modular and clear.
I'm using JWT for authentication with djangorestframework_jwt package. I can also use the builtin Django authentication system and Django admin.
I think it makes sense to have separate containers. In development you can serve the Vue app from the node container running npm run serve, and in production you can serve the production app's static files from an nginx container (you can use a multi-stage build for this part).
I would keep everything in containers with nothing on the server except the docker engine. This will simplify setup and will allow you to keep your application portable to wherever you decide to deploy it. The only thing that makes sense to keep separate is the postgres database. It is often much easier to connect to a managed database service like AWS RDS, but it is also possible to run a postgres container on your docker host machine to keep things even simpler. This will require that you do backups on your own, so you will need to be familiar with docker volumes.
I've been working with Django/Vue and this is what I do:
Create the Django project
Initialize the project's folder as new Vue project using the vue-cli
From here I can start two development servers, one for Django and the other for Vue:
python manage.py runserver
In another terminal:
npm run serve
In order to consume my API in Vue this I use this configuration in vue.config.js:
module.exports = {
baseUrl: process.env.NODE_ENV === 'production'
? '/static/'
: '/',
outputDir: '<PROJECT_BASE_DIR>/static',
indexPath: '../templates/index.html',
filenameHashing: false,
devServer: {
proxy: {
'/api': {
target: 'http://localhost:8000'
}
}
},
}
devServer redirects the requests to the API, outputDir and indexPath help to build the app to the project's folder, <PROJECT_BASE_DIR>/templates/ and <PROJECT_BASE_DIR>/static/
The next thing is to create a TemplateView and set the template_name to index.html (the file built by Vue), with this you have your SPA inside a Django view/template.
With this approach you can use a Docker container for Django.
I hope this gives you some basic guidance to continue.
Alejandro
#docker-compose.yml
version: "3.9"
services:
db:
image: postgres
environment:
- POSTGRES_NAME=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
server:
build: server/
command: python manage.py runserver 0.0.0.0:8000
ports:
- "8000:8000"
environment:
- POSTGRES_NAME=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
depends_on:
- db
client:
build: client/
command: npm run serve
ports:
- '8081:8080'
depends_on:
- server
#server django Dockerfile
# pull the official base image
FROM python:3
# set work directory
WORKDIR /usr/src/app
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
ENV DJANGO_SUPERUSER_PASSWORD="admin"
# install dependencies
RUN pip install --upgrade pip
COPY ./requirements.txt /usr/src/app
RUN pip install -r requirements.txt
# copy project
COPY . /usr/src/app
EXPOSE 8000
CMD ["python", "manage.py", "runserver"]
#client vue Dockerfile
FROM node:14.17.5 as build-stage
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY ./ .
RUN npm run build
EXPOSE 8080
CMD [ "npm", "run", "serve" ]
it's working
I am building a Python+Django development environment using docker. I defined Dockerfile files and services in docker-compose.yml for web server (nginx) and database (postgres) containers and a container that will run our app using uwsgi. Since this is a dev environment, I am mounting the the app code from the host system, so I can easily edit it in my IDE.
The question I have is where/how to run migrate command.
In case you don't know Django, migrate command creates the database structure and later changes it as needed by the project. I have seen people run migrate as part of the compose command directive command: python manage.py migrate && uwsgi --ini app.ini, but I do not want migrations to run on every container restart. I only want it to run once when I create the containers and never run again unless I rebuild.
Where/how would I do that?
Edit: there is now an open issue with the compose team. With any luck, one time command containers will get supported by compose. https://github.com/docker/compose/issues/1896
You cannot use RUN because as you mentioned in the comments your source is mounted during running of the container.
You cannot use CMD either since you don't want it to run everytime you restart the container.
I recommend using docker exec manually after running the container. I do not think there is a way to automate this inside a dockerfile or docker-compose because of the two reasons I gave above.
It sounds like what you need is a tool for managing project tasks. dobi is a tool designed to handle these tasks (disclaimer: I am the author of this tool).
You can see an example of how to run a migration here: https://github.com/dnephin/dobi/tree/master/examples/init-db-with-rails. The example uses rails, but it's basically the same idea as django.
You could setup a task called migrate which would run the command in a container and write the data to a volume. Then when you start your docker-compose containers, use that volume as the source for your database service.
https://github.com/docker/compose/issues/1896 is finally resolved now by the new service profiles introduced with docker-compose 1.28.0. With profiles you can mark services to be only started in specific profiles:
services:
nginx:
# ...
postgres:
# ...
uwsgi:
# ...
migrations:
profiles: ["cli-only"] # profile name chosen freely
# ...
docker-compose up # start only your app services, no migrations
docker-compose run migrations # run migrations on-demand
docker exec -it container-name bash
Then you will be inside the container and you can run any command you normally do when you develop without using docker.
I have quite the Docker stack at the moment, compiled from many containers, one of which is running an instance of Django.
At the moment, I'm limited to debugging by importing logging and using
logger = logging.getLogger(__name__)
logger.debug("your variable: " + variableName)
It's totally inefficient and requires me to rebuild the docker stack every time I want to re-evaluate a change.
I'm used to working in Komodo and having a robust, step-able debugger at my disposal, but I can's seem to find any documents on how to wire up a Docker container inside a vagrant VM to an IDE (or command line debugger) that will let me step through code without a rebuild.
How can I wire up a debugging IDE to a docker container inside a Vagrant VM? Thanks.
I recommend you to use Docker Compose to handle and link your containers. I'm also using a Docker stack on my dev env with a container for
- django
- postgres
- nginx
You just have to synchronize your code with the code inside your docker container. To do that, use the volumes command in your docker-compose file. Here is an example, with 2 containers (django and postgres) :
db:
image: postgres
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/webapp
ports:
- "8000:8000"
links:
- db
This portion of code will do what you want. All your project . will be synchronized with the /webapp folder of your docker container then, no more need to rebuild your docker image:
volumes:
- .:/webapp
Then, to debug, I recommend you to use pdb, which is in my opinion , the best way to debug your django app, run :
docker-compose -f [path/to/your/docker-compose.yml] --service-ports [name-of-your-django-container] python manage.py runserver
E.g :
docker-compose -f django_project/docker-compose.yml --service-ports django python manage.py runserver
Let's debug a view :
1. import pdb in the view :import pdb
2. add pdb.set_trace() in a method or a class in your view
3. request the right url
and you will be able to debug through your terminal
You should have something like that :
(Pdb) > /webapp/car/views.py(18)get()
-> for car in serializer.data:
Here is a tutorial to use Compose and Django from Docker : Quickstart Guide: Compose and Django
Can you tell me what is the best way to way to set up and configure a Django/PostgreSQL project with Docker Compose?
(with the latest versions of everything, python 3.4, django 1.8.1, etc.)
Have you looked at the examples on the Docker website? Here's a link describing exactly this.
Basically, you need two services, one for your Django app and one for your Postgres instance. You would probably want to build a Docker image for your Django app from you current folder, hence you'll need to define a Dockerfile:
# Dockerfile
FROM python:3.4-onbuild
That's the whole Dockerfile! Using the magic -onbuild image, files are automatically copied to the container and the requirements are installed with pip. For more info, read here.
Then, you simply need to define your docker-compose.yml file:
# docker-compose.yml
db:
image: postgres
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
links:
- db
Here, you've defined the Postgres service, which is built from the latest postgres image. Then, you've defined your Django app's service, built it from the current directory, exposed port 8000 so that you can access it from outside your container, linked to the database container (so that they can magically communicate without anything specific from your side - more details here) and started the container with the classic command you use to normally start your Django app. Also, a volume is defined in order to sync the code you're writing with the one from inside your container (so that you don't need to rebuild your image every time you change the code).
Related to having the latest Django version, you just have to specify it in your requirements.txt file.