I've seen other posts about this such as here but I still can't figure out my issue.
File structure of relevant files:
ba_django_project
| docker-compose.prod.yml
|
|__app
| Dockerfile.prod
| manage.py
| .env.prod
|
|__ba_django_project
| __init__.py
| settings.py
| wsgi.py
docker-compose.prod.yml:
version: '3.8'
services:
web:
build:
context: ./app
dockerfile: Dockerfile.prod
image: 370950449536.dkr.ecr.us-east-2.amazonaws.com/ba_django_project:ba_django_project_web
command: gunicorn ba_django_project.wsgi:application --bind 0.0.0.0:8000
volumes:
- static_volume:/home/app/web/staticfiles
- media_volume:/home/app/web/mediafiles
expose:
- 8000
env_file:
- ./app/.env.prod
depends_on:
- db
db:
image: postgres:13.0-alpine
volumes:
- postgres_data:/var/lib/postgresql/data/
env_file:
- ./app/.env.prod.db
nginx:
build: ./nginx
image: 370950449536.dkr.ecr.us-east-2.amazonaws.com/ba_django_project:ba_django_project_nginx
volumes:
- static_volume:/home/app/web/staticfiles
- media_volume:/home/app/web/mediafiles
ports:
- 1337:80
depends_on:
- web
volumes:
postgres_data:
static_volume:
media_volume:
settings.py:
from pathlib import Path
from decouple import config
import os
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent
STATIC_DIR = os.path.join(BASE_DIR, "static")
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/3.2/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = os.environ.get('SECRET_KEY')
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = int(os.environ.get("DEBUG", default=0))
ALLOWED_HOSTS = [
'ec2-3-20-22-254.us-east-2.compute.amazonaws.com', '0.0.0.0', 'localhost', '127.0.0.1'
]
# Application definition
INSTALLED_APPS = [
'crispy_forms',
'users.apps.UsersConfig',
'resume.apps.ResumeConfig',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
]
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
]
# etc. can post full version if needed
.env.prod:
SECRET_KEY='secretkeyfiller'
manage.py:
#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys
def main():
"""Run administrative tasks."""
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'ba_django_project.settings')
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
) from exc
execute_from_command_line(sys.argv)
if __name__ == '__main__':
main()
wsgi.py
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'ba_django_project.settings')
application = get_wsgi_application()
I wasn't getting this error before I started trying to get the app to run on a server. Only possible solutions it seems are either I messed up something with DJANGO_SETTINGS_MODULE or there's some sort or circular dependency I can't find. I've looked through all my files to try to find files importing settings and I took out their references to settings and it still won't work.
You must have messed up with your SECRET_KEY while you were trying to secure it using .env.
You must have an actual secret key. E.g: SECRET_KEY=QhJgFHhvNJFFHjFfdH....in the.env file. If you don't have it, there are ways to generate it using this guide
NOTE: no quotes around the secret key in the .env file.
Although I could have sworn I double checked this, when I went to echoed my .env file into my server, the special characters in the secret key caused it to skip that line. This can help with inputting special characters using sed.
If that wasn't the issue, I also removed the quotes around my secret key.
Related
I've been building my first Django project (just running it locally so far using runserver), and I'm taking the steps to host it on Heroku, adding gunicorn. The build succeeds, but when I try to open the app, the Heroku logs show an exception in worker process:
ModuleNotFoundError: No module named 'mysite.wsgi'
Originally my Procfile had this:
web: gunicorn mysite.wsgi
When I would try that gunicorn command locally, it would work from within the family-django/mysite directory, but when I tried from the root directory (family-django), it would give me the same 'no module named mysite.wsgi' error. According to this post, Heroku's going to try from the root directory, so I updated my Procfile as follows, to tell it to run from the mysite directory instead:
web: gunicorn --chdir mysite mysite.wsgi
This new gunicorn command does work locally when run from the root directory (family-django), hooray!! That must be the fix I need. BUT: after updating the Procfile in Heroku and trying to open the app again, it still fails with that 'no module named mysite.wsgi' error pasted above. So there's some other tweak that Heroku needs for this to run there.
My Django project structure is like this:
family-django
|-mysite
| |-familytree
| |-myauth
| |-mysite
| |-asgi.py
| |-settings.py
| |-urls.py
| |-wsgi.py
|-Procfile
|-requirements.txt
wsgi.py and asgi.py were both created at the start of the project. (Not sure if it's wrong to have both?). wsgi.py has:
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mysite.settings')
application = get_wsgi_application()
settings.py has the following info for INSTALLED_APPS and WSGI_APPLICATION:
INSTALLED_APPS = [
'familytree.apps.FamilytreeConfig',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.contrib.humanize',
"myauth",
]
WSGI_APPLICATION = 'mysite.wsgi.application'
Now that I'd like to get this up on Heroku, they recommend gunicorn, so I've added a requirements.txt:
django
django-heroku
gunicorn
Procfile has:
web: gunicorn --chdir mysite mysite.wsgi
#----------------------------Install packages----------------------------
1)pip install django-heroku
2)pip install whitenoise
#-----------------------------setting.py----------------------------------#
1)INSTALLED_APPS = [
...,
'django_heroku',
]
2)MIDDLEWARE = [
'whitenoise.middleware.WhiteNoiseMiddleware',
]
3)STATICFILES_STORAGE =
'whitenoise.storage.CompressedManifestStaticFilesStorage'
4)STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
STATIC_URL = '/static/'
# Extra places for collectstatic to find static files.
STATICFILES_DIRS = (
os.path.join(BASE_DIR, 'static'),
)
django_heroku.settings(locals())
#-----------------------urls.py---------------------------------------#
from django.conf import settings
from django.conf.urls.static import static
urlpatterns = [
...,
] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
#------------------Procfile create in Root DIR-----------------#
paste in (web: gunicorn projectname.wsgi)
#--------------create requirements.txt---------------------------#
pip freeze > requirements.txt
# runtime.txt create in Root DIR
paste in (your python version for ex.python-3.8.5)
#---------then commands in terminal-------------------------#
heroku login
heroku create YOUR_APP_NAME
##for Clone the repository.......
git init
heroku git:clone -a YOUR_APP_NAME
## for Deploy your changes......
git init
git add .
git commit - m "initial"
git push heroku master
## then
heroku run python manage.py migrate
heroku run python manage.py createsuperuser
I try to "dockerize" a Django project.
I've already dockerized a Django project ; I've tried to use the exactly same project architecture.
I have first config dev environnement.
I buil/run the 2 containers (web and db): OK.
I migrate: OK
When I try to collectstatic I got an error
FileNotFoundError: [Errno 2] No such file or directory: '/usr/src/app/core/static'
and that right, static folder is at the same level as core app.
As I config dev environnement, with Django web server, I don't have to care about serving static files. So I've try to acces localhost:8000 and got an error:
That means I have an error in path config but I can find where as I replicate the same structure...
project architecture
- my_project
|_ app
|_ core
|_ wsqi.py
|_ settings
|_ base.py
|_ dev.py
|_ prod.py
|_ urls.py
|_ requirements
|_ base.txt
|_ dev.txt
|_ prod.txt
|_ static
|_ templates
|_ layout
|_ _footer.html
|_ _nav.html
|_ base.html
|_ home.html
|_ 404.html
|_ 500.html
|_ Dockerfile
|_ Dockerfile.prod
|_ entrypoint.sh
|_ entrypoint.prod.sh
|_ manage.py
|_ .dockerignore
|_ nginx
|_ .env.dev
|_ .env.prod
|_ .gitignore
|_ docker-compose.yml
|_ docker-compose.prod.yml
base.py
import os
from django.utils.translation import ugettext_lazy as _
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
SECRET_KEY = os.environ.get('SECRET_KEY')
DEBUG = int(os.environ.get("DEBUG", default=0))
ALLOWED_HOSTS = os.environ.get("DJANGO_ALLOWED_HOSTS").split(" ")
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'crispy_forms',
'widget_tweaks',
'bootstrap4',
'registration.apps.RegistrationConfig',
'monitor.apps.MonitorConfig',
'randomization.apps.RandomizationConfig',
'parameters.apps.ParametersConfig',
'unblind.apps.UnblindConfig',
'pharmacy.apps.PharmacyConfig',
'export.apps.ExportConfig',
'django_extensions',
'debug_toolbar',
'partial_date',
'safedelete',
'simple_history',
]
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.locale.LocaleMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'randomization.middleware.stock',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'debug_toolbar.middleware.DebugToolbarMiddleware',
'simple_history.middleware.HistoryRequestMiddleware',
]
ROOT_URLCONF = 'core.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [
os.path.join(BASE_DIR,'registration/templates'),
os.path.join(BASE_DIR,'monitor/templates'),
os.path.join(BASE_DIR,'randomization/templates'),
os.path.join(BASE_DIR,'unblind/templates'),
os.path.join(BASE_DIR,'pharmacy/templates'),
os.path.join(BASE_DIR,'export/templates'),
os.path.join(BASE_DIR,'templates'),
],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'randomization.context_processors.data_context_processor',
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
WSGI_APPLICATION = 'core.wsgi.application'
# Database
# https://docs.djangoproject.com/en/2.2/ref/settings/#databases
DATABASES = {
'default': {
"ENGINE": os.environ.get("SQL_ENGINE", "django.db.backends.sqlite3"),
"NAME": os.environ.get("SQL_DATABASE", os.path.join(BASE_DIR, 'db.sqlite3')),
"USER": os.environ.get("SQL_USER", "user"),
"PASSWORD": os.environ.get("SQL_PASSWORD", "password"),
"HOST": os.environ.get("SQL_HOST", "localhost"),
"PORT": os.environ.get("SQL_PORT", "5432"),
}
}
STATIC_URL = '/static/'
MEDIA_URL = "/media/"
INTERNAL_IPS = ['127.0.0.1']
LOGIN_URL = 'home'
LOGIN_REDIRECT_URL = 'home'
LOGOUT_REDIRECT_URL = 'home'
dev.py
from .base import *
STATICFILES_DIRS = (
os.path.join(BASE_DIR,'static'),
os.path.join(BASE_DIR,'randomization/static'),
os.path.join(BASE_DIR,'unblind/static'),
os.path.join(BASE_DIR,'pharmacy/static'),
)
.env.dev
SECRET_KEY=*******************************
DEBUG=1
DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
SQL_ENGINE=django.db.backends.postgresql
SQL_DATABASE=db_dev
SQL_USER=user
SQL_PASSWORD=user
SQL_HOST=db
SQL_PORT=5432
DATABASE=postgres
DJANGO_SETTINGS_MODULE=core.settings.dev
Dockerfile
# Pull the official base image
FROM python:3.8.3-alpine
# Set a work directory
WORKDIR /usr/src/app
# Set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install psycopg2 dependencies
RUN apk update && apk add postgresql-dev gcc python3-dev musl-dev
# J LE CARROU - 21/09/2020 - Ajouts
RUN apk --update add libxml2-dev libxslt-dev libffi-dev gcc musl-dev libgcc openssl-dev curl
RUN apk add jpeg-dev zlib-dev freetype-dev lcms2-dev openjpeg-dev tiff-dev tk-dev tcl-dev
RUN pip3 install psycopg2 psycopg2-binary
# Install dependencies
COPY requirements/ requirements/
RUN pip install --upgrade pip && pip install -r requirements/dev.txt
# Copy the entrypoint.sh file
COPY ./entrypoint.sh .
# Copy the project's files
COPY . .
# Run entrypoint.sh
ENTRYPOINT [ "/usr/src/app/entrypoint.sh" ]
version: '3.7'
services:
web:
build: ./app
restart: always
command: python manage.py runserver 0.0.0.0:8000
volumes:
- ./app/:/usr/src/app
ports:
- 8000:8000
env_file:
- ./.env.dev
depends_on:
- db
db:
image: postgres:12.0-alpine
restart: always
volumes:
- postgres_data:/var/lib/postgres/data/
environment:
- POSTGRES_USER=user
- POSTGRES_PASSWORD=user
- POSTGRES_DB=db_dev
volumes:
postgres_data:
The BASE_DIR is referring to app/core, so you need to adjust the TEMPLATES directories as below:
'DIRS': [
os.path.join(BASE_DIR,'registration/templates'),
os.path.join(BASE_DIR,'monitor/templates'),
os.path.join(BASE_DIR,'randomization/templates'),
os.path.join(BASE_DIR,'unblind/templates'),
os.path.join(BASE_DIR,'pharmacy/templates'),
os.path.join(BASE_DIR,'export/templates'),
os.path.join(os.path.dirname(BASE_DIR,'templates')),
],
EDIT
- my_project
|_app
|_core
|_wsqi.py
|_settings
|_base.py <-- settings.py used for development
|_dev.py
|_prod.py
|_urls.py <-- modifications
|_requirements
|_base.txt
|_dev.txt
|_prod.txt
|_Dockerfile
|_Dockerfile.prod
|_entrypoint.sh
|_entrypoint.prod.sh
|_manage.py
|_.dockerignore
|_nginx
|_.env.dev
|_.env.prod
|_.gitignore
|_docker-compose.yml
|_docker-compose.prod.yml
# from django.conf import settings
import core
from django.conf.urls.static import static
urlpatterns = [
path('', views.home, name='home'),
path('registration/', include('django.contrib.auth.urls')),
path('randomization_management/', include('randomization_management.urls')),
path('randomization_settings/', include('randomization_settings.urls')),
path('randomization/', include('randomization.urls')),
path('export/', include('export.urls')),
path('contact/', views.contact, name='contact'),
path('admin/', admin.site.urls),
] + static(core.settings.dev.STATIC_URL, document_root=core.settings.dev.STATIC_ROOT)
I try to "dockerize" a Django apps.
All works fine except static files that are not served.
I use Django web server (dev) so I should not have to "served" static files.
Neverthelsess, I run docker exec -it coverage_africa_web_1 python manage.py collectstatic command
and get the confirmation
You have requested to collect static files at the destination
location as specified in your settings:
/usr/src/app/static
This will overwrite existing files!
Are you sure you want to do this?
Type 'yes' to continue, or 'no' to cancel: yes
Found another file with the destination path 'randomization/js/script.js'. It will be ignored since only the first encountered file is collected. If this is not what you want, make sure every static file has a unique path.
140 static files copied to '/usr/src/app/static'.
settings.base.py
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, "static")
STATICFILES_DIRS = (
os.path.join(BASE_DIR,'randomization_management/static'),
os.path.join(BASE_DIR,'randomization_settings/static'),
os.path.join(BASE_DIR,'randomization/static'),
)
MEDIA_URL = "/media/"
MEDIA_ROOT = os.path.join(BASE_DIR, 'media/')
.env.dev
SECRET_KEY=*************************************
DEBUG=1
DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
SQL_ENGINE=django.db.backends.postgresql
SQL_DATABASE=db_dev
SQL_USER=user
SQL_PASSWORD=user
SQL_HOST=db
SQL_PORT=5432
DATABASE=postgres
DJANGO_SETTINGS_MODULE=core.settings.dev
docker-compose.yml
version: '3.7'
services:
web:
build: ./app
restart: always
command: python manage.py runserver 0.0.0.0:8000
volumes:
- ./app/:/usr/src/app
ports:
- 8000:8000
env_file:
- ./.env.dev
depends_on:
- db
db:
image: postgres:12.0-alpine
restart: always
volumes:
- postgres_data:/var/lib/postgres/data/
environment:
- POSTGRES_USER=user
- POSTGRES_PASSWORD=user
- POSTGRES_DB=db_dev
volumes:
postgres_data:
I see that you don't use a web server to serve the static files, so make sure that Django is serving the static files.
from django.conf import settings
from django.conf.urls.static import static
urlpatterns = [
# ... the rest of your URLconf goes here ...
] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
have a look to to Django documentation https://docs.djangoproject.com/en/3.1/howto/static-files/#serving-static-files-during-development
OK, missing settings
STATIC_URL = '/static/'
# STATIC_ROOT = os.path.join(BASE_DIR, "static") <- removed in dev
STATICFILES_DIRS = (
os.path.join(BASE_DIR, "static"), <- added in dev
os.path.join(BASE_DIR,'randomization_management/static'),
os.path.join(BASE_DIR,'randomization_settings/static'),
os.path.join(BASE_DIR,'randomization/static'),
)
My docker contianer django server, postgres, celery worker and celery beat. Running the application by itself there is no issue with connecting to the database however when the timed task to update the database runs I get an error
celery_1 | django.db.utils.OperationalError: could not connect to server: No such file or directory
celery_1 | Is the server running locally and accepting
celery_1 | connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
below are my files
docker-compose.yaml
version: "3"
services:
redis:
image: redis:alpine
db:
image: postgres:10-alpine
environment:
- POSTGRES_DB=app
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=supersecretpassword
ports:
- "5432:5432"
volumes:
- "db:/var/lib/postgresql/data"
pgadmin:
image: dpage/pgadmin4
depends_on:
- db
ports:
- "5555:80"
environment:
- PGADMIN_DEFAULT_EMAIL=dev#localhost
- PGADMIN_DEFAULT_PASSWORD=password
dj-dividend:
container_name: dividend
build: backend
command: sh -c "python manage.py wait_for_db && python manage.py makemigrations && python manage.py migrate && python manage.py runserver 0.0.0.0:80"
environment:
- DB_HOST=db
- DB_NAME=app
- DB_USER=postgres
- DB_PASS=supersecretpassword
volumes:
- ./backend:/code
ports:
- "80:80"
depends_on:
- db
- redis
celery:
build: backend
command: celery -A backend worker -l info
volumes:
- ./backend:/code
depends_on:
- db
- redis
celery-beat:
build: backend
command: celery -A backend beat -l info
volumes:
- ./backend:/code
depends_on:
- db
- redis
- celery
volumes:
db:
settings.py
import os
from celery.schedules import crontab
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/2.2/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = '1a0#6lsfxtfrt0n=#q1&up6hf$px-$ug^n%^v!n)^iwp_n*7c%'
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True
ALLOWED_HOSTS = ['*']
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'dividends',
'rest_framework',
'django_celery_beat',
'core'
]
CORS_ORIGIN_ALLOW_ALL = True
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'corsheaders.middleware.CorsMiddleware',
]
ROOT_URLCONF = 'backend.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
WSGI_APPLICATION = 'backend.wsgi.application'
# Database
# https://docs.djangoproject.com/en/2.2/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'HOST': os.environ.get('DB_HOST'),
'NAME': os.environ.get('DB_NAME'),
'USER': os.environ.get('DB_USER'),
'PASSWORD': os.environ.get('DB_PASS'),
}
}
# Password validation
# https://docs.djangoproject.com/en/2.2/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
# Internationalization
# https://docs.djangoproject.com/en/2.2/topics/i18n/
LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'UTC'
USE_I18N = True
USE_L10N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/2.2/howto/static-files/
STATIC_URL = '/static/'
#Celery
CELERY_BROKER_URL = "redis://redis:6379"
CELERY_RESULT_BACKEND = "redis://redis:6379"
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_BEAT_SCHEDULE = {
"get_stocks": {
'task': 'dividends.tasks.get_stocks',
'schedule': crontab(minute="*/1") #every minute
}
}
Dockerfile
FROM python:3.7
# Set environment variables
ENV PYTHONUNBUFFERED 1
COPY requirements.txt /
# Install dependencies.
RUN pip install -r /requirements.txt
# Set work directory.
RUN mkdir /code
WORKDIR /code
# Copy project code.
COPY . /code/
EXPOSE 80
tasks.py
from celery import shared_task
from .utils import get_stock_data
from .models import Stock
#shared_task(default_retry_delay=5, max_retries=3)
def get_stocks():
for stock in Stock.objects.all().values_list('ticker'):
try:
get_stock_data(stock[0])
except:
continue
celery.py
from celery import Celery
import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'backend.settings')
app = Celery('backend')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
you need to add environment variables of the postgresql to the celery-beat
In your database container you set :
environment:
- POSTGRES_DB=app
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=supersecretpassword
Then you need to copy and paste these lines in your celery_beat container so that it can connect to your database.
You'd then get:
celery-beat:
build: backend
command: celery -A backend beat -l info
volumes:
- ./backend:/code
depends_on:
- db
- redis
- celery
environment:
- POSTGRES_DB=app
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=supersecretpassword
I want to use cronjob to run a script, which is to fetch data from news api and feed it into postegres which is located in other container.
so the simplified architecture is
app(in container) -> postegres(in container)
the cronjob script is inside app, and it will fetch data and then send to postegres.
in my crontab is
* * * * * cd /tourMamaRoot/tourMama/cronjob && fetch_news.py >> /var/log/cron.log 2>&1
i can run it successfully by manually run the script, but when i put it in crontab , it shows the error.
File "/usr/local/lib/python3.6/dist-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/usr/local/lib/python3.6/dist-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/usr/local/lib/python3.6/dist-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
seems it only look for database locally if use crontab, how can i set it to put data into other container like i manually run the script?
Info:
my docker container for app is Ubuntu version 18.04 , and the following is my docker file for app
FROM ubuntu:18.04
MAINTAINER Eson
ENV PYTHONUNBUFFERED 1
ENV DEBIAN_FRONTEND=noninteractive
EXPOSE 8000
# Setup directory structure
RUN mkdir /tourMamaRoot
WORKDIR /tourMamaRoot/tourMama/
COPY tourMama/requirements/base.txt /tourMamaRoot/base.txt
COPY tourMama/requirements/dev.txt /tourMamaRoot/requirements.txt
# install Python 3
RUN apt-get update && apt-get install -y \
software-properties-common
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt-get update && apt-get install -y \
python3.7 \
python3-pip
RUN python3.7 -m pip install pip
RUN apt-get update && apt-get install -y \
python3-distutils \
python3-setuptools
# install Postgresql
RUN apt-get -y install wget ca-certificates
RUN wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
RUN sh -c echo deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main >> /etc/apt/sources.list.d/pgdg.list
RUN apt-get update
RUN apt-get install -y postgresql postgresql-contrib
# Install some dep
RUN apt-get install net-tools
RUN apt-get install -y libpq-dev python-dev
RUN pip3 install -r /tourMamaRoot/requirements.txt
# Copy application
COPY ./tourMama/ /tourMamaRoot/tourMama/
docker compose file:
version: '3'
services:
app:
build:
# current directory
# if for dev, need to have Dockerfile.dev in folder
dockerfile: docker/dev/Dockerfile
context: .
ports:
#host to image
- "8000:8000"
volumes:
# map directory to image, which means if something changed in
# current directory, it will automatically reflect on image,
# don't need to restart docker to get the changes into effect
- ./tourMama:/tourMamaRoot/tourMama
command: >
sh -c "python3 manage.py wait_for_db &&
python3 manage.py makemigrations &&
python3 manage.py migrate &&
python3 manage.py runserver 0.0.0.0:8000 &&
sh initial_all.sh"
environment:
- DB_HOST=db
- DB_NAME=app
- DB_USER=postgres
- DB_PASS=supersecretpassword
depends_on:
- db
- redis
db:
image: postgres:11-alpine
ports:
#host to image
- "5432:5432"
environment:
- POSTGRES_DB=app
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=supersecretpassword
redis:
image: redis:5.0.5-alpine
ports:
#host to image
- "6379:6379"
# command: ["redis-server", "--appendonly", "yes"]
# hostname: redis
# networks:
# - redis-net
# volumes:
# - redis-data:/data
and my cronjob script is:
import os
import sys
import django
from django.db import IntegrityError
from newsapi.newsapi_client import NewsApiClient
sys.path.append("../")
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "tourMama.settings")
django.setup()
from news.models import News
from tourMama_app.models import Category
from config.script import categorization_loader
load_category = categorization_loader.load_category_data("catagorization.yml")
categories = list(load_category.keys())
countries = ["us", "gb"]
# Init
newsapi = NewsApiClient(api_key='secret')
for category in categories:
for country in countries:
category_lower = category.lower()
category_obj = Category.objects.filter(
category=category,
).get()
top_headlines = newsapi.get_top_headlines(q='',
# sources=object'bbc-news,the-verge',
category=category_lower,
language='en',
page_size=100,
country=country
)
for article in top_headlines.get("articles"):
try:
News.objects.create(
source=article["source"].get("name") if article["source"] else None,
title=article.get("title"),
author=article.get("author"),
description=article.get("description"),
url=article.get("url"),
urlToImage=article.get("urlToImage"),
published_at=article.get("publishedAt"),
content=article.get("content"),
category=category_obj
)
except IntegrityError:
print("data already exist")
else:
print("data insert successfully")
and if needed, my django setting file is as following:
"""
Django settings for tourMama project.
Generated by 'django-admin startproject' using Django 2.2.1.
For more information on this file, see
https://docs.djangoproject.com/en/2.2/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/2.2/ref/settings/
"""
import os
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
TEMPLATE_DIR = os.path.join(BASE_DIR,"templates")
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/2.2/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'd084cm20*x*&s&w)vq+7*teea540yny+fyi^dh57nxiff&a#25'
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True
COMPRESS_ENABLED = False
COMPRESS_CSS_HASHING_METHOD = 'content'
COMPRESS_FILTERS = {
'css':[
'compressor.filters.css_default.CssAbsoluteFilter',
'compressor.filters.cssmin.rCSSMinFilter',
],
'js':[
'compressor.filters.jsmin.JSMinFilter',
]
}
HTML_MINIFY = False
KEEP_COMMENTS_ON_MINIFYING = False
ALLOWED_HOSTS = ['0.0.0.0', "127.0.0.1"]
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'channels',
'bootstrap3',
'tourMama_app',
'account',
'posts',
'group',
'news',
'statistics',
'compressor',
]
AUTH_USER_MODEL = "account.UserProfile"
MIDDLEWARE = [
'django.middleware.gzip.GZipMiddleware',
'htmlmin.middleware.HtmlMinifyMiddleware',
'htmlmin.middleware.MarkRequestMiddleware',
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
]
ROOT_URLCONF = 'tourMama.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [TEMPLATE_DIR,],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages' ,
],
},
},
]
WSGI_APPLICATION = 'tourMama.wsgi.application'
ASGI_APPLICATION = 'tourMama.routing.application'
# https://stackoverflow.com/questions/56480472/cannot-connect-to-redis-container-from-app-container/56480746#56480746
CHANNEL_LAYERS = {
'default': {
'BACKEND': 'channels_redis.core.RedisChannelLayer',
'CONFIG': {
"hosts": [('redis', 6379)],
},
},
}
# Database
# https://docs.djangoproject.com/en/2.2/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'HOST': os.environ.get('DB_HOST'),
'NAME': os.environ.get('DB_NAME'),
'USER': os.environ.get('DB_USER'),
'PASSWORD': os.environ.get('DB_PASS')
}
}
# Password validation
# https://docs.djangoproject.com/en/2.2/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
# other finders..
'compressor.finders.CompressorFinder',
)
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
'LOCATION': '127.0.0.1:11211',
}
}
# Internationalization
# https://docs.djangoproject.com/en/2.2/topics/i18n/
LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'UTC'
USE_I18N = True
USE_L10N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/2.2/howto/static-files/
STATIC_URL = '/static/'
STATICFILES_DIRS = [os.path.join(BASE_DIR, 'static'),]
STATIC_ROOT = os.path.join(BASE_DIR,"static_root")
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
LOGIN_REDIRECT_URL = "home:index"
LOGOUT_REDIRECT_URL = "home:index"
environment:
- DB_HOST=db
- DB_NAME=app
- DB_USER=postgres
- DB_PASS=supersecretpassword
I see that you are passing the environment variables via docker-compose like this. This is fine when the container is running the command directly inside the shell.
However, when putting it inside the crontab, the cronjob will run your command in a separate fresh shell with no environment being passed in at all.
To work around this problem, you can create a separate shell script:
cat <<EOF > /temp/script.sh
#!/bin/bash
export DB_HOST=db
export DB_NAME=app
export DB_USER=postgres
export DB_PASS=supersecretpassword
cd /tourMamaRoot/tourMama/cronjob && fetch_news.py >> /var/log/cron.log 2>&1
EOF
chmod +x /temp/script.sh
and edit your crontab like this:
* * * * * /temp/script.sh