I have a django app running on Apache2 and WSGI. I use PyCharm as my IDE for development.
I have the wsgi python module below, and added a print statement after application = get_wsgi_application(). When watching the logs this process takes about 1 second. What is confusing, is when this gets triggered. I have a page that sends a simple text output. I hit refresh a bunch of times, and this print gets written to the log once. If I wait a few seconds, it gets written on the next page request. If I refresh successively, it does not until I wait for a period again.
My call and response is about 10 milliseconds, but when this is executed (as verified by the print in the log) it takes about a second. This is adding a tremendous amount of unnecessary load to my server and slowing things down. I have it narrowed down to the apps.populate(settings.INSTALLED_APPS) that is called in the django.setup() method. Is there a way I can prevent this from running so often or make it run faster?
Thanks for any guidance, or advice you can offer to figure this out or prevent it.
wsgi.py:
import datetime
import os
import sys
from django.core.wsgi import get_wsgi_application
root_path = os.path.abspath(os.path.split(__file__)[0])
sys.path.insert(0, os.path.join(root_path, 'project_name'))
sys.path.insert(0, root_path)
path = '/var/www/project'
if path not in sys.path:
sys.path.append(path)
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'settings')
start = datetime.datetime.now()
application = get_wsgi_application()
print('Time to Populate: ' + str(datetime.datetime.now() - start))
settings.INSTALLED_APPS:
INSTALLED_APPS = (
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.admin',
'report_builder', # Third-party tool
'business', # Internal app
'slab', # Internal app
'file', # Internal app
'training_topic', # Internal app
'item', # Internal app
'person', # Internal app
'employee', # Internal app
'school', # Internal app
'training', # Internal app
'services', # Internal app
'incident', # Internal app
'report', # Internal app
'notice', # Internal app
'county_notification', # Internal app
'utilities.fax', # Internal app
'log', # Internal app
'helptext', # Internal app
'search', # Internal app
'compensation', # Internal app
'data_export', # Internal app
'record_review', # Internal app
)
/var/log/apache2/error.log:
[Tue Apr 25 14:07:22.917665 2017] [wsgi:error] [pid 21810] Time to Populate: 0:00:00.826958
[Tue Apr 25 14:07:34.715745 2017] [wsgi:error] [pid 21817] Time to Populate: 0:00:00.822580
I was using PyCharm IDE and it was saving the files in real time, each save would prompt a recompile and this log would show up. I closed it and reran the tests and everything is fine. It only logs this function when I reload Apache2.
Related
How do you specify environment variables for an Apache site?
I have a multi-tenant Django site being served by Apache+ModWSGI. Each site uses the same Django code, but their settings differ slightly. An environment variable tells Django which settings to load. Currently, I'm getting a separate wsgi_<site_name>.py file, containing the variable appropriate for each site, but this violate the DRY rule. I'd like to, instead, put this variable in the Apache site config, and use a single wsgi.py file for all sites.
I can't find much info on how to do this. I found this one old blog post suggesting I could use the Apache config syntax:
SetEnv SITE somename
along with a wsgi file like:
import os, sys
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
PROJECT_DIR = os.path.abspath(os.path.join(BASE_DIR, '..'))
sys.path.append(PROJECT_DIR)
os.environ['DJANGO_SETTINGS_MODULE'] = 'app.settings'
from django.core.wsgi import get_wsgi_application
_application = get_wsgi_application()
def application(environ, start_response):
os.environ['SITE'] = environ['SITE']
return _application(environ, start_response)
but this has no effect. If I print out the SITE variable, Django never receives the value "somename" and displays the default value.
As far as I can tell the mod_env component is still supported, and I've confirmed it's both installed and enabled on my server.
The problem was that I was calling get_wsgi_application() too early. The correct code is:
from django.core.wsgi import get_wsgi_application
def application(environ, start_response):
os.environ['SITE'] = environ['SITE']
_application = get_wsgi_application()
return _application(environ, start_response)
I have a queryset to list today's sales
from django.utils import timezone
class VentaToday(ListView):
queryset = Venta.objects.filter(fecha=timezone.now()).order_by('-id')
template_name = 'venta/venta_today.html'
In local, this works correctly but in production (Pythonanywhere) the sales of the previous day keep appearing. To fix it, I have to go to the pythonanywhere panel and click on the ** reload ** button to solve the problem.
I changed the server time:
Image of server time
Configuration of the django project:
LANGUAGE_CODE = 'es-pe'
TIME_ZONE = 'America/Lima'
USE_I18N = True
USE_L10N = True
USE_TZ = True
Is it a server cache problem? or something am I doing wrong?
UPDATE
config WSGI:
# +++++++++++ DJANGO +++++++++++
# To use your own django app use code like this:
import os
import sys
os.environ["TZ"] = "America/Lima"
#
## assuming your django settings file is at '/home/dnicosventas/mysite/mysite/settings.py'
## and your manage.py is is at '/home/dnicosventas/mysite/manage.py'
path = '/home/dnicosventas/dnicos-ventas'
if path not in sys.path:
sys.path.append(path)
#
os.environ['DJANGO_SETTINGS_MODULE'] = 'DnicosVentas.settings'
#
## then, for django >=1.5:
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
## or, for older django <=1.4
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
and my console:
export TZ="/usr/share/zoneinfo/America/Lima"
Even so, after 12 a.m., yesterday's sales keep appearing until I click on the reload button in the pythonanywhere panel.
Views.py:
class VentaToday(ListView):
today = datetime.now(pytz.timezone('America/Lima'))
queryset = Venta.objects.filter(fecha=today).order_by('-id')
template_name = 'venta/venta_today.html'
Image of the reload button
Solution by Giles Thomas:
class VentaToday(ListView):
template_name = 'venta/venta_today.html'
def get_queryset(self):
return Venta.objects.filter(fecha=datetime.now(pytz.timezone('America/Lima'))).order_by('-id')
TLDR: I had the same issue. I fixed it by changing TIME_ZONE='' to TIME_ZONE='UTC', in the settings.py file in project folder of pythonanywhere.
Python by default uses pytz.timezone(settings.TIME_ZONE), to initiate the time zone of the webapp, and since by default pythonanywhere doesnt initiate this variable, leaving it to the end user to do it, as per their requirements. So initiate your TIME_ZONE, as per your needs, which may do the trick.
You could also try looking in your project log files, for more information on this.
Found a few versions of this question, such as Django get environment variables from apache, however the advice I've found so far doesn't seem to work with the latest LTS django (1.11).
I have an apache configuration which holds a number of environment variables, not limited to connection credentials for the DB. Im using this to make my code portable between dev/prod etc.
My apache conf just uses SetEnv to pass in some variables.
I've tried two different styles of approach to use these variables, both seem to suffer from the same issue; it needs to read the settings file before we can write to the environment, and the settings file requires values from the environment.
My two variants are;
import os
import django
from django.core.handlers.wsgi import WSGIHandler
from django.core.wsgi import get_wsgi_application
_application = get_wsgi_application()
def application(environ, start_response):
for key in [keys...]:
if environ.get(key):
os.environ[key] = environ.get(key)
return _application(environ, start_response)
and
import os
import django
from django.core.handlers.wsgi import WSGIHandler
class WSGIEnvironment(WSGIHandler):
def __call__(self, environ, start_response):
for key in [keys...]:
if environ.has_key(key):
print "Key: %s = %s" % (key,environ[key])
os.environ[key] = environ[key]
return super(WSGIEnvironment, self).__call__(environ, start_response)
os.environ.setdefault("DJANGO_SETTINGS_MODULE", 'apiClient.settings')
django.setup(set_prefix=False)
application = WSGIEnvironment()
Either way im trying to use the values in settings as;
KEY = "thing"
if os.environ.has_key("KEY"):
KEY = os.environ["KEY"]
You can't use SetEnv as the settings file is evaluated before the first request is handled. Setting environment variables from per request WSGI environ values was always a bad idea and can cause problems, so you shouldn't do that anyway.
Result is that you cannot set environment variables from the Apache configuration file. Set them from the WSGI script file. If they are things that should not be added to a Git repository, create a file on the specific host with the values in some format, and have the WSGI script file read that file from the local host to set them when first loaded and before any Django code is executed.
For Django > 1.7 get_wsgi_application calls django.setup() which initializes the settings. So if your env vars aren't set at that point you won't see them in the settings.
To get around it, don't actually call get_wsgi_application until you're ready. This works for me in wsgi.py:
def application(environ, start_response):
os.environ['APPLICATION_ENV'] = environ.get('APPLICATION_ENV', None)
return get_wsgi_application()(environ, start_response)
You have to defer initialising the Django app until the first request. Something like this in your wsgi.py:
from django.core.wsgi import get_wsgi_application
_application = None
def application(environ, start_response):
global _application
if _application == None:
for key in environ:
if key.startswith('ENV_'):
os.environ[key[4:]] = environ[key]
_application = get_wsgi_application()
return _application(environ, start_response)
It's a pity that there appears to be no option in mod_wsgi to set the initial environment when starting the daemon process, as you can with mod_fastcgi.
From what I can tell, there are two documents describing how to set up celery. There's "Running the worker as a daemon" and there's "First steps with Django".
In the Django docs, it says:
We also add the Django settings module as a configuration source for Celery. This means that you don’t have to use multiple configuration files, and instead configure Celery directly from the Django settings.
Which sounds awesome. However, from what I can tell, these are the files that are needed for a complete Celery daemonization:
/etc/init.d/celeryd
/etc/defaults/celery
/my-proj/celery.py
/my-proj/__init__.py
And possibly:
/my-proj/settings.py
Boy that's a lot of files. I think I've got them all set up properly:
/etc/init.d/celeryd has the default init.d script provided by celery.
/etc/defaults/celery has almost nothing. Just a pointer to my app:
export DJANGO_SETTINGS_MODULE='cl.settings'
/my-proj/celery.py has the recommended file from the First Steps with Django:
from __future__ import absolute_import
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')
from django.conf import settings # noqa
app = Celery('proj')
# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
/my-proj/__init__.py has the recommended code from the First Steps with Django:
from __future__ import absolute_import
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app
And I have all the celery-related settings like the following in my settings.py file:
CELERY_BIN = '/var/www/.virtualenvs/my-env/bin/celery'
CELERYD_USER = 'www-data'
CELERYD_GROUP = 'www-data'
CELERYD_CONCURRENCY = 20
BROKER_URL = 'redis://'
BROKER_POOL_LIMIT = 30
Yet, when I start celery using sudo service celeryd start, it doesn't work. Instead, it's clear that it hasn't picked up my settings from my Django project, because it says:
Nov 05 20:51:59 pounamu celeryd[30190]: celery init v10.1.
Nov 05 20:51:59 pounamu celeryd[30190]: Using config script: /etc/default/celeryd
Nov 05 20:51:59 pounamu celeryd[30190]: No passwd entry for user 'celery'
Nov 05 20:51:59 pounamu su[30206]: No passwd entry for user 'celery'
Nov 05 20:51:59 pounamu su[30206]: FAILED su for celery by root
Nov 05 20:51:59 pounamu su[30206]: - ??? root:celery
Nov 05 20:51:59 pounamu systemd[1]: celeryd.service: control process exited, code=exited status=1
Any ideas where the bailing wire isn't working? Am I missing something major?
You are attempting to run celery as the system user "celery" which is the default used by the init script. You should create this user or you can override this by setting CELERYD_USER in /etc/defaults/celery.
Personally I prefer to use supervisord to manage celery.
You are missing the CELERY_APP setting. Set it in the configuration file /etc/defaults/celery or as a parameter for the worker command:
celery worker -A my-proj
Otherwise, Celery has no idea it should look at /my-proj/celery.py. The django environment variable does not affect what Celery loads.
Trying to deploy a Django 1.4 project using mod_wsgi and virtualenv, i'm running into a 500. The Apache error_log reveals:
mod_wsgi (pid=30452): Exception occurred processing WSGI script '/path/to/project/site-packages/projectapp/wsgi.py'.
[...] Traceback (most recent call last):
[...] File "/path/to/project/env/myenv/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 219, in __call__
[...] self.load_middleware()
[...] File "/path/to/project/env/myenv/lib/python2.6/site-packages/django/core/handlers/base.py", line 47, in load_middleware
[...] raise exceptions.ImproperlyConfigured('Error importing middleware %s: "%s"' % (mw_module, e))
[...] ImproperlyConfigured: Error importing middleware projectapp.content.middleware: "cannot import name SomeModel"
From the error message i would expect that this is some kind of a path issue. However, when the offending middleware is removed from the Django settings, the site launches just fine, and there are other modules loaded from projectapp.content, SomeModel is also loaded in this case, otherwise the whole site wouldn't be able to run.
The import error raised doesn't come directly from the middleware, as it doesn't import the model. SomeModel is defined in a speparate app which is actually checked out into the src directory of the virtualenv. But the directory containing this app is also in the path.
The wsgi.py file i'm using:
import os
import sys
sys.stdout = sys.stderr
sys.path.insert(0, '/path/to/project/env/myenv/lib/python2.6/site-packages/')
# The module inside the following directory
# defines SomeModel from the error message
sys.path.insert(0, '/path/to/project/env/myenv/src/some-app/')
sys.path.insert(0, '/path/to/project/site-packages/')
import django.core.handlers.wsgi
os.environ['DJANGO_SETTINGS_MODULE'] = 'projectapp.settings'
application = django.core.handlers.wsgi.WSGIHandler()
Printing sys.path after inserting the module paths shows everything in the expected order, /path/to/project/site-packages/ is listed first and /path/to/project/env/myenv/src/some-app/ (which defines SomeModel) second.
I've also tried a different variant, based on the example from the mod_wsgi docs:
import os
import sys
import site
ALLDIRS = [
'/path/to/project/site-packages/',
'/path/to/project/env/myenv/lib/python2.6/site-packages/',
'/path/to/project/env/myenv/src/some-app/',
]
# Remember original sys.path
prev_sys_path = list(sys.path)
sys.stdout = sys.stderr
# Add each new site-packages directory
for directory in ALLDIRS:
site.addsitedir(directory)
# Reorder sys.path so new directories are at the front
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_path
#activate_this = '/path/to/project/env/myenv/bin/activate_this.py'
#execfile(activate_this, dict(__file__=activate_this))
import django.core.handlers.wsgi
os.environ['DJANGO_SETTINGS_MODULE'] = 'projectapp.settings'
application = django.core.handlers.wsgi.WSGIHandler()
The error logged by Apache is exactly the same.
I have to add that the middleware is loaded just fine when i'm running the development server on the same machine, so i have no idea what's going wrong. Is there a way to get a better traceback from mod_wsgi?
EDIT:
Fogot to mention that i'm using mod_wsgi in daemon mode. Here are the relevant parts from my vhost config:
<VirtualHost x.x.x.x:80>
WSGIDaemonProcess foo user=foo threads=10 umask=0002
WSGIProcessGroup foo
WSGIScriptAlias / /path/to/project/site-packages/projectapp/wsgi.py
</VirtualHost>
Okay after hours of debugging this turned out to be a race condition caused by the middleware. The middleware i'm using is similiar to the FlatpageFallbackMiddleware from Django contrib and actually the import of the view caused the problem.
from projectapp.content.views import generic_content_detail
class ContentFallbackMiddleware(object):
def process_response(self, request, response):
[...]
Moving the import statement inside the process_response method solved the problem for me:
class ContentFallbackMiddleware(object):
def process_response(self, request, response):
from projectapp.content.views import generic_content_detail
[...]