Would greatly appreciate input from anyone with experience configuring redis as a backend for a celery-brokered django project on heroku. My task scheduling worked fine from localhost but I'm finding it really frustrating getting it deployed on heroku:
At the moment I'm running 3 dynos, 1 web, 1 scheduler and 1 worker
I added the redistogo addon to my project. Redistogo set to the free nano plan, which gives me 10 connections, 1 DB and a 5MB size instance
I followed the redistogo documentation (https://devcenter.heroku.com/articles/redistogo#install-redis-in-python) for configuring settings.py and, alternatively, also tried implementing a variation of the solution here. Neither working for me. Here's what I have in settings.py:
redis_url = os.environ.get('REDISTOGO_URL', 'http://localhost:6959')
CACHES = {
'default': {
'BACKEND': 'redis_cache.RedisCache',,
'LOCATION': '%s:%s' % (redis_url.hostname, redis_url.port),
'OPTIONS': {
'DB': 0, # or 1?
'PASSWORD': redis_url.password,
#'PARSER_CLASS': 'redis.connection.HiredisParser'
},
},
}
CELERY_RESULT_BACKEND = redis_url
BROKER_URL = 'redis://localhost:6959/0'
Here's my heroku logs when I try to run the app:
2013-07-11T12:16:10.998516+00:00 app[web.1]: apps = settings.INSTALLED_APPS
2013-07-11T12:16:10.998516+00:00 app[web.1]: mod = importlib.import_module(self.SETTINGS_MODULE)
2013-07-11T12:16:10.998263+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
2013-07-11T12:16:10.998263+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/__init__.py", line 263, in fetch_command
2013-07-11T12:16:10.998516+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/conf/__init__.py", line 53, in __getattr__
2013-07-11T12:16:10.998516+00:00 app[web.1]: self._setup(name)
2013-07-11T12:16:10.998516+00:00 app[web.1]: self._wrapped = Settings(settings_module)
2013-07-11T12:16:10.998516+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/conf/__init__.py", line 132, in __init__
2013-07-11T12:16:10.998516+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
2013-07-11T12:16:10.998516+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/conf/__init__.py", line 48, in _setup
2013-07-11T12:16:10.998712+00:00 app[web.1]: 'LOCATION': '%s:%s' % (redis_url.hostname, redis_url.port),
2013-07-11T12:16:10.998712+00:00 app[web.1]: AttributeError: 'str' object has no attribute 'hostname'
2013-07-11T12:16:12.201202+00:00 heroku[web.1]: Process exited with status 1
2013-07-11T12:16:12.250743+00:00 heroku[web.1]: State changed from starting to crashed
how do I get redis_url treated like a URI and not a str?
my procfile:
web: python manage.py run_gunicorn -b 0.0.0.0:$PORT -w 3 --log-level info
scheduler: python manage.py celeryd -B -E
worker: python manage.py celeryd -E -B --loglevel=INFO
In requirements I have django-redis-cache==0.10.0, redis==2.7.6, django-celery==3.0.17, celery celery==3.0.20 and kombu==2.5.12
Use the python urlparse library. It parses URLs into components.
redis_url = os.environ.get('REDISTOGO_URL', 'http://localhost:6959')
redis_url = urlparse.urlparse(redis_url)
Looks like os.environ.get is returning a String (or str? not to familiar with python) and you're expecting it to be more like a URI object or something. Do normal python strings respond to methods like hostname?
The documentation also has this step:
redis = redis.from_url(redis_url)
which according to these docs that parses the string into a redis object.
Related
Good Afternoon, I am developing a Django App and trying to develop it using gunicorn and nginx as web server loadbalancer for http. when trying to start gunicorn from within an venv (the app is being developed within a venv )using
../bin/gunicorn biblio_cfv.wsgi: application
I get these errors
[2016-10-14 16:37:13 -0400] [22618] [INFO] Starting gunicorn 19.6.0
[2016-10-14 16:37:13 -0400] [22618] [INFO] Listening at: http://127.0.0.1:8000 (22618)
[2016-10-14 16:37:13 -0400] [22618] [INFO] Using worker: sync
[2016-10-14 16:37:13 -0400] [22621] [INFO] Booting worker with pid: 22621
[2016-10-14 20:37:22 +0000] [22621] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/home/eddyraz/proyectos/www/VPS_WEB_DEVEL/lib/python3.5/site-packages/gunicorn/arbiter.py", line 557, in spawn_worker
worker.init_process()
File "/home/eddyraz/proyectos/www/VPS_WEB_DEVEL/lib/python3.5/site-packages/gunicorn/workers/base.py", line 126, in init_process
self.load_wsgi()
File "/home/eddyraz/proyectos/www/VPS_WEB_DEVEL/lib/python3.5/site-packages/gunicorn/workers/base.py", line 136, in load_wsgi
self.wsgi = self.app.wsgi()
File "/home/eddyraz/proyectos/www/VPS_WEB_DEVEL/lib/python3.5/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/home/eddyraz/proyectos/www/VPS_WEB_DEVEL/lib/python3.5/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/home/eddyraz/proyectos/www/VPS_WEB_DEVEL/lib/python3.5/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/home/eddyraz/proyectos/www/VPS_WEB_DEVEL/lib/python3.5/site-packages/gunicorn/util.py", line 368, in import_app
app = eval(obj, mod.__dict__)
File "<string>", line 0
^
SyntaxError: unexpected EOF while parsing
I checked the file util.py withon gunicorn code and found nothin wrong
here is the function of the code that gives the error.
def import_app(module):
parts = module.split(":", 1)
if len(parts) == 1:
module, obj = module, "application"
else:
module, obj = parts[0], parts[1]
try:
__import__(module)
except ImportError:
if module.endswith(".py") and os.path.exists(module):
msg = "Failed to find application, did you mean '%s:%s'?"
raise ImportError(msg % (module.rsplit(".", 1)[0], obj))
else:
raise
mod = sys.modules[module]
try:
app = eval(obj, mod.__dict__)
except NameError:
raise AppImportError("Failed to find application: %r" % module)
if app is None:
raise AppImportError("Failed to find application object: %r" % obj)
if not callable(app):
raise AppImportError("Application object must be callable.")
return app
Please could anyone shed any light on this. Thanks in advance.
I know this was asked a long time ago, but I resolved mine just now.
Just 'pip install gunicorn' in that virtual environment
I have created a Django application but now have plans to use some asynchronous (real-time) functionality in some areas of the site. After doing some research I think I should use gevent-socketio and therefore it is required I switch the application server to Gunicorn.
I have fallen at the first hurdle of deploying Gunicorn, I have installed with the command sudo apt-get install gunicorn and try to run my application with gunicorn project.wsgi:application but it fails and produces the following error:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 473, in spawn_worker
worker.init_process()
File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 100, in init_process
self.wsgi = self.app.wsgi()
File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 115, in wsgi
self.callable = self.load()
File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 33, in load
return util.import_app(self.app_uri)
File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 362, in import_app
__import__(module)
File "/home/alex/django_projects/fantasymatchday_1/fantasymatchday_1/wsgi.py", line 13, in <module>
from django.core.wsgi import get_wsgi_application
ImportError: No module named django.core.wsgi
2014-11-20 17:31:45 [6605] [INFO] Worker exiting (pid: 6605)
2014-11-20 17:31:45 [6600] [INFO] Shutting down: Master
2014-11-20 17:31:45 [6600] [INFO] Reason: Worker failed to boot.
Can anybody give me a clue to what I need to do from here?
I am using python 3.4.0 and Django 1.6
You have to install django and gunicorn in the same environment.
If you use virtualenv make sure you have both in the same virtual environment.
$ python manage.py run_gunicorn 0.0.0.0:80 --settings=project_name.settings.production
<- It's run, OK.
but,
$ gunicorn_django -b 0.0.0.0:80 project_name/settings/production.py
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 473, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 100, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 106, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/djangoapp.py", line 102, in load
return mod.make_wsgi_application()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/django_wsgi.py", line 36, in make_wsgi_application
if get_validation_errors(s):
File "/usr/local/lib/python2.7/dist-packages/django/core/management/validation.py", line 35, in get_validation_errors
for (app_name, error) in get_app_errors().items():
File "/usr/local/lib/python2.7/dist-packages/django/db/models/loading.py", line 166, in get_app_errors
self._populate()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/loading.py", line 72, in _populate
self.load_app(app_name, True)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/loading.py", line 94, in load_app
app_module = import_module(app_name)
File "/usr/local/lib/python2.7/dist-packages/django/utils/importlib.py", line 35, in import_module
__import__(name)
> ImportError: No module named accounts
2013-06-29 01:56:53 [30859] [INFO] Worker exiting (pid: 30859)
2013-06-29 01:56:53 [30854] [INFO] Shutting down: Master
2013-06-29 01:56:53 [30854] [INFO] Reason: Worker failed to boot.
How can I solve this problem?
I want to run on daemon.
Thanks.
This looks like an issue related to your PYTHONPATH. Try adding the path that contains the accounts module to your python path, using gunicorn's --pythonpath.
If you have newer gunicorn/Django versions using gunicorn via management command should be the preffered way - and it should be more safe regarding issues like this.
Try running your django app using gunicorn instead of django_gunicorn:
$ gunicorn mydjangoproject.wsgi:application
There is also an alternative solution to fix the gunicorn_django script, see here. I ran into the issue when upgrading from gunicorn 18.0 to 19.1.1. Cheers!
Here is my gonicorn.conf, but I can't seem to get the command right. The command Im trying is working when I type it in the command line, but not as an Supervisor command.
[program:gunicorn]
directory = /home/USER/.virtualenvs/SITE/myApp/
command=/home/USER/.virtualenvs/SITE/bin/python /home/USER/.virtualenvs/SITE/myApp/manage.py run_gunicorn myApp.wsgi:application -c /home/USER/.virtualenvs/SITE/myApp/gunicorn.conf.py
user = USER
autostart = true
autorestart = true
redirect_stderr = true
stdout_logfile = /home/USER/logs/gunicorn.log
stderr_logfile = /home/USER/logs/gunicorn_err.log
What is the correct command for gunicorn?
Here is the error Im getting when I try to start gunicorn:
/home/USERNAME/.virtualenvs/SITE/local/lib/python2.7/site-packages/sorl/thumbnail/helpers.py:5: DeprecationWarning: django.utils.simplejson is deprecated; use json instead.
from django.utils import simplejson
2013-02-25 13:46:55 [3975] [INFO] Starting gunicorn 0.17.2
2013-02-25 13:46:55 [3975] [INFO] Listening at: http://127.0.0.1:8000 (3975)
2013-02-25 13:46:55 [3975] [INFO] Using worker: sync
2013-02-25 13:46:55 [3988] [INFO] Booting worker with pid: 3988
2013-02-25 13:46:55 [3988] [ERROR] Exception in worker process:
Traceback (most recent call last):
File "/home/USERNAME/.virtualenvs/SITE/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 485, in spawn_worker
worker.init_process()
File "/home/USERNAME/.virtualenvs/SITE/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 100, in init_process
self.wsgi = self.app.wsgi()
File "/home/USERNAME/.virtualenvs/SITE/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 103, in wsgi
self.callable = self.load()
File "/home/USERNAME/.virtualenvs/SITE/local/lib/python2.7/site-packages/gunicorn/app/djangoapp.py", line 133, in load
return mod.make_command_wsgi_application(self.admin_media_path)
File "/home/USERNAME/.virtualenvs/SITE/local/lib/python2.7/site-packages/gunicorn/app/django_wsgi.py", line 113, in make_command_wsgi_application
reload_django_settings()
File "/home/USERNAME/.virtualenvs/SITE/local/lib/python2.7/site-packages/gunicorn/app/django_wsgi.py", line 109, in reload_django_settings
logging_config_func(settings.LOGGING)
File "/usr/lib/python2.7/logging/config.py", line 777, in dictConfig
dictConfigClass(config).configure()
File "/usr/lib/python2.7/logging/config.py", line 503, in configure
raise ValueError("dictionary doesn't specify a version")
ValueError: dictionary doesn't specify a version
Traceback (most recent call last):
File "/home/USERNAME/.virtualenvs/SITE/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 485, in spawn_worker
worker.init_process()
File "/home/USERNAME/.virtualenvs/SITE/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 100, in init_process
self.wsgi = self.app.wsgi()
File "/home/USERNAME/.virtualenvs/SITE/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 103, in wsgi
self.callable = self.load()
File "/home/USERNAME/.virtualenvs/SITE/local/lib/python2.7/site-packages/gunicorn/app/djangoapp.py", line 133, in load
return mod.make_command_wsgi_application(self.admin_media_path)
File "/home/USERNAME/.virtualenvs/SITE/local/lib/python2.7/site-packages/gunicorn/app/django_wsgi.py", line 113, in make_command_wsgi_application
reload_django_settings()
File "/home/USERNAME/.virtualenvs/SITE/local/lib/python2.7/site-packages/gunicorn/app/django_wsgi.py", line 109, in reload_django_settings
logging_config_func(settings.LOGGING)
File "/usr/lib/python2.7/logging/config.py", line 777, in dictConfig
dictConfigClass(config).configure()
File "/usr/lib/python2.7/logging/config.py", line 503, in configure
raise ValueError("dictionary doesn't specify a version")
ValueError: dictionary doesn't specify a version
That specific error can be fixed by adding this variable to your settings file. I ran into this when upgrading a django project from Django 1.3 to Django 1.5.
(See https://docs.djangoproject.com/en/dev/topics/logging/ for more info)
LOGGING = {
'version': 1,
'disable_existing_loggers': True,
}
[program:projectname_gunicorn_live]
command=/home/myusername/projectname-live/bin/django run_gunicorn -b 127.0.0.1:1559
directory=/home/myusername/projectname-live/
user=myusername
autostart=True
autorestart=True
redirect_stderr=True
Link: https://gist.github.com/oaksakal/1143612
Another Link: Gunicorn Supervisor Startup Error
I followed the heroku documentation to install a django app and at first it worked fine. After a day I pushed some changes to the server. After that, I was not able to access the app at all: FATAL: password authentication failed for user "drjstoymyqyarj"
I can not even sync the db anymore:
$ heroku run python manage.py syncdb
Running `python manage.py syncdb` attached to terminal... up, run.1
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
utility.execute()
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
self.execute(*args, **options.__dict__)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
output = self.handle(*args, **options)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/base.py", line 371, in handle
return self.handle_noargs(**options)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/commands/syncdb.py", line 57, in handle_noargs
cursor = connection.cursor()
File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/backends/__init__.py", line 306, in cursor
cursor = self.make_debug_cursor(self._cursor())
File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 177, in _cursor
self.connection = Database.connect(**conn_params)
File "/app/.heroku/venv/lib/python2.7/site-packages/psycopg2/__init__.py", line 179, in connect
connection_factory=connection_factory, async=async)
psycopg2.OperationalError: FATAL: password authentication failed for user "drjstoymyqyarj"
FATAL: password authentication failed for user "drjstoymyqyarj"
I have used the database settings recommended in the heroku doc:
import dj_database_url
DATABASES = {'default': dj_database_url.config(default='postgres://localhost')}
When I check the logs after pushing code to the server, there is a suspicious Process exited with status 143 that I did not notice before. Maybe that has something to do with it?
$ heroku logs
heroku[web.1]: State changed from up to starting
heroku[web.1]: Stopping all processes with SIGTERM
heroku[web.1]: Starting process with command `python ./manage.py runserver 0.0.0.0:41048 --noreload`
app[web.1]: Validating models...
app[web.1]:
app[web.1]: 0 errors found
app[web.1]: Django version 1.4, using settings 'ClosetList.settings'
app[web.1]: Development server is running at http://0.0.0.0:41048/
app[web.1]: Quit the server with CONTROL-C.
heroku[web.1]: Process exited with status 143
heroku[web.1]: State changed from starting to up
[Edit]
Same error msg with heroku pg:psql. I am however able to open a Django shell with heroku run python manage.py shell, but I can not access any data from within it (same error of course).
[/Edit]
any help with this is appreciated.
Run
heroku config
and see if there are more than 2 heroku dbs configured and also see if DATABASE_URL point to your configured DB. If not, then you can promote your db to default DATABASE_URL by running:
heroku pg:promote HEROKU_POSTGRESQL_GREEN
Where HEROKU_POSTGRESQL_GREEN is your database name, you will see HEROKU_POSTGRESQL_GREEN_URL in your config.
Once your configured database is promoted to default database, you are ready to go.