Celery on Django not working - django

Sending emails with Celery works fine on production server.
Trying to use it on local dev (VM) and does not work.
I get this when restart:
Starting web server apache2 [ OK ]
Starting message broker rabbitmq-server [ OK ]
Starting celery task worker server celeryd [ OK ]
Starting celeryev...
: No such file or directory
Also I get this error in console when running the page:
error: [Errno 104] Connection reset by peer
Production setting:
import djcelery
djcelery.setup_loader()
BROKER_HOST = "127.0.0.1"
BROKER_PORT = 5672 # default RabbitMQ listening port
BROKER_USER = "vs_user"
BROKER_PASSWORD = "user01"
BROKER_VHOST = "vs_vhost"
CELERY_BACKEND = "amqp" # telling Celery to report the results back to RabbitMQ
CELERY_RESULT_DBURI = ""
When i ran:
sudo rabbitmqctl list_vhosts
I get this:
Listing vhosts ...
/
...done.
What i need to change in this setting to run it successfully on local VM?
UPDATE
vhost and user were definitely missing so I ran suggested commands.
They executed ok but still it does not work ,same error.
It must be one more thing that prevents it from working and celeryev is suspect.
This is what i get when stopping and starting server:
Stopping web server apache2 ... waiting . [ OK ]
Stopping message broker rabbitmq-server [ OK ]
Stopping celery task worker server celeryd start-stop-daemon: warning: failed to kill 28006: No such process
[ OK ]
Stopping celeryev...NOT RUNNING
Starting web server apache2 [ OK ]
Starting message broker rabbitmq-server [ OK ]
Starting celery task worker server celeryd [ OK ]
Starting celeryev...
: No such file or directory
Traceback (most recent call last):
File "/webapps/target/forums/json_views.py", line 497, in _send_forum_notifications
post_master_json.delay('ForumNotificationEmail', email_params)
File "/usr/local/lib/python2.6/dist-packages/celery-3.0.25-py2.6.egg/celery/app/task.py", line 357, in delay
return self.apply_async(args, kwargs)
File "/usr/local/lib/python2.6/dist-packages/celery-3.0.25-py2.6.egg/celery/app/task.py", line 474, in apply_async
**options)
File "/usr/local/lib/python2.6/dist-packages/celery-3.0.25-py2.6.egg/celery/app/amqp.py", line 250, in publish_task
**kwargs
File "/usr/local/lib/python2.6/dist-packages/kombu-2.5.16-py2.6.egg/kombu/messaging.py", line 164, in publish
routing_key, mandatory, immediate, exchange, declare)
File "/usr/local/lib/python2.6/dist-packages/kombu-2.5.16-py2.6.egg/kombu/connection.py", line 470, in _ensured
interval_max)
File "/usr/local/lib/python2.6/dist-packages/kombu-2.5.16-py2.6.egg/kombu/connection.py", line 396, in ensure_connection
interval_start, interval_step, interval_max, callback)
File "/usr/local/lib/python2.6/dist-packages/kombu-2.5.16-py2.6.egg/kombu/utils/__init__.py", line 217, in retry_over_time
return fun(*args, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/kombu-2.5.16-py2.6.egg/kombu/connection.py", line 246, in connect
return self.connection
File "/usr/local/lib/python2.6/dist-packages/kombu-2.5.16-py2.6.egg/kombu/connection.py", line 761, in connection
self._connection = self._establish_connection()
File "/usr/local/lib/python2.6/dist-packages/kombu-2.5.16-py2.6.egg/kombu/connection.py", line 720, in _establish_connection
conn = self.transport.establish_connection()
File "/usr/local/lib/python2.6/dist-packages/kombu-2.5.16-py2.6.egg/kombu/transport/pyamqp.py", line 115, in establish_connection
conn = self.Connection(**opts)
File "/usr/local/lib/python2.6/dist-packages/amqp-1.0.13-py2.6.egg/amqp/connection.py", line 136, in __init__
self.transport = create_transport(host, connect_timeout, ssl)
File "/usr/local/lib/python2.6/dist-packages/amqp-1.0.13-py2.6.egg/amqp/transport.py", line 264, in create_transport
return TCPTransport(host, connect_timeout)
File "/usr/local/lib/python2.6/dist-packages/amqp-1.0.13-py2.6.egg/amqp/transport.py", line 99, in __init__
raise socket.error(last_err)
error: timed out
I ran manage.py celeryev and got console showing workers and tasks.Everything is empty and only getting Connection Error: error(timeout('timed out',),) repeatedly.

It looks like you don't have the virtual host you specified set up on your local RabbitMQ server.
You would first need to add the virtual host.
sudo rabbitmqctl add_vhost vs_vhost
Next you need to add the permissions for your user.
sudo rabbitmqctl set_permissions -p vs_vhost vs_user ".*" ".*" ".*"
Also, make sure that you actually have a user set up, otherwise you can add one using this command.
sudo rabbitmqctl add_user vs_user user01

Related

celery worker crash when i start another

i use django with celery and redis.
I would like to have three queues und three workers.
My celery settings in the settings.py looks like this:
CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Europe/Berlin'
# CELERY QUEUES SETUP
CELERY_DEFAULT_QUEUE = 'default'
CELERY_DEFAULT_ROUTING_KEY = 'default'
CELERY_TASK_QUEUES = (
Queue('default', Exchange('default'), routing_key='default'),
Queue('manually_crawl', Exchange('manually_crawl'), routing_key='manually_crawl'),
Queue('periodically_crawl', Exchange('periodically_crawl'), routing_key='periodically_crawl'),
)
CELERY_ROUTES = {
'api.tasks.crawl_manually': {'queue': 'manually_crawl', 'routing_key': 'manually_crawl',},
'api.tasks.crawl_periodically': {'queue': 'periodically_crawl', 'routing_key': 'periodically_crawl',},
'api.tasks.crawl_firsttime': {'queue': 'default', 'routing_key': 'default',},
}
Later i will start the workers with celery multi, but in the development phase i would like to start the worker manually to see errors or so.
I start the redis server with redis-server and than i start the first worker default with:
celery -A proj worker -Q default -l debug -n default_worker
If i try to start the next worker in a new terminal with:
celery -A proj worker -Q manually_crawl -l debug -n manually_crawl
I get an error in the first default worker terminal:
[2019-10-28 09:32:58,284: INFO/MainProcess] sync with celery#manually_crawl
[2019-10-28 09:32:58,290: ERROR/MainProcess] Control command error: OperationalError("\nCannot route message for exchange 'reply.celery.pidbox': Table empty or key no longer exists.\nProbably the key ('_kombu.binding.reply.celery.pidbox') has been removed from the Redis database.\n")
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/kombu/connection.py", line 439, in _reraise_as_library_errors
yield
File "/usr/local/lib/python3.7/dist-packages/kombu/connection.py", line 518, in _ensured
return fun(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/kombu/messaging.py", line 203, in _publish
mandatory=mandatory, immediate=immediate,
File "/usr/local/lib/python3.7/dist-packages/kombu/transport/virtual/base.py", line 605, in basic_publish
message, exchange, routing_key, **kwargs
File "/usr/local/lib/python3.7/dist-packages/kombu/transport/virtual/exchange.py", line 70, in deliver
for queue in _lookup(exchange, routing_key):
File "/usr/local/lib/python3.7/dist-packages/kombu/transport/redis.py", line 877, in _lookup
exchange, redis_key))
kombu.exceptions.InconsistencyError:
Cannot route message for exchange 'reply.celery.pidbox': Table empty or key no longer exists.
Probably the key ('_kombu.binding.reply.celery.pidbox') has been removed from the Redis database.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/celery/worker/pidbox.py", line 46, in on_message
self.node.handle_message(body, message)
File "/usr/local/lib/python3.7/dist-packages/kombu/pidbox.py", line 145, in handle_message
return self.dispatch(**body)
File "/usr/local/lib/python3.7/dist-packages/kombu/pidbox.py", line 115, in dispatch
ticket=ticket)
File "/usr/local/lib/python3.7/dist-packages/kombu/pidbox.py", line 151, in reply
serializer=self.mailbox.serializer)
File "/usr/local/lib/python3.7/dist-packages/kombu/pidbox.py", line 285, in _publish_reply
**opts
File "/usr/local/lib/python3.7/dist-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/usr/local/lib/python3.7/dist-packages/kombu/connection.py", line 551, in _ensured
errback and errback(exc, 0)
File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.7/dist-packages/kombu/connection.py", line 444, in _reraise_as_library_errors
sys.exc_info()[2])
File "/usr/local/lib/python3.7/dist-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.7/dist-packages/kombu/connection.py", line 439, in _reraise_as_library_errors
yield
File "/usr/local/lib/python3.7/dist-packages/kombu/connection.py", line 518, in _ensured
return fun(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/kombu/messaging.py", line 203, in _publish
mandatory=mandatory, immediate=immediate,
File "/usr/local/lib/python3.7/dist-packages/kombu/transport/virtual/base.py", line 605, in basic_publish
message, exchange, routing_key, **kwargs
File "/usr/local/lib/python3.7/dist-packages/kombu/transport/virtual/exchange.py", line 70, in deliver
for queue in _lookup(exchange, routing_key):
File "/usr/local/lib/python3.7/dist-packages/kombu/transport/redis.py", line 877, in _lookup
exchange, redis_key))
kombu.exceptions.OperationalError:
Cannot route message for exchange 'reply.celery.pidbox': Table empty or key no longer exists.
Probably the key ('_kombu.binding.reply.celery.pidbox') has been removed from the Redis database.
Why?
there is a problem currently with the kombu library, according to this post by downgrading to 4.6.4 and for some people 4.6.3 it solves the problem
jorijinnall commented 11 days ago
Had the same issue.
I fixed by downgrading kombu from 4.6.5 to 4.6.3
I still had the bug in version 4.6.4
link github
You can start multiple workers like shown below:
$ celery -A proj worker -l info --concurrency=4 -n wkr1#hostname
$ celery -A proj worker -l info --concurrency=2 -n wkr2#hostname
$ celery -A proj worker -l info --concurrency=2 -n wkr3#hostname
In the above example, there are three workers which will be able to spawn 4,2,2 child processes. It is normally advised to run a single worker per machine and the concurrency value will define how many processes will run in parallel.
The default number of those processes is equal to a number of cores on that machine, normally.
I hope this could help you.

Cannot route message for exchange 'reply.celery.pidbox': Table empty or key no longer exists

I'm trying to spawn a few background, celery beat processes using docker-compose, but they are not working anymore. My configuration:
docker-compose-dev.yml
worker-periodic:
image: dev3_web
restart: always
volumes:
- ./services/web:/usr/src/app
- ./services/web/celery_logs:/usr/src/app/celery_logs
command: celery beat -A celery_worker.celery --schedule=/tmp/celerybeat-schedule --loglevel=DEBUG --pidfile=/tmp/celerybeat.pid
environment:
- CELERY_BROKER=redis://redis:6379/0
- CELERY_RESULT_BACKEND=redis://redis:6379/0
- FLASK_ENV=development
- APP_SETTINGS=project.config.DevelopmentConfig
- DATABASE_URL=postgres://postgres:postgres#web-db:5432/web_dev
- DATABASE_TEST_URL=postgres://postgres:postgres#web-db:5432/web_test
- SECRET_KEY=my_precious
depends_on:
- web
- redis
- web-db
links:
- redis:redis
- web-db:web-db
after I up containers, I $ docker ps and get (note how worker-periodic_1 is always up a few seconds before):
697322a621d5 dev3_web "celery worker -A ce…" 24 hours ago Up 5 minutes dev3_worker-analysis_1
d8e414aa4e5b dev3_web "celery worker -A ce…" 24 hours ago Up 5 minutes dev3_worker-learning_1
ae327266132c dev3_web "flower -A celery_wo…" 24 hours ago Up 5 minutes 0.0.0.0:5555->5555/tcp dev3_monitor_1
3ccb79e01412 dev3_web "celery beat -A cele…" 24 hours ago Up 14 seconds dev3_worker-periodic_1
a50e1276f692 dev3_web "celery worker -A ce…" 24 hours ago Up 5 minutes dev3_worker-scraping_1
All celery workers are working when endpoints are called, except when it is a celery beat, periodically automated, process. When I up containers, my logs complain at celery_logs/worker_analysis.log:
[2019-11-16 23:29:20,880: DEBUG/MainProcess] pidbox received method hello(from_node='celery#d8e414aa4e5b', revoked={}) [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': '85f4128f-2f75-3996-8375-2a19aa58d5d4'} ticket:0daa0dc4-fa09-438d-9003-ccbd39f259dd]
[2019-11-16 23:29:20,907: INFO/MainProcess] sync with celery#d8e414aa4e5b
[2019-11-16 23:29:21,018: ERROR/MainProcess] Control command error: OperationalError("\nCannot route message for exchange 'reply.celery.pidbox': Table empty or key no longer exists.\nProbably the key ('_kombu.binding.reply.celery.pidbox') has been removed from the Redis database.\n",)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 439, in _reraise_as_library_errors
yield
File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 518, in _ensured
return fun(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/kombu/messaging.py", line 203, in _publish
mandatory=mandatory, immediate=immediate,
File "/usr/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 605, in basic_publish
message, exchange, routing_key, **kwargs
File "/usr/lib/python3.6/site-packages/kombu/transport/virtual/exchange.py", line 70, in deliver
for queue in _lookup(exchange, routing_key):
File "/usr/lib/python3.6/site-packages/kombu/transport/redis.py", line 877, in _lookup
exchange, redis_key))
kombu.exceptions.InconsistencyError:
Cannot route message for exchange 'reply.celery.pidbox': Table empty or key no longer exists.
Probably the key ('_kombu.binding.reply.celery.pidbox') has been removed from the Redis database.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/celery/worker/pidbox.py", line 46, in on_message
self.node.handle_message(body, message)
File "/usr/lib/python3.6/site-packages/kombu/pidbox.py", line 145, in handle_message
return self.dispatch(**body)
File "/usr/lib/python3.6/site-packages/kombu/pidbox.py", line 115, in dispatch
ticket=ticket)
File "/usr/lib/python3.6/site-packages/kombu/pidbox.py", line 151, in reply
serializer=self.mailbox.serializer)
File "/usr/lib/python3.6/site-packages/kombu/pidbox.py", line 285, in _publish_reply
**opts
File "/usr/lib/python3.6/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 551, in _ensured
errback and errback(exc, 0)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 444, in _reraise_as_library_errors
sys.exc_info()[2])
File "/usr/lib/python3.6/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 439, in _reraise_as_library_errors
yield
File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 518, in _ensured
return fun(*args, **kwargs)
at celer
File "/usr/lib/python3.6/site-packages/kombu/messaging.py", line 203, in _publish
mandatory=mandatory, immediate=immediate,
File "/usr/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 605, in basic_publish
message, exchange, routing_key, **kwargs
File "/usr/lib/python3.6/site-packages/kombu/transport/virtual/exchange.py", line 70, in deliver
for queue in _lookup(exchange, routing_key):
File "/usr/lib/python3.6/site-packages/kombu/transport/redis.py", line 877, in _lookup
exchange, redis_key))
kombu.exceptions.OperationalError:
Cannot route message for exchange 'reply.celery.pidbox': Table empty or key no longer exists.
Probably the key ('_kombu.binding.reply.celery.pidbox') has been removed from the Redis database.
this is how celery is configured:
web/project/config.py:
class DevelopmentConfig(BaseConfig):
# CELERY
INSTALLED_APPS = ['routes']
# celery config
CELERYD_CONCURRENCY = 4
# Add a one-minute timeout to all Celery tasks.
CELERYD_TASK_SOFT_TIME_LIMIT = 60
CELERY_ENABLE_UTC = False
CELERY_TIMEZONE = 'America/Sao_Paulo'
CELERY_BROKER_URL = os.environ.get('CELERY_BROKER')
CELERY_RESULT_BACKEND = os.environ.get('CELERY_RESULT_BACKEND')
CELERY_IMPORTS = ('project.api.routes.background',)
# periodic tasks
CELERYBEAT_SCHEDULE = {
'playlist_generator_with_audio_features': {
'task': 'project.api.routes.background.playlist_generator_with_audio_features',
# Every minute
'schedule': crontab(minute=59),
'args' : [('user_id'),]
},
'cache_user_tracks_with_analysis': {
'task': 'project.api.routes.background.cache_user_tracks_with_analysis',
# Every hour
'schedule': crontab(minute=0, hour='*/1'),
'args' : ('user_id','token')
},
}
this is an example task at project/api/routes/background.py, at my Flask server:
#celery.task(queue='analysis', default_retry_delay=30, max_retries=3, soft_time_limit=1000)
def cache_user_tracks_with_analysis(user_id, token):
# business logic
return {'Status': 'Task completed!',
'features': results}
In my requirements.txt:, kombu is not declared explicitly, and I have:
celery==4.2.1
redis==3.2.0
what am I missing?
This is an open celery/kombu issue: https://github.com/celery/kombu/issues/1063
explicitly downgrading to kombu==4.5.0 fixed the error for me.

Websocket error using Django Channels with EC2

I'm trying to build an app in Django that uses Django Channels. I'm deploying on Elastic Beanstalk. My understanding is that the application load balancer supports websockets and I can route websocket traffic to the appropriate port. The websockets work on my localhost:8000. I'm using the free tier Redis Labs for my channel layer.
I followed this tutorial.
https://blog.mangoforbreakfast.com/2017/02/13/django-channels-on-aws-elastic-beanstalk-using-an-alb/#comment-43
Following this tutorial it appears I can get Daphne and the workers running on the correct ports. I SSHed into the EC2 instance to get the following output.
$ sudo /usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf status
Daphne RUNNING pid 4240, uptime 0:05:51
Worker:Worker_00 RUNNING pid 4242, uptime 0:05:51
Worker:Worker_01 RUNNING pid 4243, uptime 0:05:51
Worker:Worker_02 RUNNING pid 4244, uptime 0:05:51
Worker:Worker_03 RUNNING pid 4245, uptime 0:05:51
httpd RUNNING pid 4248, uptime 0:05:51
My daphne.out.log and workers.out.log look fine.
$ cat daphne.out.log
2017-07-20 21:41:43,693 INFO Starting server at tcp:port=5000:interface=0.0.0.0, channel layer mysite.asgi:channel_layer.
2017-07-20 21:41:43,693 INFO HTTP/2 support not enabled (install the http2 and tls Twisted extras)
2017-07-20 21:41:43,694 INFO Using busy-loop synchronous mode on channel layer
2017-07-20 21:41:43,694 INFO Listening on endpoint tcp:port=5000:interface=0.0.0.0
$ cat workers.out.log
2017-07-20 21:41:44,114 - INFO - runworker - Using single-threaded worker.
2017-07-20 21:41:44,120 - INFO - runworker - Using single-threaded worker.
2017-07-20 21:41:44,121 - INFO - runworker - Running worker against channel layer default (asgi_redis.core.RedisChannelLayer)
2017-07-20 21:41:44,121 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
2017-07-20 21:41:44,126 - INFO - runworker - Using single-threaded worker.
2017-07-20 21:41:44,126 - INFO - runworker - Running worker against channel layer default (asgi_redis.core.RedisChannelLayer)
2017-07-20 21:41:44,126 - INFO - runworker - Running worker against channel layer default (asgi_redis.core.RedisChannelLayer)
2017-07-20 21:41:44,127 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
2017-07-20 21:41:44,127 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
2017-07-20 21:41:44,133 - INFO - runworker - Using single-threaded worker.
2017-07-20 21:41:44,136 - INFO - runworker - Running worker against channel layer default (asgi_redis.core.RedisChannelLayer)
2017-07-20 21:41:44,136 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
From here I updated the security group settings and configured the load balancer and tested my webpage. Instead of /ws/ as the path for rerouting, I used /table/* because those are the pages that need websockets.
If I use /ws/ I get this error.
WebSocket connection to 'ws://example.com/table/1/' failed: Error during WebSocket handshake: Unexpected response code: 404
If I use /table/* I get this error.
WebSocket connection to 'ws://example.com/table/1/' failed: Error during WebSocket handshake: Unexpected response code: 504
When I changed the security group settings of my load balancer to also allow TCP at port 5000, the worker log changes. With the /ws/ path rule in my load balancer, I now get this in the worker log:
...
2017-07-20 21:41:44,136 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
Not Found: /ws/
Not Found: /ws/
Not Found: /ws/
Not Found: /ws/
Not Found: /ws/
Not Found: /ws/
If I use the /table/* path, I get a very long error in my log.
2017-07-21 01:22:05,270 - ERROR - worker - Error processing message with consumer deal.consumers.ws_connect:
Traceback (most recent call last):
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/sessions/backends/base.py", line 193, in _get_session
return self._session_cache
AttributeError: 'SessionStore' object has no attribute '_session_cache'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
self.connect()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 171, in connect
self.connection = self.get_new_connection(conn_params)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/postgresql/base.py", line 175, in get_new_connection
connection = Database.connect(**conn_params)
File "/opt/python/run/venv/local/lib64/python3.4/site-packages/psycopg2/__init__.py", line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/python/run/venv/local/lib/python3.4/site-packages/channels/worker.py", line 119, in run
consumer(message, **kwargs)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/channels/sessions.py", line 220, in inner
result = func(message, *args, **kwargs)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/channels/auth.py", line 71, in inner
message.user = auth.get_user(fake_request)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/auth/__init__.py", line 167, in get_user
user_id = _get_user_session_key(request)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/auth/__init__.py", line 59, in _get_user_session_key
return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/sessions/backends/base.py", line 48, in __getitem__
return self._session[key]
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/sessions/backends/base.py", line 198, in _get_session
self._session_cache = self.load()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/sessions/backends/db.py", line 33, in load
expire_date__gt=timezone.now()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/manager.py", line 122, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/query.py", line 381, in get
num = len(clone)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/query.py", line 240, in __len__
self._fetch_all()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/query.py", line 1074, in _fetch_all
self._result_cache = list(self.iterator())
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/query.py", line 52, in __iter__
results = compiler.execute_sql()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 846, in execute_sql
cursor = self.connection.cursor()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 231, in cursor
cursor = self.make_debug_cursor(self._cursor())
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 204, in _cursor
self.ensure_connection()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
self.connect()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/utils.py", line 95, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
self.connect()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 171, in connect
self.connection = self.get_new_connection(conn_params)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/postgresql/base.py", line 175, in get_new_connection
connection = Database.connect(**conn_params)
File "/opt/python/run/venv/local/lib64/python3.4/site-packages/psycopg2/__init__.py", line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
Unsurprisingly when I try to use the websockets I get this error:
WebSocket is already in CLOSING or CLOSED state.
Any idea what I'm doing wrong?
Ok so problem was when Daphne runs it doesn't see the environment variables. I added the following code to my supervisord.conf file and it worked:
[program:Daphne]
environment=PATH="/opt/python/run/venv/bin"
**command=/bin/bash -c "source /opt/python/current/env && /opt/python/run/venv/bin/daphne -b 0.0.0.0 -p 5000 mysite.asgi:channel_layer"**
directory=/opt/python/current/app
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/tmp/daphne.out.log
[program:Worker]
environment=PATH="/opt/python/run/venv/bin"
**command=/bin/bash -c "source /opt/python/current/env && python manage.py runworker"**
directory=/opt/python/current/app
process_name=%(program_name)s_%(process_num)02d
numprocs=4
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/tmp/workers.out.log

Django + Vagrant VM suddenly says Errno 111: connection refused

I'm sure this is my fault, but my system has been working for months and now does not work. I've looked in Django docs and here on stackoverflow, and I understand the issue that 'connection refused' means nobody is listening on the socket; but I don't understand why, and don't know how to go forward.
I use Vagrant + VirtualBox on a MacPro, and for six months I have been able to create virtual machines and run my app with no problems. I have some VM's where everything works as expected.
On my newly-created Vagrant VM's, I install and configure my django apps and run them using 'python manage.py runserver --settings=mumble.settings.py 0.0.0.0:8000'. Note that this is Django 1.8.3, but again,it works fine on some VM's.
For mysterious reasons, when I create a new vagrant VM and install my django payload (using deploy scripts that haven't changed), and connect to the django server, I get Errno 111 Connection refused:
+ python ./manage.py runserver --settings=mydjango.settings 0.0.0.0:8000
.....
_d^^^^^^^^^b_
.d'' ``b.
.p' `q.
.d' `b.
.d' `b. * Mezzanine 4.0.1
:: :: * Django 1.8.3
:: M E Z Z A N I N E :: * Python 2.7.12
:: :: * MySQL 5.7.18
`p. .q' * Linux 4.4.0-75-generic
`p. .q'
`b. .d'
`q.. ..p'
^q........p^
''''
Performing system checks...
System check identified no issues (0 silenced).
May 31, 2017 - 09:50:50
Django version 1.8.3, using settings 'mydjango.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.
Traceback (most recent call last):
File "/usr/lib/python2.7/wsgiref/handlers.py", line 85, in run
self.result = application(self.environ, self.start_response)
File "/vagrant/repos_here/www.mydjango.org/venv/www/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 189, in __call__
response = self.get_response(request)
File "/vagrant/repos_here/www.mydjango.org/venv/www/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 204, in get_response
'request': request
File "/usr/lib/python2.7/logging/__init__.py", line 1185, in error
Log 'msg % args' with severity 'ERROR'.
File "/usr/lib/python2.7/logging/__init__.py", line 1193, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python2.7/logging/__init__.py", line 1286, in _log
self.handle(record)
File "/usr/lib/python2.7/logging/__init__.py", line 1296, in handle
self.callHandlers(record)
File "/usr/lib/python2.7/logging/__init__.py", line 1336, in callHandlers
hdlr.handle(record)
File "/usr/lib/python2.7/logging/__init__.py", line 759, in handle
self.emit(record)
File "/vagrant/repos_here/www.mydjango.org/venv/www/local/lib/python2.7/site-packages/django/utils/log.py", line 129, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/vagrant/repos_here/www.mydjango.org/venv/www/local/lib/python2.7/site-packages/django/utils/log.py", line 132, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/vagrant/repos_here/www.mydjango.org/venv/www/local/lib/python2.7/site-packages/django/core/mail/__init__.py", line 98, in mail_admins
mail.send(fail_silently=fail_silently)
File "/vagrant/repos_here/www.mydjango.org/venv/www/local/lib/python2.7/site-packages/django/core/mail/message.py", line 303, in send
return self.get_connection(fail_silently).send_messages([self])
File "/vagrant/repos_here/www.mydjango.org/venv/www/local/lib/python2.7/site-packages/django/core/mail/backends/smtp.py", line 100, in send_messages
new_conn_created = self.open()
File "/vagrant/repos_here/www.mydjango.org/venv/www/local/lib/python2.7/site-packages/django/core/mail/backends/smtp.py", line 58, in open
self.connection = connection_class(self.host, self.port, **connection_params)
File "/usr/lib/python2.7/smtplib.py", line 256, in __init__
(code, msg) = self.connect(host, port)
File "/usr/lib/python2.7/smtplib.py", line 316, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/usr/lib/python2.7/smtplib.py", line 291, in _get_socket
return socket.create_connection((host, port), timeout)
File "/usr/lib/python2.7/socket.py", line 575, in create_connection
raise err
error: [Errno 111] Connection refused
[31/May/2017 09:50:55]"GET /donate/ HTTP/1.1" 500 59
Things I know:
'python ./manage.py' with no arguments will run fine, and shows the help message.
All forms of addressing the web server will fail:
curl http://localhost:8000
curl http://127.0.0.1:8000
curl http://192.168.33.10:8000
from a browser: http://192.168.33.10:8000
nginx is serving on port 80, no problem
my database is a copy of the production database, and it looks fine through 'mysql'
And here is a weird thing: I brought down the Django tutorial, used the 'runserver' command, and it worked!
[vagrant][~/tmp/django_tutorial/mysite]
$ python manage.py runserver 0.0.0.0:8000
Performing system checks...
System check identified no issues (0 silenced).
May 30, 2017 - 20:36:44
Django version 1.8.3, using settings 'mysite.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.
[30/May/2017 20:37:05]"GET / HTTP/1.1" 200 1767
Any suggestions on how to debug this?
The answer has nothing to do with my environment. My co-worker added a name to settings.ADMINS, which had been an empty list before that. When ADMINS is a non-empty list, any error triggers an attempt to send email to all of the email addresses in ADMINS, and my VM was not configured to send email. This is documented but not emphasized. The problem did not happen to my coworker, because that associated changes did not trigger a server error.

"password authentication failed for user" after push to Heroku

I followed the heroku documentation to install a django app and at first it worked fine. After a day I pushed some changes to the server. After that, I was not able to access the app at all: FATAL: password authentication failed for user "drjstoymyqyarj"
I can not even sync the db anymore:
$ heroku run python manage.py syncdb
Running `python manage.py syncdb` attached to terminal... up, run.1
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
utility.execute()
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
self.execute(*args, **options.__dict__)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
output = self.handle(*args, **options)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/base.py", line 371, in handle
return self.handle_noargs(**options)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/commands/syncdb.py", line 57, in handle_noargs
cursor = connection.cursor()
File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/backends/__init__.py", line 306, in cursor
cursor = self.make_debug_cursor(self._cursor())
File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 177, in _cursor
self.connection = Database.connect(**conn_params)
File "/app/.heroku/venv/lib/python2.7/site-packages/psycopg2/__init__.py", line 179, in connect
connection_factory=connection_factory, async=async)
psycopg2.OperationalError: FATAL: password authentication failed for user "drjstoymyqyarj"
FATAL: password authentication failed for user "drjstoymyqyarj"
I have used the database settings recommended in the heroku doc:
import dj_database_url
DATABASES = {'default': dj_database_url.config(default='postgres://localhost')}
When I check the logs after pushing code to the server, there is a suspicious Process exited with status 143 that I did not notice before. Maybe that has something to do with it?
$ heroku logs
heroku[web.1]: State changed from up to starting
heroku[web.1]: Stopping all processes with SIGTERM
heroku[web.1]: Starting process with command `python ./manage.py runserver 0.0.0.0:41048 --noreload`
app[web.1]: Validating models...
app[web.1]:
app[web.1]: 0 errors found
app[web.1]: Django version 1.4, using settings 'ClosetList.settings'
app[web.1]: Development server is running at http://0.0.0.0:41048/
app[web.1]: Quit the server with CONTROL-C.
heroku[web.1]: Process exited with status 143
heroku[web.1]: State changed from starting to up
[Edit]
Same error msg with heroku pg:psql. I am however able to open a Django shell with heroku run python manage.py shell, but I can not access any data from within it (same error of course).
[/Edit]
any help with this is appreciated.
Run
heroku config
and see if there are more than 2 heroku dbs configured and also see if DATABASE_URL point to your configured DB. If not, then you can promote your db to default DATABASE_URL by running:
heroku pg:promote HEROKU_POSTGRESQL_GREEN
Where HEROKU_POSTGRESQL_GREEN is your database name, you will see HEROKU_POSTGRESQL_GREEN_URL in your config.
Once your configured database is promoted to default database, you are ready to go.