I'm trying to add Celery to django to schedule tasks. I use Redis backend, and connect via unix socket. Setup was working until I have tried using password auth to redis.conf:
My settings.py:
CELERY_BROKER_URL = 'redis+socket:///home/username/domain/redis.sock?password=mypasswd'
CELERY_RESULT_BACKEND = 'redis+socket:///home/username/domain/redis.sock?password=mypasswd'
celery.py:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'public_python.settings')
# celery settings for the demo_project
app = Celery('public_python')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
Result:
[2021-07-19 21:22:14,003: ERROR/MainProcess] consumer: Cannot connect to redis+socket:///home/username/domain/redis.sock: Authentication required..
I have already tried adding:
CELERY_REDIS_PASSWORD='mypasswd'
(any any concievable combination of similar words), with no luck.
The normal format using a network based socket would look like this:
redis://:mypasswd#127.0.0.1:6379/0
For file based socket, you might want to try:
redis+socket://:mypasswd#/home/username/domain/redis.sock?virtual_host=0
Related
I can not run the celery worker + redis + django. If I run this command to check that celery worker is ready to receive tasks:
celery -A car_rental worker -l info
I got this error:
[2020-02-24 00:14:42,188: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**#127.0.0.1:5672//: [Errno 111] Connection refused.
Trying again in 2.00 seconds...
In my settings.py I have this:
BROKER_URL = 'redis://localhost:6379'
requirements.txt:
amqp==2.5.2, asgiref==3.2.3, billiard==3.6.2.0, celery==4.4.0, redis==3.4.1
celery.py:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'car_rental.settings')
app = Celery('car_rental')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
car_rental/init.py:
from __future__ import absolute_import, unicode_literals
from .celery import app as celery_app
__all__ = ('celery_app',)
and the structure of my project is like this:
car_rental
/car_rental
__init__.py
celery.py
setting.py
What I didn't understand is that I am using in the broker_url = 'redis://localhost:6379' but in the error I have: Cannot connect to amqp://guest:**#127.0.0.1:5672//
In this case it should work if you change the parameter from BROKER_URL to CELERY_BROKER_URL. When you gave it the namespace here:
app.config_from_object('django.conf:settings', namespace='CELERY')
At that point you'll want to rename your BROKER_URL parameter to CELERY_BROKER_URL.
CELERY_BROKER_URL = 'redis://localhost:6379'
Another example:
app.config_from_object('django.conf:settings', namespace='CAR')
CAR_BROKER_URL = 'redis://localhost:6379'
I had configured the celery exactly as Dantheman91 explained, and still, I faced the same issue.
Instead of having one settings.py in my project, I was using, base.py, development.py, production.py, and test.py inside the module named settings.
So, in case anyone switching between these roles/instances do not forget to change the same in celery.py.
In similar case scenario, use:
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj_name.settings.development')
It should be the same as you use in the manage.py of your project.
i'm trying found how celery is working. i have a project that have about 10 app.now i want use celery .
setting.py:
CELERY_BROKER_URL = 'amqp://rabbitmq:rabbitmq#localhost:5672/rabbitmq_vhost'
CELERY_RESULT_BACKEND = 'redis://localhost'
i created a user in rabbitmq with this info:username: rabbitq and password:rabbitmq . then i create a vhost with name rabbitmq_vhost and add rabbitmq permission to it. all is fine i think because all of error about rabbitmq disappear .
here is my test.py:
from .task import when_task_expiration
def test_celery():
result = when_task_expiration.apply_async((2, 2), countdown=3)
print(result.get())
task.py:
from __future__ import absolute_import, unicode_literals
import logging
from celery import shared_task
from proj.celery import app
#app.task
def when_task_expiration(task, x):
print(task.id, 'task done')
return True
celery.py:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')
app = Celery('proj')
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
now when i call test_celery() in python shell it's pending.i try to replace #shared_task and #app.task(bind=True) but noting changed.even i try use .delay() instead apply_async((2, 2), countdown=3) and again nothing happend.
i'm trying to use celery to call a function in specific time during this quesation that i ask in past.thank you.
You most likely forgot to run at least one Celery worker process. To do so, execute the following in the shell: celery worker -A proj.celery -c 4 -l DEBUG (here I assumed your Celery application is defined in proj/celery.py as you have Celery('proj') in there)
I'm trying to create my first Celery task. The task will send the same e-mail every one minute to the same person.
According to the documentation, I create my first task in my project.
from __future__ import absolute_import, unicode_literals
from celery import shared_task
from django.core.mail import send_mail
#shared_task
def send_message():
to = ['test#test.com', ]
send_mail('TEST TOPIC',
'TEST MESSAGE',
'test#test.com',
to)
Then, in my project's ja folder, I add the celery.py file, which looks like this:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from django.conf import settings
from celery.schedules import crontab
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'app_rama.settings')
app = Celery('app_rama')
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks(settings.INSTALLED_APPS)
app.conf.beat_schedule = {
'send-message-every-single-minute': {
'task': 'app.tasks.send_message',
'schedule': crontab(), # change to `crontab(minute=0, hour=0)` if you want it to run daily at midnight
},
}
Then in the __int__.py file of my project I added:
from __future__ import absolute_import, unicode_literals
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app
__all__ = ('celery_app',)
And the last thing I try to do is run the command:
celery -A app_rama worker -l info
And then I receive the following error:
[2019-06-27 16:01:26,750: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**#127.0.0.1:5672//: [WinError 10061]
I tried many solutions from the forum, but I did not find the correct one.
I was also not helped by adding the following settings to my settings.py file:
CELERY_BROKER_URL = 'amqp://guest:guest#localhost:5672//'
How can I solve this error so that my task works in the background of the application?
Your Celery broker is probably misconfigured. Read the "Using RabbitMQ" document to find out how to setup RabbitMQ properly (I assumed you want to use RabbitMQ as you had "amqp" protocol in your example).
I recommend learning Celery with Redis, as it is easier to setup and manage. Then once you learn the basics you may decide to move to RabbitMQ or some other supported broker...
Also, verify that your RabbitMQ server is running properly. If you use Windows, make sure some software on it does not prevent user processes to connect to the localhost:5672.
I have Django 2.0 project that is working fine, its integrated with Celery 4.1.0, I am using jquery to send ajax request to the backend but I just realized its loading endlessly due to some issues with celery.
Celery Settings (celery.py)
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'converter.settings')
app = Celery('converter', backend='amqp', broker='amqp://guest#localhost//')
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
Celery Tasks (tasks.py)
from __future__ import absolute_import, unicode_literals
from celery import shared_task
#shared_task(time_limit=300)
def add(number1, number2):
return number1 + number2
Django View (views.py)
class AddAjaxView(JSONResponseMixin, AjaxResponseMixin, View):
def post_ajax(self, request, *args, **kwargs):
url = request.POST.get('number', '')
task = tasks.convert.delay(url, client_ip)
result = AsyncResult(task.id)
data = {
'result': result.get(),
'is_ready': True,
}
if result.successful():
return self.render_json_response(data, status=200)
When I send ajax request to the Django app it is loading endlessly but when terminate Django server, and I run celery -A demoproject worker --loglevel=info that's when my tasks are running.
Question
How do I automate this so that when I run Django project my celery tasks will work automatically when I send ajax request?
If you are on development environment, you have to run manually celery worker as it does not run automatically on the background, in order to process the jobs in the queue. So if you want to have a flawless workflow, you need both Django default server and celery worker running. As stated in the documentation:
In a production environment you’ll want to run the worker in the background as a daemon - see Daemonization - but for testing and development it is useful to be able to start a worker instance by using the celery worker manage command, much as you’d use Django’s manage.py runserver:
celery -A proj worker -l info
You can read their documentation for daemonization.
http://docs.celeryproject.org/en/latest/userguide/daemonizing.html
I've got a django app that's trying to call a celery task that will eventually be executed on some remote hosts. The task codebase is completely separate to the django project, so I'm using celery.execute.send_task and calling it from a post_delete model signal. The code looks a bit like this:
class MyModel(models.Model):
#staticmethod
def do_async_thing(sender, instance, **kwargs):
celery.execute.send_task("tasks.do_my_thing", args=[instance.name])
signals.post_delete.connect(MyModel.do_async_thing, sender=MyModel)
I'm using the latest Django (1.6.1) and celery 3.1.7, so I understand that I don't need any extra module or app in my django project for it to be able to talk to celery. I've set BROKER_URL inside my settings.py to be the right url amqp://user:password#host/vhost.
When this method fires, I get a Connection Refused error. There's no indication on the celery broker that any connection was attempted - I guess it's not seeing the BROKER_URL configuration and is trying to connect to localhost.
How do I make this work? What extra configuration does send_task need to know where the broker is?
So I discovered the answer, and it was to do with not reading the tutorial (http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html) closely enough.
Specifically, I had the correct celery.py in place which I would have thought should have loaded the settings, but I'd missed the necessary changes to __init__.py in the django project, which wasn't hooking everything together.
My celery.py should be:
from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')
app = Celery('mypoject')
# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
and the __init__.py should be simply:
from __future__ import absolute_import
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app