Im wondering how to correctly use timesettings in django and celery.
Here is what I have:
TIME_ZONE = 'Europe/Oslo'
CELERY_TIMEZONE = 'Europe/Oslo'
CELERY_ENABLE_UTC = True
USE_TZ = True
TZINFO = 'UTC'
But the timestamp on my Celery task is ahead by two hours. How can I fix it?
Using:
Django - 1.6b2
celery - 3.0.23
django-celery - 3.0.23
You can use TZ default environment variable. Django will automatically use it with calling: http://docs.python.org/2/library/time.html#time.tzset
If your celery runs from django, it will work there too.
Also you could use something like:
os.environ['TZ'] = 'your timezone'
at the beginning of ( manage.py or wsgi.py ) in your local installation.
I think you might be hitting a bug in django-celery that I am also running into. There were timezone related changes in the last few releases of django-celery and this bug first showed up for me when I updated from 3.0.19 to 3.0.23.
I asked about this on the #celery IRC chat and was told that the django admin based celery task view is not that great and I should be using something like Flower (https://github.com/mher/flower) to monitor my tasks.
I installed and ran Flower and it did not suffer from the same timestamp issues that the django-celery admin based view does.
Related
I am tyring to schedule a task in Django using celery.Everything works fine when the CELERY_TIMEZONE='UTC' but doesnt work when I change the CELERY_TIMEZONE='Asia/Calcutta'.
#settings.py
CELERY_TIMEZONE='UTC'
CELERY_ENABLE_UTC = True
#tasks.py
#periodic_task(run_every=crontab(day_of_month="1-31", hour=6, minute=8), name="newtask1")
def elast():
print "test"
This works just fine but when I change my settings to
CELERY_TIMEZONE='Asia/Calcutta'
CELERY_ENABLE_UTC = False
#tasks.py
#periodic_task(run_every=crontab(day_of_month="1-31", hour=11, minute=38), name="newtask1")
def elast():
print "test"
This doesn't work.I can't seem to figure out the issue.Am I missing something? Any help would be appreciated.
Configure Celery to use a custom time zone. The timezone value can be any time zone supported by the pytz library.
kindly refer this celery reference guide
In trying to replace django-celery and upgrade celery to 4.x from an inherited project, I'm having hard time understanding the real changes to effect.
Celery is already setup as the project uses 3.x, however in removing djcelery from the app, I come across this:
CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend'
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
Reading the docs, I'm more confused about using result_backend or celery.backend.database or which:
CELERY_RESULT_BACKEND = 'celery.backends.database'
CELERYBEAT_SCHEDULER = 'beat_scheduler' OR
CELERY_RESULT_BACKEND: result_backend
CELERYBEAT_SCHEDULER: beat_scheduler
I'm new to Celery, still getting familiar with the details.
Celery 4 changed their settings as follows: http://docs.celeryproject.org/en/latest/userguide/configuration.html#new-lowercase-settings
The major difference between previous versions, apart from the lower
case names, are the renaming of some prefixes, like celerybeat_ to
beat_, celeryd_ to worker_, and most of the top level celery_ settings
have been moved into a new task_ prefix.
Celery will still be able to read old configuration files, so there’s
no rush in moving to the new settings format.
The expectation is that you use result_backend instead of CELERY_RESULT_BACKEND. Full mapping of old upper case settings to new ones are documented here: http://docs.celeryproject.org/en/latest/userguide/configuration.html#new-lowercase-settings
In other words, resut_backend is the new name of the key, NOT the new recommended value. It is the replacement for the left hand side of your assignment. These are equivalent:
CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend'
result_backend = 'djcelery.backends.database:DatabaseBackend'
Likewise these are equivalent:
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
beat_scheduler = 'djcelery.schedulers.DatabaseScheduler'
Running django-celery 3.1.16, Celery 3.1.17, Django 1.4.16. Trying to run some parallel tasks using 3 workers and collect the results using the following:
from celery import group
positions = []
jobs = group(celery_calculate_something.s(data.id) for data in a_very_big_list)
results = jobs.apply_async()
positions.extend(results.get())
The task celery_calculate_something returns an object to place the in the results list:
app.task(ignore_result=False)
def celery_calculate_something(id):
<do stuff>
No matter what I try, I always get the same result when calling get() on results:
No result backend configured. Please see the documentation for more information.
However, the results backend IS configured - I have many other tasks with ignore_result=False merrily adding to the tasks meta table in django_celery. It is something to do with using the results returned from group(). I should note it is not set explicitly in settings - it seems that django-celery has set it automatically for you.
I have the worker collecting events using:
manage.py celery worker -l info -E
and celerycam running with
python manage.py celerycam
Inspecting the results object returned (an instance of GroupResult) I can see that the backend attr is an instance of DisabledBackend. Is this the problem? What have I mis-understood?
You did not configure the results backend, so basically you need tables to store the results, since you have django-celery add it to INSTALLED_APPS in your settings.py file and then perform the migration (python manage.py migrate) After that open your celery.py file and modify your backend to djcelery.backends.database:DatabaseBackend. Here's an example
app = Celery('almanet',
broker='amqp://guest#localhost//',
backend='djcelery.backends.database:DatabaseBackend',
include=['alm_crm.tasks'] #References your tasks. Donc forget to put the whole absolute path.
)
After that you can import results from celery import result Now you can save the result and extract the result by job.id
from celery import group
positions = []
jobs = group(celery_calculate_something.s(data.id) for data in
a_very_big_list)
results = jobs.apply_async()
results.save()
some_task_result = result.GroupResult.restore(results.id)
print some_task_results.ready()
I'm trying to setup a Periodic Task that should expire after some time. I'm using Django 1.5.1, celery 3.0.19 and django-celery 3.0.17 (everything from pip).
This is the excerpt code to create the task:
from django.utils import timezone
from datetime import timedelta, datetime
from djcelery.models import PeriodicTask, IntervalSchedule
interval = IntervalSchedule.objects.get(pk=1) # Added through fixture - 3sec interval
expiration = timezone.now() + timedelta(seconds=10)
task = PeriodicTask(name='fill_%d' % profile.id,
task='fill_album',
args=[instance.id],
interval=interval,
expires=expiration) task.save()
And I'm running celery with ./manage.py celeryd -B
The task is being created just fine, and beat is running it every 3 seconds, but after 10 seconds it doesn't expire. At first I thought it was some timezone issue between django and celery, so I let it running for 3 hours (my difference to UTC) but it still wouldn't expire.
During my tests I've actually managed to make it expire once (and the logger kept repeating it was expired, every 3 seconds) but I haven't been able to reproduce it since.
Can anyone shed some light on what I'm doing wrong?
Thanks!
I'm having the same problem and I think celery beat is not honoring the expires. If you set a breakpoint in your task take a look at the current_task.request object and see if expires has a value (or just print current_task.request from within the task.)
For me, if I manually run the task, current_task.request.expires has a value, but if celery beat schedules it, it is None.
I'm using celery 3.1.11
I filed a bug: https://github.com/celery/celery/issues/2283
You can try use last_run_at as:
task = PeriodicTask(name='fill_%d' % profile.id,
task='fill_album',
args=[instance.id],
interval=interval,
expires=expiration,
last_run_at=expiration)
task.save()
I'm trying create a simple background periodic task using Django-Celery-RabbitMQ combination. I installed Django 1.3.1, I downloaded and setup djcelery. Here is how my settings.py file looks like:
BROKER_HOST = "127.0.0.1"
BROKER_PORT = 5672
BROKER_VHOST = "/"
BROKER_USER = "guest"
BROKER_PASSWORD = "guest"
....
import djcelery
djcelery.setup_loader()
...
INSTALLED_APPS = (
'djcelery',
)
And I put a 'tasks.py' file in my application folder with the following contents:
from celery.task import PeriodicTask
from celery.registry import tasks
from datetime import timedelta
from datetime import datetime
class MyTask(PeriodicTask):
run_every = timedelta(minutes=1)
def run(self, **kwargs):
self.get_logger().info("Time now: " + datetime.now())
print("Time now: " + datetime.now())
tasks.register(MyTask)
And then I start up my django server (local development instance):
python manage.py runserver
Then I start up the celerybeat process:
python manage.py celerybeat --logfile=<path_to_log_file> -l DEBUG
I can see entries like this in the log:
[2012-04-29 07:50:54,671: DEBUG/MainProcess] tasks.MyTask sent. id->72a5963c-6e15-4fc5-a078-dd26da663323
And I also can see the corresponding entries getting created in database, but I can't find where it is logging the text I specified in the actual run function in MyTask class.
I tried fiddling with the logging settings, tried using the django logger instead of celery logger, but of no use. I'm not even sure, my task is getting executed. If I print any debug information in the task, where does it go?
Also, this is first time I'm working with any type of message queuing system. It looks like the task will get executed as part of the celerybeat process - outside the django web framework. Will I still be able to access all the django models I created.
Thanks,
Venkat.
Celerybeat it stuff, which pushes task when it need, but not executing them. You tasks instances stored in RabbitMq server. You need to execute celeryd daemon for executing your tasks.
python manage.py celeryd --logfile=<path_to_log_file> -l DEBUG
Also if you using RabbitMq, I recommend to you to install special rabbitmq management plugins:
rabbitmq-plugins list
rabbitmq-enable rabbitmq_management
service rabbitmq-server restart
It will be available at http://:55672/ login: guest pass: guest. Here you can check how many tasks in your rabbit instance online.
You should check the RabbitMQ logs, since celery sends the tasks to RabbitMQ and it should execute them. So all the prints of the tasks should be in RabbitMQ logs.