i work on a celery beat task within a django project which creates Database entries periodically. I know so beacuse when i set the task up like this :
celery.py:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from celery.schedules import crontab
app = Celery("clock-backend", broker=os.environ.get("RABBITMQ_URL"))
app.config_from_object("django.conf:settings", namespace="CELERY")
app.conf.beat_schedule = {
'create_reports_monthly': {
'task': 'project_celery.tasks.create_reports_monthly',
'schedule': 10.0,
},
}
app.autodiscover_tasks()
And start my project it really creates an object every 10 seconds.
But what i really want to do is to set it up to run every first day of a month.
To do so i would change "schedule": crontab(0, 0, day_of_month="1").
Here comes my actual problem : How do i test that this really works ?
And by testing i mean actual (unit)tests.
What I've tried is to work with a package called freezegun.
A test with this looks like this :
def test_start_of_month_report_creation(self, user_object, contract_object, report_object):
# set time to the last day of January
with freeze_time("2019-01-31 23:59:59") as frozen_time:
# let one second pass
frozen_time.tick()
# give the celery task some time
time.sleep(20)
# Test Logic to check whether the object was created
# Example: assert MyModel.objects.count() > 0
But this did not work. I suspect that the celery beat does not use the time set via freezgun/python but the real "hardware" clock.
I've also tried setting the Hardwareclock like here but this did not work in my setup.
I'm thankful for any comments, remarks or help on this topic since i'd really like to implement a test for this.
Unit tests cannot test third-party libraries.
You can set the system log, to keep track.
You can check if your task is already on model PeriodicTask. This model defines a single periodic task to be run. It must be associated with a schedule, which defines how often the task should run.
Related
I am trying to execute a task with celery in Django.I want to execute the task at 12:30 pm everyday for which I have written this in my tasks.py
#periodic_task(run_every=crontab(minute=30, hour=12), name="elast")
def elast():
do something
This is not working but if I want to schedule it at every 30 seconds I write this code
#periodic_task(run_every=(timedelta(seconds=30)), name="elast")
def elast():
do something
This works.I wanted to know that what is wrong with the first piece of code?Any help would be appreciated.
As per latest celery 4.3 version , to execute the task at 12:30 pm below code will be useful
celery.py
from celery.schedules import crontab
app.conf.beat_schedule = {
# Executes every day at 12:30 pm.
'run-every-afternoon': {
'task': 'tasks.elast',
'schedule': crontab(hour=12, minute=30),
'args': (),
},
}
tasks.py
import celery
#celery.task
def elast():
do something
to start celery beat scheduler
celery -A proj worker -B
for older version around celery 2.0
from celery.task.schedules import crontab
from celery.decorators import periodic_task
#periodic_task(run_every=crontab(hour=12, minute=30))
def elast():
print("code execution started.")
please check timezone setting.
New userguide
Old userguide
Check out the documentation, especially the parts specific for Django users. Also note that using #periodic_task decorator is deprecated and should be replaced with beat_schedule configuration (see the code).
I have an Django project, that has some functionality to run as cron job several times i.e(every half an hour I need this functionality to be executed).
Till now the job is scheduling but not executing the function. Here I am attaching the code below:
from __future__ import unicode_literals
from django.apps import AppConfig
from projectApp.views import function_to_exec
from django_redis import get_redis_connection
rc = get_redis_connection('default')
from rq_scheduler import Scheduler
scheduler = Scheduler(connection=rc)
def ready():
for job in scheduler.get_jobs():
job.delete()
scheduler.schedule(datetime.utcnow(), function_to_exec, interval=60, queue_name='high')
# scheduler.cron("15 * * * *", func=get_dfp_report, queue_name='high')
ready();
The above code is in my application's apps.py
and the views.py code is like this :
#job('high')
def function_to_exec():
# some logic here
And in my django-scheduler the status is
The status is always in queued state.
Can anyone share the some reference for this to achieve.
Thanks in advance.
Have you started the rqscheduler from the command line to make sure that the jobs are executed?
The scheduler can be started with
rqscheduler
Use -v if you need verbose output
rqscheduler -v
Documentation
Suppose I have 1000 user_ids in a table and I would run every hour to get from Google API info and update 3 fields in that table. How would the impact be and how can it be done efficiently?
I've seen this variant:
m = Module.objects.get(user_id=1).update(field_one=100, field_two=200, field_three=300)
And this one:
m = Module.objects.get(user_id=1)
m.field_one = 100
m.field_two = 200
m.field_three = 300
m.save()
Also how can it be done so that it will run every hour and grab that information? Never done something like this.
Use Redis, Celery to setup asynchronous task queue every hour. Look here https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ for more info on how to setup asych task queue system for django.
Here is the code for tasks.py
from celery.task import periodic_task
from celery.schedules import crontab
#periodic_task(run_every=crontab(minute=0, hour='*/1'))
def get_data_from_google_api():
data_from_google =ping_google_api() # ping google api to get data
return Module.objects.get(user_id=1).update(field_one= data_from_google['field_one'], field_two= data_from_google['field_two'], field_three= data_from_google['field_three'])
Look here for more info :
https://www.caktusgroup.com/blog/2014/06/23/scheduling-tasks-celery/
How to run a Django celery task every 6am and 6pm daily?
Fof this purpose you need to run background queries with periodic taks.
Here is most popular in django task-queue-libs
For example, if you decide use celery, you can write simple periodic task:
from celery.schedules import crontab
from celery.task import periodic_task
#periodic_task(
name='UPDATE_USER',
run_every=crontab(
minute='1',
hour='1,4,7,10,13,16,19,22'))
def update_user():
#get some value from api
Module.objects.filter(user_id=1).update(
field_one=value, field_two=value, field_three=value)
All settings for django you can look in celery docs
How do I schedule a task to run when I start celery beat then again in 1 hours and so.
Currently I have schedule in settings.py:
CELERYBEAT_SCHEDULE = {
'update_database': {
'task': 'myapp.tasks.update_database',
'schedule': timedelta(seconds=60),
},
}
I saw a post from 1 year here on stackoverflow asking the same question:
How to run celery schedule instantly?
However this does not work for me, because my celery worker get 3-4 requests for the same task, when I run django server
I'm starting my worker and beat like this:
celery -A dashboard_web worker -B --loglevel=INFO --concurrency=10
Crontab schedule
You could try to use a crontab schedule instead which will run every hour and start 1 min after initialization of the scheduler. Warning: you might want to do it a couple of minutes later in case it takes longer to start, otherwise you might need to wait the full hour.
from celery.schedules import crontab
from datetime import datetime
CELERYBEAT_SCHEDULE = {
'update_database': {
'task': 'myapp.tasks.update_database',
'schedule': crontab(minute=(datetime.now().minute + 1) % 60),
},
}
Reference: http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html#crontab-schedules
Ready method of MyAppConfig
In order to ensure that your task is run right away, you could use the same method as before to create the periodic task without adding 1 to the minute. Then, you call your task in the ready method of MyAppConfig which is called whenever your app is ready.
#myapp/apps.py
class MyAppConfig(AppConfig):
name = "myapp"
def ready(self):
from .tasks import update_database
update_database.delay()
Please note that you could also create the periodic task directly in the ready method if you were to use django_celery_beat.
Edit: Didn't see that the second method was already covered in the link you mentioned. I'll leave it here in case it is useful for someone else arriving here.
Try setting the configuration parameter CELERY_ALWAYS_EAGER = True
Something like this
app.conf.CELERY_ALWAYS_EAGER = True
I'm trying to setup a Periodic Task that should expire after some time. I'm using Django 1.5.1, celery 3.0.19 and django-celery 3.0.17 (everything from pip).
This is the excerpt code to create the task:
from django.utils import timezone
from datetime import timedelta, datetime
from djcelery.models import PeriodicTask, IntervalSchedule
interval = IntervalSchedule.objects.get(pk=1) # Added through fixture - 3sec interval
expiration = timezone.now() + timedelta(seconds=10)
task = PeriodicTask(name='fill_%d' % profile.id,
task='fill_album',
args=[instance.id],
interval=interval,
expires=expiration) task.save()
And I'm running celery with ./manage.py celeryd -B
The task is being created just fine, and beat is running it every 3 seconds, but after 10 seconds it doesn't expire. At first I thought it was some timezone issue between django and celery, so I let it running for 3 hours (my difference to UTC) but it still wouldn't expire.
During my tests I've actually managed to make it expire once (and the logger kept repeating it was expired, every 3 seconds) but I haven't been able to reproduce it since.
Can anyone shed some light on what I'm doing wrong?
Thanks!
I'm having the same problem and I think celery beat is not honoring the expires. If you set a breakpoint in your task take a look at the current_task.request object and see if expires has a value (or just print current_task.request from within the task.)
For me, if I manually run the task, current_task.request.expires has a value, but if celery beat schedules it, it is None.
I'm using celery 3.1.11
I filed a bug: https://github.com/celery/celery/issues/2283
You can try use last_run_at as:
task = PeriodicTask(name='fill_%d' % profile.id,
task='fill_album',
args=[instance.id],
interval=interval,
expires=expiration,
last_run_at=expiration)
task.save()