I have a project using django-celery-beat, celery, and reds. For some reason one specific task out of many in my entire project never fires, with no errors or anything.
I've verified that it's autodetected and showing up in the Django Admin dropdown and is registered with #app.task.
I've tried deleting and recreating the task.
I've tried creating the PeriodicTask programmatically instead of via the Admin.
I've tried setting the last_run_at to start_time - interval
I've tried using a crontab interval.
I've made sure it's enabled and points to a correct task name.
Are there any reasons why this might be happening? It appears to be just straight up ignored.
Related
My Django app generates a complex report that can take upto 5 minutes to create. Therefore it runs once a night using a scheduled management command.
That's been ok, except I now want the user to be able to select the date range for the report, which means the report needs to be created while the user waits.
What are my options for running the tast in the background? So far I've found these:
Celery - might work but is complex
django-background-tasks looks like the right tool for the job but hasn't been updated for years, last supported Django is 2.2
The report/background task could be generated by AWS Lambda, basically in a microservice. Django calls the Microservice which can execute the background task then call the Django app back once finished. This is what I did last time but not sure it would work now as I'd need to send the microservice 10mb of data to process.
Use subprocess.popen which someone here said worked for them but other reports say it doesn't work from Django.
EDIT: Looks like Django 3.1 onwards supports ASync views and may be the simple solution for this.
I am creating a Django application for school project. I want to schedule jobs (every (work)day on 9:00 and 17:00).
I am trying to do it with Celery right now, but I stuck very hard on it, and as the deadline is in sight, I want to use alternative options: just a cronjob. I think just the cronjob works fine, but the user should be able to edit the times of the cronjobs using the Django web application (so not logging in to SSH, edit the crontab manually).
Is this possible? Can't find anything about it on the internet.
You need django-celery-beat plugin that adds new models to the django admin named "Periodic tasks" where you can manage cron schedule for your tasks.
As an alternate, if you really do not want to run a background tasks, you can create django management commands and use a library like python-crontab to add/modify/remove cron jobs in the system.
I want to schedule a task in Django which would run at a paticular time every day.I am thinking of using celery to schedule the tasks so I wanted to know is there anyway through celery-beat I could execute the task through admin panel that is make it active or inactive and give the time at which I want to execute task through my admin site.Instead of writing this in my code
crontab(day_of_week="0,6", hour="0,12", minute=15)
I want to pass it through my admin panel so that in future if I want to execute my task at some other time I could change it directly through admin panel instead of changing my code.
You will need to implement that on your own or use a third-party project. Luckily, someone already created something like that:
https://django-celery-beat.readthedocs.io/
This extension enables you to store the periodic task schedule in the
database.
The periodic tasks can be managed from the Django Admin interface,
where you can create, edit and delete periodic tasks and how often
they should run.
I have a set of Django commands that do some complex operations, and I'd like to begin tracking the duration of how long they take, in order to monitor their performance over time.
I've struggled to figure out how to instrument newrelic to capture the duration. I've tried everything from 'assuming' the standard django newrelic agent will magically capture it, to try to instrument them myself as a custom event, all with varying degrees of success.
What is the best way to do this, and does newrelic provide something to facilitate this? The docs were very lacking for this kind of scenario.
Additionally, I would consider these as background_tasks - what is the background_task field in newrelic used for?
Thanks
At New Relic we do have a prototype for some changes we are considering making to the Python agent which will add support for monitoring nominated Django admin commands. We are still working through some details of those changes but it is possible it will make it in to one of the upcoming releases over the next couple of months.
In the interim, what you can do is add something like the following to your agent configuration file:
[newrelic:django-manage]
app_name = Python Application (Django Management Commands)
startup_timeout = 10.0
[background-task:django-manage-syncdb]
enabled = true
function = django.core.management.commands.syncdb:Command.handle_noargs
name = syncdb
group = Django
You would then run the Django manage.py script as:
NEW_RELIC_CONFIG_FILE=newrelic.ini
NEW_RELIC_ENVIRONMENT=django-manage
export NEW_RELIC_CONFIG_FILE
export NEW_RELIC_ENVIRONMENT
newrelic-admin run-python manage.py syncdb
When the data is then reported into our UI, it will appear under 'Type -> Other transactions -> Django' in the 'Transactions' tab for the specified application.
The 'newrelic:django-manage' section is an environment section that allows you to override the 'app_name' setting defining which application in our UI the data should be reported to. The 'NEW_RELIC_ENVIRONMENT' variable says that the environment section should be used as overrides for the default 'newrelic' section'.
Even if you do not override 'app_name' and have it inherit the value from the default 'newrelic' section, you still must set the 'startup_timeout' setting. This is necessary as under normal circumstances the agent will only lazily register itself in the background when the first web request is being handled by a web application. Having registration being done lazily will though cause a problem when tracking Django management commands as a background task. This is because there is only the one function call to be monitored, and not waiting for agent registration to occur will result in no data being captured. Setting a non zero value for the startup timeout therefore says to wait for registration to occur.
If for some reason agent registration takes too long, the timeout will expire after the specified time and the management command allowed to proceed anyway. Either way, the need to wait means that there will be a delay in actually starting the management command. This should be kept in mind if the management command needs to be run as quickly as possible.
If necessary, a 'shutdown_timeout' can also be specified if the uploading of data captured by the agent is consistently taking longer than the default 2.5 seconds and therefore not always being reported.
As to the 'background-task' section, it defines the specific function which should be monitored and reported as a background task. In this case we are targeting the particular function implementing the Django 'syncdb' command.
I have a program that classifies text and would like to make it interactive with a user on the front-end of my django site. The problem is it takes 20 seconds for the program to load the training set and get going, and that's not feasible every time someone enters input.
Instead, I'd like Django to load the program once when the server starts, and have all user input interact with it via a view.
I looked at launching subprocesses, but if I'm not mistaken, a subprocess would only be launched when a view is called and that is undesirable for this.
Any ideas?
Thanks.
It's possible that Celery would be appropriate here. There is Django integration available with django-celery.
As Jim noticed celery is one of the best options you have for asynchronus task management, but if you want to avoid celery & its dependecies overhead you could just add a status field on the model the process takes place (e.g. text_processed boolean field with default=False) and create an application management command which would handle the process of the created db entries.
Add the command on a cron and you are done.