I am scheduling some tasks with apply_async() providing the countdown for the task. Once the tasks execute I can view the state and logs with django-celery-results's task_results model. Now, I know all the scheduled tasks are added to the message queue from where I should be able to retrieve the scheduled tasks that haven't executed yet. But is there any other way to view them with state/logs etc? Or will I need to create a model similar to task_results and implement the feature myself?
Related
The airflow I'm using, sometimes the pipelines wait for a long time to be scheduled. There have also been instances where a job was running for too long (presumably taking up resources of other jobs)
I'm trying to work out how to programatically identify the health of the scheduler and potentially monitor those in the future without any additional frameworks. I started to have a look at the metadata database tables. All I can think of now is to see start_date and end_date from dag_run, and duration of the tasks. What are the other metrics that I should be looking at? Many thanks for your help.
There is no need to go "deep" inside the database.
Airflow provide you with metrics that you can utilize for the very purpose: https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/metrics.html
If you scroll down, you will see all the useful metrics and some of them are precisely what you are looking for (especially Timers).
This can be done with the usual metrics integration. Airflow publishes the metrics via statsd, and Airflow Official Helm Chart (https://airflow.apache.org/docs/helm-chart/stable/index.html) even exposes those metrics for Prometheus via statsd exporter.
Regarding the spark job - yeah - current implementation of spark submit hook/operator is implemented in "active poll" mode. The "worker" process of airflow polls the status of the job. But Airlfow can run multiple worker jobs in parallel. Also if you want, you can implement your own task which will behave differently.
In "classic" Airflow you'd need to implement a Submit Operator (to submit the job) and "poke_reschedule" sensor (to wait for the job to complete) and implement your DAG in the way that sensort task will be triggered after the operator. The "Poke reschedule" mode works in the way that the sensor is only taking the worker slot for the time of "polling" and then it frees the slot for some time (until it checks again).
As of Airflow 2.2 you can also write a Deferrable Operator (https://airflow.apache.org/docs/apache-airflow/stable/concepts/deferring.html?highlight=deferrable) where you could write single Operator - doing submision first, and then deferring the status check - all in one operator. Defferrable operators are efficiently handling (using async.io) potentially multiple thousands of waiting/deferred operators without taking slots or excessive resources.
Update: If you really cannot use statsd (helm is not needed, statsd is enough) you should never use DB to get information about the DAGS. Use Stable Airflow REST API instead: https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html
I wanted to implement scheduled jobs/actions in my DJANGO backend.
Actions are basically deducting monthly recurring payment from the customer. Sending payment link say before 10 days etc. etc. The dates will be based on when the user buys the subscription.
I have never implemented scheduled jobs before. I know there are some ways like cron tabs and celery.
I wanted to know what will be the best strategy/tool for scheduled payments.
So basically what I think i will do is that i will run the scheduled job every day at a particular time and will check the available candidates and will run the payment module.
Is this strategy correct to run jobs everyday. Are there any better methods available. Is there a way that jobs run automatically when say the customers new billing cycle arrives.
Yes, the strategy you are following is correct. You can use celery, redis and crontab to execute the payment system.
So, firstly you can specify the schedules using crontab. Also, .delay() function will help you in triggering the jobs whenever customer`s new billing cycle arrives.
So, the flow will be, tasks get triggered when a new billing cycle arrives using .delay()
Then the celery worker will register the task and the schedules. Then you may use celery beat to run the tasks periodically. Also you may use redis as a message queue.
Read about .delay() here
Read about celery configuration setting here
Read about setting a task scheduler using celery here
I am a non-paying user on a computing cluster that uses SLURM.
Occasionally, I've had long-running and multiple jobs that clogged up the squeue for paying users. Due to this I've had jobs cancelled by admin. Currently I've had a cap on the number of nodes that are available to me. While I dont argue with the equity of this arrangement , this is a problem for me in terms of getting work done, especially because I see free nodes that are not running any jobs, while I just sit waiting for jobs to pass through the node cap....
With that as background info, here are my two questions:
Isnt it possible for admin to suspend, and then resume jobs - either a job, or all jobs of a user, or a set of jobs? Is this suspend / resume onerous from the admin's perspective?
I suppose it should be possible to create a list of paying Vs non-paying users. And when paying username submits with sbatch to automatically instruct SLURM to suspend non-paying username's job or jobs, and resume when paid user's jobs have completed. Is this even possible? IF yes, is it outside the skill scope of regular SLURM / Farm admins?
Could someone please suggest any other solutions (if what I have asked above are unreasonable or absurd)?
Thank you!
The admin can run scontrol suspend jobid and then scontrol resume jobid
The keywords here are 'QOS' and 'preemption'. Typically a QOS is created for the paying users, that has preemptive rights over the normal QOS. Jobs of the non-paying users can be cancelled, checkpointed, requeued, or suspended.
I'm writing some auction functionality for a website powered by Django, and there is an option to have the auction be extended by a few seconds every time somebody bids (so that there is no last-minute sniping). When the auction is finished, certain things need to be done like notifying the winner of their victory and notifying the seller to whom they need to send the item.
How, in Django, do I go about running an auction-completion script whenever an auction ends, especially when the auction end time may be a moving target?
Thanks.
It sounds like extending an auction_end DateTime field could easily be done on the bid() view
As far as emailing outcomes you should write a management script http://docs.djangoproject.com/en/dev/howto/custom-management-commands/ checking for auctions that have ended and haven't had notifications sent and schedule this script to run in cron at regular intervals (by minute if neccessary). An example would be pinax's django-notification emit_notices script.
http://github.com/jtauber/django-notification/blob/master/notification/management/commands/emit_notices.py
Better still you might be able to leverege django-notification itself.
Django command extensions provides jobs system (job scheduling). After you add it to the crontab it is capable of running jobs hourly/daily/weekly/monthly.
I think that it should be easy to extend job management command to acomplish your tasks.
Sounds like you probably want to have whatever code is responsible for closing the auction fire a custom signal to indicate that it's happening.
One of my view functions is a very long processing job and clearly needs to be handled differently.
Instead of making the user wait for long time, it would be best if I were able to lunch the processing job which would email the results, and without waiting for completion notify the user that their request is being processed and let them browse on.
I know I can use os.fork, but I was wondering if there is a 'right way' in terms of Django. Perhaps I can return the HTTP response, and than go on with this job somehow?
There are a couple of solutions to this problem, and the best one depends a bit on how heavy your workload will be.
If you have a light workload you can use the approach used by django-mailer which is to define a "jobs" model, save new jobs into the database, then have cron run a stand-alone script every so often to process the jobs stored in the database (deleting them when done). You can use something like django-chronograph to manage the job scheduling easier
If you need help understanding how to write a script to process the job see James Bennett's article Standalone Django Scripts for help.
If you have a very high workload, meaning you'll need more than a single server to process the jobs, then you want to use a real distribute task queue. There is a lot of competition here so I can't really detail all the options, but a good one to use with for Django apps is celery.
Why not simply start a thread to do the processing and then go on to send the response?
Before you select a solution, you need to determine how the process will be run. I.e is it the same process for every single user, the data is the same and can be scheduled regularly? or does each user request something and the results are slightly different ?
As an example, if the data will be the same for every single user and can be run on a schedule you could use cron.
See: http://www.b-list.org/weblog/2007/sep/22/standalone-django-scripts/
or
http://docs.djangoproject.com/en/dev/howto/custom-management-commands/
However if the requests will be adhoc and you need something scalable that can handle high load and is asynchronous: what you are actually looking for is a message queing system. Your view will add a request to the queue which will then get acted upon.
There are a few options to implement this in Django:
Django Queue service is purely django & python and simple, though the last commit was in April and it seems the project has been abandoned.
http://code.google.com/p/django-queue-service/
The second option if you need something that scales, is distributed and makes use of open source message queing servers: celery is what you need
http://ask.github.com/celery/introduction.html
http://github.com/ask/celery/tree