I am making a price tracker.My project structure is this:
Myapp-folder
manage.py-from flask script module
subApp-folder
__init__.py
form.py
models.py
views.py
pricemonitor-folder
main.py
__init__.py
send_email.py
price_compare_sendemail.py-with class Compare_sendemail and start_monitor function
In the main.py, I have an interation to compare the prices every 60s and send-email if needed.
from app.PriceMonitor.price_compare_sendmail import Compare_sendemail
break_time = 60 # set waiting time for one crawl round
monitor = Compare_sendemail()
monitor.start_monitor(break_time)
The manage.py is as below:
from flask_script import Manager, Server
from app import app, db
manager = Manager(app)
manager.add_command("runserver",Server(host='127.0.0.1', port=5000, use_debugger=True))
if __name__ == '__main__':
manager.run()
But the iteration doesn't work when I run python manage.py runserver while I directly run the main.py successfully. How can I make up code to run the flask server with the compare_sendemail iteration running at the background? Thanks.
I think you are looking for Celery.
you can use Celery background task. If your application has a long running task, such as processing some uploaded data or sending email, you don’t want to wait for it to finish during a request. Instead, use a task queue to send the necessary data to another process that will run the task in the background while the request returns immediately.
here you can find documentation for celery
https://flask.palletsprojects.com/en/1.1.x/patterns/celery/
and if you want to wait for Task to complete you can use Coroutines and Tasks
https://docs.python.org/3/library/asyncio-task.html
there are other options for flask background task
like
RQ
https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xxii-background-jobs
some other Alternatives
https://smirnov-am.github.io/background-jobs-with-flask/
Threads
uWSGI thread
uWSGI spooler
uSWGI spooler is great for simple tasks. like sending OTP SMS or Email.
I answer part of my own question.
In the main.py, I used while loop and time module to iterate the price_compare_sendemail.py
every 60s. While this is not an ideal background task handler, this project is currently just for my own usage so it is OK for me. My original thought was using the flask script manager to handle all the python commands-I don't know if it is the right thought though because I just started to learn Flask.
After some google search, I found the way to use manager.
from subapp.pricemonitor.main import Start_monitor
Monitor=Start_monitor()
#manager.command
def monitor_start():
break_time=10
Monitor.start_monitoring(break_time)
Then use the command 'python manage.py monitor_start' to start the background task. I don't know if it is useful but at least it fit my original thought.
Related
I have a django project with APScheduler built in it. I have proceeded to the production environment now so binded it with gunicorn and nginx in the proceess. Gunicorn has 3 workers. Problem is that gunicorn initiates the APScheduler for each worker and runs the scheduled job 3 times instead of running it for only once.
I have seen similar questions here it seems it is a common problem. Even the APScheduler original documentation acknowledges the problem and tells no way of fixing it.
https://apscheduler.readthedocs.io/en/stable/faq.html#how-do-i-share-a-single-job-store-among-one-or-more-worker-processes
I saw in other threads people recommended putting --preconfig in the settings. But I read that --preconfig initiates the workers with the current code and does not reload when there has been a change in the code.(See "when not to preload" in below link)
https://www.joelsleppy.com/blog/gunicorn-application-preloading/
I also saw someone recommended binding a TCP socket for the APScheduler. I did not understand it fully but basically it was trying to bind a socket each time APScheduler is initiated then the second and third worker hits that binded socket and throws a socketerror. Sort of
try:
"bind socket somehow"
except socketerror:
print("socket already exists")"
else:
"run apscheduler module"
configuration. Does anyone know how to do it or know if that would actually work?
Another workaround I thought is simply removing the APScheduler and do it with cron function of the server. I am using Digital Ocean so I can simply delete the APScheduler and a cron function that will run the module instead. However, I do not want to go that way because that will make break the "unity" of the whole project and make it server dependable. Does anyone have any more ideas?
Schedule module:
from apscheduler.schedulers.background import BackgroundScheduler
from RENDER.views import dailypuzzlefunc
def start():
scheduler=BackgroundScheduler()
scheduler.add_job(dailypuzzlefunc,'cron', day="*",max_instances=2,id='dailyscheduler')
scheduler.start()
In the app:
from django.apps import AppConfig
class DailypuzzleConfig(AppConfig):
default_auto_field = "django.db.models.BigAutoField"
name = "DAILYPUZZLE"
def ready(self):
from SCHEDULER import dailypuzzleschedule
dailypuzzleschedule.start()
web:
python manage.py collectstatic --no-input;
gunicorn MasjidApp.wsgi --timeout 15 --preload
use --preload.
It's working well for me.
I have a function that fetches data and needs to be run periodically.
All I care about is running it every 30 seconds.
I searched and found the following options -
celery
django-apscheduler
Apscheduler
I have tried Apscheduler using BackgroundScheduler and it has this problem that it'll run a new scheduler for each process.
I am completely new to scheduling functions and have no idea which one I should use or if there is a better way.
I experienced a similar issue and solved it by creating a custom management command and scheduling it on the web server.
Within the root of the app, create management/commands directory:
some_app/
__init__.py
models.py
management/
commands/
dosomething.py
tests.py
views.py
// dosomething.py
from django.core.management.base import BaseCommand, CommandError
class Command(BaseCommand):
help = 'Description of the action here'
def handle(self):
print('Doing something')
return
To confirm if it's working, run python manage.py dosomething
The scheduling itself depends on which web server you are using. In my case it was Heroku and I used their Scheduler add-on.
I am developing a Django app running on Heroku.
In order to update my database with some data coming from a certain API service, I need to periodically run a certain script (let's say myscript).
How can I use Heroku Scheduler to do it?
As already explained here, quick and simple way to answer this question is asking yourself how would you do to run that script periodically, as if you were the scheduler yourself.
Now, the best way to run a script in your Django app at any moment, it is to create a custom management command and to run it from your command prompt when you need it, like this:
python manage.py some_custom_command
Then, if you were the scheduler, you would run that command from your command prompt at every time written in the schedule.
So, a good idea would be to make Heroku Scheduler behave the same. Thus, the aim here is to have Heroku Scheduler run python manage.py some_custom_command at scheduled times.
Here is how you can do it:
In your_app directory, create a folder management and then inside it create another folder commands and finally, inside it, create a file some_custom_command.py
So, just to be clear
your_app/management/commands/some_custom_command.py
Then, inside some_custom_command.py insert:
from django.core.management.base import BaseCommand
from your_app.path_to_myscript_file import myscript
class Command(BaseCommand):
def handle(self, *args, **options):
# Put here some script to get the data from api service and store it into your models.
myscript()
Then go on Heroku > your_app > resources
In add-ons section select Heroku Scheduler, click on it so that its window opens, then click on add job, select the time you want, insert the command python manage.py some_custom_command and save.
I am just starting to learn about Django and have just discovered celery to run async background tasks.
I have a dummy project which I pilfered off the internet with a sample task as follows:
from djcelery import celery
from time import sleep
#celery.task
def sleeptask(i):
sleep(i)
return i
Now in my view, I have the following:
def test_celery(request):
result = tasks.sleeptask.delay(10)
return HttpResponse(result.task_id)
This runs fine and when I point the browser to it, I get some random string like 93463e9e-d8f5-46b2-8544-8d4b70108b0d which I am guessing is the task id.
However, when I do this:
def test_celery(request):
result = tasks.sleeptask.delay(10)
return HttpResponse(result.get())
The web browser goes in a loop with the message "Connecting..." and never returns. I was under the impression, this will block till the task is run and give the result but that does not seem to be the case. What am I doing wrong?
Another question is the way I am doing it, is it going to run it asynchronously i.e. not block while the task is running?
EDIT
In my settings.py file I have:
import djcelery
# Setup celery
djcelery.setup_loader()
BROKER_URL = 'redis://localhost:6379/0'
On the Django side, I do not get any errors:
System check identified no issues (0 silenced).
September 27, 2016 - 18:13:12
Django version 1.9.5, using settings 'myproject.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Thanks to the hints in the comments, I finally was able to solve the problem. I had to add the following:
CELERY_IMPORTS('myproject.tasks') to my settings.py file.
I also needed to run the worker as:
python manage.py celery worker
I've recently been playing around with django and celery. One annoying thing during development is the fact that I have to restart the celery daemon each time I modify a task. When I'm developing, I usually like to use 'manage.py runserver' which automatically reloads the django framework on modifications to my apps.
Is there a way to add a hook to the reloading process that runserver does so that it automatically restarts the celery daemon I have running?
Alternatively, does celery have a similar monitor-and-reload-on-change mode that I should be using for development?
Django-supervisor works very well for this purpose. You can have it start the Django server, Celery, and anything else you need, and have different configurations for development and production servers. It also knows to reload the celery daemon when your code changes.
https://github.com/rfk/django-supervisor
I believe you can set CELERY_ALWAYS_EAGER to true.
Yes. Django provides auto reload hook, which can be used to restart other scripts.
Here is a simple management command which prints a message on reload
import subprocess
from django.core.management.base import BaseCommand
from django.utils import autoreload
def reload():
print('Code changed. Auto reloading...')
class Command(BaseCommand):
def handle(self, *args, **options):
autoreload.main(reload)
Now you can save to a reload.py and run it with python manage.py reload. A management command to reload celery workers is available here.
Celery didn't have any feature for reload code or for auto restart when the code change, than you have to restart it manually.
There isn't a way for add an hook, and I think not worthwhile of edit the source code of django just for perform a restart.
Personally while I'm developing i prefere to see the output shell of celery that is decorated with color instead of tail the logs, is more readable.
Celery 2.5 has an experimental runtime option --autoreload that could be used for this purpose, too. Here's more detail in the release notes. That being said, I think django-supervisor (via #Lee Semel) looks like the better way of doing things. I thought I would post this alternative here in case other readers do not want to have to configure another app for asynchronous processing.