I am developing a Django app running on Heroku.
In order to update my database with some data coming from a certain API service, I need to periodically run a certain script (let's say myscript).
How can I use Heroku Scheduler to do it?
As already explained here, quick and simple way to answer this question is asking yourself how would you do to run that script periodically, as if you were the scheduler yourself.
Now, the best way to run a script in your Django app at any moment, it is to create a custom management command and to run it from your command prompt when you need it, like this:
python manage.py some_custom_command
Then, if you were the scheduler, you would run that command from your command prompt at every time written in the schedule.
So, a good idea would be to make Heroku Scheduler behave the same. Thus, the aim here is to have Heroku Scheduler run python manage.py some_custom_command at scheduled times.
Here is how you can do it:
In your_app directory, create a folder management and then inside it create another folder commands and finally, inside it, create a file some_custom_command.py
So, just to be clear
your_app/management/commands/some_custom_command.py
Then, inside some_custom_command.py insert:
from django.core.management.base import BaseCommand
from your_app.path_to_myscript_file import myscript
class Command(BaseCommand):
def handle(self, *args, **options):
# Put here some script to get the data from api service and store it into your models.
myscript()
Then go on Heroku > your_app > resources
In add-ons section select Heroku Scheduler, click on it so that its window opens, then click on add job, select the time you want, insert the command python manage.py some_custom_command and save.
Related
I was wondering why my scheduled posts were not working automatically in Wagtail, but I see in the documentation that a management command is needed to make this happen. I am unfamiliar with writing custom management commands, and I am wondering how to make the python manage.py publish_scheduled_pages command fire off automatically every hour?
Where would this code go in the document tree? Is there code that I just need to drop in and it runs from there? Or is something required on the server to run these commands on a schedule?
Any help would be appreciated. I couldn't find any existing code anywhere for this functionality in Wagtail and I'm wondering why the button is in the admin to schedule a post, but the functionality is not already built in?
You are probably familiar with management commands since python manage.py runserver and makemigrations and migrate are management commands.
You can see all available commands with python manage.py -h
publish_scheduled_pages should be called periodically. Form the Wagtail docs:
This command publishes, updates or unpublishes pages that have had these actions scheduled by an editor. We recommend running this command once an hour.
Periodically executing a command can be done in various ways. Via crontab is probably the most common. To edit the crontab:
$ crontab -e
Add (for every fist minute of the hour):
0 * * * * python /path/to/your/manage.py publish_scheduled_pages --settings=your.settings
I have a Django based web application, some functionality of the application is scheduled to be run as a part of cron jobs using django-crontab. I want to hook a debugger so that I can inspect some odd behaviours of my code. I normally use visual studio code. Is it possible to hook a debugger, since cron jobs basically run independently apart from server?
You can put a breaking point debugger in the code using pdb or ipdb. Like this:
def some_function():
# some code
import pdb;pdb.set_trace() # or use ipdb
# rest of the code
Then in shell, run python manage.py crontab show to show cronjobs with ids, then run python manage.py crontab run <id>. It will hit the debugger, then you will hit the breaking point. Thus you can use debugger here.
I am developing a web application with React for frontend and Django for backend. I use Webpack to watch for changes and bundle code for React apps.
The problem is that I have to run two commands concurrently, one for React and the other one for Django:
webpack --config webpack.config.js --watch
./manage.py runserver
Is there any way to customize runserver command to execute the npm script, like npm run start:dev? When you use Node.js as a backend platform, you can do the similar job like npm run build:client && npm run start:server.
If you are already using webpack and django, probably you can be interested in using webpack-bundle-tracker and django-webpack-loader.
Basically webpack-bundle-tracker will create an stats.json file each time the bundle is build, and django-webpack-loader will watch for those stats.json file to relaunch the dev server. This stack allows to separate the concerns between the server and the client.
There are a couple of posts out there explaining this pipeline.
I'm two and a half years late, but here's a management command that implements the solution that OP wanted, rather than a redirection to another solution. It inherits from the staticfiles runserver and runs webpack concurrently in a thread.
Create this management command at <some_app>/management/commands/my_runserver.py:
import os
import subprocess
import threading
from django.contrib.staticfiles.management.commands.runserver import (
Command as StaticFilesRunserverCommand,
)
from django.utils.autoreload import DJANGO_AUTORELOAD_ENV
class Command(StaticFilesRunserverCommand):
"""This command removes the need for two terminal windows when running runserver."""
help = (
"Starts a lightweight Web server for development and also serves static files. "
"Also runs a webpack build worker in another thread."
)
def add_arguments(self, parser):
super().add_arguments(parser)
parser.add_argument(
"--webpack-command",
dest="wp_command",
default="webpack --config webpack.config.js --watch",
help="This webpack build command will be run in another thread (should probably have --watch).",
)
parser.add_argument(
"--webpack-quiet",
action="store_true",
dest="wp_quiet",
default=False,
help="Suppress the output of the webpack build command.",
)
def run(self, **options):
"""Run the server with webpack in the background."""
if os.environ.get(DJANGO_AUTORELOAD_ENV) != "true":
self.stdout.write("Starting webpack build thread.")
quiet = options["wp_quiet"]
command = options["wp_command"]
kwargs = {"shell": True}
if quiet:
# if --quiet, suppress webpack command's output:
kwargs.update({"stdin": subprocess.PIPE, "stdout": subprocess.PIPE})
wp_thread = threading.Thread(
target=subprocess.run, args=(command,), kwargs=kwargs
)
wp_thread.start()
super(Command, self).run(**options)
For anyone else trying to write a command that inherits from runserver, note that you need to check for the DJANGO_AUTORELOAD_ENV variable to make sure you don't create a new thread every time django notices a .py file change. Webpack should be doing it's own auto-reloading anyway.
Use the --webpack-command argument to change the webpack command that runs (for example, I use --webpack-command 'vue-cli-service build --watch'
Use --webpack-quiet to disable the command's output, as it can get messy.
If you really want to override the default runserver, rename the file to runserver.py and make sure the app it lives in comes before django.contrib.static in your settings module's INSTALLED_APPS.
You shouldn't mess with the built-in management commands but you can make your own: https://docs.djangoproject.com/en/1.10/howto/custom-management-commands/.
On your place I'd leave runserver in place and create one to run your custom (npm in this case) script, i.e. with os.execvp.
In theory you could run two parallel subprocesses one that would execute for example django.core.management.execute_from_command_line and second to run your script. But it would make using tools like pbd impossible (which makes work very hard).
The way I do it is that I leverage Docker and Docker compose. Then when I use docker-compose up -d my database service, npm scripts, redis, etc run in the background (running runserver separately but that's another topic).
I have a custom admin command that emails out reports. It normally runs from a cron job. What I would like to do is add a button to my web app that when clicked will cause the the admin command to run there and then rather than wait for the cron job to call it. How do I do this? Do I have to call out to a command line such as
python manage.py myadmincmd
or can I invoke the code from within a view? It seems it would be cleaner if I could do this from within a view without needing to break out to the command line.
You can use call_command:
from django.core.management import call_command
call_command('myadmincmd')
I've recently been playing around with django and celery. One annoying thing during development is the fact that I have to restart the celery daemon each time I modify a task. When I'm developing, I usually like to use 'manage.py runserver' which automatically reloads the django framework on modifications to my apps.
Is there a way to add a hook to the reloading process that runserver does so that it automatically restarts the celery daemon I have running?
Alternatively, does celery have a similar monitor-and-reload-on-change mode that I should be using for development?
Django-supervisor works very well for this purpose. You can have it start the Django server, Celery, and anything else you need, and have different configurations for development and production servers. It also knows to reload the celery daemon when your code changes.
https://github.com/rfk/django-supervisor
I believe you can set CELERY_ALWAYS_EAGER to true.
Yes. Django provides auto reload hook, which can be used to restart other scripts.
Here is a simple management command which prints a message on reload
import subprocess
from django.core.management.base import BaseCommand
from django.utils import autoreload
def reload():
print('Code changed. Auto reloading...')
class Command(BaseCommand):
def handle(self, *args, **options):
autoreload.main(reload)
Now you can save to a reload.py and run it with python manage.py reload. A management command to reload celery workers is available here.
Celery didn't have any feature for reload code or for auto restart when the code change, than you have to restart it manually.
There isn't a way for add an hook, and I think not worthwhile of edit the source code of django just for perform a restart.
Personally while I'm developing i prefere to see the output shell of celery that is decorated with color instead of tail the logs, is more readable.
Celery 2.5 has an experimental runtime option --autoreload that could be used for this purpose, too. Here's more detail in the release notes. That being said, I think django-supervisor (via #Lee Semel) looks like the better way of doing things. I thought I would post this alternative here in case other readers do not want to have to configure another app for asynchronous processing.