Django custom command and cron - django

I want my custom made Django command to be executed every minute. However it seems like python /path/to/project/myapp/manage.py mycommand doesn't seem to work while at the directory python manage.py mycommand works perfectly.
How can I achieve this ? I use /etc/crontab with:
****** root python /path/to/project/myapp/manage.py mycommand

I think the problem is that cron is going to run your scripts in a "bare" environment, so your DJANGO_SETTINGS_MODULE is likely undefined. You may want to wrap this up in a shell script that first defines DJANGO_SETTINGS_MODULE
Something like this:
#!/bin/bash
export DJANGO_SETTINGS_MODULE=myproject.settings
./manage.py mycommand
Make it executable (chmod +x) and then set up cron to run the script instead.
Edit
I also wanted to say that you can "modularize" this concept a little bit and make it such that your script accepts the manage commands as arguments.
#!/bin/bash
export DJANGO_SETTINGS_MODULE=myproject.settings
./manage.py ${*}
Now, your cron job can simply pass "mycommand" or any other manage.py command you want to run from a cron job.

cd /path/to/project/myapp && python manage.py mycommand
By chaining your commands like this, python will not be executed unless cd correctly changes the directory.

If you want your Django life a lot more simple, use django-command-extensions within your project:
http://code.google.com/p/django-command-extensions/
You'll find a command named "runscript" so you simply add the command to your crontab line:
****** root python /path/to/project/myapp/manage.py runscript mycommand
And such a script will execute with the Django context environment.

This is what i have done recently in one of my projects,(i maintain venvs for every project i work, so i am assumning you have venvs)
***** /path/to/venvs/bin/python /path/to/app/manage.py command_name
This worked perfectly for me.

How to Schedule Django custom Commands on AWS EC-2 Instance?
Step -1
First, you need to write a .cron file
Step-2
Write your script in .cron file.
MyScript.cron
* * * * * /home/ubuntu/kuzo1/venv/bin/python3 /home/ubuntu/Myproject/manage.py transfer_funds >> /home/ubuntu/Myproject/cron.log 2>&1
Where * * * * * means that the script will be run at every minute. you can change according to need (https://crontab.guru/#*_*_*_*_*). Where /home/ubuntu/kuzo1/venv/bin/python3 is python virtual environment path. Where /home/ubuntu/kuzo1/manage.py transfer_funds is Django custom command path & /home/ubuntu/kuzo1/cron.log 2>&1 is a log file where you can check your running cron log
Step-3
Run this script
$ crontab MyScript.cron
Step-4
Some useful command
1. $ crontab -l (Check current running cron job)
2. $ crontab -r (Remove cron job)

The runscript extension wasn't well documented. Unlike the django command this one can go anywhere in your project and requires a scripts folder. The .py file requires a run() function.

If its a standalone script, you need to do this:
from django.conf import settings
from django.core.management import setup_environ
setup_environ(settings)
#your code here which uses django code, like django model
If its a django command, its easier: https://coderwall.com/p/k5p6ag
In (management/commands/exporter.py)
from django.core.management.base import BaseCommand, CommandError
class Command(BaseCommand):
args = ''
help = 'Export data to remote server'
def handle(self, *args, **options):
# do something here
And then, in the command line:
$ python manage.py exporter
Now, it's easy to add a new cron task to a Linux system, using crontab:
$ crontab -e
or $ sudo crontab -e if you need root privileges
In the crontab file, for example for run this command every 15 minutes, something like this:
# m h dom mon dow command
*/15 * * * * python /var/www/myapp/manage.py exporter

Related

In Linux, how to run Django custom management using crontab?

I have generated a bash script that activates virtaulenv and runs my custom management command in Django. I want to run the bash script every day at midnight.
Bash Script :
cd ~
cd path_to_virtualenv/
source virtualenv_name/bin/activate
cd path_to_project/
python manage.py custom_command
deactivate
When I run this script using . or source it runs perfectly. I have configured crontab to run this bash script (For testing, I have set execution time per minute). But I am not getting desired output.
crontab -e
*/1 * * * * source /path_to_bash_script/bash_script_filename
Add shebang to your shell script (i.e: #!/bin/bash, ...)
Remove source from crontab
add an extension to your script file.
I changed the script as :
#!/bin/bash
source /path_to_virutalenv/bin/activate
python /path_to_project/manage.py custom_command
In crontab :
*/1 * * * * bash /path_to_bash_script/script.sh

A clear step by step process for running a periodic task in a django application

I have been trying a long for creating a periodic task in Django but there are lot of version constraints and not a clear explanation.
I recommend Celery. What is Celery?
Celery supports scheduling tasks. Check this doc
First of all, you want to create a management command following this guide.
https://docs.djangoproject.com/en/2.1/howto/custom-management-commands/
Say we want to run the closepoll command in the example every 5 minutes.
You'll then need to create a script to run this command.
Linux / MacOS:
#!/bin/bash -e
cd path/to/your/django/project
source venv/bin/activate # if you use venv
python manage.py closepoll # maybe you want to >> /path/to/log so you can log the results
store the file as run_closepoll.sh, run chmod +x run_closepoll.sh in command line
Now we can use crontab to run our command
run crontab -e in your command line
add this line:
*/5 * * * * /path/to/run_closepoll.sh
Now the command will run every 5 minutes.
If you're not familiar with crontab, you can use this website
https://crontab-generator.org/
Windows:
Same content as the above example, but remove the first line and save as run_closepoll.bat
In your start menu, search for Task Scheduler, follow the instructions on the GUI, it should be pretty simple from there.
for more info about the task scheduler, see here: https://learn.microsoft.com/en-us/windows/desktop/taskschd/using-the-task-scheduler
This blog explains clearly
https://medium.com/#yehandjoe/celery-4-periodic-task-in-django-9f6b5a8c21c7
Thanks!!!
I'm using django-cron and It works as expected. The only caveat is that you have to set a Cron job in the Linux system to run the command python manage.py runcrons.

How to assign the python interpreter spark worker used?

How to assign the python interpreter spark worker used?
i try several method like:
1) set env Vars
export PYSPARK_DRIVER_PYTHON=/python_path/bin/python
export PYSPARK_PYTHON=/python_path/bin/python
not work. i'm sure PYSPARK_DRIVER_PYTHON PYSPARK_PYTHON env set success use:
env | grep PYSPARK_PYTHON
i want to pyspark use
/python_path/bin/python
as the starting python interpreter
but worker start use the :
python -m deamon
i don't want to link default python to /python_path/bin/python in the fact that
this may affect other devs, bcz default python and /python_path/bin/python is not same version, and both in production use.
Also set spark-env.sh not works:
spark.pyspark.driver.python=/python_path/bin/python
spark.pyspark.python=/python_path/bin/python
when start driver some warning logs like:
conf/spark-env.sh: line 63:
spark.pyspark.driver.python=/python_path/bin/python: No such file or directory
conf/spark-env.sh: line 64:
spark.pyspark.python=/python_path/bin/python: No such file or directory
1) Check permissions on your python directory. Maybe Spark doesn't have correct permissions. Try to do: sudo chmod -R 777 /python_path/bin/python
2) Spark documentation says:
Property spark.pyspark.python take precedence if it is set.
So try also set spark.pyspark.python in conf/spark-defaults.conf.
3) Also if you use cluster with more then one node you need to check if Python is installed in a correct directory on each node because you don't know where workers will be started.
4) Spark will use the first Python interpreter available on your system PATH, so like workaround you can set the path to your python in PYTHON variable.

SyntaxError if executed with cron, OK if manually

I'm trying to schedule a script with cron in kubuntu. If I execute the script manually line by line, it goes just fine, but scheduled with cron it raises the following SyntaxError:
File "/opt/django/myproject/myapp/cron/test.sh", line 4
python manage.py mycustomcommand
^
SyntaxError: invalid syntax
The content of the script test.sh is as following:
#!/bin/bash
source /opt/virtualenvs/myvirtualenv/bin/activate
cd /opt/django/myproject
python manage.py mycustomcommand
Basically the script activates a virtual enviroment where django is installed, then accesses to my project path, and then executes a custom django command. As said, this works fine if I do it manually.
I tried to schedule the script in cron with normal and root permissions also ("crontab -e" and "sudo crontab -e")
Any idea? Thanks!
The error suggests that you fire the command python manage.py mycustomcommand from within the Python interpreter (and not as a bash command).
You probably have something like
1 * * * * python /path/to/myscript/test.sh
in your crontab entry which is a mistake and it should be
1 * * * * /path/to/myscript/test.sh
instead.

Defining startup cron job with screen running django

I want to run my development django server at startup so I defined following cron job:
#reboot screen -d -m django-admin.py runserver 192.168.0.28:8000
But it didn't work.
What is really interesting, when I copy/paste directly to terminal and execute it works just fine.
I even tried something like this:
#reboot cd /home/ubuntu && /usr/bin/screen -d -m /usr/bin/python /usr/local/bin/django-admin.py runserver 192.168.0.28:8000 &> /home/ubuntu/cron.err
To be sure I'm not using some undefined commands in wrong location and examined contents of cron.err file but it's empty.
And (of course) when I fire this directly from the console it works immediately.
Please help.
Does it work if you try and run it from cron at a specific time? Eg:
50 12 2 8 * /usr/bin/screen -dmS set_from_cron