Running cron jobs on aws elastic beanstalk - django - django

I'm having trouble getting my cron jobs to execute.
Setup:
Django - 1.9
Elastic beanstalk - 64bit Amazon Linux 2016.03 v2.1.3 running Python 3.4
I've tried doing this a couple of ways so far:
Using a cron.yaml file:
Didn't touch anything else - just added a cron.yaml file to my project root folder
version: 1
cron: - name: "test"
url: "http://website.com/workers/test"
schedule: "*/10 * * * *"
Using a container command and a separate cron.txt file:
Added this line in my .ebextensions/development.config file
05_some_cron:
command: "cat .ebextensions/crontab.txt > /etc/cron.d/crontab && chmod 644 /etc/cron.d/crontab"
leader_only: true
and in .ebextensions/crontab.txt
*/10 * * * * source /opt/python/run/venv/bin/activate && python mysite/manage.py test
The app deploys successfully in both cases.
Manually (in a browser) going to http://website.com/workers/test has
the intended outcome (in the first case).
Adding source /opt/python/run/venv/bin/activate && python mysite/manage.py test as a management command runs the correct script once on deploying.
The logs do not show any GETS on that url.
What am I doing wrong? Am I missing some step of the process or some setup step on EBS?
Also what is the best ways to run cron jobs for django applications hosted on EBS? - django apps can run management commands either from the command line as in attempt 2 or by extending a GET or POST url as in attempt 1.

2018 Update using Django 2.0.6 + AWS ElasticBeanstalk + Cronjob
I found I needed to export the AWS environment variables using source /opt/python/current/env to prevent manage.py from throwing the error "The SECRET_KEY setting must not be empty".
This is because I had placed my Django Secret key in the os.environ for beanstalk which it seems is not accessible by shell/cron by default.
See https://forums.aws.amazon.com/thread.jspa?threadID=108465
My final cron job .txt script was as follows. (process_emails is my own django management function, myjob.log is where the output of the function call is logged).
*/15 * * * * source /opt/python/run/venv/bin/activate && cd /opt/python/current/app/ && source /opt/python/current/env && python manage.py process_emails >> /var/log/myjob.log 2>&1

This works for me in Django 1.10.6 + AWS ElasticBeanstalk + Cronjob
* * * * * source /opt/python/run/venv/bin/activate && cd /opt/python/current/app/ && python manage.py do_your_thing > $HOME/cron-sqs.log 2>&1
do_your_thing is a management command

Related

Cron job Django Elastic Beanstalk

I am trying to set a cron job on my project to send a email after 15 minutes.
This is in django.config
04_cronjb:
command: "cat .ebextensions/cron-linux.config > /etc/cron.d/crontab && chmod 644 /etc/cron.d/crontab"
leader_only: true
This is my cron-linux.config file
files:
"/etc/cron.d/mycron":
mode: "000644"
owner: root
group: root
content: |
* * * * * source /opt/python/current/env && python /opt/python/current/app/manage.py cronjb
It all deploys successfully but i am not receiving any email.
Cronjb script is working i have tested it. So the error is one these two files.
Is there some mistake in it?
Your 04_cronjb copies entire content of cron-linux.config into crontab. This is sadly incorrect.
Instead you should do as shown here. This includes putting all the bash commands you want to execute in a custom script called, e.g., myscript.sh and then adding myscript.sh to cron only.

Cron with elastic beanstalk and symfony - not working

I'm a newbie developer and I am messing around with Symfony and AWS. I have a Symfony app deployed via elastic beanstalk and I am trying to add a cron job every minute.
Basically I want the cron job to execute a Symfony command I created : update-pings, that is supposed to update some fields in my RDS DB.
I'm using a 02-crons.config file in my ./ebextensions directory that looks like this (greatly inspired from AWS documentation https://aws.amazon.com/fr/premiumsupport/knowledge-center/cron-job-elastic-beanstalk/ ) :
files:
"/etc/cron.d/mycron":
mode: "000644"
owner: root
group: root
content: |
* * * * * root php /var/app/current/bin/console app:update-pings
commands:
remove_old_cron:
command: "rm -f /etc/cron.d/mycron.bak"
Well, first of all, this is not working, the fields in the DB are not updating. So i connected in SSH to my EC2 instance to see what's happening.
The mycron file in /etc/cron.d/was correctly created with * * * * * root php /var/app/current/bin/console app:update-pingsinside of it.
Now, I tried replacing the content of this file by a simple * * * * * root echo test >> /tmp/cron_temp, this is working perfectly, a line with 'test' is added in my cron_temp file every minute
I tried running the initial command php /var/app/current/bin/console app:update-pingsmanually on the instance, works perfectly as ec2-user
When I switch to root user using sudo su -, the command still works. But when I try to run it with sudo, it fails, not sure if that's any relevant :
In EnvVarProcessor.php line 171:
Environment variable not found: "DATABASE_DBNAME".
I believe there is something I'm missing with linux users/rights, but I'm a total newbie with that and cannot find what's going on
Any ideas ? Thanks a lot in advance ! :)
Edit 1 : I'm know logging the sterr of the commande into a file and I get the same environment variable error as when I try to run the command manually with sudo
In EnvVarProcessor.php line 171:
Environment variable not found: "DATABASE_DBNAME".
The command works fine without sudo though

Run django management command on elastic beanstalk worker instance using crontab

I have written few django management commands that I want to run hourly, daily and weekly basis. I'm using Elastic Beanstalk and created a worker instance where the code is deployed. Can someone help me how to run the django management command with crontab using elastic beanstalk. Thanks
Here is my management command:
python manage.py command_name
Please help me write the container_command in .ebextensions/django.config file for crontab that will schedule the command on hourly basis. Thanks
Any help?
in .ebextensions create a file crontab.txt
# Set the cron to run with utf8 encoding
PYTHONIOENCODING=utf8
0 1 * * * root source /opt/python/current/env && nice /opt/python/current/app/manage.py command_name
# this file needs a blank space as the last line otherwise it will fail
This is a crontab file that will run command_name at 1 am every day. You'll need to get you path right which depends on your file structure.
In one of your config files include:
02_cron_job:
command: "cp .ebextensions/crontab.txt /etc/cron.d/my_cron_jobs && chmod 644 /etc/cron.d/my_cron_jobs"
leader_only: true
This is going to copy your cron file into the right place on the ec2 instance so the job runs.

How to set environment variables in Amazon Elastic Beanstalk when running a cron job (Node.js)?

I have been working on configuring a cron job while deploying an environment on Elastic Beanstalk. The purpose of the job is to run every 30 minutes and execute a Node.js script that processes SMS schedules in our system; when a schedule is ready, an SMS message is sent via the Twilio api.
Unfortunately, Node.js environments don't have the file /opt/elasticbeanstalk/support/envvars containing the environment variables defined (Maybe I am missing something?).
To work around this issue, I am loading the environment variables from /opt/elasticbeanstalk/bin/get-config in Python and then executing my Node.js script.
Everything is working as expected so hopefully this can help someone in the same situation; however, I am wondering if there is a better way to accomplish this... Open to suggestions.
In my .ebextensions folder I have the config file for the cron:
files:
"/etc/cron.d/process-sms-schedules":
mode: "000644"
owner: root
group: root
content: |
*/30 * * * * root /usr/local/bin/process-sms-schedules.sh
"/usr/local/bin/process-sms-schedules.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
# Execute script to process schedules
python /var/app/current/process-sms-schedules.py > /var/log/process-sms-schedules.log
exit 0
commands:
remove_old_cron:
command: "rm -f /etc/cron.d/*.bak"
Here is the Python script that gets executed as part of the cron job:
#!/usr/bin/env python
import os
import subprocess
from subprocess import Popen, PIPE, call
import simplejson as json
envData = json.loads(Popen(['/opt/elasticbeanstalk/bin/get-config', 'environment'], stdout = PIPE).communicate()[0])
for k, v in envData.iteritems():
os.environ[k] = v
call(["babel-node", "/var/app/current/process-sms-schedules.js"])
Thanks in advance for any feedback.
References
Cron Job Elastic Beanstalk
How to set environment variable in Amazon Elastic Beanstalk (Python)
I had this issue while trying to execute a PHP file inside an Elastic Beanstalk environment. In particular I was trying to execute wp-cron.php file.
Basically you should write a cron job like this:
/etc/cron.d/wp-cronjob :
PATH=/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/home/ec2-user/.local/bin:/home/ec2-user/bin
*/5 * * * * ec2-user . /opt/elasticbeanstalk/support/envvars; /usr/bin/php /var/www/html/wp-cron.php > /dev/null 2>&1
Explained:
Every 5 minutes executes commands as user "ec2-user": '*/5 * * * * ec2-user'
Loads elasticbeanstalk environment variables: '. /opt/elasticbeanstalk/support/envvars;'.
Do not print any output: '> /dev/null 2>&1'.
Also:
.ebextensions/wp-cronjob.txt
# Load paths
PATH=/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/home/ec2-user/.local/bin:/home/ec2-user/bin
# Every 5 minutes executes commands as user "ec2-user": '*/5 * * * * ec2-user'.
# Loads elasticbeanstalk environment variables: '. /opt/elasticbeanstalk/support/envvars;'.
# Do not print any output: '> /dev/null 2>&1'.
*/5 * * * * ec2-user . /opt/elasticbeanstalk/support/envvars; /usr/bin/php /var/www/html/wp-cron.php > /dev/null 2>&1
.ebextensions/cronjob.config
container_commands:
add_wp_cronjob:
command: "cat .ebextensions/wp-cronjob.txt > /etc/cron.d/wp-cronjob && chmod 644 /etc/cron.d/wp-cronjob"
leader_only: true
commands:
remove_old_cron:
command: "rm -f /etc/cron.d/*.bak"
Maybe is not the same for a Node.js environment but I am pretty sure they are similar.

.config file is being ignored by elastic beanstalk AWS

I have a web app running on elastic beanstalk. For some reason I was able to install the composer files in order to run my laravel app. The problem is that no other config file works. I have put newrelic.config into the .ebextensions/ directory, but that file got ignored.
I recently tried to create a cron job using this, AWS Elastic Beanstalk, running a cronjob, but it is not working.
Example of a .config file:
container_commands:
01_some_cron_job:
command: "cat .ebextensions/some_cron_job.txt > /etc/cron.d/some_cron_job && chmod 644 /etc/cron.d/some_cron_job"
leader_only: true
When I ssh into the ec2 instance, there is no such directory as some_cron_job.
The source gets committed to beanstalk, but beanstalk is not running the commands.
How can I make beanstalk acknowledge the .config files. Fixing this cronjob will also fix installing new relic, because both configs are being ignored and I do not know why.
Try putting it in commands section. It is more of a server command than a container command.
commands:
01_some_cron_job:
command: "cat .ebextensions/some_cron_job.txt > /etc/cron.d/some_cron_job && chmod 644 /etc/cron.d/some_cron_job"
leader_only: true
I had similar issues as well using container_commands and files, however, I deferred to the files event and it worked like a charm. My specific setup is as below.
.ebextensions/cron.config
files:
"/etc/cron.d/mycronstuff":
mode: "000644"
owner: root
group: root
content: |
# Run daily job every 8 hours
0 */8 * * * root curl -i XXXXXXXXXX
# Run nightly job at 2AM (8AM UTC)
0 8 * * * root curl -i XXXXXXXXX
Update 2019:
You must use a cron.yaml file on your project directory.
inside the file you can mention:
version: 1
cron:
- name: "task1"
url: "/scheduled"
schedule: "* * * * *"