I have a Django project running on AWS using Elastic Beanstalk. It can have between 1 and 6 instances running.
I load a crontab file to run some management/commands overnight. I have this config file:
container_commands:
01_cron_job:
command: "cp .ebextensions/crontab.txt /etc/cron.d/my_cron_jobs && chmod 644 /etc/cron.d/my_cron_jobs"
#leader_only: true
The file copied across looks like:
# Set the cron to run with utf8 encoding
PYTHONIOENCODING=utf8
# Specify where to send email
MAILTO="me#gmail.com"
1 0 * * * root source /opt/python/current/env && nice /opt/python/current/app/src/manage.py clearsessions
15 0 * * * root source /opt/python/current/env && nice /opt/python/current/app/src/manage.py update_summary_stats >> /opt/python/log/update_summary_stats.log 2>&1
# this file needs a blank space as the last line otherwise it will fail
Within the config file, if I set leader_only to false then the command doesn't run if the leader instance gets deleted at some point (for example because another instance was added during peak times and the leader deleted when it quietened). If I set leader_only to true then the crontab commands run on every instance.
What is the best way to set up crontab on AWS Elastic Beanstalk to only run once irrespective of the number of instances? Thank you
You could create a lock file (perhaps locally on a shared EFS mount, or externally using a service such as DynamoDB with Transactional Consistency or S3).
When your application creates this lock file it could then continue as normal, however should the file exist you would skip the script.
By doing this it reduces the chance of a collision, however I would also recommend adding some jitter to the start of the script (add a sleep for a random amount of seconds) to reduce the chance further that the scripts will attempt to create this lockfile at the same time.
I'm having trouble getting my cron jobs to execute.
Setup:
Django - 1.9
Elastic beanstalk - 64bit Amazon Linux 2016.03 v2.1.3 running Python 3.4
I've tried doing this a couple of ways so far:
Using a cron.yaml file:
Didn't touch anything else - just added a cron.yaml file to my project root folder
version: 1
cron: - name: "test"
url: "http://website.com/workers/test"
schedule: "*/10 * * * *"
Using a container command and a separate cron.txt file:
Added this line in my .ebextensions/development.config file
05_some_cron:
command: "cat .ebextensions/crontab.txt > /etc/cron.d/crontab && chmod 644 /etc/cron.d/crontab"
leader_only: true
and in .ebextensions/crontab.txt
*/10 * * * * source /opt/python/run/venv/bin/activate && python mysite/manage.py test
The app deploys successfully in both cases.
Manually (in a browser) going to http://website.com/workers/test has
the intended outcome (in the first case).
Adding source /opt/python/run/venv/bin/activate && python mysite/manage.py test as a management command runs the correct script once on deploying.
The logs do not show any GETS on that url.
What am I doing wrong? Am I missing some step of the process or some setup step on EBS?
Also what is the best ways to run cron jobs for django applications hosted on EBS? - django apps can run management commands either from the command line as in attempt 2 or by extending a GET or POST url as in attempt 1.
2018 Update using Django 2.0.6 + AWS ElasticBeanstalk + Cronjob
I found I needed to export the AWS environment variables using source /opt/python/current/env to prevent manage.py from throwing the error "The SECRET_KEY setting must not be empty".
This is because I had placed my Django Secret key in the os.environ for beanstalk which it seems is not accessible by shell/cron by default.
See https://forums.aws.amazon.com/thread.jspa?threadID=108465
My final cron job .txt script was as follows. (process_emails is my own django management function, myjob.log is where the output of the function call is logged).
*/15 * * * * source /opt/python/run/venv/bin/activate && cd /opt/python/current/app/ && source /opt/python/current/env && python manage.py process_emails >> /var/log/myjob.log 2>&1
This works for me in Django 1.10.6 + AWS ElasticBeanstalk + Cronjob
* * * * * source /opt/python/run/venv/bin/activate && cd /opt/python/current/app/ && python manage.py do_your_thing > $HOME/cron-sqs.log 2>&1
do_your_thing is a management command
I am running an amazon web services web server tier elastic beanstalk application, and I would like to run a script every 3 hours. I've googled around quite a bit, and the answer for this seems to be to create a ".ebextensions" folder, with a .config file that looks something like this:
commands:
01_cronjobs:
leader_only: true
command: "cat .ebextensions/etss.txt > /etc/cron.d/crontab_etss && chmod 644 /etc/cron.d/crontab_etss"
Unfortunately, when I do this, the application fails to deploy with the error:
[Instance: xxx] Command failed on instance. Return code: 1 Output: cat: .ebextensions/etss.txt: No such file or directory. command 01_cronjobs in .ebextensions/config.config failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
I have an etss.txt file in my .ebextensions folder that looks like this:
0 */3 * * * /usr/bin/php download_etss_tar.php > /dev/null
#blank newline here
I also have tried putting it in the root and referencing it there. Why can't the cron find the file that I know exists in the zip? What am I doing wrong here?
Furthermore, I also tried using the full file name, to no avail:
Command failed on instance. Return code: 1 Output: cat: /var/www/html/.ebextensions/etss.txt: No such file or directory. command 01_cronjobs in .ebextensions/etss.config failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
I have a web app running on elastic beanstalk. For some reason I was able to install the composer files in order to run my laravel app. The problem is that no other config file works. I have put newrelic.config into the .ebextensions/ directory, but that file got ignored.
I recently tried to create a cron job using this, AWS Elastic Beanstalk, running a cronjob, but it is not working.
Example of a .config file:
container_commands:
01_some_cron_job:
command: "cat .ebextensions/some_cron_job.txt > /etc/cron.d/some_cron_job && chmod 644 /etc/cron.d/some_cron_job"
leader_only: true
When I ssh into the ec2 instance, there is no such directory as some_cron_job.
The source gets committed to beanstalk, but beanstalk is not running the commands.
How can I make beanstalk acknowledge the .config files. Fixing this cronjob will also fix installing new relic, because both configs are being ignored and I do not know why.
Try putting it in commands section. It is more of a server command than a container command.
commands:
01_some_cron_job:
command: "cat .ebextensions/some_cron_job.txt > /etc/cron.d/some_cron_job && chmod 644 /etc/cron.d/some_cron_job"
leader_only: true
I had similar issues as well using container_commands and files, however, I deferred to the files event and it worked like a charm. My specific setup is as below.
.ebextensions/cron.config
files:
"/etc/cron.d/mycronstuff":
mode: "000644"
owner: root
group: root
content: |
# Run daily job every 8 hours
0 */8 * * * root curl -i XXXXXXXXXX
# Run nightly job at 2AM (8AM UTC)
0 8 * * * root curl -i XXXXXXXXX
Update 2019:
You must use a cron.yaml file on your project directory.
inside the file you can mention:
version: 1
cron:
- name: "task1"
url: "/scheduled"
schedule: "* * * * *"
I´m running an EC2 instance through AWS Elastic Beanstalk. Unfortunately it has the incorrect timezone - it´s 2 hours earlier than it should be, because timezone is set to UTC. What I need is GMT+1.
Is there a way to set up the .ebextensions configuration, in order to force the EC2 instance to use the right timezone?
Yes, you can.
Just create a file /.ebextensions/00-set-timezone.config with following content
commands:
set_time_zone:
command: ln -f -s /usr/share/zoneinfo/Australia/Sydney /etc/localtime
This is assuming your are using default Amazon Linux AMI image. If you use some other Linux distribution, just change the command to whatever it requires to set timezone in that Linux.
This is a response from the aws Support Business and this works!
---- Original message ----
How can I change the timezone of an enviroment or rather to the instances of the enviroment in Elastic Beasntalk to UTC/GMT -3 hours (Buenos Aires, Argentina)?
I´m currently using Amazon Linux 2016.03. Thanks in advance for your help.
Regards.
---------- Response ----------
Hello,Thank you for contacting AWS support regarding modifying your Elastic Beanstalk instances time zone to use UTC/GMT -3 hours (Buenos Aires, Argentina), please see below on steps on how to perform this modification.
The below example shows how to modify timezone for Elastic Beanstalk environment using .ebextensions for Amazon Linux OS:
Create .ebextensions folder in the root of your application
Create a .config file for example 00-set-timezone.config file and add the below content in yaml formatting.
container_commands:
01changePHP:
command: sed -i '/PHP_DATE_TIMEZONE/ s/UTC/America\/Argentina\/Buenos_Aires/' /etc/php.d/environment.ini
01achangePHP:
command: sed -i '/aws.php_date_timezone/ s/UTC/America\/Argentina\/Buenos_Aires/' /etc/php.d/environment.ini
02change_AWS_PHP:
command: sed -i '/PHP_DATE_TIMEZONE/ s/UTC/America\/Argentina\/Buenos_Aires/' /etc/httpd/conf.d/aws_env.conf
03php_ini_set:
command: sed -i '/date.timezone/ s/UTC/America\/Argentina\/Buenos_Aires/' /etc/php.ini
commands:
01remove_local:
command: "rm -rf /etc/localtime"
02link_Buenos_Aires:
command: "ln -s /usr/share/zoneinfo/America/Argentina/Buenos_Aires /etc/localtime"
03restart_http:
command: sudo service httpd restart
Deploy application to Elastic Beanstalk including the .ebextensions and the timezone will change as per the above.
I hope that helps
Regards!
If you are running windows in your eb environment...
.
create a folder named .ebextensions in the root of your project..
inside that folder create a file named timezone.config
in that file add the following :
commands:
set_time_zone:
command: tzutil /s "Central Standard Time"
set the time zone as needed
screenshot
I'm using custom .ini file in php.d folder along with regular recommendations from http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#change_time_zone:
The sed command inserts (rewrites) only the first line of /etc/sysconfig/clock, since the second line (UTC=true) should be left alone, per the above AWS documentation.
# .ebextensions/02-timezone.config
files:
/etc/php.d/webapp.ini:
mode: "000644"
owner: root
group: root
content: |
date.timezone="Europe/Amsterdam"
commands:
01_set_ams_timezone:
command:
- sed -i '1 s/UTC/Europe\/Amsterdam/g' /etc/sysconfig/clock
- ln -sf /usr/share/zoneinfo/Europe/Amsterdam /etc/localtime
Changing the time zone of EC2 with Elastic Beanstalk is simple:
Create a .ebextensions folder in the root
Add a file with filename end with .config (timezone.config)
Inside the file
container_commands:
time_zone:
command: ln -f -s /usr/share/zoneinfo/America/Argentina/Buenos_Aires /etc/localtime
Then you have done.
Note that the container_commands is different from commands, from the document it states:
commands run before the application and web server are set up and
the application version file is extracted.
That's the reason of your time zone command doesn't work because the server hasn't started yet.
container_commands run after the application and web server have been
set up and the application version file has been extracted, but before
the application version is deployed.
If you are runing a java/Tomcat container, just put the JVM Option on the configuration.
-Duser.timezone=America/Sao_Paulo
Possibles values: timezones
Moving to AWS Linux 2 was challenging. It took me a while to work out how to do this easily in .ebextensions.
I wrote the simple solution in another stackoverflow question .. but for anyone needing instant gratification .. add the following commands into the file .ebextensions/xxyyzz.config:
container_commands:
01_set_bne:
command: "sudo timedatectl set-timezone Australia/Brisbane"
command: "sudo systemctl restart crond.service"
These workarounds only fixes the timezone for applications. But when you have any system services like a cron run it looks at the /etc/sysconfig/clock and that is always UTC. If you tail the cron logs or aws-sqsd logs would will notice timestamps are still 2hrs behind - in my case. And a change to the clock setting would need a reboot into order to take effect - which is not an option to consider should you have autoscaling in place or should you want to use ebextensions to change the system clock's config.
Amazon is aware of this issue and I dont think they have resolved it yet.
If your EB application is using the Java/Tomcat container, you can add the JVM timezone Option to the Procfile configuration. Example:
web: java -Duser.timezone=Europe/Berlin -jar application.jar
Make sure to add all configuration options before the -jar option, otherwise they are ignored.
in the .ebextensions added below for PHP
container_commands:
00_changePHP:
command: sed -i '/;date.timezone =/c\date.timezone = \"Australia/Sydney\"' /etc/php.ini
01_changePHP:
command: sed -i '/date.timezone = UTC/c\date.timezone = \"Australia/Sydney\"' /etc/php.d/aws.ini
02_set_tz_AEST:
command: "sudo timedatectl set-timezone Australia/Sydney"
command: "sudo systemctl restart crond.service"
commands:
01remove_local:
command: "rm -rf /etc/localtime"
02change_clock:
command: sed -i 's/\"UTC\"/\"Australia\/Sydney\"/g' /etc/sysconfig/clock
03link_Australia_Sydney:
command: "ln -f -s /usr/share/zoneinfo/Australia/Sydney /etc/localtime"
cwd: /etc
Connect AMI(amazon linux instance) via putty or ssh and execute the commands below;
sudo rm /etc/localtime
sudo ln -sf /usr/share/zoneinfo/Europe/Istanbul /etc/localtime
sudo reboot
Explanation of the procedure above is simply;
remove localtime,
update the timezone,
reboot
Please notify that I've changed my timezone to Turkey's localtime, you can find your timezone by listing zoneinfo directory with the command below;
ls /usr/share/zoneinfo
or just check timezone abbrevetaions via wikipedia;
http://en.wikipedia.org/wiki/Category:Tz_database
You can also check out the related Amazon AWS documentation;
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html
Note: I'm not sure that if this is the best practice or not (probably not), however I've applied the procedure I've written above and it's working for me.