Elastic BeanStalk EC2 instance's log uses up whole disk space - amazon-web-services

I have an Elastic BeanStalk environment where I run my application on 1 EC2 instance. I've added load balancer, when I configured the environment initially, but since then I set it only use 1 instance.
Application run within container apparently produces quite a lot of logs - after several days they use up whole disk space and then application crash. Health check drops to severe.
I see that terminating instance manually helps - environment removes old instance and creates a new one that works (until it fills up the whole disk again).
What are my options? A script that regularly cleans up logs? Some log rotation? Trigger that reboots instance when disk is nearly full?
I do not write anything to file myself - my application only log to std out and std err, so writing to file is done by EC2/EBS wrapper. (I deploy the application as a ZIP containing a JAR, a bash script and Procfile if that is relevant).

By default EB will rotate some of the logs produced by the Docker containers, but not all of them. After contacting support on this issue I received the following helpful config file, to be placed in the source path .ebextensions/liblogrotate.config:
files:
"/etc/logrotate.elasticbeanstalk.hourly/logrotate.elasticbeanstalk.containers.conf":
mode: "00644"
owner: "root"
group: "root"
content: |
/var/lib/docker/containers/*/*.log {
size 10M
rotate 5
missingok
compress
notifempty
copytruncate
dateext
dateformat %s
olddir /var/lib/docker/containers/rotated
}
"/etc/cron.hourly/cron.logrotate.elasticbeanstalk.containers.conf":
mode: "00755"
owner: "root"
group: "root"
content: |
#!/bin/sh
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/logrotate.elasticbeanstalk.hourly/logrotate.elasticbeanstalk.containers.conf
container_commands:
create_rotated_dir:
command: mkdir -p /var/lib/docker/containers/rotated
test: test ! -d /var/lib/docker/containers/rotated
99_cleanup:
command: rm /etc/cron.hourly/*.bak /etc/logrotate.elasticbeanstalk.hourly/*.bak
ignoreErrors: true
What this does is install an additional log rotation configuration and cron task for the /var/lib/docker/containers/*/*.log files which are the ones not automatically rotated on EB.
Eventually, however, the rotated logs themselves will fill up the disk if the host lives long enough. For this, you can add shred in the list of logrotation options (along side compress notifempty etc).
(However, I'm not sure if the container logs that are already configured for rotation are set to be shredded, probably not - so those may accumulate too and require modification of the default EB log rotation config. Not sure how to do that yet. But the above solution in most cases would be sufficient since hosts typically do not live that long. The volume of logging and lifetime of your containers may force you to go even further.)

Logrotation is the way forward. You can create a configuration file in `/etc/logrotate.d/' where you state your options in order to avoid having large log files.
You can read more about the configurations here https://linuxconfig.org/setting-up-logrotate-on-redhat-linux
A sample configuration file would look something like this:
/var/log/your-large-log.log {
missingok
notifempty
compress
size 20k
daily
create 0600 root root
}
You can also test the new configuration file from the cli by running the follow:
logrotate -d [your_config_file]
This will test if the log rotation will be successful or not but only in debugging mode, therefore the log file will not be actually rotated.

Related

Starting NginX with my modified nginx.conf on ECS

I have an environment in AWS with an ECS cluster, an EFS source and some services running on this cluster.
One of my services is the NginX web server which I use to serve our site and our services. As a solution to keep some sensitive and static configuration files we have chosen the EFS service. So, each service creates a volume from this EFS and mount it every time a container starts.
The problem is with NginX. I want to store my nginx.conf file into an EFS folder and after the NginX service starts, we want the container to copy this file at /etc/nginx/ folder in order for my NginX server to start with my configuration.
I've tried to build my own image including my configuration with success but this is not what we want.That means that we should build a new image every time we want to change a line on nginx.conf.
I've tried to create a script to run every time the container starts and copy my configuration but i didn't manage to make it play on ECS. Either the NginX failed to reload, either the syntax is wrong, either the file is not available.
#!/bin/bash
cp /efs/nginx.conf /etc/nginx/
nginx -s reload
Ι considered to find out how to create a cron job to run every X minutes and copy my nginx.conf to etc/nginx but this seems to be a stupid approach.
I made like 60 different task definitions revisions in order to find out how this CMD Environment option works on ECS. Of course the most of them has to do with the syntax and i get bach errors like "invalid option: bash" or "invalid option: /tmp/1.sh" etc
Samples:
1.Command ["cp","/efs/nginx.conf /etc/nginx/"]
2.Entry point ["nginx","-g","daemon off;"]
Command ["cp /efs/nginx.conf /etc/nginx/"]
Entry point: ["nginx","-g","daemon off"]
Command: ["/bin/sh","cp","/efs/nginx.conf/","/etc/nginx/"]
Command ["[\"cp\"","\"/efs/nginx.conf\"","\"/etc/nginx/\"]","[\"nginx\"","\"-g\"","\"daemon off;\"]"]
Command ["cp /efs/nginx.conf /etc/nginx/","nginx -g daemon off;"]
Command ["cp","/efs/nginx.conf /etc/nginx/","nginx -g daemon off;"]
-
Does anyone knows or does anyone already implement this solution on ECS?
To replace /etc/nginx/nginx.conf with a modified one from a binded volume?
Thanks in advance
SOLUTION:
As I mention at my question above, I'd like to use a static nginx.conf file, which will be into an EFS folder, into my nginx service container.
My task definition is simple like this
FROM nginx
EXPOSE 80
RUN mkdir /etc/nginx/html
Through ECS task definition I create a volume and then a mounting point which is an easy process and works fine. The problem was in the entrypoint field which supposed to include my script's directory and to my script itself.
At ECS task definition Environment entrypoint field i putted
sh,-c,/efs/docker-cmd-nginx.sh
and my script is just the following
#!/bin/dash
cp /efs/nginx.conf /etc/nginx/ &&
nginx -g "daemon off;"
PS: The problem probably was at:
my script which I didn't use double quotes at the daemon off; part but I was using double quotes on the whole line nginx -g daemon off;
my script was trying to reload nginx which was not even running yet.
my attempt to put the commands seperately at my task's entrypoint was wrong, syntax-wise for sure and maybe strategy-wise as well.

How to add my own log to log rotation/S3 backup on Amazon Elastic Beanstalk?

We have a PHP application that lives on Amazon Elastic Beanstalk. The application has log rotation and backup to S3 enabled. Apache access and error logs do get properly rotated and backed up every hour.
However the application also creates its own log file. I want to do the same thing with it - every hour it should be rotated and backed up to S3. Following the instructions here I created the following file:
.ebextensions/publish-logs.config
files:
"/opt/elasticbeanstalk/tasks/publishlogs.d/cloud-init.conf" :
mode: "000755"
owner: root
group: root
content: |
/var/app/current/log/*.log
Then I uploaded the new version to Amazon.
The results - I see that the log file was backed up to S3 ONCE at the very first rotation. And it was not gzipped, just copied. After that, nothing. No new backups to S3. No rotation. When downloading bundle logs, the file is there, in its full glory of about 80MB now (accumulated over several days).
Amazon's documentation is pretty sparse. But it does say that:
When you configure your application's log files for log rotation, the application doesn't need to create copies of log files. Elastic Beanstalk configures logrotate to make a copy of your application's log files for each rotation.
What have I done wrong?
To do that you need to configure logrotate, and that is a bit tricky. Since highly depends on the instance you are using. But let me try. Add this two files to your configuration. First creates configuration for logrotate, and second configures cron to run logrotate with that configuration.
files:
"/etc/logrotate.d/logrotate.elasticbeanstalk.php.conf":
mode: "000655"
owner: root
group: root
content: |
/var/app/current/log/*.log {
rotate 14
size 100M
daily
compress
delaycompress
}
"/etc/cron.daily/cron.logrotate.elasticbeanstalk.php.conf":
mode: "000655"
owner: root
group: root
content: |
#!/bin/sh
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/logrotate.d/logrotate.elasticbeanstalk.php.conf
/sbin/service awslogs restart
Give it a try. If fail - please provide AMI ID you are using

Mounting Docker volumes - is the writeable layer used?

I'm new to docker and I'm trying to process a lot of data on AWS. Right now, the input for the scripts I want to run using my parent image is about 20G.
First, I tried just copying the data into my image on the writeable layer (using COPY), but then I got the error
Sending build context to Docker daemon 20.53 GB
Error response from daemon: Error processing tar file(exit status 1): write /Input/dbsnp_138.b37.vcf: no space left on device
So I thought that 20G would be too much to just store on my writeable layer.
Then I looked at mounting a volume on the docker host (using VOLUME), but wouldn't that also need to be written on the writeable layer first? Wouldn't that also give me the same error?
So the way docker build works is this, when you use something like below
docker build .
It will send the content of the current directory to the Docker daemon. So if you are sending a context of 20GB then an additional 20GB+ of free space would be needed.
When you mount volumes on a running image, it would actually mount your folder, so no extra space would be used. But if delete the directory from inside the container, the file on the host will also be deleted.
So mounting the directory is a possible solution. Also if there are files in your current directory which you would not be interested in sending to docker daemon as a context, then you should use a .dockerignore to specify which files to ignore

Dockerfile; docker build volumes: changes to volume via ADD or COPY are not discarded

I have a dockerfile with something like:
VOLUME /tmp/space
ADD local/directory/ /tmp/space/
RUN cp /tmp/space/somescript.sh /opt/real/space/
After the container is built and I get an interactive shell I notice that the /tmp/space still contains the data from local/directory.
If I add a RUN rm -rf /tmp/space/* to the end of the dockerfile and get shell access. The data is still there in /tmp/space/.
As a result, I'm left making a running container using the same volume and then committing the changed container to an updated image.
Is there a method to, during the build, have a temporarily loaded volume that doesn't bloat the resulting image?
The goal is to use source files and scripts to perform some actions during a build. The layers of docker end up recording a duplicate of the COPY/ADD step with the RUN step. So it would be better to COPY the data into a space that isn't recorded as a layer then as a single RUN step cp stuff && execute scripts to save on space.
I am not sure what you are trying to do here:
VOLUME /tmp/space - this declares a mount point and maps /tmp/space on your container to a directory on the host
ADD /local/directory /tmp/space - I think you are attempting to copy a local directory from your container to your mounted volume
RUN cp /tmp/space/* /opt/real/space/ - Are you trying to copy from your volume to your host?
After adding a VOLUME directive in a dockerfile, what happens is that the folder in the container /tmp/space is mapped to a folder on the host, say /hosttmp/hostspace. You can find out what this is on the host by running the command -
$ docker inspect -f {{.Volumes}} <your_container_name>
In order to prevent corruption of data in /hosttmp/hostspace, once a VOLUME is declared in the dockerfile, you cannot play around with the contents.
I would recommend reading this article as it explains the rather confusing docker volume concept
http://container-solutions.com/understanding-volumes-docker/

AWS Elastic Beanstalk, running a cronjob

I would like to know if there is a way to setup a cronjob/task to execute every minute. Currently any of my instances should be able to run this task.
This is what I have tried to do in the config files without success:
container_commands:
01cronjobs:
command: echo "*/1 * * * * root php /etc/httpd/myscript.php"
I'm not really sure if this is the correct way to do it
Any ideas?
This is how I added a cron job to Elastic Beanstalk:
Create a folder at the root of your application called .ebextensions if it doesn't exist already. Then create a config file inside the .ebextensions folder. I'll use example.config for illustration purposes. Then add this to example.config
container_commands:
01_some_cron_job:
command: "cat .ebextensions/some_cron_job.txt > /etc/cron.d/some_cron_job && chmod 644 /etc/cron.d/some_cron_job"
leader_only: true
This is a YAML configuration file for Elastic Beanstalk. Make sure when you copy this into your text editor that your text editor uses spaces instead of tabs. Otherwise you'll get a YAML error when you push this to EB.
So what this does is create a command called 01_some_cron_job. Commands are run in alphabetical order so the 01 makes sure it's run as the first command.
The command then takes the contents of a file called some_cron_job.txt and adds it to a file called some_cron_job in /etc/cron.d.
The command then changes the permissions on the /etc/cron.d/some_cron_job file.
The leader_only key ensures the command is only run on the ec2 instance that is considered the leader. Rather than running on every ec2 instance you may have running.
Then create a file called some_cron_job.txt inside the .ebextensions folder. You will place your cron jobs in this file.
So for example:
# The newline at the end of this file is extremely important. Cron won't run without it.
* * * * * root /usr/bin/php some-php-script-here > /dev/null
So this cron job will run every minute of every hour of every day as the root user and discard the output to /dev/null. /usr/bin/php is the path to php. Then replace some-php-script-here with the path to your php file. This is obviously assuming your cron job needs to run a PHP file.
Also, make sure the some_cron_job.txt file has a newline at the end of the file just like the comment says. Otherwise cron won't run.
Update:
There is an issue with this solution when Elastic Beanstalk scales up your instances. For example, lets say you have one instance with the cron job running. You get an increase in traffic so Elastic Beanstalk scales you up to two instances. The leader_only will ensure you only have one cron job running between the two instances. Your traffic decreases and Elastic Beanstalk scales you down to one instance. But instead of terminating the second instance, Elastic Beanstalk terminates the first instance that was the leader. You now don't have any cron jobs running since they were only running on the first instance that was terminated. See the comments below.
Update 2:
Just making this clear from the comments below:
AWS has now protection against automatic instance termination. Just enable it on your leader instance and you're good to go. – Nicolás Arévalo Oct 28 '16 at 9:23
This is the official way to do it now (2015+). Please try this first, it's by far easiest method currently available and most reliable as well.
According to current docs, one is able to run periodic tasks on their so-called worker tier.
Citing the documentation:
AWS Elastic Beanstalk supports periodic tasks for worker environment tiers in environments running a predefined configuration with a solution stack that contains "v1.2.0" in the container name. You must create a new environment.
Also interesting is the part about cron.yaml:
To invoke periodic tasks, your application source bundle must include a cron.yaml file at the root level. The file must contain information about the periodic tasks you want to schedule. Specify this information using standard crontab syntax.
Update: We were able to get this work. Here are some important gotchas from our experience (Node.js platform):
When using cron.yaml file, make sure you have latest awsebcli, because older versions will not work properly.
It is also vital to create new environment (at least in our case it was), not just clone old one.
If you want to make sure CRON is supported on your EC2 Worker Tier instance, ssh into it (eb ssh), and run cat /var/log/aws-sqsd/default.log. It should report as aws-sqsd 2.0 (2015-02-18). If you don't have 2.0 version, something gone wrong when creating your environment and you need to create new one as stated above.
Regarding jamieb's response, and as alrdinleal mentions, you can use the 'leader_only' property to ensure that only one EC2 instance runs the cron job.
Quote taken from http://docs.amazonwebservices.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html:
you can use leader_only. One instance is chosen to be the leader in an Auto Scaling group. If the leader_only value is set to true, the command runs only on the instance that is marked as the leader.
Im trying to achieve a similar thing on my eb, so will update my post if I solve it.
UPDATE:
Ok, I now have working cronjobs using the following eb config:
files:
"/tmp/cronjob" :
mode: "000777"
owner: ec2-user
group: ec2-user
content: |
# clear expired baskets
*/10 * * * * /usr/bin/wget -o /dev/null http://blah.elasticbeanstalk.com/basket/purge > $HOME/basket_purge.log 2>&1
# clean up files created by above cronjob
30 23 * * * rm $HOME/purge*
encoding: plain
container_commands:
purge_basket:
command: crontab /tmp/cronjob
leader_only: true
commands:
delete_cronjob_file:
command: rm /tmp/cronjob
Essentially, I create a temp file with the cronjobs and then set the crontab to read from the temp file, then delete the temp file afterwards. Hope this helps.
I spoke to an AWS support agent and this is how we got this to work for me. 2015 solution:
Create a file in your .ebextensions directory with your_file_name.config.
In the config file input:
files:
"/etc/cron.d/cron_example":
mode: "000644"
owner: root
group: root
content: |
* * * * * root /usr/local/bin/cron_example.sh
"/usr/local/bin/cron_example.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
/usr/local/bin/test_cron.sh || exit
echo "Cron running at " `date` >> /tmp/cron_example.log
# Now do tasks that should only run on 1 instance ...
"/usr/local/bin/test_cron.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
METADATA=/opt/aws/bin/ec2-metadata
INSTANCE_ID=`$METADATA -i | awk '{print $2}'`
REGION=`$METADATA -z | awk '{print substr($2, 0, length($2)-1)}'`
# Find our Auto Scaling Group name.
ASG=`aws ec2 describe-tags --filters "Name=resource-id,Values=$INSTANCE_ID" \
--region $REGION --output text | awk '/aws:autoscaling:groupName/ {print $5}'`
# Find the first instance in the Group
FIRST=`aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names $ASG \
--region $REGION --output text | awk '/InService$/ {print $4}' | sort | head -1`
# Test if they're the same.
[ "$FIRST" = "$INSTANCE_ID" ]
commands:
rm_old_cron:
command: "rm *.bak"
cwd: "/etc/cron.d"
ignoreErrors: true
This solution has 2 drawbacks:
On subsequent deployments, Beanstalk renames the existing cron script as .bak, but cron will still run it. Your Cron now executes twice on the same machine.
If your environment scales up, you get several instances, all running your cron script. This means your mail shots are repeated, or your database archives duplicated
Workaround:
Ensure any .ebextensions script which creates a cron also removes the .bak files on subsequent deployments.
Have a helper script which does the following: -- Gets the current Instance ID from the Metadata -- Gets the current Auto
Scaling Group name from the EC2 Tags -- Gets the list of EC2
Instances in that Group, sorted alphabetically. -- Takes the first
instance from that list. -- Compares the Instance ID from step 1
with the first Instance ID from step 4.
Your cron scripts can then use this helper script to determine if they should execute.
Caveat:
The IAM Role used for the Beanstalk instances needs ec2:DescribeTags and autoscaling:DescribeAutoScalingGroups permissions
The instances chosen from are those shown as InService by Auto Scaling. This does not necessarily mean they are fully booted up and ready to run your cron.
You would not have to set the IAM Roles if you are using the default beanstalk role.
As mentioned above, the fundamental flaw with establishing any crontab configuration is that it only happens at deployment. As the cluster gets auto-scaled up, and then back down, it is favored to also be the first server turned off. In addition there would be no fail-over, which for me was critical.
I did some research, then talked with our AWS account specialist to bounce ideas and valid the solution I came up with. You can accomplish this with OpsWorks, although it's bit like using a house to kill a fly. It is also possible to use Data Pipeline with Task Runner, but this has limited ability in the scripts that it can execute, and I needed to be able to run PHP scripts, with access to the whole code base. You could also dedicate an EC2 instance outside of the ElasticBeanstalk cluster, but then you have no fail-over again.
So here is what I came up with, which apparently is unconventional (as the AWS rep commented) and may be considered a hack, but it works and is solid with fail-over. I chose a coding solution using the SDK, which I'll show in PHP, although you could do the same method in any language you prefer.
// contains the values for variables used (key, secret, env)
require_once('cron_config.inc');
// Load the AWS PHP SDK to connection to ElasticBeanstalk
use Aws\ElasticBeanstalk\ElasticBeanstalkClient;
$client = ElasticBeanstalkClient::factory(array(
'key' => AWS_KEY,
'secret' => AWS_SECRET,
'profile' => 'your_profile',
'region' => 'us-east-1'
));
$result = $client->describeEnvironmentResources(array(
'EnvironmentName' => AWS_ENV
));
if (php_uname('n') != $result['EnvironmentResources']['Instances'][0]['Id']) {
die("Not the primary EC2 instance\n");
}
So walking through this and how it operates... You call scripts from crontab as you normally would on every EC2 instance. Each script includes this at the beginning (or includes a single file for each, as I use it), which establishes an ElasticBeanstalk object and retrieves a list of all instances. It uses only the first server in the list, and checks if it matches itself, which if it does it continues, otherwise it dies and closes out. I've checked and the list returned seems to be consistent, which technically it only needs to be consistent for a minute or so, as each instance executes the scheduled cron. If it does change, it wouldn't matter, since again it only is relevant for that small window.
This isn't elegant by any means, but suited our specific needs - which was not to increase cost with an additional service or have to have a dedicated EC2 instance, and would have fail-over in case of any failure. Our cron scripts run maintenance scripts which get placed into SQS and each server in the cluster helps execute. At least this may give you an alternate option if it fits your needs.
-Davey
If you're using Rails, you can use the whenever-elasticbeanstalk gem. It allows you to run cron jobs on either all instances or just one. It checks every minute to ensure that there is only one "leader" instance, and will automatically promote one server to "leader" if there are none. This is needed since Elastic Beanstalk only has the concept of leader during deployment and may shut down any instance at any time while scaling.
UPDATE
I switched to using AWS OpsWorks and am no longer maintaining this gem. If you need more functionality than is available in the basics of Elastic Beanstalk, I highly recommend switching to OpsWorks.
You really don't want to be running cron jobs on Elastic Beanstalk. Since you'll have multiple application instances, this can cause race conditions and other odd problems. I actually recently blogged about this (4th or 5th tip down the page). The short version: Depending on the application, use a job queue like SQS or a third-party solution like iron.io.
2017: If you are using Laravel5+
You just need 2 minutes to configure it:
create a Worker Tier
install laravel-aws-worker
composer require dusterio/laravel-aws-worker
add a cron.yaml to the root folder:
Add cron.yaml to the root folder of your application (this can be a
part of your repo or you could add this file right before deploying to
EB - the important thing is that this file is present at the time of
deployment):
version: 1
cron:
- name: "schedule"
url: "/worker/schedule"
schedule: "* * * * *"
That's it!
All your task in App\Console\Kernel will now be executed
Detailed instructions and explainations: https://github.com/dusterio/laravel-aws-worker
How to write tasks inside of Laravel: https://laravel.com/docs/5.4/scheduling
A more readable solution using files instead of container_commands:
files:
"/etc/cron.d/my_cron":
mode: "000644"
owner: root
group: root
content: |
# override default email address
MAILTO="example#gmail.com"
# run a Symfony command every five minutes (as ec2-user)
*/10 * * * * ec2-user /usr/bin/php /var/app/current/app/console do:something
encoding: plain
commands:
# delete backup file created by Elastic Beanstalk
clear_cron_backup:
command: rm -f /etc/cron.d/watson.bak
Note the format differs from the usual crontab format in that it specifies the user to run the command as.
My 1 cent of contribution for 2018
Here is the right way to do it (using django/python and django_crontab app):
inside .ebextensions folder create a file like this 98_cron.config:
files:
"/tmp/98_create_cron.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/sh
cd /
sudo /opt/python/run/venv/bin/python /opt/python/current/app/manage.py crontab remove > /home/ec2-user/remove11.txt
sudo /opt/python/run/venv/bin/python /opt/python/current/app/manage.py crontab add > /home/ec2-user/add11.txt
container_commands:
98crontab:
command: "mv /tmp/98_create_cron.sh /opt/elasticbeanstalk/hooks/appdeploy/post && chmod 774 /opt/elasticbeanstalk/hooks/appdeploy/post/98_create_cron.sh"
leader_only: true
It needs to be container_commands instead of commands
The latest example from Amazon is the easiest and most efficient (periodic tasks):
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html
where you create a separate worker tier to execute any of your cron jobs. Create the cron.yaml file and place it in your root folder. One issue I had (after cron did not seem to be executing) was finding that my CodePipeline did not have authority to perform a dynamodb modification. Based on that after adding FullDynamoDB access under IAM -> roles -> yourpipeline and redeploying (elastic beanstalk) it worked perfectly.
Someone was wondering about the leader_only auto scaling problems when new leaders arise. I can't seem to figure out how to reply to their comments, but see this link: http://blog.paulopoiati.com/2013/08/25/running-cron-in-elastic-beanstalk-auto-scaling-environment/
So we've been struggling with this for a while and after some discussion with an AWS rep I've finally come up with what I think is the best solution.
Using a worker tier with cron.yaml is definitely the easiest fix. However, what the documentation doesn't make clear is that this will put the job at the end of the SQS queue you're using to actually run your jobs. If your cron jobs are time sensitive (as many are), this isn't acceptable, since it would depend on the size of the queue. One option is to use a completely separate environment just to run cron jobs, but I think that's overkill.
Some of the other options, like checking to see if you're the first instance in the list, aren't ideal either. What if the current first instance is in the process of shutting down?
Instance protection can also come with issues - what if that instance gets locked up / frozen?
What's important to understand is how AWS itself manages the cron.yaml functionality. There is an SQS daemon which uses a Dynamo table to handle "leader election". It writes to this table frequently, and if the current leader hasn't written in a short while, the next instance will take over as leader. This is how the daemon decides which instance to fire the job into the SQS queue.
We can repurpose the existing functionality rather than trying to rewrite our own. You can see the full solution here: https://gist.github.com/dorner/4517fe2b8c79ccb3971084ec28267f27
That's in Ruby, but you can easily adapt it to any other language that has the AWS SDK. Essentially, it checks the current leader, then checks the state to make sure it's in a good state. It'll loop until there is a current leader in a good state, and if the current instance is the leader, execute the job.
The best way to do this is to use an Elastic Beanstalk Worker Environment (see "Option 1" below). However, this will add to your server costs. If you don't want to do this, see "Option 2" below for how to configure cron itself.
Option 1: Use Elastic Beanstalk Worker environments
Amazon has support for Elastic Beanstalk Worker Environments. They are Elastic Beanstalk managed environments that come with an SQS queue which you can enqueue tasks onto. You can also give them a cron config that will automatically enqueue the task on a recurring schedule. Then, rather than receiving requests from a load balancer, the servers in a worker environment each have a daemon (managed by Elastic Beanstalk) that polls the queue for tasks and calls the appropriate web endpoint when they get a message on the queue. Worker environments have several benefits over running cron yourself:
Performance. Your tasks are now running on dedicated servers instead of competing for CPU and memory with web requests. You can also have different specs for the worker servers (ex. you can have more memory on just the worker servers).
Scalability. You can also scale up your number of worker servers to more than 1 in order to handle large task loads.
Ad-hoc Tasks. Your code can enqueue ad-hoc tasks as well as scheduled ones.
Standardization. You write tasks as web endpoints rather than needing to configure your own task framework, which lets your standardize your code and tooling.
If you just want a cron replacement, all you need to do is make a file called cron.yaml at the top level of your project, with config like the following:
cron.yaml
version: 1
cron:
- name: "hourly"
url: "/tasks/hourly"
schedule: "0 */1 * * *"
This will call the url /tasks/hourly once an hour.
If you are deploying the same codebase to web and worker environments, you should have the task URLs require an environment variable that you set on worker environments and not web environments. This way, your task endpoints are not exposed to the world (task servers by default do not accept incoming HTTP requests, as the only thing making calls to them is the on-server daemon).
The full docs are here: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html
Option 2: Configure Cron
If you want to run cron, you need to make sure it's running on only one server. The leader_only flag in .ebextensions config isn't sufficient because servers don't reliably stay the leader. This can be fixed by deleting the cron config if present on any server as the first step of a deploy and then installing it on just one server using leader_only. Here is an example .ebextensions config file that accomplishes this:
.ebextensions/cron.config
container_commands:
01_remove_cron_jobs:
command: "rm /etc/cron.d/cronjobs || exit 0"
02_set_up_cron:
command: "cat .ebextensions/cronjobs.txt > /etc/cron.d/cronjobs && chmod 644 /etc/cron.d/cronjobs"
leader_only: true
This config file assumes the existence of a file .ebextensions/cronjobs.txt. This file contains your actual cron config. Note that in order to have environment variables loaded and your code in scope, you need to have code that does this baked into each command. The following is an example cron config that works on an Amazon Linux 2 based Python environment:
.ebextensions/cronjobs.txt
SHELL=/bin/bash
PROJECT_PATH=/var/app/current
ENV_PATH=/opt/elasticbeanstalk/deployment/env
# m h dom mon dow user command
0 * * * * ec2-user set -a; source <(sudo cat $ENV_PATH) && cd $PROJECT_PATH && python HOURLY_COMMAND > /dev/null
# Cron requires a newline at the end of the file
Here is a full explanation of the solution:
http://blog.paulopoiati.com/2013/08/25/running-cron-in-elastic-beanstalk-auto-scaling-environment/
To control whether Auto Scaling can terminate a particular instance when scaling in, use instance protection. You can enable the instance protection setting on an Auto Scaling group or an individual Auto Scaling instance. When Auto Scaling launches an instance, the instance inherits the instance protection setting of the Auto Scaling group. You can change the instance protection setting for an Auto Scaling group or an Auto Scaling instance at any time.
http://docs.aws.amazon.com/autoscaling/latest/userguide/as-instance-termination.html#instance-protection
I had another solution to this if a php file needs to be run through cron and if you had set any NAT instances then you can put cronjob on NAT instance and run php file through wget.
here is a fix incase you want to do this in PHP. You just need cronjob.config in your .ebextensions folder to get it to work like this.
files:
"/etc/cron.d/my_cron":
mode: "000644"
owner: root
group: root
content: |
empty stuff
encoding: plain
commands:
01_clear_cron_backup:
command: "rm -f /etc/cron.d/*.bak"
02_remove_content:
command: "sudo sed -i 's/empty stuff//g' /etc/cron.d/my_cron"
container_commands:
adding_cron:
command: "echo '* * * * * ec2-user . /opt/elasticbeanstalk/support/envvars && /usr/bin/php /var/app/current/index.php cron sendemail > /tmp/sendemail.log 2>&1' > /etc/cron.d/my_cron"
leader_only: true
the envvars gets the environment variables for the files. You can debug the output on the tmp/sendemail.log as above.
Hope this helps someone as it surely helped us!
Based on the principles of the answer from user1599237, where you let the cron jobs run on all instances but then instead in the beginning of the jobs determine if they should be allowed to run, I have made another solution.
Instead of looking at the running instances (and having to store your AWS key and secret) I'm using the MySQL database that I'm already connecting to from all instances.
It has no downsides, only positives:
no extra instance or expenses
rock solid solution - no chance of double execution
scalable - automatically works as your instances are scaled up and down
failover - automatically works in case an instance has a failure
Alternatively, you could also use a commonly shared filesystem (like AWS EFS via the NFS protocol) instead of a database.
The following solution is created within the PHP framework Yii but you can easily adapt it for another framework and language. Also the exception handler Yii::$app->system is a module of my own. Replace it with whatever you are using.
/**
* Obtain an exclusive lock to ensure only one instance or worker executes a job
*
* Examples:
*
* `php /var/app/current/yii process/lock 60 empty-trash php /var/app/current/yii maintenance/empty-trash`
* `php /var/app/current/yii process/lock 60 empty-trash php /var/app/current/yii maintenance/empty-trash StdOUT./test.log`
* `php /var/app/current/yii process/lock 60 "empty trash" php /var/app/current/yii maintenance/empty-trash StdOUT./test.log StdERR.ditto`
* `php /var/app/current/yii process/lock 60 "empty trash" php /var/app/current/yii maintenance/empty-trash StdOUT./output.log StdERR./error.log`
*
* Arguments are understood as follows:
* - First: Duration of the lock in minutes
* - Second: Job name (surround with quotes if it contains spaces)
* - The rest: Command to execute. Instead of writing `>` and `2>` for redirecting output you need to write `StdOUT` and `StdERR` respectively. To redirect stderr to stdout write `StdERR.ditto`.
*
* Command will be executed in the background. If determined that it should not be executed the script will terminate silently.
*/
public function actionLock() {
$argsAll = $args = func_get_args();
if (!is_numeric($args[0])) {
\Yii::$app->system->error('Duration for obtaining process lock is not numeric.', ['Args' => $argsAll]);
}
if (!$args[1]) {
\Yii::$app->system->error('Job name for obtaining process lock is missing.', ['Args' => $argsAll]);
}
$durationMins = $args[0];
$jobName = $args[1];
$instanceID = null;
unset($args[0], $args[1]);
$command = trim(implode(' ', $args));
if (!$command) {
\Yii::$app->system->error('Command to execute after obtaining process lock is missing.', ['Args' => $argsAll]);
}
// If using AWS Elastic Beanstalk retrieve the instance ID
if (file_exists('/etc/elasticbeanstalk/.aws-eb-system-initialized')) {
if ($awsEb = file_get_contents('/etc/elasticbeanstalk/.aws-eb-system-initialized')) {
$awsEb = json_decode($awsEb);
if (is_object($awsEb) && $awsEb->instance_id) {
$instanceID = $awsEb->instance_id;
}
}
}
// Obtain lock
$updateColumns = false; //do nothing if record already exists
$affectedRows = \Yii::$app->db->createCommand()->upsert('system_job_locks', [
'job_name' => $jobName,
'locked' => gmdate('Y-m-d H:i:s'),
'duration' => $durationMins,
'source' => $instanceID,
], $updateColumns)->execute();
// The SQL generated: INSERT INTO system_job_locks (job_name, locked, duration, source) VALUES ('some-name', '2019-04-22 17:24:39', 60, 'i-HmkDAZ9S5G5G') ON DUPLICATE KEY UPDATE job_name = job_name
if ($affectedRows == 0) {
// record already exists, check if lock has expired
$affectedRows = \Yii::$app->db->createCommand()->update('system_job_locks', [
'locked' => gmdate('Y-m-d H:i:s'),
'duration' => $durationMins,
'source' => $instanceID,
],
'job_name = :jobName AND DATE_ADD(locked, INTERVAL duration MINUTE) < NOW()', ['jobName' => $jobName]
)->execute();
// The SQL generated: UPDATE system_job_locks SET locked = '2019-04-22 17:24:39', duration = 60, source = 'i-HmkDAZ9S5G5G' WHERE job_name = 'clean-trash' AND DATE_ADD(locked, INTERVAL duration MINUTE) < NOW()
if ($affectedRows == 0) {
// We could not obtain a lock (since another process already has it) so do not execute the command
exit;
}
}
// Handle redirection of stdout and stderr
$command = str_replace('StdOUT', '>', $command);
$command = str_replace('StdERR.ditto', '2>&1', $command);
$command = str_replace('StdERR', '2>', $command);
// Execute the command as a background process so we can exit the current process
$command .= ' &';
$output = []; $exitcode = null;
exec($command, $output, $exitcode);
exit($exitcode);
}
This is the database schema I'm using:
CREATE TABLE `system_job_locks` (
`job_name` VARCHAR(50) NOT NULL,
`locked` DATETIME NOT NULL COMMENT 'UTC',
`duration` SMALLINT(5) UNSIGNED NOT NULL COMMENT 'Minutes',
`source` VARCHAR(255) NULL DEFAULT NULL,
PRIMARY KEY (`job_name`)
)