Its been a month I have started working on EKS AWS and up till now successfully deployed by code.
The steps which I follow for deployment are given below:
Create image from docker terminal.
Tag and push to ECR AWS.
Create the deployment "project.json" and service file "project-svc.json".
Save the above file in "kubectl/bin" path and deploy it with following commands below.
"kubectl apply -f projectname.json" and "kubectl apply -f projectname-svc.json".
So if I want to deployment the same project again with change, I push the new image on ECR and delete the existing deployment by using "kubectl delete -f projectname.json" without deleting the existing service and deploy it again using command "kubectl apply -f projectname.json" again.
Now, I'm in confusing that after I delete the existing deployment there is a downtime until I apply or create the deployment again. So, how to avoid this ? Because I don't want the downtime actually that is the reason why I started to use the EKS.
And one more thing is the process of deployment is a bit long too. I know I'm missing something can anybody guide me properly please?
The project is on .NET Core and if there is any simplified way to do deployment using Visual Studio please guide me for that also.
Thank You in advance!
There is actually no need to delete your deployment. Just need to update the desired state (the deployment configuration) and let K8s do its magic and apply the needed changes, like deploying a new version of your container.
If you have a single instance of your container, you will experience a short down time while changes are applied. If your application supports multiple replicas (HA), you can enjoy the rolling upgrade feature.
Start by reading the official Kubernetes documentation of a Performing a Rolling Update.
You only need to use the delete/apply if you are changing (And if you have) the ConfigMap attached to the Deployment.
Is the only change you do is the "image" of the deployment - you must use the "set-image" command.
Kubectl let you change the actual deployment image and it does the Rolling Updates all by itself and with 3+ pods you have the minimum chance for downtime.
Even more, if you use the --record flag, you can "rollback" to your previous image with no effort because it keep track of the changes.
You also have the possibility to specify the "Context" too, with no need to jump from contexts.
You can go like this:
kubectl set image deployment DEPLOYMENT_NAME DEPLOYMENT_NAME=IMAGE_NAME --record -n NAMESPACE
OR Specifying the Cluster
kubectl set image deployment DEPLOYEMTN_NAME DEPLOYEMTN_NAME=IMAGE_NAME_ECR -n NAMESPACE --cluster EKS_CLUSTER_NPROD --user EKS_CLUSTER --record
As an Eg:
kubectl set image deployment nginx-dep nginx-dep=ecr12345/nginx:latest -n nginx --cluster eu-central-123-prod --user eu-central-123-prod --record
The --record is what let you track all the changes, if you want to rollback just do:
kubectl rollout undo deployment.v1.apps/nginx-dep
More documentations about it here:
Updating a deployment
https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#updating-a-deployment
Roll Back Deployment
https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-back-a-deployment
Below is the setup of the application which runs a docker container into elasticbeanstalk.
step 1:
created the parent folder say apptest and inside that I have placed the Dockerfile, package.json and small hello world server.js node app.
step 2:
inside the parent folder apptest ran the command eb init which crated a hidden folder .elasticbeanstalk with config.yml comprising default settings.
step 3:
added .ebextensions with a config file 01_run.config, comprising the below configuration to update the instance type.
option_settings:
aws:autoscaling:launchconfiguration :
InstanceType : "m4.xlarge"
Note that till now no environment is created yet. And since I have extensions created, should override the defaults with the instance type pointed to m4.xlarge.
step 4:
Now ran the command eb create apptest-dev ( say for example ) and created the environment
Problem:
when the environment is created, it did not have the m4.xlarge, rather got created with default instance type as t2.micro. But when uploaded the zipped version of this folder contents into the environment from console ( excluding the .elasticbeanstalk folder ), ebextensions folder configuration is picked up. Its only with the option_settings not getting worked, but the other resources like files, commands are getting reflected both from command line and from the file upload.
I feel its kind of very small thing missing which I am not able to figure it out from blogs and documentation. Thanks for the help in advance.
During eb create, the EBCLI passes it's own defaults for many of the option settings, among which is the instance type. Since, the EBCLI does not parse .ebextensions, and the Beanstalk service prefers the defaults passed by the EBCLI, the instance type specified in your .ebextensions are disregarded.
There are two ways to get around this:
call eb config after eb create. In the interactive mode, change the instance type, and save and exit.
call eb create as eb create -i m4.xlarge
I found fragmented instructions here and some other places about deploying Play2 app on amazon ec2. But did not find any neat way to deploy using Beanstalk.
Play is a nice framework and AWS beanstalk is one of the most popular services then why is there no official instruction to do this?
Has anyone found any better solution?
Deploying a Play2 app on elastic beanstalk is now easy with Docker Containers in combination with sbt's experimental docker feature.
In build.sbt specify the exposed docker ports:
dockerExposedPorts in Docker := Seq(9000)
You should automate the following steps, but you can try this out manually to test that it works:
Generate a Dockerfile for the project by running the command: sbt docker:stage.
Go to the ./target/docker/ directory.
Create an elastic beanstalk Dockerrun.aws.json file with the following contents:
{
"AWSEBDockerrunVersion": "1",
"Ports": [
{
"ContainerPort": "9000"
}
]
}
Zip up everything in that directory, let's say into a file called play2-test-docker.zip. The zip file should contain the files: Dockerfile, Dockerrun.aws.json, and files/* directory.
Go to aws beanstalk console and create a new application using the m3.medium or any instance type with enough memory for the jvm to run. Any instance with too little memory will result in a JVM error.
Select "Docker Container" in the Predefined Configuration dropdown.
In the application selection screen, select "Upload" and select the zip file you created earlier. Launch the app and then go brew some tea. This can take a very long time. Minutes. Subsequent deployments of the same app version should be slightly quicker.
Once the app is running and green in the aws console, click on the app's url and you should see the welcome screen of the application (or whatever your index file is).
Here's my solution that doesn't require any additional services/containers like Docker or Jenkins.
Create a dist folder in the root of your Play application's directory. Create a Procfile file containing the following contents and put it in the dist folder (EB requires port 5000):
web: ./bin/YOUR_APP_FILE_NAME -Dhttp.port=5000 -Dconfig.file=conf/application.conf
The YOUR_APP_FILE_NAME is the name of the executable in the bin directory, which is inside the .zip created by activator dist.
After running activator dist, you can just upload the created zip file into Elastic Beanstalk and it will automatically deploy the app. You also put whatever .ebextension folders and configuration files into the dist folder that you require for Elastic Beanstalk configuration. Ex. I have dist/.ebextensions/nginx/conf.d/proxy.conf for NGINX reverse proxy settings or dist/.ebextensions/env.config for environment variables.
Edit 2016: There's now a much better way to deploy your Playframework apps onto ElasticBeanstalk using the new Java SE containers.
Here's an article that walks you through deploying step by step using Jenkins to build and deploy your project:
https://www.davemaple.com/articles/deploy-playframework-elastic-beanstalk-jenkins/
You can use custom AMIs that I keep updated here:
https://github.com/davemaple/playframework-nginx-elastic-beanstalk
These run Nginx + Playframework and support standard zip files created using "activator dist".
We also saw this as being too much of a pain and have added native Play 2 support to Boxfuse to address this.
You can now simply do boxfuse run my-play-app-1.0.zip -env=prod and this will automatically:
create a minimal AMI tailor-made for your Play 2 app
create an elastic IP
create a security group with the correct permissions
launch an instance of your app
All future updates are performed as blue/green deployments with zero downtime.
This also works with Elastic Load Balancers and Auto-Scaling Groups and the Boxfuse free tier is designed to fit the AWS free tier.
You can read more about it here: https://boxfuse.com/blog/playframework-aws
Disclaimer: I'm the founder and CEO of Boxfuse
I had some problems with other solutions found here and there. I guess that the problem is that I'm developing on Play 2.4.
Anyway, I could deploy the app to Beanstalk using Typesafe Activator and Docker:
In build.sbt I added this lines:
import com.typesafe.sbt.packager.docker.{ExecCmd, Cmd}
// [...]
dockerCommands := Seq(
Cmd("FROM","java:openjdk-8-jre"),
Cmd("MAINTAINER","myname"),
Cmd("EXPOSE","9000"),
Cmd("ADD","stage /"),
Cmd("WORKDIR","/opt/docker"),
Cmd("RUN","[\"chown\", \"-R\", \"daemon\", \".\"]"),
Cmd("RUN","[\"chmod\", \"+x\", \"bin/myapp\"]"),
Cmd("USER","daemon"),
Cmd("ENTRYPOINT","[\"bin/myapp\", \"-J-Xms128m\", \"-J-Xmx512m\", \"-J-server\"]"),
ExecCmd("CMD")
)
I went to the project's directory and ran this command in the terminal
$ ./activator clean docker:stage
I opened the [project]/target/dockerdirectory and created the file Dockerrun.aws.json. This was its content:
{
"AWSEBDockerrunVersion": "1",
"Ports": [
{
"ContainerPort": "9000"
}
]
}
In the same target/docker directory, I tested the result, built, checked and ran the image:
$ docker build -t myapp .
$ docker images
$ docker run -p 9000:9000 myapp
As everything was ok, I zipped the content:
$ zip -r myapp.zip *
My zip file had Dockerfile, Dockerrun.aws.json and stage/* files
Finally, I created a new Beanstalk app and uploaded the zip created on the last step. I took care of select "Generic Docker" on "Predefined configuration", when I was creating the app.
Beanstalk only supports WAR deployment and Play doesn't officially support WAR deployment. If you want to use EC2 then you should instead just create an EC2 instance and follow the deployment instructions: http://www.playframework.com/documentation/2.2.x/ProductionDist
Deploying play 2.* apps in aws ec2 is very diffrent until you have found this much better way to do it. I mean ansible is promising a great solution to that. though it is still needed to work with new setup of ansible, and its playbook but that must be worthy.
I have found these reads very recently and yet to apply them in my project. I hope following reads will help you to learn more:
Ansible + play + aws ec2
Read it to know more about Ansible to deply play in aws
Thanks!
Hope this will help you to kick your start. Please do share more knowledge you gain during the procedure or if there is any simple way to solve this complicated deployment problem.
I would like to know if there is a way to setup a cronjob/task to execute every minute. Currently any of my instances should be able to run this task.
This is what I have tried to do in the config files without success:
container_commands:
01cronjobs:
command: echo "*/1 * * * * root php /etc/httpd/myscript.php"
I'm not really sure if this is the correct way to do it
Any ideas?
This is how I added a cron job to Elastic Beanstalk:
Create a folder at the root of your application called .ebextensions if it doesn't exist already. Then create a config file inside the .ebextensions folder. I'll use example.config for illustration purposes. Then add this to example.config
container_commands:
01_some_cron_job:
command: "cat .ebextensions/some_cron_job.txt > /etc/cron.d/some_cron_job && chmod 644 /etc/cron.d/some_cron_job"
leader_only: true
This is a YAML configuration file for Elastic Beanstalk. Make sure when you copy this into your text editor that your text editor uses spaces instead of tabs. Otherwise you'll get a YAML error when you push this to EB.
So what this does is create a command called 01_some_cron_job. Commands are run in alphabetical order so the 01 makes sure it's run as the first command.
The command then takes the contents of a file called some_cron_job.txt and adds it to a file called some_cron_job in /etc/cron.d.
The command then changes the permissions on the /etc/cron.d/some_cron_job file.
The leader_only key ensures the command is only run on the ec2 instance that is considered the leader. Rather than running on every ec2 instance you may have running.
Then create a file called some_cron_job.txt inside the .ebextensions folder. You will place your cron jobs in this file.
So for example:
# The newline at the end of this file is extremely important. Cron won't run without it.
* * * * * root /usr/bin/php some-php-script-here > /dev/null
So this cron job will run every minute of every hour of every day as the root user and discard the output to /dev/null. /usr/bin/php is the path to php. Then replace some-php-script-here with the path to your php file. This is obviously assuming your cron job needs to run a PHP file.
Also, make sure the some_cron_job.txt file has a newline at the end of the file just like the comment says. Otherwise cron won't run.
Update:
There is an issue with this solution when Elastic Beanstalk scales up your instances. For example, lets say you have one instance with the cron job running. You get an increase in traffic so Elastic Beanstalk scales you up to two instances. The leader_only will ensure you only have one cron job running between the two instances. Your traffic decreases and Elastic Beanstalk scales you down to one instance. But instead of terminating the second instance, Elastic Beanstalk terminates the first instance that was the leader. You now don't have any cron jobs running since they were only running on the first instance that was terminated. See the comments below.
Update 2:
Just making this clear from the comments below:
AWS has now protection against automatic instance termination. Just enable it on your leader instance and you're good to go. – Nicolás Arévalo Oct 28 '16 at 9:23
This is the official way to do it now (2015+). Please try this first, it's by far easiest method currently available and most reliable as well.
According to current docs, one is able to run periodic tasks on their so-called worker tier.
Citing the documentation:
AWS Elastic Beanstalk supports periodic tasks for worker environment tiers in environments running a predefined configuration with a solution stack that contains "v1.2.0" in the container name. You must create a new environment.
Also interesting is the part about cron.yaml:
To invoke periodic tasks, your application source bundle must include a cron.yaml file at the root level. The file must contain information about the periodic tasks you want to schedule. Specify this information using standard crontab syntax.
Update: We were able to get this work. Here are some important gotchas from our experience (Node.js platform):
When using cron.yaml file, make sure you have latest awsebcli, because older versions will not work properly.
It is also vital to create new environment (at least in our case it was), not just clone old one.
If you want to make sure CRON is supported on your EC2 Worker Tier instance, ssh into it (eb ssh), and run cat /var/log/aws-sqsd/default.log. It should report as aws-sqsd 2.0 (2015-02-18). If you don't have 2.0 version, something gone wrong when creating your environment and you need to create new one as stated above.
Regarding jamieb's response, and as alrdinleal mentions, you can use the 'leader_only' property to ensure that only one EC2 instance runs the cron job.
Quote taken from http://docs.amazonwebservices.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html:
you can use leader_only. One instance is chosen to be the leader in an Auto Scaling group. If the leader_only value is set to true, the command runs only on the instance that is marked as the leader.
Im trying to achieve a similar thing on my eb, so will update my post if I solve it.
UPDATE:
Ok, I now have working cronjobs using the following eb config:
files:
"/tmp/cronjob" :
mode: "000777"
owner: ec2-user
group: ec2-user
content: |
# clear expired baskets
*/10 * * * * /usr/bin/wget -o /dev/null http://blah.elasticbeanstalk.com/basket/purge > $HOME/basket_purge.log 2>&1
# clean up files created by above cronjob
30 23 * * * rm $HOME/purge*
encoding: plain
container_commands:
purge_basket:
command: crontab /tmp/cronjob
leader_only: true
commands:
delete_cronjob_file:
command: rm /tmp/cronjob
Essentially, I create a temp file with the cronjobs and then set the crontab to read from the temp file, then delete the temp file afterwards. Hope this helps.
I spoke to an AWS support agent and this is how we got this to work for me. 2015 solution:
Create a file in your .ebextensions directory with your_file_name.config.
In the config file input:
files:
"/etc/cron.d/cron_example":
mode: "000644"
owner: root
group: root
content: |
* * * * * root /usr/local/bin/cron_example.sh
"/usr/local/bin/cron_example.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
/usr/local/bin/test_cron.sh || exit
echo "Cron running at " `date` >> /tmp/cron_example.log
# Now do tasks that should only run on 1 instance ...
"/usr/local/bin/test_cron.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
METADATA=/opt/aws/bin/ec2-metadata
INSTANCE_ID=`$METADATA -i | awk '{print $2}'`
REGION=`$METADATA -z | awk '{print substr($2, 0, length($2)-1)}'`
# Find our Auto Scaling Group name.
ASG=`aws ec2 describe-tags --filters "Name=resource-id,Values=$INSTANCE_ID" \
--region $REGION --output text | awk '/aws:autoscaling:groupName/ {print $5}'`
# Find the first instance in the Group
FIRST=`aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names $ASG \
--region $REGION --output text | awk '/InService$/ {print $4}' | sort | head -1`
# Test if they're the same.
[ "$FIRST" = "$INSTANCE_ID" ]
commands:
rm_old_cron:
command: "rm *.bak"
cwd: "/etc/cron.d"
ignoreErrors: true
This solution has 2 drawbacks:
On subsequent deployments, Beanstalk renames the existing cron script as .bak, but cron will still run it. Your Cron now executes twice on the same machine.
If your environment scales up, you get several instances, all running your cron script. This means your mail shots are repeated, or your database archives duplicated
Workaround:
Ensure any .ebextensions script which creates a cron also removes the .bak files on subsequent deployments.
Have a helper script which does the following: -- Gets the current Instance ID from the Metadata -- Gets the current Auto
Scaling Group name from the EC2 Tags -- Gets the list of EC2
Instances in that Group, sorted alphabetically. -- Takes the first
instance from that list. -- Compares the Instance ID from step 1
with the first Instance ID from step 4.
Your cron scripts can then use this helper script to determine if they should execute.
Caveat:
The IAM Role used for the Beanstalk instances needs ec2:DescribeTags and autoscaling:DescribeAutoScalingGroups permissions
The instances chosen from are those shown as InService by Auto Scaling. This does not necessarily mean they are fully booted up and ready to run your cron.
You would not have to set the IAM Roles if you are using the default beanstalk role.
As mentioned above, the fundamental flaw with establishing any crontab configuration is that it only happens at deployment. As the cluster gets auto-scaled up, and then back down, it is favored to also be the first server turned off. In addition there would be no fail-over, which for me was critical.
I did some research, then talked with our AWS account specialist to bounce ideas and valid the solution I came up with. You can accomplish this with OpsWorks, although it's bit like using a house to kill a fly. It is also possible to use Data Pipeline with Task Runner, but this has limited ability in the scripts that it can execute, and I needed to be able to run PHP scripts, with access to the whole code base. You could also dedicate an EC2 instance outside of the ElasticBeanstalk cluster, but then you have no fail-over again.
So here is what I came up with, which apparently is unconventional (as the AWS rep commented) and may be considered a hack, but it works and is solid with fail-over. I chose a coding solution using the SDK, which I'll show in PHP, although you could do the same method in any language you prefer.
// contains the values for variables used (key, secret, env)
require_once('cron_config.inc');
// Load the AWS PHP SDK to connection to ElasticBeanstalk
use Aws\ElasticBeanstalk\ElasticBeanstalkClient;
$client = ElasticBeanstalkClient::factory(array(
'key' => AWS_KEY,
'secret' => AWS_SECRET,
'profile' => 'your_profile',
'region' => 'us-east-1'
));
$result = $client->describeEnvironmentResources(array(
'EnvironmentName' => AWS_ENV
));
if (php_uname('n') != $result['EnvironmentResources']['Instances'][0]['Id']) {
die("Not the primary EC2 instance\n");
}
So walking through this and how it operates... You call scripts from crontab as you normally would on every EC2 instance. Each script includes this at the beginning (or includes a single file for each, as I use it), which establishes an ElasticBeanstalk object and retrieves a list of all instances. It uses only the first server in the list, and checks if it matches itself, which if it does it continues, otherwise it dies and closes out. I've checked and the list returned seems to be consistent, which technically it only needs to be consistent for a minute or so, as each instance executes the scheduled cron. If it does change, it wouldn't matter, since again it only is relevant for that small window.
This isn't elegant by any means, but suited our specific needs - which was not to increase cost with an additional service or have to have a dedicated EC2 instance, and would have fail-over in case of any failure. Our cron scripts run maintenance scripts which get placed into SQS and each server in the cluster helps execute. At least this may give you an alternate option if it fits your needs.
-Davey
If you're using Rails, you can use the whenever-elasticbeanstalk gem. It allows you to run cron jobs on either all instances or just one. It checks every minute to ensure that there is only one "leader" instance, and will automatically promote one server to "leader" if there are none. This is needed since Elastic Beanstalk only has the concept of leader during deployment and may shut down any instance at any time while scaling.
UPDATE
I switched to using AWS OpsWorks and am no longer maintaining this gem. If you need more functionality than is available in the basics of Elastic Beanstalk, I highly recommend switching to OpsWorks.
You really don't want to be running cron jobs on Elastic Beanstalk. Since you'll have multiple application instances, this can cause race conditions and other odd problems. I actually recently blogged about this (4th or 5th tip down the page). The short version: Depending on the application, use a job queue like SQS or a third-party solution like iron.io.
2017: If you are using Laravel5+
You just need 2 minutes to configure it:
create a Worker Tier
install laravel-aws-worker
composer require dusterio/laravel-aws-worker
add a cron.yaml to the root folder:
Add cron.yaml to the root folder of your application (this can be a
part of your repo or you could add this file right before deploying to
EB - the important thing is that this file is present at the time of
deployment):
version: 1
cron:
- name: "schedule"
url: "/worker/schedule"
schedule: "* * * * *"
That's it!
All your task in App\Console\Kernel will now be executed
Detailed instructions and explainations: https://github.com/dusterio/laravel-aws-worker
How to write tasks inside of Laravel: https://laravel.com/docs/5.4/scheduling
A more readable solution using files instead of container_commands:
files:
"/etc/cron.d/my_cron":
mode: "000644"
owner: root
group: root
content: |
# override default email address
MAILTO="example#gmail.com"
# run a Symfony command every five minutes (as ec2-user)
*/10 * * * * ec2-user /usr/bin/php /var/app/current/app/console do:something
encoding: plain
commands:
# delete backup file created by Elastic Beanstalk
clear_cron_backup:
command: rm -f /etc/cron.d/watson.bak
Note the format differs from the usual crontab format in that it specifies the user to run the command as.
My 1 cent of contribution for 2018
Here is the right way to do it (using django/python and django_crontab app):
inside .ebextensions folder create a file like this 98_cron.config:
files:
"/tmp/98_create_cron.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/sh
cd /
sudo /opt/python/run/venv/bin/python /opt/python/current/app/manage.py crontab remove > /home/ec2-user/remove11.txt
sudo /opt/python/run/venv/bin/python /opt/python/current/app/manage.py crontab add > /home/ec2-user/add11.txt
container_commands:
98crontab:
command: "mv /tmp/98_create_cron.sh /opt/elasticbeanstalk/hooks/appdeploy/post && chmod 774 /opt/elasticbeanstalk/hooks/appdeploy/post/98_create_cron.sh"
leader_only: true
It needs to be container_commands instead of commands
The latest example from Amazon is the easiest and most efficient (periodic tasks):
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html
where you create a separate worker tier to execute any of your cron jobs. Create the cron.yaml file and place it in your root folder. One issue I had (after cron did not seem to be executing) was finding that my CodePipeline did not have authority to perform a dynamodb modification. Based on that after adding FullDynamoDB access under IAM -> roles -> yourpipeline and redeploying (elastic beanstalk) it worked perfectly.
Someone was wondering about the leader_only auto scaling problems when new leaders arise. I can't seem to figure out how to reply to their comments, but see this link: http://blog.paulopoiati.com/2013/08/25/running-cron-in-elastic-beanstalk-auto-scaling-environment/
So we've been struggling with this for a while and after some discussion with an AWS rep I've finally come up with what I think is the best solution.
Using a worker tier with cron.yaml is definitely the easiest fix. However, what the documentation doesn't make clear is that this will put the job at the end of the SQS queue you're using to actually run your jobs. If your cron jobs are time sensitive (as many are), this isn't acceptable, since it would depend on the size of the queue. One option is to use a completely separate environment just to run cron jobs, but I think that's overkill.
Some of the other options, like checking to see if you're the first instance in the list, aren't ideal either. What if the current first instance is in the process of shutting down?
Instance protection can also come with issues - what if that instance gets locked up / frozen?
What's important to understand is how AWS itself manages the cron.yaml functionality. There is an SQS daemon which uses a Dynamo table to handle "leader election". It writes to this table frequently, and if the current leader hasn't written in a short while, the next instance will take over as leader. This is how the daemon decides which instance to fire the job into the SQS queue.
We can repurpose the existing functionality rather than trying to rewrite our own. You can see the full solution here: https://gist.github.com/dorner/4517fe2b8c79ccb3971084ec28267f27
That's in Ruby, but you can easily adapt it to any other language that has the AWS SDK. Essentially, it checks the current leader, then checks the state to make sure it's in a good state. It'll loop until there is a current leader in a good state, and if the current instance is the leader, execute the job.
The best way to do this is to use an Elastic Beanstalk Worker Environment (see "Option 1" below). However, this will add to your server costs. If you don't want to do this, see "Option 2" below for how to configure cron itself.
Option 1: Use Elastic Beanstalk Worker environments
Amazon has support for Elastic Beanstalk Worker Environments. They are Elastic Beanstalk managed environments that come with an SQS queue which you can enqueue tasks onto. You can also give them a cron config that will automatically enqueue the task on a recurring schedule. Then, rather than receiving requests from a load balancer, the servers in a worker environment each have a daemon (managed by Elastic Beanstalk) that polls the queue for tasks and calls the appropriate web endpoint when they get a message on the queue. Worker environments have several benefits over running cron yourself:
Performance. Your tasks are now running on dedicated servers instead of competing for CPU and memory with web requests. You can also have different specs for the worker servers (ex. you can have more memory on just the worker servers).
Scalability. You can also scale up your number of worker servers to more than 1 in order to handle large task loads.
Ad-hoc Tasks. Your code can enqueue ad-hoc tasks as well as scheduled ones.
Standardization. You write tasks as web endpoints rather than needing to configure your own task framework, which lets your standardize your code and tooling.
If you just want a cron replacement, all you need to do is make a file called cron.yaml at the top level of your project, with config like the following:
cron.yaml
version: 1
cron:
- name: "hourly"
url: "/tasks/hourly"
schedule: "0 */1 * * *"
This will call the url /tasks/hourly once an hour.
If you are deploying the same codebase to web and worker environments, you should have the task URLs require an environment variable that you set on worker environments and not web environments. This way, your task endpoints are not exposed to the world (task servers by default do not accept incoming HTTP requests, as the only thing making calls to them is the on-server daemon).
The full docs are here: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html
Option 2: Configure Cron
If you want to run cron, you need to make sure it's running on only one server. The leader_only flag in .ebextensions config isn't sufficient because servers don't reliably stay the leader. This can be fixed by deleting the cron config if present on any server as the first step of a deploy and then installing it on just one server using leader_only. Here is an example .ebextensions config file that accomplishes this:
.ebextensions/cron.config
container_commands:
01_remove_cron_jobs:
command: "rm /etc/cron.d/cronjobs || exit 0"
02_set_up_cron:
command: "cat .ebextensions/cronjobs.txt > /etc/cron.d/cronjobs && chmod 644 /etc/cron.d/cronjobs"
leader_only: true
This config file assumes the existence of a file .ebextensions/cronjobs.txt. This file contains your actual cron config. Note that in order to have environment variables loaded and your code in scope, you need to have code that does this baked into each command. The following is an example cron config that works on an Amazon Linux 2 based Python environment:
.ebextensions/cronjobs.txt
SHELL=/bin/bash
PROJECT_PATH=/var/app/current
ENV_PATH=/opt/elasticbeanstalk/deployment/env
# m h dom mon dow user command
0 * * * * ec2-user set -a; source <(sudo cat $ENV_PATH) && cd $PROJECT_PATH && python HOURLY_COMMAND > /dev/null
# Cron requires a newline at the end of the file
Here is a full explanation of the solution:
http://blog.paulopoiati.com/2013/08/25/running-cron-in-elastic-beanstalk-auto-scaling-environment/
To control whether Auto Scaling can terminate a particular instance when scaling in, use instance protection. You can enable the instance protection setting on an Auto Scaling group or an individual Auto Scaling instance. When Auto Scaling launches an instance, the instance inherits the instance protection setting of the Auto Scaling group. You can change the instance protection setting for an Auto Scaling group or an Auto Scaling instance at any time.
http://docs.aws.amazon.com/autoscaling/latest/userguide/as-instance-termination.html#instance-protection
I had another solution to this if a php file needs to be run through cron and if you had set any NAT instances then you can put cronjob on NAT instance and run php file through wget.
here is a fix incase you want to do this in PHP. You just need cronjob.config in your .ebextensions folder to get it to work like this.
files:
"/etc/cron.d/my_cron":
mode: "000644"
owner: root
group: root
content: |
empty stuff
encoding: plain
commands:
01_clear_cron_backup:
command: "rm -f /etc/cron.d/*.bak"
02_remove_content:
command: "sudo sed -i 's/empty stuff//g' /etc/cron.d/my_cron"
container_commands:
adding_cron:
command: "echo '* * * * * ec2-user . /opt/elasticbeanstalk/support/envvars && /usr/bin/php /var/app/current/index.php cron sendemail > /tmp/sendemail.log 2>&1' > /etc/cron.d/my_cron"
leader_only: true
the envvars gets the environment variables for the files. You can debug the output on the tmp/sendemail.log as above.
Hope this helps someone as it surely helped us!
Based on the principles of the answer from user1599237, where you let the cron jobs run on all instances but then instead in the beginning of the jobs determine if they should be allowed to run, I have made another solution.
Instead of looking at the running instances (and having to store your AWS key and secret) I'm using the MySQL database that I'm already connecting to from all instances.
It has no downsides, only positives:
no extra instance or expenses
rock solid solution - no chance of double execution
scalable - automatically works as your instances are scaled up and down
failover - automatically works in case an instance has a failure
Alternatively, you could also use a commonly shared filesystem (like AWS EFS via the NFS protocol) instead of a database.
The following solution is created within the PHP framework Yii but you can easily adapt it for another framework and language. Also the exception handler Yii::$app->system is a module of my own. Replace it with whatever you are using.
/**
* Obtain an exclusive lock to ensure only one instance or worker executes a job
*
* Examples:
*
* `php /var/app/current/yii process/lock 60 empty-trash php /var/app/current/yii maintenance/empty-trash`
* `php /var/app/current/yii process/lock 60 empty-trash php /var/app/current/yii maintenance/empty-trash StdOUT./test.log`
* `php /var/app/current/yii process/lock 60 "empty trash" php /var/app/current/yii maintenance/empty-trash StdOUT./test.log StdERR.ditto`
* `php /var/app/current/yii process/lock 60 "empty trash" php /var/app/current/yii maintenance/empty-trash StdOUT./output.log StdERR./error.log`
*
* Arguments are understood as follows:
* - First: Duration of the lock in minutes
* - Second: Job name (surround with quotes if it contains spaces)
* - The rest: Command to execute. Instead of writing `>` and `2>` for redirecting output you need to write `StdOUT` and `StdERR` respectively. To redirect stderr to stdout write `StdERR.ditto`.
*
* Command will be executed in the background. If determined that it should not be executed the script will terminate silently.
*/
public function actionLock() {
$argsAll = $args = func_get_args();
if (!is_numeric($args[0])) {
\Yii::$app->system->error('Duration for obtaining process lock is not numeric.', ['Args' => $argsAll]);
}
if (!$args[1]) {
\Yii::$app->system->error('Job name for obtaining process lock is missing.', ['Args' => $argsAll]);
}
$durationMins = $args[0];
$jobName = $args[1];
$instanceID = null;
unset($args[0], $args[1]);
$command = trim(implode(' ', $args));
if (!$command) {
\Yii::$app->system->error('Command to execute after obtaining process lock is missing.', ['Args' => $argsAll]);
}
// If using AWS Elastic Beanstalk retrieve the instance ID
if (file_exists('/etc/elasticbeanstalk/.aws-eb-system-initialized')) {
if ($awsEb = file_get_contents('/etc/elasticbeanstalk/.aws-eb-system-initialized')) {
$awsEb = json_decode($awsEb);
if (is_object($awsEb) && $awsEb->instance_id) {
$instanceID = $awsEb->instance_id;
}
}
}
// Obtain lock
$updateColumns = false; //do nothing if record already exists
$affectedRows = \Yii::$app->db->createCommand()->upsert('system_job_locks', [
'job_name' => $jobName,
'locked' => gmdate('Y-m-d H:i:s'),
'duration' => $durationMins,
'source' => $instanceID,
], $updateColumns)->execute();
// The SQL generated: INSERT INTO system_job_locks (job_name, locked, duration, source) VALUES ('some-name', '2019-04-22 17:24:39', 60, 'i-HmkDAZ9S5G5G') ON DUPLICATE KEY UPDATE job_name = job_name
if ($affectedRows == 0) {
// record already exists, check if lock has expired
$affectedRows = \Yii::$app->db->createCommand()->update('system_job_locks', [
'locked' => gmdate('Y-m-d H:i:s'),
'duration' => $durationMins,
'source' => $instanceID,
],
'job_name = :jobName AND DATE_ADD(locked, INTERVAL duration MINUTE) < NOW()', ['jobName' => $jobName]
)->execute();
// The SQL generated: UPDATE system_job_locks SET locked = '2019-04-22 17:24:39', duration = 60, source = 'i-HmkDAZ9S5G5G' WHERE job_name = 'clean-trash' AND DATE_ADD(locked, INTERVAL duration MINUTE) < NOW()
if ($affectedRows == 0) {
// We could not obtain a lock (since another process already has it) so do not execute the command
exit;
}
}
// Handle redirection of stdout and stderr
$command = str_replace('StdOUT', '>', $command);
$command = str_replace('StdERR.ditto', '2>&1', $command);
$command = str_replace('StdERR', '2>', $command);
// Execute the command as a background process so we can exit the current process
$command .= ' &';
$output = []; $exitcode = null;
exec($command, $output, $exitcode);
exit($exitcode);
}
This is the database schema I'm using:
CREATE TABLE `system_job_locks` (
`job_name` VARCHAR(50) NOT NULL,
`locked` DATETIME NOT NULL COMMENT 'UTC',
`duration` SMALLINT(5) UNSIGNED NOT NULL COMMENT 'Minutes',
`source` VARCHAR(255) NULL DEFAULT NULL,
PRIMARY KEY (`job_name`)
)