In Informatica, I can start a workflow but cannot get it to recognize my instance name in the session log and Workflow Monitor.
The workflow starts but in the session log it displays this:
Workflow wf_Tenter image description hereemp started with run id [22350], run instance name [], run type [Concurrent Run with Un[enter image description here][1]ique Instance Name]
Instance name is blank.
My command is:
pmcmd startworkflow -sv <service> -d <domain> -u <user> -p <password> -f <folder> -rin INST1 -paramfile <full param file path name> wf_Temp
I have edited the workflow and selected the checkbox Configure Current Execution. Inside Configure Concurrent Execution button, I have created three instances: INST1, INST2, INST3, but without any associated parameter files. All parameter files are blank.
I understand, I think, that in order to start a workflow with PMCMD I must pass in one of the configured instance names (i.e. INST1, INST2, INST3, etc.)
If I execute the PMCMD command from Putty a second time to see the second instance run, I receive a message that workflow is still running and I have to wait? Why? I have checked the Concurrent Workflow box in the workflow.
ERROR: Workflow [wf_Temp]: Could not start execution of this workflow because the current run on this Integration Service has not completed yet.
Disconnecting from Integration Service
So, I think I'm close, but am missing something. The workflow runs with the parameter file I pass in PMCMD but the instance name seems to be ignored.
Further. Do, I have to pre-configure instance names in the Workflow manager? Is the PMCMD instance and parameter file parameters enough? It doesn't seem quite so dynamic if Instances have to be pre-defined in the workflows.
Thanks.
#MacieJG
Here's the screenshots from Putty when I run the command. You can see the instance name DALLAS is being passed through the PMCMD OK. No combination ever gets the Instance name. I did not include the pics of your suggested Test 1, but results were same.. still no instance.
Here's my complete test as requested in a comment above. I tried my best to put everything you may need here, but if I missed anything, just let me know. So here goes...
I've created a very simple workflow to run with instance name. It uses a timer to wait and a command tast to write the instance name to a file:
The concurrent execution has been set up in the most simple way:
Now, I've prepared the followig batch to run the workflow (just user & password removed):
SET "PMCMD=C:\Informatica\9.5.1\clients\PowerCenterClient\CommandLineUtilities\PC\server\bin\pmcmd"
%PMCMD% startworkflow -sv Dev_IS -d Domain_vic-vpc -u ####### -p ####### -f Dev01 -rin GLASGOW wf_Instance_Test
%PMCMD% startworkflow -sv Dev_IS -d Domain_vic-vpc -u ####### -p ####### -f Dev01 -rin FRANKFURT wf_Instance_Test
%PMCMD% startworkflow -sv Dev_IS -d Domain_vic-vpc -u ####### -p ####### -f Dev01 -rin GLASGOW wf_Instance_Test
It runs three instances, two of them with same name, just to test it. I run the batch the following way to capture the output:
pmStartTestWF.bat > c:\MG\pmStartTestWF.log
Once I execute it, here what I see in workflow monitor:
Just as expected, three instances executed and properly displayed. File output looks fine as well:
The output of pmcmd can be found here. Full definition of my test workflow is available here.
I really hope this will help you somehow. Feel free to let me know if you'd find anything missing here. Good luck!
You don't need to pre-configure instance names in workflow. Passing the instance name in pmcmd along with parameter filename is enough.
try this: pmcmd startworkflow -sv (service) -d (domain) -u (user) -p (password) -f (folder) -paramfile (full param file path name) -rin INST1 wf_Temp
To be precise: when you configure Concurrent Execution, you can specify if you:
allow concurrent run with same instance name
allow concurrent run only with unique instance name
In addition to that you may, but don't have to, indicate which instance should use which parameter file, so it won't be need to mention it while executing. But that's a separate feature.
Now, if you've chosen the first one, you will be able to invoke the WF multiple times with the very same command. If you've chosen the second one and try this, you will get the 'WF is already running' error.
The trouble is that your example seems correct at first glance. As per the log message:
Workflow wf_Temp started with run id [22350], run instance name [], run type [Concurrent Run with Unique Instance Name]
So you're allowing unique instances only. It seems that the instance name has not been used. First execution does not set the instance name, so similar second execution won't use it either and will get rejected as this is the same instance name (i.e. None).
You may try to change the setting to Allow concurrent run with same instance name, this shall allow the secon execution, but does not solve the main issue. For some reason the instance name does not get passed.
Please verify your command against the docs referenced below. Try to match the order perhaps. Please share some more info if it still fails.
Looking at the docs:
pmcmd StartWorkflow
<<-service|-sv> service [<-domain|-d> domain] [<-timeout|-t> timeout]>
<<-user|-u> username|<-uservar|-uv> userEnvVar>
<<-password|-p> password|<-passwordvar|-pv> passwordEnvVar>
[<<-usersecuritydomain|-usd> usersecuritydomain|<-usersecuritydomainvar|-usdv>
userSecuritydomainEnvVar>]
[<-folder|-f> folder]
[<-startfrom> taskInstancePath]
[<-recovery|-norecovery>]
[<-paramfile> paramfile]
[<-localparamfile|-lpf> localparamfile]
[<-osprofile|-o> OSUser]
[-wait|-nowait]
[<-runinsname|-rin> runInsName]
workflow
While creating a new AWS EC2 instance using the EC2 command line API, I passed some user data to the new instance.
How can I know whether that user data executed or not?
You can verify using the following steps:
SSH on launch EC2 instance.
Check the log of your user data script in:
/var/log/cloud-init.log and
/var/log/cloud-init-output.log
You can see all logs of your user data script, and it will also create the /etc/cloud folder.
Just for reference, you can check if the user data executed by taking a look at the system log from the EC2 console. Right click on your instance -
In the new interface: Monitor and Troubleshoot > Get System Log
In the old interface: Instance Settings > Get System log
This should open a modal window with the system logs
It might also be useful for you to see what the userdata looks like when it's being executed during the bootstrapping of the instance. This is especially true if you are passing in environmental variables or flags from the CloudFormation template. You can see how the UserData is being executed in two different ways:
1. From within the instance:
# Get instance ID
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
# Print user data
sudo cat /var/lib/cloud/instances/$INSTANCE_ID/user-data.txt
2. From outside the instance
Note: this will only work if you have configured the UserData shell in such a way that it will output the commands it runs.
For bash, you can do this like as follows:
"#!/bin/bash\n",
"set -x\n",
Right click on the EC2 instance from the EC2 console -> Monitor and Troubleshoot -> Get system log. Download the log file and look for something a section that looks like this:
ip-172-31-76-56 login: 2021/10/25 17:13:47Z: Amazon SSM Agent v3.0.529.0 is running
2021/10/25 17:13:47Z: OsProductName: Ubuntu
2021/10/25 17:13:47Z: OsVersion: 20.04
[ 45.636562] cloud-init[856]: Cloud-init v. 21.2-3...
[ 47.749983] cloud-init[896]: + echo hello world
this is what you would see if the UserData was configured like this:
"#!/bin/bash\n",
"set -x\n",
"echo hello world"
Debugging user data scripts on Amazon EC2 is a bit awkward indeed, as there is usually no way to actively hook into the process, so one ideally would like to gain Real time access to user-data script output as summarized in Eric Hammond's article Logging user-data Script Output on EC2 Instances:
The recent Ubuntu AMIs still send user-data script to the console
output, so you can view it remotely, but it is no longer available in
syslog on the instance. The console output is only updated a few
minutes after the instance boots, reboots, or terminates, which forces
you to wait to see the output of the user-data script as well as not
capturing output that might come out after the snapshot.
Depending on your setup you might want to ship the logs to a remote logging facility like Loggly right away, but getting this installed early enough can obviously be kind of a chicken/egg problem (though it works great if the AMI happens to be configured like so already).
Enable logging for your user data
Eric Hammond, in "Logging user-data Script Output on EC2 Instances (2010, Hammond)", suggests:
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
Take care to put a space between the two > > characters at the beginning of the statement.
Here’s a complete user-data script as an example:
#!/bin/bash -ex
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
echo BEGIN
date '+%Y-%m-%d %H:%M:%S'
echo END
Put this in userdata
touch /tmp/file2.txt
Once the instance is up you can check whether the file is created or not. Based on this you can tell if the userdata is executed or not.
Have your user data create a file in your ec2's /tmp directory to see if it works:
bob.txt:
#!/bin/sh
echo 'Woot!' > /home/ec2-user/user-script-output.txt
Then launch with:
ec2-run-instances -f bob.txt -t t1.micro -g ServerPolicy ami-05cf5c6d -v
I would like to know if there is a way to setup a cronjob/task to execute every minute. Currently any of my instances should be able to run this task.
This is what I have tried to do in the config files without success:
container_commands:
01cronjobs:
command: echo "*/1 * * * * root php /etc/httpd/myscript.php"
I'm not really sure if this is the correct way to do it
Any ideas?
This is how I added a cron job to Elastic Beanstalk:
Create a folder at the root of your application called .ebextensions if it doesn't exist already. Then create a config file inside the .ebextensions folder. I'll use example.config for illustration purposes. Then add this to example.config
container_commands:
01_some_cron_job:
command: "cat .ebextensions/some_cron_job.txt > /etc/cron.d/some_cron_job && chmod 644 /etc/cron.d/some_cron_job"
leader_only: true
This is a YAML configuration file for Elastic Beanstalk. Make sure when you copy this into your text editor that your text editor uses spaces instead of tabs. Otherwise you'll get a YAML error when you push this to EB.
So what this does is create a command called 01_some_cron_job. Commands are run in alphabetical order so the 01 makes sure it's run as the first command.
The command then takes the contents of a file called some_cron_job.txt and adds it to a file called some_cron_job in /etc/cron.d.
The command then changes the permissions on the /etc/cron.d/some_cron_job file.
The leader_only key ensures the command is only run on the ec2 instance that is considered the leader. Rather than running on every ec2 instance you may have running.
Then create a file called some_cron_job.txt inside the .ebextensions folder. You will place your cron jobs in this file.
So for example:
# The newline at the end of this file is extremely important. Cron won't run without it.
* * * * * root /usr/bin/php some-php-script-here > /dev/null
So this cron job will run every minute of every hour of every day as the root user and discard the output to /dev/null. /usr/bin/php is the path to php. Then replace some-php-script-here with the path to your php file. This is obviously assuming your cron job needs to run a PHP file.
Also, make sure the some_cron_job.txt file has a newline at the end of the file just like the comment says. Otherwise cron won't run.
Update:
There is an issue with this solution when Elastic Beanstalk scales up your instances. For example, lets say you have one instance with the cron job running. You get an increase in traffic so Elastic Beanstalk scales you up to two instances. The leader_only will ensure you only have one cron job running between the two instances. Your traffic decreases and Elastic Beanstalk scales you down to one instance. But instead of terminating the second instance, Elastic Beanstalk terminates the first instance that was the leader. You now don't have any cron jobs running since they were only running on the first instance that was terminated. See the comments below.
Update 2:
Just making this clear from the comments below:
AWS has now protection against automatic instance termination. Just enable it on your leader instance and you're good to go. – Nicolás Arévalo Oct 28 '16 at 9:23
This is the official way to do it now (2015+). Please try this first, it's by far easiest method currently available and most reliable as well.
According to current docs, one is able to run periodic tasks on their so-called worker tier.
Citing the documentation:
AWS Elastic Beanstalk supports periodic tasks for worker environment tiers in environments running a predefined configuration with a solution stack that contains "v1.2.0" in the container name. You must create a new environment.
Also interesting is the part about cron.yaml:
To invoke periodic tasks, your application source bundle must include a cron.yaml file at the root level. The file must contain information about the periodic tasks you want to schedule. Specify this information using standard crontab syntax.
Update: We were able to get this work. Here are some important gotchas from our experience (Node.js platform):
When using cron.yaml file, make sure you have latest awsebcli, because older versions will not work properly.
It is also vital to create new environment (at least in our case it was), not just clone old one.
If you want to make sure CRON is supported on your EC2 Worker Tier instance, ssh into it (eb ssh), and run cat /var/log/aws-sqsd/default.log. It should report as aws-sqsd 2.0 (2015-02-18). If you don't have 2.0 version, something gone wrong when creating your environment and you need to create new one as stated above.
Regarding jamieb's response, and as alrdinleal mentions, you can use the 'leader_only' property to ensure that only one EC2 instance runs the cron job.
Quote taken from http://docs.amazonwebservices.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html:
you can use leader_only. One instance is chosen to be the leader in an Auto Scaling group. If the leader_only value is set to true, the command runs only on the instance that is marked as the leader.
Im trying to achieve a similar thing on my eb, so will update my post if I solve it.
UPDATE:
Ok, I now have working cronjobs using the following eb config:
files:
"/tmp/cronjob" :
mode: "000777"
owner: ec2-user
group: ec2-user
content: |
# clear expired baskets
*/10 * * * * /usr/bin/wget -o /dev/null http://blah.elasticbeanstalk.com/basket/purge > $HOME/basket_purge.log 2>&1
# clean up files created by above cronjob
30 23 * * * rm $HOME/purge*
encoding: plain
container_commands:
purge_basket:
command: crontab /tmp/cronjob
leader_only: true
commands:
delete_cronjob_file:
command: rm /tmp/cronjob
Essentially, I create a temp file with the cronjobs and then set the crontab to read from the temp file, then delete the temp file afterwards. Hope this helps.
I spoke to an AWS support agent and this is how we got this to work for me. 2015 solution:
Create a file in your .ebextensions directory with your_file_name.config.
In the config file input:
files:
"/etc/cron.d/cron_example":
mode: "000644"
owner: root
group: root
content: |
* * * * * root /usr/local/bin/cron_example.sh
"/usr/local/bin/cron_example.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
/usr/local/bin/test_cron.sh || exit
echo "Cron running at " `date` >> /tmp/cron_example.log
# Now do tasks that should only run on 1 instance ...
"/usr/local/bin/test_cron.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
METADATA=/opt/aws/bin/ec2-metadata
INSTANCE_ID=`$METADATA -i | awk '{print $2}'`
REGION=`$METADATA -z | awk '{print substr($2, 0, length($2)-1)}'`
# Find our Auto Scaling Group name.
ASG=`aws ec2 describe-tags --filters "Name=resource-id,Values=$INSTANCE_ID" \
--region $REGION --output text | awk '/aws:autoscaling:groupName/ {print $5}'`
# Find the first instance in the Group
FIRST=`aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names $ASG \
--region $REGION --output text | awk '/InService$/ {print $4}' | sort | head -1`
# Test if they're the same.
[ "$FIRST" = "$INSTANCE_ID" ]
commands:
rm_old_cron:
command: "rm *.bak"
cwd: "/etc/cron.d"
ignoreErrors: true
This solution has 2 drawbacks:
On subsequent deployments, Beanstalk renames the existing cron script as .bak, but cron will still run it. Your Cron now executes twice on the same machine.
If your environment scales up, you get several instances, all running your cron script. This means your mail shots are repeated, or your database archives duplicated
Workaround:
Ensure any .ebextensions script which creates a cron also removes the .bak files on subsequent deployments.
Have a helper script which does the following: -- Gets the current Instance ID from the Metadata -- Gets the current Auto
Scaling Group name from the EC2 Tags -- Gets the list of EC2
Instances in that Group, sorted alphabetically. -- Takes the first
instance from that list. -- Compares the Instance ID from step 1
with the first Instance ID from step 4.
Your cron scripts can then use this helper script to determine if they should execute.
Caveat:
The IAM Role used for the Beanstalk instances needs ec2:DescribeTags and autoscaling:DescribeAutoScalingGroups permissions
The instances chosen from are those shown as InService by Auto Scaling. This does not necessarily mean they are fully booted up and ready to run your cron.
You would not have to set the IAM Roles if you are using the default beanstalk role.
As mentioned above, the fundamental flaw with establishing any crontab configuration is that it only happens at deployment. As the cluster gets auto-scaled up, and then back down, it is favored to also be the first server turned off. In addition there would be no fail-over, which for me was critical.
I did some research, then talked with our AWS account specialist to bounce ideas and valid the solution I came up with. You can accomplish this with OpsWorks, although it's bit like using a house to kill a fly. It is also possible to use Data Pipeline with Task Runner, but this has limited ability in the scripts that it can execute, and I needed to be able to run PHP scripts, with access to the whole code base. You could also dedicate an EC2 instance outside of the ElasticBeanstalk cluster, but then you have no fail-over again.
So here is what I came up with, which apparently is unconventional (as the AWS rep commented) and may be considered a hack, but it works and is solid with fail-over. I chose a coding solution using the SDK, which I'll show in PHP, although you could do the same method in any language you prefer.
// contains the values for variables used (key, secret, env)
require_once('cron_config.inc');
// Load the AWS PHP SDK to connection to ElasticBeanstalk
use Aws\ElasticBeanstalk\ElasticBeanstalkClient;
$client = ElasticBeanstalkClient::factory(array(
'key' => AWS_KEY,
'secret' => AWS_SECRET,
'profile' => 'your_profile',
'region' => 'us-east-1'
));
$result = $client->describeEnvironmentResources(array(
'EnvironmentName' => AWS_ENV
));
if (php_uname('n') != $result['EnvironmentResources']['Instances'][0]['Id']) {
die("Not the primary EC2 instance\n");
}
So walking through this and how it operates... You call scripts from crontab as you normally would on every EC2 instance. Each script includes this at the beginning (or includes a single file for each, as I use it), which establishes an ElasticBeanstalk object and retrieves a list of all instances. It uses only the first server in the list, and checks if it matches itself, which if it does it continues, otherwise it dies and closes out. I've checked and the list returned seems to be consistent, which technically it only needs to be consistent for a minute or so, as each instance executes the scheduled cron. If it does change, it wouldn't matter, since again it only is relevant for that small window.
This isn't elegant by any means, but suited our specific needs - which was not to increase cost with an additional service or have to have a dedicated EC2 instance, and would have fail-over in case of any failure. Our cron scripts run maintenance scripts which get placed into SQS and each server in the cluster helps execute. At least this may give you an alternate option if it fits your needs.
-Davey
If you're using Rails, you can use the whenever-elasticbeanstalk gem. It allows you to run cron jobs on either all instances or just one. It checks every minute to ensure that there is only one "leader" instance, and will automatically promote one server to "leader" if there are none. This is needed since Elastic Beanstalk only has the concept of leader during deployment and may shut down any instance at any time while scaling.
UPDATE
I switched to using AWS OpsWorks and am no longer maintaining this gem. If you need more functionality than is available in the basics of Elastic Beanstalk, I highly recommend switching to OpsWorks.
You really don't want to be running cron jobs on Elastic Beanstalk. Since you'll have multiple application instances, this can cause race conditions and other odd problems. I actually recently blogged about this (4th or 5th tip down the page). The short version: Depending on the application, use a job queue like SQS or a third-party solution like iron.io.
2017: If you are using Laravel5+
You just need 2 minutes to configure it:
create a Worker Tier
install laravel-aws-worker
composer require dusterio/laravel-aws-worker
add a cron.yaml to the root folder:
Add cron.yaml to the root folder of your application (this can be a
part of your repo or you could add this file right before deploying to
EB - the important thing is that this file is present at the time of
deployment):
version: 1
cron:
- name: "schedule"
url: "/worker/schedule"
schedule: "* * * * *"
That's it!
All your task in App\Console\Kernel will now be executed
Detailed instructions and explainations: https://github.com/dusterio/laravel-aws-worker
How to write tasks inside of Laravel: https://laravel.com/docs/5.4/scheduling
A more readable solution using files instead of container_commands:
files:
"/etc/cron.d/my_cron":
mode: "000644"
owner: root
group: root
content: |
# override default email address
MAILTO="example#gmail.com"
# run a Symfony command every five minutes (as ec2-user)
*/10 * * * * ec2-user /usr/bin/php /var/app/current/app/console do:something
encoding: plain
commands:
# delete backup file created by Elastic Beanstalk
clear_cron_backup:
command: rm -f /etc/cron.d/watson.bak
Note the format differs from the usual crontab format in that it specifies the user to run the command as.
My 1 cent of contribution for 2018
Here is the right way to do it (using django/python and django_crontab app):
inside .ebextensions folder create a file like this 98_cron.config:
files:
"/tmp/98_create_cron.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/sh
cd /
sudo /opt/python/run/venv/bin/python /opt/python/current/app/manage.py crontab remove > /home/ec2-user/remove11.txt
sudo /opt/python/run/venv/bin/python /opt/python/current/app/manage.py crontab add > /home/ec2-user/add11.txt
container_commands:
98crontab:
command: "mv /tmp/98_create_cron.sh /opt/elasticbeanstalk/hooks/appdeploy/post && chmod 774 /opt/elasticbeanstalk/hooks/appdeploy/post/98_create_cron.sh"
leader_only: true
It needs to be container_commands instead of commands
The latest example from Amazon is the easiest and most efficient (periodic tasks):
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html
where you create a separate worker tier to execute any of your cron jobs. Create the cron.yaml file and place it in your root folder. One issue I had (after cron did not seem to be executing) was finding that my CodePipeline did not have authority to perform a dynamodb modification. Based on that after adding FullDynamoDB access under IAM -> roles -> yourpipeline and redeploying (elastic beanstalk) it worked perfectly.
Someone was wondering about the leader_only auto scaling problems when new leaders arise. I can't seem to figure out how to reply to their comments, but see this link: http://blog.paulopoiati.com/2013/08/25/running-cron-in-elastic-beanstalk-auto-scaling-environment/
So we've been struggling with this for a while and after some discussion with an AWS rep I've finally come up with what I think is the best solution.
Using a worker tier with cron.yaml is definitely the easiest fix. However, what the documentation doesn't make clear is that this will put the job at the end of the SQS queue you're using to actually run your jobs. If your cron jobs are time sensitive (as many are), this isn't acceptable, since it would depend on the size of the queue. One option is to use a completely separate environment just to run cron jobs, but I think that's overkill.
Some of the other options, like checking to see if you're the first instance in the list, aren't ideal either. What if the current first instance is in the process of shutting down?
Instance protection can also come with issues - what if that instance gets locked up / frozen?
What's important to understand is how AWS itself manages the cron.yaml functionality. There is an SQS daemon which uses a Dynamo table to handle "leader election". It writes to this table frequently, and if the current leader hasn't written in a short while, the next instance will take over as leader. This is how the daemon decides which instance to fire the job into the SQS queue.
We can repurpose the existing functionality rather than trying to rewrite our own. You can see the full solution here: https://gist.github.com/dorner/4517fe2b8c79ccb3971084ec28267f27
That's in Ruby, but you can easily adapt it to any other language that has the AWS SDK. Essentially, it checks the current leader, then checks the state to make sure it's in a good state. It'll loop until there is a current leader in a good state, and if the current instance is the leader, execute the job.
The best way to do this is to use an Elastic Beanstalk Worker Environment (see "Option 1" below). However, this will add to your server costs. If you don't want to do this, see "Option 2" below for how to configure cron itself.
Option 1: Use Elastic Beanstalk Worker environments
Amazon has support for Elastic Beanstalk Worker Environments. They are Elastic Beanstalk managed environments that come with an SQS queue which you can enqueue tasks onto. You can also give them a cron config that will automatically enqueue the task on a recurring schedule. Then, rather than receiving requests from a load balancer, the servers in a worker environment each have a daemon (managed by Elastic Beanstalk) that polls the queue for tasks and calls the appropriate web endpoint when they get a message on the queue. Worker environments have several benefits over running cron yourself:
Performance. Your tasks are now running on dedicated servers instead of competing for CPU and memory with web requests. You can also have different specs for the worker servers (ex. you can have more memory on just the worker servers).
Scalability. You can also scale up your number of worker servers to more than 1 in order to handle large task loads.
Ad-hoc Tasks. Your code can enqueue ad-hoc tasks as well as scheduled ones.
Standardization. You write tasks as web endpoints rather than needing to configure your own task framework, which lets your standardize your code and tooling.
If you just want a cron replacement, all you need to do is make a file called cron.yaml at the top level of your project, with config like the following:
cron.yaml
version: 1
cron:
- name: "hourly"
url: "/tasks/hourly"
schedule: "0 */1 * * *"
This will call the url /tasks/hourly once an hour.
If you are deploying the same codebase to web and worker environments, you should have the task URLs require an environment variable that you set on worker environments and not web environments. This way, your task endpoints are not exposed to the world (task servers by default do not accept incoming HTTP requests, as the only thing making calls to them is the on-server daemon).
The full docs are here: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html
Option 2: Configure Cron
If you want to run cron, you need to make sure it's running on only one server. The leader_only flag in .ebextensions config isn't sufficient because servers don't reliably stay the leader. This can be fixed by deleting the cron config if present on any server as the first step of a deploy and then installing it on just one server using leader_only. Here is an example .ebextensions config file that accomplishes this:
.ebextensions/cron.config
container_commands:
01_remove_cron_jobs:
command: "rm /etc/cron.d/cronjobs || exit 0"
02_set_up_cron:
command: "cat .ebextensions/cronjobs.txt > /etc/cron.d/cronjobs && chmod 644 /etc/cron.d/cronjobs"
leader_only: true
This config file assumes the existence of a file .ebextensions/cronjobs.txt. This file contains your actual cron config. Note that in order to have environment variables loaded and your code in scope, you need to have code that does this baked into each command. The following is an example cron config that works on an Amazon Linux 2 based Python environment:
.ebextensions/cronjobs.txt
SHELL=/bin/bash
PROJECT_PATH=/var/app/current
ENV_PATH=/opt/elasticbeanstalk/deployment/env
# m h dom mon dow user command
0 * * * * ec2-user set -a; source <(sudo cat $ENV_PATH) && cd $PROJECT_PATH && python HOURLY_COMMAND > /dev/null
# Cron requires a newline at the end of the file
Here is a full explanation of the solution:
http://blog.paulopoiati.com/2013/08/25/running-cron-in-elastic-beanstalk-auto-scaling-environment/
To control whether Auto Scaling can terminate a particular instance when scaling in, use instance protection. You can enable the instance protection setting on an Auto Scaling group or an individual Auto Scaling instance. When Auto Scaling launches an instance, the instance inherits the instance protection setting of the Auto Scaling group. You can change the instance protection setting for an Auto Scaling group or an Auto Scaling instance at any time.
http://docs.aws.amazon.com/autoscaling/latest/userguide/as-instance-termination.html#instance-protection
I had another solution to this if a php file needs to be run through cron and if you had set any NAT instances then you can put cronjob on NAT instance and run php file through wget.
here is a fix incase you want to do this in PHP. You just need cronjob.config in your .ebextensions folder to get it to work like this.
files:
"/etc/cron.d/my_cron":
mode: "000644"
owner: root
group: root
content: |
empty stuff
encoding: plain
commands:
01_clear_cron_backup:
command: "rm -f /etc/cron.d/*.bak"
02_remove_content:
command: "sudo sed -i 's/empty stuff//g' /etc/cron.d/my_cron"
container_commands:
adding_cron:
command: "echo '* * * * * ec2-user . /opt/elasticbeanstalk/support/envvars && /usr/bin/php /var/app/current/index.php cron sendemail > /tmp/sendemail.log 2>&1' > /etc/cron.d/my_cron"
leader_only: true
the envvars gets the environment variables for the files. You can debug the output on the tmp/sendemail.log as above.
Hope this helps someone as it surely helped us!
Based on the principles of the answer from user1599237, where you let the cron jobs run on all instances but then instead in the beginning of the jobs determine if they should be allowed to run, I have made another solution.
Instead of looking at the running instances (and having to store your AWS key and secret) I'm using the MySQL database that I'm already connecting to from all instances.
It has no downsides, only positives:
no extra instance or expenses
rock solid solution - no chance of double execution
scalable - automatically works as your instances are scaled up and down
failover - automatically works in case an instance has a failure
Alternatively, you could also use a commonly shared filesystem (like AWS EFS via the NFS protocol) instead of a database.
The following solution is created within the PHP framework Yii but you can easily adapt it for another framework and language. Also the exception handler Yii::$app->system is a module of my own. Replace it with whatever you are using.
/**
* Obtain an exclusive lock to ensure only one instance or worker executes a job
*
* Examples:
*
* `php /var/app/current/yii process/lock 60 empty-trash php /var/app/current/yii maintenance/empty-trash`
* `php /var/app/current/yii process/lock 60 empty-trash php /var/app/current/yii maintenance/empty-trash StdOUT./test.log`
* `php /var/app/current/yii process/lock 60 "empty trash" php /var/app/current/yii maintenance/empty-trash StdOUT./test.log StdERR.ditto`
* `php /var/app/current/yii process/lock 60 "empty trash" php /var/app/current/yii maintenance/empty-trash StdOUT./output.log StdERR./error.log`
*
* Arguments are understood as follows:
* - First: Duration of the lock in minutes
* - Second: Job name (surround with quotes if it contains spaces)
* - The rest: Command to execute. Instead of writing `>` and `2>` for redirecting output you need to write `StdOUT` and `StdERR` respectively. To redirect stderr to stdout write `StdERR.ditto`.
*
* Command will be executed in the background. If determined that it should not be executed the script will terminate silently.
*/
public function actionLock() {
$argsAll = $args = func_get_args();
if (!is_numeric($args[0])) {
\Yii::$app->system->error('Duration for obtaining process lock is not numeric.', ['Args' => $argsAll]);
}
if (!$args[1]) {
\Yii::$app->system->error('Job name for obtaining process lock is missing.', ['Args' => $argsAll]);
}
$durationMins = $args[0];
$jobName = $args[1];
$instanceID = null;
unset($args[0], $args[1]);
$command = trim(implode(' ', $args));
if (!$command) {
\Yii::$app->system->error('Command to execute after obtaining process lock is missing.', ['Args' => $argsAll]);
}
// If using AWS Elastic Beanstalk retrieve the instance ID
if (file_exists('/etc/elasticbeanstalk/.aws-eb-system-initialized')) {
if ($awsEb = file_get_contents('/etc/elasticbeanstalk/.aws-eb-system-initialized')) {
$awsEb = json_decode($awsEb);
if (is_object($awsEb) && $awsEb->instance_id) {
$instanceID = $awsEb->instance_id;
}
}
}
// Obtain lock
$updateColumns = false; //do nothing if record already exists
$affectedRows = \Yii::$app->db->createCommand()->upsert('system_job_locks', [
'job_name' => $jobName,
'locked' => gmdate('Y-m-d H:i:s'),
'duration' => $durationMins,
'source' => $instanceID,
], $updateColumns)->execute();
// The SQL generated: INSERT INTO system_job_locks (job_name, locked, duration, source) VALUES ('some-name', '2019-04-22 17:24:39', 60, 'i-HmkDAZ9S5G5G') ON DUPLICATE KEY UPDATE job_name = job_name
if ($affectedRows == 0) {
// record already exists, check if lock has expired
$affectedRows = \Yii::$app->db->createCommand()->update('system_job_locks', [
'locked' => gmdate('Y-m-d H:i:s'),
'duration' => $durationMins,
'source' => $instanceID,
],
'job_name = :jobName AND DATE_ADD(locked, INTERVAL duration MINUTE) < NOW()', ['jobName' => $jobName]
)->execute();
// The SQL generated: UPDATE system_job_locks SET locked = '2019-04-22 17:24:39', duration = 60, source = 'i-HmkDAZ9S5G5G' WHERE job_name = 'clean-trash' AND DATE_ADD(locked, INTERVAL duration MINUTE) < NOW()
if ($affectedRows == 0) {
// We could not obtain a lock (since another process already has it) so do not execute the command
exit;
}
}
// Handle redirection of stdout and stderr
$command = str_replace('StdOUT', '>', $command);
$command = str_replace('StdERR.ditto', '2>&1', $command);
$command = str_replace('StdERR', '2>', $command);
// Execute the command as a background process so we can exit the current process
$command .= ' &';
$output = []; $exitcode = null;
exec($command, $output, $exitcode);
exit($exitcode);
}
This is the database schema I'm using:
CREATE TABLE `system_job_locks` (
`job_name` VARCHAR(50) NOT NULL,
`locked` DATETIME NOT NULL COMMENT 'UTC',
`duration` SMALLINT(5) UNSIGNED NOT NULL COMMENT 'Minutes',
`source` VARCHAR(255) NULL DEFAULT NULL,
PRIMARY KEY (`job_name`)
)