I am new to AWS and experimenting with AWS Lambda and Fargate. I have a long-running process that I have defined as an AWS Fargate containerized task. This task is triggered using ecs.runTask(taskParams, callback) api call from an AWS Lambda function. The Lambda function is triggered by a notification of a file uploaded into an S3 bucket.
You can now configure your AWS Lambda functions to run up to 15
minutes per execution. Previously, the maximum execution time
(timeout) for a Lambda function was 5 minutes. (Source: Amazon)
My question is, does ecs.runTask() run the task asynchronously inside an on-demand container without the lambda function that triggered it waiting for its completion? Does that explain how the lambda function is no longer bound by the task running time? Is this a recommended approach for long-running processes where we don't want an ECS instance just around?
Finally, what is the difference between ecs.runTask() and ecs.startTask() api calls?
asynchronously inside an on-demand container without the lambda function that triggered it waiting for its completion?
Yes. Lambda will just start it.
what is the difference between ecs.runTask() and ecs.startTask() api calls?
startTask can be only used on EC2 launch type and requires you to explicitly choose which EC2 instance to use for your task. Can't be used for Fargate and allows you to launch a single task.
runTask can be use for both EC2 and Fargate launch types. When you use runTask ECS decides where to place your tasks, e.g. which instance. Also, you can run multiple copies of a single task at once.
There are probably more differences, but I think the above are the key ones.
I'm new to Cloudwatch events and to Fargate. I want to trigger a Fargate task (Python) to run whenever a file is uploaded to a specific S3 bucket. I can get the task to run whenever I upload a file, and can see the name in the event log; however I can't figure out a simple way to read the event data in Fargate. I've been researching this the past couple of days and haven't found solution other than reading the event log or using a lambda to invoke the task and to put the event data in a message queue.
Is there a simple way to obtain the event data in Fargate with boto3? It's likely that I'm not looking in the right places or asking the right question.
Thanks
One of the easiest options that you can configure is two targets for same s3 image upload event.
Push the Same Event to SQS
launch Fargate task at the same time
Read Message Event from SQS when Fargate is up (No Lambda in between), also same task definition that will work a normal use case, make sure you exit the process after reading the message from sqs.
So in this case whenever Fargate Task up, it will read messages from the SQS.
To do this you would need to use a input transformer.
Each time a event rule is triggered a JSON object accessible to use for in the transformation.
As the event itself is not accessible within the container (like with Lambda functions), the idea is that you would actually forward key information as environment variables and manipulate in your container.
At this time it does not look like every service supports this in the console so you have the following options:
CloudFormation
Terraform
CLI
You can view a tutorial for this exact scenario from this link.
I'm creating bunch of EBS snapshots as part of AWS Lambda. I need to capture events when these snapshots complete so I can create an ec2 instance based on these.
I could use snapshot waiter but this polls and sometimes snapshot creation can take long time. I don't want Lambda to keep running for a while and plus the maximum time for Lambda seems to be five minutes. I looked at CloudWatch and AWS Config to see if I can capture snapshot events but had no luck.
There is now a new Event when Snapshots are completed in AWS Cloudwatch Events:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-cloud-watch-events.html
You are correct -- there is no notification event that signifies completion of an EBS Snapshot. Instead, you would need to check the status until the status changes to completed.
You are also correct that AWS Lambda functions can run for a maximum of 5 minutes and having a Lambda function waiting on an external process is not a good architecture.
Instead, you could break-up the architecture:
Have your existing process trigger the EBS Snapshot(s) and then push a message into an SQS queue
Schedule a Lambda function (eg every 5 minutes) to check the SQS queue. If a message exists:
Retrieve details about the instance and snapshot(s) from the message
Check the status of the snapshot(s)
If the status is completed, perform the next step in the process
The down-side is that the scheduled Lambda function will trigger even when there are no messages in the queue. The Lambda function will exit very quickly (cost: 100ms).
The alternative is to run a cron script on an Amazon EC2 instance (or on any computer connected to the Internet). A t2.nano instance is about 15.6c per day, which might be more expensive than a schedule Lambda function. If you already have an instance being used, then there would be no additional cost.
I have a situation where I have to trigger my workflow based on this condition "It has to process all files in s3 and then start again when there are files in s3". However, I found that Data Pipeline starts every scheduled duration while SWF starts and ends the job which also shut downs my EMR Cluster. Both of them are not suitable in this case. So, for a process which has to start or trigger based on a condition neither is suitable is what I found. Is there any alternative? Or is one of SWF and Data Pipeline could perform my task.
This is more like #Chris's answer's corollary. You still make use of Lambda - listen to S3 - Put Event trigger - so every time when there is a new object being create - the lamdba function would be called.
The Lambda Function can pick up the S3 object's key and put it in SQS; you can run a separate Worker Process which can pick items from the Queue.
To reiterate your statement,
It has to process all files in s3 [ Can be Done by Lambda ]
and then start again when there are files in s3 [Can be Done by SQS & EC2 ]
Look at Lambda. You can set up a trigger so that your code is invoked each time a new object is uploaded to S3.
Data Pipeline supports the concept of Preconconditions which can trigger your execution based on conditions. The S3KeyExists preconditions seems like what you're looking for. This will begin the execution of your activity when a particular S3 key exists.
Data Pipeline will also manage the creation and termination of your resource (EC2 or EMR) based on the activity's execution. If you wish to use your own EC2 instance or EMR cluster you can look into worker groups. Worker group resources are managed by you and will not be terminated by the service.
Currently I have a single server in amazon where I put all my cronjobs. I want to eliminate this single point of failure, and expose all my tasks as web services. I'd like to expose the services behind a VPC ELB to a few servers that will run the tasks when called.
Is there some service that Amazon (AWS) offers that can run a reoccurring job (really call a webservice) at scheduled intervals? I'd really like to be able to keep the cron functionality in terms of time/day specification, but farm out the HA of the driver (thing that calls endpoints at the right time) to AWS.
I like how SQS offers web endpoint(s), but from what I can tell you cant schedule them. SWF doesn't seem to be a good fit either.
AWS announced support for scheduled functions in Lambda at its 2015 re:Invent conference. With this feature users can execute Lambda functions on a scheduled basis using a cron-like syntax. The Lambda docs show an example of using Python to perform scheduled events.
Currently, the minimum resolution that a scheduled lambda can run at is 1 minute (the same as cron, but not as fine grained as systemd timers).
The Lambder project helps to simplify the use of scheduled functions on Lambda.
λ Gordon's cron example has perhaps the simplest interface for deploying scheduled lambda functions.
Original answer, saved for posterity.
As Eric Hammond and others have stated, there is no native AWS service for scheduled tasks. There are only workarounds and half solutions as mentioned in other answers.
To recap the current options:
The single-instance autoscale group that starts and stops on a schedule, as described by Eric Hammond.
Using a Simple Workflow Service timer, which is not at all intuitive. This case study mentions that JPL used SWF to build a distributed cron, but there are no implementation details. There is also a reference to a code example buried in the SWF code samples.
Run it yourself using something like cronlock.
Use something like the Unreliable Town Clock (UTC) to run Lambda functions on a schedule. Remember that Lambda cannot currently access resources within a VPC
Hopefully a better solution will come along soon.
Introducing Events in AWS Cloudwatch
You can schedule by minute, hourly, days or using CRON expression using console and without Lambda or any programming.
I just scheduled my ASP.net WEB API(HTTP Post) using SNS HTTP endpoint to execute every minute and it's working perfectly.
Is there some service that Amazon (AWS) offers that can run a reoccurring job at scheduled intervals?
This is one of a few single points of failure that people (including me) keep mentioning when designing architectures with AWS. Until Amazon solves it with a service, here's a hack I've published which is actively used by some companies.
AWS Auto Scaling can run and terminate instances using a recurring schedule specified in the cron format.
http://docs.amazonwebservices.com/AutoScaling/latest/APIReference/API_PutScheduledUpdateGroupAction.html
You can have the instance automatically run a process on startup.
If you don't know how long the job will last, you can set things up so that your job terminates the instance when it has completed.
Here's an article I wrote that walks through exact commands needed to set this up:
Running EC2 Instances on a Recurring Schedule with Auto Scaling
http://alestic.com/2011/11/ec2-schedule-instance
Starting a whole instance just to kick off a set of jobs seems a bit like overkill, but if it's a t1.micro, then it only costs a couple pennies.
That t1.micro doesn't have to do the actual work either. Your instance could inject messages into SQS or through SNS so that the other redundant servers pick up the tasks.
This a hosted third party site that can regularly call scheduled scripts on your domain.
This will not work if you need your script to run in the shell, and not as Apache.
Sounds like this might be useful to you:
http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-using-task-runner.html
Task Runner is a task agent application that polls AWS Data Pipeline
for scheduled tasks and executes them on Amazon EC2 instances, Amazon
EMR clusters, or other computational resources, reporting status as it
does so. Depending on your application, you may choose to:
Allow AWS Data Pipeline to install and manage one or more Task Runner
applications for you on computational resources that it manages
automatically. In this case, you do not need to install or configure
Task Runner as described in this section. This is the recommended
configuration.
Manually install and configure Task Runner on a computational resource
such as a long-running EC2 instance or a physical server. To do so,
use the procedures in this section.
Develop and install a custom task agent instead of Task Runner. The
procedures for doing so will depend on the implementation of the
custom task agent.
Amazon has introducted Lambda last year for NodeJS, yesterday Amazon added the features Scheduled Functions, VPC Support, and Python Support.
By leveraging Scheduled Function - a proper replacement for CRON can be attained.
More Info - http://aws.amazon.com/lambda/details/
As of August 2020, Amazon has moved the Lambda/CloudWatch events to a service called EventBridge (https://aws.amazon.com/eventbridge/). It was launched in July 2019, after most of the answers to this question.
Looks like this is a relatively new option from AWS BeanStalk:
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html#worker-periodictasks
Basically, they act like regular SQS receivers, but they're called on a cron schedule instead of in response to a SQS message.
SWF is a Web service from AWS that can be used to schedule tasks. Most of the work goes into specifying what a task and a schedule is.
http://milindparikh.blogspot.com/2015/07/introducing-diksha-aws-lambda-function.html is a scalable scheduler written against SWF.
CloudWatch Events are great, but there is a limit on their number. If you need a scale and willing to sacrifice the precision you could use DynamoDB's TTL as a timer.
The idea is to put items into a DynamoDB table with a TTL set to the time you need to run a task. DynamoDB will delete those items somewhere around the specified time (within 48 hours of expiration). Those deleted items will appear in the DynamoDB stream, associated with a table. A lambda function could listen the stream and take appropriate actions upon the deletions.
Read more in "DynamoDB TTL as an ad-hoc scheduling mechanism" by theburningmonk.com.
The AWS Elastic Load Balancers will ping your instances to check that they're healthy. You can add your cron-like tasks to the script that the ELB is pinging, and it will execute very regularly.
You'd want to add some logic so that each tasks is executed the right amount of times and at the right interval, but this could be accomplished with a database table that tracks executions. Each time the ELB pings your server, your server would check the database to see if any job is pending, and then execute that job.
The ELB will timeout if the script takes too long to execute, so it's important to not create a situation where your ELB health check will take many seconds to process the cron tasks. To overcome this, you can employ the AWS Simple Notification Service. Your ELB health check script can simply publish a message to an SNS topic, and then that topic can deliver the message via an HTTP request to your web server.
In other words:
ELB pings your EC2 instance...
EC2 instance checks for pending jobs and sends a message to SNS if any are found...
SNS notifies your app via HTTP...
The HTTP call from SNS is what actually processes the cron job