What example belong does is, a cronjob runs every minute to trigger a lambda function written in Golang. If the lambda returns an error, a message is put into a DLQ straight away. However, what I am struggling to work out is that the retry logic. A message should go to DLQ only after third lambda try which is what I am trying to accomplish. If you see I am missing something in AWS commands below please let me know.
What I tried so far is that, I created an additional normal queue on top of DQL and linked it to lambda instead with --dead-letter-config. Then linked DLQ to target with DeadLetterConfig with RetryPolicy. I am not sure if this is how the whole thing is designed to work but I think there may be more components required for this. Not even sure if this is correct either!
Lambda (main.go)
package main
import (
"context"
"fmt"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
)
func main() {
lambda.Start(handle)
}
func handle(_ context.Context, event events.CloudWatchEvent) error {
detail, err := event.Detail.MarshalJSON()
if err != nil {
return err
}
if string(detail) == `{"ok": "yes"}` {
return nil
}
return fmt.Errorf("not ok")
}
AWS steps
GOOS=linux CGO_ENABLED=0 go build -ldflags "-s -w" -o main main.go
zip main.zip main
# Create rule
aws --profile localstack --endpoint-url http://localhost:4566 events put-rule \
--name test-rule \
--schedule-expression 'cron(* * * * *)'
# Create DLQ
aws --profile localstack --endpoint-url http://localhost:4566 sqs create-queue \
--queue-name test-dead-letter-queue \
--attributes '{}'
# Create lambda with DLQ
aws --profile localstack --endpoint-url http://localhost:4566 lambda create-function \
--function-name test-lambda \
--handler main \
--runtime go1.x \
--role test-role \
--dead-letter-config '{"TargetArn":"arn:aws:sqs:eu-west-1:000000000000:test-dead-letter-queue"}' \
--zip-file fileb://main.zip
# Create lambda rule (purposely causes lambda error!)
aws --profile localstack --endpoint-url http://localhost:4566 events put-targets \
--rule test-rule \
--targets '[{"Id":"1","Arn":"arn:aws:lambda:eu-west-1:000000000000:function:test-lambda","Input":"{\"ok\":\"no\"}"}]'
I am not seeing what AWS doc says happening.
Error handling for a given event source depends on how Lambda is invoked. Amazon CloudWatch Events is configured to invoke a Lambda function asynchronously.
Asynchronous invocation – Asynchronous events are queued before being used to invoke the Lambda function. If AWS Lambda is unable to fully process the event, it will automatically retry the invocation twice, with delays between retries.
You have a asynchronous schedule rule -> lambda setup. Note that async retries are made with a delay (see below). Perhaps you are not noticing the retries because the backoff time >= your per-minute schedule?
Lambda error handling and automatic retry logic is different for async and sync invocation patterns:
Asynchronous (e.g. event-triggered) Lambda Invocations
Two async lambda retries are made with a 1 minute and 2 minute backoff, respectively. Async lambdas will be retried in the case of both invocation and function errors. After the retries are exhausted, the event will be sent to the lambda's DLQ or the lambda's failure destination*, if configured.
Synchronous (e.g. SQS-triggered) Lambda Invocations
An alternative is to use SQS as an event source (schedule rule -> Queue -> Lambda). In this synchronous scenario, retry logic and the DLQ is configured on the Queue itself, not on the Lambda.
* Destinations are a newer alternative to async lambda DLQs: "Destinations and DLQs can be used together and at the same time although Destinations should be considered a more preferred solution."
I'm working on a Lambda function to process AWS GuardDuty findings.
I'd like to generate sample events, which is easily done using the CreateSampleFindings API call or create-sample-findings cli command.
I have a custom cloudwatch rule that responds to the following event Pattern which triggers my Lambda function:
{
"detail-type": [
"GuardDuty Finding"
],
"source": [
"aws.guardduty"
]
}
Generating the first sample finding easily triggers a cloudwatch event
$ aws guardduty create-sample-findings \
--detector-id abcd12345efgh6789 \
--finding-types Recon:EC2/PortProbeUnprotectedPort
However when I call this same command again, the count of the finding in guard duty increments, but no more cloudwatch events are generated.
$ aws guardduty get-findings \
--detector-id abcd12345efgh6789 \
--finding-ids zyxwv987654acbde1234 \
--query "Findings[].Service.Count" --output text
$ 2
I understand why this behavior is in place, as the findings are grouped by unique signature and triggering cloudwatch events for each instance of a unique finding would be too much noise
However for developing/debugging purposes, is there a way I can generate multiple sample events that will trigger a cloudwatch event?
For anyone that comes across this for testing purposes disabling GuardDuty and then reenabling allows you to regenerate sample findings that trigger the CloudWatch event. This method has worked for me while creating a log forwarder for GuardDuty.
As #jl-dos has pointed out you can just disable/enable GD. But what that effectively does is to delete all findings for this GD instance, so when you go to create sample findings they are brand new an trigger the CloudWatch events.
The other option I've found is to archive the current findings. Then when you create new sample findings they will come out as brand new ones and not just increment the counters. This should also trigger a CloudWatch event.
To do that use a combination of aws guardduty get-findings and aws guardduty archive-findings commands.
Is there a way to stream an AWS Log Group to multiple Elasticsearch Services or Lambda functions?
AWS only seems to allow one ES or Lambda, and I've tried everything at this point. I've even removed the ES subscription service for the Log Group, created individual Lambda functions, created the CloudWatch Log Trigger, and I can only apply the same CloudWatch Log trigger on one Lambda function.
Here is what I'm trying to accomplish:
CloudWatch Log Group ABC -> No Filter -> Elasticsearch Service #1
CloudWatch Log Group ABC -> Filter: "XYZ" -> Elasticsearch Service #2
Basically, I need one ES cluster to store all logs, and another to only have a subset of filtered logs.
Is this possible?
I've ran into this limitation as well. I have two Lambda's (doing different things) that need to subscribe to the same CloudWatch Log Group.
What I ended up using is to create one Lambda that subscribes to the Log Group and then proxy the events into an SNS topic.
Those two Lambdas are now subscribed to the SNS topic instead of the Log Group.
For filtering events, you could implement them inside the Lambda.
It's not a perfect solution but it's a functioning workaround until AWS allows multiple Lambdas to subscribe to the same CloudWatch Log Group.
I was able to resolve the issue using a bit of a workaround through the Lambda function and also using the response provided by Kannaiyan.
I created the subscription to ES via the console, and then unsubscribed, and modified the Lambda function default code.
I declared two Elasticsearch endpoints:
var endpoint1 = '<ELASTICSEARCH ENDPOINT 1>';
var endpoint2 = '<ELASTICSEARCH ENDPOINT 2>';
Then, declared an array named "endpoint" with the contents of endpoint1 and endpoint2:
var endpoint = [endpoint1, endpoint2];
I modified the "post" function which calls the "buildRequest" function that then references "endpoint"...
function post(body, callback) {
for (index = 0; index < endpoint.length; ++index) {
var requestParams = buildRequest(endpoint[index], body);
...
So every time the "post" function is called it cycles through the array of endpoints.
Then, I modified the buildRequest function that is in charge of building the request. This function by default calls the endpoint variable, but since the "post" function cycles through the array, I renamed "endpoint" to "endpoint_xy" to make sure its not calling the global variable and instead takes the variable being inputted into the function:
function buildRequest(endpoint_xy, body) {
var endpointParts = endpoint_xy.match(/^([^\.]+)\.?([^\.]*)\.?([^\.]*)\.amazonaws\.com$/);
...
Finally, I used the response provided by Kannaiyan on using the AWS CLI to implement the subscription to the logs, but corrected a few variables:
aws logs put-subscription-filter \
--log-group-name <LOG GROUP NAME> \
--filter-name <FILTER NAME>
--filter-pattern <FILTER PATTERN>
--destination-arn <LAMBDA FUNCTION ARN>
I kept the filters completely open for now, but will now code the filter directly into the Lambda function like dashmug suggested. At least I can split one log to two ES clusters.
Thank you everyone!
Seems like AWS console limitation,
You can do it via command line,
aws logs put-subscription-filter \
--log-group-name /aws/lambda/testfunc \
--filter-name filter1 \
--filter-pattern "Error" \
--destination-arn arn:aws:lambda:us-east-1:<ACCOUNT_NUMBER>:function:SendToKinesis
You also need to add permissions as well.
Full detailed instructions,
http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html
Hope it helps.
As of September 2020 CloudWatch now allows two subscriptions to a single CloudWatch Log Group, as well as multiple Metric filters for a single Log Group.
Update: AWS posted October 2, 2020, on their "What's New" blog that "Amazon CloudWatch Logs now supports two subscription filters per log group".
Amazon announced AWS Lambda (http://aws.amazon.com/lambda/).
The product description includes:
Scheduled Tasks
AWS Lambda functions can be triggered by external event timers, so
functions can be run during regularly scheduled maintenance times or
non-peak hours. For example, you can trigger an AWS Lambda function to
perform nightly archive cleanups during non-busy hours.
When I read this, I understood I could finally have a way to consistently do "cron-like" tasks. I want to run a specific query everyday at 5PM let's say.
However I do not find this anywhere in the documentation. They only mention triggers on programatical events, or events from other AWS services.
Did I misunderstand? Or can someone point me to the documentation?
Native Support for Scheduled Events added October 8, 2015:
As announced in this AWS blog post, scheduling is now supported as an event source type (also called triggers) called "CloudWatch Events - Schedule", and can be expressed as a rate or a cron expression.
Add Scheduled Event to a new lambda
Navigate to the 'Configure triggers' step of creation, and specify the 'CloudWatch Event - Schedule' trigger. Example configuration below:
Add Scheduled Event to an existing lambda
Navigate to the 'Triggers' tab of your lambda, select 'Add Trigger', and specify the 'CloudWatch Event - Schedule' trigger. Example screenshot where I have an existing lambda with an SNS trigger:
Once loaded, the UI to configure this trigger is identical to the screenshot in the above "Add Scheduled Event to a new lambda" section above.
Discussion
For your example case, you'll want to use cron() instead of rate(). Cron expressions in lambda require all fields and are expressed in UTC. So to run a function every day at 5pm (UTC), use the following cron expression:
cron(0 17 * * ? *)
Further Resources
AWS Documentation - Schedule Expressions Using Rate or Cron
AWS Documentation - Run an AWS Lambda Function on a Schedule Using the AWS CLI
AWS Documentation - Tutorial: Using AWS Lambda with Scheduled Events
AWS has provided a sample "blueprint" that uses a cron expression called lambda-canary that can be selected during function creation from the AWS console.
This tutorial walks you through configuration of this blueprint.
Notes
The name of this event type has changed from "Scheduled Event" to "CloudWatch Events - Schedule" since this feature was first released.
Prior to the release of this feature, the recommended solution to this issue (per "Getting Started with AWS Lambda" at 42min 50secs) was to use SWF to create a timer, or to create a timer with an external application.
The Lambda UI has been overhauled since the scheduled event blog post came out, and the screenshots within are no longer exact. See my updated screenshots above from 3/10/2017 for latest revisions.
Since the time of this post, there seems to have risen another solution: Schedule Recurring AWS Lambda Invocations With The Unreliable Town Clock (UTC) in which the author proposes subscribing to the SNS topic Unreliable Town Clock. I've used neither SWF nor SNS, but it seems to me that the SNS solution is simpler. Here's an excerpt from the article
Unreliable Town Clock (UTC)
The Unreliable Town Clock (UTC) is a new, free, public SNS Topic
(Amazon Simple Notification Service) that broadcasts a “chime” message
every quarter hour to all subscribers. It can send the chimes to AWS
Lambda functions, SQS queues, and email addresses.
You can use the chime attributes to run your code every fifteen
minutes, or only run your code once an hour (e.g., when minute ==
"00") or once a day (e.g., when hour == "00" and minute == "00") or
any other series of intervals.
You can even subscribe a function you only want to run only once at a
specific time in the future: Have the function ignore all invocations
until it’s after the time it wants. When it is time, it can perform
its job, then unsubscribe itself from the SNS Topic.
Connecting your code to the Unreliable Town Clock is fast and easy. No
application process or account creation is required
NEW SOLUTION: Lambda Scheduled Jobs
Werner Vogel has announced tonight (10/08) at re:Invent that AWS Lambda now has it's own scheduler.
Se the AWS Lambda release note on 2015-10-08 :
You can also set up AWS Lambda to invoke your code on a regular,
scheduled basis using the AWS Lambda console. You can specify a fixed
rate (number of hours, days, or weeks) or you can specify a cron
expression. For an example, see Walkthrough 5: Using Lambda Functions
to Process Scheduled Events (Python).
OLD SOLUTION: Scheduling with AWS Data Pipeline
You can use AWS Data Pipeline to schedule a task with a given period. The action can be any command when you configure your Pipeline with the ShellCommandActivity.
You can for example run an AWS CLI command to:
Put a message to SQS
or directly invoke a Lambda function (see invoke)
You can easily create the AWS Data Pipeline scheduled task directly within AWS console (e.g. with an AWS CLI command) :
You can also use the API to define your scheduling:
{
"pipelineId": "df-0937003356ZJEXAMPLE",
"pipelineObjects": [
{
"id": "Schedule",
"name": "Schedule",
"fields": [
{ "key": "startDateTime", "stringValue": "2012-12-12T00:00:00" },
{ "key": "type", "stringValue": "Schedule" },
{ "key": "period", "stringValue": "1 hour" },
{ "key": "endDateTime", "stringValue": "2012-12-21T18:00:00" }
]
}, {
"id": "DoSomething",
"name": "DoSomething",
"fields": [
{ "key": "type", "stringValue": "ShellCommandActivity" },
{ "key": "command", "stringValue": "echo hello" },
{ "key": "schedule", "refValue": "Schedule" }
]
}
]
}
Limits: Minimum scheduling interval is 15 minutes.
Pricing: About $1.00 per month.
Here is how I do it:
Create Lambda which:
purges given SQS
sends there message with delay 10 minutes
https://gist.github.com/mikeplavsky/5ffe7e33e0d70a248537
Create CloudWatch Alarm for: ApproximateNumberOfMessagesVisible > 0 for 1 minute
Subscribe SNS Topic to the Alarm
Subscribe Lambda to SNS Topic
Now you have a timer with approximately 15 minutes resolution.
Then other Lambda functions are subscribed to SNS Topic and called every 15 minutes.
Since it is now easily possible to trigger lambda functions over HTTP (e.g. using GET or curl) a simple solution is to use a managed CRON like easycron: https://www.easycron.com/ to trigger your lambda function into running.
We had the same problem and ended up running a cron service on Google App Engine in python since this allowed for more flexibility and complexity in the CRON job itself.
In the Function page, Add trigger, you can add a CloudWatch Events, and make it as a schedule type
Run as cron in AWS
An example to setup cloudwatch schedule event trigger for you lambda using cloudformation.
LambdaSchedule:
Type: "AWS::Events::Rule"
Properties:
Description: A schedule for the Lambda function..
ScheduleExpression: rate(5 minutes)
State: ENABLED
Targets:
- Arn: !Sub ${LambdaFunction.Arn}
Id: LambdaSchedule
LambdaSchedulePermission:
Type: "AWS::Lambda::Permission"
Properties:
Action: 'lambda:InvokeFunction'
FunctionName: !Sub ${LambdaFunction.Arn}
Principal: 'events.amazonaws.com'
SourceArn: !Sub ${LambdaSchedule.Arn}
LambdaFunction:
Type: "AWS::Lambda::Function"
Properties:
Description: Scheduled lambda to run every 5 minutes
CodeUri: ./build/package.zip
Handler: index.lambda_handler
MemorySize: 128
Runtime: python3.6
AWS Recently(10-Nov-2022) launched a new service called EventBridge Scheduler or you can choose EventBridge Rules for this as well. As per your example here I'm going to trigger an event every day at 5.00 A.M. As you can see it shows us the next 10 trigger dates and times as well, this will really help us to manually check our cron before doing anything.
Please note, if you want to start this schedule on a specific date and time, please choose EventBridge Scheduler for that. It has a Timeframe option. If you want to know more information about Timeframeplease have look at this answer.
In the target section, you can select 35 AWS Lambda function options.
Hope this will help you.
You could schedule it with cloudWatch events too. Create rule -> attach target (lambda) and set up cron/rate wise schedule on your rule.
The web-console way is pretty straightforward. Just create a CloudWatch rule for the lambda and add it in the lambda's Triggers tab.
For those who needs to automate that with aws cli, we can
create the function,
create the rule,
grant the permission,
link rule and function
Create function
aws lambda create-function --function-name ${FUNCTION-NAME} \
--runtime java8 \
--role 'arn:aws:iam::${Account}:role/${ROLE}' \
--handler org.yourCompany.LambdaApp \
--code '{"S3Bucket":"yourBucket", "S3Key": "RC/yourapp.jar"}' \
--description 'check hive connection' \
--timeout 180 \
--memory-size 384 \
--publish \
--vpc-config '{"SubnetIds": ["subnet-1d2e3435", "subnet-0df4547a"], "SecurityGroupIds": ["sg-cb17b1ae", "sg-0e7ae277"]}' \
--environment Variables={springEnv=dev}
Create rules
## create
aws events put-rule --name ${ruleName} \
--schedule-expression 'rate(5 minutes)' \
--state ENABLED \
--description 'check hive connection'
# grant permission to the Rule to allow it to trigger the function
aws lambda add-permission --function-name ${functionName} \
--statement-id 123 \
--action 'lambda:InvokeFunction' \
--principal events.amazonaws.com \
--source-arn arn:aws:events:us-east-1:acc:rule/${ruleName}
# link rule and function
aws events put-targets --rule ${ruleName} \
--targets '[{"Id":"1", "Arn":"arn:aws:lambda:us-east-1:acc:function:RC-checkhive"}]'
simple way to run your query in lambda for particular time interval is to set rule for your lambda function. for that after creating lambda function go to cloudwatch>>rules>>schedule. and define cron expression and in the target section select lambda function which you want to trigger.
Posted - 27 June 2021
You can schedule AWS Lambda functions using Amazon EventBridge
Here I am using AWS Management Console
Select your Lambda function and in configuration select "Triggers"
Select EventBridge(CloudWatch Events) - Basically this is the latest version of one of the popular answers(using CloudWatch triggers).
Create a new rule - Add details. My lambda will be triggered at 4pm UTC everday.
EventsBridge (CloudWatch) Solution:
You can create an AWS Events Bridge Rule and set a Lambda function as the target using its ARN. You can specify a rate or cron schedule expression. For example, the following expression will run your Lambda function after every ten minutes on all weekdays.
schedule = "cron(0/10 * ? * MON-FRI *)"
Note that your EventsBridge role will also require the lambda:InvokeFunction permission so EventsBridge can trigger your Lambda function.
Here's a full tutorial for the Terraform setup for this architecture: https://medium.com/geekculture/terraform-setup-for-scheduled-lambda-functions-f01931040007
While creating the lambda function create trigger "CloudWatch Events - Schedule"
Now you can either use AWS presets in schedule expression like rate = 15 min or you can use a cron expression.
For your requirement the Cron Schedule is "0 0 17 1/1 * ? *"
Here's an example of deploying up a Scheduled Lambda to run every 10 minutes using Serverless. The function handler is located at src/scheduled/index.handler and the rate is specified in the Lambda's settings. AWS uses EventBridge now to control when the Lambda should be invoked. That is all setup automatically for you when using Serverless. You can see the setup in the AWS console by viewing the Lambda or by looking at the "default" EventBridge in the EventBridge section.
https://carova.io/snippets/serverless-aws-lambdafunction-scheduled-cronjob
Diksha is AWS Lambda Scheduler based on AWS SWF Trigger as recommended by AWS Team. One can schedule jobs using cron expressions and can also specify how many time you want to run, when to start or when to end. You can view status as well as history of scheduled jobs. Security is managed by AWS policies.
Once you set up diksha engine, you can schedule functions using cron expression in following way:
java -jar diksha-client-0.0.1.jar -lcfg cf1 -cj "jobName|functionName|context|0 0-59 * * * *|10"
In this job job will run every minute for 10 times. AWS SWF will trigger function by itself.
Details: https://github.com/milindparikh/diksha
Disclaimer: I am contributor to the project.