A rule in AWS EventBridge allows us to provide upto 5 targets. For each of these targets, we have some options to choose the input - based on the event that matched the rule. Is there a way to pass the output of the first target (lambda function) as the input to the next (another lambda function).
I know we can do this by triggering an SNS at the end of the first lambda function. But, I am looking for a way to do this within the EventBridge.
Thanks for your help
A cleaner way to do this would be to have Eventbridge hand over the event to a Step Functions state machine and have 5 steps in that state machine.
Step functions allow you to consume output from first step in the second step and so on.
More on Step Functions here.
More on Lambda with Step Functions here.
I'd agree with #paritosh answer that Step-functions makes more sense for a workflow but if you need something more light-weight (and don't want to learn one more thing); you can use set Eventbridge as the lambda destination. Lambda should then send the event back to Eventbridge via a PutEvents api call https://aws.amazon.com/blogs/compute/introducing-aws-lambda-destinations/
If you want to change the input before triggering the lambda, you can use InputTransformer https://docs.aws.amazon.com/eventbridge/latest/userguide/transform-input.html
Related
I am planning to use a DynamoDB Stream to do a bunch of task on a very complex single table. The task include actions such as sending notifications, updating other entries in the database and indexing some entries using AWS OpenSearch. I can already see that the Lambda function to handle all of this will end up being massively complex and very hard to maintain. It also feels very wasteful since most actions that would be included in this lambda only apply to very specific stream events and are not required for most others.
My thought was that I could maybe use a StateMachine to split these task up into individual Lambdas and then only run them when they are needed for a certain item in the stream.
So I'm curious if there is any easy way to poll a DynamoDB Stream with a Step Function in the same way you would do with a Lambda.
(PS. I know this could be considered an opinion question, but I would also be curious about anyone's thoughts on this solution. Is it a good idea or bad idea ?)
The lambda must be unique for specific treatment, this is why the best practice is to split your lambda to many sub components
You can use this architecture for example :
DynamoDB Stream --> Lambda --> Step Function (parralele treatment)
Or use multiple lambdas with SNS fanout like :
DynamoDB Stream --> Lambda --> SNS --> multiple sub ( lambdas)
There is a Lambda say my Lambda Name is - XYZ which has a s3 file upload trigger. Now lets say if I upload multiple files say around 100 and the lambda execution starts.
Is there any way I can track if the lambda is running or has processed all the files?
The reason for this is, once the lambda has completed processing all the file, I would want to trigger a step function, so for me to trigger my Step Function I would want to do that only once all the files are processed by my lambda (XYZ).
FYI: There is no current way to track how many files have been uploaded
I think it's not a good design to run a step function state machine after the lambda completes the job without having a perfect logical event.
Because you can for example receive a bunch of files (say 100) completed by the lambda, then using alarms with Cloudwatch once we complete the 100 values of a custom metric, or check value in DynamoDb or number of object in a custom folder, fires a job step function, at the same time you receive another file with +5 sec delay to complete the 101, in this case maybe you miss this file.
If you don't have a special event or a condition to have the completion of the files, then you can work with time scheduling and trigger your Step function with Cloudwatch event ( scheduled event) like every 15 min, check if there is work if not, exit the job.
Otherwise, either include the lambda (file process) in your step function as a step or change your design.
I want to add a Lambda target to an existing CloudWatch rule. I used the following to refer to the existing rule:
rule = events.Rule.from_event_rule_arn(self, "Rule",event_rule_arn='')
Later I add a target with:
rule.add_target(targets.LambdaFunction(lambdaFn))
When I execute a cdk synth or deploy, I get the following error:
AttributeError: '+' object has no attribute 'add_target'
I know the IRule element does not have that method but I cannot find a clear way how to achieve what I need.
I also try using event source in Lambda but got the following error:
Unrecognized event source, must be kinesis, dynamodb stream or sqs.
I do not think this is possible. You need to reference the lambda function and manage the rule from the stack to which the rule belongs.
As MilanG suggests, it is not possible to do.
My use case needs to create several Lambda functions and set the same trigger to them and CloudWatch Rules is not a fit for it because of a 5 targets per rule hard limit. I use SNS instead as follows:
sns_topic = aws_sns.Topic.from_topic_arn(scope, id, topic_arn=config)
lambdaFn.add_event_source(aws_lambda_event_sources.SnsEventSource(sns_topic))
I have 100 lambda functions. All of them have 2 distinct keywords in their function name -
From_Table
To_Table
For example, my function-names are: function1_To_Table, function2_From_Table etc. I have 100 such functions, 50 each side (To, From)
I want to create one event which invokes all functions that have above keywords in their name every 15 minutes. I don't want to create 100 events for all 100 lambdas.
Something like this: Run my function if function name contains - To_Table or From_Table
Can anyone help me with this?
I think your approach is wrong. Creating 1 event for each lambda not the right solution, I agree with you there. Instead create one lambda function that can iterate over and call each of the 50 other functions on its own, then schedule that function in Cloudwatch events to trigger every 15 minutes. One event, one function that executes each 50 to_from or from_to functions.
Reference Can an AWS Lambda function call another.
EDIT: Just thought about something, are your functions long running? If they take a while to complete then the first function may time out before completing all 50 other functions. You may need to setup SNS or SQS to trigger the other functions.
first of all AWS doesn't provide any function to invoke a lambda with regex or using wildcard.
In order to invoke a all your functions,
first you need to get the list of lambda functions. using aws cli you can run this command
1 - aws lambda list-functions
2 - you extract the functions which matches the keywords i.e to_from, from_to from the response of list-functions
3 - create a look to invoke all the extracted functions (aws lambda invoke --function-name)
aws also has api to perform all the functions in aws-sdk or in boto3
I'm interested in seeing whether I can invoke an AWS Lambda when one of my DynamoDB tables grows to a certain size. Nothing in the DynamoDB Events/Triggers docs nor the Lambda Developer Guide suggests this is possible, but I find that hard to believe. Anyone ever deal with anything like this before?
You will have to do it manually.
I see two out-of-the box ways to achieve this though:
1) You can create a CloudWatch Event that runs every X min (replace X with whatever you think is necessary for your business case) to trigger your Lambda Function. Your function then needs to invoke the describeTable API and run a check against that value. Once it has run, you can disable the event since your table has reached the size you wanted to be notified about. This is the easiest and most cost effective since most of time your tables size will be lower than your predefined limit.
2) You could also use DynamoDB streams and invoke the describeTable API, but then your function would be triggered upon every new event in your table. This is cost ineffective and, in my opinion, overkilling.