I have been trying to address the issue (dropping an event in a queue with “cron") and need to get someone else’s insight on the issue.
Current status:
Currently, I have two lambdas. The thing that I am trying to do is to set the schedule (cloudwatch - every 12PM) to trigger the Lambda A, and the Lambda A triggers Lambda B via Event bridge.
It is all working if I use schedule-expression as "cron(0/5 * * * ? *)" which means it triggers the Lambda A every 5 mins and I can get the end result from Lambda B.
Since the 5 mins interval is working with the whole flow (CloudWatch schedule -> Lambda A -> Lambda B), I can tell all the rules and event bridge setting etc are working.
Problem:
The problem is that when I change the Schedule expression to cron(0 12 * * ? *) from cron(0/5 * * * ? *), it only worked until Lambda A - which means the “cron" triggers the Lambda A at 12PM correctly but the following event to trigger the Lambda B is not working for some reason.
The event on SQS (calling Lambda B) is dropped and put it to Deadletter queue.
I have tried:
I have compared the event(calling Lambda B) detail between using cron 12PM and 5mins interval expressions and no differences found.
I have checked UTC now(let's say this moment is 1am), then I have set cron(5 1 * * ? *), but it only calls the Lambda A.
I have checked the individual Lambdas and event bridge and setting. It worked fine individually.
I have used rate(5mins) instead of cron without any other changes, the Lambda B got correctly triggered.
Questions:
I would like to know what could be the cause of this issue and what could be a way to get proper logs for it.
Please let me know if you have any thought or ideas on this issue.
Related
So we have a couple of lambdas that listen to changes on the db, in every lambda we had to filter in the beginning because this particular lambda don't care about all changes just a particular one.
To be more explicit, we are applying event sourcing and lambdas are supposed to be event handlers. I want lambda A to be triggered ONLY when event A is inserted in the db rather than whenever an event is inserted! And same for lambda B and event B and so on.
In the time being we have a filter in the beginning of every lambda:
lambdaA handler:
const eventsToBeProcessed = events.filter(
(event) => event.eventName === 'EventA'
);
Now that we have a good bunch of lambdas it doesnt make sense to trigger all of them when I need only to trigger one!
I'm not very experienced with AWS but I'm assuming the solution will either be:
Being able to filter on DynamoDb Triggers
Instead of making lambdas triggered by dynamodb, have a AWS service in between that's triggered on every change and it will know which lambda to trigger depending on what's the update
EDIT:
I was not very happy with the solution I marked as an answer, the main reason because the Single Point Of Failure in the design, also it will need to count on an SNS to do a publish/subscribe and SNS is not highly available and it can fail you if you are trying to have a highly available system (4 9s or so)
The solution I ended up adapting (for now at least) is that when I push my events into the event store, if I want to trigger a side effect, I will just push it into AWS' EventBridge (an Event bus with rules). The best part about the EventBridge is that you can set the rules you want, the bus you want, and then have (for example) Lambda A be invoked when the bus have Event A and so on.
There is now a way to do that,
https://aws.amazon.com/blogs/compute/filtering-event-sources-for-aws-lambda-functions/
It's a new feature of lambda, it uses the same syntax as AWS Eventbdridge.
aws lambda create-event-source-mapping \
--function-name fleet-tire-pressure-evaluator \
--batch-size 100 \
--starting-position LATEST \
--event-source-arn YOUR_DB_STREAM_ARN \
--filter-criteria '{"Filters": [{"eventName": ["EventA"]}]}'
You can refer to this documentation to know the exact syntax
I think it would make sense to have one Lambda function responding to additions to the DynamoDB stream, dispatching the events to the respective downstream functions.
------------
--> Function A
-------- --------------------- ------------
Stream --> Dispatcher Function
-------- --------------------- ------------
--> Function B
------------
i created an AWS Lambda function, and created a scheduled event rule in Cloudwatch to trigger it every 5 minutes :
Schedule Cron expression : 0/5 * * * ? *
The problem is that when looking at the logs, the Lambda seems to be executed every 1~2 minute.
I checked the CRON expression, it seem correct because AWS shows previews of the next triggers. I also tried with a rate expression, but i still have the same issue.
I tooks a look at cloudwatch metrics :
- The scheduled event seems fine, it triggered once every 10minutes
- The lambda invocation metric show that it is invoked more often
Any help ?
Thanks #JohnRotenstein for the lead,
the issue came from the returned result from the NodeJS promise.
If the JS handler doesn't give back any result inside the returned promise, lambda considers the execution as a failure.
In my case i simply had to return a value (even null) so that the execution completes successfully.
Otherwise it looks like lambda will to re-execute the code 1~2 minutes later
I need my python lambda to invoke multiple lambda so they can run in parallel. I have it working - kind of. To test it, I invoke the code below for 5 different lambda calls (could be more in the future)... when i look at the Step Function console, I only see 2 lamdbas running concurrently and the other 3 are ignored after the first 2 finished. In researching concurrency issues, I didnt find a limit that is this low (5 should be nothing for AWS!)..
response = lambdaClient.invoke(
FunctionName=lambdaToInvoke <---this var is an ARN,
InvocationType='Event',
Payload=json.dumps(event) <--just passing state
)
All IAM is provisioned...Any thoughts are appreciated!
Usually when I need to do a "fan out" operation like you are describing I don't invoke the lambda functions directly.
Rather I would recommend you setup your 5 "slave" lambda functions to get activated from either SNS or SQS. From your "master" function you can use the SDK to create a SNS alert or SQS message.
What happens when you try to manually invoke the 3 functions that don't run?
Do you have logging setup in CW logs? Can you post?
I would like to create a Scheduled Events to run a Lambda to execute an api call every 1 minute (cron-line behaviour).
The caveat to this setup is that; the external api is un-reliable / slow and the api call sometimes could last longer than 1 minute.
So, my question here is; given the setup & scenario - would AWS run another Scheduled Event and execute the lambda before the previous executing finished? I.e. overlap?
If it does; is there a way to configure the scheduled event to not "overlap"?
I did some initial research into this and came across this article:
https://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html
It looks like you can set concurrency limits at function level? Is this the way to achieve non-overlapping scheduled lambda executions? i.e. set the function's concurrency limit to 1?
Yes by default it will execute your Lambda function every 1 minute, regardless if the previous invocation has completed or not.
To enforce no more than one running instance of your Lambda function at a time, set the Concurrency setting of your Lambda function to 1.
Problem : Fetch 2000 items from Dynamo DB and process(Create a POST req from 100 items) it batch by batch (Batch size = 100).
Question : Is there anyway that I can achieve it from any configuration in AWS.
PS : I've configured a cron schedule to run my Lambda function. I'm using Java. I've made multi-threaded application which synchronously does so, but this eventually increases my computation time drastically.
I have the same problem and thinking of solving it in following way. Please let me know if you try it.
Schedule Job to fetch N items from DynamoDB using Lambda function
Lambda function in #1 will submit M messages to SQS to process each
item and trigger lambda functions, in this case it should call
lambda functions M times Each lambda function will process request
given in the message
In order to achieve this you need to schedule an event via CloudWatch, setup SQS and create lambda function triggered by SQS events.
Honestly, I am not sure if this is price effective but it should be working. Assuming your fetch size is so low, this should be reasonable.
Also you can try using SNS in this case you don't need to worry about SQS message polling.