I'm trying to understand the behavior of S3 Event Notification trigger. I have s3 events to trigger lambda. Lambda captures the event and file metadata to dynamodb. There would be around 50k event triggers in short burst across the day. If I had to add SNS in the workflow and have SNS trigger lambda, what are the advantages with sns vs s3 directly invoking lambda?
There is no gained advantage. Both S3 and SNS events are asynchronous event sources and behave the same way. See: Lambda supported event sources And: Lambda Retry on Errors (Asynchronous invocation part), which highlights nicely the lambda behavior with specific types of event sources.
Simply doing S3 -> Lambda is sufficient.
The advantage is flexibility for the future. If you use SNS in the middle, you can easily send (fan-out) the notifications to multiple destinations with more SNS topic subscriptions -- another Lambda function, an SQS Queue, an HTTPS endpoint, or even email, which can be very useful for non-intrusive observation, testing, troubleshooting, and developing new capabilities that need the same notification.
Related
I see this question has been asked few times but has not been answered yet. Making another attempt.
What is the basic difference between an S3 event and Cloudwatch events ?
Is one is preferred over the other ?
Appreciate an answer.
Thanks !
S3 Event Notifications are for events that are specific to S3 buckets. S3 Events Notifications can publish events for
New object created
Object removal
Restore object
Reduced Redundancy Storage (RRS) object lost events
Replication events
And it can send notifications to:
SNS topics
SQS queues
Lambda functions
CloudWatch Events, and the associated (preferred, actually) service, Amazon EventBridge, are much broader, and apply to the entire AWS platform. CloudWatch and EventBridge use the same underlying CloudWatch Events API, but EventBridge has more features.
You can use CloudWatch Events/EventBridge to react to any event published by AWS CloudTrail as well as from a very long list of integrated AWS services. These events can also be published on a schedule using a cron-like schedule expression syntax. It can send notifications to more targets as well, including Amazon EC2, Kinesis data streams, ECS tasks, Systems Manager, and much more.
Generally, it's preferable to use EventBridge for anything other than S3. Since EventBridge shares the same underlying API as CloudWatch Events, any change you make to either one will show up in the other. You should use S3 Events for any of the events listed above (see the docs for up an to date list of events).
guys need small help, I have a use case, where I want to set up a communication service.
using SQS, SQs is going to receive a different type of events to be communicated. Now we have a single lambda function which does a single communication. let's say one email Lambda, Slack lambda, etc.
how I can invoke different lambda based on queue attributes. I was planning to use SQS as an event source and something kind of this architecture link to sample architeture
here in the above, we can handle rate limiting and concurrency at the lambda service level
simplified works if event type is A invoke Lambda A if the event type is B invoke a lambda B
and both events are in same SQS
all suggestions are welcome
Your problem is a SQS message can only be read by one service at a time. When it is being read, it is invisible to anyone else. You can only have one Lambda consumer and there isn't any partitioning or routing in SQS besides setting up another SQS topic. Multiple consumers are implemented Kensis or AWS MSK (Kafka)
What you are trying to accomplish is called a fan out. This is a common cloud architecture. What you probably want to do is publish initially to SNS. Then with SNS you can filter and route to multiple SQS topics for each of the message types and each SQS topic would then be consumed by it's own Lambda.
Check out a tutorial here:
https://docs.aws.amazon.com/sns/latest/dg/sns-common-scenarios.html
I have a Lambda function that takes a list of tasks to be run at the time specified. This time can vary.
I am using SNS to trigger another Lambda function that in turn runs the tasks.
These tasks need to be run at specified time. Is it possible to publish a message to SNS using Lambda at the specified time?
Or send the message to SNS, but SNS in turn triggers Lambda at the specified time?
Any option would do.
P.S. I know there is an option of using Cloud Watch events, but I do not want to use any more services.
It appears that your requirement is to trigger an AWS Lambda function at specific times.
This can be done by using Amazon CloudWatch Events, which can take a cron-like expression to run a Lambda function at desired time intervals or specific times. This functionality was originally in the Lambda console, but was moved to CloudWatch Events when more services added scheduling capabilities.
However, CloudWatch Events cannot trigger an Amazon SNS message. If you need SNS to trigger Lambda, then you'll need CloudWatch Events to trigger a Lambda function that sends a message to SNS (which then triggers Lambda functions). Obviously, it would be cleaner to avoid SNS altogether unless you specifically need to fan-out the message to multiple subscriptions/Lambda functions.
At the moment I'm are pooling AWS SQS from our back-end and doing business logic once payload is received.
I would like to move this to AWS Lambda and start automating business logic via SQS/SNS.
As I can not subscribe to AWS SQS events, what is the best practice in implementing SQS pooling with Lambda (node.js)?
SQS doesn't really work well with Lambda since you cannot automatically trigger Lambda functions from SQS queues messages.
I would rather remove the SQS/SNS logic and go for a DynamoDB Streams based solution that would cover the queueing, archiving & Lambda triggering tasks natively: your producer puts messages in a DynamoDB table while your Lambda is triggered for any new entry with Streams (it's an AWS native mechanism)
Of course a Kinesis based solution may be considered as well.
It is possible to even simplify the whole polling process by using the built-in SQS event source for lambda.
Lambda will automatically scale out horizontally consume the messages
in my queue. Lambda will try to consume the queue as quickly and
effeciently as possible by maximizing concurrency within the bounds of
each service. As the queue traffic fluctuates the Lambda service will
scale the polling operations up and down based on the number of
inflight messages.
see AWS Blog
As I can not subscribe to AWS SQS events,
Why?
Lambda can be triggered on SQS messages. Lambda internally handles the scaling, batching and retries for you. Check this AWS documentation on how to use Lambda with SQS.
I want to trigger Lambda function whenever new message added to SQS.
Note that I don't want to add new message (events) to SQS.
What I'm trying to do:
My app will send message to SQS
Whenever new message added to queue CloudWatch event gets generated
CloudWatch Event triggers lambda
Problem:
In AWS console while configuring CloudWatch Events I haven't found any option to add source of event i.e. URL or Name of my SQS queue.
I'm not sure if this use case is valid but please help me out.
EDIT: AWS now supports SQS as an event source to trigger Lambda functions. See this blog post for more details.
ORIGINAL ANSWER:
SQS is not supported as a direct event source for AWS Lambda functions. If there are properties of a queueing system that you need for your use case, then you could have a "cron-job" type Lambda function that runs on a schedule, receives messages from the queue, and calls your worker Lambda function in response to each message received. The problem with this approach is that you must continually poll SQS even during periods when you don't expect messages, which incurs unnecessary cost.
The easiest approach is to use SNS instead. Create a topic, publish events to that topic instead of adding a message to an SQS queue, and have your Lambda function subscribe to that SNS topic. It will then be invoked each time a message is published to that SNS topic. There's a tutorial on this approach here:
http://docs.aws.amazon.com/lambda/latest/dg/with-sns-example.html
I would recommend to change your approach.
Your application should publish a message to an existing SNS topic. Your SQS and Lambda should than subscribe to this SNS topic.
Application -> publish -> SNS_TOPIC
-> SQS is notified
-> Lambda is notified