Can someone explain to me the advantage or disadvantage of using SNS -> Lambda vs. SNS -> SQS -> Lambda.
I'm looking to setup an architecture for pub/sub micro-service messaging, but having a queue in front of every Lambda seems excessive.
Unless something has changed, the question of whether to it makes more sense to deploy SNS → Lambda, or SNS → SQS → Lambda, is based on a premise with a significant flaw.
As is indicated in Supported Event Sources in the Lambda documentation, Lambda events can be sourced from S3, DynamoDB, Kinesis, SNS, SES, Cognito, CloudFormation, CloudWatch & Events (including Scheduled Events), AWS Config, Amazon Echo, and API Gateway.
And, of course, you can invoke them directly.
But SQS is not a supported Lambda event source.
Amazon SQS is a message queue service used by distributed applications to exchange messages through a polling model, and can be used to decouple sending and receiving components—without requiring each component to be concurrently available. By using Amazon SNS and Amazon SQS together, messages can be delivered to applications that require immediate notification of an event, and also persisted in an Amazon SQS queue for other applications to process at a later time.
Untill you don't want to decouple sending and receiving components and just want to achieve your use case in the question it will work in both case SNS- Lambda and SNS - SQS - Lambda.
Related
Is it possible to receive notifications in a .NET Core application about bucket/object creation/deletion?
How to do it?
S3 bucket can generate SNS and SQS event notifications as well as trigger Lambda function on misc. events. Go to Bucket Properties->Events.
In your .NET code you'll need to react to these events, for instance by receiving SQS messages.
Amazon S3 Events can send a notification to:
An AWS Lambda function (Trigger): Does not appear relevant since your code is running elsewhere.
An Amazon SQS queue (Pull): Your application could regularly poll the Amazon SQS queue to retrieve a message, then act on that message.
An Amazon SNS topic (Push): Your application could subscribe to the Amazon SNS topic to receive the message via an HTTP endpoint. For example, this could point to your back-end web server.
If your application has a web server that is accessible from the Internet, then use the SNS push. Otherwise, your app will need to poll the SQS queue.
I would like to know if it's possible to persist all unacknowledged messages from an SNS topic to an S3 file given a certain time window. These messages don't necessarily need to follow the original order in the S3 file, a timestamp attribute is enough.
If all you want is to save all messages published to your SNS topic in an S3 bucket, then you can simply subscribe to your SNS topic the AWS Event Fork Pipeline for Storage & Backup:
https://docs.aws.amazon.com/sns/latest/dg/sns-fork-pipeline-as-subscriber.html#sns-fork-event-storage-and-backup-pipeline
** Jan 2021 Update: SNS now supports Kinesis Data Firehose as a native subscription type. https://aws.amazon.com/about-aws/whats-new/2021/01/amazon-sns-adds-support-for-message-archiving-and-analytics-via-kineses-data-firehose-subscriptions/
There is no in-built capability to save messages from Amazon SNS to Amazon S3.
However, this week AWS introduced Dead Letter Queues for Amazon SNS.
From Amazon SNS Adds Support for Dead-Letter Queues (DLQ):
You can now set a dead-letter queue (DLQ) to an Amazon Simple Notification Service (SNS) subscription to capture undeliverable messages. Amazon SNS DLQs make your application more resilient and durable by storing messages in case your subscription endpoint becomes unreachable.
Amazon SNS DLQs are standard Amazon SQS queues.
So, if Amazon SNS is unable to deliver a message, it can automatically send it to an Amazon SQS queue. You can later review/process those failed messages. For example, you could create an AWS Lambda function that is triggered when a message arrives in the Dead Letter Queue. The function could then store the message in Amazon S3.
We are planning to use AWS SES with SQS. The idea is to push the email content to SQS then SES fetch the queue item and send the emails. We are using PHP.
We just want the idea to implement this without hampering the webserver performance.
SES cannot fetch stuff from SQS.
You will have to have a compute tier in between (either on EC2 or Lambda or some other server), which could poll the SQS and then call SES.
Scheduled Lambda can be ideal for this use case.
Out of curiosity, who is putting the message in SQS and do tou have any specific use case for putting the message in SQS. Why can't the routine that places the message on SQS, simply call the SES and get the email sent
Answer is summed up in previous answer but still I want to put a word for it.
Flow that I implemented was like this: Invoking sender lambda of sqs -> sqs queue triggers reciever lambda of ses -> ses sends mail.
The answer to why it is to be done? To avoid bottlenek at SES.
A Lambda function can process items from multiple queues (using one Lambda event source for each queue). You can use the same queue with multiple Lambda functions.
Other ways to maximize throughput of SES given in docs: https://docs.aws.amazon.com/ses/latest/dg/troubleshoot-throughput-problems.html
Is that possible if the user create a ticket in freshdesk that needs to be trigger the AWS lambda function.
That shouldn't be that hard. I would like to recommend using the following architecture
FreshDesk Ticket Trigger
FreshDesk Ticket Trigger Handler Published Message to SNS Topic
AWS Lambda Configured to SNS Topic as Event Source
AWS Lambda Code Accepts the SNS topic message (as Input) and performs the necessary processing
The advantages of using SNS rather directly calling Lambda are
Reducing the exposure of AWS API to only SNS topic and completely sealing rest of the API (IAM Privileges)
Possibility of Fan-Out Architecture [Multiple Lambda Functions can listen to the same SNS topic - near zero configuration]
For anyone landing on this topic.
It's possible with Freshdesk Marketplace app. With onTicketCreate product event, any actions can be written to execute with a Serverless function. It's completely run in Freshworks platform cloud.
If required, it can call your AWS Lambda.
Is it possible to auto send/push the messages in Amazon SQS to DynamoDB? I wish to send my messages to SQS and for period of time I want to send this to DynamoDB. Another service should fetch the DynamoDB table and send it as email using SES.
Kindly help me out to achieve this. I will be using it for the User notification purpose from a Social networking site.
Thanks.
There is no AWS mechanism to automatically publish SQS messages to DynamoDB; but you can use an AWS Lambda event source mapping to automatically pull SQS messages and invoke a Lambda function, and it's pretty straightforward to write a Lambda function that writes those messages to DynamoDB. (Here's an example using Node.js: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/dynamodb-example-table-read-write.html.)
Yes I agree Hyangelo, you can do this with Simple Workflow Service (SWF).
SWF will give you a control feature over your application enabling you to distribute and execute different services or tasks when you want.
Here is the link to the documentation: http://aws.amazon.com/swf/
Sounds like a workflow system from how you describe what you want, have you considered Simple Workflow Service?
SQS can't be processed w/o pulling messages.
You can either use SWF to solve your use-case OR use SNS.
SNS<=>SQS binding is free by AWS.
Send your messages to SNS, bind your SNS with SQS & lambda-function.
On triggering lambda function - you can create dynamodb-record and send it to another SNS2.
Bind SNS2 <=> SES which will trigger the email.
checkout: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-sns-ses-dynamodb/