Can we have a poller for SNS Topic which would get all the messages from SNS Topic in regular intervals, like we can have for SQS?
Amazon SNS is a Publish/Subscribe system. When a message is sent to a topic, it is immediately sent out to all relevant subscribers. Messages are not retained in Amazon SNS.
If you wish to retain messages for later polling, you can use Amazon SQS.
However, Amazon SQS can subscribe to an Amazon SNS topic. This would allow you to store messages for later polling. The flow would be:
Message -> SNS Topic -> SQS Queue subscription -> Available for polling by worker
Related
I have created a SNS Topic and subscribed a SQS Queue with to this Topic using CDK.
I would like to understand does SQS Queue automatically polls messages published through this SNS Topic?
SNS automatically "pushes" messages to subscribed SQS queues:
Docs: Amazon SNS allows applications to send time-critical messages to multiple subscribers through a “push” mechanism, eliminating the need to periodically check or “poll” for updates.
The integration works this way whether created with the CDK or not.
How to create in AWS SQS something like 'direct' exchange in RabbitMQ:
1 message -> N receivers queues.
Each client app connects to server and creates its own queue, a publisher sends one messages to exchange (direct+routing key) and its sent to all N queues, then each user reads its own queue and the queue is emptied.
This can be done with a 'fan-out' pattern combining Amazon SNS and Amazon SQS:
Create all desired Amazon SQS queues
Create an Amazon SNS Topic
Subscribe all of the Amazon SQS queues to the Amazon SNS Topic
Send a message to the Amazon SNS topic -- this will be sent to all subscribing queues. Each queue will have its own copy of the message.
Make sure you use Amazon SNS raw message delivery to preserve the format of the initial message as it goes from Amazon SNS to the SQS queues.
See also: How to Fan-Out to Different SQS Queues Using SNS Message Filtering | by Lorenz Vanthillo | Better Programming
I would like to know if it's possible to persist all unacknowledged messages from an SNS topic to an S3 file given a certain time window. These messages don't necessarily need to follow the original order in the S3 file, a timestamp attribute is enough.
If all you want is to save all messages published to your SNS topic in an S3 bucket, then you can simply subscribe to your SNS topic the AWS Event Fork Pipeline for Storage & Backup:
https://docs.aws.amazon.com/sns/latest/dg/sns-fork-pipeline-as-subscriber.html#sns-fork-event-storage-and-backup-pipeline
** Jan 2021 Update: SNS now supports Kinesis Data Firehose as a native subscription type. https://aws.amazon.com/about-aws/whats-new/2021/01/amazon-sns-adds-support-for-message-archiving-and-analytics-via-kineses-data-firehose-subscriptions/
There is no in-built capability to save messages from Amazon SNS to Amazon S3.
However, this week AWS introduced Dead Letter Queues for Amazon SNS.
From Amazon SNS Adds Support for Dead-Letter Queues (DLQ):
You can now set a dead-letter queue (DLQ) to an Amazon Simple Notification Service (SNS) subscription to capture undeliverable messages. Amazon SNS DLQs make your application more resilient and durable by storing messages in case your subscription endpoint becomes unreachable.
Amazon SNS DLQs are standard Amazon SQS queues.
So, if Amazon SNS is unable to deliver a message, it can automatically send it to an Amazon SQS queue. You can later review/process those failed messages. For example, you could create an AWS Lambda function that is triggered when a message arrives in the Dead Letter Queue. The function could then store the message in Amazon S3.
I am using Amazon SES for a project and have set up a Receipt rule to send messages from SES to SNS. The SNS has my API end point as a subscriber but to ensure that I do not miss any message I have also set up an SQS Queue and have subscribed the queue to SNS topic.
With this set up I receive each SES email twice. (One from SNS and one with a poll in SQS). Is there a way to send only the failed SNS messages to SQS queue so that I don't have to check for duplicates always ?
Rule which sends SES messages to SNS:
SQS queue subscribed to SNS topic:
With your setup, it's not possible to do so. But there are other approaches that you can try out. It might introduce complexity into your application, but it's worth trying. Some of the approaches are given below.
SES to SNS and send all messages from SNS to SQS, and poll SQS for the messages. If some fail, put them to a Deadletterqueue (similar to SQS), and poll that queue separately from time to time, to look for failed messages. This makes the messages more persistent, but slightly inefficient due to polling.
SES to SNS and let SNS use it's delivery policy as needed. You can avoid an SQS, and ask SNS to look for the delivery status, and retry if it is a failure. You can define the retry policy as needed and it's given in this link. After a lot of trying, the message can be discarded as a total failure. You can also write to AWS CloudWatch Logs on the status of the retries.
Short answer, no.
If you've subscribed an SQS queue to an SNS topic, it'll receive all messages published to the latter. SNS doesn't know which messages were successfully processed by your API, so can't selectively send to SQS.
I know I can configure an Amazon S3 bucket to publish events to a SQS topic and to a SNS topic.
But, is it possible to configure the bucket to publish the event to SQS first, and then, when the message has been sent to SQS, have the bucket publish the event to SNS (kind of publish these events synchronously)?
An Amazon S3 bucket can publish a notification to one of:
Amazon Simple Notification Service (SNS)
Amazon Simple Queue Service (SQS)
AWS Lambda
However, SNS can also send a message to SQS. (More accurately, SQS can be added as a subscriber to an SNS topic).
Therefore, you could choose to send the event to SNS, which can on-send the event to an SQS queue. This is a good way to "fork" the event, sending it to multiple SNS subscribers.