With the introduction of FIFO queues in SQS we are guaranteed that SQS messages will be delivered in order.
Now, could there be an out-of-order delivery when I publish messages through SNS? Here's what I'm thinking that could happen:
_____ _____ __________
| App |-- Publish msg 1 --> | SNS | --> Queue msg 2 --> | FIFO SQS | --> Consume 2
| |-- Publish msg 2 --> | | --> Queue msg 1 --> | | --> Consume 1
----- ----- ----------
Is that scenario possible?
This scenario is currently impossible because SNS is not compatible with FIFO SQS in AWS. It is only supported for standard queues. This is clearly mentioned in the AWS docs in this link, under special notes.
EDIT
It's also listed in the SNS FAQs, under Are Amazon SQS FIFO queues compatible with Amazon Simple Notification Service (SNS)?
The answer's still the same as of July 26, 2020: Amazon SNS does not currently support forwarding messages to Amazon SQS FIFO queues. You can use SNS to forward messages to standard queues.
SNS FIFO just launched. SNS can now integrate with SQS FIFO https://aws.amazon.com/blogs/aws/introducing-amazon-sns-fifo-first-in-first-out-pub-sub-messaging/
It is possible with the latest SNS support for FIFO,
As per the excerpt from amazon doc
FIFO topics support fanning out messages to multiple subscriptions
with high durability, filtering, encryption, and privacy, while FIFO
topics provide the added benefit of ordering and deduplication of
message
I have listed some important links for you to refer
Amazon SNS introduces First-in-First-out (FIFO) topics with strict ordering and deduplication of messages
Introducing Amazon SNS FIFO – First-In-First-Out Pub/Sub Messaging
Building event-driven architectures with Amazon SNS FIFO
As your SNS topic protocol endpoint is Amazon SQS which perfectly works fine.
NOTE for other protocol endpoints Ref
Currently, the endpoint protocol must be Amazon SQS, with an Amazon SQS FIFO queue's Amazon Resource Name (ARN) as the endpoint.
SNS FIFO topics can't deliver messages to customer managed endpoints, such as email addresses, mobile apps, phone numbers for
text messaging (SMS), or HTTP(S) endpoints. These endpoint types
aren't guaranteed to preserve strict message ordering. Attempts to
subscribe customer managed endpoints to SNS FIFO topics result in
errors.
My team was also looking for SNS integration with SQS FIFO queue but unfortunately it's not compatible till now.
Related
I'm using AWS SES for sending emails. To track email delivery status information I have followig pipeline:
SES Config set -> SNS -> SQS -> Lambda -> Our system
Sometimes the events comes in wrong order. For example the event Send comes 30 seconds later after event Delivery, so in our system is the status Delivery replaced by Send.
I tried to set the SNS topic and SQS queue as FIFO. But in SES config set it is not possible to add SNS FIFO topic as destination. AWS says: FIFO topics are not supported.
The SES is located in different account than SNS, but I believe, that this has no effect.
Maybe I can remove the SQS, the lambda is probably fast enough, so it does not need the queue. But anyway, the standard SNS is high-throughput, best-effort delivery, so it does not keep the order.
I would like to solve this issue on the AWS side. How can I configure the AWS, so the events (resp. lambda) are fired in correct order? Does the SES publish the events in specific order?
I have created a SNS Topic and subscribed a SQS Queue with to this Topic using CDK.
I would like to understand does SQS Queue automatically polls messages published through this SNS Topic?
SNS automatically "pushes" messages to subscribed SQS queues:
Docs: Amazon SNS allows applications to send time-critical messages to multiple subscribers through a “push” mechanism, eliminating the need to periodically check or “poll” for updates.
The integration works this way whether created with the CDK or not.
How to create in AWS SQS something like 'direct' exchange in RabbitMQ:
1 message -> N receivers queues.
Each client app connects to server and creates its own queue, a publisher sends one messages to exchange (direct+routing key) and its sent to all N queues, then each user reads its own queue and the queue is emptied.
This can be done with a 'fan-out' pattern combining Amazon SNS and Amazon SQS:
Create all desired Amazon SQS queues
Create an Amazon SNS Topic
Subscribe all of the Amazon SQS queues to the Amazon SNS Topic
Send a message to the Amazon SNS topic -- this will be sent to all subscribing queues. Each queue will have its own copy of the message.
Make sure you use Amazon SNS raw message delivery to preserve the format of the initial message as it goes from Amazon SNS to the SQS queues.
See also: How to Fan-Out to Different SQS Queues Using SNS Message Filtering | by Lorenz Vanthillo | Better Programming
I would like to know if it's possible to persist all unacknowledged messages from an SNS topic to an S3 file given a certain time window. These messages don't necessarily need to follow the original order in the S3 file, a timestamp attribute is enough.
If all you want is to save all messages published to your SNS topic in an S3 bucket, then you can simply subscribe to your SNS topic the AWS Event Fork Pipeline for Storage & Backup:
https://docs.aws.amazon.com/sns/latest/dg/sns-fork-pipeline-as-subscriber.html#sns-fork-event-storage-and-backup-pipeline
** Jan 2021 Update: SNS now supports Kinesis Data Firehose as a native subscription type. https://aws.amazon.com/about-aws/whats-new/2021/01/amazon-sns-adds-support-for-message-archiving-and-analytics-via-kineses-data-firehose-subscriptions/
There is no in-built capability to save messages from Amazon SNS to Amazon S3.
However, this week AWS introduced Dead Letter Queues for Amazon SNS.
From Amazon SNS Adds Support for Dead-Letter Queues (DLQ):
You can now set a dead-letter queue (DLQ) to an Amazon Simple Notification Service (SNS) subscription to capture undeliverable messages. Amazon SNS DLQs make your application more resilient and durable by storing messages in case your subscription endpoint becomes unreachable.
Amazon SNS DLQs are standard Amazon SQS queues.
So, if Amazon SNS is unable to deliver a message, it can automatically send it to an Amazon SQS queue. You can later review/process those failed messages. For example, you could create an AWS Lambda function that is triggered when a message arrives in the Dead Letter Queue. The function could then store the message in Amazon S3.
Can we have a poller for SNS Topic which would get all the messages from SNS Topic in regular intervals, like we can have for SQS?
Amazon SNS is a Publish/Subscribe system. When a message is sent to a topic, it is immediately sent out to all relevant subscribers. Messages are not retained in Amazon SNS.
If you wish to retain messages for later polling, you can use Amazon SQS.
However, Amazon SQS can subscribe to an Amazon SNS topic. This would allow you to store messages for later polling. The flow would be:
Message -> SNS Topic -> SQS Queue subscription -> Available for polling by worker