Several different questions have been asked on how to subscribe an SQS FIFO queue to an SNS topic and their answers were "it's impossible" or, more recently, "it's now possible using an SNS FIFO topic".
This question is a bit more specific in its premises:
Using an SNS FIFO topic is not a viable option, it has to be a "standard" topic
Using a "standard" SQS queue is not a viable option, it has to be a FIFO queue
As of time of writing, AWS does not support a direct subscription.
How to achieve it in the next best way possible? Using an AWS Lambda? EventBridge?
As you said, this is currently not possible through direct subscription. A work around to achieve that is to subscribe a lambda to the standard topic and have that lambda sending to the FIFO Queue. It's not ideal since it adds an extra hop, but it does unblock your use-case
Related
In our system, an SNS topic has several subscribers, each one of them doing different processing on the messages.
For some of the subscribers, the order in which the original events occurred and as they were received through the SNS is significant. So the SNS topic was created as FIFO and the order-sensitive subscribers use FIFO SQS.
For the other subscribers, the order does not matter and duplication is acceptable, so those would prefer to use standard SQS to avoid the higher FIFO queue prices and other restrictions.
However, AWS seems to allow subscribing only
FIFO SQS to FIFO SNS
Standard SQS to Standard SNS
Why does not AWS allow subscribing standard SQS to FIFO SNS, as it seems to be simply relaxing the ordering requirement after the messages arrive in SQS?
Are there any workarounds? I see a few suggestions of using lambda to receive messages from SNS and send them to SQS but that means additional costs for lambda execution.
Is there any effort in progress to support this scenario?
The CloudFormation also returns similar error:
Invalid parameter: Endpoint Reason: Please use FIFO SQS queue
Looking at the SNS & SQS documentation, I couldn't find a specific article that mentioned this combination.
Physically I found it's possible to create a subscription of a FIFO queue to a Standard Topic. However, in the AWS console for SQS FIFO queue, under SNS subscriptions it's always empty.
It does seem as if you can publish a message and it will arrive at the lambda triggered via the FIFO queue, but is message ordering/grouping basically discarded? Like what is the point of using this combination? Ideally you want to use FIFO topic + FIFO Queue?
What I found out was that you need a FIFO Topic to ensure strict ordering of insertion into a FIFO queue, but that you still get message grouping on a FIFO queue regardless of whether the message came from Standard/FIFO topic
My AWS application does not allows duplicates. In my application (fan-out) SNS triggers multiple lambda services. Since SNS follows at least once execution, there are chances for triggering same service multiple times.
If i use SNS FIFO fixes duplicate issue? or any best alternatives?
With SNS FIFO we can subscribe only SQS, any alternatives to trigger lambda directly?
My intention is to SNS==> different lambdas (based on input msg type, without duplicates)
Thanks in advance,
Anil
Yes, if you provide a deduplication ID or if you enable content-based message deduplication on the topic. Also, you will have to have an SQS FIFO queue.
The AWS docs has this to say about the deduplication:
Amazon SNS FIFO topics and Amazon SQS FIFO queues support message deduplication, which provides exactly-once message delivery and processing as long as the following conditions are met:
The subscribed SQS FIFO queue exists and has permissions that allow the Amazon SNS service principal to deliver messages to the queue.
The SQS FIFO queue consumer processes the message and deletes it from the queue before the visibility timeout expires.
The Amazon SNS subscription topic has no message filtering. When you configure message filtering, SNS FIFO topics support at-most-once delivery, as messages can be filtered out based on your subscription filter policies.
There are no network disruptions that prevent acknowledgment of the message delivery.
The answer should be obvious, for this one. No, at this this point in time, you can have only SQS FIFO as a subscriber for the topic. The AWS documentation is pretty specific on this:
To fan out messages from Amazon SNS FIFO topics to AWS Lambda functions, extra steps are required. First, subscribe Amazon SQS FIFO queues to the topic. Then configure the queues to trigger the functions.
The AWS FAQs for SNS says:
Q: Will messages be delivered to me in the exact order they were published?
The Amazon SNS service will attempt to deliver messages from the
publisher in the order they were published into the topic. However,
network issues could potentially result in out-of-order messages at
the subscriber end.
Does it apply to SQS consumers, specially a FIFO SQS? I have a use case where one of the consumers needs to maintain the order in which the messages were sent. If this is not the case, I would need to use something else.
Amazon SNS does not currently support delivering messages to SQS FIFO queues. This is documented here.
Important
Amazon SNS isn't currently compatible with FIFO queues.
So since SNS does not guarantee order, and regular SQS queues do not guarantee order,
you have no guarantee of message delivery order when using SNS to fan out messages to SQS.
As of yesterday, SNS also supports strict message ordering and deduplication with FIFO topics.
https://aws.amazon.com/about-aws/whats-new/2020/10/amazon-sns-introduces-fifo-topics-with-strict-ordering-and-deduplication-of-messages/
I am not sure this is possible so I just writing my question here. I am working with SNS/SQS architecture where messages going through an SNS topic and then being delivered to SQS queues that are subscribed to that topic. I want to set timers on some specific message. Is it possible to do it when they routed by the SNS topic to the SQS queue?
I don't think this is possible, especially since you only want it "on some specific message".
There is a default Delay setting on an SQS queue, but that would not be applicable for some messages only.
There is no capability to specify Delay settings on a message going from Amazon SNS to Amazon SQS. Your only choice might be to send it to a different queue using Amazon SNS Message Filtering.