My AWS application does not allows duplicates. In my application (fan-out) SNS triggers multiple lambda services. Since SNS follows at least once execution, there are chances for triggering same service multiple times.
If i use SNS FIFO fixes duplicate issue? or any best alternatives?
With SNS FIFO we can subscribe only SQS, any alternatives to trigger lambda directly?
My intention is to SNS==> different lambdas (based on input msg type, without duplicates)
Thanks in advance,
Anil
Yes, if you provide a deduplication ID or if you enable content-based message deduplication on the topic. Also, you will have to have an SQS FIFO queue.
The AWS docs has this to say about the deduplication:
Amazon SNS FIFO topics and Amazon SQS FIFO queues support message deduplication, which provides exactly-once message delivery and processing as long as the following conditions are met:
The subscribed SQS FIFO queue exists and has permissions that allow the Amazon SNS service principal to deliver messages to the queue.
The SQS FIFO queue consumer processes the message and deletes it from the queue before the visibility timeout expires.
The Amazon SNS subscription topic has no message filtering. When you configure message filtering, SNS FIFO topics support at-most-once delivery, as messages can be filtered out based on your subscription filter policies.
There are no network disruptions that prevent acknowledgment of the message delivery.
The answer should be obvious, for this one. No, at this this point in time, you can have only SQS FIFO as a subscriber for the topic. The AWS documentation is pretty specific on this:
To fan out messages from Amazon SNS FIFO topics to AWS Lambda functions, extra steps are required. First, subscribe Amazon SQS FIFO queues to the topic. Then configure the queues to trigger the functions.
Related
In our system, an SNS topic has several subscribers, each one of them doing different processing on the messages.
For some of the subscribers, the order in which the original events occurred and as they were received through the SNS is significant. So the SNS topic was created as FIFO and the order-sensitive subscribers use FIFO SQS.
For the other subscribers, the order does not matter and duplication is acceptable, so those would prefer to use standard SQS to avoid the higher FIFO queue prices and other restrictions.
However, AWS seems to allow subscribing only
FIFO SQS to FIFO SNS
Standard SQS to Standard SNS
Why does not AWS allow subscribing standard SQS to FIFO SNS, as it seems to be simply relaxing the ordering requirement after the messages arrive in SQS?
Are there any workarounds? I see a few suggestions of using lambda to receive messages from SNS and send them to SQS but that means additional costs for lambda execution.
Is there any effort in progress to support this scenario?
The CloudFormation also returns similar error:
Invalid parameter: Endpoint Reason: Please use FIFO SQS queue
I have this situation where I am using Amazon SNS + SQS in order to handle domain events.
Basically on domain event I publish a message to SNS and two SQS queues are subscribed to SNS. Since i noticed SQS supports FIFO, but SNS doesn't support FIFO, I am trying to find a resolution on how to simultaneously deliver message A to multiple SQS FIFO queues?
What I had so far
Publish Message A to SNS
Distribute Message A to SQS 1 and SQS 2
All I can think of now is
Publish message A to SQS A
Use code to pull message A from SQS and publish it to SQS 1 and SQS 2
Not really an atomic process I was looking for...
Is there an alternative to this approach?
Today, we launched Amazon SNS FIFO topics, which can fan out messages to multiple Amazon SQS FIFO queues!
https://aws.amazon.com/about-aws/whats-new/2020/10/amazon-sns-introduces-fifo-topics-with-strict-ordering-and-deduplication-of-messages/
You can think about using the AWS Kinesis Data stream. One feature of it is an ordering.
From faq: https://aws.amazon.com/kinesis/data-streams/faqs/
When should I use Amazon Kinesis Data Streams, and when should I use Amazon SQS?
Ordering of records. For example, you want to transfer log data from the application host to the processing/archival host while maintaining the order of log statements.
You can process events from Kinesis to SQSs.
If your goal is to have a message be pushed to two Amazon SQS FIFO queues, I'd recommend:
Have Amazon SNS trigger an AWS Lambda function
The Lambda function can send the same message to both Amazon SQS queues
It is effectively doing the fan-out via Lambda rather than SNS.
The Lambda function might also be able to extract a Message Group ID that it can provide with the SQS message, which will enable parallel processing of messages while maintaining FIFO within the message group. For example, all messages coming from a particular source will be FIFO, but can be processed in parallel with messages from other sources. It's a very powerful capability that would not be available just by having Amazon SNS forward the message.
The AWS FAQs for SNS says:
Q: Will messages be delivered to me in the exact order they were published?
The Amazon SNS service will attempt to deliver messages from the
publisher in the order they were published into the topic. However,
network issues could potentially result in out-of-order messages at
the subscriber end.
Does it apply to SQS consumers, specially a FIFO SQS? I have a use case where one of the consumers needs to maintain the order in which the messages were sent. If this is not the case, I would need to use something else.
Amazon SNS does not currently support delivering messages to SQS FIFO queues. This is documented here.
Important
Amazon SNS isn't currently compatible with FIFO queues.
So since SNS does not guarantee order, and regular SQS queues do not guarantee order,
you have no guarantee of message delivery order when using SNS to fan out messages to SQS.
As of yesterday, SNS also supports strict message ordering and deduplication with FIFO topics.
https://aws.amazon.com/about-aws/whats-new/2020/10/amazon-sns-introduces-fifo-topics-with-strict-ordering-and-deduplication-of-messages/
I am not sure this is possible so I just writing my question here. I am working with SNS/SQS architecture where messages going through an SNS topic and then being delivered to SQS queues that are subscribed to that topic. I want to set timers on some specific message. Is it possible to do it when they routed by the SNS topic to the SQS queue?
I don't think this is possible, especially since you only want it "on some specific message".
There is a default Delay setting on an SQS queue, but that would not be applicable for some messages only.
There is no capability to specify Delay settings on a message going from Amazon SNS to Amazon SQS. Your only choice might be to send it to a different queue using Amazon SNS Message Filtering.
As there is a limitation on SQS to support multiple consumers to process messages in parallel. ie. m1 to m10 picked by process 1 and m11 to m20 picked by process 2 and so on.. with duplication. Since this is not supported by SQS, I am thinking of using SNS + SQS (list of queues subscribed), where each process listens to its specific queue and processes records.
Is there an option to set between SNS and SQS like round-robin so that SNS distributes messages to SQS in a round robin fashion, So that each queue would have unique messages without duplication across queues?
Thanks in advance!
Regards,
Kumar
If you don't want your SNS publish to go to all subscribers (queues), look in to SNS Message Filtering. Message filtering allows you to define logic controlling which subscribers receive a given message.
By default, a subscriber of an Amazon SNS topic receives every message
published to the topic. To receive only a subset of the messages, a
subscriber assigns a filter policy to the topic subscription.
A filter policy is a simple JSON object. The policy contains
attributes that define which messages the subscriber receives. When
you publish a message to a topic, Amazon SNS compares the message
attributes to the attributes in the filter policy for each of the
topic's subscriptions. If there is a match between the attributes,
Amazon SNS sends the message to the subscriber. Otherwise, Amazon SNS
skips the subscriber without sending the message to it. If a
subscription lacks a filter policy, the subscription receives every
message published to its topic.
Unless you are using SQS FIFO queues, your assumption about the limitation of SQS not supporting multiple parallel consumers is not correct
Standard SQS do support multiple parallel consumers.
Regarding the SQS FIFO queues they don't serve messages from the same message group to more than one consumer at a time. However, if your FIFO queue has multiple message groups, you can take advantage of parallel consumers, allowing Amazon SQS to serve messages from different message groups to different consumers.