Currently we have a system that processes messages from an external JMS message queue. With our blue-green deployments, part of the activation of a stack is modifying security groups to enable/disable the rules needed to connect to the external JMS queue so that only the active stack can process messages. This works great for our current environment however the external queue is not that reliable so we are looking to replace it with SNS topics + SQS queues. We plan to have a single static SQS queue that is subscribed to the SNS topic. The problem we are having is how to enable/disable access to read from the SQS queue so that we can do blue-green and have only 1 stack able to read from the queue at a time?
Would have multiple instance roles, and part of the activation would be modifying the role to give permission to read from SQS work? Is there a better solution?
you can should create SQS queue for each new deployment. But SNS is static.
So you have running version v1. You have SNS topic "SNS" and SQS queue "v1",that subscribed to "SNS".
Then you deployed new version with new SQS "v2". So you have to subcribe it to "SNS". And unsubscribe "v1" from "SNS".
You can receive duplication during this switching.I recommend to use deduplication, for example Elasticache
Related
My AWS application does not allows duplicates. In my application (fan-out) SNS triggers multiple lambda services. Since SNS follows at least once execution, there are chances for triggering same service multiple times.
If i use SNS FIFO fixes duplicate issue? or any best alternatives?
With SNS FIFO we can subscribe only SQS, any alternatives to trigger lambda directly?
My intention is to SNS==> different lambdas (based on input msg type, without duplicates)
Thanks in advance,
Anil
Yes, if you provide a deduplication ID or if you enable content-based message deduplication on the topic. Also, you will have to have an SQS FIFO queue.
The AWS docs has this to say about the deduplication:
Amazon SNS FIFO topics and Amazon SQS FIFO queues support message deduplication, which provides exactly-once message delivery and processing as long as the following conditions are met:
The subscribed SQS FIFO queue exists and has permissions that allow the Amazon SNS service principal to deliver messages to the queue.
The SQS FIFO queue consumer processes the message and deletes it from the queue before the visibility timeout expires.
The Amazon SNS subscription topic has no message filtering. When you configure message filtering, SNS FIFO topics support at-most-once delivery, as messages can be filtered out based on your subscription filter policies.
There are no network disruptions that prevent acknowledgment of the message delivery.
The answer should be obvious, for this one. No, at this this point in time, you can have only SQS FIFO as a subscriber for the topic. The AWS documentation is pretty specific on this:
To fan out messages from Amazon SNS FIFO topics to AWS Lambda functions, extra steps are required. First, subscribe Amazon SQS FIFO queues to the topic. Then configure the queues to trigger the functions.
I have this situation where I am using Amazon SNS + SQS in order to handle domain events.
Basically on domain event I publish a message to SNS and two SQS queues are subscribed to SNS. Since i noticed SQS supports FIFO, but SNS doesn't support FIFO, I am trying to find a resolution on how to simultaneously deliver message A to multiple SQS FIFO queues?
What I had so far
Publish Message A to SNS
Distribute Message A to SQS 1 and SQS 2
All I can think of now is
Publish message A to SQS A
Use code to pull message A from SQS and publish it to SQS 1 and SQS 2
Not really an atomic process I was looking for...
Is there an alternative to this approach?
Today, we launched Amazon SNS FIFO topics, which can fan out messages to multiple Amazon SQS FIFO queues!
https://aws.amazon.com/about-aws/whats-new/2020/10/amazon-sns-introduces-fifo-topics-with-strict-ordering-and-deduplication-of-messages/
You can think about using the AWS Kinesis Data stream. One feature of it is an ordering.
From faq: https://aws.amazon.com/kinesis/data-streams/faqs/
When should I use Amazon Kinesis Data Streams, and when should I use Amazon SQS?
Ordering of records. For example, you want to transfer log data from the application host to the processing/archival host while maintaining the order of log statements.
You can process events from Kinesis to SQSs.
If your goal is to have a message be pushed to two Amazon SQS FIFO queues, I'd recommend:
Have Amazon SNS trigger an AWS Lambda function
The Lambda function can send the same message to both Amazon SQS queues
It is effectively doing the fan-out via Lambda rather than SNS.
The Lambda function might also be able to extract a Message Group ID that it can provide with the SQS message, which will enable parallel processing of messages while maintaining FIFO within the message group. For example, all messages coming from a particular source will be FIFO, but can be processed in parallel with messages from other sources. It's a very powerful capability that would not be available just by having Amazon SNS forward the message.
We are evaluating SNS for our messaging requirements to integrate multiple applications. we have a single producer that publishes messages to multiple topics on SNS. Each topic has 2-5 subscribers. In event of subscriber failures (down for maintenance) I have a few questions on the recommended strategy of using SQS queues per consumer
Is it possible to configure SNS to push to SQS only in event of failure in delivering the message to a subscriber? Dumping all the messages in SQS queue creates a problem for the consumer to analyze all messages in the queue when it restarts.
In event of subscriber failure, it can read messages from SQS queue on restart but how would it know that it missed messages from SNS when it was overloaded?
Any suggestions on handling subscriber failures are welcome.
Thanks!
No, it is not possible to "configure SNS to push to SQS only in event of failure".
Rather than trying to recover a message after a failure, you can configure the Amazon SNS retry policies.
From Setting Amazon SNS Delivery Retry Policies for HTTP/HTTPS Endpoints:
You can use delivery policies to control not only the total number of retries, but also the time delay between each retry. You can specify up to 100 total retries distributed among four discrete phases. The maximum lifetime of a message in the system is one hour. This one hour limit cannot be extended by a delivery policy.
So, you don't need to worry as long as the destination is back online within an hour.
If it is likely to be offline for more than an hour, you will need to find a way to store and "replay" the messages, possibly by inspecting CloudWatch Logs.
Or, here's another idea...
Push initially to SQS. Have an AWS Lambda function triggered by SQS. The Lambda function can do the 'push' that would normally be done by SNS. If it fails, then the standard SQS invisibility process will retry it later, eventually going to a Dead Letter Queue.
If Service1 wants to send a command asynchronously towards a service2, which would be ideal:
service1 -> sns -> sqs -> service2
or
service1 -> sqs -> service2
What would be the basis on deciding on this?
If you want your command to be received by only one consumer use SQS.
If you want it to be received by many consumers publish your command to SNS and subscribe with SQS. Many different subscription types are available in addition to SQS e.g. Lambda, Email, Webhook etc.
In both cases commands (or messages) will be delivered asynchronously.
If you want only a single consumer and also ordered delivery you can use FIFO SQS queues. Ordered delivery cannot be achieved with SNS + SQS.
Amazon SQS will not "deliver" a message. Rather, the application needs to poll SQS to request a message.
Thus, if you wish to "push" a message to your application, then Amazon SNS is more appropriate, but it will not queue messages.
Some people prefer to implement the first option since other consumers can be added with no change to existing code or configuration. The new consumers would simply subscribe to the existing Amazon SNS topic.
However, if you know for sure that there will never be another consumer, then you could feel comfortable going direct to an Amazon SQS queue.
I am not sure this is possible so I just writing my question here. I am working with SNS/SQS architecture where messages going through an SNS topic and then being delivered to SQS queues that are subscribed to that topic. I want to set timers on some specific message. Is it possible to do it when they routed by the SNS topic to the SQS queue?
I don't think this is possible, especially since you only want it "on some specific message".
There is a default Delay setting on an SQS queue, but that would not be applicable for some messages only.
There is no capability to specify Delay settings on a message going from Amazon SNS to Amazon SQS. Your only choice might be to send it to a different queue using Amazon SNS Message Filtering.