I have a lambda that is triggered by an SQS but I want to disable the event trigger in the lower environments.
If there are other parts that publishes to that SQS, what would happens to the the message? There is no DLQ on the SQS. Will the message disappear after the MessageRetentionPeriod is up?
Per the SQS FAQ:
Q: How long can I keep my messages in Amazon SQS message queues?
A: You can configure the Amazon SQS message retention period to a value from 1 minute to 14 days. The default is 4 days. Once the message retention quota is reached, your messages are
automatically deleted.
The word 'quota' is a bad choice, imo, but it means the message retention period.
Note: if a message happens to have been moved from a regular SQS queue to its associated Dead Letter Queue then the retention period is considered to start when the message first arrived on the underlying SQS queue, not when it was transferred to the DLQ.
Related
I'm wondering what would happen if there was an SNS topic having messages written to it, but for a period of time, there is no SQS queue. Let's say there was a container which normally was subscribed to the SNS topic to handle such messages, but it crashed and burned and spent 10 minutes getting resurrected; what would happen to any messages written to that topic, during which there is no queue? Do they disappear forever, or do they wait politely until some queue comes along, subscribes and picks up said messages?
They disappear forever.
SNS cannot know that some subscriber wants to subscribe but simply cannot right now. The topic either has subscribers or it does not. All current subscriber get the message, all future ones do not.
If you have subscriber but the delivery fails there is some SNS specific behaviour in regards to retries: https://docs.aws.amazon.com/sns/latest/dg/sns-message-delivery-retries.html
If the subscriber fails to get the message, a retry mechanism in SNS kicks in as explained in the AWS docs:
When the delivery policy is exhausted, Amazon SNS stops retrying the delivery and discards the message—unless a dead-letter queue is attached to the subscription.
For SQS subscriber retry can be up to 100,015 times, over 23 days
If SQS Queue goes down then message won't disappear , Let's discuss this scenario:
Retry Policy :-** Let's say you set "Number of retries" as n and "Retry-backoff function" as Linear(you can select any other retry-backoff function) in SNS topic , then if SQS is not available then SNS will retry to send that message to subscriber(SQS) n number of times based on the "Retry-backoff function" .
But if you set Number of retries as 0 then your message will delete from SNS topic immediately if Subscriber(SQS) is not available
We are evaluating SNS for our messaging requirements to integrate multiple applications. we have a single producer that publishes messages to multiple topics on SNS. Each topic has 2-5 subscribers. In event of subscriber failures (down for maintenance) I have a few questions on the recommended strategy of using SQS queues per consumer
Is it possible to configure SNS to push to SQS only in event of failure in delivering the message to a subscriber? Dumping all the messages in SQS queue creates a problem for the consumer to analyze all messages in the queue when it restarts.
In event of subscriber failure, it can read messages from SQS queue on restart but how would it know that it missed messages from SNS when it was overloaded?
Any suggestions on handling subscriber failures are welcome.
Thanks!
No, it is not possible to "configure SNS to push to SQS only in event of failure".
Rather than trying to recover a message after a failure, you can configure the Amazon SNS retry policies.
From Setting Amazon SNS Delivery Retry Policies for HTTP/HTTPS Endpoints:
You can use delivery policies to control not only the total number of retries, but also the time delay between each retry. You can specify up to 100 total retries distributed among four discrete phases. The maximum lifetime of a message in the system is one hour. This one hour limit cannot be extended by a delivery policy.
So, you don't need to worry as long as the destination is back online within an hour.
If it is likely to be offline for more than an hour, you will need to find a way to store and "replay" the messages, possibly by inspecting CloudWatch Logs.
Or, here's another idea...
Push initially to SQS. Have an AWS Lambda function triggered by SQS. The Lambda function can do the 'push' that would normally be done by SNS. If it fails, then the standard SQS invisibility process will retry it later, eventually going to a Dead Letter Queue.
I have a lambda function triggered by an SQS queue. When the lambda fails to process a message, it redrives the message to a dead letter queue. The dead letter queue is configured with a delivery delay of 5 minutes, but any messages are visible and processed immediately.
The delivery delay seems to be getting ignored. Is this supposed to happen? Is there a way to configure a "redrive delay" for an SQS?
Looks like this is working as intended per this post from 2016 - https://forums.aws.amazon.com/thread.jspa?messageID=702896򫦰
According to documentation the maximum SQS message retention period(MessageRetentionPeriod) is 14 days. After that time message will be deleted from the queue.
Is any way with SQS to not lose these messages after their retention period expired? For example, it is not clear or is it possible to use Dead Letter Queue for this purpose?
Well 14 days is the max limit you can keep the message. After 14 days you can move that message to S3 Bucket for backup. Also there is a hack you can do with DLQ.
Here is a quick hack where you send that message back to the main queue. This is definitely not the best or recommended option.
Set the main SQS queue as the DLQ for the actual DLQ with Maximum
Receives as 1.
View the content in DLQ (This will move the messages to the main
queue as this is the DLQ for the actual DLQ)
Remove the setting so that the main queue is no more the DLQ of the
actual DLQ
How does the Amazon SQS's DLQ work when it comes to old messages?
Do messages older than 14days() get moved to DLQ instead of being deleted ?
I dont see any documentation relating to how older messages are handled.
From the documentation it looks like just the errored messages are moved to DLQ, is my assumption right ?
Your understanding is correct; Messages that are older than the retention period you have set (max of 14 days), will be deleted, not moved to the DLQ.
SQS automatically deletes messages that have been in a queue for more
than maximum message retention period. The default message retention
period is 4 days. However, you can set the message retention period to
a value from 60 seconds to 1209600 seconds (14 days) with
SetQueueAttributes.
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/MessageLifecycle.html
All the messages that are not processed/consumed will gets pushed to DLQ.
Amazon SQS supports dead-letter queues (DLQ), which other queues (source queues) can target for messages that can't be processed (consumed) successfully.
Reference - https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html