How does the Amazon SQS's DLQ work when it comes to old messages?
Do messages older than 14days() get moved to DLQ instead of being deleted ?
I dont see any documentation relating to how older messages are handled.
From the documentation it looks like just the errored messages are moved to DLQ, is my assumption right ?
Your understanding is correct; Messages that are older than the retention period you have set (max of 14 days), will be deleted, not moved to the DLQ.
SQS automatically deletes messages that have been in a queue for more
than maximum message retention period. The default message retention
period is 4 days. However, you can set the message retention period to
a value from 60 seconds to 1209600 seconds (14 days) with
SetQueueAttributes.
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/MessageLifecycle.html
All the messages that are not processed/consumed will gets pushed to DLQ.
Amazon SQS supports dead-letter queues (DLQ), which other queues (source queues) can target for messages that can't be processed (consumed) successfully.
Reference - https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
Related
I have a lambda that is triggered by an SQS but I want to disable the event trigger in the lower environments.
If there are other parts that publishes to that SQS, what would happens to the the message? There is no DLQ on the SQS. Will the message disappear after the MessageRetentionPeriod is up?
Per the SQS FAQ:
Q: How long can I keep my messages in Amazon SQS message queues?
A: You can configure the Amazon SQS message retention period to a value from 1 minute to 14 days. The default is 4 days. Once the message retention quota is reached, your messages are
automatically deleted.
The word 'quota' is a bad choice, imo, but it means the message retention period.
Note: if a message happens to have been moved from a regular SQS queue to its associated Dead Letter Queue then the retention period is considered to start when the message first arrived on the underlying SQS queue, not when it was transferred to the DLQ.
I'm wondering what would happen if there was an SNS topic having messages written to it, but for a period of time, there is no SQS queue. Let's say there was a container which normally was subscribed to the SNS topic to handle such messages, but it crashed and burned and spent 10 minutes getting resurrected; what would happen to any messages written to that topic, during which there is no queue? Do they disappear forever, or do they wait politely until some queue comes along, subscribes and picks up said messages?
They disappear forever.
SNS cannot know that some subscriber wants to subscribe but simply cannot right now. The topic either has subscribers or it does not. All current subscriber get the message, all future ones do not.
If you have subscriber but the delivery fails there is some SNS specific behaviour in regards to retries: https://docs.aws.amazon.com/sns/latest/dg/sns-message-delivery-retries.html
If the subscriber fails to get the message, a retry mechanism in SNS kicks in as explained in the AWS docs:
When the delivery policy is exhausted, Amazon SNS stops retrying the delivery and discards the message—unless a dead-letter queue is attached to the subscription.
For SQS subscriber retry can be up to 100,015 times, over 23 days
If SQS Queue goes down then message won't disappear , Let's discuss this scenario:
Retry Policy :-** Let's say you set "Number of retries" as n and "Retry-backoff function" as Linear(you can select any other retry-backoff function) in SNS topic , then if SQS is not available then SNS will retry to send that message to subscriber(SQS) n number of times based on the "Retry-backoff function" .
But if you set Number of retries as 0 then your message will delete from SNS topic immediately if Subscriber(SQS) is not available
The host that my SQS app runs on recently experienced some external DNS resolution issues. This meant that suddenly, I couldn't hit the SQS API endpoints. As a part of figuring out what was going on, I logged into the AWS console only to find the messages count slowly declining.
If the messages could not have been consumed by my app, how could the number of messages in the queue be declining?
Amazon SQS automatically deletes messages that have been in a queue for more than the maximum message retention period.
By default, the message retention period is 4 days. However, you can set the message retention period to any value from 60 seconds to 1,209,600 seconds (14 days) in the AWS console.
Link
According to the Cloud PubSub documentation messages cease to be stored if not consumed within 7 days.
Retains unacknowledged messages in persistent storage for 7 days from the moment of publication.
What happens to PubSub messages when the retention delay is over ? Are they simply deleted ?
Is there a log entry generated for a lost message ?
Is there a way to know how many messages were lost ?
When the seven-day retention expires, the messages are deleted. No long entry is generated for these deletions. There is not a metric to determine how many messages are deleted, though you could set up alerting on the subscription/oldest_unacked_message_age metric in Stackdriver to know when you have messages that are close to the seven-day retention limit.
According to documentation the maximum SQS message retention period(MessageRetentionPeriod) is 14 days. After that time message will be deleted from the queue.
Is any way with SQS to not lose these messages after their retention period expired? For example, it is not clear or is it possible to use Dead Letter Queue for this purpose?
Well 14 days is the max limit you can keep the message. After 14 days you can move that message to S3 Bucket for backup. Also there is a hack you can do with DLQ.
Here is a quick hack where you send that message back to the main queue. This is definitely not the best or recommended option.
Set the main SQS queue as the DLQ for the actual DLQ with Maximum
Receives as 1.
View the content in DLQ (This will move the messages to the main
queue as this is the DLQ for the actual DLQ)
Remove the setting so that the main queue is no more the DLQ of the
actual DLQ