SQS has delay queues that can add a delay before the message is delivered. However, they have a 120,000 cap on the total number of 'in flight' messages. The documentation recommends falling back to another queue when a client gets an OverLimit error.
Is there any way to automatically fallback to another queue by the client publishing to a single SNS topic connected to several SQS delay queues? By that, I mean that the message would generally be pushed from SNS to one of the SQS queues that has available 'in flight' capacity.
Are you actually worried about exceeding the 120,000 'in flight' messages, or are you possibly confusing that with the maximum queue size? (of which there is none).
There is no limit to the number of messages you can have in a queue, the 120,000 limit has to do with the number of messages where you queue consumers have requested messages to consume, but those messages have not yet been processed/deleted?
This is the defintion of 'in flight' from AWS:
Messages are inflight after they have been received from the
queue by a consuming component, but have not yet been deleted from the
queue. If you reach the 120,000 limit, you will receive an OverLimit
error message from Amazon SQS. To help avoid reaching the limit, you
should delete the messages from the queue after they have been
processed. You can also increase the number of queues you use to
process the messages.
Here is a link to confirm that the queue size itself has no limit:
Q: How big can Amazon SQS queues be?
A: A single queue may contain an unlimited number of messages, and you
can create an unlimited number of queues.
http://aws.amazon.com/sqs/faqs/#How_big_can_Amazon_SQS_queues_be
My apologies ahead of time if you are NOT confusing the two issues - if thats the case and you really are talking about exceeding the 120K 'inflight' limit, I'll delete my post.
By the way, found this question/answer, which will confirm for you that just because a message is in the delay queue, they are not 'in flight':
Do Delay Queue messages count as "In Flight" in SQS?
Using SNS wouldn't help in this case, because a copy of the message posted to a SNS topic would be delivered to each SQS subscription.
Some pointers that might help:
Use Dead Letter Queues to store failures:
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/SQSDeadLetterQueue.html
Use Apache Camel to configure your routing and additional capabilities over SQS:
http://camel.apache.org/aws-sqs.html
Also, make sure you understand the difference pointed out in #E.J. Brennan answer.
Related
I am creating an AWS SQS standard queue to store some data temporarily and then retrieve it later. I plan to enqueue about 10^5 to 10^6 messages. I was wondering what is the maximum number of messages the queue can store?
From AWS Docs:
Q: How large can Amazon SQS message queues be?
A single Amazon SQS message queue can contain an unlimited number of messages. However, there is a 120,000 quota for the number of inflight messages for a standard queue and 20,000 for a FIFO queue. Messages are inflight after they have been received from the queue by a consuming component, but have not yet been deleted from the queue.
https://aws.amazon.com/sqs/faqs/
Did any one notice issue while consuming large number of messages from SQS Queues using spring boot listener. Some of the messages are going directly to Dead Letter Queue.
The messages in the Dead Letter Queue show the MessageDeliveryCount as 6.
Please check the visibilityTimeout attribute in DLQ. When processing time exceeds this, messages will move to DLQ
In an AWS SQS standard queue you can set a redrive policy which will cause messages to be retried if there is a failure where by the message is not deleted from the queue.
In my case i have > 1,000,000 messages on the queue which take a couple of hours to process. When a message fails and is put back on the queue will it be put to the end of the queue or the front. Will the messages get retried in a minute or two or in two or three hours when all the other messages have been processed?
There is no guarantee which order messages are returned, so once you return a message it could be retried immediately, when all the others are processed, or anywhere in the middle - there may be some undocumented common patterns for when retries happen, but its not something you can count on or design around.
Q: Does Amazon SQS provide message ordering?
Yes. FIFO (first-in-first-out) queues preserve the exact order in
which messages are sent and received. If you use a FIFO queue, you
don't have to place sequencing information in your messages. For more
information, see FIFO Queue Logic in the Amazon SQS Developer Guide.
Standard queues provide a loose-FIFO capability that attempts to
preserve the order of messages. However, because standard queues are
designed to be massively scalable using a highly distributed
architecture, receiving messages in the exact order they are sent is
not guaranteed.
https://aws.amazon.com/sqs/faqs/
Simple question:
I want to run an autoscale group on Amazon, which fires up multiple instance which processes the messages from a SQS queue. But how do I know that the instances aren't processing the same messages?
I can delete a message from the queue when it's processed. But if it's not deleted yet and still being processed by an instance, another instance CAN download that same message and processing it also, to my opinion.
Aside from the fairly remote possibility of SQS incorrectly delivering the same message more than once (which you still need to account for, even though it is unlikely), I suspect your question stems from a lack of familiarity with SQS's concept of "visibility timeout."
Immediately after the component receives the message, the message is still in the queue. However, you don't want other components in the system receiving and processing the message again. Therefore, Amazon SQS blocks them with a visibility timeout, which is a period of time during which Amazon SQS prevents other consuming components from receiving and processing that message.
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html
This is what keeps multiple queue runners from seeing the same message. Once the visibility timeout expires, the message will be delivered again to a queue consumer, unless you delete it, or it exceeds the maximum configured number of deliveries (at which point it's deleted or goes into a separate dead letter queue if you have configured one). If a job will take longer than the configured visibility timeout, your consumer can also send a request to SQS to change the visibility timeout for that individual message.
Update:
Since this answer was originally written, SQS has introduced FIFO Queues in some of the AWS regions. These operate with the same logic described above, but with guaranteed in-order delivery and additional safeguards to guarantee that occasional duplicate message delivery cannot occur.
FIFO (First-In-First-Out) queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can't be tolerated. FIFO queues also provide exactly-once processing but are limited to 300 transactions per second (TPS).
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html
Switching an application to a FIFO queue does require some code changes, and requires that a new queue be created -- existing queues can't be changed over to FIFO.
You can receive duplicate messages, but only "on rare occasions". And so you should aim for idempotency.
An instance can receive duplicate messages only once the SQS visibility time out has expired. By default the visibility timeout is 30 seconds. So you have 30 seconds to make sure that your processing is done, else other instances may welcome new messages.
See AWS SQS Timeout for timeout details.
For Amazon SQS - If the number of requests a cron job can make in a given window is exceeded by the number of messages received in SQS, how would you ensure all messages are processed in that window?
Standard Queues can hold upto 120,000 Messages and Fifo queues can hold upto 20,000 Messages. These messages can be set to be retained upto 14days which is pretty much.
So Even If the number of requests a cron job can make in a given window is exceeded by the number of messages received in SQS:
All the Incoming messages are still stored in the queue.
The Cron Job/Jobs just takes the first message from the queue and processes it.
Make sure you know and set the Visibity time out while reading from the queue and delete the message from the queue after it is processes. Otherwise, the cron job keeps processing the same message again and again.
Refer other AWS SQS features for better processing And utilizing.
To ensure all messages are processed in that window, As I mentioned You just need run the CRON Job with proper setting.
If you have a Delivery deadline which we will be having in most of the cases, Configure the environment to be Autoscaled.
We can even configure scale-in and scale-out based on SQS message.
Happy to Help.. :)