Did any one notice issue while consuming large number of messages from SQS Queues using spring boot listener. Some of the messages are going directly to Dead Letter Queue.
The messages in the Dead Letter Queue show the MessageDeliveryCount as 6.
Please check the visibilityTimeout attribute in DLQ. When processing time exceeds this, messages will move to DLQ
Related
Trying to write some tests for my AWS SQS Queue and its associated Dead letter queue. I want to somehow in my tests force the message from the queue to its DLQ, then read from the dlq to see if the message is there.
Reading from the DLQ is no problem. But does anyone know a quick and easy way I can programmatically force an sqs queue to send a message to its associated DLQ?
The Dead Letter Queue is simply a SQS Queue, so you could send a message to it like you would any other queue.
The DLQ is configured when you create your normal queue and you need to pass a arn of a queue that will be used as DLQ.
When you configure your DLQ you set the maxReceiveCount (Maximum receives on the console) that is the number of times a message is delivered to the source queue before being moved to the dead-letter queue. When the ReceiveCount for a message exceeds the maxReceiveCount for a queue, Amazon SQS moves the message to the dead-letter-queue.
If want to test the process to send messages to DLQs, you need to force in your tests an error on the queue messages' processing to send a message to the DLQ queue, this will be the best way to understand if the errors are going to the queue correctly.
The process to send messages to the DLQ can be done in the following ways:
You explicitly send a message to the DLQ, if you found some error and do not want to process the message or delete at that time.
If you read the messages more times than the maxReceiveCount and do not process the message (read and delete from the queue) the AWS SQS service will understand that you are having problems on that message and will send automatically to the DLQ for you. (eg. maxReceiveCount equals 1 and your read the message and did not delete 2 times)
To understand more about DLQs, take a look here: Amazon SQS dead-letter queues.
we have currently a sqs queue for processing incoming data. Is there a recommended way for managing two DLQs for one queue?
if there is a parsing error of the incoming data, then I want to move the message directly into a "userInput" DLQ, without redrives
if our mongo is on maxConnections, or any other error occurs, then the configured redrive policy should take place
Do I have to put the message manually into the dlq for the first szenario, or is there a better way?
Thanks!
An Amazon SQS queue only has one Dead Letter Queue.
If a message is read from an SQS queue more than a defined number of times, the message can be moved to the Dead Letter Queue for later processing. However, there is no control over what conditions will send the message to the Dead Letter Queue. It is simply based on a message being retrieved more than the maxReceiveCount.
See: Amazon SQS dead-letter queues
Please note that SQS itself does not process the message. Rather, you will have an app or an AWS Lambda function that reads the message from the queue and processes the message. Therefore, you could program your desired functionality (checking incoming data, responding to Mongo maxConnections) into the code that is processing the message from SQS. If it detects such a problem, that program could send the message to a specific queue, and then delete the original message from the source SQS queue.
This would have the same behaviour as having "multiple DLQs", except that your code is responsible for the logic of moving the messages to these queues, rather than Amazon SQS doing it.
SQS Supports only Single DLQ .
Alternatively what you could do is, Let the Consumer of the **Queue** Handle your first case. Meaning "if there is a parsing error of the incoming data" Let the Consumer Move it to another queue.
And The Second case of redrive policy will be handled Automatically and Moved to Real DLQ after the maxReceiveCount
You can have only one DLQ for an queue.
However, you could subscribe a lambda function to that one DLQ.
The lambda function could process the "bad" messages and distributed to other DQLs queues. So you could have additional DLQs for which the function would filter the messages.
Need a way for an admin to place a message back for reprocessing after he reviewed it in dead letter queue. We are using both AWS SQS and Active MQ for different pieces of the system. Assume there was some connectivity problem that prevented the message from being processed that has been resolved.
There is no command to send a message from an Amazon SQS Dead Letter Queue back to the original queue. In fact, there is no command to send messages between any queues.
Your application will need to send a new message to the queue, then delete the 'dead' message from the DLQ.
Currently we are using 'RabbitMQ' for reliable messaging delivery,we have plan to move SQS.RMQP monitors its consumers through TCP when any consumer of the Queue goes down it will automatically Re-Queue the messaging for processing.
Will SQS monitor all its slaves? Will the message is visible in the queue if one of its consumer goes down while processing the message?
I have tried to find out the same from documentation,i could not find any.
If by 'slaves', you mean SQS consumers, then no, SQS does not monitor the consumers from the queue.
In a nutshell, SQS works like this:
A consumer requests a message from the queue.
SQS sends the message to the consumer to process and makes that message temporarily invisible to other consumers.
When the consumer is finished processing the message, it sends a 'DeleteMessage' requests back to SQS and SQS removes that item from the queue.
If a consumer does not send the deletemessage back soon enough (within its configurable timeout period), then SQS will put the message back into the queue automatically.
So SQS doesn't monitor you consumers, but if a consumer requests messages - and does nothing with them - they will eventually end up back in the queue to be processed by someone else.
But if your queue doesn't have any consumers, then sooner or later (14 days max), the messages will be deleted altogether (or sent to a dead-letter-queue if you set that up).
It is usually a good idea to setup your queue consumers in an auto-scaling group, with a health-check that can verify that it is running/processing properly. If an instance fails a health check, it will be terminated and a new instance spun up to continue the work in the queue. Optionally, you can spin up extra instances if the size of the SQS queue grows to meet peak demand.
SQS has delay queues that can add a delay before the message is delivered. However, they have a 120,000 cap on the total number of 'in flight' messages. The documentation recommends falling back to another queue when a client gets an OverLimit error.
Is there any way to automatically fallback to another queue by the client publishing to a single SNS topic connected to several SQS delay queues? By that, I mean that the message would generally be pushed from SNS to one of the SQS queues that has available 'in flight' capacity.
Are you actually worried about exceeding the 120,000 'in flight' messages, or are you possibly confusing that with the maximum queue size? (of which there is none).
There is no limit to the number of messages you can have in a queue, the 120,000 limit has to do with the number of messages where you queue consumers have requested messages to consume, but those messages have not yet been processed/deleted?
This is the defintion of 'in flight' from AWS:
Messages are inflight after they have been received from the
queue by a consuming component, but have not yet been deleted from the
queue. If you reach the 120,000 limit, you will receive an OverLimit
error message from Amazon SQS. To help avoid reaching the limit, you
should delete the messages from the queue after they have been
processed. You can also increase the number of queues you use to
process the messages.
Here is a link to confirm that the queue size itself has no limit:
Q: How big can Amazon SQS queues be?
A: A single queue may contain an unlimited number of messages, and you
can create an unlimited number of queues.
http://aws.amazon.com/sqs/faqs/#How_big_can_Amazon_SQS_queues_be
My apologies ahead of time if you are NOT confusing the two issues - if thats the case and you really are talking about exceeding the 120K 'inflight' limit, I'll delete my post.
By the way, found this question/answer, which will confirm for you that just because a message is in the delay queue, they are not 'in flight':
Do Delay Queue messages count as "In Flight" in SQS?
Using SNS wouldn't help in this case, because a copy of the message posted to a SNS topic would be delivered to each SQS subscription.
Some pointers that might help:
Use Dead Letter Queues to store failures:
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/SQSDeadLetterQueue.html
Use Apache Camel to configure your routing and additional capabilities over SQS:
http://camel.apache.org/aws-sqs.html
Also, make sure you understand the difference pointed out in #E.J. Brennan answer.