I want to get all the messages in the queue to process them. However the property for MaxNumberOfMessages is 10 (based on documentation)
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html
How can I read in all messages so I can process them? Or how would I know when queue is empty?
thanks
When you receive messages from the queue, they are marked as "in flight." After you successfully process them, you send a call to the queue to delete them. This call will include IDs of each of the messages.
When the queue is empty, the next read will have an empty Messages array.
Usually when I do this I wrap my call to read the queue in a loop (a while loop) and only keep processing if I have Messages after doing a read.
It shouldn't make any difference if it's a FIFO queue or a standard one.
To check if the queue is empty you have to verify the total number of messages in the queue is zero. SQS does not provide a single metric for this, rather you have to calculate the sum of three different metrics.
From the docs:
To confirm that a queue is empty (AWS CLI, AWS API)
Stop all producers from sending messages.
Repeatedly run one of the following commands:
AWS CLI: get-queue-attributes
AWS API: GetQueueAttributes
Observe the metrics for the following attributes:
ApproximateNumberOfMessagesDelayed
ApproximateNumberOfMessagesNotVisible
ApproximateNumberOfMessages
When all of them are 0 for several minutes, the queue is empty.
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/confirm-queue-is-empty.html
Getting an empty response from a ReceiveMessage call does NOT necessarily mean the queue is empty. You can have messages in the queue and still receive an empty response if:
Messages are delayed - You can set delays on individual messages for standard queues or at the queue level for standard and FIFO queues. During the delay period messages are invisible to consumers.
Messages are in-flight - When a consumer receives a message, that message remains in the queue until the consumer deletes it by calling DeleteMessage. While the message is in this state it is considered in-flight and is not available for other consumers.
Multiple messages have the same message group id in a FIFO queue - When a consumer receives a message from a FIFO queue, no other consumer can receive messages from the same message group. This ensures messages are processed in FIFO order.
By summing the metrics listed above, you can account for all of these scenarios.
Related
I've a FIFO SQS queue with a lambda polling it.
The lambda had this error for one of the msgs of queue:
Runtime exited with error: signal: killed Runtime.ExitError
I know this happened due to max memory reached.
Now, any msg sent to queue after this error stays in queue with the faulty msg stuck in Flight as seen on the AWS console.
What I expected:
Msg with max memory reached should have exited, allowing next msgs to be in flight.
What happened:
AWS console showed 1 in flight msg and the new msgs sent after delay were shown under "Messages available" until I had to purge the queue to empty it.
I'm trying to understand if runtime is killed, does the msg is kept in flight permanently?
Lambda is at default concurrency capacity.
Since you are using an Amazon SQS FIFO queue, it sounds like all of your messages have the same MessageGroupId.
When using a FIFO queue, SQS will ensure that all messages with the same MessageGroupId will be processed in order. If the first message is not being correctly processed, it will not send any other messages with the same MessageGroupId for processing, since they will be out-of-order.
If you do not strictly need these messages to be processed in order, then you could supply different values for MessageGroupId, or even use a standard (non-FIFO) queue.
To fix your current situation, you could activate a Dead Letter Queue on this SQS queue, configured to send any messages with repeated failures to an alternate ('Dead Letter') queue. This would then allow subsequent messages to be processed.
I am using SQS queues in two places of my Spring boot application :
In one queue, I would like the messages to be routed to DLQ when maximum numbers of receives for a given message > = 3
For the second case, I don't like to configure a DLQ.
In (1) and (2), however, I would like to delete the message from DLQ and normal queue respectively after 3 times receives.
As of now, I cannot find any such configurations in SQS, that allows me to delete a message from the queue after a certain number of receives.
Maybe, I am missing something. Could anyone please help here?
There is no mechanism for "automated" deletion of messages from SQS queue upon a given number of unsuccessful received, if you don't want to use DLQ.
Without DQL, SQS will keep messages in the queue till they expire. Thus, if you want to do what you wish, you have to create your own solution for that. You have to store number of times the message got received, e.g., in DynamoDB, and then upon third receive, the consumer must explicitly delete the message from the queue.
You can explore sqs message attributes. Once you received the message, delete it from the queue and send it back to the queue with an added message attribute stating how many times you have received the message.
Ref:https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-java-send-message-with-attributes.html
I'm learning AWS SQS and I've sent 6 messages to a FIFO queue, with the same GroupId. But when I try to poll for messages, I can only receive 2 of them (Why? I set the MaxNumberOfMessages=10 using boto3 API, but I can only receive 2. How can I receive all of the messages?).
(As shown in this picture, I have 5 messages available, but I can only receive 2 messages.)
I tried to delete one of two received messages and poll again. The deleted one is gone, and I received a new message. But in total, it's still 2 messages.
Using an Amazon SQS FIFO queue means that you want to receive messages in order. It will also try to ensure ordering within a Message Group.
This means that, if some messages for a given Message Group ID are currently being processed ("in flight"), no more messages for that Message Group will be provided since an earlier message might be returned to the queue if not fully processed. This could result in messages being processed out-of-order.
From Using the Amazon SQS message group ID - Amazon Simple Queue Service:
To interleave multiple ordered message groups within a single FIFO queue, use message group ID values (for example, session data for multiple users). In this scenario, multiple consumers can process the queue, but the session data of each user is processed in a FIFO manner.
When messages that belong to a particular message group ID are invisible, no other consumer can process messages with the same message group ID.
Therefore, your choices are:
Don't uses a FIFO queue, or
Use different Message Group IDs, or
Be happy with what it is doing because that is desired FIFO behaviour
From AWS Docs:
The maximum number of messages to return. Amazon SQS never returns more messages than this value (however, fewer messages might be returned).
Just like doc's write, you can get less messages. You have to call ReceiveMessage multiple times, usually done in a loop. You can also increase WaitTimeSeconds so that the ReceiveMessage does not return immedietly if there are no messages.
My application consists of:
1 Amazon SQS message queue
n workers
The workers have the following logic:
1. Wait for message from SQS queue
2. Perform task described in message
3. Delete the message from the SQS queue
4. Go to (1)
I want each message to be received by only one worker to avoid redundant work.
Is there a mechanism to mark a message as "in progress" using SQS, so that other pollers do not receive it?
Alternatively, is it appropriate to delete the message as soon as it is received?
1. Wait for message from SQS queue
2. Delete the message from the SQS queue
3. Perform task described in message
4. Go to (1)
If I follow this approach, is there a way to recover received but unprocessed messages in case a worker crashes (step (3) fails)?
This question is specific to Spring, which contains all sorts of magic.
An SQS message is considered to be "inflight" after it is received from a queue by a consumer, but not yet deleted from the queue. These messages are not visible to other consumers.
In SQS messaging, a message is considered in "inflight" if:
You the consumer have received it, and
the visibility timeout has not expired and
you have not deleted it.
SQS is designed so that you can call ReceiveMessage and a message is given to you for processing. You have some amount of time (the visibility timeout) to perform the processing on this message. During this "visibility" timeout, if you call ReceiveMessage again, no worker will be returned the message you are currently working with. It is hidden.
Once the visibility timeout expires the message will be able to be returned to future ReceiveMessage calls. This could happen if the consumer fails in some way. If the process is successful, then you can delete the message.
The number of messages that are hidden from ReceiveMessage call is the "inflight" number. Currently a SQS queue is set by default to allow a max of 120,000 messages to be "inflight".
http://docs.amazonwebservices.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html
There will be ReciepientHandle String which will be sent with the message. It will be having a expiry time based on queue visibility timeout.
You can use the this ReciepientHandle to delete message from queue
I have one Node.js application that pool messages from an AWS SQS queue and process these messages.
However some of these messages are not relevant to my service and they will be filtered out and not processed.
I am wondering if I can do this filtering stuffs from AWS before receiving these irrelevant messages ...
For example if the message does not have the following attribute data.name than this message will be deleted before reaching my application ...
Filtering these messages before sending them to to the queue is not possible (according to my client).
No that is not possible without polling the message itself. So you would need some other consumer polling the messages and returning them to queue (not calling DeleteMessage on the received delete handle) if they meet your requirements but that would be overkill in most of the cases, depending on the ratio of "good" and "bad" messages but still you would have to process "good" messages twice.
Better way would be to set up additional consumer and 2 queues. Producer sends messages to the first queue which is polled by the first consumer whose sole purpose is to filter messages and to send "good" messages to the second queue which would then be polled by your current consumer application. But again, this is much more costly.
If you can't filter messages before sending them to queue then filter them in your consuming application or you will have to pay some extra for this extra functionality.