I'm learning AWS SQS and I've sent 6 messages to a FIFO queue, with the same GroupId. But when I try to poll for messages, I can only receive 2 of them (Why? I set the MaxNumberOfMessages=10 using boto3 API, but I can only receive 2. How can I receive all of the messages?).
(As shown in this picture, I have 5 messages available, but I can only receive 2 messages.)
I tried to delete one of two received messages and poll again. The deleted one is gone, and I received a new message. But in total, it's still 2 messages.
Using an Amazon SQS FIFO queue means that you want to receive messages in order. It will also try to ensure ordering within a Message Group.
This means that, if some messages for a given Message Group ID are currently being processed ("in flight"), no more messages for that Message Group will be provided since an earlier message might be returned to the queue if not fully processed. This could result in messages being processed out-of-order.
From Using the Amazon SQS message group ID - Amazon Simple Queue Service:
To interleave multiple ordered message groups within a single FIFO queue, use message group ID values (for example, session data for multiple users). In this scenario, multiple consumers can process the queue, but the session data of each user is processed in a FIFO manner.
When messages that belong to a particular message group ID are invisible, no other consumer can process messages with the same message group ID.
Therefore, your choices are:
Don't uses a FIFO queue, or
Use different Message Group IDs, or
Be happy with what it is doing because that is desired FIFO behaviour
From AWS Docs:
The maximum number of messages to return. Amazon SQS never returns more messages than this value (however, fewer messages might be returned).
Just like doc's write, you can get less messages. You have to call ReceiveMessage multiple times, usually done in a loop. You can also increase WaitTimeSeconds so that the ReceiveMessage does not return immedietly if there are no messages.
Related
We have provisioned a single AWS SQS FIFO queue. There is a single process that adds items to this queue. All items added have the same group message id.
We start two independent identical processes, Consumer A and Consumer B. The only thing the consumers do is pull items off the queue and throw away the results. Assume that there are no network or service interruptions on either the AWS end or at our end.
I have looked carefully through AWS's documentation and cannot find an answer to this question: When can two AWS SQS FIFO queue consumers process messages with the same group message id?
I did see this text:
When messages that belong to a particular message group ID are invisible, no other consumer can process messages with the same message group ID.
Does the above apply only when all messages with the same message group ID are invisible? Or is it enough for just some of the messages for a given group message id be invisible.
For example, imagine that when the above two consumers start there are already 10,000 messages on the queue all with the same group message id. Since a consumer can be sent a maximum of 10 messages at once, does this mean that while consumer A is processing 10 messages consumer B can't get any messages?
I am looking for a reference in the AWS documentation that clarifies this, or perhaps, someone has done an experiment that decides this one way or the other.
If any single message of a group is invisible (in-flight / currently being processed) no other consumer can process / receive any message(s).
That is the only way to ensure in-order / FIFO processing because if the processing of the currently invisible message fails it needs to be reprocessed and it must not happen that another message within the same group was already processed in the meantime.
Don't have a doc for that because in my mind this is obvious and the only logical thing to do. Anything else will break the FIFO principle.
And to answer "When can two AWS SQS FIFO queue consumers process messages with the same group message id?" - never. Unless the visibility timeout of one message expires, it becomes visible again and the second consumer picks it up. But at that point from the perspective of the queue only the second consumer is working on the message, the first does not count anymore.
I created a test SQS FIFO queue test.fifo - see screenshot below.
Then:
I sent three messages to it with message-group-id = A, and message bodies A1, A2 and A3 respectively using AWS Console (via Send and receive messages button).
Using AWS Console again, I polled for messages with 10 seconds polling, two times in a row. Every time, I saw all three messages in results, and I could open and see the message bodies for all.
Without deleting any message after receiving, how was I able to see all messages with same message-group-id at once? Isn't that a violation of FIFO nature of the queue?
.
Isn't that a violation of FIFO nature of the queue?
No, of course not.
FIFO's main guarantee is around ordering, which would have been preserved in the console and you would have seen the messages in the same order you sent them.
It guarantees exactly-once processing, not exactly-one-message-being-received-at-a-time.
You can receive multiple messages at once, as mentioned in the FIFO docs:
It is possible to receive up to 10 messages in a single call using the MaxNumberOfMessages request parameter of the ReceiveMessage action. These messages retain their FIFO order and can have the same message group ID. Thus, if there are fewer than 10 messages available with the same message group ID, you might receive messages from another message group ID, in the same batch of 10 messages, but still in FIFO order.
I have a process which reads a message from SQS and process it. Each message has a group_message_key, and the processing of each message is relatively fast, but if I read a message with a different group_message_key, there is an extra processing time. Therefore I try to group the messages in order to avoid having these context switches.
I would like to have the option to initially read from a general queue, where all the messages are queued, and only after i have read the first message, query the message queue to deliver only messages with the specific group_message_key.
I am currently using Amazon SQS, but I don't mind at all changing to other message broker which can provide the feature I am missing (ie: Rabbit, Kafka).
I read from the same queue in parallel from many different processes so the solution would still need to support this. Amazon SQS has a FIFO queue which although it doesn't allow requesting messages with a specific group_message_key, it does tries to group these messages together, the problem is that it doesn't allow many workers to process the same message_group_id
I presume that you are referring to the Amazon SQS Message Group ID.
This Message Group ID is unique to FIFO queues and ensures that messages with the same Message Group ID are always processed in-order.
To provide an example, imagine a number of buses sending their GPS coordinates to an Amazon SQS queue. A queue consumer wishes to retrieve these coordinates from the queue and plot the path of each bus on a map. It is important to always retrieve messages from a specific bus in the order that the messages were sent, but messages from multiple buses can also be processed.
This need can be accomplished by having each bus send a unique ID in the Message Group ID when sending its coordinates. When a consumer pulls a message from the queue, no further messages with the same Message Group ID will be provided to a queue consumer until this particular message has been processed. This ensures that the messages from a given bus are never processed out-of-order.
However, it is not possible to request messages with a specific Message Group ID. If your request to receive message from the queue indicates that you are willing to receive a Batch of messages, then multiple messages with the same Message Group ID might be provided.
I have a FIFO SQS queue, and I am sending 4 messages one after another.
After sending first 4 messages, I ran a program to receive them, but only 2 messages were returned - even with long polling and the max messages = 10.
I've sent 4 more messages - now I had 8 messages total. Both the AWS SQS UI for receiving messages, and my code for receiving messages - showed only 2 messages, but said that 8 messages were available.
After sending 4 more messages, you can see in the attached screenshot that SQS UI shows 12 messages available, but lists only 2 messages, and I have C# code to receive messages, with long-polling, that also returns only 2 messages.
What am I doing wrong that I can see all available messages?
You need to delete the message with the ReceiptHandle to get the other messages. FIFO queues guarantee ordering inside a message group and a message is not delivered until previous messages are processed. You are seeing only some messages but not others because they are the first in the queue for their message groups.
I've encountered it a while back and blogged about how it works.
You have select to use a First-In First-Out (FIFO) queue.
To ensure that messages are processed in the correct order, it is not possible to retrieve messages with the same Message Group ID if there are currently messages with that ID being processed. Otherwise, there is a risk that a message might be processed out-of-order.
This is why your request for more messages is not providing back any messages.
If you simply want the messages to be in FIFO order without specific ordering within message groups, it is recommended that you provide a random Message Group ID so that each message can be processed individually.
See: Amazon SQS FIFO (First-In-First-Out) queues - Amazon Simple Queue Service
I want to get all the messages in the queue to process them. However the property for MaxNumberOfMessages is 10 (based on documentation)
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html
How can I read in all messages so I can process them? Or how would I know when queue is empty?
thanks
When you receive messages from the queue, they are marked as "in flight." After you successfully process them, you send a call to the queue to delete them. This call will include IDs of each of the messages.
When the queue is empty, the next read will have an empty Messages array.
Usually when I do this I wrap my call to read the queue in a loop (a while loop) and only keep processing if I have Messages after doing a read.
It shouldn't make any difference if it's a FIFO queue or a standard one.
To check if the queue is empty you have to verify the total number of messages in the queue is zero. SQS does not provide a single metric for this, rather you have to calculate the sum of three different metrics.
From the docs:
To confirm that a queue is empty (AWS CLI, AWS API)
Stop all producers from sending messages.
Repeatedly run one of the following commands:
AWS CLI: get-queue-attributes
AWS API: GetQueueAttributes
Observe the metrics for the following attributes:
ApproximateNumberOfMessagesDelayed
ApproximateNumberOfMessagesNotVisible
ApproximateNumberOfMessages
When all of them are 0 for several minutes, the queue is empty.
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/confirm-queue-is-empty.html
Getting an empty response from a ReceiveMessage call does NOT necessarily mean the queue is empty. You can have messages in the queue and still receive an empty response if:
Messages are delayed - You can set delays on individual messages for standard queues or at the queue level for standard and FIFO queues. During the delay period messages are invisible to consumers.
Messages are in-flight - When a consumer receives a message, that message remains in the queue until the consumer deletes it by calling DeleteMessage. While the message is in this state it is considered in-flight and is not available for other consumers.
Multiple messages have the same message group id in a FIFO queue - When a consumer receives a message from a FIFO queue, no other consumer can receive messages from the same message group. This ensures messages are processed in FIFO order.
By summing the metrics listed above, you can account for all of these scenarios.