I want to distinguish the messages after retreiving from aws fifo sqs for routing to correct processor. And I planned to utilize the MessageGroupId , which I set while sending the request.
I am using Java sdk.
sendMessageRequest.setMessageGroupId(messageGroupId);
Documentation says it can be retrieved as request parameter of AttributeName
MessageGroupId - Returns the value provided by the sender that calls the SendMessage action. Messages with the same MessageGroupId are returned in sequence.
However, I could not find the correct way/method to do so. There is no attribute
MessageGroupId present and size is 0.
List<Message> messages = sqs.receiveMessage(receiveMessageRequest).getMessages();
for (Message message : messages)
{
System.out.println(" Size getAttributes: " + message.getAttributes() .size());
}
Only if I set any attribute explicitly then it is retreivable (like with MESSAGEPRIORITY), but how to receive the values of attributes like MessageGroupId?
Help is appreciated.
You have to tell SQS which attributes you want it to give you, before getAttributes() will be able to show them to you.
List<Message> messages = sqs.receiveMessage(receiveMessageRequest.withAttributeNames("All")).getMessages();
Note that SQS messages have two different kinds of attributes. Attributes are system-generated (like MessageGroupId), while Message Attributes are user-generated, custom key/value pairs included when the message is sent but transported external to the message body itself.
Related
Regarding AWS SQS dead letter queues, if two messages have the exact same string, will the second message be considered a reattempt of the first message?
So like if I put retry attempts as 2, would the second message count as the retry event?
Amazon SQS does not look at the 'content' of messages.
Message retry counts would be based on the message's 'metadata' itself (with a unique ID supplied by AWS) rather than the content of the message.
I'm learning AWS SQS and I've sent 6 messages to a FIFO queue, with the same GroupId. But when I try to poll for messages, I can only receive 2 of them (Why? I set the MaxNumberOfMessages=10 using boto3 API, but I can only receive 2. How can I receive all of the messages?).
(As shown in this picture, I have 5 messages available, but I can only receive 2 messages.)
I tried to delete one of two received messages and poll again. The deleted one is gone, and I received a new message. But in total, it's still 2 messages.
Using an Amazon SQS FIFO queue means that you want to receive messages in order. It will also try to ensure ordering within a Message Group.
This means that, if some messages for a given Message Group ID are currently being processed ("in flight"), no more messages for that Message Group will be provided since an earlier message might be returned to the queue if not fully processed. This could result in messages being processed out-of-order.
From Using the Amazon SQS message group ID - Amazon Simple Queue Service:
To interleave multiple ordered message groups within a single FIFO queue, use message group ID values (for example, session data for multiple users). In this scenario, multiple consumers can process the queue, but the session data of each user is processed in a FIFO manner.
When messages that belong to a particular message group ID are invisible, no other consumer can process messages with the same message group ID.
Therefore, your choices are:
Don't uses a FIFO queue, or
Use different Message Group IDs, or
Be happy with what it is doing because that is desired FIFO behaviour
From AWS Docs:
The maximum number of messages to return. Amazon SQS never returns more messages than this value (however, fewer messages might be returned).
Just like doc's write, you can get less messages. You have to call ReceiveMessage multiple times, usually done in a loop. You can also increase WaitTimeSeconds so that the ReceiveMessage does not return immedietly if there are no messages.
I am using AWS SQS and Spring JMS in my project. I have my method with #JmsListener(destination = "queue_name"). I want to use this queue for two different types of messages.
Since this listener is configured to this queue it receives both types of messages. What I am trying to achieve is to ignore message of one type. (Sender is adding a MessageAttribute while sending message to Queue). So, is there a way to just ignore message coming from sender 2 so this method won't process them.
Also, I have DLQ set on this queue with max receives as 5. So if message is not processed in first 5 attempts it gets moved to DLQ.
Please do share your suggestion.
Thanks.
The correct solution is to use 2 different queues; SQS can't filter the messages delivered by any property, so as you are seeing, when the client reads the message and doesn't process it, it is going to end up in your DLQ quicker.
Queues are free, so having multiple won't cost any more.
What is the purpose of using message body in SQS while you're already able to add message attributes?
Let's take an example, we want to push a message to new-user queue when a new user registered, I imagine the message will have an attribute userId, I don't see the use of body here.
Message attributes are supposed to be used as message metadata (like timestamp or possibly some category) and not the message itself.
Ideally, message payload should be given in the message body
So, for example if you are supporting JSON and XML payloads then possibly you can put payload type as message attribute and then when you fetch the message, based on this payload type attribute you decide between the JSON message processor or XML message processor. This is just a superficial example to explain the usage of attributes and body
Following is the extract from AWS Doc
Amazon SQS provides support for message attributes. Message attributes allow you to provide structured metadata items (such as timestamps, geospatial data, signatures, and identifiers) about the message. Message attributes are optional and separate from, but sent along with, the message body. This information can be used by the receiver of the message to help decide how to handle the message without having to first process the message body. Each message can have up to 10 attributes. To specify message attributes, you can use the AWS Management Console, AWS software development kits (SDKs), or query API.
To map with the traditional queue provider such as rabbitMQ or Kafka world.
We can understand as below:
message_body=message_payload
message_attributes=message_headers ( can be used to apply different routing and filtering message using their headers information)
In fact, I prefer the term payload and headers more than what terms used in aws sqs, its abit confusing.
Message attributes sound more like the attributes of message payload
I am wondering how does Amazon SQS handle the SentTimestamp while sending a message to a queue?
If the SentTimestamp attribute is not specified in the request, does Amazon take their system timestamp when it receives the message in the sqs to be the SentTimestamp value?
From my experiments, even if I specify the SentiTimestamp attribute in the request, I do see Amazon does not take the value as message SentTimestamp.
If my local system clock is out of synch with NTP server, does it give any influence on the SentTimestamp attribute?
Can someone please help clarify these questions?
Thanks.
The SentTimestamp attribute is set by SQS and is read-only, you cannot override the SentTimestamp attribute with some other value.
Note that the SQS documentation only mentions SentTimestamp in the ReceiveMessage section. SentTimestamp comes from the Attributes field, which is distinct from the MessageAttributes field. While you can specify MessageAttributes when sending a message, you cannot specify Attributes when sending. Therefore, the Attributes field must be populated by SQS.
If the sender's clock is wrong, it will have no effect on the SentTimestamp value.
However, if the receiver's clock is wrong you may get some incorrect results when trying to process the SentTimestamp value.
Source: SQS API Documentation.