I read the SQS Doc, it stated that A single message batch request can include a maximum of 10 messages
Do this apply to both send and receive? Because I was planning to receive 200 message at one time.
According to this: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-batch-api-actions.html
There are only following batch requests:
SendMessageBatch
DeleteMessageBatch
ChangeMessageVisibilityBatch
Single ReceiveMessage action is limited up to 10.
For receive 200 messages - you need to poll queue multiple times until you read them all.
Related
I have a FIFO SQS queue, and I am sending 4 messages one after another.
After sending first 4 messages, I ran a program to receive them, but only 2 messages were returned - even with long polling and the max messages = 10.
I've sent 4 more messages - now I had 8 messages total. Both the AWS SQS UI for receiving messages, and my code for receiving messages - showed only 2 messages, but said that 8 messages were available.
After sending 4 more messages, you can see in the attached screenshot that SQS UI shows 12 messages available, but lists only 2 messages, and I have C# code to receive messages, with long-polling, that also returns only 2 messages.
What am I doing wrong that I can see all available messages?
You need to delete the message with the ReceiptHandle to get the other messages. FIFO queues guarantee ordering inside a message group and a message is not delivered until previous messages are processed. You are seeing only some messages but not others because they are the first in the queue for their message groups.
I've encountered it a while back and blogged about how it works.
You have select to use a First-In First-Out (FIFO) queue.
To ensure that messages are processed in the correct order, it is not possible to retrieve messages with the same Message Group ID if there are currently messages with that ID being processed. Otherwise, there is a risk that a message might be processed out-of-order.
This is why your request for more messages is not providing back any messages.
If you simply want the messages to be in FIFO order without specific ordering within message groups, it is recommended that you provide a random Message Group ID so that each message can be processed individually.
See: Amazon SQS FIFO (First-In-First-Out) queues - Amazon Simple Queue Service
I am trying to find a document of how requests will be processed if there is some network error between request from Client -> AWS SQS?
Because if one request can contain up to 10 messages, how would it work if there is an http error on the request while the 10 messages are being processed to SQS? Does SQS fail the whole request, and all 10 messages are being re-published? OR some messages get processed, and we will reprocess the whole thing again (which will have some duplication)?
I am trying to determine if I can cut cost with aggregating messages into 1 single request, but wonder if this will hurt idempotency.
So when you're sending messages as batch you'll be processing it one by one so each message will have its unique receipthandle so you've delete each message programmatically after processing it to avoid duplication
The ReceiveMessage call can return up to 10 messages from a queue (http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html). I am using long-polling.
I will be looping through the list of received messages and processing them in a single thread. Does this mean I should set the VisibilityTimeout for the queue to be (expected time to process single message) * 10.
Or, does SQS take care of this for me automagically? If so, I would be curious to learn about this as well, since I have not been able to find any clarification in the docs.
Does this mean I should set the VisibilityTimeout for the queue to be
(expected time to process single message) * 10.
Yes
Or, does SQS take care of this for me automagically?
No. How would that be possible? Once you've received the 10 messages SQS has no visibility into what you are doing with them. It doesn't know what constitutes the "processing" of each message in your application. All SQS knows is that it handed 10 messages to your application.
Alternatively, set the MaxNumberOfMessages to 1 so that your application only receives one message at a time from SQS.
For example, a message is sent to a queue at 2016-05-19 23:29:05. And it being received by a consumer at 2016-05-19 23:30:05. After processed, the message is deleted at 2016-05-19 23:30:08. That means it takes 60 seconds from message sent to queue to consumer received it, and it takes 63 seconds from the message born to dead.
This metrics is helpful to monitor how fast consumer can consume messages.
To get lifetime of a message, I have to manually add timestamp attribute to each message when it is sent to queue. Consumer will retrieve the timestamp and calculate the time interval.
Does SQS provide similar API to get lifetime of a message (I checked SQS API and with no luck), or is there any other good idea?
There are attributes on an SQS message which will allow you to measure the time spent by a message in the queue.
Some of the attributes you can use are
SentTimestamp, CreatedTimestamp, LastModifiedTimestamp etc
I was looking at the javascript SDK, and it seems like some attributes are only returned when explicitly requested for
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SQS.html#receiveMessage-property
For the list of all attributes available within an SQS message please refer to http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html
I have large number of messages in AWS SQS Queue. These messages will be pushed to it constantly by other source. There are no proper dynamic on how often those messages will be pushed to queue. Currently, I keep polling SQS every second and checking if there are any messages available in there. Is there any better way of handling this, like receiving notification from SQS or SNS that some messages are available so that I only request SQS when I needed instead of constant polling?
The way to do what you want is to use long polling - rather than constantly poll every second, you open a request that stays open until it either times out or a message comes into the queue. Take a look at the documentation for ReceiveMessageRequest
ReceiveMessageRequest req = new ReceiveMessageRequest()
.withWaitTimeSeconds(Integer.valueOf(20)); // set long poll timeout to 20 sec
// set other properties on the request as well
ReceiveMessageResult result = amazonSQS.receiveMessage(req);
A common usage pattern for this is to have a background thread running the long poll and pushing the results into an internal queue (such as LinkedBlockingQueue or an ExecutorService) for a worker thread to read from.
PS. Don't forget to call deleteMessage once you're done processing the result so you don't end up receiving it again.
You can also use the worker functionality in AWS Elastic Beanstalk. It allows you to build a worker to process each message, and when you use Elastic Beanstalk to deploy it to an EC2 instance, you can define it as subscribed to a specific queue. Then each message will be POST to the worker, without your need to call receive-message on it from the queue.
It makes your system wiring much easier, as you can also have auto scaling rules that will allow you to spawn multiple workers to handle more messages in time of peak load, and scale down back to a single worker, when the load is low. It will also delete the message automatically, if you respond with OK from your worker.
See more information about it here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html
You could also have a look at Shoryuken and the property delay:
delay: 25 # The delay in seconds to pause a queue when it's empty
But being honest we use delay: 0 here, the cost of SQS is inexpensive:
First 1 million Amazon SQS Requests per month are free
$0.50 per 1 million Amazon SQS Requests per month thereafter ($0.00000050 per SQS Request)
A single request can have from 1 to 10 messages, up to a maximum total payload of 256KB.
Each 64KB ‘chunk’ of payload is billed as 1 request. For example, a single API call with a 256KB payload will be billed as four requests.
You will probably spend less than 10 dollars monthly polling messages every second 24x7 in a single host.
One of the advantages of Shoryuken is that it fetches in batch, so it saves some money compared with a fetch per message solutions.