The ReceiveMessage call can return up to 10 messages from a queue (http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html). I am using long-polling.
I will be looping through the list of received messages and processing them in a single thread. Does this mean I should set the VisibilityTimeout for the queue to be (expected time to process single message) * 10.
Or, does SQS take care of this for me automagically? If so, I would be curious to learn about this as well, since I have not been able to find any clarification in the docs.
Does this mean I should set the VisibilityTimeout for the queue to be
(expected time to process single message) * 10.
Yes
Or, does SQS take care of this for me automagically?
No. How would that be possible? Once you've received the 10 messages SQS has no visibility into what you are doing with them. It doesn't know what constitutes the "processing" of each message in your application. All SQS knows is that it handed 10 messages to your application.
Alternatively, set the MaxNumberOfMessages to 1 so that your application only receives one message at a time from SQS.
Related
Apologies for the title. It's hard to summarise what I'm trying to accomplish.
Basically let me define a service to be an SQS Queue + A Lambda function.
A service (represented by square brackets below) performs a given task, where the queue is the input interface, processes the input, and outputs on to the queue of the subsequent service.
Service 1 Service 2 Service 3
[(APIG) -> (Lambda)] -> [(SQS) -> (Lambda)] -> [(SQS) -> (Lambda)] -> ...
Service 1: Consumes the request and payload, splits it into messages and passes on to the queue of the next service.
Service 2: This service does not have a reserved concurrency. It validates each message on the queue, and if valid, passes on to the next service.
Service 3: Processes each message in the queue (ideally in batches of approximately 100). The lambda here must have a reserved concurrency of 1 (as it hits an API that can't process multiple requests concurrently).
Currently I have the following configuration on Service 3.
Default visibility timeout of queue = 5 minutes
Lambda timeout = 5 minutes
Lambda reserved concurrency = 1
Problem 1: Service 3 consumes x items off the queue and if it finishes processing them within 30 seconds I expect the queue to process the next x items off the queue immediately (ideally x=100). Instead, it seems to always wait 5 minutes before taking the next batch of messages off the queue, even if the lambda completes in 30 seconds.
Problem 2: Service 3 typically consumes a few messages at a time (inconsistent) rather than batches of 100.
A couple of more notes:
In service 3 I do not explicitly delete messages off the queue using the lambda. AWS seems to do this itself when the lambda successfully finishes processing the messages
In service 2 I have one item per message. And so when I send messages to Service 3 I can only send 10 items at a time, which is kind of annoying. Because queue.send_messages(Entries=x), len(x) cannot exceed 10.
Does anyone know how I solve Problem 1 and 2? Is it an issue with my configuration? If you require any further information please ask in comments.
Thanks
Both your problems and notes indicate misconfigured SQS and/or Lambda function.
In service 3 I do not explicitly delete messages off the queue using
the lambda. AWS seems to do this itself when the lambda successfully
finishes processing the messages.
This is definitely not the case here as it would go agains the reliability of SQS. How would SQS know that the message was successfully processed by your Lambda function? SQS doesn't care about consumers and doesn't really communicate with them and that is exactly the reason why there is a thing such as visibility timeout. SQS deletes message in two cases, either it receives DeleteMessage API call specifying which message to be deleted via ReceiptHandle or you have set up redrive policy with maximum receive count set to 1. In such case, SQS will automatically send message to dead letter queue when if it receives it more than 1 time which means that every message that was returned to the queue will be send there instead of staying in the queue. Last thing that can cause this is a low value of Message Retention Period (min 60 seconds) which will drop the message after x seconds.
Problem 1: Service 3 consumes x items off the queue and if it finishes
processing them within 30 seconds I expect the queue to process the
next x items off the queue immediately (ideally x=100). Instead, it
seems to always wait 5 minutes before taking the next batch of
messages off the queue, even if the lambda completes in 30 seconds.
This simply doesn't happen if everything is working as it should. If the lambda function finishes in 30 seconds, if there is reserved concurrency for the function and if there are messages in the queue then it will start processing the message right away.
The only thing that could cause is that your lambda (together with concurrency limit) is timing out which would explain those 5 minutes. Make sure that it really finishes in 30 seconds, you can monitor this via CloudWatch. The fact that the message has been successfully processed doesn't necessarily mean that the function has returned. Also make sure that there are messages to be processed when the function ends.
Problem 2: Service 3 typically consumes a few messages at a time
(inconsistent) rather than batches of 100.
It can never consume 100 messages since the limit is 10 (messages in the sense of SQS message not the actual data that is stored within the message which can be anywhere up to 256 KB, possibly "more" using extended SQS library or similar custom solution). Moreover, there is no guarantee that the Lambda will receive 10 messages in each batch. It depends on the Receive Message Wait Time setting. If you are using short polling (1 second) then only subset of servers which are storing the messages will be polled and a single message is stored only on a subset of those servers. If those two subsets do not match when the message is polled, the message is not received in that batch. You can control this by increasing polling interval, Receive Message Wait Time, (max 20 seconds) but even if there are not enough messages in the queue when the timer finishes, the batch will still be received with fewer messages, possibly zero.
And as it was mentioned in the comments, using this strategy with concurrency set to low number can lead to some problems. Another thing is that you need to ensure that rate at which messages are produced is somehow consistent with the time it takes for one instance of lambda function to process the message otherwise you will end up with constantly growing queue, possibly losing messages after they outlive the Message Retention Period.
I would like to publish a message on SQS and process that message after a few hours.
How can i schedule a message delivery or select messages from SQS based on some attribute?
I've implemented a SQS consumer but I'm receiving every message from SQS queue. Is possible to implement something like that on SQS? I was thinking about to receive every message and send to queue again if it's not time to process that message.
There is a feature called as Delay Queues in SQS, wherein if you set the delay on the queue then any message that is put on queue is available to consumers only after the delay duration has elapsed. However, the maximum delay that you can set there is 15 minutes and if you are looking for a delay of few hours this may not directly work for you.
The other option is to set a visibilty timeout for the messages higher than the delay time that you want. Then when you read the message you can get the message timestamp. If there is still some time left for your delay then you can sleep your consumer for the remaining time and after it has woken up you can process that message. However this is not a recommended way and would be highly inefficient because your threads are getting blocked. In fact what can as well be done is if there is still some time left for your delay then you just hold the message in a local List/Array and check for other messages and process this message after your delay. But all this would require entire logic to reside in your code and you don't get any ready-made feature from AWS
I have seen an option called visibility timeout in AWS SQS which sets a time for other consumers to ignore the message which is being processed by one.
Is there an option to set a time before the message actually becomes available for consumers to use it, may be set while message actually is inserted into Queue
There is such an option, but it is a queue-level option, not a message-level option:
You can use the CreateQueue action to create a delay queue by setting the DelaySeconds attribute to any value between 0 and 900 (15 minutes). You can also change an existing queue into a delay queue using the SetQueueAttributes action to set the queue's DelaySeconds attribute
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-delay-queues.html
When the value is nonzero, all messages are delayed by the specified number of seconds before they are initially visible to any consumer.
Found an option called Amazon SQS Message Timers
Amazon SQS message timers allow you to specify an initial invisibility period for a message that you add to a queue. For example, if you send a message with the DelaySeconds parameter set to 45, the message isn't visible to consumers for the first 45 seconds during which the message stays in the queue.
The default value for DelaySeconds is 0.
To set a delay period that applies to all messages in a queue, use delay queues. A message timer setting for an individual message overrides any DelaySeconds value that applies to the entire delay queue.
For more info on SQS check this detailed PDF by AWS
When you are processing the message, you can actually use ChangeMessageVisibility() . This allows you to change the timeout period while holding the message either up or down. For example, if you KNOW that your message failed the first time, you can set it to 0 and it will put it back on the queue. If you wanted a specifically longer timeout, you could check for that type of message and set it to 10 minutes for example.
I set visibility time out 12 hours and max message 3, delay time 15 min, I'm get sqs message few minute after automatically I get same message again.
Why do I get multiple sqs message without timeout?
After visibility time out it delete message in queue or send again sqs message?
When ReceiveMessage() is called on an Amazon SQS queue, up to 10 messages (configurable) will be retrieved from the queue.
These messages will be marked as Invisible or In-Flight. This means that the messages are still in the queue, but will not be returned via another ReceiveMessage() call. The messages will remain invisible for a period of time. The default period is configured on the queue ("Default Visibility Timeout") or when the messages are retrieved (VisibilityTimeout).
When an application has finished handling a message, it should call DeleteMessage(), passing the MessageHandle that was provided with the message. The message will then be deleted from the queue.
If the invisibility period expires before a message is deleted, it will be placed on the queue again and applications can retrieve it again. Therefore, be sure to set your invisibility timeout to be longer than an application normally takes to process a message.
It is possible that a message may be retrieved more than once from Amazon SQS. It is rare, but can happen where there are multiple processes retrieving messages simultaneously. Thus, SQS is "At least once delivery". If this is a problem, you can use FIFO Queues (not yet available in every region) that will guarantee that each message is delivered only once, but there are throughput restrictions on FIFO queues.
So, if you are receiving a message more than once:
You should check your invisibility timeout setting (both the default setting and the value that can be passed when you call ReceiveMessage())
Consider using FIFO queues
Have your application check whether a message has already been processed before processing it again (eg via a unique ID)
What is the meaning of "Visibility Timeout" for Amazon SQS service ? What factors determine an ideal value for this field ?
I have looked at http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/Welcome.html
When using sqs as queuing service, when you read the message off the queue it does not automatically delete the message off the queue.
So when you are processing the message, sqs will wait for the time period defined as visibility timeout before other consumers will get the same message again.
The best time value to set for the visibility timeout will be at least the timeout value for the consumer process. If the consumer is able to successfully complete the processing then it would delete the message off the queue else if it times out then the message reappears in the queue for other consumer to pick it again.
Visibility timeout is the time-period or duration you specify for the queue item which when is fetched and processed by the consumer is made hidden from the queue and other consumers.
The main purpose is to avoid multiple consumers (or the same consumer), consuming the same item repetitively.
The key factor to be considered while arriving at this value is the time & effort taken by the consumer(s) to process a single queue item.
Basically the time taken by the consumer to process the message. And during the same time the message is unvailable for any other consumer (since this is a distributed system). Although the time priod is configurable, the default visibility timeout for a message is 30 seconds. The minimum is 0 seconds. The maximum is 12 hours.
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
Visibility Timeout is an important functionality of Amazon SQS. It helps to ensure the integrity and reliability of message processing in distributed systems.
You can know more about it here https://link.medium.com/u3A8aId0Swb