I am trying to find a document of how requests will be processed if there is some network error between request from Client -> AWS SQS?
Because if one request can contain up to 10 messages, how would it work if there is an http error on the request while the 10 messages are being processed to SQS? Does SQS fail the whole request, and all 10 messages are being re-published? OR some messages get processed, and we will reprocess the whole thing again (which will have some duplication)?
I am trying to determine if I can cut cost with aggregating messages into 1 single request, but wonder if this will hurt idempotency.
So when you're sending messages as batch you'll be processing it one by one so each message will have its unique receipthandle so you've delete each message programmatically after processing it to avoid duplication
Related
I have a pretty standard setup of feeding SQS to Lambda. The lambda reads the message and makes a web request to a defined endpoint.
If I encounter an exception during processing of the SQS message that is due to the form of the message then I put the message on a dead letter queue.
If I encounter an error with the web request, I put the message back on the feeding queue to make the HTTP request at a later time.
This seems to work fine, but we just ran into an issue where an HTTP endpoint was down for 4 days and the feeding queue dropped the message. I imagine this has something to do with the retention period setting of the queue.
Questions
Is there a way to know, in the lambda, how many times a message has been replayed?
How did the feeder queue know that the message that was re-enqueued was the same as the one that was originally put on the queue?
I'm currently not explicitly deleting a message off the queue. Not having that, hasn't seemed to cause any issues, no re-processing of messages or anything. Should I be explicitly deleting them?
The normal process would be:
The AWS Lambda function is triggered, with the message(s) passed via the event parameter
If the Lambda function successfully processes the message(s), it should return a 'success' code (200) and the message is automatically removed from the queue
If the Lambda function is unable to process the message, it should return a 'failure' code (eg 400) and Amazon SQS will automatically attempt to re-process the message (unless it has exceeded the retry count)
If the Lambda function fails (eg due to a timeout), Amazon SQS will automatically attempt to re-process the message (unless it has exceeded the retry count)
If a message has exceeded its retry count, Amazon SQS will move the message to the Dead Letter Queue
To answer your questions:
If you wish to take responsibility for these activities yourself, you can use the ApproximateReceiveCount attribute on the message. In the request, it appears that you should add AttributeNames=['ApproximateReceiveCount'], but the documentation is a bit contradictory. You might need to use All instead.
Since you are sending a new message to the queue, Amazon SQS is not aware that it is the same message. The message is not 're-enqueued' since it is a new message.
When your Lambda function returns 'success' (200), the message is being deleted off the queue for you.
You might consider using the standard functionality for retries and Dead Letter Queues rather than implementing that logic yourself.
I have a process which reads a message from SQS and process it. Each message has a group_message_key, and the processing of each message is relatively fast, but if I read a message with a different group_message_key, there is an extra processing time. Therefore I try to group the messages in order to avoid having these context switches.
I would like to have the option to initially read from a general queue, where all the messages are queued, and only after i have read the first message, query the message queue to deliver only messages with the specific group_message_key.
I am currently using Amazon SQS, but I don't mind at all changing to other message broker which can provide the feature I am missing (ie: Rabbit, Kafka).
I read from the same queue in parallel from many different processes so the solution would still need to support this. Amazon SQS has a FIFO queue which although it doesn't allow requesting messages with a specific group_message_key, it does tries to group these messages together, the problem is that it doesn't allow many workers to process the same message_group_id
I presume that you are referring to the Amazon SQS Message Group ID.
This Message Group ID is unique to FIFO queues and ensures that messages with the same Message Group ID are always processed in-order.
To provide an example, imagine a number of buses sending their GPS coordinates to an Amazon SQS queue. A queue consumer wishes to retrieve these coordinates from the queue and plot the path of each bus on a map. It is important to always retrieve messages from a specific bus in the order that the messages were sent, but messages from multiple buses can also be processed.
This need can be accomplished by having each bus send a unique ID in the Message Group ID when sending its coordinates. When a consumer pulls a message from the queue, no further messages with the same Message Group ID will be provided to a queue consumer until this particular message has been processed. This ensures that the messages from a given bus are never processed out-of-order.
However, it is not possible to request messages with a specific Message Group ID. If your request to receive message from the queue indicates that you are willing to receive a Batch of messages, then multiple messages with the same Message Group ID might be provided.
I have a FIFO SQS queue, and I am sending 4 messages one after another.
After sending first 4 messages, I ran a program to receive them, but only 2 messages were returned - even with long polling and the max messages = 10.
I've sent 4 more messages - now I had 8 messages total. Both the AWS SQS UI for receiving messages, and my code for receiving messages - showed only 2 messages, but said that 8 messages were available.
After sending 4 more messages, you can see in the attached screenshot that SQS UI shows 12 messages available, but lists only 2 messages, and I have C# code to receive messages, with long-polling, that also returns only 2 messages.
What am I doing wrong that I can see all available messages?
You need to delete the message with the ReceiptHandle to get the other messages. FIFO queues guarantee ordering inside a message group and a message is not delivered until previous messages are processed. You are seeing only some messages but not others because they are the first in the queue for their message groups.
I've encountered it a while back and blogged about how it works.
You have select to use a First-In First-Out (FIFO) queue.
To ensure that messages are processed in the correct order, it is not possible to retrieve messages with the same Message Group ID if there are currently messages with that ID being processed. Otherwise, there is a risk that a message might be processed out-of-order.
This is why your request for more messages is not providing back any messages.
If you simply want the messages to be in FIFO order without specific ordering within message groups, it is recommended that you provide a random Message Group ID so that each message can be processed individually.
See: Amazon SQS FIFO (First-In-First-Out) queues - Amazon Simple Queue Service
I read the SQS Doc, it stated that A single message batch request can include a maximum of 10 messages
Do this apply to both send and receive? Because I was planning to receive 200 message at one time.
According to this: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-batch-api-actions.html
There are only following batch requests:
SendMessageBatch
DeleteMessageBatch
ChangeMessageVisibilityBatch
Single ReceiveMessage action is limited up to 10.
For receive 200 messages - you need to poll queue multiple times until you read them all.
I've set up an S3 bucket to emit an event on PUT object to SQS, and I'm handling the SQS queue in an EB worker tier.
The schema for the message that SQS sends is here: http://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html
Records is an array, implying that there can be multiple records sent in one POST to my worker's endpoint. Does this actually happen? Or will my worker only ever receive one record per message?
The worker can only return one response, either 200 (message handled successfully) or non-200 (message not handled successfully, which puts it back into the queue), regardless of how many records in the message it receives.
So if my worker receives multiple records in a message, and it handles some successfully (say by doing something with side effects such as inserting into a database) but fails on one or more, how should I handle that? If I return 200, then the ones that failed will not be retried. But if I return non-200, then the ones that were handled successfully will be retried unnecessarily, and possibly re-inserted. So I'd have to make my worker smart enough to retry only the failed ones -- which is logic I'd prefer not having to write.
This would be much easier if only one record was ever sent per message. So if that's the case in practice, despite records being an array, I'd really like to know!
To be clear, it's not the records that "SQS sends." It's the records that S3 sends to SQS (or to SNS, or to Lambda).
Currently, all S3 event notifications have a single event per notification message. We might include multiple records as we add new event types in the future. This is also a message format that is shared across other AWS services, and other services can include multiple records.
— https://forums.aws.amazon.com/thread.jspa?messageID=592264򐦈
So, for the moment, it appears there's only one record per message.
But... you are making a mistake if you assume your application need not be prepared to handle repeated or duplicate messages. In any massive and distributed system like SQS it is extremely difficult to absolutely guarantee that this can never happen, however unlikely:
Q: How many times will I receive each message?
Amazon SQS is engineered to provide “at least once” delivery of all messages in its queues. Although most of the time each message will be delivered to your application exactly once, you should design your system so that processing a message more than once does not create any errors or inconsistencies.
— http://aws.amazon.com/sqs/faqs/
Incidentally, in my platform, more than one entry in the records array is considered an error, causing the message to be abandoned and sent to the dead letter queue for review.