Do RabbitMQ queues have a AWS SQS-like - "message visibility timeout" ?
From the AWS SQS documentation :
"The visibility timeout clock starts ticking once Amazon SQS returns the message. During that time, the component processes and deletes the message. But what happens if the component fails before deleting the message? If your system doesn't call DeleteMessage for that message before the visibility timeout expires, the message again becomes visible to the ReceiveMessage calls placed by the components in your system and it will be received again"
Thanks!
I believe you are looking for the RabbitMQ manual acknowledgment feature. This feature allows you get messages from the queue and once you have receive them ack'ed them. If something happens in the middle of this process, the message will be available again in the queue after a certain amount of time. Also, in the meantime since you get the message until you ack it, the message is not available for other consumers to consume.
I think this is the same behavior as Message Visibility Timeout of SQS.
There aren't any message timeouts; RabbitMQ will redeliver the message only when the worker connection dies. It's fine even if processing a message takes a very, very long time.There aren't any message timeouts; RabbitMQ will redeliver the message only when the worker connection dies. It's fine even if processing a message takes a very, very long time.
I believe the answer can be found # a discussion of MQ vs SQS generally this is a considered a feature of MQ (that it can handle slow consumers) but using a destination policy of "slowConsumerStrategy" with "abortSlowConsumerStrategy" might solve your problem. A fuller explanation can be found at redhat's MQ documentation and i supposed we have to hope that rabbitMQ and AmazonMQ both support that strategy.
Related
How do I configure visibility timeout so that a message in SQS can be read again?
I have Amazon SQS as a message queue. Messages are being sent by multiple applications. I am now using Spring listener to read message in queue as below:
public DefaultMessageListenerContainer jmsListenerContainer() {
SQSConnectionFactory sqsConnectionFactory = SQSConnectionFactory.builder()
.withAWSCredentialsProvider(new DefaultAWSCredentialsProviderChain())
.withEndpoint(environment.getProperty("aws_sqs_url"))
.withAWSCredentialsProvider(awsCredentialsProvider)
.withNumberOfMessagesToPrefetch(10).build();
DefaultMessageListenerContainer dmlc = new DefaultMessageListenerContainer();
dmlc.setConnectionFactory(sqsConnectionFactory);
dmlc.setDestinationName(environment.getProperty("aws_sqs_queue"));
dmlc.setMessageListener(queueListener);
return dmlc;
}
The class queueListener implements javax.jms.MessageListener which uses onMessage() method further.
I have also configured a scheduler to read the queue again after a certain period of time. It uses receiveMessage() of com.amazonaws.services.sqs.AmazonSQS.
As soon as message reach the queue the listener reads the message. I want to read the message again after certain period of time i.e. through scheduler, but once a message is read by listener it does not become visible or read again. As per Amazon's SQS developer guide the default visibility timeout is 30 seconds, but that message is not becoming visible even after 30 seconds. I have tried setting custom visibility timeout in SQS QUEUE PARAMETER CONSOLE, but it's not working.
For information, nobody is deleting the message from the queue.
I only have a passing familiarity with Amazon SQS, but I can say that typically in messaging use-cases when a consumer receives and acknowledges the message then that message is removed (i.e. deleted) from the queue. Given that your Spring application is receiving the message I would suspect it is also acknowledging the message and therefore removing it from the queue which prevents your scheduler from receiving it later. Note that Spring's DefaultMessageListenerContainer uses JMS' AUTO_ACKNOWLEDGE mode by default.
This documentation from Amazon essentially states that if a message is acknowledged in a JMS context that it is "deleted from the underlying Amazon SQS queue."
Does the deletion AWS SQS message (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_DeleteMessage.html) remove/acknowledge all messages that was earlier fetched by any of consumer?
I'm asking because I would like to make sure delete operation is not working in similar way to acknowledge described in (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/getting-started.html)
In this mode, when a message is acknowledged, all messages received before this message are implicitly acknowledged as well. For example, if 10 messages are received, and only the 10th message is acknowledged (in the order the messages are received), then all of the previous nine messages are also acknowledged.
The acknowledgements are a feature of JMS. When you work with SQS directly, there are no acknowledgements sent to the producer of the messages.
If you require such acknowledgements you would have to develop request-response system yourself, or use pre-existing solutions for that (such as with JMS).
I have a system where a Lambda is triggered with event source as an SQS Queue.Each message gets our own internal unique id to differentiate between two requests .
Now lambda deletes the message from the queue automatically after sqs invocation and keeps the message in inflight while processing it so duplicate processing of a unique message should never occur ideally.
But when I checked my logs a message with the same unique id was processed within 100 milliseconds of the time frame of each other.
So This seems like two lambdas were triggered for one message and something failed at the end of aws it was either visibility timeout or something else.I have read online that few others have gone through the same situation.
Can anyone who has gone through the same situation explain how did they solve it or people with current scalable systems who don't have this kind of issue can help me out with the reasons why I could be having it ?
Note:- One single message was successfully executed Twice this wasn't the case of retry on failure.
I faced a similar issue, where a lambda (let's call it lambda-1) is triggered through a queue, and lambda-1 further invokes lambda-2 'synchronously' (https://docs.aws.amazon.com/lambda/latest/dg/invocation-sync.html) and the message basically goes to inflight and return back after visibility timeout expiry and triggers lambda-1 again. This goes on in a loop.
As per the link above:
"For functions with a long timeout, your client might be disconnected
during synchronous invocation while it waits for a response. Configure
your HTTP client, SDK, firewall, proxy, or operating system to allow
for long connections with timeout or keep-alive settings."
Making async calls in lambda-1 can resolve this issue. In the case above, invoking lambda-2 with InvocationType='Event' returns back, which in-turn deletes the item from queue.
I was looking for how celery and sqs deletion works?
When does celery delete message from sqs?
Does it delete when message is picked from sqs or after completion of tasks?
What happens if there is some error in task or it raises error?
Will the message (task) will be there if the tasks is taking too long like and 20 mins.
When does celery delete message from sqs?
Message will be deleted after completing the task.
What happens if there is some error in task or it raises error?
Message is still with broker, and deleted after max_retries reaches.
Will the message (task) will be there if the tasks is taking too long like and 20 mins?
This depends on visibility timeout. Message goes to "Not Visible" state, till your visibility timeout, after that it is available to worker.
(if visibility timeout is less than retry time, worker will consume same message many times).
Best Practice is (visibility timeout) > (max_retries * retry_time)
The selected answer is (unfortunately) incorrect for SQS, as this open issue indicates.
There was an attempt at fixing the issue, as evidenced by this merged PR.
However, there are bugs with the above implementation.
Long story short, messages will be deleted from an SQS queue 100% of the time, regardless of any exception that occurs within the task.
edit: this may have been resolved, per this PR
I'll update this answer after I've confirmed via personal testing that this functions correctly
While using AWS SQS, does "SentTimestamp" attribute of a message change after it is received from the queue, but not deleted and returned back to the queue after the visibility timeout expired?
No, and that behavior would be undesirable because SentTimestamp specifically describes when the message was first sent to the queue.
This documentation that describes the visibility timeout offers some insight:
Immediately after the component receives the message, the message is
still in the queue. However, you don't want other components in the
system receiving and processing the message again. Therefore, Amazon
SQS blocks them with a visibility timeout, which is a period of time
during which Amazon SQS prevents other consuming components from
receiving and processing that message.
The important takeaway here is that the message never really leaves the queue, it is simply hidden from other clients that are receiving messages. So message contents like the MessageID and SentTimeout won't change. On the other hand, things related to receiving the message like RecieptHandle and Receive Count do change each time the message is received.
You can self-verify this from the AWS web console by:
Creating a message in a queue.
Viewing the message.
Waiting for the visibility timeout to expire. Once it has, open the SQS console again in a new tab.
Viewing the message again, in the new tab. Compare the contents of both received messages.