I need to develop an application which will read messages from MSMQ asynchronously. I found good examples of this the official one being
http://msdn.microsoft.com/en-us/library/windows/desktop/ms701332(v=vs.85).aspx
http://msdn.microsoft.com/en-us/library/windows/desktop/ms701349(v=vs.85).aspx
But the additional requirement is MSMQ should keep the message in message queue until I send acknowledgement message. Is there a way to do that? If yes, is there a example out there which demonstrates this?
One more advanced requirement also is that there might be more than one receiver of the same message (or subscriber of the same message queue) and I want MSMQ to delete the message only after all of the subscribers have read the message. Is that possible?
Thanks in advance,
-Neel.
Related
I have a dead-letter queue for a pubsub cloud function that is receiving messages using PUSH subscription and at the moment the service is successfully sending those messages to the dead-letter topic and the dead-letter subscription when fails. However, I am uncertain on how to carry on once it reaches this dead-letter subscription.
I don't want to lose the messages that have been sent to the dead letter so my idea would be that in case of failures from the service to acknowledge the message after predefined delivery attempts, the message will be forwarded to a dead letter topic. The same service, when back to life, can pull the messages from the dead-letter topic as well to see what it missed during the times of unavailability.
There is a similar post in here but the answer only points to the options but not the solutions, and unfortunately, I haven't been able to find it.
There is also a mention in here about this issue where it's actually from where I have taken my question.
Please, could somebody point me in the right direction? Is there a better way?
If the main process aren't able to process the messages, you have to rely on the retry mechanism of PubSub.
If you put the messages in a Dead letter topic, it's because you can't process the message with the main function. So, it's another process, another function. You can't process with the same function the dead letter message (In fact you can, but it's an anti pattern).
A good pattern is to save your dead letter message somewhere. When your main function is up again, trigger a process that read the messages and re publish the messages in the main topic.
Does the deletion AWS SQS message (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_DeleteMessage.html) remove/acknowledge all messages that was earlier fetched by any of consumer?
I'm asking because I would like to make sure delete operation is not working in similar way to acknowledge described in (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/getting-started.html)
In this mode, when a message is acknowledged, all messages received before this message are implicitly acknowledged as well. For example, if 10 messages are received, and only the 10th message is acknowledged (in the order the messages are received), then all of the previous nine messages are also acknowledged.
The acknowledgements are a feature of JMS. When you work with SQS directly, there are no acknowledgements sent to the producer of the messages.
If you require such acknowledgements you would have to develop request-response system yourself, or use pre-existing solutions for that (such as with JMS).
So I am building a small application that uses SQS. I have a simple handler process that determines if a given message is considered processed, marked for retry (to be re-queued) or is not able to be processed (should be sent to dead letter).
However based on the docs it would appear the only way to truly send a message to DL is by using a redrive policy which operates over # of receives a message has racked up. Because of the nature of my application, I could have several valid retries if my process isn't ready to handle a given message, but there are also times I may want to DL a message I have just received. Does AWS/Boto3 not provide a way to mark a specific message for DL?
I know I can just send the message myself to another queue I consider my own DL, I would just rather use AWS' built in tools for this.
I don't believe there is any limitation that would prevent you from sending the message to the deal-letter-queue by yourself.
So just read the message from the Q, if you know it needs to go to the DLQ directly, send it to the DLQ and remove it from the regular Q.
I am using AWS SQS and Spring JMS in my project. I have my method with #JmsListener(destination = "queue_name"). I want to use this queue for two different types of messages.
Since this listener is configured to this queue it receives both types of messages. What I am trying to achieve is to ignore message of one type. (Sender is adding a MessageAttribute while sending message to Queue). So, is there a way to just ignore message coming from sender 2 so this method won't process them.
Also, I have DLQ set on this queue with max receives as 5. So if message is not processed in first 5 attempts it gets moved to DLQ.
Please do share your suggestion.
Thanks.
The correct solution is to use 2 different queues; SQS can't filter the messages delivered by any property, so as you are seeing, when the client reads the message and doesn't process it, it is going to end up in your DLQ quicker.
Queues are free, so having multiple won't cost any more.
In IEventProcessor.ProcessEventsAsync I want to store events in a persisted store. It's possible this store is unavailable and messages cannot be persisted. How to sign these messages to be redelivered later?
The store may be down only for some hours, but until it's up again every message is affected and cannot be persisted.
I don't think you can mark a particular event to be delivered in eventhub, unlike ServiceBus queue. However, eventhub does provide retention policy and offset for each event, which make possible to reprocess an old event. You can read more in the "checkpointing" section from this document: https://azure.microsoft.com/en-us/documentation/articles/event-hubs-overview/
Adding to Tyler response, i suppose that you could use the some kind of "Poison Message"/Dead letter queue approaches. Event Hub does not have that functionality, but Service Bus Queues do.
Anyway, i think that it should be a programmatic approach, not something inside of the backend.
There is a good article about something else, but approach is alike what i meant:
https://www.dougv.com/2015/07/handling-poison-messages-in-an-azure-service-bus-queue/