In WSO2ESB the message sampling processor de-queues the messages from the queue and forward it to a sequence where we can do transformation to the request and send to the back - end. And
The main purpose of using MSMP is for archiving guaranteed delivery, or in other words, to archive reliable messaging. [Source]
But when we use Message Sampling Processor it just forward the message to sequence and the processor doesn't know whether the message successfully delivered to the back end (using send | call mediator) or the processor knows this and restore the message in the queue?
I tried this scenario using JMS-Message Store (WSO2MB) and what I found is, message get lost when the back end service is down.
Message Sampling processor does not support reliable messaging[1]. For reliable messaging you will have to use Message Forwarding processor[2]
[1] - https://docs.wso2.com/display/ESB481/Message+Sampling+Processor
[2] - https://docs.wso2.com/display/ESB481/Message+Forwarding+Processor
Related
I have already gone through the client library provided by google in the below doc. The given client library is just to poll the message from PubSub, But it will not poll continuously until we create the Unbounded Source Connector.
https://cloud.google.com/pubsub/docs/pull#synchronous_pull
Since the source connector I'm trying to build is Unbounded source,For that I need to take care of Checkpoint Marker, implement PubSub reader, PubSub split source and implement ACK and NACK logic and I believe it will take a good amount to time to create my own Unbounded source connector. Right now PubSubIO(Beam api) only supports asynchronous pull. So is there any way I can just implement ACK n NACK logic logic on top of PubSubIO api provided by Apache Beam. Is there any kind of already developed api which is more suitable for this kind of use-cases.
With synchronous pull, you can acknowledge the intended message and NACK the consumed message in case of any parsing failure.
The feature that you expect doesn't exist, and should not exist.
The current behavior, the async pull, get the message and as soon as the message is persisted (in the worker or sink in the pipeline, the first of both), the message is ACK.
In your case, you expect to ACK manually the message according with the pipeline processing. However, you are limited to 10 minutes to ACK your messages with PubSub. Imagine if you build a pipeline with windows of 15 minutes (or more). You need to wait the end of the processing of the windows to ACK the messages; impossible!
The correct design, in your case, is to manage your errors in your pipeline.
Is it possible to ensure that a message was successfully delivered to an Event Hub when sending it with the log-to-eventhub policy in API Management?
Edit: In our solution we cannot allow any request to proceed if a message was not delivered to the Event Hub. As far as I can tell the log-to-eventhub policy doesn't check for this.
Welcome to Stackoveflow!
Note: Once the data has been passed to an Event Hub, it is persisted and will wait for Event Hub consumers to process it. The Event Hub does not care how it is processed; it just cares about making sure the message will be successfully delivered.
For more details, refer “Why send to an Azure Event Hub?”.
Hope this helps.
Event Hubs is built on top of Service Bus. According to the Service Bus documentation,
Using any of the supported Service Bus API clients, send operations into Service Bus are always explicitly settled, meaning that the API operation waits for an acceptance result from Service Bus to arrive, and then completes the send operation.
If the message is rejected by Service Bus, the rejection contains an error indicator and text with a "tracking-id" inside of it. The rejection also includes information about whether the operation can be retried with any expectation of success. In the client, this information is turned into an exception and raised to the caller of the send operation. If the message has been accepted, the operation silently completes.
When using the AMQP protocol, which is the exclusive protocol for the .NET Standard client and the Java client and which is an option for the .NET Framework client, message transfers and settlements are pipelined and completely asynchronous, and it is recommended that you use the asynchronous programming model API variants.
A sender can put several messages on the wire in rapid succession without having to wait for each message to be acknowledged, as would otherwise be the case with the SBMP protocol or with HTTP 1.1. Those asynchronous send operations complete as the respective messages are accepted and stored, on partitioned entities or when send operation to different entities overlap. The completions might also occur out of the original send order.
I think this means the SDK is getting a receipt for each message.
This theory is further aided by the RetryPolicy Class used in the ClientEntity.RetryPolicy Property of the EventHubSender Class.
In the API Management section on logging-to-eventhub, there is also a section on retry intervals. Below that are sections on modifying the return response or taking action on certain status codes.
Once the status codes of a failed logging attempt are known, you can modify the policies to take action on failed logging attempts.
Running WSO2 EI 6.2.0
I have a simple use case (Sequence) for WSO2 EI ESB:
Extract some parameters from the original request
Call an Async REST API
Extract an Execution ID from the Async Call Payload
Poll Loop another Sync API to check Execution Status based on Execution ID
Halt polling when the Sync API says that the request is completed
Extract some parameters from the last Sync Call
Response
My problem lies on the Poll a Sync API until it returns some parameter saying that the previous Async execution is finnished.
Is there any WSO2 EI Sequence mediator for this sort of Poll Loop?
The ESB mediations (sequences) are not really intended to keep the state and wait for anything. I'd believe it is even intention not having any sort "do/while" loop. We had a project requiring many polling steps and we used a process server to do so. So - with pure mediation it is very difficult to accomplish what are you asking for. Even you may check this one http://bsenduran.blogspot.com/2017/08/while-loop-in-wso2-esb.html
I will propose a few things you could do:
write a custom polling mediator (I really do not advice to do so)
use a process server (requires additional no-so-lightweight server)
use messaging with message processor (send a message to a queue, a message processor will poll, call and send the back to the queue or to response)
In all cases - if a client is waiting for a synchronous response, you need to finish the polling before the client times out. IMHO the best option return a message to a client (we are working on it) and avoid polling if possible..
Im using WSO2 EI 6.1.1 with Message Broker, and trying to create message queue with message store and message process with an endpoint.
When I shutdown my endpoint, the message processor is deactivated and the messages stay in the queue and they are not moved to DLC.
What should I do to make it work ?
Thanks,
Faris Shomou
This is the expected behavior with message processor / message store :
A scheduled message processor will try to send the message until the
delivery is successfull (and offers you a way to implement guaranteed
delivery pattern)
A sampling message processor will send the message
in a non reliable way (it can be lost)
If you want to manage a JMS transaction and have the message to go in DLQ, use a jms inbound endpoint or jms proxy and set required parameters (transport.jms.SessionTransacted, transport.jms.SessionAcknowledgement : have a look to wso2 documentation https://docs.wso2.com/display/EI611/JMS+Transactions)
Message store / processor is used to implements Dead letter channel EIP : the jms store host the dead message and you don't want it to be moved elsewhere
I'm quite a newbie in WSO2 so sorry for the mistakes (and for my english too ... )
I need to implement a proxy with delivery-garantee pattern and here you are my solution (I'm started from this post http://charith.wickramaarachchi.org/2012/05/another-message-redelivery-pattern-with.html):
a proxy invoke an external service giving, as input, the initial
client message
if the external service is running all works fine and
the reply is given to the client
if the external service is down or generate a SOAP fault, I'll
put the message in a store (retry store), and then, using a sampling
processor (after a time "t"), I'll try again for "n" max attempts:
at any attempt, if the external service is down or generate a SOAP
fault, I'll put the message again in the retry store, and the
process is repeated
after "n" attempts, if the external service is still out of
service, the message is stored in another store (garbage store)
All works fine when I try to test with one message, but when I try to test with more messages (> 20 but this number is variable ... ), the sampling processor hangs completely, nothing is shown in the logs. Looking in the console, sometimes (but not always ...), the processor is off, deactivate and in this case, to restore, I need to undeploy, stop and restart, and then deploy again my .car.
NOTE: I've to use the sampling processor and not the forwarding processor because this processor, after "n" attempts deactive itself and I can't use it for my goals.
I can't put here the complete code because is too long, but I can give you a sample .car that you can deploy and execute on your WSO2 installation (to simulate the external service I've used the echo service ...).
Here you are the sample car that you can download
Thank you very much in advance: all suggestions are appreciated!!!
Cesare
Message Forwarding Processor
Retrieves the messages stored in a message store and reliably forwards them to a specified endpoint. This processor attempts to send one message at a time and it does not dequeue a message from the store until it receives a response from the target endpoint. Therefore this processor is ideal for implementing in-order delivery scenarios and guaranteed delivery scenarios.
Sampling Processor
Retrieves the messages stored in a message store and injects them to a given sequence at specified intervals. This processor utilizes the Quartz scheduler framework for periodically processing messages. This can be used to implement message rate throttling scenarios.
--> You can use the forwarding processor and configure it so that it will never be deactivated, just add this parameter : <parameter name="max.delivery.attempts">-1</parameter>