I have been doing some experimentation with siddhi CEP and found results that led me to believe that it is a blocking CEP meaning that it does not queue up events at a higher rate than it can process them.
Can someone tell me more about this?
Siddhi by default does not buffer events. However, by using #Async annotation you can instruct siddhi to buffer events before processing, this will cause events arriving at a given event stream to get buffered before processing and a seperate thread will take the event from the buffer and process the events.
Please refer siddhi architecture document for more information.
Related
Is it possible to ensure that a message was successfully delivered to an Event Hub when sending it with the log-to-eventhub policy in API Management?
Edit: In our solution we cannot allow any request to proceed if a message was not delivered to the Event Hub. As far as I can tell the log-to-eventhub policy doesn't check for this.
Welcome to Stackoveflow!
Note: Once the data has been passed to an Event Hub, it is persisted and will wait for Event Hub consumers to process it. The Event Hub does not care how it is processed; it just cares about making sure the message will be successfully delivered.
For more details, refer “Why send to an Azure Event Hub?”.
Hope this helps.
Event Hubs is built on top of Service Bus. According to the Service Bus documentation,
Using any of the supported Service Bus API clients, send operations into Service Bus are always explicitly settled, meaning that the API operation waits for an acceptance result from Service Bus to arrive, and then completes the send operation.
If the message is rejected by Service Bus, the rejection contains an error indicator and text with a "tracking-id" inside of it. The rejection also includes information about whether the operation can be retried with any expectation of success. In the client, this information is turned into an exception and raised to the caller of the send operation. If the message has been accepted, the operation silently completes.
When using the AMQP protocol, which is the exclusive protocol for the .NET Standard client and the Java client and which is an option for the .NET Framework client, message transfers and settlements are pipelined and completely asynchronous, and it is recommended that you use the asynchronous programming model API variants.
A sender can put several messages on the wire in rapid succession without having to wait for each message to be acknowledged, as would otherwise be the case with the SBMP protocol or with HTTP 1.1. Those asynchronous send operations complete as the respective messages are accepted and stored, on partitioned entities or when send operation to different entities overlap. The completions might also occur out of the original send order.
I think this means the SDK is getting a receipt for each message.
This theory is further aided by the RetryPolicy Class used in the ClientEntity.RetryPolicy Property of the EventHubSender Class.
In the API Management section on logging-to-eventhub, there is also a section on retry intervals. Below that are sections on modifying the return response or taking action on certain status codes.
Once the status codes of a failed logging attempt are known, you can modify the policies to take action on failed logging attempts.
I am using the EventHubStream provider in a project based on Orleans.
After the system has been running a few minutes Orleans starts throwing a QueueCacheMissException while trying to push an event to OnNext from a producer.
i have tried to increase the size of the cache but that helped only for a while.
Is this a normal behavior due to the size of the cache?
In this situation should i unsubscribe and subscribe again? i have tried to resume stream but that didn't work, the stream was in faulted state... any ideas?
It is likely that the service is reading events from eventhub faster than grains are processing them. EventHub can deliver events at a rate of ~1k/second per partition.
The latest version of the EventHub stream provider supports backpressure that should prevent this problem, but it has not been released. You can however build your own nugets.
I am trying to use AWS Kinesis stream for one of our data streams. I would like to monitor pending messages on my stream for ops purposes(scale downstream according to backlog), but unable to find any API that gives (approx) pending messages in my stream.
This looks strange as messages get expired after 7 days and if the producers and consumers are isolated and can't communicate, how do you know messages are expiring. How do you handle this problem?
Thanks!
There is no such concept as "pending" message in Kinesis. All the incoming data will be placed on a shard.
Your consumer application should be in running state all the time, to keep track of changes in your stream. The application (with the help of KCL) will continue to poll "Shard Iterator" in the background, thus you will be notified about the new data when it comes.
Roughly; you can see Kinesis as a FIFO queue and the messages will disappear in a short time if you don't pop them.
If your application will process a few messages in an hour, you should think about changing your architecture. Kinesis is probably not the correct tool for you.
Is there any document/article explaining the event lifecycle in WSO2 CEP?
I dont quite understand how the events are discarded from the event streams.
Thank you,
Hugo Calado
Events will be discarded immediately. Basic flow is stream will receive events from and receivers and it will immediately push events to publisher without storing. If you want to collect event for certain time periods you can use somwthing like time windows in Siddhi Execution Plans [1].
In following Siddhi query it collects events for 10 minutes and insert into AvgTempStream by calculating average stream. In that case events will be stored for 10 minutes in memory.
from TempStream#window.time(10 min)
select avg(temp) as avgTemp, roomNo, deviceID
insert all events into AvgTempStream;
[1] https://docs.wso2.com/display/CEP400/SiddhiQL+Guide+3.0#SiddhiQLGuide3.0-Window
I am working with wso2cep3.0. I went through the docs of wso2cep but there is no exaplanation about wso2cep how would i use it
1.what is the use of Input Event Adapter:-in my concern for getting the data from client.
2.Event Builder means incoming data format specifier.
3.Event Formatter means Outgoing data format specifier.
4.Output Event Adapter out put handler.
but how can i use this thing means any program or any event writer most important how would i publish this to external world example as http endpoint or https or jms.
I am unable to understand how would i start and where can i start.
Please suggest me i know the ESB,DSS,IS,BPS
For a typical CEP usecase, you configure the input event adaptor to connect to an event source, such as a JMS endpoint, Thrift endpoint(WSO2Event adaptor in CEP 3.0.0) etc. Event builder specifies how the incoming message will be mapped.
Next the execution plans will have the actual query (the processing part) of the execution flow. This is where the CEP engine actually processes the events.
Event formatter formats it to an output format as needed. The output event adaptor connects to the actual endpoint to which the processed result would be published. It would go and publish to a JMS endpoint, email, database etc.
To get started, you need to create an input event adaptor, then an event builder that uses the input event adaptor, then the execution plan, then the output event adaptor, and an event formatter that would format the result from the execution plan and send to the relevant output adaptor. You can find the flow in [1].
You can find example configurations in /samples/artifacts directory. You can find an overview of samples in [2] and can find how to run them [3]. Each sample has an associated producer and a consumer that simulates real world event producers/consumers. Samples would be the best place to learn more about CEP configurations.
[1] http://docs.wso2.org/display/CEP300/CEP+Configuration+Overview
[2] http://docs.wso2.org/display/CEP300/Overview+of+Samples
[3] http://docs.wso2.org/display/CEP300/Setting+up+CEP+Samples