I am using Sidhhi CEP as Java library in my project . Now i need to analyse my data with High available system (Similar to Esper HA). I have done little bit study about Siddhi High availability
http://wso2.com/library/articles/2014/05/high-availability-deployment-in-wso2-complex-event-processor-0/
Also gone through with the above links
is that possible to the same task using Siddhi java library ???
Above document demonstrates how WSO2 CEP has achieved high availability by using multiple nodes with hazelcast clustering. If you are just using Siddhi CEP as a Java library, then you need to implement your clustering by using hazelcast or any other.
More about WSO2 CEP Clustering [1]
[1] https://docs.wso2.com/display/CLUSTER420/Clustering+Complex+Event+Processor
Related
I am looking for an Open source Business Intelligent (BI) Solutions for my organization. So I am trying WSO2 Stream Processor and I could not find any graphical Interface for building RDBMS Queries.
I check editor, portal and widgets.
widgets were very nice for visualizing data but samples were limited and I could not find what I am looking for.
Especially I need an Interface that shows me my DB (or multiple DBs) and when I select Them to show me Tables and I Could select Tables and building my query graphically.
As of now, this is not available in the Editor/Tooling interface. However, it will be easier if this can be viewed on the Editor itself. You can raise a feature request in Siddhi distribution repo and the team will see if we can incorporate it into the roadmap.
Please note, WSO2SP is not currently under active development and you can try the latest WSO2 Streaming Integrator or OSS option of Siddhi Cloud-native Stream Processor. However, Streaming Integrator focuses on streaming data integration. Whereas Siddhi Cloud-native Stream Processor bundles the newest version of Siddhi and it's a tool for building fully-fledged event-driven applications.
I have a little bit of confusion about the Stream Processor.
I've previously used the CEP and now I'm using the Stream Processor.
if I'm not mistaken, the Data Analytics Server, the CEP and the Machine Learner merged into the Stream Processor, is it true?
Because I found some inconsistencies, for example the SP can't publish directly in the dashboard, while CEP could.
So, my question is, all the feutures in the CEP and in the ML, are going to flow in the SP?
DAS, CEP and ML have not been completely merged into the Stream Processor.
In DAS, the real time analytics were handled by Siddhi and the batch analytics were done through Spark. However, in Stream Processor, only Siddhi acts as the core processor and Spark is not used.
Stream processor processes data in streaming manner through siddhi. In order to fulfill the requiremnts for batch analytics, incremental processing[1] which has been introduced to Siddhi 4.0.0 can be used.
Also ML support is provided through ml extentions written for Siddhi 4.0.0.
In das/cep it is required to define several artifacts like receivers, execution plans, publishers etc.. in order to create a analytic work flow.
But in Stream Processor,it is possible to define the whole flow in a single Siddhi-App.
For further clarification, please refer to the DAS to SP migration guide[2] and WSO2 analytics site[3].
[1] https://wso2.github.io/siddhi/documentation/siddhi-4.0/#incremental-aggregation
[2] https://docs.wso2.com/display/SP4xx/Upgrading+from+a+Previous+Release
[3] https://wso2.com/analytics
WSO2 Stream Processor is the latest WSO2 analytics offering. It has a super set of functionalities that WSO2 CEP had. Following is a comparison of capabilities of WSO2 CEP vs WSO2 SP.
General
The core of SP 4.x is the latest siddhi 4.x which is more stable and has improved performance. While CEP is powered by Siddhi 3.x.
SP is based on C5 and it's lean and light weight than CEP which was based on C4.
SP is designed to be container friendly and could native. Where as CEP had some challenges when deployed in containerised environments.
Everything is now contained in a Siddhi App, which is a single file which can be deployed and executed on it's own.
Incremental Analysis
New siddhi has the incremental analysis feature which is designed to cater batch analytics. With this feature users can easily do time series aggregations without having to integrate with other platforms such as Spark.
Incremental analysis smoothly federates real time analytics with batch analytics by allowing both forms of analytics to be done in the same message flow.
Distributed Deployment
SP 4.x has a distributed architecture which is highly scalable. SP's container friendly nature let's it be scaled massively.
The distributed deployment is fault tolerant and it supports exactly once processing with the aid of Apache Kafka.
CEP distributed architecture was based on Apache Storm.
Also, SP has in build support for Multi data center deployment. While CEP does not.
Tooling
SP has a rich editor which supports auto completion, event simulation, debugging of siddhi queries, etc. CEP only has the query editor UI in the management console.
Status Dashboard of SP let's users monitor their deployment with comprehensive set of statistics related to performance, resource consumption etc of Siddhi Apps and JVM. CEP had the carbon metric support which shows only JVM stats.
Business Rules
SP has Business rules feature where non-tech users can build processing logics through a graphical wizard-like UI without having to rite queries.
Developers can use this feature to present complex problems in a abstract manner which is understandable to business users.
CEP did not have feature focusing on business users.
So, my question is, all the feutures in the CEP and in the ML, are going to flow in the SP?
I don't believe so. StreamProcessor has only subset of capabilies of CEP, DAS or ML. IMHO it t is promoted currently as it is new, more lightweight and faster
When clustering WSO2 products, you create a database for the registry and other items that WSO2 product use for operations. With the combined WSO2 Enterprise Integrator, it consist of multiple elements (ESB, Business Process Manager, Message Broker, Analytics, and MSF4J).
Do you create different registry database for each sub-product or you use only one that is created for the first?
OPTION #1: WSO2_USER_DB, REGISTRY_DB, REGISTRY_LOCAL1, REGISTRY_LOCAL2
OPTION #2: ESB_WSO2_USER_DB, ESB_REGISTRY_DB, ESB_REGISTRY_LOCAL1, ESB_REGISTRY_LOCAL2, MB_WSO2_USER_DB, MB_REGISTRY_DB, MB_REGISTRY_LOCAL1, MB_REGISTRY_LOCAL2 ... etc.
I understand that user database can be shared since the authentication manager is similar. But is it the case with the registry database?
I'm new to clustering so this question might be a little not appropriate for advanced users.
WSO2 EI can offer various services, usually separately. For example WSO2 EI for integration or WSO2 EI for process automation.
When you install this product in clustering you do it under a specific role and not combined.
In essence you have local registry for each node and one shared for the synchronization of artifacts.
I hope it helps you.
Each of the profiles included in the EI is separate runtimes. You need to configure the profiles only according to your use case.
For example: If you are using Integrator profile (ESB) and MB profile (MB) you need to maintain two different registry data sources for ESB and MB as defined in your second option.
OPTION #2: ESB_WSO2_USER_DB, ESB_REGISTRY_DB, ESB_REGISTRY_LOCAL1, ESB_REGISTRY_LOCAL2, MB_WSO2_USER_DB, MB_REGISTRY_DB, MB_REGISTRY_LOCAL1, MB_REGISTRY_LOCAL2.
If you want to share the users across both applications, you can use one USER_DB instead of using two separate USER_DBs for ESB_WSO2_USER_DB and MB_WSO2_USER_DB.
EI clustering guide can be found from https://docs.wso2.com/display/EI610/Clustered+Deployment
We have a large number of algorithms written in C++ and Java.
Now we want to introduce the WSO2 DAS platform.
Where and how can we integrate these algorithms with DAS?
I should explain more info: these algorithms are running realtime analytics, but we found that DAS uses siddhi for realtime analytics, and it doesn't provide any interface or API that we can use for these algorithms.
WSO2DAS consist with event receivers where you Java and C++ applications can push data by using several protocols such as thrift, http, soap, mqtt, jms, kafka and etc [1]
Siddhi can be used as a Java Library within your applications or else you can create a event flow with streams and receivers in WSO2 DAS and add Siddhi queries in an Execution Plan. For more details please refer Siddhi documentation [2]
[1] https://docs.wso2.com/display/DAS300/Configuring+Event+Receivers
[2] https://docs.wso2.com/display/CEP400/SiddhiQL+Guide+3.0
I am trying to understand the whole WSO2 SOA topology, but not able to understand
how the CEP and BAM fit together
Can CEP provide visual monitoring of processed events e.g. integration with WSO2 GS
Although WSO2 website says CEP is tightly integrated with BAM for post processing I couldnt
find any scenario explaining the same or how its done..( can CEP feed BAM ? how to configure the same)
Why would you have CEP + BAM together ? Any use case
Answers
All WSO2 projects are capable of integrating with each other because they are based on the same underlying platform (WSO2 Carbon). In this particular case, WSO2 CEP and GS. One way is, persisting processed results from CEP in a data store or file, and reading it from a Gadget backend so that the gadget (the frontend) can visualize it in the GS. If you want, you can install GS features (dashboard, gadget repo, etc) on top of CEP as well and use the same server runtime. But, for the latter it has to be based on the same Carbon version
This means, that the same data agent can send events to BAM as well as CEP. They both share the Thrift and REST APIs. Similar to 1, CEP and BAM can exist in the same runtime or can be downloaded and used separately. One related article is at here
The primary use case was processing the same event for real time analytics for CEP and a just-in-time (near real time) batch based processing for BAM. Ex: Processing up time related analytics for servers can be broken down to fit both servers. For CEP the query can do, Alert me a server does not respond for 3 requests in 30 secs. For BAM, you can plot the uptime trend within a hour/day/week.