I am trying to understand the whole WSO2 SOA topology, but not able to understand
how the CEP and BAM fit together
Can CEP provide visual monitoring of processed events e.g. integration with WSO2 GS
Although WSO2 website says CEP is tightly integrated with BAM for post processing I couldnt
find any scenario explaining the same or how its done..( can CEP feed BAM ? how to configure the same)
Why would you have CEP + BAM together ? Any use case
Answers
All WSO2 projects are capable of integrating with each other because they are based on the same underlying platform (WSO2 Carbon). In this particular case, WSO2 CEP and GS. One way is, persisting processed results from CEP in a data store or file, and reading it from a Gadget backend so that the gadget (the frontend) can visualize it in the GS. If you want, you can install GS features (dashboard, gadget repo, etc) on top of CEP as well and use the same server runtime. But, for the latter it has to be based on the same Carbon version
This means, that the same data agent can send events to BAM as well as CEP. They both share the Thrift and REST APIs. Similar to 1, CEP and BAM can exist in the same runtime or can be downloaded and used separately. One related article is at here
The primary use case was processing the same event for real time analytics for CEP and a just-in-time (near real time) batch based processing for BAM. Ex: Processing up time related analytics for servers can be broken down to fit both servers. For CEP the query can do, Alert me a server does not respond for 3 requests in 30 secs. For BAM, you can plot the uptime trend within a hour/day/week.
Related
I am having some trouble with Monitoring System Statistics in WSO2 Identity Server 5.3.3. There is no activity being reported in the Service Summary. We are about to go live with the WSO2 Identity server in a couple weeks, so I really want to keep an eye on the response time and counts.
Our current production system is not heavily used, but currently it shows Total Response Count: 0, when in fact I have tested several logins earlier today.
I know the monitoring was updating a few weeks ago, but something happened. Do I need to enable this via a setting or is it possible it was turned off?
This is the official documentation entry point on setting up and using statistics/analytics for latest WSO2 Identity server version. Please make sure you have followed all the steps accurately. You will have to separately enable it and configure in below files. After enabling you should be able to view stats.
<IS_HOME>/repository/conf/identity/identity.xml
<IS_HOME>/repository/deployment/server/eventpublishers/IsAnalytics-Publisher-wso2event-*
I have a little bit of confusion about the Stream Processor.
I've previously used the CEP and now I'm using the Stream Processor.
if I'm not mistaken, the Data Analytics Server, the CEP and the Machine Learner merged into the Stream Processor, is it true?
Because I found some inconsistencies, for example the SP can't publish directly in the dashboard, while CEP could.
So, my question is, all the feutures in the CEP and in the ML, are going to flow in the SP?
DAS, CEP and ML have not been completely merged into the Stream Processor.
In DAS, the real time analytics were handled by Siddhi and the batch analytics were done through Spark. However, in Stream Processor, only Siddhi acts as the core processor and Spark is not used.
Stream processor processes data in streaming manner through siddhi. In order to fulfill the requiremnts for batch analytics, incremental processing[1] which has been introduced to Siddhi 4.0.0 can be used.
Also ML support is provided through ml extentions written for Siddhi 4.0.0.
In das/cep it is required to define several artifacts like receivers, execution plans, publishers etc.. in order to create a analytic work flow.
But in Stream Processor,it is possible to define the whole flow in a single Siddhi-App.
For further clarification, please refer to the DAS to SP migration guide[2] and WSO2 analytics site[3].
[1] https://wso2.github.io/siddhi/documentation/siddhi-4.0/#incremental-aggregation
[2] https://docs.wso2.com/display/SP4xx/Upgrading+from+a+Previous+Release
[3] https://wso2.com/analytics
WSO2 Stream Processor is the latest WSO2 analytics offering. It has a super set of functionalities that WSO2 CEP had. Following is a comparison of capabilities of WSO2 CEP vs WSO2 SP.
General
The core of SP 4.x is the latest siddhi 4.x which is more stable and has improved performance. While CEP is powered by Siddhi 3.x.
SP is based on C5 and it's lean and light weight than CEP which was based on C4.
SP is designed to be container friendly and could native. Where as CEP had some challenges when deployed in containerised environments.
Everything is now contained in a Siddhi App, which is a single file which can be deployed and executed on it's own.
Incremental Analysis
New siddhi has the incremental analysis feature which is designed to cater batch analytics. With this feature users can easily do time series aggregations without having to integrate with other platforms such as Spark.
Incremental analysis smoothly federates real time analytics with batch analytics by allowing both forms of analytics to be done in the same message flow.
Distributed Deployment
SP 4.x has a distributed architecture which is highly scalable. SP's container friendly nature let's it be scaled massively.
The distributed deployment is fault tolerant and it supports exactly once processing with the aid of Apache Kafka.
CEP distributed architecture was based on Apache Storm.
Also, SP has in build support for Multi data center deployment. While CEP does not.
Tooling
SP has a rich editor which supports auto completion, event simulation, debugging of siddhi queries, etc. CEP only has the query editor UI in the management console.
Status Dashboard of SP let's users monitor their deployment with comprehensive set of statistics related to performance, resource consumption etc of Siddhi Apps and JVM. CEP had the carbon metric support which shows only JVM stats.
Business Rules
SP has Business rules feature where non-tech users can build processing logics through a graphical wizard-like UI without having to rite queries.
Developers can use this feature to present complex problems in a abstract manner which is understandable to business users.
CEP did not have feature focusing on business users.
So, my question is, all the feutures in the CEP and in the ML, are going to flow in the SP?
I don't believe so. StreamProcessor has only subset of capabilies of CEP, DAS or ML. IMHO it t is promoted currently as it is new, more lightweight and faster
When clustering WSO2 products, you create a database for the registry and other items that WSO2 product use for operations. With the combined WSO2 Enterprise Integrator, it consist of multiple elements (ESB, Business Process Manager, Message Broker, Analytics, and MSF4J).
Do you create different registry database for each sub-product or you use only one that is created for the first?
OPTION #1: WSO2_USER_DB, REGISTRY_DB, REGISTRY_LOCAL1, REGISTRY_LOCAL2
OPTION #2: ESB_WSO2_USER_DB, ESB_REGISTRY_DB, ESB_REGISTRY_LOCAL1, ESB_REGISTRY_LOCAL2, MB_WSO2_USER_DB, MB_REGISTRY_DB, MB_REGISTRY_LOCAL1, MB_REGISTRY_LOCAL2 ... etc.
I understand that user database can be shared since the authentication manager is similar. But is it the case with the registry database?
I'm new to clustering so this question might be a little not appropriate for advanced users.
WSO2 EI can offer various services, usually separately. For example WSO2 EI for integration or WSO2 EI for process automation.
When you install this product in clustering you do it under a specific role and not combined.
In essence you have local registry for each node and one shared for the synchronization of artifacts.
I hope it helps you.
Each of the profiles included in the EI is separate runtimes. You need to configure the profiles only according to your use case.
For example: If you are using Integrator profile (ESB) and MB profile (MB) you need to maintain two different registry data sources for ESB and MB as defined in your second option.
OPTION #2: ESB_WSO2_USER_DB, ESB_REGISTRY_DB, ESB_REGISTRY_LOCAL1, ESB_REGISTRY_LOCAL2, MB_WSO2_USER_DB, MB_REGISTRY_DB, MB_REGISTRY_LOCAL1, MB_REGISTRY_LOCAL2.
If you want to share the users across both applications, you can use one USER_DB instead of using two separate USER_DBs for ESB_WSO2_USER_DB and MB_WSO2_USER_DB.
EI clustering guide can be found from https://docs.wso2.com/display/EI610/Clustered+Deployment
I want to deploy only one carbon product (one JVM) containing the ESB component and the api gateway component.
Is it possible to build this kind of application ? Is there any reference documentation explaining how to do that ?
Thanks.
Every Carbon-based product comprise of a set of installable/uninstallable features. You can get an ESB pack and install API manager features on it (can do this via the management console UI). However you won't be able to put together mismatching versions of the features (features released for different carbon platform versions/components) so this is subject to availability of matching components. This documentation on feature management will give you some insights on how to do it. (There can be cases where components may be in total conflict though)
But if your requirement is just running two products on the same machine, you can consider running them with port offsets, which is easier.
A simple question, yet I couldn't find much information on the subject. How is business activity monitoring related to business analytics? I always thought business analytics is a subsystem of the activity monitoring systems. But that's only my limited view so I was wondering. In that trail of thought, how are for instance WSO2 BAM and Google Analytics compared to each other?
Initially WSO2 BAM 2.x.x was just a data analytic framework that can process big data offline (as batch processes with Apache Hadoop) which can also receive data and visualize data.
But from BAM 2.4.0 it comprises WSO2 Complex Event Processing features (CEP) that can monitor events real-time, process them and visualize them in a relatively low latency according to [1].
In Google Analytics most analytics and dashboards are available out of the box but with WSO2 BAM you may need to write some hive queries and dashboards to come up with a great solution.
WSO2 BAM is open source (Apache Licences) and you can use it as you wish with great flexibility although it lacks some out of the box features compare to the Google Analytics.
From BAM 2.4.0 it comes with an inbuilt Activity Monitoring feature [2] that is based on the concept of an Activity ID. This can be used out of the box when your business process is properly configured for activity monitoring use case.
[1] https://docs.wso2.org/display/BAM240/Realtime+Analytics
[2] https://docs.wso2.org/display/BAM240/Activity+Monitoring+Dashboard