I am wondering if I can use BAM and CEP to monitor requests from client, and even find the bottleneck of the service.
I found zipkin, a project that could do this, but the base of my application is WSO2, I don't want to get other projects from scratch.
Yes, you can use BAM/CEP for this. If you need real time monitoring you can use CEP and you can use BAM for batch process. From BAM 2.4.0 onwards, CEP features have been added inside BAM also hence you can use BAM and do real time analytics.
What type of services are involved with your scenario? Depends on this you can use already existing data publisher or write new data publisher for BAM/CEP to publish your request details. For example if you are having chain of axis2 webservice calls for a request from client, and you want to monitor where the bottle neck/more time consumed, then you may use the service stats publishing, and monitor the average time take to process the message which will help you to see where the actual delay has been introduced. For this you can use existing service statistics publisher feature. Also BAM will allow you to create your own dashboard to visualize, hence you can customize the dashboard.
Also with BAM 2.4.0 we have introduced notifications feature also which you can define some threshold value and configure to send notification if that cross that threshold value.
Related
I have 100 devices that do simple calculation.
The only way to extract data from those devices is by their REST API.
I want to schedule a task every minute to collect every new data from those 100 devices.
Each device have its own API endpoint and all the payloads to collect the data are identical for each device. To be able to invoke the REST API I need to provide a valid token. This token can be acquired by calling the authentication (/auth/token) function from each REST API endpoints with specific usernames and passwords
They have all the same version, so the exact same logic is needed to collect the data. I found out we can use WSO2-ESB to collect data.
What I've done so far:
I create an Entreprise Integration Connector for the devices.
I create a New Integration project in the Integration Studio.
I use the Connector and Schedule a task that do the sequence logic to test data collection from 1 device.
Now I need to scale from collecting 1 device to 100 devices at the same time.
How can I collect all devices at once using the same logic with WSO2-ESB?
It seems that you have followed the correct approach. Yes there is a significant change between EI 6 series and the EI 7 series. The EI 6 series has the ESB, BPS, MB and Analytics profiles in the same server. The EI 7 series only has the ESB server. For your use case you need the ESB. Therefore you can use either EI 6.6.0 server or EI 7.1.0 server.
If you need to invoke a REST API no need to use a connector. The schedule task and the sequence would be sufficient. To implement the logic for 100 devices we need more information.
Do you have different API endpoints for each device
Do you need different payloads to get information from different APIs
It depends what do you want to have on the output - collected data or single per device... As source of your endpoint devices you can use Local-entries, or embedded in scheduler task as message. Maybe you should also look at the
Split-aggregate pattern
I want to get the current number of connections from within an akka-http application. Is there any way to get hold of it?
The end goal is to be able to publish it to prometheus regularly for monitoring the load of the service.
Have you tried using Kamon ? It has a nice build in Context instrumentation that you can use to have the metrics you want to use.
Documentation here : https://kamon.io/docs/latest/core/context/
I am working on an integration project where we want to use JIRA tickets for business follow up operations. The JIRA (externally hosted) is not always available hence I want to use some Guaranteed delivery patterns. So the question, is it possible WSO2 ESB to use existing connectors (JIRA) in the message processor?
Message processors and connectors are independent. This is what you have to do (you are in right track at the moment too).
Put your message to a message store. This can be the in-memory message store (which looses messages upon a server restart) or a persistent message store such as an activemq queue.
Then, configure a message processor to consume messages from this store. There are two types of message processors namely forwarding and sampling processors. Here you need a sampling processor.https://docs.wso2.com/display/ESB490/Message+Processors
These consumed messages can be handed over to a sequence where the sequence can use the jira connector to create the jira.
Problem I see with this approach is, sampling processors do not support guaranteed delivery (but the forwarding processor do). But, AFAIK, we cannot use connectors with forwarding processors because we need to provide an endpoint in the forwarding processors configs.
You will understand the difference and the pros and cons of two types when you go through the docs. As a workaround, I can suggest following.
Create a proxy service which uses jira connector to create the jira
Then use the forwarding processor to send the consumed message to that proxy service.
I think, with above approach, you will be able to achieve guaranteed delivery.
I am new with wso2 products. I am using EMM for device and app. But i want to monitor my devices continuously.
As some document suggest, i can done this using CEP in real time and also can use BAM. But i did not found any proper document to install or configure these system together (EMM, CEP, BAM).
My EMM server is already live. I am now tring to analysis the http access log to find out registration failed or other error.
But I actually need to monitor the EMM server in real time.
Do I need to run these servers separately and also have to configure them separately? Is there any way to merge all(EMM,CEP,BAM and even ESS) application and run all of them by running one carbon server?
If I need to run these server separately, then how I can configure CEP/BAM as this will monitor EMM server's API/log files/services etc?
Is there documentation for this? I can not find any document related to CEP/BAM related to EMM.
Can any one please help me for this?
WSO2 BAM product has been replaced by WSO2 DAS (Data Analytics Server). Again WSO2 DAS consist with WSO2 CEP features where as it is capable of doing both real time and batch time processing. Therefore you can DAS for both batch processing and real time processing
Both DAS and CEP deals with event streams where as in DAS streams could be persisted to event tables as data needs to be stored for batch processing [2]. On the other hand both can be integrated with other applications by using Receivers for streams. You can protocols such as thrift, soap, http, mqtt, jms and kafka when creating receivers[3]
For real time processing you need to write Execution Plans [4] which uses Siddhi Query Language [5]. DAS supports Apache spark for batch processing. Both products consist with Analytic dashboard which could be used to visualize real time and persistent data [6].
[1] http://wso2.com/products/data-analytics-server/
[2] https://docs.wso2.com/display/DAS300/Understanding+Event+Streams+and+Event+Tables
[3] https://docs.wso2.com/display/DAS300/Configuring+Event+Receivers
[4] https://docs.wso2.com/display/CEP400/Creating+a+Standalone+Execution+Plan
[5] https://docs.wso2.com/display/CEP400/SiddhiQL+Guide+3.0
[6] https://docs.wso2.com/display/DAS300/Analytics+Dashboard
Yes its possible. You will need to use a data publisher to publish events to DAS.
Currently we are using wso2 4.1.0 version. we are using soap based services authentication admin, entitlements service for getting policy decision , getting claims values. we are using cxf for our webservice clients. when we make calls with 500 concurrent users from a single machine everything works fine. but when we make it to 1000 concurrent users we are seeing a huge response time for these service calls from wso2 is. can you tell if there is any configuration change that we need to make for tuning. we increased the wso2 is axis client no. of connections per host but we still didn't see any improvement. By the way we are using the default configuration of wso2 is out of the box
Thanks
Kishore
There are some places you can improve the performances.
1.Increase the memory setting. you can find it in the wso2server script file in /bin. can u increase default value and see.. such as
-Xms1024m -Xmx2048m -XX:MaxPermSize=1024m
2.Increase max thread pool size in catalina-server.xml file which can find in /repository/conf/tomcat
maxThreads="250"
3.Increase caching timeout value. entitlement.properties file which can find in /repository/conf/security
4.Please check you are not authenticating for each request. You need to call AuthenticationAdmin 1st time and get the cookie and then use cookie for subsequence requests.
Else could do use some Jconsole or Jprofiler and see what is going wrong? Also according to your environment, this can be the max load that one server can handle, then you need to do horizontal scaling. (adding WSO2 Identity Server more instance in a cluster)