We have implemented Api-Manager Clustering and when we try to hit the api with large data some events are not captured by analytics
So we want to implement clustering for analytics - since there is no document available for apimanager - analytics clustering we have followed https://docs.wso2.com/display/CLUSTER44x/Clustering+Data+Analytics+Server+3.1.0.
We have implemented the Spark Deployment pattern - 2 node spark clustering but everytime in one node we are getting out of memory error
since the above one is not working we are planning to do the https://docs.wso2.com/display/CLUSTER44x/Fully+Distributed+High+Availability+Deployment+-+DAS+3.1.0 but it doesn't answer few question
How do they communicate each other - I mean some where we need to mention In reciver that Analyzer stays in some ip
And do the gateway need to point to reciever ?
Can Anyone help us
Thanks
Related
I am familiar with firebase platform, but I am relatively a new user of the google cloud platform as whole.
I am working on a project built using a microservices structure, and I do have so many question for which I cannot find an answer or better I cannot find any example.
Unfortunately all the example that I am able to find are way to simple to be able to extrapolate a viable answer for my issues.
I adopted the new cloud run offer, and I decided to play with the full managed version (not kubernetes). I built few microservices (each service is built using express for node or flask for python - depending on what the services does). Each microservices expose it's own endpoint and has it's own api to call the methods - and I use a service account to allow the application to perform the internal calls.
I now want to expose the application to the external (specifically to my client built using vuejs technology), and I was trying to leverage another google product to create and expose an api: the google endpoints.
My question (specifically referred to the cloud run structure) is related to how is possible and what I need to do to create an api endpoints to communicate with the client app, that internally calls multiple services and combine their response in one.
Just to be clear, let's make an example:
Cloud run service 1 -> crud user api
Cloud run service 2 -> crud product api
Cloud endpoint external visible api -> get user from service 1, and after get products from service 2 and return the combined response all green products for user Jane Doe.
How I can aggregate the response directly in the endpoint gateway, check for failure and if everything goes smooth send the aggregate response to the client?
I need to build the aggregate endpoint in something else, like a cloud function for example? or I can do it directly in the google endpoints gateway?
Note that for cloud run the google endpoints is another cloud run container.
Thanks guys for some help, running pretty much out of option here.
As per my understanding, API Gateway should just work as a proxy, presenting all micro services as a single endpoint. To this scenarios I think you can have following 2 approaches :
1: Implement a new micro service (or on any of the existing one) which will do invocations and aggregation of responses.
2: Client(like UI) can invoke the services and do the aggregation on their side as well.
I feel, it is not a good idea to do it at api-gateway.
In my opinion, from an architectural point of view, the best option for you is to create a new microservice which will take the responses from the other two and then, it will aggregate them.
I understand that you want to aggregate the responses in a api-geteway and you are not able to find code examples for it. Here I was able to find a guide on what are you wanting to implement. The full code implementation can be found in this repository.
Keep in mind though, this idea of implementation is not a best practice.
This is ok, only if those two services that are going to be combined are independent. Meaning there is no functional/business relation between them and the concurrency or inconsistency problem will not occur in the process of aggregating.
What is the best way to implement ETL jobs using WSO2.
We've been trying to leverage data services within WSO2 EI 6.4.
Our objective is to fetch data from web services as well as RDBMS and to store it to an RDBMS.
Any suggestions / ideas will be much appreciated.
In My Experience with WSO2 middleware, Data services may not be the best fit for ETL jobs.
We had similar case where we wanted to copy data from one Databse to another Application.
For that we wrote integration (java) web service to fetch data from database send to the applciation using application Interface (web service exposed from application). and configured the Integration web service in EI scheduler to periodically run the service to sync the data.
Yes, Stream processor is better but need to really see if it fits in for ETL
I am using WSO2 APIM 1.10.0 on a single server deployment and would like to move to a clustering one. Looking at this documentation I could found a lot of information, howevre something is boring me; do I really have to always do all of it?
I mean, I don't want to split all my workers in multiple instances, all I want is configure two full setup configurations (key manager + publisher + store + gateway), each one on its own host and make sure I can put a load balance in front of it.
Thre requiremenst are simple: I would like to share the load on both of them, and guarantee a better availability in case of one of the hosts goes down. Is it a MUST break down the whole installation on both nodes so I have to start each component independently with offset ports configured?
I coud see that on version 2.0.0 a lot have been simplified, any way to reach the same on 1.10.0 one?
Regards
Splitting into profiles is not mandatory. This is designed in this way to scale API Manager based on the TPS. If you have a low TPS count and prefer to have 2 node HA setup, you can do the following.
Cluster the two nodes using wka, aws, etc.
Use dep-sync to share API artifacts between two nodes.
Use one node as the Publisher. You need to handle the publisher node traffic using single node. This is to avoid getting SVN conflicts.
You can serve API requests from both nodes.
You do not want to always use the same deployment pattern mentioned in the docuemtnation that you have pointed there. There are various Other deployment patterns that you can use according to the scalability and the requirement of yours.
Please refer the following documentation [1] for different deployment patterns you can use for WSO2 API Manager and [2] for more information on worker Manager separation and Load balancing.
[1] https://docs.wso2.com/display/CLUSTER44x/API+Manager+Deployment+Patterns
[2] https://docs.wso2.com/display/CLUSTER44x/Separating+the+Worker+and+Manager+Nodes
I am trying to set up a clustered environment for WSO2 API Manager. In the environment I need there is no need for an external store. I looking to start with the least amount of nodes and JVMs that yet is scalable with growing number of API requests.
Having looked at the WSO2 documentation
Clustering WSO2 API Manager and specifically the "Store and Publisher components in a single server node" model.
Some questions on this deployment model
Where is the Gateway Manager deployed?
I understand the publisher and store are on the same server node. Can they be run in the same JVM? If so would you use the default profile that also starts up KM and Gateway or soemthing else?
(Appologies but I can't post the image due to my low reputation value. I would have thought the image of the model would have helped)
Yes - API Store and Publisher will be running in the same JVM. As there is no profile for Store & Publisher (see [1] for available profiles), we need to start API Manager in the default profile. And yes it will start KM & Gateway components as well. But you can block (not expose) gateway ports. And regarding gateway manager, I guess one gateway node can act as both manager and worker in this deployment pattern.
[1] https://docs.wso2.com/display/AM180/Product+Profiles
As per the design publisher is a subset of store. So, If you start with profile api-store you will eventually get publisher as well. In this case you can start the server with following option.
-Dprofile=api-store
A simple question, yet I couldn't find much information on the subject. How is business activity monitoring related to business analytics? I always thought business analytics is a subsystem of the activity monitoring systems. But that's only my limited view so I was wondering. In that trail of thought, how are for instance WSO2 BAM and Google Analytics compared to each other?
Initially WSO2 BAM 2.x.x was just a data analytic framework that can process big data offline (as batch processes with Apache Hadoop) which can also receive data and visualize data.
But from BAM 2.4.0 it comprises WSO2 Complex Event Processing features (CEP) that can monitor events real-time, process them and visualize them in a relatively low latency according to [1].
In Google Analytics most analytics and dashboards are available out of the box but with WSO2 BAM you may need to write some hive queries and dashboards to come up with a great solution.
WSO2 BAM is open source (Apache Licences) and you can use it as you wish with great flexibility although it lacks some out of the box features compare to the Google Analytics.
From BAM 2.4.0 it comes with an inbuilt Activity Monitoring feature [2] that is based on the concept of an Activity ID. This can be used out of the box when your business process is properly configured for activity monitoring use case.
[1] https://docs.wso2.org/display/BAM240/Realtime+Analytics
[2] https://docs.wso2.org/display/BAM240/Activity+Monitoring+Dashboard