We are working on a solution built on top of WSO2 APIM.
Problem Statement:
We want to add custom capability in existing APIM. Right now if we consume a single api endpoint it gives us meaningful statics some of them are:
API Usage API
API Usage per Application
Top Users per Application
Resource Usage per Application
But what we need is to store some/all incoming request data into database and extract meaningful information over time.
Example:
Suppose we have a api endpoint like below:
HTTP-Verb: POST
URL: http://localhost:9444/subscriptions/1/create_subscription
Params: name:sample_name, type:sample_type, user_id:12345
What we want?
It could be WSO2, OpenAM or any other opensource API manager.
We need to store parameter values e.g. sample_name, sample_type
and user_id in database.
Observations?
Looks like WSO Stream Processor can provide this functionality.
As it has a solution for HTTP Analytics. But we are not certain
to proceed with this option.
Another possible candidate is to Extend API Manager. But docs doesn't have anything to target what we wanted to do.
I am not sure but we can use RequestInterceptor?
Anyone has a experience with this kind of problem please let me know.
There are two problems you have to solve here,
1. Capture the required data from the APIM side
2. Publish to Analytics Server
3. Perform Analytics on received events
As you have observed WSO2 SP can be used for 3.
For 1 & 2, please see blog (https://medium.com/#naduni_pamudika/how-to-publish-custom-attributes-from-wso2-api-manager-to-wso2-analytics-38dd83e8aea2) for extending the publishers to capture and publish additional data.
You can also take a look at OpenIG, it is easy to extend and you can implement your own filter to store request data in database. Another approach is develop your own gateway based on Netflix Zuul or Spring Cloud Gateway
Related
I am absolutly new in the WSO2 world and I have the following problem.
I have to developer a toolkit application and the customer says to me that, if possible, they want introduce WSO2 technologies in this project. I will try to explain what exactly this toolkit project have to do to try to understand if and what WSO2 technologies eventually can match with the requirement.
The toolkint project have to do:
1) When executed have to connect to a local database and perform some queries.
2) Use the data obtained from the previous query to create an XML document.
3) Send this XML document into a request to a web service (I think that the content of this XML will be put into the request body of the sended request).
4) Use the web service response to update some data in the local database.
So I am trying to understand if there are some WSO2 technologies that match these requirements or if it is better to create this toolkit as a simple standard Java program.
Searching online I have identified WSO2 Enterprise Integration and WSO2 Data Services Server (that I think it is included into WSO2 Enterprise Integration).
At the beginning I had thought to use Data Services Server to access to the DB, perform the queries and so retrieve data from the DB tables.
But reading the official documentation (https://docs.wso2.com/display/DSS351/About+Data+Services+Server) it seems to me that it is more a technology thinked to expose a DB using web service, infact in this documentation I can read:
Data services provide a convenient mechanism to configure a Web
service interface for data in various datasources such as relational
databases, CSV files, Microsoft Excel sheets, Google spreadsheets
etc...
Infact reading this official tutorial: https://docs.wso2.com/display/DSS351/Quick+Start+Guide
it seems to me that if does the opposite of what I have to do.
IT DOES:
SET THE DB CONNECTION.
PERFORM THE QUERY.
EXPOSE THE DATA RETURNED BY THIS QUERY AS A WEB SERVICE.
I HAVE TO DO:
SET THE DB CONNECTION.
PERFORM THE QUERY.
SEND THE OBTAINED DATA TO A WEB SERVICE (in XML format).
So it seems to me that Data Services Server is not the right choice because it seems to me that more than help me to do data access, data manipulation and send these data to an endpoing it is useful to do data access and to expose these data as a web service (but maybe I am missing something).
So I am trying to understand if WSO2 Enterprise Integration contains some modules that better fulfills my requirements.
Reading the official documentation (https://docs.wso2.com/display/EI600/Enterprise+Integration+Patterns) it seems to me that I can use:
1) BPEL/BPMN process to define the operations flow.
2) It seems to me that I can do DB queries but I always have the same doubt (that it is used to expose the retrieved data as service and not to use internally, it seems to me that this is WSO2 Data Services Server putted into WSO2 Enterprise Integration).
3) I think that this module provides me a way to aggragate data from some queries into an XML document: https://docs.wso2.com/display/EI600/Transforming+Message+Content
4) I can send a request with the previous generated XML usingsomething like this: https://docs.wso2.com/display/EI600/Sending+a+Simple+Message
So can you help me to identify what WSO2 techonlogies I can use to implement something like this or if it is better to implement it manually in pure Java?
1) When executed have to connect to a local database and perform some queries.
R:/For this point you can create a data service using DSS or EI.
2) Use the data obtained from the previous query to create an XML document.
R:/You can create a proxy service using the ESB or EI, inside the proxy service you call the data service created in the previous point(using call mediator) and get the response message in a property mediator to create a new payload message(using payload, enrich or/and a xslt mediator).
3) Send this XML document into a request to a web service (I think that the content of this XML will be put into the request body of the sended request).
R:/Inside the proxy service created in the previous point and after the creation of the new payload message you can send that payload to another webservice using call mediator.
4) Use the web service response to update some data in the local database.
R:/ You can get the response message, create a new payload message and call another data service, or another operation of the same data service created in the first point.
UPDATE 1:
you can use the IDE or the web UI, as you like. for the implementation check this out: Combine two queries using WSO2 ESB as a start.
Inside the IDE you can go to Developer Studio menu, clic in OpenDashboard and create a "ESB Solution Project" or a "ESB Config Project" for the ESB stuff, and a Data Service Project for the data service.
Our employee salary data are stored in the cloud provided by the third party. Currently, the third party need to send us a copy of the database to us and we recover it to our local server for reporting purpose. I need to build a application to retrieve the data directly from the cloud and use the only tables we need for report purpose. the third party provide restful service to do this, but since I am new to web service programming, I am kind of lost here to decide what to do. So my question is what is the best way that i could proceed this.
Any help will be appreciated.
The first place to start is to familiarize yourself with a coding language you'd like to use. You can usually find documentation from the REST API provider on the 'developer' section of their website. Once on their site, you can usually setup a developer sandbox account and grab a few quick examples from the documentation in the most popular languages i.e Java, PHP etc. Then, using the documentation, tweak the example to get the data you want for your app.
It is common for REST APIs to assign you a key/token that you will use with each request. Refer to their documentation for more examples.
My requirement is - I have an application and need to get product details from amazon, ebay & few more websites for a product. Please let me know the process for getting the product info & also let me know how could i achieve this using C/C++ language. Does amazon support soap still? some where i saw its not supported. If soap is not supported then whats the next alternative?
You can use eBay API, and amazon API, and if you want to compare data and categories between the two (amazon and eBay), you can Try using http://www.ecommerceapi.org
it's an API that combine between them
For example if you have one categoryId of eBay, Ecommerceapi can tell you what is the match for it in amazon
What ever you choose depends on your requirements of the project, API is simple technology that allows you to query the source over http.
I am working on WSO2 ESB 4.0.3 on MAC OS X Lion (10.7.4)
I would like to know what are the best practices for development for WSO2 ESB 4.0.3.
Currently I am using Data Services Feature in it and existing tomcat application, which we are trying to port to WSO2 ESB, does the SQL query in 2-3 seconds where as WSO2 ESB 4.0.3 with Data Services feature taking around 16-17 secodns.
I would be thankful if some body can let me know best practices for WSO2 and in perticular XSLT transformation.
Hoping for answer.
thanks
Hi Prabath
Here is how my environment is
I am using WSO2 ESB 4.0.3 with Data Services Feature 3.2.2. Proxy service front ends the DS service. Data sources are defined as carbon data sources in datasources.properties.
I tried to run the same service in the WSO2 Data Services Server 2.6.3 and the performance is comparable to what existing tomcat application does but the ESB 4.0.3 with Data Services Feature 3.2.2 takes 8 times more time than tomcat application. Looks like XSLT is not a issue as I thought earlier.
I have all the error handling & input validation in the proxy service which calls this DS.
Also I tried changing it to local for the transport but still same performance issue. Also I have to make sure the format of the forwarded XML is SOAP 12 in the end point definition otherwise proxy service does not forward with local transport.
Can you please suggest so that I can use WSO2 ESB with Data Services Feature 3.2.2 and get comparable performance?
Help really appreciated.
thanks
Abhijit
Hi Prabath
Thanks for reply.
Proxy service validation and transformation is not a problem. Looking at the logs it looks like Data Service deployed in ESB with Data Services feature is taking 8 times more time than the tomcat application. So it is Data Services Feature which is problem I believe and not the proxy service.
Even if we remove the proxy service where you will do the input validations and error handling?
Please let me know.
thanks
Abhijit
Abhijit,
I'm not quite clear of whether this problem is related to executing SQL using dbReport/dbLookup mediators against doing the same thing having data services features installed in ESB OR transforming responses using XSLT at the ESB layer against doing it at the DSS layer.
If it's the former, then you should be able to effectively use the db mediator pair (namely dbLookup and dbreport) to execute simple SQL queries such as SELECT, INSERT, UPDATE, DELETE, etc. However, it is not recommended to do use those mediators to do much complex queries such as stored procedures with "OUT and INOUT" parameters etc as WSO2 DSS is specifically designed to serve any sort of complex queries like that. However, this (using data services) comes at the cost of network latency. Because, you're invoking a data service endpoint through the network which obviously adds the network latency to the end-to-end time taken to get your task done. However, if you're using Data Services features installed in the WSO2 ESB, you always have the option of using "local" transport instead of "http/https" which does an in-JVM call and thus would not dispatch the request over the network.
If this is related to the later, meaning, if you refer to the XSLT transformations, I believe there's no such hard and fast rules in doing this and this would completely depend on your requirements and the usecase. For example, if you're only using WSO2 DSS and want to get some request transformed into a particular format that is expected by the client side, it would only be enough for you to get it done at the WSO2 DSS layer. Because, dispatching it into ESB ONLY for the sake of getting the XSLT transformation done, would add an additional unwanted overhead to the end-to-end completion time of your task. On the other hand, if you're doing this as a part of a configuration flow at the ESB side, then it's perfectly okay to use something like XSLT mediator inside the flow itself.
Hope this helps!
Regards.
Prabath
I hope Prabath already gave the answer to your question.
However, it is not recommended to do use those mediators to do much complex queries such as stored procedures with "OUT and INOUT" parameters etc as WSO2 DSS is specifically designed to serve any sort of complex queries like that. However, this (using data services) comes at the cost of network latency. Because, you're invoking a data service endpoint through the network which obviously adds the network latency to the end-to-end time taken to get your task done. However, if you're using Data Services features installed in the WSO2 ESB, you always have the option of using "local" transport instead of "http/https" which does an in-JVM call and thus would not dispatch the request over the network.
I have been reviewing the setup of a current Amazon Web Store implementation to figure out some of the systems capabilities. The end goal is to pull a feed that i can reuse and transform for Channel Adviser. The documentation on Amazon Web Store is a little sparse and I'm wondering if anyone here has a little experience with this specific product scenario (pull inventory from Web Store Account).
The current goal is to pull down the inventory for the store in a feed via RSS or any other suitable format. I have been looking at the RSS web feeds for Tags # amazon.com page but I dont think it's available for the web store product or feasible to pull an entire store's inventory this way.
-my research has led me to the amazon product advertising API but looking at the WSDL for the service, the only relevant service calls I see are ItemSearch, ItemLookup and SimilarityLookup; none of these seem to explicitly support Web Store and they dont really seem entirely appropriate for pulling the Web Store's entire inventory (although they do appear to accept a merchant ID). Does anyone know of a common or appropriate way to get access to this data?
Thanks in advance for any specific guidance you can provide
After a couple of contacts with Amazon Seller support, it appears that the most appropriate solution is to set up an Amazon Marketplace Web Services account and use those apis. The 'Inventory' api seems to be exactly what we were after (although the support email suggested the Reports section).
Response from Amazon Seller Support
We checked and we have a different program that fits your needs called MWS (Market Place Web Service), you would have to sign up for an account first which is for free. Please visit this link to find out more information:
https://developer.amazonservices.com/
Click on reports and then on Amazon MWS Developer Guide (PDF) and you will be able to get more information related to your concern.