I have a big Symfony3 project, with a REST API and many webservices.
I'm looking for a bundle, tool, and/or best practices to get usage analytics on my webservices.
Most cases, you don't need special bundles for that. You can analyze your webserver's log. It will be even much easier if you have more than 1 server under load-balancer.
For analytics you can use ElasticSearch + Logstash + Kibana (ELK). There are a lot of tutorials how to set up it on the internet (1, 2, 3)
Related
I have a log analysis tool as a Java based web application with html/jsp front end. Java backend receives real-time log data from few different agents/datasources and does ML/analysis and then stores insights on a file system, accessible only to Java backend. Now my users login to the system using a browser and access Java backend and insights using rest endpoints in the form of HTML/jsp reports/dashboards.
All sounds good but my users are complaining that UI is not production grade. So I am searching for different solutions to build a nicer UI experience for my users and I am looking into Power BI embedded solution.
How to integrate power bi embedded to serve insights that continuously change behind the scenes? how does it work in this use case knowing that my insights are only accessible to my Java backend and can only be served using Rest End points?
Also are there any better alternative solutions?
We are working on a solution built on top of WSO2 APIM.
Problem Statement:
We want to add custom capability in existing APIM. Right now if we consume a single api endpoint it gives us meaningful statics some of them are:
API Usage API
API Usage per Application
Top Users per Application
Resource Usage per Application
But what we need is to store some/all incoming request data into database and extract meaningful information over time.
Example:
Suppose we have a api endpoint like below:
HTTP-Verb: POST
URL: http://localhost:9444/subscriptions/1/create_subscription
Params: name:sample_name, type:sample_type, user_id:12345
What we want?
It could be WSO2, OpenAM or any other opensource API manager.
We need to store parameter values e.g. sample_name, sample_type
and user_id in database.
Observations?
Looks like WSO Stream Processor can provide this functionality.
As it has a solution for HTTP Analytics. But we are not certain
to proceed with this option.
Another possible candidate is to Extend API Manager. But docs doesn't have anything to target what we wanted to do.
I am not sure but we can use RequestInterceptor?
Anyone has a experience with this kind of problem please let me know.
There are two problems you have to solve here,
1. Capture the required data from the APIM side
2. Publish to Analytics Server
3. Perform Analytics on received events
As you have observed WSO2 SP can be used for 3.
For 1 & 2, please see blog (https://medium.com/#naduni_pamudika/how-to-publish-custom-attributes-from-wso2-api-manager-to-wso2-analytics-38dd83e8aea2) for extending the publishers to capture and publish additional data.
You can also take a look at OpenIG, it is easy to extend and you can implement your own filter to store request data in database. Another approach is develop your own gateway based on Netflix Zuul or Spring Cloud Gateway
How can i get twitter information (number of followers, following, etc.) about a set of twitter handles using the Twitter API?
i have already used Python-Twitter library but this only gives me information about my own twitter account, but i need the same for other twitter users (i have a list).
Can you please guide me in the right direction? or refer to some good blogs/articles
If you want the latest tweets from specific users, Twitter offers the Streaming API.
The Streaming API is the real-time sample of the Twitter Firehose. This API is for those developers with data intensive needs. If you're looking to build a data mining product or are interested in analytics research, the Streaming API is most suited for such things.
If you're trying to access old information, the REST API with its severe request limits is the only way to go.
I am newbie here and confused by few things
Some websites (twitter, foursquare, etc) provide API to third-party developer to call. are those APIs the web services that the sites provide?
Are those web sites themselves built on top of those public APIs/web services? theoretically is it possible?
Comparing the traditionally built website and the websites build on top of web service, pros and cons? are there any performance, scalability, etc differences?
Thanks in advance!
I'm sure somebody can give you a more exact answer but reading your question and applying my self-taught knowledge:
The simple technical definition of Web Services according to W3C:
A Web service is a software system designed to support interoperable machine-to-machine interaction over a network.
http://www.w3.org/TR/2004/NOTE-ws-gloss-20040211/
I like to think of web services as the interactive elements of a site that its customer base utilizes. For example, Twitter's web services include: tweeting, messages, hashtags, etc. Web services are what users get to DO or DATA passed back and forth.
A public web API provides means for developers to utilize the web services on their own site. For example, Twitter's API allows example.com site to utilize tweeting, messaging, hashtags, etc from within their own domain. An API is how developers get external access to web services to make apps using those services.
I have no idea about this question. I wouldn't do that. I would use the methods the public API exposes access to. But, I've never written my own API, let alone on the scale of Twitter or foursquare.
I hope this helps.
First of all, maybe you need some more info about what an API is: please take a look at the Wikipedia api page.
To answer to you questions (these are only general thoughts and not best practices):
An API, in this case, is a way that a developer uses to access a webservice, and it's not the service itself.
The websites you mention are not using their own APIs, as these APIs are meant for remote users (clients), and offer limited data sets, while the websites need maximum performance, access to the full database, and (almost) always use server-side code. The websites you mentioned, probably use other, server-side, high-performance APIs.
See the previous point: although it depends highly on which APIs you use, what you call "traditionally built websites" (that is, web applications using server-side APIs) can afford higher performance than websites totally built on top of remote APIs, because they do not depend on the bottleneck of the network connection (because, again usually, the web server and the database server either run on the same machine, or communicate faster than the client's browser and the server).
The reason that would make most people choose to develop a webapp the traditional way is that free APIs provide limited functionality (e.g. Google custom Search, limited to 100 reults).
When I want to find a "Big" Web service (WS-*/WSDL based) of required functionality I can simply google for ".... filetype:wsdl" because Google indexes also WSDL definitions. Or I can simply use some registries that already crawled the Web for WSDL definitions for me, e.g. SeekDa.com or ServiceFinder.
When I want to find RESTful Web service (RESTful Web API) I may count only on the community, because it is not possible to distinguish between URI of RESTful Web service and other Web resources (e.g. Web sites URLs) and hence any focused crawler cannot tell for 100% sure: this is URI of RESTful Web service. So, I come e.g. to the ProgrammableWeb.com and I hope someone already used/found the service I'm looking for and registered it there.
Recently, I was looking for the Web service for abbreviation expansion. I couldn't find it in portals like ProgrammableWeb, because no-one has put it there. Finally I was forced to browse through several abbreviation dictionaries sites, dig and drill there and I finally found Abbreviations.com Web service API in RESTful form. That took me a lot of time!
Is there any other Way to discover RESTful Web services on the Web?
There is a convention (recommended in REST API Design Handbook for example) that you expose a /api endpoint at the root of your service. This returns an XML or JSON response containing the "child" resources that your service supports e.g. /api/products
The REST ideal is that services are as consumable as Web Pages are, they don't need registries. [Got to admit I don't fully "get" the implications of this RESTful world, no WSDL, no registry feels like riding a bike with no stabilisers.]
In the old Web, pre-Google, how did we find pages? Effectively word-of-mouth and a few key starting points. REST services so far as I can see are pretty much in the pre-Google stage.
I don't agree that "it is not possible to distinguish between URI of RESTful Web service and other Web resources" - if we follow the link we get certain types of content application/xml and application/json would be pretty strong indicators wouldn't they?
There's a WSDL-like file for REST webservices: it's called WADL.
SoapUI now can discover the REST services. It works as a proxy, writing down all requests/responses that pass through. Having requests and responses SoapUI recreates descriptions/definitions of the services.
Now the definitions can be stored in WADL and WSDL formats (as WSDL and XML-Schema in XML world). It also can be stored in Swagger format. I prefer Swagger.
Swagger can be stored in SwaggerHub right from SoapUI, which is like GitHub for source code. SwaggerHub is one of many API management systems.