We have several systems that are used by a few hundred users. We would like to collect insights that cannot be collected by Google Analytics.
For example, we want to store data such as Customer X, click on button Y, in system Z, at time YYYY-MM-DD:HH:mm:ss
We would like to:
Show a summary of metrics on a dashboard. e.g. live stats on a Grafana dashboard
Pull detailed reports e.g. Top 5 users that use feature X in system Y (and their usage)
Is there an insight system that I can use and avoid building it from scratch? If I have to build this what is the best way suggested? I am thinking My SQL or SQL Server can be good candidates for the database as Grafana has built-in support for them.
Related
I have a product for which I would like to create a dashboard to show
its availability/uptime over time and display any outages.
Specifically I am looking for
ability to report historical information on service uptime
provide details on any service outages
The product is running on a fleet of linux servers and connects to a DB running
on a separate instance, also we have some dedicated instances that run nightly
batch jobs. My system also relies on some external services to provide
additional functionality for select customers. There is redis cache also for
caching data for multiple customers.
We replicate all the above setup (application servers, DB, jobs servers, redis
cache etc) into dedicated clusters for large customers. Small customers are put
on one of the shared clusters to keep costs low.
Currently we are running health checks on application servers only and providing
that information in a simple HTML page. This is a go to page for end-users/customers
and support teams.
Since the product is constructed using multiple systems/services our current HTML
page often times says that the system is up and running fine while can be experiencing
issues with some of its components or external services.
Current health check is using a simple HTTP request and looks for a 200
status code, this check runs every minute and we plot this data into a simple
chart to show last 30 days. We also show a list of outages with timestamp and
additional static information that is manually added.
We would like to build a more robust solution that monitors much more than the HTTP port
and where we have more details like what part
of the system is having issues and how those issues are impacting the system and
which customers are impacted.
Appreciate any guidance or help. We prefer to build the solution using
open source tools since we dont have much budget. Goal is to improve things for
my team members who are already overloaded.
I'm not sure if this will be overkill or not for your setup, given that I don't know your product, but have a look at the ELK Stack and see if you can use some components or at least some ideas from there:
What is the ELK Stack?
The Complete Guide to the ELK Stack
Our product requires support for a high level dashboard highlighting metrics pertaining to the business modeled by the product. Its a B2B product with a web application as a front-end.
We use Amazon RDS to store business data. What is the best approach to build a customizable dashboard in AWS?
The traditional approach would be to create metrics, process business data (analysis) and store the metrics/results in output tables. And then use a fancy charts library in the web application to build a support a panel/chart/graphic view per such metric. Support drill downs etc.
However, I believe this approach is outdated and too much work. Are there any readmade solutions available? Ideally, I should just be able to push results data to a third party solution, and the third party solution should allow for creation of custom dashboards (custom implies that the user can choose to select particular metrics/panels he would like to see in the dashboard, and their relative order& position in the dashboard) and embedding them in the products web application, to enable to end user to view as well as customize dashboards.
I'm aware of AWS quicksight, but it does not support embedding output views/dashboards into a web application.
I like Azure Power BI; At least you get to embed the dashboard in the web application, but I do not believe the end user can customize the embedded view. And ofcourse, Azure PowerBI is not on AWS ;).
Is there a mature 3rd party solution that we can evaluate? The scale of the data is very very small. So we are not looking for a high performance enterprise solution (might be too expensive anyway!). However, the need for customizability of dashboards view is high.
Embedded dashboards were added to QuickSight in November 2018, so you can now use that tool if you'd prefer.
https://docs.aws.amazon.com/quicksight/latest/user/embedded-dashboards-setup.html
Tableau is the first possibility, but it is expensive.
Infogram may also be an option: https://infogram.com/examples/dashboards
If you have small data volumes and you want cusomisation you may unfortunately have to look into writing or at least customising an existing solution.
I operate a number of content websites that have several million user sessions and need a reliable way to monitor some real-time metrics on particular pieces of content (key metrics being: pageviews/unique pageviews over time, unique users, referrers).
The use case here is for the stats to be visible to authors/staff on the site, as well as to act as source data for real-time content popularity algorithms.
We already use Google Analytics, but this does not update quickly enough (4-24 hours depending on traffic volume). Google Analytics does offer a real-time reporting API, but this is currently in closed beta (I have requested access several times, but no joy yet).
New Relic appears to offer a few analytics products, but they are quite expensive ($149/500k pageviews - we have several times this).
Other answers I found on StackOverflow suggest building your own, but this was 3-5 years ago. Any ideas?
Heard some good things about Woopra and they offer 1.2m page views for the same price as Relic.
https://www.woopra.com/pricing/
If that's too expensive then it's live loading your logs and using an elastic search service to read them to get he data you want but you will need access to your logs whilst they are being written to.
A service like Loggly might suit you which would enable you to "live tail" your logs (view whilst being written) but again there is a cost to that.
Failing that you could do something yourself or get someone on freelancer to knock something up for you enabling logs to be read and displayed in a format you recognise.
https://www.portent.com/blog/analytics/how-to-read-a-web-site-log-file.htm
If the metrics that you need to track are just limited to the ones that you have listed (Page Views, Unique Users, Referrers) you may think of collecting the logs of your web servers and using a log analyzer.
There are several free tools available on the Internet to get real-time statistics out of those logs.
Take a look at www.elastic.co, for example.
Hope this helps!
Google Analytics offers real time data viewing now, if that's what you want?
https://support.google.com/analytics/answer/1638635?hl=en
I believe their API is now released as we are now looking at incorporating this!
If you have access to web server logs then you can actually set up Elastic Search as a search engine and along with log parser as Logstash and Kibana as Front end tool for analyzing the data.
For more information: please go through the elastic search link.
Elasticsearch weblink
I would like to build an app and collect some events from the app, and then show some event statistics like frequency, duration etc.
I`ve just investigated the aws Cognito web service, but it stores only a set of key-value pairs of a limited total size.
I can build, of course, my own REST web service on the top of the database and store all my events there. But I wonder if there are some aws web service(s) that I can leverage to build such a solution. (In case if someone familiar with Azure, it would be nice to see the possible solution there too!)
Any ideas, suggestions?
Haven't used any packaged web service for this; however, I do use REST methods for statistics in my apps and find it works well....low overhead and easy to add, change and collect.
I would suggest you to have a look at AWS Mobile Analytics service (http://aws.amazon.com/mobileanalytics/)
Have a look at the Getting Started page http://aws.amazon.com/mobileanalytics/getting-started/
Seb
Nowadays a lot of web applications are providing API for other applications to use.
I am new to the usage of API so I want to understand the use cases for it.
Lets take Basecamp as an example.
What are the use cases for using their API in my web application?
For inserting current data in my web application into a newly created Basecamp account instead of inserting everything manually which could take days or weeks if the data is huge?
For updating my application data when the user changes something in Basecamp. If so, how do I know for example when a user add/edit/remove a contact in Basecamp. Do I make a request and check every minute from the backend?
For making backup of the Basecamp data so I can move it to other applications if necessary?
Are all the above examples good use cases for the usage of API?
Are there more use cases?
I want to have a clear picture of why it's good to use another web service API and how I can leverage that on my application.
Thanks.
I've found the biggest reason to use and provide web services is to be able to programmatically drive the application with another process. This allows the coupling of different actions in different applications driven by one event/process/trigger.
For example I could create a use a webservice provided by Basecamp, my bug tracking database and the continuous integration server. I could tie all those things together and kick them off from a commit hook script.
I can have a monitor in production automatically open a ticket in our ticket tracker. This could trigger an autoremediation process from the ticket tracker which logs into the box remotely and restarts the service.
The other major reason I've seen to use and provide web service is to reduce double entry. If you do change management in your production environment that usually means you create Change tickets. The changes that occur may also need to be reflected in the Change Management Database which is usually a model of how production is suppose to look. Most of these systems don't automatically drive the update of your configuration item with the data from the change. Using web services you can stitch them together to eliminate the double (manual) entry that would normally occur.
APIs are used any time you want to get data to/from an application without using the default interface.
*I'd bet there's a mobile app would use the basecamp api.
*You could use the api to pull information from basecamp into another application (like project manager software or an individual's todo webpage)
*the geekiest of us may prefer to update basecamp from a script/command line rather than interrupting our work flow to open a web page and click around.