I am using the amazon services S3 and CloudFront for a web application and I would like to have various statistics about accessing the data that I am providing through the logs of those services (there is logging activated in both services).
I did a bit of googling and the only thing I could find is how to manage my S3 storage. I also noticed that newrelic offers monitoring for many amazon services but not for those 2.
Is there something that you use? A service that could read my logs periodically and provide me with some nice analytics that would make developers and managers happy?
e.g.
I am trying to avoid writing my own log parsers.
I believe Piwik supports the Amazon S3 log format. Take a look at their demo site to see some example reports.
Well, this may not be what you expect but I use qloudstat for my cloudfront distributions.
The $5 plan covers my needs thats less than a burrito here where I live.
Best regards.
Well, we have a SaaS product Cloudlytics which offers you many reports including, Geo, IP tracking, SPAM, CloudFront cost analysis. You can try it for free for upto 25 MB of logs.
I might be answering this very late. But I have worked on a golang library that can run analysis of CDN and S3 usages and store them in a backend of your choice varying from influxdb, MongoDB or Cassandra for later time series evaluations. The project is hosted at http://github.com/meson10/cdnlysis
See if this fits.
Popular 3rd party analytics packages include S3stat, Cloudlytics and Qloudstat. They all run around $10/month for low traffic sites.
Several stand-alone analytics packages support Amazon's logfile format if you want to download logs each night and feed them in directly. Others might need pre-processing to transform to Combined Logfile Format (CLF) first.
I've written about how to do that here:
https://www.expatsoftware.com/articles/2007/11/roll-your-own-web-stats-for-amazon-s3.html
Related
I have users uploading videos in all kinds of formats, some not supported by chrome.
I just want to transcode all videos so they play in the browser, e.g. h.264.
I have looked at AWS Elastic Media Convert that does not have any documentation that is actually explaining how to go from zero to hero, let alone having meaningful templates for jobs.
Is it at all possible to do that with media convert and if not, what would be an appropriate service?
In general, yes. Existing AWS Blog articles describe workflows using S3 "watch folders" to convert new files as they arrive, using MediaConvert to covert files to a ubiquitous output format and deliver the converted output to a specified S3 location which can support web access both directly and as an origin for CDN distributions.
There are several considerations (for example: very long input files, exotic file formats, nature of content, etc) for which you may want to account. The design and testing of a multi-step workflow with error handling can be complex. If you wish to use outside expertise, you have at least three options:
[a] AWS Paid Professional Services -
There is a large global AWS ProServices team able to help via paid service engagements. The fastest way to start this dialog is by submitting the AWS Sales team 'contact me' form found at this link and specifying 'Sales Support' : https://aws.amazon.com/contact-us/
[b] AWS Certified Consulting Partners -- The partner search tool & listings are here: https://iq.aws.amazon.com/services/aws/medialive
[c] AWS Solutions Architects -- for AWS Enterprise Support customers. The TAM or the Sales contact linked in item [a] is the best way to engage them. Purchasing AWS Enterprise Support will entitle the customer to a dedicated TAM / SA combination.
Here is the documentation for AWS Elemental MediaConvert and following the getting started steps will get you a basic understanding of the service.
If there is any specific issue with the usage of the service, please create a new question.
I am using S3 and cloudfront to deliver videos to my users, however is there a way for me to see how much bandwidth a video is using? I can't seem to see it in the aws panal its self and I've found nothing with a good search and can't see anything in there SDK. I'm posting here just to double check before I give up.
Thanks :)
Edit: I've found that you can see file usage in Cloudfront on Popular objects page and can download it to a CSV but it only shows the most popular 50 items. I also can't find anything about it on the SDK...
I think there is no such report (bandwidth usage per file)
however it should be easy to implement it using s3 events, lambda and dynamodb (with counters)
I operate a number of content websites that have several million user sessions and need a reliable way to monitor some real-time metrics on particular pieces of content (key metrics being: pageviews/unique pageviews over time, unique users, referrers).
The use case here is for the stats to be visible to authors/staff on the site, as well as to act as source data for real-time content popularity algorithms.
We already use Google Analytics, but this does not update quickly enough (4-24 hours depending on traffic volume). Google Analytics does offer a real-time reporting API, but this is currently in closed beta (I have requested access several times, but no joy yet).
New Relic appears to offer a few analytics products, but they are quite expensive ($149/500k pageviews - we have several times this).
Other answers I found on StackOverflow suggest building your own, but this was 3-5 years ago. Any ideas?
Heard some good things about Woopra and they offer 1.2m page views for the same price as Relic.
https://www.woopra.com/pricing/
If that's too expensive then it's live loading your logs and using an elastic search service to read them to get he data you want but you will need access to your logs whilst they are being written to.
A service like Loggly might suit you which would enable you to "live tail" your logs (view whilst being written) but again there is a cost to that.
Failing that you could do something yourself or get someone on freelancer to knock something up for you enabling logs to be read and displayed in a format you recognise.
https://www.portent.com/blog/analytics/how-to-read-a-web-site-log-file.htm
If the metrics that you need to track are just limited to the ones that you have listed (Page Views, Unique Users, Referrers) you may think of collecting the logs of your web servers and using a log analyzer.
There are several free tools available on the Internet to get real-time statistics out of those logs.
Take a look at www.elastic.co, for example.
Hope this helps!
Google Analytics offers real time data viewing now, if that's what you want?
https://support.google.com/analytics/answer/1638635?hl=en
I believe their API is now released as we are now looking at incorporating this!
If you have access to web server logs then you can actually set up Elastic Search as a search engine and along with log parser as Logstash and Kibana as Front end tool for analyzing the data.
For more information: please go through the elastic search link.
Elasticsearch weblink
What is a better mBaaS that supports offline sync and caching?
I am evaluating several mBaaS solutions for my hybrid mobile app under development. I looked at Kinvey, Kii, buddy, and Telerik BackEnd platform. I have also came across some open source solutions like openmobster and dreamfactory. I am looking to store data in sql-lite on mobile app and then sync it back with an online data store. Kinvey has this support, but their pricing model (per user) is not suitable in my scenario. I can see that openmobster does this but, how is what I need to understand? Can I host in on Azure VM or something? Also please suggest if there is any other solution commercial/open source capable of doing offline sync and caching with push notifications and data storage?
DreamFactory could be a good fit for your scenario. It is open source and comes with a full 30 days of free support. After which it's only like $25/month for a developer account - and this isn't even a requirement to use its product. It's specifically a support package.
To address your question a little more in-depth... I don't believe DreamFactory supports offline syncing at the moment, though they plan to very soon. In regards to sql-lite, DreamFactory's (DSP) product has a built in sql-lite driver to connect to that DB. However, it hasn't been tested enough for them to say it is a fully supported RDBMS. One of the beautiful things about DreamFactory is you're able to host the DSP (DreamFactory Service Platform) on Azure and Amazon EC2 instances (cloud solutions), host locally on your own server, or even use its own free hosted edition!
I would definitely take a little time to look into DF. It doesn't seem to me like you have much to lose. Especially, considering it's a free open-source product!
Feel free to ask me any questions you may have about DreamFactory!
-Mark
Is there any tools that can be used to monitor SOAP, REST services for basic as well as service uptime etc.,
I keep track of my systems with nagios. Basically I scripted some queries to
the relevant services so that I get warnings if they don't respond properly / in reasonable time.
In a development environment I use either the TcpMonitor that came with Apache Axis or WireShark which is less invasive in the sense that you don't have to change port numbers or hostnames in your configuration.
Also, for services implemented via Java servlets, I have a filter that I can put in front of the service which logs the payload of each request and response.
I've seen Nagios used successfully in the past.
If you want only to monitor my suggestion is nagios (just see the other answers)
But if you want a tool to display performance data on graphs, a good one is Cacti, it uses RRDTool and can help you generating graphs.
It keeps a historical graphing data.