I have to write a report about Cloud Computing and QoS, and I need information about the tools that Providers provide customers to measure the performances of services. Does anyone know if Google provides any such tool?
I apologize for any grammatical errors :)
Thank you for support
It really depends on what you mean by "performance", because there are many aspects, factors, etc. that can all be considered related to performance, including compute, storage, network, boot times, price/performance, etc.
Since your question is generic, you might want to read a couple of references below to get an overview of the types of performance measurements that can be performed and what tools, frameworks, etc. are generally used.
In no particular order, here are some references to tools and APIs from Google:
gsutil (storage) performance diagnostics
Measuring network performance with Resource Timing API
benchmark suites from third parties:
AMPLab Big Data Benchmark
Intel Cloud Object Storage Benchmark
and benchmark results from third parties:
Google Compute Engine benchmarking by Scalr
The Cloud Performance Dashboard: A Quick Market Overview
By the numbers: How Google Compute Engine stacks up to Amazon EC2
MongoDB on Google Compute Engine – tips and benchmarks
Performance analysis of GAE and AWS
Google Compute Engine Performance Test with RightScale and Apica
Measuring and Comparing the Performance of 5 Cloud Platforms
Cassandra performance benchmark by Stackdriver
DataStax Enterprise Testing on Google Compute Engine
Google Compute Engine vs Amazon EC2 Part 2: Synthetic CPU and Memory Benchmarks
Related
so I have these Airflow DAGs which consists of several tasks. Basically each task executes some independent analysis steps against a given code snippet and finally it is decided if this snippet can be used from a regulatory point of view.
Each tasks - depending on the code snippet - is quite short (1-25 minutes at most) and mostly it boils down to executing some external analysis tool (open source and internally) and processing the output of this tool.
All this works quite nice on my development machine but since we are analyzing quite a lot of code snippets during working hours (~50 per hour) and none outside of working hours, I'd like to get all of this up and running somewhere in the cloud (I don't really care if on google cloud, aws or azure).
So my question is what would be an economic way of getting this up and running in the cloud? I thought about using google cloud composer and these google preemptible VMs (the ones that shut down randomly but are super cheap) but it seems that I can not use the PVMs together with cloud composer.
Since the various steps in the DAG are independent the PVMs would be IMO great - if during task execution they are shut down I just retry this one task on a different PVM.
Thank you
On Google Cloud, there are a few options for you.
Run self-managed Airflow on a Compute Engine VM
Run Cloud Composer
The best option will be a mix of how much you want to spend and what features you need. Self-managed Airflow is a great option if you want to have very low cost (less than $100 per month) and are OK self-managing the VM and taking on the risk that the SLA from Google will only cover the VM, so if Airflow malfunctions, you're going to have to detect it and fix it.
The benefit of Composer is the fact that it's integrated so you get things like IAM, Stackdriver, WebUI proxying and so on. You will pay more for the service, however, since it's managed. Presently there is no way to run Composer with preemptible VMs.
Your use case sounds like it could run on a default size cluster on Cloud Composer, though.
It's worth noting that if you go self-managed, you also get the benefit that Google actively contributes to Airflow, so things like the operators should work against the current product APIs. Google also contributes fixes and new operators pretty regularly.
I understand both are built over Jupyter noteboooks but run in cloud. Why do we have two then?
Jupyter is the only thing these two services have in common.
Colaboratory is a tool for education and research. It doesn’t require any setup or other Google products to be used (although notebooks are stored in Google Drive). It’s intended primarily for interactive use and long-running background computations may be stopped. It currently only supports Python.
Cloud Datalab allows you to analyse data using Google Cloud resources. You can take full advantage of scalable services such as BigQuery and Machine Learning Engine to analyse, manipulate and visualise data. You can use it with Python, SQL, and JavaScript.
Google Colaboratory is free. But, you are limited to one spec of cpu/ram/disk/gpu.
Google Datalab is paid. You pay for whatever specs you want.
The notebook interface is also a bit different between the two.
As Google Cloud Prediction API is deprecated (https://cloud.google.com/prediction/docs/end-of-life-faq), does ml-engine provide a similar black-box?
Google Cloud ML Engine is managed TensorFlow and supports higher level APIs (see Datalab notebooks for regression and image classification - runnable in Datalab). Compared to Prediction API, there are some capability differences between the data types and some user experience delta that is being addressed in the near term.
Note that TensorFlow and ML Engine allow you a greater degree of freedom to select and tune the model & much larger scale than a blackbox - albeit with some added complexity at present. That too will be addressed soon.
Dinesh Kulkarni
Product Manager, Google Cloud ML & Datalab
Ho would you do the below
Please share your thoughts.
There are many ways to perform stress test on AWS architecture some of them are Jmeter, Blazemeter etc. Regarding restriction you have to let AWS support know before hand regarding the stress test or penetration test you are going to perform on the AWS infrastructure you have created. Check this link for more details.
https://aws.amazon.com/security/penetration-testing/
Since you pay for bytes into and out of the Amazon infrastructure, to maintain low costs keep your load generators in the same data center as your application under test. There are some drawbacks to this, but the primary one is that your network will lack the complexity of impairment that real users will experience. If you are using a tool which includes modeling of network impairment with the virtual users then this drawback is reduced.
No matter what tool you use, if you have the load generators running on virtual machines in AWS, you will face the issue of clock float on the virtual machine clock. Periodically this virtual clock will need to be resync'd to the system clock on the hypervisor host. This will result in clock jump. This will happen while you have a timing record open - it is unavoidable. The net impact of this is that you will have higher average, percentile values, standard deviation and maximums than if you were running on physical hardware.
I recently read this article about Cassandra performance on three large cloud infrastructures (AWS, GCloud, and Rackspace): http://www.stackdriver.com/cassandra-aws-gce-rackspace/
Google has since published an article claiming 1 million writes/sec but using a setup that most people would never use, being 330 nodes.
Following up on that, they have taken the tuning lessons from that exercise and made a Cassandra 2.0 deploy available: http://googlecloudplatform.blogspot.com/2014/07/click-to-deploy-apache-cassandra-on-google-compute-engine.html
I'm wondering if anyone has more recent benchmarks taking a mixed IO test or had real world experience with a nearly identical deployments on 2 or all 3 infrastructures. What was your experience? What do you think about price/performance for the given infrastructures? How about more subjective issues such as Cassandra manageability/maintainability on each?