Because there is no GCP Monitoring metric that can count the total number of rows in a BigQuery table I want to create a custom metric for that. My goal is to use that metric for visualization in a dashboard.
According to the documentation (https://cloud.google.com/monitoring/custom-metrics/creating-metrics#monitoring_create_metric-java) you can write a java script using the Monitoring API to create a custom metric.
In addition to that I was wondering if I can use the BigQuery API and do a simple sql query to count the number of rows in a BigQuery table and use this as data points for the custom metric.
Is this possible?
Thanks for any answer in advance.
Yes, of course, you can add a side jobs that create these custom metrics. In addition, on BigQuery, when you perform a select count(*) from TABLE, you pay nothing because this data is retrieve from the table metadata.
Related
Please what is the best or recommended way how to visualize data from the DynamoDB table ? We need to create some simple dashboard with graphs connected to data table on the AWS account.
We prefer to use one of the services from AWS to keep everything in one place. I read about the QuickSight but it would be great to know some experience.
You can use quicksight to visualize your table by using the Athena-DynamoDB connector. This will allow you to use DynamoDB as a table source in Athena which can then act as a source for Quicksight.
https://docs.aws.amazon.com/athena/latest/ug/connectors-dynamodb.html
Can Google BigQuery data transfer service allow me to transfer specific app data automatically?
For example, I have 10 apps in my Google play console, I only want to transfer to BQ within only 3 apps. Is it possible to make this work or any approach?
Also, I just read the price of doc, The monthly charge is $25 per unique Package Name in the Installs_country table.
I don't quite understand how to calculate my cost with that example.
Thank you.
For your requirement, you can download the reports in Cloud Storage of a specific app by selecting the app from Google Play Store for which you want to get the data and then send it to BigQuery using BigQuery Data Transfer Service. The cost calculation of Google Play, it is calculated as $25 per month per unique package and stored in the Installs_country table in BigQuery.
For selecting the specific app, follow the steps given below :
Go to the Play Console.
Click on Download Reports and select the type of report you want.
Under "Select an application," type and select the app for which you want to get the data.
Select the year and month for which you want to download the report.
If you are storing data in a Cloud Storage bucket then that will incur cost and the pricing for data transfer from one storage bucket to another storage bucket can be checked in this link and since you are storing and querying in BigQuery that will also be chargeable.For BigQuery pricing details you can check this documentation. You can use the Billing Calculator to calculate your costs.
I am trying to build AWS QuickSight reports using AWS Athena that builds the specific views for said reports. however, I seem to only be able to select a single table in creating the Glue job despite being able to select all tables i need for the crawler of the entire DB from Dynamo.
What is the simplest route to get a complete extract of all tables that is queryable in Athena.
I dont want to connect the reports direct to dynamoDB as it s a production database and want to create some separation to avoid any performance degradation by a poor query etc.
I have a use case where I need to sync spanner table with Big Query tables. So I need to update the Spanner tables based on the updated data in Big Query tables. I am planning to using Cloud data fusion for this. But I do not see any example available for this scenario. Any pointers on this?
I need to store Amazon Athena query results into New Amazon Athena Table.
Updates:
Athena now supports Create Table as Select Queries (CTAS). Examples are available here.
They have implemented several nice feautes, namely the ability to apply compression to outputs (GZIP, SNAPPY) and supply output format.
Recently they have added Create Table as CTAS support to Athena.