I added resource labels to a few VMs to be able to pull a more granular billing breakdown by label. However, when I go to the billing report, I don't see any option to filter by Label. Is this a permission issue or am I missing something?
If I embed "label=" in the url, the label option will show, but it still doesn't retrieve the matching key pair.
As per my analysis your issue can be due to below reasons :
As per the official doc it says that
When filtering your billing breakdown by label keys, you are not able
to select labels applied to a project. You can select other
user-created labels that you set up and applied to Google Cloud
services.
This might be the reason you are unable to filter the label.
Google does not recommend creating large numbers of unique labels,
such as for timestamps or individual values for every API call. Refer
to these common use cases for labels and Refer this link for
requirements of label.
You need to enable “resourcemanager.projects.get “ permissions and
also enable “resourcemanager.projects.update” to add or modify the
label.
Refer to this link to create the label.
Related
In Google Cloud Platform, you can add labels to several resources and also add labels to the query jobs you execute. I did this second option. A typical code looks like this:
bq query --label=my_label:{parameter} --label=my_label2:{parameter2} --format=json --use_legacy_sql=false '{query}'"
But, by mistake, the first time I did like this:
bq query --label=my_label{parameter} --label=my_label2:{parameter2} --format=json --use_legacy_sql=false '{query}'"
which created several jobs (I regularly ran this command) having a label named my_labelFoo with an empty value instead of a label named my_label with a value of Foo. This was detected when, in the Billing UI, we noticed several labels as options for filtering, being all of them:
my_labelFoo
my_labelBar
my_labelBaz
my_labelJohn
my_labelGeorge
my_labelRingo
my_labelPaul
...
What I tried to do, then, is to delete the metadata of those wrong jobs. So I tried this query in BigQuery (having the appropriate permissions):
SELECT job_id, query, labels FROM `my-project`.`region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT WHERE ARRAY_LENGTH(labels) > 0 AND EXISTS(SELECT * FROM UNNEST(labels) l WHERE l.key = 'my_labelRingo')"
For each job_id retrieved this way, I tried invoking:
from google.cloud.bigquery import Client
Client().delete_job_metadata(job_id, location="us")
What I can say for good, is that the job entries were removed (they were a few amount), but...
...when I go back to the Billing UI, I still see the my_labelRingo as a selectable label from there. I don't want that label to exist anymore.
So, my question is:
How do I delete the wrong labels from the Billing UI?
Is there, perhaps, a time I have to wait for my_labelRingo to cease to exist?
The situation you are experiencing with the labels in the Billing console is something specific to Cloud Billing Support and you will need to directly engage them using this link
so they can fully investigate why it is happening.
The solutions shared below are different alternatives to delete labels in the GCP BigQuery console.
You can delete a table or view label in the following ways:
Using the console
Using SQL DDL statements
Using the bq command-line tool's bq update command
Calling the tables.patch API method
Because views are treated like table resources, tables.patch is used to modify both views and tables.
Using the client libraries
But you would need to have the next permissions:
bigquery.tables.get
bigquery.tables.update
For example to delete a label through the console you need to follow the next steps:
On the console, select the dataset you want to edit.
Click on the details tab and then click the pencil icon to the right of labels.
On the edit labels dialog
For each label you want to delete, click delete (X)
Click on update to save the changes,
Also you can see more ways to delete labels.
I'm building a custom report in GA UA to check which pages aren't passing specific custom dimensions. The Dimensions of the report are:
Hostname
Page
The metrics are:
Pageviews
Sessions
Users
What I'm expecting is to be able to add a filter with the following syntax to only get pages where the custom dimension isn't populating:
Exclude [Custom_dimension] Regex .*
When I do this however, the report excludes all sessions. If I change from exclude to include though, I only get the subset of sessions where it's populating, roughly 30% of sessions.
Is there a way to get the 70% of sessions not covered by [Custom_dimension] by changing the regex?
GA explicitly doesn't show rows in custom reports for any instances where any of the dimensions is not set. While it's definitely an odd solution on GA's part, you still can do a few things.
You can view, say, a default page report and drop a secondary hostname dimension there and then see where it's not set, or even try to filter those with advanced filter's regex.
Or you can export some of the data in BQ and inspect it there with no UI limitations.
Or you can export the data in an excel or google sheet to analyze it there. there are free options.
GCP provides a mechanism to export billing data to BigQuery. This is really helpful but what it lacks is to provide cost line items at the Instance level (or at least I could not figure out a way). We can get cost aggregates at SKU, Project, Service level, but more granularity is required. This is very much possible with Azure and AWS.
Following are the columns I see in the exported BigQuery Billing table;
billing_account_id, invoice.month, cost_type, service.id, service.description, sku.id, sku.description, usage_start_time, usage_end_time, project.id, project.name, project.ancestry_numbers, project.labels.key, project.labels.value, labels.key, labels.value, system_labels.key, system_labels.value, location.location, location.country, location.region, location.zone, cost, currency, currency_conversion_rate, , usage.amount, usage.unit, usage.amount_in_pricing_units, usage.pricing_unit, credits.name, credits.amount, export_time
Is there a workaround to fetch cost aggregates at Instance level?
Example: If I have subscribed for 2 Compute Engines of a specific SKU. Is there a mechanism to get cost aggregates for each Compute Engine separately?
At the moment its not possible to filter your reports in instance level and SKU is the most granular filter.
An approach you can use to identify your instances and a get a better understanding of your data is using labels. As you can see here :
A label is a key-value pair that helps you organize your Google Cloud
instances. You can attach a label to each resource, then filter the
resources based on their labels. Information about labels is forwarded
to the billing system, so you can break down your billing charges by
label.
In this document which explains the billing data table's schema you can see that the labels attached in your resource will be present in your data.
Basically I am trying to setup Google Cloud Billing Report Demo for multiple projects.
Example mentioned in this link
In it there are 3 steps to configure datasource for data studio
Create the Billing Export Data Source
Create the Spending Trends Data Source
Create the BigQuery Audit Data Source
Now 1st point is quite clear.
For 2nd point the query example which is provided in demo is based on a single project. In my case I wanted to have spending datasource from multiple projects.
Does doing UNION of query based on each project works in this case?
For 3rd point, I need Bigquery Audit log from all my projects. I thought setting the external single dataset sink as shown below for bigquery in all my project should be able to do the needful.
bigquery.googleapis.com/projects/myorg-project/datasets/myorg_cloud_costs
But I see that in my dataset tables are creating with a suffix _(1) as shown below
cloudaudit_googleapis_com_activity_ (1)
cloudaudit_googleapis_com_data_access_ (1)
and these tables doesn't contain any data despite running bigquery queries in all projects multiple times.In fact it shows below error on previewing.
Unable to find table: myorg-project:cloud_costs.cloudaudit_googleapis_com_activity_20190113
I think auto generated name with suffix _ (1) is causing some issue and because of that data is also not getting populated.
I believe there should be a very simple solution for it, but it just I am not able to think in correct way.
Can somebody please provide some information on how to solve 2nd and 3rd requirement for multiple projects in gcp datastudio billing report demo?
For 2nd point the query example which is provided in demo is based on
a single project. In my case I wanted to have spending datasource from
multiple projects. Does doing UNION of query based on each project
works in this case?
That project is the project you specify for the bulling audit logs in BigQuery. The logs are attached to the billing account, which can contain multiple projects underneath it. All projects in the billing account will be captured in the logs - more specifically, the column project.id.
For 3rd point, I need Bigquery Audit log from all my projects. I
thought setting the external single dataset sink as shown below for
bigquery in all my project should be able to do the needful.
You use the includeChildren property. See here. If you don't have an organisation or use folders, then you will need to create a sink per project and point it at the dataset in BigQuery where you want all the logs to go. You can script this up using the gcloud tool. It's easy.
I think auto generated name with suffix _ (1) is causing some issue and because of that data is also not getting populated.
The suffix normal. Also, it can take a few hours for your logs/sinks to start flowing.
Is it possible to share a dashboard chart and keep its configuration?
Whenever I try it the resulting iFrame just shows the default configuration for that metric. Aggregations, type of chart, etc, are gone.
I could only find URL parameters to enable auto refresh (isAutoRefresh) and changing the timeframe.
You can share a chart of a custom dashboard by following these instructions.
If you have already followed the instructions provided and it is not working as expected, I would then suggest to file a PIT here. You can file a PIT if you have encountered unexpected behavior with one of GCP's products or to request for a new feature to be implemented within GCP.