Felipe Hoffa wrote this very helpful guide on how to turn on custom cost control for a project in BigQuery. However, according to the doc, it should be possible to configure custom cost control as user level as well. I really need to do this for my production data warehouse project because I can't let one person's mishap stop all the other users from using the data warehouse. Please help!
Go to console.cloud.google.com > I&AM > Quotas. Then filter by bigquery in the services dropdown. You will find a row like the one bellow:
You are looking to edit the Query usage per day per user. To calculate the number of Bytes you can use a service like: https://convertlive.com/u/convert/terabytes/to/bytes#1
Related
Is it possible to create a bigquery service account to limit access to only 1 dataset? When I go through the service account generation process it appears to give access to an entire project and does not show options to limit to a specific data set.
Short answer is yes. But to do it you do not assign the privileges at the project level. You need to actually go and modify the dataset to do it.
Check the documentation here:
https://cloud.google.com/bigquery/docs/dataset-access-controls
It outlines the process with a few different methods.
I am working on the reporting project that uses PowerBI as the data visualization tool.
I need create a processing approval workflow on the PowerBI tool. After seeing the Dashboard, the employer can approve some exception cases and the workflow can direct connect with email or ticket system.
There are 2 cases:
Approve for the whole dashboard that supports to be easy. I don't have any problem here.
Approve for singular object/row in a table chart. So I must generate number of buttons according to the number of row, which I need help. I don't know how to generate dynamic number of buttons and attached to row. And how to program/code it to create a view or action to become an approval step.
Button PowerBI
In this screenshot, my plan is create buttons in each row and each button has the same function with parameter is username or IP. And after that I can send email to the user and notice him/her that his/her case is approve for exception.
I find this https://community.powerbi.com/t5/Community-Blog/A-simple-and-fun-guide-to-Microsoft-Flow-and-Power-BI/ba-p/151530. But it doesn't seem helpful. Anyone here has ever dealt with approval case like this.
Is PowerBI able to do the approval process like I want?
Thank you so much.
First: This kind of goes against the spirit of BI in general. BI is for data visualization, exploration, etc. It's not really a UI for inserting data and executing tasks. Maybe you want instead to have a front end that lets you do things, and only needs to handle a very limited dataset? PowerApps is good for that. If the dataset is less than 1000 rows, this could work.
Second: I'm pretty sure it's not possible to create a button (like one that you'd see in an HTML page) that does what you want it to do in a Power BI table visual.
Third: There is a "drill through" button capability, but this just lets you navigate from one area in the report to another, not send an email or execute a Power Automate flow or anything like that. You may have seen a button on a table visual, but it's misleading. It's not really programmable like an HTML/JS button on a website.
https://www.c-sharpcorner.com/article/create-a-drill-through-button-in-power-bi/
That said, within the last 1.5 years or so, we now have the PowerApps add-in available. You could create an app that utilizes your streaming dataset, create a gallery that looks at that dataset and creates a kind of table with buttons on it, and then each button is set to execute the flow you've created in Power Automate.
PowerApp Add in chiclet
All of this is very, extremely straightforward, but beware, the PowerApp will start to cost you extra money depending on where your data is housed. If it's a SQL server, you'll need both a premium PowerApp license and Power Automate license too.
Sorry for the not so great news, but this is kind of a limitation of Power BI.
Where/how can I easily see how many BigQuery analysis queries have been run per month. How about storage usage overall/changes-over-time(monthly)?
I've had a quick look at "Monitoring > Dashboards > Bigquery". Is that the best place to explore? It only seems to go back to early October - was that when it was released or does it only display the last X weeks of data? Trying metrics explorer for Queries Count (Metric:bigquery.googleapis.com/job/num_in_flight) was giving me a weird unlabelled y-axis, e.g. a scale of 0 to 0.08? Odd as I expect to see a few hundred queries run per week.
Context: It would be good to have a high level summary of BigQuery, as the the months progress, to give an idea to the wider organisation and management on the scale of usage.
You can track your bytes billed by exporting BigQuery usage logs.
Setup logs export (this is using the Legacy Logs Viewer)
Open Logging -> Logs Viewer
Click Create Sink
Enter "Sink Name"
For "Sink service" choose "BigQuery dataset"
Select your BigQuery dataset to monitor
Create sink
Create sink
Once Logs is enabled, all queries to be executed will store data usage logs in table "cloudaudit_googleapis_com_data_access_YYYYMMDD" under the BigQuery dataset you selected in your sink.
Created cloudaudit_googleapis_com_* tables
Here is a sample query to get bytes used per user
#standardSQL
WITH data as
(
SELECT
protopayload_auditlog.authenticationInfo.principalEmail as principalEmail,
protopayload_auditlog.metadataJson AS metadataJson,
CAST(JSON_EXTRACT_SCALAR(protopayload_auditlog.metadataJson,
"$.jobChange.job.jobStats.queryStats.totalBilledBytes") AS INT64) AS totalBilledBytes,
FROM
`myproject_id.training_big_query.cloudaudit_googleapis_com_data_access_*`
)
SELECT
principalEmail,
SUM(totalBilledBytes) AS billed_bytes
FROM
data
WHERE
JSON_EXTRACT_SCALAR(metadataJson, "$.jobChange.job.jobConfig.type") = "QUERY"
GROUP BY principalEmail
ORDER BY billed_bytes DESC
Query results
NOTES:
You can only track the usage starting at the date when you set up the logs export
Table "cloudaudit_googleapis_com_data_access_YYYYMMDD" is created daily to track all logs
I think Cloud Monitoring is the only place to create and view metrics. If you are not happy with what they provide for BigQuery by default, the only other alternative is to create your own customized carts and dashboards that satisfy your need. You can achieve that using Monitoring Query Language. Using MQL you can achieve the stuff you described in you question. Here are the links for more detailed information.
Introduction to BigQuery monitoring
Introduction to Monitoring Query Language
Is it possible to count number of occurrences of a specific log message over a specific period of time from GCP Stackdriver logging? To answer the question "How many times did this event occur during this time period." Basically I would like the integral of the curve in the chart below.
It doesn't have to be a moving window, this time it's more of a one-time-task. A count-aggregator or similar on the advanced log query would also work if that would be available.
The query looks like this:
(resource.type="container"
logName="projects/xyz-142842/logs/drs"
"Publish Message for updated entity"
) AND (timestamp>="2018-04-25T06:20:53Z" timestamp<="2018-04-26T06:20:53Z")
My log based metric for the graph above looks like this:
My Dashboard is setup like this:
I ended up building stacked bars.
With correct zoom level I can sum up the number of occurrences easy enough. It would have been a nice feature to get the count directly from a graph (the integral), but this works for now.
There are multiple ways to do so, the two that I saw actually working and that can apply to your situation are the following:
Making use of Logs-based Metrics. They can, for example, record the number of log entries containing particular error messages, or they can extract latency information reported in log entries.
Stackdriver Logging logs-based metrics can be one of two metric types: counter or distribution. [...] Counter metrics count the number of log entries matching an advanced logs filter. [...] Distribution metrics accumulate numeric data from log entries matching a filter.
I would advise you to go through the Documentation to check this feature completely cover your use case.
You can export your logs to Big query, once you have them there you can make use of the classical tools like groupby, select and all the tool that BigQuery offers you.
Here you can find a very minimal step to step guide regarding how to export the logs and how to Analyzing Audit Logs Using BigQuery, but I am sure you can find online many resources.
The product and the approaches are really different, I would say that BigQuery is more flexible, but also more complex to be configure and to properly use it. If you find a third better way please update your question with those information.
At first you have to create a metric :
Go to Log explorer.
Type your query
Go to Actions >> Create Metric.
In the monitoring dashboard
Create a chart.
Select the resource and metric.
Go to "Advanced" and provide the details as given below :
Preprocessing step : Rate
Alignment function : count
Alignment period : 1
Alignment unit : minutes
Group by : log
Group by function : count
This will give you the visualisation in a bar chart with count of the desired events.
There is one more option.
You can read your custom metric using Stackdriver Monitoring API ( https://cloud.google.com/monitoring/api/v3/ ) and process it in script with whatever aggregation you need.
If you are working with python - you may look into gcloud python library https://github.com/GoogleCloudPlatform/google-cloud-python/tree/master/monitoring
It will be very simple script and you can stream results of calculation into bigquery table and use it in your dashboard
With PacketAI, you can send logs of arbitrary formats, including from GCP. then the logs dashboard will automatically parse and group into patterns as shown in this video. https://streamable.com/n50kr8
Counts and trends of different log patterns are also displayed
Disclaimer: I work for PacketAI
I'm new to SharePoint but most of it seems pretty straight forward but I've hit a problem and haven't been able to find a way around it so far.
I'm trying to set/enforce Service Level Agreements (SLA's) for different departments based on the department the task is assigned to. I was going to do this based on the workflow status that generates the task but am open to any other suggestions.
My workflow for requesting funds for an approved project goes through several stages (Project management validation, Finance Admin validation; Finance manager validation, Fixed assets authorization) and each one has a slightly different SLA. For this reason, I cant just add an arbitrary value to the start date for the calculated column associated to the task.
Any suggestions?
The option I'd go with here is to use If/Then blocks in my workflow code based on the current stage. Something like this:
If Stage = Project Management Validation Then
Set DueDate to Today+5
Else If Stage = Finance Admin Validation Then
Set DueDate to Today+3
etc.
Hope this helps!