Is there a way to list all projects in an account and sort those projects by the last date they had any activity?
Related
Problem Statement
We're a large organisation (7000+ people) with many BigQuery projects. My team own a highly used set of approx 250 tables. We are aware of some data quality issues, but need to prioritise which tables we focus our efforts on.
In order to prioritise our effort, we plan to calculate two metrics for each table:
Monthly total count of query jobs referencing that table
Total number of distinct destination tables referencing that table
However, we are stuck on one aspect -- how do you access all the query jobs across the entire org that reference a specific table?
What we've tried
We've tried using the following query to find all query jobs referencing a table:
select count(*) from
`project-a`.`region-qualifier`.INFORMATION_SCHEMA.JOBS
where job_type = 'QUERY'
and referenced_tables.project_id = 'project-a'
and referenced_tables.dataset_id = 'dataset-b'
and referenced_tables.table_id = 'table-c'
Unfortunately, this is only showing query jobs that are kicked off with project-a as the billing project (afaik).
Summary
Imagine we have 50+ GCP projects that could be executing queries referencing a table we own, what we want is to see ALL those query jobs across all those projects.
Currently it's not possible to access all the query jobs across the entire organization that reference a specific table.
As you have mentioned you can list query jobs within the project using the query as:
select * from `PROJECT_ID`.`region-REGION_NAME`.INFORMATION_SCHEMA.JOBS
where job_type = 'QUERY'
PROJECT_ID is the ID of your Cloud project. If not specified, the default project is used.
You can use the query without the project-id as:
select * from `region-REGION_NAME`.INFORMATION_SCHEMA.JOBS
where job_type = 'QUERY'
For more information you can refer to this document.
If you want the feature to list query jobs across the entire organization be implemented, you can open a new feature request on the issue tracker describing your requirement.
Turns out that you can get this information through Google Cloud Logging.
The following command extracted the logs of all queries across the org referencing tables within <DATASET_ID>.
gcloud logging read 'timestamp >= "2022-09-01T00:00:00Z" AND resource.type=bigquery_dataset AND resource.labels.dataset_id=<DATASET_ID> AND severity=INFO'
Important that this command needs to be run from the project on which <DATASET_ID> exists and you need the roles/logging.admin role.
Worth to note that I was not able to test the INFORMATION_SCHEMA.JOBS_BY_ORGANIZATION which should do the trick.
I'm making a Utilization file for our team.
I'm having a bit of difficulty in identifying what kind of workflow that the agent did that day.
I need to identify first the workflow done by that agent for a specific day because each workflow has a different AHT (average handling time) for the computation of their capacity for that day.
I have this file where
Column A = agent's name
Column B = date
Column C = workflow
is there a way to identify the workflows that the agent did that day?
note: there are agents that are working with different workflows each day.
Here's a sample of what I was trying to do.
Sample 2
try:
=IF((I2="")*(I3=""),,UNIQUE(IFERROR(FILTER(D2:D, B2:B=I2, C2:C=I3), "no data")))
spreadsheet demo
Basically I am trying to setup Google Cloud Billing Report Demo for multiple projects.
Example mentioned in this link
In it there are 3 steps to configure datasource for data studio
Create the Billing Export Data Source
Create the Spending Trends Data Source
Create the BigQuery Audit Data Source
Now 1st point is quite clear.
For 2nd point the query example which is provided in demo is based on a single project. In my case I wanted to have spending datasource from multiple projects.
Does doing UNION of query based on each project works in this case?
For 3rd point, I need Bigquery Audit log from all my projects. I thought setting the external single dataset sink as shown below for bigquery in all my project should be able to do the needful.
bigquery.googleapis.com/projects/myorg-project/datasets/myorg_cloud_costs
But I see that in my dataset tables are creating with a suffix _(1) as shown below
cloudaudit_googleapis_com_activity_ (1)
cloudaudit_googleapis_com_data_access_ (1)
and these tables doesn't contain any data despite running bigquery queries in all projects multiple times.In fact it shows below error on previewing.
Unable to find table: myorg-project:cloud_costs.cloudaudit_googleapis_com_activity_20190113
I think auto generated name with suffix _ (1) is causing some issue and because of that data is also not getting populated.
I believe there should be a very simple solution for it, but it just I am not able to think in correct way.
Can somebody please provide some information on how to solve 2nd and 3rd requirement for multiple projects in gcp datastudio billing report demo?
For 2nd point the query example which is provided in demo is based on
a single project. In my case I wanted to have spending datasource from
multiple projects. Does doing UNION of query based on each project
works in this case?
That project is the project you specify for the bulling audit logs in BigQuery. The logs are attached to the billing account, which can contain multiple projects underneath it. All projects in the billing account will be captured in the logs - more specifically, the column project.id.
For 3rd point, I need Bigquery Audit log from all my projects. I
thought setting the external single dataset sink as shown below for
bigquery in all my project should be able to do the needful.
You use the includeChildren property. See here. If you don't have an organisation or use folders, then you will need to create a sink per project and point it at the dataset in BigQuery where you want all the logs to go. You can script this up using the gcloud tool. It's easy.
I think auto generated name with suffix _ (1) is causing some issue and because of that data is also not getting populated.
The suffix normal. Also, it can take a few hours for your logs/sinks to start flowing.
I have the following structure in my Sharepoint:
Parent Site (site collection)
Project A (subsite)
Orders List
Project B (subsite)
Orders List
Project C (subsite)
Orders List
Orders List is the same for all subsites.
I'd like to create a single workflow that will work for all Orders List.
If I create a workflow for each list on each subsite, maintenance will be hard.. Will have to update all workflows separately once one is changed.
I've looked into workflow attached to content types but it doesn't work in SP 2013 Workflow.
What is the best way I can do this? Maybe create a workflow on the parent site that works for all subsites calling same list.
I'm working on a small project in Salesforce where a customer can be the master in a master-detail relationship to more than a single project object.
Projects have a currency 'Max_Budget__c' and what I want to do is calculate the max budget per customer by adding the Max_Budget__c per project. This means that I need to collect a List per customer but also need customer stored so I can update the Max_budget__c object of it.
I was thinking about storing it in a Map with Customer__c as key and Set as value, But I'm not sure how to do this in a single SOQL Query or if i'm even looking in the right direction. Any ideas?
Thanks in advance
Let suppose you have multiple projects like project1, project2 etc .
Each has many to one relationship with customer.
i.e many project1 records to single customer record.
many project2 records to single customer record and so on.
so each customer would have many project1 records,project2 records etc..
To calculate max budget per customer you need to calculate max budget from all project1 records,max budget from all project2 records etc..from each customer and add.
So the requirement is simple write a roll up summary on each child(each project) and add all roll up summaries into a custom field.
I hope this will help you or please let me know if anything else needed.
Regards,
Naveen
Automated deployments , Dataloader , Sandbox back-ups, test automation for Salesforce applications