I need to list out all the instance, container, function, notebooks, bucket, dataproc and composer running under project in all the region/locations.
Is it possible to list resources of all the regions location. Gcloud or python script both can work for me
My ultimate goal after listing is to put tag as per its name of the resource.
Thanks
You can use Google Asset inventory feature and query your project like this
gcloud asset search-all-resources --scope=projects/<PROJECT_ID> --page-size=500 --format=json
More detail in the documentation about the query format.
All the ressources aren't supported. You can find the full list here (For example, Cloud Run isn't yet supported, but it's coming soon!)
If you want to access through console, you could go to IAM & Admin Menu, then select Asset Inventory.
Then you could see bunch of asset list.
Click Resource tab if you want download all the details in csv format.
In search asset you will get abundance of irrelevant data. Better to use resource api of the resource you think relevant to you. Like
compute.googleapis.com/Instance
storage.googleapis.com/Bucket
dataproc.googleapis.com/Cluster
container.googleapis.com/Cluster
cloudfunctions.googleapis.com/CloudFunction
dataflow.googleapis.com/Job //Notebook
gcloud asset search-all-resources --asset-types='compute.googleapis.com/Instance,storage.googleapis.com/Bucket' --query='labels.name:*' --format='table(name, assetType, labels)'”
Related
How can we get the cluster owner details in GKE. Logging part only contains the entry with service account operations and there is no entry with principal email of userId anywhere.
It seems very difficult to get the name of the user who created the GKE cluster.
we have exported complete json file of logs but did not the user entry who actually click on create cluster button. I think this is very common use case to know GKE cluster creator, not sure if we are missing something.
Query:
resource.type="k8s_cluster"
resource.labels.cluster_name="clusterName"
resource.labels.location="us-central1"
-protoPayload.methodName="io.k8s.core.v1.configmaps.update"
-protoPayload.methodName="io.k8s.coordination.v1.leases.update"
-protoPayload.methodName="io.k8s.core.v1.endpoints.update"
severity=DEFAULT
-protoPayload.authenticationInfo.principalEmail="system:addon-manager"
-protoPayload.methodName="io.k8s.apiserver.flowcontrol.v1beta1.flowschemas.status.patch"
-protoPayload.methodName="io.k8s.certificates.v1.certificatesigningrequests.create"
-protoPayload.methodName="io.k8s.core.v1.resourcequotas.delete"
-protoPayload.methodName="io.k8s.core.v1.pods.create"
-protoPayload.methodName="io.k8s.apiregistration.v1.apiservices.create"
I have referred the link below, but it did not help either.
https://cloud.google.com/blog/products/management-tools/finding-your-gke-logs
Audit Logs and specifically Admin Activity Logs
And, there's a "trick": The activity audit log entries include the API method. You can find the API method that interests you. This isn't super straightforward but it's relatively easy. You can start by scoping to the service. For GKE, the service is container.googleapis.com.
NOTE APIs Explorer and Kubenetes Engine API (but really container.googleapis.com) and projects.locations.clusters.create. The mechanism breaks down a little here as the protoPayload.methodName is a variant of the underlying REST method name.
And so you can use logs explorer with the following very broad query:
logName="projects/{PROJECT}/logs/cloudaudit.googleapis.com%2Factivity"
container.googleapis.com
NOTE replace {PROJECT} with the value.
And then refine this based on what's returned:
logName="projects/{PROJECT}/logs/cloudaudit.googleapis.com%2Factivity"
protoPayload.serviceName="container.googleapis.com"
protoPayload.methodName="google.container.v1beta1.ClusterManager.CreateCluster"
NOTE I mentioned that it isn't super straightforward because, as you can see in the above, I'd used gcloud beta container clusters create and so I need the google.container.v1beta1.ClusterManager.CreateCluster method but, it was easy to determine this from the logs.
And, who dunnit?
protoPayload: {
authenticationInfo: {
principalEmail: "{me}"
}
}
So:
PROJECT="[YOUR-PROJECT]"
FILTER="
logName=\"projects/${PROJECT}/logs/cloudaudit.googleapis.com%2Factivity\"
protoPayload.serviceName=\"container.googleapis.com\"
protoPayload.methodName=\"google.container.v1beta1.ClusterManager.CreateCluster\"
"
gcloud logging read "${FILTER}" \
--project=${PROJECT} \
--format="value(protoPayload.authenticationInfo.principalEmail)"
For those who are looking for a quick answer.
Use the log filter in Logs Explorer & use below to check the creator of the cluster.
resource.type="gke_cluster"
protoPayload.authorizationInfo.permission="container.clusters.create"
resource.labels.cluster_name="your-cluster-name"
From gcloud command, you can get the creation date of the cluster.
gcloud container clusters describe YOUR_CLUSTER_NAME --zone ZONE
I need to find out which resources (Asset-Types) in entire GCP organization can be labelled.
In short, i do not want resources which doesn't have a column Label in the schema. Is there a way to find columns of every asset-type ? or any other way to extract only resources that have column/attribute Label?
gcloud asset search-all-resources --scope=organizations/Org-ID
--filter=-labels:* --format='csv(name, assetType, labels)' --sort-by=name > notLabels.csv
i use this command to get the resources but it returns also the resources that can't be labelled.
You can find the list of services that support labels in GCP in this documentation.
And you can filter it with the following format below as an example:
gcloud asset search-all-resources --filter labels.env:*
The above command lists the services that has env as key and anything that has value on it.
gcloud asset search-all-resources --filter=-labels.*
The second sample command above lists the resources with no labels value by adding - before the label parameter.
You can find more information on using filter searches using labels here.
I have a large number of projects to handle on Google Cloud Platform. To clean them up, I want to pull a list of all projects incl. information on usage so I can filter and identify e.g. outdated projects.
Especially info on "last access" would help a lot. I couldn't find a way yet to pull a datetime variable giving me the last use of e.g. "data access" or "configuration" activity.
Any idea on how to perform such a query? Even alternative ways of determining recent activity within projects would help. Most used resources are BigQuery, ComputeEngine, Buckets.
Thanks!
You can achieve this through:
Audit Logs Or
Cloud Asset Inventory (better than audit logs for your case).
You will have the ability to view activity at project level or at folder/organization level.
Edit:
Including the Cloud Asset Inventory query that #nordlicht.22 created to solve the issue:
gcloud asset search-all-resources \
--scope='projects/{ProjectID}' \
--query='updateTime > 1643155200' \
--order-by='createTime DESC' \
--limit='1'`
I have exported the dashboards using gcloud alpha monitoring dashboards list --format=json, but using gcloud dashboard create using file is not working, basically I want to export the dashboards from one project and import that in other project.
The output of the list sub command probably (didn't test this) has too many dashboards for the create command.
Also, you should remove two fields (name and etag). No need to export as json, yaml will also work and is easier to edit anyway.
I did the following:
gcloud monitoring dashboards list and find the dashboard I was looking for
Note it's name property and get the id from the last part in the name property (a large decimal number or guid)
gcloud monitoring dashboards describe $DASHBOARD_ID > dashboard-$DASHBOARD_ID.yaml the dashboard
Edit the file to remove the etag and name field (the name is usually located at the end of the file)
gcloud monitoring dashboards create --config-from-file dashboard-$DASHBOARD_ID.yaml
I am trying to identify the users who created tables in BigQuery.
Is there any command line or API that would provide this information. I know that audit logs do provide this information, but I was looking for a command line which could do the job so that i could wrap this in a shell script and run them against all the tables at one time. Same for Google Storage Buckets as well. I did try
gsutil iam get gs://my-bkt and looked for "role": "roles/storage.admin" role, but I do not find the admin role with all buckets. Any help?
This is a use case for audit logs. BigQuery tables don't report metadata about the original resource creator, so scanning via tables.list or inspecting the ACLs don't really expose who created the resource, only who currently has access.
What's the use case? You could certainly export the audit logs back into BigQuery and query for table creation events going forward, but that's not exactly the same.
You can find it out using Audit Logs. You can access them both via Console/Log Explorer or using gcloud tool from the CLI.
The log filter that you're interested in is this one:
resource.type = ("bigquery_project" OR "bigquery_dataset")
logName="projects/YOUR_PROJECT/logs/cloudaudit.googleapis.com%2Factivity"
protoPayload.methodName = "google.cloud.bigquery.v2.TableService.InsertTable"
protoPayload.resourceName = "projects/YOUR_PROJECT/datasets/curb_tracking/tables/YOUR_TABLE"
If you want to run it from the command line, you'd do something like this:
gcloud logging read \
'
resource.type = ("bigquery_project" OR "bigquery_dataset")
logName="projects/YOUR_PROJECT/logs/cloudaudit.googleapis.com%2Factivity"
protoPayload.methodName = "google.cloud.bigquery.v2.TableService.InsertTable"
protoPayload.resourceName = "projects/YOUR_PROJECT/datasets/curb_tracking/tables/YOUR_TABLE"
'\
--limit 10
You can then post-process the output to find out who created the table. Look for principalEmail field.