How we can find the details programmatically about GCP Infrastructure like various Folders, Projects, Compute Instances, datasets etc. which can help to have a better understanding of GCP platform.
Regards,
Neeraj
There is a service in GCP called Cloud Asset Inventory. Cloud Asset Inventory is a storage service that keeps a five week history of Google Cloud Platform (GCP) asset metadata.
It allows you to export all asset metadata at a certain timestamp to Google Cloud Storage or BigQuery.
It also allows you to search resources and IAM policies.
It supports a wide range of resource types, including:
Resource Manager
google.cloud.resourcemanager.Organization
google.cloud.resourcemanager.Folder
google.cloud.resourcemanager.Project
Compute Engine
google.compute.Autoscaler
google.compute.BackendBucket
google.compute.BackendService
google.compute.Disk
google.compute.Firewall
google.compute.HealthCheck
google.compute.Image
google.compute.Instance
google.compute.InstanceGroup
...
Cloud Storage
google.cloud.storage.Bucket
BigQuery
google.cloud.bigquery.Dataset
google.cloud.bigquery.Table
Find the full list here.
The equivalent service in AWS is called AWS Config.
I have found open source tool named as "forseti Security", which is easy to install and use. It has 5 major components in it.
Inventory : Regularly collects the data from GCP and store the results in cloudSQL under the table “gcp_inventory”. In order to refer to the latest inventory information you can refer to the max value of column : inventory_index_id.
Scanner : It periodically compares the policies applied on GCP resources with the data collected from Inventory. It stores the scanner information in table “scanner_index”
Explain : it helps to manage the cloud IAM policies.
Enforcer : This component use Google Cloud API to enforce the policies you have set in GCP platform.
Notifier : It helps to send notifications to Slack, Cloud Storage or SendGrid as show in Architecture diagram above.
You can find the official documentation here.
I tried using this tool and found it really useful.
Related
Is there a way to list/get the owner(creator) of all the resource under a project ?
I have already looked at the answers here and tried the cloud assets api gcloud asset search-all-resources --scope=projects/123 but this doesn't list the creator of each resource. I have also referred to the searching resources samples queries here but again this doesn't suffice my needs.
Ideally I need the following, for example -
asset type -storage bucket
resource name - test_bucket
owner/creator/user - user123#org1.com or test#gservice_account.com
created - 02-02-2018
same for other asset types like compute instance, BigQuery datasets etc.
Has anyone ever tried this ?
What you are looking for is the Audit logs.
As mentioned in docs:
Google Cloud services write audit logs that record administrative activities and accesses within your Google Cloud resources. Audit logs help you answer "who did what, where, and when?" within your Google Cloud resources with the same level of transparency as in on-premises environments
Also,
Here, you can find a list of all services that produces audit-logs.
Take a look here on the Best practices and things to be taken into consideration while working with audit-logs.
We are in the process of remodelling our cloud infrastructure. In an endeavor to do so, we need a list of all cloud resources in the project, along with below properties:
VPC/Subnet
Region
Zone
Is there a quick approach to do so?
The GCP Cloud Asset Inventory service seems like a good match for your needs. This is an API/commands for working with the database of assets (things) in your GCP environment. From this you can dump your resource definitions as well as request feeds for changes within the environment.
To get a current dump of resources, we can use gcloud asset list see Listing assets.
Outline
We are running an ecommerce platform on Google Cloud on a dedicated VM Instance. Most of our traffic happens on Monday, as we then send our newsletters to our customer-base. Because of that we have huge traffic-peaks each Monday.
Goal
Because of this peak-traffic we need to make sure, that we understand how much server-load a single user is generating on average. To achieve this, we want to correlate our VM Instance Monitoring Data with our Google Analytics Data in Google Datastudio. To get a better understanding of the mentioned dynamics.
Problem
As far as we are aware (based on the docs), there is no direct data-consumption from the gcloud sdk possible in Google Datastudio. With that as a fact, we tried to extract the data via. BigQuery, but also there didn't found a possibility to access the monitoring data of our VM Instance.
Therefore we are looking for a solution, how we can extract our monitoring data of our VM Instances to Google Datastudio (preferably via BigQuery). Thank you for your help.
Here is Google official solution for monitoring export.
This page describes how to export monitoring metrics to bigquery dataset.
Solution deployments use pub/sub, app engine, Cloud scheduler and some python codes.
I think you only need to export the metrics listed in here.
If you complete exporting process successfully, then you can use Google Data studio for visualizing your metric data.
I generated a billing report(export to csv) for GCP but not able to map billing entry to the resource as there is no resource id present. How am I supposed to know the cost for the individual resource
AWS gives to option to explicitly include the resource id.
Any Help is appreciated
You can use labels to do it. Have a look at the documentation Creating and managing labels:
A label is a key-value pair that helps you organize your Google Cloud
instances. You can attach a label to each resource, then filter the
resources based on their labels. Information about labels is forwarded
to the billing system, so you can break down your billing charges by
label.
You can create labels by using Resource Manager API, Cloud Console or gcloud command (here an example for GCE VM instances).
Services currently supporting labels:
BigQuery
Cloud Bigtable
Dataflow
Dataproc
Cloud Deployment Manager
Cloud Functions
Cloud Healthcare API
Cloud Key Management Service
Pub/Sub
Cloud Spanner
Cloud SQL
Cloud Storage
Compute Engine
Google Kubernetes Engine
Cloud Run (fully managed)
Networking
Resource Manager (projects only)
App Engine (both Flexible and Standard environments)
More information about using labels you can find at the documentation View your billing reports and cost trends, View and download the cost details of your invoice or statement and Export Cloud Billing data to BigQuery
Thank your Response.
I did try it with labels, but in CSV report there is a column for Project Labels only. I exported the same report in Big query and i was able to see columns for tree types of labels project labels, system labels, and labels. The labels which I was setting in VMs I could see them in the third column.
Not sure what are project labels
In general any cloud service provider, GCP in this context, is it not relevant and mandatory for Google to specifically allow consumers to choose data residency and data processing region option for all services? Else serverless option will have serious adoption issue. Please clarify.
Google Cloud have two types of the products available: that have specified location and available globally.
You can deploy resources in specific location, multi-regional for:
Compute: Compute Engine, App Engine, Google Kubernetes Engine, Cloud Functions
Storage & Databases: Cloud Storage, Bigtable, Spanner, Cloud SQL, Firestore, Memorystore, Persistent Disk...
BigData & Machine learning: BigQuery, Composer, Dataflow, Dataproc, AI training,
Networking: VPC, Cloud Load Balancing,
Developer Tools...
Following products are available only globally: Networking, Big Data Pub/Sub, Machine Learning like vision API, Management Tools, Developer Tools, IAM.
For detailed list please check Google Cloud Locations Documentation
Even if the product is available globally, for example PubSub: it is possible to specify where messages are stored.
If the data in transit are the concern, you have to be aware that Google Cloud Platform uses data encryption at Rest. It consists on several layers of encryption to protect customer data.