what is the GCP CLI Command to remove (detach) a Data Catalog Tag from a BigQuery Dataset, and also CLI Command to Update Tag.
I am able to do it manually how to do it using Cloud Shell CLI gcloud command?
You can use gcloud data-catalog tags update and gcloud data-catalog tags delete commands.
The tricky part here is obtaining values for --entry-group and --entry parameters - BigQuery entries are automatically ingested to Data Catalog, and have automatically assigned entry group id and entry id. To get these values use gcloud data-catalog entries lookup command.
Related
Using the AWS CLI, how can I get the most recent snapshot for a particular DB instance?
I can get them through the GUI easily but I'd like to automate it.
You can use the aws rds describe-db-snapshots CLI command to get a list of DB snapshots, and then run a local query using --query to get the latest DB snapshot using the SnapshotCreateTime field.
SnapshotCreateTime -> (timestamp)
Specifies when the snapshot was taken in Coordinated Universal Time (UTC). Changes for the copy when the snapshot is copied.
Something like this:
aws rds describe-db-snapshots \
--db-instance-identifier your-id \
--query "sort_by(DBSnapshots, &SnapshotCreateTime)[-1].{id:DBSnapshotIdentifier,time:SnapshotCreateTime}"
Note that this query sorts snapshots by their ascending SnapshotCreateTime and then simply takes the last one in the list (as dictated by [-1]), which will be the one that was last created.
[Added] if you're looking for snapshots of Aurora DB clusters then you'd have to use describe-db-cluster-snapshots in place of describe-db-snapshots, but otherwise the process is similar: use DBClusterSnapshots and DBClusterSnapshotIdentifier (in place of DBSnapshots and DBSnapshotIdentifier).
I am very new to Google Cloud Platform, and I am doing a POC for moving a hive application (tables and jobs) to Google Dataproc. The data has already been moved to Google cloud Storage.
Is there an inbuilt way to create all the tables from hive in dataproc in bulk, instead of creating one by one using the hive prompt?
Dataproc support Hive job type, so you can use the gcloud command:
gcloud dataproc jobs submit hive --cluster=CLUSTER \
-e 'create table t1 (id int, name string); create table t2 ...;'
or
gcloud dataproc jobs submit hive --cluster=CLUSTER -f create_tables.hql
You can also SSH into the master node, then use beeline to execute the script:
beeline -u jdbc:hive2://localhost:10000 -f create_tables.hql
I have logging enabled on my 2nd Generation CloudSQL instances in GCP - however I'm attempting to read these using the CLI and drawing a blank.
If I run $ gcloud logging logs list I can see the logs I want to read, example as follows:
projects/<project name>/logs/cloudsql.googleapis.com%2Fmysql-slow.log
projects/<project name>/logs/cloudsql.googleapis.com%2Fmysql.err
The docs are confusing, but it looks like I should be able to read them if I run:
gcloud logging read "logName=projects/<project name>/logs/cloudsql.googleapis.com%2Fmysql.err" --limit 10 --format json
However this only returns a blank array as []
I just want to read out the logs.
What am I doing wrong?
You have to execute the gcloud logging read projects/<project name>/logs/cloudsql.googleapis.com%2Fmysql-slow.log and gcloud logging read projects/<project name>/logs/cloudsql.googleapis.com%2Fmysql.err. As it is specified in Quickstart using Cloud SDK documentation. Do not use the " in the command.
I would like to automate exports of our Spanner database to Google Cloud Storage. Is this possible using the gcloud SDK? I could not find a command for this.
Is there any other recommended way to back up Spanner databases?
The export and Import pipelines are Dataflow templates that can be started using the Gcloud command.
See the third paragraph in:
https://cloud.google.com/spanner/docs/export
And how to run the template in:
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates#cloud_spanner_to_gcs_avro
(Select the Gcloud tab in the executing the template section).
Yes, it is possible to do this using gcloud, but it is not a direct Cloud Spanner command. The detailed documentation is here.
Essentially you use gcloud to run a Cloud Dataflow job to export or backup your data to GCS using a command like the following:
gcloud dataflow jobs run [JOB_NAME] \
--gcs-location='gs://dataflow-templates/latest/Cloud_Spanner_to_GCS_Avro' \
--region=[DATAFLOW_REGION] \
--parameters='instanceId=[YOUR_INSTANCE_ID],databaseId=[YOUR_DATABASE_ID],outputDir=[YOUR_GCS_DIRECTORY]
I am new to Google Cloud and am facing a challenge while adding ssh-keys to google metadata (project-wide) with gcloud command line.
When I try to add ssh-key into Google metadata (with command :: gcloud compute project-info add-metadata --metadata-from-file ssh-keys=[LIST_PATH]) along with the new ssh-key which I am trying to add, I also have to specify all existing ssh-keys in the source file. (the source file is the file where we store ssh-key value). because I will add all the ssh-keys which are present in source file so if I do not keep existing ssh-keys in source file and keep only one key, it will add only this single key into metadata and rest of the existing keys will be removed.
So what I am trying to achieve is to add any single ssh-key to the metadata without affecting existing keys. Because this will be a repeated process for many of the machines in my environment, and I cannot track existing keys every time.
I've had the same question.
According to the the official doc (https://cloud.google.com/compute/docs/instances/adding-removing-ssh-keys), it is not possible to manipulate individual keys from the gcloud tool.
Here is an example shell to add a key:
gcloud compute project-info add-metadata \
--metadata-from-file ssh-keys=<(\
gcloud compute project-info describe --format json \
| jq -r '.commonInstanceMetadata.items[]
| if .key == "ssh-keys" then .value else empty end';
echo "john:ssh-rsa mykey john")
It:
grabs the existing values (gcloud describe | jq).
adds a key (echo "john...").
feeds it as a pseudo-file to gcloud add-metadata.
Up to you to separate the steps, keep a local list of your keys, or whatever suits your need.
This example lacks a few features, like key de-duplication. That's just an experiment at the moment, I'll have to create a more robust script for real use.