How to retrieve the compute engine name from snapshot API response? - google-cloud-platform

I need to list all the compute instance snapshots successfully created in a project (only for compute instance types), along with the compute engine names.
I am using this API: https://compute.googleapis.com/compute/v1/projects/my-project/global/snapshots
It lists the snapshot and I get the response like this:
"items": [
{
"id": "36734343434334343",
"creationTimestamp": "2020-09-16T11:38:54.780-07:00",
"name": "backup-data1-us-central1-c-3234234324-202009161",
"status": "READY",
"sourceDisk": "https://www.googleapis.com/compute/v1/projects/my-project/zones/us-central1-c/disks/backup-data1",
"sourceDiskId": "323434232434970709",
"diskSizeGb": "10",
"storageBytes": "452416",
"storageBytesStatus": "UP_TO_DATE",
"selfLink": "https://www.googleapis.com/compute/v1/projects/my-project/global/snapshots/amtest-backup-data1-us-central1-c-3234234324-202009161",
"labelFingerprint": "23WmSpBrSM=",
"storageLocations": [
"us-central1"
],
"autoCreated": true,
"downloadBytes": "456717",
"kind": "compute#snapshot"
},
{
"id": "343486082509657007",
"creationTimestamp": "2020-09-17T11:38:56.840-07:00",
"name": "backup-data1-us-central1-c-3234234324-202009161",
"status": "READY",
"sourceDisk": "https://www.googleapis.com/compute/v1/projects/my-project/zones/us-central1-c/disks/backup-data1",
"sourceDiskId": "323434232434970709",
"diskSizeGb": "10",
"storageBytes": "0",
"storageBytesStatus": "UP_TO_DATE",
"selfLink": "https://www.googleapis.com/compute/v1/projects/my-project/global/snapshots/amtest-backup-data1-us-central1-c-20200917183856-n2ipabzb",
"labelFingerprint": "23WmSpB8rSM=",
"storageLocations": [
"us-central1"
],
"autoCreated": true,
"downloadBytes": "456717",
"kind": "compute#snapshot"
}
From this information, I need to find out what is the VM which is associated with this snapshot. How can I find out the compute engine for which this snapshot is created? Is there any REST API for finding the compute engine from the snapshot?

There is a little misunderstanding here: You snapshot a disk, not a VM. Indeed, you can detach the disk and attach it to another VM. You can also set the disk in multi-reader and attach it to several VM.
So, your question is wrong. You can list, among all your VM, the disk attache to them. Then check if a snapshot exists for each of these disks.

Related

How to fetch GCP billing by instance name? (Not by type) and is there any association between resource and billing sku id?

Can I see GCP billing by instance name? (Not by type)
I am trying to filter GCP billing by instance name, is it possible? I only succeeded to filter by GCP Compute Engine and instance type (n1-standard) etc...
I am trying to programmatically match the machineType associated with a GCP compute instance to the corresponding billing SKU, but am unable to find a key for direct association. For example, here is the response from the machineType API:
{
"kind": "compute#machineType",
"name": "n1-standard-32",
"description": "32 vCPUs, 120 GB RAM",
"guestCpus": 32,
"memoryMb": 122880,
"imageSpaceGb": 0,
"maximumPersistentDisks": 128,
"maximumPersistentDisksSizeGb": "65536",
"zone": "us-east1-b",
"isSharedCpu": false
}
And here is the corresponding SKU from the cloudbilling APIs:
"name": "services/XXXX/skus/XXXX",
"skuId": "XXXX",
"description": "Standard Intel N1 32 VCPU running in Americas",
"category": {
"serviceDisplayName": "Compute Engine",
"resourceFamily": "Compute",
"resourceGroup": "N1Standard",
"usageType": "OnDemand"
},
"serviceRegions": [
"us-central1",
"us-east1",
"us-west1"
],
"pricingInfo": [
{
"summary": "",
"pricingExpression": {
"usageUnit": "h",
"usageUnitDescription": "hour",
"baseUnit": "s",
"baseUnitDescription": "second",
"baseUnitConversionFactor": 3600,
"displayQuantity": 1,
"tieredRates": [
{
"startUsageAmount": 0,
"unitPrice": {
"currencyCode": "USD",
"units": "1",
"nanos": 520000000
}
}
]
},
"currencyConversionRate": 1,
"effectiveTime": "2018-02-22T12:00:16.647Z"
}
],
"serviceProviderName": "Google"
There doesn't seem to be a field with value n1-standard-32 in the billing SKU. How do we tie these two together as this page seems to do: https://cloud.google.com/compute/pricing?
You can create labels and add to your instances in order to get a breakdown of the charges per instance. This label needs to be added on each instance, once added, you will be able to see the charges per instance on the billing reports by sorting it per label.
Creating and managing labels can be found here
You can use the Resource Manager API and perform a request such as
POST https://cloudresourcemanager.googleapis.com/v1beta1/projects
{
"labels": {
"color": "red"
},
"name": "myproject",
"projectId": "our-project-123"
}
You can also add and edit labels for your Compute Engine Instances using the gcloud commands

Aws cli command substitution

I want to list volume's snapshots but in the output I would like to also see the Name of that volume (I mean tag).
So far I was using:
aws ec2 describe-snapshots
And in the reply I got something like:
Snapshots: [
{
"Description": "some description",
"Encrypted": false,
"OwnerId": "someownerid",
"Progress": "100%",
"SnapshotId": "snap-example",
"StartTime": "start time",
"State": "completed",
"VolumeId": "volume id",
"VolumeSize": 32
}
]
But what I would like to have in that output is also a volume name:
Snapshots: [
{
"Description": "some description",
"Encrypted": false,
"OwnerId": "someownerid",
"Progress": "100%",
"SnapshotId": "snap-example",
"StartTime": "start time",
"State": "completed",
"VolumeId": "volume id",
"VolumeSize": 32,
"VolumeName": "Volume Name" #additional key:val
}
]
The aws ec2 describe-snapshots does return tags on snapshots if they are present.
Something similar to this:
{
"Description": "This snapshot is created by the AWS Backup service.",
"Tags": [
{
"Value": "On",
"Key": "Backup"
},
{
"Value": "Jenkins_Machine",
"Key": "Name"
},
{
"Value": "*********",
"Key": "aws:backup:source-resource"
}
],
"Encrypted": false,
"VolumeId": "vol-*****",
"State": "completed",
"VolumeSize": 250,
"StartTime": "2019-08-01T11:29:31.654Z",
"Progress": "100%",
"OwnerId": "******",
"SnapshotId": "snap-******"
}
To be able to see the name (Assuming your snapshots have them) do this:
aws ec2 describe-snapshots --snapshot-id snap-**** --query 'Snapshots[*].{Description:Description,Name:Tags[?Key==`Name`].Value|[0],State:State}'
This should give you output like this:
[
{
"State": "completed",
"Description": "This snapshot is created by the AWS Backup service.",
"Name": "Jenkins_Machine"
}
]
The fields are curtailed but you can add fields that you need at the end of the query like this ...State:State,VolumeId:VolumeId}, where I have newly added the VolumeId.
If you remove the --snapshot-id parameter, the above command should return you all snapshots, however for snapshots that don't have the Name tag its going to print null.
Edit:
As #krishna_mee2004 pointed out, the OP is probably looking for snapshots for a particular volume. If that is the case you can still do it using this command. The filters option can be used to filter based on volume ID.
aws ec2 describe-snapshots --filters Name=volume-id,Values=vol-***** --query 'Snapshots[*].{Description:Description,Name:Tags[?Key==`Name`].Value|[0],State:State,VolumeId:VolumeId}'
If you are referring to snapshot's name tag, you can write a simple python or ruby script using aws sdk. For example, a ruby code to list the snapshot id and value of its name tag will look like this:
require 'aws-sdk'
# provide region and credentials in parameter
ec2 = Aws::EC2::Client.new
# paginate if you have a big list
resp = ec2.describe_snapshots
# iterate snapshots
resp.snapshots.each do |snapshot|
# iterate tags and print if it has a name tag.
snapshot.tags.each do |tag|
# print whatever is required/available in the response structure
puts "#{snapshot.snapshot_id} has the name tag with value #{tag.value}" if tag.key.casecmp? 'Name'
end
end
Refer the respective language api documentation to understand more about the usage of the sdk and the api calls. Make sure to setup the sdk before using and it varies based on the language you choose. For example, steps for setting up the ruby sdk is outlined here. You may also want to checkout the API reference for describe_snaphots used in the above code.

API/SDK for EC2 Spot advisor

The EC2 Spot advisor which is available in the url - https://us-west-2.console.aws.amazon.com/ec2sp/v1/spot/advisor?region=us-west-2 provides recommendation on which instance-type in an availability zone has the least likelihood of getting intervened(Terminated) and the spot price.
Is there a programmable way(api/sdk, etc) to get this data from a script to automate provisioning instances using this data?
I am having the same issue, I noticed in the network tab that the data is coming from an API get_advice, but I cannot find that API anywhere in the AWS SDK/CLI docs.
GET https://us-west-2.console.aws.amazon.com/ec2sp/services/get_advice?product=linux&cheapest=90&min_cpu=2&min_ram=3&same_size=false&parallelability=90&az=any
It returns machine-parseable JSON data like this:
[
{
"instanceType": "t2.medium",
"azs": [
"us-west-2a",
"us-west-2b",
"us-west-2c"
],
"averagePrice": 0.0139,
"pricePerUnit": 0.0139,
"downtime": null,
"terminationProbability": 0,
"averageQuality": 0.01251
},
{
"instanceType": "t3.2xlarge",
"azs": [
"us-west-2a",
"us-west-2b",
"us-west-2c"
],
"averagePrice": 0.1002,
"pricePerUnit": 0.0251,
"downtime": null,
"terminationProbability": 0,
"averageQuality": 0.024372972972972973
},
{
"instanceType": "t2.large",
"azs": [
"us-west-2a",
"us-west-2b"
],
"averagePrice": 0.0278,
"pricePerUnit": 0.0278,
"downtime": null,
"terminationProbability": 0,
"averageQuality": 0.02502
}
]
This could be used to populate LaunchSpecifications in request-spot-fleet, however the API is probably not stable, and needs a console authentication token.

EMR cluster created with CloudFormation not shown

I have added an EMR cluster to a stack. After updating the stack successfully (CloudFormation), I can see the master and slave nodes in EC2 console and I can SSH into the master node. But AWS console does not show the new cluster. Even aws emr list-clusters doesn't show the cluster. I have triple checked the region and I am certain I'm looking at the right region.
Relevant CloudFormation JSON:
"Spark01EmrCluster": {
"Type": "AWS::EMR::Cluster",
"Properties": {
"Name": "Spark01EmrCluster",
"Applications": [
{
"Name": "Spark"
},
{
"Name": "Ganglia"
},
{
"Name": "Zeppelin"
}
],
"Instances": {
"Ec2KeyName": {"Ref": "KeyName"},
"Ec2SubnetId": {"Ref": "PublicSubnetId"},
"MasterInstanceGroup": {
"InstanceCount": 1,
"InstanceType": "m4.large",
"Name": "Master"
},
"CoreInstanceGroup": {
"InstanceCount": 1,
"InstanceType": "m4.large",
"Name": "Core"
}
},
"Configurations": [
{
"Classification": "spark-env",
"Configurations": [
{
"Classification": "export",
"ConfigurationProperties": {
"PYSPARK_PYTHON": "/usr/bin/python3"
}
}
]
}
],
"BootstrapActions": [
{
"Name": "InstallPipPackages",
"ScriptBootstrapAction": {
"Path": "[S3 PATH]"
}
}
],
"JobFlowRole": {"Ref": "Spark01InstanceProfile"},
"ServiceRole": "MyStackEmrDefaultRole",
"ReleaseLabel": "emr-5.13.0"
}
}
The reason is missing VisibleToAllUsers property, which defaults to false. Since I'm using AWS Vault (i.e. using STS AssumeRole API to authenticate), I'm basically a different user every time, so I couldn't see the cluster. I couldn't update the stack to add VisibleToAllUsers either as I was getting Job flow ID does not exist.
The solution was to login as root user and fix things from there (I had to delete the cluster manually, but removing it from the stack template JSON and updating the stack would probably have worked if I hadn't messed things up already).
I then added the cluster back to the template (with VisibleToAllUsers set to true) and updated the stack as usual (AWS Vault).

Create tagged vm instance snapshots

I'm trying to create snapshots of my vm instance on google cloud platform with a custom tag, but it's currently not working as expected. I'm sending the following Post request body to API referring to this documentation Google docs
{
"name":"<SnapshotName>",
"labels": {
"<LabelKey>":"<LabelValue>"
}
}
this gives me a positive 200 OK response, but no label appears.
{
"kind": "compute#operation",
"id": "<id>",
"name": "<name>",
"zone": "<Zone Link>",
"operationType": "createSnapshot",
"targetLink": "<Target Link>",
"targetId": "<Target ID>",
"status": "PENDING",
"user": "<User>",
"progress": 0,
"insertTime": "<Time>",
"selfLink": "<Self Link>"
}
additionally I tried to use syntax described in "Labeling Resources" documentation Google Labeling Resources
{
"name":"<SnapshotName>",
"labels": [{
"<Key>":"<LabelKey>"
"<Value>":"<LabelValue>"
}]
}
this gave me the same result.
In web interface it's possible to create snapshots and label it manually, but I would like to create them with a custom label via API.
Am I doing something wrong, or is it just broken?