GCP has a deep learning VM available to run on their cloud compute platform. The details about the image is here
So, I am using the google python client to launch my instances and the documentation for this is available here. Now, the way one specifies the disk and the boot image is through this JSON blob:
'disks': [
{
'boot': True,
'autoDelete': True,
'initializeParams': {
'sourceImage': source_disk_image,
}
}
]
Now the source_disk_image is specified by the path to some public image like:
projects/debian-cloud/global/images/family/debian-9 or some variant of this type. Now, my question is how can I specify some marketplace image to be used for my instance?
If you're not attached to using the marketplace to create the VM, there's a lot of documentation about all the available Google Deep Learning images.
They live in the deeplearning-platform-release project, so, for example, I think (but am not sure) the default image you are referring to from the Marketplace you linked is projects/deeplearning-platform-release/global/images/tf-1-14-cu100-20191004 but you can also pull them by family and just get the latest versions, for example, projects/deeplearning-platform-release/global/images/family/tf-latest-gpu.
The gcloud images command is also pretty illuminating to see the description of a given family choice or image, e.g.:
$ gcloud compute images describe-from-family tf-latest-gpu --project deeplearning-platform-release
archiveSizeBytes: '322993843200'
creationTimestamp: '2019-10-06T13:57:56.932-07:00'
description: "Google, Deep Learning Image: TensorFlow 1.14.0, m36, TensorFlow 1.14.0\
\ with CUDA 10.0 and Intel\xAE MKL-DNN, Intel\xAE MKL."
diskSizeGb: '30'
...
Which looks a lot like the Marketplace description.
That said, it looks like the Marketplace might be doing other things though (e.g. there are checkboxes about installing particular drivers separate from choosing the image).
I think that #Ernesto's tip about creating an instance off the marketplace, and then viewing that instance via the REST link at the bottom of the instance page to find exactly how it was created is also good advice. However, in this case you probably want to view the disk that was created (not the instance, since once it is created it only references the disk resource), click on the rest link, and look for the "sourceImage" portion of the REST response.
e.g. from a regular old debian-9 disk (I don't have GPU quota so I can't actually create the marketplace deployment):
I was able to find the SourceImage of a Deep Learning found in the marketplace, for this example I'm using
NVIDIA GPU Cloud Image for Deep Learning, Data Science, and HPC
"name": "nvidia-gpu-cloud-image-20190809",
"selfLink": "projects/nvidia-ngc-public/global/images/nvidia-gpu-cloud-image-20190809",
"sourceDisk": "projects/nvidia-ngc-dev/zones/us-central1-a/disks/chetan-official-base-image"
Deploy an instance from the MarketPlace
Go to the instance and inspect the details from the UI
In the Boot disk section click on the image name
nvidia-gpu-cloud-image-20190809 it will take you to the image
details page
Click on REST at the bottom of the description
Find SelfLink or SourceDisk entry
Related
For a project, I have to create a Dataproc cluster that has one of the outdated versions (for example, 1.3.94-debian10) that contain the vulnerabilities in Apache Log4j 2 utility. The goal is to get the alert related (DATAPROC_IMAGE_OUTDATED), in order to check how SCC works (it is just for a test environment).
I tried to run the command gcloud dataproc clusters create dataproc-cluster --region=us-east1 --image-version=1.3.94-debian10 but got the following message ERROR: (gcloud.dataproc.clusters.create) INVALID_ARGUMENT: Selected software image version 1.3.94-debian10 is vulnerable to remote code execution due to a log4j vulnerability (CVE-2021-44228) and cannot be used to create new clusters. Please upgrade to image versions >=1.3.95, >=1.4.77, >=1.5.53, or >=2.0.27. For more information, see https://cloud.google.com/dataproc/docs/guides/recreate-cluster, which makes sense, in order to protect the cluster.
I did some research and discovered that I will have to create a custom image with said version and generate the cluster from that. The thing is, I have tried to read the documentation or find some tutorial, but I still can't understand how to start or to run the file generate_custom_image.py, for example, since I am not confortable with cloud shell (I prefer the console).
Can someone help? Thank you
I published a gene mapping pipeline that can be downloaded from Github and run on a local machine. The problem is that server setting is not always available (or easy) to people. I created an instance and an image (or machineImage which I don't understand the difference) on Google cloud that can run the pipeline, however I could not make it public so people can use it to create an instance, upload their specific data (fastq files) and run it to map their mutants.
Any help would be highly appreciated. Thank you
As John Hanley mentioned, you cannot make an image public.
According to the official documentation:
Compute Engine Images
Public images are provided and maintained by Google, open source
communities, and third-party vendors. By default, all Google Cloud
projects have access to these images and can use them to create
instances.
Therefore you can use the public images maintained by Google to create your instances.
I've looked for this across the web a few times, and I feel like this hasn't been asked exactly, or I may just be getting bogged down with the wrong syntax. Hoping to get an easy answer here (yes, you can't get this, is an acceptable answer).
The variations from the base CentOS image are listed here: Link to GCP
However, they don't actually provide a download for this image. I'm trying to get a local VM running in VMWare with this image.
I feel as though they'd provide this to their clients to make it easier to prepare for use of their product, but I'm not finding it anywhere.
If anyone could toss me a link to a pre-configured CentOS ISO with the minor changes, I'd definitely take that as an alternative. I'm just not confident in my skills with Linux enough to configure the firewall properly :)
GCP doesn't support Google-provied images for exporting. However, they support exporting images for custom images.
I don't have any experience about image exporting, but I think this works.
Create custom images
You can create custom images based on your GCE VM instance.
Go navigation -> Compute engine -> images page.
You can create custom image via disk or snapshot in this page.
Select one and create a custom image.
Export your image
After creating custom image successfully, Go custom image page and click "export" on upper side.
Select export format and GCS destination. then click export.
Now you have an image in the Google Cloud storage.
Download image file and import to your local VM machine.
I want to build a Google Cloud image using Packer, but can't seem to find a way for packer to add additional disks with googlecompute builder. This is required as want a persistent disk for application to store data on it.
Is it something that can be done through startup_script or any other way?
GCE images only support one disk.
Please check here for an open feature request to support this.
I am just looking into using GCP for cloud computing stuff. So far I have been using AWS and the boto3 library and was trying to use the google python client API for launching instances.
So an example I came across was from their docs here. The instance machine type is specified as:
machine_type = "zones/%s/machineTypes/n1-standard-1" % zone
and then it passed to the configuration as:
config = {
'name': name,
'machineType': machine_type,
....
I wonder how does one go about specifying machines with GPU and custom RAM and processors etc. from the python API?
The Python API is basically a wrapper around the REST API, so in the example code you are using, the config object is being built using the same schema as would be passed in the insert request.
Reading that document shows that the guestAccelerators structure is the relevant one for GPUs.
Custom RAM and CPUs are more interesting. There is a format for specifying a custom machine type name (you can see it in the gcloud documentation for creating a machine type). The format is:
[GENERATION]custom-[NUMBER_OF_CPUs]-[RAM_IN_MB]
Generation refers to the "n1" or "n2" in the predefined names. For n1, this block is empty, for n2, the prefix is "n2-". That said, experimenting with gcloud seems to indicate that "n1-" as a prefix also works as you would expect.
So, for a 1 CPU n1 machine with 5GB of ram, it would be a custom-1-5120. This is what you would replace the n1-standard-1 in your example with.
You are, of course, subject to the limits of how to specify a custom machine such as the fact that RAM must be a multiple of 256MB.
Finally, there's a neat little feature at the bottom of the console "create instance" page:
Clicking on the relevant link will show you the exact REST object you need to create the machine you have defined in the console at that very moment, so it can be very useful to see how a particular parameter is used.
You can create a Compute Engine instance using the Compute Engine API. Specifically, we can use the insert API request. This accepts a JSON payload in a REST request that describes the desired VM instance that you desire. A full specification of the request is found in the docs. It includes:
machineType - specs of different (common) machines including CPUs and memory
disks - specs of disks to be added including size and type
guestAccelerators - specs for GPUs to add
many more options ...
One can also create a template description of the machine structure you want and simplify the creation of an instance by naming the template to use and thereby abstracting the configuration details out of code and into configuration.
Beyond using REST requests (which can be passed from a python), you also have the capability to create Compute Engines from:
GCP Console - web interface
gcloud - command line (which I suspect can also be driven from within Python)
Deployment Manager - configuration driven deployment which includes Python as a template language
Terraform - popular environment for creating Infrastructure as Code environments