I was looking on the prices on the calculator when see on Free saying "... or User Provided OS". And I want know how I upload a OS to Google Compute Engine.
There are a number of ways to import/migrate a custom instance to GCP. Two possible and popular solutions would be either migrating with Velostrata or using CloudEndure.
To upload or import a boot disk image to Compute Engine, you may use the following process:
Plan your import path. You must identify where you are going to prepare your boot disk image before you upload it, and how you are going to connect to that image after it boots in the Compute Engine environment.
Prepare your boot disk so it can boot within the Compute Engine environment and so you can access it after it boots.
Create and compress the boot disk image file.
Upload the image file to Google Cloud Storage [1] and import the image to Compute Engine as a new custom image [2].
Use the imported image to create a virtual machine instance and make sure it boots properly.
If the image does not successfully boot, you can troubleshoot the issue by attaching the boot disk image to another instance and reconfiguring it.
Optimize the image [3] and install the Linux Guest Environment [4] so that your imported operating system image can communicate with the metadata server and use additional Compute Engine features.
You may visit this link to learn on how to import Boot Disk Images to Compute Engine [5].
[1] https://cloud.google.com/storage/
[2] https://cloud.google.com/compute/docs/images#custom_images
[3] https://cloud.google.com/compute/docs/images/configuring-imported-images
[4] https://cloud.google.com/compute/docs/images/configuring-imported-images#install_guest_environment
[5] https://cloud.google.com/compute/docs/images/import-existing-image
Related
I want to build a Google Cloud image using Packer, but can't seem to find a way for packer to add additional disks with googlecompute builder. This is required as want a persistent disk for application to store data on it.
Is it something that can be done through startup_script or any other way?
GCE images only support one disk.
Please check here for an open feature request to support this.
I'm preparing to get in to the world of cloud computing.
My first question is:
Is it possible to programmatically create a new, or duplicate an existing VM from my server?
Project Background
I provide a file processing service, and as it's been growing I need to offer a better service.
Project Requirement
Machine specs:
HDD: Min 16gb
CPU: Min 1 core
RAM: Min 2
GB GPU: Min CUDA 10.1 compatible
What I'm thinking is the following steps:
User uploads a file
A dedicated VM is created for that specific file inside Google Cloud Compute
The file is sent to the VM
File is processed using a Anaconda environment
Results are downloaded to local server
Dedicated VM is removed
Results are served to user
How is this accomplished?
PS: I'm looking for resources and advice. Not code.
Your question is a perfect formulation of the concept of Google Cloud Run. At the highest level concept, you create a Docker image (think of it like a VM) and then register that Docker image with GCP Cloud Run. When a trigger occurs, GCP will spin up an instance of that Docker container and pass in information about the cause of that trigger (a file created in GCS or a REST request or others ...). What you do in your container is up to you. You have full power of the Linux environment (under Docker) to do as you like. When your request ends, the container is spun down. You are only billed for the compute resources you use. If your container (VM) isn't being used, you pay nothing until the next trigger.
An alternative to Cloud Run is Cloud Functions. This is a higher level abstraction where instead of providing a Docker container, you provide the body of a function (JavaScript, Java, Python or others) and the request is passed to that function when a trigger occurs. Which you use is mostly personal choice (you didn't elaborate on "File is processed").
References:
Cloud Run
Cloud Functions
GCP has a deep learning VM available to run on their cloud compute platform. The details about the image is here
So, I am using the google python client to launch my instances and the documentation for this is available here. Now, the way one specifies the disk and the boot image is through this JSON blob:
'disks': [
{
'boot': True,
'autoDelete': True,
'initializeParams': {
'sourceImage': source_disk_image,
}
}
]
Now the source_disk_image is specified by the path to some public image like:
projects/debian-cloud/global/images/family/debian-9 or some variant of this type. Now, my question is how can I specify some marketplace image to be used for my instance?
If you're not attached to using the marketplace to create the VM, there's a lot of documentation about all the available Google Deep Learning images.
They live in the deeplearning-platform-release project, so, for example, I think (but am not sure) the default image you are referring to from the Marketplace you linked is projects/deeplearning-platform-release/global/images/tf-1-14-cu100-20191004 but you can also pull them by family and just get the latest versions, for example, projects/deeplearning-platform-release/global/images/family/tf-latest-gpu.
The gcloud images command is also pretty illuminating to see the description of a given family choice or image, e.g.:
$ gcloud compute images describe-from-family tf-latest-gpu --project deeplearning-platform-release
archiveSizeBytes: '322993843200'
creationTimestamp: '2019-10-06T13:57:56.932-07:00'
description: "Google, Deep Learning Image: TensorFlow 1.14.0, m36, TensorFlow 1.14.0\
\ with CUDA 10.0 and Intel\xAE MKL-DNN, Intel\xAE MKL."
diskSizeGb: '30'
...
Which looks a lot like the Marketplace description.
That said, it looks like the Marketplace might be doing other things though (e.g. there are checkboxes about installing particular drivers separate from choosing the image).
I think that #Ernesto's tip about creating an instance off the marketplace, and then viewing that instance via the REST link at the bottom of the instance page to find exactly how it was created is also good advice. However, in this case you probably want to view the disk that was created (not the instance, since once it is created it only references the disk resource), click on the rest link, and look for the "sourceImage" portion of the REST response.
e.g. from a regular old debian-9 disk (I don't have GPU quota so I can't actually create the marketplace deployment):
I was able to find the SourceImage of a Deep Learning found in the marketplace, for this example I'm using
NVIDIA GPU Cloud Image for Deep Learning, Data Science, and HPC
"name": "nvidia-gpu-cloud-image-20190809",
"selfLink": "projects/nvidia-ngc-public/global/images/nvidia-gpu-cloud-image-20190809",
"sourceDisk": "projects/nvidia-ngc-dev/zones/us-central1-a/disks/chetan-official-base-image"
Deploy an instance from the MarketPlace
Go to the instance and inspect the details from the UI
In the Boot disk section click on the image name
nvidia-gpu-cloud-image-20190809 it will take you to the image
details page
Click on REST at the bottom of the description
Find SelfLink or SourceDisk entry
When an image is created from a compressed RAW image stored in a gcs bucket, is an instance spun up in the background to validate the image? I would like to understand how the image creation process works and if Google adds some software on top of what's in the RAW image.
According to our documentation regarding the importation of Boot Disk Images to Compute Engine. In the overview section, we explain all the needed steps to understand how the image creating process works. This will reply to your question “I would like to understand how the image creation process works”.
Verifying this steps details will allow you to address the remaining questions and you’ll know that we don’t spin up an instance in the background to validate the image and that Google doesn’t add any software on top of what’s in the RAW image.
Customers are responsible for:
1- Planning for the import path
2- Preparing the boot disk so it can boot in Compute Engine environment
3- Creating and compressing the boot disk
4- Uploading the image
5- Using the imported image to create a VM
I Have successfully implemented predictionIO engine templates. I can deploy an engine with
$pio deploy -- --driver-memory xG
But How can I run recommendation (or any other) engine as a service? I want to log all entries to a file specified for reference if some issues occur.
Also its mentioned that for small deployments it is better not to use a distributed setup. I have a json formatted dataset for text classification template about 2MB in size and it requires about 8GB of memory for the engine to train and deploy. Does this fit into small deployment category?
pio deploy
Will run the engine as a service. Logging goes to pio.log and you could add your own custom logging. I'd say so long as you have more than 8GB in RAM to spare, stick to the single machine aka "small deployment".