In AWS I will create an AMI, copy to other regions and make them all public. My customers can then choose from Community AMIs.
I have been trying to replicate this workflow in GCP and I found that GCP does not have an option of community images. And you cannot make it 'public' either. But you can use gcloud compute images export command to export an image to an external file and upload it to a bucket.
But how to use this to create an instance? I checked console to 'create VM Instance' but it does not have an option to upload or choose from drive. Only public and custom images already in your account.
To share custom image publicly.
Make custom image public using below command
gcloud compute images add-iam-policy-binding [image-name] \
--member='allAuthenticatedUsers' \
--role='roles/compute.imageUser'
Get public URL of custom image
gcloud compute images list --uri | grep [image-name]
It will be in https://www.googleapis.com/compute/v1/projects/[project-name]/global/images/[image-name] format
Create VM using public image URL
gcloud beta compute instances create instance-1 --zone=us-central1-a \
--machine-type=n1-standard-1 --subnet=default \
--image=https://www.googleapis.com/compute/v1/projects/[project-name]/global/images/[image-name] \
--boot-disk-size=10GB --boot-disk-device-name=instance-1
For details, gcp manage images here
Related
When I do something in GCP console (by clicking in GUI), I imagine some gcloud command is executed underneath. Is it possible to view this command?
(I created a notebooks instance on Vertex AI and wanted to know what exactly I should put after gcloud notebooks instances create... to get the same result)
I think it's not possible to view a gcloud command from GUI.
You should test your gcloud command to create another instance alongside the current with all the needed parameters.
When the 2 instances are the same, you know that your gcloud command is ready.
The documentation seems to be clear and complete for this :
https://cloud.google.com/vertex-ai/docs/workbench/user-managed/create-new#gcloud
If it's possible for you, you can also think about Terraform to automate this creation for you with a state management.
Try this for a Python based User Managed Notebook (GUI version of Python instance is using the base image as boot disk, which does not containg Pythong.
The Python suite is installed explicitly via Metadata parameters):
export NETWORK_URI="NETWORK URI"
export SUBNET_URI="SUBNET URI"
export INSTANCE_NAME="instance-name-of-your-liking"
export VM_IMAGE_PROJECT="deeplearning-platform-release"
export VM_IMAGE_FAMILY="common-cpu-notebooks-debian-10"
export MACHINE_TYPE="n1-standard-4"
export LOCATION="europe-west3-b"
gcloud notebooks instances create $INSTANCE_NAME \
--no-public-ip \
--vm-image-project=$VM_IMAGE_PROJECT \
--vm-image-family=$VM_IMAGE_FAMILY \
--machine-type=$MACHINE_TYPE \
--location=$LOCATION \
--network=$NETWORK_URI \
--subnet=$SUBNET_URI \
--metadata=framework=NumPy/SciPy/scikit-learn,report-system-health=true,proxy-mode=service_account,shutdown-script=/opt/deeplearning/bin/shutdown_script.sh,notebooks-api=PROD,enable-guest-attributes=TRUE
To get a list of Network URIs in your project:
gcloud compute networks list --uri
To get a list of Subnet URIs in your project:
gcloud compute networks subnets list --uri
Put the corresponding URIs in between the quotation marks in the first two variables:
export NETWORK_URI="NETWORK URI"
export SUBNET_URI="SUBNET URI"
Name the instance (keep the quotation marks):
export INSTANCE_NAME="instance-name-of-your-liking"
When done copy paste the complete block in your Google Cloud Shell (assuming you are in a correct project).
To additionally enable secure boot (which is a thick box in the GUI setup):
gcloud compute instances stop $INSTANCE_NAME
gcloud compute instances update $INSTANCE_NAME --shielded-secure-boot
Hope it works for you, as it does for me.
I'm currently using windows VM instance in GCP(project1), i want to create same VM with data in my another project(project2)(project in another gmail account). The project2 have access to my project1 which i had setup IAM role.
So what the next step to copy the instance between the project1 to project2 (without loosing the data)?
You can copy a VM instance from one project to another either using a custom image or using disk.
Method 1: Using Custom image
You have to use the following steps to create a mirror VM in another project:
In your source project, create a snapshot of the VM's boot disk, using one of the below command:
$ gcloud compute snapshots create <SNAPSHOT_NAME> --source-disk <SOURCE_DISK> --source-disk-zone <SOURCE_DISK_ZONE>
Create a custom image from the snapshot using the following command:
$ gcloud compute images create <IMAGE_NAME> --source-snapshot=<SOURCE_SNAPSHOT> [--storage-location=<LOCATION>]
In your destination project, create a VM from the custom image using the following command:
$ gcloud compute instances create <VM_NAME> --image-project <IMAGE_PROJECT> [--image <IMAGE> | --image-family <IMAGE_FAMILY>]
Method 2: Using disk
In your source project, create a snapshot of the VM's boot disk, using one of the below command:
$ gcloud compute snapshots create <SNAPSHOT_NAME> --source-disk <SOURCE_DISK> --source-disk-zone <SOURCE_DISK_ZONE>
Then create a disk in the destination project with --source-snapshot:
$ gcloud compute disks create <DISK_NAME> --source-snapshot <SOURCE_SNAPSHOT> --project <destination-project>
Create a VM based on the new disk from step 2
$ gcloud compute instances create <VM_NAME> --project <destination-project> --disk name=<DISK_NAME>,boot=yes
Refer Copying VMs between projects for information.
I have two GCP projects, projectA and projectB. In projectB, I have a machine image B-machine-image. My goal was to make a copy of the machine image in projectA. But since there is no way to copy the machine image from projectB to projectA directly, I thought I can create a instance in projectA using the machine image in projectB, then use the new instance in projectA to create a machine image locally.
I found a Google Doc for this: https://cloud.google.com/compute/docs/machine-images/create-instance-from-machine-image and followed the steps to first grant the service account in projectA admin role to the machine image B-machine-image in projectB, then run the command to make the instance like:
gcloud beta compute instances create my-new-instance --project=projectA --zone us-central1-b --source-machine-image projects/projectB/global/machineImages/B-machine-image --service-account 123456789000-compute#developer.gserviceaccount.com
and I am getting the error:
ERROR: (gcloud.beta.compute.instances.create) Could not fetch resource:
Invalid value for field 'resource.networkInterfaces[0].network': 'projects/123456789000/global/networks/projectB-vpc'. The referenced network resource cannot be found.
In projectB, I do have a VPC called projectB-vpc.
What I do not understand is that in "projects/123456789000/global/networks/projectB-vpc", 123456789000 seems is for projectA as it is part of the email for projectA service account, while projectB-vpc is for projectB. In my command I have "projects/projectB/....", why the command replaced "projectB" with 123456789000?
Any idea where did I do wrong?
Thanks,
Philip
As per the GCP documentation you can create VM using a machine image from a different project directly. Please note that, when you create a VM by using a machine image from a different project, you might not have access to the service account attached to that source project. In my reproduction steps I followed this steps:
Created an Multi-regional Machine Image
2. gcloud beta compute machine-images add-iam-policy-binding rhel-machine-image \
--project=source-project \
--member='serviceAccount:Source-compute#developer.gserviceaccount.com' \
--role='roles/compute.admin'
Added the source project service account to destination project as iam.serviceAccountUser
4. gcloud beta compute instances create machine-image-vm --project=destination-project --zone us-central1-a --source-machine-image projects/source-project/global/machineImages/rhel-machine-image --service-account Destination-compute#developer.gserviceaccount.com
On the last step I used destination project Compute Engine Default Service Account.
Created [https://www.googleapis.com/compute/beta/projects/destination-project/zones/us-central1-a/instances/machine-image-vm].
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
machine-image-vm us-central1-a e2-medium x0.xxx.0.18 35.2x3.18x.x59 RUNNING
I believe if you keep the "default" network and subnet when you created the projects, GCP does that for you(I also believe most people will keep the default around),then the commands from the Google doc will work. But in my case, I deleted those defaults and created my own network and subnet, so I have to use the extra parameters --network and --subnet in the gcloud command to make it work.
I want to create image with packer build command but when I do that I face this error :
googlecompute: * The user does not have access to service account 'service-76100000418#compute-system.iam.gserviceaccount.com'. User: 'farzin-second-test-project#farzin-second-test-project.iam.gserviceaccount.com'. Ask a project owner to grant you the iam.serviceAccountUser role on the service account
I have added the service account user role and compute admin and Compute Instance Admin (v1) to my service account.
Also I enabled the compute API ,but in my virtual box I used centos, and here I use this packer build which does not work properly.
There I create one file with name credentials and put json key in there, and also I make my packer template, I should also say my template is validated.
Does anybody have any Idea? The odd thing is it asks me to give this permission to the user, but still it is not working after adding that.
You have to enable all fallowing API's on your GCP account:
gcloud services enable sourcerepo.googleapis.com
gcloud services enable compute.googleapis.com
gcloud services enable servicemanagement.googleapis.com
gcloud services enable storage-api.googleapis.com
Also make sure your service account has the proper permissions (and you are using a proper account):
CLOUD_BUILD_ACCOUNT=$(gcloud projects get-iam-policy $PROJECT --filter="(bindings.role:roles/cloudbuild.builds.builder)" --flatten="bindings[].members" --format="value(bindings.members[])")
gcloud projects add-iam-policy-binding $PROJECT \
--member $CLOUD_BUILD_ACCOUNT \
--role roles/editor
You may also have a look at the documentation : Building VM images using Packer and Create a Cloud Build image factory using Packer.
I have a private image called 'test-1' and I can use it to create an instance
$ gcloud compute instances create demo --image test-1
However I cannot use a public image:
$ gcloud compute instances create demo --image ubuntu-1804-bionic-v20190204
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
- Invalid value for field 'resource.disks[0].initializeParams.sourceImage': 'https://www.googleapis.com/compute/v1/projects/szabgab-149/global/images/ubuntu-1804-bionic-v20190204'. The referenced image resource cannot be found.
Nor a public image family:
$ gcloud compute instances create demo --image-family ubuntu-1804-lts
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
- The resource 'projects/szabgab-149/global/images/family/ubuntu-1804-lts' was not found
What am I doing wrong here?
I've noticed that in the generated error message, that you are generating the location of the image as if it is located in your project, which I understand it is named szabgab-149:
The resource
'projects/szabgab-149/global/images/family/ubuntu-1804-lts'
The thing is that public images are located in its own projects, see the output of the command gcloud compute images list.
For example, the ubuntu-1804-bionic-v20190204 is under the project ubuntu-os-cloud and the image family ubuntu-1804-lts.
To solve this, you can simply add the --image-project flag, from the information of the previous gcloud command, you execute this command, to use this specific image:
gcloud compute instances create demo \
--image ubuntu-1804-bionic-v20190204 \
--image-project ubuntu-os-cloud
Or this one, to use the default image under the project ubuntu-os-cloud and family ubuntu-1804-lts:
gcloud compute instances create demo \
--image-family ubuntu-1804-lts \
--image-project ubuntu-os-cloud
https://github.com/khushbuparakh/gcp/blob/master/instance.py
You can use this. It creates a bucket. Upload your private file to the bucket. Create an image using that file and spin up an instance