I have two GCP projects, projectA and projectB. In projectB, I have a machine image B-machine-image. My goal was to make a copy of the machine image in projectA. But since there is no way to copy the machine image from projectB to projectA directly, I thought I can create a instance in projectA using the machine image in projectB, then use the new instance in projectA to create a machine image locally.
I found a Google Doc for this: https://cloud.google.com/compute/docs/machine-images/create-instance-from-machine-image and followed the steps to first grant the service account in projectA admin role to the machine image B-machine-image in projectB, then run the command to make the instance like:
gcloud beta compute instances create my-new-instance --project=projectA --zone us-central1-b --source-machine-image projects/projectB/global/machineImages/B-machine-image --service-account 123456789000-compute#developer.gserviceaccount.com
and I am getting the error:
ERROR: (gcloud.beta.compute.instances.create) Could not fetch resource:
Invalid value for field 'resource.networkInterfaces[0].network': 'projects/123456789000/global/networks/projectB-vpc'. The referenced network resource cannot be found.
In projectB, I do have a VPC called projectB-vpc.
What I do not understand is that in "projects/123456789000/global/networks/projectB-vpc", 123456789000 seems is for projectA as it is part of the email for projectA service account, while projectB-vpc is for projectB. In my command I have "projects/projectB/....", why the command replaced "projectB" with 123456789000?
Any idea where did I do wrong?
Thanks,
Philip
As per the GCP documentation you can create VM using a machine image from a different project directly. Please note that, when you create a VM by using a machine image from a different project, you might not have access to the service account attached to that source project. In my reproduction steps I followed this steps:
Created an Multi-regional Machine Image
2. gcloud beta compute machine-images add-iam-policy-binding rhel-machine-image \
--project=source-project \
--member='serviceAccount:Source-compute#developer.gserviceaccount.com' \
--role='roles/compute.admin'
Added the source project service account to destination project as iam.serviceAccountUser
4. gcloud beta compute instances create machine-image-vm --project=destination-project --zone us-central1-a --source-machine-image projects/source-project/global/machineImages/rhel-machine-image --service-account Destination-compute#developer.gserviceaccount.com
On the last step I used destination project Compute Engine Default Service Account.
Created [https://www.googleapis.com/compute/beta/projects/destination-project/zones/us-central1-a/instances/machine-image-vm].
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
machine-image-vm us-central1-a e2-medium x0.xxx.0.18 35.2x3.18x.x59 RUNNING
I believe if you keep the "default" network and subnet when you created the projects, GCP does that for you(I also believe most people will keep the default around),then the commands from the Google doc will work. But in my case, I deleted those defaults and created my own network and subnet, so I have to use the extra parameters --network and --subnet in the gcloud command to make it work.
Related
I'm currently using windows VM instance in GCP(project1), i want to create same VM with data in my another project(project2)(project in another gmail account). The project2 have access to my project1 which i had setup IAM role.
So what the next step to copy the instance between the project1 to project2 (without loosing the data)?
You can copy a VM instance from one project to another either using a custom image or using disk.
Method 1: Using Custom image
You have to use the following steps to create a mirror VM in another project:
In your source project, create a snapshot of the VM's boot disk, using one of the below command:
$ gcloud compute snapshots create <SNAPSHOT_NAME> --source-disk <SOURCE_DISK> --source-disk-zone <SOURCE_DISK_ZONE>
Create a custom image from the snapshot using the following command:
$ gcloud compute images create <IMAGE_NAME> --source-snapshot=<SOURCE_SNAPSHOT> [--storage-location=<LOCATION>]
In your destination project, create a VM from the custom image using the following command:
$ gcloud compute instances create <VM_NAME> --image-project <IMAGE_PROJECT> [--image <IMAGE> | --image-family <IMAGE_FAMILY>]
Method 2: Using disk
In your source project, create a snapshot of the VM's boot disk, using one of the below command:
$ gcloud compute snapshots create <SNAPSHOT_NAME> --source-disk <SOURCE_DISK> --source-disk-zone <SOURCE_DISK_ZONE>
Then create a disk in the destination project with --source-snapshot:
$ gcloud compute disks create <DISK_NAME> --source-snapshot <SOURCE_SNAPSHOT> --project <destination-project>
Create a VM based on the new disk from step 2
$ gcloud compute instances create <VM_NAME> --project <destination-project> --disk name=<DISK_NAME>,boot=yes
Refer Copying VMs between projects for information.
So, I created a GCP Compute optimized VM and gave it full access to all cloud apis as well as full HTTP and HTTPS traffic access. I now want to create a TPU from inside this VM i.e. run the following command:
gcloud compute tpus create node-1 --zone us-central1-a --project $PROJECT_NAME --version 2.5.0 --accelerator-type v3-8 --no-async
and it constantly errors with:
ERROR: (gcloud.compute.tpus.create) PERMISSION_DENIED: Permission 'tpu.nodes.create' denied on 'projects/$PROJECT_NAME/locations/us-central1-a/nodes/node-1'
I only ever get this error in the VM, but when I run this command on my local machine with my local install of gcloud, everything works fine. It is really weird because all other commands like gcloud list and gsutil all work fine, but creating TPUs doesn't work. I even tried adding a service account into ~/.credentials and setting that in my bashrc:
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.credentials/service-account.googleapis.com.json
but this doesn't solve the problem. I even tried with the execution groups as well:
gcloud compute tpus execution-groups create --name=node-1 --zone=us-central1-a --tf-version=2.5.0 --accelerator-type=v3-8 --tpu-only --project $PROJECT_NAME
but this also fails.
Below are two possible reasons why you have Permission denied Error:
Service Account does not have Allow full access to all Cloud APIs.
Account doesn't have a role TPU ADMIN.
I tried to create TPU using your command. I got the same error before modifying the service account. Here is the output that TPU has been created.
$ gcloud compute tpus create node-1 --zone us-central1-a --project $PROJECT_NAME --version 2.5.0 --accelerator-type v3-8 --no-async \
Create request issued for: [node-1]
Waiting for operation [projects/project-id/locations/us-central1-a/operations/operation-1634780772429-5ced30f39edf6-105ccd39-96d571fa] to complete...done.
Created tpu [node-1].
Try creating the TPU again after following these instructions:
a. Make sure to Enable TPU API
b. Go to VM Instance and stop/down VM before editing service account.
c. Refresh VM instance page and click Edit
d. At the bottom of Instance details page Select Compute Engine Service Account and Allow full Access to all Cloud APIs and Save.
(As recommended by #John Hanley)
e. On your Instance Page check and note your Service Account.
f. Go to IAM page and look for the Service Account and Edit
g. Click Add Role and select TPU ADMIN and Save
h. Start your VM instance and SSH to Server
i. Run this command
gcloud compute tpus create node-1 --zone us-central1-a --project $PROJECT_NAME --version 2.5.0 --accelerator-type v3-8 --no-async
I encountered error at first because there was existing TPU on the same zone I entered. Make sure that your TPU has not been created with the same zone.
I'm trying to create a Compute Engine VM instance sample in Google Cloud that has an associated startup script startup_script.sh. On startup, I would like to have access to files that I have stored in a Cloud Source Repository. As such, in this script, I clone a repository using
gcloud source repos clone <repo name> --project=<project name>
Additionally, startup_script.sh also runs commands such as
gcloud iam service-accounts keys create key.json --iam-account <account>
which creates .json credentials, and
EXTERNAL_IP = $(gcloud compute instances describe sample --format='get(networkInterfaces[0].accessConfigs[0].natIP)' --zone=us-central1-a)
to get the external IP of the VM within the VM. To run these commands without any errors, I found that I need partial or full access to multiple Cloud API access scopes.
If I manually edit the scopes of the VM after I've already created it to allow for this and restart it, startup_script.sh runs fine, i.e. I can see the results of each command completing successfully. However, I would like to assign these scopes upon creation of the VM and not have to manually edit scopes after the fact. I found in the documentation that in order to do this, I can run
gcloud compute instances create sample --image-family=ubuntu-1804-lts --image-project=ubuntu-os-cloud --metadata-from-file=startup-script=startup_script.sh --zone=us-central1-a --scopes=[cloud-platform, cloud-source-repos, default]
When I run this command in the Cloud Shell, however, I can either only add one scope at a time, i.e. --scopes=cloud_platform, or if I try to enter multiple scopes as shown in the command above, I get
ERROR: (gcloud.compute.instances.create) unrecognized arguments:
cloud-source-repos,
default]
Adding multiple scopes as the documentation suggests doesn't seem to work. I get a similar error when use the scope's URI instead of it's alias.
Any obvious reasons as to why this may be happening? I feel this may have to do with the service account (or lack thereof) associated with the sample VM, but I'm not entirely familiar with this.
BONUS: Ideally I would like to run the VM creation cloud shell command in a cloudbuild.yaml file, which I have as
steps:
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: gcloud
args: ['compute', 'instances', 'create', 'sample', '--image-family=ubuntu-1804-lts', '--image-project=ubuntu-os-cloud', '--metadata-from-file=startup-script=startup_sample.sh', '--zone=us-central1-a', '--scopes=[cloud-platform, cloud-source-repos, default]']
I can submit the build using
gcloud builds submit --config cloudbuild.yaml .
Are there any issues with the way I've setup this cloudbuild.yaml?
Adding multiple scopes as the documentation suggests doesn't seem to work
Please use the this command with --scopes=cloud-platform,cloud-source-reposCreated and not --scopes=[cloud-platform, cloud-source-repos, default]:
gcloud compute instances create sample --image-family=ubuntu-1804-lts --image-project=ubuntu-os-cloud --zone=us-central1-a --scopes=cloud-platform,cloud-source-reposCreated
[https://www.googleapis.com/compute/v1/projects/wave25-vladoi/zones/us-central1-a/instances/sample].
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
sample us-central1-a n1-standard-1 10.128.0.17 35.238.166.75 RUNNING
Also consider #John Hanley comment.
I am new to google cloud. I have seen the similar question but I couldn't understand the answer. It will be great if someone could give easy instruction to tackle this problem.
I have two linux VM instances under same project on google cloud. I want to copy files from one VM to other VM.
I tried copy-files command. It threw error "deprecated, use scp instead"
I tried "gcloud compute scp user#vm2_instance_name:vm2_instance_file_path"
other answers say use "service account". I read about them and created one and created key as well in .json format but not sure what to do after that. Appreciate any suggestions.
If you are in one instance, don't worry about Google Cloud. Simply perform a scp to copy file from VM to another one.
If you don't have customize users on the VM, you can omit it
scp <my local file path> <vm name>:<destination path>
About service account, if your VM are in Google Cloud, they have the compute engine service account by default <projectNumber>-compute#developer.gserviceaccount.com
You can customize this service account if you want. This service account is mandatory to identify the VM which perform API call or gcloud command
Google's documentation addresses this. Personally, I have always preferred using gcloud compute scp as it provides both a simplistic way of performing transfers while not necessarily taking away any of the complexities and features that other transferring options provide.
In any case, in the documentation provided you will most likely find the method that are more in-line with what you want.
This is the solution that worked for me:
1. gcloud compute instances list
NAME ZONE MACHINE_TYPE PREEMPTIBLE
INTERNAL_IP EXTERNAL_IP STATUS
instance-1 us-central1-a n2-standard-8
10.128.0.60 34.66.177.187 RUNNING
instance-2 us-central1-a n1-standard-1
10.128.15.192 34.69.216.153 STAGING
2. gcloud compute ssh instance-1 --zone=us-central1-a
3. user#instance-1:~$ ls
myfile
4. usernstance-1:~$ gcloud compute scp myfile user#instance-2:myfile
5. gcloud compute ssh instance-2 --zone=us-central1-a
6. user#instance-2:~$ ls
myfile
I'm now learning Google Cloud Platform instance creation. As part of learning, trying to launch RHEL 6 instance on a f1.micro instance-type in us-east1-b region.
Here's is the Gcloud command I've used:
gcloud compute --project=<project-id> instances create cldinit-vm --zone=us-east1-b --machine-type=f1-micro--subnet=default --network-tier=PREMIUM --metadata-from-file startup-script=initscript.sh --maintenance-policy=MIGRATE --service-account=<account-id>#developer.gserviceaccount.com --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append --min-cpu-platform="Intel Broadwell" --tags=http-server --image=rhel-6-v20181210 --image-project=rhel-cloud --boot-disk-size=10GB --boot-disk-type=pd-standard --boot-disk-device-name=cldinit-vm --labels=name=cloudinit-vm
When I run the command, it is showing the error below,
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
- Invalid value for field 'resource.machineType': 'https://www.googleapis.com/compute/v1/projects/<project-id>/zones/us-east1-b/machineTypes/f1-micro--subnet=default'.
Machine type with name 'f1-micro--subnet=default' does not exist in zone 'us-east1-b'.
I've two questions:
I could not modify the Subnet settings from "default", as it is the only option available to choose from "network" in instance launching page.
So could anyone help to resolve the issue please?
Since I'm learning GCP, I've launched the CLI command into "CloudShell" directly from the link located at bottom of GCP compute engine - instance launching page.
Is there a correction needs to be done from "Google" to provide the working command ?
As part of learning, found that there was a missing space in between the option value f1-micro and --subnet.
So here is the corrected command snippet
gcloud compute --project=<project-id> instances create cldinit-vm --zone=us-east1-b --machine-type=f1-micro --subnet=default ....