I would like to use autossh instead of the ssh command run by google cloud while doing
gcloud compute ssh --zone myzone --project myproject mymachine
What is the command line to replace it?
A replacement for the command line
Here is the two commands that allow you to use autossh:
gcloud compute config-ssh
autossh -M9042 mymachine.myzone.myproject
In my opinion, this is the easiest way to do it. The first line export the ssh key and configuration of your machines, and the second just use this alias.
More on ssh configuration
This also allow you to use the shortcut mymachine.myzone.myproject for scp, ssh, and all ssh related tools. Indeed, the list of your machines are exported in your ~/.ssh/config.
This is a list of aliases you can also configure by hand.
For more information on gcloud's config-ssh, you can refer to the official documentation.
Notice that some shell environment will provide you with ssh command autocompletion, which is really helpful when it comme to remembering the name of your machines. Here is a reference for zsh.
Related
I use gcloud cloud-shell ssh to connect to Google Cloud Shell. However, this spawns a new window, (PuTTY) which bothers me. Is there a way (some -- flag / &c.) to use it from the same console window?
Thanks in advance!
The CLI is just a wrapper for launching Putty (on Windows) with the correct command line options. The SSH functionality is not built into gcloud.
Using --command flag, you can run the command on the virtual machine without opening new console window.
gcloud compute ssh your-instance --zone=your-zone --command="ps aux"
It runs the command on the target instance and then exits.
Use gcloud compute ssh --help command to learn more.
Instead of directly opening the SSH session with gcloud beta compute ssh you can set up a tunnel from a local port to the GCP instance and then use your preferred terminal to ssh over the tunnel.
You set up the tunnel with:
gcloud compute start-iap-tunnel --zone <zone> <name of instance> 22 --project <name of project> --local-host-port=localhost:4226
You open an SSH connection to localhost:4226 as you would do to the instance with:
ssh <user>#localhost -p 4226 -i <identity file>
(EDIT: make sure to include the identity file for GCP access)
I've somehow ended up as two different users depending on where I'm connecting from. I think it's the result of my org creating users for different projects. If I execute whoami from my local terminal I'm foo but if I execute the command from the ssh.cloud terminal I'm foo_foobar.
I have a folder projects on the VM and I can see it from both terminals, but all the subfolders that belong to foo are not visible to foo_foobar. OK, I get it.
The biggest issue is that from my local terminal, as foo I can't pull from or push to a cloud repo.
So my ask is: does there exist a cli command that will let me connect as foo_foobar from my local? I've looked at my config with gcloud config list and the email and project ID are correct. Thanks
The solution was to use this ssh command: gcloud compute ssh foo_foobar#foo-vm --zone us-west1-b
The confusion was caused by the command provided in the dashboard from the SSH dropdown View gcloud command which is: gcloud beta compute ssh with no "foo_foobar#foo-vm". In other words the provided ssh command does not indicate that username#instance_name should be part of the command
I am new to google cloud. I have seen the similar question but I couldn't understand the answer. It will be great if someone could give easy instruction to tackle this problem.
I have two linux VM instances under same project on google cloud. I want to copy files from one VM to other VM.
I tried copy-files command. It threw error "deprecated, use scp instead"
I tried "gcloud compute scp user#vm2_instance_name:vm2_instance_file_path"
other answers say use "service account". I read about them and created one and created key as well in .json format but not sure what to do after that. Appreciate any suggestions.
If you are in one instance, don't worry about Google Cloud. Simply perform a scp to copy file from VM to another one.
If you don't have customize users on the VM, you can omit it
scp <my local file path> <vm name>:<destination path>
About service account, if your VM are in Google Cloud, they have the compute engine service account by default <projectNumber>-compute#developer.gserviceaccount.com
You can customize this service account if you want. This service account is mandatory to identify the VM which perform API call or gcloud command
Google's documentation addresses this. Personally, I have always preferred using gcloud compute scp as it provides both a simplistic way of performing transfers while not necessarily taking away any of the complexities and features that other transferring options provide.
In any case, in the documentation provided you will most likely find the method that are more in-line with what you want.
This is the solution that worked for me:
1. gcloud compute instances list
NAME ZONE MACHINE_TYPE PREEMPTIBLE
INTERNAL_IP EXTERNAL_IP STATUS
instance-1 us-central1-a n2-standard-8
10.128.0.60 34.66.177.187 RUNNING
instance-2 us-central1-a n1-standard-1
10.128.15.192 34.69.216.153 STAGING
2. gcloud compute ssh instance-1 --zone=us-central1-a
3. user#instance-1:~$ ls
myfile
4. usernstance-1:~$ gcloud compute scp myfile user#instance-2:myfile
5. gcloud compute ssh instance-2 --zone=us-central1-a
6. user#instance-2:~$ ls
myfile
I have managed to set up a VM instance on Google cloud platform using the following instructions:
https://towardsdatascience.com/running-jupyter-notebook-in-google-cloud-platform-in-15-min-61e16da34d52
I am then able to run a Jupyter notebook as per the instructions.
Now I want to be able to use my own data in the notebook....this is where I am really struggling. I downloaded the Cloud SDK onto my mac and ran this from the terminal (as per https://cloud.google.com/compute/docs/instances/transfer-files)
My-MacBook-Air:~ me$ gcloud compute scp /Users/me/Desktop/my_data.csv aml-test:~/amlfolder
where aml-test is the name of my instance and amlfolder a folder I created on the VM instance. I don't get any error messages and it seems to work (the terminal displays the following after I run it >> 100% 66MB 1.0MB/s 01:03 )
However when I connect to my VM instance via the SSH button on the google console and type
cd amlfolder
ls
I cannot see any files! (nor can I see them from the jupyter notebook homepage)
I cannot figure out how to use my own data in a python jupyter notebook on a GCP VM instance. I have been trying/googling for an entire day. As you might have guessed I'm a complete newbie to GCP (and cd, ls and mkdir is the extent of my linux command knowledge!)
I also tried using Google Cloud Storage - I uploaded the data into a google storage bucket (as per https://cloud.google.com/compute/docs/instances/transfer-files) but don't know how to complete the last step '4. On your instance, download files from the bucket.'
If anyone can figure out what i am doing wrong, or an easier method to get my own data running into a python jupyter notebook on GCP than using gcloud scp command please help!
Definitely try writing
pwd
to verify you're in the path you think you are, there's a chance that your scp command and the console SSH command login as different users.
To copy data from a bucket to the instance, do
gsutil cp gcs://bucket-name/you-file .
As you can see in gcloud compute docs , gcloud compute scp /Users/me/Desktop/my_data.csv aml-test:~/amlfolder will use your local environment username, so the tilde in your command refers to the home directory of a username that is the same name as your local.
But when you SSH from the Browser as you can see from docs that your Gmail username will be used.
So, you should check the home directory of the user used by gcloud compute scp ... command.
The easiest way to check, SSH to your VM and run
ls /home/ --recursive
I have two projects in Google Cloud and I need to copy files from an instance in one project to an instance in another project. I tried to using the 'gcloud compute copy-files' command but I'm getting this error:
gcloud compute copy-files test.tgz --project stack-complete-343 instance-IP:/home/ubuntu --zone us-central1-a
ERROR: (gcloud.compute.copy-files) Could not fetch instance: - Insufficient Permission
I was able to replicate your issue with a brand new VM instance, getting the same error. Here are a few steps that I took to correct the problem:
Make sure you are authenticated and have rights to both projects with the same account!
$ gcloud config list (if you see the service account #developer.gserviceaccount.com, you need to switch to the account that is enabled on both projects. you can check that from the Devlopers Console > Permissions)
$ gcloud auth login (copy the link to a new window, login, copy the code and paste it back in the prompt)
$ gcloud compute scp test.tgz --project stack-complete-343 instance-IP:/home/ubuntu --zone us-central1-a (I would also use the instance name instead of the IP)
This last command should also generate your ssh keys. You should see something like this, but do not worry about entering a passphrase :
WARNING: [/usr/bin/ssh-keygen] will be executed to generate a key.
Generating public/private rsa key pair
Enter passphrase (empty for no passphrase):
Go to the permissions tab on the remote instance(i.e. the instance you WON'T be running gcloud compute copy-files on). Then go to service accounts and create a new one, give it a name and check the box to get a key file for it and leave JSON selected. Upload that key file from your personal machine using gcloud compute copy-files and your personal account to the local instance(i.e. the machine you're SSHing into and running the gcloud compute copy-files command on.) Then run this from the local instance via SSH. gcloud auth activate-service-account ACCOUNT --key-file KEY-FILE replacing ACCOUNT with the email like address that was generated and KEY-FILE with the path to the key file you uploaded from your personal machine earlier. Then you should be able to access the instance that setup the account. These steps have to be repeated on every instance you want to copy files between. If these instructions weren't clear let me know and I'll try to help out.
It's not recommended to auth your account on Compute Engine instances because that can expose your credentials to anybody with access to the machine.
Instead, you can let your service accounts use the Compute Engine API. First, stop the instance. Once stopped you can edit Cloud API access scopes from the console. Modify the Compute Engine scope from Disabled to Read Only.
You should be able to just use the copy-files command now. This lets your service account access the Compute Engine API.
The most simple way to to this will be using 'scp' command and .pem file. Here's as example
sudo scp -r -i your/path_to/.pem your_username#ip_address_of_instance:path/to/copy/file
If both of them are in the same project this is the simplest way
gcloud compute copy-files yourFileName --project yourProjectName instance-name:~/folderInInstance --zone europe-west1-b
Obviously you should edit the zone according to your instances.
One of the approaches to get permissions is to enable Cloud API access scopes. You may set them to Allow full access to all Cloud APIs.
In console click on the instance and use EDIT button above. Scroll to the bottom and change Cloud API access scopes. See also this answer.