How would I copy files from a remote server to a google bucket? For example,
gcloud compute scp username#server:/path/to/file gs://my-bucket
This method gives the error: All sources must be local files when destination is remote.
Additionally, gsutil only provides support for cp and not scp.
Thanks in advance!
You can also directly execute gsutil command on your GCE VM (Most VM images have Cloud SDK preinstalled). For example:
gcloud compute ssh user#server --zone my_zone \
--command='gsutil cp path/to/my_file gs://MY_BUCKET'
Note that for this to work your service account associated with VM must have appropriate access scope to GCS. If you run
gcloud beta compute instances describe my_instance --zone my_zone \
--format="value(serviceAccounts.scopes)"
It will show list of scopes set for VM service account. Make sure you have https://www.googleapis.com/auth/cloud-platform or https://www.googleapis.com/auth/devstorage.full_control or https://www.googleapis.com/auth/devstorage.read_write. If not you can use
set-scopes beta command to reset them or go to console and edit VM in question.
gsutil supports streaming data, so you can try to pipe the data in.
Not 100% certain this works with arbitrary binary data (and I'm on a phone so I can't test it)
gcloud compute ssh username#server --command='cat /path/to/file' | gsutil cp - gs://my-bucket
Related
I am trying to transfer the file samtools.tar.bz2 from inside a VM instance to another server outside GCP called username#server.uni.no: However, I get the following error:
gcloud beta compute scp --project "absolute-bison-xxxx" --zone "europe-west4-a" samtools.tar.bz2 username#server.uni.no:/home/local/user
error
ERROR: (gcloud.beta.compute.scp) Could not fetch resource: - Invalid value 'server.uni.no'. Values must match the following regular expression: '[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?|[1-9][0-9]{0,19}'
Take a look at the documentation:
(BETA) gcloud beta compute scp securely copies files between a virtual machine instance and your local machine using the scp command.
(...)
When the destination of your transfer is local, all source files must be from the same virtual machine.
When the destination of your transfer is remote instead, all sources must be local.
It won't work with two remote machines, even if they are both in GCP.
A crude workaround is to download the file to your local machine:
gcloud beta compute scp --project "absolute-bison-xxxx" --zone "europe-west4-a" USERNAME#INSTANCE_NAME:~/samtools.tar.bz2 samtools.tar.bz2
and then upload it to the external server:
scp samtools.tar.bz username#server.uni.no:/home/local/user
I am new to google cloud. I have seen the similar question but I couldn't understand the answer. It will be great if someone could give easy instruction to tackle this problem.
I have two linux VM instances under same project on google cloud. I want to copy files from one VM to other VM.
I tried copy-files command. It threw error "deprecated, use scp instead"
I tried "gcloud compute scp user#vm2_instance_name:vm2_instance_file_path"
other answers say use "service account". I read about them and created one and created key as well in .json format but not sure what to do after that. Appreciate any suggestions.
If you are in one instance, don't worry about Google Cloud. Simply perform a scp to copy file from VM to another one.
If you don't have customize users on the VM, you can omit it
scp <my local file path> <vm name>:<destination path>
About service account, if your VM are in Google Cloud, they have the compute engine service account by default <projectNumber>-compute#developer.gserviceaccount.com
You can customize this service account if you want. This service account is mandatory to identify the VM which perform API call or gcloud command
Google's documentation addresses this. Personally, I have always preferred using gcloud compute scp as it provides both a simplistic way of performing transfers while not necessarily taking away any of the complexities and features that other transferring options provide.
In any case, in the documentation provided you will most likely find the method that are more in-line with what you want.
This is the solution that worked for me:
1. gcloud compute instances list
NAME ZONE MACHINE_TYPE PREEMPTIBLE
INTERNAL_IP EXTERNAL_IP STATUS
instance-1 us-central1-a n2-standard-8
10.128.0.60 34.66.177.187 RUNNING
instance-2 us-central1-a n1-standard-1
10.128.15.192 34.69.216.153 STAGING
2. gcloud compute ssh instance-1 --zone=us-central1-a
3. user#instance-1:~$ ls
myfile
4. usernstance-1:~$ gcloud compute scp myfile user#instance-2:myfile
5. gcloud compute ssh instance-2 --zone=us-central1-a
6. user#instance-2:~$ ls
myfile
I have tried using this code, but it didn't work.
gcloud compute ssh user#ftp.example.com --zone my_zone \
--command='gsutil cp path/to/my_file gs://MY_BUCKET'
I tried even copying files from the server via http request too, no results. I have made the files available to public and tried this code.
gsutil cp http://example.com/dir1/ gs://MY_BUCKET
I am getting errors as such the http is unrecognized.
I also have the ssh access to the remote server.
gsutil cp allows you to stream in data it receives from stdin per documentation here: https://cloud.google.com/storage/docs/gsutil/commands/cp Pasted below:
Streaming Transfers Use '-' in place of src_url or dst_url to perform
a streaming transfer. For example:
long_running_computation | gsutil cp - gs://my-bucket/obj Streaming
uploads using the JSON API (see gsutil help apis) are buffered in
memory part-way back into the file and can thus retry in the event of
network or service problems.
Using this, we can stream the scp copy from your remote server to standard out (if you're on linux or mac) and pipe it into the gsutil cp like this:
scp <USER>#<YOUR_SERVER>:/<PATH>/<FILE_NAME> /dev/stdout | gsutil cp - gs://<BUCKET_NAME>/<FILE_NAME>
What you should be doing is not a regular copy, but an scp - secured copy.
gcloud compute scp securely copies files between a virtual machine instance and your local machine using the scp command.
More information on all available switches and descriptions can be found on online docs
Combining the other answers, to copy a file from a Compute Engine instance to Google Cloud Storage, you need the following command:
gcloud compute scp --zone [ZONE] [USER]#[INSTANCE_NAME]:/path/to/file /dev/stdout | gsutil cp - gs://[BUCKET_NAME]/[FILE_NAME]
If the file is located in a remote server other than a Compute Engine instance, you can use "scp" as Nicholas said.
I have managed to set up a VM instance on Google cloud platform using the following instructions:
https://towardsdatascience.com/running-jupyter-notebook-in-google-cloud-platform-in-15-min-61e16da34d52
I am then able to run a Jupyter notebook as per the instructions.
Now I want to be able to use my own data in the notebook....this is where I am really struggling. I downloaded the Cloud SDK onto my mac and ran this from the terminal (as per https://cloud.google.com/compute/docs/instances/transfer-files)
My-MacBook-Air:~ me$ gcloud compute scp /Users/me/Desktop/my_data.csv aml-test:~/amlfolder
where aml-test is the name of my instance and amlfolder a folder I created on the VM instance. I don't get any error messages and it seems to work (the terminal displays the following after I run it >> 100% 66MB 1.0MB/s 01:03 )
However when I connect to my VM instance via the SSH button on the google console and type
cd amlfolder
ls
I cannot see any files! (nor can I see them from the jupyter notebook homepage)
I cannot figure out how to use my own data in a python jupyter notebook on a GCP VM instance. I have been trying/googling for an entire day. As you might have guessed I'm a complete newbie to GCP (and cd, ls and mkdir is the extent of my linux command knowledge!)
I also tried using Google Cloud Storage - I uploaded the data into a google storage bucket (as per https://cloud.google.com/compute/docs/instances/transfer-files) but don't know how to complete the last step '4. On your instance, download files from the bucket.'
If anyone can figure out what i am doing wrong, or an easier method to get my own data running into a python jupyter notebook on GCP than using gcloud scp command please help!
Definitely try writing
pwd
to verify you're in the path you think you are, there's a chance that your scp command and the console SSH command login as different users.
To copy data from a bucket to the instance, do
gsutil cp gcs://bucket-name/you-file .
As you can see in gcloud compute docs , gcloud compute scp /Users/me/Desktop/my_data.csv aml-test:~/amlfolder will use your local environment username, so the tilde in your command refers to the home directory of a username that is the same name as your local.
But when you SSH from the Browser as you can see from docs that your Gmail username will be used.
So, you should check the home directory of the user used by gcloud compute scp ... command.
The easiest way to check, SSH to your VM and run
ls /home/ --recursive
I would like to verify my website in Google Search Console by uploading a file to my website - example.com which is hosted on Google Cloud Compute Engine (Wordpress site deployed with Google Cloud Launcher) . It has to be placed on a website as example.com/123googlesearchconsoleexample.html , but I cannot upload it via Cloud Shell from a Google Cloud Storage bucket.
Is there a way to do this?
I tried gcloud compute copy-files command
gcloud compute copy-files gs://example/123googlesearchconsoleexample.html example-prd-vm:~/
, but it seems that it does not work in the browser and this is how I would like to do it (without installing SDK).
The error I get is:
(gcloud.compute.copy-files) All sources must be local files when destination is remote.
I don't know if there's a way of doing it in a single step, but it looks like it should be easy to do in two steps:
$ gsutil cp gs://example/123googlesearchconsoleexample.html .
$ gcloud compute copy-files 123googlesearchconsoleexample.html example-prd-vm:~/
The first command copies from Google Cloud Storage to the VM running Cloud Shell. The second command then copies from that VM to example-prd-vm... it's now copying a local file, so the error you're currently getting won't be applicable.
First list all the buckets you have using the following command
gsutil ls
Use the following command to download a file from your Google Cloud Storage bucket to your local location
gsutil cp gs://bucketname/filename local-location
You can use the -r option to download a folder from GCS
gsutil cp -r gs://bucketname/folder-name local-location
You can also use the -m option to download large number of files which performs a parallel (multi-threaded/multi-processing) copy
gsutil -m cp -r gs://bucketname/folder-name local-location
In order to copy one file:
gsutil -m cp "gs://yourbucketnam/folder/../filename.txt"
In order to copy the whole folder:
gsutil -m cp "gs://yourbucketnam/folder"
In order to copy a folder and all its contents.