How to open files using code editor which is stored in compute engine created under google cloud? I can open for files stored using temporary cloud shell but not for the ones stored in compute engine.
Google Cloud Shell code editor can only open files that are stored in your Cloud Shell home directory (which, by the way, is not temporary; unlike Cloud Shell VMs that are recycled, home directory persists between sessions).
Cloud Shell code editor is based on the open source Orion editor. If you like that editor, you can install it on your GCE VM and connect to it directly.
Related
I am trying to create DAG using SSH terminal of a VM, but I am unable to get where I have write the DAG script I am using cloud and installed airflow through terminal only. Please guide me step-by-step, and Do I need to install any text editor? and how to link that editor to airflow?
I have used this tutorial and understood it but in there he is using a text editor. how to connect that to airflow as I am using Cloud?
from picture, I have created a dag_folder. but how to link that with AIRFLOW_HOME/dags and also I am unable to find PATH where that AIRFLOW_HOME/dags is there.
When I download files to the bucket I use the command: gsutil -u absolute-bison-xxxxx cp $FILE gs://bucket_1 which works fine.
I am running downstream programs that I want the output to be saved to the same bucket, but when I for instance type: -output gs://bucket_1/file.out to specify the folder for the output, it does not recognise the bucket as a place to store the output. How do I set the path to the bucket?
From the command you have posted on your question, I think you are using Cloud SDK as stated in the documentation to copy files from your computer to your bucket.
If that is the case, following Kolban’s comment, you could use Cloud Storage FUSE to mount your Google Cloud Storage bucket as a file system.
In order to use Cloud Storage Fuse, you should be running an OS based on Linux kernel version 3.10 or newer, or Mac OS X 10.10.2 as stated on the installation notes; also, you could run it from Google Compute Engine VMs.
You could begin installing it following the installation notes and following this YouTube tutorial.
For more information about using Cloud Storage FUSE, or to file an issue, go to the Google Cloud [GitHub repository].
For a fully-supported file system product in Google Cloud, see Filestore.
You could use Firestore as recommended in the documentation. To be used locally on your machine, you could follow this guide.
On SSH, I've tried using gsutil -m cp filename* Desktop to copy a file from the VM Instance to my computer desktop, like Google Cloud's own example in its documentation. I got a message saying that the file was copied successfully, but no mention of downloading anything, and I don't see the relevant file on my desktop. I've tried specifying the full desktop address instead of just 'Desktop', but SSH does not recognize the address.
Is there a way I can directly download files from the VM Instance to my desktop without having to go through a Google Cloud bucket?
In my opinion this is the easiest way. Directly to local machine in a few steps.
I have been using some packages by installing them using the sudo apt-get command in the cloud shell. But now I want to make it permanent. I got this message in the shell
You are running apt-get inside of Cloud Shell. Note that your Cloud Shell
machine is ephemeral and no system-wide change will persist beyond session end.
You can customize your environment to permanently include this package by
updating your environment at https://cloud.google.com/console/cloudshell/environment/view.
So how to customize the cloud environment to include a package permanently?
You have several options.
1) Reinstall everything each time you launch Cloud Shell. This sounds bad but if you keep your files on GCS, the copy happens very fast.
2) Cloud Shell is a Docker container. You can modify that container so that you launch Cloud Shell using your customized container. Launch Cloud Shell. In the title bar on the right hand side is a icon that looks like a laptop. Click it. This will open a window with details on configuring the Docker container.
3) Keep everything local to your home directory. You home directory tree is persistent and will be restored each time your Cloud Shell VM is recreated.
I just started to use cloud to do training for my deep learning program. For now every time I modified my local .py file I have to remove the old one in the remote virtual machine of Google Cloud Platform and upload the new one. I am just curious if there is a way that I can actually open the .py file in the remote visual machine through the command line? That would be very high efficiency.
Thank you very much!
To edit a file on a machine you can SSH into there are many potential solutions.
Use scp to copy files. E.g. scp mylocalfile ssh://my-host-address/myfolder
Use ssh mounting solutions: How do you edit files over SSH?
Edit using nano your-file-to-edit (my favorite) https://www.howtogeek.com/howto/42980/the-beginners-guide-to-nano-the-linux-command-line-text-editor/
Edit using vi or vim http://vim.wikia.com/wiki/Tutorial