I just started to use cloud to do training for my deep learning program. For now every time I modified my local .py file I have to remove the old one in the remote virtual machine of Google Cloud Platform and upload the new one. I am just curious if there is a way that I can actually open the .py file in the remote visual machine through the command line? That would be very high efficiency.
Thank you very much!
To edit a file on a machine you can SSH into there are many potential solutions.
Use scp to copy files. E.g. scp mylocalfile ssh://my-host-address/myfolder
Use ssh mounting solutions: How do you edit files over SSH?
Edit using nano your-file-to-edit (my favorite) https://www.howtogeek.com/howto/42980/the-beginners-guide-to-nano-the-linux-command-line-text-editor/
Edit using vi or vim http://vim.wikia.com/wiki/Tutorial
Related
I have some pdf's into linux server and I want to download them into my pc. I am trying to use somthing like this
scp username#instance_name:/path_to_this_file/XXX.pdf C:\Users\username\Desctop
but isn't working, do you have any idea how to deal with that?
It would be helpful if you include in your question the errors you're receiving.
scp is a *nix command (not Windows) and so you likely can't use that command on Windows.
See: Transferring files to (from) Linux VMs
If you have a Linux shell (e.g. WSL) then the easiest way is probably to use gcloud compute scp which provides a wrapper around scp for you.
On SSH, I've tried using gsutil -m cp filename* Desktop to copy a file from the VM Instance to my computer desktop, like Google Cloud's own example in its documentation. I got a message saying that the file was copied successfully, but no mention of downloading anything, and I don't see the relevant file on my desktop. I've tried specifying the full desktop address instead of just 'Desktop', but SSH does not recognize the address.
Is there a way I can directly download files from the VM Instance to my desktop without having to go through a Google Cloud bucket?
In my opinion this is the easiest way. Directly to local machine in a few steps.
I made a fairly standard deployment of the Single-Node File Server on Google Cloud. It works fine as I can mount the file server's disk from other instances.
However, now I want to add another disk to the same file server. The documentation says I should use the following command to add another file system:
zfs create storagepool_name/file_system_name
I tried to run this command on the VM that is acting as the file server, but I get the error that the command zfs is not found.
Now I can probably install zfs myself, but I wonder whether that will somehow collide with whatever the deployment has already set up on the machine.
Is installing and setting up zfs myself a problem? If so, how do I add another disk to the file server?
I figured out what went wrong with my setup of the Single-Node File Server.
First, the default deployment settings seems to choose xfs as the default file system instead of zfs. The file server I had was using xfs, as can be seen in the metadata of the instance itself.
Secondly, as user John Hanley commented in my question, even with zfs selected as the file system, only the root user has its PATH variable set-up properly to be able to directly use the zfs command.
I am stuck in a technical issue on a project and I think you the forum could help me out.
I have an EC2 Instance Type:p2.xlarge running on AWS, I cloned a repository in this instance which requires pytorch and cuda dependencies(this point has been taken care of).
Now, The issue is that I wanna work & run this code-base(which is is AWS instance now) somehow in my local pyCHARM IDE. In short, I didn't have proper resources on my laptop to run the repository, so I have to run in an AWS instance but for debugging purposes the local IDE would be a great option.
Is it possible to do that?. In other words, we can do SSH into AWS instance and run code, but all will be done through command line, if we could SSH through PYCHARM and can see the code in AWS here in local machine within PYCHARM and change, debug or run it as it was local but actually it gets executed in the instance.
Please suggest a solution to it.
Thanks in advance.
EDIT-1:
After following, #Cromulent suggestion, I have arrived here
Setting the remote:
Upload happening within the local & remote repo.
I still didn't understand the requirement of syncing the local and remote folders, when I only want to open the remote folder in my PYCHARM IDE and work on it.
I think after this setup, I have to change the code in local copy and the PYCHARM will sync the code in remote copy. How will I be running(using resources-GPUs of the remote Instance, not my local machine.) the remote code in PYCHARM in this scenario, I am just syncing it, for running again I have to ssh through command line and run the script(This does not serve the purpose)?
EDIT-2:
After #Cromulent suggestions.
Actually, it did work, but still, I am not able to run the remote code locally.
I am getting the below error while running any remote script. If I run the same script using ssh in the terminal, the scripts run normally. I tried to fix the problem using this post on StackOverflow, but it didn't work too.
ssh://ubuntu#ec2-52-41-247-169.us-west-2.compute.amazonaws.com:22/home/ubuntu/anaconda3/bin/python -u <08ad9807-3477-4916-96ce-ba6155e3ff4c>/home/ubuntu/InsightProject/scripts/download_flownet2.py
/home/ubuntu/anaconda3/bin/python: can't open file '<08ad9807-3477-4916-96ce-ba6155e3ff4c>/home/ubuntu/InsightProject/scripts/download_flownet2.py': [Errno 2] No such file or directory
The below is the screenshot for the above problem:
PyCharm Professional supports remote Python interpreters (either the globally installed Python interpreter or a virtualenv). It works by creating an SSH connection to the server and then running the code on the remote host. The results are then displayed locally in PyCharm Professional. You can also do remote debugging as well.
You MUST be using the professional version of PyCharm though. The free community version does not support this feature.
You can find the documentation here:
https://www.jetbrains.com/help/pycharm/configuring-remote-interpreters-via-ssh.html
One more solution is to deploy a Jupyter Notebook on your remote server. Then you will be able to use it from PyCharm Professional Edition.
Don't forget to make rules for the jupyter ports (e.g. allow all 8888) in your AWS console and in your instance.
To configure a remote interpreter for your notebook do this (source):
Open the Jupyter Notebook page of the Settings/Preferences dialog.
On this page, select or clear the Markdown cells rendering enabled option, and specify the username and password. Note that for the
single-user notebooks these fields are optional - leave them blank.
Fill in the username (for JupyterHub) and password.
Click the link Configure remote interpreter. You'll find yourself at the Project Interpreter page.
Configure the remote interpreter, as described in the section Configuring Python Interpreter.
You will want to configure a remote interpreter.
I tried the above approach but it didn't work for me. I have edited my post so that I can get additional input from the community, but I didn't any after the first answer was posted.
My friend actually figured out a secondary way to fix the issue. He actually uses "NOMACHINE" on the local machine and open connection to the remote desktop. Then you can directly install PYCHARM in the remote machine and work in there. I hope this will help others.
The solution is in his blog post. (Thanks to Shaobo Guan)
Another solution would be to use VNC instead of NoMachine
I am building a (manually)bootable .vmdk file, but would like to further automate the process by using the .vmdk to create all necessary VM files and make it boot.
How can I do this?
Thank you for reading.
Solved. I was hoping for an elegant way to do this through an API, but the solution
was as simple as creating a VM manually, then swapping out all file references in the generated files, and completely replacing the .vmdk.