I have a google cloud Deep Learning Virtual Machine Image for PyTorch that uses an SSH connection to connect to the Jupyter Notebook on it. How can I change what I am currently doing so that the Jupyter Notebook remains alive even when I close my laptop/temporarily disconnect from internet?
Currently after turning my VM and opening a tmux window I start up the Jupyter Notebook and its SSH connection with this command:
gcloud compute ssh <my-server-name> -- -L 8080:localhost:8080
This code is taken from the official docs for the deep learning images here: https://cloud.google.com/deep-learning-vm/docs/jupyter
I can then connect at localhost:8080 and do what I need to. However, if I start training a model for a long time and need to close my laptop, when I re-open it my ssh connection breaks, the Jupyter Notebook is turned off, and my model that is training is interrupted.
How can I keep this Juptyer Notebook live and be able to reconnect to it later?
NB. I used to use the Google Cloud browser SSH option and once in the server start a tmux window and the jupyter notebook within it. This worked great and meant the notebook was always alive. However, with the Google Cloud images that have CUDA and Jupyter preinstalled, this doesn't work and the only way I have been able to connect is through the above command.
I have faced this problem before on GCP too and found a simple way to resolve this. Once you have ssh'd into the compute engine, run the linux screen command and you will find yourself in a virtual terminal (you may open many terminals in parallel) and it is here you will want to run your long running job.
Once you have started the job, detach from the screen using the keys the keys Ctrl+a and then d. Once detached, you can exit out of the VM, reconnect to the VM and run screen -r and you will find that your job is still running.
Of course, you can do a lot of cool stuff with the screen command and would encourage you to read some of the tutorials found here.
NOTE: Please ensure that your Compute Engine instance is not a Pre-emptible machine!
Let me know if this helps!
I thinks it's better install Jupyter as server . so your job can keep running even when you disconnect.
There are something you might also want to know.
This is not the multi-user server you are looking for. This document describes how you can run a public server with a single user. This should only be done by someone who wants remote access to their personal machine. Even so, doing this requires a thorough understanding of the set-ups limitations and security implications. If you allow multiple users to access a notebook server as it is described in this document, their commands may collide, clobber and overwrite each other.
If you want a multi-user server, the official solution is JupyterHub. To use JupyterHub, you need a Unix server (typically Linux) running somewhere that is accessible to your users on a network. This may run over the public internet, but doing so introduces additional security concerns.
Related
I'm trying to run my application on GCE VM. It uses nodeJs as frontend and a Java backend. I use this server to communicate with my local computer using MQTT. This is working but after some time (one hour and a half), the server seems to go to sleep (or the ports close ?).
Both MQTT and ssh terminal interface connections are lost.
When I connect back, the application is not running anymore, it seems like the VM restarted.
Do you have any idea on how to keep my server alive ? I can give further details.
Answering my own question as John Hanley explained the solution in comments:
"By what method are you running your frontend and backend code. VMs do not go to sleep. If you are starting your software via an SSH session, that will be an issue. Your code needs to run as a service. Edit your question with more details."
I was indeed running my application via the ssh terminal which caused the problem. The solution for me was to remotely access the VM via vncserver and to launch the application using the VM's terminal.
I am trying to train a deep learning model on GCP VM which is supposed to run 10-12 hours. But every time it runs for few epoch (few hours), it gives ssh timeout error.
I have set the following parameters on the config file but still the same error.
Can this be a code issue or something to do with VM config parameter.
The first step is to understand the problem, then select the solution.
When you create an SSH connection to a host (your VM), this opens a TCP connection on port 22. The SSH server then runs a shell program in this SSH session. When the SSH session closes or is interrupted, the shell program is killed, which kills most processes that the shell started.
Home Internet providers often reset long-running connections. This is to prevent you from running servers or large downloads over a home Internet connection. There are other reasons that the Internet provider might reset connections that are normal. You need to understand that this will happen and select options to manage this.
There are several programs that are available. Look into tmux and screen.
I use both programs and prefer tmux. Both programs are free and very popular. There are additional methods and programs from running your program in the background & to using nohup.
From Wikipedia:
tmux is a terminal multiplexer for Unix-like operating systems. It allows multiple terminal sessions to be accessed simultaneously in a single window. It is useful for running more than one command-line program at the same time. It can also be used to detach processes from their controlling terminals, allowing SSH sessions to remain active without being visible.
GNU Screen is a terminal multiplexer, a software application that can be used to multiplex several virtual consoles, allowing a user to access multiple separate login sessions inside a single terminal window, or detach and reattach sessions from a terminal. It is useful for dealing with multiple programs from a command line interface, and for separating programs from the session of the Unix shell that started the program, particularly so a remote process continues running even when the user is disconnected.
I'm new at GCP and I'm trying to keep my process running on Jupyter Notebook after shutting down my local PC. Does anyone know how can I do it? Nowaday I open a terminal on my VM run jupter notebook and then after start the process on jupyter I'd like to turn my machine off.
I keep following the process on my cellphone and shutdown on there. Does anyone know how to turn this off automatically when it stops?
Sorry to make two questions at once, but I think that one is related with another. If it does not I can edit and make another one.
This is a technical limitation of Jupyter Notebooks unfortunately. The browser window contains the code which updates the notebook itself, so if you close the browser window then there is not process running to update the notebook.
However, there is one workaround which you may find useful.
There is a library called Fairing that you can use with GCP's new AI Platform Notebooks which allows you to pack up your notebook and run it remotely, and that library will save the results of that execution in a GCP Storage bucket. No active internet connection required (once you kick of the notebook run).
You can learn how to use it by creating a new GCP AI Platform Notebook and looking at the tutorials folder inside it. You can also find additional tutorials for Fairing here
Typically to keep your remote sessions up in the event of network connectivity loss (which also covers shutting down the local computer) you'd use a terminal multiplexer application. From Known issues:
Intermittent disconnects: At this time, we do not offer a specific SLA for connection lifetimes. Use terminal multiplexers like tmux
or screen if you plan to keep the terminal window open for an
extended period of time.
But these multiplexers are terminal/text-mode apps, so you'd have to launch the notebook with the --no-browser and then connect your local browser to its port.
You can find a recipe based on tmux and a local browser connection to the notebook using an SSH tunnel at Using Jupyter notebooks securely on remote linux machines.
As for shutting down the session - you'd just have to instruct the multiplexer application to end the session (or terminate the multiplexer app itself) - which you could do automatically via a wrapper script first invoking your process and immediately after the process ends invoking the commands to shutdown the session.
The title explains it all. I'm currently working within AWS and am setting up anaconda for my projects. I'm trying to work on https://XYZ.compute.amazonaws.com:8888 since that is the only port that will load in chrome.
Every time I exit my jupyter window and close my Notebook session, It shows that this port is still in use.
Any help on how to vacate these ports would be much appreciated.
Did you use 'Ctrl + c' on the terminal to stop the Jupyter server? If not, every time you try running Jupyter again it will automatically add 1 to the previously used port number.
Another way around is to modify the AWS EC2 instance access rules and allow other ports like 9999
You have used screen.
follow below steps to close the running sessions.
Reattach to the screen session (screen -r)
Hit Ctrl +c to terminate the Jupyter notebook session
If nothing works power off and on your server to free the ports
Normally I've developed locally (on my own machine) and pushed to wherever things needed to go via mapped drives, ftp, github, etc. I have done a bit of work with vagrant/virtualbox (but again, locally) with a shared/mirrored folder.
I am now in a situation where everyone here has access to their own dev box (a vm on the network). I see some working in Vim directly via SSH, I believe, but I'm not there yet. So I'm left with the question: What's the best way for (more of a front end guy) to approach this?
I have heard of doing an SSH-mount from my workstation... if that's a viable thing. I'm curious what everyone's take on this kind of environment is and (perhaps) any best practices. Tips, links, and reading is highly welcome and appreciated, too... any pointing in a good direction would be wonderful.
Thank you.
The best answer will come from what virtual resources do you want to capitalize on for the virtual networked VMs. If you just want the storage space, then share the VM's drives, and mount them locally, treat them as local, end of story. If you want to run all the processing on the remote machine, and connect from a thin client, you have a couple of options, but they all take the same form. Connect to the machine, edit the files on the remote machine. Depending on your OS, you will have different options available.
If the remote machine doesn't have an graphical client installed you are stuck with either, mounting the remote share locally (you can use whatever editor you want) or ssh to the remote machine and using a commandline editor (vim, nano, emacs).
If there is a graphical client installed you have more options:
Remote in the server using any visual viewer (mstsc for windows, vnc is an option), and then use any remotely installed editor of your choice.
Remote in using ssh -X, and then run the remotely installed editor. Assuming you have an X-Server locally (if you are running linux you already do), the GUI part of the application will be run on the client side of the ssh tunnel, and the process will be run on the server. This is probably the best option.
So:
Make sure the remote server has a desktop client software (gtk, kde, gnome, almost any windows os, etc...)
install GUI editor of your choice on that server
ssh -X to that server
install sublime text, geany, or your choice of editor
run subl, geany, or other to start the application.
SSH mounting would indeed allow you to use all of the files on the VM as if they were stored in your local machine, letting you edit and update files without having to manually copy them every time you perform changes. You will run into a speed bump though, since files changed will have to be synchronized/copied to your remote machine every time and that takes a couple of seconds. Check this post by DigitalOcean, they explain how to get the SSH mount working.
A better option you have (IMHO) is to use an IDE in your local machine that allows you to push changes to a server after saving or by manually doing so. This would allow you to develop faster by using your local resources (local web server) since no files would have to be copied over the network to the remote VM; and would also allow you to test on that remote VM when needed by uploading the files when you are ready to test on that environment.
PS: Exporting visual apps or environments form the remote machine to your local one can be slow (depending on your network and the VM host load running your machine). If you still like that approach, you could also install something to access that VM over something more standard and lightweight like RDP for GNU/Linux (xrdp).