I'm new at GCP and I'm trying to keep my process running on Jupyter Notebook after shutting down my local PC. Does anyone know how can I do it? Nowaday I open a terminal on my VM run jupter notebook and then after start the process on jupyter I'd like to turn my machine off.
I keep following the process on my cellphone and shutdown on there. Does anyone know how to turn this off automatically when it stops?
Sorry to make two questions at once, but I think that one is related with another. If it does not I can edit and make another one.
This is a technical limitation of Jupyter Notebooks unfortunately. The browser window contains the code which updates the notebook itself, so if you close the browser window then there is not process running to update the notebook.
However, there is one workaround which you may find useful.
There is a library called Fairing that you can use with GCP's new AI Platform Notebooks which allows you to pack up your notebook and run it remotely, and that library will save the results of that execution in a GCP Storage bucket. No active internet connection required (once you kick of the notebook run).
You can learn how to use it by creating a new GCP AI Platform Notebook and looking at the tutorials folder inside it. You can also find additional tutorials for Fairing here
Typically to keep your remote sessions up in the event of network connectivity loss (which also covers shutting down the local computer) you'd use a terminal multiplexer application. From Known issues:
Intermittent disconnects: At this time, we do not offer a specific SLA for connection lifetimes. Use terminal multiplexers like tmux
or screen if you plan to keep the terminal window open for an
extended period of time.
But these multiplexers are terminal/text-mode apps, so you'd have to launch the notebook with the --no-browser and then connect your local browser to its port.
You can find a recipe based on tmux and a local browser connection to the notebook using an SSH tunnel at Using Jupyter notebooks securely on remote linux machines.
As for shutting down the session - you'd just have to instruct the multiplexer application to end the session (or terminate the multiplexer app itself) - which you could do automatically via a wrapper script first invoking your process and immediately after the process ends invoking the commands to shutdown the session.
Related
I'm trying to run my application on GCE VM. It uses nodeJs as frontend and a Java backend. I use this server to communicate with my local computer using MQTT. This is working but after some time (one hour and a half), the server seems to go to sleep (or the ports close ?).
Both MQTT and ssh terminal interface connections are lost.
When I connect back, the application is not running anymore, it seems like the VM restarted.
Do you have any idea on how to keep my server alive ? I can give further details.
Answering my own question as John Hanley explained the solution in comments:
"By what method are you running your frontend and backend code. VMs do not go to sleep. If you are starting your software via an SSH session, that will be an issue. Your code needs to run as a service. Edit your question with more details."
I was indeed running my application via the ssh terminal which caused the problem. The solution for me was to remotely access the VM via vncserver and to launch the application using the VM's terminal.
I am performing some UI Automation on GCP using a Windows Server.
The process is as follows:
=> Machine Switches on at a defined time
=> RDP Connection to Machine
=> UI Interaction Script Runs on Startup
=> Process Ends
=> Machine Switches off at a defined time
All the components have been fulfilled except for automating the RDP connection in some way or other. I referred to this link but didn't find much insights or documentations.
Does anyone know a way to Automate an RDP connection right after instance turns on in GCP?
There is a windows application called IAP Desktop, using that you can manage multiple remote Desktop connection to Windows VM. While connecting to the VM you can save the credentials which will allow you to access the Windows VM using RDP just after boot on.
Also to automate the Windows password generation here is the documentation related to 1, inside of that document there are both options available automate or manually.
How are you deploying your startup script?
During the boot sequence, a script will either run before, after or during the boot process. By declaring Windows-specific metadata keys, you can run startup scripts after the instance turns on.
If that doesn't work, there is a paid Cloud Automation service that sounds like it will meet your requirements.
Tried using startup-scripts but no luck IAP Desktop didn't work due to scheduling as well. Finally Managed to solve it via using Windows 10 Auto login settings. This skips login screen and the best part was that out of all the users, it allows you to login via user of your choice. After I Login to the system, I added a startup a bat file by running shell:startup and it worked great.
I am trying to train a deep learning model on GCP VM which is supposed to run 10-12 hours. But every time it runs for few epoch (few hours), it gives ssh timeout error.
I have set the following parameters on the config file but still the same error.
Can this be a code issue or something to do with VM config parameter.
The first step is to understand the problem, then select the solution.
When you create an SSH connection to a host (your VM), this opens a TCP connection on port 22. The SSH server then runs a shell program in this SSH session. When the SSH session closes or is interrupted, the shell program is killed, which kills most processes that the shell started.
Home Internet providers often reset long-running connections. This is to prevent you from running servers or large downloads over a home Internet connection. There are other reasons that the Internet provider might reset connections that are normal. You need to understand that this will happen and select options to manage this.
There are several programs that are available. Look into tmux and screen.
I use both programs and prefer tmux. Both programs are free and very popular. There are additional methods and programs from running your program in the background & to using nohup.
From Wikipedia:
tmux is a terminal multiplexer for Unix-like operating systems. It allows multiple terminal sessions to be accessed simultaneously in a single window. It is useful for running more than one command-line program at the same time. It can also be used to detach processes from their controlling terminals, allowing SSH sessions to remain active without being visible.
GNU Screen is a terminal multiplexer, a software application that can be used to multiplex several virtual consoles, allowing a user to access multiple separate login sessions inside a single terminal window, or detach and reattach sessions from a terminal. It is useful for dealing with multiple programs from a command line interface, and for separating programs from the session of the Unix shell that started the program, particularly so a remote process continues running even when the user is disconnected.
I have a google cloud Deep Learning Virtual Machine Image for PyTorch that uses an SSH connection to connect to the Jupyter Notebook on it. How can I change what I am currently doing so that the Jupyter Notebook remains alive even when I close my laptop/temporarily disconnect from internet?
Currently after turning my VM and opening a tmux window I start up the Jupyter Notebook and its SSH connection with this command:
gcloud compute ssh <my-server-name> -- -L 8080:localhost:8080
This code is taken from the official docs for the deep learning images here: https://cloud.google.com/deep-learning-vm/docs/jupyter
I can then connect at localhost:8080 and do what I need to. However, if I start training a model for a long time and need to close my laptop, when I re-open it my ssh connection breaks, the Jupyter Notebook is turned off, and my model that is training is interrupted.
How can I keep this Juptyer Notebook live and be able to reconnect to it later?
NB. I used to use the Google Cloud browser SSH option and once in the server start a tmux window and the jupyter notebook within it. This worked great and meant the notebook was always alive. However, with the Google Cloud images that have CUDA and Jupyter preinstalled, this doesn't work and the only way I have been able to connect is through the above command.
I have faced this problem before on GCP too and found a simple way to resolve this. Once you have ssh'd into the compute engine, run the linux screen command and you will find yourself in a virtual terminal (you may open many terminals in parallel) and it is here you will want to run your long running job.
Once you have started the job, detach from the screen using the keys the keys Ctrl+a and then d. Once detached, you can exit out of the VM, reconnect to the VM and run screen -r and you will find that your job is still running.
Of course, you can do a lot of cool stuff with the screen command and would encourage you to read some of the tutorials found here.
NOTE: Please ensure that your Compute Engine instance is not a Pre-emptible machine!
Let me know if this helps!
I thinks it's better install Jupyter as server . so your job can keep running even when you disconnect.
There are something you might also want to know.
This is not the multi-user server you are looking for. This document describes how you can run a public server with a single user. This should only be done by someone who wants remote access to their personal machine. Even so, doing this requires a thorough understanding of the set-ups limitations and security implications. If you allow multiple users to access a notebook server as it is described in this document, their commands may collide, clobber and overwrite each other.
If you want a multi-user server, the official solution is JupyterHub. To use JupyterHub, you need a Unix server (typically Linux) running somewhere that is accessible to your users on a network. This may run over the public internet, but doing so introduces additional security concerns.
The title explains it all. I'm currently working within AWS and am setting up anaconda for my projects. I'm trying to work on https://XYZ.compute.amazonaws.com:8888 since that is the only port that will load in chrome.
Every time I exit my jupyter window and close my Notebook session, It shows that this port is still in use.
Any help on how to vacate these ports would be much appreciated.
Did you use 'Ctrl + c' on the terminal to stop the Jupyter server? If not, every time you try running Jupyter again it will automatically add 1 to the previously used port number.
Another way around is to modify the AWS EC2 instance access rules and allow other ports like 9999
You have used screen.
follow below steps to close the running sessions.
Reattach to the screen session (screen -r)
Hit Ctrl +c to terminate the Jupyter notebook session
If nothing works power off and on your server to free the ports