I'm working on creating a tool that allows users to run a jupyter-notebook w/ pyspark on an AWS server and forward the port to their localhost to connect to the notebook.
I've been using subprocess.Popen to ssh into the remote server and kick off the pyspark shell/notebook, but I'm unable to avoid having it print everything to the terminal. I WANT to perform an action per line to retrieve the port number.
For example, running this (following the most popular answer here: Read streaming input from subprocess.communicate())
command = "jupyter-notebook"
con = subprocess.Popen(['ssh', node, command], stdout=subprocess.PIPE, bufsize=1)
with con.stdout:
for line in iter(con.stdout.readline, b''):
print(line),
con.wait()
this ignores the context manager, and the con portion starts printing off stdout so that this is immediately printed to terminal
[I 16:13:20.783 NotebookApp] [nb_conda_kernels] enabled, 0 kernels found
[I 16:13:21.031 NotebookApp] JupyterLab extension loaded from /home/*****/miniconda3/envs/aws/lib/python3.7/site-packages/jupyterlab
[I 16:13:21.031 NotebookApp] JupyterLab application directory is /data/data0/home/*****/miniconda3/envs/aws/share/jupyter/lab
[I 16:13:21.035 NotebookApp] [nb_conda] enabled
...
...
...
I can get the context manager to function when I call a random script like the below instead of "jupyter-notebook" (where command="bash random_script.sh")
# random_script.sh
for i in $(seq 1 100)
do
echo "some output: $i"
sleep 2
done
This acts as expected, and I can actually perform an action per line within the with statement. Is there something fundamentally different about the jupyter version that prevents this from acting similarly?
The issue turned out to have everything to do with the fact that the console output produced by jupyter was actually going to STDERR instead of stdout. I'm not sure why. But regardless, this change totally fixed the issue:
con = subprocess.Popen(['ssh', node, command],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, # <-- redirect stderr to stdout
bufsize=1)
Related
In preemptive VM instances in Google Cloud Platform, a forced shut down can be called at any time. They allow to run a shutdown script to avoid file loss. But how do I use the script to cause a specific interrupt in my jupyter notebook?
I have come up with a solution.
from os import getpid, kill
from time import sleep
import signal
import ipykernel
import psutil
def get_active_kernels():
active_kernels = []
pids = psutil.pids()
my_pid = getpid()
for pid in pids:
if pid == my_pid:
continue
try:
p = psutil.Process(pid)
cmd = p.cmdline()
for arg in cmd:
if arg.count('ipykernel'):
active_kernels.append(pid)
except psutil.AccessDenied:
continue
return active_kernels
if __name__ == '__main__':
kernels = get_active_kernels()
for kernel in kernels:
kill(kernel, signal.SIGINT)
One can use this code as a shut-down script. It invokes a keyboard interrupt to all existing jupyter kernels. So, a simple try-except block that excepts KeyboardInterrupt can be used inside the jupyter notebook.
try:
... #regular code
except KeyboardInterrupt:
... #code to save the progress
Jupyter notebooks work with an instance VM on its backend and you can access them through ssh protocol like the other instances from the compute engine. This means that any script which works in a computer engine instance must work with a jupyter notebook.
In your description I understand you are referring to this shutdown script. This scripts saves a checkpoint while your instance is being shutdown, so it doesn't trigger the shutdown command itself.
There are many ways to shutdown an instance, either from inside the instance (script) as from outside (cloud shell, console UI ...).
Could you explain which is your specific purpose so I can help you further?
I'm trying to run a command when I start my gcloud vm server. I added the following command in the console custom metadata
key:
startup-script
value:
#! /bin/bash
parallel 'screen -d -m python3 Documents/myscript.py' ::: arg1 arg2
But when I start the vm instance, it doesn't run because I don't see any screens when I type screen -list. However, when I connect with the gcp tool over SSH using the following command to connect
gcloud beta compute ssh --zone "us-west1-c" "instance-1" --project "unique-kakaw-123456"
and then run
parallel 'screen -d -m python3 Documents/myscript.py' ::: arg1 arg2
consecutively, it runs properly.
How do I either, get my script to run on start up on gcp, or failing which, use the gcloud command to run the command in the same line so then when I hit enter, it will connect to the server and run the command right after. Because if I put both the above commands in the same line and add a ; it won't work obviously.
For some reason, I can add the script to rc.d and update on my ubuntu pc at home and it works fine. I don't know why it doesn't work on gcp.
The startup script should be working. Here's what could be happening:
1. Python script is not being executed because it cannot be found
Compute Engine documentation says:
The instance always executes startup scripts as root after the network is available.
So, if the Python script is in your home directory, provide the full path (replace [USER] with the actual user):
#! /bin/bash
parallel 'screen -d -m python3 /home/[USER]/Documents/myscript.py' ::: arg1 arg2
2. Python script runs and then exits, so screen terminates the window
Screen's User Manual says:
When a program terminates, screen (per default) kills the window that contained it. If this window was in the foreground, the display switches to the previously displayed window; if none are left, screen exits.
Thus if your Python script exits prematurely, add this to your /etc/screenrc:
zombie qr
Here is what this parameter does:
When a string of two keys is specified to the zombie command, ‘dead’ windows will remain in the list.
For the record, I replicated your startup script configuration in my GCP instance (providing the full path) and I can confirm it does work: there were two screens running with my Python script, each with its own argument.
I am connecting to SSH via terminal (on Mac) and run a Paramiko Python script and for some reason, the two sessions seem to behave differently. The PATH environment variable is different in these cases.
This is the code I run:
import paramiko
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('host', username='myuser',password='mypass')
stdin, stdout, stderr =ssh.exec_command('echo $PATH')
print (stdout.readlines())
Any idea why the environment variables are different?
And how can I fix it?
The SSHClient.exec_command by default does not allocate a pseudo terminal for the session. As a consequence a different set of startup scripts is (might be) sourced (particularly for non-interactive sessions, .bash_profile is not sourced). And/or different branches in the scripts are taken, based on an absence/presence of TERM environment variable.
To emulate the default Paramiko behavior with the ssh, use the -T switch:
ssh -T myuser#host
See the ssh man:
-T Disable pseudo-tty allocation.
Contrary, to emulate the default ssh behavior with Paramiko, set the get_pty parameter of the exec_command to True:
def exec_command(self, command, bufsize=-1, timeout=None, get_pty=False):
Though rather than working around the issue by allocating the pseudo terminal in Paramiko, you should better fix your startup scripts to set the same PATH for all sessions.
For that see Some Unix commands fail with "<command> not found", when executed using Python Paramiko exec_command.
Working with the Channel object instead of the SSHClient object solved my problem.
chan=ssh.invoke_shell()
chan.send('echo $PATH\n')
print (chan.recv(1024))
For more details, see the documentation
I'm trying to run a batch script remotely
python C:\FtpServer.py
when i start it manually it works fine, but when i use the remote script the python process wont start or terminates directly.
my python code is
import wmi
import win32con
connection = wmi.WMI("10.60.2.244", user="XY", password="XY")
startup = connection.Win32_ProcessStartup.new(ShowWindow=win32con.SW_SHOWNORMAL)
process_id, return_value = connection.Win32_Process.Create(CommandLine="C:\\startFtpServer.bat", ProcessStartupInformation=startup)
I get a pid and return value is 0
When i tasklist in cmd on the remote machine the process is not listed. Just starting python.exe instead of that batch file with that script works fine though.
I have a html file with one button. When the button is clicked, a javascript function "run" is called:
function run() {
window.open("http://localhost/cgi-bin/run.py", "_self");
}
run.py is simply trying to run a helloworld.exe program, that outputs in a terminal the string "helloworld", but nothing happens, and browser keeps "waiting for localhost" indefinitely.
#!python
import sys, string, os, cgitb
cgitb.enable()
os.system("helloworld.exe")
I have tried helloworld.exe alone and it works, I have run run.py on the terminal, and it worked, and also I have tested on the browser the test site http://localhost/cgi-bin/helloworld.py, and it worked fine (helloworld.py is another script just to see if my apache is configured OK).
I am using wamp.
What I am trying to do is a bigger program that allows a client connect to a server, and "interact" with a program on the server side. The program is already done in c++, and won't be translated into php or javascript.
EDIT: I have been trying with the functions: subprocess.Popen, subprocess.call and os.system. I have also tested the code to run .exe files created by me living at apache/cgi-bin folder or executables like wordpad, living at c:\windows. And it always succeeds when the python script runs from the terminal, and it never works when trying from the browser. Is it possible that it is because of the server I am using? I use the apache from wamp, and have added the sentence "AddHandler cgi-script .exe" to the httpd.conf file.
I am sure it doesn't work locally. os.system returns you the exit code of the command, not its output.
You will need to use subprocess.Popen and pipeline the output to read the output.
import subprocess
p = subprocess.Popen("whatever.exe",stdout=subprocess.PIPE)
print p.stdout.read()
p.wait()
Also, the output of your CGI script is not valid HTTP protocol, and depending on your server that could be causing problems (some servers sanitize the output of the scripts, some others expect them to be written properly).
To me this code example works (on GNU/Linux using weborf as server, but it should be the same). You can try setting the content type and sending the \r\n\r\n final sequence. Servers are not expected to send that because the CGI script might want to add some more HTTP headers.
#!/usr/bin/env python
import cgitb
import subprocess
import sys
cgitb.enable()
sys.stdout.write("Content-type: text/plain\r\n\r\n")
p = subprocess.Popen("/usr/games/fortune",stdout=subprocess.PIPE)
data = p.stdout.read()
p.wait()
print data
In general my experience has been that if something works locally and doesn't work when you install all the same code on a different server, you are looking at a permissions problem. Check the permissions that visitors on your site have and make sure that the user coming through your web server has the right permissions to run helloworld.exe and run.py.