Python Multiprocessing - Passing values between child process - python-2.7

I have a single method named grep_phalanx_log whose functionality is to SSH to a machine and grep for some values.
My main method will call this method with different host names/credentials, log file name, grep pattern.
So, I need to grep for a SAME pattern in two different servers in PARALLEL. If the match is found in one server, I want the other server to stop grep-ing. If the pattern is not found in both the servers for a specified time, my method grep_phalanx_log will return a negative value. Based on the negative value, I have to proceed with some other requirement.
class eventFlowTestNfx(object)
def grep_phalanx_log(self, host_name, username, password, grep_cmd, timeout=10, time_to_monitor=20):
ssh_client = paramiko.SSHClient()
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
Log.info("Grep command to be executed: %r" % grep_cmd)
try:
ssh_client.connect(host_name, username=username, password=password, timeout=timeout)
end_time = time.time() + time_to_monitor
while time.time() < end_time:
ssh_stdin, ssh_stdout, ssh_stderr = ssh_client.exec_command(grep_cmd)
output = ssh_stdout.read()
if not output:
time.sleep(1)
else:
Log.info("NFX: Match message from %r is %r" % (host_name, output))
return output
if not output:
Log.error("FAILED: Message not processed.")
Log.error("Host Name: %r and grep command: %r" % (host_name, grep_cmd))
raise Exception("NFX agent could not process message")
except:
Log.error("End to End flow is broken, check the logs!")
return -1
def main(self):
for cr_dict in correlation_list:
cr_process = multiprocessing.Process(target=self.grep_phalanx_log(), args=(cr_dict["host"], cr_dict["username"], cr_dict["password"], cr_received_cmd_skeleton,))
cr_process.start()
So, I have my code stared 2 process, I am not sure how they will talk to each other and terminate the other.

You could replace time.sleep(1) with:
if not output.strip(): # blank output
is_found = found.wait(1) # sleep >= 1 second unless found
if is_found:
break # stop grepping
else: # found something
found.set()
...
return output
where found = multiprocessing.Event(): creat it in the parent process and pass to each child.

Related

Check health of Docker using Python Script - Python 2.7 [duplicate]

Here's the Python code to run an arbitrary command returning its stdout data, or raise an exception on non-zero exit codes:
proc = subprocess.Popen(
cmd,
stderr=subprocess.STDOUT, # Merge stdout and stderr
stdout=subprocess.PIPE,
shell=True)
communicate is used to wait for the process to exit:
stdoutdata, stderrdata = proc.communicate()
The subprocess module does not support timeout--ability to kill a process running for more than X number of seconds--therefore, communicate may take forever to run.
What is the simplest way to implement timeouts in a Python program meant to run on Windows and Linux?
In Python 3.3+:
from subprocess import STDOUT, check_output
output = check_output(cmd, stderr=STDOUT, timeout=seconds)
output is a byte string that contains command's merged stdout, stderr data.
check_output raises CalledProcessError on non-zero exit status as specified in the question's text unlike proc.communicate() method.
I've removed shell=True because it is often used unnecessarily. You can always add it back if cmd indeed requires it. If you add shell=True i.e., if the child process spawns its own descendants; check_output() can return much later than the timeout indicates, see Subprocess timeout failure.
The timeout feature is available on Python 2.x via the subprocess32 backport of the 3.2+ subprocess module.
I don't know much about the low level details; but, given that in
python 2.6 the API offers the ability to wait for threads and
terminate processes, what about running the process in a separate
thread?
import subprocess, threading
class Command(object):
def __init__(self, cmd):
self.cmd = cmd
self.process = None
def run(self, timeout):
def target():
print 'Thread started'
self.process = subprocess.Popen(self.cmd, shell=True)
self.process.communicate()
print 'Thread finished'
thread = threading.Thread(target=target)
thread.start()
thread.join(timeout)
if thread.is_alive():
print 'Terminating process'
self.process.terminate()
thread.join()
print self.process.returncode
command = Command("echo 'Process started'; sleep 2; echo 'Process finished'")
command.run(timeout=3)
command.run(timeout=1)
The output of this snippet in my machine is:
Thread started
Process started
Process finished
Thread finished
0
Thread started
Process started
Terminating process
Thread finished
-15
where it can be seen that, in the first execution, the process
finished correctly (return code 0), while the in the second one the
process was terminated (return code -15).
I haven't tested in windows; but, aside from updating the example
command, I think it should work since I haven't found in the
documentation anything that says that thread.join or process.terminate
is not supported.
jcollado's answer can be simplified using the threading.Timer class:
import shlex
from subprocess import Popen, PIPE
from threading import Timer
def run(cmd, timeout_sec):
proc = Popen(shlex.split(cmd), stdout=PIPE, stderr=PIPE)
timer = Timer(timeout_sec, proc.kill)
try:
timer.start()
stdout, stderr = proc.communicate()
finally:
timer.cancel()
# Examples: both take 1 second
run("sleep 1", 5) # process ends normally at 1 second
run("sleep 5", 1) # timeout happens at 1 second
If you're on Unix,
import signal
...
class Alarm(Exception):
pass
def alarm_handler(signum, frame):
raise Alarm
signal.signal(signal.SIGALRM, alarm_handler)
signal.alarm(5*60) # 5 minutes
try:
stdoutdata, stderrdata = proc.communicate()
signal.alarm(0) # reset the alarm
except Alarm:
print "Oops, taking too long!"
# whatever else
Here is Alex Martelli's solution as a module with proper process killing. The other approaches do not work because they do not use proc.communicate(). So if you have a process that produces lots of output, it will fill its output buffer and then block until you read something from it.
from os import kill
from signal import alarm, signal, SIGALRM, SIGKILL
from subprocess import PIPE, Popen
def run(args, cwd = None, shell = False, kill_tree = True, timeout = -1, env = None):
'''
Run a command with a timeout after which it will be forcibly
killed.
'''
class Alarm(Exception):
pass
def alarm_handler(signum, frame):
raise Alarm
p = Popen(args, shell = shell, cwd = cwd, stdout = PIPE, stderr = PIPE, env = env)
if timeout != -1:
signal(SIGALRM, alarm_handler)
alarm(timeout)
try:
stdout, stderr = p.communicate()
if timeout != -1:
alarm(0)
except Alarm:
pids = [p.pid]
if kill_tree:
pids.extend(get_process_children(p.pid))
for pid in pids:
# process might have died before getting to this line
# so wrap to avoid OSError: no such process
try:
kill(pid, SIGKILL)
except OSError:
pass
return -9, '', ''
return p.returncode, stdout, stderr
def get_process_children(pid):
p = Popen('ps --no-headers -o pid --ppid %d' % pid, shell = True,
stdout = PIPE, stderr = PIPE)
stdout, stderr = p.communicate()
return [int(p) for p in stdout.split()]
if __name__ == '__main__':
print run('find /', shell = True, timeout = 3)
print run('find', shell = True)
Since Python 3.5, there's a new subprocess.run universal command (that is meant to replace check_call, check_output ...) and which has the timeout= parameter as well.
subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None, shell=False, cwd=None, timeout=None, check=False, encoding=None, errors=None)
Run the command described by args. Wait for command to complete, then return a CompletedProcess instance.
It raises a subprocess.TimeoutExpired exception when the timeout expires.
timeout is now supported by call() and communicate() in the subprocess module (as of Python3.3):
import subprocess
subprocess.call("command", timeout=20, shell=True)
This will call the command and raise the exception
subprocess.TimeoutExpired
if the command doesn't finish after 20 seconds.
You can then handle the exception to continue your code, something like:
try:
subprocess.call("command", timeout=20, shell=True)
except subprocess.TimeoutExpired:
# insert code here
Hope this helps.
surprised nobody mentioned using timeout
timeout 5 ping -c 3 somehost
This won't for work for every use case obviously, but if your dealing with a simple script, this is hard to beat.
Also available as gtimeout in coreutils via homebrew for mac users.
I've modified sussudio answer. Now function returns: (returncode, stdout, stderr, timeout) - stdout and stderr is decoded to utf-8 string
def kill_proc(proc, timeout):
timeout["value"] = True
proc.kill()
def run(cmd, timeout_sec):
proc = subprocess.Popen(shlex.split(cmd), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
timeout = {"value": False}
timer = Timer(timeout_sec, kill_proc, [proc, timeout])
timer.start()
stdout, stderr = proc.communicate()
timer.cancel()
return proc.returncode, stdout.decode("utf-8"), stderr.decode("utf-8"), timeout["value"]
Another option is to write to a temporary file to prevent the stdout blocking instead of needing to poll with communicate(). This worked for me where the other answers did not; for example on windows.
outFile = tempfile.SpooledTemporaryFile()
errFile = tempfile.SpooledTemporaryFile()
proc = subprocess.Popen(args, stderr=errFile, stdout=outFile, universal_newlines=False)
wait_remaining_sec = timeout
while proc.poll() is None and wait_remaining_sec > 0:
time.sleep(1)
wait_remaining_sec -= 1
if wait_remaining_sec <= 0:
killProc(proc.pid)
raise ProcessIncompleteError(proc, timeout)
# read temp streams from start
outFile.seek(0);
errFile.seek(0);
out = outFile.read()
err = errFile.read()
outFile.close()
errFile.close()
Prepending the Linux command timeout isn't a bad workaround and it worked for me.
cmd = "timeout 20 "+ cmd
subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(output, err) = p.communicate()
I added the solution with threading from jcollado to my Python module easyprocess.
Install:
pip install easyprocess
Example:
from easyprocess import Proc
# shell is not supported!
stdout=Proc('ping localhost').call(timeout=1.5).stdout
print stdout
Here is my solution, I was using Thread and Event:
import subprocess
from threading import Thread, Event
def kill_on_timeout(done, timeout, proc):
if not done.wait(timeout):
proc.kill()
def exec_command(command, timeout):
done = Event()
proc = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
watcher = Thread(target=kill_on_timeout, args=(done, timeout, proc))
watcher.daemon = True
watcher.start()
data, stderr = proc.communicate()
done.set()
return data, stderr, proc.returncode
In action:
In [2]: exec_command(['sleep', '10'], 5)
Out[2]: ('', '', -9)
In [3]: exec_command(['sleep', '10'], 11)
Out[3]: ('', '', 0)
The solution I use is to prefix the shell command with timelimit. If the comand takes too long, timelimit will stop it and Popen will have a returncode set by timelimit. If it is > 128, it means timelimit killed the process.
See also python subprocess with timeout and large output (>64K)
if you are using python 2, give it a try
import subprocess32
try:
output = subprocess32.check_output(command, shell=True, timeout=3)
except subprocess32.TimeoutExpired as e:
print e
I've implemented what I could gather from a few of these. This works in Windows, and since this is a community wiki, I figure I would share my code as well:
class Command(threading.Thread):
def __init__(self, cmd, outFile, errFile, timeout):
threading.Thread.__init__(self)
self.cmd = cmd
self.process = None
self.outFile = outFile
self.errFile = errFile
self.timed_out = False
self.timeout = timeout
def run(self):
self.process = subprocess.Popen(self.cmd, stdout = self.outFile, \
stderr = self.errFile)
while (self.process.poll() is None and self.timeout > 0):
time.sleep(1)
self.timeout -= 1
if not self.timeout > 0:
self.process.terminate()
self.timed_out = True
else:
self.timed_out = False
Then from another class or file:
outFile = tempfile.SpooledTemporaryFile()
errFile = tempfile.SpooledTemporaryFile()
executor = command.Command(c, outFile, errFile, timeout)
executor.daemon = True
executor.start()
executor.join()
if executor.timed_out:
out = 'timed out'
else:
outFile.seek(0)
errFile.seek(0)
out = outFile.read()
err = errFile.read()
outFile.close()
errFile.close()
Once you understand full process running machinery in *unix, you will easily find simplier solution:
Consider this simple example how to make timeoutable communicate() meth using select.select() (available alsmost everythere on *nix nowadays). This also can be written with epoll/poll/kqueue, but select.select() variant could be a good example for you. And major limitations of select.select() (speed and 1024 max fds) are not applicapable for your task.
This works under *nix, does not create threads, does not uses signals, can be lauched from any thread (not only main), and fast enought to read 250mb/s of data from stdout on my machine (i5 2.3ghz).
There is a problem in join'ing stdout/stderr at the end of communicate. If you have huge program output this could lead to big memory usage. But you can call communicate() several times with smaller timeouts.
class Popen(subprocess.Popen):
def communicate(self, input=None, timeout=None):
if timeout is None:
return subprocess.Popen.communicate(self, input)
if self.stdin:
# Flush stdio buffer, this might block if user
# has been writing to .stdin in an uncontrolled
# fashion.
self.stdin.flush()
if not input:
self.stdin.close()
read_set, write_set = [], []
stdout = stderr = None
if self.stdin and input:
write_set.append(self.stdin)
if self.stdout:
read_set.append(self.stdout)
stdout = []
if self.stderr:
read_set.append(self.stderr)
stderr = []
input_offset = 0
deadline = time.time() + timeout
while read_set or write_set:
try:
rlist, wlist, xlist = select.select(read_set, write_set, [], max(0, deadline - time.time()))
except select.error as ex:
if ex.args[0] == errno.EINTR:
continue
raise
if not (rlist or wlist):
# Just break if timeout
# Since we do not close stdout/stderr/stdin, we can call
# communicate() several times reading data by smaller pieces.
break
if self.stdin in wlist:
chunk = input[input_offset:input_offset + subprocess._PIPE_BUF]
try:
bytes_written = os.write(self.stdin.fileno(), chunk)
except OSError as ex:
if ex.errno == errno.EPIPE:
self.stdin.close()
write_set.remove(self.stdin)
else:
raise
else:
input_offset += bytes_written
if input_offset >= len(input):
self.stdin.close()
write_set.remove(self.stdin)
# Read stdout / stderr by 1024 bytes
for fn, tgt in (
(self.stdout, stdout),
(self.stderr, stderr),
):
if fn in rlist:
data = os.read(fn.fileno(), 1024)
if data == '':
fn.close()
read_set.remove(fn)
tgt.append(data)
if stdout is not None:
stdout = ''.join(stdout)
if stderr is not None:
stderr = ''.join(stderr)
return (stdout, stderr)
You can do this using select
import subprocess
from datetime import datetime
from select import select
def call_with_timeout(cmd, timeout):
started = datetime.now()
sp = subprocess.Popen(cmd, stdout=subprocess.PIPE)
while True:
p = select([sp.stdout], [], [], timeout)
if p[0]:
p[0][0].read()
ret = sp.poll()
if ret is not None:
return ret
if (datetime.now()-started).total_seconds() > timeout:
sp.kill()
return None
python 2.7
import time
import subprocess
def run_command(cmd, timeout=0):
start_time = time.time()
df = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while timeout and df.poll() == None:
if time.time()-start_time >= timeout:
df.kill()
return -1, ""
output = '\n'.join(df.communicate()).strip()
return df.returncode, output
Example of captured output after timeout tested in Python 3.7.8:
try:
return subprocess.run(command, shell=True, capture_output=True, timeout=20, cwd=cwd, universal_newlines=True)
except subprocess.TimeoutExpired as e:
print(e.output.decode(encoding="utf-8", errors="ignore"))
assert False;
The exception subprocess.TimeoutExpired has the output and other members:
cmd - Command that was used to spawn the child process.
timeout - Timeout in seconds.
output - Output of the child process if it was captured by run() or
check_output(). Otherwise, None.
stdout - Alias for output, for symmetry with stderr.
stderr - Stderr output of the child process if it was captured by
run(). Otherwise, None.
More info: https://docs.python.org/3/library/subprocess.html#subprocess.TimeoutExpired
I've used killableprocess successfully on Windows, Linux and Mac. If you are using Cygwin Python, you'll need OSAF's version of killableprocess because otherwise native Windows processes won't get killed.
Although I haven't looked at it extensively, this decorator I found at ActiveState seems to be quite useful for this sort of thing. Along with subprocess.Popen(..., close_fds=True), at least I'm ready for shell-scripting in Python.
This solution kills the process tree in case of shell=True, passes parameters to the process (or not), has a timeout and gets the stdout, stderr and process output of the call back (it uses psutil for the kill_proc_tree). This was based on several solutions posted in SO including jcollado's. Posting in response to comments by Anson and jradice in jcollado's answer. Tested in Windows Srvr 2012 and Ubuntu 14.04. Please note that for Ubuntu you need to change the parent.children(...) call to parent.get_children(...).
def kill_proc_tree(pid, including_parent=True):
parent = psutil.Process(pid)
children = parent.children(recursive=True)
for child in children:
child.kill()
psutil.wait_procs(children, timeout=5)
if including_parent:
parent.kill()
parent.wait(5)
def run_with_timeout(cmd, current_dir, cmd_parms, timeout):
def target():
process = subprocess.Popen(cmd, cwd=current_dir, shell=True, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
# wait for the process to terminate
if (cmd_parms == ""):
out, err = process.communicate()
else:
out, err = process.communicate(cmd_parms)
errcode = process.returncode
thread = Thread(target=target)
thread.start()
thread.join(timeout)
if thread.is_alive():
me = os.getpid()
kill_proc_tree(me, including_parent=False)
thread.join()
There's an idea to subclass the Popen class and extend it with some simple method decorators. Let's call it ExpirablePopen.
from logging import error
from subprocess import Popen
from threading import Event
from threading import Thread
class ExpirablePopen(Popen):
def __init__(self, *args, **kwargs):
self.timeout = kwargs.pop('timeout', 0)
self.timer = None
self.done = Event()
Popen.__init__(self, *args, **kwargs)
def __tkill(self):
timeout = self.timeout
if not self.done.wait(timeout):
error('Terminating process {} by timeout of {} secs.'.format(self.pid, timeout))
self.kill()
def expirable(func):
def wrapper(self, *args, **kwargs):
# zero timeout means call of parent method
if self.timeout == 0:
return func(self, *args, **kwargs)
# if timer is None, need to start it
if self.timer is None:
self.timer = thr = Thread(target=self.__tkill)
thr.daemon = True
thr.start()
result = func(self, *args, **kwargs)
self.done.set()
return result
return wrapper
wait = expirable(Popen.wait)
communicate = expirable(Popen.communicate)
if __name__ == '__main__':
from subprocess import PIPE
print ExpirablePopen('ssh -T git#bitbucket.org', stdout=PIPE, timeout=1).communicate()
I had the problem that I wanted to terminate a multithreading subprocess if it took longer than a given timeout length. I wanted to set a timeout in Popen(), but it did not work. Then, I realized that Popen().wait() is equal to call() and so I had the idea to set a timeout within the .wait(timeout=xxx) method, which finally worked. Thus, I solved it this way:
import os
import sys
import signal
import subprocess
from multiprocessing import Pool
cores_for_parallelization = 4
timeout_time = 15 # seconds
def main():
jobs = [...YOUR_JOB_LIST...]
with Pool(cores_for_parallelization) as p:
p.map(run_parallel_jobs, jobs)
def run_parallel_jobs(args):
# Define the arguments including the paths
initial_terminal_command = 'C:\\Python34\\python.exe' # Python executable
function_to_start = 'C:\\temp\\xyz.py' # The multithreading script
final_list = [initial_terminal_command, function_to_start]
final_list.extend(args)
# Start the subprocess and determine the process PID
subp = subprocess.Popen(final_list) # starts the process
pid = subp.pid
# Wait until the return code returns from the function by considering the timeout.
# If not, terminate the process.
try:
returncode = subp.wait(timeout=timeout_time) # should be zero if accomplished
except subprocess.TimeoutExpired:
# Distinguish between Linux and Windows and terminate the process if
# the timeout has been expired
if sys.platform == 'linux2':
os.kill(pid, signal.SIGTERM)
elif sys.platform == 'win32':
subp.terminate()
if __name__ == '__main__':
main()
Late answer for Linux only, but in case someone wants to use subprocess.getstatusoutput(), where the timeout argument isn't available, you can use the built-in Linux timeout on the beginning of the command, i.e.:
import subprocess
timeout = 25 # seconds
cmd = f"timeout --preserve-status --foreground {timeout} ping duckgo.com"
exit_c, out = subprocess.getstatusoutput(cmd)
if (exit_c == 0):
print("success")
else:
print("Error: ", out)
timeout Arguments:
--preserve-status : Preserving the Exit Status
--foreground : Running in Foreground
25 : timeout value in seconds
Unfortunately, I'm bound by very strict policies on the disclosure of source code by my employer, so I can't provide actual code. But for my taste the best solution is to create a subclass overriding Popen.wait() to poll instead of wait indefinitely, and Popen.__init__ to accept a timeout parameter. Once you do that, all the other Popen methods (which call wait) will work as expected, including communicate.
https://pypi.python.org/pypi/python-subprocess2 provides extensions to the subprocess module which allow you to wait up to a certain period of time, otherwise terminate.
So, to wait up to 10 seconds for the process to terminate, otherwise kill:
pipe = subprocess.Popen('...')
timeout = 10
results = pipe.waitOrTerminate(timeout)
This is compatible with both windows and unix. "results" is a dictionary, it contains "returnCode" which is the return of the app (or None if it had to be killed), as well as "actionTaken". which will be "SUBPROCESS2_PROCESS_COMPLETED" if the process completed normally, or a mask of "SUBPROCESS2_PROCESS_TERMINATED" and SUBPROCESS2_PROCESS_KILLED depending on action taken (see documentation for full details)
for python 2.6+, use gevent
from gevent.subprocess import Popen, PIPE, STDOUT
def call_sys(cmd, timeout):
p= Popen(cmd, shell=True, stdout=PIPE)
output, _ = p.communicate(timeout=timeout)
assert p.returncode == 0, p. returncode
return output
call_sys('./t.sh', 2)
# t.sh example
sleep 5
echo done
exit 1
Sometimes you need to process (ffmpeg) without using communicate() and in this case you need asynchronous timeout, a practical way to do this using ttldict
pip install ttldict
from ttldict import TTLOrderedDict
sp_timeout = TTLOrderedDict(default_ttl=10)
def kill_on_timeout(done, proc):
while True:
now = time.time()
if sp_timeout.get('exp_time') == None:
proc.kill()
break
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, text=True, stderr=subprocess.STDOUT)
sp_timeout['exp_time'] = time.time()
done = Event()
watcher = Thread(target=kill_on_timeout, args=(done, process))
watcher.daemon = True
watcher.start()
done.set()
for line in process.stdout:
.......

How do I retrieve last console output of a jenkins job using python-jenkins?

I'm unable to extract the jenkins console output of a given job using python-jenkins. I must to extract the full console output of a given build #.
Here is how my jenkins url looks like:
http://jenkins.abc.xyz.io:8080/job/myapp-build/46/consoleFull
and It has valid output.
As per https://python-jenkins.readthedocs.io/en/latest/api.html#jenkins.Jenkins.get_build_console_output
get_build_console_output(name, number)
Get build console text.
Parameters:
name – Job name, str
number – Build number, int
Returns:
Build console output, str
I tried
import jenkins
server = jenkins.Jenkins('http://jenkins.abc.xyz.io:8080', username='foo', password='foo!')
jenkins_url = "http://jenkins.abc.xyz.io:8080"
user = server.get_whoami()
version = server.get_version()
print('Hello %s from Jenkins %s' % (user['fullName'], version))
#
name='/myapp-build'
number=46
print server.get_build_console_output(name, number)
But it aborts saying
Traceback (most recent call last):
File "test_jenkins.py", line 20, in <module>
print server.get_build_console_output(name, number)
File "/home/myapp/jenkins_py/env/jenkins/local/lib/python2.7/site-packages/jenkins/__init__.py", line 1630, in get_build_console_output
% (name, number))
jenkins.JenkinsException: job[/myapp-build] number[46] does not exist
Expected result would be the console output.
What am I missing?
Please help me extract the console output from jenkins using python.
are there any alternate ways or libraries to accomplish it?
Ok. I feel really stupid at 2:18am.
The script works.
Just needed to remove the preceding '/' in front of the job name.
Now I'm able to retrieve the full output.
server = jenkins.Jenkins('http://jenkins.abc.xyz.io:8080', username='foo', password='foo!')
jenkins_url = "http://jenkins.abc.xyz.io:8080"
user = server.get_whoami()
version = server.get_version()
print('Hello %s from Jenkins %s' % (user['fullName'], version))
#
name='myapp-build' #without the /
number=46
print server.get_build_console_output(name, number)
This code will continuously compare old job == current job number. Once the current job will finish. The whole output will be in console_output.
def jenkins_wait():
global BuildID, BuildStatus
server = jenkins.Jenkins(Jenkins_url, username=jenkins_user, password=jenkins_pwd)
time.sleep(5)
while True:
temptime = datetime.now()
Job_time = temptime.strftime("%H:%M:%S")
print('Jenkins build ID :',(server.get_job_info(jOBnAmE)['lastBuild']['number'])," Time:", Job_time)
time.sleep(5)
processLoading() #Display processing bar.
if server.get_job_info(jOBnAmE)['lastCompletedBuild']['number'] == server.get_job_info(jOBnAmE)['lastBuild']['number']:
print("Last ID %s, Current ID %s" % (server.get_job_info(jOBnAmE)['lastCompletedBuild']['number'], server.get_job_info(jOBnAmE)['lastBuild']['number']))
BuildID = server.get_job_info(jOBnAmE)['lastBuild']['number']
print (BuildID)
break
time.sleep(25)
print('Job execution has been completed.')
console_output = server.get_build_console_output(jOBnAmE, server.get_job_info(jOBnAmE)['lastBuild']['number'])
console_output = re.split('\r\n',console_output)

Audio Timeout error in Speech to text API of Google Cloud

I aim to make my jarvis, which listens all the time and activates when I say hello. I learned that Google cloud Speech to Text API doesn't listen for more than 60 seconds, but then I found this not-so-famous link, where this listens for infinite duration. The author of github script says that, he has played a trick that script refreshes after 60 seconds, so that program doesn't crash.
https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/speech/cloud-client/transcribe_streaming_indefinite.py
Following is the modified version, since I wanted it to answer of my questions, followed by "hello", and not answer me all the time. Now if I ask my Jarvis, a question, which while answering takes more than 60 seconds and it doesn't get the time to refresh, the program crashes down :(
#!/usr/bin/env python
# Copyright 2018 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Google Cloud Speech API sample application using the streaming API.
NOTE: This module requires the additional dependency `pyaudio`. To install
using pip:
pip install pyaudio
Example usage:
python transcribe_streaming_indefinite.py
"""
# [START speech_transcribe_infinite_streaming]
from __future__ import division
import time
import re
import sys
import os
from google.cloud import speech
from pygame.mixer import *
from googletrans import Translator
# running=True
translator = Translator()
init()
import pyaudio
from six.moves import queue
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "C:\\Users\\mnauf\\Desktop\\rehandevice\\key.json"
from commands2 import commander
cmd=commander()
# Audio recording parameters
STREAMING_LIMIT = 55000
SAMPLE_RATE = 16000
CHUNK_SIZE = int(SAMPLE_RATE / 10) # 100ms
def get_current_time():
return int(round(time.time() * 1000))
def duration_to_secs(duration):
return duration.seconds + (duration.nanos / float(1e9))
class ResumableMicrophoneStream:
"""Opens a recording stream as a generator yielding the audio chunks."""
def __init__(self, rate, chunk_size):
self._rate = rate
self._chunk_size = chunk_size
self._num_channels = 1
self._max_replay_secs = 5
# Create a thread-safe buffer of audio data
self._buff = queue.Queue()
self.closed = True
self.start_time = get_current_time()
# 2 bytes in 16 bit samples
self._bytes_per_sample = 2 * self._num_channels
self._bytes_per_second = self._rate * self._bytes_per_sample
self._bytes_per_chunk = (self._chunk_size * self._bytes_per_sample)
self._chunks_per_second = (
self._bytes_per_second // self._bytes_per_chunk)
def __enter__(self):
self.closed = False
self._audio_interface = pyaudio.PyAudio()
self._audio_stream = self._audio_interface.open(
format=pyaudio.paInt16,
channels=self._num_channels,
rate=self._rate,
input=True,
frames_per_buffer=self._chunk_size,
# Run the audio stream asynchronously to fill the buffer object.
# This is necessary so that the input device's buffer doesn't
# overflow while the calling thread makes network requests, etc.
stream_callback=self._fill_buffer,
)
return self
def __exit__(self, type, value, traceback):
self._audio_stream.stop_stream()
self._audio_stream.close()
self.closed = True
# Signal the generator to terminate so that the client's
# streaming_recognize method will not block the process termination.
self._buff.put(None)
self._audio_interface.terminate()
def _fill_buffer(self, in_data, *args, **kwargs):
"""Continuously collect data from the audio stream, into the buffer."""
self._buff.put(in_data)
return None, pyaudio.paContinue
def generator(self):
while not self.closed:
if get_current_time() - self.start_time > STREAMING_LIMIT:
self.start_time = get_current_time()
break
# Use a blocking get() to ensure there's at least one chunk of
# data, and stop iteration if the chunk is None, indicating the
# end of the audio stream.
chunk = self._buff.get()
if chunk is None:
return
data = [chunk]
# Now consume whatever other data's still buffered.
while True:
try:
chunk = self._buff.get(block=False)
if chunk is None:
return
data.append(chunk)
except queue.Empty:
break
yield b''.join(data)
def search(responses, stream, code):
responses = (r for r in responses if (
r.results and r.results[0].alternatives))
num_chars_printed = 0
for response in responses:
if not response.results:
continue
# The `results` list is consecutive. For streaming, we only care about
# the first result being considered, since once it's `is_final`, it
# moves on to considering the next utterance.
result = response.results[0]
if not result.alternatives:
continue
# Display the transcription of the top alternative.
top_alternative = result.alternatives[0]
transcript = top_alternative.transcript
# music.load("/home/pi/Desktop/rehandevice/end.mp3")
# music.play()
# Display interim results, but with a carriage return at the end of the
# line, so subsequent lines will overwrite them.
# If the previous result was longer than this one, we need to print
# some extra spaces to overwrite the previous result
overwrite_chars = ' ' * (num_chars_printed - len(transcript))
if not result.is_final:
sys.stdout.write(transcript + overwrite_chars + '\r')
sys.stdout.flush()
num_chars_printed = len(transcript)
else:
#print(transcript + overwrite_chars)
# Exit recognition if any of the transcribed phrases could be
# one of our keywords.
if code=='ur-PK':
transcript=translator.translate(transcript).text
print("Your command: ", transcript + overwrite_chars)
if "hindi assistant" in (transcript+overwrite_chars).lower():
cmd.respond("Alright. Talk to me in urdu",code=code)
main('ur-PK')
elif "english assistant" in (transcript+overwrite_chars).lower():
cmd.respond("Alright. Talk to me in English",code=code)
main('en-US')
cmd.discover(text=transcript + overwrite_chars,code=code)
for i in range(10):
print("Hello world")
break
num_chars_printed = 0
def listen_print_loop(responses, stream, code):
"""Iterates through server responses and prints them.
The responses passed is a generator that will block until a response
is provided by the server.
Each response may contain multiple results, and each result may contain
multiple alternatives; for details, see https://cloud.google.com/speech-to-text/docs/reference/rpc/google.cloud.speech.v1#streamingrecognizeresponse. Here we
print only the transcription for the top alternative of the top result.
In this case, responses are provided for interim results as well. If the
response is an interim one, print a line feed at the end of it, to allow
the next result to overwrite it, until the response is a final one. For the
final one, print a newline to preserve the finalized transcription.
"""
responses = (r for r in responses if (
r.results and r.results[0].alternatives))
music.load(r"C:\\Users\\mnauf\\Desktop\\rehandevice\\coins.mp3")
num_chars_printed = 0
for response in responses:
if not response.results:
continue
# The `results` list is consecutive. For streaming, we only care about
# the first result being considered, since once it's `is_final`, it
# moves on to considering the next utterance.
result = response.results[0]
if not result.alternatives:
continue
# Display the transcription of the top alternative.
top_alternative = result.alternatives[0]
transcript = top_alternative.transcript
# Display interim results, but with a carriage return at the end of the
# line, so subsequent lines will overwrite them.
#
# If the previous result was longer than this one, we need to print
# some extra spaces to overwrite the previous result
overwrite_chars = ' ' * (num_chars_printed - len(transcript))
if not result.is_final:
sys.stdout.write(transcript + overwrite_chars + '\r')
sys.stdout.flush()
num_chars_printed = len(transcript)
else:
print("Listen print loop", transcript + overwrite_chars)
# Exit recognition if any of the transcribed phrases could be
# one of our keywords.
if re.search(r'\b(hello)\b', transcript.lower(), re.I):
#print("Give me order")
music.play()
search(responses, stream,code)
break
elif re.search(r'\b(ہیلو)\b', transcript, re.I):
music.play()
search(responses, stream,code)
break
num_chars_printed = 0
def main(code):
cmd.respond("I am Rayhaan dot A Eye. How can I help you?",code=code)
client = speech.SpeechClient()
config = speech.types.RecognitionConfig(
encoding=speech.enums.RecognitionConfig.AudioEncoding.LINEAR16,
sample_rate_hertz=SAMPLE_RATE,
language_code='en-US',
max_alternatives=1,
enable_word_time_offsets=True)
streaming_config = speech.types.StreamingRecognitionConfig(
config=config,
interim_results=True)
mic_manager = ResumableMicrophoneStream(SAMPLE_RATE, CHUNK_SIZE)
print('Say "Quit" or "Exit" to terminate the program.')
with mic_manager as stream:
while not stream.closed:
audio_generator = stream.generator()
requests = (speech.types.StreamingRecognizeRequest(
audio_content=content)
for content in audio_generator)
responses = client.streaming_recognize(streaming_config,
requests)
# Now, put the transcription responses to use.
try:
listen_print_loop(responses, stream, code)
except:
listen
if __name__ == '__main__':
main('en-US')
# [END speech_transcribe_infinite_streaming]
You can call your functions after recognition in different thread. Example:
new_thread = Thread(target=music.play)
new_thread.daemon = True # Not always needed, read more about daemon property
new_thread.start()
Or if you want just to prevent exception - you can always use try/except. Example:
with mic_manager as stream:
while not stream.closed:
try:
audio_generator = stream.generator()
requests = (speech.types.StreamingRecognizeRequest(
audio_content=content)
for content in audio_generator)
responses = client.streaming_recognize(streaming_config,
requests)
# Now, put the transcription responses to use.
listen_print_loop(responses, stream, code)
except BaseException as e:
print("Exception occurred - {}".format(str(e)))

Multiprocessing and a list as output returns error: local variable '' referenced before assignment

I want to use multiprocessing for a project of mine. I have a code that already works fine an i want to adapt it. I realised that when there is a list in the workers method i get the following error; local variable 'RouteInfo' referenced before assignment. Any suggestions?
if __name__ == '__main__':
time_start = time.clock()
AREA_ID, PARENT_ID, routes_data, routingPlanID, StationName = getRouteDetails(StationName, PARENT_ID, AREA_ID)
copyDailyABPDetail(PARENT_ID, AREA_ID)
AREA_ID=AREA_ID.decode('utf-8')
StationsList = [unicode(item[0]) for item in routes_data]
func = partial(testMethod, AREA_ID, PARENT_ID, routingPlanID, StationNameDic)
# declare number of cores to use
cpuNum = multiprocessing.cpu_count()
# Create the pool object
pool = multiprocessing.Pool(processes=cpuNum)
# Fire off list to worker function.
results=pool.map(func,StationsList)
pool.close()
pool.join()
print results
time_elapsed = (time.clock() - time_start)
def testMethod(AREA_ID, PARENT_ID, routingPlanID, StationNameDic, StationName):
try:
print "Processing: ",StationName
RouteInfo = ["ALL", "ALL", "PRESERVE_BOTH", ["OnewaySelective","Footway","Steps","Path","Tolls"], "OR", "Α,Β,Γ", "NGIS", "RA"]
print RouteInfo
except Exception as e:
# If an error occurred, print line number and error message
import traceback, sys
tb = sys.exc_info()[2]
print "An error occured on line %i" % tb.tb_lineno
print str(e)
return StationName, RouteInfo
Solved there was an error in my code (variable StatioName had an utf-8 issue), so the variable RouteInfo was never been assigned.

stopping execution of code in clean way python

I have a GUI created using PyQt. In the GUI their is a button which when pressed send some data to client. Following is my code
class Main(QtGui.QTabWidget, Ui_TabWidget):
def __init__(self):
QtGui.QTabWidget.__init__(self)
self.setupUi(self)
self.pushButton_8.clicked.connect(self.updateActual)
def updateActual():
self.label_34.setText(self.comboBox_4.currentText())
HOST = '127.0.0.1' # The remote host
PORT = 8000 # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
s.connect((displayBoard[str(self.comboBox_4.currentText())], PORT))
except socket.error as e:
err1 = str(self.comboBox_4.currentText()) + " is OFF-LINE"
reply2 = QtGui.QMessageBox.critical(self, 'Error', err1, QtGui.QMessageBox.Ok)
if reply2 == QtGui.QMessageBox.Ok:
pass #stop execution at this point
fileName = str(self.comboBox_4.currentText()) + '.txt'
f = open(fileName)
readLines = f.readlines()
line1 = int(readLines[0])
f.close()
Currently if a user clicks 'ok' in QMessageBox the program will continue code execution in case their is socket exception. Thus my question is how can I stop the execution of code after 'except' in a clean way such that my UI doesn't crash and user can continue using it?
Yes, you can simply return from the if block:
if reply2 == QtGui.QMessageBox.Ok:
return
Alternatively, move your code for when it doesn't raise socket.error into an else block:
try: # this might fail
s.connect(...)
except socket.error as e: # what to do if it fails
err1 = ...
reply2 = QtGui.QMessageBox.critical(...)
else: # what to do if it doesn't
with open(fileName) as f:
line1 = int(f.readline().strip())
Note that:
You don't actually need to deal with the return from the message box, as it could only be OK and you have no else option;
you should generally use with for file handling, it will automatically close at the end of the block; and
you can simplify your file handling code by only reading the first line.