How to stream output in realtime in Twisted[autobahn] websocket server? - python-2.7

Question

I want to execute a C program using subprocess.Popen() and stream it's output in realtime and send it to the client. However, the output is buffered and is sent together at the end of execution (Blocking nature). How can I recieve the output in realtime and then send it immediatly in Twisted Autobahn.
def onConnect(self, request):
try:
self.cont_name = ''.join(random.choice(string.lowercase) for i in range(5))
self.file_name = self.cont_name
print("Connecting...")
except Exception:
print("Failed"+str(Exception))
def onOpen(self):
try:
print("open")
except Exception:
print("Couldn't create container")
def onMessage(self, payload,isBinary=False):
cmd = "docker exec "+self.cont_name+" /tmp/./"+self.file_name
a = subprocess.Popen([cmd], shell=True, stdout=subprocess.PIPE, bufsize=1)
for line in iter(a.stdout.readline, b''):
line = line.encode('utf8')
self.sendMessage(line)
def onClose(self, wasClean, code, reason):
try:
print("Closed container...")
except Exception:
print(str(Exception))
When the docker command is executed using subprocess,the entire output of the c code is returned at once rather than as it happens. For ex:
#include <stdio.h>
#include <unistd.h>
int main(){
int i=0;
for(i=0;i<5;i++){
fflush(stdout);
printf("Rounded\n");
sleep(3);
}
}
After running this in the container,the program should return 'Rounded' after 3 secs to client. However, it ends up sending all the 'Rounded' at the end of execution.

Answer 1

The misbehavior comes from the loop in this method:
def onMessage(self, payload,isBinary=False):
cmd = "docker exec "+self.cont_name+" /tmp/./"+self.file_name
a = subprocess.Popen([cmd], shell=True, stdout=subprocess.PIPE, bufsize=1)
for line in iter(a.stdout.readline, b''):
line = line.encode('utf8')
self.sendMessage(line)
Twisted is a cooperative multitasking system. By default, everything runs in a single thread ("the reactor thread"). That means all code has to periodically (and usually quickly) give up control so other code (application code or Twisted-implementation code) gets a chance to run. The loop in this function reads from the child process and tries to send the data using an Autobahn API - over and over again, never giving up control.
Blocking reads from the Popen object may also cause problems. You won't know how long the read will block and so you won't know how long you'll prevent other code from running in the reactor thread. You could either move your Popen reads to a new thread where they won't block the reactor thread:
def onMessage(self, payload,isBinary=False):
cmd = "docker exec "+self.cont_name+" /tmp/./"+self.file_name
popen_in_thread(
lambda line: reactor.callFromThread(
lambda: self.sendMessage(line.encode("utf-8"))
),
[cmd], shell=True, stdout=subprocess.PIPE, bufsize=1
)
def popen_in_thread(callback, *args, **kwargs):
def threaded():
a = subprocess.Popen(*args, **kwargs)
for line in iter(a.stdout.readline, b''):
callback(line)
reactor.callInThread(threaded)
Or, better, use Twisted's own process support:
def onMessage(self, payload,isBinary=False):
class ProcessLinesToMessages(ProcessProtocol):
def outReceived(self, output):
buf = self.buf + output
lines = buf.splitlines()
self.buf = lines.pop()
for line in lines:
self.sendMessage(line.encode("utf-8"))
while True:
line, self.buf = self.buf.splitline
reactor.spawnProcess(
ProcessLinesToMessages(),
"docker",
[
"docker",
"exec",
self.cont_name,
"/tmp/./ + self.file_name,
],
)
(neither version tested, hopefully the idea is clear though)