Related
I'm writing a custom shell where I try to add support for input, output redirections and pipes just like standard shell. I stuck at point where I cannot do input redirection, but output redirection is perfectly working. My implementation is something like this (only related part), you can assume that (string) input is non-empty
void execute() {
... // stuff before execution and initialization of variables
int *fds;
std::string content;
std::string input = readFromAFile(in_file); // for input redirection
for (int i = 0; i < commands.size(); i++) {
fds = subprocess(commands[i]);
dprintf(fds[1], "%s", input.data()); // write to write-end of pipe
close(fds[1]);
content += readFromFD(fds[0]); // read from read-end of pipe
close(fds[0]);
}
... // stuff after execution
}
int *subprocess(std::string &cmd) {
std::string s;
int *fds = new int[2];
pipe(fds);
pid_t pid = fork();
if (pid == -1) {
std::cerr << "Fork failed.";
}
if (pid == 0) {
dup2(fds[1], STDOUT_FILENO);
dup2(fds[0], STDIN_FILENO);
close(fds[1]);
close(fds[0]);
system(cmd.data());
exit(0); // child terminates
}
return fds;
}
My thought is subprocess returns a pipe (fd_in, fd_out) and parent can write to write-end and read-from read-end afterwards. However when I try an input redirection something like sort < in.txt, the program just hangs. I think there is a deadlock because one waiting other to write, and other one to read, however, after parent writes to write-end it closes, and then read from read-end. How should I consider this case ?
When I did a bit of searching, I saw this answer, which my original thinking was similar except that in the answer it mentions creating two pipes. I did not quite understand this part. Why do we need two separate pipes ?
I have the helper function below, used to execute a command and get the return value on posix systems. I used to use popen, but it is impossible to get the return code of an application with popen if it runs and exits before popen/pclose gets a chance to do its work.
The following helper function creates a process fork, uses execvp to run the desired external process, and then the parent uses waitpid to get the return code. I'm seeing odd cases where it's refusing to run.
When called with wait = true, waitpid should return the exit code of the application no matter what. However, I'm seeing stdout output that specifies the return code should be non-zero, yet the return code is zero. Testing the external process in a regular shell, then echoing $? returns non-zero, so it's not a problem w/ the external process not returning the right code. If it's of any help, the external process being run is mount(8) (yes, I know I can use mount(2) but that's besides the point).
I apologize in advance for a code dump. Most of it is debugging/logging:
inline int ForkAndRun(const std::string &command, const std::vector<std::string> &args, bool wait = false, std::string *output = NULL)
{
std::string debug;
std::vector<char*> argv;
for(size_t i = 0; i < args.size(); ++i)
{
argv.push_back(const_cast<char*>(args[i].c_str()));
debug += "\"";
debug += args[i];
debug += "\" ";
}
argv.push_back((char*)NULL);
neosmart::logger.Debug("Executing %s", debug.c_str());
int pipefd[2];
if (pipe(pipefd) != 0)
{
neosmart::logger.Error("Failed to create pipe descriptor when trying to launch %s", debug.c_str());
return EXIT_FAILURE;
}
pid_t pid = fork();
if (pid == 0)
{
close(pipefd[STDIN_FILENO]); //child isn't going to be reading
dup2(pipefd[STDOUT_FILENO], STDOUT_FILENO);
close(pipefd[STDOUT_FILENO]); //now that it's been dup2'd
dup2(pipefd[STDOUT_FILENO], STDERR_FILENO);
if (execvp(command.c_str(), &argv[0]) != 0)
{
exit(EXIT_FAILURE);
}
return 0;
}
else if (pid < 0)
{
neosmart::logger.Error("Failed to fork when trying to launch %s", debug.c_str());
return EXIT_FAILURE;
}
else
{
close(pipefd[STDOUT_FILENO]);
int exitCode = 0;
if (wait)
{
waitpid(pid, &exitCode, wait ? __WALL : (WNOHANG | WUNTRACED));
std::string result;
char buffer[128];
ssize_t bytesRead;
while ((bytesRead = read(pipefd[STDIN_FILENO], buffer, sizeof(buffer)-1)) != 0)
{
buffer[bytesRead] = '\0';
result += buffer;
}
if (wait)
{
if ((WIFEXITED(exitCode)) == 0)
{
neosmart::logger.Error("Failed to run command %s", debug.c_str());
neosmart::logger.Info("Output:\n%s", result.c_str());
}
else
{
neosmart::logger.Debug("Output:\n%s", result.c_str());
exitCode = WEXITSTATUS(exitCode);
if (exitCode != 0)
{
neosmart::logger.Info("Return code %d", (exitCode));
}
}
}
if (output)
{
result.swap(*output);
}
}
close(pipefd[STDIN_FILENO]);
return exitCode;
}
}
Note that the command is run OK with the correct parameters, the function proceeds without any problems, and WIFEXITED returns TRUE. However, WEXITSTATUS returns 0, when it should be returning something else.
Probably isn't your main issue, but I think I see a small problem. In your child process, you have...
dup2(pipefd[STDOUT_FILENO], STDOUT_FILENO);
close(pipefd[STDOUT_FILENO]); //now that it's been dup2'd
dup2(pipefd[STDOUT_FILENO], STDERR_FILENO); //but wait, this pipe is closed!
But I think what you want is:
dup2(pipefd[STDOUT_FILENO], STDOUT_FILENO);
dup2(pipefd[STDOUT_FILENO], STDERR_FILENO);
close(pipefd[STDOUT_FILENO]); //now that it's been dup2'd for both, can close
I don't have much experience with forks and pipes in Linux, but I did write a similar function pretty recently. You can take a look at the code to compare, if you'd like. I know that my function works.
execAndRedirect.cpp
I'm using the mongoose library, and grepping my code for SIGCHLD revealed that using mg_start from mongoose results in setting SIGCHLD to SIG_IGN.
From the waitpid man page, on Linux a SIGCHLD set to SIG_IGN will not create a zombie process, so waitpid will fail if the process has already successfully run and exited - but will run OK if it hasn't yet. This was the cause of the sporadic failure of my code.
Simply re-setting SIGCHLD after calling mg_start to a void function that does absolutely nothing was enough to keep the zombie records from being immediately erased.
Per #Geoff_Montee's advice, there was a bug in my redirect of STDERR, but this was not responsible for the problem as execvp does not store the return value in STDERR or even STDOUT, but rather in the kernel object associated with the parent process (the zombie record).
#jilles' warning about non-contiguity of vector in C++ does not apply for C++03 and up (only valid for C++98, though in practice, most C++98 compilers did use contiguous storage, anyway) and was not related to this issue. However, the advice on reading from the pipe before blocking and checking the output of waitpid is spot-on.
I've found that pclose does NOT block and wait for the process to end, contrary to the documentation (this is on CentOS 6). I've found that I need to call pclose and then call waitpid(pid,&status,0); to get the true return value.
I am trying to figure out a generalized way for Asynchronous Bidirectional IO Redirection of a child process. Basically, I would like to spawn an interactive child process that waits for input and any output should be read back. I tried to experiment with python.subprocess by spawning a new python process. A base simplistic example tried to achieve is as follows
process = subprocess.Popen(['/usr/bin/python'],shell=False,stdin=subprocess.PIPE, stdout=subprocess.PIPE)
while True:
output = process.stdout.readline()
print output
input = sys.stdin.readline()
process.stdin.write(input)
and executing the above code snippet simply hangs without any output. I tried running with /usr/bash and /usr/bin/irb but the result is all the same. My guess is, buffered IO is simply not gelling well with IO redirection.
So my question is, is it feasible to read the output of a child process without flushing the buffer or quitting the subprocess?
The following post mentions IPC sockets but for that I would have to change the child process which may not be feasible. Is there any other way to achieve it?
Note*** My ultimate goal is to create a server REPL process which can interact with a remote web client. Though the example given is of Python, my ultimate goal is to wrap all available REPL by a generalized wrapper.
With the help of some of the suggestion in the answers I came up with the following
#!/usr/bin/python
import subprocess, os, select
proc = subprocess.Popen(['/usr/bin/python'],shell=False,stdin=subprocess.PIPE, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
for i in xrange(0,5):
inputready, outputready, exceptready = select.select([proc.stdout, proc.stderr],[proc.stdout, proc.stderr],[proc.stdout, proc.stderr],0)
if not inputready: print "No Data",
print inputready, outputready, exceptready
for s in inputready: print s.fileno(),s.readline()
proc.terminate()
print "After Terminating"
for i in xrange(0,5):
inputready, outputready, exceptready = select.select([proc.stdout, proc.stderr],[proc.stdout, proc.stderr],[proc.stdout, proc.stderr],0)
if not inputready: print "No Data",
print inputready, outputready, exceptready
for s in inputready: print s.fileno(),s.readline()
now, though the programs is not in deadlock but unfortunately there is no output. Running the above code I get
No Data [] [] []
No Data [] [] []
No Data [] [] []
No Data [] [] []
No Data [] [] []
After Terminating
No Data [] [] []
No Data [] [] []
No Data [] [] []
No Data [] [] []
No Data [] [] []
Just FYI,
running python as
/usr/bin/python 2>&1|tee test.out
seems to be working just fine.
I also came up with a 'C' code. But the result is not different.
int kbhit() {
struct timeval tv;
fd_set fds;
tv.tv_sec = tv.tv_usec = 0;
FD_ZERO(&fds);
FD_SET(STDIN_FILENO, &fds);
select(STDIN_FILENO+1, &fds, NULL, NULL, &tv);
return FD_ISSET(STDIN_FILENO, &fds);
}
void receive(char *str) {
char ch;
fprintf(stderr,"IN1\n");
if(!kbhit()) return;
fprintf(stderr,"IN2\n");
fprintf(stderr,"%d\n",kbhit());
for(;kbhit() && (ch=fgetc(stdin))!=EOF;) {
fprintf(stderr,"%c,%d",ch,kbhit());
}
fprintf(stderr,"Done\n");
}
int main(){
pid_t pid;
int rv, pipeP2C[2],pipeC2P[2];
pipe(pipeP2C);
pipe(pipeC2P);
pid=fork();
if(pid){
dup2(pipeP2C[1],1); /* Replace stdout with out side of the pipe */
close(pipeP2C[0]); /* Close unused side of pipe (in side) */
dup2(pipeC2P[0],0); /* Replace stdin with in side of the pipe */
close(pipeC2P[1]); /* Close unused side of pipe (out side) */
setvbuf(stdout,(char*)NULL,_IONBF,0); /* Set non-buffered output on stdout */
sleep(2);
receive("quit()\n");
wait(&rv); /* Wait for child process to end */
fprintf(stderr,"Child exited with a %d value\n",rv);
}
else{
dup2(pipeP2C[0],0); /* Replace stdin with the in side of the pipe */
close(pipeP2C[1]); /* Close unused side of pipe (out side) */
dup2(pipeC2P[1],1); /* Replace stdout with the out side of the pipe */
close(pipeC2P[0]); /* Close unused side of pipe (out side) */
setvbuf(stdout,(char*)NULL,_IONBF,0); /* Set non-buffered output on stdout */
close(2), dup2(1,2); /*Redirect stderr to stdout */
if(execl("/usr/bin/python","/usr/bin/python",NULL) == -1){
fprintf(stderr,"execl Error!");
exit(1);
}
}
return 0;
}
In the Python code you posted, you're not using the right streams:
inputready, outputready, exceptready = select.select(
[proc.stdout, proc.stderr], # read list
[proc.stdout, proc.stderr], # write list
[proc.stdout, proc.stderr], # error list.
0) # time out.
I haven't tried fixing it, but I bet reading and writing to the same set of streams is incorrect.
There are multiple things going wrong in your sample. The first is that the python executable that you launch as as a child process produces no output. The second is that there is a race condition since you can invoke select() 5 times in a row before the child process produces output, in which case you will kill the process before reading anything.
I fixed the three problems mentioned above (write list, starting a process that produces output and race condition). Try out this sample and see if it works for you:
#!/usr/bin/python
import subprocess, os, select, time
path = "/usr/bin/python"
proc = subprocess.Popen([path, "foo.py"], shell=False,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
for i in xrange(0,5):
time.sleep(1)
inputready, outputready, exceptready = select.select(
[proc.stdout, proc.stderr], [proc.stdin,],
[proc.stdout, proc.stderr, proc.stdin], 0)
if not inputready:
print "No Data",
print inputready, outputready, exceptready
for s in inputready:
print s.fileno(),s.readline()
proc.terminate()
print "After Terminating"
for i in xrange(0,5):
inputready, outputready, exceptready = select.select(
[proc.stdout, proc.stderr], [proc.stdin,],
[proc.stdout, proc.stderr, proc.stdin], 0)
if not inputready:
print "No Data",
print inputready, outputready, exceptready
for s in inputready:
print s.fileno(),s.readline()
The foo.py file I used contained this:
#!/usr/bin/python
print "Hello, world!"
The following version (mostly removed redundant output to make results easier to read):
#!/usr/bin/python
import subprocess, os, select, time
path = "/usr/bin/python"
proc = subprocess.Popen([path, "foo.py"], shell=False,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
for i in xrange(0,5):
time.sleep(1)
inputready, outputready, exceptready = select.select(
[proc.stdout, proc.stderr], [proc.stdin,],
[proc.stdout, proc.stderr, proc.stdin], 0)
for s in inputready:
line = s.readline()
if line:
print s.fileno(), line
proc.terminate()
print "After Terminating"
for i in xrange(0,5):
time.sleep(1)
inputready, outputready, exceptready = select.select(
[proc.stdout, proc.stderr], [proc.stdin,],
[proc.stdout, proc.stderr, proc.stdin], 0)
for s in inputready:
line = s.readline()
if line:
print s.fileno(), line
Gives the following output:
5 Hello, world!
After Terminating
Note that for some reason, using the timeout parameter in select.select() did not produce the expected results on my system, and I resorted to using time.sleep() instead.
Just FYI, running python as
/usr/bin/python 2>&1|tee test.out
seems to be working just fine.
You cannot get this effect because this example still gives the python interpreter a controlling tty. Without the controlling tty, the python interpreter does not print the Python version and does not display the >>> prompt.
A close example would be something like the following. You can replace the /dev/null with a file containing commands to send to the interpreter.
/usr/bin/python </dev/null 2>&1|tee test.out
If you redirect anything other than the controlling tty (keyboard) as the standard input to the process, you will get no output from the python interpreter. This is why your code appears not to work.
There are different way to do this.
You can, for example:
use SysV message queues and poll with timeout on the queue for message to arrive
create a pipe() for the child and a pipe() for the father both using the O_NONBLOCK flag and then select() on the file descriptors for data to arrive (to can even handle timeouts if no data arrives)
use socket() AF_UNIX or AF_INET, set it non blocking and select() or epoll() for data to arrive
mmap() MAP_SHARED memory segments and signal the other process when data is arrived, pay attention to the shared segment with a locking mechanism.
I wrote a sample in C with double pipes:
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <unistd.h>
#include <sys/time.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <sys/stat.h>
#include <sys/select.h>
#include <fcntl.h>
#include <signal.h>
#define BUFLEN (6*1024)
#define EXECFILE "/usr/bin/python"
char *itoa(int n, char *s, int b) {
static char digits[] = "0123456789abcdefghijklmnopqrstuvwxyz";
int i=0, sign;
if ((sign = n) < 0)
n = -n;
do {
s[i++] = digits[n % b];
} while ((n /= b) > 0);
if (sign < 0)
s[i++] = '-';
s[i] = '\0';
return s;
}
/*
int set_nonblock(int sockfd) { // set socket to non blocking
int arg,i;
if ((arg=fcntl(sockfd, F_GETFL, NULL)) < 0) {
printf("error getting socket flag for fd %i: fcntl(..., F_GETFL): %i\n", sockfd, errno);
return -1;
}
// set O_NONBLOCK flag
arg |= O_NONBLOCK;
if ((i=fcntl(sockfd, F_SETFL, arg)) < 0) {
printf("error setting socket flag for fd %i: fcntl(..., F_SETFL): %i\n", sockfd, errno);
return -1;
}
return i;
}
int set_block(int sockfd) { // set socket to blocking
int arg,i;
if ((arg=fcntl(sockfd, F_GETFL, NULL)) < 0) {
printf("error getting socket flag for fd %i: fcntl(..., F_GETFL): %i\n", sockfd, errno);
return -1;
}
// clean O_NONBLOCK flag
arg &= (~O_NONBLOCK);
if ((i=fcntl(sockfd, F_SETFL, arg)) < 0) {
printf("error setting socket flag for fd %i: fcntl(..., F_SETFL): %i\n", sockfd, errno);
return -1;
}
return i;
}
*/
int main() {
FILE *input;
char slice[BUFLEN];
int status = 0;
pid_t pid;
int err;
int newfd;
// if you want you can pass arguments to the program to execute
// char *const arguments[] = {EXECFILE, "-v", NULL};
char *const arguments[] = {EXECFILE, NULL};
int father2child_pipefd[2];
int child2father_pipefd[2];
char *read_data = NULL;
FILE *retclam;
fd_set myset;
int x=1;
signal(SIGPIPE, SIG_IGN);
newfd = dup(0);
input = fdopen(newfd, "r");
pipe(father2child_pipefd); // Father speaking to child
pipe(child2father_pipefd); // Child speaking to father
pid = fork();
if (pid > 0) { // Father
close(father2child_pipefd[0]);
close(child2father_pipefd[1]);
// Write to the pipe reading from stdin
retclam = fdopen(child2father_pipefd[0], "r");
// set the two fd non blocking
//set_nonblock(0);
//set_nonblock(child2father_pipefd[0]);
//set_nonblock(fileno(retclam));
while(x==1) {
// clear the file descriptor set
FD_ZERO(&myset);
// add the stdin to the set
FD_SET(fileno(input), &myset);
// add the child pipe to the set
FD_SET(fileno(retclam), &myset);
// here we wait for data to arrive from stdin or from the child pipe. The last argument is a timeout, if you like
err = select(fileno(retclam)+1, &myset, NULL, NULL, NULL);
switch(err) {
case -1:
// Problem with select(). The errno variable knows why
//exit(1);
x=0;
break;
case 0:
// timeout on select(). Data did not arrived in time, only valid if the last attribute of select() was specified
break;
default:
// data is ready to be read
bzero(slice, BUFLEN);
if (FD_ISSET(fileno(retclam), &myset)) { // data ready on the child
//set_block(fileno(retclam));
read_data = fgets(slice, BUFLEN, retclam); // read a line from the child (max BUFLEN bytes)
//set_nonblock(fileno(retclam));
if (read_data == NULL) {
//exit(0);
x=0;
break;
}
// write data back to stdout
write (1, slice, strlen(slice));
if(feof(retclam)) {
//exit(0);
x=0;
break;
}
break;
}
bzero(slice, BUFLEN);
if (FD_ISSET(fileno(input), &myset)) { // data ready on stdin
//printf("father\n");
//set_block(fileno(input));
read_data = fgets(slice, BUFLEN, input); // read a line from stdin (max BUFLEN bytes)
//set_nonblock(fileno(input));
if (read_data == NULL) {
//exit (0);
close(father2child_pipefd[1]);
waitpid(pid, &status, 0);
//fclose(input);
break;
}
// write data to the child
write (father2child_pipefd[1], slice, strlen(slice));
/*
if(feof(input)) {
exit(0);
}*/
break;
}
}
}
close(father2child_pipefd[1]);
fclose(input);
fsync(1);
waitpid(pid, &status, 0);
// child process terminated
fclose (retclam);
// Parse output data from child
// write (1, "you can append somethind else on stdout if you like");
if (WEXITSTATUS(status) == 0) {
exit (0); // child process exited successfully
}
}
if (pid == 0) { // Child
close (0); // stdin is not needed
close (1); // stdout is not needed
// Close the write side of this pipe
close(father2child_pipefd[1]);
// Close the read side of this pipe
close(child2father_pipefd[0]);
// Let's read on stdin, but this stdin is associated to the read pipe
dup2(father2child_pipefd[0], 0);
// Let's speak on stdout, but this stdout is associated to the write pipe
dup2(child2father_pipefd[1], 1);
// if you like you can put something back to the father before execve
//write (child2father_pipefd[1], "something", 9);
//fsync(child2father_pipefd[1]);
err = execve(EXECFILE, arguments, NULL);
// we'll never be here again after execve succeeded!! So we get here only if the execve() failed
//fprintf(stderr, "Problem executing file %s: %i: %s\n", EXECFILE, err, strerror(errno));
exit (1);
}
if (pid < 0) { // Error
exit (1);
}
fclose(input);
return 0;
}
I use 2-way io in bash like this:
mkfifo hotleg
mkfifo coldleg
program <coldleg |tee hotleg &
while read LINE; do
case $LINE in
*)call_a_function $LINE;;
esac
done <hotleg |tee coldleg &
(note that you can just ">" instead of tee, but you may want to see the output at first)
Your guess that buffered I/O is to blame is most likely correct. The way you wrote your loop, the read will block until it fills the required buffer, and you won't be able to process any input until it returns. This can easily cause a deadlock.
Popen.communicate deals with this by making a thread to work with each pipe, and by making sure it has all the data to be written to stdin, so that the actual write cannot be delayed while the file object waits for a buffer to fill or for the file object to be flushed/closed. I think you could make a solution involving threads work if you needed to, but that's not really asynchronous and probably not the easiest solution.
You can get around python's buffering by not using the file objects provided by Popen to access the pipes, and instead grabbing their fd's using the fileno() method. You can then use the fd's with os.read, os.write, and select.select. The os.read and os.write functions will do no buffering, but they will block until at least one byte can be read/written. You need to make sure the pipe is readable/writeable before calling them. The simplest way to do this is to use select.select() to wait for all the pipes you want to read/write, and make a single read or write call to every pipe that's ready when select() returns. You should be able to find examples of select loops if you search (they'll probably be using sockets instead of pipes, but the principle is the same). (Also, never do a read or write without checking first that it won't block, or you can end up with cases where you cause a deadlock with the child process. You have to be ready to read data even when you haven't yet written everything you want.)
If you need to control a Python interpreter session, you're probably better off with
embedding Python into your program (plain evals if it's in Python itself), or
using RPC facilities like rpyc like PyScripter does.
Btw in the latter case, the server can be run anywhere and PyScripter already has a working server module (client module is in Pascal, will need to translate).
This question already has answers here:
Can popen() make bidirectional pipes like pipe() + fork()?
(6 answers)
Closed 3 years ago.
Is it possible to read and write to a file descriptor returned by popen. I have an interactive process I'd like to control through C. If this isn't possible with popen, is there any way around it?
As already answered, popen works in one direction. If you need to read and write, You can create a pipe with pipe(), span a new process by fork() and exec functions and then redirect its input and outputs with dup2(). Anyway I prefer exec over popen, as it gives you better control over the process (e.g. you know its pid)
EDITED:
As comments suggested, a pipe can be used in one direction only. Therefore you have to create separate pipes for reading and writing. Since the example posted before was wrong, I deleted it and created a new, correct one:
#include<unistd.h>
#include<sys/wait.h>
#include<sys/prctl.h>
#include<signal.h>
#include<stdlib.h>
#include<string.h>
#include<stdio.h>
int main(int argc, char** argv)
{
pid_t pid = 0;
int inpipefd[2];
int outpipefd[2];
char buf[256];
char msg[256];
int status;
pipe(inpipefd);
pipe(outpipefd);
pid = fork();
if (pid == 0)
{
// Child
dup2(outpipefd[0], STDIN_FILENO);
dup2(inpipefd[1], STDOUT_FILENO);
dup2(inpipefd[1], STDERR_FILENO);
//ask kernel to deliver SIGTERM in case the parent dies
prctl(PR_SET_PDEATHSIG, SIGTERM);
//replace tee with your process
execl("/usr/bin/tee", "tee", (char*) NULL);
// Nothing below this line should be executed by child process. If so,
// it means that the execl function wasn't successfull, so lets exit:
exit(1);
}
// The code below will be executed only by parent. You can write and read
// from the child using pipefd descriptors, and you can send signals to
// the process using its pid by kill() function. If the child process will
// exit unexpectedly, the parent process will obtain SIGCHLD signal that
// can be handled (e.g. you can respawn the child process).
//close unused pipe ends
close(outpipefd[0]);
close(inpipefd[1]);
// Now, you can write to outpipefd[1] and read from inpipefd[0] :
while(1)
{
printf("Enter message to send\n");
scanf("%s", msg);
if(strcmp(msg, "exit") == 0) break;
write(outpipefd[1], msg, strlen(msg));
read(inpipefd[0], buf, 256);
printf("Received answer: %s\n", buf);
}
kill(pid, SIGKILL); //send SIGKILL signal to the child process
waitpid(pid, &status, 0);
}
The reason popen() and friends don't offer bidirectional communication is that it would be deadlock-prone, due to buffering in the subprocess. All the makeshift pipework and socketpair() solutions discussed in the answers suffer from the same problem.
Under UNIX, most commands cannot be trusted to read one line and immediately process it and print it, except if their standard output is a tty. The reason is that stdio buffers output in userspace by default, and defers the write() system call until either the buffer is full or the stdio stream is closed (typically because the program or script is about to exit after having seen EOF on input). If you write to such a program's stdin through a pipe, and now wait for an answer from that program's stdout (without closing the ingress pipe), the answer is stuck in the stdio buffers and will never come out - This is a deadlock.
You can trick some line-oriented programs (eg grep) into not buffering by using a pseudo-tty to talk to them; take a look at libexpect(3). But in the general case, you would have to re-run a different subprocess for each message, allowing to use EOF to signal the end of each message and cause whatever buffers in the command (or pipeline of commands) to be flushed. Obviously not a good thing performance-wise.
See more info about this problem in the perlipc man page (it's for bi-directional pipes in Perl but the buffering considerations apply regardless of the language used for the main program).
You want something often called popen2. Here's a basic implementation without error checking (found by a web search, not my code):
// http://media.unpythonic.net/emergent-files/01108826729/popen2.c
#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
#include "popen2.h"
int popen2(const char *cmdline, struct popen2 *childinfo) {
pid_t p;
int pipe_stdin[2], pipe_stdout[2];
if(pipe(pipe_stdin)) return -1;
if(pipe(pipe_stdout)) return -1;
//printf("pipe_stdin[0] = %d, pipe_stdin[1] = %d\n", pipe_stdin[0], pipe_stdin[1]);
//printf("pipe_stdout[0] = %d, pipe_stdout[1] = %d\n", pipe_stdout[0], pipe_stdout[1]);
p = fork();
if(p < 0) return p; /* Fork failed */
if(p == 0) { /* child */
close(pipe_stdin[1]);
dup2(pipe_stdin[0], 0);
close(pipe_stdout[0]);
dup2(pipe_stdout[1], 1);
execl("/bin/sh", "sh", "-c", cmdline, NULL);
perror("execl"); exit(99);
}
childinfo->child_pid = p;
childinfo->to_child = pipe_stdin[1];
childinfo->from_child = pipe_stdout[0];
close(pipe_stdin[0]);
close(pipe_stdout[1]);
return 0;
}
//#define TESTING
#ifdef TESTING
int main(void) {
char buf[1000];
struct popen2 kid;
popen2("tr a-z A-Z", &kid);
write(kid.to_child, "testing\n", 8);
close(kid.to_child);
memset(buf, 0, 1000);
read(kid.from_child, buf, 1000);
printf("kill(%d, 0) -> %d\n", kid.child_pid, kill(kid.child_pid, 0));
printf("from child: %s", buf);
printf("waitpid() -> %d\n", waitpid(kid.child_pid, NULL, 0));
printf("kill(%d, 0) -> %d\n", kid.child_pid, kill(kid.child_pid, 0));
return 0;
}
#endif
popen() can only open the pipe in read or write mode, not both. Take a look at this thread for a workaround.
In one of netresolve backends I'm talking to a script and therefore I need to write to its stdin and read from its stdout. The following function executes a command with stdin and stdout redirected to a pipe. You can use it and adapt it to your liking.
static bool
start_subprocess(char *const command[], int *pid, int *infd, int *outfd)
{
int p1[2], p2[2];
if (!pid || !infd || !outfd)
return false;
if (pipe(p1) == -1)
goto err_pipe1;
if (pipe(p2) == -1)
goto err_pipe2;
if ((*pid = fork()) == -1)
goto err_fork;
if (*pid) {
/* Parent process. */
*infd = p1[1];
*outfd = p2[0];
close(p1[0]);
close(p2[1]);
return true;
} else {
/* Child process. */
dup2(p1[0], 0);
dup2(p2[1], 1);
close(p1[0]);
close(p1[1]);
close(p2[0]);
close(p2[1]);
execvp(*command, command);
/* Error occured. */
fprintf(stderr, "error running %s: %s", *command, strerror(errno));
abort();
}
err_fork:
close(p2[1]);
close(p2[0]);
err_pipe2:
close(p1[1]);
close(p1[0]);
err_pipe1:
return false;
}
https://github.com/crossdistro/netresolve/blob/master/backends/exec.c#L46
(I used the same code in Can popen() make bidirectional pipes like pipe() + fork()?)
Use forkpty (it's non-standard, but the API is very nice, and you can always drop in your own implementation if you don't have it) and exec the program you want to communicate with in the child process.
Alternatively, if tty semantics aren't to your liking, you could write something like forkpty but using two pipes, one for each direction of communication, or using socketpair to communicate with the external program over a unix socket.
You can't use popen to use two-way pipes.
In fact, some OSs don't support two-way pipes, in which case a socket-pair (socketpair) is the only way to do it.
popen works for me in both directions (read and write)
I have been using a popen() pipe in both directions..
Reading and writing a child process stdin and stdout with the file descriptor returned by popen(command,"w")
It seems to work fine..
I assumed it would work before I knew better, and it does.
According posts above this shouldn't work.. which worries me a little bit.
gcc on raspbian (raspbery pi debian)
This question already has answers here:
How do I execute a command and get the output of the command within C++ using POSIX?
(12 answers)
Closed 7 years ago.
I'm trying to start an external application through system() - for example, system("ls"). I would like to capture its output as it happens so I can send it to another function for further processing. What's the best way to do that in C/C++?
From the popen manual:
#include <stdio.h>
FILE *popen(const char *command, const char *type);
int pclose(FILE *stream);
Try the popen() function. It executes a command, like system(), but directs the output into a new file. A pointer to the stream is returned.
FILE *lsofFile_p = popen("lsof", "r");
if (!lsofFile_p)
{
return -1;
}
char buffer[1024];
char *line_p = fgets(buffer, sizeof(buffer), lsofFile_p);
pclose(lsofFile_p);
EDIT: misread question as wanting to pass output to another program, not another function. popen() is almost certainly what you want.
System gives you full access to the shell. If you want to continue using it, you can
redirect it's output to a temporary file, by system("ls > tempfile.txt"), but choosing a secure temporary file is a pain. Or, you can even redirect it through another program: system("ls | otherprogram");
Some may recommend the popen() command. This is what you want if you can process the output yourself:
FILE *output = popen("ls", "r");
which will give you a FILE pointer you can read from with the command's output on it.
You can also use the pipe() call to create a connection in combination with fork() to create new processes, dup2() to change the standard input and output of them, exec() to run the new programs, and wait() in the main program to wait for them. This is just setting up the pipeline much like the shell would. See the pipe() man page for details and an example.
The functions popen() and such don't redirect stderr and such; I wrote popen3() for that purpose.
Here's a bowdlerised version of my popen3():
int popen3(int fd[3],const char **const cmd) {
int i, e;
int p[3][2];
pid_t pid;
// set all the FDs to invalid
for(i=0; i<3; i++)
p[i][0] = p[i][1] = -1;
// create the pipes
for(int i=0; i<3; i++)
if(pipe(p[i]))
goto error;
// and fork
pid = fork();
if(-1 == pid)
goto error;
// in the parent?
if(pid) {
// parent
fd[STDIN_FILENO] = p[STDIN_FILENO][1];
close(p[STDIN_FILENO][0]);
fd[STDOUT_FILENO] = p[STDOUT_FILENO][0];
close(p[STDOUT_FILENO][1]);
fd[STDERR_FILENO] = p[STDERR_FILENO][0];
close(p[STDERR_FILENO][1]);
// success
return 0;
} else {
// child
dup2(p[STDIN_FILENO][0],STDIN_FILENO);
close(p[STDIN_FILENO][1]);
dup2(p[STDOUT_FILENO][1],STDOUT_FILENO);
close(p[STDOUT_FILENO][0]);
dup2(p[STDERR_FILENO][1],STDERR_FILENO);
close(p[STDERR_FILENO][0]);
// here we try and run it
execv(*cmd,const_cast<char*const*>(cmd));
// if we are there, then we failed to launch our program
perror("Could not launch");
fprintf(stderr," \"%s\"\n",*cmd);
_exit(EXIT_FAILURE);
}
// preserve original error
e = errno;
for(i=0; i<3; i++) {
close(p[i][0]);
close(p[i][1]);
}
errno = e;
return -1;
}
The most efficient way is to use stdout file descriptor directly, bypassing FILE stream:
pid_t popen2(const char *command, int * infp, int * outfp)
{
int p_stdin[2], p_stdout[2];
pid_t pid;
if (pipe(p_stdin) == -1)
return -1;
if (pipe(p_stdout) == -1) {
close(p_stdin[0]);
close(p_stdin[1]);
return -1;
}
pid = fork();
if (pid < 0) {
close(p_stdin[0]);
close(p_stdin[1]);
close(p_stdout[0]);
close(p_stdout[1]);
return pid;
} else if (pid == 0) {
close(p_stdin[1]);
dup2(p_stdin[0], 0);
close(p_stdout[0]);
dup2(p_stdout[1], 1);
dup2(::open("/dev/null", O_WRONLY), 2);
/// Close all other descriptors for the safety sake.
for (int i = 3; i < 4096; ++i) {
::close(i);
}
setsid();
execl("/bin/sh", "sh", "-c", command, NULL);
_exit(1);
}
close(p_stdin[0]);
close(p_stdout[1]);
if (infp == NULL) {
close(p_stdin[1]);
} else {
*infp = p_stdin[1];
}
if (outfp == NULL) {
close(p_stdout[0]);
} else {
*outfp = p_stdout[0];
}
return pid;
}
To read output from child use popen2() like this:
int child_stdout = -1;
pid_t child_pid = popen2("ls", 0, &child_stdout);
if (!child_pid) {
handle_error();
}
char buff[128];
ssize_t bytes_read = read(child_stdout, buff, sizeof(buff));
To both write and read:
int child_stdin = -1;
int child_stdout = -1;
pid_t child_pid = popen2("grep 123", &child_stdin, &child_stdout);
if (!child_pid) {
handle_error();
}
const char text = "1\n2\n123\n3";
ssize_t bytes_written = write(child_stdin, text, sizeof(text) - 1);
char buff[128];
ssize_t bytes_read = read(child_stdout, buff, sizeof(buff));
The functions popen() and pclose() could be what you're looking for.
Take a look at the glibc manual for an example.
In Windows, instead of using system(), use CreateProcess, redirect the output to a pipe and connect to the pipe.
I'm guessing this is also possible in some POSIX way?
Actually, I just checked, and:
popen is problematic, because the process is forked. So if you need to wait for the shell command to execute, then you're in danger of missing it. In my case, my program closed even before the pipe got to do it's work.
I ended up using system call with tar command on linux. The return value from system was the result of tar.
So: if you need the return value, then not no only is there no need to use popen, it probably won't do what you want.
In this page: capture_the_output_of_a_child_process_in_c describes the limitations of using popen vs. using fork/exec/dup2/STDOUT_FILENO approach.
I'm having problems capturing tshark output with popen.
And I'm guessing that this limitation might be my problem:
It returns a stdio stream as opposed to a raw file descriptor, which
is unsuitable for handling the output asynchronously.
I'll come back to this answer if I have a solution with the other approach.
I'm not entirely certain that its possible in standard C, as two different processes don't typically share memory space. The simplest way I can think of to do it would be to have the second program redirect its output to a text file (programname > textfile.txt) and then read that text file back in for processing. However, that may not be the best way.