Child process is blocked by full pipe, cannot read in parent process - c++

I have roughly created the following code to call a child process:
// pipe meanings
const int READ = 0;
const int WRITE = 1;
int fd[2];
// Create pipes
if (pipe(fd))
{
throw ...
}
p_pid = fork();
if (p_pid == 0) // in the child
{
close(fd[READ]);
if (dup2(fd[WRITE], fileno(stdout)) == -1)
{
throw ...
}
close(fd[WRITE]);
// Call exec
execv(argv[0], const_cast<char*const*>(&argv[0]));
_exit(-1);
}
else if (p_pid < 0) // fork has failed
{
throw
}
else // in th parent
{
close(fd[WRITE]);
p_stdout = new std::ifstream(fd[READ]));
}
Now, if the subprocess does not write too much to stdout, I can wait for it to finish and then read the stdout from p_stdout. If it writes too much, the write blocks and the parent waits for it forever.
To fix this, I tried to wait with WNOHANG in the parent, if it is not finished, read all available output from p_stdout using readsome, sleep a bit and try again. Unfortunately, readsome never reads anything:
while (true)
{
if (waitid(P_PID, p_pid, &info, WEXITED | WNOHANG) != 0)
throw ...;
else if (info.si_pid != 0) // waiting has succeeded
break;
char tmp[1024];
size_t sizeRead;
sizeRead = p_stdout->readsome(tmp, 1024);
if (sizeRead > 0)
s_stdout.write(tmp, sizeRead);
sleep(1);
}
The question is: Why does this not work and how can I fix it?
edit: If there is only child, simply using read instead of readsome would probably work, but the process has multiple children and needs to react as soon as one of them terminates.

As sarnold suggested, you need to change the order of your calls. Read first, wait last. Even if your method worked, you might miss the last read. i.e. you exit the loop before you read the last set of bytes that was written.
The problem might be is that ifstream is non-blocking. I've never liked iostreams, even in my C++ projects, I always liked the simplicity of C's stdio functions (i.e. FILE*, fprintf, etc). One way to get around this is to read if the descriptor is readable. You can use select to determine if there is data waiting on that pipe. You're going to need select if you are going to read from multiple children anyway, so might as well learn it now.
As for a quick isreadable function, try something like this (please note I haven't tried compiling this):
bool isreadable(int fd, int timeoutSecs)
{
struct timeval tv = { timeoutSecs, 0 };
fd_set readSet;
FD_ZERO(&readSet);
return select(fds, &readSet, NULL, NULL, &tv) == 1;
}
Then in your parent code, do something like:
while (true) {
if (isreadable(fd[READ], 1)) {
// read fd[READ];
if (bytes <= 0)
break;
}
}
wait(pid);

I'd suggest re-writing the code so that it doesn't call waitpid(2) until after read(2) calls on the pipe return 0 to signify end-of-file. Once you get the end-of-file return from your read calls, you know the child is dead, and you can finally waitpid(2) for it.
Another option is to de-couple the reading from the reaping even further and perform the wait calls in a SIGCHLD signal handler asynchronously to the reading operations.

Related

Do input redirection and capture command output (Custom shell-like program)

I'm writing a custom shell where I try to add support for input, output redirections and pipes just like standard shell. I stuck at point where I cannot do input redirection, but output redirection is perfectly working. My implementation is something like this (only related part), you can assume that (string) input is non-empty
void execute() {
... // stuff before execution and initialization of variables
int *fds;
std::string content;
std::string input = readFromAFile(in_file); // for input redirection
for (int i = 0; i < commands.size(); i++) {
fds = subprocess(commands[i]);
dprintf(fds[1], "%s", input.data()); // write to write-end of pipe
close(fds[1]);
content += readFromFD(fds[0]); // read from read-end of pipe
close(fds[0]);
}
... // stuff after execution
}
int *subprocess(std::string &cmd) {
std::string s;
int *fds = new int[2];
pipe(fds);
pid_t pid = fork();
if (pid == -1) {
std::cerr << "Fork failed.";
}
if (pid == 0) {
dup2(fds[1], STDOUT_FILENO);
dup2(fds[0], STDIN_FILENO);
close(fds[1]);
close(fds[0]);
system(cmd.data());
exit(0); // child terminates
}
return fds;
}
My thought is subprocess returns a pipe (fd_in, fd_out) and parent can write to write-end and read-from read-end afterwards. However when I try an input redirection something like sort < in.txt, the program just hangs. I think there is a deadlock because one waiting other to write, and other one to read, however, after parent writes to write-end it closes, and then read from read-end. How should I consider this case ?
When I did a bit of searching, I saw this answer, which my original thinking was similar except that in the answer it mentions creating two pipes. I did not quite understand this part. Why do we need two separate pipes ?

C++ both input and output pipe to the external program

I am trying to invoke external program with some input and retrieve the output from it within a program.
It will be look like;
(some input) | (external program) | (retrieve output)
I first thought about using a popen() but it seems like, it is not possible because the pipe is not bidirectional.
Is there any easy way to handle this kind of stuff in linux?
I can try making a temp file but it will be great if it can be handled clearly without accessing the disk.
Any Solution? Thanks.
On linux you can use pipe function: Open two new pipes, one for each direction, then create a child process using fork, afterwards, you typically close the file descriptors not in use (read end on parent, write end on child of the pipe for parent sending to child and vice versa for the other pipe) and then start your application using execve or one of its front ends.
If you dup2 the pipes' file descriptors to the standard console file handles (STDIN_FILENO/STDOUT_FILENO; each process separately), you should even be able to use std::cin/std::cout for communicating with the other process (you might want to do so only for the child, as you might want to keep your console in parent). I have no tested this, though, so that's left to you.
When done, you'd yet wait or waitpid for your child process to terminate. Might look like similar to the following piece of code:
int pipeP2C[2], pipeC2P[2];
// (names: short for pipe for X (writing) to Y with P == parent, C == child)
if(pipe(pipeP2C) != 0 || pipe(pipeC2P) != 0)
{
// error
// TODO: appropriate handling
}
else
{
int pid = fork();
if(pid < 0)
{
// error
// TODO: appropriate handling
}
else if(pid > 0)
{
// parent
// close unused ends:
close(pipeP2C[0]); // read end
close(pipeC2P[1]); // write end
// use pipes to communicate with child...
int status;
waitpid(pid, &status, 0);
// cleanup or do whatever you want to do afterwards...
}
else
{
// child
close(pipeP2C[1]); // write end
close(pipeC2P[0]); // read end
dup2(pipeP2C[0], STDIN_FILENO);
dup2(pipeC2P[1], STDOUT_FILENO);
// you should be able now to close the two remaining
// pipe file desciptors as well as you dup'ed them already
// (confirmed that it is working)
close(pipeP2C[0]);
close(pipeC2P[1]);
execve(/*...*/); // won't return - but you should now be able to
// use stdin/stdout to communicate with parent
}
}

child process hanging while reading from stdin (fork/dup2 race condition)?

I have a process that forks in order to execute a subprocess, which receive an entry from stdin and writes to stdout.
My code in short is as follows:
int fd[2];
int fd2[2];
if (pipe(fd) < 0 || pipe(fd2) < 0)
throws exception;
pid_t p = fork();
if (p == 0) // child
{
close(fd[0]); //not needed
dup2( fd[1],STDOUT_FILENO);
dup2( fd[1],STDERR_FILENO);
close(fd2[1]); //not needed
//what if write calls on parent process execute first?
//how to handle that situation
dup2( fd2[0],STDIN_FILENO);
string cmd="./childbin";
if (execl(cmd.c_str(),(char *) NULL) == -1)
{
exit (-1);
}
exit(-1);
}
else if (p > 0) // parent
{
close(fd[1]); //not needed
close(fd2[0]);
if (write(fd2[1],command.c_str(),command.size())<0)
{
throw exception;
}
close(fd2[1]);
//waits for child to finish.
//child process actually hangs on reading for ever from stdin.
pidret=waitpid(p,&status,WNOHANG))==0)
.......
}
The child process remains waiting forever for data in STDIN. Is there maybe a race condition between the child and parent process? I think that could be the problem but not quite sure and also not sure how to fix it.
Thanks in advance.
Update:
Some useful information.
The parent process is a daemon and this code runs several times per second. It works 97% of the times (~3% of the cases, the child process remains in the state described before).
UPDATE 2
After added validation in dup2 call, there is no error there, next condition is never raised.
if(dup2(...) == -1) {
syslog(...)
}
Your missing a wait that is why you in 3% of the cases run the parent before the child. See the example at the bottom.
Also you should call close on the fd's you don't use before doing anything else.

read() on a pipe is not blocking

I have following piece of code where I am using the pipe for two way read and write between parent and child process.
From what I have read, if I dont use O_NONBLOCK, the read should block until the data is written to the pipe from the other side.
However, I notice that the read on the parent side did not block. I know that, since I am debugging in gdb, I have put a sleep as a first statement inside the child.
Why did the read() by parent not block here? Also, is there anything else that I need to do to synchronize the read/write as below between the two processes?
typedef struct
{
int x;
int y;
}PayLoad;
PayLoad pl;
bool b = false;
int pipe_fds[2];
void p(int i, int j)
{
pl.x = i;
pl.y = j;
pipe(pipe_fds);
pid_t cpid = fork();
if (cpid == 0) // child process
{
std::this_thread::sleep_for(std::chrono::seconds(100)); // just for debugging
close(pipe_fds[1]);
read(pipe_fds[0], &pl, sizeof(Payload));
//... do some processing on read data
close(pipe_fds[0]);
write(pipe_fds[1], &b, sizeof(bool));
close(pipe_fds[1]);
}
else if (cpid > 0) // parent process
{
close(pipe_fds[0]);
write(pipe_fds[1], &pl, sizeof(Payload));
close(pipe_fds[1]);
read(pipe_fds[0], &b, sizeof(bool)); <------ did not block!
close(pipe_fds[0]);
}
}
If O_NONBLOCK is set, read() will return a -1 and set errno to [EAGAIN].
The real problem is you are closing the file descriptors before using them. For example, in the child process, you are closing pipe_fds[1] and you are using it for writing some value. In the parent process, you are closing pipe_fds[0] and you are using it for reading some value. Once the process closes the file descriptor, the process shouldn't use it for reading or writing. Usually pipe concept is one process (either parent or child) will write using one of file descriptors created by pipe and the other process (either parent or child) will read the data using another file descriptor.

waitpid/wexitstatus returning 0 instead of correct return code

I have the helper function below, used to execute a command and get the return value on posix systems. I used to use popen, but it is impossible to get the return code of an application with popen if it runs and exits before popen/pclose gets a chance to do its work.
The following helper function creates a process fork, uses execvp to run the desired external process, and then the parent uses waitpid to get the return code. I'm seeing odd cases where it's refusing to run.
When called with wait = true, waitpid should return the exit code of the application no matter what. However, I'm seeing stdout output that specifies the return code should be non-zero, yet the return code is zero. Testing the external process in a regular shell, then echoing $? returns non-zero, so it's not a problem w/ the external process not returning the right code. If it's of any help, the external process being run is mount(8) (yes, I know I can use mount(2) but that's besides the point).
I apologize in advance for a code dump. Most of it is debugging/logging:
inline int ForkAndRun(const std::string &command, const std::vector<std::string> &args, bool wait = false, std::string *output = NULL)
{
std::string debug;
std::vector<char*> argv;
for(size_t i = 0; i < args.size(); ++i)
{
argv.push_back(const_cast<char*>(args[i].c_str()));
debug += "\"";
debug += args[i];
debug += "\" ";
}
argv.push_back((char*)NULL);
neosmart::logger.Debug("Executing %s", debug.c_str());
int pipefd[2];
if (pipe(pipefd) != 0)
{
neosmart::logger.Error("Failed to create pipe descriptor when trying to launch %s", debug.c_str());
return EXIT_FAILURE;
}
pid_t pid = fork();
if (pid == 0)
{
close(pipefd[STDIN_FILENO]); //child isn't going to be reading
dup2(pipefd[STDOUT_FILENO], STDOUT_FILENO);
close(pipefd[STDOUT_FILENO]); //now that it's been dup2'd
dup2(pipefd[STDOUT_FILENO], STDERR_FILENO);
if (execvp(command.c_str(), &argv[0]) != 0)
{
exit(EXIT_FAILURE);
}
return 0;
}
else if (pid < 0)
{
neosmart::logger.Error("Failed to fork when trying to launch %s", debug.c_str());
return EXIT_FAILURE;
}
else
{
close(pipefd[STDOUT_FILENO]);
int exitCode = 0;
if (wait)
{
waitpid(pid, &exitCode, wait ? __WALL : (WNOHANG | WUNTRACED));
std::string result;
char buffer[128];
ssize_t bytesRead;
while ((bytesRead = read(pipefd[STDIN_FILENO], buffer, sizeof(buffer)-1)) != 0)
{
buffer[bytesRead] = '\0';
result += buffer;
}
if (wait)
{
if ((WIFEXITED(exitCode)) == 0)
{
neosmart::logger.Error("Failed to run command %s", debug.c_str());
neosmart::logger.Info("Output:\n%s", result.c_str());
}
else
{
neosmart::logger.Debug("Output:\n%s", result.c_str());
exitCode = WEXITSTATUS(exitCode);
if (exitCode != 0)
{
neosmart::logger.Info("Return code %d", (exitCode));
}
}
}
if (output)
{
result.swap(*output);
}
}
close(pipefd[STDIN_FILENO]);
return exitCode;
}
}
Note that the command is run OK with the correct parameters, the function proceeds without any problems, and WIFEXITED returns TRUE. However, WEXITSTATUS returns 0, when it should be returning something else.
Probably isn't your main issue, but I think I see a small problem. In your child process, you have...
dup2(pipefd[STDOUT_FILENO], STDOUT_FILENO);
close(pipefd[STDOUT_FILENO]); //now that it's been dup2'd
dup2(pipefd[STDOUT_FILENO], STDERR_FILENO); //but wait, this pipe is closed!
But I think what you want is:
dup2(pipefd[STDOUT_FILENO], STDOUT_FILENO);
dup2(pipefd[STDOUT_FILENO], STDERR_FILENO);
close(pipefd[STDOUT_FILENO]); //now that it's been dup2'd for both, can close
I don't have much experience with forks and pipes in Linux, but I did write a similar function pretty recently. You can take a look at the code to compare, if you'd like. I know that my function works.
execAndRedirect.cpp
I'm using the mongoose library, and grepping my code for SIGCHLD revealed that using mg_start from mongoose results in setting SIGCHLD to SIG_IGN.
From the waitpid man page, on Linux a SIGCHLD set to SIG_IGN will not create a zombie process, so waitpid will fail if the process has already successfully run and exited - but will run OK if it hasn't yet. This was the cause of the sporadic failure of my code.
Simply re-setting SIGCHLD after calling mg_start to a void function that does absolutely nothing was enough to keep the zombie records from being immediately erased.
Per #Geoff_Montee's advice, there was a bug in my redirect of STDERR, but this was not responsible for the problem as execvp does not store the return value in STDERR or even STDOUT, but rather in the kernel object associated with the parent process (the zombie record).
#jilles' warning about non-contiguity of vector in C++ does not apply for C++03 and up (only valid for C++98, though in practice, most C++98 compilers did use contiguous storage, anyway) and was not related to this issue. However, the advice on reading from the pipe before blocking and checking the output of waitpid is spot-on.
I've found that pclose does NOT block and wait for the process to end, contrary to the documentation (this is on CentOS 6). I've found that I need to call pclose and then call waitpid(pid,&status,0); to get the true return value.