What is fork and what is pipe?
Any scenarios explaining why their use is necessary will be appreciated.
What are the differences between fork and pipe in C?
Can we use them in C++?
I need to know this is because I want to implement a program in C++ which can access live video input, convert its format and write it to a file.
What would be the best approach for this?
I have used x264 for this. So far I have implemented the part of conversion on a file format.
Now I have to implement it on a live stream.
Is it a good idea to use pipes? Capture video in another process and feed it to the other?
A pipe is a mechanism for interprocess communication. Data written to the pipe by one process can be read by another process. The primitive for creating a pipe is the pipe function. This creates both the reading and writing ends of the pipe. It is not very useful for a single process to use a pipe to talk to itself. In typical use, a process creates a pipe just before it forks one or more child processes. The pipe is then used for communication either between the parent or child processes, or between two sibling processes. A familiar example of this kind of communication can be seen in all operating system shells. When you type a command at the shell, it will spawn the executable represented by that command with a call to fork. A pipe is opened to the new child process and its output is read and printed by the shell. This page has a full example of the fork and pipe functions. For your convenience, the code is reproduced below:
#include <sys/types.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
/* Read characters from the pipe and echo them to stdout. */
void
read_from_pipe (int file)
{
FILE *stream;
int c;
stream = fdopen (file, "r");
while ((c = fgetc (stream)) != EOF)
putchar (c);
fclose (stream);
}
/* Write some random text to the pipe. */
void
write_to_pipe (int file)
{
FILE *stream;
stream = fdopen (file, "w");
fprintf (stream, "hello, world!\n");
fprintf (stream, "goodbye, world!\n");
fclose (stream);
}
int
main (void)
{
pid_t pid;
int mypipe[2];
/* Create the pipe. */
if (pipe (mypipe))
{
fprintf (stderr, "Pipe failed.\n");
return EXIT_FAILURE;
}
/* Create the child process. */
pid = fork ();
if (pid == (pid_t) 0)
{
/* This is the child process.
Close other end first. */
close (mypipe[1]);
read_from_pipe (mypipe[0]);
return EXIT_SUCCESS;
}
else if (pid < (pid_t) 0)
{
/* The fork failed. */
fprintf (stderr, "Fork failed.\n");
return EXIT_FAILURE;
}
else
{
/* This is the parent process.
Close other end first. */
close (mypipe[0]);
write_to_pipe (mypipe[1]);
return EXIT_SUCCESS;
}
}
Just like other C functions you can use both fork and pipe in C++.
there are stdin and stdout for common input and output.
A common style is like this:
input->process->output
But with pipe, it becomes:
input->process1->(tmp_output)->(tmp-input)->process2->output
pipe is the function that returns the two temporary tmp-input and tmp-output, i.e. fd[0] and fd[1].
Related
A child process runs a bin file, which is provided by Qualcomm.
The child process is invoked by my parent process, which is developed by me.
When the child process is running, it always prints lots pieces of logs in shell command.
So, am I able to redirect Qualcomm's outstream from stdout to another file in the parent process?
As you know, it's nearly impossible to push Qualcomm to update this bin file.
The key piece here is the POSIX function dup2, which lets you essentially replace one file descriptor with another. And if you use fork (not system), you actually have control of what happens in the child process between the fork and the exec* that loads the other executable.
#include <cstdlib>
extern "C" {
#include <fcntl.h>
#include <unistd.h>
}
#include <stdexcept>
#include <iostream>
pid_t start_child(const char* program, const char* output_filename)
{
pid_t pid = fork();
if (pid < 0) {
// fork failed!
std::perror("fork");
throw std::runtime_error("fork failed");
} else if (pid == 0) {
// This code runs in the child process.
int output_fd = open(output_filename, O_WRONLY | O_CREAT | O_TRUNC);
if (output_fd < 0) {
std::cerr << "Failed to open log file " << output_filename << ":"
<< std::endl;
std::perror("open");
std::exit(1);
}
// Replace the child's stdout and stderr handles with the log file handle:
if (dup2(output_fd, STDOUT_FILENO) < 0) {
std::perror("dup2 (stdout)");
std::exit(1);
}
if (dup2(output_fd, STDERR_FILENO) < 0) {
std::perror("dup2 (stderr)");
std::exit(1);
}
if (execl(program, program, (char*)nullptr) < 0) {
// These messages will actually go into the file.
std::cerr << "Failed to exec program " << program << ":"
<< std::endl;
std::perror("execl");
std::exit(1);
}
}
return pid;
}
It is possible, for POSIX, because the POSIX shells do this. Executing a program has two steps, for POSIX. First use fork to clone the parent process to create the child process. Then have the child process use one of the exec family of system calls to execute the chosen program instead of the program of the parent. In between those two steps the code executing for the child process can do additional operations, which will affect the environment of the program to be executed. In particular, the code could open a file descriptor to the file to be redirected to, close the stdout file descriptor, then duplicate the file's file descriptor to the value (1) used for stdout.
You could create own pipes and attach them to the child process.
Create 3 pipes. they are going to replace stdin, stdout, stderr of the child.
fork()
In subprocess close() the parent end of the pipes. Close stdin,stdout and stderr.
The parent process close() the child end of the pipes.
dup2() the pipe ends in the child process that are intended to work as the new stdin,out,err
exec() the child.
Now you got all Output from the child to the pipe in the parent. Ofcourse you need to read from the pipes that come from the child or it will block on any write to the stdout/stderr. For this you could use a select(), poll(), epoll() multiplexing algorithm.
See
https://linux.die.net/man/2/pipe
https://linux.die.net/man/2/dup2
https://linux.die.net/man/2/execve
https://linux.die.net/man/2/fork
I am trying to create a inter process communication bus using pipes.
I can send messages to the client, and i can receive messages at the client.
I also tried to create another pipe which goes the other way, child-> parent.
However I need this communication to be non blocking and I want to send messages in both directions with some undefined time gap in between.
It appears i can not send more then one message, I take it it is due to the fclose(); closing the stream, however if i remove the close, the program stops working...
How can I make this code nonblocking two way parent -> child communication through pipes?
Later I want to connect a child-binary using exec(), and connecting this binary's stdin, stdout to my parent, so that I can pass messages to the stdin of the child and retrieve answers through the "childParent" descriptor.
#include <sys/types.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#define READ_SIDE 0
#define WRITE_SIDE 1
/* Read characters from the pipe and echo them to stdout. */
void
read_from_pipe (int file)
{
FILE *stream;
int c;
stream = fdopen (file, "r");
while ((c = fgetc (stream)) != EOF)
putchar (c);
fclose (stream);
}
/* Write some random text to the pipe. */
void
write_to_pipe (int file, char * message)
{
FILE *stream;
stream = fdopen (file, "w");
fprintf (stream, message);
//fprintf (stream, "goodbye, world!\n");
//Dont forget to close
fclose (stream);
}
void end(){
//kill process and close pipes
}
int
main (void)
{
pid_t pid;
int parentToChild[2];
int childToParent[2];
/* Create the pipes. */
if (pipe (parentToChild))
{
fprintf (stderr, "Pipe failed.\n");
return EXIT_FAILURE;
}
if (pipe (childToParent))
{
fprintf (stderr, "Pipe failed.\n");
return EXIT_FAILURE;
}
/* Create the child process. */
pid = fork ();
if (pid == (pid_t) 0)
{
/* This is the child process.
Close other end first. */
close(childToParent[READ_SIDE]);
close (parentToChild[WRITE_SIDE]);
read_from_pipe (parentToChild[READ_SIDE]);
write_to_pipe(childToParent[WRITE_SIDE], "Child sent message to parent");
return EXIT_SUCCESS;
}
else if (pid < (pid_t) 0)
{
/* The fork failed. */
fprintf (stderr, "Fork failed.\n");
return EXIT_FAILURE;
}
else
{
/* This is the parent process.
Close other end first. */
close(childToParent[WRITE_SIDE]);
close (parentToChild[READ_SIDE]);
write_to_pipe (parentToChild[WRITE_SIDE],"Parent sent message to child");
read_from_pipe(childToParent[READ_SIDE]);
//Send more messages...
return EXIT_SUCCESS;
}
}
UPDATE
It appears that sometimes the communication only happens one way, depending on which process gets to read/write first I guess.
There are several issues to be considered. First, pipes have a finite
maximum length (4096 was common in the distant past); any write to a
pipe which has more data in it than that will block, as will any read
from an empty pipe. These are fundamental to the way pipes work, so
when you speak of non-blocking, you have to take them into consideration
(or use asynchronous IO).
Second, if you really need to control things at this level, you might
want to consider using Posix level IO, rather than streams. If you do
use streams (either iostream or FILE*), then you have to take into
account the buffering they use. In particular, when you want to be sure
that the data is output to the pipe, you need to flush (fflush on
the FILE*). With regards to sending more than one message, and the
code not working without the fclose, replacing the fclose with
fflush should be all you need to fix this.
And while I'm at it: your use of fprintf for output is extremely
dangerous. If you're getting a message from an external source, and
(possibly) don't know what it contains, you should use fputs to output
it, or possibly something like fprintf( fd, "%s\n", message ) (if you
want to append a new line). As you've done it, if the message contains
a '%', strange things will happen. More generally, you should prefer
the much safer iostream, but for simple things like this, it doesn't
matter much. And for what you're actually doing, Posix level IO is
probably just as appropriate, or more so. With Posix level IO, you
don't get any formatting or buffering: but since you're not using the
formatting capabilities, and the buffering it part of your problem...
I am trying to write a program that runs an external program.
I know that I can catch stdout, and I can catch stdout and stderr together BUT the question is can I catch the stderr and stdout separated?
I mean for example, stderr in variable STDERR and stdout in variable STDOUT. I mean I want them separated.
Also I need the exit code of the external program in a variable.
On Windows you must fill STARTUPINFO for the CreateProcess to catch standart streams, and you can use GetExitCodeProcess function to get the termination status. There is an example how to redirect standart streams into the parent process http://msdn.microsoft.com/en-us/library/windows/desktop/ms682499.aspx
On Linux-like OS you probably want to use fork instead of execve, and working with a forked process is another story.
In Windows and Linux redirecting streams has general approach - you must create several pipes (one for each stream) and redirect child process streams into that pipes, and the parent process can read data from that pipes.
Sample code for Linux:
int fd[2];
if (pipe(fd) == -1) {
perror("pipe");
exit(EXIT_FAILURE);
}
pid_t cpid = fork();
if (cpid == -1) {
perror("fork");
exit(EXIT_FAILURE);
}
if (cpid == 0) { // child
dup2(fd[1], STDERR_FILENO);
fprintf(stderr, "Hello, World!\n");
exit(EXIT_SUCCESS);
} else { // parent
char ch;
while (read(fd[0], &ch, 1) > 0)
printf("%c", ch);
exit(EXIT_SUCCESS);
}
EDIT: If you need to catch streams from another program, use the same stragey as above, first fork, second - use pipes (as in code above), then execve another progrram in child process and use this code in parent process to wait an execution end and catch a return code:
int status;
if (waitpid(cpid, &status, 0) < 0) {
perror("waitpid");
exit(EXIT_FAILURE);
}
You can find more details in man pages pipe, dup2 and waitpid.
This question already has answers here:
Can popen() make bidirectional pipes like pipe() + fork()?
(6 answers)
Closed 3 years ago.
Is it possible to read and write to a file descriptor returned by popen. I have an interactive process I'd like to control through C. If this isn't possible with popen, is there any way around it?
As already answered, popen works in one direction. If you need to read and write, You can create a pipe with pipe(), span a new process by fork() and exec functions and then redirect its input and outputs with dup2(). Anyway I prefer exec over popen, as it gives you better control over the process (e.g. you know its pid)
EDITED:
As comments suggested, a pipe can be used in one direction only. Therefore you have to create separate pipes for reading and writing. Since the example posted before was wrong, I deleted it and created a new, correct one:
#include<unistd.h>
#include<sys/wait.h>
#include<sys/prctl.h>
#include<signal.h>
#include<stdlib.h>
#include<string.h>
#include<stdio.h>
int main(int argc, char** argv)
{
pid_t pid = 0;
int inpipefd[2];
int outpipefd[2];
char buf[256];
char msg[256];
int status;
pipe(inpipefd);
pipe(outpipefd);
pid = fork();
if (pid == 0)
{
// Child
dup2(outpipefd[0], STDIN_FILENO);
dup2(inpipefd[1], STDOUT_FILENO);
dup2(inpipefd[1], STDERR_FILENO);
//ask kernel to deliver SIGTERM in case the parent dies
prctl(PR_SET_PDEATHSIG, SIGTERM);
//replace tee with your process
execl("/usr/bin/tee", "tee", (char*) NULL);
// Nothing below this line should be executed by child process. If so,
// it means that the execl function wasn't successfull, so lets exit:
exit(1);
}
// The code below will be executed only by parent. You can write and read
// from the child using pipefd descriptors, and you can send signals to
// the process using its pid by kill() function. If the child process will
// exit unexpectedly, the parent process will obtain SIGCHLD signal that
// can be handled (e.g. you can respawn the child process).
//close unused pipe ends
close(outpipefd[0]);
close(inpipefd[1]);
// Now, you can write to outpipefd[1] and read from inpipefd[0] :
while(1)
{
printf("Enter message to send\n");
scanf("%s", msg);
if(strcmp(msg, "exit") == 0) break;
write(outpipefd[1], msg, strlen(msg));
read(inpipefd[0], buf, 256);
printf("Received answer: %s\n", buf);
}
kill(pid, SIGKILL); //send SIGKILL signal to the child process
waitpid(pid, &status, 0);
}
The reason popen() and friends don't offer bidirectional communication is that it would be deadlock-prone, due to buffering in the subprocess. All the makeshift pipework and socketpair() solutions discussed in the answers suffer from the same problem.
Under UNIX, most commands cannot be trusted to read one line and immediately process it and print it, except if their standard output is a tty. The reason is that stdio buffers output in userspace by default, and defers the write() system call until either the buffer is full or the stdio stream is closed (typically because the program or script is about to exit after having seen EOF on input). If you write to such a program's stdin through a pipe, and now wait for an answer from that program's stdout (without closing the ingress pipe), the answer is stuck in the stdio buffers and will never come out - This is a deadlock.
You can trick some line-oriented programs (eg grep) into not buffering by using a pseudo-tty to talk to them; take a look at libexpect(3). But in the general case, you would have to re-run a different subprocess for each message, allowing to use EOF to signal the end of each message and cause whatever buffers in the command (or pipeline of commands) to be flushed. Obviously not a good thing performance-wise.
See more info about this problem in the perlipc man page (it's for bi-directional pipes in Perl but the buffering considerations apply regardless of the language used for the main program).
You want something often called popen2. Here's a basic implementation without error checking (found by a web search, not my code):
// http://media.unpythonic.net/emergent-files/01108826729/popen2.c
#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
#include "popen2.h"
int popen2(const char *cmdline, struct popen2 *childinfo) {
pid_t p;
int pipe_stdin[2], pipe_stdout[2];
if(pipe(pipe_stdin)) return -1;
if(pipe(pipe_stdout)) return -1;
//printf("pipe_stdin[0] = %d, pipe_stdin[1] = %d\n", pipe_stdin[0], pipe_stdin[1]);
//printf("pipe_stdout[0] = %d, pipe_stdout[1] = %d\n", pipe_stdout[0], pipe_stdout[1]);
p = fork();
if(p < 0) return p; /* Fork failed */
if(p == 0) { /* child */
close(pipe_stdin[1]);
dup2(pipe_stdin[0], 0);
close(pipe_stdout[0]);
dup2(pipe_stdout[1], 1);
execl("/bin/sh", "sh", "-c", cmdline, NULL);
perror("execl"); exit(99);
}
childinfo->child_pid = p;
childinfo->to_child = pipe_stdin[1];
childinfo->from_child = pipe_stdout[0];
close(pipe_stdin[0]);
close(pipe_stdout[1]);
return 0;
}
//#define TESTING
#ifdef TESTING
int main(void) {
char buf[1000];
struct popen2 kid;
popen2("tr a-z A-Z", &kid);
write(kid.to_child, "testing\n", 8);
close(kid.to_child);
memset(buf, 0, 1000);
read(kid.from_child, buf, 1000);
printf("kill(%d, 0) -> %d\n", kid.child_pid, kill(kid.child_pid, 0));
printf("from child: %s", buf);
printf("waitpid() -> %d\n", waitpid(kid.child_pid, NULL, 0));
printf("kill(%d, 0) -> %d\n", kid.child_pid, kill(kid.child_pid, 0));
return 0;
}
#endif
popen() can only open the pipe in read or write mode, not both. Take a look at this thread for a workaround.
In one of netresolve backends I'm talking to a script and therefore I need to write to its stdin and read from its stdout. The following function executes a command with stdin and stdout redirected to a pipe. You can use it and adapt it to your liking.
static bool
start_subprocess(char *const command[], int *pid, int *infd, int *outfd)
{
int p1[2], p2[2];
if (!pid || !infd || !outfd)
return false;
if (pipe(p1) == -1)
goto err_pipe1;
if (pipe(p2) == -1)
goto err_pipe2;
if ((*pid = fork()) == -1)
goto err_fork;
if (*pid) {
/* Parent process. */
*infd = p1[1];
*outfd = p2[0];
close(p1[0]);
close(p2[1]);
return true;
} else {
/* Child process. */
dup2(p1[0], 0);
dup2(p2[1], 1);
close(p1[0]);
close(p1[1]);
close(p2[0]);
close(p2[1]);
execvp(*command, command);
/* Error occured. */
fprintf(stderr, "error running %s: %s", *command, strerror(errno));
abort();
}
err_fork:
close(p2[1]);
close(p2[0]);
err_pipe2:
close(p1[1]);
close(p1[0]);
err_pipe1:
return false;
}
https://github.com/crossdistro/netresolve/blob/master/backends/exec.c#L46
(I used the same code in Can popen() make bidirectional pipes like pipe() + fork()?)
Use forkpty (it's non-standard, but the API is very nice, and you can always drop in your own implementation if you don't have it) and exec the program you want to communicate with in the child process.
Alternatively, if tty semantics aren't to your liking, you could write something like forkpty but using two pipes, one for each direction of communication, or using socketpair to communicate with the external program over a unix socket.
You can't use popen to use two-way pipes.
In fact, some OSs don't support two-way pipes, in which case a socket-pair (socketpair) is the only way to do it.
popen works for me in both directions (read and write)
I have been using a popen() pipe in both directions..
Reading and writing a child process stdin and stdout with the file descriptor returned by popen(command,"w")
It seems to work fine..
I assumed it would work before I knew better, and it does.
According posts above this shouldn't work.. which worries me a little bit.
gcc on raspbian (raspbery pi debian)
I'm trying to use a socketpair to have a parent process provide input to a child process that execs a different program (e.g., grep) and then read the resulting output. The program hangs in the while loop that reads the output from the program that the child execs.. The child dupes stdin and stdout on to its end of the socketpair and the parent and the child both close their unused end of the pair.
Interestingly, if the child execs a program that I wrote (OK, I ripped it off from Stevens Advanced Programming in the Unix Environment) everything works as expected. However, if the child execs grep (or some other standard program) the parent invariably hangs in trying to read the output. I can't tell if the input is not reaching grep or if the grep cannot determine the end of the input or if the output is somehow being lost.
Here's the code:
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/wait.h>
#include <unistd.h>
#include <signal.h>
#include <cstdio>
#include <cerrno>
#include <iostream>
using namespace std;
void
sigpipe_handler(int sig, siginfo_t *siginfo, void * context) {
cout << "caught SIGPIPE\n";
pid_t pid;
if (errno == EPIPE) {
throw "SIGPIPE caught";
}
}
int main(int argc, char** argv) {
struct sigaction sa;
memset(&sa, '\0', sizeof(struct sigaction));
sa.sa_sigaction = sigpipe_handler;
sa.sa_flags = SA_SIGINFO | SA_RESTART;
sigaction(SIGPIPE, &sa, NULL);
int sp[2];
socketpair(PF_UNIX, SOCK_STREAM, AF_UNIX, sp);
pid_t childPid = fork();
if (childPid == 0) {
close(sp[0]);
if (dup2(sp[1], STDIN_FILENO) != STDIN_FILENO) throw "dup2 error to stdin";
if (dup2(sp[1], STDOUT_FILENO) != STDOUT_FILENO) throw "dup2 error to stdout";
execl("/bin/grep", "grep", "-n", "namespace", (char*)NULL);
} else {
close(sp[1]);
char line[80];
int n;
try {
while (fgets(line, 80, stdin) != NULL) {
n = strlen(line);
if (write(sp[0], line, n) != n) {
throw "write error to pipe";
}
if ((n=read(sp[0], line, 80)) < 0) { // hangs here
throw "read error from pipe";
}
if (n ==0) {
throw "child closed pipe";
break;
}
line[n] = 0;
if (fputs(line, stdout) == EOF) {
throw "puts error";
}
if (ferror(stdin)) {
throw "fgets error on stdin";
}
exit(0);
}
} catch (const char* e) {
cout << e << endl;
}
int status;
waitpid(childPid, &status, 0);
}
}
Your code hangs as grep's output may be less than 80 bytes and you are issuing a blocking read on sp[0]. The proper way of doing this is by marking both sockets as non-blocking and selecting() over both of them.
You also forgot to close(sp[0]) before you wait(), which will leave your child process waiting for input.
You cannot achieve deadlock-free bidirectional communication with a subprocess using UNIX pipes or socketpairs, because you don't have control over buffering in the subprocess.
It just so happens that cat can be trusted to read one line and immediately print it, regardless of whether its standard output is a tty, a pipe or a socket. This is not the case with grep (and actually most programs using stdio), which will buffer output in-process (in the stdio buffers) and defer the write() call until either the buffer is full or the stdio stream is closed (typically because grep is about to exit after having seen EOF on input).
You can trick line-oriented programs (including grep) into not buffering by using a pseudo-tty instead; take a look at libexpect(3). But in the general case, you would have to re-run a different subprocess for each message, which allows to use EOF to signal the end of each message and cause whatever buffers in the command (or pipeline of commands) to be flushed.
See more info about this problem in the perlipc man page (it's for bi-directional pipes in Perl but the buffering considerations apply regardless of the language used for the main program).
It works fine with cat, so the problem is with grep. May be grep output behave differently when connected to something else than a terminal. Or it is not detecting the pattern for some reason.