I am using the following code to redirect stdout to a pipe, then read all the data from the pipe to a buffer. I have 2 problems:
first problem: when i send a string (after redirection) bigger then the pipe's BUFF_SIZE, the program stops responding (deadlock or something).
second problem: when i try to read from a pipe before something was sent to stdout. I get the same response, the program stops responding - _read command stuck's ...
The issue is that i don't know the amount of data that will be sent to the pipe after the redirection.
The first problem, i don't know how to handle and i'll be glad for help. The second problem i solved by a simple workaround, right after the redirection i print space character to stdout. but i guess that this solution is not the correct one ...
#include <fcntl.h>
#include <io.h>
#include <iostream>
#define READ 0
#define WRITE 1
#define BUFF_SIZE 5
using namespace std;
int main()
{
int stdout_pipe[2];
int saved_stdout;
saved_stdout = _dup(_fileno(stdout)); // save stdout
if(_pipe(stdout_pipe,BUFF_SIZE, O_TEXT) != 0 ) // make a pipe
{
exit(1);
}
fflush( stdout );
if(_dup2(stdout_pipe[1], _fileno(stdout)) != 0 ) //redirect stdout to the pipe
{
exit(1);
}
ios::sync_with_stdio();
setvbuf( stdout, NULL, _IONBF, 0 );
//anything sent to stdout goes now to the pipe
//printf(" ");//workaround for the second problem
printf("123456");//first problem
char buffer[BUFF_SIZE] = {0};
int nOutRead = 0;
nOutRead = _read(stdout_pipe[READ], buffer, BUFF_SIZE); //second problem
buffer[nOutRead] = '\0';
// reconnect stdout
if (_dup2(saved_stdout, _fileno(stdout)) != 0 )
{
exit(1);
}
ios::sync_with_stdio();
printf("buffer: %s\n", buffer);
}
Your problem is that you are using blocking I/O calls, while both ends of the pipe are connected to the same process. If you don't know how much data there will be, this is just a deadlock situation waiting to happen.
printf is a blocking call, which means that it will not return until all data has been written to the output device (the pipe in this case), or until a write error is signalled (for example, the other end of the pipe is closed).
_read works similarly. It only returns when it has a full buffer worth of data or it knows that the end of the input has been reached (which can be signalled by closing the write-end of the pipe).
The only ways around this are
to use non-blocking I/O (which is not feasible if you don't have access to the code that calls printf), or
to ensure the reading and writing happens in different processes or threads, or
to use a temporary file for buffering, instead of the buffer of a pipe.
Pipes are unidirectional. Ie. you can either write to a pipe (x)or you can read from it.
To simulate a pipeline, try the following (the below is C, not C++):
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc)
{
int pfds[2];
pipe(pfds);
if (!fork()) {
close(1); /* close stdout, check for errors */
dup(pfds[1]); /* make stdout same as pfds[1], dup reuses lowest fd */
close(pfds[0]); /* not needed */
execlp("ls", "ls", NULL); /* or write() in whatever way you want */
} else {
close(0); /* close stdin, check for errors please! */
dup(pfds[0]); /* make stdin same as pfds[0] */
close(pfds[1]); /* not needed on this end */
execlp("wc", "wc", "-l", NULL); /* or read() */
}
return 0;
}
[edit] By the way, your code does not overflow a buffer. Its only relation to buffer overflow is that you're reading into a statically allocated array...if you read() more than sizeof buffer elements, then you'll run into problems.
You must use non-blocking I/O if you don't want read or write to be blocked in this case.
Related
I have two separate programs in C++, one that writes to two named pipes in unpredictable intervals and one that should wait to read new content from the pipes whenever available. For simplicity, here my writer only writes two times to the pipes (1st: "One" "Tree" , 2nd: "Two" "Fogs").
My writer program is:
#include <unistd.h>
#include <iostream>
int main()
{
int fd1, fd2;
const char* myfifo1 = "/tmp/myfifo1";
const char* myfifo2 = "/tmp/myfifo2";
mkfifo(myfifo1, 0666);
mkfifo(myfifo2, 0666);
fd1 = open(myfifo1, O_WRONLY);
fd2 = open(myfifo2, O_WRONLY);
write(fd1, "One", sizeof("One"));
write(fd2, "Tree", sizeof("Tree"));
sleep(5);
write(fd1, "Two", sizeof("Two"));
write(fd2, "Fogs", sizeof("Fogs"));
close(fd1);
close(fd2);
unlink(myfifo1);
unlink(myfifo2);
}
My reader program is:
#include <fcntl.h>
#include <stdio.h>
#include <sys/stat.h>
#include <unistd.h>
#define MAX_BUF 1024
int main()
{
int fd1, fd2;
const char * myfifo1 = "/tmp/myfifo1";
const char * myfifo2 = "/tmp/myfifo2";
char buf1[MAX_BUF], buf2[MAX_BUF];
fd1 = open(myfifo1, O_RDONLY);
fd2 = open(myfifo2, O_RDONLY);
while (1) {
read(fd1, buf1, MAX_BUF);
read(fd2, buf2, MAX_BUF);
printf("Received: %s\n", buf1);
printf("Received: %s\n", buf2);
}
close(fd1);
close(fd2);
return 0;
}
I do not want the reader to terminate or close the connection between the reads, I need it to remain active and wait until the writer writes something new in the named pipe.
However, by simultaneously running these two programs (in different cores, first the writer and then the reader) I get:
Received: One
Received: Tree
Received: Two
Received: Fogs
Received: Two
Received: Fogs
Received: Two
Received: Fogs
Received: Two
Received: Fogs
...
...
The wanted behavior would be:
Received: One
Received: Tree
Received: Two
Received: Fogs
and then nothing (the reader should wait for another write).
I understand that I should somehow clear the buffer after I read it, but isn't it the default behavior of read? Since it finished reading "Two" (and also "Fogs"), why does it keep reading it and does not wait for new content?
What modifications should I make?
Thank you very much in advance!
Closing a pipe signals end-of-file to the other end.
read will read zero bytes (and return zero) whenever it encounters an end-of-file condition. Any subsequent read after the end-of-file is a no-op.
If you want to continue reading from the same pipe, you need to check the result of read, and if it returns zero, close the pipe and open it again.
Of course you should not unlink the pipe if you need someone else to continue using it.
One other thing to note: read (1) doesn't terminate the buffer with the 0 byte and (2) doesn't necessarily read the exact amount of bytes that was written by a single write. You should detect the end-of-message yourself, rather than relying on read and write to do it for you. It is entirely possible that read will read a half of the string without the terminating null character, then your printf will print garbage.
I'm trying to multiplex an unnamed pipe with some other file descriptors.
The problem is that the pipe file descriptor always appears in the result of select. In other words the event-loop reads from pipe for infinite times. Here is a metaphor of what I want to do, and what actually happens.
#include <iostream>
#include <stdlib.h>
#include <algorithm>
#include <stdio.h>
#include <errno.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <unistd.h>
using namespace std;
int main(){
fd_set master; // master file descriptor list
fd_set read_fds; // temp file descriptor list for select()
FD_ZERO(&master); // clear the master and temp sets
FD_ZERO(&read_fds);
int fdmax; // maximum file descriptor number
int pfd[2];
if(pipe(pfd)!=0) {cout<<"Unable to create a pipe.\n",exit(1);};
FD_SET(0, &master);
FD_SET(pfd[0],&master);
fdmax=pfd[0];
if(fork()){//Parent
for (;;){
read_fds = master; // copy it
if (select(fdmax+1, &read_fds, NULL, NULL, NULL) == -1) {
perror("select");
exit(4);
}
for(int i = 0; i <= fdmax; i++) {
if (FD_ISSET(i, &read_fds)) {
int n;
char buff[200];
if (i==pfd[0]){
close(pfd[1]);
n=read(pfd[0],buff,sizeof(buff));
buff[n]=0;
cout<<"Read from pipe:"<<buff<<endl;
}else if(i==0){
n=read(0,buff,sizeof(buff));
buff[n]=0;
cout<<"Read from std:"<<buff<<endl;
}
}
}
}
}else{//Child
usleep(50000);
char buff[200]="This is a simple sample.";
close(pfd[0]);
write(pfd[1],buff,sizeof(buff));
close(pfd[1]);
exit(0);
}
}
First of all, the read() call can read less than number of bytes specified in the last argument ant id does not automatically append zero-byte terminator, so your receiving code can easily access uninitialized memory in buff[] and after it (if there is no zero byte). You need to check return value when calling read and use only so many bytes from buffer.
Then, the select call returns when any file descriptor in the readfds set won't block on subsequent read. Which includes end-of-file condition. This likely happens in your case, when the forked process closes its fd. See this SO question too.
Could it be the reason for the problem you encounter? Checking return value when calling read make this clear to you, as read returns zero if-and-only-if the fd reached end of file.
Last detail – it does not make much sense to close pfd[1] only after pfd[0] is returned in the readfds. You should close it immediately after the fork, so it remains open in the child process only if you have no use for it in the parent process.
In my code below, I forked my process into a parent and child process. In the child process, I sent the c string argv[1] to the parent process to be printed. Then I made the child process sleep for 4 seconds before printing "This is the child process. Closing\n".
In the parent process, I want the string from the child process to be printed to stdout as soon as I receive it from the child process.
The problem arises here. Instead of immediately printing argv[1] in the parent process before the string "This is the child process. Closing\n" is printed 4 seconds later, what happens is this:
$ g++ -std=c++11 printchild.cpp -o printchild
$ ./printchild helloworld
1) 4 seconds passes
2) "This is the child process. Closing\n" is printed
3) "helloworld" is printed
Why is the output from the parent process blocked by the child process?
// printchild.cpp
#include <chrono>
#include <thread>
#include <cstdio>
#include <unistd.h>
#include <cstdlib>
int main(int argc, char * argv[]){
int pipefd[2];
pid_t cpid;
if(argc != 2){
fprintf(stderr, "Usage: %s <string>\n", argv[0]);
exit(EXIT_FAILURE);
}
if(pipe(pipefd) == -1){
perror("fork");
exit(EXIT_FAILURE);
}
cpid = fork();
if(cpid == 0){
close(pipefd[0]);
FILE *fpc = fdopen(pipefd[1],"wb");
fprintf(fpc,"%s",argv[1]);
fflush(fpc);
std::this_thread::sleep_for(std::chrono::seconds(4));
fprintf(stdout,"This is the child process. Closing\n");
return 0;
}else if(cpid == -1){
perror("Error in forking");
exit(EXIT_FAILURE);
}else{
char str[80];
close(pipefd[1]);
FILE* fp = fdopen(pipefd[0], "rb");
fgets(str,80,fp);
fprintf(stdout,"%s\n",str);
return 0;
}
}
The parent process is reading the child's message via fgets(). It will continue to read until one of three things happens:
enough bytes have been read to fill the buffer, less one for a string terminator
a newline is read
end-of-file is encountered
The child does not send enough bytes to exhaust the buffer, and it does not send a newline, so the parent's fgets() does not return until the child's end of the pipe is closed upon its exit.
You can fix this in the child by having it either terminate the message with a newline or close the stream immediately after writing.
If you close the file in the client right after flushing, it will work as expected:
fflush(fpc);
fclose(fpc);
Output:
[dbush] /tmp/x1 hello
hello
[dbush] This is the child process. Closing
Your problem is the fgets call.
It tries to fulfill your request fully, and that's the cause of the blocking.
If you're prototyping some IPC, I recommend you ditch stdio and go for the raw IO syscalls.
If you substitute, e.g.:
char buf[100];
read(pipefd[0], buf, 100);
fprintf(stdout,"%s\n",buf);
for the fgets part, you'll get your message right away, because read will give you whatever's available and return (if there's at least 1 byte in the pipe buffer at the time you make the read request), whereas fgets won't back off until it has read all the requested bytes or until it figures out it can't (EOF or an IO error).
(To split hairs, it's not fgets that's blocking (userspace code alone can't block) -- it's another momentarily unsatisfiable read syscall that fgets has issued).
Before the below code I do:
create 1 pipe to read output from forked process
fork()
execv() a python script
Then in the parent process I do:
//set pipes to non-Blocking
File * cout_f = fdopen(cout_pipe[0], "r");
int flags = fcntl(cout_pipe[0], F_GETFL, 0);
fcntl(cout_pipe[0], F_SETFL, flags|O_NONBLOCK);
// read from pipe and send it up through a callback method
int stat;
size_t size = 0;
char buffer [ 1000 ];
do
{
while((size = fread(buffer, sizeof(char), sizeof(char)*1000, cout_f))!=0)
{
call_back_function(buffer, size);
}
}while(waitpid(child_pid, &stat, WNOHANG) != -1)
//Do 1 extra read
while((size = fread(buffer, sizeof(char), sizeof(char)*1000, cout_f))!=0)
{
call_back_function(buffer, size);
}
The problem I am facing happens when the child process prints to stdout and exits (normally) before flushing. I miss what was sent in the pipe.
Here are my questions:
Is the above code safe/correct or can it be improved?
Is there a way to read the entire pipe at the last moment when the subprocess dies, even if it doesn't flush its stdout?
You don't have to wait. Just send everything as you read it. That's what your code already does. Get rid of non-blocking mode; get rid of the while(wait(...)) condition; get rid of the final read; and just perform the first read loop until end of stream. Then call wait() to get the exit code.
If the process also produces on stderr you will need to read that in another thread, otherwise when the stderr buffer fills, it will block.
I use a function (L) to execute another program (K) through execlp().
In the K program, the result is written into stream 1:
write(1, (char *)&o, sizeof(int));
Since after execlp(), the remaining part of L won't be executed, how could I get the result written in stream 1?
Don't ask me why I need to do it this way. It's the requirement for a project.
I followed your guys advices, but now the problem is, the way the K program get the arguments are from streams(one standard stream, one other stream), I'm using pipes to write arguments into corresponding streams (it's done by the parent).
After the child exec, in the parent part, I read from stream 0 (the K program writes its result back in stream 1). But what I could get is what the parent wrote into the stream, not what the K program write back.
What's wrong? Do I need to add another pipe?
Thanks!!
The key insight, which Jonathan Leffler mentioned in his comment, is that you need to fork the program which is running L before you call execlp().
After the fork, the parent continues to execute the rest of L, and the child morphs into the program K by calling execlp(), which should never return unless there is an error.
Thus, the assertion that "the remaining part of L won't be executed" is incorrect. It will get executed, in the parent process if you write the function L correctly.
Update: Since the OP made his question more specific, I am appending to this answer.
If you want to retrieve what the child process wrote to stdout (fd 1), you need to create a new pipe before the fork, and copy the writing end of this pipe into the child's stdout.
Here is an example program, slightly modified from the pipe man page.
#include <sys/types.h>
#include <sys/wait.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
int
main(int argc, char *argv[])
{
int pipefd[2];
pid_t cpid;
char buf;
if (pipe(pipefd) == -1) {
perror("pipe");
exit(EXIT_FAILURE);
}
cpid = fork();
if (cpid == -1) {
perror("fork");
exit(EXIT_FAILURE);
}
if (cpid == 0) { /* Child writes into the pipe */
close(pipefd[0]); /* Close unused read end */
// Copy the writing end of the pipe into STDOUT for the child.
dup2(pipefd[1], STDOUT_FILENO);
// Execute your program L here, and its stdout will be captured by the pipe.
const char* message = "Child is speaking to stdout!";
write(STDOUT_FILENO, message, strlen(message));
write(STDOUT_FILENO, "\n", 1);
close(pipefd[1]);
_exit(EXIT_SUCCESS);
} else { /* Parent reads child's stdout from the pipe */
close(pipefd[1]); /* Close unused write end */
// Here the parent process is reading the child's stdout.
while (read(pipefd[0], &buf, 1) > 0)
write(STDOUT_FILENO, &buf, 1);
wait(NULL); /* Wait for child */
exit(EXIT_SUCCESS);
}
}