I am programming a shell in c++. It needs to be able to pipe the output from one thing to another. For example, in linux, you can pipe a textfile to more by doing cat textfile | more.
My function to pipe one thing to another is declared like this:
void pipeinput(string input, string output);
I send "cat textfile" as the input, and "more" as the output.
In c++ examples that show how to make pipes, fopen() is used. What do I send as my input to fopen()? I have seen c++ examples of pipeing using dup2 and without suing dup2. What's dup2 used for? How do you know if you need to use it or not?
Take a look at popen(3), which is a way to avoid execvp.
For a simple, two-command pipeline, the function interface you propose may be sufficient. For the general case of an N-stage pipeline, I don't think it is flexible enough.
The pipe() system call is used to create a pipe. In context, you will be creating one pipe before forking. One of the two processes will arrange for the write end of the pipe to become its standard output (probably using dup2()), and will then close both of the file descriptors originally returned by pipe(). It will then execute the command that writes to the pipe (cat textfile in your example). The other process will arrange for the read enc of the pipe to become its standard input (probably using dup2() again), and will then close both of the file descriptor originally returned by pipe(). It will then execute the command that reads from the pipe (more in your example).
Of course, there will be still a third process around - the parent shell process - which forked off a child to run the entire pipeline. You might decide you want to refine the mechanisms a bit if you want to track the statuses of each process in the pipeline; the process organization is then a bit different.
fopen() is not used to create pipes. It can be used to open the file descriptor, but it is not necessary to do so.
Pipes are created with the pipe(2) call, before forking off the process. The subprocess has a little bit of file descriptor management to do before execing the command. See the example in pipe's documentation.
Related
I need to learn how to create a pipe and use fork, and also how to write to a pipe and read, in VC++ 2010.
Are there any tutorials on how to do that?
This question is already answered in detail here.
Quoting verbatim from the same answer
A pipe is a mechanism for interprocess communication. Data written to the pipe by one process can be read by another process. The primitive for creating a pipe is the pipe function. This creates both the reading and writing ends of the pipe. It is not very useful for a single process to use a pipe to talk to itself. In typical use, a process creates a pipe just before it forks one or more child processes. The pipe is then used for communication either between the parent or child processes, or between two sibling processes. A familiar example of this kind of communication can be seen in all operating system shells. When you type a command at the shell, it will spawn the executable represented by that command with a call to fork. A pipe is opened to the new child process and its output is read and printed by the shell. This page has a full example of the fork and pipe functions...
Though I'm reasonably used to UNIX and have programmed on it for a long time, I'm not used to file manipulation.
I know that 0/1/2 file descriptors are standard in, out, and error. I'm aware that whenever a process opens a file, it is given a descriptor with the smallest value that isn't yet used - and I understand some things about using dup/dup2.
I get confused about file descriptors between processes though. Does each process have its own 0/1/2 descriptors for in/out/error or are those 3 descriptors shared between all processes? How come you can run 3 programs in 3 different shells and they all get only their programs output if they are shared?
If two programs open myfile.txt after start-up, will they both use file descriptor #3, or would the second program use #4 since 3 was taken?
I know I asked the same question in a couple ways there, but I just wanted to be clear. The more detail the better :) I've never run into problems with these things while programming, but I'm reading through a UNIX book to understand more and I suddenly realized this confused me a lot and I'd never though about it in detail before.
Each file descriptor is local to the process. However, some file descriptors can refer to the same file - for example, if you create a child process using fork() it would share the files opened by the parent. It would have its own set of file descriptors, initially identical to the parent's ones, but they can change with closing/dup-ing, etc.
If two programs open the same file, in general they get separate file descriptors, pointing to separate internal structures. However, using certain techniques (fork, FD passing, etc.) you can have file descriptors in different processes point to the same internal entity. Generally, though, it is not the case.
Answering your question, both programs would have FD #3 for newly open file.
File descriptors in Unix (normally) persist through fork() and exec() calls. So yes, several processes can share file descriptors.
For example, a shell might do a command like:
foo | bar
In this case, foo's stdout must be connected to bar's stdin. To do this, the shell will most likely use pipe() to create reader- and writer file descriptors. It fork()s twice. The descriptors persist. The fork() which will call up foo, will close(1); dup(writer_fd); to make writer_fd descriptor 1. It will then exec(), and process foo will output to the pipe we created. For bar, we close(0); dup(reader); then exec(). And voila, foo will output to bar.
Don't confuse the file descriptors with the resources they represent. You can have ten different processes, each with a file descriptor of '3' open, and each refer to a different open file. When a process does I/O using its file descriptor, the OS knows which process is doing the I/O and is able to disambiguate which file is being referred to.
I'm looking at the code for a c++ program which pipes the contents of a file to more. I don't quite understand it, so I was wondering if someone could write pseudocode for a c++ program that pipes something to something else? Why is it necessary to use fork?
create pipe
fork process
if child:
connect pipe to stdin
exec more
write to pipe
You need fork() so that you can replace stdin of the child before calling, and so that you don't wait for the process before continuing.
You will find your answer precisely here
Why is it necessary to use fork?
When you run a pipeline from the shell, eg.
$ ls | more
what happens? The shell runs two processes (one for ls, one for more). Additionally, the output (STDOUT) of ls is connected to the input (STDIN) of more, by a pipe.
Note that ls and more don't need to know anything about pipes, they just write to (and read from) their STDOUT (and STDIN) respectively. Further, because they're likely to do normal blocking reads and writes, it's essential that they can run concurrently. Otherwise ls could just fill the pipe buffer and block forever before more gets a chance to consume anything.
... pipes something to something else ...
Note also that aside from the concurrency argument, if your something else is another program (like more), it must run in another process. You create this process using fork. If you just run more in the current process (using exec), it would replace your program.
In general, you can use a pipe without fork, but you'll just be communicating within your own process. This means you're either doing non-blocking operations (perhaps in a synchronous co-routine setup), or using multiple threads.
I have a program that creates pipes between two processes. One process constantly monitors the output of the other and when specific output is encountered it gives input through the other pipe with the write() function. The problem I am having, though is that the contents of the pipe don't go through to the other process's stdin stream until I close() the pipe. I want this program to infinitely loop and react every time it encounters the output it is looking for. Is there any way to send the input to the other process without closing the pipe?
I have searched a bit and found that named pipes can be reopened after closing them, but I wanted to find out if there was another option since I have already written the code to use unnamed pipes and I haven't yet learned to use named pipes.
Take a look at using fflush.
How are you reading the other end? Are you expecting complete strings? You aren't sending terminating NULs in the snippet you posted. Perhaps sending strlen(string)+1 bytes will fix it. Without seeing the code it's hard to tell.
Use fsync. http://pubs.opengroup.org/onlinepubs/007908799/xsh/fsync.html
From http://www.delorie.com/gnu/docs/glibc/libc_239.html:
Once write returns, the data is enqueued to be written and can be read back right away, but it is not necessarily written out to permanent storage immediately. You can use fsync when you need to be sure your data has been permanently stored before continuing. (It is more efficient for the system to batch up consecutive writes and do them all at once when convenient. Normally they will always be written to disk within a minute or less.) Modern systems provide another function fdatasync which guarantees integrity only for the file data and is therefore faster. You can use the O_FSYNC open mode to make write always store the data to disk before returning.
I am wrapping existing C++ code from a BSD project in our own custom wrapper and I want to integrate it to our code with as few changes as possible. This code uses fprintf to print to stderr in order to log / report errors.
I want to redirect this to an alternative place within the same process. On Unix I have done this with a socketpair and a thread: one end of the socket is where I send stderr (via a call to dup2) and the other end is monitored in a thread, where I can then process the output.
This does not work on Windows though because a socket is not the same as a file handle.
All documents I have found on the web show how to redirect output from a child process, which is not what I want. How can I redirect stderr within the same process getting a callback of some sort when output is written? (and before you say so, I've tried SetStdHandle but cannot find any way to make this work)...
You can use a similar technique on Windows, you just need to use different words for the same concepts. :) This article: http://msdn.microsoft.com/en-us/library/ms682499.aspx uses a win32 pipe to handle I/O from another process, you just have to do the same thing with threads within the same process. Of course, in your case all output to stderr from anywhere in the process will be redirected to your consumer.
Actually, other pieces of the puzzle you may need are _fdopen and _open_osfhandle. In fact, here's a related example from some code I released years ago:
DWORD CALLBACK DoDebugThread(void *)
{
AllocConsole();
SetConsoleTitle("Copilot Debugger");
// The following is a really disgusting hack to make stdin and stdout attach
// to the newly created console using the MSVC++ libraries. I hope other
// operating systems don't need this kind of kludge.. :)
stdout->_file = _open_osfhandle((long)GetStdHandle(STD_OUTPUT_HANDLE), _O_TEXT);
stdin->_file = _open_osfhandle((long)GetStdHandle(STD_INPUT_HANDLE), _O_TEXT);
debug();
stdout->_file = -1;
stdin->_file = -1;
FreeConsole();
CPU_run();
return 0;
}
In this case, the main process was a GUI process which doesn't start with stdio handles at all. It opens a console, then shoves the right handles into stdout and stdin so the debug() function (which was designed as a stdio interactive function) can interact with the newly created console. You should be able to open some pipes and do the same sort of thing to redirect stderr.
You have to remember that what MSVCRT calls "OS handles" are not Win32 handles, but another layer of handles added just to confuse you. MSVCRT tries to emulate the Unix handle numbers where stdin = 0, stdout = 1, stderr = 2 and so on. Win32 handles are numbered differently and their values always happen to be a multiple of 4. Opening the pipe and getting all the handles configured properly will require getting your hands messy. Using the MSVCRT source code and a debugger is probably a requirement.
You mention that you don't want to use a named pipe for internal use; it's probably worth poining out that the documentation for CreatePipe() states, "Anonymous pipes are implemented using a named pipe with a unique name. Therefore, you can often pass a handle to an anonymous pipe to a function that requires a handle to a named pipe." So, I suggest that you just write a function that creates a similar pipe with the correct settings for async reading. I tend to use a GUID as a string (generated using CoCreateGUID() and StringFromIID()) to give me a unique name and then create the server and client ends of the named pipe with the correct settings for overlapped I/O (more details on this, and code, here: http://www.lenholgate.com/blog/2008/02/process-management-using-jobs-on-windows.html).
Once I have that I wire up some code that I have to read a file using overlapped I/O with an I/O Completion Port and, well, then I just get async notifications of the data as it arrives... However, I've got a fair amount of well tested library code in there that makes it all happen...
It's probably possible to set up the named pipe and then just do an overlapped read with an event in your OVERLAPPED structure and check the event to see if data was available... I don't have any code available that does that though.