Yes, I can't. It seems weird ostream has no close, since istream can detect end of file.
Here's my situation: I am capturing all the output on Posix fd2, in this process, and its children, by creating a pipe and dup2'ing the pipe output end onto fd2. A thread then reads the read end of the pipe using an associated C stream (and happens to write each line with a timestamp to the original fd2 via another associated C stream).
When all the children are dead, I write a closing message to cerr, then I need to close it so the thread echoing it to the original error file will close the pipe and terminate.
The thread is not detecting eof(), even though I am closing both stderr and fd2.
I have duplicated my main program using a simple one, and using C streams instead of C++ iostreams, and everything works just fine by fclosing stderr (there are no child processes in that simplified test though).
Edit: hmm .. do I need to close the original pipe fd after dup2'ing it onto channel 2? I didn't do that, so the underlying pipe still has an open fd attached. Aha .. that's the answer!
When you duplicate a file descriptor with dup2 the original descriptor remains a valid reference to the underlying file. The file won't be closed and the associated resources freed until all file descriptors associated with a particular file are closed (with close).
If you are using dup2 to copy a file descriptor to a well known number (such as 2 for stderr), you usually want to call close on the original file descriptor immediately after a successful dup2.
The streams used for the standard C++ streams are the same as those controlled by the corresponding stdio files. That is, if you fclose(stderr) you also close the stream used for std::cerr. ... and since you seem to play with the various dup() functions you can also close(2) to close this stream.
The best is to put a wrapper around your resource and then have the destructor close it when it goes out of scope. The the presention from Bjarne Stoustup
Related
Background: I'm working in C (and very C-ish C++) on Linux. The parent process has an open file descriptor (edit: not file descriptor, actually a FILE pointer) that it writes data to in a "sectioned" format. The child process uses it for this same purpose. As long as the child process is running, it is guaranteed that the parent will not attempt to write more data to its copy of the FILE pointer. The child exits, the parent waits for it, and then it writes more data to the file.
It appears to be working correctly, but I'm still suspicious of it. Do I need to re-seek to the end in the parent? Are there any synchronization issues I need to handle?
The question changed from 'file descriptors' to 'file pointers' or 'file streams'. That complicates any answer.
File Descriptors
File descriptors come from Unix in general, and Linux in particular, and a lot of the behaviour is standardized by POSIX. You say that the parent and child processes share the same file descriptor. That's not possible; file descriptors are specific to one process.
Suppose the parent opens a file (assume the file descriptor is 3) and therefore there is also a new open file description; then the parent process forks. After the fork, each process has a separate file descriptor (but they're both using file descriptor 3), but they share the same open file description. Yes: 'open file descriptors' and 'open file descriptions' are different! Every open file descriptor has an open file description, but a single open file description can be associated with many open file descriptors, and those descriptors need not all be associated with the same process.
One of the critical bits of data in the open file description is the current position (for reading or writing — writing is what matters here).
Consequently, when the child writes, the current position moves for both the parent and the child. Therefore, whenever the parent does write, it writes after the location where the child finished writing.
What you are seeing is guaranteed. At least under the circumstances where the parent opens the file and forks.
Note that in the scenario discussed, the O_APPEND flag was not needed or relevant. However, if you are worried about it, you could open the file with the O_APPEND flag, and then each normal write (not written via pwrite()) will write the data at the current end of the file. This will work even if the two processes do not share the same open file description.
POSIX specification:
open()
fork()
write()
dup2()
pwrite()
File Streams
File streams come with buffering which makes their behaviour more complex than file descriptors (which have no buffering).
Suppose the scenario is like this pseudo-code (error handling omitted):
FILE *fp = fopen(filename, "w");
…code block 1…
pid_t pid = fork();
if (pid == 0)
{
…child writes to file…
…child exits…
}
else
{
…parent waits for child to exit…
…parent writes to file…
}
The de facto implementation of a file stream uses a file descriptor (and you can find the file descriptor using fileno(fp)). When the fork() occurs, it is important that there is no pending data in the file stream — use fflush(fp) before the fork() if any of the code in '…code block 1…' has written to the file stream.
After the fork(), the two processes share the same open file description, but they have independent file descriptors. Of necessity, they have identical copies of the file stream structure.
When the child writes to its copy of the file stream, the data is stored in its buffer. When the buffer fills, or when the child closes the stream (possibly by exiting in a coordinated manner, not using _exit() or its relatives or as a result of a signal), the child's file data is written to the file. That process will move the current position in the shared open file description.
When the parent is notified that the child has exited, then it can write to its file buffer. That information will be written to disk when the buffer fills or is flushed, or when the parent closes the file stream. Since it will be using the same open file description as the child was using, the write position will be where the child left it.
So, as before, what you're seeing is guaranteed as long as you are careful enough.
In particular, calling fflush(fp) before the fork() is crucial if the file stream has been used by the parent before the fork(). If you don't ensure that the stream is flushed, you will get unflushed data written twice, once by the child and once by the parent.
It is also crucial that the child exits cleanly — closing the file stream and hence flushing any unwritten data to the file. If the child does not exit cleanly, there may be buffered data that never gets written to the file. Similarly, if the parent doesn't exit cleanly, there may be buffered data from the parent that never gets written to the file.
If you are talking about POSIX file descriptors, then each write call to a file descriptor is atomic and affects the underlying kernel resource object indepently of what other processes might do with file descriptors that refer to the same object. If two processes do write at approximately the same time, the operations will get ordered by the kernel with one happening completely (though it might write less data than requested) and then the other happening.
In your case, it sounds like you are synchronizing such that you know all parent writes happen either before the child has started (before fork) or after it has completed (after wait), which guarentees the ordering of the write calls.
Are standard input and standard output independent or not?
Consider a parent program had launched a child, and the parent's standard output was attached to the child's standard input, and the child's standard output was attached to the parent's standard input.
stdin <- stdout
parent child
stdout -> stdin
If the child (asynchronously) continually read from its standard input and wrote data to its standard output, but the parent just wrote to the child's standard input and didn't read from the child's standard output at all:
stdin| << stdout
parent child
stdout ==>==> stdin
would there eventually be a blockage? Do standard input and standard output share a buffer of any kind? Specifically via C++ std::cin (istream) and std::cout (ostream) if that's needed to answer. Does the standard require they do or do not share such a thing, or does it leave it up to the implementation?
What would happen?
You can't "attach" a file descriptor from a process to a file descriptor of a different process. What you do (if your operating system supports it) is to assign the two file descriptors to the ends of a "pipe". Pipes are not specified anywhere in the C/C++ standard (they are defined by POSIX), and you won't find any standard C/C++ library function which makes any reference to them at all.
As implemented by Unix (and Unix-like) systems, a pipe is little more than a buffer somewhere in the operating system. While the buffer is not full, a process can write data to the input end of the pipe; the data is simply added to the buffer. While the buffer is not empty, a process can read data from the output end of the buffer; the data is removed from the buffer and handed off to the reading process. If a process tries to write to a pipe whose buffer is full or read from a pipe whose buffer is empty, the process "blocks": that is, it is marked by the kernel scheduler as not runnable, and it stays in that state until the pipe can handle its request.
The scenario described in the question needs to involve two pipes. One pipe is used to allow the parent's stdout to send data to the child's stdin, and the other is used to allow the child's stdout to send data to the parent's stdin. These two pipes are wholly independent of each other.
Now, if the parent stops reading from its stdin, but the child continues writing to its stdout, then eventually the pipe buffer will become full. (It actually won't take very long. Pipe buffers are not very big, and they don't grow.) At that point, the child will block trying to write to the pipe. If the child is not multithreaded, then once it blocks, that's it. It stops running, so it won't read from its stdin any more. And if the child stops reading from its stdin, then the other pipe will soon become full and the parent will also block trying to write to its stdout.
So there's no requirement that resources be shared in order to achieve deadlock.
This is a very well-known bug in processes which spawn a child and try to feed data to the child while reading the child's response. If the reader does not keep up with the data produced, then deadlock is likely. You'll find lots of information about it by searching for, for example, "pipe buffer deadlock". Here are a few sample links, just at random:
Raymond Chen, on MSDN: http://blogs.msdn.com/b/oldnewthing/archive/2011/07/07/10183884.aspx
Right here on StackOverflow (with reference to Python but the issue is identical): Can someone explain pipe buffer deadlock?
David Glasser, from 2006: http://web.mit.edu/6.033/2006/wwwdocs/writing-samples/unix-DavidGlasser.html ("These limitations are not merely theoretical — they can be seen in practice by the fact that no major form of inter-process communication later developed in Unix is layered on top of pipe.")
I am programming a shell in c++. It needs to be able to pipe the output from one thing to another. For example, in linux, you can pipe a textfile to more by doing cat textfile | more.
My function to pipe one thing to another is declared like this:
void pipeinput(string input, string output);
I send "cat textfile" as the input, and "more" as the output.
In c++ examples that show how to make pipes, fopen() is used. What do I send as my input to fopen()? I have seen c++ examples of pipeing using dup2 and without suing dup2. What's dup2 used for? How do you know if you need to use it or not?
Take a look at popen(3), which is a way to avoid execvp.
For a simple, two-command pipeline, the function interface you propose may be sufficient. For the general case of an N-stage pipeline, I don't think it is flexible enough.
The pipe() system call is used to create a pipe. In context, you will be creating one pipe before forking. One of the two processes will arrange for the write end of the pipe to become its standard output (probably using dup2()), and will then close both of the file descriptors originally returned by pipe(). It will then execute the command that writes to the pipe (cat textfile in your example). The other process will arrange for the read enc of the pipe to become its standard input (probably using dup2() again), and will then close both of the file descriptor originally returned by pipe(). It will then execute the command that reads from the pipe (more in your example).
Of course, there will be still a third process around - the parent shell process - which forked off a child to run the entire pipeline. You might decide you want to refine the mechanisms a bit if you want to track the statuses of each process in the pipeline; the process organization is then a bit different.
fopen() is not used to create pipes. It can be used to open the file descriptor, but it is not necessary to do so.
Pipes are created with the pipe(2) call, before forking off the process. The subprocess has a little bit of file descriptor management to do before execing the command. See the example in pipe's documentation.
After redirecting standard input and output to some files(for example) using freopen how can I redirect them back to where they were at the very beginning?
You can duplicate/clone the initial FD's using id = fcntl(blah,F_DUPFD,0), and then use dup2(0,id) to copy it back (after closing stdin!), then reopen the file using fdopen(). Repeat for the others. However, that probably doesn't get you what you want exactly - it gets you a random filehandle that is associated with FD 0, not changing 'stdin'.
Another kind-of ugly option is to spawn off a thread (which implicitly dups stdin/etc), fdreopen them, do your processing, then exit the thread (closing the reopened stdin/etc) and unblock the main process (which never was changed). This is indirect but probably both portable and guaranteed to work.
You can not, since underlying file descriptors was closed. But on windows you can try to open special file "CON", which corresponds to console.
I have a program that creates pipes between two processes. One process constantly monitors the output of the other and when specific output is encountered it gives input through the other pipe with the write() function. The problem I am having, though is that the contents of the pipe don't go through to the other process's stdin stream until I close() the pipe. I want this program to infinitely loop and react every time it encounters the output it is looking for. Is there any way to send the input to the other process without closing the pipe?
I have searched a bit and found that named pipes can be reopened after closing them, but I wanted to find out if there was another option since I have already written the code to use unnamed pipes and I haven't yet learned to use named pipes.
Take a look at using fflush.
How are you reading the other end? Are you expecting complete strings? You aren't sending terminating NULs in the snippet you posted. Perhaps sending strlen(string)+1 bytes will fix it. Without seeing the code it's hard to tell.
Use fsync. http://pubs.opengroup.org/onlinepubs/007908799/xsh/fsync.html
From http://www.delorie.com/gnu/docs/glibc/libc_239.html:
Once write returns, the data is enqueued to be written and can be read back right away, but it is not necessarily written out to permanent storage immediately. You can use fsync when you need to be sure your data has been permanently stored before continuing. (It is more efficient for the system to batch up consecutive writes and do them all at once when convenient. Normally they will always be written to disk within a minute or less.) Modern systems provide another function fdatasync which guarantees integrity only for the file data and is therefore faster. You can use the O_FSYNC open mode to make write always store the data to disk before returning.