When calling WriteFile can the returned lpNumberOfBytesWritten ever be less than nNumberOfBytesToWrite in synchronous write mode where an error has not occured (return is TRUE)? Writing is done to a proper file (not a socket, pipe, or other special handle). lpOverlapped is not used.
It is not entirely clear from the docs.
The documentation states:
The WriteFile function returns when
one of the following conditions occur:
The number of bytes requested is written.
A read operation releases buffer space on the read end of the pipe (if
the write was blocked). For more
information, see the Pipes section.
An asynchronous handle is being used and the write is occurring
asynchronously.
An error occurs.
Only the first one of these meets your criteria so the answer is that the documentation is clear *lpNumberOfBytesWritten is always equal to nNumberOfBytesToWrite when the function returns when called as you specify.
Related
I program in C++ and found that pread works very interestingly.
pread() returns the number of bytes read. The returned number of bytes read can be different from the specified number of bytes intended to be read.
Why pread does not guarantee that it reads all the specified bytes?
Where does this limitation come from?
Why pread does not guarantee that it reads all the specified bytes?
Because it is designed like that.
As it's mentioned here:
Note that is not an error for a successful call to transfer fewer
bytes than requested (see read(2) and write(2)).
So you simply going to call that function again in such case.
this may happen for example because fewer bytes are actually available
right now (maybe because we were close to end-of-file, or because we
are reading from a pipe, or from a terminal), or because read() was
interrupted by a signal. On error, -1 is returned, and errno is set
appropriately. In this case it is left unspecified whether the file
position (if any) changes.
from https://linux.die.net/man/2/read
I am writing a process on Linux with two threads. They communicate using an anonymous pipe, created with the pipe() call.
One end is copying a C structure into the pipe:
struct EventStruct e;
[...]
ssize_t n = write(pipefd[1], &e, sizeof(e));
The other end reads it from the pipe:
struct EventStruct e;
ssize_t n = read(pipefd[0], &e, sizeof(e));
if(n != -1 && n != 0 && n < sizeof(e))
{
// Is a partial read possible here??
}
Can partial reads occur with the anonymous pipe?
The man page (man 7 pipe) stipulates that any write under PIPE_BUF size is atomic. But what they mean is atomic regarding other writers threads... I am not concerned with multiple writers issues. I have only one writer thread, and only one reader thread.
As a side note, my structure is 56 bytes long. Well below the PIPE_BUF size, which is at least 4096 bytes on Linux. It looks like it's even higher on most recent kernel.
Told otherwise: on the reading end, do I have to deal with partial read and store them meanwhile I receive a full structure instance?
As long as you are dealing with fixed size units, there isn't a problem. If you write a unit of N bytes on the pipe and the reader requests a unit of N bytes from the pipe, then there will be no issue. If you can't read all the data in one fell swoop (you don't know the size until after you've read its length, for example), then life gets trickier. However, as shown, you should be fine.
That said, you should still detect short reads. There's a catastrophe pending if you get a short read but assume it is full length. However, you should not expect to detect short reads — code coverage will be a problem. I'd simply test n < (ssize_t)sizeof(e) and anything detected is an error or EOF. Note the cast; otherwise, the signed value will be converted to unsigned and -1 won't be spotted properly.
For specification, you'll need to read the POSIX specifications for:
read()
write()
pipe()
and possibly trace links from those pages. For example, for write(), the specification says:
Write requests to a pipe or FIFO shall be handled in the same way as a regular file with the following exceptions:
There is no file offset associated with a pipe, hence each write request shall append to the end of the pipe.
Write requests of {PIPE_BUF} bytes or less shall not be interleaved with data from other processes doing writes on the same pipe. Writes of greater than {PIPE_BUF} bytes may have data interleaved, on arbitrary boundaries, with writes by other processes, whether or not the O_NONBLOCK flag of the file status flags is set.
Or from the specification of read():
Upon successful completion, where nbyte is greater than 0, read() shall mark for update the last data access timestamp of the file, and shall return the number of bytes read. This number shall never be greater than nbyte. The value returned may be less than nbyte if the number of bytes left in the file is less than nbyte, if the read() request was interrupted by a signal, or if the file is a pipe or FIFO or special file and has fewer than nbyte bytes immediately available for reading. For example, a read() from a file associated with a terminal may return one typed line of data.
So, the write() will write atomic units; the read() will only read atomic units because that's what was written. There won't be a problem, which is what I said at the start.
From Linux documentation, POLLOUT means Normal data may be written without blocking. Well, but this explanation is ambigous.
How much data is it possible to write without blocking after poll reported this event? 1 byte? 2 bytes? Gigabyte?
After POLLOUT event on blocking socket, how to check how much data I can send to socket without block?
poll system call only tells you that there is something happen in the file descriptor(physical device) but it doesn't tell you how much space is available for you to read or write. In order to know exactly how many bytes data is available to be used for reading or writing, you must use read() or write() system call to get the return value which says the number of bytes you have actually been read or written.
Thus,poll() is mainly used for applications that must use multiple input or output streams without getting stuck on any one of them. You can't use write() or read() in this case since you can't monitor multiple descriptors at the same time within one thread.
BTW,for device driver,the underlying implementation for POLL in driver usually likes this(code from ldd3):
static unsigned int scull_p_poll(struct file *filp, poll_table *wait)
{
poll_wait(filp, &dev->inq, wait);
poll_wait(filp, &dev->outq, wait);
...........
if (spacefree(dev))
mask |= POLLOUT | POLLWRNORM; /* writable */
up(&dev->sem);
return mask;
}
If poll() sets the POLLOUT flag then at least one byte may be written without blocking. You may then find that a write() operation performs only a partial write, so indicated by returning a short count. You must always be prepared for partial reads and writes when multiplexing I/O via poll() and/or select().
I have a C++11 program that uses std::fprintf to write to stderr as a log and debug info. I know fprintf can fail and return a negative value, but I can't found if the operation is atomic (if fails, has no effect) or it can write some part of the text and then fails (or any other side effect).
The function that uses fprintf looks like this:
void writeToConsole (std::string const &message)
{
std::fprintf(stderr, "%s\n", message.c_str());
}
I am developing using Clang and GCC on Linux (for now), but my question is more about the standard, so...
Question:
If std::fprintf fails, is still possible that some characters had been written to stderr? Is this behaviour a C/C++ standard or is implementation-defined?
Even more, if std::fprintf fails, should I abort the program or can continue execution silently without side effects (other than impossibility of write to stderr)?
Keep in mind that the printf family of functions (almost always) eventually turns into a write(2) function call (or other such low-level, OS/implementation-provided equivalent). This function can be partially successful. If at least one byte is written, the function succeeds (if no error from the underlying destination can be detected - for example, interruption by a signal handler) and it will return the number of bytes actually written:
The number of bytes written may be less than count if, for example, there is insufficient space on the underlying physical medium, or the RLIMIT_FSIZE resource limit is encountered (see setrlimit(2)), or the call was interrupted by a signal handler after having written less than count bytes. [...]
For information about partial writes, see write(2), for example at http://man7.org/linux/man-pages/man2/write.2.html. Values for errno or other effects of a partial write may be dependent on the output medium (or what the file descriptor represents - memory mapped file, regular file, etc.), as well as the specific reason for failure. For example, ENOMEM, EIO, EDQUOT are all possibilities.
Also see the linked man page for additional information about atomicity with regard to multiple threads.
Your other question:
Even more, if std::printf fails, should I abort the program or can continue execution silently without side effects (other than impossibility of write to stderr)?
This really depends on your program.
For fprintf the C++11 standard falls back to C99 since it is part of the C standard library and the C99 draft standard says the following:
The fprintf function returns the number of characters transmitted, or
a negative value if an output or encoding error occurred.
but does not actually specify whether an error means no character were transmitted or not, so that will end up dependent on the implementation.
For POSIX compliant systems, which in this case should cover Linux, the reference for fprintf says:
Upon successful completion, the fprintf() and printf() functions shall return the number of bytes transmitted.
[...]
If an output error was encountered, these functions shall return a negative value.
There are several errors listed that could lead to partial output such as:
[ENOMEM]
Insufficient storage space is available.
Whether an error indicates that you should exit your application depends on your application, do you have an alternate logging mechanism other than stderr? Does your application have legal requirements that mandate everything is logged or are the logs purely informational, etc...
I need to put the standard output of a process (binary data) to a string buffer and consume it in another thread.
Here is the producer:
while (ReadFile(ffmpeg_OUT_Rd, cbBuffer, sizeof(cbBuffer), &byteRead, NULL)){
tByte += byteRead; //total bytes
sb->sputn(cbBuffer, byteRead);
}
m_bIsFinished = true;
printf("%d bytes are generated.\n", tByte);
Here is the consumer:
while (!PCS_DISPATCHER_INSTANCE->IsFinished())
Sleep(200);
Sleep(5000);
Mystringbuf* sb = PCS_DISPATCHER_INSTANCE->sb;
printf("Avail: %d\n", sb->in_avail());
It turns out that the consumer cannot get all the bytes of the produced by the producer.
( tByte <> sb->in_avail() )
Is it a kind of internal buffering problem? If yes, how to force the stringbuf to flush its internal buffer?
A streambufhas nothing like flush: writes are done directly into the buffer. There is a pubsync() member that could help, if you would use an object derived such as a filebuf. But this does not apply to your case.
Your issue certainly comes from a a data race on sputn() or is_avail(). Either protect access to the streambuf via a mutex, or via an atomic. If m_bIsFinished is not an atomic, and depending on your implementation of isFinished(), the synchronisation between the threads might not be guaranteed (for example: producer could write to memory, but consumer still obtains an outdated value from the CPU memory cache), which could conduct to such a data race.
Edit:
If you'd experience the issue within a single thread, thus eliminating any potential racing condition, it may come from implementation of streambuf. I could experience this with MSVC13 in a single thread application:
tracing showed that number of bytes read were accurate, but in_avail() result was always smaller or equal to tByte through the whole loop.
when reading the streambuf, the correct total number of bytes were read (thus more than indicated by in_avail()).
This behaviour is compliant. According to C++ standard: in_avail() shall return egptr() - gptr() if a read position is available, and otherwhise showmanyc(). The latter is defined as returning an estimate of the number of characters available in the sequence. The only guarantee given is that you could read at least in_avail() bytes without encountering eof.
Workaround use sb->pubseekoff(0,ios::end)- sb->pubseekoff(0,ios::beg); to count the number of bytes available, and make sure you're repositioned at sb->pubseekoff(0,ios::beg) beofre you read your streambuf.