I'm using SetFileCompletionNotificationModes() API to optimize my I/O completion ports loops, but it seems to not work properly.
Even if I set FILE_SKIP_COMPLETION_PORT_ON_SUCCESS for sockets and HANDLEs, an I/O port's completion callback is still called also if ReadFile() WriteFile() WSARecv() WSASend() do return synchronosly.
I'm sure that the 3 conditions that MSDN says must be true (A completion port is associated with the file handle, The file is opened for asynchronous I/O, A request returns success immediately without returning ERROR_PENDING) are met, and are all true, so why I still receive I/O completion calls?
When i call SetFileCompletionNotificationModes() it returns success, so no errors or whatsoever, and the system is Windows 7.
How I can replicate a scenario when after I have activated SetFileCompletionNotificationModes() on my socket/HANDLEs, I can clearly see that the I/O completion callback won't be called?
I guessed that it happens when I write few bytes on a socket, since the socket's buffer is quite bigger, I didn't full it, so another write of another few bytes shouldn't block, since there are still a lot of room in the buffer, so it should return immediately, but not with ERROR_IO_PENDING, just in a synchronous way, right? (more or less, in a similar way of unix EWOULDBLOCK/EAGAIN: when i call write() for few bytes it returns immediately, and doesn't return EAGAIN, because there is still space in the write buffer).
Well it doesn't do that. Even for writing multiple times few bytes on a socket, it still calls the I/O completion callback, avoding the benefits of setting FILE_SKIP_COMPLETION_PORT_ON_SUCCESS
Am I missing something important? Any clue about that?
NOTE: I know that this wont work if the socket is only compatible with Layered Service Providers (LSP) that return Installable File Systems (IFS) handles, but that's not my case, it should work.
Btw I'm trying this also with pipes and files.
Shouldn't files never call I/O completion callbacks because they never block, just like in unix read() and write() calls on local files never returns EWOULDBLOCK/EAGAIN, so ReadFile() and WriteFile() with an handle set with FILE_SKIP_COMPLETION_PORT_ON_SUCCESS should return immediately?
A network write completion is only generated when the data buffer that you have provided is no longer required by the network stack. It's somewhat hard to reason about when this will be and it's also somewhat irrelevant and not something to worry about. When you issue an overlapped write with FILE_SKIP_COMPLETION_PORT_ON_SUCCESS set then your write operation will return 0 if and only if the completion has occurred synchronously. Write code to handle that case correctly (it's the same code that you need for the recv side) and forget about it. You'll get the performance and context switching advantages when it's possible and your code with work correctly when it isn't.
As for file system access, that depends on the file system drivers and, likely the actual hardware. See here for some information about how hard it can be to get some hardware to do async file writes at all, ever. Then note that when I switched the test that I talk about there from a workstation with 'normal' SATA disks to a server with hardware raid everything was different and all of the writes were always fully async...
Make sure you validate the 4th condition mentioned in the documentation:
When the FileHandle parameter is a socket, this mode is only compatible with Layered Service Providers (LSP) that return Installable File Systems (IFS) handles. To detect whether a non-IFS LSP is installed, use the WSAEnumProtocols function and examine the dwServiceFlag1 member in each returned WSAPROTOCOL_INFO structure. If the XP1_IFS_HANDLES (0x20000) bit is cleared then the specified LSP is not an IFS LSP. Vendors that have non-IFS LSPs are encouraged to migrate to the Windows Filtering Platform (WFP).
Also read this MSDN Support ticket:
SetFileCompletionNotificationModes API causes an I/O completion port not to work correctly if a non-IFS LSP is installed
Related
I am currently working on an application in VxWorks that talks to a device file using the read() function and uses a set of ioctl()to configure the communication port. It happens sometimes that the read() is stuck forever because it waits for a message, but never gets the message. A following ioctl(fd,FIOCANCEL,0) will then never be reached.
My question is how I can use select() in combination with the FIOCANCEL to properly disconnect such that it is possible to do a read() again later.
I would also like to know the implementation of FIOCANCEL if that is possible, such I can modify it.
select() on VxWorks is the same as select on any other POSIX platform (though I can't remember whether it modifies the timeout parameter like Linux does; POSIX say's either behaviour is fine).
Presumably you're talking about a serial port? Then that should be fine to include in a call to select(). You then won't need the ioctl(fd, FIOCANCEL, 0), because you'd only ever call read() if select() has indicated that there is indeed something to read. If select() times out, don't call read().
I downloaded the source code for a simple static web server from
http://www.ibm.com/developerworks/systems/library/es-nweb/sidefile1.html
However, I'm confused by line 130:
#ifdef LINUX
sleep(1); /* to allow socket to drain */
#endif
exit(1);
Since there is no close for the socket, does it mean I need to wait the client close the socket?
Regardless of the author's intent, it is needless and incorrect. exit() is sufficient. When close() is called on a TCP socket, or exit() is called to terminate the process, unless SO_LINGER socket option has been set to a non-default setting, the kernel will keep the socket(s) in a wait state and attempt to deliver any undelivered / buffered data. You can see this with netstat, and is the reason that fast restarting a TCP server that isn't written for fast restart will have a problem reopening the port quickly (there is a proper way to accomplish this too).
I disagree with a couple of things in the accepted answer.
close() and exit() should have the same effect on the socket, traditionally it has only been a matter of style whether to close sockets if you were about to exit.
It should have nothing to do with overflowing a TCP send buffer, since it happens after all the writes. Full write buffer will return an error immediately by the write() return code; sleeping at the end will be irrelevant to that.
sleep(1) should have no effect on the socket buffer or reliable data delivery. If anything, this code throttles the web server child processes after the writes, so really has no good effect, and could actually increase the potential of a denial of service attack.
I am describing default operation. The defaults can be changed via the many options.
For the "bible" on socket programming, see W. Richard Steven's UNIX Network Programming - Networking APIs: Sockets and XTI where he covers this in detail.
That looks like a bit of sloppy code to me.
If a process with an open socket terminates, and the socket has some unwritten data, the kernel is going to tear down the socket without flushing out the unsent data.
When you write something to a socket, the written data will not necessarily be transmitted immediately. The kernel maintains a small buffer that collects the data being written to a socket. Or a pipe, too. It's more efficient to have the process go on, and then the kernel will take care of actually transmitting the written data, when it has time to do that.
A process can obviously write data to a socket much faster than it can be transmitted over a typical network interface, and the size of the internal socket buffer is limited, so if the process keeps writing data to the socket, at some point it will fill up the internal buffer, and will have to wait until the kernel actually transmits the data, and removes the written data from the internal buffer, before there's room to write more.
[*] I am omitting some technical details, such as that the data isn't considered written until the receiver ACKs it.
Anyway, the intent of that sleep() call appears to be to allow some time for the internal buffer to actually be transmitted, before the process terminates, because if it does before the actual data gets written, the kernel won't bother sending it and terminate the socket, as I just mentioned.
This is a bad example. This is not the right way to do these kinds of things. The socket should simply be close()d. This will correctly flush things out and make sure that everything goes where it's supposed to go. I can't see any valid reason why that example simply didn't properly close the socket, instead of engaging in this kind of hackery.
I'm working on my own FTP client in C++, but I'm stuck at function recv(). When I get data with recv(), they can be incomplete, because I'm using TCP protocol, so I have to use recv in loop. Problem is that when I call recv after everything that should be received was received server blocks, and my program is stuck.
I don't know how many bytes im going to recieve so I can't control it and stop it when its done. I found two not very elegant solutions right now:
is to use string.substr() (or TR1 regex) to find needed
expression and then stop calling recv before it blocks
second is to
set up timeval structure and then control socket through
setsockopt() function. Problem is long response time when i can get
incomplete corrupted data.
Question is, is there any clean and elegant solution for this?
The obvious thing to do is to transmit the length of the to-be-received message ahead (many protocols, including for example HTTP do that, to address the exact same issue). That way, you know that when you have received amount X, no more will come.
This will work fine 99.9% of the time and will catastrophically fail in the 0.1% of cases where the server is lying to you or where the server crashes unexpectedly or someone stumbles over the network cable (or something similar happens). Sadly, the "connection" established by TCP is an illusion, and you don't have much of a means to detect when the connection dies. The other end can go down, and you will not notice anything, unless you try to send and get an error (or until several hours later).
Therefore, you also need a backup strategy for when things don't go quite as good as expected. You might either use select or poll to know when data is available, so you don't block forever for a message that will never come.
Using threads to solve the block-at-end problem (as proposed in other answers) is not a very good option since blocking isn't the actual problem. The actual problem is that you don't know when you have reached the end of the transmission. Having a worker thread block at the end of the transmission will "work", but will leave the worker thread blocked indefinitely, consuming resources and with an uncertain, system-dependent fate.
You cannot join the thread before exiting, since it is blocked (so trying to join it would deadlock your main thread). When your process exits and the socket is closed, the thread will unblock, but will (at least on some operating systems, e.g. Windows) be terminated immediately after. This likely won't do much evil, but terminating a thread in an uncontrolled way is always less desirable than having it exit properly. On other operating systems, you may have a lingering thread remaining.
Since you are using C++, there are alternative libraries that greatly simplify network programming compared to stock C. My personal favourite is Boost::Asio, however others are available. These libraries not only save you the pain of coding in C, but also provide asynchronous capabilities to work around your blocking problem.
The typical approach is to use select()/pselect() or poll()/ppoll(). Both allow to specify a timeout in order to exit if there are no incoming data.
However I don't see how you should "call recv after everything that should be received". It would be extremely inefficient to rely on the timeout also when there are not network problems...
Or you send the size of data being sent, before the data, and that's what you read, or the data connection is terminated with an EOF. In this case read() will return -1 and you exit.
I can think of two options that will not require a major rewrite of your existing code and a third one which is more radical:
use non-blocking I/O and poll for data periodically. You can do other work while a message remains incomplete or no further data can be read from the socket.
use a separate worker thread to do the I/O. Even if it blocks on synchronous recv() calls, your main thread can continue to do work. The worker thread can transfer the data it receives to the main thread for processing once a complete message is received via TCP.
use an OS specific feature (I/O completion ports on Windows or aio on Linux), but these are far more complex and you should definitely consider Boost.Asio before going this route.
You can put the recv function in it's own thread and do the processing in another thread.
So I have a service method that uses io_service::poll() to process any currently-ready read/write events:
void Foo::service()
{
io.poll(); // boost::asio::io_service
if (socket.is_open())
if (socket.available())
notifyReadyToRead();
else
notifyReadyToWrite();
}
However, I'd also like to trigger behavior if the remote socket closes (gracefully or otherwise). I know that Winsock can handle this, as it raises a network event in this circumstance. Boost Asio sockets, however, have no built-in support for this as far as I can see.
I've tried reading and writing a zero-byte packet to the socket to test it for errors, but it appears that the implementation simply bails successfully if you pass it boost::asio::buffer(socket, 0) without actually checking the socket.
I've also heard that leaving a hanging read operation could work, but I'm not sure how to manage this while still allowing regular boost::asio::read and 'boost::asio::write` operations to work correctly.
Finally, adding a ping operation to the app-level protocol is a no-go; this class is intended to be used generically, and even if it wasn't I cannot change the remote side's protocol.
While I'd prefer a portable solution, a Windows-specific solution would be sufficient, even if it means grabbing the native handle and enumerating events that way. (This occurred to me, but WSAEnumNetworkEvents clears the internal record of events, so I suspect it would cause problems if the boost::asio implementation depended on reading it.)
In general case, you have to implement some application-level mechanism to detect all kinds of disconnection.
There were numerous discussions on boost/asio ML on this topic, here is on of them.
You can enable the socket option TCP_KEEPALIVE, but still need a pending read operation in order to receive a notification of the failed read.
I am writing an application needs to use large audio multi-samples, usually around 50 mb in size. One file contains approximately 80 individual short sound recordings, which can get played back by my application at any time. For this reason all the audio data gets loaded into memory for quick access.
However, when loading one of these files, it can take many seconds to put into memory because I need to read a large amount of data with ifstream, meaning my program GUI is temporarily frozen. I have tried memory mapping my file but this causes huge CPU spikes and a mess of audio every time I need to jump to a different area of the file, which is not acceptable.
So this has led me to think that performing an Asynchronous file read will solve my problem, that is the data gets read in a different process and calls a function on completion. This needs to be both compatible for Mac OS X and Windows and in C++.
EDIT: Don't want to use the Boost library because I want to keep a small code base.
boost has an asio library, which I've not used before (it's not on NASA's list of approved third-party libraries).
My own approach has been to write the file reading code twice, once for Windows, once for the POSIX aio API, and then just pick the right one to link with.
For Windows, use OVERLAPPED (you have to enable it in the CreateFile call, then pass an OVERLAPPED structure when you read). You can either have it set an event on completion (ReadFile) or call a completion callback (ReadFileEx). You'll probably need to change your main event loop to use MsgWaitForMultipleObjectsEx so you can either wait for the I/O events or allow callbacks to run, in addition to receiving WM_ window messages. MSDN has documentation for these functions.
For Linux, there's either fadvise and epoll, which will use the readahead cache, or aio_read which will allow actual async read requests. You'll get a signal when the request completes, which you should use to post an XWindows message and wake up your event processing loop.
Both are a little different in the details, but the net effect is the same -- you request a read which completes in the background, then your event dispatch loop gets woken up when the I/O finishes.
Boost.Asio library has limited implementation of asynchronous file I/O operations (only Windows wrapper for HANDLE) therefore it not suitable for you. See this question also.
You could easily implement your own asynchronous reading using standard streams and Boost.Thread library (or platform specific threads support).