I want to monitor the modifications in a specified directory, and retrieve the exact change information. So I've decided to use the ReadDirectoryChangesW() function. But I want to use it asynchronously, which means I don't want my worker thread to be blocked if there are no changes in the directory.
How can I do this?
I tend to use ReadDirectoryChangesW() with an I/O Completion Port, which is good for when you want to monitor multiple directories because you can have a single thread servicing the IOCP and dealing with the results from all of your calls to ReadDirectoryChangesW().
First create an IOCP, once, by calling CreateIoCompletionPort(). Start a thread that calls GetQueuedCompletionStatus() and deals with your async directory changes.
Now for each directory that you want to monitor. First, open your directory handle as normal. Next, associate it with your IOCP by calling CreateIoCompletionPort() and passing the IOCP handle in (see docs for more details). Finally, call ReadDirectoryChangesW() and pass an OVERLAPPED as detailed in the docs. The call will return, and if successful, the results will become available as the results of GetQueuedCompletionStatus() in the thread that you created to service the IOCP.
From the MSDN documentation for ReadDirectoryChnagesW():
For asynchronous completion, you can receive notification in one of
three ways:
Using the GetOverlappedResult function. To receive notification
through GetOverlappedResult, do not
specify a completion routine in the
lpCompletionRoutine parameter. Be sure
to set the hEvent member of the
OVERLAPPED structure to a unique
event.
Using the GetQueuedCompletionStatus function. To
receive notification through
GetQueuedCompletionStatus, do not
specify a completion routine in
lpCompletionRoutine. Associate the
directory handle hDirectory with a
completion port by calling the
CreateIoCompletionPort function.
Using a completion routine. To receive notification through a
completion routine, do not associate
the directory with a completion port.
Specify a completion routine in
lpCompletionRoutine. This routine is
called whenever the operation has been
completed or canceled while the thread
is in an alertable wait state. The
hEvent member of the OVERLAPPED
structure is not used by the system,
so you can use it yourself.
Related
In IOCP, when starting an IO operation such as WSARecv(), a completion packet will be sent to the completion port when the IO operation completes.
What I want to know is what IO operations causes completion packets to be sent to the completion port when using sockets, for example, I know that WSASend(), WSARecv(), AcceptEx(), and PostQueuedCompletionStatus() causes completion packets to be sent. Is there other IO operations that does that?
A completion will be queued to the IOCP associated with a socket only if an API call that can generate completions is called in a way that requests a completion to be queued. So you will know which API calls can generate completions by the fact that you've read the documentation and you're passing an OVERLAPPED structure to them.
Thus you don't really need to know the answer to your question as you will never get a completion that you do not expect to get as you have to have called an appropriate API with appropriate parameters for a completion to be generated.
You can then differentiate between the API that caused the completion to be generated by adding some form of identifying "per operation data" to the OVERLAPPED either by making an 'extended overlapped stucture' or by using the event handle as opaque data. Either way you get a chance to send some context from the API call site to the IOCP completion handling site. This context is of your own design and can tell you what initiated the completion.
Then you get to use the return value from the GetQueuedCompletionStatus() call to determine if the completion is a success or failure and you can then access the error code for failures using WSAGetLastError() (though see this answer for more detail on an additional hoop that you could jump through to get more accurate error codes).
This then lets you determine which of the events listed in EJP's answer you have.
The actual set of functions that can generate a completion for socket operations can change with changes in the OS. The easiest way to determine what these are for the operating system that you're targeting is to either read the MSDN docs or do a search of the SDK headers for lpOverlapped... As you'll see from the current VS2013 headers there are quite a few that relate to sockets; AcceptEx(), ConnectEx(), DisconnectEx(), TransmitFile(), the HTTP.sys API, the RIO API, etc.
You're missing the point. What causes completion packets to be sent is events, not API calls. There are basically only a few TCP events:
inbound connection
outbound connection complete
data
write finished
timeout
end of stream, and
error.
Copied from the site
Supported I/O Functions
The following functions can be used to start I/O operations that complete by using I/O completion ports. You must pass the function an instance of the OVERLAPPED structure and a file handle previously associated with an I/O completion port (by a call to CreateIoCompletionPort) to enable the I/O
completion port mechanism:
ConnectNamedPipe
DeviceIoControl
LockFileEx
ReadDirectoryChangesW
ReadFile
TransactNamedPipe
WaitCommEvent
WriteFile
WSASendMsg
WSASendTo
WSASend
WSARecvFrom
WSARecvMsg
WSARecv
I want to use Overlapped I/O with Completion Routine to handle client connections.
In my UI thread I want to use WSASend(), but in order for the system to call my callback function to inform me that data has been sent, the UI thread must be in a wait state, but this will freeze my UI!
How should I fix this problem?
I agree with #DavidHeffernan - the UI thread should be doing UI things. The IO thread surely needs a binding and port, (server), or peer address and port(client). The socket from ConnectEx or AcceptEx is surely better loaded in the IO thread, but a Socket class with, (at this time, undefined), socket member could surely be created in the UI thread and signaled into the IO thread for handling. Whether buffers form part of your Socket class, or a separate Buffer class, is a design consideration.
One implementation, (that I have used successfully):
Design/define an 'Inter Thread Comms', (ITC'), message class. This has a 'command' enum member that can tell other threads to do stuff, together with any other useful stuff that might be required in such a message
Derive a 'Socket' class from ITC. This has string members for the IP/port, the socket handle and anything else that may be required.
Derive a 'Buffer' class from ITC. This has a 'BoundSocket' member, buffer-space and an 'OVERLAPPED' struct.
Comms with the IO thread is fairly easy. Since it has to wait on something altertably, it can wait on a semaphore that manages a 'Commands' ConcurrentQueue.
If you UI wishes to instruct the IO thread to, say, connect to a server, it creates a Socket instance, (new), loads the IP and Port members from UI elements, sets the Command enum to 'Connect', pushes the socket onto the Commands queue and signals the semaphore, (ReleaseSemaphore).
The alertable wait in the IO thread then returns with WAIT_OBJECT_0, (it needs to ignore returns with WAIT_IO_COMPLETION) and so knows that a command has ben queued. It pops it from the Commands queue and acts upon the command enum, (maybe switching on it), to perform the required action/s. For connect, this would involve an overlapped 'ConnectEx' call to queue up a connect request and set up the connect completion handler.
The connect completion handler, when called, checks for a succesfull connect and, if so, could new up a Buffer, load it, issue a WSARecv with it for the server to send stuff and store the returned Socket object in a container. If failed, it could load the Socket
with a suitable error message and PostMessage it back to the UI thread to inform the user of the fail.
See - it's not that difficult and does not need 10000 lines of code:)
The only thing I don't know how to do immediately is getting the 'this' for the socket object back from the OVERLAPPED struct that is returned in the completion routine. On 32-bit systems, I shoved the Buffer 'this' into the hEvent field of the overlapped struct in the Buffer instance and cast it back in the completion routine. The Buffer instance has a Socket reference, so the job was done. On 64-bit systems, hEvent has not enough room to store the 48/64-bit 'this' Buffer pointer and, (aparrently), this required an extended OVERLAPPED struct:( Not sure how that is done - maybe you will find out:)
[edit] #BenVoigt has advice on the 32/64 bit 'getting the Socket context 'this' back in the completion routine' issue - it's easier than I thought:):
https://stackoverflow.com/a/28660537/758133
In my c++ windows application I'm using the CreateFile function to access a file. I'm using also the following functions to control my file:
DeviceIOControl
ReadFile
SetFilePointer
since I want that my application will access many positions in the file at the same time (from requests I get to my application) I want that my application will be async. I saw there is a flag for this called FILE_FLAG_OVERLAPPED but I don't know how to use it and when do I know when it complete its operation.
is there any callback that I have to register?
if you can copy here a sample it will be helpful
thanks
You can just spawn a new thread and call ReadFie() synchronously from that second thread. You need to synchronize both threads using a synchronization object, such as an event or a critical section.
If you insist on the single-threaded asynchonous approach, then call the ReadFile() or ReadFileEx() and pass an OVERLAPPED structure. The structure will provide a callback function that will be called once the file read is complete.
However for the callback function to be called, the calling thread (the one that called ReadFile() or ReadFileEx()) has to be in an alertable state.
You just use one of the wait functions to suspend the thread and wait for the read function to call the callback function.
Did you read this MSDN page on Synchronization and Overlapped Input and Output which is describing how it works and linking to the Named Pipe Server Using Overlapped I/O sample?
We have an API that handles event timers. This API says that it uses OS callbacks to handle timed events (using select(), apparently).
The api claims this order of execution as well:
readable events
writable events
timer events
This works by creating a point to a Timer object, but passing the create function a function callback:
Something along these lines:
Timer* theTimer = Timer::Event::create(timeInterval,&Thisclass::FunctionName);
I was wondering how this worked?
The operating system is handling the timer itself, and when it sees it fired how does it actually invoke the callback? Does the callback run in a seperate thread of execution?
When I put a pthread_self() call inside the callback function (Thisclass::FunctionName) it appears to have the same thread id as the thread where theTimer is created itself! (Very confused by this)
Also: What does that priority list above mean? What is a writable event vs a readable event vs a timer event?
Any explanation of the use of select() in this scenario is also appreciated.
Thanks!
This looks like a simple wrapper around select(2). The class keeps a list of callbacks, I guess separate for read, write, and timer expiration. Then there's something like a dispatch or wait call somewhere there that packs given file descriptors into sets, calculates minimum timeout, and invokes select with these arguments. When select returns, the wrapper probably goes over read set first, invoking read callback, then write set, then looks if any of the timers have expired and invokes those callbacks. This all might happen on the same thread, or on separate threads depending on the implementation of the wrapper.
You should read up on select and poll - they are very handy.
The general term is IO demultiplexing.
A readable event means that data is available for reading on a particular file descriptor without blocking, and a writable event means that you can write to a particular file descriptor without blocking. These are most often used with sockets and pipes. See the select() manual page for details on these.
A timer event means that a previously created timer has expired. If the library is using select() or poll(), the library itself has to keep track of timers since these functions accept a single timeout. The library must calculate the time remaining until the first timer expires, and use that for the timeout parameter. Another approach is to use timer_create(), or an older variant like setitimer() or alarm() to receive notification via a signal.
You can determine which mechanism is being used at the OS layer using a tool like strace (Linux) or truss (Solaris). These tools trace the actual system calls that are being made by the program.
At a guess, the call to create() stores the function pointer somewhere. Then, when the timer goes off, it calls the function you specified via that pointer. But as this is not a Standard C++ function, you should really read the docs or look at the source to find out for sure.
Regarding your other questions, I don't see mention of a priority list, and select() is a sort of general purpose event multiplexer.
Quite likely there's a framework that works with a typical main loop, the driving force of the main loop is the select call.
select allows you to wait for a filedescriptor to become readable or writable (or for an "exception" on the filedeescriptor) or for a timeout to occur. I'd guess the library also allow you to register callbacks for doing async IO, if it's a GUI library it'll get the low primitive GUI events via a file descriptor on unixes.
To implement timer callbacks in such a loop, you just keep a priority queue of timers and process them on select timeouts or filedescriptor events.
The priority means it processes the file i/o before the timers, which in itself takes time, could result in GUI updates eventually resulting in GUI event handlers being run, or other tasks spending time servicing I/O.
The library is more or less doing
for(;;) {
timeout = calculate_min_timeout();
ret = select(...,timeout); //wait for a timeout event or filedescriptor events
if(ret > 0) {
process_readable_descriptors();
process_writable_descriptors();
}
process_timer_queue(); //scan through a timer priority queue and invoke callbacks
}
Because of the fact that the thread id inside the timer callback is the same as the creator thread I think that it is implemented somehow using signals.
When a signal is sent to a thread that thread's state is saved and the signal handler is called which then calls the event call back.
So the handler is called in the creator thread which is interrupted until the signal handler returns.
Maybe another thread waits for all timers using select() and if a timer expires it sends a signal to the thread the expired timer was created in.
To simplify, this is a situation where a NamedPipe SERVER is waiting for a NamedPipe CLIENT to write to the pipe (using WriteFile())
The Windows API that is blocking is ReadFile()
The Server has created the synchronous pipe (no overlapped I/O) with blocking enabled
The client has connected, and now the server is waiting for some data.
In the normal flow of things, the client sends some data and the server processes it and then returns to ReadFile() to wait for the next chunk of data.
Meanwhile an event occurs (user input for example) and the NamedPipe SERVER must now execute some other code, which it cannot do while the ReadFile() is blocking.
At this point I need to mention that the NamedPipe Client is not my application, so I have no control over it. I cannot make it send a few bytes to unblock the server. It is just going to sit there and send no data. Since I do not have control of the Client implementation I cannot change anything on that end.
One solution would be to create a separate thread in which all ReadFile() operations are performed. That way when the event occurs, I can just process the code. The problem with that, is that the event also requires a separate thread, so now I have two additional threads for each instance of this server. Since this needs to be scalable, this is undesirable.
From another thread I have tried calling
DisconnectNamedPipe()
and
CloseHandle()
they both will not return (until the client writes to the pipe.)
I cannot connect to the same pipe and write a few bytes because:
"All instances of a named pipe share the same pipe name, but each instance has
its own buffers and handles, and provides a separate conduit for client/server
communication."
http://msdn.microsoft.com/en-us/library/aa365590.aspx
I need a way to fake it out, So the $64k dollar question is:
How can I break the blocking of ReadFile()?
Try this before ReadFile :
BOOL WINAPI PeekNamedPipe(
__in HANDLE hNamedPipe,
__out_opt LPVOID lpBuffer,
__in DWORD nBufferSize,
__out_opt LPDWORD lpBytesRead,
__out_opt LPDWORD lpTotalBytesAvail,
__out_opt LPDWORD lpBytesLeftThisMessage
);
if(TotalBytesAvail > 0)
ReadFile(....);
-AV-
Take a look on CancelSynchronousIo
Marks pending synchronous I/O
operations that are issued by the
specified thread as canceled.
And CancelIo/CancelIoEx:
To cancel all pending asynchronous I/O
operations, use either:
CancelIo — this function only cancels
operations issued by the calling
thread for the specified file handle.
CancelIoEx — this function cancels all
operations issued by the threads for
the specified file handle.
http://msdn.microsoft.com/en-us/library/aa363794(VS.85).aspx
http://msdn.microsoft.com/en-us/library/aa365467(VS.85).aspx
Mike,
You can't cancel synchronous ReadFile. But you can switch to asynchronous (overlapped) operations. By doing this, you can implement a pretty scalable architecture.
Possible algorithm (just an idea):
For each new client call ReadFile
WaitForMultipleObjects where the handles are overlapped.hEvent + your
custom events
Iterate over signalled events, and schedule them for execution by threads from a threads pool.
This way you can have only few threads to receive connections and read data, while the actual data processing can be done by the threads pool.
The problem with that, is that the
event also requires a separate thread,
so now I have two additional threads
for each instance of this server.
Since this needs to be scalable, this
is undesirable.
Never in my career have I found that "more threads" == "less scalable". How many of these "server" instances do you have?
Normally, an operation needs to be performed in a separate thread if that operation is going to block and the system needs to be responsive while the operation is blocked.
Asynchronous I/O operations do not have to block any thread if they use I/O Completion Ports. See: http://msdn.microsoft.com/en-us/library/aa365198(VS.85).aspx
What happening is the server outbound pipe is left open waiting for connection while your client is trying to connect to the server inbound pipe (which is no longer existent)... What you need to do is flush out your outbound pipe in order to loop back to your inbound. You can flush out on the client side by reading the file (remember to loop the connect establishment because there is a "handshake" in there, and it will never work the first time)
Just use SetNamedPipeHandleState function
https://learn.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-setnamedpipehandlestate
Use the PIPE_NOWAIT flag when calling this function.
hNamedPipe should be the handle that returned from CreateFile function.
After that, the call to ReadFile will not block the thread when no data available.