Breaking ReadFile() blocking - Named Pipe (Windows API) - c++

To simplify, this is a situation where a NamedPipe SERVER is waiting for a NamedPipe CLIENT to write to the pipe (using WriteFile())
The Windows API that is blocking is ReadFile()
The Server has created the synchronous pipe (no overlapped I/O) with blocking enabled
The client has connected, and now the server is waiting for some data.
In the normal flow of things, the client sends some data and the server processes it and then returns to ReadFile() to wait for the next chunk of data.
Meanwhile an event occurs (user input for example) and the NamedPipe SERVER must now execute some other code, which it cannot do while the ReadFile() is blocking.
At this point I need to mention that the NamedPipe Client is not my application, so I have no control over it. I cannot make it send a few bytes to unblock the server. It is just going to sit there and send no data. Since I do not have control of the Client implementation I cannot change anything on that end.
One solution would be to create a separate thread in which all ReadFile() operations are performed. That way when the event occurs, I can just process the code. The problem with that, is that the event also requires a separate thread, so now I have two additional threads for each instance of this server. Since this needs to be scalable, this is undesirable.
From another thread I have tried calling
DisconnectNamedPipe()
and
CloseHandle()
they both will not return (until the client writes to the pipe.)
I cannot connect to the same pipe and write a few bytes because:
"All instances of a named pipe share the same pipe name, but each instance has
its own buffers and handles, and provides a separate conduit for client/server
communication."
http://msdn.microsoft.com/en-us/library/aa365590.aspx
I need a way to fake it out, So the $64k dollar question is:
How can I break the blocking of ReadFile()?

Try this before ReadFile :
BOOL WINAPI PeekNamedPipe(
__in HANDLE hNamedPipe,
__out_opt LPVOID lpBuffer,
__in DWORD nBufferSize,
__out_opt LPDWORD lpBytesRead,
__out_opt LPDWORD lpTotalBytesAvail,
__out_opt LPDWORD lpBytesLeftThisMessage
);
if(TotalBytesAvail > 0)
ReadFile(....);
-AV-

Take a look on CancelSynchronousIo
Marks pending synchronous I/O
operations that are issued by the
specified thread as canceled.
And CancelIo/CancelIoEx:
To cancel all pending asynchronous I/O
operations, use either:
CancelIo — this function only cancels
operations issued by the calling
thread for the specified file handle.
CancelIoEx — this function cancels all
operations issued by the threads for
the specified file handle.
http://msdn.microsoft.com/en-us/library/aa363794(VS.85).aspx
http://msdn.microsoft.com/en-us/library/aa365467(VS.85).aspx

Mike,
You can't cancel synchronous ReadFile. But you can switch to asynchronous (overlapped) operations. By doing this, you can implement a pretty scalable architecture.
Possible algorithm (just an idea):
For each new client call ReadFile
WaitForMultipleObjects where the handles are overlapped.hEvent + your
custom events
Iterate over signalled events, and schedule them for execution by threads from a threads pool.
This way you can have only few threads to receive connections and read data, while the actual data processing can be done by the threads pool.

The problem with that, is that the
event also requires a separate thread,
so now I have two additional threads
for each instance of this server.
Since this needs to be scalable, this
is undesirable.
Never in my career have I found that "more threads" == "less scalable". How many of these "server" instances do you have?
Normally, an operation needs to be performed in a separate thread if that operation is going to block and the system needs to be responsive while the operation is blocked.

Asynchronous I/O operations do not have to block any thread if they use I/O Completion Ports. See: http://msdn.microsoft.com/en-us/library/aa365198(VS.85).aspx

What happening is the server outbound pipe is left open waiting for connection while your client is trying to connect to the server inbound pipe (which is no longer existent)... What you need to do is flush out your outbound pipe in order to loop back to your inbound. You can flush out on the client side by reading the file (remember to loop the connect establishment because there is a "handshake" in there, and it will never work the first time)

Just use SetNamedPipeHandleState function
https://learn.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-setnamedpipehandlestate
Use the PIPE_NOWAIT flag when calling this function.
hNamedPipe should be the handle that returned from CreateFile function.
After that, the call to ReadFile will not block the thread when no data available.

Related

What IO operations causes completion packets to be sent to the completion port when using sockets?

In IOCP, when starting an IO operation such as WSARecv(), a completion packet will be sent to the completion port when the IO operation completes.
What I want to know is what IO operations causes completion packets to be sent to the completion port when using sockets, for example, I know that WSASend(), WSARecv(), AcceptEx(), and PostQueuedCompletionStatus() causes completion packets to be sent. Is there other IO operations that does that?
A completion will be queued to the IOCP associated with a socket only if an API call that can generate completions is called in a way that requests a completion to be queued. So you will know which API calls can generate completions by the fact that you've read the documentation and you're passing an OVERLAPPED structure to them.
Thus you don't really need to know the answer to your question as you will never get a completion that you do not expect to get as you have to have called an appropriate API with appropriate parameters for a completion to be generated.
You can then differentiate between the API that caused the completion to be generated by adding some form of identifying "per operation data" to the OVERLAPPED either by making an 'extended overlapped stucture' or by using the event handle as opaque data. Either way you get a chance to send some context from the API call site to the IOCP completion handling site. This context is of your own design and can tell you what initiated the completion.
Then you get to use the return value from the GetQueuedCompletionStatus() call to determine if the completion is a success or failure and you can then access the error code for failures using WSAGetLastError() (though see this answer for more detail on an additional hoop that you could jump through to get more accurate error codes).
This then lets you determine which of the events listed in EJP's answer you have.
The actual set of functions that can generate a completion for socket operations can change with changes in the OS. The easiest way to determine what these are for the operating system that you're targeting is to either read the MSDN docs or do a search of the SDK headers for lpOverlapped... As you'll see from the current VS2013 headers there are quite a few that relate to sockets; AcceptEx(), ConnectEx(), DisconnectEx(), TransmitFile(), the HTTP.sys API, the RIO API, etc.
You're missing the point. What causes completion packets to be sent is events, not API calls. There are basically only a few TCP events:
inbound connection
outbound connection complete
data
write finished
timeout
end of stream, and
error.
Copied from the site
Supported I/O Functions
The following functions can be used to start I/O operations that complete by using I/O completion ports. You must pass the function an instance of the OVERLAPPED structure and a file handle previously associated with an I/O completion port (by a call to CreateIoCompletionPort) to enable the I/O
completion port mechanism:
ConnectNamedPipe
DeviceIoControl
LockFileEx
ReadDirectoryChangesW
ReadFile
TransactNamedPipe
WaitCommEvent
WriteFile
WSASendMsg
WSASendTo
WSASend
WSARecvFrom
WSARecvMsg
WSARecv

What happens when sending multiple messages on WSADuplicateSocket?

I'm looking for some information on what happens when sending at the same time on a WSA socket that had been duplicated? Is that safe? Can you point me to any specific docs?
Do I need to have some sort of message across the processes to tell which process to be active for sends? I assume I have to do that for receives.
Can anyone give me an example of what this portion of the documentation means?
Notification on shared sockets is subject to the usual constraints of WSAAsyncSelect and WSAEventSelect. Issuing either of these calls using any of the shared descriptors cancels any previous event registration for the socket, regardless of which descriptor was used to make that registration. Thus, a shared socket cannot deliver FD_READ events to process A and FD_WRITE events to process B. For situations when such tight coordination is required, developers would be advised to use threads instead of separate processes.
As the documentation states:
The descriptors that reference a shared socket can be used independently for I/O. However, the Windows Sockets interface does not implement any type of access control, so it is up to the processes involved to coordinate their operations on a shared socket. Shared sockets are typically used to having one process that is responsible for creating sockets and establishing connections, and other processes that are responsible for information exchange.
If you have two processes sending on a shared socket at the same time, they are going to overlap each other. The same as if two threads within a single process send to the same socket at the same time. So you need to coordinate the sends to avoid the overlap. You can use a shared named mutex for that, for instance.
As for the quote you asked about, it should be self-explanatory if you read the relevant documentation:
WSAAsyncSelect() function
Issuing a WSAAsyncSelect for a socket cancels any previous WSAAsyncSelect or WSAEventSelect for the same socket. For example, to receive notification for both reading and writing, the application must call WSAAsyncSelect with both FD_READ and FD_WRITE, as follows:
rc = WSAAsyncSelect(s, hWnd, wMsg, FD_READ|FD_WRITE);
It is not possible to specify different messages for different events. The following code will not work; the second call will cancel the effects of the first, and only FD_WRITE events will be reported with message wMsg2:
rc = WSAAsyncSelect(s, hWnd, wMsg1, FD_READ);
rc = WSAAsyncSelect(s, hWnd, wMsg2, FD_WRITE);
WSAEventSelect() function
Issuing a WSAEventSelect for a socket cancels any previous WSAAsyncSelect or WSAEventSelect for the same socket and clears the internal network event record. For example, to associate an event object with both reading and writing network events, the application must call WSAEventSelect with both FD_READ and FD_WRITE, as follows:
rc = WSAEventSelect(s, hEventObject, FD_READ|FD_WRITE);
It is not possible to specify different event objects for different network events. The following code will not work; the second call will cancel the effects of the first, and only the FD_WRITE network event will be associated with hEventObject2:
rc = WSAEventSelect(s, hEventObject1, FD_READ);
rc = WSAEventSelect(s, hEventObject2, FD_WRITE); //bad
So, if you have process S sharing a socket with processes A and B, you can't have A listening for FD_READ events and B listening for FD_WRITE events, or vice versa. It is an all-or-nothing deal.
It is as safe as writing to the same socket from two threads. The actual send() function is atomic: no-one else can enter it for the same socket while you're in it. So you won't get data interleaving, provided you are writing entire messages with a single send(), and you're in blocking mode: see comments below.
Only if you're violating the condition above.
It implies that really only one of the processes can use select() on that socket concurrently. As soon as one process does that, the other process loses its 'interest' in whatever it was selecting on, so none of those events will fire. This is usually exactly what you want, as you don't want both processes reacting to the same read event. However it's more problematic if you're also selecting on write. If you're not selecting at all, don't worry about this.

What is an overlapped I/O alternative to WaitNamedPipe?

The WaitNamedPipe function allows a pipe client application to synchronously wait for an available connection on a named pipe server. You then call CreateFile to open the pipe as a client. Pseudocode:
// loop works around race condition with WaitNamedPipe and CreateFile
HANDLE hPipe;
while (true) {
if (WaitNamedPipe says connection is ready) {
hPipe = CreateFile(...);
if (hPipe ok or last error is NOT pipe busy) {
break; // hPipe is valid or last error is set
}
} else {
break; // WaitNamedPipe failed
}
}
The problem is that these are all blocking, synchronous calls. What is a good way to do this asynchronously? I can't seem to find an API that uses overlapped I/O to do this, for example. For example, for pipe servers the ConnectNamedPipe function provides an lpOverlapped parameters allowing for a server to asynchronously wait for a client. The pipe server can then call WaitForMultipleObjects and wait for the I/O operation to complete, or any other event to be signaled (for example, an event signaling the thread to cancel pending I/O and terminate).
The only way I can think of is to call WaitNamedPipe in a loop with a short, finite timeout and check other signals if it times out. Alternatively, in a loop call CreateFile, check other signals, and then call Sleep with a short delay (or WaitNamedPipe). For example:
HANDLE hPipe;
while (true) {
hPipe = CreateFile(...);
if (hPipe not valid and pipe is busy) {
// sleep 100 milliseconds; alternatively, call WaitNamedPipe with timeout
Sleep(100);
// TODO: check other signals here to see if we should abort I/O
} else
break;
}
But this method stinks to high heaven in my opinion. If a pipe isn't available for awhile, the thread continues to run - sucking up CPU, using power, requiring memory pages to remain in RAM, etc. In my mind, a thread that relies on Sleep or short timeouts does not perform well and is a sign of sloppy multi-threaded programming.
But what's the alternative in this case?
WaitNamedPipe is completely useless, and will just use all the cpu if you specify a timeout and there's no server waiting for it.
Just call CreateFile over and over with a Sleep like you're doing, and move it to other threads as you see appropriate. There is no API alternative.
The only "benefit" WaitNamedPipe provides is if you want to know if you can connect to a named pipe but you explicitly don't want to actually open a connection. It's junk.
If you really want to be thorough, your only options are
Ensure that whatever program is opening the named pipe is always calling CreateNamedPipe again immediately after it's named pipe is connected to.
Have your program actually check if that program is running.
If your intent is really not to have additional connections, still call CreateNamedPipe, and when someone connects, tell them to go away until they're waited a given amount of time, the close the pipe.
Why can't the server just create more pipes? The performance hit in the scenario you describe isn't a problem if it is rare.
I.e. if there are usually enough pipes to go round what does it matter if you use CreateFile/Sleep instead of WaitForMultipleObjects? The performance hit will not matter.
I also have to question the need for overlapped IO in a client. How many servers is it communicating with at a time? If the answer is less than, say, 10 you could reasonably create a thread per connection.
Basically I am saying I think the reason there is no overlapped WaitforNamedPipe is because there is no reasonable use-case which requires it.
You can open the pipe file system at \\.\pipe\ and then use DeviceIoControl to send FSCTL_PIPE_WAIT.

EvtSubscribe and threading

I am trying to write a log forwarded for Windows. The plan is simple - receive an event notification and then write it over a TCP socket. This MSDN example shows that I should be using EvtSubscribe. However, I am confused as to how I should share the file descriptor for the open TCP socket. Will the EvtSubscribe callback block by default or will it thread or...?
Thank you in advance for any tips, picking up C++ on Windows after C on Linux has been a bit of a challenge for me :)
The docs are quite sparse in details, but I reckon that it works as follows:
If you use the subscription callback, then it will be called in a dedicated thread. So, if you delay in it, it will block further callbacks, but not other thread of the program
If you use the SignalEvent, it will get signaled when the event arrives, and no threads are created automatically.
You can check that it is really another thread by calling GetCurrentThreadId() from the code that calls EvSubscribe() and from the callback, and compare the values.
My recommendation is to use the thread options, as the Event handlers in Windows are so difficult to be programmed correctly.
About sharing the TCP socket, you can share a socket between threads, but you should not write to it from more than one thread at a time. Nor read.
You can, however, read from one thread and write from another. Also, you can close the socket from one thread while other is in a blocking operation: it will get cancelled.
If you find this limiting, you should create a user thread and use it to send and/or receive data, while communicating with the other threads with queues, or similar.

Asynchronous ReadDirectoryChangesW()?

I want to monitor the modifications in a specified directory, and retrieve the exact change information. So I've decided to use the ReadDirectoryChangesW() function. But I want to use it asynchronously, which means I don't want my worker thread to be blocked if there are no changes in the directory.
How can I do this?
I tend to use ReadDirectoryChangesW() with an I/O Completion Port, which is good for when you want to monitor multiple directories because you can have a single thread servicing the IOCP and dealing with the results from all of your calls to ReadDirectoryChangesW().
First create an IOCP, once, by calling CreateIoCompletionPort(). Start a thread that calls GetQueuedCompletionStatus() and deals with your async directory changes.
Now for each directory that you want to monitor. First, open your directory handle as normal. Next, associate it with your IOCP by calling CreateIoCompletionPort() and passing the IOCP handle in (see docs for more details). Finally, call ReadDirectoryChangesW() and pass an OVERLAPPED as detailed in the docs. The call will return, and if successful, the results will become available as the results of GetQueuedCompletionStatus() in the thread that you created to service the IOCP.
From the MSDN documentation for ReadDirectoryChnagesW():
For asynchronous completion, you can receive notification in one of
three ways:
Using the GetOverlappedResult function. To receive notification
through GetOverlappedResult, do not
specify a completion routine in the
lpCompletionRoutine parameter. Be sure
to set the hEvent member of the
OVERLAPPED structure to a unique
event.
Using the GetQueuedCompletionStatus function. To
receive notification through
GetQueuedCompletionStatus, do not
specify a completion routine in
lpCompletionRoutine. Associate the
directory handle hDirectory with a
completion port by calling the
CreateIoCompletionPort function.
Using a completion routine. To receive notification through a
completion routine, do not associate
the directory with a completion port.
Specify a completion routine in
lpCompletionRoutine. This routine is
called whenever the operation has been
completed or canceled while the thread
is in an alertable wait state. The
hEvent member of the OVERLAPPED
structure is not used by the system,
so you can use it yourself.