I'm looking for some information on what happens when sending at the same time on a WSA socket that had been duplicated? Is that safe? Can you point me to any specific docs?
Do I need to have some sort of message across the processes to tell which process to be active for sends? I assume I have to do that for receives.
Can anyone give me an example of what this portion of the documentation means?
Notification on shared sockets is subject to the usual constraints of WSAAsyncSelect and WSAEventSelect. Issuing either of these calls using any of the shared descriptors cancels any previous event registration for the socket, regardless of which descriptor was used to make that registration. Thus, a shared socket cannot deliver FD_READ events to process A and FD_WRITE events to process B. For situations when such tight coordination is required, developers would be advised to use threads instead of separate processes.
As the documentation states:
The descriptors that reference a shared socket can be used independently for I/O. However, the Windows Sockets interface does not implement any type of access control, so it is up to the processes involved to coordinate their operations on a shared socket. Shared sockets are typically used to having one process that is responsible for creating sockets and establishing connections, and other processes that are responsible for information exchange.
If you have two processes sending on a shared socket at the same time, they are going to overlap each other. The same as if two threads within a single process send to the same socket at the same time. So you need to coordinate the sends to avoid the overlap. You can use a shared named mutex for that, for instance.
As for the quote you asked about, it should be self-explanatory if you read the relevant documentation:
WSAAsyncSelect() function
Issuing a WSAAsyncSelect for a socket cancels any previous WSAAsyncSelect or WSAEventSelect for the same socket. For example, to receive notification for both reading and writing, the application must call WSAAsyncSelect with both FD_READ and FD_WRITE, as follows:
rc = WSAAsyncSelect(s, hWnd, wMsg, FD_READ|FD_WRITE);
It is not possible to specify different messages for different events. The following code will not work; the second call will cancel the effects of the first, and only FD_WRITE events will be reported with message wMsg2:
rc = WSAAsyncSelect(s, hWnd, wMsg1, FD_READ);
rc = WSAAsyncSelect(s, hWnd, wMsg2, FD_WRITE);
WSAEventSelect() function
Issuing a WSAEventSelect for a socket cancels any previous WSAAsyncSelect or WSAEventSelect for the same socket and clears the internal network event record. For example, to associate an event object with both reading and writing network events, the application must call WSAEventSelect with both FD_READ and FD_WRITE, as follows:
rc = WSAEventSelect(s, hEventObject, FD_READ|FD_WRITE);
It is not possible to specify different event objects for different network events. The following code will not work; the second call will cancel the effects of the first, and only the FD_WRITE network event will be associated with hEventObject2:
rc = WSAEventSelect(s, hEventObject1, FD_READ);
rc = WSAEventSelect(s, hEventObject2, FD_WRITE); //bad
So, if you have process S sharing a socket with processes A and B, you can't have A listening for FD_READ events and B listening for FD_WRITE events, or vice versa. It is an all-or-nothing deal.
It is as safe as writing to the same socket from two threads. The actual send() function is atomic: no-one else can enter it for the same socket while you're in it. So you won't get data interleaving, provided you are writing entire messages with a single send(), and you're in blocking mode: see comments below.
Only if you're violating the condition above.
It implies that really only one of the processes can use select() on that socket concurrently. As soon as one process does that, the other process loses its 'interest' in whatever it was selecting on, so none of those events will fire. This is usually exactly what you want, as you don't want both processes reacting to the same read event. However it's more problematic if you're also selecting on write. If you're not selecting at all, don't worry about this.
Related
I'm trying to create a server with blocking sockets (one new thread for each new client). This thread should be able to receive commands from the client (and send back the result) and periodically send commands to the client (and request back the result).
What I've thought is creating two threads for each client, one for recv, second for send. However:
it's double of the normal thread overhead.
due to request/response design, recv I do in the first thread (to wait for client's commands) can be the request I look for in the second thread (client's result to my send) and vice versa. Making it all properly synced is probably a hell story. So now I'm thinking to do that from a single thread this way:
In a loop:
setsockopt(SO_RCVTIMEO, &small_timeout); // set the timeout for the recv (like 1000 ms).
recv(); // check for client's requests first. if returns WSAETIMEDOUT than I assume no data is requested and do nothing. if I get a normal request I handle it.
if (clientbufferToSend != nullptr) send(clientbufferToSend); // now when client's request has been processed we check the command list we have to send to the client. if there is commands in queue, we send them. SO_SNDTIMEO timeout can be set to a large value so we don't deadlock if client looses connection.
setsockopt(SO_RCVTIMEO, &large_timeout); // set the timeout for the recv (as large as SO_SNDTIMEO, just to not deadlock if anything).
recv(); // now we wait the response from the client.
Is this the legal way to do what I want? Or are there better alternatives (preferrably with blocking sockets and threads)?
P.S. Does recv() with timeout returns WSAETIMEDOUT only if no data is available? Can it return this error if there is the data, but recv() wasn't fast enough to handle it all, thus returning partial data?
One approach is only create a background thread for reading from that socket. Write on whatever random thread your unsolicited events are raised.
You’ll need following stuff.
A critical section or mutex per socket to serialize writes, like when background thread is sending response to client-initiated message, and other thread wants to send message to the same client.
Some other synchronization primitive like a conditional variable for client thread to sleep while waiting for responses.
The background thread which receives messages needs to distinguish client-initiated messages (which need to be responded by the same background thread) from responses to server-initiated messages. If your network protocol doesn’t have that data you’ll have to change the protocol.
This will work OK if your server-initiated events are only happening on a single thread, e.g. they come from some serialized source like a device or OS interface.
If however the event source is multithreaded as well, and you want good performance, you gonna need non-trivial complexity to dispatch the responses to the correct server thread, like 1 conditional variable per client thread, maybe some queues, etc.
I want to use Overlapped I/O with Completion Routine to handle client connections.
In my UI thread I want to use WSASend(), but in order for the system to call my callback function to inform me that data has been sent, the UI thread must be in a wait state, but this will freeze my UI!
How should I fix this problem?
I agree with #DavidHeffernan - the UI thread should be doing UI things. The IO thread surely needs a binding and port, (server), or peer address and port(client). The socket from ConnectEx or AcceptEx is surely better loaded in the IO thread, but a Socket class with, (at this time, undefined), socket member could surely be created in the UI thread and signaled into the IO thread for handling. Whether buffers form part of your Socket class, or a separate Buffer class, is a design consideration.
One implementation, (that I have used successfully):
Design/define an 'Inter Thread Comms', (ITC'), message class. This has a 'command' enum member that can tell other threads to do stuff, together with any other useful stuff that might be required in such a message
Derive a 'Socket' class from ITC. This has string members for the IP/port, the socket handle and anything else that may be required.
Derive a 'Buffer' class from ITC. This has a 'BoundSocket' member, buffer-space and an 'OVERLAPPED' struct.
Comms with the IO thread is fairly easy. Since it has to wait on something altertably, it can wait on a semaphore that manages a 'Commands' ConcurrentQueue.
If you UI wishes to instruct the IO thread to, say, connect to a server, it creates a Socket instance, (new), loads the IP and Port members from UI elements, sets the Command enum to 'Connect', pushes the socket onto the Commands queue and signals the semaphore, (ReleaseSemaphore).
The alertable wait in the IO thread then returns with WAIT_OBJECT_0, (it needs to ignore returns with WAIT_IO_COMPLETION) and so knows that a command has ben queued. It pops it from the Commands queue and acts upon the command enum, (maybe switching on it), to perform the required action/s. For connect, this would involve an overlapped 'ConnectEx' call to queue up a connect request and set up the connect completion handler.
The connect completion handler, when called, checks for a succesfull connect and, if so, could new up a Buffer, load it, issue a WSARecv with it for the server to send stuff and store the returned Socket object in a container. If failed, it could load the Socket
with a suitable error message and PostMessage it back to the UI thread to inform the user of the fail.
See - it's not that difficult and does not need 10000 lines of code:)
The only thing I don't know how to do immediately is getting the 'this' for the socket object back from the OVERLAPPED struct that is returned in the completion routine. On 32-bit systems, I shoved the Buffer 'this' into the hEvent field of the overlapped struct in the Buffer instance and cast it back in the completion routine. The Buffer instance has a Socket reference, so the job was done. On 64-bit systems, hEvent has not enough room to store the 48/64-bit 'this' Buffer pointer and, (aparrently), this required an extended OVERLAPPED struct:( Not sure how that is done - maybe you will find out:)
[edit] #BenVoigt has advice on the 32/64 bit 'getting the Socket context 'this' back in the completion routine' issue - it's easier than I thought:):
https://stackoverflow.com/a/28660537/758133
using socket with the overlapped operation selected the event-based completion notification;
Have 2 events, one for data, the other to cancel long send/recv:
HANDLE events[] = { m_hDataEvent, m_hInterruptEvent };
then calling WSASend,
WSASend(m_Socket, &DataBuf, 1, NULL, 0, &SendOverlapped, NULL);
followed by
WSAWaitForMultipleEvents(2, events, FALSE, INFINITE, FALSE);
which is setup to return on any one event signaled.
Now assume send is in progress, and m_hInterruptEvent is signaled.
WSAWaitForMultipleEvents returns, technically the function calling send can return as well and delete internally allocated buffers.
What is not clear to me, the WSASend may still be working in background, and deleting buffers will cause data corruption in best case.
What would be the proper way to stop the background Send/Receive, if the socket needs to be used for something else immediately?
I looked at the CancelIO(), but the MSDN never mentions it in relation to Sockets, does it work with file based IO only?
It makes no sense to try to cancel it once sent. Even if you succeeded you would have a problem because the receiving application would not have any idea that the transmission was interrupted. Your new message will be mistaken for the end of the old message.
If you feel the need to cancel long sends, you should probably look at your application design.
Send in chunks and check for cancellation in between chunks. Ensure you have a way of communicating to the receiver that the transmission was cancelled.
Close the socket to cancel. Again, ensure the client has a way to know that this is an interrupted transmission (for example if the client knows the total length in advance they will recognise an interrupted transmission).
Just wait for it to succeed in the background and don't worry. If you have urgent messages use a separate connection for them.
For your particular question "What would be the proper way to stop the background Send/Receive, if the socket needs to be used for something else immediately", the answer is: Sockets are cheap - Just use two - one for the slow transmission the other for the urgent messages.
I am trying to write a log forwarded for Windows. The plan is simple - receive an event notification and then write it over a TCP socket. This MSDN example shows that I should be using EvtSubscribe. However, I am confused as to how I should share the file descriptor for the open TCP socket. Will the EvtSubscribe callback block by default or will it thread or...?
Thank you in advance for any tips, picking up C++ on Windows after C on Linux has been a bit of a challenge for me :)
The docs are quite sparse in details, but I reckon that it works as follows:
If you use the subscription callback, then it will be called in a dedicated thread. So, if you delay in it, it will block further callbacks, but not other thread of the program
If you use the SignalEvent, it will get signaled when the event arrives, and no threads are created automatically.
You can check that it is really another thread by calling GetCurrentThreadId() from the code that calls EvSubscribe() and from the callback, and compare the values.
My recommendation is to use the thread options, as the Event handlers in Windows are so difficult to be programmed correctly.
About sharing the TCP socket, you can share a socket between threads, but you should not write to it from more than one thread at a time. Nor read.
You can, however, read from one thread and write from another. Also, you can close the socket from one thread while other is in a blocking operation: it will get cancelled.
If you find this limiting, you should create a user thread and use it to send and/or receive data, while communicating with the other threads with queues, or similar.
To simplify, this is a situation where a NamedPipe SERVER is waiting for a NamedPipe CLIENT to write to the pipe (using WriteFile())
The Windows API that is blocking is ReadFile()
The Server has created the synchronous pipe (no overlapped I/O) with blocking enabled
The client has connected, and now the server is waiting for some data.
In the normal flow of things, the client sends some data and the server processes it and then returns to ReadFile() to wait for the next chunk of data.
Meanwhile an event occurs (user input for example) and the NamedPipe SERVER must now execute some other code, which it cannot do while the ReadFile() is blocking.
At this point I need to mention that the NamedPipe Client is not my application, so I have no control over it. I cannot make it send a few bytes to unblock the server. It is just going to sit there and send no data. Since I do not have control of the Client implementation I cannot change anything on that end.
One solution would be to create a separate thread in which all ReadFile() operations are performed. That way when the event occurs, I can just process the code. The problem with that, is that the event also requires a separate thread, so now I have two additional threads for each instance of this server. Since this needs to be scalable, this is undesirable.
From another thread I have tried calling
DisconnectNamedPipe()
and
CloseHandle()
they both will not return (until the client writes to the pipe.)
I cannot connect to the same pipe and write a few bytes because:
"All instances of a named pipe share the same pipe name, but each instance has
its own buffers and handles, and provides a separate conduit for client/server
communication."
http://msdn.microsoft.com/en-us/library/aa365590.aspx
I need a way to fake it out, So the $64k dollar question is:
How can I break the blocking of ReadFile()?
Try this before ReadFile :
BOOL WINAPI PeekNamedPipe(
__in HANDLE hNamedPipe,
__out_opt LPVOID lpBuffer,
__in DWORD nBufferSize,
__out_opt LPDWORD lpBytesRead,
__out_opt LPDWORD lpTotalBytesAvail,
__out_opt LPDWORD lpBytesLeftThisMessage
);
if(TotalBytesAvail > 0)
ReadFile(....);
-AV-
Take a look on CancelSynchronousIo
Marks pending synchronous I/O
operations that are issued by the
specified thread as canceled.
And CancelIo/CancelIoEx:
To cancel all pending asynchronous I/O
operations, use either:
CancelIo — this function only cancels
operations issued by the calling
thread for the specified file handle.
CancelIoEx — this function cancels all
operations issued by the threads for
the specified file handle.
http://msdn.microsoft.com/en-us/library/aa363794(VS.85).aspx
http://msdn.microsoft.com/en-us/library/aa365467(VS.85).aspx
Mike,
You can't cancel synchronous ReadFile. But you can switch to asynchronous (overlapped) operations. By doing this, you can implement a pretty scalable architecture.
Possible algorithm (just an idea):
For each new client call ReadFile
WaitForMultipleObjects where the handles are overlapped.hEvent + your
custom events
Iterate over signalled events, and schedule them for execution by threads from a threads pool.
This way you can have only few threads to receive connections and read data, while the actual data processing can be done by the threads pool.
The problem with that, is that the
event also requires a separate thread,
so now I have two additional threads
for each instance of this server.
Since this needs to be scalable, this
is undesirable.
Never in my career have I found that "more threads" == "less scalable". How many of these "server" instances do you have?
Normally, an operation needs to be performed in a separate thread if that operation is going to block and the system needs to be responsive while the operation is blocked.
Asynchronous I/O operations do not have to block any thread if they use I/O Completion Ports. See: http://msdn.microsoft.com/en-us/library/aa365198(VS.85).aspx
What happening is the server outbound pipe is left open waiting for connection while your client is trying to connect to the server inbound pipe (which is no longer existent)... What you need to do is flush out your outbound pipe in order to loop back to your inbound. You can flush out on the client side by reading the file (remember to loop the connect establishment because there is a "handshake" in there, and it will never work the first time)
Just use SetNamedPipeHandleState function
https://learn.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-setnamedpipehandlestate
Use the PIPE_NOWAIT flag when calling this function.
hNamedPipe should be the handle that returned from CreateFile function.
After that, the call to ReadFile will not block the thread when no data available.