Say that I have two threads, the main thread and a thread that is dedicated to continuously listening on a standard TCP socket. Now, say that at some point I want to shutdown everything. From the main thread, I would like to close the connection the listening thread is working on, then join the thread and end the program.
However, this is tricky, since I don't know how to make the listening thread return from the call to read. That call won't return unless something actually is received, and in principle I could be waiting for a long long time until the other endpoint decides to send me something.
When I was working with UDP sockets, I used to work around this problem by sending a packet on that port from my loopback interface, therefore triggering a return from recvfrom. However, this is terribly inelegant and it cannot be done on a TCP socket.
I know that another workaround could be to set a timeout with setsockopt: in this way I am guaranteed that the call will eventually return, but this is inelegant as well, and also quite inefficient since I could be waiting for several seconds before being able to join the thread.
So I was wondering if there is some way to trigger an EAGAIN on a socket read call, not unlike the one I would get on a timeout, so that on my main thread I could just call some force_returnon my socket descriptor and the call to read on the other thread would return?
I usually solve this problem by creating a pipe() and using select() in the reading thread. The reading thread must select on both the TCP socket and one end of the pipe. Whenever you want to shut down the reader, set a flag and write some data to the other end of the pipe.
Setup:
#include <unistd.h>
int signalPipe[2];
...
pipe(signalPipe);
Reader:
while(running)
{
FD_ZERO(&fds);
FD_SET(tcpSocket, &fds);
FD_SET(signalPipe[0], &fds);
select(max(tcpSocket, signalPipe[0]) + 1, &fds, NULL, NULL, NULL);
...
}
Other thread:
// We want to stop now.
running = false;
write(signalPipe[1], "foo", 3);
Related
I have a loop which basically calls this every few seconds (after the timeout):
while(true){
if(finished)
return;
switch(select(FD_SETSIZE, &readfds, 0, 0, &tv)){
case SOCKET_ERROR : report bad stuff etc; return;
default : break;
}
// do stuff with the incoming connection
}
So basically for every few seconds (which is specified by tv), it reactivates the listening.
This is run on thread B (not a main thread). There are times when I want to end this acceptor loop immediately from thread A (main thread), but seems like I have to wait until the time interval finishes..
Is there a way to disrupt the select function from another thread so thread B can quit instantly?
The easiest way is probably to use pipe(2) to create a pipe and add the read end to readfds. When the other thread wants to interrupt the select() just write a byte to it, then consume it afterward.
Yes, you create a connected pair of sockets. Then thread B writes to one side of socket and thread A adds the other side socket to select. So once B writes to socket A exits select, do not forget to read this byte from socket.
This is the most standard and common way to interrupt selects.
Notes:
Under Unix, use socketpair to create a pair of sockets, under windows it is little bit tricky but googling for Windows socketpair would give you samples of code.
Can't you just make the timeout sufficiently short (like 10ms or so?).
These "just create a dummy connection"-type solution seem sort of hacked. I personally think that if an application is well designed, concurrent tasks never have to be interrupted forcefully, the just has worker check often enough (this is also a reason why boost.threads do not have a terminate function).
Edit Made this answer CV. It is bad, but it might help other to understand why it is bad, which is explained in the comments.
You can use shutdown(Sock, SHUT_RDWR) call from main thread to come out of waiting select call which will also exit your another thread before the timeout so you don't need to wait till timeout expires.
cheers. :)
I have a Windows service written in C++ that functions as a TCP server listening for incoming connections.
I initialized the server socket and put the accept code in a separate thread. This will accept and process the incoming connections.
However, I also need to stop this thread in case the service receives the STOP signal. So I thought of creating an event object using CreateEvent and waiting for it to be signaled. This waiting would happen in the thread that creates the accept thread. So I could use the TerminateThread function to stop the accept thread when the STOP signal is received.
However, MSDN says that
TerminateThread is a dangerous function that should only be used in the most extreme cases.
How strictly should this be followed and is my approach correct? What could be another way of doing this?
In Windows, you can wake up a blocking accept call from another thread simply by calling closesocket. The blocking accept call will return -1 and your code has a chance to break out of whatever loop it is in by checking some other exit condition that you have already set (e.g. global variable)
This also works with Mac (and likely BSD derivatives) with the close function, but not Linux. The more universal UNIX solution to this problem is here.
Some pseduo code for the Windows solution below.
SOCKET _listenSocket;
bool _needToExit = false;
HANDLE _hThread;
void MakeListenThreadExit()
{
_needToExit = true;
closesocket(_listenSocket);
_listenSocket = INVALID_SOCKET;
// wait for the thread to exit
WaitForSingleObject(_hThread, INFINITE);
}
DWORD __stdcall ListenThread(void* context)
{
while (_needToExit == false)
{
SOCKET client = accept(_listenSocket, (sockaddr*)&addr, &addrSize);
if ((client == -1) || _needToExit)
{
break;
}
ProcessClient(client);
}
return 0;
}
In this situation, don't use accept() on a blocking socket. Use a non-blocking socket instead. Then you can use select() with a timeout so your thread can check for a termination condition periodically. Or better, use WSACreateEvent() with WSASelectEvent(). Create two event objects, one to detect client connections, and one to detect thread termination. You can then use WSAWaitForMultipleEvents() to wait on both events at the same time. Use WSASetEvent() to signal the termination event when needed, and call accept() or WSAAccept() whenever the other event is signalled. WSAWaitForMultipleEvents() will tell you which event to act on.
The WaitNamedPipe function allows a pipe client application to synchronously wait for an available connection on a named pipe server. You then call CreateFile to open the pipe as a client. Pseudocode:
// loop works around race condition with WaitNamedPipe and CreateFile
HANDLE hPipe;
while (true) {
if (WaitNamedPipe says connection is ready) {
hPipe = CreateFile(...);
if (hPipe ok or last error is NOT pipe busy) {
break; // hPipe is valid or last error is set
}
} else {
break; // WaitNamedPipe failed
}
}
The problem is that these are all blocking, synchronous calls. What is a good way to do this asynchronously? I can't seem to find an API that uses overlapped I/O to do this, for example. For example, for pipe servers the ConnectNamedPipe function provides an lpOverlapped parameters allowing for a server to asynchronously wait for a client. The pipe server can then call WaitForMultipleObjects and wait for the I/O operation to complete, or any other event to be signaled (for example, an event signaling the thread to cancel pending I/O and terminate).
The only way I can think of is to call WaitNamedPipe in a loop with a short, finite timeout and check other signals if it times out. Alternatively, in a loop call CreateFile, check other signals, and then call Sleep with a short delay (or WaitNamedPipe). For example:
HANDLE hPipe;
while (true) {
hPipe = CreateFile(...);
if (hPipe not valid and pipe is busy) {
// sleep 100 milliseconds; alternatively, call WaitNamedPipe with timeout
Sleep(100);
// TODO: check other signals here to see if we should abort I/O
} else
break;
}
But this method stinks to high heaven in my opinion. If a pipe isn't available for awhile, the thread continues to run - sucking up CPU, using power, requiring memory pages to remain in RAM, etc. In my mind, a thread that relies on Sleep or short timeouts does not perform well and is a sign of sloppy multi-threaded programming.
But what's the alternative in this case?
WaitNamedPipe is completely useless, and will just use all the cpu if you specify a timeout and there's no server waiting for it.
Just call CreateFile over and over with a Sleep like you're doing, and move it to other threads as you see appropriate. There is no API alternative.
The only "benefit" WaitNamedPipe provides is if you want to know if you can connect to a named pipe but you explicitly don't want to actually open a connection. It's junk.
If you really want to be thorough, your only options are
Ensure that whatever program is opening the named pipe is always calling CreateNamedPipe again immediately after it's named pipe is connected to.
Have your program actually check if that program is running.
If your intent is really not to have additional connections, still call CreateNamedPipe, and when someone connects, tell them to go away until they're waited a given amount of time, the close the pipe.
Why can't the server just create more pipes? The performance hit in the scenario you describe isn't a problem if it is rare.
I.e. if there are usually enough pipes to go round what does it matter if you use CreateFile/Sleep instead of WaitForMultipleObjects? The performance hit will not matter.
I also have to question the need for overlapped IO in a client. How many servers is it communicating with at a time? If the answer is less than, say, 10 you could reasonably create a thread per connection.
Basically I am saying I think the reason there is no overlapped WaitforNamedPipe is because there is no reasonable use-case which requires it.
You can open the pipe file system at \\.\pipe\ and then use DeviceIoControl to send FSCTL_PIPE_WAIT.
I have a loop which basically calls this every few seconds (after the timeout):
while(true){
if(finished)
return;
switch(select(FD_SETSIZE, &readfds, 0, 0, &tv)){
case SOCKET_ERROR : report bad stuff etc; return;
default : break;
}
// do stuff with the incoming connection
}
So basically for every few seconds (which is specified by tv), it reactivates the listening.
This is run on thread B (not a main thread). There are times when I want to end this acceptor loop immediately from thread A (main thread), but seems like I have to wait until the time interval finishes..
Is there a way to disrupt the select function from another thread so thread B can quit instantly?
The easiest way is probably to use pipe(2) to create a pipe and add the read end to readfds. When the other thread wants to interrupt the select() just write a byte to it, then consume it afterward.
Yes, you create a connected pair of sockets. Then thread B writes to one side of socket and thread A adds the other side socket to select. So once B writes to socket A exits select, do not forget to read this byte from socket.
This is the most standard and common way to interrupt selects.
Notes:
Under Unix, use socketpair to create a pair of sockets, under windows it is little bit tricky but googling for Windows socketpair would give you samples of code.
Can't you just make the timeout sufficiently short (like 10ms or so?).
These "just create a dummy connection"-type solution seem sort of hacked. I personally think that if an application is well designed, concurrent tasks never have to be interrupted forcefully, the just has worker check often enough (this is also a reason why boost.threads do not have a terminate function).
Edit Made this answer CV. It is bad, but it might help other to understand why it is bad, which is explained in the comments.
You can use shutdown(Sock, SHUT_RDWR) call from main thread to come out of waiting select call which will also exit your another thread before the timeout so you don't need to wait till timeout expires.
cheers. :)
In my program there is one thread (receiving thread) that is responsible for receiving requests from a TCP socket and there are many threads (worker threads) that are responsible for processing the received requests. Once a request is processed I need to send an answer over TCP.
And here is a question. I would like to send TCP data in the same thread that I use for receiving data. This thread after receiving data usually waits for new data in select(). So once a worker thread finished processing a request and put an answer in the output queue it has to signal the receiving thread that there are data to send. The problem is that I don't know how to cancel waiting in select() in order to get out of waiting and to call send() .
Or shall I use another thread solely for sending data over TCP?
Updated
MSalters, Artyom thank you for you answers!
MSalters, having read your answer I found this site: Winsock 2 I/O Methods and read about WSAWaitForMultipleEvents(). My program in fact must work both on HP-UX and Windows I finally decided to use the approach that had been suggested by Artyom.
You need to use something similar to safe-pipe trick, but in your case you need to use a pair of connected TCP sockets.
Create a pair of sockets.
Add one to the select and wait on it as well
Notify by writing to other socket from other threads.
Select is immediately waken-up as one of the sockets is readable, reads all the
data in this special socket and check all data in queues to send/recv
How to create pair of sockets under Windows?
inline void pair(SOCKET fds[2])
{
struct sockaddr_in inaddr;
struct sockaddr addr;
SOCKET lst=::socket(AF_INET, SOCK_STREAM,IPPROTO_TCP);
memset(&inaddr, 0, sizeof(inaddr));
memset(&addr, 0, sizeof(addr));
inaddr.sin_family = AF_INET;
inaddr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
inaddr.sin_port = 0;
int yes=1;
setsockopt(lst,SOL_SOCKET,SO_REUSEADDR,(char*)&yes,sizeof(yes));
bind(lst,(struct sockaddr *)&inaddr,sizeof(inaddr));
listen(lst,1);
int len=sizeof(inaddr);
getsockname(lst, &addr,&len);
fds[0]=::socket(AF_INET, SOCK_STREAM,0);
connect(fds[0],&addr,len);
fds[1]=accept(lst,0,0);
closesocket(lst);
}
Of course some checks should be added for return values.
select is not the native API for Windows. The native way is WSAWaitForMultipleEvents. If you use this to create an alertable wait, you can use QueueUserAPC to instruct the waiting thread to send data. (This might also mean you don't have to implement your own output queue)
See also this post:
How to signal select() to return immediately?
For unix, use an anonymous pipe. For Windows:
Unblocking can be achieved by adding a dummy (unbound) datagram socket to fd_set and then closing it. To make this thread safe, use QueueUserAPC:
The only way I found to make this multi-threadsafe is to close and recreate the socket in the same thread as the select statement is running. Of course this is difficult if the thread is blocking on the select. And then comes in the windows call QueueUserAPC. When windows is blocking in the select statement, the thread can handle Asynchronous Procedure Calls. You can schedule this from a different thread using QueueUserAPC. Windows interrupts the select, executes your function in the same thread, and continues with the select statement. You can now in your APC method close the socket and recreate it. Guaranteed thread safe and you will never loose a signal.
The typical model is for the worker to handle its own writing. Is there a reason why you want to send all the output-IO through selecting thread?
If you're sure of this model, you could have your workers send data back to the master thread using file descriptors as well (pipe(2)) and simply add those descriptors to your select() call.
And, if you're especially sure that you're not going to use pipes to send data back to your master process, the select call allows you to specify a timeout. You can busy-wait while checking your worker threads, and periodically call select to figure out which TCP sockets to read from.
Another quick&dirty solution is to add localhost sockets to the set. Now use those sockets as the inter-thread communication queues. Each worker thread simply sends something to its socket, which ends up on the corresponding socket in your receiving thread. This wakes up the select(), and your receiving thread can then echo the message on the appropriate outgoing socket.