In my program there is one thread (receiving thread) that is responsible for receiving requests from a TCP socket and there are many threads (worker threads) that are responsible for processing the received requests. Once a request is processed I need to send an answer over TCP.
And here is a question. I would like to send TCP data in the same thread that I use for receiving data. This thread after receiving data usually waits for new data in select(). So once a worker thread finished processing a request and put an answer in the output queue it has to signal the receiving thread that there are data to send. The problem is that I don't know how to cancel waiting in select() in order to get out of waiting and to call send() .
Or shall I use another thread solely for sending data over TCP?
Updated
MSalters, Artyom thank you for you answers!
MSalters, having read your answer I found this site: Winsock 2 I/O Methods and read about WSAWaitForMultipleEvents(). My program in fact must work both on HP-UX and Windows I finally decided to use the approach that had been suggested by Artyom.
You need to use something similar to safe-pipe trick, but in your case you need to use a pair of connected TCP sockets.
Create a pair of sockets.
Add one to the select and wait on it as well
Notify by writing to other socket from other threads.
Select is immediately waken-up as one of the sockets is readable, reads all the
data in this special socket and check all data in queues to send/recv
How to create pair of sockets under Windows?
inline void pair(SOCKET fds[2])
{
struct sockaddr_in inaddr;
struct sockaddr addr;
SOCKET lst=::socket(AF_INET, SOCK_STREAM,IPPROTO_TCP);
memset(&inaddr, 0, sizeof(inaddr));
memset(&addr, 0, sizeof(addr));
inaddr.sin_family = AF_INET;
inaddr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
inaddr.sin_port = 0;
int yes=1;
setsockopt(lst,SOL_SOCKET,SO_REUSEADDR,(char*)&yes,sizeof(yes));
bind(lst,(struct sockaddr *)&inaddr,sizeof(inaddr));
listen(lst,1);
int len=sizeof(inaddr);
getsockname(lst, &addr,&len);
fds[0]=::socket(AF_INET, SOCK_STREAM,0);
connect(fds[0],&addr,len);
fds[1]=accept(lst,0,0);
closesocket(lst);
}
Of course some checks should be added for return values.
select is not the native API for Windows. The native way is WSAWaitForMultipleEvents. If you use this to create an alertable wait, you can use QueueUserAPC to instruct the waiting thread to send data. (This might also mean you don't have to implement your own output queue)
See also this post:
How to signal select() to return immediately?
For unix, use an anonymous pipe. For Windows:
Unblocking can be achieved by adding a dummy (unbound) datagram socket to fd_set and then closing it. To make this thread safe, use QueueUserAPC:
The only way I found to make this multi-threadsafe is to close and recreate the socket in the same thread as the select statement is running. Of course this is difficult if the thread is blocking on the select. And then comes in the windows call QueueUserAPC. When windows is blocking in the select statement, the thread can handle Asynchronous Procedure Calls. You can schedule this from a different thread using QueueUserAPC. Windows interrupts the select, executes your function in the same thread, and continues with the select statement. You can now in your APC method close the socket and recreate it. Guaranteed thread safe and you will never loose a signal.
The typical model is for the worker to handle its own writing. Is there a reason why you want to send all the output-IO through selecting thread?
If you're sure of this model, you could have your workers send data back to the master thread using file descriptors as well (pipe(2)) and simply add those descriptors to your select() call.
And, if you're especially sure that you're not going to use pipes to send data back to your master process, the select call allows you to specify a timeout. You can busy-wait while checking your worker threads, and periodically call select to figure out which TCP sockets to read from.
Another quick&dirty solution is to add localhost sockets to the set. Now use those sockets as the inter-thread communication queues. Each worker thread simply sends something to its socket, which ends up on the corresponding socket in your receiving thread. This wakes up the select(), and your receiving thread can then echo the message on the appropriate outgoing socket.
Related
I want a server to accept a client with this code
SOCKADDR_IN cliaddr = { 0 };
int len = sizeof(cliaddr);
_sockClient = accept(_sockServer, (SOCKADDR*)&cliaddr, &len);
Both _sockClient and _sockServer are SOCKETs, _sockServer's SOCKET already connected to the server.
Before I accept, I want to check _sockServer if there is an incoming client request to join. I'm pretty sure there is a way to do this, though I don't know how. Does anyone know?
By default, a socket operates in blocking mode. So, you could simply call accept() and let it block the calling thread until a new connection is accepted. That is the easiest approach. If you don't want to block the calling thread, you could call accept() in another thread instead.
But, provided you don't want to block any thread at all, then yes, there are other ways to handle this, depending on your programming model:
Poll the socket using the readfds parameter of select(). The socket will be in a readable state if it has any pending connections waiting to be accepted.
Use WSAEventSelect() to signal a waitable WSAEVENT object whenever an FD_ACCEPT event occurs, indicating that pending connections are available to accept. You can then poll/wait on that object using WSAWaitForMultipleEvents(), and use WSAEnumNetworkEvents() to reset the object's state for the next wait.
Use WSAAsyncSelect() to receive a window message whenever an FD_ACCEPT event occurs.
Use AcceptEx() instead of accept() to start an asynchronous acceptance in the background. It will report its completion to you via Overlapped I/O or an I/O Completion Port.
See Overlapped I/O and Event Objects in Winsock's documentation for more details.
Say that I have two threads, the main thread and a thread that is dedicated to continuously listening on a standard TCP socket. Now, say that at some point I want to shutdown everything. From the main thread, I would like to close the connection the listening thread is working on, then join the thread and end the program.
However, this is tricky, since I don't know how to make the listening thread return from the call to read. That call won't return unless something actually is received, and in principle I could be waiting for a long long time until the other endpoint decides to send me something.
When I was working with UDP sockets, I used to work around this problem by sending a packet on that port from my loopback interface, therefore triggering a return from recvfrom. However, this is terribly inelegant and it cannot be done on a TCP socket.
I know that another workaround could be to set a timeout with setsockopt: in this way I am guaranteed that the call will eventually return, but this is inelegant as well, and also quite inefficient since I could be waiting for several seconds before being able to join the thread.
So I was wondering if there is some way to trigger an EAGAIN on a socket read call, not unlike the one I would get on a timeout, so that on my main thread I could just call some force_returnon my socket descriptor and the call to read on the other thread would return?
I usually solve this problem by creating a pipe() and using select() in the reading thread. The reading thread must select on both the TCP socket and one end of the pipe. Whenever you want to shut down the reader, set a flag and write some data to the other end of the pipe.
Setup:
#include <unistd.h>
int signalPipe[2];
...
pipe(signalPipe);
Reader:
while(running)
{
FD_ZERO(&fds);
FD_SET(tcpSocket, &fds);
FD_SET(signalPipe[0], &fds);
select(max(tcpSocket, signalPipe[0]) + 1, &fds, NULL, NULL, NULL);
...
}
Other thread:
// We want to stop now.
running = false;
write(signalPipe[1], "foo", 3);
BSD stream sockets are full duplex, meaning two connected parties can both send/receive at the same time.
A QTcpSocket (qt socket implementation) has asynchronous support, non blocking mode, but can only belong to one thread, see qt docs.
Event driven objects may only be used in a single thread.
Specifically, this applies to the timer mechanism and the network
module.
Let's say I want a transmit/tx thread and a separate receive/rx thread to use the same socket and send/receive data at the same time.
In my understanding this can be 'done' via qt signals/slots, but the socket thread will never really perform the send() and the receive() simultaneously. It just runs the event loop which will do this in a serial fashion and emit the signals when send/receive is done.
Yes, my rx and tx threads can work concurrently and handle the notifications via qt slots, but the socket itself is never really used in full duplex mode.
Is it correct to say that: considering one endpoint only, in the socket thread, its send() and receive() calls are always serial, never simultaneous?
(because the event loop thread is one thread only)
In my understanding this can be 'done' via qt signals/slots, but the
socket thread will never really perform the send() and the receive()
simultaneously. It just runs the event loop which will do this in a
serial fashion and emit the signals when send/receive is done.
True, but keep in mind that the kernel buffers incoming and outgoing data, and QTCPSocket sets the socket to non-blocking, so that the send() and recv() calls always return immediately and never block the event-loop. That means that the actual processes of sending and receiving data will happen simultaneously (inside the kernel), even if the (more-or-less instantaneous) send() and recv() calls technically do not. (*)
Yes, my rx and tx threads can work concurrently and handle the
notifications via qt slots, but the socket itself is never really used
in full duplex mode. Is this correct?
That is not correct -- the socket's data streams can (and do) flow both ways across the network simultaneously, so the socket really is full-duplex. The full-duplex capability is present whether you are using a single thread or multiple threads.
(*) You can test this with a single-threaded Qt program that uses a QTCPSocket to send or receive data, by simply disconnecting your computer's Ethernet cable during a large data transfer. If the QTCPSocket's send() or recv() calls are blocking until completion, that would block the GUI thread and cause your GUI to become unresponsive until you reconnect the cable (or until the TCP connection times out after several minutes).
Hi I'm working on a networking project. I've a socket that is listening incoming data. Now I want to archive this: Socket will receive only 100 packets. And there is 3-4 clients. They are sending random data packets infinitely. I'll receive 100 packets and later I'll process them. After process I'll re-start receiving. But at this time there are some pending send() >> recv() operations. Now I want to cancel/discard pending recv operations. I think we'll recv datas and we'll not process them. Any other suggestions? (sorry for bad question composition)
Shutdown and close the connection. That will cancel everything immediately.
Better yet, rearchitect your application and network protocol so that you can reliably tell how much data to receive.
On Windows you can cancel outstanding receives using CancelIO, but that might result in lost data if the receive just happened to read something.
You can use select() or poll() loops.
you can use signal. recv() will return on receiving a signal so you can send a signal from another task to the task that blocks on recv(). But you need to make sure you don't specify SA_RESTART (see http://pubs.opengroup.org/onlinepubs/9699919799/functions/sigaction.html)
Read http://en.wikipedia.org/wiki/Asynchronous_I/O for more details
I would go with non-blocking sockets + cancellation socket.
You'll have to read into dedicated incremental buffer (as recv() may not receive all the data expected at once - this would be the case if you can only process full messages) and return to select()/poll() in your loop, where you can safely sit and wait for:
next data
next connection
cancellation event from a cancellation socket, to which your other thread will send a cancellation signal (some trivial send()).
UPD: the trivial event may be the number of the socket in the array or its handle - something to identify which one you'd like to cancel.
I have a listening thread that waiting for reading on few socket using select and FD_SET. The story is. At some point I will add another socket to the pool and need to abort select and re-initialize FD_SET array fo select. I have an Event to signal pool changes. But how select can react to my Event?
select() at this point of time use timeval with waiting interval of 20 sec and I don't want to changed time to lower value. I don't want frequently re-start select() by timeout...
Is there any way to abort select? What would be the right approach to inform/restart select and force using of new list of socket(at least one socket will be added to pool)
And another question - Msdn says "The select function determines the status of one or more sockets, waiting if necessary, to perform synchronous I/O."
Does that mean that select is not designed to work with sockets that turned to use using async operation?
Use WSAEventSelect() and WSAWaitForMultipleEvents() instead of select(). That way, your pool can create a separate event with WSACreateEvent() and signal it with WSASetEvent() to wake up WSAWaitForMultipleEvents() when needed.
If you want select() to wake up, the easiest way is to send a byte to one of the sockets that select() is waiting on for read access. One way to implement that without affecting functionality of the existing sockets is to create a pair of sockets specifically for that purpose and connect() one to the other.