I have two detached threads in server: for receiving and sending data.
All work fine, until setting client's sending frequency to 16 ms (< ~200 ms). In this state one thread always wins the race and answers to only one client with ~1 us ping. What I need to do to send and receive data in separate threads with two (or one) UDP sockets?
Part of server's code:
void Server::receive() {
while(true) {
if (recvfrom(getReceiver()->getSocketDesc(), buffer, getBufferLength(), 0, (struct sockaddr *) ¤tClient, getReceiver()->getLength()) < 0) {
// add message to client's own thread-safe buffer
}
}
}
void Server::send() {
while(true) {
// get message from thread-safe general buffer after processing
// dequeue one
if (message != nullptr)
sendto(getDispatcher()->getSocketDesc(), "hi", 2, 0, message->_addr, *getDispatcher()->getLength());
}
}
Related
I am relatively new to network programming and multithreading in C++. Currently my recv() call returns an unknown error. I'm not quite sure where the error coming from at the moment and would appreciate some help.
I used putty to connect to the server locally
class Threaded_TCPListener{
int Threaded_TCPListener::Init()
{
// Initializing WinSock
WSADATA wsData;
WORD ver = MAKEWORD(2,2);
int winSock = WSAStartup(ver, &wsData);
if(winSock != 0)
return winSock;
// Creating listening socket
this->socket = ::socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if(this->socket == INVALID_SOCKET)
return WSAGetLastError();
// Fill sockaddr with ip addr and port
sockaddr_in hint;
hint.sin_family = AF_INET;
hint.sin_port = htons(this->port);
inet_pton(AF_INET, this->ipAddress, &hint.sin_addr);
// Bind hint to socket
if(bind(this->socket, (sockaddr*)&hint, sizeof(hint)) == SOCKET_ERROR)
return WSAGetLastError();
// Start listening on socket
if(listen(this->socket, SOMAXCONN) == SOCKET_ERROR)
return WSAGetLastError();
// Accept first client
this->createAcceptThread();
return 0;
}
int Threaded_TCPListener::Run()
{
bool isRunning = true;
// Read from all clients
std::vector<std::thread> threads;
threads.reserve(this->clients.size());
// Recv from client sockets
for (int i=0; i < this->clients.size(); ++i)
{
threads.emplace_back(std::thread(&Threaded_TCPListener::receiveFromSocket, this, socket));
}
// Wait for all threads to finish
for(std::thread& t : threads)
{
t.detach();
}
return 0;
}
void Threaded_TCPListener::onMessageReceived(int clientSocket, const char* msg, int length)
{
Threaded_TCPListener::broadcastToClients(clientSocket, msg, length);
std::thread t(&Threaded_TCPListener::receiveFromSocket, this, clientSocket);
t.detach();
return;
}
void Threaded_TCPListener::sendMessageToClient(int clientSocket, const char * msg, int length)
{
send(clientSocket, msg, length, 0);
return;
}
void Threaded_TCPListener::broadcastToClients(int senderSocket, const char * msg, int length)
{
std::vector<std::thread> threads;
threads.reserve(clients.size());
// Iterate over all clients
for (int sendSock : this->clients)
{
if(sendSock != senderSocket)
threads.emplace_back(std::thread(&Threaded_TCPListener::sendMessageToClient, this,sendSock, msg, length));
}
// Wait for all threads to finish
for(std::thread& t : threads)
t.join();
return;
}
void Threaded_TCPListener::createAcceptThread()
{
// Start accepting clients on a new thread
this->listeningThread = std::thread(&Threaded_TCPListener::acceptClient, this);
this->listeningThread.detach();
return;
}
void Threaded_TCPListener::acceptClient()
{
int client = accept(this->socket, nullptr, nullptr);
// Error
if(client == INVALID_SOCKET)
{
std::printf("Accept Err: %d\n", WSAGetLastError());
}
// Add client to clients queue
else
{
// Add client to queue
this->clients.emplace(client);
// Client Connect Confirmation
onClientConnected(client); // Prints msg on server
// Create another thread to accept more clients
this->createAcceptThread();
}
return;
}
void Threaded_TCPListener::receiveFromSocket(int receivingSocket)
{
// Byte storage
char buff[MAX_BUFF_SIZE];
// Clear buff
memset(buff, 0, sizeof(buff));
// Receive msg
int bytesRecvd = recv(receivingSocket, buff, MAX_BUFF_SIZE, 0);
if(bytesRecvd <= 0)
{
char err_buff[1024];
strerror_s(err_buff, bytesRecvd);
std::cerr << err_buff;
// Close client
this->clients.erase(receivingSocket);
closesocket(receivingSocket);
onClientDisconnected(receivingSocket); // Prints msg on server
}
else
{
onMessageReceived(receivingSocket, buff, bytesRecvd);
}
}
}
I am trying to create a multithreaded TCP 'server' that will handle incoming clients by having an accept thread continuously running (listening for new connections), and a thread waiting with a recv block for each client connected to the server.
Your Init looks fine:
create socket, bind it, listen on it, start accept thread
In your accept thread's acceptClient looks sort of OK:
print some message
add the client socket to clients queue
create a new accept thread
Your Run makes no sense:
create one thread per element in clients to receive from the listening socket
It looks like you are spawning a new thread for every single socket action. That is a pretty wasteful design. As soon as the thread is done it can go back to doing something else.
So creating a new accept thread in acceptClient is a waste, you could just loop back to the beginning to ::accept the next client. Like so:
acceptClient() {
while (alive) {
int client = accept(socket, ...);
createClientHandler(client);
}
}
What seems to be missing is spawning a new client thread to service the client socket. You currently do this in Run, but that's before any of the clients are actually accepted. And you do it for the wrong socket! Instead, you should be spawning the receiveFromSocket threads in acceptClient, and passing it the client socket. So that's a bug.
In your receiveFromSocket you also need not create another thread to receiveFromSocket again -- just loop back to the beginning.
The biggest concern with this thread-per-action design is that you are spawning sender threads on every incoming message. This means you could actually have several sender threads attempting to ::send on the same TCP socket. That's not very safe.
The order of calls made to WSASend is also the order in which the buffers are transmitted to the transport layer. WSASend should not be called on the same stream-oriented socket concurrently from different threads, because some Winsock providers may split a large send request into multiple transmissions, and this may lead to unintended data interleaving from multiple concurrent send requests on the same stream-oriented socket.
https://learn.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsasend
Similarly, instead of spawning threads in broadcastToClients, I suggest you just spawn one persistent sender thread per client socket in acceptClient (together with the receiveFromSocket thread within some createClientHandler).
To communicate with the sender threads you should use thread-safe blocking queues. Each sender thread would look like this:
while (alive) {
msg = queue.next_message();
send(client_socket, msg);
}
Then on message received you just do:
for (client : clients) {
client.queue.put_message(msg);
}
So to summarize, to handle each client you need a structure like this:
struct Client {
int client_socket;
BlockingQueue queue;
// optionally if you want to keep track of your threads
// to be able to safely clean up
std::thread recv_thread, send_thread;
};
Safe cleanup is a whole other story.
Finally, a remark on this comment in your code:
// Wait for all threads to finish
for(std::thread& t : threads)
{
t.detach();
}
That's almost the opposite to what std::thread::detach does:
https://en.cppreference.com/w/cpp/thread/thread/detach
It allows you to destroy the thread object without having to wait for the thread to finish execution.
There is a misconception in the code in how a TCP server has to be implemented:
You seem to assume that you can have a single server socket file descriptor which can handle all communication. This is not the case. You must have a single dedicated socket file descriptor which is just used for listening and accepting incoming connections, and then you have one additional file descriptor for each existing connection.
In your code I see that you invoke receiveFromSocket() always with the listening socket. This is wrong. Also invoking receiveFromSocket() in a loop for all clients is wrong.
What you rather need to do is:
- Have one dedicated thread which call accept() in a loop. There is no performance benefit in calling accept() from multiple threads.
- One accept() returns a new connection you spawn a new thread which calls recv() in a loop. This will then block and wait for new data as you expect in your question.
You also need to drop the habit of calling individual functions from new threads. This is not multithreaded programming. A thread usually contains a loop. Everything else is usually a design flaw.
Also note that multithreaded programming is still rocket science in 2019, especially in C++. If you are not an absolute expert you will not be able to do it. Also note that absolute experts in multithreaded programming will try to avoid multithreaded programming whenever possible. A lot seemingly concurrent tasks which are I/O bound can better be handled by a single threaded event based system.
I remake two detached threads to poll() cycle in server: for receiving and sending data.
All work fine, until setting client's sending frequency to 16 ms (< ~200 ms). In this state one thread always wins the race and answers to only one client with ~1 us ping. What I need to do to send and receive data in poll() with two (or one) UDP sockets?
Part of server's code:
struct pollfd pollStruct[2];
// int timeout_sec = 1;
pollStruct[0].fd = getReceiver()->getSocketDesc();
pollStruct[0].events = POLLIN;
pollStruct[0].revents = 0;
pollStruct[1].fd = getDispatcher()->getSocketDesc();
pollStruct[1].events = POLLOUT;
pollStruct[1].revents = 0;
while(true) {
if (poll(pollStruct, 2, 0 /* (-1 / timeout) */) > 0) {
if (pollStruct[0].revents & POLLIN) {
recvfrom(getReceiver()->getSocketDesc(), buffer, getBufferLength(), 0, (struct sockaddr *) ¤tClient, getReceiver()->getLength());
// add message to client's own thread-safe buffer
}
if (pollStruct[1].revents & POLLOUT) {
// get message from thread-safe general buffer after processing
// dequeue one
if (!getBuffer().isEmpty()) {
auto message = getBuffer().dequeue();
if (message != nullptr) {
sendto(getDispatcher()->getSocketDesc(), "hi", 2, 0, message->_addr, *getDispatcher()->getLength());
}
}
}
}
}
I implemented a program that receives from one socket and sends/receives from the other socket.
For this i use polling of select(), in socket 1, i receive data at a high data rate, while in the other socket i receive periodic message and requests to receive data from the first socket.
When there is no request "from socket 2" to delegate the data from socket 1 to socket2 , i receive data from socket 1 normal and with no problem. However, say i received two requests "socket 2" while data is being received in socket 1, the second request breaks the the data reception as if it could no longer keep up with rate "rate isn't high really is only 150 Hz".
The pseudo code i do in the main():
fd_set readfds, rd_fds, writefds, wr_fds;
struct timeval tv;
do
{
do
{
rd_fds = readfds;
wr_fds = writefds;
FD_ZERO (&rd_fds);
FD_SET (sock1, &rd_fds);
FD_SET (sock2, &rd_fds);
FD_SET (sock1, &wr_fds);
tv.tv_sec = 0;
tv.tv_usec = 20;
int ls = sock2 + 1;
rslt = select (ls, &rd_fds, &wr_fds, NULL, &tv);
}
while (rslt == -1 && errno == EINTR);
if (FD_ISSET (sock1, &rd_fds))
{
rs1 = recvfrom (sock1, buff, size of the buff, ....);
if (rs1 > 0)
{
if (rs1 = alive message)
{
/* system is alive; */
}
else if (rs1 == request message)
{
/* store Request info (list or vector) */
}
else {}
}
}
if (FD_ISSET (StructArg.sock2, &rd_fds))
{
rs2 = recv (sock2, ..., 0);
if (rs2 > 0)
{
if ( /* Message (high rate) is from sock 2 */ )
{
/* process this message and do some computation */
int sp1 = sendto (sock1, .....);
if (sp1 < 0)
{
perror ("Failed data transmission ");
}
else
{
/* increase some counters */
}
}
}
}
if (FD_ISSET (sock1, &wr_fds))
{
/*
if there info stored in the list
do some calculaitons then send to sock 1
*/
if (sendto (sock1, ... ...) < 0)
{
perror ("Failed data transmission");
}
else
{
/* increase counter */
}
}
FD_CLR (sock1, &rd_fds);
FD_CLR (sock2, &rd_fds);
}
while (1);
Again, the question is, why does receiving from sock1 is interrupted if a request is received from sock2, while i am receiving from sock1 (fast messages), i expect interleaved messages in the output based on the timestamps in the message.
Note that nearly all socket functions can block execution unless you've created the socket with the O_NONBLOCK option:
http://pubs.opengroup.org/onlinepubs/009695399/functions/sendto.html
And you'll also have to handle the case where recvfrom only gives you a partial read - unless you use MSG_WAITALL:
http://pubs.opengroup.org/onlinepubs/009695399/functions/recvfrom.html
Personally, I'd use a multi-threaded implementation which can have threads just sit and wait for data on each socket.
As to your final question:
why does receiving from sock1 is interrupted if a request is received from sock2, while i am receiving from sock1 (fast messages), i expect interleaved messages in the output based on the timestamps in the message.
You are slave to the network stack's implementation and there are nearly no guarantees about the sending or receiving of data on one socket relative to another. You are only guaranteed that the data within a socket is properly ordered.
I expect interleaved messages in the output based on the timestamps in the message.
Your expectation is without foundation. If there is data in either socket receive buffer, select() will fire. That's all you can rely on. You don't have any guarantee about timestamps being observed and ordered as between multiple sockets.
I am having trouble using the std::async to have tasks execute in parallel when the task involves a socket.
My program is a simple TCP socket server written in standard C++ for Linux. When a client connects, a dedicated port is opened and separate thread is started, so each client is serviced in their own thread.
The client objects are contained in a map.
I have a function to broadcast a message to all clients. I originally wrote it like below:
// ConnectedClient is an object representing a single client
// ConnectedClient::SendMessageToClient opens a socket, connects, writes, reads response and then closes socket
// broadcastMessage is the std::string to go out to all clients
// iterate through the map of clients
map<string, ConnectedClient*>::iterator nextClient;
for ( nextClient = mConnectedClients.begin(); nextClient != mConnectedClients.end(); ++nextClient )
{
printf("%s\n", nextClient->second->SendMessageToClient(broadcastMessage).c_str());
}
I have tested this and it works with 3 clients at a time. The message gets to all three clients (one at a time), and the response string is printed out three times in this loop. However, it is slow, because the message only goes out to one client at a time.
In order to make it more efficient, I was hoping to take advantage of std::async to call the SendMessageToClient function for every client asynchronously. I rewrote the code above like this:
vector<future<string>> futures;
// iterate through the map of clients
map<string, ConnectedClient*>::iterator nextClient;
for ( nextClient = mConnectedClients.begin(); nextClient != mConnectedClients.end(); ++nextClient )
{
printf("start send\n");
futures.push_back(async(launch::async, &ConnectedClient::SendMessageToClient, nextClient->second, broadcastMessage, wait));
printf("end send\n");
}
vector<future<string>>::iterator nextFuture;
for( nextFuture = futures.begin(); nextFuture != futures.end(); ++nextFuture )
{
printf("start wait\n");
nextFuture->wait();
printf("end wait\n");
printf("%s\n", nextFuture->get().c_str());
}
The code above functions as expected when there is only one client in the map. That you see "start send" quickly followed by "end send", quickly followed by "start wait" and then 3 seconds later (I have a three second sleep on the client response side to test this) you see the trace from the socket read function that the response comes in, and then you see "end wait"
The problem is that when there is more than one client in the map. In the part of the SendMessageToClient function that opens and connects to the socket, it fails in the code identified below:
// connected client object has a pipe open back to the client for sending messages
int clientSocketFileDescriptor;
clientSocketFileDescriptor = socket(AF_INET, SOCK_STREAM, 0);
// set the socket timeouts
// this part using setsockopt is omitted for brevity
// host name
struct hostent *server;
server = gethostbyname(mIpAddressOfClient.c_str());
if (server == 0)
{
close(clientSocketFileDescriptor);
return "";
}
//
struct sockaddr_in clientsListeningServerAddress;
memset(&clientsListeningServerAddress, 0, sizeof(struct sockaddr_in));
clientsListeningServerAddress.sin_family = AF_INET;
bcopy((char*)server->h_addr, (char*)&clientsListeningServerAddress.sin_addr.s_addr, server->h_length);
clientsListeningServerAddress.sin_port = htons(mPortNumberClientIsListeningOn);
// The connect function fails !!!
if ( connect(clientSocketFileDescriptor, (struct sockaddr *)&clientsListeningServerAddress, sizeof(clientsListeningServerAddress)) < 0 )
{
// print out error code
printf("Connected client thread: fail to connect %d \n", errno);
close(clientSocketFileDescriptor);
return response;
}
The output reads: "Connected client thread: fail to connect 4".
I looked this error code up, it is explained thus:
#define EINTR 4 /* Interrupted system call */
I searched around on the internet, all I found were some references to system calls being interrupted by signals.
Does anyone know why this works when I call my send message function one at a time, but it fails when the send message function is called using async? Does anyone have a different suggestion how I should send a message to multiple clients?
First, I would try to deal with the EINTR issue. connect ( ) has been interrupted (this is the meaning of EINTR) and does not try again because you are using and asynch descriptor.
What I usually do in such a circumstance is to retry: I wrap the function (connect in this case) in a while cycle. If connect succeeds I break out of the cycle. If it fails, I check the value of errno. If it is EINTR I try again.
Mind that there are other values of errno that deserve a retry (EWOULDBLOCK is one of them)
I am wondering if there is a way to clean up window socket internal buffer, because what I want to achieve is this
while(1){
for(i=0;i<10;i++){
sendto(...) //send 10 UDP datagrams
}
for(i=0;i<10;i++){
recvfrom (Socket, RecBuf, MAX_PKT_SIZE, 0,
(SOCKADDR*) NULL, NULL);
int Status = ProcessBuffer(RecBuf);
if (Status == SomeCondition)
MagicalSocketCleanUP(Socket); //clean up the rest of stuff in the socket, so that it doesn't effect the reading for next iteration of the outer while loop
break; //occasionally the the receive loop needs to terminate before finishing off all 10 iteration
}
}
so I am asking for is there a function to clean up whatever remaining in the socket so that it won't effect my next reading? Thank you
The way to clean up data from the internal receive socket buffer is to read data until there is no more data to read. If you do this in a non-blocking way, you do not need to wait for more data in select(), because the EWOUDBLOCK error value means the internal receive socket buffer is empty.
int MagicalSocketCleanUP(SOCKET Socket) {
int r;
std::vector<char> buf(128*1024);
do {
r = recv(Socket, &buf[0], buf.size(), MSG_DONTWAIT);
if (r < 0 && errno == EINTR) continue;
} while (r > 0);
if (r < 0 && errno != EWOULDBLOCK) {
perror(__func__);
//... code to handle unexpected error
}
return r;
}
But this is not exactly safe. The other end of the socket may have sent good data into the socket buffer too, so this routine may discard more than what you want to discard.
Instead, the data on the socket should be framed in such a way that you know when the data of interest arrives. So instead of a cleanup API, you could extend ProcessBuffer() to discard input until it finds data of interest.
A simpler mechanism would be a message exchange between the two sides of the socket. When the error state is entered, the sender sends a "DISCARDING UNTIL <TOKEN>" message. The receiver sends back "<TOKEN>" and knows that only the data after the "<TOKEN>" message will be processed. The "<TOKEN>" can be a random sequence.