C++ Multithreaded TCP Server Issue - c++

I'm writing a simple TCP server. The model that I though of is the server accepting client connections in the main thread and handing them over to another thread so that server can listen for connections again. The relevant parts of the code that I used are posted below:
Accepting connections:
void startServer () {
int serverSideSocket = 0;
int clientSideSocket = 0;
serverSideSocket = socket(AF_INET, SOCK_STREAM, 0);
if (serverSideSocket < 0) {
error("ERROR opening socket");
exit(1);
}
clientAddressLength = sizeof(clientAddress);
memset((char *) &serverAddress, 0, sizeof(serverAddress));
memset((char *) &clientAddress, 0, clientAddressLength);
serverAddress.sin_family = AF_INET;
serverAddress.sin_addr.s_addr = INADDR_ANY;
serverAddress.sin_port = htons(32000);
if (bind(serverSideSocket, (struct sockaddr *) &serverAddress, sizeof(serverAddress)) < 0) {
error("ERROR on binding");
exit(1);
}
listen(serverSideSocket, SOMAXCONN);
while(true) {
clientSideSocket = accept(serverSideSocket, (struct sockaddr *) &clientAddress, &clientAddressLength);
if (clientSideSocket < 0)
error("ERROR on accept");
processingThreadGroup->create_thread(boost::bind(process, clientSideSocket, this));
}
}
Here, the processingThreadGroup is a boost::thread_group instance. In the process method:
void process (int clientSideSocket, DataCollector* collector) {
int numberOfCharactersRead = 0;
string buffer;
do {
char msgBuffer[1000];
numberOfCharactersRead = recv(clientSideSocket, msgBuffer, (1000 - 1), 0);
if (numberOfCharactersRead < 0) {
//display error
close(clientSideSocket);
}
else if (numberOfCharactersRead == 0)
close(clientSideSocket);
else {
printf("%s", msgBuffer);
memset(msgBuffer, 0, 1000);
}
} while (numberOfCharactersRead > 0);
}
However, when I debug the code, I saw that when the processing thread is invoked, the main thread is not accepting connections anymore. The data is read inside the process() method only. The main thread seem to be not running anymore. What is the issue with the approach I took and any suggestions to correct it?
EDIT: I think I found the issue here, and have updated it as an answer. Will not accept it since I answered my own question. Thank you for the help everyone!

Think I found the issue. I was using this as a server to accept syslog messages. The code I use for the syslog message generator is as follows:
openlog ("MyProgram", LOG_CONS | LOG_PID | LOG_NDELAY, LOG_LOCAL0);
cout << "opened the log" << endl;
for (int i = 0 ; i < 10 ; i++)
{
syslog (LOG_INFO, "Program started by User %d \n", getuid ());
syslog (LOG_WARNING, "Beware of the WARNING! \n");
syslog (LOG_ERR, "fatal ERROR! \n");
}
closelog ();
cout << "closed the log" << endl;
and I use an entry in the rsyslog.conf file to direct all syslog LOG_LOCAL0 application traffic to be sent to the relevant TCP port where the server is listening. Somehow, syslog allows only one connection to be made, not multiple connections. Therefore, it only used one connection in a single thread. If that connection was closed, a new connection is craeted.
I checked with a normal tcp client. That works fine, with multiple threads being spawned for each connection accepted.

Related

bind() fails occasionally with Address already in use even after setting SO_REUSEADDR and SO_REUSEPORT

We have a streaming server implemented using live555 library. This server is deployed on CentOS instance.
Recently we wanted to modify the server socket options, so that it can accept the requests immediately after re-starting the process (either after crash or manual restart).
I have referred to man pages and few web links and set the socket options (SO_REUSEADDR and SO_REUSEPORT) before bind() call.
int setupStreamSocket(UsageEnvironment& env,
Port port, Boolean makeNonBlocking) {
int newSocket = createSocket(SOCK_STREAM);
if (newSocket < 0) {
socketErr(env, "unable to create stream socket: ");
return newSocket;
}
int reuseFlag = 1;
fprintf(stderr,"reuseFlag : %d\n",reuseFlag);
if (setsockopt(newSocket, SOL_SOCKET, SO_REUSEADDR,
(const char*)&reuseFlag, sizeof reuseFlag) < 0) {
socketErr(env, "setsockopt(SO_REUSEADDR) error: ");
closeSocket(newSocket);
return -1;
}
if (setsockopt(newSocket, SOL_SOCKET, SO_REUSEPORT,
(const char*)&reuseFlag, sizeof reuseFlag) < 0) {
socketErr(env, "setsockopt(SO_REUSEPORT) error: ");
closeSocket(newSocket);
return -1;
}
int flag = 1;
setsockopt( newSocket, IPPROTO_TCP, TCP_NODELAY, &flag, sizeof(flag) );
if (port.num() != 0 || ReceivingInterfaceAddr != INADDR_ANY) {
MAKE_SOCKADDR_IN(name, ReceivingInterfaceAddr, port.num());
int reuse_addr_val, reuse_port_val;
socklen_t reuse_addr_len = sizeof(reuse_addr_val);
socklen_t reuse_port_len = sizeof(reuse_port_val);
getsockopt(newSocket, SOL_SOCKET, SO_REUSEADDR, &reuse_addr_val, &reuse_addr_len);
getsockopt(newSocket, SOL_SOCKET, SO_REUSEPORT, &reuse_port_val, &reuse_port_len);
fprintf(stderr,"reuse_addr_val = %d, reuse_port_val = %d\n", reuse_addr_val, reuse_port_val);
if (bind(newSocket, (struct sockaddr*)&name, sizeof name) != 0) {
char tmpBuffer[100];
sprintf(tmpBuffer, "bind() error (port number: %d): ",
ntohs(port.num()));
socketErr(env, tmpBuffer);
closeSocket(newSocket);
return -1;
}
}
if (makeNonBlocking) {
if (!makeSocketNonBlocking(newSocket)) {
socketErr(env, "failed to make non-blocking: ");
closeSocket(newSocket);
return -1;
}
}
return newSocket;
}
This code works as expected (binding to the address even when a socket is in TIME_WAIT state) if I restart the server with above options.
If I replace previous build (without socket options) with the build created with above code then I have noticed that occasionally bind() is failing with Address already in use error.
When the bind() failed the port/address is in TIME_WAIT state. So the server should able to bind the socket to the address.
tcp 0 0 10.0.1.24:8554 10.0.1.89:27085 TIME_WAIT -
And the getsockopt() of my code printed the flag value (corresponding to SO_REUSEADDR and SO_REUSEPORT) as 1.
reuse_addr_val = 1, reuse_port_val = 1
bind() error (port number: 8554): Address already in use
So I'm wondering why it's failing only few times. Did I miss something in my code ? or is this an expected behavior ?

Non Blocking socket in C++ - Resource temporarily unavailable error

I am currently working in a server program in linux, which has to connect to a single client and has to listen to it for commands. Once it receives certain command I have to send some values in return to the client. The values are generated in the main function and the server program is run in a thread.
So, in this case I thought I have to implement non-blocking socket. But I am getting Resource temporarily unavailable error. If any anyone point me in a direction for further research it will would helpful , as I am stuck in this implementation for a long time.
Here is the while loop code,
void server::serve() {
struct sockaddr_in clientAddress;
socklen_t sin_size;
sin_size = sizeof(struct sockaddr_in);
if(client = accept(sockId,(struct sockaddr *)&clientAddress,&sin_size) == -1){
perror("accept");
}
fcntl(sockId,F_SETFL,O_NONBLOCK);
fcntl(client,F_SETFL,O_NONBLOCK);
while (1) {
if(client = accept(sockId,(struct sockaddr *)&clientAddress,&sin_size) == -1){
perror("accept");
}
cout <<"client "<<client<<endl;
fcntl(client,F_SETFL,O_NONBLOCK);
getMessage(client);
if(transmitFlag != -1)
sendRequest(client);
}
closeSocket();
}

Blocking sockets to non-blocking sockets

Can anyone help me, how to change the below code to non-blocking
struct sockaddr_un server_address;
int server_len, err;
int ret = 1;
int ipc_sockfd = socket(AF_UNIX, SOCK_STREAM, 0);
if (ipc_sockfd < 0) {
printf("%s\n","SHM_IPC: socket creation failed");
return 0;
}
server_address.sun_family = AF_UNIX;
strcpy(server_address.sun_path, SHM_IPC_SOCKET_NAME);
server_len = sizeof(server_address);
err = connect(ipc_sockfd, (struct sockaddr*)&server_address,
server_len);
if (err < 0) {
printf("%s %d\n", "IPC socket server not ready for"
". Try after few moments, Errno:", errno);
close(ipc_sockfd);
return 0;
}
err = write(ipc_sockfd, (void *)msg, sizeof(shm_ipc_msg));
if (err <=0) {
printf("%s %d\n", "SHM_IPC: socket write failed:",errno);
ret = 0;
}
close(ipc_sockfd);
return ret;
This is my client side socket, i need my client side write() and connect() to be non-blocking (I dont care about server side), do I need to change the server socket also to non-blocking to take effect?
Really appreciate your help!
This thread might help you: Does connect() block for TCP socket?
You can make your connect non-blocking but for what reason? if the client is not connected to the server you cannot expect socket.write to work. You need to wait until the connection it's done. The write operation though it is not a blocking operation. If the connection is established the write function should return immediately after sending your data.

Multiple connections from same host cause socket disconnection issues

I'm writting a C++ app for Linux. It uses sockets for TCP communication. I read data from remote hosts as follows (a bit simplified for this topic's needs):
Packet Connection::receiveRawData()
{
int length = recv(socketDescriptor, receiveBuffer, MAXBUF, 0);
if(length < 0)
{
std::cout << getConnectionDetails() << " - nothing to read.\n";
return Packet(); //Empty packet object (with no data)
}
else if(length == 0)
{
std::cout << getConnectionDetails() << " - disconnected suddenly!\n";
//Do something about this closed connection
}
return Packet(receiveBuffer, length);
}
getConnectionDetails() shows me adress and port number (like this: xx.xx.xx.xx:yy) so I would know that the message is about specific connection (since even if they're from same host they'll still have different port numbers). Couts are just for debuging purpose.
The thing is, it doesn't work quite well when I close one of the connections from the same host (ie. after making three different connection to the application, but all of them come from same machine). Only when I close them in reversed order (from newest to oldest) it works as expected - it shows "xx.xx.xx.xx:yy - disconnected suddenly!" in app's console. But when I try from oldest to newest, I keep getting "xx.xx.xx.xx:yy - nothing to read." for every connection, even though these connections should be closed!
What's going on? recv() is supposed to return 0 when remote host closed its connection, so why doesn't it happen when I close connection (from the remote host side) from oldest or even from the "middle" (neither oldest nor newest)?
It doesn't occur when I close connections from different machines (so that no machine makes more than one connection to the app) - order of these operations don't matter then, it just works fine as it should.
Edit: some more code so you can see how I set up connection sockets:
//Create "empty" socket object associated with the given port number
Connection::Connection(PortNumberFormat portNum)
: address(), port(portNum), socketDescriptor(socket(AF_INET, SOCK_STREAM, 0)), isConnected(false), isListening(false), isShuttingDown(false)
{
if(socketDescriptor < 0)
throw std::string("Fatal error: cannot create socket! The error was: " + std::string( strerror(errno) ));
}
//Create socket object using specified socket descriptor and mark it as connected to the server
//available on specified address and port
Connection::Connection(SocketDescriptor sock, char *addr, PortNumberFormat portNum)
: address(addr), port(portNum), socketDescriptor(sock), isConnected(true), isListening(false), isShuttingDown(false)
{
setTimeouts();
}
Connection::~Connection()
{
if(socketDescriptor < 0) return;
if(!isShuttingDown) signalShutdown();
close(socketDescriptor);
}
//Turn socket into listening mode by binding it to the specified port
void Connection::startListen()
{
if(isConnected || isListening || socketDescriptor < 0)
throw std::string("Socket can not be set as listening!");
struct sockaddr_in bindData;
int dummy = 1;
memset(&bindData, 0, sizeof(bindData));
bindData.sin_family = AF_INET;
bindData.sin_addr.s_addr = INADDR_ANY;
bindData.sin_port = htons(port);
setsockopt(socketDescriptor, SOL_SOCKET, SO_REUSEADDR, &dummy, sizeof(dummy)); //Release socket ASAP on closing
if(bind(socketDescriptor, (struct sockaddr*)&bindData, sizeof(bindData)) < 0)
throw std::string("Fatal error: cannot bind socket for listening! The error was: " + std::string( strerror(errno) ));
if(listen(socketDescriptor, MAXCONN) < 0)
throw std::string("Fatal error: cannot listen on bound socket! The error was: " + std::string( strerror(errno) ));
isListening = true;
}
//Get new Connection object that represents external client connection
//This method will block until client connects or the socket shutdown occurs
Connection Connection::getClientConnection()
{
if(!isListening || socketDescriptor < 0)
throw std::string("Can't get connection from non-listening socket!");
struct sockaddr_in clientData;
socklen_t len = sizeof(clientData);
SocketDescriptor clientSocket = accept(socketDescriptor, (struct sockaddr*)&clientData, &len);
if(isShuttingDown) //An shutdown occured
{
if(clientSocket < 0)
throw std::string("Information: listening connection is shutting down peacefully.");
//Else...
close(clientSocket); //Just to be extra sure
throw std::string("Warning: listening connection is shutting down. Client connection has been discarded.");
}
if(clientSocket < 0)
throw std::string("Fatal error: cannot open client connection socket! The error was: " + std::string( strerror(errno) ));
return Connection(clientSocket, inet_ntoa(clientData.sin_addr), ntohs(clientData.sin_port));
}
//Invalidates socket and prepares it to be closed
void Connection::signalShutdown()
{
if(socketDescriptor < 0) return;
isShuttingDown = true;
shutdown(socketDescriptor, SHUT_RDWR);
}
inline void Connection::setTimeouts()
{
struct timeval timeout;
timeout.tv_sec = 2;
timeout.tv_usec = 0;
if(setsockopt(socketDescriptor, SOL_SOCKET, SO_RCVTIMEO, (void*)&timeout, sizeof(timeout)) < 0
|| setsockopt(socketDescriptor, SOL_SOCKET, SO_SNDTIMEO, (void*)&timeout, sizeof(timeout)) < 0)
{
//Do something about it
}
}
std::string Connection::getConnectionDetails()
{
return std::string(address + ":" + std::to_string(port));
}
Listening connection is created using first constructor (port number only), then I call startListen() on it. Connections from hosts are returned to external code with getClientConnection() method (they are created using second constructor). Now these are our troublemakers.

libev sets sockets to blocking with no timeout

Rant: I really dislike boost::asio, So I've been looking at alternatives and came across libev. Which seems simple enough for me, but is doing a few things I cannot understand. If those are too many questions in one thread, please let me know.
1) I set the listening socket to NON_BLOCK, I also set each accepted incoming connection as NON_BLOCK, yet somewhere in the code the socket(s) turns into BLOCK.
Ex:
bool Server::Start()
{
// Setup event loop
loop = ev_default_loop(EVBACKEND_SELECT); //EVFLAG_AUTO ?
// Create Socket
sockfd = socket(PF_INET, SOCK_STREAM, 0);
addr_len = sizeof(addr)
// Set Socket to non blocking
fcntl(sockfd, F_SETFL, fcntl(sockfd, F_GETFL) | O_NONBLOCK);
if (fcntl(sockfd, F_GETFL) & O_NONBLOCK) std::cout << "Socket is NONBLOCK" << std::endl;
else std::cout << "Socket is BLOCK" << std::endl;
if (sockfd < 0) {
std::cout << "ERROR opening socket" << std::endl;
return false;
}
bzero((char *)&addr, sizeof(addr));
addr.sin_family = AF_INET;
addr.sin_port = htons(port);
addr.sin_addr.s_addr = INADDR_ANY;
// Bind port to socket
if (bind(sockfd,(struct sockaddr*)&addr, sizeof(addr))!=0) {
std::cout << "bind error" << std::endl;
return false;
}
// Listen
if (listen(sockfd, 2) < 0) {
std::cout << "listen error" << std::endl;
return false;
}
// Initialize and start a watcher to accepts client requests
ev_io_init(&w_accept, accept_cb, sockfd, EV_READ);
ev_io_start(loop, &w_accept);
return true;
}
I have tried to make the main loop also not to block:
void Server::MainLoop()
{
// Start infinite loop
while (1) {
ev_loop(loop, EVLOOP_NONBLOCK);
}
}
But it doesnt seem to have made a different. PLEASE DO NOT redirect me to the documentation (the only available source of documentation on the internet) I have read it.
I do this for the client socket that has been accepted:
void accept_cb(struct ev_loop *loop, struct ev_io *watcher, int revents)
....
c->client_sd = accept(watcher->fd, (struct sockaddr *)&c->client_addr, &c->client_len);
....
ev_io *w_client = (struct ev_io*) malloc (sizeof(struct ev_io));
ev_io_init(w_client, read_cb, c->client_sd, EV_READ);
ev_io_start(loop, w_client);
fcntl(watcher->fd, F_SETFL, fcntl(watcher->fd, F_GETFL) | O_NONBLOCK);
Yet every time my read callback is executed, the socket is magically set to BLOCK
2) I have tried setting a timeout for the socket:
struct timeval timeout;
timeout.tv_sec = 10;
timeout.tv_usec = 0;
if (setsockopt (sockfd, SOL_SOCKET, SO_RCVTIMEO, (char *)&timeout,
sizeof(timeout)) < 0)
error("setsockopt failed\n");
if (setsockopt (sockfd, SOL_SOCKET, SO_SNDTIMEO, (char *)&timeout,
sizeof(timeout)) < 0)
error("setsockopt failed\n");
(Taken from here: this question)
It simply doesn't work. Is this because the sockets are reset to BLOCKing mode ?
3) I have seen a C++ wrapper for libev. I absolutely hate the fact I have to make the callbacks static functions, it ruins everything for me. Yet all the examples I have seen use:
signal.loop.break_loop();
and
loop.run(0);
which, funnily enough produces:
error: ‘struct ev::loop_ref’ has no member named ‘break_loop’ error:
‘struct ev::default_loop’ has no member named ‘run’
on Debian Squeeze.
So, what I am asking is:
What, who, where is the socket changed from NON_BLOCK to BLOCK ?
How (if) can I set a timeout for the socket (blocking or non-blocking)
What is wrong with ev++.h and why are those nice people using the wrappers I can't use?
Please, bear in mind that I can use the sockets to read and send data, but in a blocking manner, without timeouts. Furthermore, as this is a server, I NEED to keep the code in classes, as I have to save messages per connected clients. Making this static or non-class methods simply ruins it, or forces me to take a very different approach.
PS: Any alternatives to libev ?
You aren't setting the client FD to non-blocking mode. You are setting the listening socket FD.