Multiple connections from same host cause socket disconnection issues - c++

I'm writting a C++ app for Linux. It uses sockets for TCP communication. I read data from remote hosts as follows (a bit simplified for this topic's needs):
Packet Connection::receiveRawData()
{
int length = recv(socketDescriptor, receiveBuffer, MAXBUF, 0);
if(length < 0)
{
std::cout << getConnectionDetails() << " - nothing to read.\n";
return Packet(); //Empty packet object (with no data)
}
else if(length == 0)
{
std::cout << getConnectionDetails() << " - disconnected suddenly!\n";
//Do something about this closed connection
}
return Packet(receiveBuffer, length);
}
getConnectionDetails() shows me adress and port number (like this: xx.xx.xx.xx:yy) so I would know that the message is about specific connection (since even if they're from same host they'll still have different port numbers). Couts are just for debuging purpose.
The thing is, it doesn't work quite well when I close one of the connections from the same host (ie. after making three different connection to the application, but all of them come from same machine). Only when I close them in reversed order (from newest to oldest) it works as expected - it shows "xx.xx.xx.xx:yy - disconnected suddenly!" in app's console. But when I try from oldest to newest, I keep getting "xx.xx.xx.xx:yy - nothing to read." for every connection, even though these connections should be closed!
What's going on? recv() is supposed to return 0 when remote host closed its connection, so why doesn't it happen when I close connection (from the remote host side) from oldest or even from the "middle" (neither oldest nor newest)?
It doesn't occur when I close connections from different machines (so that no machine makes more than one connection to the app) - order of these operations don't matter then, it just works fine as it should.
Edit: some more code so you can see how I set up connection sockets:
//Create "empty" socket object associated with the given port number
Connection::Connection(PortNumberFormat portNum)
: address(), port(portNum), socketDescriptor(socket(AF_INET, SOCK_STREAM, 0)), isConnected(false), isListening(false), isShuttingDown(false)
{
if(socketDescriptor < 0)
throw std::string("Fatal error: cannot create socket! The error was: " + std::string( strerror(errno) ));
}
//Create socket object using specified socket descriptor and mark it as connected to the server
//available on specified address and port
Connection::Connection(SocketDescriptor sock, char *addr, PortNumberFormat portNum)
: address(addr), port(portNum), socketDescriptor(sock), isConnected(true), isListening(false), isShuttingDown(false)
{
setTimeouts();
}
Connection::~Connection()
{
if(socketDescriptor < 0) return;
if(!isShuttingDown) signalShutdown();
close(socketDescriptor);
}
//Turn socket into listening mode by binding it to the specified port
void Connection::startListen()
{
if(isConnected || isListening || socketDescriptor < 0)
throw std::string("Socket can not be set as listening!");
struct sockaddr_in bindData;
int dummy = 1;
memset(&bindData, 0, sizeof(bindData));
bindData.sin_family = AF_INET;
bindData.sin_addr.s_addr = INADDR_ANY;
bindData.sin_port = htons(port);
setsockopt(socketDescriptor, SOL_SOCKET, SO_REUSEADDR, &dummy, sizeof(dummy)); //Release socket ASAP on closing
if(bind(socketDescriptor, (struct sockaddr*)&bindData, sizeof(bindData)) < 0)
throw std::string("Fatal error: cannot bind socket for listening! The error was: " + std::string( strerror(errno) ));
if(listen(socketDescriptor, MAXCONN) < 0)
throw std::string("Fatal error: cannot listen on bound socket! The error was: " + std::string( strerror(errno) ));
isListening = true;
}
//Get new Connection object that represents external client connection
//This method will block until client connects or the socket shutdown occurs
Connection Connection::getClientConnection()
{
if(!isListening || socketDescriptor < 0)
throw std::string("Can't get connection from non-listening socket!");
struct sockaddr_in clientData;
socklen_t len = sizeof(clientData);
SocketDescriptor clientSocket = accept(socketDescriptor, (struct sockaddr*)&clientData, &len);
if(isShuttingDown) //An shutdown occured
{
if(clientSocket < 0)
throw std::string("Information: listening connection is shutting down peacefully.");
//Else...
close(clientSocket); //Just to be extra sure
throw std::string("Warning: listening connection is shutting down. Client connection has been discarded.");
}
if(clientSocket < 0)
throw std::string("Fatal error: cannot open client connection socket! The error was: " + std::string( strerror(errno) ));
return Connection(clientSocket, inet_ntoa(clientData.sin_addr), ntohs(clientData.sin_port));
}
//Invalidates socket and prepares it to be closed
void Connection::signalShutdown()
{
if(socketDescriptor < 0) return;
isShuttingDown = true;
shutdown(socketDescriptor, SHUT_RDWR);
}
inline void Connection::setTimeouts()
{
struct timeval timeout;
timeout.tv_sec = 2;
timeout.tv_usec = 0;
if(setsockopt(socketDescriptor, SOL_SOCKET, SO_RCVTIMEO, (void*)&timeout, sizeof(timeout)) < 0
|| setsockopt(socketDescriptor, SOL_SOCKET, SO_SNDTIMEO, (void*)&timeout, sizeof(timeout)) < 0)
{
//Do something about it
}
}
std::string Connection::getConnectionDetails()
{
return std::string(address + ":" + std::to_string(port));
}
Listening connection is created using first constructor (port number only), then I call startListen() on it. Connections from hosts are returned to external code with getClientConnection() method (they are created using second constructor). Now these are our troublemakers.

Related

bind() fails occasionally with Address already in use even after setting SO_REUSEADDR and SO_REUSEPORT

We have a streaming server implemented using live555 library. This server is deployed on CentOS instance.
Recently we wanted to modify the server socket options, so that it can accept the requests immediately after re-starting the process (either after crash or manual restart).
I have referred to man pages and few web links and set the socket options (SO_REUSEADDR and SO_REUSEPORT) before bind() call.
int setupStreamSocket(UsageEnvironment& env,
Port port, Boolean makeNonBlocking) {
int newSocket = createSocket(SOCK_STREAM);
if (newSocket < 0) {
socketErr(env, "unable to create stream socket: ");
return newSocket;
}
int reuseFlag = 1;
fprintf(stderr,"reuseFlag : %d\n",reuseFlag);
if (setsockopt(newSocket, SOL_SOCKET, SO_REUSEADDR,
(const char*)&reuseFlag, sizeof reuseFlag) < 0) {
socketErr(env, "setsockopt(SO_REUSEADDR) error: ");
closeSocket(newSocket);
return -1;
}
if (setsockopt(newSocket, SOL_SOCKET, SO_REUSEPORT,
(const char*)&reuseFlag, sizeof reuseFlag) < 0) {
socketErr(env, "setsockopt(SO_REUSEPORT) error: ");
closeSocket(newSocket);
return -1;
}
int flag = 1;
setsockopt( newSocket, IPPROTO_TCP, TCP_NODELAY, &flag, sizeof(flag) );
if (port.num() != 0 || ReceivingInterfaceAddr != INADDR_ANY) {
MAKE_SOCKADDR_IN(name, ReceivingInterfaceAddr, port.num());
int reuse_addr_val, reuse_port_val;
socklen_t reuse_addr_len = sizeof(reuse_addr_val);
socklen_t reuse_port_len = sizeof(reuse_port_val);
getsockopt(newSocket, SOL_SOCKET, SO_REUSEADDR, &reuse_addr_val, &reuse_addr_len);
getsockopt(newSocket, SOL_SOCKET, SO_REUSEPORT, &reuse_port_val, &reuse_port_len);
fprintf(stderr,"reuse_addr_val = %d, reuse_port_val = %d\n", reuse_addr_val, reuse_port_val);
if (bind(newSocket, (struct sockaddr*)&name, sizeof name) != 0) {
char tmpBuffer[100];
sprintf(tmpBuffer, "bind() error (port number: %d): ",
ntohs(port.num()));
socketErr(env, tmpBuffer);
closeSocket(newSocket);
return -1;
}
}
if (makeNonBlocking) {
if (!makeSocketNonBlocking(newSocket)) {
socketErr(env, "failed to make non-blocking: ");
closeSocket(newSocket);
return -1;
}
}
return newSocket;
}
This code works as expected (binding to the address even when a socket is in TIME_WAIT state) if I restart the server with above options.
If I replace previous build (without socket options) with the build created with above code then I have noticed that occasionally bind() is failing with Address already in use error.
When the bind() failed the port/address is in TIME_WAIT state. So the server should able to bind the socket to the address.
tcp 0 0 10.0.1.24:8554 10.0.1.89:27085 TIME_WAIT -
And the getsockopt() of my code printed the flag value (corresponding to SO_REUSEADDR and SO_REUSEPORT) as 1.
reuse_addr_val = 1, reuse_port_val = 1
bind() error (port number: 8554): Address already in use
So I'm wondering why it's failing only few times. Did I miss something in my code ? or is this an expected behavior ?

C++ WINSOCK tcpaccept stops accepting connections after an "attack"

I have a game server in C++ and I'm using a network library that uses winsock in Windows.
I've been stress-testing my server to see how many connections it can accept at a time. It works fine when I connect using my game clients but my game clients can no longer connect after I do a stress-test described below.
The stress test is, I connected to my server about 1000 times using a simple program for loop that just starts a tcp connection with my game server and closes it right away. They all connect. Then, after, I try to connect with my game. The game does not connect at all.
I checked the tcpaccept() function from the library (see below), no output. For some reason, accept() stops accepting connections after my "attack" of 1000 connections.
What could possibly make my server just stop accepting connections?
Here's my summary of my loop that listens and accepts connections and closes them:
bool serverIsOn = true;
double listen = tcplisten(12345, 30000, 1);
setnagle(listen, true);
...
while(serverIsOn){
double playerSocket = tcpaccept(listen, 1);
if(playerSocket > -1){
cout << "Got a new connection, socket ID: " << playerSocket << endl;
//add their sockID to list here!
addSockIDToList(playerSocket);
}
//Loop through list of socks and parse their messages here..
//If their message size == 0, we close their socket via closesocket(sockID);
loopThroughSocketIdsAndCloseOnLeave();
}
cout << "Finished!" << endl;
Here's the definitions for tcplisten, tcpaccept, CSocket::CSocket(SOCKET), CSocket::tcplisten(...) and CSocket::tcpaccept(...):
double tcplisten(int port, int max, int mode)
{
CSocket* sock = new CSocket();
if(sock->tcplisten(port, max, mode))
return AddSocket(sock);
delete sock;
return -1;
}
double tcpaccept(int sockid, int mode)
{
CSocket*sock = (CSocket*)sockets.item(sockid);
if(sock == NULL)return -1;
CSocket*sock2 = sock->tcpaccept(mode);
if(sock2 != NULL)return AddSocket(sock2);
return -1;
}
...
CSocket::CSocket(SOCKET sock)
{
sockid = sock;
udp = false;
format = 0;
}
bool CSocket::tcplisten(int port, int max, int mode)
{
if((sockid = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)) == INVALID_SOCKET) return false;
SOCKADDR_IN addr;
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = INADDR_ANY;
addr.sin_port = htons(port);
if(mode)setsync(1);
if(bind(sockid, (LPSOCKADDR)&addr, sizeof(SOCKADDR_IN)) == SOCKET_ERROR)
{
closesocket(sockid);
return false;
}
if(listen(sockid, max) == SOCKET_ERROR)
{
closesocket(sockid);
sockid = INVALID_SOCKET;
return false;
}
return true;
}
CSocket* CSocket::tcpaccept(int mode)
{
if(sockid == INVALID_SOCKET) return NULL;
SOCKET sock2;
if((sock2 = accept(sockid, (SOCKADDR *)&SenderAddr, &SenderAddrSize)) != INVALID_SOCKET)
{
//This does NOT get output after that 1000-'attack' test.
std::cout << "Accepted new connection!" << std::endl;
CSocket*sockit = new CSocket(sock2);
if(mode >=1)sockit->setsync(1);
return sockit;
}
return NULL;
}
What can I do to figure out why accept() no longer accepts connections after my 1000-connection stress test? Does it have something to do with the way I close connections after their finished? When I do that, all I do is just call: closesocket(sockID).
Please ask for any other code needed!
EDIT:
I just noticed that my "stress-test" java program is getting an exception after its connected around 668 times. Here's the exception:
Exception in thread "main" java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:208)
at sockettest.SocketTest.main(SocketTest.java:63)
Java Result: 1
Because your server side is closing the sockets, they are most likely sitting in time_wait for several minutes. Windows has various parameters controlling maximum sockets and various states. I am guessing your program starts working again after several minutes, and potentially there are some warnings in event viewer.
An alternative might be to simply ignore these sockets for several minutes and hope they go away. ie the client calls closesocket when you dont respond at all, which means you do not incur time_wait. This often works but not always. If they do not, then you call closesocket() slowly on them in the background.
If you really want too though, you can reset the connection, see TCP option SO_LINGER (zero) - when it's required for details, but reseting connections is not normal so definitely read widely about So_linger and how tcp teardown works.
It turns out this library has it's own method of closing a socket:
int closesock(int sockid)
{
CSocket*sock = (CSocket*)sockets.item(sockid);
if(sock == NULL)return -1;
delete sock;
sockets.set((int)sockid, NULL);
return 1;
}
So it gets the current socket via the sockID in the list of sockets.
Then if the sockID was related to a valid socket, delete the sock object and set it to NULL in the list of sockets.
The problem was I was only calling closesocket(sockID) instead of closesock(sockID) which performed the necessary operations needed to close a socket.
Thanks everyone for your help.

bind returns address in use even if no connection is established

I have a c++ code in which I am trying to establish a connection on a socket. But I firstly need to check if a connection already exist on a given port, and if it exists I need to close the connection. I have the code below and my problem is that when checking if the port is already connected it returs that it is even if connect has failed previously.
connected = false;
int sockfd;
void conn(int port) {
struct sockaddr_in addr;
addr.sin_family = AF_INET;
addr.sin_port = htons(port);
.....
int sockfd_t;
if ( (sockfd_t = socket(PF_INET, SOCK_STREAM, 0)) < 0) {
cout << "Error opening socket_test " << endl;
return;
}
// check if address already in use
if (bind(sockfd_t, (struct sockaddr *) &addr, sizeof(addr)) < 0) {
if(errno == EADDRINUSE) {
cout << "address in use: bind fail, port=" << port << endl;
}
// do something - close the connection if already connected
}
else {
cout << "bind ok, port=" << port << endl;
}
close(sockfd_t);
if ( (sockfd = socket(PF_INET, SOCK_STREAM, 0)) < 0) {
cout << "Error opening socket " << endl;
return;
}
if (connect(sockfd, (struct sockaddr *) &adresse, sizeof(adresse))) {
cout << "Error connecting" << endl;
close(sockfd);
return;
}
connected = true;
}
int main() {
int port=3590;
while (!connected) {
conn(port);
}
cout << "CONNECTED";
// ..........
}
After running the program this is the output printed:
bind ok, port=3590;
Error connecting
bind ok, port=3590;
Error connecting
address in use: bind fail, port=3590 //???
CONNECTED!
I don't know why on the 5-th line of the print it displays "address in use:..." as the connect fails the first two times?
I think you have a misconception about what these socket operations do.
But I firstly need to check if a connection already exist on a given port, and if it exists I need to close the connection.
bind() gives a socket a local address, rather than having anything to do with checking if a remote address you are trying to connect to is accessible.
connect() connects the socket to a remote address.
When connecting a socket as a client (which is what I think you are trying to do), you don't need to check if there is already a connection, remote server can handle multiple incoming client connections to the same port. Binding is usually only important for servers.
if you don't bind before connecting, a socket will be assigned a random local port.
So, if you are a client, you do:
socket()
connect()
If you are a server, you do:
socket()
bind()
listen()
In your own question, the output makes sense when there is no server listening, but then a server comes online.
First two times, you bind a socket and it's successful, because nobody is using it to listen, then you fail to connect, because you just bound, and did not start a server (by calling listen()).
Then a real server on the same host binds that socket and starts listening, therefore you can't bind that port anymore (it fails), but you can connect, because the server is listening.

C++ Multithreaded TCP Server Issue

I'm writing a simple TCP server. The model that I though of is the server accepting client connections in the main thread and handing them over to another thread so that server can listen for connections again. The relevant parts of the code that I used are posted below:
Accepting connections:
void startServer () {
int serverSideSocket = 0;
int clientSideSocket = 0;
serverSideSocket = socket(AF_INET, SOCK_STREAM, 0);
if (serverSideSocket < 0) {
error("ERROR opening socket");
exit(1);
}
clientAddressLength = sizeof(clientAddress);
memset((char *) &serverAddress, 0, sizeof(serverAddress));
memset((char *) &clientAddress, 0, clientAddressLength);
serverAddress.sin_family = AF_INET;
serverAddress.sin_addr.s_addr = INADDR_ANY;
serverAddress.sin_port = htons(32000);
if (bind(serverSideSocket, (struct sockaddr *) &serverAddress, sizeof(serverAddress)) < 0) {
error("ERROR on binding");
exit(1);
}
listen(serverSideSocket, SOMAXCONN);
while(true) {
clientSideSocket = accept(serverSideSocket, (struct sockaddr *) &clientAddress, &clientAddressLength);
if (clientSideSocket < 0)
error("ERROR on accept");
processingThreadGroup->create_thread(boost::bind(process, clientSideSocket, this));
}
}
Here, the processingThreadGroup is a boost::thread_group instance. In the process method:
void process (int clientSideSocket, DataCollector* collector) {
int numberOfCharactersRead = 0;
string buffer;
do {
char msgBuffer[1000];
numberOfCharactersRead = recv(clientSideSocket, msgBuffer, (1000 - 1), 0);
if (numberOfCharactersRead < 0) {
//display error
close(clientSideSocket);
}
else if (numberOfCharactersRead == 0)
close(clientSideSocket);
else {
printf("%s", msgBuffer);
memset(msgBuffer, 0, 1000);
}
} while (numberOfCharactersRead > 0);
}
However, when I debug the code, I saw that when the processing thread is invoked, the main thread is not accepting connections anymore. The data is read inside the process() method only. The main thread seem to be not running anymore. What is the issue with the approach I took and any suggestions to correct it?
EDIT: I think I found the issue here, and have updated it as an answer. Will not accept it since I answered my own question. Thank you for the help everyone!
Think I found the issue. I was using this as a server to accept syslog messages. The code I use for the syslog message generator is as follows:
openlog ("MyProgram", LOG_CONS | LOG_PID | LOG_NDELAY, LOG_LOCAL0);
cout << "opened the log" << endl;
for (int i = 0 ; i < 10 ; i++)
{
syslog (LOG_INFO, "Program started by User %d \n", getuid ());
syslog (LOG_WARNING, "Beware of the WARNING! \n");
syslog (LOG_ERR, "fatal ERROR! \n");
}
closelog ();
cout << "closed the log" << endl;
and I use an entry in the rsyslog.conf file to direct all syslog LOG_LOCAL0 application traffic to be sent to the relevant TCP port where the server is listening. Somehow, syslog allows only one connection to be made, not multiple connections. Therefore, it only used one connection in a single thread. If that connection was closed, a new connection is craeted.
I checked with a normal tcp client. That works fine, with multiple threads being spawned for each connection accepted.

winsock: connect fails with error 10049 when using localhost (127.0.0.1)

i wrote a class encapsulating some of the winsock functions to imitate a simple TCP socket for my needs...
When i try to run a simple connect-and-send-data-to-server test the "client" fails on its call to connect with the error code of 10049 (WSAEADDRNOTAVAIL) connect function on MSDN
What I am doing is (code below):
Server:
Create a Server Socket -> Bind it to Port 12345
Put the Socket in listen mode
Call accept
Client
Create a socket -> Bind it to a random port
Call Connect: connect to localhost, port 12345
=> the call to connect fails with Error 10049, as described above
Here is the main function including the "server":
HANDLE hThread = NULL;
Inc::CSocketTCP ServerSock;
Inc::CSocketTCP ClientSock;
try
{
ServerSock.Bind(L"", L"12345");
ServerSock.Listen(10);
//Spawn the senders-thread
hThread = (HANDLE)_beginthreadex(nullptr, 0, Procy, nullptr, 0, nullptr);
//accept
ServerSock.Accept(ClientSock);
//Adjust the maximum packet size
ClientSock.SetPacketSize(100);
//receive data
std::wstring Data;
ClientSock.Receive(Data);
std::wcout << "Received:\t" << Data << std::endl;
}
catch(std::exception& e)
{...
Client thread function
unsigned int WINAPI Procy(void* p)
{
Sleep(1500);
try{
Inc::CSocketTCP SenderSock;
SenderSock.Bind(L"", L"123456");
SenderSock.Connect(L"localhost", L"12345");
//Adjust packet size
SenderSock.SetPacketSize(100);
//Send Data
std::wstring Data = L"Hello Bello!";
SenderSock.Send(Data);
}
catch(std::exception& e)
{
std::wcout << e.what() << std::endl;
}...
The Connect-Function
int Inc::CSocketTCP::Connect(const std::wstring& IP, const std::wstring& Port)
{
//NOTE: assert that the socket is valid
assert(m_Socket != INVALID_SOCKET);
//for debuggin: convert WStringToString here
std::string strIP = WStringToString(IP), strPort = WStringToString(Port);
Incgetaddrinfo AddyResolver;
addrinfo hints = {}, *pFinal = nullptr;
hints.ai_family = AF_INET;
//resolve the ip/port-combination for the connection process
INT Ret = AddyResolver(strIP.c_str(), strPort.c_str(), &hints, &pFinal);
if(Ret)
{
//error handling: throw an error description
std::string ErrorString("Resolving Process failed (Connect): ");
ErrorString.append(Inc::NumberToString<INT>(Ret));
throw(std::runtime_error(ErrorString.c_str()));
}
/*---for debbuging---*/
sockaddr_in *pP = (sockaddr_in*)(pFinal->ai_addr);
u_short Porty = ntohs(pP->sin_port);
char AddBuffer[20] = "";
InetNtopA(AF_INET, (PVOID)&pP->sin_addr, AddBuffer, 20);
/*--------------------------------------------------------*/
if(connect(m_Socket, pFinal->ai_addr, pFinal->ai_addrlen) == SOCKET_ERROR)
{
int ErrorCode = WSAGetLastError();
if((ErrorCode == WSAETIMEDOUT) || (ErrorCode == WSAEHOSTUNREACH) || (ErrorCode == WSAENETUNREACH))
{
//Just Unreachable
return TCP_TARGETUNREACHABLE;
}
//real errors now
std::string ErrorString("Connection Process failed: ");
ErrorString.append(Inc::NumberToString<int>(ErrorCode));
throw(std::runtime_error(ErrorString.c_str()));
}
return TCP_SUCCESS;
}
Additional Information:
-Incgetaddrinfo is a function object encapuslating getaddrinfo...
-Noone of the server functions return any error and work as expected when stepping through them using the debugger or when letting them run solely...
I'd be glad to hear your suggestions what the rpoblem might be...
Edit: It works when I dont connect to ("localhost","12345"), but to ("",12345)...
When look into the address resolution process of getaddrinfo it gives 127.0.0.1 for "localhost" and my real IP for ""
Why doesn't it work with my loopback-IP?
You have the answer in your question:
... it gives 127.0.0.1 for "localhost" and my real IP for ""
This means your server binds to the real IP of the interface instead of INADDR_ANY, i.e. it doesn't listen on the loopback.
Edit 0:
You don't really need name resolution when creating listening socket. Just bind() it to INADDR_ANY to listen on all available interfaces (including the loopback).
In my case, this error was caused by not calling htonl on INADDR_LOOPBACK before assigning it to address.sin_addr.s_addr.
Make sure you convert to network byte order, or you'll get 0x0100007f (1.0.0.127) instead of 0x7f000001 (127.0.0.1) for your loopback address and the bind will fail with code 10049 (WSAEADDRNOTAVAIL).