I have a game server in C++ and I'm using a network library that uses winsock in Windows.
I've been stress-testing my server to see how many connections it can accept at a time. It works fine when I connect using my game clients but my game clients can no longer connect after I do a stress-test described below.
The stress test is, I connected to my server about 1000 times using a simple program for loop that just starts a tcp connection with my game server and closes it right away. They all connect. Then, after, I try to connect with my game. The game does not connect at all.
I checked the tcpaccept() function from the library (see below), no output. For some reason, accept() stops accepting connections after my "attack" of 1000 connections.
What could possibly make my server just stop accepting connections?
Here's my summary of my loop that listens and accepts connections and closes them:
bool serverIsOn = true;
double listen = tcplisten(12345, 30000, 1);
setnagle(listen, true);
...
while(serverIsOn){
double playerSocket = tcpaccept(listen, 1);
if(playerSocket > -1){
cout << "Got a new connection, socket ID: " << playerSocket << endl;
//add their sockID to list here!
addSockIDToList(playerSocket);
}
//Loop through list of socks and parse their messages here..
//If their message size == 0, we close their socket via closesocket(sockID);
loopThroughSocketIdsAndCloseOnLeave();
}
cout << "Finished!" << endl;
Here's the definitions for tcplisten, tcpaccept, CSocket::CSocket(SOCKET), CSocket::tcplisten(...) and CSocket::tcpaccept(...):
double tcplisten(int port, int max, int mode)
{
CSocket* sock = new CSocket();
if(sock->tcplisten(port, max, mode))
return AddSocket(sock);
delete sock;
return -1;
}
double tcpaccept(int sockid, int mode)
{
CSocket*sock = (CSocket*)sockets.item(sockid);
if(sock == NULL)return -1;
CSocket*sock2 = sock->tcpaccept(mode);
if(sock2 != NULL)return AddSocket(sock2);
return -1;
}
...
CSocket::CSocket(SOCKET sock)
{
sockid = sock;
udp = false;
format = 0;
}
bool CSocket::tcplisten(int port, int max, int mode)
{
if((sockid = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)) == INVALID_SOCKET) return false;
SOCKADDR_IN addr;
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = INADDR_ANY;
addr.sin_port = htons(port);
if(mode)setsync(1);
if(bind(sockid, (LPSOCKADDR)&addr, sizeof(SOCKADDR_IN)) == SOCKET_ERROR)
{
closesocket(sockid);
return false;
}
if(listen(sockid, max) == SOCKET_ERROR)
{
closesocket(sockid);
sockid = INVALID_SOCKET;
return false;
}
return true;
}
CSocket* CSocket::tcpaccept(int mode)
{
if(sockid == INVALID_SOCKET) return NULL;
SOCKET sock2;
if((sock2 = accept(sockid, (SOCKADDR *)&SenderAddr, &SenderAddrSize)) != INVALID_SOCKET)
{
//This does NOT get output after that 1000-'attack' test.
std::cout << "Accepted new connection!" << std::endl;
CSocket*sockit = new CSocket(sock2);
if(mode >=1)sockit->setsync(1);
return sockit;
}
return NULL;
}
What can I do to figure out why accept() no longer accepts connections after my 1000-connection stress test? Does it have something to do with the way I close connections after their finished? When I do that, all I do is just call: closesocket(sockID).
Please ask for any other code needed!
EDIT:
I just noticed that my "stress-test" java program is getting an exception after its connected around 668 times. Here's the exception:
Exception in thread "main" java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:208)
at sockettest.SocketTest.main(SocketTest.java:63)
Java Result: 1
Because your server side is closing the sockets, they are most likely sitting in time_wait for several minutes. Windows has various parameters controlling maximum sockets and various states. I am guessing your program starts working again after several minutes, and potentially there are some warnings in event viewer.
An alternative might be to simply ignore these sockets for several minutes and hope they go away. ie the client calls closesocket when you dont respond at all, which means you do not incur time_wait. This often works but not always. If they do not, then you call closesocket() slowly on them in the background.
If you really want too though, you can reset the connection, see TCP option SO_LINGER (zero) - when it's required for details, but reseting connections is not normal so definitely read widely about So_linger and how tcp teardown works.
It turns out this library has it's own method of closing a socket:
int closesock(int sockid)
{
CSocket*sock = (CSocket*)sockets.item(sockid);
if(sock == NULL)return -1;
delete sock;
sockets.set((int)sockid, NULL);
return 1;
}
So it gets the current socket via the sockID in the list of sockets.
Then if the sockID was related to a valid socket, delete the sock object and set it to NULL in the list of sockets.
The problem was I was only calling closesocket(sockID) instead of closesock(sockID) which performed the necessary operations needed to close a socket.
Thanks everyone for your help.
Related
I'm writting a C++ app for Linux. It uses sockets for TCP communication. I read data from remote hosts as follows (a bit simplified for this topic's needs):
Packet Connection::receiveRawData()
{
int length = recv(socketDescriptor, receiveBuffer, MAXBUF, 0);
if(length < 0)
{
std::cout << getConnectionDetails() << " - nothing to read.\n";
return Packet(); //Empty packet object (with no data)
}
else if(length == 0)
{
std::cout << getConnectionDetails() << " - disconnected suddenly!\n";
//Do something about this closed connection
}
return Packet(receiveBuffer, length);
}
getConnectionDetails() shows me adress and port number (like this: xx.xx.xx.xx:yy) so I would know that the message is about specific connection (since even if they're from same host they'll still have different port numbers). Couts are just for debuging purpose.
The thing is, it doesn't work quite well when I close one of the connections from the same host (ie. after making three different connection to the application, but all of them come from same machine). Only when I close them in reversed order (from newest to oldest) it works as expected - it shows "xx.xx.xx.xx:yy - disconnected suddenly!" in app's console. But when I try from oldest to newest, I keep getting "xx.xx.xx.xx:yy - nothing to read." for every connection, even though these connections should be closed!
What's going on? recv() is supposed to return 0 when remote host closed its connection, so why doesn't it happen when I close connection (from the remote host side) from oldest or even from the "middle" (neither oldest nor newest)?
It doesn't occur when I close connections from different machines (so that no machine makes more than one connection to the app) - order of these operations don't matter then, it just works fine as it should.
Edit: some more code so you can see how I set up connection sockets:
//Create "empty" socket object associated with the given port number
Connection::Connection(PortNumberFormat portNum)
: address(), port(portNum), socketDescriptor(socket(AF_INET, SOCK_STREAM, 0)), isConnected(false), isListening(false), isShuttingDown(false)
{
if(socketDescriptor < 0)
throw std::string("Fatal error: cannot create socket! The error was: " + std::string( strerror(errno) ));
}
//Create socket object using specified socket descriptor and mark it as connected to the server
//available on specified address and port
Connection::Connection(SocketDescriptor sock, char *addr, PortNumberFormat portNum)
: address(addr), port(portNum), socketDescriptor(sock), isConnected(true), isListening(false), isShuttingDown(false)
{
setTimeouts();
}
Connection::~Connection()
{
if(socketDescriptor < 0) return;
if(!isShuttingDown) signalShutdown();
close(socketDescriptor);
}
//Turn socket into listening mode by binding it to the specified port
void Connection::startListen()
{
if(isConnected || isListening || socketDescriptor < 0)
throw std::string("Socket can not be set as listening!");
struct sockaddr_in bindData;
int dummy = 1;
memset(&bindData, 0, sizeof(bindData));
bindData.sin_family = AF_INET;
bindData.sin_addr.s_addr = INADDR_ANY;
bindData.sin_port = htons(port);
setsockopt(socketDescriptor, SOL_SOCKET, SO_REUSEADDR, &dummy, sizeof(dummy)); //Release socket ASAP on closing
if(bind(socketDescriptor, (struct sockaddr*)&bindData, sizeof(bindData)) < 0)
throw std::string("Fatal error: cannot bind socket for listening! The error was: " + std::string( strerror(errno) ));
if(listen(socketDescriptor, MAXCONN) < 0)
throw std::string("Fatal error: cannot listen on bound socket! The error was: " + std::string( strerror(errno) ));
isListening = true;
}
//Get new Connection object that represents external client connection
//This method will block until client connects or the socket shutdown occurs
Connection Connection::getClientConnection()
{
if(!isListening || socketDescriptor < 0)
throw std::string("Can't get connection from non-listening socket!");
struct sockaddr_in clientData;
socklen_t len = sizeof(clientData);
SocketDescriptor clientSocket = accept(socketDescriptor, (struct sockaddr*)&clientData, &len);
if(isShuttingDown) //An shutdown occured
{
if(clientSocket < 0)
throw std::string("Information: listening connection is shutting down peacefully.");
//Else...
close(clientSocket); //Just to be extra sure
throw std::string("Warning: listening connection is shutting down. Client connection has been discarded.");
}
if(clientSocket < 0)
throw std::string("Fatal error: cannot open client connection socket! The error was: " + std::string( strerror(errno) ));
return Connection(clientSocket, inet_ntoa(clientData.sin_addr), ntohs(clientData.sin_port));
}
//Invalidates socket and prepares it to be closed
void Connection::signalShutdown()
{
if(socketDescriptor < 0) return;
isShuttingDown = true;
shutdown(socketDescriptor, SHUT_RDWR);
}
inline void Connection::setTimeouts()
{
struct timeval timeout;
timeout.tv_sec = 2;
timeout.tv_usec = 0;
if(setsockopt(socketDescriptor, SOL_SOCKET, SO_RCVTIMEO, (void*)&timeout, sizeof(timeout)) < 0
|| setsockopt(socketDescriptor, SOL_SOCKET, SO_SNDTIMEO, (void*)&timeout, sizeof(timeout)) < 0)
{
//Do something about it
}
}
std::string Connection::getConnectionDetails()
{
return std::string(address + ":" + std::to_string(port));
}
Listening connection is created using first constructor (port number only), then I call startListen() on it. Connections from hosts are returned to external code with getClientConnection() method (they are created using second constructor). Now these are our troublemakers.
I have been trying to figure out this problem for over a month now. I have no where else to turn.
I have a server that listens to many multicast channels (100ish). Each socket is its own thread. Then I have a client listener (single threaded) that handles all incoming connections, disconnects, and client messaging within the same server. The idea is that a client comes in, connects, requests data from a multicast channels and I send the data back to the client. The client stays connected and I relay the UDP data back to the client. The client can either request UDP or TCP has the protocol for the data relay. At one point this was working beautifully for a couple of weeks. We did some code and kernel changes, and now we cant figure out whats gone wrong.
The server will run for hours and have hundreds of clients connected throughout the day. But at some point, randomly, the server will just stop. And by stop, I mean: all UDP sockets stop receiving/handling data (tcpdump shows data still coming to the box), the client_listener thread stops receiving client packets. BUT!!! the main client_listener socket can still receive new connections and new disconnects on the main socket. On a new connection, the main sockets is able to send a "Connection Established" packet back to the client, but then when the client responds, the select never returns.
I can post code if someone would like. If anyone has any suggestions where to look or if this sounds like something. Please let me know.
If you have any questions, please ask.
Thank you.
I would like to share my TCP Server code:
This is a single thread. Works fine for hours and then I will only receive "New Connections" and "Disconnects". NO CLIENT PACKETS WILL COME IN.
int opt = 1;
int addrlen;
int sd;
int max_sd;
int valread;
int activity;
int new_socket;
char buffer[MAX_BUFFER_SIZE];
int client_socket[m_max_clients];
struct sockaddr_in address;
fd_set readfds;
for(int i = 0; i<m_max_clients; i++)
{
client_socket[i]=0;
}
if((m_master_socket = socket(AF_INET,SOCK_STREAM,0))==0)
LOG(FATAL)<<"Unable to create master socket";
if(setsockopt(m_master_socket,SOL_SOCKET,SO_REUSEADDR,(char*)&opt,sizeof(opt))<0)
LOG(FATAL)<<"Unable to set master socket";
address.sin_family = AF_INET;
address.sin_addr.s_addr = INADDR_ANY;
address.sin_port = htons(m_listenPort);
if(bind(m_master_socket,(struct sockaddr*)& address, sizeof(address))!=0)
LOG(FATAL)<<"Unable to bind master socket";
if(listen(m_master_socket,SOMAXCONN)!=0)
LOG(FATAL)<<"listen() failed with err";
addrlen = sizeof(address);
LOG(INFO)<<"Waiting for connections......";
while(true)
{
FD_ZERO(&readfds);
FD_SET(m_master_socket, &readfds);
max_sd = m_master_socket;
for(int i = 0; i<m_max_clients; i++)
{
sd = client_socket[i];
if(sd > 0)
FD_SET(sd, &readfds);
if(sd>max_sd)
max_sd = sd;
}
activity = select(max_sd+1,&readfds,NULL,NULL,NULL);
if((activity<0)&&(errno!=EINTR))
{
// int err = errno;
// LOG(ERROR)<<"SELECT ERROR:"<<activity<<" "<<err;
continue;
}
if(FD_ISSET(m_master_socket, &readfds))
{
if((new_socket = accept(m_master_socket,(struct sockaddr*)&address, (socklen_t*)&addrlen))<0)
LOG(FATAL)<<"ERROR:ACCEPT FAILED!";
LOG(INFO)<<"New Connection, socket fd is (" << new_socket << ") client_addr:" << inet_ntoa(address.sin_addr) << " Port:" << ntohs(address.sin_port);
for(int i =0;i<m_max_clients;i++)
{
if(client_socket[i]==0)
{
//try to set the socket to non blocking, tcp nagle and keep alive
if ( !SetSocketBlockingEnabled(new_socket, false) )
LOG(INFO)<<"UNABLE TO SET NON-BLOCK: ("<<new_socket<<")" ;
if ( !SetSocketNoDelay(new_socket,false) )
LOG(INFO)<<"UNABLE TO SET DELAY: ("<<new_socket<<")" ;
// if ( !SetSocketKeepAlive(new_socket,true) )
// LOG(INFO)<<"UNABLE TO SET KeepAlive: ("<<new_socket<<")" ;
ClientConnection* con = new ClientConnection(m_mocSrv, m_udpPortGenerator, inet_ntoa(address.sin_addr), ntohs(address.sin_port), new_socket);
if(con->login())
{
client_socket[i] = new_socket;
m_clientConnectionSocketMap[new_socket] = con;
LOG(INFO)<<"Client Connection Logon Complete";
}
else
delete con;
break;
}
}//for
}
else
{
try{
for(int i = 0; i<m_max_clients; i++)
{
sd = client_socket[i];
if(FD_ISSET(sd,&readfds))
{
if ( (valread = recv(sd, buffer, sizeof(buffer),MSG_DONTWAIT|MSG_NOSIGNAL)) <= 0 )
{
//remove from the fd listening set
LOG(INFO)<<"RESET CLIENT_SOCKET:("<<sd<<")";
client_socket[i]=0;
handleDisconnect(sd,true);
}
else
{
std::map<int, ClientConnection*>::iterator client_connection_socket_iter = m_clientConnectionSocketMap.find(sd);
if(client_connection_socket_iter != m_clientConnectionSocketMap.end())
{
client_connection_socket_iter->second->handle_message(buffer, valread);
if(client_connection_socket_iter->second->m_logoff)
{
LOG(INFO)<<"SOCKET LOGGED OFF:"<<sd;
client_socket[i]=0;
handleDisconnect(sd,true);
}
}
else
{
LOG(ERROR)<<"UNABLE TO FIND SOCKET DESCRIPTOR:"<<sd;
}
}
}
}
}catch(...)
{
LOG(ERROR)<<"EXCEPTION CATCH!!!";
}
}
}
From the information given I would state the following:
Do not use a thread for each connection. Since you're on Linux use EPOLL Edge Triggered Multiplexing. Most newer web frameworks use this technology. For more info check 10K Problem.
By eliminating threads from the equation you're eliminating the possibilities of a deadlock and reducing the complexity of debugging / worrying about thread safe variables.
Ensure each connection when finished is completely closed.
Ensure that you do not have some new firewall rules that popped up in iptables since the upgrade.
Check any firewalls on the network to see if they are restricting certain types of activity (is your server on a new IP since the upgrade?)
In short I would put my money on a thread deadlock and / or starvation. I've personally conducted experiments in which I've created a multithreaded server vs a single threaded server using Epoll. The results where night and day, Epoll blows away multithreaded implementation (for I/O) and makes the code simpler to write, debug and maintain.
So I recently decided to dabble into winsock and network programming a bit by using a guide I found and searching the net, but I have run into a problem which I am not really sure how i should solve.
I'm trying to make a very simple chat system, I've got a working server program and client program, and if I only use on client (sending the message back to the same client) It seems to work perfectly fine. The problem appears when I try to have multiple clients connect. I get error 10048 from WSAgetlasterror and it seems to be the bind function that is the source, more specificly the fact that I am trying to bind on the same port twice, (once for each client). From looking around on msdn and forums it seems to be possible to get around this problem by using setsockopt, but I'm not really sure what changes I should make, and also if this is the smartest solution.
I mean I want the clients to connect to the same port don't I? How else will the client program know what to connect to? Or am I just missing something? As I said I have no prior experience with winsock or any other network programming so I might be doing things in a stupid way.
int listenOnPort(int portno, SOCKET& reciever, SOCKET s){
int error = WSAStartup(0x0202, &w);
if (error)
{
cout << "Error starting WSA";
return false; //for some reason we couldn't start Winsock
}
if (w.wVersion != 0x0202) //Wrong Winsock version?
{
cout << "Wrong Winsock version";
WSACleanup();
return false;
}
SOCKADDR_IN addr; // The address structure for a TCP socket
addr.sin_family = AF_INET; // Address family
addr.sin_port = htons(portno); // Assign port no to this socket
//Accept a connection from any IP using INADDR_ANY
//You could pass inet_addr("0.0.0.0") instead to accomplish the
//same thing. If you want only to watch for a connection from a
//specific IP, specify that //instead.
addr.sin_addr.s_addr = htonl(INADDR_ANY);
s = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); // Create a socket
if (s == INVALID_SOCKET)
{
cout << "Couldn't create the socket";
return false; //Don't continue if we couldn't create a //socket!!
}
if (bind(s, (LPSOCKADDR)&addr, sizeof(addr)) == SOCKET_ERROR)
{
//We couldn't bind (this will happen if you try to bind to the same
//socket more than once)
cout << "Error binding the socket";
return false;
}
//Now we can start listening (allowing as many connections as possible to
//be made at the same time using SOMAXCONN). You could specify any
//integer value equal to or lesser than SOMAXCONN instead for custom
//purposes). The function will not //return until a connection request is
//made
listen(s, 1);
reciever = accept(s, NULL, NULL);
cout << "connection established\n\n";
//Don't forget to clean up with CloseConnection()!}
int main(){
e = WSAGetLastError();
listenOnPort(1337, r1, s1);
cout << "First Client connected\n\n";
e = WSAGetLastError();
listenOnPort(1338, r2, s2);
cout << "Second Client connected\n\n";
e = WSAGetLastError();
std::thread com1(communicate, r1, r2);
std::thread com2(communicate, r2, r1);
com1.join();
com2.join();
//system("pause");
closeConnection();}
You have to have 1 thread in the server dedicated to accept new connections.
The process listens to new connections and assigns a new port (in the server) for that connection, making the default port available for new connections.
in the server you will have N+1 socket ports open at any time when N is the number of clients the server has and 1 is the socket listening to new connections.
Take a look at this:
http://www.codeproject.com/Articles/7785/Single-Server-With-Multiple-Clients-a-Simple-C-Imp
My program freezes when trying to connect to an offline server , it doesn't do that if it's online, i know it's trying to connect several times.
is there a way to do that without blocking the main code?
my connection function
bool WSockClient::ConnectServer(int PortNumber, char *IP)
{
SetClientSockAddr(&sockAddr, PortNumber, IP); // Settings
if((hSocket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)) == INVALID_SOCKET)
return false;
if(connect(hSocket, (sockaddr*)(&sockAddr), sizeof(sockAddr)) != 0) // Connect to the server
return false;
//cout << "Attempting to connect to " << inet_ntoa(sockAddr.sin_addr) << endl;
return true;
}
By default, a socket will be 'blocking', meaning that certain calls (like connect) will block the execution of your program until the operation has been completed. On MS-Windows, you can change the socket to 'non-blocking' using a call to ioctlsocket.
For a non-blocking socket, the connect call will return immediately and you'll have to use select to find out if the connection was successful. You can find some additional info here
I'm attempting to create a method that listens for a connection request to a specific port using a TCP protocol, with no libraries other than those that come with the Windows OS. The method seems to work fine with creating a socket and binding to a port; the problem seems to be with the listen() function. Even with no connection request to any port, it continually returns the value of zero, meaning, straight off of Microsoft's website -
If no error occurs, listen returns zero.
The strange part is that this happens with all port values; it seems to find a connection request for randomly attempted ports, ranging from 1234, to 8000, to -154326. For each of these, it's returning a value of zero.
What it should be doing is continually running until a connection request is found (this is what SOMAXCONN apparently indicates); once again, straight off of Microsoft's website -
If there are no available socket descriptors, listen attempts to continue to function.
Here is the method itself -
bool listenOnPort(SOCKET networkSocket, int portNumber) {
WSADATA wsadata;
int error = WSAStartup(0x0202, &wsadata);
if(error) {
cout << "Failed to start up Windows Sockets API." << endl;
return false;
}
if(wsadata.wVersion != 0x0202) {
WSACleanup();
cout << "Failed to find a valid Windows Sockets API." << endl;
return false;
}
SOCKADDR_IN address;
address.sin_family = AF_INET;
address.sin_port = htons(portNumber);
address.sin_addr.s_addr = htonl(INADDR_ANY);
networkSocket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if(networkSocket == INVALID_SOCKET) {
cout << "Failed to create a network socket." << endl;
return false;
}
if(bind(networkSocket, (LPSOCKADDR)&address, sizeof(address)) == SOCKET_ERROR) {
cout << "Failed to bind to the port." << endl;
return false;
}
cout << "Listening for a connection to port " << portNumber <<"..." << endl;
listen(networkSocket, SOMAXCONN);
cout << "Found a connection!" << endl;
}
Any explanation/word of advice is appreciated - thank you ahead of time!
You've confused listen with accept. listen reserves the port for your application, and queues incoming connections. accept waits for an incoming connection (if one isn't already queued).
listen will succeed when there is no incoming connection attempt.
http://linux.die.net/man/2/listen
listen() marks the socket referred to by sockfd as a passive socket, that is, as a socket that will be used to accept incoming connection requests using accept(2).
You must call "listen()" before you can call "accept()"; but "accept()" is the call that accepts new connections (and gives you a new socket for each new connection).
Here's the man page for "accept()":
http://linux.die.net/man/2/accept
Better, look at Beej's Guide for an excellent introduction to sockets programming:
http://beej.us/guide/bgnet/output/html/multipage/
PS:
And don't forget to call WSAStartup() if you're using Windows sockets :)