I am writing on a small tcp chat server, but I am encountering some problems I canĀ“t figure out how to solve "elegantly".
Below is the code for my main loop: it does:
1.Initiates a vector with the basic event, which is flagged, when a new tcp connection is made.
2. gets this connection and pushes it back into a vector, too. Then with the socket it creates a CSingleConnection object and passes the socket into it.
2.1. gets the event from the CSingleConnection, which is flagged when the connection receives data...
3. when it receives data. the wait is fullfilled and returns the number of the handle in the array... with all those other vectors it seems i can determine which one is sending now...
but as everybody can see: this methodology is really poorly... I cant figure out how to do all this better, with getting the connection socket, creating a single connection and so on :/...
Any suggestions, improvements, etc?...
void CServer::MainLoop()
{
DWORD dwResult = 0;
bool bMainLoop = true;
std::vector<std::string> vecData;
std::vector<HANDLE> vecEvents; //Contains the handles to wait on
std::vector<SOCKET> vecSocks; //contains the sockets
enum
{
ACCEPTOR = 0, //First element: sequence is mandatory
EVENTSIZE //Keep as the last element!
};
//initiate the vector with the basic handles
vecEvents.clear();
GetBasicEvents(vecEvents);
while(bMainLoop)
{
//wait for event handle(s)
dwResult = WaitForMultipleObjects(vecEvents.size(), &vecEvents[0], true, INFINITE);
//New connection(s) made
if(dwResult == (int)ACCEPTOR)
{
//Get the sockets for the new connections
m_pAcceptor->GetOutData(vecSocks);
//Create new connections
for(unsigned int i = 0; i < vecSocks.size(); i++)
{
//Add a new connection
CClientConnection Conn(vecSocks[i]);
m_vecConnections.push_back(Conn);
//Add event
vecEvents.push_back(Conn.GetOutEvent());
}
}
//Data from one of the connections
if(dwResult >= (int)EVENTSIZE)
{
Inc::MSG Msg;
//get received string data
m_vecConnections[dwResult].GetOutData(vecData);
//handle the data
for(unsigned int i = 0; i < vecData.size(); i++)
{
//convert data into message
if(Inc::StringToMessage(vecData[i], Msg) != Inc::SOK)
continue;
//Add the socket to the sender information
Msg.Sender.sock = vecSocks[dwResult];
//Evaluate and delegate data and task
EvaluateMessage(Msg);
}
}
}
}
Do not re-invent the wheel, use Boost.ASIO. It is well optimized utilizing kernel specific features of different operating systems, designed the way which makes client code architecture simple. There are a lot of examples and documentation, so you just cannot get it wrong.
Related
So I have this winsock application (a server, able to accept multiple clients), where in the main thread I setup the socket and create another thread where I listen for clients (listen_for_clients function).
I also constantly receive data from a device in the main thread, which I afterwards concatenate to char arrays (buffers) of Client objects (BroadcastSample function). Currently I create a thread for each connected client (ProcessClient function), where I initialize a Client object and push it to a global vector of clients after which I send data to this client through the socket whenever the buffer in the corresponding Client object exceeds 4000 characters.
Is there a way I can send data from the main thread to the separate client threads so I don't have to use structs/classes (also to send a green light if I want to send the already accumulated data) and also if I'm going to keep a global container of objects, what is a good way to remove a disconnected client object from it without crashing the program because another thread is using the same container?
struct Client{
int buffer_len;
char current_buffer[5000];
SOCKET s;
};
std::vector<Client*> clientBuffers;
DWORD WINAPI listen_for_clients(LPVOID Param)
{
SOCKET client;
sockaddr_in from;
int fromlen = sizeof(from);
char buf[100];
while(true)
{
client = accept(ListenSocket,(struct sockaddr*)&from,&fromlen);
if(client != INVALID_SOCKET)
{
printf("Client connected\n");
unsigned dwThreadId;
HANDLE hThread = (HANDLE)_beginthreadex(NULL, 0, &ProcessClient, (void*)client, 0, &dwThreadId);
}
}
closesocket(ListenSocket);
WSACleanup();
ExitThread(0);
}
unsigned __stdcall ProcessClient(void *data)
{
SOCKET ClientSocket = (SOCKET)data;
Client * a = new Client();
a->current_buffer[0] = '\0';
a->buffer_len = 0;
a->s = ClientSocket;
clientBuffers.push_back(a);
char szBuffer[255];
while(true)
{
if(a->buffer_len > 4000)
{
send(ClientSocket,a->current_buffer,sizeof(a->current_buffer),0);
memset(a->current_buffer,0,5000);
a->buffer_len = 0;
a->current_buffer[0] = '\0';
}
}
exit(1);
}
//function below is called only in main thread, about every 100ms
void BroadcastSample(Sample s)
{
for(std::vector<Client*>::iterator it = clientBuffers.begin(); it != clientBuffers.end(); it++)
{
strcat((*it)->current_buffer,s.to_string);
(*it)->buffer_len += strlen(s.to_string);
}
}
This link has some Microsoft documentation on MS-style mutexes (muticies?).
This other link has some general info on mutexes.
Mutexes are the general mechanism for protecting data which is accessed by multiple threads. There are data structures with built-in thread safety, but in my experience, they usually have caveats that you'll eventually miss. That's just my two cents.
Also, for the record, you shouldn't use strcat, but rather strncat. Also, if one of your client servicing threads accesses one of those buffers after strncat overwrites the old '\0' but before it appends the new one, you'll have a buffer overread (read past end of allocated buffer).
Mutexes will also solve your current busy-waiting problem. I'm not currently near a windows compiler, or I'd try to help more.
I'm coding a TCP Server class based on the I/O multiplexing (select) way.
The basic idea is explained in this chunk of code:
GenericApp.cpp
TServer *server = new Tserver(/*parameters*/);
server->mainLoop();
For now the behavior of the server is independent from the context but in a way that i nedd to improove.
Actual Status
receive(sockFd , buffer);
MSGData * msg= MSGFactory::getInstance()->createMessage(Utils::getHeader(buffer,1024));
EventHandler * rightHandler =eventBinder->getHandler(msg->type());
rightHandler->callback(msg);
At this version the main loop reads from the socket, instantiates the right type of message object and calls the appropriate handler(something may not work properly because it compiles but i have not tested it).
As you can notice this allows a programmer to define his message types and appropriate handlers but once the main loop is started nothing can be done.
I need to make this part of the server more customizable to adapt this class to a bigger
quantity of problems.
MainLoop Code
void TServer::mainLoop()
{
int sockFd;
int connFd;
int maxFd;
int maxi;
int i;
int nready;
maxFd = listenFd;
maxi = -1;
for(i = 0 ; i< FD_SETSIZE ; i++) clients[i] = -1; //Should be in the constructor?
FD_ZERO(&allset); //Should be in the constructor?
FD_SET(listenFd,&allset); //Should be in the constructor?
for(;;)
{
rset = allset;
nready = select (maxFd + 1 , &rset , NULL,NULL,NULL);
if(FD_ISSET( listenFd , &rset ))
{
cliLen = sizeof(cliAddr);
connFd = accept(listenFd , (struct sockaddr *) &cliAddr, &cliLen);
for (i = 0; i < FD_SETSIZE; i++)
{
if (clients[i] < 0)
{
clients[i] = connFd; /* save descriptor */
break;
}
}
if (i == FD_SETSIZE) //!!HANDLE ERROR
FD_SET(connFd, &allset); /* add new descriptor to set */
if (connFd > maxFd) maxFd = connFd; /* for select */
if (i > maxi) maxi = i; /* max index in client[] array */
if (--nready <= 0) continue;
}
for (i = 0; i <= maxi; i++)
{
/* check all clients for data */
if ( (sockFd = clients[i]) < 0) continue;
if (FD_ISSET(sockFd, &rset))
{
//!!SHOULD CLEAN BUFFER BEFORE READ
receive(sockFd , buffer);
MSGData * msg = MSGFactory::getInstance()->createMessage(Utils::getHeader(buffer,1024));
EventHandler * rightHandler =eventBinder->getHandler(msg->type());
rightHandler->callback(msg);
}
if (--nready <= 0) break; /* no more readable descriptors */
}
}
}
Do you have any suggestions on a good way to do this?
Thanks.
Your question requires more than just a stack overflow question. You can find good ideas in these book:
http://www.amazon.com/Pattern-Oriented-Software-Architecture-Concurrent-Networked/dp/0471606952/ref=sr_1_2?s=books&ie=UTF8&qid=1405423386&sr=1-2&keywords=pattern+oriented+software+architecture
http://www.amazon.com/Unix-Network-Programming-Volume-Networking/dp/0131411551/ref=sr_1_1?ie=UTF8&qid=1405433255&sr=8-1&keywords=unix+network+programming
Basically what you're trying to do is a reactor. You can find open source library implementing this pattern. For instance:
http://www.cs.wustl.edu/~schmidt/ACE.html
http://pocoproject.org/
If you want yout handler to have the possibility to do more processing you could give them a reference to your TCPServer and a way to register a socket for the following events:
read, the socket is ready for read
write, the socket is ready for write
accept, the listening socket is ready to accept (read with select)
close, the socket is closed
timeout, the time given to wait for the next event expired (select allow to specify a timeout)
So that the handler can implement all kinds of protocols half-duplex or full-duplex:
In your example there is no way for a handler to answer the received message. This is the role of the write event to let a handler knows when it can send on the socket.
The same is true for the read event. It should not be in your main loop but in the socket read handler.
You may also want to add the possibility to register a handler for an event with a timeout so that you can implement timers and drop idle connections.
This leads to some problems:
Your handler will have to implement a state-machine to react to the network events and update the events it wants to receive.
You handler may want to create and connect new sockets (think about a Web proxy server, an IRC client with DCC, an FTP server, and so on...). For this to work it must have the possibility to create a socket and to register it in your main loop. This means the handler may now receive callbacks for one of the two sockets and there should be a parameter telling the callback which socket it is. Or you will have to implement a handler for each socket and they will comunnicate with a queue of messages. The queue is needed because the readiness of one socket is independent of the readiness of the other. And you may read something on one and not being ready to send it on the other.
You will have to manage the timeouts specified by each handlers which may be different. You may end up with a priority queue for timeouts
As you see this is no simple problem. You may want to reduce the genericity of your framework to simplify its design. (for instance handling only half-duplex protocols like simple HTTP)
I have written a client/server application where the server spawns multiple threads depending upon the request from client.
These threads are expected to send some data to the client(string).
The problem is, data gets overwritten on the client side. How do I tackle this issue ?
I have already read some other threads on similar issue but unable to find the exact solution.
Here is my client code to receive data.
while(1)
{
char buff[MAX_BUFF];
int bytes_read = read(sd,buff,MAX_BUFF);
if(bytes_read == 0)
{
break;
}
else if(bytes_read > 0)
{
if(buff[bytes_read-1]=='$')
{
buff[bytes_read-1]='\0';
cout<<buff;
}
else
{
cout<<buff;
}
}
}
Server Thread code :
void send_data(int sd,char *data)
{
write(sd,data,strlen(data));
cout<<data;
}
void *calcWordCount(void *arg)
{
tdata *tmp = (tdata *)arg;
string line = tmp->line;
string s = tmp->arg;
int sd = tmp->sd_c;
int line_no = tmp->line_no;
int startpos = 0;
int finds = 0;
while ((startpos = line.find(s, startpos)) != std::string::npos)
{
++finds;
startpos+=1;
pthread_mutex_lock(&myMux);
tcount++;
pthread_mutex_unlock(&myMux);
}
pthread_mutex_lock(&mapMux);
int t=wcount[s];
wcount[s]=t+finds;
pthread_mutex_unlock(&mapMux);
char buff[MAX_BUFF];
sprintf(buff,"%s",s.c_str());
sprintf(buff+strlen(buff),"%s"," occured ");
sprintf(buff+strlen(buff),"%d",finds);
sprintf(buff+strlen(buff),"%s"," times on line ");
sprintf(buff+strlen(buff),"%d",line_no);
sprintf(buff+strlen(buff),"\n",strlen("\n"));
send_data(sd,buff);
delete (tdata*)arg;
}
On the server side make sure the shared resource (the socket, along with its associated internal buffer) is protected against the concurrent access.
Define and implement an application level protocol used by the server to make it possible for the client to distinguish what the different threads sent.
As an additional note: One cannot rely on read()/write() reading/writing as much bytes as those two functions were told to read/write. It is an essential necessity to check their return value to learn how much bytes those functions actually read/wrote and loop around them until all data that was intended to be read/written had been read/written.
You should put some mutex to your socket.
When a thread use the socket it should block the socket.
Some mutex example.
I can't help you more without the server code. Because the problem is probably in the server.
I am trying to implement an authentication system using C++/QtTcpSocket for a personal project (A Multiplayer Chess Game).
My friend suggested a method for verifying a user but I wanted to ask if there was an easier or better way. Coming from a Python background and mostly doing this project to develop a deeper understanding of C++.
I will post the method my friend suggested and ask for maybe a better solution.
He built it in a kind of pseudo code fashion. The server is mostly built, I am now hoping to implement Authentication
*cheers
void process_packet(PACKET *pkt)
{
switch(pkt->PacketID)
{
case 0: // let's say packet id 0 is the logon packet; packet contents are username and password
{
//let's say packet size is 101 bytes; packet id was already received, so get the other 100 bytes
unsigned char BUFFER[101] = {0}; // i always add an extra byte to the end of the buffer to allow for off-by-one errors ^_^
int result = recv_packet(pkt->cSocket, 100, BUFFER);
if(result <= 0)
return; // connection error; no packet data was received
unsigned char *UserName = BUFFER+0; //+0 is not neccessary, but the username starts at the beginning. just getting the point across.
unsigned char *PassWord = BUFFER+50;
//side note: if we did "unsigned long *blah = BUFFER+4" or something, we would have to make sure the byte order is right. network byte order is BIG ENDIAN
// WINDOWS byte order is LITTLE ENDIAN
result = QueryDatabase("SELECT username, password FROM chess_players WHERE username = '%s'", FILTER_INVALID_CHARS(UserName));
// check result
unsigned char ServerResponse[2] = {0};
if(result['password'] == PassWord)
{
ServerResponse[0] = 1; // packet id will be 1. the next byte can be 1 or 0 to indicate logon success or failure.
ServerResponse[1] = true; // so packet 0x0101 mean logon success, packet 0x0100 means logon failure
send_packet(pkt->cSocket, ServerResponse, 2);
} else {
ServerResponse[0] = 1;
ServerResponse[1] = false;
send_packet(pkt->cSocket, ServerResponse, 2);
}
}
break;
default:
{
// received an unknown packet id; send a packet to the client that indicates an error_status_t
unsigned char *ServerResponse[2] = {0};
ServerResponse[0] = 2; // packet id 2 means server error
ServerResponse[1] = 0; // error code 0 means 'unknown packet id'
send_packet(pkt_cSocket, ServerResponse, 2);
}
break;
}
delete pkt; // must delete pkt, was created with 'new' in get_client_packets()
}
This seems rather C-stylish and not like the Qt way of doing things.
There is no general answer to your question but my suggestions are the following:
Listen to the newConnection() signal of the QTcpServer. Your handler has to call the nextPendingConnection() to get the next client waiting in the queue. The first thing you will do is probably your user authentication.
Once authenticated, you keep the QTcpSocket in your list of active connections.
Take a look at e.g. the fortune client/server examples how to actually write/read packets.
You might also want to look into the stream operators << to serialize your objects. This is much easier and less error prone than the low-level method you posted. ALso, QDataStream will take care of host and network byte orders automatically.
If you have followed the fortune client/server examples, you should have a QTcpServer (Rfserver) with a QThread subclass (Rfdevice, its instance variable is called thread in the following code) that contains a QTcpSocket (listenSocket).
Having said that, in your server class, listen for incoming connections, my setup looks like this:
void Rfserver::incomingConnection(int socketDescriptor){
if(thread){ //if thread exists, there is probably still an open connection
if(thread->listenSocket){//if thread exists and the listenSocket is filled, there is definately an open connection
if(thread->listenSocket->state() == QAbstractSocket::UnconnectedState){
//but alas, it could just be in the unconnected state, if so kill it.
this->disconnect();
thread->terminate();
thread=0;
connected=false;
}//otherwise, do nothing, because the software is happily connected to a device
}
}
if(!thread){ //if no thread exists, we are by no means connected
thread = new rfdevice(socketDescriptor, this); //set up a new thread
//this first connection communicates the string from your socket to the server parent...use it if you want.
connect( thread, SIGNAL(RemoteButton(QString)),this,SLOT(remoteButton(QString)),Qt::BlockingQueuedConnection);
connect( thread, SIGNAL(error(QTcpSocket::SocketError)),this,SLOT(tcpError(QTcpSocket::SocketError)),Qt::AutoConnection);
connect( thread, SIGNAL(finished()), this, SLOT(threadZero())); //I have a threadZero function that deletes all the data then schedules the socket for deletion.
thread->start();
connected=true;
QString *welcome = new QString("Enter your password:\r\n");
echoCommand(welcome); //this is a function you will implement that sends the welcome message to the pending device.
}
}
Okay, so now, when a device tries to connect to the server the device is presented with "Enter your password:\r\n". Your device will respond to this with a password and username perhaps. But the Qt side of things would look like this:
/*
FUNCTION:read
this is a polling runloop that listens for data as long as the socket is connected or connecting. If a
write is ever scheduled, it will be called from this runloop..
*/
void Rfdevice::read(void){
while((listenSocket->state() == QAbstractSocket::ConnectedState) || (listenSocket->state() == QAbstractSocket::ConnectingState)){
//if there is data available to send write it to the socket
if(dataToSend) this->write();
if(listenSocket->waitForReadyRead(50)) readBytes();
//wait for 50ms for data from the device
//if there is ever data available to be read, read it.
}
}
Your device responds with a username/password in the format username---password\r\n. Then the socket does this:
/*
FUNCTION:readBytes
this is like a callback function because it only gets called when there is data available for read.
It basically converts the data to a string.
*/
void Rfdevice::readBytes(void){
QByteArray newData;
newData = listenSocket->readAll();
QString *recieved = new QString(newData);
QStringList userAndPass = recieved.split("---");//this is your delimiter
QString username = userAndPass.at(0);
QString password = userAndPass.at(1);
//NOW, check the username and password vs your SQL or wherever it's saved.
}
The pseudo-code is pretty complete on the particulars. Hopefully you can put it all together! Let me know if you need more code.
I'm not sure if this is a known issue that I am running into, but I couldn't find a good search string that would give me any useful results.
Anyway, here's the basic rundown:
we've got a relatively simple application that takes data from a source (DB or file) and streams that data over TCP to connected clients as new data comes in. its a relatively low number of clients; i would say at max 10 clients per server, so we have the following rough design:
client: connect to server, set to read (with timeout set to higher than the server heartbeat message frequency). It blocks on read.
server: one listening thread that accepts connections and then spawns a writer thread to read from the data source and write to the client. The writer thread is also detached(using boost::thread so just call the .detach() function). It blocks on writes indefinetly, but does check errno for errors before writing. We start the servers using a single perl script and calling "fork" for each server process.
The problem(s):
at seemingly random times, the client will shutdown with a "connection terminated (SUCCESFUL)" indicating that the remote server shutdown the socket on purpose. However, when this happens the SERVER application ALSO closes, without any errors or anything. it just crashes.
Now, to further the problem, we have multiple instances of the server app being started by a startup script running different files and different ports. When ONE of the servers crashes like this, ALL the servers crash out.
Both the server and client using the same "Connection" library created in-house. It's mostly a C++ wrapper for the C socket calls.
here's some rough code for the write and read function in the Connection libary:
int connectionTimeout_read = 60 * 60 * 1000;
int Socket::readUntil(char* buf, int amount) const
{
int readyFds = epoll_wait(epfd,epEvents,1,connectionTimeout_read);
if(readyFds < 0)
{
status = convertFlagToStatus(errno);
return 0;
}
if(readyFds == 0)
{
status = CONNECTION_TIMEOUT;
return 0;
}
int fd = epEvents[0].data.fd;
if( fd != socket)
{
status = CONNECTION_INCORRECT_SOCKET;
return 0;
}
int rec = recv(fd,buf,amount,MSG_WAITALL);
if(rec == 0)
status = CONNECTION_CLOSED;
else if(rec < 0)
status = convertFlagToStatus(errno);
else
status = CONNECTION_NORMAL;
lastReadBytes = rec;
return rec;
}
int Socket::write(const void* buf, int size) const
{
int readyFds = epoll_wait(epfd,epEvents,1,-1);
if(readyFds < 0)
{
status = convertFlagToStatus(errno);
return 0;
}
if(readyFds == 0)
{
status = CONNECTION_TERMINATED;
return 0;
}
int fd = epEvents[0].data.fd;
if(fd != socket)
{
status = CONNECTION_INCORRECT_SOCKET;
return 0;
}
if(epEvents[0].events != EPOLLOUT)
{
status = CONNECTION_CLOSED;
return 0;
}
int bytesWrote = ::send(socket, buf, size,0);
if(bytesWrote < 0)
status = convertFlagToStatus(errno);
lastWriteBytes = bytesWrote;
return bytesWrote;
}
Any help solving this mystery bug would be great! at the VERY least, I would like it to NOT crash out the server even if the client crashes (which is really strange for me, since there is no two-way communication).
Also, for reference, here is the server listening code:
while(server.getStatus() == connection::CONNECTION_NORMAL)
{
connection::Socket s = server.listen();
if(s.getStatus() != connection::CONNECTION_NORMAL)
{
fprintf(stdout,"failed to accept a socket. error: %s\n",connection::getStatusString(s.getStatus()));
}
DATASOURCE* dataSource;
dataSource = open_datasource(XXXX); /* edited */ if(dataSource == NULL)
{
fprintf(stdout,"FATAL ERROR. DATASOURCE NOT FOUND\n");
return;
}
boost::thread fileSender(Sender(s,dataSource));
fileSender.detach();
}
...And also here is the spawned child sending thread:
::signal(SIGPIPE,SIG_IGN);
//const int headerNeeds = 29;
const int BUFFERSIZE = 2000;
char buf[BUFFERSIZE];
bool running = true;
while(running)
{
memset(buf,'\0',BUFFERSIZE*sizeof(char));
unsigned int readBytes = 0;
while((readBytes = read_datasource(buf,sizeof(unsigned char),BUFFERSIZE,dataSource)) == 0)
{
boost::this_thread::sleep(boost::posix_time::milliseconds(1000));
}
socket.write(buf,readBytes);
if(socket.getStatus() != connection::CONNECTION_NORMAL)
running = false;
}
fprintf(stdout,"socket error: %s\n",connection::getStatusString(socket.getStatus()));
socket.close();
fprintf(stdout,"sender exiting...\n");
Any insights would be welcome! Thanks in advance.
You've probably got everything backwards... when the server crashes, the OS will close all sockets. So the server crash happens first and causes the client to get the disconnect message (FIN flag in a TCP segment, actually), the crash is not a result of the socket closing.
Since you have multiple server processes crashing at the same time, I'd look at resources they share, and also any scheduled tasks that all servers would try to execute at the same time.
EDIT: You don't have a single client connecting to multiple servers, do you? Note that TCP connections are always bidirectional, so the server process does get feedback if a client disconnects. Some internet providers have even been caught generating RST packets on connections that fail some test for suspicious traffic.
Write a signal handler. Make sure it uses only raw I/O functions to log problems (open, write, close, not fwrite, not printf).
Check return values. Check for negative return value from write on a socket, but check all return values.
Thanks for all the comments and suggestions.
After looking through the code and adding the signal handling as Ben suggested, the applications themselves are far more stable. Thank you for all your input.
The original problem, however, was due to a rogue script that one of the admins was running as root that would randomly kill certain processes on the server-side machine (i won't get into what it was trying to do in reality; safe to say it was buggy).
Lesson learned: check the environment.
Thank you all for the advice.