I'm trying to build a simulation for multiple socket clients.
My server has the following code to listen to multiple clients
My socket are from a very simple class drive from CAsyncSocket and my environment is windows MFC.
m_server.Create(....); // with the correct values
if (m_server.Listen()==FALSE)
and later on the OnSocketAccept() function
if (m_server.Accept(tempSock))
{
CSocketThread* pThread = (CSocketThread*)AfxBeginThread(RUNTIME_CLASS(CSocketThread), THREAD_PRIORITY_NORMAL, 0, CREATE_SUSPENDED);
...
My simulation apps has the following code:
for (int i = 0; i < numOfClients; i++)
{
m_sConnected[i].Create();
int rVal = m_sConnected[i].Connect(csIPAddress.GetString(), m_port);
That doesn't work.
In WireShark I can see that my (numOfClients = 10 for example) 10 clients are connected with different client source port.
But each new socket of m_sConnected[i] is becoming NULL after the second connection to all sockets including m_sConnected[0].
Closing the sockets or destroy the simulation app create socket close at the server side for all open threads for the listen sockets.
What is the problem?
Can I use the same process/thread for all my socket clients?
10x
UrAv.
your problem is that you are not using the CSocketThread object the right way.
as mentiend in microsoft documention
after the accept function you need to do the following :
CSockThread* pSockThread = (CSockThread*)AfxBeginThread( RUNTIME_CLASS(CSockThread), THREAD_PRIORITY_NORMAL, 0, CREATE_SUSPENDED);
if (NULL != pSockThread) {
// Detach the newly accepted socket and save
//the SOCKET handle in our new thread object.
//After detaching it, it should no longer be
//used in the context of this thread.
pSockThread->m_hConnected = sConnected.Detach();
pSockThread->ResumeThread();
} }
when you attach your socket to the thread then it will run.
link to microsoft doc:
https://msdn.microsoft.com/en-us/library/wxzt95kb.aspx
your solution has worked for me. I have used multiple threads to stress test the server in c++ under linux. Pasting my code, it will be helpful to somebody...Experts can improve my code, if they find any flaws in my handling of code. I know, I am doing something wrong but no other go to test the server as no one has provided the solution for this till now. I am able to test the server for 100000 clients using this code. - Kranti.
include //for threading , link with lpthread
void *connection_handler(void *);
#define PORT 9998
#define SERVER_IP "127.0.0.1"
#define MAXSZ 100
#define MAXSOCK 70000
int main()
{
int sockfd[MAXSOCK];//to create socket
int socket_desc , new_socket[MAXSOCK], *new_sock;
struct sockaddr_in serverAddress;//client will connect on this
int n;
char msg1[MAXSZ];
char msg2[MAXSZ];
int NoOfClients = MAXSOCK;
memset(msg2,0,100);
void *ret;
for(int i=0;i<NoOfClients;i++){
//create socket
sockfd[i]=socket(AF_INET,SOCK_STREAM,0);
//initialize the socket addresses
memset(&serverAddress,0,sizeof(serverAddress));
serverAddress.sin_family=AF_INET;
serverAddress.sin_addr.s_addr=inet_addr(SERVER_IP);
serverAddress.sin_port=htons(PORT);
//client connect to server on port
new_socket[i] = connect(sockfd[i],(struct sockaddr *)&serverAddress,sizeof(serverAddress));
printf("new socket connected= %d",new_socket[i]);
pthread_t sniffer_thread[MAXSOCK];
new_sock = malloc(sizeof(int));
*new_sock = new_socket[i];
int p=-1;
if( p = pthread_create( &sniffer_thread[i] , NULL , connection_handler , (void*) new_sock) < 0)
{
perror("could not create thread");
return 1;
}
}
return 0;
}
/*
* This will handle connection for each client
* */
void *connection_handler(void *socket_desc)
{
//Get the socket descriptor
int sock = *(int*)socket_desc;
int read_size;
char *message , client_message[50];
printf("we are in connection handler");
//Send some messages to the server
message = "Greetings! I am your connection handler\n";
int wlen = write(sock , message , strlen(message));
printf("write length is %d", wlen);
//Free the socket pointer
//close(sock);
free(sock);
return 0;
}
Related
I have a server which uses a ZMQ_ROUTER to communicate with ZMQ_DEALER clients. I set the ZMQ_HEARTBEAT_IVL and ZMQ_HEARTBEAT_TTL options on the client socket to make the client and server ping pong each other. Beside, because of the ZMQ_HEARTBEAT_TTL option, the server will timeout the connection if it does not receive any pings from the client in a time period, according to zmq man page:
The ZMQ_HEARTBEAT_TTL option shall set the timeout on the remote peer for ZMTP heartbeats. If
this option is greater than 0, the remote side shall time out the connection if it does not
receive any more traffic within the TTL period. This option does not have any effect if
ZMQ_HEARTBEAT_IVL is not set or is 0. Internally, this value is rounded down to the nearest
decisecond, any value less than 100 will have no effect.
Therefore, what I expect the server to behave is that, when it does not receive any traffic from a client in a time period, it will close the connection to that client and discard all the messages in the outgoing queue after the linger time expires. I create a toy example to check if my hypothesis is correct and it turns out that it is not. The chain of events is as followed:
The server sends a bunch of data to the client.
The client receives and processes the data, which is slow.
All send commands return successfully.
While the client is still receiving the data, I unplug the internet cable.
After a few seconds (set by the ZMQ_HEARTBEAT_TTL option), the server starts sending FIN signals to the client, which are not being ACKed back.
The outgoing messages are not discarded (I check the memory consumption) even after a while. They are discarded only if I call zmq_close on the router socket.
So my question is, is this suppose to be how one should use the ZMQ heartbeat mechanism? If it is not then is there any solution for what I want to achieve? I figure that I can do heartbeat myself instead of using ZMQ's built in. However, even if I do, it seems that ZMQ does not provide a way to close a connection between a ZMQ_ROUTER and a ZMQ_DEALER, although that another version of ZMQ_ROUTER - ZMQ_STREAM provides a way to do this by sending an identity frame followed by an empty frame.
The toy example is below, any help would be thankful.
Server's side:
#include <zmq.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
int main(int argc, char **argv)
{
void *context = zmq_ctx_new();
void *router = zmq_socket(context, ZMQ_ROUTER);
int router_mandatory = 1;
zmq_setsockopt(router, ZMQ_ROUTER_MANDATORY, &router_mandatory, sizeof(router_mandatory));
int hwm = 0;
zmq_setsockopt(router, ZMQ_SNDHWM, &hwm, sizeof(hwm));
int linger = 3000;
zmq_setsockopt(router, ZMQ_LINGER, &linger, sizeof(linger));
char bind_addr[1024];
sprintf(bind_addr, "tcp://%s:%s", argv[1], argv[2]);
if (zmq_bind(router, bind_addr) == -1) {
perror("ERROR");
exit(1);
}
// Receive client identity (only 1)
zmq_msg_t identity;
zmq_msg_init(&identity);
zmq_msg_recv(&identity, router, 0);
zmq_msg_t dump;
zmq_msg_init(&dump);
zmq_msg_recv(&dump, router, 0);
printf("%s\n", (char *) zmq_msg_data(&dump)); // hello
zmq_msg_close(&dump);
char buff[1 << 16];
for (int i = 0; i < 50000; ++i) {
if (zmq_send(router, zmq_msg_data(&identity),
zmq_msg_size(&identity),
ZMQ_SNDMORE) == -1) {
perror("ERROR");
exit(1);
}
if (zmq_send(router, buff, 1 << 16, 0) == -1) {
perror("ERROR");
exit(1);
}
}
printf("OK IM DONE SENDING\n");
// All send commands have returned successfully
// While the client is still receiving data, I unplug the intenet cable on the client machine
// After a while, the server starts sending FIN signals
printf("SLEEP before closing\n"); // At this point, the messages are not discarded (memory usage is high).
getchar();
zmq_close(router);
zmq_ctx_destroy(context);
}
Client's side:
#include <zmq.h>
#include <stdlib.h>
#include <string.h>
int main(int argc, char **argv)
{
void *context = zmq_ctx_new();
void *dealer = zmq_socket(context, ZMQ_DEALER);
int heartbeat_ivl = 3000;
int heartbeat_timeout = 6000;
zmq_setsockopt(dealer, ZMQ_HEARTBEAT_IVL, &heartbeat_ivl, sizeof(heartbeat_ivl));
zmq_setsockopt(dealer, ZMQ_HEARTBEAT_TIMEOUT, &heartbeat_timeout, sizeof(heartbeat_timeout));
zmq_setsockopt(dealer, ZMQ_HEARTBEAT_TTL, &heartbeat_timeout, sizeof(heartbeat_timeout));
int hwm = 0;
zmq_setsockopt(dealer, ZMQ_RCVHWM, &hwm, sizeof(hwm));
char connect_addr[1024];
sprintf(connect_addr, "tcp://%s:%s", argv[1], argv[2]);
zmq_connect(dealer, connect_addr);
zmq_send(dealer, "hello", 6, 0);
size_t size = 0;
int i = 0;
while (size < (1ll << 16) * 50000) {
zmq_msg_t msg;
zmq_msg_init(&msg);
if (zmq_msg_recv(&msg, dealer, 0) == -1) {
perror("ERROR");
exit(1);
}
size += zmq_msg_size(&msg);
printf("i = %d, size = %ld, total = %ld\n", i, zmq_msg_size(&msg), size); // This causes the cliet to be slow
// Somewhere in this loop I unplug the internet cable.
// The client starts sending FIN signals as well as trying to reconnect. The recv command hangs forever.
zmq_msg_close(&msg);
++i;
}
zmq_close(dealer);
zmq_ctx_destroy(context);
}
PS: I know that setting the highwater mark to unlimited is bad practice, however I figure that the problem will be the same even if the highwater mark is low so let's ignore it for now.
I have been trying to figure out this problem for over a month now. I have no where else to turn.
I have a server that listens to many multicast channels (100ish). Each socket is its own thread. Then I have a client listener (single threaded) that handles all incoming connections, disconnects, and client messaging within the same server. The idea is that a client comes in, connects, requests data from a multicast channels and I send the data back to the client. The client stays connected and I relay the UDP data back to the client. The client can either request UDP or TCP has the protocol for the data relay. At one point this was working beautifully for a couple of weeks. We did some code and kernel changes, and now we cant figure out whats gone wrong.
The server will run for hours and have hundreds of clients connected throughout the day. But at some point, randomly, the server will just stop. And by stop, I mean: all UDP sockets stop receiving/handling data (tcpdump shows data still coming to the box), the client_listener thread stops receiving client packets. BUT!!! the main client_listener socket can still receive new connections and new disconnects on the main socket. On a new connection, the main sockets is able to send a "Connection Established" packet back to the client, but then when the client responds, the select never returns.
I can post code if someone would like. If anyone has any suggestions where to look or if this sounds like something. Please let me know.
If you have any questions, please ask.
Thank you.
I would like to share my TCP Server code:
This is a single thread. Works fine for hours and then I will only receive "New Connections" and "Disconnects". NO CLIENT PACKETS WILL COME IN.
int opt = 1;
int addrlen;
int sd;
int max_sd;
int valread;
int activity;
int new_socket;
char buffer[MAX_BUFFER_SIZE];
int client_socket[m_max_clients];
struct sockaddr_in address;
fd_set readfds;
for(int i = 0; i<m_max_clients; i++)
{
client_socket[i]=0;
}
if((m_master_socket = socket(AF_INET,SOCK_STREAM,0))==0)
LOG(FATAL)<<"Unable to create master socket";
if(setsockopt(m_master_socket,SOL_SOCKET,SO_REUSEADDR,(char*)&opt,sizeof(opt))<0)
LOG(FATAL)<<"Unable to set master socket";
address.sin_family = AF_INET;
address.sin_addr.s_addr = INADDR_ANY;
address.sin_port = htons(m_listenPort);
if(bind(m_master_socket,(struct sockaddr*)& address, sizeof(address))!=0)
LOG(FATAL)<<"Unable to bind master socket";
if(listen(m_master_socket,SOMAXCONN)!=0)
LOG(FATAL)<<"listen() failed with err";
addrlen = sizeof(address);
LOG(INFO)<<"Waiting for connections......";
while(true)
{
FD_ZERO(&readfds);
FD_SET(m_master_socket, &readfds);
max_sd = m_master_socket;
for(int i = 0; i<m_max_clients; i++)
{
sd = client_socket[i];
if(sd > 0)
FD_SET(sd, &readfds);
if(sd>max_sd)
max_sd = sd;
}
activity = select(max_sd+1,&readfds,NULL,NULL,NULL);
if((activity<0)&&(errno!=EINTR))
{
// int err = errno;
// LOG(ERROR)<<"SELECT ERROR:"<<activity<<" "<<err;
continue;
}
if(FD_ISSET(m_master_socket, &readfds))
{
if((new_socket = accept(m_master_socket,(struct sockaddr*)&address, (socklen_t*)&addrlen))<0)
LOG(FATAL)<<"ERROR:ACCEPT FAILED!";
LOG(INFO)<<"New Connection, socket fd is (" << new_socket << ") client_addr:" << inet_ntoa(address.sin_addr) << " Port:" << ntohs(address.sin_port);
for(int i =0;i<m_max_clients;i++)
{
if(client_socket[i]==0)
{
//try to set the socket to non blocking, tcp nagle and keep alive
if ( !SetSocketBlockingEnabled(new_socket, false) )
LOG(INFO)<<"UNABLE TO SET NON-BLOCK: ("<<new_socket<<")" ;
if ( !SetSocketNoDelay(new_socket,false) )
LOG(INFO)<<"UNABLE TO SET DELAY: ("<<new_socket<<")" ;
// if ( !SetSocketKeepAlive(new_socket,true) )
// LOG(INFO)<<"UNABLE TO SET KeepAlive: ("<<new_socket<<")" ;
ClientConnection* con = new ClientConnection(m_mocSrv, m_udpPortGenerator, inet_ntoa(address.sin_addr), ntohs(address.sin_port), new_socket);
if(con->login())
{
client_socket[i] = new_socket;
m_clientConnectionSocketMap[new_socket] = con;
LOG(INFO)<<"Client Connection Logon Complete";
}
else
delete con;
break;
}
}//for
}
else
{
try{
for(int i = 0; i<m_max_clients; i++)
{
sd = client_socket[i];
if(FD_ISSET(sd,&readfds))
{
if ( (valread = recv(sd, buffer, sizeof(buffer),MSG_DONTWAIT|MSG_NOSIGNAL)) <= 0 )
{
//remove from the fd listening set
LOG(INFO)<<"RESET CLIENT_SOCKET:("<<sd<<")";
client_socket[i]=0;
handleDisconnect(sd,true);
}
else
{
std::map<int, ClientConnection*>::iterator client_connection_socket_iter = m_clientConnectionSocketMap.find(sd);
if(client_connection_socket_iter != m_clientConnectionSocketMap.end())
{
client_connection_socket_iter->second->handle_message(buffer, valread);
if(client_connection_socket_iter->second->m_logoff)
{
LOG(INFO)<<"SOCKET LOGGED OFF:"<<sd;
client_socket[i]=0;
handleDisconnect(sd,true);
}
}
else
{
LOG(ERROR)<<"UNABLE TO FIND SOCKET DESCRIPTOR:"<<sd;
}
}
}
}
}catch(...)
{
LOG(ERROR)<<"EXCEPTION CATCH!!!";
}
}
}
From the information given I would state the following:
Do not use a thread for each connection. Since you're on Linux use EPOLL Edge Triggered Multiplexing. Most newer web frameworks use this technology. For more info check 10K Problem.
By eliminating threads from the equation you're eliminating the possibilities of a deadlock and reducing the complexity of debugging / worrying about thread safe variables.
Ensure each connection when finished is completely closed.
Ensure that you do not have some new firewall rules that popped up in iptables since the upgrade.
Check any firewalls on the network to see if they are restricting certain types of activity (is your server on a new IP since the upgrade?)
In short I would put my money on a thread deadlock and / or starvation. I've personally conducted experiments in which I've created a multithreaded server vs a single threaded server using Epoll. The results where night and day, Epoll blows away multithreaded implementation (for I/O) and makes the code simpler to write, debug and maintain.
I have to create a basic p2p connection with c++ sockets, which means each user has a server for listening onto connections and and a client for connecting, right?
For now I'm trying to create a master client which has a dedicated server and is a client too.
This means creating the server and client in the same program and I have used fork() which creates a child process of the server and the parent is the client. Now, fork works fine and I'm using select() to check sockets for reading data and i have modeled the server on this http://beej.us/guide/bgnet/output/html/multipage/advanced.html#select
Now when I run the program, the master client is able to connect to its own dedicated server, but the messages don't always get received by the server. Sometimes, it receives it, sometimes it doesn't. Any idea why?
Also, when a second client gets connected to the master client, and it doesn't have it's own server for now, the server shows that it gets a new connection, but when I write the message and send it, it doesn't receive any message from the second client, but it receives a message from the master client sometimes and not always.
EDIT: Added cout.flush
EDIT: I think forking the process causes some delay when a client and server run on the same program.
UPDATE: Added the new server code which causes a delay by one message (in response to the comments)
Here's the code.
SERVER CODE
while (1) {
unsigned int s;
readsocks = socks;
if (select(maxsock + 1, &readsocks, NULL, NULL, NULL) == -1) {
perror("select");
return ;
}
for (s = 0; s <= maxsock; s++) {
if (FD_ISSET(s, &readsocks)) {
//printf("socket %d was ready\n", s);
if (s == sock) {
/* New connection */
cout<<"\n New Connection";
cout.flush();
int newsock;
struct sockaddr_in their_addr;
socklen_t size = sizeof(their_addr);
newsock = accept(sock, (struct sockaddr*)&their_addr, &size);
if (newsock == -1) {
perror("accept");
}
else {
printf("Got a connection from %s on port %d\n",
inet_ntoa(their_addr.sin_addr), htons(their_addr.sin_port));
FD_SET(newsock, &socks);
if (newsock > maxsock) {
maxsock = newsock;
}
}
}
else {
/* Handle read or disconnection */
handle(s, &socks);
}
}
}
}
void handle(int newsock, fd_set *set)
{
char buf[256];
bzero(buf, 256);
/* send(), recv(), close() */
if(read(newsock, buf, 256)<=0){
cout<<"\n No data";
FD_CLR(newsock, set);
cout.flush();
}
else {
string temp(buf);
cout<<"\n Server: "<<temp;
cout.flush();
}
/* Call FD_CLR(newsock, set) on disconnection */
}
I've programmed a simple c++ client which should communicate with another program using a listener thread (a downloaded EchoServer in Java) over the local loop on Ubuntu 14.04. However, there's a problem in the following situation:
Client connects to server
Server sends greeting message
Client receives message and sends a new message to server
Client waits for an answer; the main thread sleeps and the listener thread waits for an answer using recv()
In the next step, the server should receive the message sent by the server, but it doesn't. Instead it first receives the message once the client terminates.
I think that the problem is that the client blocks resources and thus not allowing the server to receive the message, but I'm not sure. Unfortunately I don't have the option to test this on two separate machines.
Code snippets:
// main method
int main(void) {
Client client("127.0.0.1", 13050);
std::cout << client.open() << std::endl;
client.attachListener(foo);
usleep(1000 * 1000 * 2);
std::cout << client.send("hello") << std::endl;
usleep(1000 * 1000 * 5);
}
// send method
int Client::send(const char* msg) {
return write(sockfd, msg, strlen(msg));
}
// listener function
void* Client::listen() {
char buffer[256];
unsigned int receive_size = 0;
while(true) {
receive_size = 0;
while((receive_size = recv(sockfd, &buffer, 256, 0)) > 0) {
buffer[receive_size] = '\0';
msgHandler(buffer);
bzero(&buffer, 256);
}
if(receive_size == 0) {
msgHandler("Server disconnected");
} else if(receive_size == 1) {
msgHandler("Connection failure!");
}
}
return NULL;
}
Output:
1
Welcome to the Java EchoServer. Type 'bye' to close.
6
The EchoServer implementations typically want to see a newline on the message you send before they'll echo it back. Instead of client.send("hello") try client.send("hello\n").
Also, though this isn't really necessary for an application you're just experimenting with, you might want to turn off the Nagle algorithm on your client socket so that small messages get sent immediately. Add code like this just after the point where you call connect with client socket:
int flag = 1;
int res = setsockopt(sockfd, IPPROTO_TCP, TCP_NODELAY, &flag, sizeof flag);
if (res < 0) // handle setsockopt failure here...
This code requires these header files:
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
Firstly I'm coding in c++ and running in Linux/CentOS 6.4
So after a really long time and reading a lot of different books on sockets, I finally have at least my client and my server partially working.
First I want to continuously accept messages from different clients, I have already setup the client, and it finally successfully compiled at least. Now I need to set up my server so that I can properly test.
What I'm doing is implementing the dining philosopher problem with sockets, with each client/philosopher representing a different process. I was going to go through this whole thing, where the server was going to keep track of everything, like the states of all the client. That was too difficult, I have now just created the client just to send their status to the server and the server prints it out.
I was thinking of putting a do/while loop to continuously accept messages, but not sure what I should use to stop the loop. Note that I will have a while loop set up in my client, which is signaled to stop after an elapsed amount of time. It should then close that particular client. I do have a signal in my serve, but I am not sure it works.
#include "helper.h"
char buffer[4096];
void sigchld_handler(int signo)
{
while (waitpid(-1, NULL, WNOHANG) > 0);
}
void client(int &newsock, int nread)
{
do
{
int nread = recv(newsock, buffer,sizeof(buffer), 0);
puts(buffer);
}while(nread!=0);
}
int main(int argc, char *argv[])
{
struct sockaddr_in sAddr, cli_addr;
socklen_t client_len;
int listensock;
int newsock;
int result;
int nread=1;
pid_t childid; ;
int status;
if((listensock = socket(AF_INET, SOCK_STREAM, 0))<0)
{
perror("Problem in creating socket");
exit(2);
}
sAddr.sin_family = AF_INET;
sAddr.sin_port = htons(3333);
sAddr.sin_addr.s_addr = htonl(INADDR_ANY);
bind(listensock, (struct sockaddr *) &sAddr, sizeof(sAddr));
if (result < 0) {
perror("exserver2");
return 0;
}
result = listen(listensock, 5);
if (result < 0) {
perror("exserver2");
return 0;
}
signal(SIGCHLD, sigchld_handler);
while (1) {
client_len = sizeof(cli_addr);
newsock = accept(listensock,(struct sockaddr *)&cli_addr, &client_len);
if ((childid = fork()) == 0) {
printf("child process %i created.\n", getpid());
close(listensock);
client(newsock, nread);
}
if(status<0)
{
printf("%s\n" "Read error");
exit(1);
}
close(newsock);
}
}
You need a multiplexing syscall like poll(2) (or the old, nearly obsolete, select(2)syscall). You may want to use some (or implement your own) event loop. See this & that answer. Read about the C10K problem.
Every server needs an event loop.
Read Advanced Linux Programming (or some Posix network programming book).
You may want to simply run your server program under tcpserver (see http://cr.yp.to/ucspi-tcp.html). This will spawn a new instance of your program each time a client connects to your program (on the port that you specify). This way, you can focus on the core logic of your program, and let tcpserver handle all of the heavy lifting as far as the socket programming, etc. tcpserver will pass input from the client to your program's stdin, and output from your programs stdout will be sent back to the client.