I remake two detached threads to poll() cycle in server: for receiving and sending data.
All work fine, until setting client's sending frequency to 16 ms (< ~200 ms). In this state one thread always wins the race and answers to only one client with ~1 us ping. What I need to do to send and receive data in poll() with two (or one) UDP sockets?
Part of server's code:
struct pollfd pollStruct[2];
// int timeout_sec = 1;
pollStruct[0].fd = getReceiver()->getSocketDesc();
pollStruct[0].events = POLLIN;
pollStruct[0].revents = 0;
pollStruct[1].fd = getDispatcher()->getSocketDesc();
pollStruct[1].events = POLLOUT;
pollStruct[1].revents = 0;
while(true) {
if (poll(pollStruct, 2, 0 /* (-1 / timeout) */) > 0) {
if (pollStruct[0].revents & POLLIN) {
recvfrom(getReceiver()->getSocketDesc(), buffer, getBufferLength(), 0, (struct sockaddr *) ¤tClient, getReceiver()->getLength());
// add message to client's own thread-safe buffer
}
if (pollStruct[1].revents & POLLOUT) {
// get message from thread-safe general buffer after processing
// dequeue one
if (!getBuffer().isEmpty()) {
auto message = getBuffer().dequeue();
if (message != nullptr) {
sendto(getDispatcher()->getSocketDesc(), "hi", 2, 0, message->_addr, *getDispatcher()->getLength());
}
}
}
}
}
Related
I am learning to make a client-server communication model through socket use Winsock (C++) in my school.
And I am facing problem how to make server serve multi client with limit resources, and my mission is handle this problem using multithreading and exploration technical and Specific here is use two function in C++ and winSock is: select() and _beginthreadex(). (with limited assumptions about resources)
If I use only select() function so my server only serve max 1024 client and if I use only _beginthreadex() so it will be limited on resources.
But I need to serve more than 4096 clients just use select() and _beginthreadex() (can not use only one of them because the above reason)
This is my struct of program use select():
SOCKET client[FD_SETSIZE], connSock;
fd_set readfds, initfds; //use initfds to initiate readfds at the begining of every loop step
sockaddr_in clientAddr;
int nEvents, clientAddrLen;
bool ret;
for (int i = 0; i < FD_SETSIZE; i++)
client[i] = 0; // 0 indicates available entry
FD_ZERO(&initfds);
//Step 5: Communicate with clients
while (1) {
FD_SET(listenSock, &initfds);
for (int i = 0; i < FD_SETSIZE; ++i) {
if (client[i] > 0) {
FD_SET(client[i], &initfds);
}
}
readfds = initfds; /* structure assignment */
nEvents = select(0, &readfds, 0, 0, 0);
if (nEvents < 0) {
printf("\nError! Cannot poll sockets: %d", WSAGetLastError());
break;
}
//new client connection
if (FD_ISSET(listenSock, &readfds)) {
clientAddrLen = sizeof(clientAddr);
if ((connSock = accept(listenSock, (sockaddr *)&clientAddr, &clientAddrLen)) < 0) {
printf("\nError! Cannot accept new connection: %d", WSAGetLastError());
break;
}
else {
printf("You got a connection from %s\n", inet_ntoa(clientAddr.sin_addr)); /* prints client's IP */
cout << "\tconnSocket: " << (int)connSock << endl;
int i;
for (i = 0; i < FD_SETSIZE; i++)
if (client[i] == 0) {
client[i] = connSock;
FD_SET(client[i], &initfds);
break;
}
if (i == FD_SETSIZE) {
printf("\nToo many clients.");
closesocket(connSock);
}
if (--nEvents == 0)
continue; //no more event
}
}
//receive data from clients
for (int i = 0; i < FD_SETSIZE; i++) {
if (client[i] == 0)
continue;
if (FD_ISSET(client[i], &readfds)) {
ret = business(client[i]); // the function handle logic of my server
if (ret == false ) {
FD_CLR(client[i], &initfds);
closesocket(client[i]);
client[i] = 0;
}
}
if (--nEvents <= 0)
continue; //no more event
}
}
//Step 5: Close socket
closesocket(listenSock);
With program above my server can not serve more than 4096 client. Somebody can give a solution for my problem ? Thank you very much!
One thread can handle 4096 as you show. That is because a FD_SET is limited to 4096 entries. However you can have multiple threads, and multiple FD_SETs.
Each thread would have its FD_SET to read from and reply, handling up to 4k clients. When the client 4097 arrives, you create a new thread to handle a new set of sockets. You'll need synchronisation around accepting and connecting. But apart from that, the thread handling the first 4k can run independently of the thread handling the next 4k clients.
A full example is out of scope. :-)
I have two detached threads in server: for receiving and sending data.
All work fine, until setting client's sending frequency to 16 ms (< ~200 ms). In this state one thread always wins the race and answers to only one client with ~1 us ping. What I need to do to send and receive data in separate threads with two (or one) UDP sockets?
Part of server's code:
void Server::receive() {
while(true) {
if (recvfrom(getReceiver()->getSocketDesc(), buffer, getBufferLength(), 0, (struct sockaddr *) ¤tClient, getReceiver()->getLength()) < 0) {
// add message to client's own thread-safe buffer
}
}
}
void Server::send() {
while(true) {
// get message from thread-safe general buffer after processing
// dequeue one
if (message != nullptr)
sendto(getDispatcher()->getSocketDesc(), "hi", 2, 0, message->_addr, *getDispatcher()->getLength());
}
}
I am trying to check if a client has send some new data. This actually tells me that i always have new data:
bool ClientHandle::hasData()
{
fd_set temp;
FD_ZERO(&temp);
FD_SET(m_sock, &temp);
//setup the timeout to 1000ms
struct timeval tv;
tv.tv_sec = 0;
tv.tv_usec = 1000;
//temp.fd_count possible?
if (select(m_sock+1, &temp, nullptr, nullptr, &tv) == -1)
{
return false;
}
if (FD_ISSET(m_sock, &temp))
return true;
return false;
}
I am connecting with a java client and send a "connection" message which i read inside of the ctor:
ClientHandle::ClientHandle(SOCKET s) : m_sock(s)
{
while (!hasData())
{
}
char buffer[5];
recv(m_sock, buffer, 4, NULL);
auto i = atoi(buffer);
LOG_INFO << "Byte to receive: " << i;
auto dataBuffer = new char[i + 1]{'\0'};
recv(m_sock, dataBuffer, i, NULL);
LOG_INFO << dataBuffer;
//clean up
delete[] dataBuffer;
}
This seems to work right. After that i keep checking if there is new data which always is true even if the java client does not send any new data.
Here is the java client. Don't judge me it's just for checking the connections. It wont stay like this to send the size information as char[].
public static void main(String[] args) throws UnknownHostException,
IOException {
Socket soc = null;
soc = new Socket("localhost", 6060);
PrintWriter out = new PrintWriter(soc.getOutputStream(), true);
BufferedReader in = new BufferedReader(new InputStreamReader(
soc.getInputStream()));
if (soc != null)
System.out.println("Connected");
out.write("10\0");
out.flush();
out.write("newCon\0");
out.flush();
out.close();
in.close();
soc.close();
}
So what is wrong with the hasData FD_ISSET method?
So what is wrong with the hasData FD_ISSET method?
Actually no. There is a problem with your use of recv().
recv() will return 0 if the client is disconnected and will return this until you close the socket (server-side). You can find this information in the manual.
Even if recv() returns 0, it will "trigger" select().
Knowing that, it's easy to find out the problem: you never check the return value of recv() and so you're unable to say if the client is still connected or not. However, you still add it with FD_SET!
#include <sys/types.h> // for ssize_t
#include <stdio.h> // for perror()
ClientHandle::ClientHandle(SOCKET s) : m_sock(s)
{
while (!hasData())
{
}
char buffer[5];
ssize_t ret = recv(m_sock, buffer, 4, NULL);
if (ret == -1) // error
{
perror("recv");
return ;
}
else if (ret == 0) // m_sock disconnects
{
close(m_sock);
// DO NOT FD_SET m_sock since the socket is now closed
}
else
{
auto i = atoi(buffer);
LOG_INFO << "Byte to receive: " << i;
auto dataBuffer = new char[i + 1]{'\0'};
recv(m_sock, dataBuffer, i, NULL);
LOG_INFO << dataBuffer;
//clean up
delete[] dataBuffer;
}
}
From Steven's book UNIX Networking Programming:
A socket is ready for reading if any of the following four conditions is true:
The number of bytes of data in the socket receive buffer is greater than or equal to the current size of the low-water mark for the socket receive buffer. A read operation on the socket will not block and will return a value greater than 0 (i.e., the data that is ready to be read). We can set this low-water mark using the SO_RCVLOWAT socket option. It defaults to 1 for TCP and UDP sockets.
The read half of the connection is closed (i.e., a TCP connection that has received a FIN). A read operation on the socket will not block and will return 0 (i.e., EOF).
The socket is a listening socket and the number of completed connections is nonzero. An accept on the listening socket will normally not block, although we will describe a timing condition in Section 16.6 under which the accept can block.
A socket error is pending. A read operation on the socket will not block and will return an error (–1) with errno set to the specific error condition. These pending errors can also be fetched and cleared by calling getsockopt and specifying the SO_ERROR socket option.
ISSET is going to return true in all the cases above. After your Java client closes the connection, the socket will be ready for reading in the server.
In ClientHandle::ClientHandle you are not checking the return value of recv and if any data is returned.
Is it blocking in the second call to recv?
You don't check the return value of recv and you don't handle receiving fewer bytes than you asked for. So what do you expect to happen when the connection is closed?
I implemented a program that receives from one socket and sends/receives from the other socket.
For this i use polling of select(), in socket 1, i receive data at a high data rate, while in the other socket i receive periodic message and requests to receive data from the first socket.
When there is no request "from socket 2" to delegate the data from socket 1 to socket2 , i receive data from socket 1 normal and with no problem. However, say i received two requests "socket 2" while data is being received in socket 1, the second request breaks the the data reception as if it could no longer keep up with rate "rate isn't high really is only 150 Hz".
The pseudo code i do in the main():
fd_set readfds, rd_fds, writefds, wr_fds;
struct timeval tv;
do
{
do
{
rd_fds = readfds;
wr_fds = writefds;
FD_ZERO (&rd_fds);
FD_SET (sock1, &rd_fds);
FD_SET (sock2, &rd_fds);
FD_SET (sock1, &wr_fds);
tv.tv_sec = 0;
tv.tv_usec = 20;
int ls = sock2 + 1;
rslt = select (ls, &rd_fds, &wr_fds, NULL, &tv);
}
while (rslt == -1 && errno == EINTR);
if (FD_ISSET (sock1, &rd_fds))
{
rs1 = recvfrom (sock1, buff, size of the buff, ....);
if (rs1 > 0)
{
if (rs1 = alive message)
{
/* system is alive; */
}
else if (rs1 == request message)
{
/* store Request info (list or vector) */
}
else {}
}
}
if (FD_ISSET (StructArg.sock2, &rd_fds))
{
rs2 = recv (sock2, ..., 0);
if (rs2 > 0)
{
if ( /* Message (high rate) is from sock 2 */ )
{
/* process this message and do some computation */
int sp1 = sendto (sock1, .....);
if (sp1 < 0)
{
perror ("Failed data transmission ");
}
else
{
/* increase some counters */
}
}
}
}
if (FD_ISSET (sock1, &wr_fds))
{
/*
if there info stored in the list
do some calculaitons then send to sock 1
*/
if (sendto (sock1, ... ...) < 0)
{
perror ("Failed data transmission");
}
else
{
/* increase counter */
}
}
FD_CLR (sock1, &rd_fds);
FD_CLR (sock2, &rd_fds);
}
while (1);
Again, the question is, why does receiving from sock1 is interrupted if a request is received from sock2, while i am receiving from sock1 (fast messages), i expect interleaved messages in the output based on the timestamps in the message.
Note that nearly all socket functions can block execution unless you've created the socket with the O_NONBLOCK option:
http://pubs.opengroup.org/onlinepubs/009695399/functions/sendto.html
And you'll also have to handle the case where recvfrom only gives you a partial read - unless you use MSG_WAITALL:
http://pubs.opengroup.org/onlinepubs/009695399/functions/recvfrom.html
Personally, I'd use a multi-threaded implementation which can have threads just sit and wait for data on each socket.
As to your final question:
why does receiving from sock1 is interrupted if a request is received from sock2, while i am receiving from sock1 (fast messages), i expect interleaved messages in the output based on the timestamps in the message.
You are slave to the network stack's implementation and there are nearly no guarantees about the sending or receiving of data on one socket relative to another. You are only guaranteed that the data within a socket is properly ordered.
I expect interleaved messages in the output based on the timestamps in the message.
Your expectation is without foundation. If there is data in either socket receive buffer, select() will fire. That's all you can rely on. You don't have any guarantee about timestamps being observed and ordered as between multiple sockets.
I have written simple C/S applications to test the characteristics of non-blocking sockets, here is some brief information about the server and client:
//On linux The server thread will send
//a file to the client using non-blocking socket
void *SendFileThread(void *param){
CFile* theFile = (CFile*) param;
int sockfd = theFile->GetSocket();
set_non_blocking(sockfd);
set_sock_sndbuf(sockfd, 1024 * 64); //set the send buffer to 64K
//get the total packets count of target file
int PacketCOunt = theFile->GetFilePacketsCount();
int CurrPacket = 0;
while (CurrPacket < PacketCount){
char buffer[512];
int len = 0;
//get packet data by packet no.
GetPacketData(currPacket, buffer, len);
//send_non_blocking_sock_data will loop and send
//data into buffer of sockfd until there is error
int ret = send_non_blocking_sock_data(sockfd, buffer, len);
if (ret < 0 && errno == EAGAIN){
continue；
} else if (ret < 0 || ret == 0 ){
break;
} else {
currPacket++;
}
......
}
}
//On windows, the client thread will do something like below
//to receive the file data sent by the server via block socket
void *RecvFileThread(void *param){
int sockfd = (int) param; //blocking socket
set_sock_rcvbuf(sockfd, 1024 * 256); //set the send buffer to 256
while (1){
struct timeval timeout;
timeout.tv_sec = 1;
timeout.tv_usec = 0;
fd_set rds;
FD_ZERO(&rds);
FD_SET(sockfd, &rds)'
//actually, the first parameter of select() is
//ignored on windows, though on linux this parameter
//should be (maximum socket value + 1)
int ret = select(sockfd + 1, &rds, NULL, NULL, &timeout );
if (ret == 0){
// log that timer expires
CLogger::log("RecvFileThread---Calling select() timeouts\n");
} else if (ret) {
//log the number of data it received
int ret = 0;
char buffer[1024 * 256];
int len = recv(sockfd, buffer, sizeof(buffer), 0);
// handle error
process_tcp_data(buffer, len);
} else {
//handle and break;
break;
}
}
}
What surprised me is that the server thread fails frequently because of socket buffer full, e.g. to send a file of 14M size it reports 50000 failures with errno = EAGAIN. However, via logging I observed there are tens of timeouts during the transfer, the flow is like below:
on the Nth loop, select() succeeds and read 256K's data successfully.
on the (N+1)th loop, select() failed with timeout.
on the (N+2)th loop, select() succeeds and read 256K's data successfully.
Why there would be timeouts interleaved during the receving? Can anyone explain this phenomenon?
[UPDATE]
1. Uploading a file of 14M to the server only takes 8 seconds
2. Using the same file with 1), the server takes nearly 30 seconds to send all data to the client.
3. All sockets used by the client are blocking. All sockets used by the server are non-blocking.
Regarding #2, I think timeouts are the reason why #2 takes much more time then #1, and I wonder why there would be so many timeouts when the client is busy in receiving data.
[UPDATE2]
Thanks for comments from #Duck, #ebrob, #EJP, #ja_mesa , I will do more investigation today
then update this post.
Regarding why I send 512 bytes per loop in the server thread, it is because I found the server thread sends data much faster than the client thread receiving them. I am very confused that why timeout happened to the client thread.
Consider this more of a long comment than an answer but as several people have noted the network is orders of magnitude slower than your processor. The point of non-blocking i/o is that the difference is so great that you can actually use it to do real work rather than blocking. Here you are just pounding on the elevator button hoping that makes a difference.
I'm not sure how much of your code is real and how much is chopped up for posting but in the server you don't account for (ret == 0) i.e. normal shutdown by the peer.
The select in the client is wrong. Again, not sure if that was sloppy editing or not but if not then the number of parameters are wrong but, more concerning, the first parameter - i.e. should be the highest file descriptor for select to look at plus one - is zero. Depending on the implementation of select I wonder if that is in fact just turning select into a fancy sleep statement.
You should be calling recv() first and then call select() only if recv() tells you to do so. Don't call select() first, that is a waste of processing. recv() knows if data is immediately available or if it has to wait for data to arrive:
void *RecvFileThread(void *param){
int sockfd = (int) param; //blocking socket
set_sock_rcvbuf(sockfd, 1024 * 256); //set the send buffer to 256
char buffer[1024 * 256];
while (1){
int ret = 0;
int len = recv(sockfd, buffer, sizeof(buffer), 0);
if (len == -1) {
if (WSAGetLastError() != WSAEWOULDBLOCK) {
//handle error
break;
}
struct timeval timeout;
timeout.tv_sec = 1;
timeout.tv_usec = 0;
fd_set rds;
FD_ZERO(&rds);
FD_SET(sockfd, &rds)'
//actually, the first parameter of select() is
//ignored on windows, though on linux this parameter
//should be (maximum socket value + 1)
int ret = select(sockfd + 1, &rds, NULL, &timeout );
if (ret == -1) {
// handle error
break;
}
if (ret == 0) {
// log that timer expires
break;
}
// socket is readable so try read again
continue;
}
if (len == 0) {
// handle graceful disconnect
break;
}
//log the number of data it received
process_tcp_data(buffer, len);
}
}
Do something similar on the sending side as well. Call send() first, and then call select() waiting for writability only if send() tells you to do so.