Best & Fast way to find out if an ip address is reachable - c++

I need the fastest way to see if an ip address is reachable. On each ip address theres a server listening on a specific port so let me know if your method is about to find if a server is listening on a port.
The reason for this is that suppose I have 10 ip addresses with 10 server listening on port 101 on each ip address. I want my client be able to find a Reachable ip address and connect to it as fast as he can(I don't want him to wait 30 seconds to find out if a ip address is reachable and then try the next ip address in the list)
May be it has to be done in treads simultaneously.

While you can quickly determine that an IP is reachable, your problem is determining that an IP is not reachable. The reason why is that you can't always definitively determine that an IP is not reachable. While there are some conditions where you will be given an affirmative notice that the IP is not reachable, usually your code will just not hear an answer and after waiting for some amount of time, your code will assume the IP is not reachable.
The problem in deciding the timeout is network topology. If you have a large topology (such as the Internet), you will need a large timeout to deal with potentially high latencies if you try to connect to an IP that is 'far' away.
From your description, the best idea would be to try to connect to all servers at the same time and use the first one that accepts the connection. You can use threads or you can use non-blocking sockets. In a non-blocking connect, the connect call returns immediately and you then use select to efficiently determine when the connect call has completed (either successfully or with an error).

You could use threads, but it would introduce unnecessary overhead for this task.
Use non-blocking sockets here (and avoid non-blocking sockets wherever you can! really, but they make sense in this case):
// initiate tcp connects...
for( each of your target host+port pairs ) {
int socket = socket( AF_INET, SOCK_STREAM );
...
#ifdef WIN32
unsigned long mode = 1;
ioctlsocket( socket, FIONBIO, &mode );
#else
int value = fcntl( socket, F_GETFL, 0 );
fcntl( socket, F_SETFL, value | O_NONBLOCK );
#endif
...
int code = connect( s, target, ... );
if( code == 0 ) { /* ok, this one did connect, DONE */ }
// now remember the socket in a list ...
}
// now find the first socket that was successfully connected
while( still not found ) {
struct timeval tval;
memset( &tval, 0, sizeof(tval) );
fd_set write_set, error_set;
int largest_socket = 0;
// add sockets to write and error set, and determine largest socket no.
for( each socket that did not fail until now ) {
FD_SET( socket, write_set );
FD_SET( socket, error_set );
if( socket > largest_socket ) largest_socket = socket;
}
// now use select to wait until something happens on the sockets
int code = select( largest_socket+1, NULL, &write_set, &error_set, &tval );
if( code < 0 ) { something went terribly wrong }
else {
for( each socket that did not fail until now ) {
if( FD_ISSET( socket, write_set ) ) { you've found it! DONE }
if( FD_ISSET( socket, error_set ) ) { error, remove this socket from list (for next iteration) }
}
}
}
Check documentation for connect and select for more details!

Typically randomly trying to connect with a short time out is sufficient.
Reachability is not very important, the fact that a route exists from you to the server isn't what matters, whether you can connected to said server is. Typically your own code will run just as fast as any other reachability method you can devise.
If you are having problems with it taking too long then try adjusting the length of your response, or having tighter timeouts.
Simple algorithm:
shuffle IP addresses
foreach IP in addresses
attempt connect
if succeed then
break

Try to open a socket using the connect() function from a BSD socket library. Its as fast as you can get, if the port is not open it wont respond to the SYN packet.
The key issue, as you realize, is tying up a thread which has to wait for a SYN-ACK before it can do anything else. Luckily, you do not need threads to parallelise IO anymore; however programming asynchronous operations can be subtle; therefore, I would recommend the libevent library for dispatching TCP/IP connect operations in parallel... since the kernel is doing the heavy lifting you only need one thread to do it on. You could probably do 100's or thousands of connects a second using libevent -- depending on your network hardware.
Another alternative is Boost::ASIO which is more complicated. But since you are using C++ might suite you better.

Below is code that you can use to create outgoing connections concurrently.
Iterate over your IPs and SpawnOutgoing connections in your loop.
Each connection conn_t* is posted as LParam in a window message - concurrently.
You should monitor the messages and save somewhere only the first connection - ignore (delete) other connections.
#define MSG_NEW_CONNECTION (WM_USER + 1)
struct conn_t {
SOCKET s;
sockaddr_in server;
};
static
UINT OutgoingWorker(LPVOID param)
{
// `param` holds "conn_t*"
assert(param);
if (!param) return 0;
conn_t* c = (conn_t*)param;
if (SOCKET_ERROR == connect(c->s, (SOCKADDR*)&c->server, sizeof c->server)) {
closesocket(c->s);
return 0;
}
PostMessage(mainwnd, MSG_NEW_CONNECTION, 0, (LPARAM)c); // <-- mainwnd handle
return 0;
}
conn_t*
SpawnOutgoing(const char* dest_ip, const USHORT dest_port)
{
if (!dest_ip) return NULL;
SOCKET s = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if (INVALID_SOCKET == s) {
return NULL;
}
conn_t* c = new conn_t;
// Create the socket here but connect it in the worker
memset(&c->server, 0, sizeof sockaddr_in);
c->server.sin_family = AF_INET;
c->server.sin_addr.s_addr = inet_addr(dest_ip);
c->server.sin_port = htons(dest_port);
c->s = s;
CreateThread(0, 0, OutgoingWorker, c);
return c;
}

Related

Switching between TCP and UDP simultaneously

I am working on robot control project where I should primarily (1)send some data about speed and fixed positions to the robot then (2)start a realtime control using my PC and repeat.
I chose to use TCP/IP for the first step and for realtime UDP/IP.
I tried to code this, but I believe something is missing.
I have some problems:
1- When there is no incoming (clients) connections accept function didn't return valid socket nor Message[10] is printed!!
2- Is it ok to Sendto using the same server IP and port?
3- Is this the right way to switch between UDP and TCP back and forth or there are some considerations should be undertaken?
while(Comm_Loop)
{
while (TCP_Loop)
{
TCP_Client = accept(TCP_Server, (struct sockaddr*) &Client_Address, &Client_Address_Length) ;
if ( TCP_Client != INVALID_SOCKET)
{
if (Data_Trans)
{
State = send(TCP_Client, Msg, sizeof(Msg), 0) ;
if (State != SOCKET_ERROR)
TCP_Loop = 0 ;
else
cout << Message[8] ;
}
}
else cout << Message[10] ;
}
Check_Send = sendto(UDP_Socket,
UDP_Send_Packet,
sizeof(UDP_Send_Packet),
0,
(LPSOCKADDR) &Socket_Address,
sizeof(Socket_Address));
fd_set SockSet ;
timeval Time_Out ;
FD_ZERO(&SockSet) ;
FD_SET(UDP_Socket, &SockSet);
Time_Out.tv_sec = 5;
Time_Out.tv_usec = 0 ;
State = select(0, &SockSet, (fd_set *)NULL, (fd_set *)NULL, &Time_Out) ;
if((State > 0) && (FD_ISSET(UDP_Socket, &SockSet) != 0))
{
Check_Recieve = recvfrom(UDP_Socket,
UDP_Recieve_Packet,
Buffer_Length,
0,
NULL,
NULL);
TCP_Loop = 1 ;
}
else
{
// Reception timeout
Comm_Loop = 0;
}
}
closesocket(TCP_Client) ;
closesocket(TCP_Server) ;
closesocket(UDP_Socket) ;
WSACleanup() ;
return 0;
}
1- When there is no incoming (clients) connections accept function
didn't return valid socket nor Message[10] is printed!!
Normal behavior by accept() is to block (i.e. not return) until either an incoming TCP connection is received, or an error occurs. If you don't want accept() to block, you can set the TCP_Server socket to be non-blocking, so that instead of blocking it returns -1 and sets errno to EWOULDBLOCK instead.
As far as Message[10] not printing, keep in mind that the stdout/cout stream is buffered, which means any text you send to it won't actually show up in the Terminal window until either you've printed a newline character ('\n' or std::endl) or manually flushed the stream. So if you aren't seeing the output you expected in a timely manner, you might want to verify that your Message[10] string contains a newline character, or alternatively append << std::endl; to the end of your cout << blah calls.
2- Is it ok to Sendto using the same server IP and port?
That question is ambiguous -- same server IP and port as what?
If you are asking if you can send your UDP sockets to the same IP address and port that were set in the accept() call (i.e. Client_Address in your code), you can do that, but it will only work if your robot has set up a UDP socket to bind to that port number in the UDP port-space. (Note that a UDP socket and a TCP socket can both bind to the same port without interfering with each other since they have different port-spaces, but by the same token you won't ever see TCP-data appear on your UDP socket or vice versa)
3- Is this the right way to switch between UDP and TCP back and forth
or there are some considerations should be undertaken?
It looks like it could more or less work, although a more robust design would be one that uses non-blocking sockets and always passes all of the sockets (i.e. the original TCP socket that you pass in to accept(), the per-TCP-connection TCP socket(s) that were returned by accept(), and your UDP socket) into every select call's read-fd_set, and then always checks all of those sockets (via FD_ISSET()) after select() returns and reacts appropriately when any of them is ready-for-read. That way your program could always handle incoming TCP data while simultaneously handling incoming UDP data, rather than switching from a mode where it handles only TCP to a mode where it handles only UDP (and ignoring any data that doesn't correspond to its current mode; because usually ignoring incoming data makes for a bad experience for the client/robot whose requests you are ignoring). This would also allow you to accept() incoming TCP connections at any time (since an incoming TCP connection would cause your TCP_Server socket to select as ready-for-read, at which point you could then call accept() on it again).

Unable to accept a new connection properly using IOCP - Invalid socket handle

I am learning about IOCP and have decided to write my own wrapper class based on the following article:
http://www.codeproject.com/Articles/13382/A-simple-application-using-I-O-Completion-Ports-an
My project is a C++ TCP server using IOCP. The client uses send() and recv() to send and receive data which I cannot change (from what I've been told this shouldn't cause any problem, but I am mentioning it just in case). It also creates a socket using socket() (and not WSASocket()).
Everything seems to be working fine (no error with CreateIoCompletionPort, I can add a socket descriptor to the existing completion port without any error. I've checked everything by adding a call to WSAGetLastError() after each of these functions).
(Before anything, please don't mind the inconsistent coding style. I like to make stuff work first and then clean it all up.)
socket_ = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_IP, NULL, 0, WSA_FLAG_OVERLAPPED);
setsockopt(socket_, IPPROTO_IP, SO_DEBUG | TCP_NODELAY, sockopt, 4);
ioctlsocket(socket_, FIONBIO, &ulSockMode_);
sin_.sin_family = AF_INET;
sin_.sin_port = htons((uint16_t)uiPort_));
hAccept_[0] = WSACreateEvent(); //only 1 event, I'm using an array for convenience
if (hAccept_ == WSA_INVALID_EVENT)
{
//this is never executed
}
WSAEventSelect(socket_, hAccept_[0], FD_ACCEPT);
After an incoming connection is detected (I use WSAWaitForMultipleEevents and WSAEnumNetworkEvents which work don't trigger any error), I use the following code to accept the client (and this is where the problems start):
SOCKET sock_client{ INVALID_SOCKET };
int32_t len_si{ sizeof(SOCKADDR_IN) };
//sock_client = accept(socket_, reinterpret_cast<SOCKADDR*>(pSockAddr), &len_si); // this works fine
//sock_client = sock_client = WSAAccept(socket_, reinterpret_cast<SOCKADDR*>(pSockAddr), &len_si, NULL, 0);//works fine too
char buf[2 * (sizeof(SOCKADDR_IN) + 16)];
WSAOVERLAPPED wsaovl;
uint32_t bytes{ 0 };
BOOL b = AcceptEx(socket_, sock_client, (PVOID)buf, 0, sizeof(SOCKADDR_IN) + 16, sizeof(SOCKADDR_IN) + 16, reinterpret_cast<LPDWORD>(&bytes), &wsaovl); //this fails, returns 0
int32_t test = WSAGetLastError(); // this returns 6 (WSA_INVALID_HANDLE)
I have no idea why it works with accept() and WSAAccept(), however it doesn't with AcceptEx().
If I use accept() though, after accepting the client I need to call WSARecv() immediately. I'm not sending anything back to the client just yet but I read that it needs to be called before GetQueuedCompletionStatus() in the worker thread:
WSABUF* buf = new WSABUF;
OVERLAPPED* ovl = new OVERLAPPED;
int32_t flags{ 0 };
int32_t bytes{ 0 };
int32_t bytes_recv = WSARecv(client_socket, buf, 1, &flags, &bytes, ovl, NULL); // this returns -1
int32_t err = WSAGetLastError(); // this returns 6 (WSA_INVALID_HANDLE)
And since this doesn't work, the GetQueuedCompletionStatus() routine in my worker thread keeps on hanging (or at least, I assume this is the reason)
Is there anything wrong with what I'm doing? I've been trying to search around and fix it since yesterday night, I know it's not a lot of time but I really don't see what I'm not doing correctly.
UPDATE:
I have changed the way I initialize my socket for AcceptEx().
SOCKET sock_client = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_IP, NULL, 0, WSA_FLAG_OVERLAPPED);
and
WSAOVERLAPPED wsaovl = {};
AcceptEx() still returns false, however the error returned by WSAGetLastError() is now 997 (WSA_IO_PENDING). I'm not sure what I/O operation is pending exactly and how I would go about fixing it.
I had stumbled upon similar hurdle when I was learning I/O Completion Ports (IOCP)...
I think the problem is that, in the scheme of IOCP socket model, the most complicated part is the beginning phase of 'socket acceptance'. That is why most tutorial skip over it and begin the discussion on how to handle send/recv, instead.
If you want to develop sufficient understanding of IOCP so that you could implement a production software then my advice to you is to study it until you completely grasp it (this answer below is not enough). One document that I would recommend is chapter 5 of 'Network Programming for Microsoft Windows - 2nd edition'. The book may be old but valid for IOCP. Also, the article 'Windows via C/C++: Synchronous and Asynchronous Device I/O' touches some aspects of IOCP, though not enough information to do production software.
I will try to explain as best as I can, however, I must warn you that this may not be enough. Here it goes...
So, the part you are missing is "How to do 'socket acceptance' in an IOCP socket model".
First off, lets examine the typical Winsock (non-IOCP) sequence of calls on server;
// (1) Create listen socket on server.
WSASocket()
// (2) Bind an address to your listen socket.
bind()
// (3) Associate the listen socket with an event object on FD_ACCEPT event.
WSAEventSelect(,, FD_ACCEPT )
// (4) Put socket in listen state - now, Windows listening for new
// connection requests. If new requests comes, the associated
// event object will be set.
listen()
// (5) Wait on the event object associated on listen socket. This
// will get signaled when a new connection request comes.
WaitForSingleObject() {
// (6) A network activity has occurred. Verify that FD_ACCEPT has
// raised the event object. This also resets the event object
// so WaitForSingleObject() does not loop non-stop.
WSAEnumNetworkEvents()
// (7) Understanding this part is important. The WSAAccept() doesn't
// just accept connection, it first creates a new socket and
// then associates it with the newly accepted connection.
WSAAccept()
}
The step (7) is ok for non-IOCP based models. However, when looking it from the performance point of view - socket creation is expensive. And it slows down the connection acceptance process.
In IOCP model, sockets are created in advance for new incoming connection requests. Not only sockets are created in advance they are associated with the listen socket even before the connection request comes. To achieve this Microsoft has provided extension functions. Two such functions that are required for IOCP model are AcceptEx() & GetAcceptExSockaddrs().
Note: When using these extension functions they need to be loaded at runtime in order to avoid performance penalty. This can be achieved using WSAIoctl(). For further read refer to the MSDN documentation on AcceptEx().
Caveat: AcceptEx() can be used to set the new socket to receive some data as part of connection-acceptance process. This feature needs to be disabled as it makes application susceptible to DoS attack i.e., a connection request is issued but no data is sent. The receiving application will wait on that socket indefinitely. To avoid that just pass 0 value for its 'dwReceiveDataLength' parameter.
How to setup connection-acceptance code for IOCP model?
One way to do this is;
// (1) Create IO completion port
CreateIoCompletionPort()
// (2) Have a method that creates worker threads say 'CreateWorkerThreads()'.
// This assign same method (say WorkerThread_Func()) to all worker threads.
// In the WorkerThread_Func() threads are blocked on call to
// GetQueuedCompletionStatus().
CreateWorkerThreads()
// (3) Create listen socket.
WSASocket()
// (4) Associate listen socket to IO Completion Port created earlier.
CreateIoCompletionPort()
// (5) Bind an address to your listen socket.
bind()
// (6) Put socket in listen state - now, Windows listening for new
// connection requests. If a new request comes, GetQueuedCompletionStatus()
// will release a thread.
listen()
// (7) Create sockets in advance and call AcceptEx on each of
// these sockets. If a new connection requests comes
// Windows will pick one of these sockets and associate the
// connection with it.
//
// As an example, below loop will create 1000 sockets.
GUID GuidAcceptEx = WSAID_ACCEPTEX;
DWORD dwBytes;
LPFN_ACCEPTEX lpfnAcceptEx;
// First, load extension method.
int retCode = WSAIoctl(listenSocket,
SIO_GET_EXTENSION_FUNCTION_POINTER,
&GuidAcceptEx,
sizeof(GuidAcceptEx),
&lpfnAcceptEx,
sizeof(lpfnAcceptEx),
&dwBytes,
NULL,
NULL
);
for( /* loop for 1000 times */ ) {
SOCKET preemptiveSocket = WSASocket(, , , , , WSA_FLAG_OVERLAPPED);
lpfnAcceptEx(listenSocket, preemptiveSocket,,,,,,);
}
This essentially prepare your application to accept sockets in IOCP way. When a new connection requests comes one of the worker threads, that are waiting on GetQueuedCompletionStatus(), will be released and handed over the pointer to the data structure. This will have the socket that was created by lpfnAcceptEx().
Is the process complete? Not yet. The socket accepted through AcceptEx() call does not inherit properties of listenSocket. To do that you need to call;
setsockopt( acceptSocket, SOL_SOCKET, SO_UPDATE_ACCEPT_CONTEXT,
(char*)&listenSocket, sizeof(listenSocket) );
Now, acceptSocket is good to use for WSASend / WSARecv!
Something is missing! I skipped the part about how does a worker thread gets the acceptedSocket from GetQueuedCompletionStatus() ?
The answer is, by passing your specially crafted structure to lpfnAcceptEx(). When the GetQueuedCompletionStatus() returns it will have this data structure containing the socket that you will have passed.
How to make such a structure? By creating a structure having 'WSAOVERLAPPED' as its first member. You can have any members of your own after the first member. For example, my structure looked like;
typedef struct _WSAOVERLAPPEDPLUS
{
WSAOVERLAPPED ProviderOverlapped; // 'WSAOVERLAPPED' has to be the first member.
SOCKET client; // Use this to pass preemptive socket.
SOCKET listenSocket; // Use this to pass the listenSocket.
DWORD dwBytes;
SOCKET_OPERATION operation; // Enum to assist in knowing what socket operation ...
} WSAOVERLAPPEDPLUS, *LPWSAOVERLAPPEDPLUS;
...
typedef enum SOCKET_OPERATION {
UNINITIALIZED_ENUM, // To protect against memory leaks and uninitialized buffers.
OP_ACCEPTEX,
OP_RECEIVE,
OP_SEND
};
...
//
// So the previously mentioned for() loop will become;
//
for( /* loop for 1000 times */ ) {
SOCKET preemptiveSocket = WSASocket(, , , , , WSA_FLAG_OVERLAPPED);
LPWSAOVERLAPPEDPLUS pOl = new WSAOVERLAPPEDPLUS();
// Initialize our "extended" overlapped structure
memset(pOl, 0, sizeof(WSAOVERLAPPEDPLUS));
pOl->operation = OP_ACCEPTEX;
pOl->client = preemptiveSocket;
pOl->listenSocket = listenSocket;
int buflen = (sizeof(SOCKADDR_IN) + 16) * 2;
char* pBuf = new char[buflen];
memset(pBuf, 0, buflen);
m_lpfnAcceptEx(listenSocket,
preemptiveSocket,
pBuf,
0, // Passed 0 to avoid reading data on accept which in turn
// avoids DDoS attack i.e., connection attempt without data will
// cause AcceptEx to wait indefinitely.
sizeof(SOCKADDR_IN) + 16,
sizeof(SOCKADDR_IN) + 16,
&pOl->dwBytes,
&pOl->ProviderOverlapped
);
}
... and in the worker thread when GetQueuedCompletionStatus() returns;
while (TRUE)
{
bOk = ::GetQueuedCompletionStatus(hCompPort, &bytes_transferred, &completion_key, &pOverlapped, INFINITE);
if (bOk) {
// Process a successfully completed I/O request
if (completion_key == 0) {
// Safe way to extract the customized structure from pointer
// is to use 'CONTAINING_RECORD'. Read more on 'CONTAINING_RECORD'.
WSAOVERLAPPEDPLUS *pOl = CONTAINING_RECORD(pOverlapped, WSAOVERLAPPEDPLUS, ProviderOverlapped);
if (pOl->operation == OP_ACCEPTEX) {
// Before doing any WSASend/WSARecv, inherit the
// listen socket properties by calling 'setsockopt()'
// as explained earlier.
// The listenSocket and the preemptive socket are available
// in the 'pOl->listenSocket' & 'pOl->client', respectively.
}
delete pOl;
}
}
else {
// Handle error ...
}
I hope this gave you idea on how AcceptEx() is utilized with IOCP.

Create multiple listening sockets

I'm using winsocks and I am coding an IDS/Honeypot, this is just a small section of it, because at the moment I want the server to listen on multiple sockets (7) and accept the connections, but I've tried to dynamically create the sockets with an array (and the listener etc) but I am still having trouble - I've tried it multiple ways but so far, all I've managed to do is get it working successfully on ONE socket, and LISTEN to all sockets, but not accept them.
So, this was my last attempt but not sure, maybe I need to use threads or declare the sockets differently?
So far, in this small test code, I want:
Initialize server
listen on all 7 ports (1111,2222 ...etc)
Accept an incoming connection on ANY of them
display both messages on client/server
drop the connection
and continue
It's a little sloppy I know, but here is the code so far and I think you can see where I am going with it:
#include <iostream>
#include <winsock2.h>
#include <string>
#pragma comment(lib, "ws2_32.lib")
int main()
{
std::cout<<"Honeypot server [test #1] by Dreamwalker"<<std::endl;
WSADATA wsa;
SOCKET s[7] , new_socket[7];
struct sockaddr_in server , client;
int c, port[7] = {1111,2222,3333,4444,5555,6666,7777};
char *message;
std::cout<<"\nInitialising Winsock and other components...";
if (WSAStartup(MAKEWORD(2,2),&wsa) != 0)
{
std::cout<<"Failed. Error Code :"<<WSAGetLastError()<<std::endl;
return 1;
}
//!IMPORTANT: create multiple new sockets on different ports
int i = 0;
for( i = 0; i < 7; i++)
{
//Create socket
if((s[i] = socket(AF_INET , SOCK_STREAM , 0 )) == INVALID_SOCKET)
{
std::cout<<"Could not create socket : "<< WSAGetLastError()<<std::endl;
}
//Prepare the sockaddr_in structure
server.sin_family = AF_INET;
server.sin_addr.s_addr = INADDR_ANY;
server.sin_port = htons( port[i] );
//Bind
if( bind(s[i] ,(struct sockaddr *)&server , sizeof(server)) == SOCKET_ERROR)
{
std::cout<<"Bind failed with error code : "<< WSAGetLastError()<<std::endl;
}
/*!ALL CREATION CHECKING DONE, now create multiple sockets on the server
and listen for connections*/
c = sizeof(struct sockaddr_in);
listen(s[i] , SOMAXCONN);
}
///ALL INITIALIZED
std::cout<<"DONE!"<<std::endl;
//Listen/accept incoming connections
std::cout<<"Now listening for connections"<<std::endl;
new_socket[i] = accept(s[i] , (struct sockaddr *)&client, &c);
if (new_socket[i] == INVALID_SOCKET)
{
std::cout<<"accept failed with error code : "<< WSAGetLastError()<<std::endl;
}
//Accepted connection
else{
std::cout<<"Someone has connected to this machine!"<<std::endl;
message = "Hello Client , I have received your connection.\n";
send(new_socket[i] , message , strlen(message) , 0);
closesocket(s[i]);
}
std::cout<<"FINISHED"<<std::endl;
WSACleanup();
getchar();
return 0;
}
And now it's throwing a runtime error as well:
WSAENOTSOCK
10038
Socket operation on nonsocket.
An operation was attempted on something that is not a socket. Either the socket handle parameter did not reference a valid socket,
or for select, a member of an fd_set was not valid.
Which (including debugging) indicates that the socket isn't declared properly when creating on an array, advice?
You code to create/bind/listen is all good. Then:
new_socket[i] = accept(s[i] , (struct sockaddr *)&client, &c);
Firstly, by the time this runs you're outside the loop, and i is 7 which is past the end of the array of sockets, which is why you get the not-a-socket error.
Secondly, accept() is a blocking call, so you can't just call accept() on all the sockets from the same thread the way you did for listen. You need to either have a separate thread block in accept() for each of the ports, or find out which one has a client connection attempt in progress using e.g. select (or epoll - does Windows have that?), then accept() a client on that specific socket (but then you've still got to either create a thread to handle the client read/recvs and write/sends or use select/epoll to find out when there's input ready to read, or more space in output buffers for transmission). There's also a race condition to be wary of if you use select/epoll - a listening socket might signal readiness for accepting a client connection, but by the time you call accept() that connection attempt's failed and forgotten, then if the listening socket hasn't been set to non-blocking mode it'll hang there waiting for another client to connect to that specific socket. IMHO, this is a case where threading is actually easier.
I think it's more "Windowsy" to use IO Completion Ports (you might want to Google), but AFAIK they're totally unportable. Winsock's not an exact match for BSD sockets, but the porting or dual-support effort's small.

standard C++ TCP socket, connect fails with EINTR when using std::async

I am having trouble using the std::async to have tasks execute in parallel when the task involves a socket.
My program is a simple TCP socket server written in standard C++ for Linux. When a client connects, a dedicated port is opened and separate thread is started, so each client is serviced in their own thread.
The client objects are contained in a map.
I have a function to broadcast a message to all clients. I originally wrote it like below:
// ConnectedClient is an object representing a single client
// ConnectedClient::SendMessageToClient opens a socket, connects, writes, reads response and then closes socket
// broadcastMessage is the std::string to go out to all clients
// iterate through the map of clients
map<string, ConnectedClient*>::iterator nextClient;
for ( nextClient = mConnectedClients.begin(); nextClient != mConnectedClients.end(); ++nextClient )
{
printf("%s\n", nextClient->second->SendMessageToClient(broadcastMessage).c_str());
}
I have tested this and it works with 3 clients at a time. The message gets to all three clients (one at a time), and the response string is printed out three times in this loop. However, it is slow, because the message only goes out to one client at a time.
In order to make it more efficient, I was hoping to take advantage of std::async to call the SendMessageToClient function for every client asynchronously. I rewrote the code above like this:
vector<future<string>> futures;
// iterate through the map of clients
map<string, ConnectedClient*>::iterator nextClient;
for ( nextClient = mConnectedClients.begin(); nextClient != mConnectedClients.end(); ++nextClient )
{
printf("start send\n");
futures.push_back(async(launch::async, &ConnectedClient::SendMessageToClient, nextClient->second, broadcastMessage, wait));
printf("end send\n");
}
vector<future<string>>::iterator nextFuture;
for( nextFuture = futures.begin(); nextFuture != futures.end(); ++nextFuture )
{
printf("start wait\n");
nextFuture->wait();
printf("end wait\n");
printf("%s\n", nextFuture->get().c_str());
}
The code above functions as expected when there is only one client in the map. That you see "start send" quickly followed by "end send", quickly followed by "start wait" and then 3 seconds later (I have a three second sleep on the client response side to test this) you see the trace from the socket read function that the response comes in, and then you see "end wait"
The problem is that when there is more than one client in the map. In the part of the SendMessageToClient function that opens and connects to the socket, it fails in the code identified below:
// connected client object has a pipe open back to the client for sending messages
int clientSocketFileDescriptor;
clientSocketFileDescriptor = socket(AF_INET, SOCK_STREAM, 0);
// set the socket timeouts
// this part using setsockopt is omitted for brevity
// host name
struct hostent *server;
server = gethostbyname(mIpAddressOfClient.c_str());
if (server == 0)
{
close(clientSocketFileDescriptor);
return "";
}
//
struct sockaddr_in clientsListeningServerAddress;
memset(&clientsListeningServerAddress, 0, sizeof(struct sockaddr_in));
clientsListeningServerAddress.sin_family = AF_INET;
bcopy((char*)server->h_addr, (char*)&clientsListeningServerAddress.sin_addr.s_addr, server->h_length);
clientsListeningServerAddress.sin_port = htons(mPortNumberClientIsListeningOn);
// The connect function fails !!!
if ( connect(clientSocketFileDescriptor, (struct sockaddr *)&clientsListeningServerAddress, sizeof(clientsListeningServerAddress)) < 0 )
{
// print out error code
printf("Connected client thread: fail to connect %d \n", errno);
close(clientSocketFileDescriptor);
return response;
}
The output reads: "Connected client thread: fail to connect 4".
I looked this error code up, it is explained thus:
#define EINTR 4 /* Interrupted system call */
I searched around on the internet, all I found were some references to system calls being interrupted by signals.
Does anyone know why this works when I call my send message function one at a time, but it fails when the send message function is called using async? Does anyone have a different suggestion how I should send a message to multiple clients?
First, I would try to deal with the EINTR issue. connect ( ) has been interrupted (this is the meaning of EINTR) and does not try again because you are using and asynch descriptor.
What I usually do in such a circumstance is to retry: I wrap the function (connect in this case) in a while cycle. If connect succeeds I break out of the cycle. If it fails, I check the value of errno. If it is EINTR I try again.
Mind that there are other values of errno that deserve a retry (EWOULDBLOCK is one of them)

How to do TCP connection pooling in C/C++

I'm designing a distributed server/client system with C++, in which many clients send request to many servers through TCP and server throw a thread to handle the request and send back it's response. In my use case only limited number of clients will access the server and I need very high performance.The data sent from client and server are all small, but are very frequent. So creating a connection and tearing it down it after use is expensive. So I want to use connection caching to solve this problem: once connection created, it will be stored in a cache for future use.(Assume that the number of clients will not beyond the size of cache).
My question is:
I saw someone said that connection pooling is a client side technique. If this connection pooling is only used in client side, then first time it make connection to a server, and send data. This action of making connection triggers the accept() function in server side which return a socket for receiving from client. So when client wants to use a existing connection(in cache), it doesn't make new connection, but just send data. The problem is, if no making connection, who would trigger accept() in server side and to throw a thread?
If connection pooling also need to be implemented in server side, how can I know where a request come from? Since only from accept() I can get the client address, but meanwhile accept() already make a new socket for that request, so no point to use a cached connection.
Any answer and suggestion will be appreciated. Or any one can give me an example of connection pool or connection caching?
I saw someone said that connection pooling is a client side technique. ... if no making connection, who would trigger accept() in server side and to throw a thread?
Firstly, connection pooling is not just a client-side technique; it's a connection-mode technique. It applies to both types of peer (the "server" and the "client").
Secondly, accept doesn't need to be called to start a thread. Programs can start threads for any reason they like... They could start threads just to start more threads, in a massively parallelised loop of thread creation. (edit: we call this a "fork bomb")
Finally, an efficient thread-pooling implementation won't start a thread for each client. Each thread typically occupies between 512KB-4MB (counting stack space and other context information), so if you have 10000 clients each occupying that much, that's a lot of wasted memory.
I want to do so, but just don't know how to do it in multithreading case.
You shouldn't use multithreading here... At least, not until you have a solution that uses a single thread, and you decide that it's not fast enough. At the moment you don't have that information; you're just guessing, and guessing doesn't guarantee optimisation.
At the turn of the century there were FTP servers that solved the C10K problem; they were able to handle 10000 clients at any given time, browsing, downloading or idling as users tend to do on FTP servers. They solved that problem not by using threads, but by using non-blocking and/or asynchronous sockets and/or calls.
To clarify, those web servers handled thousands of connections on a single thread! One typical way is to use select, but I'm not particularly fond of that method because it requires a rather ugly series of loops. I prefer to use ioctlsocket for Windows and fcntl for other POSIX OSes to set the file descriptor into non-blocking mode, e.g.:
#ifdef WIN32
ioctlsocket(fd, FIONBIO, (u_long[]){1});
#else
fcntl(fd, F_SETFL, fcntl(fd, F_GETFL, 0) | O_NONBLOCK);
#endif
At this point, recv and read won't block when operating on fd; if there's no data available, they'll return an error value immediately rather than waiting for data to arrive. That means you can loop on multiple sockets.
If connection pooling also need to be implemented in server side, how can I know where a request come from?
Store the client fd along-side its struct sockaddr_storage and any other stateful information you need to store about clients, in a struct that you declare however you feel. If this ends up being 4KB (which is a fairly large struct, usually about as large as they need to get) then 10000 of these will only occupy about 40000KB (~40MB). Even the mobile phones of today should have no problems handling that. Consider completing the following code for your needs:
struct client {
struct sockaddr_storage addr;
socklen_t addr_len;
int fd;
/* Other stateful information */
};
#define BUFFER_SIZE 4096
#define CLIENT_COUNT 10000
int main(void) {
int server;
struct client client[CLIENT_COUNT] = { 0 };
size_t client_count = 0;
/* XXX: Perform usual bind/listen */
#ifdef WIN32
ioctlsocket(server, FIONBIO, (u_long[]){1});
#else
fcntl(server, F_SETFL, fcntl(server, F_GETFL, 0) | O_NONBLOCK);
#endif
for (;;) {
/* Accept connection if possible */
if (client_count < sizeof client / sizeof *client) {
struct sockaddr_storage addr = { 0 };
socklen_t addr_len = sizeof addr;
int fd = accept(server, &addr, &addr_len);
if (fd != -1) {
# ifdef WIN32
ioctlsocket(fd, FIONBIO, (u_long[]){1});
# else
fcntl(fd, F_SETFL, fcntl(fd, F_GETFL, 0) | O_NONBLOCK);
# endif
client[client_count++] = (struct client) { .addr = addr
, .addr_len = addr_len
, .fd = fd };
}
}
/* Loop through clients */
char buffer[BUFFER_SIZE];
for (size_t index = 0; index < client_count; index++) {
ssize_t bytes_recvd = recv(client[index].fd, buffer, sizeof buffer, 0);
# ifdef WIN32
int closed = bytes_recvd == 0
|| (bytes_recvd < 0 && WSAGetLastError() == WSAEWOULDBLOCK);
# else
int closed = bytes_recvd == 0
|| (bytes_recvd < 0 && errno == EAGAIN) || errno == EWOULDBLOCK;
# endif
if (closed) {
close(client[index].fd);
client_count--;
memmove(client + index, client + index + 1, (client_count - index) * sizeof client);
continue;
}
/* XXX: Process buffer[0..bytes_recvd-1] */
}
sleep(0); /* This is necessary to pass control back to the kernel,
* so it can queue more data for us to process
*/
}
}
Supposing you want to pool connections on the client-side, the code would look very similar, except obviously there would be no need for the accept-related code. Supposing you have an array of clients that you want to connect, you could use non-blocking connect calls to perform all of the connections at once like this:
size_t index = 0, in_progress = 0;
for (;;) {
if (client[index].fd == 0) {
client[index].fd = socket(/* TODO */);
# ifdef WIN32
ioctlsocket(client[index].fd, FIONBIO, (u_long[]){1});
# else
fcntl(client[index].fd, F_SETFL, fcntl(client[index].fd, F_GETFL, 0) | O_NONBLOCK);
# endif
}
# ifdef WIN32
in_progress += connect(client[index].fd, (struct sockaddr *) &client[index].addr, client[index].addr_len) < 0
&& (WSAGetLastError() == WSAEALREADY
|| WSAGetLastError() == WSAEWOULDBLOCK
|| WSAGetLastError() == WSAEINVAL);
# else
in_progress += connect(client[index].fd, (struct sockaddr *) &client[index].addr, client[index].addr_len) < 0
&& (errno == EALREADY
|| errno == EINPROGRESS);
# endif
if (++index < sizeof client / sizeof *client) {
continue;
}
index = 0;
if (in_progress == 0) {
break;
}
in_progress = 0;
}
As for optimisation, given that this should be able to handle 10000 clients with perhaps a few minor tweaks, you shouldn't need multiple threads.
Nonetheless, by associating items from a mutex collection with clients and preceding the non-blocking socket operation with a non-blocking pthread_mutex_trylock, the above loops could be adapted to run simultaneously in multiple threads whilst processing the same group of sockets. This provides a working model for all POSIX-compliant platforms, be it Windows, BSD or Linux, but it's not a perfectly optimal one. To achieve optimality, we must step into the asynchronous world, which varies from system to system:
Windows uses WSA* functions with call-backs.
BSD and Linux use the somewhat similar kqueue and epoll, respectively.
It may pay to codify that "non-blocking socket operation" abstraction mentioned earlier, as the two asynchronous mechanisms vary significantly in respect to their interface. Like everything else, unfortunately we must write abstractions so that our Windows-relevant code remains legible on POSIX-compliant systems. As a bonus, this'll allow us to mingle server-processing (i.e. accept and anything that follows) with client-processing (i.e. connect and anything that follows), so our server loop can become a client loop (or vice-versa).