I am having trouble using the std::async to have tasks execute in parallel when the task involves a socket.
My program is a simple TCP socket server written in standard C++ for Linux. When a client connects, a dedicated port is opened and separate thread is started, so each client is serviced in their own thread.
The client objects are contained in a map.
I have a function to broadcast a message to all clients. I originally wrote it like below:
// ConnectedClient is an object representing a single client
// ConnectedClient::SendMessageToClient opens a socket, connects, writes, reads response and then closes socket
// broadcastMessage is the std::string to go out to all clients
// iterate through the map of clients
map<string, ConnectedClient*>::iterator nextClient;
for ( nextClient = mConnectedClients.begin(); nextClient != mConnectedClients.end(); ++nextClient )
{
printf("%s\n", nextClient->second->SendMessageToClient(broadcastMessage).c_str());
}
I have tested this and it works with 3 clients at a time. The message gets to all three clients (one at a time), and the response string is printed out three times in this loop. However, it is slow, because the message only goes out to one client at a time.
In order to make it more efficient, I was hoping to take advantage of std::async to call the SendMessageToClient function for every client asynchronously. I rewrote the code above like this:
vector<future<string>> futures;
// iterate through the map of clients
map<string, ConnectedClient*>::iterator nextClient;
for ( nextClient = mConnectedClients.begin(); nextClient != mConnectedClients.end(); ++nextClient )
{
printf("start send\n");
futures.push_back(async(launch::async, &ConnectedClient::SendMessageToClient, nextClient->second, broadcastMessage, wait));
printf("end send\n");
}
vector<future<string>>::iterator nextFuture;
for( nextFuture = futures.begin(); nextFuture != futures.end(); ++nextFuture )
{
printf("start wait\n");
nextFuture->wait();
printf("end wait\n");
printf("%s\n", nextFuture->get().c_str());
}
The code above functions as expected when there is only one client in the map. That you see "start send" quickly followed by "end send", quickly followed by "start wait" and then 3 seconds later (I have a three second sleep on the client response side to test this) you see the trace from the socket read function that the response comes in, and then you see "end wait"
The problem is that when there is more than one client in the map. In the part of the SendMessageToClient function that opens and connects to the socket, it fails in the code identified below:
// connected client object has a pipe open back to the client for sending messages
int clientSocketFileDescriptor;
clientSocketFileDescriptor = socket(AF_INET, SOCK_STREAM, 0);
// set the socket timeouts
// this part using setsockopt is omitted for brevity
// host name
struct hostent *server;
server = gethostbyname(mIpAddressOfClient.c_str());
if (server == 0)
{
close(clientSocketFileDescriptor);
return "";
}
//
struct sockaddr_in clientsListeningServerAddress;
memset(&clientsListeningServerAddress, 0, sizeof(struct sockaddr_in));
clientsListeningServerAddress.sin_family = AF_INET;
bcopy((char*)server->h_addr, (char*)&clientsListeningServerAddress.sin_addr.s_addr, server->h_length);
clientsListeningServerAddress.sin_port = htons(mPortNumberClientIsListeningOn);
// The connect function fails !!!
if ( connect(clientSocketFileDescriptor, (struct sockaddr *)&clientsListeningServerAddress, sizeof(clientsListeningServerAddress)) < 0 )
{
// print out error code
printf("Connected client thread: fail to connect %d \n", errno);
close(clientSocketFileDescriptor);
return response;
}
The output reads: "Connected client thread: fail to connect 4".
I looked this error code up, it is explained thus:
#define EINTR 4 /* Interrupted system call */
I searched around on the internet, all I found were some references to system calls being interrupted by signals.
Does anyone know why this works when I call my send message function one at a time, but it fails when the send message function is called using async? Does anyone have a different suggestion how I should send a message to multiple clients?
First, I would try to deal with the EINTR issue. connect ( ) has been interrupted (this is the meaning of EINTR) and does not try again because you are using and asynch descriptor.
What I usually do in such a circumstance is to retry: I wrap the function (connect in this case) in a while cycle. If connect succeeds I break out of the cycle. If it fails, I check the value of errno. If it is EINTR I try again.
Mind that there are other values of errno that deserve a retry (EWOULDBLOCK is one of them)
Related
I am relatively new to network programming and multithreading in C++. Currently my recv() call returns an unknown error. I'm not quite sure where the error coming from at the moment and would appreciate some help.
I used putty to connect to the server locally
class Threaded_TCPListener{
int Threaded_TCPListener::Init()
{
// Initializing WinSock
WSADATA wsData;
WORD ver = MAKEWORD(2,2);
int winSock = WSAStartup(ver, &wsData);
if(winSock != 0)
return winSock;
// Creating listening socket
this->socket = ::socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if(this->socket == INVALID_SOCKET)
return WSAGetLastError();
// Fill sockaddr with ip addr and port
sockaddr_in hint;
hint.sin_family = AF_INET;
hint.sin_port = htons(this->port);
inet_pton(AF_INET, this->ipAddress, &hint.sin_addr);
// Bind hint to socket
if(bind(this->socket, (sockaddr*)&hint, sizeof(hint)) == SOCKET_ERROR)
return WSAGetLastError();
// Start listening on socket
if(listen(this->socket, SOMAXCONN) == SOCKET_ERROR)
return WSAGetLastError();
// Accept first client
this->createAcceptThread();
return 0;
}
int Threaded_TCPListener::Run()
{
bool isRunning = true;
// Read from all clients
std::vector<std::thread> threads;
threads.reserve(this->clients.size());
// Recv from client sockets
for (int i=0; i < this->clients.size(); ++i)
{
threads.emplace_back(std::thread(&Threaded_TCPListener::receiveFromSocket, this, socket));
}
// Wait for all threads to finish
for(std::thread& t : threads)
{
t.detach();
}
return 0;
}
void Threaded_TCPListener::onMessageReceived(int clientSocket, const char* msg, int length)
{
Threaded_TCPListener::broadcastToClients(clientSocket, msg, length);
std::thread t(&Threaded_TCPListener::receiveFromSocket, this, clientSocket);
t.detach();
return;
}
void Threaded_TCPListener::sendMessageToClient(int clientSocket, const char * msg, int length)
{
send(clientSocket, msg, length, 0);
return;
}
void Threaded_TCPListener::broadcastToClients(int senderSocket, const char * msg, int length)
{
std::vector<std::thread> threads;
threads.reserve(clients.size());
// Iterate over all clients
for (int sendSock : this->clients)
{
if(sendSock != senderSocket)
threads.emplace_back(std::thread(&Threaded_TCPListener::sendMessageToClient, this,sendSock, msg, length));
}
// Wait for all threads to finish
for(std::thread& t : threads)
t.join();
return;
}
void Threaded_TCPListener::createAcceptThread()
{
// Start accepting clients on a new thread
this->listeningThread = std::thread(&Threaded_TCPListener::acceptClient, this);
this->listeningThread.detach();
return;
}
void Threaded_TCPListener::acceptClient()
{
int client = accept(this->socket, nullptr, nullptr);
// Error
if(client == INVALID_SOCKET)
{
std::printf("Accept Err: %d\n", WSAGetLastError());
}
// Add client to clients queue
else
{
// Add client to queue
this->clients.emplace(client);
// Client Connect Confirmation
onClientConnected(client); // Prints msg on server
// Create another thread to accept more clients
this->createAcceptThread();
}
return;
}
void Threaded_TCPListener::receiveFromSocket(int receivingSocket)
{
// Byte storage
char buff[MAX_BUFF_SIZE];
// Clear buff
memset(buff, 0, sizeof(buff));
// Receive msg
int bytesRecvd = recv(receivingSocket, buff, MAX_BUFF_SIZE, 0);
if(bytesRecvd <= 0)
{
char err_buff[1024];
strerror_s(err_buff, bytesRecvd);
std::cerr << err_buff;
// Close client
this->clients.erase(receivingSocket);
closesocket(receivingSocket);
onClientDisconnected(receivingSocket); // Prints msg on server
}
else
{
onMessageReceived(receivingSocket, buff, bytesRecvd);
}
}
}
I am trying to create a multithreaded TCP 'server' that will handle incoming clients by having an accept thread continuously running (listening for new connections), and a thread waiting with a recv block for each client connected to the server.
Your Init looks fine:
create socket, bind it, listen on it, start accept thread
In your accept thread's acceptClient looks sort of OK:
print some message
add the client socket to clients queue
create a new accept thread
Your Run makes no sense:
create one thread per element in clients to receive from the listening socket
It looks like you are spawning a new thread for every single socket action. That is a pretty wasteful design. As soon as the thread is done it can go back to doing something else.
So creating a new accept thread in acceptClient is a waste, you could just loop back to the beginning to ::accept the next client. Like so:
acceptClient() {
while (alive) {
int client = accept(socket, ...);
createClientHandler(client);
}
}
What seems to be missing is spawning a new client thread to service the client socket. You currently do this in Run, but that's before any of the clients are actually accepted. And you do it for the wrong socket! Instead, you should be spawning the receiveFromSocket threads in acceptClient, and passing it the client socket. So that's a bug.
In your receiveFromSocket you also need not create another thread to receiveFromSocket again -- just loop back to the beginning.
The biggest concern with this thread-per-action design is that you are spawning sender threads on every incoming message. This means you could actually have several sender threads attempting to ::send on the same TCP socket. That's not very safe.
The order of calls made to WSASend is also the order in which the buffers are transmitted to the transport layer. WSASend should not be called on the same stream-oriented socket concurrently from different threads, because some Winsock providers may split a large send request into multiple transmissions, and this may lead to unintended data interleaving from multiple concurrent send requests on the same stream-oriented socket.
https://learn.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsasend
Similarly, instead of spawning threads in broadcastToClients, I suggest you just spawn one persistent sender thread per client socket in acceptClient (together with the receiveFromSocket thread within some createClientHandler).
To communicate with the sender threads you should use thread-safe blocking queues. Each sender thread would look like this:
while (alive) {
msg = queue.next_message();
send(client_socket, msg);
}
Then on message received you just do:
for (client : clients) {
client.queue.put_message(msg);
}
So to summarize, to handle each client you need a structure like this:
struct Client {
int client_socket;
BlockingQueue queue;
// optionally if you want to keep track of your threads
// to be able to safely clean up
std::thread recv_thread, send_thread;
};
Safe cleanup is a whole other story.
Finally, a remark on this comment in your code:
// Wait for all threads to finish
for(std::thread& t : threads)
{
t.detach();
}
That's almost the opposite to what std::thread::detach does:
https://en.cppreference.com/w/cpp/thread/thread/detach
It allows you to destroy the thread object without having to wait for the thread to finish execution.
There is a misconception in the code in how a TCP server has to be implemented:
You seem to assume that you can have a single server socket file descriptor which can handle all communication. This is not the case. You must have a single dedicated socket file descriptor which is just used for listening and accepting incoming connections, and then you have one additional file descriptor for each existing connection.
In your code I see that you invoke receiveFromSocket() always with the listening socket. This is wrong. Also invoking receiveFromSocket() in a loop for all clients is wrong.
What you rather need to do is:
- Have one dedicated thread which call accept() in a loop. There is no performance benefit in calling accept() from multiple threads.
- One accept() returns a new connection you spawn a new thread which calls recv() in a loop. This will then block and wait for new data as you expect in your question.
You also need to drop the habit of calling individual functions from new threads. This is not multithreaded programming. A thread usually contains a loop. Everything else is usually a design flaw.
Also note that multithreaded programming is still rocket science in 2019, especially in C++. If you are not an absolute expert you will not be able to do it. Also note that absolute experts in multithreaded programming will try to avoid multithreaded programming whenever possible. A lot seemingly concurrent tasks which are I/O bound can better be handled by a single threaded event based system.
I am working on robot control project where I should primarily (1)send some data about speed and fixed positions to the robot then (2)start a realtime control using my PC and repeat.
I chose to use TCP/IP for the first step and for realtime UDP/IP.
I tried to code this, but I believe something is missing.
I have some problems:
1- When there is no incoming (clients) connections accept function didn't return valid socket nor Message[10] is printed!!
2- Is it ok to Sendto using the same server IP and port?
3- Is this the right way to switch between UDP and TCP back and forth or there are some considerations should be undertaken?
while(Comm_Loop)
{
while (TCP_Loop)
{
TCP_Client = accept(TCP_Server, (struct sockaddr*) &Client_Address, &Client_Address_Length) ;
if ( TCP_Client != INVALID_SOCKET)
{
if (Data_Trans)
{
State = send(TCP_Client, Msg, sizeof(Msg), 0) ;
if (State != SOCKET_ERROR)
TCP_Loop = 0 ;
else
cout << Message[8] ;
}
}
else cout << Message[10] ;
}
Check_Send = sendto(UDP_Socket,
UDP_Send_Packet,
sizeof(UDP_Send_Packet),
0,
(LPSOCKADDR) &Socket_Address,
sizeof(Socket_Address));
fd_set SockSet ;
timeval Time_Out ;
FD_ZERO(&SockSet) ;
FD_SET(UDP_Socket, &SockSet);
Time_Out.tv_sec = 5;
Time_Out.tv_usec = 0 ;
State = select(0, &SockSet, (fd_set *)NULL, (fd_set *)NULL, &Time_Out) ;
if((State > 0) && (FD_ISSET(UDP_Socket, &SockSet) != 0))
{
Check_Recieve = recvfrom(UDP_Socket,
UDP_Recieve_Packet,
Buffer_Length,
0,
NULL,
NULL);
TCP_Loop = 1 ;
}
else
{
// Reception timeout
Comm_Loop = 0;
}
}
closesocket(TCP_Client) ;
closesocket(TCP_Server) ;
closesocket(UDP_Socket) ;
WSACleanup() ;
return 0;
}
1- When there is no incoming (clients) connections accept function
didn't return valid socket nor Message[10] is printed!!
Normal behavior by accept() is to block (i.e. not return) until either an incoming TCP connection is received, or an error occurs. If you don't want accept() to block, you can set the TCP_Server socket to be non-blocking, so that instead of blocking it returns -1 and sets errno to EWOULDBLOCK instead.
As far as Message[10] not printing, keep in mind that the stdout/cout stream is buffered, which means any text you send to it won't actually show up in the Terminal window until either you've printed a newline character ('\n' or std::endl) or manually flushed the stream. So if you aren't seeing the output you expected in a timely manner, you might want to verify that your Message[10] string contains a newline character, or alternatively append << std::endl; to the end of your cout << blah calls.
2- Is it ok to Sendto using the same server IP and port?
That question is ambiguous -- same server IP and port as what?
If you are asking if you can send your UDP sockets to the same IP address and port that were set in the accept() call (i.e. Client_Address in your code), you can do that, but it will only work if your robot has set up a UDP socket to bind to that port number in the UDP port-space. (Note that a UDP socket and a TCP socket can both bind to the same port without interfering with each other since they have different port-spaces, but by the same token you won't ever see TCP-data appear on your UDP socket or vice versa)
3- Is this the right way to switch between UDP and TCP back and forth
or there are some considerations should be undertaken?
It looks like it could more or less work, although a more robust design would be one that uses non-blocking sockets and always passes all of the sockets (i.e. the original TCP socket that you pass in to accept(), the per-TCP-connection TCP socket(s) that were returned by accept(), and your UDP socket) into every select call's read-fd_set, and then always checks all of those sockets (via FD_ISSET()) after select() returns and reacts appropriately when any of them is ready-for-read. That way your program could always handle incoming TCP data while simultaneously handling incoming UDP data, rather than switching from a mode where it handles only TCP to a mode where it handles only UDP (and ignoring any data that doesn't correspond to its current mode; because usually ignoring incoming data makes for a bad experience for the client/robot whose requests you are ignoring). This would also allow you to accept() incoming TCP connections at any time (since an incoming TCP connection would cause your TCP_Server socket to select as ready-for-read, at which point you could then call accept() on it again).
I am learning about IOCP and have decided to write my own wrapper class based on the following article:
http://www.codeproject.com/Articles/13382/A-simple-application-using-I-O-Completion-Ports-an
My project is a C++ TCP server using IOCP. The client uses send() and recv() to send and receive data which I cannot change (from what I've been told this shouldn't cause any problem, but I am mentioning it just in case). It also creates a socket using socket() (and not WSASocket()).
Everything seems to be working fine (no error with CreateIoCompletionPort, I can add a socket descriptor to the existing completion port without any error. I've checked everything by adding a call to WSAGetLastError() after each of these functions).
(Before anything, please don't mind the inconsistent coding style. I like to make stuff work first and then clean it all up.)
socket_ = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_IP, NULL, 0, WSA_FLAG_OVERLAPPED);
setsockopt(socket_, IPPROTO_IP, SO_DEBUG | TCP_NODELAY, sockopt, 4);
ioctlsocket(socket_, FIONBIO, &ulSockMode_);
sin_.sin_family = AF_INET;
sin_.sin_port = htons((uint16_t)uiPort_));
hAccept_[0] = WSACreateEvent(); //only 1 event, I'm using an array for convenience
if (hAccept_ == WSA_INVALID_EVENT)
{
//this is never executed
}
WSAEventSelect(socket_, hAccept_[0], FD_ACCEPT);
After an incoming connection is detected (I use WSAWaitForMultipleEevents and WSAEnumNetworkEvents which work don't trigger any error), I use the following code to accept the client (and this is where the problems start):
SOCKET sock_client{ INVALID_SOCKET };
int32_t len_si{ sizeof(SOCKADDR_IN) };
//sock_client = accept(socket_, reinterpret_cast<SOCKADDR*>(pSockAddr), &len_si); // this works fine
//sock_client = sock_client = WSAAccept(socket_, reinterpret_cast<SOCKADDR*>(pSockAddr), &len_si, NULL, 0);//works fine too
char buf[2 * (sizeof(SOCKADDR_IN) + 16)];
WSAOVERLAPPED wsaovl;
uint32_t bytes{ 0 };
BOOL b = AcceptEx(socket_, sock_client, (PVOID)buf, 0, sizeof(SOCKADDR_IN) + 16, sizeof(SOCKADDR_IN) + 16, reinterpret_cast<LPDWORD>(&bytes), &wsaovl); //this fails, returns 0
int32_t test = WSAGetLastError(); // this returns 6 (WSA_INVALID_HANDLE)
I have no idea why it works with accept() and WSAAccept(), however it doesn't with AcceptEx().
If I use accept() though, after accepting the client I need to call WSARecv() immediately. I'm not sending anything back to the client just yet but I read that it needs to be called before GetQueuedCompletionStatus() in the worker thread:
WSABUF* buf = new WSABUF;
OVERLAPPED* ovl = new OVERLAPPED;
int32_t flags{ 0 };
int32_t bytes{ 0 };
int32_t bytes_recv = WSARecv(client_socket, buf, 1, &flags, &bytes, ovl, NULL); // this returns -1
int32_t err = WSAGetLastError(); // this returns 6 (WSA_INVALID_HANDLE)
And since this doesn't work, the GetQueuedCompletionStatus() routine in my worker thread keeps on hanging (or at least, I assume this is the reason)
Is there anything wrong with what I'm doing? I've been trying to search around and fix it since yesterday night, I know it's not a lot of time but I really don't see what I'm not doing correctly.
UPDATE:
I have changed the way I initialize my socket for AcceptEx().
SOCKET sock_client = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_IP, NULL, 0, WSA_FLAG_OVERLAPPED);
and
WSAOVERLAPPED wsaovl = {};
AcceptEx() still returns false, however the error returned by WSAGetLastError() is now 997 (WSA_IO_PENDING). I'm not sure what I/O operation is pending exactly and how I would go about fixing it.
I had stumbled upon similar hurdle when I was learning I/O Completion Ports (IOCP)...
I think the problem is that, in the scheme of IOCP socket model, the most complicated part is the beginning phase of 'socket acceptance'. That is why most tutorial skip over it and begin the discussion on how to handle send/recv, instead.
If you want to develop sufficient understanding of IOCP so that you could implement a production software then my advice to you is to study it until you completely grasp it (this answer below is not enough). One document that I would recommend is chapter 5 of 'Network Programming for Microsoft Windows - 2nd edition'. The book may be old but valid for IOCP. Also, the article 'Windows via C/C++: Synchronous and Asynchronous Device I/O' touches some aspects of IOCP, though not enough information to do production software.
I will try to explain as best as I can, however, I must warn you that this may not be enough. Here it goes...
So, the part you are missing is "How to do 'socket acceptance' in an IOCP socket model".
First off, lets examine the typical Winsock (non-IOCP) sequence of calls on server;
// (1) Create listen socket on server.
WSASocket()
// (2) Bind an address to your listen socket.
bind()
// (3) Associate the listen socket with an event object on FD_ACCEPT event.
WSAEventSelect(,, FD_ACCEPT )
// (4) Put socket in listen state - now, Windows listening for new
// connection requests. If new requests comes, the associated
// event object will be set.
listen()
// (5) Wait on the event object associated on listen socket. This
// will get signaled when a new connection request comes.
WaitForSingleObject() {
// (6) A network activity has occurred. Verify that FD_ACCEPT has
// raised the event object. This also resets the event object
// so WaitForSingleObject() does not loop non-stop.
WSAEnumNetworkEvents()
// (7) Understanding this part is important. The WSAAccept() doesn't
// just accept connection, it first creates a new socket and
// then associates it with the newly accepted connection.
WSAAccept()
}
The step (7) is ok for non-IOCP based models. However, when looking it from the performance point of view - socket creation is expensive. And it slows down the connection acceptance process.
In IOCP model, sockets are created in advance for new incoming connection requests. Not only sockets are created in advance they are associated with the listen socket even before the connection request comes. To achieve this Microsoft has provided extension functions. Two such functions that are required for IOCP model are AcceptEx() & GetAcceptExSockaddrs().
Note: When using these extension functions they need to be loaded at runtime in order to avoid performance penalty. This can be achieved using WSAIoctl(). For further read refer to the MSDN documentation on AcceptEx().
Caveat: AcceptEx() can be used to set the new socket to receive some data as part of connection-acceptance process. This feature needs to be disabled as it makes application susceptible to DoS attack i.e., a connection request is issued but no data is sent. The receiving application will wait on that socket indefinitely. To avoid that just pass 0 value for its 'dwReceiveDataLength' parameter.
How to setup connection-acceptance code for IOCP model?
One way to do this is;
// (1) Create IO completion port
CreateIoCompletionPort()
// (2) Have a method that creates worker threads say 'CreateWorkerThreads()'.
// This assign same method (say WorkerThread_Func()) to all worker threads.
// In the WorkerThread_Func() threads are blocked on call to
// GetQueuedCompletionStatus().
CreateWorkerThreads()
// (3) Create listen socket.
WSASocket()
// (4) Associate listen socket to IO Completion Port created earlier.
CreateIoCompletionPort()
// (5) Bind an address to your listen socket.
bind()
// (6) Put socket in listen state - now, Windows listening for new
// connection requests. If a new request comes, GetQueuedCompletionStatus()
// will release a thread.
listen()
// (7) Create sockets in advance and call AcceptEx on each of
// these sockets. If a new connection requests comes
// Windows will pick one of these sockets and associate the
// connection with it.
//
// As an example, below loop will create 1000 sockets.
GUID GuidAcceptEx = WSAID_ACCEPTEX;
DWORD dwBytes;
LPFN_ACCEPTEX lpfnAcceptEx;
// First, load extension method.
int retCode = WSAIoctl(listenSocket,
SIO_GET_EXTENSION_FUNCTION_POINTER,
&GuidAcceptEx,
sizeof(GuidAcceptEx),
&lpfnAcceptEx,
sizeof(lpfnAcceptEx),
&dwBytes,
NULL,
NULL
);
for( /* loop for 1000 times */ ) {
SOCKET preemptiveSocket = WSASocket(, , , , , WSA_FLAG_OVERLAPPED);
lpfnAcceptEx(listenSocket, preemptiveSocket,,,,,,);
}
This essentially prepare your application to accept sockets in IOCP way. When a new connection requests comes one of the worker threads, that are waiting on GetQueuedCompletionStatus(), will be released and handed over the pointer to the data structure. This will have the socket that was created by lpfnAcceptEx().
Is the process complete? Not yet. The socket accepted through AcceptEx() call does not inherit properties of listenSocket. To do that you need to call;
setsockopt( acceptSocket, SOL_SOCKET, SO_UPDATE_ACCEPT_CONTEXT,
(char*)&listenSocket, sizeof(listenSocket) );
Now, acceptSocket is good to use for WSASend / WSARecv!
Something is missing! I skipped the part about how does a worker thread gets the acceptedSocket from GetQueuedCompletionStatus() ?
The answer is, by passing your specially crafted structure to lpfnAcceptEx(). When the GetQueuedCompletionStatus() returns it will have this data structure containing the socket that you will have passed.
How to make such a structure? By creating a structure having 'WSAOVERLAPPED' as its first member. You can have any members of your own after the first member. For example, my structure looked like;
typedef struct _WSAOVERLAPPEDPLUS
{
WSAOVERLAPPED ProviderOverlapped; // 'WSAOVERLAPPED' has to be the first member.
SOCKET client; // Use this to pass preemptive socket.
SOCKET listenSocket; // Use this to pass the listenSocket.
DWORD dwBytes;
SOCKET_OPERATION operation; // Enum to assist in knowing what socket operation ...
} WSAOVERLAPPEDPLUS, *LPWSAOVERLAPPEDPLUS;
...
typedef enum SOCKET_OPERATION {
UNINITIALIZED_ENUM, // To protect against memory leaks and uninitialized buffers.
OP_ACCEPTEX,
OP_RECEIVE,
OP_SEND
};
...
//
// So the previously mentioned for() loop will become;
//
for( /* loop for 1000 times */ ) {
SOCKET preemptiveSocket = WSASocket(, , , , , WSA_FLAG_OVERLAPPED);
LPWSAOVERLAPPEDPLUS pOl = new WSAOVERLAPPEDPLUS();
// Initialize our "extended" overlapped structure
memset(pOl, 0, sizeof(WSAOVERLAPPEDPLUS));
pOl->operation = OP_ACCEPTEX;
pOl->client = preemptiveSocket;
pOl->listenSocket = listenSocket;
int buflen = (sizeof(SOCKADDR_IN) + 16) * 2;
char* pBuf = new char[buflen];
memset(pBuf, 0, buflen);
m_lpfnAcceptEx(listenSocket,
preemptiveSocket,
pBuf,
0, // Passed 0 to avoid reading data on accept which in turn
// avoids DDoS attack i.e., connection attempt without data will
// cause AcceptEx to wait indefinitely.
sizeof(SOCKADDR_IN) + 16,
sizeof(SOCKADDR_IN) + 16,
&pOl->dwBytes,
&pOl->ProviderOverlapped
);
}
... and in the worker thread when GetQueuedCompletionStatus() returns;
while (TRUE)
{
bOk = ::GetQueuedCompletionStatus(hCompPort, &bytes_transferred, &completion_key, &pOverlapped, INFINITE);
if (bOk) {
// Process a successfully completed I/O request
if (completion_key == 0) {
// Safe way to extract the customized structure from pointer
// is to use 'CONTAINING_RECORD'. Read more on 'CONTAINING_RECORD'.
WSAOVERLAPPEDPLUS *pOl = CONTAINING_RECORD(pOverlapped, WSAOVERLAPPEDPLUS, ProviderOverlapped);
if (pOl->operation == OP_ACCEPTEX) {
// Before doing any WSASend/WSARecv, inherit the
// listen socket properties by calling 'setsockopt()'
// as explained earlier.
// The listenSocket and the preemptive socket are available
// in the 'pOl->listenSocket' & 'pOl->client', respectively.
}
delete pOl;
}
}
else {
// Handle error ...
}
I hope this gave you idea on how AcceptEx() is utilized with IOCP.
I'm using winsocks and I am coding an IDS/Honeypot, this is just a small section of it, because at the moment I want the server to listen on multiple sockets (7) and accept the connections, but I've tried to dynamically create the sockets with an array (and the listener etc) but I am still having trouble - I've tried it multiple ways but so far, all I've managed to do is get it working successfully on ONE socket, and LISTEN to all sockets, but not accept them.
So, this was my last attempt but not sure, maybe I need to use threads or declare the sockets differently?
So far, in this small test code, I want:
Initialize server
listen on all 7 ports (1111,2222 ...etc)
Accept an incoming connection on ANY of them
display both messages on client/server
drop the connection
and continue
It's a little sloppy I know, but here is the code so far and I think you can see where I am going with it:
#include <iostream>
#include <winsock2.h>
#include <string>
#pragma comment(lib, "ws2_32.lib")
int main()
{
std::cout<<"Honeypot server [test #1] by Dreamwalker"<<std::endl;
WSADATA wsa;
SOCKET s[7] , new_socket[7];
struct sockaddr_in server , client;
int c, port[7] = {1111,2222,3333,4444,5555,6666,7777};
char *message;
std::cout<<"\nInitialising Winsock and other components...";
if (WSAStartup(MAKEWORD(2,2),&wsa) != 0)
{
std::cout<<"Failed. Error Code :"<<WSAGetLastError()<<std::endl;
return 1;
}
//!IMPORTANT: create multiple new sockets on different ports
int i = 0;
for( i = 0; i < 7; i++)
{
//Create socket
if((s[i] = socket(AF_INET , SOCK_STREAM , 0 )) == INVALID_SOCKET)
{
std::cout<<"Could not create socket : "<< WSAGetLastError()<<std::endl;
}
//Prepare the sockaddr_in structure
server.sin_family = AF_INET;
server.sin_addr.s_addr = INADDR_ANY;
server.sin_port = htons( port[i] );
//Bind
if( bind(s[i] ,(struct sockaddr *)&server , sizeof(server)) == SOCKET_ERROR)
{
std::cout<<"Bind failed with error code : "<< WSAGetLastError()<<std::endl;
}
/*!ALL CREATION CHECKING DONE, now create multiple sockets on the server
and listen for connections*/
c = sizeof(struct sockaddr_in);
listen(s[i] , SOMAXCONN);
}
///ALL INITIALIZED
std::cout<<"DONE!"<<std::endl;
//Listen/accept incoming connections
std::cout<<"Now listening for connections"<<std::endl;
new_socket[i] = accept(s[i] , (struct sockaddr *)&client, &c);
if (new_socket[i] == INVALID_SOCKET)
{
std::cout<<"accept failed with error code : "<< WSAGetLastError()<<std::endl;
}
//Accepted connection
else{
std::cout<<"Someone has connected to this machine!"<<std::endl;
message = "Hello Client , I have received your connection.\n";
send(new_socket[i] , message , strlen(message) , 0);
closesocket(s[i]);
}
std::cout<<"FINISHED"<<std::endl;
WSACleanup();
getchar();
return 0;
}
And now it's throwing a runtime error as well:
WSAENOTSOCK
10038
Socket operation on nonsocket.
An operation was attempted on something that is not a socket. Either the socket handle parameter did not reference a valid socket,
or for select, a member of an fd_set was not valid.
Which (including debugging) indicates that the socket isn't declared properly when creating on an array, advice?
You code to create/bind/listen is all good. Then:
new_socket[i] = accept(s[i] , (struct sockaddr *)&client, &c);
Firstly, by the time this runs you're outside the loop, and i is 7 which is past the end of the array of sockets, which is why you get the not-a-socket error.
Secondly, accept() is a blocking call, so you can't just call accept() on all the sockets from the same thread the way you did for listen. You need to either have a separate thread block in accept() for each of the ports, or find out which one has a client connection attempt in progress using e.g. select (or epoll - does Windows have that?), then accept() a client on that specific socket (but then you've still got to either create a thread to handle the client read/recvs and write/sends or use select/epoll to find out when there's input ready to read, or more space in output buffers for transmission). There's also a race condition to be wary of if you use select/epoll - a listening socket might signal readiness for accepting a client connection, but by the time you call accept() that connection attempt's failed and forgotten, then if the listening socket hasn't been set to non-blocking mode it'll hang there waiting for another client to connect to that specific socket. IMHO, this is a case where threading is actually easier.
I think it's more "Windowsy" to use IO Completion Ports (you might want to Google), but AFAIK they're totally unportable. Winsock's not an exact match for BSD sockets, but the porting or dual-support effort's small.
It's hard for me to describe my question, though, the title above maybe somehow not clear.
I have a ordinary tcp server which have a listen socket, and can accept up to 32 clients, I have created sockets for each client, and I use select system call to monitor which client could be read, following code is a snippet of my program.
where, rset is a member variable of my class and its type is fd_set. I have zeroed it using FD_ZERO in the constructor.
timeo is also a member variable, type is struct timeval initialized with 10 seconds.
this->sock is again a member variable, used for listening and accepting new clients. I have called FD_SET(this->sock, &rset) before.
print_trace is just a macro which print the message and append a '\n'.
while(1) {
int count = select(FD_SETSIZE, &rset, /*&wset*/ NULL, NULL, &timeo);
printf("%d fds\n", count);
if(count) {
if(FD_ISSET(this->sock, &rset)) {
// new connection comes and now this line will not blocked
if((csocks[sock_count] = accept(this->sock, NULL, NULL))) {
FD_SET(csocks[sock_count], &rset);
++sock_count;
}
} else {
print_trace("there are clients can be read.");
for(int i = 0 ; i < sock_count ; ++i) {
if(FD_ISSET(csocks[i], &rset)) {
char buffer[512] = {0};
recv(csocks[i], buffer, 512, 0);
printf("here client socket number: %d, i=%d, message: %s\n", csocks[i], i, buffer);
}
}
}
}
timeo.tv_sec = 10;
timeo.tv_usec = 0;
}
I know I am not re-enable the this->sock using FD_SET, for select will clear all the bits when timeout, but it have no concern with my trouble.
my trouble is, when I run the server program, and in 10 seconds, I run a client program to connect to this server, the select returns 1 normally, so that the client socket be created and be added to rset, and then, the server goes to next loop, caution! now, right now, terminate the client program immediately, don't wait for the select returns.
Okay, now the trouble will reappear, the server program keep printing the following info:
1 fds
there are clients can be read.
here client socket number: 6, i=2, message:
1 fds
there are clients can be read.
here client socket number: 6, i=2, message:
1 fds
there are clients can be read.
here client socket number: 6, i=2, message:
1 fds
there are clients can be read.
here client socket number: 6, i=2, message:
...
...
I have use tcpdump monitor the connection, when terminate the client, it just send an FIN packet, and the server program just send an ACK packet, there is no any other data flowing the connection.
why the select keep finding the client socket can be read while it just read empty message(as the printing message showing)?
Any help will be appreciated.
update: I know the usage of select method clearly, I thought I don't use this method properly so I have had spent about one hour to study this method, in order to solve my problem, but found no result still.
You're ignoring the result returned by recv(), which is itself an error, and specifically you're ignoring the possibility that it's zero, which is end of stream, on which you should close the socket, so you don't just select on EOS again.