multiple epoll control with thread avoid EEXIST - c++

I made TCP/UDP epoll server socket and each socket listening in each other threads. But, epoll_ctl_return exist.
I use different ports so, this problem may be solved using fork, but
I want to handle multiple servers socket in one process and multiple threads.
I read an article in c# multiple server socket.
is it impossible using epoll?
Or multiple server sockets add epoll?
that's my pseudo code
listener* tcpListener = new listener(ip,port, protocolType.tcp);
tcpListener.bind();
std::thread t{listener::listen, tcpListener};
listener* udpListener = new listener(ip,port,protocolType.udp);
udpListener.bind();
std::thread u{listener::listen, udpListener};

I'm not sure where epoll fits in to your description. If every socket is owned by a separate thread then there is no point to using epoll. epoll() is used for asynchronous programming, where multiple sockets can be managed by a single thread of execution.

Related

multiple boost io_service for multiple network cards on separate threads

I have available multiple network cards on the pc I am working on. I'd like to use a thread per each network card to achieve better performance. My application will receive UDP traffic from multiple sources that I want to group across these network cards/threads.
With such architecture in mind, if use boost::asio does make sense to have multiple boost::asio::io_service objects, each running on a separate thread assigned to a single network card?
Essentially a single boost::asio::io_service will run in a thead to execute all asynchronous methods for all UDP sockets binding on the same network card.
Is this a good idea to process the network traffic received on each card independently?
NIC A ---> boost::io_service nic_a;
socket one(nic_a);
one.bind(nic_a, 0);
socket two(nic_a);
two.bind(nic_a);
// all sockets call async_read
-------------------------------------------------------
nic_a.run(); /// call this in Thread #a
-------------------------------------------------------
NIC B ---> boost::io_service nic_b;
socket three(nic_b);
three.bind(nic_b, 0);
socket four(nic_b);
four.bind(nic_b);
// all sockets call async_read
-------------------------------------------------------
nic_b.run(); /// call this in Thread #b
-------------------------------------------------------
Instead of using an io_service in a thread per pair of network cards, you may be better off wrapping your sockets in an asio::io_service::strand and using a single io_service in a thread pool, see: Strands: Use Threads Without Explicit Locking and Asynchronous IO with boost asio.
It is easiest to put the sockets and strands together in a class as in this example: Timer 5 example. There is some code that supports asio UDP sockets and strands here.

threads for tcp communication

My question is about usage of threads. Im making an application that connects to a device over TCP/IP. Im using boost::asio lib. I have decided to use a read or listening thread and a write thread for listening and writing to the device respectively. My confusion is should the function creating the socket that handles the communication also be a thread.
Thanks :)
In my client class, I create 2 worker threads to handle sending and receiving messages which are used for multiple connections to multiple servers. The thread that creates those 2 worker threads happens to be the user interface thread. This is what my code looks like:
// Create the resolver and query objects to resolve the host name in serverPath to an ip address.
boost::asio::ip::tcp::resolver resolver(*IOService);
boost::asio::ip::tcp::resolver::query query(serverPath, port);
boost::asio::ip::tcp::resolver::iterator EndpointIterator = resolver.resolve(query);
// Set up an SSL context.
boost::asio::ssl::context ctx(*IOService, boost::asio::ssl::context::tlsv1_client);
// Specify to not verify the server certificiate right now.
ctx.set_verify_mode(boost::asio::ssl::context::verify_none);
// Init the socket object used to initially communicate with the server.
pSocket = new boost::asio::ssl::stream<boost::asio::ip::tcp::socket>(*IOService, ctx);
//
// The thread we are on now, is most likely the user interface thread. Create a thread to handle all incoming socket work messages.
if (!RcvThreadCreated)
{
WorkerThreads.create_thread(boost::bind(&SSLSocket::RcvWorkerThread, this));
RcvThreadCreated = true;
WorkerThreads.create_thread(boost::bind(&SSLSocket::SendWorkerThread, this));
}
// Try to connect to the server. Note - add timeout logic at some point.
boost::asio::async_connect(pSocket->lowest_layer(), EndpointIterator,
boost::bind(&SSLSocket::HandleConnect, this, boost::asio::placeholders::error));
The worker threads handle all socket I/O. It depends on what you are doing, but the 2 worker threads for servicing the socket will need to be created from another thread. That other thread can be the user interface thread or main thread if you want since it will be returning pretty quickly. If you have multiple connections to servers or clients, then It is up to you to decide whether or not you want more than one set of threads to service them.
That depends on whether you want to read and write at the same time. In that case you would need one thread for reading and one for writing, but you would have to properly synchronize those in case both streams to and from the device have something to do with each other (what they probably do). However, talking to a device sounds to me like a task where you establish a connection, send some request, wait for and read the answer, send another request, wait for and read the next answer, etc. In that case using just one thread is sufficient and makes your life a lot easier.

Writing multithreaded TCP server on Linux

At work I have been tasked with implementing a TCP server as part of a Modbus slave device. I have done a lot of reading both here on stack exchange and on the internet in general (including the excellent http://beej.us/guide/bgnet/) but I am struggling with a design issue. In summary, my device can accept just 2 connections and on each connection will be incoming modbus requests which I must process in my main controller loop and then reply with success or failure status. I have the following ideas of how to implement this.
Have a listener thread that creates, binds, listens and accepts connections, then spawns a new pthread to listen on the connection for incoming data and close connection after an idle timeout period. If the number of active threads is currently 2, new connections are instantly closed to ensure only 2 are allowed.
Do not spawn new threads from the listener thread, instead use select() to detect incoming connection requests as well as incoming modbus connects on active connections (similar to the approach in Beejs guide).
Create 2 listener threads each of which creates a socket (same IP and port number) which can block on accept() calls, then close the socket fd and deal with the connection. Here I am (perhaps naively) assuming that this will only allow max of 2 connections which I can deal with using blocking reads.
I have been using C++ for a long time but I am fairly new to Linux development. I would really welcome any suggestions as to which of the above approaches is best (if any) and if my inexperience with Linux means that any of them are really really bad ideas. I am keen to avoid fork() and stick to pthreads as incoming modbus requests are going to be queued and read off a main controller loop periodically. Thanks in advance for any advice.
The third alternative won't work, you can only bind to the local address once.
I would probably use your second alternative, unless you need to do a lot of processing in which case a combination of the first to alternatives might be useful.
The combination of the two first alternative I'm thinking of is to have the main thread (the one you always have when a program starts) create two worker threads, then go a blocking accept call to wait for a new connection. When a new connection arrives, tell one of the threads to start working on the new connection and go back to block on accept. When the second connection is accepted you tell the other thread to work on that connection. If both connections are open already, either don't accept until one connection is closed, or wait for new connections but close them immediately.
All of the design option you propose are not very object oriented, and they're all geared more towards C than C++. If your work allows you to use boost, then the Boost.Asio library is fantastic for making simple (and complex) socket servers. You could take nearly any of their examples and trivially extend it to only allow 2 active connections, closing all others as soon as they are opened.
Off the top of my head, their simple HTTP server could be modified to do this by keeping a static counter in the connection class (inc in the constructor, dec in the destructor), and when a new one is created check the count and decide whether to close the connection. The connection class could also gain a boost::asio::deadline_timer to keep track of timeouts.
This would most closely resemble your first design choice, boost could do this in 1 thread and in the background does something similar to select() (usually epoll()). But this is the "C++ way", and in my opinion using select() and raw pthreads is the C way.
Since you are only dealing with 2 connections, thread per connection is perfect for this kind of application. Object oriented approaches using non-blocking or asynchronous I/O would be better if you needed to scale up to thousands of connections. 2 listener threads makes sense, you don't need to close the accept fd. Just come back to accept on it when the connection is completed. In fact, a variation is to have three threads blocked doing accept. If two of the threads are actively handling connections, then the third resets the newly created connection (or returns busy response, whatever is appropriate for your device).
To have all three threads block on accept, you need to have the main thread create and bind your socket before the three threads launch to do their accept/handle processing.
The man page for pthreads on Linux indicates that accept is thread-safe. (The section under thread-safe functions lists the functions that are not thread-safe, go figure.)

Multi threaded client server

Hi I am working on an assignment writing multi threaded client server.
So far I have done is open a socket in a port and forked two thread for listening and writing to client. But I need to connect two type of clients to the server and service them differently. My question is what would be my best approach?
I am handling connection in a class which has a infinite loop to accept connection. When ever a connection is accepted, the class create two thread to read and write to client? Now if I wnat to handle another client of different type, what should we do?
Do I need to open another port? or is it possible to service through same port? May be if it is possible to identify the type of client in the socket than I can handle messages differently.
Or do you suggest like this?
Fork two thread for two type of client and monitor inbound connection in each thread in different port.
when a connection accepted each thread spawn another two thread for listening and writing.
please make a suggestion.
Perhaps you'll get a better answer from a Unix user, but I'll provide what I know.
Your server needs a thread that opens a 'listening' socket that waits for incoming connections. This thread can be the main thread for simplicity, but can be an alternate thread if you are concerned about UI interaction, for example (in Windows, this would be a concern, not sure about Unix). It sounds like you are at least this far.
When the 'listening' socket accepts a connection, you get a 'connected' socket that is connected to the 'client' socket. You would pass this 'connected' socket to a new thread that manages the reading from and writing to the 'connected' socket. Thus, one change I would suggest is managing the 'connected' socket in a single thread, not two separate threads (one for reading, one for writing) as you have done. Reading and writing against the same socket can be accomplished using the select() system call, as shown here.
When a new client connects, your 'listening' socket will provide a new 'connected' socket, which you will hand off to another thread. At this point, you have two threads - one that is managing the first connection and one that is managing the second connection. As far as the sockets are concerned, there is no distinction between the clients. You simply have two open connections, one to each of your two clients.
At this point, the question becomes what does it mean to "service them differently". If the clients are expected to interact with the server in unique ways, then this has to be determined somehow. The interactions could be determined based on the 'client' socket's IP address, which you can query, but this seems arbitrary and is subject to network changes. It could also be based on the initial block of data received from the 'client' socket which indicates the type of interaction required. In this case, the thread that is managing the 'connected' socket could read the socket for the expected type of interaction and then hand the socket off to a class object that manages that interaction type.
I hope this helps.
You can handle the read-write on a single client connection in one thread. The simplest solution based on multiple-threads will be this:
// C++ like pseudo-code
while (server_running)
{
client = server.accept();
ClientHandlingThread* cth = CreateNewClientHandlingThread(client);
cth->start();
}
class ClientHandlingThread
{
void start()
{
std::string header = client->read_protocol_header();
// We get a specific implementation of the ProtocolHandler abstract class
// from a factory, which create objects by inspecting some protocol header info.
ProtocolHandler* handler = ProtocolHandlerFactory.create(header);
if (handler)
handler->read_write(client);
else
log("unknown protocol")
}
};
To scale better, you can use a thread pool, instead of spawning a new thread for each client. There are many free thread pool implementations for C++.
while (server_running)
{
client = server.accept();
thread_pool->submit(client);
cth->start();
}
The server could be improved further by using some framework that implements the reactor pattern. They use select or poll functions under the hood. You can use these functions directly. But for a production system it is better to use an existing reactor framework. ACE is one of the most widely known C++ toolkits for developing highly scalable concurrent applications.
Different protocols are generally serviced on different ports. However, you could service both types of clients over the same port by negotiating the protocol to be used. This can be as simple as the client sending either HELO or EHLO to request one or another kind of service.

How to cancel waiting in select() on Windows

In my program there is one thread (receiving thread) that is responsible for receiving requests from a TCP socket and there are many threads (worker threads) that are responsible for processing the received requests. Once a request is processed I need to send an answer over TCP.
And here is a question. I would like to send TCP data in the same thread that I use for receiving data. This thread after receiving data usually waits for new data in select(). So once a worker thread finished processing a request and put an answer in the output queue it has to signal the receiving thread that there are data to send. The problem is that I don't know how to cancel waiting in select() in order to get out of waiting and to call send() .
Or shall I use another thread solely for sending data over TCP?
Updated
MSalters, Artyom thank you for you answers!
MSalters, having read your answer I found this site: Winsock 2 I/O Methods and read about WSAWaitForMultipleEvents(). My program in fact must work both on HP-UX and Windows I finally decided to use the approach that had been suggested by Artyom.
You need to use something similar to safe-pipe trick, but in your case you need to use a pair of connected TCP sockets.
Create a pair of sockets.
Add one to the select and wait on it as well
Notify by writing to other socket from other threads.
Select is immediately waken-up as one of the sockets is readable, reads all the
data in this special socket and check all data in queues to send/recv
How to create pair of sockets under Windows?
inline void pair(SOCKET fds[2])
{
struct sockaddr_in inaddr;
struct sockaddr addr;
SOCKET lst=::socket(AF_INET, SOCK_STREAM,IPPROTO_TCP);
memset(&inaddr, 0, sizeof(inaddr));
memset(&addr, 0, sizeof(addr));
inaddr.sin_family = AF_INET;
inaddr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
inaddr.sin_port = 0;
int yes=1;
setsockopt(lst,SOL_SOCKET,SO_REUSEADDR,(char*)&yes,sizeof(yes));
bind(lst,(struct sockaddr *)&inaddr,sizeof(inaddr));
listen(lst,1);
int len=sizeof(inaddr);
getsockname(lst, &addr,&len);
fds[0]=::socket(AF_INET, SOCK_STREAM,0);
connect(fds[0],&addr,len);
fds[1]=accept(lst,0,0);
closesocket(lst);
}
Of course some checks should be added for return values.
select is not the native API for Windows. The native way is WSAWaitForMultipleEvents. If you use this to create an alertable wait, you can use QueueUserAPC to instruct the waiting thread to send data. (This might also mean you don't have to implement your own output queue)
See also this post:
How to signal select() to return immediately?
For unix, use an anonymous pipe. For Windows:
Unblocking can be achieved by adding a dummy (unbound) datagram socket to fd_set and then closing it. To make this thread safe, use QueueUserAPC:
The only way I found to make this multi-threadsafe is to close and recreate the socket in the same thread as the select statement is running. Of course this is difficult if the thread is blocking on the select. And then comes in the windows call QueueUserAPC. When windows is blocking in the select statement, the thread can handle Asynchronous Procedure Calls. You can schedule this from a different thread using QueueUserAPC. Windows interrupts the select, executes your function in the same thread, and continues with the select statement. You can now in your APC method close the socket and recreate it. Guaranteed thread safe and you will never loose a signal.
The typical model is for the worker to handle its own writing. Is there a reason why you want to send all the output-IO through selecting thread?
If you're sure of this model, you could have your workers send data back to the master thread using file descriptors as well (pipe(2)) and simply add those descriptors to your select() call.
And, if you're especially sure that you're not going to use pipes to send data back to your master process, the select call allows you to specify a timeout. You can busy-wait while checking your worker threads, and periodically call select to figure out which TCP sockets to read from.
Another quick&dirty solution is to add localhost sockets to the set. Now use those sockets as the inter-thread communication queues. Each worker thread simply sends something to its socket, which ends up on the corresponding socket in your receiving thread. This wakes up the select(), and your receiving thread can then echo the message on the appropriate outgoing socket.