Handling multiple connections using QThreadPool - c++

Question

Consider a situation where you need to maintain 256 tcp connections with devices just for ocassionally sending commands. I want to do this in parallel(It needs to block until it gets the response), I'm trying to use QThreadPool for this purpose but I have some doubts if it is possible.
I tried to use QRunnable but I'm not sure how sockets will behave between threads (sockets should be used only in thread that they were created in?)
I'm also worried about efficiency of this solution, I would be glad if somebody could propose some alternatives, not necessarily using QT.
Below I'm posting some snippets of the code.
class Task : public QRunnable {
Task(){
//creating TaskSubclass instance and socket in it
}
private:
TaskSubclass *sub;
void run() override {
//some debug info and variable setting...
sub->doSomething( args );
return;
}
};
class TaskSubclass {
Socket *sock; // socket instance
//...
void doSomething( args )
{
//writing to socket here
}
}
class MainProgram : public QObject{
Q_OBJECT
private:
QThreadPool *pool;
Task *tasks;
public:
MainProgram(){
pool = new QThreadPool(this);
//create tasks here
}
void run(){
//decide which task to start
pool->start(tasks[i]);
}
};

Answer 1

My favorite solution for this problem is by multiplexing your sockets using select(). That way you don't need to create additional threads, and it is a "very POSIX" way to do it.
See for example see this tutorial:
http://www.binarytides.com/multiple-socket-connections-fdset-select-linux/
Or a related question in:
Using select(..) on client

Answer 2

As OMD_AT has allready pointed out the best solution is to use Select() and let the kernel do the job for you :-)
here you have an example of an Async approach and an Syncron multi thread approach.
In this example we create 10 connection to a google webservice and make a simple get request to the server, we measure how long all connections in each approach needed to receive the response from the google server.
Be aware that you should use a more faster webserver to make a real test, like the localhost because the network latency has a big impact on the result.
#include <QCoreApplication>
#include <QTcpSocket>
#include <QtConcurrent/QtConcurrentRun>
#include <QElapsedTimer>
#include <QAtomicInt>
class Task : public QRunnable
{
public:
Task() : QRunnable() {}
static QAtomicInt counter;
static QElapsedTimer timer;
virtual void run() override
{
QTcpSocket* socket = new QTcpSocket();
socket->connectToHost("www.google.com", 80);
socket->write("GET / HTTP/1.1\r\nHost: www.google.com\r\n\r\n");
socket->waitForReadyRead();
if(!--counter) {
qDebug("Multiple Threads elapsed: %lld nanoseconds", timer.nsecsElapsed());
}
}
};
QAtomicInt Task::counter;
QElapsedTimer Task::timer;
int main(int argc, char *argv[])
{
QCoreApplication app(argc, argv);
// init
int connections = 10;
Task::counter = connections;
QElapsedTimer timer;
/// Async via One Thread (Select)
// handle the data
auto dataHandler = [&timer,&connections](QByteArray data) {
Q_UNUSED(data);
if(!--connections) qDebug(" Single Threads elapsed: %lld nanoseconds", timer.nsecsElapsed());
};
// create 10 connection to google.com and send an http get request
timer.start();
for(int i = 0; i < connections; i++) {
QTcpSocket* socket = new QTcpSocket();
socket->connectToHost("www.google.com", 80);
socket->write("GET / HTTP/1.1\r\nHost: www.google.com\r\n\r\n");
QObject::connect(socket, &QTcpSocket::readyRead, [dataHandler,socket]() {
dataHandler(socket->readAll());
});
}
/// Async via Multiple Threads
Task::timer.start();
for(int i = 0; i < connections; i++) {
QThreadPool::globalInstance()->start(new Task());
}
return app.exec();
}
Prints:
Multiple Threads elapsed: 62324598 nanoseconds
Single Threads elapsed: 63613967 nanoseconds

Answer 3

Although, the answer is already accepted, I would like to share my)
What I understood from your question: Having 256 currently active connections, from time to time you send a request ("command" as you named it) to one of them and wait for the response. Meanwhile, you want to make this process multithreaded and, though you said "It needs to block until it gets the response", I assume you implied blocking a thread which handles request-response process, but not the main thread.
If I indeed understand the question right, here is how I suggest to do it using Qt:
#include <functional>
#include <QObject> // need to add "QT += core" in .pro
#include <QTcpSocket> // QT += network
#include <QtConcurrent> // QT += concurrent
#include <QFuture>
#include <QFutureWatcher>
class CommandSender : public QObject
{
public:
// Sends a command via connection and blocks
// until the response arrives or timeout occurs
// then passes the response to a handler
// when the handler is done - unblocks
void SendCommand(
QTcpSocket* connection,
const Command& command,
void(*responseHandler)(Response&&))
{
const int timeout = 1000; // milliseconds, set it to -1 if you want no timeouts
// Sending a command (blocking)
connection.write(command.ToByteArray()); // Look QByteArray for more details
if (connection.waitForBytesWritten(timeout) {
qDebug() << connection.errorString() << endl;
emit error(connection);
return;
}
// Waiting for a response (blocking)
QDataStream in{ connection, QIODevice::ReadOnly };
QString message;
do {
if (!connection.waitForReadyRead(timeout)) {
qDebug() << connection.errorString() << endl;
emit error(connection);
return;
}
in.startTransaction();
in >> message;
} while (!in.commitTransaction());
responseHandler(Response{ message }); // Translate message to a response and handle it
}
// Non-blocking version of SendCommand
void SendCommandAsync(
QTcpSocket* connection,
const Command& command,
void(*responseHandler) (Response&&))
{
QFutureWatcher<void>* watcher = new QFutureWatcher<void>{ this };
connect(watcher, &QFutureWatcher<void>::finished, [connection, watcher] ()
{
emit done(connection);
watcher->deleteLater();
});
// Does not block,
// emits "done" when finished
QFuture<void> future
= QtConcurrent::run(this, &CommandSender::SendCommand, connection, command, responseHandler);
watcher->setFuture(future);
}
signals:
void done(QTcpSocket* connection);
void error(QTcpSocket* connection);
}
Now you can send a command to a socket using a separate thread taken from a thread pool: under the hood QtConcurrent::run() uses the global instance of QThreadPool provided by Qt for you. That thread blocks until it gets a response back and than handles it with responseHandler . Meanwhile, your main thread managing all your commands and sockets stays unblocked. Just catch done() signal which tells that response was received and handled successfully.
One thing to note: asynchronous version sends request only when there is a free thread in the thread pool and waits for it otherwise. Of course, that is the behavior for any thread pool (that is exactly the point of such pattern) but just do not forget about that.
Also I was writing code without Qt in handy so may contain some errors.
Edit: As it turned out, this is not thread safe as sockets are not reentrant in Qt.
What you can do about it is to associate a mutex with a socket and lock it each time you execute its function. This can be done easily creating a wrapper around QTcpSocket class. Please, correct me if I wrong.