I am trying to perform multiple long-running initializers on threads. I would like to get results as soon as they are available, regardless of order.
I have tried with a simple thread but got stuck where I needed to create multiple threads... and manage them... And the idea of thread pool, futures, thread watcher kept coming up in my research. So, this is my latest try:
struct Initializers {
BigObject *newBigObject;
bool newValue;
int initValue;
};
class BigObject {
public:
BigObject(QString) {}
void initialize(bool, int) {}
};
class MyClass : public QObject {
Q_OBJECT
public:
MyClass() {
connect(&m_futureWatcher, SIGNAL(resultReadyAt(int)), this, SLOT(initUsingFutureSlot(int)));
connect(&m_futureWatcher, SIGNAL(finished()), this, SLOT(initUsingFutureSlot()));
}
void doImportantWork(QStringList someNames);
private slots:
void initUsingFutureSlot(int);
void initUsingFutureSlot();
private:
void createThreadConnections();
//void initComplete(BigObject *);
QFutureWatcher<void> m_futureWatcher;
QVector<Initializers> m_initializersVector;
};
void longInitializer(Initializers &initializers) {
qDebug() << "Running in thread" << QThread::currentThreadId();
initializers.newBigObject->initialize(initializers.newValue, initializers.initValue);
}
void MyClass::doImportantWork(QStringList someNames) {
qDebug() << QString("Long running process using %1 thread(s)").arg(QThread::idealThreadCount());
// Prepare the vector.
foreach (QString someName, someNames) {
BigObject *newBigObject = new BigObject(someName);
Initializers initializers;
initializers.newBigObject = newBigObject;
initializers.newValue = m_appValue;
initializers.initValue = m_appInit;
m_initializersVector.append(initializers);
}
// Start the work
m_futureWatcher.setFuture(QtConcurrent::map(m_initializersVector, longInitializer));
//m_futureWatcher.waitForFinished();
}
// This does not get called
void MyClass::initUsingFutureSlot(int index) {
qDebug() << "Finished" << index;
//initComplete(m_initializersVector.at(index).newBigObject);
}
// This does not get called
void MyClass::initUsingFutureSlot() {
qDebug() << "Finished"
}
int main(int argc, char *argv[]) {
QApplication app(argc, argv);
MyClass task;
task.doImportantWork(QStringList() << "Hello" << "World");
return app.exec();
}
This version of my code is based on QT5 Tutorial QProgressDialog with QtConcurrent which is the only complete example I was able to find that works with multiple threads.
One problem - their QFutureWatcher's type is void - and I could not find any way to modify the code to use a different type - and not give compile errors. That is why I made the vector they use a class member, to grab the values from there.
My understanding is that the vector gets the slot applied to every member of it - based on documentation.
So, I ran it, but nothing happens. The debug message shows that the threads are running. But the resultReadyAt or finished and their corresponding slots are not called. In addition, closing the program says QThread: Destroyed while thread is still running, so they don't finish and close, as I expected them being handled by the Watcher.
I can make things work with QtConcurrent::run !!!!
I am stuck.
Questions:
How can I make this QFuture / QFutureWatcher work ? What am I missing in my code or what am I doing wrong to make the threads not finish ?
How can I make the template non-void (to retrieve results easier ?)
I have m_futureWatcher.waitForFinished(); commented because that would block the GUI thread, I think - is that correct ?
Is there a better option to run multiple threads ? the word ThreadPool comes to mind but I have seen no examples that I can use - that can notify of completion. I did find a nice example to emulate a ThreadPool (NetResultsIT), but I expect Qt should have something functional.
When closing program, I get QThread: Destroyed while thread is still running. I was under the assumption that threads are closed when done, how should I clean up ? Is deleting the watcher enough ?
I am so lost, fighting these threads for over a week, and they're winning :( Thank you for any help or example.
Using Qt 5.15
Here is a simple example how QThreadPool works
#include <QThreadPool>
#include <QDebug>
#include <QApplication>
#include <QRunnable>
class MyRunnable : public QRunnable
{
public:
MyRunnable():QRunnable(){}
void run(){
for(int i = 0; i < 3; i++)
qDebug() << QThread::currentThreadId() << " i " << i;
}
};
int main(int argc, char *argv[]){
QApplication a(argc, argv);
QThreadPool::globalInstance()->start(new MyRunnable);
QThreadPool::globalInstance()->start(new MyRunnable);
QThreadPool::globalInstance()->start(new MyRunnable);
return 0;
}
output:
0x32c0 i 0
0x32f8 i 0
0x32f8 i 1
0x32f8 i 2
0x32c0 i 1
0x32c0 i 2
0x25a0 i 0
0x25a0 i 1
0x25a0 i 2
Related
I'm using QtConcurrent::run (I know other APIs of QtConcurrent has built in support for progress reporting but I can't use them for other reasons). to run an operation inside a different than the main GUI thread. I also need this operation to notify the GUI thread of the progress made. So what I did is that created a separate function for the operation I want which accepts a callback that carries the information about the progress of the operation. This callback then calls the signal on a QObject living in the main thread.
Here is a full working example that shows my structure:
#include <QCoreApplication>
#include <QObject>
#include <QThread>
#include <QtConcurrent/QtConcurrent>
namespace Operations {
template<typename Callback>
void longOperation(Callback progressCallback)
{
qint64 sum = 0;
for(int i = 0; i < 100; ++i){
QThread::msleep(400);
sum += i;
progressCallback(i/100.0);
}
}
}
class Emitter : public QObject
{
Q_OBJECT
public:
Q_INVOKABLE void doSomething()
{
auto progressCallback = [&](qreal p){
emit progress(p);
};
auto lambda = [progressCallback](){
Operations::longOperation(progressCallback);
};
QtConcurrent::run(lambda);
}
signals:
void progress(qreal);
};
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
Emitter emitter;
QObject::connect(&emitter, &Emitter::progress, [](qreal progress){
qDebug() << "Progress" << progress;
});
emitter.doSomething();
return a.exec();
}
#include "main.moc"
Now my question is using the progressCallback as defined above thread safe? The callback will clearly be triggered from a thread different than the GUI thread, so effectively it's calling emitter.progress() directly on the QObject.
Ok, so I now realised that the code above may not be thread safe. Part of my confusion was what does the emit keyword actually does. It turns out it's actually not much at all. So calling the signal from another thread is not really the best idea.
Instead, one of way of improve the situation is replace the progressCallback with:
auto progressCallback = [&](qreal p){
QMetaObject::invokeMethod(this, [this, p](){ emit progress(p);}, Qt::QueuedConnection);
};
This way the signal is emitted on the thread where the Emitter lives as the lambda slot will be executed "when control returns to the event loop of the receiver's thread" as per the Qt documentation.
If I have a main thread and a separate, permanent worker QThread:
// Main thread where the event loop and QCoreApplication runs.
class NetworkController : public QObject {
public:
Q_OBJECT
void connect_to_ap()
{
WifiAP *myAP = netMgr.get_best_ap();
myAP->connect("my_psk_password");
}
// This is a class from a 3rd-party library, which contains, owns
// and manages the lifetime of "WifiAP" instances.
NetworkManager netMgr;
};
// Separate thread where a state machine runs.
class StateMachineWorker : public QObject {
public:
Q_OBJECT
void on_ready_to_connect_event()
{
// HERE: How to trigger NetworkController::connect_to_ap() and
// *block* until it returns.
}
NetworkController *pNetCtrlr;
}
When the state machine class enters a certain state it should connect to an AP (Access Point). The NetworkController has the functionality to connect to an AP.
I am trying to figure out a way that the state machine can do this in a thread-safe way. The problem is NetworkManager is always updating its list of WifiAP instances: they are created and destroyed often.
It would not be thread-safe for StateMachineWorker to call pNetCtrlr->connect_to_ap() directly (as the NetworkManager in the NetworkController thread could at the same time delete the WifiAP instance).
So what I would like is in StateMachineWorker::on_ready_to_connect_event() to somehow signal the NetworkController to run its connect_to_ap() method in NetworkController's own thread, and to block the StateMachineWorker thread until connect_to_ap() has finished doing its stuff. The reason I want the state machine to be blocked is if I did not block and let it enter the event loop, it could receive some event that would make it transition to another state before connect_to_ap() has finished executing; this must not happen.
Mutex locks to protect the list of WifiAP in NetworkManager would not work as they would need to be inserted inside the 3rd-party library.
You can use QMetaObject::invokeMethod with parameter Qt::BlockingQueuedConnection. This connection type adds all the blocking logic for you and you don't have to change the third party library at all.
A general example:
objects.h
#include <QObject>
#include <QThread>
#include <iostream>
class Object : public QObject {
Q_OBJECT
public slots:
void foo() {
std::cout << "Hello";
thread()->msleep(2000);
std::cout << " world!" << std::endl;
}
};
class Caller : public QObject {
Q_OBJECT
public:
void call(Object* o) {
std::cout << "Calling..." << std::endl;
metaObject()->invokeMethod(o, "foo", Qt::BlockingQueuedConnection);
std::cout << "Finished!" << std::endl;
}
};
main.cpp
#include <QCoreApplication>
#include "objects.h"
int main(int argc, char* argv[])
{
QCoreApplication a(argc, argv);
QThread t;
Object o;
o.moveToThread(&t);
t.start();
Caller().call(&o);
return a.exec();
}
The counterpart is that the method to be called must be a slot. If connect_to_ap is not already a slot you can create a bridge object that do the job, as explained below.
Have in mind that this bridge object must live in the same thread as the NetworkController (in your case the main thread), so the slot is queued in the correct events loop. You can take a look at QObject::moveToThread for further information.
A quick draft would be something like:
class NetworkControllerBridge : public QObject {
Q_OBJECT
NetworkController* nc;
public:
NetworkControllerBridge(NetworkController* nc_) : nc(nc_) {}
public slots:
void connect_to_ap() {
nc->connect_to_ap();
}
};
// ...
void on_ready_to_connect_event()
{
NetworkControllerBridge bridge(pNetCtrlr);
bridge.moveToThread(qApp->thread());
metaObject()->invokeMethod(&bridge, "connect_to_ap", Qt::BlockingQueuedConnection);
}
Update
Another way to call a method through invokeMethod is to mark it as Q_INVOKABLE. Although you still need the bridge since you cannot modify the library, I mention this for completeness of the answer.
I'm trying to understand and use RAII and wanted opinions on this implementation:
I want the RAII PauseProcessRAII to emit a signal and another one in the destructor. Example:
// Header
public:
PauseProcessRAII(QObject *parent = 0);
void Execute();
~PauseProcessRAII();
signals:
void PauseProcess(bool pause_process);
// Source
PauseProcessRAII::~PauseProcessRAII()
{
emit PauseProcess(false);
}
void PauseProcessRAII::Execute()
{
emit PauseProcess(true);
}
// MainWindow code
void MainWindow::OnPauseProcessRAII(bool pause_process)
{
qDebug() << "pause_process: " << pause_process;
}
void MainWindow::OnButtonSaveClicked()
{
PauseProcessRAII pauseProcessRAII(this);
connect(&pauseProcessRAII, &PauseProcessRAII::PauseProcess, this, &MainWindow::OnPauseProcess);
pauseProcessRAII.Execute();
// ... Some code runs
// ... pauseRAII desctructor is called
}
When I run the code both emits are firing as expected. My question is this a good solution? At first I though the emit call in PauseProcessRAII destructor wouldn't have worked because it may have destroyed the signal and slot connection. Of course that would mean I would have to add the connect to every function that I use it on.
The whole idea is fundamentally broken if your premise is that the code indicated by "Some code runs" runs long enough to block the GUI. If so: don't do that. The only code that is supposed to ever execute in the GUI thread is short, run-to-completion code that doesn't take long enough for the user to notice.
If the time taken by "Some code runs" is very short - on the order of single milliseconds at most - then you can certainly use such a construct. The term RAII doesn't apply here, as you're dealing with some sort of a scope guard.
If you wish, you can forgo the execute method and perform the connection in the constructor:
// https://github.com/KubaO/stackoverflown/tree/master/questions/scopeguard-signal-34910879
// main.cpp
#include <QtCore>
class ScopeSignaller : public QObject {
Q_OBJECT
public:
Q_SIGNAL void inScope(bool);
template <typename F>
ScopeSignaller(QObject * target, F && slot, QObject * parent = 0) : QObject(parent) {
connect(this, &ScopeSignaller::inScope, target, std::forward<F>(slot));
inScope(true);
}
~ScopeSignaller() {
inScope(false);
}
};
int main(int argc, char ** argv) {
QCoreApplication app{argc, argv};
ScopeSignaller s(&app, +[](bool b){ qDebug() << "signalled" << b; });
}
#include "main.moc"
due to requirements at my job, I had to revert by build from Qt5.3 to Qt4.8 and it all went well with a few function changes, except I got QMetaProperty::read: Unable to handle unregistered datatype errors all over. Has anyone else encountered this? I'm baffled, I have no idea where to start. As far as I know, Q_DECLARE_METATYPE is a valid macro in Qt4.8 and works the same way. I haven't changed any includes or anything.
The code below works identically in Qt 4 and Qt 5. I'm not sure what you're doing wrong - it's hard to imagine one cause from a vast array of snafus without seeing some code. "I just wrote a bunch of code in Qt 5 and it doesn't work in Qt 4" is a bit vague, you must admit.
#include <QtCore>
struct T {};
class C : public QObject {
Q_OBJECT
Q_PROPERTY(QList<T*> stuff READ stuff)
QList<T*> m_stuff;
public:
C(QObject * parent = 0) : QObject(parent) { m_stuff << (T*)10 << (T*)11; }
Q_SIGNAL void aSignal(const QList<T*>&);
Q_SLOT void aSlot(const QList<T*>& data) {
m_stuff = data;
}
QList<T*> stuff() const {
return m_stuff;
}
};
Q_DECLARE_METATYPE(QList<T*>)
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
qRegisterMetaType<QList<T*> >();
C c;
c.connect(&c, SIGNAL(aSignal(QList<T*>)), SLOT(aSlot(QList<T*>)), Qt::QueuedConnection);
// Test reading a property
QMetaProperty p = c.metaObject()->property(c.metaObject()->indexOfProperty("stuff"));
auto s = p.read(&c).value<QList<T*> >();
Q_ASSERT(s == c.stuff());
// Test queued signal-slot connection
auto const data = QList<T*>() << (T*)1 << (T*)2;
emit c.aSignal(data);
a.processEvents();
Q_ASSERT(data == c.stuff());
return 0;
}
#include "main.moc"
I have a very difficult time of understanding how to make a simplest possible working
multithreaded Qt console application.
I have read tons of stuff on how to use QThread class.
Some of them say subclass QThread, others say use worker class wrapper for QThread.
After several tries and retries I could still not make a working multithreaded
Qt console application.
Right now I don't need any fancy Qt Gui.
Can someone please help me to fill the threading parts of the example code ?
It just reads one line at the time from text file and the idea is that each thread (I want to use 4 threads) that is not busy at the moment will print that line to stdout with std::cout as soon as possible. Just print it, no other fancy processing stuff for now to keep this simple for me.
#include <QCoreApplication>
#include <QFile>
#include <iostream>
/* QThread stuff here */
/* Don't know how to make it */
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
/* Create four instances of threads here and
put them to wait readed lines */
QFile file("file.txt");
file.open(QIODevice::ReadOnly | QIODevice::Text);
while(!file.atEnd()) {
/* Read lines here but where should they be saved?
Into a global variable like QList<QByteArray> list ?
So that each thread can read them from there or where ???? */
??? = file.readLine();
}
file.close();
a.exit();
}
Putting the Functionality in a Slot in a QObject
The key points are:
Remember that each QObject has a certain thread() that it "lives" in. Each thread can have an event loop running there. This event loop will deliver the events sent to the objects that "live" in this thread.
Do not derive from QThread. Start stock QThreads. They'll start an even event loop in the default implementation of QThread::run().
Implement your functionality in a slot (or a Q_INVOKABLE) method. The class obviously has to derive from QObject.
The magic happens when you send signals (using signal-slot connection, not directly) to the slot in #3. The connection from the notifier, running in the GUI thread, to the notified objects is done automatically using the Qt::QueuedConnection since the sender and the receiver objects live in different threads.
Sending a signal to such an object results in posting an event to the event queue of the thread the object is in. The event loop's event dispatcher will pick those events and call the appropriate slots. This is the power of Qt - a lot of useful stuff can be done for you.
Note that there is no notion of a "currently busy" thread. The threads execute short slots of the objects that live there. If you want to move threads between a "busy" and "not busy" states, then you'd need extra code for that.
Another way of implementing it would be to derive from QRunnable and use QThreadPool. That's in another answer.
main.cpp
#include <QCoreApplication>
#include <QTextStream>
#include <QThread>
#include <QFile>
#include <cstdio>
class Notified : public QObject {
Q_OBJECT
QTextStream m_out;
public:
Q_SLOT void notify(const QString & text) {
m_out << "(" << this << ") " << text << endl;
}
Notified(QObject *parent = 0) : QObject(parent), m_out(stdout) {}
};
class Notifier : public QObject {
Q_OBJECT
Q_SIGNAL void notification(const QString &);
public:
Notifier(QObject *parent = 0) : QObject(parent) {}
void notifyLines(const QString & filePath) {
QFile file(filePath);
file.open(QIODevice::ReadOnly | QIODevice::Text);
while (! file.atEnd()) {
emit notification(file.readLine());
}
file.close();
}
};
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
QObjectList notifieds;
QList<QThread*> threads;
Notifier notifier;
for (int i = 0; i < 4; ++i) {
QThread * thread = new QThread(&a); // thread owned by the application object
Notified * notified = new Notified; // can't have an owner before it's moved to another thread
notified->moveToThread(thread);
thread->start();
notifieds << notified;
threads << thread;
notified->connect(¬ifier, SIGNAL(notification(QString)), SLOT(notify(QString)));
}
notifier.notifyLines("file.txt");
foreach (QThread *thread, threads) {
thread->quit();
thread->wait();
}
foreach (QObject *notified, notifieds) delete notified;
a.exit();
}
#include "main.moc"
For your purposes I would not use QThread at all but the classes from QtConcurrent.
Something simple like (assuming you have C++11):
while(!file.atEnd()) {
QString line = file.readLine();
QtConcurrent::run([line]
{
qDebug() << line;
});
}
Though I'm still not sure what this should give you on a high level.
Below link can be useful for you for the information related to using threads in Qt
http://mayaposch.wordpress.com/2011/11/01/how-to-really-truly-use-qthreads-the-full-explanation/
If you only want file reading to be done in Asynchronous ways Qt is having several alternate techniques like QtConcurrent.
http://qt-project.org/doc/qt-4.8/threads-qtconcurrent.html
Here is some example code to help you for using QtConcurrent
Running a Function in a Separate Thread
extern QString aFunction();
QFuture<void> future = QtConcurrent::run(aFunction);
aFunction should contain the code for reading the file .
You can return the read data in the following way
QFuture<QString> future = QtConcurrent::run(aFunction);
...
QString result = future.result();
Note that the QFuture::result() function blocks and waits for the result to become available. Use QFutureWatcher to get notification when the function has finished execution and the result is available.
Hope this helps. All the above code is taken from Qt documentation.
Putting the functionality in a QRunnable
Perhaps the solution that's closest to your explicit needs would use a QThreadPool. It does what you want: it picks a non-busy thread from its pool, and runs the worker there. If there are no free threads, it'll add the runnable to a run queue that gets drained each time a free thread becomes available.
Note, though, that your explicit wish of having a thread state, namely busy/non-busy, does not really mesh at all with a network penetration system that needs to wait for a reply before trying each new password. You'll want it based on QObjects. I'll modify my other answer to show how you might do it while managing network connections. It's very, very wasteful to waste threads on busy waiting for network answers. You do not want to do that. It will perform poorly.
Your application is I/O bound and could, pretty much, run on one thread without much in the way of undue performance loss. Only if you have a huge network pipe and are testing tens of thousands of accounts at the same time would you need more than one thread. I'm serious.
#include <QCoreApplication>
#include <QTextStream>
#include <QRunnable>
#include <QThreadPool>
#include <QFile>
#include <cstdio>
class Data : public QString {
public:
Data(const QString & str) : QString(str) {}
};
class Worker : public QRunnable {
QTextStream m_out;
Data m_data;
public:
void run() {
// Let's pretend we do something serious with our data here
m_out << "(" << this << ") " << m_data << endl;
}
Worker(const Data & data) : m_out(stdout), m_data(data) {}
};
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
QThreadPool * pool = QThreadPool::globalInstance();
QFile file("file.txt");
file.open(QIODevice::ReadOnly | QIODevice::Text);
while (! file.atEnd()) {
const Data data(file.readLine());
Worker * worker = new Worker(data);
pool->start(worker);
}
file.close();
pool->waitForDone();
}