Here's the problem: let's say we have a lot of threads (>1000) receiving data via some protocol and I want to write this data in database (sql server).
For database connection, we use QSqlDatabase. According to the documentation of QSqlDatabase:
A connection can only be used from within the thread that created it. Moving connections between threads or creating queries from a different thread is not supported.
Considering we cannot create 1000 connections (which can cause performance problems), what are the options here?
What comes to mind is to create a separate thread for database writings handling.
But how to run this thread in background and wait for signals from data receiving threads? What are the other approaches that can be used in this situation?
i used a database connection as a global logger, so you might be able to adapt it to fit your needs.
the skeleton looks like:
//Logger.h
class Logger : public QObject
{
Q_OBJECT
...
public:
void log(const QString& msg):
private:
LoggerWorker* w;
QThread t;
}
// Logger.cpp
Logger::Logger(QObject* p)
:QObject(p)
{
w = new LoggerWorker;
w->moveToThread(&t);
connect(&t, SIGNAL(started()), w, SLOT(init()));
t.start();
}
...
void
Logger::log(const QString& msg)
{
QMetaObject::invokeMethod(w, "log", Q_ARG(QString, msg));
}
// LoggerWorker.h
class LoggerWorker : public QObject
{
Q_OBJECT
...
public slots:
void init()
{
// create database and connect to it
}
void log(const QString& msg)
{
// insert to database
}
private:
QSqlDatabase db;
}
note: code above might contain nuts and bugs
cheers
The best option is to have a class for working with database which resides in separate thread.You can perform all the database related operations in relevant slots. Different threads can interact with object of the class through connecting their signals to it's slots. Since the emitter and receiver exist in different threads, you will have a queued connection type. So different signals that are emitted from different threads would be queued in the database class to be processed.
Related
I am trying to setup an infinite loop in another thread. The purpose is to download some data from an URL, send the data to the main thread and sleep for some seconds:
DataFetcher::DataFetcher(QUrl url, int fetchRateSec) :
url {url},
fetchRateSec {fetchRateSec}
{
}
void DataFetcher::run()
{
QNetworkAccessManager* manager = new QNetworkAccessManager();
QObject::connect(manager, &QNetworkAccessManager::finished, this, &DataFetcher::onReply);
while (true) {
QNetworkRequest req;
req.setUrl(url);
manager->get(req);
qDebug() << "run";
sleep(fetchRateSec);
}
}
void DataFetcher::onReply(QNetworkReply* reply)
{
qDebug() << "repl";
emit fetched(reply->readAll());
}
class DataFetcher : public QThread
{
Q_OBJECT
public:
DataFetcher(QUrl, int);
void run() override;
signals:
void fetched(QString);
private:
const QUrl url;
const int fetchRateSec;
private slots:
void onReply(QNetworkReply*);
};
But onReply is never called.
It is interesting because the qDebug in the while loop is executed as it should.
Im a bit nooby with QT so i might have missed something regarding how to connect the slots / signals but i think i got it right following some other examples.
It looks about right from the accepted answer in How do I write a Qt HTTP GET request?
what could be the problem here?
Per the discussion in the comments, it seems like the whole business of threading is not needed.
One of Qt's strongest attributes is that it is event-driven. You generally create a program by describing what you want to happen (slots) in response to certain events (signals). Explicitly waiting or sleeping is very rare in Qt-based applications (usually only testing), and is generally considered a no-no in event-driven development.
For your specific problem, a solution might look like this. You can create a QTimer in the main thread and connect its timeout signal to a function to make your HTTP request. You can then connect a slot to the QNetworkAccessManager::finished signal, which will run when your response completes. All of this can take place in the main thread, relying on the thread's event loop to manage the callbacks. No need to manage a separate thread yourself, and no looping, sleeping, blocking, or anything like that.
Disclaimer: I am relatively new to Qt and any type of programming that revolves around Threads and Networking. I have also adopted a lot of code from Qt Examples, API, and other online examples.
All code can be found on GitHub. This code is relatively as simple as it can get minus striping out GUI. I figure supplying it this way would help as well versus just pasting the code below.
I want to use and believe I need to use Threads as I need multiple clients send a request to the server, the server run some SQL code, then spit out the results back to the client (basically deriving a MySQL Server, but specific to what I am doing). Right now though, I am just working on learning the workings of it all.
With all that being said, as the Title states.. My client can connect to the server, the server sets up the thread, and will receive data (a String) through the readReady. After the data is read in, for right now I am just trying to echo it back to the client. It will do this, but only once. Then it spits out:
QObject: Cannot create children for a parent that is in a different thread.
(Parent is QNativeSocketEngine(0x266cca92ea0), parent's thread is serverThread(0x266cca9ed60), current thread is QThread(0x266cac772e0)
I cannot send any further data to the server unless I have the client reconnect, then after the data is sent, it will do its job but then spit out the same error and cease functioning. I have tried quite a bit of different things, but cannot seem to fix the issue. I even tried setting up a SIGNAL/SLOT for this as suggested in API:
It is important to remember that a QThread instance lives in the old thread that instantiated it, not in the new thread that calls run(). This means that all of QThread's queued slots will execute in the old thread. Thus, a developer who wishes to invoke slots in the new thread must use the worker-object approach; new slots should not be implemented directly into a subclassed QThread.
Anyway, any help would be greatly appreciated! My Code is below..
Server
ServerThread.cpp
// Project
#include "ServerDialog.h"
#include "ServerThread.h"
ServerThread::ServerThread(qintptr _socketDiscriptor, QObject *parent /*= 0*/)
: QThread(parent)
{
socketDiscriptor = _socketDiscriptor;
}
void ServerThread::run()
{
emit threadStarted(socketDiscriptor);
// Start Thread
clientSocket = new QTcpSocket;
// Set SocketDisc
if (!clientSocket->setSocketDescriptor(socketDiscriptor))
{
emit error(clientSocket->error());
return;
}
// Connect Socket and Signal
connect(clientSocket, SIGNAL(readyRead()), this, SLOT(readyRead()));
connect(clientSocket, SIGNAL(disconnected()), this, SLOT(disconnected()));
//// Loop Thread to Stay Alive for Signals and Slots
exec();
}
void ServerThread::readyRead()
{
QDataStream in(clientSocket);
in.setVersion(QDataStream::Qt_5_7);
in.startTransaction();
QString dataReceived;
in >> dataReceived;
if (!in.commitTransaction())
{
emit readyReadError(socketDiscriptor);
return;
}
emit readyReadMessage(socketDiscriptor, dataReceived);
echoData(dataReceived);
}
void ServerThread::disconnected()
{
emit threadStopped(socketDiscriptor);
clientSocket->disconnect();
clientSocket->deleteLater();
this->exit(0);
}
void ServerThread::echoData(QString &data)
{
QByteArray block;
QDataStream out(&block, QIODevice::WriteOnly);
out.setVersion(QDataStream::Qt_5_7);
out << data;
clientSocket->write(block);
}
So in ServerThread.cpp when echoData is called, that is when the error shows up and the Socket ceases functioning.
Any and all help will be appreciated. I know there are a few other posts regarding "Cannot create children for..." in regards to Threads. But I did not find any of them helpful. The one thing that I did find interesting but did not understand was maybe using moveToThread() but a lot of mixed comments on that.
I learn best through code examples along with explanation versus just an explanation or pointer to API. Thank you!
Most of Qt network functions are asynchronous; they do not block the calling thread. There is no need to mess up with threads if you are using QTcpSockets. In fact, creating a thread for every socket is an overkill, since that thread will spend most of its time just waiting for some network operation to finish. Here is how I would implement a single-threaded echo server in Qt:
#include <QtNetwork>
#include <QtCore>
//separate class for the protocol's implementation
class EchoSocket : public QTcpSocket{
Q_OBJECT
public:
explicit EchoSocket(QObject* parent=nullptr):QTcpSocket(parent){
connect(this, &EchoSocket::readyRead, this, &EchoSocket::EchoBack);
connect(this, &EchoSocket::disconnected, this, &EchoSocket::deleteLater);
}
~EchoSocket() = default;
Q_SLOT void EchoBack(){
QByteArray receivedByteArray= readAll();
write(receivedByteArray);
disconnectFromHost();
}
};
class EchoServer : public QTcpServer{
public:
explicit EchoServer(QObject* parent= nullptr):QTcpServer(parent){}
~EchoServer() = default;
//override incomingConnection() and nextPendingConnection()
//to make them deal with EchoSockets instead of QTcpSockets
void incomingConnection(qintptr socketDescriptor){
EchoSocket* socket= new EchoSocket(this);
socket->setSocketDescriptor(socketDescriptor);
addPendingConnection(qobject_cast<QTcpSocket*>(socket));
}
EchoSocket* nextPendingConnection(){
QTcpSocket* ts= QTcpServer::nextPendingConnection();
return qobject_cast<EchoSocket*>(ts);
}
};
int main(int argc, char* argv[]){
QCoreApplication a(argc, argv);
EchoServer echoServer;
echoServer.listen(QHostAddress::Any, 9999);
QObject::connect(&echoServer, &EchoServer::newConnection, [&](){
EchoSocket* socket= echoServer.nextPendingConnection();
qDebug() << "Got new connection from: " << socket->peerAddress().toString();
});
return a.exec();
}
#include "main.moc"
Notes:
This server has the ability to handle more than one client at the same time, since there is no blocking. The thread will just respond to the event that happens with the appropriate action; So, if that event was a new connection, it will create a new EchoSocket object to handle it and prints a statement out to qDebug(), and if that event was receiving something on a previously created socket, the same thread will echo received data back and close the connection. It will never block on a single connection waiting for data to arrive nor it will block waiting for a new connection to arrive.
Since you mention using some SQL queries in response for some connections later in your project. Please avoid threading since an SQL database connection in Qt can be used only from the thread that created it, see docs here. So, You'll have to either create a new database connection for each thread (and thus for each connection) in your application (and this is beyond just overkill), or switch later to a single threaded design.
In this section, I am explaining why threading does not work for you the way you are doing it:
You should not be declaring slots in your QThread subclass, Instead, use worker QObjects and move them to QThreads as needed.
The quote you have provided in your question is the exact explanation for why you get this warning. The ServerThread instance you create will be living in the main thread (or whatever thread that created it). Now let's consider this line from your code:
connect(clientSocket, SIGNAL(readyRead()), this, SLOT(readyRead()));
The signal readyRead() will be emitted from the current ServerThread instance (since the clientSocket object that emits it lives there), However, the receiver object is the current ServerThread instance, But that lives in the main thread. Here is what the documentation says:
If the receiver lives in the thread that emits the signal, Qt::DirectConnection is used. Otherwise, Qt::QueuedConnection is used.
Now, the main point of Qt::QueuedConnection is executing the slot in the receiver object's thread. This means that, your slots ServerThread::readyRead() and ServerThread::disconnected will get executed in the main thread. This is most likely not what you meant to do, since you'll end up accessing clientSocket from the main thread. After that, any call on clientSocket that results in child QObjects being created will result in the warning you get (you can see that QTcpSocket::write() does this here).
Mixed comments of movetothread are linked mostly to usage of it to move thread object to itself.
The quote hints that the members of QThread aren't designed to be called from worker. Strictly proper way to call signal would be by using worker object model, that was shown in Qt examples and explained a few times on QT-related blogs:
class Worker : public QObject
{
Q_OBJECT
private slots:
void onTimeout()
{
qDebug()<<"Worker::onTimeout get called from?: "<<QThread::currentThreadId();
}
};
class Thread : public QThread
{
Q_OBJECT
private:
void run()
{
qDebug()<<"From work thread: "<<currentThreadId();
QTimer timer;
Worker worker;
connect(&timer, SIGNAL(timeout()), &worker, SLOT(onTimeout()));
timer.start(1000);
exec();
}
};
worker constructed inside run() is "property" of the thread it created, so figuratively speaking, it is slaved to its context. The same effect maybe achieved if you create worker in other thread, then move it to this thread before connection was made. When you connect signal to slot of the QThread itself, you connect child thread to thread it was created by.
Use of
connect(&timer, SIGNAL(timeout()), this, SLOT(onTimeout()), Qt::DirectConnection);
or creating connection from your thread sometimes seems to achieve proper result, but not in this case, where you try use objects constructed in different threads together. Calling moveToThread(this) in constructor is a thing not recommended to do.
I'm running a C++ program, build with Qt, that never can stop.
The program always fetches data from database and if there is a result sends an SMS.
I'm able to connect to database, but after some hours (+/- 10), it doesn't work anymore.
I don't know if the problem is because I lose connection with database or because my computer goes standby...
I'm not able in Qt to see database status: db.open() always returns true when tested inside while loop.
QSqlDatabase db = QSqlDatabase::addDatabase("QMYSQL");
db.setHostName("");
db.setPort();
db.setDatabaseName("");
db.setUserName("");
db.setPassword("");
if (db.open())
{
while (true)
{
// MySQL Request
// If data -> send SMS
}
}
There's always the possibility to loose a DB connection for whatever reason. You just can't rely on it. You have to check your connection inside the loop and implement some kind of re-connection scheme if the connection gets lost. As far as I know Qt doesn't do that for you.
Qt provides an event driven framework; events occur and the program reacts to those events.
When you have a never ending loop, events are queued, waiting until they can be processed. In your case, this is never going to happen, so the queue of events will just keep increasing, taking up resources such as memory.
There are two possible ways of solving this. The first is to call QApplication::processEvents every now and again in your loop.
However, the better method would be to remove the while(true) and instead use a QTimer which will periodically call a function to process any available data.
Assuming you have a class, derived from QObject, here's skeleton code using QObject's own timer
class MyObject : public QObject
{
Q_OBJECT
public:
MyObject(QObject *parent = 0);
protected:
// this will be called periodically from the timer
void timerEvent(QTimerEvent *event);
private:
m_timerId = 0; // C++ 11 initialisation
};
MyObject::MyObject(QObject *parent)
: QObject(parent)
{
m_timerId = startTimer(50); // 50-millisecond timer
}
void MyObject::timerEvent(QTimerEvent *event)
{
if(event->timerId() == m_timerId)
{
// MySQL Request
// If data -> send SMS
}
}
I have a multithreaded Qt5 app written in C++ where each thread is writing to the same database.
I share the same database QSqlDatabase variable across threads, but each thread creates its own query. I am regularly getting crashes and I've captured this info at the time of a crash:
Driver error [QMYSQL3: Unable to store statement results]
Database error [Commands out of sync; you can't run this command now]
Type [2]
Number [2014]
First of all, can I make simultaneous MySQL queries in multiple threads? If so, do I need to protect SQL calls with a mutex? Or do I need to post more info....
UPDATE: I have a seperate DBManager thread which handles opening/closing the DB, and simple database writes. I did this because my DB often goes offline, and I don't want other threads to hang for up to 2 minutes while a db open fails. No I have more threads which do reports, and must retrieve substantial amounts of data from the db.
As noted below, sharing the db handle across threads is not permitted. So now perhaps this is more of a design question - what is the best way to handle this situation? I don't want each thread that does DB access to attempt it's own open and wait for 2 minutes in case the DB is offline
Read the documentation for the SQL module, in particular the threads section that states each connection can only be used within the thread that created it.
So adding a mutex is not good enough, you will have to either create a new connection object in each thread, or pass the required data to/from a dedicated thread that performs all DB queries.
I generally handle this problem by providing a factory method on my database manager class, something that looks similar to this (semi-pseudocode):
static QHash<QThread, QSqlDatabase> DatabaseManager::s_instances;
static QMutex DatabaseManager::s_databaseMutex;
QSqlDatabase DatabaseManager::database() {
QMutexLocker locker(s_databaseMutex);
QThread *thread = QThread::currentThread();
// if we have a connection for this thread, return it
if (s_instances.contains(thread))
return s_instances[thread];
}
// otherwise, create a new connection for this thread
QSqlDatabase connection =
QSqlDatabase::cloneDatabase(existingConnection, uniqueNameForNewInstanceOnThisThread);
// open the database connection
// initialize the database connection
s_instances.insert(thread, connection);
return connection;
}
As long as you make sure you create an initial connection (you can modify the factory to include that, or just create the initial connection in your DatabaseManager constructor), you should be able to replace most existing cases where you're using a QSqlDatabase with:
QSqlQuery query(DatabaseManager::database());
query.exec(<some sql>);
without worrying about which thread you're calling from. Hope that helps!
This is just mbroadst's excellent answer with minor changes. This code should be complete and directly compilable:
database_manager.h:
#ifndef DATABASEMANAGER_H
#define DATABASEMANAGER_H
#include <QMutex>
#include <QHash>
#include <QSqlDatabase>
class QThread;
class DatabaseManager
{
public:
static QSqlDatabase database(const QString& connectionName = QLatin1String(QSqlDatabase::defaultConnection));
private:
static QMutex s_databaseMutex;
static QHash<QThread*, QSqlDatabase> s_instances;
};
#endif // DATABASEMANAGER_H
database_manager.cpp:
#include "database_manager.h"
#include <QSqlDatabase>
#include <QMutexLocker>
#include <QThread>
#include <stdexcept>
QMutex DatabaseManager::s_databaseMutex;
QHash<QThread*, QHash<QString, QSqlDatabase>> DatabaseManager::s_instances;
QSqlDatabase DatabaseManager::database(const QString& connectionName)
{
QMutexLocker locker(&s_databaseMutex);
QThread *thread = QThread::currentThread();
// if we have a connection for this thread, return it
auto it_thread = s_instances.find(thread);
if (it_thread != s_instances.end()) {
auto it_conn = it_thread.value().find(connectionName);
if (it_conn != it_thread.value().end()) {
return it_conn.value();
}
}
// otherwise, create a new connection for this thread
QSqlDatabase connection = QSqlDatabase::cloneDatabase(
QSqlDatabase::database(connectionName),
QString("%1_%2").arg(connectionName).arg((int)thread));
// open the database connection
// initialize the database connection
if (!connection.open()) {
throw std::runtime_error("Unable to open the new database connection.");
}
s_instances[thread][connectionName] = connection;
return connection;
}
Usage: Instead of QSqlDatabase::database(), use DatabaseManager::database()
Edited 2016-07-01: fixed bug with multiple connection names
I'm trying to create a periodic callback using QT. I'm thinking, something like OpenGL's periodic callback.
Basically, I have a micro controller which is waiting for an event to happen. When the event Happens it sends the time over the serial port to the PC. On the PC side, I want the GUI to update whenever this time information is received, or to periodically check to see if there is anything in the Rx buffer.
Is there any built in functionality of Qt to periodically call a methood?
If not is there a way to create a signal between QextSerial, and The GUI?
Or, will I have to make a multi-threaded app with an observer/observable relationship between the serialPort, and the GUI?.
Have a look at QTimer. They also list an example of an analog clock
For my specific needs I ended up using a QextSerialPort.
I attached a signal to the Port so that whenever a message was received it would add the time to the Model, and update the GUI.
Port = new QextSerialPort();
//... various Port Setup
connect(Port, SIGNAL(readyRead()), this, SLOT(onReadyRead()));
This way I was able to keep the Data Model and GUI more separated.
You could try using QSocketNotifier.
I've used it to read and process standard input. You just need to change your the file you read from (in my case STDIN_FILENO)
---hpp
class QSocketNotifier;
class ConsoleReader : public QObject
{
Q_OBJECT
public:
explicit ConsoleReader(QObject *parent = 0);
Q_SIGNALS:
void textReceived(QString message);
public Q_SLOTS:
void text();
private:
QSocketNotifier* notifier;
};
---cpp
#include <QSocketNotifier>
#include <unistd.h> //Provides STDIN_FILENO
ConsoleReader::ConsoleReader(QObject *parent)
: QObject(parent)
{
notifier = new QSocketNotifier(STDIN_FILENO, QSocketNotifier::Read);
connect(notifier, SIGNAL(activated(int)), this, SLOT(text()));
}
void ConsoleReader::text()
{
QTextStream qin(stdin);
QString line = qin.readLine();
Q_EMIT textReceived(line);
}