so, I'm trying to run some simple code in Qt to return the contents of a given web page. After doing quick research, I was able to develop my own class to simplify the process:
WebFetch::WebFetch()
{
nam = new QNetworkAccessManager(this);
connect(nam, SIGNAL(finished(QNetworkReply*)), this, SLOT(finished(QNetworkReply*)));
}
QString WebFetch::get(QString url)
{
nam->get(QNetworkRequest(QUrl(url)));
}
void WebFetch::finished(QNetworkReply* reply)
{
QByteArray data = reply->readAll();
QString str(data);
}
However, there big problem that I'm finding with the above code is that the call is asynchronous. I would like the "get" function to simply return the string after it is retrieved, which seems impossible on the account that it needs to wait for the finished signal, at which point there's no way of having "get" return whatever content is retrieved by the "finished" slot. Is there any alternative to the above method or is there a way I can get "get" to return the content retrieved by "finished"? Any help would be greatly appreciated. Thanks!
The call being asynchronous is not a problem - it's a big win. With a synchronous call, you're essentially wasting potentially hundreds ok KB of RAM, and an entire thread, just idly waiting for something to come back. You can't write such code while pretending that things happen synchronously or even "quickly" for that matter. I won't even comment on the insanity of running such synchronous code in the GUI thread. It's also a very bad idea to run a local event loop, since suddenly all of your GUI code becomes reentrant. My bet is that you neither design nor test for that.
You have to break down whatever code is expecting the result into two parts: the first part needs to place the request. The second part, in a slot, is notified when the request is finished and continues doing whatever is to be done.
If you wish to have it all in a single method, use C++11:
QNetworkAccessManager * mgr = ...;
QObject::connect(mgr, &QNetworkAccessManager::finished,
[this, mgr](QNetworkReply * reply){
// here you can do things with the reply
});
mgr->get(QNetworkRequest("....");
For a complete example, see this 300-line photographic mosaic generator that pulls random images from imgur. It extensively uses asynchronous, multithreaded processing and lambdas in the above style.
Related
I have this code:
QNetworkAccessManager man;
QNetworkRequest req(QUrl("URL"));
QString ua("HttpRequestDemo/0.1 (Win64) Qt/5.14.0");
req.setHeader(QNetworkRequest::UserAgentHeader, QVariant(ua));
QNetworkReply* reply = man.get(req);
QObject::connect(reply){
QByteArray read = reply->readLine();
QFile out("file.txt");
out.open(QIODevice::WriteOnly|QIODevice::Text);
out.write(read);
out.close();
})
This works on the main.cpp file, using the QCoreApplication, but I want to use the QApplication and download a specific data while pressing a button.
I put the same code on the on_pushButton_clicked() in the mainwindow.cpp file and it didn't even generate the file from the url.
The problem is that man and req go out of scope and are destroyed as soon as your on_pushButton_clicked() function returns, at which point the request probably hasn't even been sent yet.
You need to make sure that these objects outlive the current scope, either by making them members of the window class, or by allocating them on the heap and setting some QObject (maybe also the window class) as the parent.
The problem is that if you put the same code in a method like X you make QNetworkAccessManager a local variable that will be removed instantly that the connection is asynchronous. The solution is to make QNetworkAccessManager an attribute of the class.
*.h
private:
QNetworkAccessManager man;
*.cpp
void Klass::on_pushButton_clicked(){
QNetworkRequest req(QUrl("URL"));
QString ua("HttpRequestDemo/0.1 (Win64) Qt/5.14.2");
req.setHeader(QNetworkRequest::UserAgentHeader, QVariant(ua));
QNetworkReply* reply = man.get(req);
connect(reply, &QNetworkReply::finished, [&]() {
QByteArray read = reply->readAll();
QFile out("file.txt");
out.open(QIODevice::WriteOnly|QIODevice::Text);
out.write(read);
out.close();
reply->close();
reply->deleteLater();
})
}
If you are planning on potentially queuing very many downloads, I strongly recommend using libcurl in your Qt app. I was using QNetworkAccessManager to down 100+ financial quote files, and it would fail downloading ~ 1/3 of the time, and take a while to download. I switched to libcurl, and after figuring out how to get my crypto root certificates setup for https, it runs much faster, and almost never fails. I run it as a dll.
And yes, you will need to make sure the network manager, whether QNetworkManager or curl, doesn't go out of scope upon exiting the button handler. A more conventional pattern, although not necessarily better, is to either have a pointer to e.g. QNetworkManager in your parent class, and new it, or use a std::unique_ptr and std::make_unique (purportedly safer). Creating large objects on the stack can cause problems (in the old days, dare I say, stack overflows), and so is usually done on the heap. In this case, it's not very big, so it doesn't really matter. Alternatively, a form creating big objects might itself be created on the heap.
I am currently working on an editor program; there's a feature I need to write, which requires loading several files in a row using the project's asynchronous file API, then performing some more computations once those files are loaded.
In another language, this would probably be implemented with an async/await workflow, eg:
let firstFile = await FileAPI.loadFile("Foo.xxx", ...);
let otherFile = await FileAPI.loadFile("Bar/Foobar.xxx", ...);
The Qt equivalent to this code would be to spawn a new thread using QtConcurrent::run, returning a QFuture, and waiting for that future to yield a result.
However, in the project I work on, the file-opening API runs on a single worker thread, which means I can't use QtConcurrent::run. This is an established, non-negotiable part of the codebase. Eg the constructor of the file API looks like:
FileApiWorker* worker = new FileApiWorker();
m_workerThread = new QThread();
worker->moveToThread( m_workerThread );
// Input signals
connect( this, &FileApi::loadFile, worker, &FileApiWorker::loadFile);
connect( this, &FileApi::loadData, worker, &FileApiWorker::loadData);
connect( this, &FileApi::loadDir, worker, &FileApiWorker::loadDir);
Which means my only way of accessing filesystem data is to call a method which emits a signal, which starts the computation on another thread, which eventually emits its own signal at the end to pass on the loaded data.
This is extremely impractical for the use case above, because instead of saying "do thing, load data, wait, keep doing things", I essentially need to say "do thing, load data (with call back 'keep doing things')" and "keep doing things" in another function, which introduces all sorts of brittleness in the code. (and, well, you know, that's exactly the sort of workflow we invented futures for)
Is there some way I could create a QFuture, or some future-equivalent object (that can be awaited inside a method) from the loadFile method, given that loadFile always runs on the same worker thread and I am not allowed to create new threads?
The simplest way to create a QFuture in Qt is with the undocumented QFutureInterface class.
Example code:
Q_DECLARE_METATYPE( QFutureInterface<FileData> );
// ...
qRegisterMetaType<QFutureInterface<FileData>>();
FileApiWorker* worker = new FileApiWorker();
connect( this, &FileApi::loadFile_signal, worker, &FileApiWorker::loadFile_signal);
// ...
QFuture<FileData> FileApi::loadFile()
{
QFutureInterface<FileData> futureInterface;
// IMPORTANT: This line is necessary to be able to wait for the future.
futureInterface.reportStarted();
emit loadFile_signal(futureInterface);
return futureInterface.future();
}
FileApiWorker::loadFile_signal(QFutureInterface<FileData>& futureInterface)
{
// Do some things
// ...
futureInterface.reportResult(...);
// IMPORTANT: Without this line, future.waitForFinished() never returns.
futureInterface.reportFinished();
}
Some factors to account for:
The above code uses Q_DECLARE_METATYPE; which is necessary to be able to pass QFutureInterface through a cross-threads signal. To be precise, the connect line will fail to compile if Q_DECLARE_METATYPE isn't included; and the emit loadFile_signal line will fail at runtime if qRegisterMetaType isn't called. See the Qt documentation on metatypes for details.
You can propagate errors, in such a way that calling loadFile().waitForFinished() throws on error. To achieve this, you need to create a special-purpose class inheriting QException, then call:
futureInterface.reportException( MyException(...) );
futureInterface.reportFinished();
in your error path.
QException is essentially a wrapper for actual exceptions that need to be transferred between threads. See the documentation for details.
While QFutureInterface is stable, and mostly has the same API as QFuture and QFutureWatcher, it's still an undocumented feature, which may surprise contributors coming across it in a shared codebase. The class can be counter-intuitive, and fail silently if you don't respect the points above (which I had to learn through trial and error). This must be stressed in the comments of any shared code using QFutureInterface. The class's source code can be found here.
IMO, it is strange not to use ready-to-use solutions (AsyncFuture) and try to rewrite from scratch.
But I can suggest my own "wheel": lambda as a slot.
void FileApi::awaitLoadFile()
{
qDebug() << "\"await\" thread is" << thread();
emit loadFile("Foo.xxx");
static bool once = connect(m_worker, &FileApiWorker::loadFileDone, this, // there is possible to avoid the third "this" parameter, but it is important to specify the lifetime of the connection and the receiver context while using lambdas
[=](QByteArray result)
{
qDebug() << "\"comeback-in-place\" thread is" << thread(); // will be the same as "await" thread was
// do what you need with your result
},
Qt::QueuedConnection // do not forget
);
qDebug() << "here is an immediate return from the \"await\" slot";
}
Useful arcticle New Signal Slot Syntax - Qt Wiki
I need to update the content of a field on my QWidget via a JSON file (updated in real time). I've read about functions readLine() and readAll() of QFile, but when I try a loop like :
while(true):
jsfile.readLine()
creation of objects, update of values, display etc ...
I lost the focus on my window. But I want to keep the control of the application with my buttons and obviously to watch the evolution of the JSON values.
I have thought that Qt manages itself the events and keeps the focus on the current window, but like I've said, it's not the case.
Is there a good solution (multi threads maybe) to use my window while the application reads the file (with new informations in real time)?
(With the constraint "real time" I can't read the whole file every time and I've no choice about the format of this file)
Update
I tried the thread method.
So, I choose to create my thread instance into the main (with my main window) and connect here. But, when I run the program, I've this error :
no matching member function for call to 'connect'
Reader reader;
QObject::connect(controler, SIGNAL(ready()),
reader, SLOT(received()));
According to this error, I've thought that the reason was main don't inherits of Object, and so, I've move the connection ans the creation of thread instance into my main window.
Reader reader;
QObject::connect(reader, SIGNAL(newobject(QJsonObject)),
this, SLOT(displayJSON(QJsonObject)));
With this one, I've the same error while I've already connect lot of widget into this class without any error.
What can be the problem ?
Update 2
I've a solution when I give as argument my main window (controler) in reader's constructor and connect into this one but, if possible, I would an explanation for the previous problem.
The current problem that I have is that signals are emit well but slots are executed after the end the application (so after the end of the thread's execution and not during)
This isn't really the subject of this topic so we can close this one.
You can use QThread (Qt documentation: QThread) class to create a thread, which will read your file. The main thread will execute your GUI application and it will be available during file reading.
You can find a simple example in documentation for creating your thread:
class WorkerThread : public QThread
{
Q_OBJECT
void run() Q_DECL_OVERRIDE {
QString result;
/* ... here is the expensive or blocking operation ... */
emit resultReady(result);
}
signals:
void resultReady(const QString &s);
};
void MyObject::startWorkInAThread()
{
WorkerThread *workerThread = new WorkerThread(this);
connect(workerThread, &WorkerThread::resultReady, this, &MyObject::handleResults);
connect(workerThread, &WorkerThread::finished, workerThread, &QObject::deleteLater);
workerThread->start();
}
You can modify this example for your purpose. For example, WorkerThread for your task may be something like this:
class WorkerThread : public QThread
{
Q_OBJECT
void run() Q_DECL_OVERRIDE {
while(!stopFlag)
{
// read JSON file to QByteArray. Use QFile and QTextStream
// use QJsonDocument to read JSON content
// find what is new in JSON
emit signalSomethingNew(/*parameters*/);
QThread::currentThread()->msleep(/*timeout*/);
}
}
signals:
void signalSomethingNew(/*parameters*/);
};
At the end you must implement slot on your QWidget for signalSomethingNew(/*parameters*/) and make connection:
connect(yourThread, &WorkingThread::signalSomethingNew, youWidget, &YouWidget::yourSlot);
For working with JSON data: QJsonDocument
I'm interpreting your question as "my application is unresponsive whilst doing work" rather than "my focus jumped to another window" - please comment if you meant something different.
You have a choice of options:
Create and run a background QThread to do the work. Have it emit signals (connected to your widgets using Qt::QueuedConnection - the default) when it has results to display.
This is a good solution when the worker has a lot of computation to do, or needs all the input to be read before it can start. It works very well when the target system has processors available with no other work to do.
Use a QSocketNotifier to signal your GUI thread when some of the input becomes available (note that the name is misleading - it actually works on all kinds of file descriptor, not just sockets).
This is appropriate when the algorithm is simple and incremental - i.e. if a small chunk of input can be read and processed quickly.
Incorporate periodic calls to processEvents() in your algorithm:
auto *const dispatcher = QThread::currentThread()->eventDispatcher;
while (line = json.readLine()) {
doSomethingWith(line);
if (dispatcher)
dispatcher->processEvents();
}
This won't work unless you can modify the algorithm like this - if the loop is in somebody else's (closed) code, then you'll need one of the other solutions.
When using QTcpSocket to receive data, the signal to use is readyRead(), which signals that new data is available.
However, when you are in the corresponding slot implementation to read the data, no additional readyRead() will be emitted.
This may make sense, as you are already in the function, where you are reading all the data that is available.
Problem description
However assume the following implementation of this slot:
void readSocketData()
{
datacounter += socket->readAll().length();
qDebug() << datacounter;
}
What if some data arrives after calling readAll() but before leaving the slot?
What if this was the last data packet sent by the other application (or at least the last one for some time)?
No additional signal will be emitted, so you have to make sure to read all the data yourself.
One way to minimize the problem (but not avoid it totally)
Of course we can modify the slot like this:
void readSocketData()
{
while(socket->bytesAvailable())
datacounter += socket->readAll().length();
qDebug() << datacounter;
}
However, we haven't solved the problem. It is still possible that data arrives just after the socket->bytesAvailable()-check (and even placing the/another check at the absolute end of the function doesn't solve this).
Making sure to be able to reproduce the problem
As this problem of course happens very rarely, I stick to the first implementation of the slot, and I'll even add a an artificial timeout, to be sure that the problem occurs:
void readSocketData()
{
datacounter += socket->readAll().length();
qDebug() << datacounter;
// wait, to make sure that some data arrived
QEventLoop loop;
QTimer::singleShot(1000, &loop, SLOT(quit()));
loop.exec();
}
I then let another application send 100,000 bytes of data.
This is what happens:
new connection!
32768 (or 16K or 48K)
The first part of the message is read, but the end isn't read anymore, as readyRead() won't be called again.
My question is: what is the best way to be sure, this problem never occurs?
Possible solution
One solution I came up with is calling the same slot again at the end again, and to check at the beginning of the slot, if there is any more data to read:
void readSocketData(bool selfCall) // default parameter selfCall=false in .h
{
if (selfCall && !socket->bytesAvailable())
return;
datacounter += socket->readAll().length();
qDebug() << datacounter;
QEventLoop loop;
QTimer::singleShot(1000, &loop, SLOT(quit()));
loop.exec();
QTimer::singleShot(0, this, SLOT(readSocketDataSelfCall()));
}
void readSocketDataSelfCall()
{
readSocketData(true);
}
As I don't call the slot directly, but use QTimer::singleShot(), I assume that the QTcpSocket can't know that I'm calling the slot again, so the problem that readyRead() isn't emitted can't happen anymore.
The reason why I have included the parameter bool selfCall is that the slot which is called by the QTcpSocket isn't allowed to exit sooner, else the same problem can occur again, that data arrives exactly at the wrong moment and readyRead() isn't emitted.
Is this really the best solution to solve my problem?
Is the existence of this problem a design error in Qt or am I missing something?
Short answer
The documentation of QIODevice::readyRead() states:
readyRead() is not emitted recursively; if you reenter the event loop or call waitForReadyRead() inside a slot connected to the readyRead() signal, the signal will not be reemitted.
Thus, make sure that you
don't instantiate a QEventLoop inside your slot,
don't call QApplication::processEvents() inside your slot,
don't call QIODevice::waitForReadyRead() inside your slot,
don't use the same QTcpSocket instance within different threads.
Now you should always receive all data sent by the other side.
Background
The readyRead() signal is emitted by QAbstractSocketPrivate::emitReadyRead() as follows:
// Only emit readyRead() when not recursing.
if (!emittedReadyRead && channel == currentReadChannel) {
QScopedValueRollback<bool> r(emittedReadyRead);
emittedReadyRead = true;
emit q->readyRead();
}
The emittedReadyRead variable is rolled back to false as soon as the if block goes out of scope (done by the QScopedValueRollback). So the only chance to miss a readyRead() signal is when the control flow reaches the if condition again before the processing of the last readyRead() signal has finished (in other words, when there would be a recursion).
And a recursion should only be possible in the situations listed above.
I think scenario mentioned in this topic has two major cases which works differently, but in general QT doesn't have this problem at all and I will try to explain below why.
First case: Single threaded application.
Qt uses select() system call to poll open file descriptor for any change happened or operations available. Simple saying on every loop Qt checks if any of opened file descriptors have data available to read/closed etc. So on single threaded application flow looks like that (code part simplified)
int mainLoop(...) {
select(...);
foreach( descriptor which has new data available ) {
find appropriate handler
emit readyRead;
}
}
void slotReadyRead() {
some code;
}
So what will happend if new data arrived while program still inside slotReadyRead.. honestly nothing special. OS will buffer data, and as soon as control will return to next execute of select() OS will notify software that there are data available for particular file handle. It works in absolutely the same way for TCP sockets/files etc.
I can imaging situations where (in case of really long delays in slotReadyRead and a lot of data coming) you can experience an overrun within OS FIFO buffers (for example for serial ports) but that has more to do with a bad software design rather then QT or OS problems.
You should look on slots like readyRead like on a interrupt handlers and keep their logic only within fetch functionality which fills your internals buffers while processing should be done in separate threads or while application on idle etc.. Reason is that any such application in general is a mass service system and if it spends more time on serving one request then a time interval between two requests it's queue will overrun anyway.
Second scenario: multithreaded application
Actually this scenario is not that much differ from 1) expect that you should design right what happens in each of your threads. If you keep main loop with light wighted 'pseudo interrupt handlers' you will be absolutely fine and keep processing logic in other threads, but this logic should work with your own prefetch buffers rather then with QIODevice.
The problem is quite interesting.
In my program the usage of QTcpSocket is very intensive. So I've written the whole library, that breaks outgoing data into packages with a header, data identifier, package index number and maximum size, and when the next piece of data comes, I know exactly where it belongs to. Even if I miss something, when the next readyRead comes, the receiver reads all and compose received data correctly. If the communication between your programs is not so intense, you could do the same, but with timer (which is not very fast, but solves the problem.)
About your solution. I don't think it's better then this:
void readSocketData()
{
while(socket->bytesAvailable())
{
datacounter += socket->readAll().length();
qDebug() << datacounter;
QEventLoop loop;
QTimer::singleShot(1000, &loop, SLOT(quit()));
loop.exec();
}
}
The problem of both methods is the code right after leaving the slot, but before returning from emitting the signal.
Also you could connect with Qt::QueuedConnection.
Here are some examples of ways to get the whole file, but using some other parts of the QNetwork API:
http://qt-project.org/doc/qt-4.8/network-downloadmanager.html
http://qt-project.org/doc/qt-4.8/network-download.html
These examples show a stronger way to handle the TCP data, and when buffers are full, and better error handling with a higher level api.
If you still want to use the lower level api, here is a post with a great way to handle the buffers:
Inside your readSocketData() do something like this:
if (bytesAvailable() < 256)
return;
QByteArray data = read(256);
http://www.qtcentre.org/threads/11494-QTcpSocket-readyRead-and-buffer-size
EDIT: Additional examples of how to interact with QTCPSockets:
http://qt-project.org/doc/qt-4.8/network-fortuneserver.html
http://qt-project.org/doc/qt-4.8/network-fortuneclient.html
http://qt-project.org/doc/qt-4.8/network-blockingfortuneclient.html
Hope that helps.
If a QProgressDialog shall be shown while receiving data from a socket it only works if any QApplication::processEvents() are sent (e.g. by the QProgessDialog::setValue(int) methode). This of course leads to the loss of readyRead signals as mentioned above.
So my workaround was a while loop including the processEvents command such as:
void slot_readSocketData() {
while (m_pSocket->bytesAvailable()) {
m_sReceived.append(m_pSocket->readAll());
m_pProgessDialog->setValue(++m_iCnt);
}//while
}//slot_readSocketData
If the slot is called once any additional readyRead signals can be ignored because the bytesAvailable() always returns the actual number after the processEvents call. Only on pausing of the stream the while loop ends. But then the next readReady is not missed and starts it again.
I had the same problem right away with the readyRead slot. I disagree with the accepted answer; it doesn't solve the problem. Using bytesAvailable as Amartel described was the only reliable solution I found. Qt::QueuedConnection had no effect. In the following example, I'm deserializing a custom type, so it's easy to predict a minimum byte size. It never misses data.
void MyFunExample::readyRead()
{
bool done = false;
while (!done)
{
in_.startTransaction();
DataLinkListStruct st;
in_ >> st;
if (!in_.commitTransaction())
qDebug() << "Failed to commit transaction.";
switch (st.type)
{
case DataLinkXmitType::Matrix:
for ( int i=0;i<st.numLists;++i)
{
for ( auto it=st.data[i].begin();it!=st.data[i].end();++it )
{
qDebug() << (*it).toString();
}
}
break;
case DataLinkXmitType::SingleValue:
qDebug() << st.value.toString();
break;
case DataLinkXmitType::Map:
for (auto it=st.mapData.begin();it!=st.mapData.end();++it)
{
qDebug() << it.key() << " == " << it.value().toString();
}
break;
}
if ( client_->QIODevice::bytesAvailable() < sizeof(DataLinkListStruct) )
done = true;
}
}
i got the same problem, rather use signal readyRead() and socket.readall, i' m trying the following, just after connect without be sure :
QByteArray RBuff;
if(m_socket->waitForConnected(3000))
{
while (m_socket->ConnectedState == QAbstractSocket::ConnectedState) {
RBuff = m_socket->read(2048);
SocketRead.append(RBuff);
if (!SocketRead.isEmpty() && SocketRead.length() == 2048)
{
readData(SocketRead);
SocketRead.remove(0,2048);
}
QCoreApplication::processEvents(QEventLoop::AllEvents, 100);
}
//m_socket->close();*/
}
else
{
This is my first post here, and I'm fairly new to Qt.
I am using Qwebkit in order to load a web page, and I'm interested in NOT fully load some resources from web. To be specific, I'm trying to get only the size of jpg files and not the image data from within the jpg's binary data (not HTML tags). For doing so, I have re-implemented the createRequest method of QNAM to do as follow:
QNetworkReply *NetworkAccessManager::createRequest(Operation op,const QNetworkRequest & req,QIODevice * outgoingData )
{
if (req.url().path().endsWith("jpg"))
{
CustomReply *reply = new CustomReply(QNetworkAccessManager::createRequest(op, req, outgoingData));
return reply->getQNR();
}else{
return QNetworkAccessManager::createRequest(op, req, outgoingData);
}
}
Then I connect some signal in my CustomReply class to append the coming data into a QByteArray, then I process the QByteArray to see if I have the marker I'm looking for. Now here I don't know how to proceed. What I want to do after this is closing the connection (to not download more) and passing the reply with the data I have received through CustomReply::getQNR(). I need to implement a function to set the content of my reply to the QByteArray I stored, and I have read this and that but couldn't solve my problem.
Thank you in advance.
I'm not 100% clear on your question but if you're trying to return the value of the reply you received to another QByteArray just set up an additional signal which is either fired on completion or called with emit and pass the value across that way.
I did as described here and solved my problem. I was missing the offset, so every time readData() was called, the data was read from the beginning. In order to close the connection I connect the finish signal of CustomReply into original QNReply. When I get enough data, I emit finish signal to close the connection. Calling abort or close will result in ERROR 5 (although you can handle the error but I find working with signal a bit cooler).
Thank you everybody.