I'm writing an application that scrapes some web pages using QWebPage. I'm having some trouble when the response is a Http redirect (e.g 302, 303, etc). The QWebPage simply does not follow the redirect.
To work around this issue I've connected to the page's network manager's finished signal to capture the status of the response and load any redirect, however, when I call the load method for the second time on the QWebPage, it justs sets the url to blank and doesn't issue any request whatsoever.
Here are some relevant bits of code:
connect(page->networkAccessManager(), SIGNAL(finished(QNetworkReply*)), SLOT(gotReply(QNetworkReply*)));
connect(page, SIGNAL(loadFinished(bool)), SLOT(doneLoading(bool)));
page->mainFrame()->load(url);
My slot:
void Snapshot::gotReply(QNetworkReply *reply)
{
if(reply->header(QNetworkRequest::ContentTypeHeader).toString().contains(QString("text/html")))
{
qDebug() << "Got reply " + reply->url().toString() + " - " + reply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toString() + " - " + reply->header(QNetworkRequest::ContentTypeHeader).toString();
}
if(!statusCode && reply->header(QNetworkRequest::ContentTypeHeader).toString().contains(QString("text/html"))) {
statusCode = reply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toInt();
redirectUrl = QUrl(reply->header(QNetworkRequest::LocationHeader).toUrl());
}
}
void Snapshot::doneLoading(bool)
{
// A reasonable waiting time for any script to execute
timer->start(3000);
}
void Snapshot::doneWaiting()
{
if( statusCode != 0 &&
statusCode != 301 &&
statusCode != 302 &&
statusCode != 303
) {
qDebug() << page->mainFrame()->url().toString();
qDebug() << page->mainFrame()->toHtml();
QImage image(page->viewportSize(), QImage::Format_ARGB32);
QPainter painter(&image);
page->mainFrame()->render(&painter);
painter.end();
image.save(*outputFilename);
delete outputFilename;
QApplication::quit();
}
else if(statusCode != 0) {
statusCode = 0;
qDebug() << "Redirecting to: " + redirectUrl.toString();
if(page->mainFrame()->url().toString().isEmpty()) {
qDebug() << "about:blank";
page->mainFrame()->load(this->redirectUrl); // No network activity after this
qDebug() << "Loading";
}
}
// This should ensure that the program never hangs
if(statusCode == 0) {
if(tries > 5) {
qDebug() << "Giving up.";
QApplication::quit();
}
tries++;
}
}
The problem was that the page I was testing was redirecting to https and had a self-signed certificate.
The solution was to make the QNetworkReply ignore the ssl errors:
void Snapshot::sslErrors(QNetworkReply *reply, const QList<QSslError> &errors)
{
reply->ignoreSslErrors();
}
Related
A proxy application which the client side redirect all connections to the server side application.
At the server side, QTcpServer::newConnection() signal is emitted and we connect QTcpSocket::readyRead() signal to our slot which receives the data from client and redirect the data to 3rd party application for handling HTTP/HTTPS requests. readyRead() signal emitted when client sends HTTP request but this does not happen with HTTPS requests.
Tried: Set QTcpServer::proxy() and QTcpSocket::proxy() to QNetworkProxy::NoProxy, use QSslSocket instead of QTcpSocket with ignoreSslErrors().
Code:
Client::Client(QObject *parent, QTcpSocket* clientCon)
: QObject(parent)
{
client = clientCon;
tunnel = new QSslSocket();
if (serverApp) // If application running as server
tunnel->connectToHost("127.0.0.1", 808); // Connect to CCProxy which handle the requests for us
else
{
tunnel->setProxy(QNetworkProxy::NoProxy);
tunnel->ignoreSslErrors();
tunnel->connectToHost(QHostAddress("serverip"), 9090); // Connect to the server side if running as client
}
tunnel->waitForConnected(5000); // 5 seconds timeout
if (tunnel->state() == tunnel->ConnectedState)
{
QObject::connect(client, SIGNAL(readyRead()), this, SLOT(ClientDataReceived()));
QObject::connect(tunnel, SIGNAL(readyRead()), this, SLOT(TunnelDataReceived()));
QObject::connect(client, SIGNAL(disconnected()), this, SLOT(ClientDisconnected()));
QObject::connect(tunnel, SIGNAL(disconnected()), this, SLOT(ClientDisconnected()));
}
}
Client::~Client()
{
}
bool Client::IsTunnelEstablished()
{
return tunnel->state() == tunnel->ConnectedState ? true : false;
}
QString Client::GetTunnelErrorString()
{
return tunnel->errorString();
}
void Client::ClientDataReceived()
{
QByteArray data = client->readAll();
qDebug() << "Client [" << client->peerAddress().toString().remove(" ") << ":" << client->peerPort() << "] Data Received: " << data.size();
qint64 wr = tunnel->write(data, data.size());
if (wr != data.size())
qDebug() << "QTcpSocket::write() failed!"; // Never happened
data.clear();
tunnel->flush();
}
void Client::TunnelDataReceived()
{
QByteArray data = tunnel->readAll();
qDebug() << "Client Tunnel [" << client->peerAddress().toString().remove(" ") << ":" << client->peerPort() << "] Data Received: " << data.size();
client->write(data);
data.clear();
client->flush();
}
void Client::ClientDisconnected()
{
/*if (tunnel->state() == tunnel->ConnectedState)
tunnel->close();*/
qDebug() << "[" << client->peerAddress().toString().remove(" ") << ":" << client->peerPort() << "] Disconnected";
QObject::disconnect(client, SIGNAL(readyRead()), this, SLOT(ClientDataReceived()));
QObject::disconnect(tunnel, SIGNAL(readyRead()), this, SLOT(TunnelDataReceived()));
QObject::disconnect(client, SIGNAL(disconnected()), this, SLOT(ClientDisconnected()));
QObject::disconnect(tunnel, SIGNAL(disconnected()), this, SLOT(ClientDisconnected()));
client->deleteLater();
tunnel->deleteLater();
delete this; // Self delete
}
Server Initialization:
void BaseProtocol::StartServer()
{
server = new QTcpServer();
server->setProxy(QNetworkProxy::NoProxy);
server->listen(QHostAddress("0.0.0.0"), serverApp ? 9090 : 5050); // 9090 for server, 5050 for client
qDebug() << QObject::connect(server, SIGNAL(newConnection()), this, SLOT(ClientConnecting()));
}
void BaseProtocol::ClientConnecting()
{
QTcpSocket *client = server->nextPendingConnection(); // Accept the connection
qDebug() << "Client Connected: " << client->peerAddress().toString() << ":" << client->peerPort();
Client* clientHandler = new Client(this, client);
if (!clientHandler->IsTunnelEstablished())
{
qDebug() << "Tunnel Connection Failed: " << clientHandler->GetTunnelErrorString() << "\n";
client->disconnect();
client->deleteLater();
delete clientHandler;
}
}
As you see the client and server application are the same but start with different ports. The reason why I use a client side application instead of connecting to the server as proxy directly is because I want to encrypt the data between client and server later. (same problem also exist when connecting directly to server side)
This is supposed to be a transparent proxy so there is no need for SSL/TLS at all.
"Internet Options" are used to set the proxy server to "127.0.0.1:5050" at the client side.
I am trying to get data out of slot with a signal readyRead(). But my method doesn't seem to work. I googled a lot but still I can't solve the problem.
Here what I have:
In my main function I call the method sendPOST() to get cookies. I got cookies from this method using inside of it SIGNAL finished(QNetworkReply *) and SLOT replyFinishedSlot_(QNetworkReply *) :
connect(manager_, SIGNAL(finished(QNetworkReply *)), this, SLOT(replyFinishedSlot_(QNetworkReply *)));
I made a public static bool variable isFinished = false by default to write if slot is finished it's job.
replyFinishedSlot_(QNetworkReply ):
if(reply->error())
qDebug() << "Error: " << reply->errorString();
else
{
cookie = reply->manager()->cookieJar()->cookiesForUrl(webReportsUrl);
QString cookieString = cookie[0].name() + "=" + cookie[0].value() + "; domain=" + cookie[0].domain() + "; path=" + cookie[0].path() + ";";
if(reply->isFinished()) isFinished = true; //isFinished is static public variable
}
reply->deleteLater();
And then I check in my main function if isFinished is true, and if it is I connect to another slot:
manager_ = new QNetworkAccessManager(this);
sendPOST("http://url");
if(isFinished)
{
QNetworkAccessManager *man = new QNetworkAccessManager();
QNetworkRequest request(webReportsUrl);
request.setHeader(QNetworkRequest::CookieHeader, QVariant::fromValue(cookie));
getReply = man->get(request);
connect(getReply, SIGNAL(readyRead()), this, SLOT(readyReadSlot_()));
if(isRead)
qDebug() << "reading";
else qDebug() << "not reading";
}
and isFinished in here works very well (but I am not sure if this is the right way to check finished or not like this). I get isFinished == true, and I can get cookies from replyFinishedSlot_.
But the problem is to get data from readyReadSlot_(). I tried different ways to receive the data from this slot, but there's no successful result.
I tried even something like this:
QEventLoop loop;
connect(getReply, SIGNAL(readyRead()), &loop, SLOT(readyReadSlot_()));
loop.exec();
But I got the error:
QObject::connect: No such slot QEventLoop::readyReadSlot_() in ...
Inside readyReadSlot_() I have to receive all the data from the page:
if(getReply->isReadable())
{
if(getReply->error() != QNetworkReply::NoError)
{
qDebug() << "Error: " << getReply->errorString();
}
else {
isRead = true;
response = getReply->readAll(); //here the data I need outside of this slot
qDebug() << "response: " << response;
}
}
getReply->deleteLater();
And I get it successfully inside, but I need to get response outside of this slot, in my main function, for example.
I know here's something with a threads, and I just don't wait till the data recieved, but I don't know how can I fix it.
I found a problem solvation for me.
void DataMartsModel::replyFinishedSlot_(QNetworkReply *reply)
{
static bool isRead = false;
if(reply->error())
qDebug() << "Error: " << reply->errorString();
else
{
cookie = reply->manager()->cookieJar()->cookiesForUrl(webReportsUrl);
QString cookieString = cookie[0].name() + "=" + cookie[0].value() + "; domain=" + cookie[0].domain() + "; path=" + cookie[0].path() + ";";
QNetworkAccessManager *man = new QNetworkAccessManager();
QNetworkRequest request(webReportsUrl);
request.setHeader(QNetworkRequest::CookieHeader, QVariant::fromValue(cookie));
getReply = man->get(request);
connect(getReply, &QNetworkReply::readyRead, [=](){
if(getReply->isReadable())
{
if(getReply->error() != QNetworkReply::NoError) qDebug() << "Error: " << getReply->errorString();
else {
isRead = true;
}
}
});
if(reply->isFinished() && getReply->isReadable()) isFinished = true; //here is the problem solvation I wanted
}
reply->deleteLater();
}
main function
manager_ = new QNetworkAccessManager(this);
sendPOST("http://url");
if(isFinished)
{
QByteArray array = getReply->readAll(); //here I got the data I needed to get from readyReady
qDebug() << array; //here I display it and I can use them in the future
}
If you know better way to solve the problem, I would like to check it, too.
Im trying to find out what goes wrong when doing a get request in qt.
I have the following slots attached to my networkmanager:
connect(mgr,SIGNAL(finished(QNetworkReply*)),this,SLOT(requestFinished(QNetworkReply*)));
connect(mgr, SIGNAL(error(QNetworkReply::NetworkError)),
this, SLOT(slotError(QNetworkReply::NetworkError)));
The request finished is like:
void FirebaseInteractor::requestFinished(QNetworkReply *rep)
{
QVariant statusCode = rep->attribute( QNetworkRequest::HttpStatusCodeAttribute );
int status = statusCode.toInt();
if ( status != 200 )
{
QString reason = rep->attribute( QNetworkRequest::HttpReasonPhraseAttribute ).toString();
qDebug() << "Pushnotification Request failed : " <<reason;
}
else{
qDebug() << "Pushnotification has been send: ";
}
if ( !statusCode.isValid() )
{
QString status = statusCode.toString(); // or status_code.toInt();
qDebug() << "Failing " << status;
int code = statusCode.toInt();
qDebug() << "Pushnotification Request failed invalid status code." << QString::number(code);
QString reason = rep->attribute( QNetworkRequest::HttpReasonPhraseAttribute ).toString();
qDebug() << "reason " << reason;
return;
}
}
However Status is always empty Failing and reason is printed but there is no value after it (i was expecting a reason e.g. timeout, 401 etc).
I also tried:
int status = statusCode.toInt();
if ( status != 200 )
{
QString reason = rep->attribute( QNetworkRequest::HttpReasonPhraseAttribute ).toString();
qDebug() << "Pushnotification Request failed : " <<reason;
}
else{
qDebug() << "Pushnotification has been send: ";
}
But all reasons/codes are empty.
I also added:
void FirebaseInteractor::slotError(QNetworkReply::NetworkError error)
{
qDebug() << "slotError" << error;
}
but this is not called.
How can I find out whats going wrong?
You can get the error directly using:
qDebug() << reply->error();
This won't work if the network request never happened:
if ( !statusCode.isValid() )
Because this means that the QVariant itself is invalid and has type QMetaType::UnknownType Documentation. Hence it will not give any information about what went wrong in the http request. To fix this, here's a simple example:
if (statusCode.isValid()) { // it needs to be valid
qDebug() << "Status Code: " << statusCode.toString();
}
Alternatively, you can switch on QNetworkReply::Error()
void FirebaseInteractor::requestFinished(QNetworkReply *rep)
{
switch (rep->error())
{
case QNetworkReply::NoError:
// No error
return;
case QNetworkReply::TimeoutError:
{
auto httpStatus =
reply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toInt();
auto httpStatusMessage = reply->attribute(
QNetworkRequest::HttpReasonPhraseAttribute).toByteArray();
//...
break;
}
}
}
I would have to send Modbus request for specific data, my problem is that I have to use mobile communication, wifi, connect to a custom electronic card, which is right in Modbus RTU.
My working code connecting to the electronic board:
#include "connessione.h"
#include <QModbusTcpClient>
#include <QVariant>
#include <QModbusDataUnit>
#include <QDebug>
connessione::connessione(QObject *parent) : QObject(parent)
{
qDebug() << "here debug " << "ok";
clientX = new QModbusTcpClient();
clientX->setConnectionParameter(QModbusDevice::NetworkAddressParameter, "192.168.222.1");
clientX->setConnectionParameter(QModbusDevice::NetworkPortParameter, 5555);
if (clientX->connectDevice())
{
qDebug() << "connected: " << clientX->state();
}
else
{
qDebug() << "ERRORE" << clientX->errorString();
}
}
void connessione::clickButton(){
QModbusDataUnit readUnit(QModbusDataUnit::HoldingRegisters, 0, 1); // just read input register 40006
//qDebug() << "readUnit" << readUnit.RegisterType;
qDebug() << "readUnit" << clientX->state();
if (auto *reply = clientX->sendReadRequest(readUnit, 255)) // client id 255
{
if (!reply->isFinished())
{
// connect the finished signal of the request to your read slot
qDebug() << "connected" << reply->errorString();
connect(reply, &QModbusReply::finished, this, &connessione::readReady);
}
else
{
qDebug() << "Errore" << reply->errorString();
delete reply; // broadcast replies return immediately
}
}
else
{
qDebug() << "Errore" << reply->errorString();
// request error
}
}
void connessione::readReady()
{
QModbusReply *reply = qobject_cast<QModbusReply *>(sender());
if (!reply)
return;
if (reply->error() == QModbusDevice::NoError)
{
const QModbusDataUnit unit = reply->result();
int startAddress = unit.startAddress(); // the start address,
int value = unit.value(0); // value of the start address + 0
qDebug() << "NESSUN ERRORE" << reply->errorString();
}
else
{
qDebug() << "Errore readReady" << reply->errorString();
// reply error
}
reply->deleteLater(); // delete the reply
}
log string TCP sent:
D/libmodbusMobile.so( 8042): (null):0 ((null)): qt.modbus: (TCP
client) Sent TCP PDU: 0x0300000001 with tId: 2
this is right: 0x0300000001
But unfortunately, my electronic card, the integrated firmware I can not modify, is right with Modbus RTU, so I should change 0x0300000001 to 0x010300000001C1C2 where C1 and C2 are the checksums.
I believe that QModbusDataUnit generate buffer to send. So how to change it? Exist manual solution where I build the buffer?
how to change it and create custom send buffer like the example?
Thanks
I want to see the results of a GET request. By my understanding, this code should do it. What am I doing wrong?
void getDoc::on_pushButton_2_clicked()
{
manager = new QNetworkAccessManager(this);
connect(manager, SIGNAL(finished(QNetworkReply*)), this, SLOT(replyFinished(QNetworkReply*)));
manager->get(QNetworkRequest(QUrl("http://www.google.com")));
}
void getDoc::replyFinished(QNetworkReply *reply)
{
qDebug() << reply->error(); //prints 0. So it worked. Yay!
QByteArray data=reply->readAll();
qDebug() << data; // This is blank / empty
QString str(data);
qDebug() << "Contents of the reply: ";
qDebug() << str; //this is blank or does not print.
}
The code compiles and runs fine. It just doesn't work.
Try modifying your replyFinished slot to look like this:
QByteArray bytes = reply->readAll();
QString str = QString::fromUtf8(bytes.data(), bytes.size());
int statusCode = reply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toInt();
You can then print the statusCode to see if you are getting a 200 response:
qDebug() << QVariant(statusCode).toString();
If you are getting a 302 response, you are getting a status redirect. You will need to handle it like this:
if(statusCode == 302)
{
QUrl newUrl = reply->attribute(QNetworkRequest::RedirectionTargetAttribute).toUrl();
qDebug() << "redirected from " + replyUrl + " to " + newUrl.toString();
QNetworkRequest newRequest(newUrl);
manager->get(newRequest);
return;
}
I'm returning when encountering a status code of 302 since I don't want the rest of the method to execute.
I hope this helps!