Proper way to handle multiple QNetworkRequests - c++

im grabbing url's from a list and then sending each to a QNetworkRequest and recieving the HTML back to process. I however have thousands of requests to process. So my application kept hanging until i stopped it from producing all these requests at once.
Is this the rite way to handle the que for a great number of requests?
I tried using a Qqueue of Urls, which i would then link to a SLOT which was triggered after each QNetworkReply reponse.
Create the job list and add to que
QQueue<QString> jobs;
for (int i = 1; i <= totalPages; i++){
QString pageUrl = url + "&page=" + QString::number(i);
jobs.enqueue(pageUrl);
}
qDebug() << "Total Jobs : " << jobs.count() << endl;
for (int i = 0; i < 5; i++){
processQueue();
}
then inside the getHtml function
QNetworkAccessManager *manager = new QNetworkAccessManager(this);
QNetworkRequest *getHtmlRequest = new QNetworkRequest(pageUrl);
getHtmlRequest = new QNetworkRequest(url);
getHtmlRequest->setRawHeader( "User-Agent", "Mozilla/5.0 (X11; U; Linux i686 (x86_64); "
"en-US; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1" );
getHtmlRequest->setRawHeader( "charset", "utf-8" );
getHtmlRequest->setRawHeader( "Connection", "keep-alive" );
connect(manager, SIGNAL(finished(QNetworkReply*)), this, SLOT(replyGetPageHtmlFinished(QNetworkReply*)));
connect(manager, SIGNAL(finished(QNetworkReply*)), this, SLOT(processQueue()));
manager->get(*getHtmlRequest);
which triggers
void checkNewArrivalWorker::processQueue(){
if (jobs.isEmpty()){
qDebug() << "Jobs Completed" << endl;
emit finished();
} else {
QString pageUrl = jobs.dequeue();
QString pageNumber = pageUrl.mid(pageUrl.indexOf("page=") + 5);
getHtml(pageUrl, pageNumber);
}
}

Related

QTcpSocket readyRead() Signal emitted twice

I have QTcpServer. I want to send large data from client side and how to catch signal, when all data is received on server?
"while (socket->bytesavailable)" doesn't work.
For example:
when qbytearray size is 9000, which is send from client, on the server it's 4000 or 5000...
Example Two:
In this Case readyRead() SIGNAL Is Emited 8 times.
void Client::SendMessage(std::vector<QString>)
{
MyClass _Send;
_Send.Age = 22;
_Send.School = 14;
_Send.Name = "Taz";
QVector<MyClass2> vv;
for (int i = 0; i < 15000; i++) {
vv.push_back(MyClass2(24, "leri"));
vv.push_back(MyClass2(22, "tazo"));
}
_Send.vctr = vv;
QByteArray bytes;
QDataStream stream(&bytes, QIODevice::WriteOnly);
int FunctionUID = 331;
int ij, ik = ij = 169;
MyClass2 faf(-31, "15");
stream << FunctionUID << _Send << faf << ij << ik;
socket->write(bytes);
}
void Server::ImReady()
{
QByteArray buf;
buf = socket->readAll();
QDataStream stream(&buf, QIODevice::ReadOnly);
int FunctionUID, ij, ik;
MyClass vv;
MyClass2 vv1;
stream >> FunctionUID >> vv >> vv1 >> ij >> ik;
qDebug() << vv.vctr.size() << "faf";
}
void Server::incomingConnection(qintptr val)
{
qDebug() << "Client Connected" << val;
socket = new QTcpSocket;
socket->setSocketDescriptor(val);
if (!socket) {
return;
}
connect(socket, SIGNAL(disconnected()), socket, SLOT(deleteLater()));
connect(socket, SIGNAL(readyRead()), this, SLOT(ImReady()));
}
TCP does not send messages, it is a stream of data. This means you can't just read all data that's there, and consider this one message. Instead one usually sends a header, containing the message size, and then the receiver knows how much to read to get the whole message (and just this one, not read into the next).
This means your slot would do something like this
void Server::ImReady()
{
uint32 size;
socket->read(&size, 4);
uint32 readTotal = 0;
do {
readTotal += socket->read(buffer, size-readTotal);
} while (readTotal < size);
}
You could put a line like
if (socket->bytesAvailable() == 0)
return;
at the beginning of the slot, and then you would not care at all if the signal is emitted more than once per message.
Note that the code above needs additional error handling, e.g. you must make sure the first read reads all 4 bytes, and always handle a return value of -1.

Can not send udp packet in QT

all
I want to use QUdpSocket to send udp packet to get config parameter from specific server,but It failed and I can not capture the sending packet using wireshark.here is my code:
CGetConfig.cpp:
CGetConfig::CGetConfig(const QString &conf_server,const uint16_t port)
:m_conf_server(conf_server)
,m_port(port)
{
m_socket = NULL;
}
CGetConfig::~CGetConfig()
{}
void CGetConfig::init()
{
// create a QUDP socket
m_socket = new QUdpSocket(this);
m_socket->bind(QHostAddress::LocalHost, 12345);
connect(m_socket, SIGNAL(readyRead()), this, SLOT(readyRead()));
m_ip_addr = get_ip_address(m_conf_server);
}
bool CGetConfig::get_reflector(const QString &mac)
{
qDebug() << "CGetConfig::get_reflector():Entry\n";
if(m_ip_addr.isEmpty())
{
qDebug() << "CGetConfig::get_reflector():ip address of cofnig server could not be resolved\n";
return 0;
}
QString msg("id=1&mac=");
msg+= mac;
msg+= "&get_config=fw_type=v.1,cfg_ver=4,set_ver=0,ip=192.168.1.101";
qDebug() << m_ip_addr;
qDebug() << m_port;
qDebug() << msg.toLatin1();
int count = 0;
while(count < 3)
{
int t = m_socket->writeDatagram(msg.toLatin1(), QHostAddress(m_ip_addr), m_port);
count++;
}
}
Main.cpp
CGetConfig cfg(cfg_server,cfg_port);
cfg.init();
local_mac = "00d033120001";
cfg.get_reflector(local_mac);
Can anyone help me figure out the problem?

QT NetworkManager initialazation and asynch problems

Firstly, I'd like to trigger an initialization process whenever this class been created. (DNS lookups for known hosts, and and pre-handshake with hosts over a specified number of slots - this shall not be hardcoded once it has been tested) - while all the classes are being created, config is being read, etc.
Secondly, I'd like to allow parallelized request sending depending on the allowed (and opened) slots present for the specific host.
Quite frankly there's multiple issues with the following code;
Google seems to redirect, upon multiple requests at the same time to this host seems to result in a catastrophic failure, when x number or request triggers the finished signal, and that signals invokes my redirect checking and the first request get redirected, and I do seem to get the first reply back, with x error messages.
I cant seem to invoke deleteLater() from httpFinished for the same reason
I've could not find a way to wait for the slot number of request to be finished to start the next request in queue
The main goal would be to gather data from multiple APIs using a single network class, those requests would maybe occur the same time, may not.
Also there would be another request emitted just as POST, etc. - maybe 1 class / API?
Please note that the code is just test of concept and for testing purposes only, and this shall be cleaned after it has been fixed, this is not how I intended it to work at all.
nebula.pro
QT += core network
QT -= gui
TARGET = nebula
CONFIG += console
CONFIG -= app_bundle
TEMPLATE = app
SOURCES += main.cpp \
network.cpp
HEADERS += \
network.h
network.h
#ifndef NETWORK_H
#define NETWORK_H
#include <QObject>
#include <QtNetwork/QNetworkAccessManager>
#include <QtNetwork/QNetworkReply>
#include <QtNetwork/QHostInfo>
#include <QtNetwork/QSslConfiguration>
#include <QEventLoop>
class network : public QObject
{
Q_OBJECT
public:
explicit network(QObject *parent = 0);
~network();
void sendGet(const QUrl &url);
private:
QNetworkAccessManager networkAccessManager;
QHash<QString, QByteArray> usedSession;
QNetworkReply *reply;
QUrl url;
Q_INVOKABLE void init();
void replyFinished(QUrl &url, QNetworkReply *reply);
QList<QString> provideTcpHosts();
QList<QString> provideSslHosts();
QEventLoop eventLoop;
private slots:
void httpError();
void sslErrors(const QList<QSslError> &errors);
void httpFinished();
};
#endif // NETWORK_H
network.cpp
#include "network.h"
/**
* #brief network::network
* initialazing pre-networking initialization once the eventpool is ready
* #param parent
*/
network::network(QObject *parent) : QObject(parent)
{
QMetaObject::invokeMethod(this, "init", Qt::QueuedConnection);
}
network::~network()
{
networkAccessManager.deleteLater();
}
/**
* #brief network::init
* dns chache warming for all hosts and pre-connecting to all hosts via 4 sockets
*/
void network::init()
{
QList<QString> tcpHosts = provideTcpHosts();
QList<QString> sslHosts = provideSslHosts();
// tcp hosts initialazation
for( int i=0; i<tcpHosts.count(); ++i )
{
// pre-dns lookup cache warming
QHostInfo::lookupHost(tcpHosts[i], 0, 0);
qDebug() << "pre-dns lookup cache warming for: " + tcpHosts[i];
// tcp pre-handshake with known hosts over 4 sockets with known http hosts
for(int a=0; a<4; ++a)
{
networkAccessManager.connectToHost(tcpHosts[i], 80);
qDebug() << "connecting " + QString::number(a+1) + "th socket for " + tcpHosts[i];
}
}
// tcp hosts initialazation
for( int i=0; i<sslHosts.count(); ++i )
{
// pre-dns lookup cache warming
QHostInfo::lookupHost(sslHosts[i], 0, 0);
qDebug() << "pre-dns lookup cache warming for: " + sslHosts[i];
// tcp pre-handshake with known hosts over 4 sockets with known http hosts
for(int a=0; a<4; ++a)
{
networkAccessManager.connectToHostEncrypted(sslHosts[i], 443);
qDebug() << "connecting " + QString::number(a+1) + "th socket for " + sslHosts[i];
}
}
}
/**
* #brief network::replyFinished
* storing previous ssl session tickets for re-use, and error handling for finished requests
* #param url
* #param reply
*/
void network::replyFinished(QUrl &url, QNetworkReply *reply)
{
if(!usedSession.contains((QString)url.toString()))
{
usedSession.insert((QString)url.toString(), (QByteArray)reply->sslConfiguration().sessionTicket());
qDebug() << "saved ssl session ticket for" + url.toString();
}
reply->deleteLater();
}
/**
* #brief network::sendGet
* sending a simple GET request to specified url
* #param url
*/
void network::sendGet(const QUrl &url)
{
connect(&networkAccessManager, SIGNAL(finished(QNetworkReply*)), &eventLoop, SLOT(quit()));
qDebug() << "Sending a GET request to" + (QString)url.toString();
QNetworkRequest request;
request.setUrl(url);
request.setRawHeader("User-Agent", "nebula");
// reusing an ssl session ticket if exists for url
if(usedSession.contains((QString)url.toString()))
{
QSslConfiguration qssl;
qssl.setSslOption(QSsl::SslOptionDisableSessionPersistence, false);
qssl.setSessionTicket(usedSession.value((QString)url.toString()));
request.setSslConfiguration(qssl);
qDebug() << "used ssl session ticket for" + url.toString();
}
reply = networkAccessManager.get(request);
connect(reply, SIGNAL(error(QNetworkReply::NetworkError)), this, SLOT(httpError()));
connect(reply, SIGNAL(sslErrors(QList<QSslError>)), this, SLOT(sslErrors(QList<QSslError>)));
connect(reply, SIGNAL(finished()), this, SLOT(httpFinished()));
}
/**
* #brief network::provideTcpHosts
* #return
*/
QList<QString> network::provideTcpHosts()
{
return QList<QString>()
<< (QString)"http://www.google.com"
<< (QString)"http://www.bing.com";
}
/**
* #brief network::provideSslHosts
* #return
*/
QList<QString> network::provideSslHosts()
{
return QList<QString>()
<< (QString)"https://www.ssllabs.com/ssltest/";
}
/*SLOTS*/
/**
* #brief network::slotTcpError
* #param tcpError
*/
void network::httpError()
{
qDebug() << "QNetworkRequest::HttpStatusCodeAttribute " << reply->attribute(QNetworkRequest::HttpStatusCodeAttribute ) << "received";
qDebug() << reply->errorString();
reply->deleteLater();
}
/**
* #brief network::slotSslErrors
* #param sslErrors
*/
void network::sslErrors(const QList<QSslError> &errors)
{
QString errorString;
foreach (const QSslError &error, errors) {
if (!errorString.isEmpty())
errorString += ", ";
errorString += error.errorString();
}
qDebug() << "ssl error recieved: " + errorString;
reply->deleteLater();
}
void network::httpFinished()
{
// possible redirect url
QVariant redirectionTarget = reply->attribute(QNetworkRequest::RedirectionTargetAttribute);
if (!redirectionTarget.isNull()) {
this->sendGet(reply->url().resolved(reply->attribute(QNetworkRequest::RedirectionTargetAttribute).toUrl()));
} else {
qDebug() << "QNetworkRequest::HttpStatusCodeAttribute " << reply->attribute(QNetworkRequest::HttpStatusCodeAttribute ) << "received ";
qDebug() << reply->errorString();
qDebug() << reply->readAll();
}
}
main.cpp
#include <QCoreApplication>
#include <network.h>
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
network networkAccessManager;
for(int i = 0; i < 50; ++i)
{
networkAccessManager.sendGet((QString)"http://www.google.com");
}
return a.exec();
}
Please, note I'm quite new to advanced C++ projects and to QT, this project partially -but will be used for my awesome project once its un-screwed- is to help me being familiarized with those topics.
Could you help me out, oh glorious stranger? :)
Regards;
OneEyedSpaceFish

Get Image URL From String

I am trying to figure out a way to get a image url from a web page source.
I can get the web page source into a string and parse it line by line to find the line with the URL.
However, I haven't been able to figure out a good way to pull just the URL from the line.
I'm think this can be done with QRegExp, but have been unable to figure out how to use it.
Line I am trying to parse
<img width="980" height="1515" id="mainImg" src="//test/123.jpg" alt="test">
Final Working Code
void MainWindow::on_btnDownload_clicked()
{
QString url = "http://test.foo.com";
QUrl qURL = url;
QNetworkAccessManager manager;
QNetworkReply *response = manager.get(QNetworkRequest(QUrl(url)));
QEventLoop event;
connect(response,SIGNAL(finished()),&event,SLOT(quit()));
event.exec();
QString html = response->readAll();
QStringList str;
str = html.split("\n");
//qDebug() << url;
for (int i = 0; i < str.size(); ++i){
if(str.at(i).contains("id=\"mainImg\"", Qt::CaseInsensitive)){
QString pic;
pic = str.at(i);
pic = pic.remove(QRegExp("<img[^>]*src=['|\"]", Qt::CaseInsensitive));
pic = pic.remove(QString::fromStdString("//"), Qt::CaseInsensitive);
pic = pic.remove('"');
pic = pic.remove("'");
pic = pic.remove('<');
pic = pic.remove('>');
pic = pic.remove(';');
pic = pic.left(pic.length()-1);
//qDebug() << str.at(i);
qDebug() << pic;
}
}
qDebug() << "Lines: " << str.size();
}

The connect (in a QT project) doesn't work

I'm starting to create my first multithread application, using the QT libraries.
Following the qt guide about QTcpServer and QTcpSocket, i wrote a server application that create the connection with this constructor:
Connection::Connection(QObject *parent) : QTcpServer(parent)
{
server = new QTcpServer();
QString ipAddress;
if (ipAddress.isEmpty())
ipAddress = QHostAddress(QHostAddress::LocalHost).toString();
if (!server->listen(QHostAddress(ipAddress),41500))
{
qDebug() << "Enable to start server";
server->close();
return;
}
connect(server,SIGNAL(newConnection()),this,SLOT(incomingConnection()));
}
This is the incomingConnection() function which create a new thread everytime a new client try to connect:
void Connection::incomingConnection()
{
QTcpSocket *socket = new QTcpSocket();
socket = this->nextPendingConnection();
MyThreadClass *thread = new MyThreadClass(socket, server);
qDebug() << "connection required by client";
if (thread != 0)
{
connect(thread, SIGNAL(finished()), thread, SLOT(deleteLater()));
thread->start();
}
else
qDebug() << "Error: Could not create server thread.";
}
Now, this is MyThreadClass:
MyThreadClass::MyThreadClass(QTcpSocket *socket, QTcpServer *parent) : QThread(parent)
{
tcpSocket = new QTcpSocket();
database = new Db();
blockSize = 0;
tcpSocket = socket;
qDebug() << "creating new thread";
}
MyThreadClass::~MyThreadClass()
{
database->~Db();
}
void MyThreadClass::run()
{
qDebug() << "Thread created";
connect(tcpSocket, SIGNAL(readyRead()), this, SLOT(dataAvailable()));
exec();
}
void MyThreadClass::dataAvailable()
{
qDebug() << "data available";
QDataStream in(tcpSocket);
in.setVersion(QDataStream::Qt_4_0);
if (blockSize == 0) {
if (tcpSocket->bytesAvailable() < (int)sizeof(qint16))
return;
in >> blockSize;
}
if (tcpSocket->bytesAvailable() < blockSize)
return;
QString string;
in >> string;
//[...]
}
The code compiles fine but when i start a client (after starting the server), i receive the following error by server:
QObject::connect: Cannot connect (null)::readyRead() to QThread::dataAvailable()
Then the server cannot receive data by the client.
does anyone have any idea?
thanks in advance
Daniele
socket = this->nextPendingConnection();
should be:
socket = server->nextPendingConnection();
because you are using the server member and not this as the active QTcpServer, the class Connection shouldn't even inherit from QTcpServer, but only from QObject.
Also, you are misusing QThread. You should read Signals and slots across threads, and probably Threads and the SQL Module, if Db is using the QtSql module.