QNetworkAccessManager multiple uploads fail - c++

In my App, I have a method to upload files to the server, this works fine.
But when I call this method multiple times at once (like iterating over the result of a chooseFilesDialog) the first 7 (more or less) files are uploaded correctly, the others never get uploaded.
I think this has to be linked with the fact the server doesn't allow more than X connections from the same source maybe?
How can I make sure the upload waits for a free, established connection?
this is my method:
QString Api::FTPUpload(QString origin, QString destination)
{
qint64 timestamp = QDateTime::currentMSecsSinceEpoch();
QUrl url("ftp://ftp."+getLSPro("domain")+destination);
url.setUserName(getLSPro("user"));
url.setPassword(getLSPro("pwd"));
QFile *data = new QFile(origin, this);
if (data->open(QIODevice::ReadOnly))
{
QNetworkAccessManager *nam = new QNetworkAccessManager();
QNetworkReply *reply = nam->put(QNetworkRequest(url), data);
reply->setObjectName(QString::number(timestamp));
connect(reply, SIGNAL(uploadProgress(qint64, qint64)), SLOT(uploadProgress(qint64, qint64)));
return QString::number(timestamp);
}
else
{
qDebug() << "Could not open file to FTP";
return 0;
}
}
void Api::uploadProgress(qint64 done, qint64 total) {
QNetworkReply *reply = (QNetworkReply*)sender();
emit broadCast("uploadProgress","{\"ref\":\""+reply->objectName()+"\" , \"done\":\""+QString::number(done)+"\", \"total\":\""+QString::number(total)+"\"}");
}

First, don't create a QNetworkManager every time you start an upload.
Second, you definitely have to delete everything you new(), otherwise you are left with memory leaks. This includes the QFile, the QNetworkManager AND the QNetworkReply (!).
Third, you have to wait for the finished() signal.
Api::Api() { //in the constructor create the network access manager
nam = new QNetworkAccessManager()
QObject::connect(nam, SIGNAL(finished(QNetworkReply*)), this, SLOT(finished(QNetworkReply*)));
}
Api::~Api() { //in the destructor delete the allocated object
delete nam;
}
bool Api::ftpUpload(QString origin, QString destination) {
qint64 timestamp = QDateTime::currentMSecsSinceEpoch();
QUrl url("ftp://ftp."+getLSPro("domain")+destination);
url.setUserName(getLSPro("user"));
url.setPassword(getLSPro("pwd"));
//no allocation of the file object;
//will automatically be destroyed when going out of scope
//I use readAll() (see further) to fetch the data
//this is OK, as long as the files are not too big
//If they are, you should allocate the QFile object
//and destroy it when the request is finished
//So, you would need to implement some bookkeeping,
//which I left out here for simplicity
QFile file(origin);
if (file.open(QIODevice::ReadOnly)) {
QByteArray data = file.readAll(); //Okay, if your files are not too big
nam->put(QNetworkRequest(url), data);
return true;
//the finished() signal will be emitted when this request is finished
//now you can go on, and start another request
}
else {
return false;
}
}
void Api::finished(QNetworkReply *reply) {
reply->deleteLater(); //important!!!!
}

Related

How do I upload multiple files in parallel through FTP using QNetworkAccessManager?

I have written a test class that I'm using to learn more about QNetworkAccessManager and its functionality related to FTP uploading. I'm trying to upload files from my Raspberry Pi to my Windows PC through FTP and this works well when I just want to upload a single file. However, I want to transmit multiple files at once; the same way FileZilla does this when you drag a folder between client and host; it starts multiple up/downloads instead of one after the other sequentially. Below are my ftp_test.h and ftp_test.cpp files. If I provide upload() with a list of filenames, it will for each name in this list, do a put request on the server. However, only the first file gets uploaded, the second, third, fourth. etc. get ignored.
class ftp_test: public QObject
{
Q_OBJECT
public:
ftp_test (QObject *p = 0): QObject(p) { }
void upload(QString video_directory, std::vector<QString> filename);
public slots:
void uploadProgress(qint64 bytesSent, qint64 bytesTotal);
void uploadDone();
private:
QNetworkAccessManager nam;
QFile *data;
QNetworkReply *reply;
QEventLoop loop;
};
And my ftp_test.cpp
void ftp_test::upload(QString video_directory, std::vector<QString> filename)
{
for(int i = 0; i < filename.size(); i++)
{
QUrl url("ftp://xx.xx.xx.xxx:xxxxx/" + filename[i]);
url.setUserName("pi");
if(QFile::exists(video_directory + filename[i]))
{
data = new QFile(video_directory + filename[i], this);
if (data->open(QIODevice::ReadOnly))
{
reply = nam.put(QNetworkRequest(url), data);
connect(reply, SIGNAL(uploadProgress(qint64, qint64)), SLOT(uploadProgress(qint64, qint64)));
connect(reply, SIGNAL(finished()), SLOT(uploadDone()));
connect(reply, SIGNAL(error(QNetworkReply::NetworkError)), SLOT(someError(QNetworkReply::NetworkError)));
}
else
{
return;
}
}
else
{
qDebug() << "Oops";
return;
}
}
loop.exec();
}
void ftp_test::uploadProgress(qint64 bytesSent, qint64 bytesTotal)
{
qDebug() << "Uploaded" << bytesSent << "of" << bytesTotal;
}
void ftp_test::uploadDone()
{
qDebug() << "Finished" << reply->error();
data->deleteLater();
reply->deleteLater();
reply = 0;
loop.quit();
}
I thought about making multiple threads for this to work, but before I delve into that, maybe there is something I'm missing.
I'm aware that it will stop uploading after the first upload was succesful, but it doesn't matter because I'm still not seeing parallel uploads.
Is this the way it's supposed to work? How do I achieve concurrent transfer of files instead of sequential?

How to use QNetworkManager for REST api?

I would like to make a class for accessing data via REST API, for example:
class MeteoStation{
int getLatestTemperature();
int getLatestPessure();
private:
QNetworkManager nmng;
}
How could I implement this methods? Usually I was using something like:
int MeteoStation::getLatestTemperature(){
...
QEventLoop eventLoop;
connect(&m_nam,SIGNAL(finished(QNetworkReply*)),&eventLoop,SLOT(quit()));
QNetworkReply *reply = m_nam.get( req );
eventLoop.exec();
reply->readAll()
...
}
But since using inner QEventLoop is not recommended, how should I see to whom the response belong to?
MeteoStation::MeteoStation(){
connect(&nmam, SIGNAL(finished(QNetworkReply*)),
this, SLOT(parseNetworkResponse(QNetworkReply*)));
}
void MeteoStation::parseNetworkResponse( QNetworkReply *finished )
{
QByteArray data = finished->readAll();
...
Yes and it would be nice to have the class thread save. How are you solving that in your code?
How bad is making the call synchronous with:
QNetworkRequest req(url);
QScopedPointer<QNetworkReply> reply(nam.get(req));
QTime timeout= QTime::currentTime().addSecs(10);
while( QTime::currentTime() < timeout && !reply->isFinished()){
QCoreApplication::processEvents(QEventLoop::AllEvents, 100);
}
if (reply->error() != QNetworkReply::NoError) {
qDebug() << "Failure" <<reply->errorString();
}
QByteArray data = reply->readAll();
I've resolved my problem using QCoreApplication::processEvents(). The response is there within ms and I'm able to implement functionality close to libcurl.
QNetworkRequest req(url);
QScopedPointer<QNetworkReply> reply(nam.get(req));
QTime timeout= QTime::currentTime().addSecs(10);
while( QTime::currentTime() < timeout && !reply->isFinished()){
QCoreApplication::processEvents(QEventLoop::AllEvents, 100);
}
if (reply->error() != QNetworkReply::NoError) {
qDebug() << "Failure" <<reply->errorString();
}
QByteArray data = reply->readAll();
The Qt docs should provide all info you need.
You creat a nam, connect the finished signal, send the request.
QNetworkAccessManager *manager = new QNetworkAccessManager(this);
connect(manager, SIGNAL(finished(QNetworkReply*)),
this, SLOT(parseNetworkResponse(QNetworkReply*)));
manager->get(QNetworkRequest(QUrl("http://qt-project.org")));
Detecting to which request a reply belongs should not be too hard. The reply contains the url. It might be different, but not that different:
...but for a variety of reasons it can be different (for example, a
file path being made absolute or canonical).
QUrl QNetworkReply::url() const
Returns the URL of the content downloaded or uploaded. Note that the
URL may be different from that of the original request.

Qt Download File - QNetworkAccessManager, not getting data

I'm trying to have my application download a file from a URL, typically an EXE or a Jar, not that this should change much though. I have this all running in a thread, but I don't think that will make a difference (if it does let me know).
So Do_Download is my function that creates the manager, sets the URL and request, and performs get. I then try to connect the finished signal to the slot the will write the file.
void DownloadManager::Do_Download() {
QNetworkAccessManager *netManager = new QNetworkAccessManager(this);
QUrl url(install_mirror); //istall_mirror is the URL provided by user
QNetworkRequest req(url);
QNetworkReply *reply = netManager->get(req);
connect(reply, SIGNAL(finished()), this, SLOT(writeData()));
}
My writeData function checks for errors, and if there are no errors it writes the data to file.
void DownloadManager::writeData() {
QNetworkReply *reply = qobject_cast<QNetworkReply*>(sender());
if (reply) {
if (reply->error() == QNetworkReply::NoError) {
QFile file(location);
if(file.open(QIODevice::WriteOnly)) {
file.write(reply->readAll());
} else {
errorMessage = "Error writing downloaded file for mirror installation";
}
} else {
//get http status code
int httpStatus = reply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toInt();
errorMessage = "HTTP Error code while downloading from mirror: " + httpStatus;
}
reply->deleteLater();
} else {
errorMessage = "Error downloading file from installation mirror";
}
}
The problem being there is no data being written. It just creates a 0Kb file.
I tried adding a download progress slot so I could see what was going on recieving the data. So I added this to my Do_Download method.
connect(reply, SIGNAL(downloadProgress(qint64,qint64)), this, SLOT(DL_Progress(qint64,qint64)));
void DownloadManager::DL_Progress(qint64 recieved, qint64 total) {
std::cout << recieved << " / " << total << endl;
}
The output displays one time as 0 / 01
What am I doing wrong?
The only problem I see in your code is you are not waiting for the download to be finished. The NetworkRequest object would be destructed at the end of function call.
So, I would rewrite Do_Download like this (QEventLoop syncronizes the network request):
void DownloadManager::Do_Download() {
QEventLoop eventLoop;
QNetworkAccessManager *netManager = new QNetworkAccessManager(this);
QUrl url(install_mirror); //istall_mirror is the URL provided by user
QNetworkRequest req(url);
QNetworkReply *reply = netManager->get(req);
connect(reply, SIGNAL(finished()), &eventLoop, SLOT(quit()));
eventLoop.exec();
writeData(reply);
}

QNetworkAccessManager threads never finish

I know that in version 4.8 each http request gets its own thread to run.
I'm doing a links checker app that does a lot of http requests in a while loop and I notice in the windows task manager that my app is using more than 1600 threads over time and the number never goes down, only up until it crashes the app. (I'm guessing that is the cause.)
My question is, does QNetworkAccessManager have an option to use thread pool?
Or does it have an option to clean its threads after it finishes its http request?
This is the main loop:
while(!rpm_urlStack->isEmpty())
{
QString url = rpm_urlStack->top();
//define the reply
QNetworkReply *reply;
rpm_urlStack->pop();
QString urlForReq(url);
bool returnVal = true;
QNetworkRequest request;
request.setUrl(QUrl(urlForReq));
request.setRawHeader("User-Agent", USER_AGENT.toUtf8());
request.setRawHeader("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
request.setRawHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
request.setRawHeader("Accept-Language", "en-us,en;q=0.5");
request.setRawHeader("Connection", "Keep-Alive");
QEventLoop loop;
reply = m_networkManager->get(request);
connect(reply, SIGNAL(finished()), &loop, SLOT(quit()));
loop.exit();
if(!loop.isRunning()) {
loop.exec();
}
RequestFinishedHandler(reply);
// this is how I delete the reply object
delete reply;
}
RequestFinishedHandler(QNetworkReply *reply)
{
if (reply->error() > 0) {
QNetworkReply::NetworkError networkError = reply->error();
QString err = reply->errorString();
} else {
QVariant vStatusCodeV = reply->attribute(QNetworkRequest::HttpStatusCodeAttribute);
QMutexLocker lock(_pMutex); // _pMutex defined as class member
char *buffer;
buffer = getCurrentDateTime();
QTextStream out(m_file);
out << buffer << " " << _sCurrentUrl << "\n";
lock.unlock();
if(vStatusCodeV.toInt() == 200) {
QString ApiResponse;
QByteArray data;
data=reply->readAll();
ApiResponse.append(QString::fromUtf8(data));
}
}
}
It appears that to be effective, the deleteLater method must be called from within an event loop, which must regain control of execution to handle garbage collection.
Maybe you should refactor your code to have the event loop replace your while loop. Alternatively, since you're not using the finished slot to process the reply, perhaps you can delete the reply directly at the end of the RequestFinishedHandler function.

QNetworkReply reply has no members in finishedSlot

Qt Creator is used as the ide for this small app that is being developed
I am attempting to use QNetworkAccessManager to retrieve some information from a website. After the request is 'posted' to the web, the finished() signal is triggered, however the pointer that is passed to the finishedSlot() function does not appear to be pointing to an instantiated object, it is just address of the ponter. The code for the button click that starts the request and the code for the finishedSlot() method is shown below.
In the watch window, I would have expected to see a triangle next to 'reply' that when expaned would show all the data member of QNetworkReply object. Instead it has a single value of #0x80c770 which looks like the pointer address.
I'd appreciate input from anyone who can help me to understand why my pointer doesn't appeart to be pointing the the QNetworkReply object.
void MainWindow::on_btnGetOAuthToken_clicked()
{
QUrl serviceUrl("https://api.ProPhotoWebsite.com/services/oauth/authorize.mg");
QUrl postData;
postData.addQueryItem("method", "ProPhotoWebsite.auth.getRequestToken");
postData.addQueryItem("oauth_consumer_key", "AAAAAAAAAAAAAAAAAAAAAAAA"); //example key
postData.addQueryItem("oauth_nonce",QUuid::createUuid().toString());
postData.addQueryItem("oauth_signature_method","PLAINTEXT");
postData.addQueryItem("oauth_signature","999999999999999999999999999"); //example
postData.addQueryItem("oauth_timestamp", QString::number(QDateTime::currentMSecsSinceEpoch()/1000));
postData.addQueryItem("oauth_version","1.0");
//...
QNetworkRequest request(serviceUrl);
request.setHeader(QNetworkRequest::ContentTypeHeader,
"application/x-www-form-urlencoded");
// Call the webservice
QNetworkAccessManager *nam = new QNetworkAccessManager(this);
connect(nam, SIGNAL(finished(QNetworkReply*)),
SLOT(finishedSlot(QNetworkReply*)));
nam->post(request,postData.encodedQuery());
}
void MainWindow::finishedSlot(QNetworkReply *reply)
{
// Reading attributes of the reply
// e.g. the HTTP status code
QVariant statusCodeV =
reply->attribute(QNetworkRequest::HttpStatusCodeAttribute);
// Or the target URL if it was a redirect:
QVariant redirectionTargetUrl =
reply->attribute(QNetworkRequest::RedirectionTargetAttribute);
// see CS001432 on how to handle this
// no error received?
if (reply->error() == QNetworkReply::NoError)
{
QByteArray bytes = reply->readAll(); // bytes
QString string(bytes); // string
ui->lblWarning->setText(string);
}
else
{
// handle errors here
}
// need to dispose reply
delete reply;
}