Qt reading file and mapping to QVector very slow (crashes) - c++

So what I am trying to do is read a file and map it to a two dimensional QVector. Here is my code so far
void dataModel::parseFileByLines()
{
QVector<QVector<QString> > dataSet;
lastError = "";
QRegExp reg(fileDelimiter);
QFile inFile(inputFile);
if (inFile.open(QIODevice::ReadOnly)){
QTextStream fread(&inFile);
long totalSize = inFile.size();
QString line;
while(!fread.atEnd()){
line = fread.readLine();
dataSet.append(line.split(reg,QString::KeepEmptyParts).toVector());
}
}else{
lastError = "Could not open "+inputFile+" for reading";
}
}
My issue is that when dealing with 1000,000 lines or more the program crashes with a message saying "This application has requested the Runtime to terminate it in an unusual way". Is there a more efficient way I can achieve my goal ? If so how ?
The input file may be in a format like so
ID,NAME,AGE,GENDER...etc
1,Sam,12
...
...
1000000
I would really appreciate any help or advice

I have tested this (QList) version here on my computer and it runs much faster then the QVector version and I also believe it will not crash.
void parseFileByLines(QString inputFile)
{
QList<QList<QString> > dataSet;
QString lastError = "";
QFile inFile(inputFile);
if (inFile.open(QIODevice::ReadOnly)){
QTextStream fread(&inFile);
long totalSize = inFile.size();
QString line;
while(!fread.atEnd()){
line = fread.readLine();
QList<QString> record = line.split('\t',QString::KeepEmptyParts);
dataSet.append(record);
}
}else{
lastError = "Could not open "+inputFile+" for reading";
}
}

Related

Infinite cycle due to QTextStream

So, I get infinite cycle while trying to read lines from file (line by line). I was trying to use do{}while(); cycle like that:
QTextStream stream(stdin);
QString line;
do {
line = stream.readLine();
} while (!line.isNull());
but I get empty string.
Sure, I checked file path (it is right). I was trying to use /Users/user/tts.txt path but without changes. I was trying to read other files (like m3u). And it's not working on macOS Catalina, Windows 10, Linux (Debian).
So, why did I get infinite cycle?
QStringList Manager::GetLinesFromFile(const QString &nameOfFile)
{
QStringList lines = {};
//path to file
const QString path = QCoreApplication::applicationDirPath() + "/bin/" + "tts.txt";
//"/Users/user/tts.txt"
QFile buffer;
buffer.QFile::setFileName(path);
#ifndef Q_DEBUG
qDebug() << path;
#endif
if(buffer.QFile::exists())
{
if(!buffer.QIODevice::open(QIODevice::ReadOnly))
{
#ifndef Q_DEBUG
qCritical() << "error: can't open file";
#endif
}
else
{
QTextStream stream(&buffer);
// both conditions
// (!stream.QTextStream::atEnd())
while(!buffer.QFileDevice::atEnd())
lines.QList::push_back(stream.QTextStream::readLine());
buffer.QFile::close();
}
}
else
{
#ifndef Q_DEBUG
qCritical() << "error: file not exists";
#endif
}
return lines;
}
Have a look at the QTextstream documentation https://doc.qt.io/qt-5/qtextstream.html. There is an example of reading line by line. Your while loop should read until the stream reaches the end of the buffer and many of the in built read functions will return false when his happens
So, I got it. I opened the file incorrectly.
I was using:
if(!file.QIODevice::open(QIODevice::ReadOnly))
but it should be like that:
if(!file.QFile::open(QFile::ReadOnly))

Converting valid QFile to QString - QString is empty

So I am trying to convert a QFile into a QString by doing the following:
void MainWindow::openTemplateFile(QString location)
{
if (location.isEmpty())
return;
else
{
QString variable;
templateFile.setFileName(location);
if (!templateFile.open(QFile::ReadOnly | QFile::Text))
{
QMessageBox::information(this, "Unable to open template",
templateFile.errorString());
return;
}
else // file opened and ready to read from
{
QTextStream in(&templateFile);
QString fileText = in.readAll();
qDebug() << templateFile.size() << in.readAll();
}
}
}
However, in I get the following result in the debug console:
48 ""
templateFile does exist and is part of the MainWindow class. This is also simplified code - in the actual program I read chars from the file and it works correctly. The location string is a result of the QFileDialog::getOpenFileName function, which I open a txt file with.
You call readAll() twice. The second time, the stream is positioned at end-of-file, and so readAll() has nothing to read and returns an empty string. Print fileText in your debug output instead.

Qt getting network requests with QNetworkReply that download data to temp file

I am having some issues reading an online file. I'm trying to read what is in the file after it gets downloaded to a temporary file. Here is my code:
void MainWindow::fileIsReady( QNetworkReply * reply)
{
QTemporaryFile tmpFile;
tmpFile.write(reply->readAll());
QByteArray asdf = reply->readAll();
qDebug() (QString("%1").arg(asdf.length())); // returns 0
if (tmpFile.open())
{
qDebug << "attempting to read file";
QTextStream stream(&tmpFile);
QString value = stream.readAll();
qDebug << value; // value is returning nothing
}
else
{
qDebug() << "failed to open internet file";
}
}
// in MainWindow constructor (MainWindow::MainWindow)...
QNetworkAccessManager * manager = new QNetworkAccessManager(this);
connect(manager, SIGNAL(finished(QNetworkReply*)), this, SLOT(fileIsReady(QNetworkReply*)) );
manager->get(QNetworkRequest(QUrl("https://www.website.com/stuff/file.exe.md5")));
I'm going to be using this to compare two md5 strings.
There are several issues in your code:
You need to open tmpFile before writing to it.
reply->readAll() will return data only once. Further calls will return empty array. Once you received data using readAll, it's your responsibility to store it in a variable if you need it later.
After you wrote someting to file, the file pointer is at its end. You cannot read anything from it because there is no data there. You can use seek to move pointer to the beginning of the file and read its content.
There is no point in reading from file just after you wrote data to it. You can use QTextStream on QNetworkReply directly to read text from it. (Maybe this was just for debugging, I don't know.)
It's hard to believe that you need to create a temporary file just to calculate md5. There are simplier ways to do that.
So turns out I was dumb and forgot to open the reply first. Also, it was unnecessary for me to create a temp file. Here is my solution::
void MainWindow::fileIsReady( QNetworkReply * reply)
{
if (reply->error() == QNetworkReply::NoError)
{
if (reply->open(QIODevice::ReadOnly))
{
QByteArray asdf = reply->readAll();
qDebug() << (QString("asdf %1").arg(asdf.length()));
qDebug() << (QString(asdf));
}
else
{
qDebug << "cant open reply";
}
}
}

How to get the SHA-1/MD5 checksum of a file with Qt?

Is there a way to get the MD5 or SHA-1 checksum/hash of a file on disk in Qt?
For example, I have the file path and I might need to verify that the contents of that file matches a certain hash value.
Open the file with QFile, and call readAll() to pull it's contents into a QByteArray. Then use that for the QCryptographicHash::hash(const QByteArray& data, Algorithm method) call.
In Qt5 you can use addData():
// Returns empty QByteArray() on failure.
QByteArray fileChecksum(const QString &fileName,
QCryptographicHash::Algorithm hashAlgorithm)
{
QFile f(fileName);
if (f.open(QFile::ReadOnly)) {
QCryptographicHash hash(hashAlgorithm);
if (hash.addData(&f)) {
return hash.result();
}
}
return QByteArray();
}
If you are using Qt4, you can try this.
QByteArray fileChecksum(const QString &fileName, QCryptographicHash::Algorithm hashAlgorithm)
{
QFile sourceFile(fileName);
qint64 fileSize = sourceFile.size();
const qint64 bufferSize = 10240;
if (sourceFile.open(QIODevice::ReadOnly))
{
char buffer[bufferSize];
int bytesRead;
int readSize = qMin(fileSize, bufferSize);
QCryptographicHash hash(hashAlgorithm);
while (readSize > 0 && (bytesRead = sourceFile.read(buffer, readSize)) > 0)
{
fileSize -= bytesRead;
hash.addData(buffer, bytesRead);
readSize = qMin(fileSize, bufferSize);
}
sourceFile.close();
return QString(hash.result().toHex());
}
return QString();
}
Because
bool QCryptographicHash::addData(QIODevice *device)
Reads the data from the open QIODevice device until it ends and hashes it. Returns true if reading was successful.
This function was introduced in Qt 5.0.
References: https://www.qtcentre.org/threads/47635-Calculate-MD5-sum-of-a-big-file

How to create a dynamic filename for an SD card on Arduino

I'd like to log my data on my Arduino one file at a time. I'd like the filename to be a combination of the number of milliseconds that have passed + some ID. For example, GPS data would be millis()+"GPS".
I tried the following code, but it doesn't like the fact that I am using a String. I could use a char array, but the length would always be dynamic. Is there a way to do this with at string somehow?
static void writeToSD()
{
String logEntry = " GPS: ";
logEntry += GPSString;
String filename = String(millis());
filename += "GPS";
Serial.println(logEntry);
Serial.println(filename);
File dataFile = SD.open(filename, FILE_WRITE);
// If the file is available, write to it:
if (dataFile) {
dataFile.println(logEntry);
dataFile.close();
Serial.println("Closed");
}
// If the file isn't open, pop up an error:
else {
Serial.println("error opening file");
}
}
You could try the following
char fileNameCharArray[filename.length()];
filename.toCharArray(fileNameCharArray, filename.length())
File dataFile = SD.open(fileNameCharArray, FILE_WRITE);
sprintf (filename, "%ld-GPS", millis());
Note that the use of String on Arduino is discouraged because of the well documented memory leak/fragmentation problems.