Iterating through files in folder is very slow in qt - c++

I am iterating through all files in a folder (recursively though all its sub folders) but this is extremely slow..it almost never returns. I could put this in a thread but even than it will not be practical with this speed. Here is my function
int MainWindow::searchXmlFiles(QString rootDir)
{
xmlFileModel->clear(); // xmlFileModel is QStanardItemModel
int count = 0;
XmlFile xmlFile;
QString xmlFilePath;
QDirIterator iter( rootDir, QStringList() << "*.xml", QDir::Files | QDir::NoDotAndDotDot, QDirIterator::Subdirectories);
QDir dir( ui->lineEditSourceFolder->text() );
while(iter.hasNext() )
{
qDebug() << iter.next();
xmlFilePath = iter.filePath();
xmlCollection = xmlFile.openXmlFile( xmlFilePath );
QStandardItem * item = new QStandardItem( QIcon(":/icons/xmlfile.ico") , dir.relativeFilePath( xmlFilePath) );
xmlFileModel->appendRow( item );
count++;
}
QString fileCount = QString("Total of %1 file(s) will be converted from `%2` folder").arg( QString::number(count), rootDir);
ui->labelXmlFileCount->setText( fileCount );
return count;
}
Is this slow speed really expected? Is there way to do the same thing faster?
Update
After comment by Jeremey, if I comment all the work inside the loop (xmlFile.openXmlFile() and QStandardItem creation), it gets quite fast, takes like 3 seconds. If I leave QStandardItem creation there, it takes about 7 seconds/

You can use QDir::entryList() to get full list of XML files at once and then iterate through QStringList of file names.

Related

QDirIterator - Skip Folders and its subfolders

How can you skip folders with the QDirIterator?
I've tried it with:
QString nameFilter = "*.h";
QDirIterator dirIterator(folder, nameFilter, QDir::Files, QDirIterator::Subdirectories);
QString str("folder");
QStringList filenames;
while (dirIterator.hasNext())
{
if(dirIterator.next() == str) continue;
filenames.append(dirIterator.next());
}
but it only ignores the specific folder but not its subdirectories.
Any idea?
Bellow instruction working 100 percent :
This method QString QDirIterator::next() Advances the iterator to the next entry, and returns the "file path" of this new entry. If hasNext() returns false, this function does nothing, and returns a null QString.
If QDirIterator found a file with specific filter (based on const QStringList &nameFilters parameter) then QDirIterator::next() returns "file-path" and you cannot compare the whole "file-path" with "skiped-folder" to iterate!
Because of this i had to write a function as a directory parser as bellow :
QStringList Widget::getCurrDirsOfFile(const QFileInfo &file_info, const QString &default_folder)
{
QString file_path = file_info.filePath();
QString file_name = file_info.fileName();
file_path.truncate(file_path.lastIndexOf("/" + file_name));
file_path = file_path.mid(file_path.lastIndexOf("/") + 1);
return file_path;
}
This function get "file-path" and return last "folder-name's" of it like bellow :
input : /home/msi/Desktop/123/untitled/123/widget.h
output(list) : 123, 123
We have QString skiped_dir("untitled"); as skiped folders!
In this state when you got something excluding 123 "folder-name's", you can append that file to QStringList filenames :
if(folder_list.indexOf(QRegExp(skiped_dir)) == -1)
filenames.append(it.filePath());
So try this (notice in comments) :
void Widget::btn_iterate_clicked()
{
// folder to iterate
QString folder("/home/msi/Desktop/");
// folder-name you want to skip
QString skiped_dir("untitled");
// file-names which are match with .h filter stored in filenames list
QStringList filenames;
// this list used in QStringList getCurrDirsOfFile() method
QStringList folder_list;
// iterator
QDirIterator it(folder, QStringList() << "*.h" , QDir::Files, QDirIterator::Subdirectories);
while (it.hasNext()) // if next object exist for iterate
{
// we have the file with .h extension
it.next();
// get previous folders that exist in filepath address of it.fileInfo()
folder_list = getCurrDirsOfFile(it.fileInfo(), folder);
// if folder_list contained skiped folder then reject that or appent it
if(folder_list.indexOf(QRegExp(skiped_dir)) == -1)
filenames.append(it.filePath());
}
for(int i = 0;i < filenames.count();i++)
{
QMessageBox::information(this, "", filenames.at(i));
}
}
Suppose we have bellow tree (directories) :
This program just shows :
/home/msi/Desktop/build-untitled-Desktop_Qt_5_0_2_GCC_64bit-Debug/ui_widget.h
/home/msi/Desktop/1/1.h
I know this is a bit old, but your original code was close...there's two problems. First, the iterator covers every entry and what it returns is a full path and filename, not just the entry name within the current directory. Checking for an exact match is the problem rather than for containment. The second issue is that you have two calls to "next" without a call to "hasNext" between them, so it's possible you'll have a crash.
while (dirIterator.hasNext())
{
QString fname = dirIterator.next ();
if (fname.contains (str)) continue;
filenames.append (fname);
}
Note that you may need to be more specific with the value of "str". Any filename could contain the word "folder", so you'll probably want to check for str.endsWith ("/folder") as well as str.contains ("/folder/").
If you inspect the values returned from "next", you'll better see how to compare what you're excluding to the current entry, so just tweak the string comparison to get what you want.

Getting full path from QListWidget

I have 2 listwidgets, lets call them listwidgetinput and listwidgetoutput. I have alot of files(only file name) on listwidgetinput. And i trim the file name before adding it to listwidgetinput like this it.fileName(). and i transfer the selected files to listdigetoutput like:
QList <QListWidgetItem*> items=ui->listWidgetinput->selectedItems();
for(int j=0;j<items.count();j++)
{
list= items.at(j)->text();
ui->listWidgetOutput->insertItem(j,list);
After i transfer the file can i get the path for all the files?. If Yes, how?
edit: code where whole path is available.
QString Dir, Type;
QStringList Files;
Qlistwidget wid
if (index==0)
{
Dir.append(C:\desktop....);
type.append(".txt")
wid = ui->listwidgetinput_txt;
}
if (index ==1)
{
Dir.append(C:\desktop....);
type.append(".doc")
wid = ui->listwidgetinput_doc
}
QDirIterator it(Dir, QStringList() << Type, QDir::Files, QDirIterator::Subdirectories);
while (it.hasNext())
{
it.next();
Files.append(it.fileName());
}
wid->additems(Files);
Use QListWidgetItem::setData() to pass additional "invisible" properties like the full path when creating the item:
auto item = new QListWidgetItem;
item->setText(fileInfo.fileName());
item->setData(Qt::UserRole, fileInfo.absoluteFilePath());
...
Later you can retrieve it via QListWidgetItem::data():
const auto fullPath = item->data(Qt::UserRole).toString();

How to iterate through only certain type of files using QDirIterator

I want to iterate through only .xml files (all files under selected folder and its sub-directories), is that possible with QDirIterator?
QDirIterator iter( rootDir, QDir::Files | QDir::Dirs | QDir::NoDotAndDotDot);
QString fileName;
while(iter.hasNext() )
{
qDebug() << iter.next();
fileName = iter.fileName();
// now I have to check if fileName is indeed a .xml extension
}
As can be seen in code above, if my iterator can jump to .xml files only than I don't have to check for file extension in the loop..is it possible?
One of the constructors of QDirIterator allows a nameFilters argument:
QDirIterator::QDirIterator ( const QString & path, const QStringList & nameFilters, QDir::Filters filters = QDir::NoFilter, IteratorFlags flags = NoIteratorFlags )
Constructs a QDirIterator that can iterate over path, using
nameFilters and filters.
The nameFilters argument is not properly documented, but there is a good chance it works like in QDir::setNameFilters.
QDirIterator it(rootdir, QStringList() << "*.xml", QDir::Files, QDirIterator::Subdirectories);
while (it.hasNext()) {
qDebug() << it.next();
}

Delete all files in a directory

I need to delete all files in a directory using Qt.
All of the files in the directory will have the extension ".txt".
I don't want to delete the directory itself.
Does anyone know how I can do this? I've looked at QDir but am having no luck.
Bjorns Answer tweeked to not loop forever
QString path = "whatever";
QDir dir(path);
dir.setNameFilters(QStringList() << "*.*");
dir.setFilter(QDir::Files);
foreach(QString dirFile, dir.entryList())
{
dir.remove(dirFile);
}
Ignoring the txt extension filtering... Here's a way to delete everything in the folder, including non-empty sub directories:
In QT5, you can use removeRecursively() on dirs. Unfortunately, that removes the whole directory - rather than just emptying it. Here is basic a function to just clear a directory's contents.
void clearDir( const QString path )
{
QDir dir( path );
dir.setFilter( QDir::NoDotAndDotDot | QDir::Files );
foreach( QString dirItem, dir.entryList() )
dir.remove( dirItem );
dir.setFilter( QDir::NoDotAndDotDot | QDir::Dirs );
foreach( QString dirItem, dir.entryList() )
{
QDir subDir( dir.absoluteFilePath( dirItem ) );
subDir.removeRecursively();
}
}
Alternatively, you could use removeRecursively() on the directory you want to clear (which would remove it altogether). Then, recreate it with the same name after that... The effect would be the same, but with fewer lines of code. This more verbose function, however, provides more potential for detailed exception handling to be added if desired, e.g. detecting access violations on specific files / folders...
Call QDir::entryList(QDir::Files) to get a list of all the files in the directory, and then for each fileName that ends in ".txt" call QDir::remove(fileName) to delete the file.
You started in a good way, look at entryList and of course pass the namefilter you want.
To improve on #user3191791's answer (which removes all files and directories), this answer:
Modernises the code with a range-based for loop
Provides optional error checking
The code:
struct FileOperationResult
{
bool success;
QString errorMessage;
};
FileOperationResult removeDirContents(const QString &dirPath)
{
QDir dir(dirPath);
dir.setFilter(QDir::NoDotAndDotDot | QDir::Files);
const QStringList files = dir.entryList();
for (const QString &fileName : files) {
if (!dir.remove(fileName)) {
const QString failureMessage = QString::fromUtf8(
"Failed to remove file %1 from %2").arg(fileName, dirPath);
return { false, failureMessage };
}
}
dir.setFilter(QDir::NoDotAndDotDot | QDir::Dirs);
const QStringList dirs = dir.entryList();
for (const QString &dirName : dirs) {
QDir subDir(dir.absoluteFilePath(dirName));
if (!subDir.removeRecursively()) {
const QString failureMessage = QString::fromUtf8(
"Failed to recursively remove directory %1 from %2").arg(dirName, dirPath);
return { false, failureMessage };
}
}
return { true, QString() };
}
Usage:
const FileOperationResult removeResult = removeDirContents(path);
if (!removeResult.success)
qWarning() << removeResult.errorMessage;
This is how I would do it:
QString path = "name-of-directory";
QDir dir(path);
dir.setNameFilters(QStringList() << "*.txt");
dir.setFilters(QDir::Files);
while(dir.entryList().size() > 0){
dir.remove(dir.entryList().first());
}
Other variant of rreeves's code:
QDir dir("/path/to/file");
dir.setNameFilters(QStringList() << "*.*");
dir.setFilter(QDir::Files);
for(const QString &dirFile: dir.entryList()) {
dir.remove(dirFile);
}
You can achieve this without using Qt: to do so, opendir, readdir, unlink, and even rmdir will be your friends. It's easy to use, just browse the man pages ;).

Failing to Parse System Directories using QFileSystemModel for different Audio/Video Files

I am newbie in Qt and have started working on it from past few days. I have come across a situation where I need to parse/traverse my System Directories and search for files like .mp3, .mpg etc.
Well in my .ui I have 2 Tree Views, one towards right and other towards left. The left one displays System Directories and on selecting the drive, the right treeview displays the .mp3, .mpg etc files.
Here is my Cpp class:
//Gets called on App startup
void PanasonicViewer::onCamStartup()
{
m_SystemModel = new QFileSystemModel(this);
m_SystemListViewModel = new QFileSystemModel(this);
m_SystemModel->setRootPath(QDir::homePath());
ui->DriveView->setModel(m_SystemModel); //Left side TreeView
ui->DriveListView->setModel(m_SystemListViewModel); //Right Side TreeView
// regard less how many columns you can do this using for:
for(int nCount = 1; nCount < m_SystemModel->columnCount(); nCount++)
ui->DriveView->hideColumn(nCount);
}
void PanasonicViewer::on_DriveView_clicked(const QModelIndex &index)
{
QStringList sDriveFilters;
QString sPath = m_SystemModel->fileInfo(index).absoluteFilePath();
ui->DriveListView->setRootIndex(m_SystemListViewModel->setRootPath(sPath));
m_SystemModel->setRootPath(QDir::homePath());
m_SystemModel->setFilter(QDir::NoDotAndDotDot | QDir::AllDirs );
m_SystemListViewModel->setFilter( QDir::Files | QDir::NoDotAndDotDot );
sDriveFilters << "*.aac" << "*.wmv" << "*.avi" << "*.mpeg" << "*.mov" << "*.3gp" << "*.flv" << "*.mp3" ;
m_SystemListViewModel->setNameFilters(sDriveFilters);
m_SystemListViewModel->setNameFilterDisables(false);
}
Here is my .h file:
QFileSystemModel *m_SystemModel;
QFileSystemModel *m_SystemListViewModel;
When I run my app, it displays only those audio/video files which are present in the Drive. I mean it doesn’t parse the folders present inside the drive where audio/video files are present. It just displays the files which are in the drive and not subdirectories. How can i achieve it? :)
"Computer do what you said, but not you want"
If you want get all files present inside the drive -- you must recursively go thru all your directories.
Here is similar and answered question: SO question.
By one of answers, in Qt you can use this approach:
#include <qapplication.h>
#include <qdir.h>
#include <iostream>
int main( int argc, char **argv )
{
QApplication a( argc, argv );
QDir currentDir = QDir::current();
currentDir.setFilter( QDir::Dirs );
QStringList entries = currentDir.entryList();
for( QStringList::ConstIterator entry=entries.begin(); entry!=entries.end(); ++entry)
{
std::cout << *entry << std::endl;
}
return 0;
}