I have a QStandardItemModel with several 100,000 records of data, and a QSortFilterProxyModel on top of it for filtering and sorting capabilities. I want to remove a substantial number of records, say 100,000, based on the value of one of the columns.
The current implementation iterates over the source model, tests for the value in the appropriate column, and calls removeRow. This turns out to be an extremely slow approach, I don't know why (I've already turned off the signalling of the source model and the sortfilterproxymodel).
What is a more efficient approach?
Can the QSortFilterProxyModel help, e.g. by creating a selection of records to be deleted, and using removeRows?
Thanks,
Andreas
QAbstractItemModel::removeRows() is a candidate, provided that the rows are contiguous. If the model is sorted by the column you are using to do the removal test, then you should be able to use this.
The more effecient approach would be implementing your own model with QAbstractItemModel interface instead of using QStandardItemModel. Then you can build custom indexes which will help you increase performance while deleting items.
Related
I have a problem I'm trying to solve but I'm at a stand still due to the fact that I'm in the process of learning Qt, which in turn is causing doubts as to what's the 'Qt' way of solving the problem. Whilst being the most efficient in term of time complexity. So I read a file line by line ( file qty ranging between 10-2000,000). At the moment my approach is to dump ever line to a QVector.
Qvector <QString> lines;
lines.append("id,name,type");
lines.append("1,James,A");
lines.append("2,Mark,B");
lines.append("3,Ryan,A");
Assuming the above structure I would like to give the user with three views that present the data based on the type field. The data is comma delimited in its original form. My question is what's the most elegant and possibly efficient way to achieve this ?
Note: For visual aid , the end result kind of emulates Microsoft access. So there will be the list of tables on the left side.In my case these table names will be the value of the grouping field (A,B). And when I switch between those two list items the central view (a table) will refill to contain the particular groups data.
Should I split the data into x amount of structures ? Or would that cause unnecessary overhead ?
Would really appreciate any help
In the end, you'll want to have some sort of a data model that implements QAbstractItemModel that exposes the data, and one or more views connected to it to display it.
If the data doesn't have to be editable, you could implement a custom table model derived from QAbstractTableModel that maps the file in memory (using QFile::map), and incrementally parses it on the fly (implement canFetchMore and fetchMore).
If the data is to be editable, you might be best off throwing it all into a temporary sqlite table as you parse the file, attaching a QSqlTableModel to it, and attaching some views to it.
When the user wants to save the changes, you simply iterate over the model and dump it out to a text file.
I have a QTableWidget that can display a huge number of elements (like, 20000 elements). Displaying per se and scrolling up and down works find, however, populating the widget works extremely slowly. I found out that constructing the QVector of elements (strings) that are displayed works very fast, however, inserting the elements into the QTableWidget is very slow.
I need to implement filtration over the elements, so if the user filters half of the elements out with a wildcard, it's still necessary to clean the QTreeWidget and insert 10000 elements back (or hide 10000 elements which is equally slow). Reasonably fast performance is critical here because the user can't wait for several minutes every time he presses a button.
Valgrind doesn't help much as apparently much of the resources are eaten by some implicitly called functions, particularly QHeaderView::sectionSize() and QHeaderView::isSectionHidden()
Migrate Your code to model-view pattern.
Create a model (subclass QStandardItemModel) and place all Your data there.
Display all the data in QTableView, ensure everything is OK
Now, You can use QSortFilterModel model for fast data-filtering, or you can subclass QProxyModel for more complex filters.
I have a QTableView thats using a model i made which extends the QAbstractTableModel. The model uses a
QList< QVector<QString> * >
as the collection. The table view is used to display logging messages from the application so the collection will eventually get extremely large... After it inserts a few thousand rows i notice the table view starts to slow down a lot and eventually the view freezes for a few seconds before refreshing.. Is it the type of collection im using thats making it slow down soo much? Is there a better way to store the data thats being inserted? Does the QTableView support a large amount of data?
Edit
Posted code on Qt forumn:
http://www.qtforum.org/article/37326/qttableview-slows-down-when-a-lot-of-data-is-inserted.html
I have successfully used QTableView displaying ~10000 rows so QTableView is capable of supporting it but your collection leaves to be desired.
QList is very expensive to insert in the middle since you have to reallocate everything sitting below the index you are trying to insert at granted you are only shifting pointers around but still.
Normally for data storage I would use an std::vector< data_struct * > rather then using vectors of strings. QVariant is quite capable of presenting integers and other types so you don't have to do the conversion up front.
The best suggestion that I can come up with would be to run gprof or a similar tool to time where exactly you are spending time and then address that particular piece.
Greetings,
I've been writing some nasty code to support the undo/redo of deletion of an arbitrary set of objects from my model. I feel like I'm going about this correctly, as all the other mutators (adding/copy-pasting) are subsets of this functionality.
The code is nastier than it needs to me, mostly because the only way to mutate the model involves calling beginInsertRows/beginRemoveRows and removing the rows in a range (just doing 1 row at a time, no need to optimize "neighbors" into a single call yet)
The problem with beginInsertRows/beginRemoveRows is that removal of a row could affect another QModelIndex (say, one cached in a list). For instance:
ParentObj
->ChildObj1
->ChildObj2
->ChildObj3
Say I select ChildObj1 and ChildObj3 and delete them, if I remove ChildObj1 first I've changed ChildObj3's QModelIndex (row is now different). Similar issues occur if I delete a parent object (but I've fixed this by "pruning" children from the list of objects).
Here are the ways I've thought of working around this interface limitation, but I thought I'd ask for a better one before forging ahead:
Move "backwards", assuming a provided list of QModelIndices is orderered from top to bottom just go from bottom up. This really requires sorting to be reliable, and the sort would probably be something naive and slow (maybe there's a smart way of sorting a collection of QModelIndexes? Or does QItemSelectionModel provide good (ordered) lists?)
Update other QModelIndeces each time an object is removed/added (can't think of a non-naive solution, search the list, get new QModelIndeces where needed)
Since updating the actual data is easy, just update the data and rebuild the model. This seems grotesque, and I can imagine it getting quite slow with large sets of data.
Those are the ideas I've got currently. I'm working on option 1 right now.
Regards,
Dan O
Think of beginRemoveRows/endRemoveRows, etc. as methods to ask the QAbstractItemModel base class to fix up your persistent model indexes for you instead of just a way of updating views, and try not to confuse the QAbstractItemModel base class in its work on those indexes. Check out http://labs.trolltech.com/page/Projects/Itemview/Modeltest to exercise your model and see if you are keeping the QAbstractItemModel base class happy.
Where QPersistentModelIndex does not help is if you want to keep your undo/redo data outside of the model. I built a model that is heavily edited, and I did not want to try keeping everything in the model. I store the undo/redo data on the undo stack. The problem is that if you edit a column, storing the persistent index of that column on the undo stack, and then delete the row holding that column, the column's persistent index becomes invalid.
What I do is keep both a persistent model index, and a "historical" regular QModelIndex. When it's time to undo/redo, I check if the persistent index has become invalid. If it has, I pass the historical QModelIndex to a special method of my model to ask it to recreate the index, based on the row, column, and internalPointer. Since all of my edits are on the undo stack, by the time I've backed up to that column edit on the undo stack, the row is sure to be there in the model. I keep enough state in the internalPointer to recreate the original index.
I would consider using a "all-data" model and a filter proxy model with the data model. Data would only be added to the all-data model, and never removed. That way, you could store your undo/redo information with references to that model. (May I suggest QPersistentModelIndex?). Your data model could also track what should be shown, somehow. The filter model would then only return information for the items that should be shown at a given time.
I am trying to create a QTreeView which displays some sorted information. To do this I use a QSortFilterProxyModel between the view and my model.
The problem is that I want to limit the number of rows to the first n rows (after sorting). The filter function from the model receives the original sourceRow so I cannot use it.
I've tried chaining two QSortFilterProxyModel: the first for the sorting and the second for the filtering. But it seems that the second proxymodel(filtering) doesn't receive the rows sorted....
Is there another way to do it?
Has anyone use this technique(chaining of 2 proxy models) and it works?
thank you
EDIT:
I've tried with the rowCount and it doesn't work.
I've also tried to chain 2 proxy models but the problem is that the view calls the sort function for the model it receives. So if the first proxy sorts and the second filters the sort will be called on the filter model and the data won't be sorted.
EDIT2: I've looked into the qt source code and the filtering is done before sorting, so in the filterAcceptsRow() I don't know any sorting order.
Just out of curiousity, have you tried overriding the rowCount method and just return 25 (or whatever n is in your case)? It might be as simple as that... well, if you'll always have at least n items.
Otherwise, you could try chaining the models. I don't know why it wouldn't work, but I've never tried something like it myself.
After trying a number of overcomplicated ways to solve this I've done a small hack for my problem: after I insert/remove a row I call setRowHidden to hide the first n rows.
This is not the most elegant solution and is particular for my needs, but I am unable to find a better alternative.
I like to mention that on gtk, because the filter and the sort proxy models are separated, this can be done fairly easy.
I'm still hoping someone can provide a better solution to this.