Very slow QTableView multi-select on PySide6 for large dataset

[M.Bat]

Hello,
I am implementing a large dataset viewer using PySide6 6.9.0 and Python 3.11.
I used a derived QTableView associated with a derived QAbstractTableModel.
It works well and is responsive as long as I do not have more than ~100 000 rows (I only have less than a dozen columns).
However it become very laggy and slow as soon as I have more than that and if I use ctrl+a for a selectAll() call, even with optimization such as setting setUpdatesEnabled(False).
Here is a working POC that will trigger a selectAll() equivalent (using select on all rows/columns), I use a somewhat equivalent structure in my own code so if anyone could find what I am doing wrong here I could replicate that on my end and I would be very grateful !
Below the code you can find the ouput of a cProfile on this snippet, which shows a long time spent in the flags method, which is mandatory, not sure how I could optimize that one..
Thank you

import sys
from PySide6.QtWidgets import QApplication, QMainWindow, QTableView, QVBoxLayout, QWidget
from PySide6.QtCore import QAbstractTableModel, Qt, QModelIndex, QItemSelectionModel, QItemSelection
import random

class LargeDatasetModel(QAbstractTableModel):
    def __init__(self, row_count, column_count, parent=None):
        super().__init__(parent)
        self.row_count = row_count
        self.column_count = column_count
        
        self._data = [
            [random.randint(0, 100) for _ in range(column_count)] for _ in range(row_count)
        ]

    def rowCount(self, parent=QModelIndex()):
        return self.row_count

    def columnCount(self, parent=QModelIndex()):
        return self.column_count

    def data(self, index, role=Qt.DisplayRole):
        if role == Qt.DisplayRole:
            row = index.row()
            column = index.column()
            return self._data[row][column]
        return None

    def headerData(self, section, orientation, role):
        if role == Qt.DisplayRole:
            if orientation == Qt.Horizontal:
                return f"Column {section}"
            else:
                return f"Row {section}"
        return None

    def flags(self, index):
        return Qt.ItemIsSelectable | Qt.ItemIsEnabled

class LargeDatasetTableView(QTableView):
    def __init__(self, parent=None):
        super().__init__(parent)

    def keyPressEvent(self, event):
        keycode = event.key()
        if keycode == ord('A'):
            self.select_all()
            return
        super().keyPressEvent(event)

    def select_all(self):
        selection_model = self.selectionModel()
        self.setUpdatesEnabled(False)
        
        model = self.model()
        row_count = model.rowCount()
        column_count = model.columnCount()
        selection_model.clearSelection()

        top_left_index = model.index(0, 0)
        bottom_right_index = model.index(row_count - 1, column_count - 1)
        
        selection_range = QItemSelection(top_left_index, bottom_right_index)
        selection_model.select(selection_range, QItemSelectionModel.Select)
        
        self.setUpdatesEnabled(True)

class MainWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("Large Dataset Viewer")

        self.table_view = LargeDatasetTableView()
        self.model = LargeDatasetModel(row_count=200000, column_count=5)
        self.table_view.setModel(self.model)
        
        layout = QVBoxLayout()
        layout.addWidget(self.table_view)

        container = QWidget()
        container.setLayout(layout)
        self.setCentralWidget(container)

if __name__ == '__main__':
    app = QApplication(sys.argv)
    window = MainWindow()
    window.show()
    app.exec()

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   36.654   36.654 {built-in method builtins.exec}
        1    0.005    0.005   36.654   36.654 <string>:1(<module>)
        1    0.026    0.026   36.649   36.649 test.py:85(main)
        1   18.124   18.124   35.058   35.058 {built-in method exec}
  2419149   12.239    0.000   15.275    0.000 test.py:37(flags)
  2419149    1.448    0.000    3.036    0.000 enum.py:1491(__or__)
        1    0.000    0.000    1.506    1.506 test.py:69(__init__)
        1    0.000    0.000    1.494    1.494 test.py:7(__init__)
        1    0.057    0.057    1.493    1.493 test.py:12(<listcomp>)
  2457871    0.845    0.000    1.478    0.000 enum.py:688(__call__)
   160000    0.178    0.000    1.437    0.000 test.py:13(<listcomp>)
   800000    0.187    0.000    1.258    0.000 random.py:358(randint)
   800000    0.514    0.000    1.072    0.000 random.py:284(randrange)
  2419150    0.699    0.000    0.920    0.000 enum.py:192(__get__)
  2457868    0.624    0.000    0.624    0.000 enum.py:1093(__new__)
   800000    0.331    0.000    0.435    0.000 random.py:235(_randbelow_with_getrandbits)
  2411789    0.242    0.000    0.242    0.000 test.py:16(rowCount)
  2412594    0.239    0.000    0.239    0.000 test.py:19(columnCount)
  2419150    0.221    0.000    0.221    0.000 enum.py:1245(value)
  2420557    0.145    0.000    0.145    0.000 {built-in method builtins.isinstance}
    47908    0.120    0.000    0.123    0.000 test.py:22(data)
  2400000    0.123    0.000    0.123    0.000 {built-in method _operator.index}
    38719    0.104    0.000    0.104    0.000 test.py:29(headerData)
  1013301    0.064    0.000    0.064    0.000 {method 'getrandbits' of '_random.Random' objects}
        1    0.056    0.056    0.058    0.058 {method 'show' of 'PySide6.QtWidgets.QWidget' objects}
   800003    0.040    0.000    0.040    0.000 {method 'bit_length' of 'int' objects}
        3    0.000    0.000    0.009    0.003 enum.py:841(_create_)
        1    0.004    0.004    0.007    0.007 {method 'setModel' of 'PySide6.QtWidgets.QTableView' objects}
        3    0.000    0.000    0.006    0.002 enum.py:485(__new__)
        1    0.005    0.005    0.005    0.005 test.py:41(__init__)
    193/4    0.000    0.000    0.005    0.001 {built-in method __new__ of type object at 0x...}
      189    0.003    0.000    0.005    0.000 enum.py:237(__set_name__)
      195    0.002    0.000    0.003    0.000 enum.py:353(__setitem__)
     6844    0.002    0.000    0.002    0.000 {method 'row' of 'PySide6.QtCore.QModelIndex' objects}
     6844    0.001    0.000    0.001    0.000 {method 'column' of 'PySide6.QtCore.QModelIndex' objects}
      200    0.000    0.000    0.001    0.000 {built-in method builtins.setattr}
      195    0.000    0.000    0.001    0.000 enum.py:78(_is_private)
      197    0.000    0.000    0.001    0.000 {built-in method builtins.delattr}
        2    0.000    0.000    0.000    0.000 test.py:44(keyPressEvent)
      189    0.000    0.000    0.000    0.000 enum.py:37(_is_descriptor)
      209    0.000    0.000    0.000    0.000 enum.py:828(__setattr__)
        1    0.000    0.000    0.000    0.000 test.py:51(select_all)
     1130    0.000    0.000    0.000    0.000 {method 'get' of 'mappingproxy' objects}
      197    0.000    0.000    0.000    0.000 enum.py:747(__delattr__)
      195    0.000    0.000    0.000    0.000 enum.py:47(_is_dunder)
      195    0.000    0.000    0.000    0.000 enum.py:58(_is_sunder)
      757    0.000    0.000    0.000    0.000 {built-in method builtins.hasattr}
      978    0.000    0.000    0.000    0.000 {built-in method builtins.len}
      195    0.000    0.000    0.000    0.000 enum.py:69(_is_internal_class)
        3    0.000    0.000    0.000    0.000 enum.py:470(__prepare__)
        9    0.000    0.000    0.000    0.000 enum.py:942(_get_mixins_)
      376    0.000    0.000    0.000    0.000 {method 'append' of 'list' objects}
        1    0.000    0.000    0.000    0.000 {method 'selectionModel' of 'PySide6.QtWidgets.QAbstractItemView' objects}
      160    0.000    0.000    0.000    0.000 {method 'startswith' of 'str' objects}
      196    0.000    0.000    0.000    0.000 {built-in method builtins.issubclass}
      193    0.000    0.000    0.000    0.000 {method 'setdefault' of 'dict' objects}
        9    0.000    0.000    0.000    0.000 enum.py:978(_find_data_type_)
      189    0.000    0.000    0.000    0.000 enum.py:234(__init__)
        3    0.000    0.000    0.000    0.000 enum.py:1006(_find_new_)
        2    0.000    0.000    0.000    0.000 {method 'setUpdatesEnabled' of 'PySide6.QtWidgets.QWidget' objects}
        2    0.000    0.000    0.000    0.000 {method 'index' of 'PySide6.QtCore.QAbstractTableModel' objects}
       12    0.000    0.000    0.000    0.000 enum.py:932(_check_for_existing_members_)
        1    0.000    0.000    0.000    0.000 {method 'setLayout' of 'PySide6.QtWidgets.QWidget' objects}
       69    0.000    0.000    0.000    0.000 {built-in method builtins.getattr}
        1    0.000    0.000    0.000    0.000 {method 'addWidget' of 'PySide6.QtWidgets.QBoxLayout' objects}
      189    0.000    0.000    0.000    0.000 enum.py:1144(__init__)
        1    0.000    0.000    0.000    0.000 {function LargeDatasetTableView.keyPressEvent at 0x...}
        1    0.000    0.000    0.000    0.000 enum.py:1376(_missing_)
        1    0.000    0.000    0.000    0.000 {method 'clearSelection' of 'PySide6.QtCore.QItemSelectionModel' objects}
        3    0.000    0.000    0.000    0.000 {method 'update' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'select' of 'PySide6.QtCore.QItemSelectionModel' objects}
        1    0.000    0.000    0.000    0.000 {method 'setCentralWidget' of 'PySide6.QtWidgets.QMainWindow' objects}
        1    0.000    0.000    0.000    0.000 enum.py:1438(<listcomp>)
        1    0.000    0.000    0.000    0.000 {method 'setWindowTitle' of 'PySide6.QtWidgets.QWidget' objects}
        3    0.000    0.000    0.000    0.000 enum.py:346(__init__)
        2    0.000    0.000    0.000    0.000 {method 'key' of 'PySide6.QtGui.QKeyEvent' objects}
        3    0.000    0.000    0.000    0.000 enum.py:1356(_iter_member_by_value_)
        3    0.000    0.000    0.000    0.000 enum.py:772(__getattr__)
        3    0.000    0.000    0.000    0.000 enum.py:964(_find_data_repr_)
        3    0.000    0.000    0.000    0.000 enum.py:116(_iter_bits_lsb)
       21    0.000    0.000    0.000    0.000 {method 'add' of 'set' objects}
        1    0.000    0.000    0.000    0.000 {method 'model' of 'PySide6.QtWidgets.QAbstractItemView' objects}
        1    0.000    0.000    0.000    0.000 enum.py:652(<listcomp>)
        3    0.000    0.000    0.000    0.000 {built-in method sys._getframe}
        6    0.000    0.000    0.000    0.000 {method 'pop' of 'dict' objects}
       10    0.000    0.000    0.000    0.000 enum.py:92(_is_single_bit)
        5    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
        3    0.000    0.000    0.000    0.000 {method 'pop' of 'set' objects}
        2    0.000    0.000    0.000    0.000 {built-in method builtins.ord}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.sorted}
        3    0.000    0.000    0.000    0.000 {method 'items' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 enum.py:794(__iter__)
        4    0.000    0.000    0.000    0.000 enum.py:798(<genexpr>)
        1    0.000    0.000    0.000    0.000 {method 'join' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'values' of 'dict' objects}

[JonB]

@M.Bat
The following may or may not be relevant to your situation.

A few months(?) ago someone using PySide6/Python found "very slow performance" on a data table. That user spent a lot of his own time investigating. It turned out (IIRC) that there were "millions" of calls to rowCount() (understandable that Qt internals need to access that a lot). You would think that would be fast, but it was not, at least from Python (I cannot recall now whether this was a Python-only issue or applied generally in C++ too). He looked at Qt source code, made a change, and sped up his application by a huge factor.

I am pretty sure he submitted his "patch" and it was accepted and rolled out in the next version of Qt6. I do not know at which version, but you do not say what version of Qt6/PySide6 you are using. If you can find the thread it might eb worth investigating.

UPDATE:

OK, the thread is https://forum.qt.io/topic/159449/qtreeview-with-lots-of-items-is-really-slow-can-it-be-optimised-or-is-something-buggy. There I think @Christian-Ehrlicher indicated that https://codereview.qt-project.org/c/qt/qtbase/+/601341 was the "patch" to fix, and that was released in Qt 6.8?

[M.Bat]

@JonB Thank you for your answer ! Indeed on the profiling we can see ~2 million calls to rowCount which can be quite consuming if the method is not optimized..
I will look into that and come back to you.

[JonB]

@M.Bat I have just updated my answer to refer you to the thread. Are you >= Qt6.8, or are you earlier?

[M.Bat]

@JonB I just updated my initial message with the version : I am in 6.9.0 so I should have that fix, I am reading the thread you linked it might contain intersting clues, thanks for that !
The huge time spent in flags does not seem suspicious to you ? (I added the output of cProfile when doing a ctrl+a in my initial message)

[JonB]

@M.Bat
Don't know, never looked at code. But the guy who got the dramatic improvement did not look at flags() and still got his hugely better performance.

On another matter: I have never understood people who put anything like 100k lines into a table view, it's going to be slow and not useful to a user. Let alone why you would then want to select all rows....

[M.Bat]

@JonB I do have the improvements he brought to Qt in the 6.8 update so there's that, however unlike him my issue was not at the loading but once the load is done, to do a ctrl+a to select all rows.. so I still have to work on that !
Regarding the many lines in a table view.. well I need that and I cannot really use another way without spending a lot of time rewriting a lot of already existing codebase so I will try to make it work.

I did managed to go from the first line to the second in the table below with a small change : I do not call the Qt enum in flags but instead I return an early memorized value of that enum.

ncalls	tottime	percall	cumtime	percall	filename:lineno(function)
4128660	2.678	0.000	7.760	0.000	test.py:38(flags)
4120819	0.396	0.000	0.396	0.000	test.py:38(flags)

In init :

self.data_flag = Qt.ItemFlag.ItemIsSelectable | Qt.ItemFlag.ItemIsEnabled

The flag method :

    def flags(self, index):
        return self.data_flag

However even though the time spent in flags went down, the app is still very laggy with a lot of rows, I am still looking for an idea if anyone think of something reading this !

[JonB]

@M.Bat
I tried your code on my (slow) Linux machine. Displaying the rows was pretty fast. I agree selecting all takes a few seconds. However even if you/the user get that far, perhaps with an improvement, what will happen then? I tried right-click on all selected, expecting context menu, and basically never got that far, it stopped responding and kept telling me it was busy and did I want to wait. Had to kill it. Would not surprise me if you have some problems. I am guessing this would be an issue from C++ too, not a Python/PySide behaviour?

Meanwhile your time improvement for the enum flags would be amusing if it were not so alarming! It seems that evaluating Qt.ItemFlag.ItemIsSelectable | Qt.ItemFlag.ItemIsEnabled from Python/PySide instead of maintaining it in a variable speeds up by a factor of 6x! In C++ that would be just a compile-time constant expression. I wonder how Python/PySide/PyQt evaluates it, does it do some "look up" for Qt.ItemFlag....? Really ought be a constant even in Python? I am surprised by so much difference. If this is indicative of other areas where Python/PySide can be slow unless you take some minor action like what you found then you may be chasing your tail trying to find such things in code. Might be nice if you could try your code for 100k things in C++ to see whether it's Qt or Python/PySide which is slow.

[jeremy_k]

@JonB said in Very slow QTableView multi-select on PySide6 for large dataset:

@M.Bat
Meanwhile your time improvement for the enum flags would be amusing if it were not so alarming! It seems that evaluating Qt.ItemFlag.ItemIsSelectable | Qt.ItemFlag.ItemIsEnabled from Python/PySide instead of maintaining it in a variable speeds up by a factor of 6x!

Taking a naive interpretation, this speedup doesn't surprise me. Qt.ItemFlag.ItemIsSelectable could be evaluated by an interpreter as:

1. Look up the the value of Qt
2. Look up the value of ItemFlag within Qt
3. Look up the value of ItemIsSelectable within ItemFlag

Repeat for ItemIsEnabled, before combining the two values with |. Perform the same process on each iteration, because any of the symbols could be pointed at a new object during runtime.

From a quick skim of the release notes, cpython is now experimenting with a jit compiler, which might improve the situation.

[JonB]

@jeremy_k
Oh I agree, and I am guessing that is what is happening. I am not a regular Python/PySide user, only occasional. At design time in the IDE I thought if you click on Qt.ItemFlag.ItemIsSelectable is takes you straight to an imported file where that is defined as a number in an enum. Which makes you (me) kind of think that can be done as a constant. But I suppose that is wishful thinking.

Assuming the slow lookup is indeed required, it makes one reflect at just how many seemingly innocuous expressions, especially Qt ones, may be dotted around the code which could be in this situation and could be optimized if the coder happens to notice ad think about them. Which the OP may wish to look into if speed is such an issue.