title: QListView with millions of items slow with keyboard
body: <p>I'm using a <code>QListView</code> with a custom model derived from <code>QAbstractItemModel</code>. I have on the order of millions of items. I have called <code>listView-&gt;setUniformItemSizes(true)</code> to prevent a bunch of layout logic from being called when I'm adding items to the model. So far, everything works as expected.</p>
  
  <p>The problem is that <strong>using the keyboard to navigate the list is slow</strong>. If I select an item in the list, then press up/down, the selection moves fast <em>until</em> the selection needs to scroll the list. Then it becomes extremely laggy. Pressing page-up or page-down is also very laggy. The problem seems to be when an item is selected (aka the "current item") with the keyboard and the list is also scrolled up/down.</p>
  
  <p>If I use the mouse, navigating the list is fast. I can use the mouse wheel, which is fast. I can drag the scroll bar up/down as fast as I want--from the top of the list to the bottom--and the list view updates wickedly fast.</p>
  
  <p>Any ideas on why the combination of changing selections and scrolling the list is so slow? Is there a viable work-around?</p>
  
  <h2>Update 9/9/15</h2>
  
  <p>In order to better illustrate the issue, I'm providing amplifying information in this update.</p>
  
  <h1>Performance Issues with KEYBOARD + SCROLLING</h1>
  
  <p>This is mostly a performance question, although it does tie in with the user experience (UX) somewhat. Check out what happens as I use the keyboard to scroll through a <code>QListView</code>:</p>
  
  <p><a href="https://i.stack.imgur.com/BVIhM.gif" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/BVIhM.gif" alt="Slow Scrolling Issue"></a></p>
  
  <p><strong>Notice the slow-down near the bottom?</strong> This is the focal point of my question. Let me explain how I am navigating the list.</p>
  
  <p><strong>Explanation:</strong></p>
  
  <ol>
  <li>Starting at the top, the first item in the list is selected.</li>
  <li>Pressing and <em>holding</em> the down arrow key, the current item (selection) is changed to the next item.</li>
  <li>Changing selection is fast for all of the items that are currently in view.</li>
  <li>As soon as the list needs to bring the next item into view, the selection rate slows down significantly.</li>
  </ol>
  
  <p>I expect that the list should be able to scroll as fast as the typematic rate of my keyboard--in other words, the time it takes to select the next item should not slow down when the list is scrolled.</p>
  
  <h1>Fast Scrolling with MOUSE</h1>
  
  <p>Here's what it looks like when I use the mouse:</p>
  
  <p><a href="https://i.stack.imgur.com/e72mu.gif" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/e72mu.gif" alt="Fast Mouse Navigation"></a></p>
  
  <p><strong>Explanation:</strong></p>
  
  <ol>
  <li>Using the mouse, I select the scroll bar handle.</li>
  <li>Quickly dragging the scroll bar handle up and down, the list is scrolled accordingly.</li>
  <li>All movements are extremely fast.</li>
  <li>Note that <strong>no selections are being made</strong>.</li>
  </ol>
  
  <p>This proves two main points:</p>
  
  <ol>
  <li><p><strong>The model is not the problem.</strong> As you can see, the model has no problem whatsoever performance-wise. It can deliver the elements faster than they can be displayed.</p></li>
  <li><p><strong>Performance is degraded when selecting AND scrolling.</strong> The "perfect storm" of selecting and scrolling (as illustrated by using the keyboard to navigate through the list) causes the slowdown. As a result, I surmise that Qt is somehow doing a lot of processing when <em>selections are being made during scrolling</em> that aren't normally performed.</p></li>
  </ol>
  
  <h1>Non-Qt Implementation is FAST</h1>
  
  <p>I want to point out that my issue seems to be specific to Qt.</p>
  
  <p><strong>I have already implemented this type of thing before using a different framework.</strong> What I am trying to do is within the scope of model-view theory. I can do exactly what I am describing at blazing fast speeds using <a href="http://learn.juce.com/doc/classListBoxModel.php" rel="nofollow noreferrer" title="juce::ListBoxModel">juce::ListBoxModel</a> with a <a href="http://learn.juce.com/doc/classListBox.php" rel="nofollow noreferrer" title="juce::ListBox">juce::ListBox</a>. It's stupid fast (plus, there's no need to create a duplicate index such as a <code>QModelIndex</code> for every single item when each item already has a unique index). I get that Qt needs a <code>QModelIndex</code> for each item for its model-view architecture, and although I don't like the overhead cost, I think I get the rational and I can live with it. Either way, I don't suspect that these <code>QModelIndex</code>es are what is causing my performance slow-down.</p>
  
  <p>With a JUCE implementation, I can even use the page-up &amp; page-down keys to navigate the list, and it just blazes through the list. Using the Qt <code>QListView</code> implementation, it chugs along and is laggy, even with a release build.</p>
  
  <p>A model-view implementation using the <a href="http://www.juce.com/" rel="nofollow noreferrer" title="JUCE framework">JUCE framework</a> is extremely fast. <strong>Why is the Qt <code>QListView</code> implementation such a dog?!</strong></p>
  
  <h1>Motivating Example</h1>
  
  <p>Is it hard to imagine why you'd need so many items in a list view? Well, we've all seen this kind of thing before:</p>
  
  <p><a href="https://i.stack.imgur.com/25snC.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/25snC.png" alt="Visual Studio Index"></a></p>
  
  <p>This is the Visual Studio Help Viewer index. Now, I haven't counted all of the items--but I think we'd agree that there are a lot of them! Of course to make this list "useful," they added a filter box that narrows down what is in the list view according to an input string. There aren't any tricks here. It's all practical, real-world stuff we've all seen for decades in desktop applications.</p>
  
  <p>But are there <em>millions</em> of items? I'm not sure it matters. Even if there were "only" 150k items (which is roughly accurate based on some crude measurements), it's easy to point out that you have to do something to make it useable--which is what a filter will do for you. </p>
  
  <p>My specific example uses <a href="http://sourceforge.net/projects/germandict/files/" rel="nofollow noreferrer" title="a list of German words as a plain text file with slightly more than 1.7 million entries &#40;including inflected forms&#41;.">a list of German words as a plain text file with slightly more than 1.7 million entries (including inflected forms).</a> This is probably only a partial (but still significant) <em>sample</em> of words from the German <a href="https://en.wikipedia.org/wiki/Text_corpus" rel="nofollow noreferrer" title="text corpus">text corpus</a> that was used to assemble this list. For linguistic study, this is a reasonable use case.</p>
  
  <p>Concerns about improving the UX (user experience) or filtering are great design goals, but they are out of the scope of this question (I'll certainly address them later in the project).</p>
  
  <h1>Code</h1>
  
  <p>Want a code example? You got it! I'm not sure how useful it will be; it's as vanilla as it gets (about 75% boilerplate), but I suppose it will provide some context. I realize that I'm using a <code>QStringList</code> and that there is a <code>QStringListModel</code> for this, but the <code>QStringList</code> that I'm using to hold the data is a placeholder--the model will eventually be somewhat more complicated, so in the end, I need to use a custom model derived from <code>QAbstractItemModel</code>.</p>
  
  <pre><code>//
  // wordlistmodel.h ///////////////////////////////////////
  //
  class WordListModel : public QAbstractItemModel
  {
      Q_OBJECT
  public:
      WordListModel(QObject* parent = 0);
  
      virtual QModelIndex index(int row, int column, const QModelIndex&amp; parent = QModelIndex()) const;
      virtual QModelIndex parent(const QModelIndex&amp; index) const;
      virtual int rowCount(const QModelIndex&amp; parent = QModelIndex()) const;
      virtual int columnCount(const QModelIndex &amp; parent = QModelIndex()) const;
      virtual QVariant data(const QModelIndex&amp; index, int role = Qt::DisplayRole) const;
  
  public slots:
      void loadWords();
  
  signals:
      void wordAdded();
  
  private:
      // TODO: this is a temp backing store for the data
      QStringList wordList;
  };
  
  
  //
  // wordlistmodel.cpp ///////////////////////////////////////
  //
  WordListModel::WordListModel(QObject* parent) :
      QAbstractItemModel(parent)
  {
      wordList.reserve(1605572 + 50); // testing purposes only!
  }
  
  void WordListModel::loadWords()
  {
      // load items from file or database
  
      // Due to taking Kuba Ober's advice to call setUniformItemSizes(true),
      // loading is fast. I'm not using a background thread to do
      // loading because I was trying to visually benchmark loading speed.
      // Besides, I am going to use a completely different method using
      // an in-memory file or a database, so optimizing this loading by
      // putting it in a background thread would obfuscate things.
      // Loading isn't a problem or the point of my question; it takes
      // less than a second to load all 1.6 million items.
  
      QFile file("german.dic");
      if (!file.exists() || !file.open(QIODevice::ReadOnly))
      {
          QMessageBox::critical(
              0,
              QString("File error"),
              "Unable to open " + file.fileName() + ". Make sure it can be located in " +
                  QDir::currentPath()
          );
      }
      else
      {
          QTextStream stream(&amp;file);
          int numRowsBefore = wordList.size();
          int row = 0;
          while (!stream.atEnd())
          {
              // This works for testing, but it's not optimal.
              // My real solution will use a completely different
              // backing store (memory mapped file or database),
              // so I'm not going to put the gory details here.
              wordList.append(stream.readLine());    
  
              ++row;
  
              if (row % 10000 == 0)
              {
                  // visual benchmark to see how fast items
                  // can be loaded. Don't do this in real code;
                  // this is a hack. I know.
                  emit wordAdded();
                  QApplication::processEvents();
              }
          }
  
          if (row &gt; 0)
          {
              // update final word count
              emit wordAdded();
              QApplication::processEvents();
  
              // It's dumb that I need to know how many items I
              // am adding *before* calling beginInsertRows().
              // So my begin/end block is empty because I don't know
              // in advance how many items I have, and I don't want
              // to pre-process the list just to count the number
              // of items. But, this gets the job done.
              beginInsertRows(QModelIndex(), numRowsBefore, numRowsBefore + row - 1);
              endInsertRows();
          }
      }
  }
  
  QModelIndex WordListModel::index(int row, int column, const QModelIndex&amp; parent) const
  {
      if (row &lt; 0 || column &lt; 0)
          return QModelIndex();
      else
          return createIndex(row, column);
  }
  
  QModelIndex WordListModel::parent(const QModelIndex&amp; index) const
  {
      return QModelIndex(); // this is used as the parent index
  }
  
  int WordListModel::rowCount(const QModelIndex&amp; parent) const
  {
      return wordList.size();
  }
  
  int WordListModel::columnCount(const QModelIndex&amp; parent) const
  {
      return 1; // it's a list
  }
  
  QVariant WordListModel::data(const QModelIndex&amp; index, int role) const
  {
      if (!index.isValid())
      {
          return QVariant();
      }    
      else if (role == Qt::DisplayRole)
      {
          return wordList.at(index.row());
      }
      else
      {    
          return QVariant();
      }
  }
  
  
  //
  // mainwindow.h ///////////////////////////////////////
  //    
  class MainWindow : public QMainWindow
  {
      Q_OBJECT
  
  public:
      explicit MainWindow(QWidget *parent = 0);
      ~MainWindow();
  
  public slots:
      void updateWordCount();
  
  private:
      Ui::MainWindow *ui;
      WordListModel* wordListModel;
  };
  
  //
  // mainwindow.cpp ///////////////////////////////////////
  //
  MainWindow::MainWindow(QWidget *parent) :
      QMainWindow(parent),
      ui(new Ui::MainWindow)
  {
      ui-&gt;setupUi(this);
      ui-&gt;listView-&gt;setModel(wordListModel = new WordListModel(this));
  
      // this saves TONS of time during loading,
      // but selecting/scrolling performance wasn't improved
      ui-&gt;listView-&gt;setUniformItemSizes(true);
  
      // these didn't help selecting/scrolling performance...
      //ui-&gt;listView-&gt;setLayoutMode(QListView::Batched);
      //ui-&gt;listView-&gt;setBatchSize(100);
  
      connect(
          ui-&gt;pushButtonLoadWords,
          SIGNAL(clicked(bool)),
          wordListModel,
          SLOT(loadWords())
      );
  
      connect(
          wordListModel,
          SIGNAL(wordAdded()),
          this,
          SLOT(updateWordCount())
      );
  }
  
  MainWindow::~MainWindow()
  {
      delete ui;
  }
  
  void MainWindow::updateWordCount()
  {
      QString wordCount;
      wordCount.setNum(wordListModel-&gt;rowCount());
      ui-&gt;labelNumWordsLoaded-&gt;setText(wordCount);
  }
  </code></pre>
  
  <p>As noted, I've already reviewed and taken Kuba Ober's advice:</p>
  
  <p><a href="https://stackoverflow.com/a/18579171/2452084">QListView takes too long to update when given 100k items</a></p>
  
  <p><strong>My question is not a duplicate of that question!</strong> In the other question, the OP was asking about <em>loading</em> speed, which as I've noted in my code above, is not a problem due to the call to <code>setUniformItemSizes(true)</code>.</p>
  
  <h1>Summary Questions</h1>
  
  <ol>
  <li>Why is navigating a <code>QListView</code> (with millions of items in the model) using the keyboard so slow when the list is scrolled?</li>
  <li>Why does the combination of <em>selecting and scrolling</em> items cause a slow-down?</li>
  <li>Are there any implementation details that I am missing, or have I reached a performance threshold for <code>QListView</code>?</li>
  </ol>