title: Speeding up iteration over a large QuerySet in Django
body: <p>I'm trying my hand at the <a href="https://developer.riotgames.com/discussion/riot-games-api/show/bX8Z86bm" rel="nofollow noreferrer">Riot API challenge</a>, and I'm trying to use Django as a backend hosted on <a href="http://PythonAnywhere.com" rel="nofollow noreferrer">PythonAnywhere.com</a>.</p>
  
  <p>I have setup a database which uses a structure similar to the one below</p>
  
  <pre><code>class MatchDetails(models.Model):
      # Data fields
  
  class Participant(models.Model):
      match = models.ForeignKey(MatchDetails)
      # Data fields
  
  class Timeline(models.Model):
      participant = models.ForeignKey(Participant)
      # Data fields
  
  # More fields, most with MatchDetails as foreign key.
  </code></pre>
  
  <p>I have written a function which retrieves and stores data, and I now have almost 40 000 games stored, with 10 participants in each game. My goal is to extract some statistics from this data, and I basically do something like:</p>
  
  <pre><code>allMatches = MatchDetails.objects.all()
  for m in allMatches:
      participants = m.participant_set.all()
      for p in participants:
          # Increment some values
  # save the result to the database
  </code></pre>
  
  <p>Currently it takes a little more than 2 hours.</p>
  
  <blockquote>
    <p>2015-04-11 03:47:35 -- Completed task, took 7942.00 seconds, return
    code was 0.</p>
  </blockquote>
  
  <p>This is a ridiculous amount of time, isn't it? Is there some way for me to speed it up?</p>
  
  <p>I've tried to use iterator, and I also tried using .value_list and .all.values() to iterate over, but I am unable to get objects related through foreign key this way.</p>
  
  <p><a href="https://stackoverflow.com/questions/5656238/how-do-i-speed-up-iteration-of-large-datasets-in-django">How do I speed up iteration of a large dataset in Django</a></p>
  
  <p>Is there any way for me to access my foreign key objects when using value_list? Or is there anything else I can do to speed it up? Any pointers would be appreciated. </p>
  
  <p>Thanks for reading!</p>