title: postgresql: offset + limit gets to be very slow
body: <p>I have a table <code>tmp_drop_ids</code> with one column, <code>id</code>, and 3.3 million entries. I want to iterate over the table, doing something with every 200 entries. I have this code:</p>
  
  <pre><code>LIMIT = 200
  for offset in xrange(0, drop_count+LIMIT, LIMIT):
      print "Making tmp table with ids %s to %s/%s" % (offset, offset+LIMIT, drop_count)
      query = """DROP TABLE IF EXISTS tmp_cur_drop_ids; CREATE TABLE tmp_cur_drop_ids AS
      SELECT id FROM tmp_drop_ids ORDER BY id OFFSET %s LIMIT %s;""" % (offset, LIMIT)
      cursor.execute(query)
  </code></pre>
  
  <p>This runs fine, at first, (~0.15s to generate the tmp table), but it will slow down occasionally, e.g. around 300k tickets it started taking 11-12 seconds to generate this tmp table, and again around 400k. It basically seems unreliable. </p>
  
  <p>I will use those ids in other queries so I figured the best place to have them was in a tmp table. Is there any better way to iterate through results like this?</p>