title: Python shelve OutOfMemory error
body: <p>I have some data stored in a DB that I want to process. DB access is painfully slow, so I decided to load all data in a dictionary before any processing. However, due to the huge size of the data stored, I get an out of memory error (I see more than 2 gigs being used). So I decided to use a disk data structure, and found out that using shelve is an option. Here's what I do (pseudo python code)</p>
  
  <pre><code>def loadData():
      if (#dict exists on disk):
          d = shelve.open(name)
          return d
      else:
          d = shelve.open(name, writeback=True)
          #access DB and write data to dict
          # d[key] = value 
          # or for mutable values
          # oldValue = d[key]
          # newValue = f(oldValue)
          # d[key] = newValue 
          d.close()
          d = shelve.open(name, writeback=True)
          return d
  </code></pre>
  
  <p>I have a couple of questions,</p>
  
  <p>1) Do I really need the writeBack=True? What does it do?</p>
  
  <p>2) I still get an OutofMemory exception, since I do not exercise any control over when the data is being written to disk. How do I do that? I tried doing a sync() every few iterations but that didn't help either.</p>
  
  <p>Thanks!</p>