title: Shelve raises out of overflow pages
body: <p>I have a lot of data that I read from a XML file, but only a part of it will be saved in the database later.</p>
  
  <p>The XML is structured like that:</p>
  
  <pre><code>&lt;element id="1" other_attrib=""&gt;
  &lt;element id="2" other_attrib=""&gt;
  ...
  &lt;other_element&gt;
    &lt;elem id="1"&gt;
    &lt;elem id="100"&gt;
  &lt;/other_element&gt;
  &lt;other_element&gt;
    &lt;elem id...&gt;
  &lt;/other_element&gt;
  </code></pre>
  
  <p>all of the <code>element</code> tags precede the <code>other_elements</code> tags, (this XML is third-party, I can't restructure it)</p>
  
  <p>I have to read all of the <code>elements</code> first because they contain additional attributes, but only save those that are referenced by <code>other_element</code>s. </p>
  
  <p>So I opened a <code>shelve</code> with <code>writeback=False</code>, use <code>lxml.iterparse()</code> to parse the XML and save elements on the go, but after many elements get added (I don't know the exact number, but it goes into hundreds of thousands) I receive the following error:</p>
  
  <pre><code>HASH: Out of overflow pages.  Increase page size
  Traceback (most recent call last):
    File "/Library/Python/2.7/site-packages/celery/app/trace.py", line 240, in trace_task
      R = retval = fun(*args, **kwargs)
    File "/Library/Python/2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
      return self.run(*args, **kwargs)
    ...
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shelve.py", line 133, in __setitem__
      self.dict[key] = f.getvalue()
  </code></pre>
  
  <p>Basically what I do is for every element <code>iterparse</code> returns:</p>
  
  <pre><code>if elem.tag == 'element':
    shlv[elem['id']] = {'attrib1': elem['other_attrib'], 'attrib2': elem['attrib2']}
  elif elem.tag == 'other_element':
    # Here I iterate through this tags children and find references in shlv object
    for ref in elem:
      save_in_database(shlv[ref.attrib['id']])
  </code></pre>
  
  <p>What can I change so that <code>shelve</code> handles more data ? Or should I use something else to store that data ?</p>