title: LevelDB for 100s of millions entries
body: <p>What are the top factors to consider when tuning inserts for a LevelDB store?</p>
  
  <p>I'm inserting 500M+ records in the form:</p>
  
  <ol>
  <li>key="rs1234576543" <em>very predictable structure. rs&lt;1+ digits></em></li>
  <li>value="1,20000,A,C" <em>string can be much longer but usually ~ 40 chars</em></li>
  <li>keys are unique</li>
  <li>key insert order is random</li>
  </ol>
  
  <p>into a LevelDB store using the python plyvel, and see dramatic drop in speed as the number of records grows. I guess this is expected but are there tuning measures I could look at to make it scale better?</p>
  
  <p>Example code:</p>
  
  <pre><code>import plyvel
  BATCHSIZE = 1000000
  
  db = plyvel.DB('/tmp/lvldbSNP151/', create_if_missing=True)
  wb = db.write_batch()
  # items not in any key order
  for key, value in DBSNPfile:
      wb.put(key,value)
      if i%BATCHSIZE==0:
          wb.write()
  wb.write()
  </code></pre>
  
  <p>I've tried various batch sizes, which helps bit, but am hoping there's something else I've missed. For example, can knowing the max length of a key (or value) be leveraged?</p>