title: File indexing (using Binary trees?) in Python
body: <h2>Background</h2>
  
  <p>I have many (thousands!) of data files with a standard field based format (think tab-delimited, same fields in every line, in every file).  I'm debating various ways of making this data available / searchable.  (Some options include RDBMS, NoSQL stuff, using the grep/awk and friends, etc.).  </p>
  
  <h2>Proposal</h2>
  
  <p>In particular, one idea that appeals to me is "indexing" the files in some way. Since these files are read-only (and static), I was imagining some persistent files containing binary trees (one for each indexed field, just like in other data stores).  I'm open to ideas about how to this, or to hearing that this is simply insane.  Mostly, my favorite search engine hasn't yielded me any pre-rolled solutions for this.  </p>
  
  <p>I realize this is a little ill-formed, and solutions are welcome.</p>
  
  <h2>Additional Details</h2>
  
  <ul>
  <li>files long, not wide
  
  <ul>
  <li>millions of lines per hour, spread over 100 files per hour</li>
  <li>tab seperated, not many columns (~10) </li>
  <li>fields are short (say &lt; 50 chars per field)</li>
  </ul></li>
  <li>queries are on fields, combinations of fields, and can be historical</li>
  </ul>
  
  <h2>Drawbacks to various solutions:</h2>
  
  <p>(All of these are based on my observations and tests, but I'm open to correction)</p>
  
  <p><strong>BDB</strong></p>
  
  <ul>
  <li>has problems with scaling to large file sizes (in my experience, once they're 2GB or so, performance can be terrible)</li>
  <li>single writer (if it's possible to get around this, I want to see code!)</li>
  <li>hard to do multiple indexing, that is, indexing on different fields at once (sure you can do this by copying the data over and over). </li>
  <li>since it only stores strings, there is a serialize / deserialize step</li>
  </ul>
  
  <p><strong>RDBMSes</strong></p>
  
  <p><em>Wins:</em></p>
  
  <ul>
  <li>flat table model is excellent for querying, indexing</li>
  </ul>
  
  <p><em>Losses:</em></p>
  
  <ul>
  <li>In my experience, the problem comes with indexing.  From what I've seen (and please correct me if I am wrong), the issue with rdbmses I know (sqlite, postgres) supporting either batch load (then indexing is slow at the end), or row by row loading (which is low).  Maybe I need more performance tuning.  </li>
  </ul>