<p>I need to index up to 500,000 entries for fastest read.  The index needs to be rebuilt  periodically , on disk.  I am trying to decide between a simple file like a hash on disk or a single table in an embedded database.  I have no need for an RDBMS engine.  </p>

[{'answer_id': 68201, 'author': 'sblundy', 'author_id': 4893, 'author_profile': 'https://Stackoverflow.com/users/4893', 'pm_score': 0, 'selected': False, 'text': '<p>This is what <a href="http://en.wikipedia.org/wiki/MapReduce" rel="nofollow noreferrer">MapReduce</a> was invented for. <a href="http://hadoop.apache.org/" rel="nofollow noreferrer">Hadoop</a> is a cool java implementation. </p>\n'}, {'answer_id': 68208, 'author': 'kbaribeau', 'author_id': 8085, 'author_profile': 'https://Stackoverflow.com/users/8085', 'pm_score': 1, 'selected': False, 'text': "<p>I'm assuming you're referring to indexing tables on a relational DBMS (like mySql, Oracle, or Postgres).</p>\n\n<p>Indexes are secondary data stores that keep a record of a subset of fields for a table in a specific order.</p>\n\n<p>If you create an index, any query that includes the subset of fields that are indexed in its WHERE clause will perform faster.</p>\n\n<p>However, adding indexes will reduce INSERT performance.</p>\n\n<p>In general, indexes don't need to be rebuilt unless they become corrupted.  They should be maintained on the fly by your DBMS.</p>\n"}, {'answer_id': 68242, 'author': '1800 INFORMATION', 'author_id': 3146, 'author_profile': 'https://Stackoverflow.com/users/3146', 'pm_score': 0, 'selected': False, 'text': '<p>If the data doesn\'t need to be completely up to date, you might also like to think about using a data warehousing tool for OLAP purposes (such as <a href="http://www.mosha.com/msolap/" rel="nofollow noreferrer">MSOLAP</a>). The can perform lightning fast read-only queries based on pre-calculated data.</p>\n'}, {'answer_id': 68353, 'author': '1800 INFORMATION', 'author_id': 3146, 'author_profile': 'https://Stackoverflow.com/users/3146', 'pm_score': 1, 'selected': False, 'text': '<p>Perhaps <a href="http://en.wikipedia.org/wiki/Berkeley_DB" rel="nofollow noreferrer">BDB</a>? It is a high perf. database that doesn\'t use a DBMS.</p>\n'}, {'answer_id': 68384, 'author': 'sblundy', 'author_id': 4893, 'author_profile': 'https://Stackoverflow.com/users/4893', 'pm_score': 1, 'selected': False, 'text': '<p>If you\'ve storing state objects by key, how about <a href="http://en.wikipedia.org/wiki/Berkeley_DB" rel="nofollow noreferrer">Berkeley DB</a>.</p>\n'}, {'answer_id': 68577, 'author': 'Allan Wind', 'author_id': 9706, 'author_profile': 'https://Stackoverflow.com/users/9706', 'pm_score': 1, 'selected': False, 'text': '<p><a href="http://cr.yp.to/cdb.html" rel="nofollow noreferrer">cdb</a> if the data does not change.</p>\n\n<p>/Allan</p>\n'}, {'answer_id': 68619, 'author': 'Wheat', 'author_id': 70142, 'author_profile': 'https://Stackoverflow.com/users/70142', 'pm_score': 1, 'selected': False, 'text': '<p>PyTables Pro claims that "for situations that don\'t require fast updates or deletions, OPSI is probably one of the best indexing engines available". However I\'ve not personally used it, but the F/OSS version of PyTables gives already gives you good performance:</p>\n\n<p><a href="http://www.pytables.org/moin/PyTablesPro" rel="nofollow noreferrer">http://www.pytables.org/moin/PyTablesPro</a></p>\n'}]
['https://Stackoverflow.com/questions/68174', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/9411/']