title: Memory-mapped file IList implementation, for storing large datasets "in memory"?
body: <p>I need to perform operations chronologically on huge time series implemented as IList. The data is ultimately stored into a database, but it would not make sense to submit tens of millions of queries to the database. </p>
  
  <p>Currently the in-memory IList triggers an OutOfMemory exception when trying to store more than 8 million (small) objects, though I would need to deal with tens of millions.</p>
  
  <p>After some research, it looks like the best way to do it would be to store data on disk and access it through an IList wrapper.</p>
  
  <p><a href="http://msdn.microsoft.com/fr-fr/library/system.io.memorymappedfiles.memorymappedfile.aspx" rel="nofollow noreferrer">Memory-mapped files</a> (introduced in .NET 4.0) seem the right interface to use, but I wonder what is the best way to write a class that should implement IList (for easy access) and internally deal with a memory-mapped file.</p>
  
  <p>I am also curious to hear if you know about other ways ! I thought for example of an IList wrapper using data from db4o (<a href="http://developer.db4o.com/Forums/tabid/98/aft/2915/Default.aspx" rel="nofollow noreferrer">someone mentionned here</a> using a memory-mapped file as the IoAdapterFile, though using db4o probably adds a performance cost vs. dealing directly with the memory-mapped file).</p>
  
  <p>I have come across <a href="https://stackoverflow.com/questions/1103360/array-which-pages-to-disk/7421227#7421227">this question</a> asked in 2009, but it did not yield useful answers or serious ideas.</p>