title: C# OutOfMemory, Mapped Memory File or Temp Database
body: <p>Seeking some advice, best practice etc...  </p>
  
  <p>Technology: C# .NET4.0, Winforms, 32 bit</p>
  
  <p>I am seeking some advice on how I can best tackle large data processing in my C# Winforms application which experiences high memory usage (working set) and the occasional OutOfMemory exception.</p>
  
  <p>The problem is that we perform a large amount of data processing "in-memory" when a "shopping-basket" is opened. <strong>In simplistic terms when a "shopping-basket" is loaded we perform the following calculations;</strong></p>
  
  <ol>
  <li><p>For each item in the "shopping-basket" retrieve it's historical price going all the way back to the date the item first appeared in-stock (could be two months, two years or two decades of data). Historical price data is retrieved from text files, over the internet, any format which is supported by a price plugin.    </p></li>
  <li><p>For each item, for each day since it first appeared in-stock calculate various metrics which builds a historical profile for each item in the shopping-basket.</p></li>
  </ol>
  
  <p>The result is that we can potentially perform hundreds, thousand and/or millions of calculations depending upon the number of items in the "shopping-basket". If the basket contains too many items we run the risk of hitting a "OutOfMemory" exception.</p>
  
  <p><strong>A couple of caveats</strong>;</p>
  
  <ol>
  <li><p>This data needs to be calculated for each item in the "shopping-basket" and the data is kept until the "shopping-basket" is closed.</p></li>
  <li><p>Even though we perform steps 1 and 2 in a background thread, speed is important as the number of items in the "shopping-basket" can greatly effect overall calculation speed.</p></li>
  <li><p>Memory is salvaged by the .NET garbage collector when a "shopping-basket" is closed. We have profiled our application and ensure that all references are correctly disposed and closed when a basket is closed.</p></li>
  <li><p>After all the calculations are completed the resultant data is stored in a IDictionary. "CalculatedData is a class object whose properties are individual metrics calculated by the above process.</p></li>
  </ol>
  
  <p><strong>Some ideas I've thought about;</strong>  </p>
  
  <p>Obviously my main concern is to reduce the amount of memory being used by the calculations however the volume of memory used can only be reduced if I<br>
  1) reduce the number of metrics being calculated for each day or<br>
  2) reduce the number of days used for the calculation.  </p>
  
  <p>Both of these options are not viable if we wish to fulfill our business requirements.</p>
  
  <ul>
  <li><p>Memory Mapped Files<br>
  One idea has been to use memory mapped files which will store the data dictionary. Would this be possible/feasible and how can we put this into place?</p></li>
  <li><p>Use a temporary database<br>
  The idea is to use a separate (not in-memory) database which can be created for the life-cycle of the application. As "shopping-baskets" are opened we can persist the calculated data to the database for repeated use, alleviating the requirement to recalculate for the same "shopping-basket".  </p></li>
  </ul>
  
  <p>Are there any other alternatives that we should consider? What is best practice when it comes to calculations on large data and performing them outside of RAM?</p>
  
  <p>Any advice is appreciated....</p>