How to handle large memory footprint in Python?
<p>I have a scientific application that reads a potentially huge data file from disk and transforms it into various Python data structures such as a map of maps, list of lists etc. <code>NumPy</code> is called in for numerical analysis. The problem is, the memory usage can grow rapidly. As swap space is called in, the system slows down significantly. The general strategy I have seen:</p>

<ol>
<li>lazy initialization: this doesn't seem to help in the sense that many operations require in memory data anyway. </li>
<li>shelving: this Python standard library seems support writing data object into a datafile (backed by some db) . My understanding is that it dumps data to a file, but if you need it, you still have to load all of them into memory, so it doesn't exactly help. Please correct me if this is a misunderstanding.</li>
<li>The third option is to leverage a database, and offload as much data processing to it</li>
</ol>

<p>As an example: a scientific experiment runs several days and have generated a huge (tera bytes of data) sequence of:</p>

<blockquote>
  <p>co-ordinate(x,y) observed event E at time t.</p>
</blockquote>

<p>And we need to compute a histogram over t for each (x,y) and output a 3-dimensional array.</p>

<p>Any other suggestions? I guess my ideal case would be the in-memory data structure can be phased to disk based on a soft memory limit and this process should be as transparent as possible. Can any of these caching frameworks help? </p>

<p><strong>Edit:</strong></p>

<p>I appreciate all the suggested points and directions. Among those, I found user488551's comments to be most relevant. As much as I like Map/Reduce, to many scientific apps, the setup and effort for parallelization of code is even a bigger problem to tackle than my original question, IMHO. It is difficult to pick an answer as my question itself is so open ... but Bill's answer is more close to what we can do in real world, hence the choice. Thank you all.</p>
