title: visualizing large datasets in python
body: <p>My original post is below but I think it probably wasn't as clear as it should have been. So, clarification:</p>
  
  <p>I'd like to be able to quickly access 3D views of large datasets. The 3D fields I'm currently working with are gigabytes when stored as text, but I may want to move on to larger fields in the future. It isn't practical to work with many gigabytes of data in memory and it isn't reasonable to plot millions of points on a display where they won't give you additional information. Instead, I'd like to have an environment where most of the data is stored to disk and only the values that are relevant to a given plot are read into memory. I was hoping mayavi could do this easily by simply replacing a normal array with a numpy memmap, but that didn't work.</p>
  
  <p>I could split up the fields into smaller volumes and work with those volumes independently or do separate plots that have lower resolution for a larger volume, but I'm hoping that there are tools available which will handle this for me. I don't necessarily need an interactive 3D environment to explore the data, but it would be nice if some of the cutting or averaging of data could be handled "under the hood" when I declare plot parameters.</p>
  
  <p>Is this possible?</p>
  
  <hr>
  
  <p>I'm working on visualizing 3D fields in mayavi. I'm storing the fields on disk in text files with one value per line (or multiple values if I'm working with more than one field). These files can get pretty big: right now I'm looking at one or two gigabytes but this could increase in the future. Since I can't load the fields entirely into memory, I thought I'd use numpy memory maps to handle the data. Unfortunately mayavi does array operations on the entire dataset, so when I try to do <code>mlab.pipeline.scalar_field(memmap)</code>, I get errors like this:</p>
  
  <pre><code>Traceback (most recent call last):
    File "C:\Anaconda\lib\site-packages\traits\trait_notifiers.py", line 340, in __call__
      self.handler( *args )
    File "C:\Anaconda\lib\site-packages\mayavi\sources\array_source.py", line 181, in _scalar_data_changed
      img_data.point_data.scalars = numpy.ravel(numpy.transpose(data))
    File "C:\Anaconda\lib\site-packages\numpy\core\fromnumeric.py", line 1367, in ravel
      return asarray(a).ravel(order)
  MemoryError
  </code></pre>
  
  <p>Is there a good way around this or am I up against the limits of what mayavi can handle? I've seen vtk recommended for visualization in python and since (as far as I know) mayavi is just a front end for vtk, I guess I could get around the array operations mayavi is trying to use. Is that the way to go? Note that I'm not very experienced with any of this, so if I should be using a completely different tool to get this done I'm happy to look into it (as long as it's free). </p>
  
  <p>Thanks!</p>