title: Handling very large netCDF files in python
body: <p>I am trying to work with data from very large netCDF files (~400 Gb each). Each file has a few variables, all much larger than the system memory (e.g. 180 Gb vs 32 Gb RAM). I am trying to use numpy and netCDF4-python do some operations on these variables by copying a slice at a time and operating on that slice. Unfortunately, it is taking a really long time just to read each slice, which is killing the performance. </p>
  
  <p>For example, one of the variables is an array of shape <code>(500, 500, 450, 300)</code>. I want to operate on the slice <code>[:,:,0]</code>, so I do the following:</p>
  
  <pre><code>import netCDF4 as nc
  
  f = nc.Dataset('myfile.ncdf','r+')
  myvar = f.variables['myvar']
  myslice = myvar[:,:,0]
  </code></pre>
  
  <p>But the last step takes a really long time (~5 min on my system). If for example I saved a variable of shape <code>(500, 500, 300)</code> on the netcdf file, then a read operation of the same size will take only a few seconds. </p>
  
  <p>Is there any way I can speed this up? An obvious path would be to transpose the array so that the indices that I am selecting would come up first. But in such a large file this would not be possible to do in memory, and it seems even slower to attempt it given that a simple operation already takes a long time. What I would like is a quick way to read a slice of a netcdf file, in the fashion of the Fortran's interface get_vara function. Or some way of efficiently transposing the array.</p>