title: Correct memory (and performance) management for large numpy arrays of images
body: <p>I am trying to transform a large number of images in this case the <code>shape = (42000,784)</code> and I am running into <code>MemoryError</code> when I do it repeatedly in my code (i.e. if I call the function scale_xy several times). I am running a 32-bit Python on a 16 GB RAM machine so I assume the error is due to the 32-bit architecture of Python. This is my code for the moment. I would like to avoid MemoryError and ideally not compromise on speed (i.e. by saving the intermediate step to disk for example). I know that calling for shape needs to load the array from memory so I tried to limit that in my code to no effect. Also if there is a way how I can do this faster, a suggestion would be very appreciated.</p>
  
  <pre><code>import numpy as np
  import skimage.transform as tf
  import time
  
  def initialize(data, img_shape):
      """
      Helper function to save some code to create arrays to be transformed.
      """
      shape = data.shape
      reshaped = data.reshape((shape[0],img_shape[0],img_shape[1]))
      new = np.zeros((shape[0],img_shape[0],img_shape[1]))
  
      return shape, reshaped, new
  
  def batch_scale_xy(data, scale_x = 1, scale_y = 1, img_shape = (28,28),
               random = False, interval = (0.0,1.0) ):
      """
      Scales vectors in both x and y by given amount. If random is set to True
      it will randomly select values from uniform distribution on a given interval 
      for each individual image.
      """    
  
      shape, reshaped, new = initialize(data,img_shape)
  
      #All images are scaled evenly
      if random == False:
          for loc in xrange(shape[0]):
              new[loc,:,:] = tf.warp(reshaped[loc,:,:],
                                     tf.AffineTransform(scale=(scale_x,scale_y)))
  
      #Randomly scale each image by a factor
      elif random == True:
          for loc in xrange(shape[0]):
              scale_x, scale_y = np.random.uniform(low = interval[0], 
                                                   high = interval[1], size = (2,1))
              new[loc,:,:] = tf.warp(reshaped[loc,:,:],
                                     tf.AffineTransform(scale=(scale_x[0],scale_y[0])))
  
      new = new.reshape((shape[0],img_shape[0]*img_shape[1]))
      return new
  
  X = np.random.uniform(low=0.0,high=1.0,size=(42000,784))
  
  start = time.time()
  new = scale_xy(X,scale_x = 0.5, scale_y = 0.5)
  print "Time: %s seconds" % (time.time()-start)
  
  start = time.time()
  new = scale_xy(X,random=True, interval = (0.7,2.0))
  print "Time: %s seconds" % (time.time()-start)
  </code></pre>
  
  <p>The error is as follows:</p>
  
  <pre><code>Time: 4.06599998474 seconds
  Traceback (most recent call last):
    File "preprocessing.py", line 100, in &lt;module&gt;
      new = batch_scale_xy(X,random=True, interval = (0.7,2.0))
    File "preprocessing.py", line 75, in batch_scale_xy
      shape, reshaped, new = initialize(data,img_shape)
    File "preprocessing.py", line 34, in initialize
      new = np.zeros((shape[0],img_shape[0],img_shape[1]))
  MemoryError
  </code></pre>