title: iPython notebook - Memory Error when loading CIFAR - 10 dataset
body: <p>I'm currently following the Convolutional neural network course of Stanford: <a href="http://cs231n.github.io/" rel="nofollow noreferrer">http://cs231n.github.io/</a>. I'm running the assignments on a Lenovo SL500 with 2GBs of RAM. But when I load the Cifar-10 dataset using the following code: </p>
  
  <pre><code>import cPickle as pickle
  import numpy as np
  import os
  from scipy.misc import imread
  
  def load_CIFAR_batch(filename):
    """ load single batch of cifar """
    with open(filename, 'rb') as f:
      datadict = pickle.load(f)
      X = datadict['data']
      Y = datadict['labels']
      X = X.reshape(10000, 3, 32, 32).transpose(0,2,3,1).astype("float")
      Y = np.array(Y)
      return X, Y
  
  def load_CIFAR10(ROOT):
    """ load all of cifar """
    xs = []
    ys = []
    for b in range(1,6):
      f = os.path.join(ROOT, 'data_batch_%d' % (b, ))
      X, Y = load_CIFAR_batch(f)
      xs.append(X)
      ys.append(Y)    
    Xtr = np.concatenate(xs)
    Ytr = np.concatenate(ys)
    del X, Y
    Xte, Yte = load_CIFAR_batch(os.path.join(ROOT, 'test_batch'))
    return Xtr, Ytr, Xte, Yte
  </code></pre>
  
  <p>Actually here is the part that calls the functions: </p>
  
  <pre><code># Load the raw CIFAR-10 data.
  cifar10_dir = 'cs231n/datasets/cifar-10-batches-py'
  X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)
  
  # As a sanity check, we print out the size of the training and test data.
  print 'Training data shape: ', X_train.shape
  print 'Training labels shape: ', y_train.shape
  print 'Test data shape: ', X_test.shape
  print 'Test labels shape: ', y_test.shape
  </code></pre>
  
  <p>I get this error:</p>
  
  <pre><code>MemoryError                               Traceback (most recent call last)
  &lt;ipython-input-3-76ab1121c87e&gt; in &lt;module&gt;()
        1 # Load the raw CIFAR-10 data.
        2 cifar10_dir = 'cs231n/datasets/cifar-10-batches-py'
  ----&gt; 3 X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)
        4 
        5 # As a sanity check, we print out the size of the training and test data.
  
  C:\Users\son\assignment1\cs231n\data_utils.py in load_CIFAR10(ROOT)
       20   for b in range(1,6):
       21     f = os.path.join(ROOT, 'data_batch_%d' % (b, ))
  ---&gt; 22     X, Y = load_CIFAR_batch(f)
       23     xs.append(X)
       24     ys.append(Y)
  
  C:\Users\son\assignment1\cs231n\data_utils.py in load_CIFAR_batch(filename)
       10     X = datadict['data']
       11     Y = datadict['labels']
  ---&gt; 12     X = X.reshape(10000, 3, 32, 32).transpose(0,2,3,1).astype("float")
       13     Y = np.array(Y)
       14     return X, Y
  
  MemoryError: 
  </code></pre>
  
  <p>And the thing is the memory usage quickly rises to 2GB ! Even though the CIFAR-10 dataset is only about 177MB! 
  <a href="https://i.stack.imgur.com/z8QbU.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/z8QbU.jpg" alt="enter image description here"></a></p>
  
  <p>I have tried loading only a single batch, and it worked, but I had to be very careful not to mess up any step (don't run the same cell twice, even if it does not appears to be running,...). Does anyone have a solution for this? Thank you very much!</p>
  
  <p>UPDATE: In the <code>load_CIFAR_batch</code> function, if I comment out the reshape line: <code># X = X.reshape(10000, 3, 32, 32).transpose(0,2,3,1).astype("float")</code>
  then the program is able to load:
  <a href="https://i.stack.imgur.com/GKLBx.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/GKLBx.jpg" alt="enter image description here"></a>
  So maybe the reason my program couldn't run was because of the reshape line, which creates a copy of X and Y therefore doubling the data. Is this hypothesis correct and if it does, what should I do about it? </p>