title: How load a big file in TensorFlow?
body: <p>I'm trying to train a neural network to recognize images NSFW(Not Safe For Work), i have about 13k images NSFW ans 13k images SFW (Safe For Work). For the training, I'm using 8.000 images (50% NSFW , 50% SFW) with size 64x64, with 6000 for training and 2000 for validation (Same proportion), but I'm not happy with my results.
  Now I'm trying to train with the same quantity and proportion, but with images of size 128x128. The execution, however, is giving me an <code>out of memory</code> exception. But the file with images is not even close to 3gb, and my memory is 16gb.</p>
  
  <p>Can I separate the files in other files like:  <code>store1.pckl</code>, <code>store2.pckl</code>, etc. and in my code load them like:</p>
  
  <pre><code>load('store1.pckl')
  train(...)
  load('store2.pckl')
  train(...)
  </code></pre>
  
  <p>Or should I train with <code>store1.pckl</code>, save model, reexecute the code loading the model and train with <code>store2.pckl</code> and repeat this for all the dataset pieces:</p>
  
  <pre><code>loadModel('image-classifier.tfl')
  load('store2.pckl')
  train()
  saveModel('image-classifier.tfl')
  </code></pre>
  
  <p>Or is there is a way not to load all this files in my memory and load like StreamFiles like the native open()?</p>
  
  <p>Sorry for my bad English.</p>
  
  <p>The file <code>store.pckl</code> is created converting images files to binary and put in a file:</p>
  
  <pre><code>def resize_image(input_image_path,size):
      original_image = Image.open(input_image_path)
      original_image = original_image.convert('RGB')
      resized_image = original_image.resize(size)
      return resized_image
  #Example: label = [(0.0,1.0),(1.0,0.0),......
  f = open('store.pckl','wb')
  xt , yt = shuffle(listatrain,labelt)
  xv , yv = shuffle(listavali,labelv)
  pickle.dump((xt,yt,xv,yv),f)
  </code></pre>
  
  <p>This is the code:</p>
  
  <pre><code>from __future__ import division, print_function, absolute_import
  import tflearn
  from tflearn.data_utils import shuffle
  from tflearn.layers.core import input_data, dropout, fully_connected
  from tflearn.layers.conv import conv_2d, max_pool_2d
  from tflearn.layers.estimator import regression
  from tflearn.data_preprocessing import ImagePreprocessing
  from tflearn.data_augmentation import ImageAugmentation
  import pickle
  import os
  
  X, Y, X_test, Y_test = pickle.load(open("store.pckl", "rb"))
  X, Y = shuffle(X, Y)
  
  img_prep = ImagePreprocessing()
  img_prep.add_featurewise_zero_center()
  img_prep.add_featurewise_stdnorm()
  img_aug = ImageAugmentation()
  img_aug.add_random_flip_leftright()
  img_aug.add_random_rotation(max_angle=25.)
  img_aug.add_random_blur(sigma_max=3.)
  
  network = input_data(shape=[None, 128, 128, 3],
                   data_preprocessing=img_prep,
                   data_augmentation=img_aug)
  
  network = conv_2d(network, 128, 3, activation='relu')
  network = max_pool_2d(network, 2)
  network = conv_2d(network, 256, 3, activation='relu')
  network = conv_2d(network, 256, 3, activation='relu')
  network = max_pool_2d(network, 2)
  network = fully_connected(network, 2048, activation='relu')
  network = dropout(network, 0.5)
  network = fully_connected(network, 2, activation='softmax')
  network = regression(network, optimizer='adam',
                   loss='categorical_crossentropy',
                   learning_rate=0.001)
  model = tflearn.DNN(network, tensorboard_verbose=0, checkpoint_path='image-classifier.tfl.ckpt')
  model.fit(X, Y, n_epoch=100, shuffle=True, validation_set=(X_test, Y_test),
        show_metric=True, batch_size=96,
        snapshot_epoch=True,
        run_id='image-classifier')
  model.save("image-classifier.tfl")
  </code></pre>