title: Keras goes out of memory when running my toy examples
body: <p>I'm learning Keras after having worked in other frameworks for a while and I have the following code toy code example: </p>
  
  <pre><code>import keras.layers 
  import keras.models 
  import numpy as np
  from keras.optimizers import Adam
  import random 
  import os
  import cv2
  import tensorflow as tf
  
  TrainingDirectory="/home/thijser/host/ImageConverter/trainingdata" 
  def main():
      model = define_feedforward_network1()
      compileModel(model)
      model.summary()
      train(model)
  
  def define_feedforward_network1():
      inp = keras.engine.input_layer.Input(shape=(None,None,3))
      init = keras.initializers.RandomNormal(stddev=0.02)
      layer1=keras.layers.Conv2D(64, (4,4), activation='relu', padding='same', kernel_initializer=init)(inp)
      layer2 = keras.layers.Conv2D(128, (4,4), activation='relu', padding='same', kernel_initializer=init)(layer1)
      outp = keras.layers.Conv2D(3, (4,4), activation='relu', padding='same', kernel_initializer=init)(layer2)
  
      return keras.Model(input=inp,output=outp)
  
  
  def compileModel(model):
      adam=keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, amsgrad=False)
      model.compile(loss="mse", optimizer=adam,
      metrics=["mse"])
  
  def train(model):
      model.fit_generator(mygenerator(),steps_per_epoch=100,epochs=16)
  
  
  def mygenerator():
  
      batch_features=None
      batch_labels=None
      while True:
          target=random.choice(os.listdir(TrainingDirectory))
          batch_features=cv2.imread(TrainingDirectory+'/'+target+"/input.jpg", cv2.IMREAD_COLOR)
          batch_labels=cv2.imread(TrainingDirectory+'/'+target+"/labelcol.png", cv2.IMREAD_COLOR)
          yield np.array([batch_features]), np.array([batch_labels])
  
  main()  
  </code></pre>
  
  <p>Which is a simple neural network for (in this case) turning color images into grayscale images (just to get a feel for this). It can be run by placing several folders into the trainingDirectory folder and placing 1 image called labelcol.png and the other input.jpg (same size as the first) inside each of these folders. </p>
  
  <p>However when I run it it goes out of error rather quickly (right around the end of the first 16 images into training). If I turn on the report_tensor_allocations_upon_oom I get a segfault instead so that doesn't provide me with useful information, I can however see that the memory usage jumps by about 100-300 MB at a time suggesting that entire tensors might not be getting deallocated. </p>
  
  <p>Anybody see what I'm doing wrong? </p>