title: Prepare VGG Perceptual Loss on the fly for super-resolution with keras
body: <p>Basically I am implementing the same model for single image super-resolution from the paper <a href="https://arxiv.org/abs/1603.08155" rel="nofollow noreferrer">https://arxiv.org/abs/1603.08155</a>. I ran into memory issue when I tried to generate to activation output of the ground truth patches, which will be used to compute the perceptual loss during the training. I wonder how I can generate the outputs on the fly.</p>
  
  <p>I use 10k 288x288 image patches as ground truths and the corresponding blurred and down-sampled 72x72 patches as training data. For the loss network I use VGG-16 and the output from Relu2-2 layer. I tried to use model.predict() to feed in the ground truth patches and generate the corresponding activation outputs, which can then be passed to model.fit() for training. However the data-set seems to be too large and it ran into memory issue. I understand it is a common problem in real practice with data-set being too large, and the solution is to use fit.generator() and imagedataGenrator to generate data on the fly. However I am not sure how exactly I can implement such function in my case. Could someone explain to me how I should implement such function or what other methods should I adopt to handle this problem?</p>
  
  <pre class="lang-py prettyprint-override"><code>### Create Image Transformation Model ###
  mainModel = ResnetBuilder.build((3,72,72), 5, basic_block, [1, 1, 1, 1, 1])
  
  ### Create Loss Model (VGG16) ###
  lossModel = VGG16(include_top=False, weights='imagenet', input_tensor=None, input_shape=(288,288,3))
  lossModel.trainable=False
  for layer in lossModel.layers:
      layer.trainable=False
  
  ### Create New Loss Model (Use Relu2-2 layer output for perceptual loss)
  lossModel = Model(lossModel.inputs,lossModel.layers[5].output)
  lossModelOutputs = lossModel(mainModel.output)
  
  ### Create Full Model ###
  fullModel = Model(mainModel.input, lossModelOutputs)
  
  ### Compile FUll Model
  fullModel.compile(loss='mse', optimizer='adam',metrics=['mse'])
  trained_epochs=0
  print("fullModel compiled!")
  </code></pre>
  
  <pre class="lang-py prettyprint-override"><code>y_train_lossModel = lossModel.predict(y_train,batch_size=1)
  </code></pre>
  
  <pre><code>MemoryError                               Traceback (most recent call last)
  &lt;ipython-input-11-1f5c849e454a&gt; in &lt;module&gt;
  ----&gt; 1 y_train_lossModel = lossModel.predict(y_train,batch_size=1)
        2 print(y_train_lossModel.shape)
        3 with h5py.File('y_train_lossModel.h5', 'w') as hf:
        4     hf.create_dataset('y_train_lossModel', data=y_train_lossModel)
  
  ~/anaconda3/envs/fyp/lib/python3.6/site-packages/keras/engine/training.py in predict(self, x, batch_size, verbose, steps)
     1167                                             batch_size=batch_size,
     1168                                             verbose=verbose,
  -&gt; 1169                                             steps=steps)
     1170 
     1171     def train_on_batch(self, x, y,
  
  ~/anaconda3/envs/fyp/lib/python3.6/site-packages/keras/engine/training_arrays.py in predict_loop(model, f, ins, batch_size, verbose, steps)
      298                 for batch_out in batch_outs:
      299                     shape = (num_samples,) + batch_out.shape[1:]
  --&gt; 300                     outs.append(np.zeros(shape, dtype=batch_out.dtype))
      301             for i, batch_out in enumerate(batch_outs):
      302                 outs[i][batch_start:batch_end] = batch_out
  
  MemoryError: 
  </code></pre>
  
  <pre class="lang-py prettyprint-override"><code>### Train the full model
  epochs=5
  for n in range(trained_epochs+1,trained_epochs+epochs+1):
      print("Epoch",n)
      fullModel.fit(x_train, y_train_lossModel, batch_size=4, epochs=1)
      fullModel.save('full_model.h5')
  trained_epochs=n
  </code></pre>