title: Keras GAN (generator) not training well despite accurate discriminator
body: <p>I've tried sorting this out for a few days now, following many pieces of advice found on forums etc, and now would welcome any suggestions to what is wrong!</p>
  
  <p>I'm attempting to get my first GAN training - a simple feedforward deep net - very similar to using MNIST dataset, but with spectrum power windows derived from the VCTK-Corpus (size(1, 513)).</p>
  
  <p>You can see from the Tensorboard graphs below that the networks seem to be interacting, and there is some sort of training going on:
  <a href="https://i.stack.imgur.com/p4sLS.png" rel="nofollow noreferrer">Tensorboard graph overview</a>. <a href="https://i.stack.imgur.com/YYRz7.png" rel="nofollow noreferrer">Tensorboard graph zoom</a>.</p>
  
  <p>However, results are poor and noisy: <a href="https://i.stack.imgur.com/DIJgL.png" rel="nofollow noreferrer">generated and validation comparison</a></p>
  
  <p>The generator takes normal noise (usually 30 to 100 vectors) with a mean of 0 and stdev of 0.5.</p>
  
  <pre><code>def gan_generator(x_shape, frame_size):
      g_input = Input(shape=x_shape)
      H = BatchNormalization()(g_input)
      H = Dense(128)(H)
      H = LeakyReLU()(H)
      H = BatchNormalization()(H)
      H = Dense(128)(H)
      H = LeakyReLU()(H)
      H = BatchNormalization()(H)
      H = Dense(256)(H)
      H = LeakyReLU()(H)
      H = BatchNormalization()(H)
      H = Dense(256)(H)
      H = LeakyReLU()(H)
      H = BatchNormalization()(H)
      H = Dense(256)(H)
      H = LeakyReLU()(H)
      H = BatchNormalization()(H)
      out = Dense(frame_size[1], activation='linear')(H)
  
      generator = Model(g_input, out)
      generator.summary()
      return generator
  </code></pre>
  
  <p>The discriminator determines a one-hot categorisation of generated frames:
  (not sure about batch normalisation here - I've read it shouldn't be used if you're mixing real and generated into one batch. However, the generator makes much more convincing results with it than without - despite having a higher loss.)</p>
  
  <pre><code>def gan_discriminator(input_shape):
      d_input = Input(shape=input_shape)
      H = Dropout(0.1)(d_input)
      H = Dense(256)(H)
      H = Dropout(0.1)(H)
      H = LeakyReLU()(H)
      H = BatchNormalization()(H)
      H = Dense(128)(H)
      H = Dropout(0.1)(H)
      H = LeakyReLU()(H)
      H = BatchNormalization()(H)
      H = Dense(100)(H)
      H = Dropout(0.1)(H)
      H = LeakyReLU()(H)
      H = BatchNormalization()(H)
      H = Dense(100)(H)
      H = Dropout(0.1)(H)
      H = LeakyReLU()(H)
      H = BatchNormalization()(H)
      H = Reshape((1, -1))(H)
      d_V = Dense(2, activation='softmax')(H)
  
      discriminator = Model(d_input,d_V)
      discriminator.summary()
      return discriminator
  </code></pre>
  
  <p>The GAN is easy:</p>
  
  <pre><code>def init_gan(generator, discriminator):
      x = Input(shape=generator.inputs[0].shape[1:])
  
      #Generator makes a prediction
      pred = generator(x)
  
      #Discriminator attempts to categorise prediction
      y = discriminator(pred)
  
      GAN = Model(x, y)
      return GAN
  </code></pre>
  
  <p>Some training variables:</p>
  
  <ul>
  <li>GAN (generator): Adam, lr=1e-4, categorical_crossentropy</li>
  <li>discriminator: Adam, lr=1e-3, categorical_crossentropy</li>
  <li>batch size: around 8000 samples</li>
  <li>Mini-batch (weight update cycle): 32</li>
  </ul>
  
  <p>The training loop:</p>
  
  <pre><code>#Pre-training Discriminator Network
  #Load new batch of real frames
  frames = load_data(data_dir)
  frames_label = np.zeros((frames.shape[0], 1, 2))
  frames_label[:, :, 0] = 1 #mark as real frames
  
  #Generate Frames from noise vector
  X_noise = noisegen((frames.shape[0], 1, n_noise))
  generated_frames = generator.predict(X_noise)
  generated_label = np.zeros((generated_frames.shape[0], 1, 2))
  generated_label[:, :, 1] = 1 #mark as false frames
  
  #Prep Data - concat real and false data
  dis_batch_x = np.concatenate((frames, generated_frames), axis=0)
  dis_batch_y = np.concatenate((frames_label, generated_label), axis=0)
  
  #Make discriminator trainable and train for 8 epochs
  make_trainable(discriminator, True)
  discriminator.compile(optimizer=dis_optimizer, loss=dis_loss)
  fit_model(discriminator, dis_batch_x, dis_batch_y, 8)
  
  #Training Loop
  for d in range(data_sets):
      print "Starting New Dataset: {0}/{1}".format(d+1, data_sets)
  
      """ Fit Discriminator """
      #Load new batch of real frames
      frames = load_data(data_dir)
      frames_label = np.zeros((frames.shape[0], 1, 2))
      frames_label[:, :, 0] = 1 #mark as real frames
  
      #Generate Frames from noise vector
      X_noise = noisegen((frames.shape[0], 1, n_noise))
      generated_frames = generator.predict(X_noise)
      generated_label = np.zeros((generated_frames.shape[0], 1, 2))
      generated_label[:, :, 1] = 1 #mark as false frames
  
      #Prep Data - concat real and false data
      dis_batch_x = np.concatenate((frames, generated_frames), axis=0)
      dis_batch_y = np.concatenate((frames_label, generated_label), axis=0)
  
      #Make discriminator trainable &amp; fit
      make_trainable(discriminator, True)
      discriminator.compile(optimizer=dis_optimizer, loss=dis_loss)
      fit_model(discriminator, dis_batch_x, dis_batch_y)
  
  
      """ Fit Generator """
      #Prep Data
      X_noise = noisegen((frames.shape[0], 1, n_noise))
      generated_label = np.zeros((generated_frames.shape[0], 1, 2))
      generated_label[:, :, 1] = 1 #mark as false frames
  
      make_trainable(discriminator, False)
      GAN.layers[2].trainable = False #done twice just to be sure
      GAN.compile(optimizer=GAN_optimizer, loss=GAN_loss) 
      fit_model(GAN, X_noise, generated_label)
  </code></pre>
  
  <p>And finally a little bit of system info:</p>
  
  <ul>
  <li>OSX 10.12</li>
  <li>Tensorflow 1.5.0 (GPU)</li>
  <li>Keras 2.1.3</li>
  <li>Python 2.7</li>
  </ul>
  
  <p>Many thanks in advance!</p>