title: Why is my generator loss function increasing with iterations
body: <p>I'm trying to train a DC-GAN on CIFAR-10 Dataset. I'm using Binary Cross Entropy as my loss function for both discriminator and generator (appended with non-trainable discriminator). If I train using Adam optimizer, the GAN is training fine. But if I replace the optimizer by SGD, the training is going haywire. The generator accuracy starts at some higher point and with iterations, it goes to 0 and stays there. The discriminator accuracy starts at some lower point and reaches somewhere around 0.5 (expected, right?). The peculiar thing is the generator loss function is increasing with iterations. I though may be the step is too high. I tried changing the step size. I tried using momentum with SGD. In all these cases, the generator may or may not decrease in the beginning, but then increases for sure. So, I think there is something inherently wrong in my model. I know training Deep Models is difficult and GANs still more, but there has to be some reason/heuristic as to why this is happening. Any inputs in appreciated. I'm new to Neural Networks, Deep Learning and hence new to GANs as well.</p>
  
  <p>Here is my code:
  Cifar10Models.py</p>
  
  <pre><code>from keras import Sequential
  from keras.initializers import TruncatedNormal
  from keras.layers import Activation, BatchNormalization, Conv2D, Conv2DTranspose, Dense, Flatten, LeakyReLU, Reshape
  from keras.optimizers import SGD
  
  
  class DcGan:
      def __init__(self, print_model_summary: bool = False):
          self.generator_model = None
          self.discriminator_model = None
          self.concatenated_model = None
          self.print_model_summary = print_model_summary
  
      def build_generator_model(self):
          if self.generator_model:
              return self.generator_model
  
          self.generator_model = Sequential()
          self.generator_model.add(Dense(4 * 4 * 512, input_dim=100,
                                         kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
          self.generator_model.add(BatchNormalization(momentum=0.5))
          self.generator_model.add(Activation('relu'))
          self.generator_model.add(Reshape((4, 4, 512)))
  
          self.generator_model.add(Conv2DTranspose(256, 3, strides=2, padding='same',
                                                   kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
          self.generator_model.add(BatchNormalization(momentum=0.5))
          self.generator_model.add(Activation('relu'))
  
          self.generator_model.add(Conv2DTranspose(128, 3, strides=2, padding='same',
                                                   kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
          self.generator_model.add(BatchNormalization(momentum=0.5))
          self.generator_model.add(Activation('relu'))
  
          self.generator_model.add(Conv2DTranspose(64, 3, strides=2, padding='same',
                                                   kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
          self.generator_model.add(BatchNormalization(momentum=0.5))
          self.generator_model.add(Activation('relu'))
  
          self.generator_model.add(Conv2D(3, 3, padding='same',
                                          kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
          self.generator_model.add(Activation('tanh'))
  
          if self.print_model_summary:
              self.generator_model.summary()
  
          return self.generator_model
  
      def build_discriminator_model(self):
          if self.discriminator_model:
              return self.discriminator_model
  
          self.discriminator_model = Sequential()
          self.discriminator_model.add(Conv2D(128, 3, strides=2, input_shape=(32, 32, 3), padding='same',
                                              kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
          self.discriminator_model.add(LeakyReLU(alpha=0.2))
  
          self.discriminator_model.add(Conv2D(256, 3, strides=2, padding='same',
                                              kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
          self.generator_model.add(BatchNormalization(momentum=0.5))
          self.discriminator_model.add(LeakyReLU(alpha=0.2))
  
          self.discriminator_model.add(Conv2D(512, 3, strides=2, padding='same',
                                              kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
          self.generator_model.add(BatchNormalization(momentum=0.5))
          self.discriminator_model.add(LeakyReLU(alpha=0.2))
  
          self.discriminator_model.add(Conv2D(1024, 3, strides=2, padding='same',
                                              kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
          self.generator_model.add(BatchNormalization(momentum=0.5))
          self.discriminator_model.add(LeakyReLU(alpha=0.2))
  
          self.discriminator_model.add(Flatten())
          self.discriminator_model.add(Dense(1, kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
          self.generator_model.add(BatchNormalization(momentum=0.5))
          self.discriminator_model.add(Activation('sigmoid'))
  
          if self.print_model_summary:
              self.discriminator_model.summary()
  
          return self.discriminator_model
  
      def build_concatenated_model(self):
          if self.concatenated_model:
              return self.concatenated_model
  
          self.concatenated_model = Sequential()
          self.concatenated_model.add(self.generator_model)
          self.concatenated_model.add(self.discriminator_model)
  
          if self.print_model_summary:
              self.concatenated_model.summary()
  
          return self.concatenated_model
  
      def build_dc_gan(self):
          self.build_generator_model()
          self.build_discriminator_model()
          self.build_concatenated_model()
  
          self.discriminator_model.trainable = True
          optimizer = SGD(lr=0.0002)
          self.discriminator_model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])
          self.discriminator_model.trainable = False
          optimizer = SGD(lr=0.0001)
          self.concatenated_model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])
          self.discriminator_model.trainable = True
  </code></pre>
  
  <p>Cifar10Trainer.py:</p>
  
  <pre><code># Shree KRISHNAya Namaha
  # Based on https://towardsdatascience.com/gan-by-example-using-keras-on-tensorflow-backend-1a6d515a60d0
  
  import os
  
  import datetime
  import numpy
  import time
  from keras.datasets import cifar10
  from keras.utils import np_utils
  from matplotlib import pyplot as plt
  
  import Cifar10Models
  
  log_file_name = 'logs.csv'
  
  
  class Cifar10Trainer:
      def __init__(self):
          self.x_train, self.y_train = self.get_train_and_test_data()
          self.dc_gan = Cifar10Models.DcGan()
          self.dc_gan.build_dc_gan()
  
      @staticmethod
      def get_train_and_test_data():
          (x_train, y_train), _ = cifar10.load_data()
          x_train = x_train.reshape(x_train.shape[0], x_train.shape[1], x_train.shape[2], 3)
          # Generator output has tanh activation whose range is [-1,1]
          x_train = (x_train.astype('float32') * 2 / 255) - 1
          y_train = np_utils.to_categorical(y_train, 10)
          return x_train, y_train
  
      def train(self, train_steps=10000, batch_size=128, log_interval=10, save_interval=100,
                output_folder_path='./Trained_Models/'):
          self.initialize_log(output_folder_path)
          self.sample_real_images(output_folder_path)
          for i in range(train_steps):
              # Get real (Database) Images
              images_real = self.x_train[numpy.random.randint(0, self.x_train.shape[0], size=batch_size), :, :, :]
  
              # Generate Fake Images
              noise = numpy.random.uniform(-1.0, 1.0, size=[batch_size, 100])
              images_fake = self.dc_gan.generator_model.predict(noise)
  
              # Train discriminator on both real and fake images
              x = numpy.concatenate((images_real, images_fake), axis=0)
              y = numpy.ones([2 * batch_size, 1])
              y[batch_size:, :] = 0
              d_loss = self.dc_gan.discriminator_model.train_on_batch(x, y)
  
              # Train generator i.e. concatenated model
              noise = numpy.random.uniform(-1.0, 1.0, size=[batch_size, 100])
              y = numpy.ones([batch_size, 1])
              g_loss = self.dc_gan.concatenated_model.train_on_batch(noise, y)
  
              # Print Logs, Save Models, generate sample images
              if (i + 1) % log_interval == 0:
                  self.log_progress(output_folder_path, i + 1, g_loss, d_loss)
              if (i + 1) % save_interval == 0:
                  self.save_models(output_folder_path, i + 1)
                  self.generate_images(output_folder_path, i + 1)
  
      @staticmethod
      def initialize_log(output_folder_path):
          log_line = 'Iteration No, Generator Loss, Generator Accuracy, Discriminator Loss, Discriminator Accuracy, ' \
                     'Time\n'
          with open(os.path.join(output_folder_path, log_file_name), 'w') as log_file:
              log_file.write(log_line)
  
      @staticmethod
      def log_progress(output_folder_path, iteration_no, g_loss, d_loss):
          log_line = '{0:05},{1:2.4f},{2:0.4f},{3:2.4f},{4:0.4f},{5}\n' \
              .format(iteration_no, g_loss[0], g_loss[1], d_loss[0], d_loss[1],
                      datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
          with open(os.path.join(output_folder_path, log_file_name), 'a') as log_file:
              log_file.write(log_line)
          print(log_line)
  
      def save_models(self, output_folder_path, iteration_no):
          self.dc_gan.generator_model.save(
              os.path.join(output_folder_path, 'generator_model_{0}.h5'.format(iteration_no)))
          self.dc_gan.discriminator_model.save(
              os.path.join(output_folder_path, 'discriminator_model_{0}.h5'.format(iteration_no)))
          self.dc_gan.concatenated_model.save(
              os.path.join(output_folder_path, 'concatenated_model_{0}.h5'.format(iteration_no)))
  
      def sample_real_images(self, output_folder_path):
          filepath = os.path.join(output_folder_path, 'CIFAR10_Sample_Real_Images.png')
          i = numpy.random.randint(0, self.x_train.shape[0], 16)
          images = self.x_train[i, :, :, :]
          plt.figure(figsize=(10, 10))
          for i in range(16):
              plt.subplot(4, 4, i + 1)
              image = images[i, :, :, :]
              image = numpy.reshape(image, [32, 32, 3])
              plt.imshow(image)
              plt.axis('off')
          plt.tight_layout()
          plt.savefig(filepath)
          plt.close('all')
  
      def generate_images(self, output_folder_path, iteration_no, noise=None):
          filepath = os.path.join(output_folder_path, 'CIFAR10_Gen_Image{0}.png'.format(iteration_no))
          if noise is None:
              noise = numpy.random.uniform(-1, 1, size=[16, 100])
          # Generator output has tanh activation whose range is [-1,1]
          images = (self.dc_gan.generator_model.predict(noise) + 1) / 2
          plt.figure(figsize=(10, 10))
          for i in range(16):
              plt.subplot(4, 4, i + 1)
              image = images[i, :, :, :]
              image = numpy.reshape(image, [32, 32, 3])
              plt.imshow(image)
              plt.axis('off')
          plt.tight_layout()
          plt.savefig(filepath)
          plt.close('all')
  
  
  def main():
      cifar10_trainer = Cifar10Trainer()
      cifar10_trainer.train(train_steps=10000, log_interval=10, save_interval=100)
      del cifar10_trainer.dc_gan
      return
  
  
  if __name__ == '__main__':
      start_time = time.time()
      print('Program Started at {0}'.format(time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(start_time))))
      main()
      end_time = time.time()
      print('Program Ended at {0}'.format(time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(end_time))))
      print('Total Execution Time: {0}s'.format(datetime.timedelta(seconds=end_time - start_time)))
  </code></pre>
  
  <p>Some of the graphs are as below:</p>
  
  <ol>
  <li><p>Discriminator Optimizer: SGD(lr=0.0001, beta1=0.5)<br>
  Generator Optimizer: Adam(lr=0.0001, beta1=0.5)
  <a href="https://i.stack.imgur.com/u04Yz.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/u04Yz.png" alt="enter image description here"></a></p></li>
  <li><p>Discriminator Optimizer: SGD(lr=0.0001)<br>
  Generator Optimizer: SGD(lr=0.0001)
  <a href="https://i.stack.imgur.com/QQ0NZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/QQ0NZ.png" alt="enter image description here"></a></p></li>
  <li><p>Discriminator Optimizer: SGD(lr=0.0001)<br>
  Generator Optimizer: SGD(lr=0.001)
  <a href="https://i.stack.imgur.com/DX2q3.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/DX2q3.png" alt="enter image description here"></a></p></li>
  <li><p>Discriminator Optimizer: SGD(lr=0.0001)<br>
  Generator Optimizer: SGD(lr=0.0005)
  <a href="https://i.stack.imgur.com/a8gQg.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/a8gQg.png" alt="enter image description here"></a></p></li>
  </ol>