not necessarily exist, or agree. Main theorems for GAN game The original GAN paper proved the following two theorems: Interpretation: For any fixed generator strategy , the optimal discriminator keeps track of the likelihood ratio between the reference distribution and the generator distribution:where is the logistic function. In particular, if the prior probability for an image to come from the reference distribution is equal to , then is just the posterior probability that came from the reference distribution: Training and evaluating GAN Training Unstable convergence While the GAN game has a unique global equilibrium point when both the generator and discriminator have access to their entire strategy sets, the equilibrium is no longer guaranteed when they have a restricted strategy set. In practice, the generator has access only to measures of form , where is a function computed by a neural network with parameters , and is an easily sampled distribution, such as the uniform or normal distribution. Similarly, the discriminator has access only to functions of form , a function computed by a neural network with parameters . These restricted strategy sets take up a vanishingly small proportion of their entire strategy sets. Further, even if an equilibrium still exists, it can only be found by searching in the high-dimensional space of all possible neural network functions. The standard strategy of using gradient descent to find the equilibrium often does not work for GAN, and often the game "collapses" into one of several failure modes. To improve the convergence stability, some training strategies start with an easier task, such as generating low-resolution images or simple images (one object with uniform background), and gradually increase the difficulty of the task during training. This essentially translates to applying a curriculum learning scheme. Mode collapse GANs often suffer from mode collapse where they fail to generalize properly, missing entire modes from the input data. For example, a GAN trained on the MNIST dataset containing many samples of each digit might only generate pictures of digit 0. This was named in the first paper as the "Helvetica scenario". One way this can happen is if the generator learns too fast compared to the discriminator. If the discriminator is held constant, then the optimal generator would only output elements of . So for example, if during GAN training for generating MNIST dataset, for a few epochs, the discriminator somehow prefers the digit 0 slightly