cycle consistency loss:The generators aim to minimize the objective, and the discriminators aim to maximize it: Unlike previous work like pix2pix, which requires paired training data, cycleGAN requires no paired data. For example, to train a pix2pix model to turn a summer scenery photo to winter scenery photo and back, the dataset must contain pairs of the same place in summer and winter, shot at the same angle; cycleGAN would only need a set of summer scenery photos, and an unrelated set of winter scenery photos. GANs with particularly large or small scales BigGAN The BigGAN is essentially a self-attention GAN trained on a large scale (up to 80 million parameters) to generate large images of ImageNet (up to 512 x 512 resolution), with numerous engineering tricks to make it converge. Invertible data augmentation When there is insufficient training data, the reference distribution cannot be well-approximated by the empirical distribution given by the training dataset. In such cases, data augmentation can be applied, to allow training GAN on smaller datasets. Naïve data augmentation, however, brings its problems. Consider the original GAN game, slightly reformulated as follows:Now we use data augmentation by randomly sampling semantic-preserving transforms and applying them to the dataset, to obtain the reformulated GAN game:This is equivalent to a GAN game with a different distribution , sampled by , with . For example, if is the distribution of images in ImageNet, and samples identity-transform with probability 0.5, and horizontal-reflection with probability 0.5, then is the distribution of images in ImageNet and horizontally-reflected ImageNet, combined. The result of such training would be a generator that mimics . For example, it would generate images that look like they are randomly cropped, if the data augmentation uses random cropping. The solution is to apply data augmentation to both generated and real images:The authors demonstrated high-quality generation using just 100-picture-large datasets. The StyleGAN-2-ADA paper points out a further point on data augmentation: it must be invertible. Continue with the example of generating ImageNet pictures. If the data augmentation is "randomly rotate the picture by 0, 90, 180, 270 degrees with equal probability", then there is no way for the generator to know which is the true orientation: Consider two generators , such that for any latent , the generated image is a 90-degree rotation of . They would have exactly the same expected loss, and so neither is preferred over the other.