GAN - am I seeing mode collapse? Common fixes not working I have a 2 part question.



Context
I am learning about GANs and writing my own starting from the very simplest example of adversarial learning (1-parameter node), then implementing a very simple 1-dimensional pattern (1010) learning GAN .. and now I am trying to implement an MNIST learning GAN, before I proceed to more realistic photos.

I have some background in machine learning and data mining (masters from a long time ago) and moderately understand how neural networks work.

You can read about my progress on the initial steps here:


GANs - Part 1
GANs - Part 2


I've read lots of the latest blogs, articles, and watched youtube tutorials but still can't get past 2 key problems with implementing MNIST learning GANS.



Q 1. Am I seeing mode collapse?

After some tweaking and iteration I have a GAN which does learn to generate images which look like they might come from the MNIST dataset. Actually they're not digits yet but they are recognisable pen strokes, and certainly not random noise.

You can see a recent iteration of my pytorch code here: github notebook.

When fed random noise, the trained GAN always generates the same, or extremely similar image. Feeding it 1-shot noise (00001000...) also generates a similar image.

Are GANs supposed to generate just one image? Are the different images we see reported from separate training of the GAN? I thought the idea was a single trained GAN could generate many different images from random input noise. Have I misunderstood it?



Q 2. How To Escape Mode-Collapse?

If the answer above is that a train GAN should output many diverse but valid images, then I have mode-collapse.

I've read extensively and tried many approaches to avoiding mode-collapse but none have worked:


with / without batch normalization
with / without maxpooling
with / without dropout
with / without label softening
with / without noise added to both input and target labels, including noise that decays over training time
various widths and depths for the generator, less so the discriminator
increasing training time (but poor compute power only allows me about 6-10 epochs on the full dataset)


What I've observed or noted is:


the discriminator width/depth/architecture is tested first to ensure it has the capacity to learn the multi-class MNIST before it is used in the GAN to avoid the case of a discriminator that can't actually learn to discriminate MNIST
plotting the error as training proceeds (D error, D error on G input) is useful to show that the D error approaches 1/2 as it should, and that D error on G input approaches 1 (well more like 0.8). 
plotting errors also shows stability or collapse of some sort which helps tune the learning rates for example
I would have thought adding noise to the input or the target labels might kick the GAN out of any local minima which is mode-collapse but I suspect the theory is more complicated than this
trying different suggestions for the optimiser parameters doesn't help, I have to find my own optimal tuning, and the learning rates are much lower than what others use, eg Adam lr=0.00002 not 0.001 which causes instability
higher training epochs result in high contrast images which don't look like the MNIST data which has softer edged strokes .. I was hoping higher training epochs would improve diversity of outputs


One area where I can't find much guidance is on the actual architecture of the discriminator and generator:


Do they have to be matched but opposite? mine aren't - the discriminator is proven to have the learning capacity and that's it. After that the shallower the better to ensure easier back propagation to the generator.
The use of deconvolution is common in the generator, but some examples online use simple fully connected mapping. Computationally the deconvolutions have fewer learning parameters and intuitively make sense in building an image.




I'd welcome your thoughts and suggestions.

Example output which I think looks good, but is mode-collapsed:
Since this question I have made significant progress, which I have written up as part 3 of the series:

http://makeyourownalgorithmicart.blogspot.com/2019/05/generative-adversarial-networks-part-iii.html

Although my GAN doesn't now have mode collapse - the reasons are still unclear. The post reports visually the impact of combinations of SGD->Adam, none->LayerNorm, Sigmoid->ReLU which I hope others find useful: