Synthetic Gradients good number of Layers & neurons
Several months later, I have a couple of insights on it:


  Also, how should we avoid our DNI overfitting, how to monitor and ensure it's not happening?


I don't think it matters, because DNI overfitting on the gradient is actually what we want. We want it to figure-out the pattern which reduces the error in the fastest way for our data.  However, as always, we should pay attention to the Validation of our entire network while doing so.

That probably means the more DNI neurons the better, as long as the designated training data stays unmodified o_O
If new or extra data is used for training, we should throw away our DNI, and just train once more, to overfit them on this new "adjusted" training-data.

Also, more on Synthetic Grads here

wish for a better answer, so won't select this one - plz post if you have a better one