Synthetic Gradients - what's the practical benefit?
But, hey! These predicted "synthetic gradients" will be very small (negligible) anyway - after all, that's why we start a new backprop session B. Weren't they too small, we would just parse all 6 timesteps in a single long bkprop "session A".


-That's not necessarily correct. We typically truncate and start a new backprop because of hardware constraints, such as memory or computational speed. Vanishing-gradient can be improved by other means, such as Gradient Normalization - scaling-up the gradient vector if it gets too small beyond certain layers, or scaling-down if it's about to explode. Or even using Batch Normalization



It's important to understand how to update any DNI module. To clear things up, consider an example of network with several layers and 3 DNI modules:

 input
   |
   V
Layer_0 &amp; DNI_0
Layer_1
Layer_2
Layer_3 &amp; DNI_3
Layer_4
Layer_5 &amp; DNI_5
Layer_6
Layer_7
   | 
   V
output



  The DNI_0 is always trained with a synthetic gradient arriving from DNI_3 (flowing through Layer_2, and Layer_1 of course), siting several layers further. 
  
  Likewise, the DNI_3 is always trained with a synthetic gradient arriving from DNI_5
  
  DNI_0 or DNI_3 will never get to see the true gradient, because true grad is only delivered to DNI_5, and not earlier.
  
  For anyone still struggling to understand them, read this awesome blogpost, part 3


Earlier layers will have to be content with synthetic gradients, because they or their DNI will never witness the "true gradient".



Regarding training in parallel with minibatches instead of Parallelizing via Synthetic Grads:

Longer sequences are more precise than minibatches, however minibatches add a regulization effect. But, given some technique to prevent gradient from exploding or vanishing, training longer sequences can provide a lot better insight into context of the problem. That's because network infers output after considering a longer sequence of input, so the outcome is more rational.

For the comparison of benefits granted by SG refer to the diagrams page 6 of the Paper, mainly being able to solve longer sequences, which I feel is most beneficial (we can already parallelize via Minibatches anyway, and thus SG shouldn't speed up the process when performed on the same machine - even if we indeed only propagate up to the next DNI). 



However, the more DNI modules we have, the noisier the signal should
 be. So it might be worth-while to train the layers and DNI all by the legacy backprop, and only after some epochs have elapsed we start using the DNI-bootstrapping discussed above. 

That way, the earliest DNI will acquire at least some sense of what to expect at the start of the training.  That's because the following DNIs are themselves unsure of what the true gradient actually looks like, when the training begins, so initially, they will be advising "garbage" gradient to anyone sitting earlier than them.

Don't forget that authors also experimented with predicting the actual  inputs for every layer.



If your layers have expensive backprop (perhaps you have Batch-Normalization or some fancy activation functions), correcting with DNI might be a lot cheaper, once it's sufficiently well-trained.  Just remember that DNI is not free - it requires matrix multiplication and probably won't provide much speed-up on a simple dense layer.



Minibatches gives us speedup (via parallelization) and also give us the  regularization.
Synthetic Gradients allow us to infer better by working with longer sequences and (potentially) less-expensive gradient. All together this is a very powerful system.