Why does decreasing the SGD learning rate cause a massive increase in accuracy? In papers such as this I often see training curves with this kind of shape:



In this case SGD was used with a factor of 0.9 and learning rate decreasing by a factor of 10 every 30 epochs.


Why is there such a large decrease in error when the learning rate is changed?
Why does the validation error begin to increase after the initial drop, whereas the training error continues decreasing?
Can the same results be obtained by moving the 2nd and subsequent learning rate changes closer together? That is, why the delay in doing further drops?
With a higher learning rate, you take bigger steps towards the solution. However, when you are close, you might jump over the solution and then the next step, you jump over it again causing an oscillation around the solution.
Now, if you lower the learning rate correctly, you will stop the oscillation and continue towards the solution once again. That is, until you start oscillating again.
To keep in mind is that a larger learning rate can jump over smaller local minima and help you find a better minima, which it can't jump over.
Also, it is generally the training error that becomes better, and the validation error becomes worse as you start to overfit on the training data.