Is the following a good method for determining learning rate in a neural network?
Starting with a high value is not a good idea. Since we are not aware of where the training starts on the Loss surface, so it can easily diverge.
Also, in between we need a higher LR to quickly move on saddle points (plateaus).
 Both start and end should be at a small value
A well-known approach is called Cyclical-Learning-Rate suggested by Leslie N. Smith in this paper Link
It suggests following a Cyclic path e.g. Triangle, Sinusoidal, etc.
It says to first increase from a minimum bound and starts decreasing when reached a maximum bound.
$\hspace{3cm}$
Paper has clearly explained -
- How to find the bounds
- How to decide the stepsize