Theoretical proof of convergence of sequential weight update procedure (Neural Networks and Machine Learning)
As pointed out by Rong, for gradient descent to converge the learning rate $\eta$ has to be chosen appropriately. It has to be large enough that every update determines a big enough change in the weights, but it has to be small enough not to overshoot the local minimum of the cost function. You may want to take a look at this paper that analyzes some sufficient conditions for gradient descent to work


Armijo, Larry (1966). "Minimization of functions having Lipschitz continuous first partial derivatives". Pacific J. Math. 16 (1): 1–3.