Optimization methods used in machine learning
In my reply here

Does gradient descent always converge to an optimum?

it is explained that standard gradient descent works well because backtracking gradient descent works well (proven in our recent paper mentioned in the post) and in the long run backtracking gradient descent behaves like the standard gradient descent. 

The main issue with other methods, I think, is that they require too strong conditions for convergence or no convergence is proven at all. In both cases, these make them less applicable to realistic applications.