Why Gradient methods work in finding the parameters in Neural Networks?
If you have a convex cost function, gradient descent will find the global minimizer. Typically, the cost functions associated with neural networks are not convex - technically. However, if you read this question on Stats.SE, you find that the cost function of many neural networks is "close enough" to convex so that gradient descent still works. You'll find, technically, a local minimizer, but neural networks are often invariant to node-switching (as mentioned in the answer to the question to which I linked), so that you're still finding a very good minimizer.