In Neural Nets, why Use Gradient Methods as Opposed to Other Metaheuristics?
It would be a waste of information; the gradient is available, so use it and save time.
There is reason to believe that the local optima are good; see, for example, Choromanska et al. (notes).
Over-optimizing for the training set leads to worse generalization, so sometimes we deliberately don't even try by stopping early. Probably the best free lunch in machine learning.