Why use gradient descent on Deep Nets / RNNs when cost function is not convex?
Even if your cost function is not convex, any minimization algorithm will do its job - including gradient descent. That said, a standard gradient descent is a local optimization procedure and can only ever guarantee a local minimum, not a global one. For quite a few applications, a good local minimum is good enough - for others, a lot of effort is put into using “global” optimization algorithms (such as genetic/evolutionary, simulated annealing, scatter search, multi start techniques, direct/multilevel searches etc...). Those algorithms are definitely more powerful but they are also significantly slower in converging to a good, potentially global, minimum. And none of them can guarantee global optimality unless you allow them a veeeeeeery large number of functions evaluations.