Why use Gradient Descent when Gradient just solve the problem? (With Neural Nets)
You might be mistaking the gradient itself with the mathematical approach to find the critical points of a differentiable function.

In the latter approach, you take the derivative of the function with respect to its parameters and find the values of the parameters that make it zero. These points in parameter space are critital points, that is, where the function has either a local maxima, a local minima or a saddle point. In order to know the actual type, you have to take the second derivative.

The problem with applying such an approach to neural networks is not finding the gradient of the loss function with respect to the parameters; currently we have automatic differentiation software that computes the gradients for you. The actual problem is solving the equation where such gradient equals to zero. We don't know how to do that but in the trivial cases. Furthermore, this only gives you local optima, not global ones.

A solution to that problem is using numerical optimization techniques, like the gradient descent family. They basically explore the parameter space in the descending direction of the gradient of the loss function, hoping to reach a minimum. This cannot not assured for non-convex problems. Nevertheless, in practice it works quite well.

The reasons why gradient descent techniques work well in non-convex problems are an active line of research.