How does the brain train its neural network?
The answer to this question is probably Hebbian Learning.

Hebbian learning can be nicely summarised with "Cells that fire together, wire together". So basically the synapses of neurons are strengthened if they fire in sync, and weakened otherwise. 

One can easily see that a) this kind of local learning mechanism makes a lot more sense for the brain then some global method like gradient descent and b) this mechanism leads to stable representations of patterns.

In artificial neural networks this kind of learning is modelled in Hopfield networks and Restricted Boltzmann Machines.

Of course this simple rule barely scratches at the surface of what goes on in the human brain, when we are learning something. A complete picture would probably involve complex feedback mechanisms of inhibitory and excitatory connections between neurons, functional modules of neurons, and different parts of the brain. 

But I fear these details are not yet well understood …