Why don't we find the analytical function of the cost function?
The power of networks come from hidden layers with non-linear activation functions. Said non-linear activation function make the calculation of an analytical solution impossible (except maybe for some very specific cases that are not really usefull).
If you need more convincing look at logistic regression : a one-neuron network with sigmoid activation which does not have an analytical solution.
As to why there is no closed form solution, you need to understand that the linear regression you gave in exemple is a very simple problem of projecting a 'point' on a 'plane'. Using an activation function on one neuron is equivalent to projecting a point on a complex manifold, which do not yield an analytical solution (except, again for some trivial cases). For neurons in succesions I am not even sure there is a geometrical interpretation.