Why is loss displayed as a parabola in mean squared error with gradient descent?
Mean Square Error (MSE) is a quadratic function and the further you go away from your optimum the bigger (quadratic) the MSE gets. Take $o_{expected}=20$ and $o_{net}=40$ as example. Your MSE is then 400, because
$MSE = (o_{expected}-o_{net})^2$.
Just imagine $y = x^2$ with x being the output of your network.
If you want to shift the parabola with optimum at $20$ the formula you get is $y = (20-x)^2$.
For every new case you train the  net on, you get a different parabola with different parameters.