Why is loss displayed as a parabola in mean squared error with gradient descent? I'm looking at the loss function: mean squared error with gradient descent in machine learning. I'm building a single-neuron network (perceptron) that outputs a linear number. For example:
Input * Weight + Bias &gt; linear activation &gt; output.
Let's say the output is 40 while I expect the number 20. That means the loss function has to correct the weights+bias from 40 towards 20.
What I don't understand about mean squared error + gradient descent is: why is this number 40 displayed as a point on a parabola?
Does this parabola represent all possible outcomes? Why isn't it just a line? How do I know where on the parabola the point &quot;40&quot; is?
Mean Square Error (MSE) is a quadratic function and the further you go away from your optimum the bigger (quadratic) the MSE gets. Take $o_{expected}=20$ and $o_{net}=40$ as example. Your MSE is then 400, because
$MSE = (o_{expected}-o_{net})^2$.
Just imagine $y = x^2$ with x being the output of your network.
If you want to shift the parabola with optimum at $20$ the formula you get is $y = (20-x)^2$.
For every new case you train the  net on, you get a different parabola with different parameters.