Is the mean-squared error always convex in the context of neural networks? Multiple resources I referred to mention that MSE is great because it's convex. But I don't get how, especially in the context of neural networks.

Let's say we have the following:


$X$: training dataset
$Y$: targets
$\Theta$: the set of parameters of the model $f_\Theta$ (a neural network model with non-linearities)


Then:

$$\operatorname{MSE}(\Theta) = (f_\Theta(X) - Y)^2$$

Why would this loss function always be convex? Does this depend on $f_\Theta(X)$?
Answer in short:  MSE is convex on its input and parameters by itself. But on an arbitrary neural network it is not always convex due to the presence of non-linearities in the form of activation functions. Source for my answer is here.