Conceptual Question on Least Squares Estimation Method
Gauss did a two-way proof.  In rough terms, he not only showed that least squares gives the best fit to data containing a standard normal distribution of errors, but that if least squares is always giving the best fit, that implies that the distribution is indeed standard normal (also now called Gaussian).

The correlation coefficient, often computed in conjunction with a least square fit, is usually assumed to be parameterizing an underlying Gaussian distribution.  If the distribution is otherwise, it may well infer nonsense.