How do standardization and normalization impact the coefficients of linear models? One benefit of creating a linear model is that you can look at the coefficients the model learns and interpret them. For example, you can see which features have the most predictive power and which do not.
How, if at all, does feature interpretability change if we normalize (scale all features to 0-1) all our features vs. standardizing (subtract mean and divide by the standard deviation) them all before fitting the model.
I have read elsewhere that you 'lose feature interpretability if you normalize your features' but could not find an explanation as to why. If that is true, could you please explain?
Here are two screenshots of the coefficients for two multiple linear regression models I built. It uses Gapminder 2008 data and statistics about each country to predict its fertility rate.
In the first, I scaled features using StandardScaler. In the second, I used MinMaxScaler. The Region_ features are categorical and were one-hot encoded and not scaled.
Not only did the coefficients change based on different scaling, but their ordering (of importance?) did too! Why is this the case? What does it mean?
When you have a linear regression (without any scaling, just plain numbers) and you have a model with one explanatory variable $x$ and coefficients $\beta_0=0$ and $\beta_1=1$, then you essentially have a (estimated) function:
$$y = 0 + 1x .$$
This tells you that when $x$ goes up (down) by one unit, $y$ goes up (down) by one unit. In this case it is just a linear function with slope 1.
Now when you scale $x$ (the plain numbers) like:
scale(c(1,2,3,4,5))
           [,1]
[1,] -1.2649111
[2,] -0.6324555
[3,]  0.0000000
[4,]  0.6324555
[5,]  1.2649111

you essentially have different units or a different scale (with mean=0, sd=1).
However, the way OLS works will be the same, it still tells you &quot;if $x$ goes up (down) by one unit, $y$ will change by $\beta_1$ units. So in this case (given a different scale of $x$), $\beta_1$ will be different.
The interpretation here would be &quot;if $x$ changes by one standard deviation...&quot;. This is very handy when you have several $x$ with different units. When you standardise all the different units, you make them comparable to some extent. I.e. the $\beta$ coefficients of your regression will be compareable in terms of how strong the variables impact on $y$ is. This is sometimes calles Beta-Coefficients or Standardised Coefficients.
A very similar thing happens when you normalise. In this case you will also change the scale of $x$, so the way how $x$ is measured.
Also see this handout.