Problem with basic understanding of polynomial regression I have an understanding of simple linear regression. Clear that  results in a fitted line like this:


However, studying polynomial regression is a bit of a challenge having some questions about the process. I understand the idea that fitting a curviliear "line" could follow the data more precisely. But seeing the following Python code in scikit-learn:

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

x = 2 - 3 * np.random.normal(0, 1, 20)
y = x - 2 * (x ** 2) + 0.5 * (x ** 3) + np.random.normal(-3, 3, 20)

x = x[:, np.newaxis]
y = y[:, np.newaxis]

polynomial_features= PolynomialFeatures(degree=2)
x_poly = polynomial_features.fit_transform(x)

model = LinearRegression()
model.fit(x_poly, y)
y_poly_pred = model.predict(x_poly)


suggests for me that here we "just" projecting the original features into 2nd degree new polynomial features, while still fitting the simple (straight) line on it. Still, we get the following solution with magically curved line:



Here arise my questions:


How could the straight line in figure #1 become a curved one in
figure #2 when we just introduce some new features but still fit the
original LinearRegression model? A cannot see why couldn't the same
estimator found curve in case 1 when it is able to find it in case
2? There is no hint for the estimator in the syntax in case 2 that "ok, let's apply a curve now instead of a line", right?
I read that polynomial regression is still linear, what is the exact
interpretation of this statement? "Linear" means no straight line but
any curve, is this what this want to say? If so, what is "nonlinear"?
Additionally, having read about penalty terms for polynomial regressions, I read that the
introduction of higher order features has the effect that
coefficients tend to grow along the magnitude. This is illustrated
here:




I cannot figure out here why does coefficients for a specific same feature (like x_1) increase just because there are more additional polynomials (e. g. x_2, x_3 etc) while stay low just being alone? Quite confusing.
"Linear regression" (aka. "ordinary least squares", OLS) refers to the type of estimator. Linear here means that you minimise the sum of squared residuals for a given (linear additive) regression equation. You can write a simple model:

$$y = \beta_0 + \beta_1 x_1 + u. $$ 

This would fit a linear function with intercept $\beta_0$ and slope $\beta_1$. So the "fitted" function would be:

$$ f(x) = \beta_0 + \beta_1 x. $$

You can "propose" basically any functional form to be fitted by the linear model, e.g. with a squared term (just add $x^2$ as feature):

$$y = \beta_0 + \beta_1 x_1 + \beta_2 x^2_1 + u. $$ 

This will give you the (estimated) quadratic function:

$$ f(x) = \beta_0 + \beta_1 x + \beta_2 x^2. $$

In essence, you can add any kind of linear transformation of $x$, e.g. $log(x)$, $exp(x)$ etc.

"Non-linear" models (there is no really clear definition), would be e.g. logistic regresion, non-parametric regression, tree-based models, neural nets etc. While each coefficient $\beta$ directly gives you a "marginal effect" in linear (OLS) regression, this is not the case for the other (non-linear) models. 

When you use regularisation in linear models, you simply add a penalty term to the loss function, so you minimise the sum of squared residuals given a penalty term. However, regarding your last question, I'm not quite sure what your actual question is.