Linear Regression with vs without polynomial features
Occam’s razor principle [ref]:

Having two hypotheses, that has the same empirical risk (here, training error), a short explanation (here, a boundary with fewer parameters) tends to be more valid than a long explanation.

Hence, complexity must add additional comparable accuracy. Otherwise, we should go with the simple model
Let's say you have 50-50 chance of Linear and Non-Linear data. With your approach, you will end up using regularization in 50% of the model. First, make a complex model and then regularize to make it simple.
Overall, you will increase the complexity which might impact future activities e.g Maintenance
Interpretability
 Many businesses demand a reason for the decision e.g. Credit card related decision. Linear regression will give you a simple explanation for each feature
Also, read these SE answers