Should I use regularization every time?
Normally you use regularization. The exception is if you know the data generating process and can model it exactly. Then you merely estimate the model parameters. In general you will not know the process, so you will have to approximate with a flexible enough model. If the model is not flexible enough you will not need to regularize but you won't approximate well anyway. If you use a more flexible model, you will get closer on average (low bias) but you will have more variance, thus the need for increased regularization. In other words, the amount of regularization you need depends on the model. This is related to the bias-variance trade-off.

Welcome to DataScience.SE.