Over fitting and association with regularization
The short answer is that regularization penalizes model complexity. It does this by adding some function of your parameters to your basic cost function as a regularization term. This forces a compromise between a good fit (low error) on your training data and model simplicity by driving the parameters closer to zero. This reduces overfitting because large parameters (in terms of magnitude) are the main culprit behind overfitting. 

For example, L1 regularization uses the sum of the absolute values of your weights, while L2 regularization uses the squares of your weights. More detailed comparison here.

For mathematical details, have a look at this answer.