training deep leaning models it must be noted that norm penalties are applied to the weight vectorsnot the bias terms the reasoning behind this is that any regularization is tradeoff between overfitting and underfittingand regularizing the bias term leads to bad tradeoff due to too much underfitting while training deep learning networksdifferent values of can be used for different layers and the appropriate value of is determined via an experiment using the validation set as guide figure - norm leads to th closer to zero tha is picked by the optimization algorithms because of regularizationwithout itthb would be picked dropout dropout is essentially computational cheap alternative of model ensemble/averaging let us first consider the key concept of model ensemble/averaging while individual models with sufficient capacity can overfitif we average or take majority voting on the predictions of multiple models (trained over subsets of dataor different weight initializations or different hyper parameters)we can address overfitting model ensembleaveraging is an extremely useful form of regularization that helps us deal with overfitting howeverit is quite computationally expensive