overfitting and underfitting the bigger network starts overfitting almost immediatelyafter just one epochand it overfits much more severely its validation loss is also noisier meanwhilefigure shows the training losses for the two networks as you can seethe bigger network gets its training loss near zero very quickly the more capacity the network hasthe more quickly it can model the training data (resulting in low training loss)but the more susceptible it is to overfitting (resulting in large difference between the training and validation lossfigure effect of model capacity on raining losst rying bigger model adding weight regularization you may be familiar with the principle of occam' razorgiven two explanations for somethingthe explanation most likely to be correct is the simplest one--the one that makes fewer assumptions this idea also applies to the models learned by neural networksgiven some training data and network architecturemultiple sets of weight values (multiple modelscould explain the data simpler models are less likely to overfit than complex ones simple model in this context is model where the distribution of parameter values has less entropy (or model with fewer parametersas you saw in the previous sectionthus common way to mitigate overfitting is to put constraints on the complexity of network by forcing its weights to take only small valueswhich makes the distribution of weight values more regular this is called weight regularizationand it' done by adding to the loss function of the network cost associated with having large weights this cost comes in two flavorsl regularization--the cost added is proportional to the absolute value of the weight coefficients (the norm of the weightsl regularization--the cost added is proportional to the square of the value of the weight coefficients (the norm of the weightsl regularization is also called weight decay in the context of neural networks don' let the different name confuse youweight decay is mathematically the same as regularization licensed to