fundamentals of machine learning overfitting and underfitting in all three examples in the previous -predicting movie reviewstopic classificationand house-price regression--the performance of the model on the held-out validation data always peaked after few epochs and then began to degradethe model quickly started to overfit to the training data overfitting happens in every machine-learning problem learning how to deal with overfitting is essential to mastering machine learning the fundamental issue in machine learning is the tension between optimization and generalization optimization refers to the process of adjusting model to get the best performance possible on the training data (the learning in machine learning)whereas generalization refers to how well the trained model performs on data it has never seen before the goal of the game is to get good generalizationof coursebut you don' control generalizationyou can only adjust the model based on its training data at the beginning of trainingoptimization and generalization are correlatedthe lower the loss on training datathe lower the loss on test data while this is happeningyour model is said to be underfitthere is still progress to be madethe network hasn' yet modeled all relevant patterns in the training data but after certain number of iterations on the training datageneralization stops improvingand validation metrics stall and then begin to degradethe model is starting to overfit that isit' beginning to learn patterns that are specific to the training data but that are misleading or irrelevant when it comes to new data to prevent model from learning misleading or irrelevant patterns found in the training datathe best solution is to get more training data model trained on more data will naturally generalize better when that isn' possiblethe next-best solution is to modulate the quantity of information that your model is allowed to store or to add constraints on what information it' allowed to store if network can only afford to memorize small number of patternsthe optimization process will force it to focus on the most prominent patternswhich have better chance of generalizing well the processing of fighting overfitting this way is called regularization let' review some of the most common regularization techniques and apply them in practice to improve the movie-classification model from section reducing the network' size the simplest way to prevent overfitting is to reduce the size of the modelthe number of learnable parameters in the model (which is determined by the number of layers and the number of units per layerin deep learningthe number of learnable parameters in model is often referred to as the model' capacity intuitivelya model with more parameters has more memorization capacity and therefore can easily learn perfect dictionary-like mapping between training samples and their targets-- mapping without any generalization power for instancea model with , binary parameters could easily be made to learn the class of every digit in the mnist training setlicensed to