quicklybut may not approximate gvery wellwhereas learner from rich class that contains gmay require lot of computing resources to train to better understand the relation between model complexitycomputational simplicityand estimation accuracyit is useful to decompose the generalization risk into several partsso that the tradeoffs between these parts can be studied we will consider two such decompositionsthe approximation-estimation tradeoff and the bias-variance tradeoff we can decompose the generalization risk ( into the following three components`(ggt `|{zirreducible risk irreducible risk approximation error statistical (estimationerror approximationestimation tradeoff `(gg ``(ggt `(gg ){ { approximation error ( statistical error where `:`(gis the irreducible risk and gg :argmingg `(gis the best learner within class no learner can predict new response with smaller risk than `the second component is the approximation errorit measures the difference between the irreducible risk and the best possible risk that can be obtained by selecting the best prediction function in the selected class of functions determining suitable class and minimizing `(gover this class is purely problem of numerical and functional analysisas the training data are not present for fixed that does not contain the optimal gthe approximation error cannot be made arbitrarily small and may be the dominant component in the generalization risk the only way to reduce the approximation error is by expanding the class to include larger set of possible functions the third component is the statistical (estimationerror it depends on the training set andin particularon how well the learner ggt estimates the best possible prediction functiongg within class for any sensible estimator this error should decay to zero (in probability or expectationas the training size tends to infinity the approximation-estimation tradeoff pits two competing demands against each other the first is that the class has to be simple enough so that the statistical error is not too large the second is that the class has to be rich enough to ensure small approximation error thusthere is tradeoff between the approximation and estimation errors for the special case of the squared-error lossthe generalization risk is equal to `(ggt ( ggt ( )) that isthe expected squared error between the predicted value ggt (xand the response recall that in this case the optimal prediction function is given by (xe[ xthe decomposition ( can now be interpreted as follows the first component` ( ( )) is the irreducible erroras no prediction function will yield smaller expected squared error the second componentthe approximation error `(gg `( )is equal to (gg (xg( )) we leave the proof (which is similar to that of theorem as an exercisesee exercise thusthe approximation error (defined as risk differencecan here be interpreted as the expected squared error between the optimal predicted value and the optimal predicted value within the class for the third componentthe statistical error`(ggt `(gg there is no direct interpretation as an expected squared error unless is the class of linear functionsthat isg(xxb for some vector in this case we can write (see exercise the statistical error as `(ggt `(gg (ggt (xgg ( )) colloquially called mean squared error