remark (early stoppingbayes iteration is an example of an "early stoppingheuristic for maximum likelihood optimizationwhere we exit after only one step as observed aboveif we keep iteratingwe obtain the maximum likelihood estimate (mlein sense the bayes rule provides regularization of the mle regularization is discussed in more detail in see also example the early stopping rule is also of benefit in regularizationsee exercise in on the one handthe initial guess (thconveys the priori (prior to training the bayesian learnerinformation about the optimal density in that minimizes the kl risk using this prior (th)the bayesian approximation to (xis the prior predictive densityz (xg( thg(thdth on the other handthe posterior pdf conveys improved knowledge about this optimal density in after training with using the posterior (th )the bayesian learner of (xis the posterior predictive densityz ( thg(th tdthgt (xg( tprior predictive density posterior predictive density where we have assumed that ( thtg( th)that isthe likelihood depends on only through the parameter th the choice of the prior is typically governed by two considerations the prior should be simple enough to facilitate the computation or simulation of the posterior pdf the prior should be general enough to model ignorance of the parameter of interest priors that do not convey much knowledge of the parameter are said to be uninformative the uniform or flat prior in example (to followis frequently used for the purpose of analytical and numerical computationswe can view th as random vector with prior density (th)which after training is updated to the posterior density (th tr the above thinking allows us to write ( tg( thg( thg(thdthfor examplethus ignoring any constants that do not depend on the argument of the densities example (normal modelsuppose that the training data { xn is modeled using the likelihood ( ththat is the pdf of th (us )where th :[us ]nextwe need to specify the prior distribution of th to complete the model we can specify prior distributions for and separately and then take their product to obtain the prior for vector th (assuming independencea possible prior distribution for is (nph ( uninformative prior