suppose that we have rough guessdenoted (th)for the best that minimizes the kullback-leibler risk we can always increase the resulting likelihood : ( thw (thr dth by instead using the density (th: (thg( th)/ giving likelihood : ( thw (thdth to see thiswrite and as expectations with respect to in particularwe can write it follows that ew ( thand ew ( thew ( th)/ ew ( thl varw [ ( th) ( we may thus expect to obtain better predictions using instead of because has taken into account the observed data and increased the likelihood of the model in factif we iterate this process (see exercise and create sequence of densities such that wt (thwt- (thg( th)then wt (thconcentrates more and more of its probability mass at the maximum likelihood estimator th (see ( )and in the limit equals (degenb eratepoint-mass pdf at th in other wordsin the limit we recover the maximum likelihood methodgt (xg( ththusunless the class of densities is restricted to be nondegeneratemaximizing the likelihood as much as possible leads to degenerate choice for (thin many situationsthe maximum likelihood estimate ( this either not an appropriate approximation to ( (see example )or simply fails to exist (see exercise in in such casesgiven an initial non-degenerate guess (thg(th)one can obtain more appropriate and non-degenerate approximation to (tby taking (thw (thg( thg(thin ( )giving the following bayesian learner of ( ) ( thg(thdth( gt ( : ( thr ( thg(thdth where ( thg(thdth (tusing bayesformula for probability densitiesl (th tg( thg(thg( ( we can write (thg(th twith this notationwe have the following definitions definition priorlikelihoodand posterior let and :{ (th)th thbe the training set and family of approximating functions prior pdf (ththat reflects our priori beliefs about th is called the prior pdf likelihood the conditional pdf ( this called the likelihood posterior inference about th is given by the posterior pdf (th )which is proportional to the product of the prior and the likelihoodg(th tg( thg(th