the expected value of this loss (that isthe riskis thus (xf ( `(ge ln (xln dx ( thg( thkullbackleibler divergence ( the integral in ( provides fundamental way to measure the distance between two densities and is called the kullback-leibler (kldivergence between and (thnote that the kl divergence is not symmetric in and (thmoreoverit is always greater than or equal to (see exercise and equal to when (thusing similar notation as for the supervised learning setting in table define gg as the global minimizer of the risk in the class that isgg argmingg `(gif we define th argmin lossf ( ) ( th)argmin ln (xln ( thf (xdx th th argmax (xln ( thdx argmax ln ( th)th th then gg (thand learning gg is equivalent to learning (or estimatingthto learn thfrom training set { xn we then minimize the training lossn lossf (xi ) (xi th)ln (xi thln (xi ) = = = givingn thn :argmax ln (xi thn = th ( as the logarithm is an increasing functionthis is equivalent to thn :argmax th = (xi th)qn maximum likelihood estimate score where = (xi this the likelihood of the datathat isthe joint density of the {xi evaluated at the points {xi we therefore have recovered the classical maximum likelihood estimate of thwhen the risk `( (th)is convex in th over convex set thwe can find the maximum likelihood estimator by setting the gradient of the training loss to zerothat iswe solve (xi th = thwhere ( th:ln ( is the gradient of ln ( thwith respect to th and is often called th the score example (exponential modelsuppose we have the training data tn { xn }which is modeled as realization of positive iid random variablesx xn ~iid (xwe select the class of approximating functions to be the parametric class { ( th sometimes called cross-entropy distance