risk and loss in unsupervised learning the random score ( th)with (th)is of particular interest in many casesits expectation is equal to the zero vectornamelyz ( thth eth ( thg( thdx ( th( ( thdx ( thdx th th th provided that the interchange of differentiation and integration is justified this is true for large number of distributionsincluding the normalexponentialand binomial distributions notable exceptions are distributions whose support depends on the distributional parameterfor example the ( thdistribution it is important to see whether expectations are taken with respect to (thor we use the expectation symbols eth and to distinguish the two cases fisher information matrix from now on we simply assume that the interchange of differentiation and integration is permittedseee [ for sufficient conditions the covariance matrix of the random score ( this called the fisher information matrixwhich we denote by or (thto show its dependence on th since the expected score is we have (theth [ ( ths( th) related matrix is the expected hessian matrix of ln ( th) ln ( th ln ( th th **th th ln ( th ln ( ths( th th (th: - th th th ln ( th ln ( thth th th th ( ln ( thth th ln ( thth th ln ( th( th note that the expectation here is with respect to it turns out that if (th)the two matrices are the samethat isf(thh(th)information matrix equality ( provided that we may swap the order of differentiation and integration (expectationthis result is called the information matrix equality we leave the proof as exercise the matrices (thand (thplay important roles in approximating the cross-entropy risk for large to set the scenelet gg (thbe the minimizer of the cross-entropy risk (th:- ln ( thwe assume that ras function of this well-behavedin particularthat in the neighborhood of thit is strictly convex and twice continuously differentiable (this holds truefor exampleif is gaussian densityit follows that this root of ( th)because ln ( thln ( thr(th=- - ( th) th th th