three Lagrange parameters 20, A1, , 22 from the constraints in Eq. (6.23) is a bit tedious, albeit elementary, which I leave as a good math-training exercise. We eventually find (6.34) Substitution of the above Lagrange parameters in Eqs. (6.32)-(6.33) yields: (6.35) and with o2 (x2) - (x)2, as defined in nat/symbol: Hmax = In / 2reo2 (6.36) We recognize in Eq. (6.35) the definition of the Gaussian (normal) distribution. It is a fundamental result in information theory that the optimal continuous PDF for which entropy is maximized, under a constraint in the first two moments ((x), o2), is precisely the Gaussian (normal) distribution. The entropy of any continuous PDF (as defined over the event space X = [-00, +oo]), therefore, has Hmax = In ² as an upper bound. Differential entropy The issue of entropy maximization is key to the solution of several problems in statistical physics and engineering (and many other fields as well) where the PDF is unknown, while a certain set of constraints gk(x), (gk) are known from real-life or experimental observation. See previous discussion in Chapter 4 regarding the maximum entropy principle. Exercises 6.1 (B): Calculate the differential entropy of the continuous source defined over the real interval X = [a, b], with a # b, assuming a uniform PDF. 6.2 (B): Calculate the differential entropy associated with an exponential PDF. 6.3 (T): Show that the differential entropy with < p(x) < 1, is always finite. 6.4 (T): Assume the probability distribution p(x) = ae-ix, which is defined for x E [0, m],m, a, a being nonnegative real. (a) Determine the PDF constant a, the PDF mean (x) and the PDF entropy H. (b) Show that in the limit 2 0 the PDF becomes uniformly distributed with p(x) 1/m. (c) Calculate the relative entropy, or KL distance D[pl|q] between the two PDFs, q(x) = ae-ix = Be-ux (d) Determine the KL distance D[pllq] in thelimità->0orq(x)->1/m. (e) Show that in the limit A, u 0 the distance D[p||q] vanishes. 6.5 (M): Show that the relative entropy, or KL distance, between any co