prove that (a) H(X, Y) = H(X) + H(Y), (b) H(X|Y) = H(X) and H(Y|X) = H(Y). 5.3 (M): Prove the entropy chain rules, which apply to the most general case: H(X,Y)=H(YX)+H(X) H(X,Y)=H(XY)+. H(Y). 5.4 (M): Prove that for any single-event source X, H(X|X) = 0. 5.5 (M): Prove that for any single-event source X, H(X;X) = H(X). 5.6 (M): Establish the following three equivalent properties for mutual information: H(X;Y)=H(X)-H(XY) =H(Y)-H(Y)X) = H(X)+H(Y)-H(X, Y). 5.7 (M): Two sources, A = {a1,a2} and B = {b1, b2}, have a joint probability distri- bution defined as follows: p(a,b1)=0.3,p(a,b2)=0.4, = = p(a2,b1 = 0.1, p(a2, b2) = 0.2. Calculate the joint entropy H(A, B), the conditional entropy H(A|B) and H(B|A), and the mutual information H(A; B). 5.4 Exercises 5.8 (B): Show that D(X, Y), as defined by D(X,Y)=H(X,Y)-H(X;Y), can also be expressed as: D(X,Y)=H(XY)H(YX). 5.9 (T): With the use of the Venn diagrams, prove that D(X,Y)=H(X,Y)-H(X;Y) satisfies the triangle inequality, d(X,Z)<d(Y,X)+d(Y,Z), for any sources X,Y,Z. 5.10 (M): Prove that the Kullback-Leibler distance between two PDFs, p and q, is always nonnegative, or D[p||q] > 0. Clue: assume that f = p/q satisfies Jensen's inequality, (f(u)) (f ( (u)) Differential entropy So far, we have assumed that the source of random symbols or events, X, is discrete, meaning that the source is made of a set (finite or infinite) of discrete elements, Xj. To such a discrete source is associated a PDF of discrete variable, p(x = xj), which I have called p(x), for convenience. In Chapter 4, I have defined the source's entropy according to Shannon as H(X) = - -p(xi) logp(xi), and described several other entropy vari- ants for multiple discrete sources, such as joint entropy, H(X, Y), conditional entropy, H(X|Y), mutual information, H(X; Y), relative entropy (or Kullback-Leibler [KL] distance) for discrete single or multivariate PDFs, D[p||q], and conditional relative entropy, D[p(y)x)||q(y)x)]. In this chapter, we shall expand our conceptual horizons by considering the entropy of con