definition in Eq. (5.27) yields: Pp(((58 = log N - H(X). Since the distance D(pllq) is always nonnegative, it follows from the above that H(X) < log N. This result shows that the entropy of a source X with N elements has log N for its upper bound, which (in the absence of any other constraint) represents the entropy maximum. This is consistent with the conclusion reached earlier in Chapter 4, where I addressed the issue of maximizing entropy for discrete sources. Assume next that p and q are joint distributions of two variables X, y. Similarly to the definition in Eq. (5.27), the relative entropy between the two joint distributions is: (5.29) We note that the expectation value is computed over the distribution p(x,y), and, therefore, D[p(x, y)llq(x,y)] # D[q(x, 011p(x,y)) in the general case. 5.3 Relative entropy The relative entropy is also related to the mutual information. Indeed, recalling the definition of mutual information, Eq. (2.37), we get (5.30) which shows that the mutual information between two sources X, Y is the relative entropy (or KL distance) between the joint distribution p(x, y) and the distribution product p(x)p(y). Since the relative entropy (or KL distance) is always nonnegative (D(.I|.) > 0), it follows that mutual information is always nonnegative (H(X;) Y) > 0). This is consistent with results obtained in Section 5.2. Indeed, recalling the chain rules in Eq. (5.13) and combining them with the property of nonnegativity (obtained in section herein) yields: H(X;Y)=H(X)-H(XY)=H(Y)-H(YX)0, (5.31) which thus implies the two inequalities H(X|Y) < H(X) (5.32) H(Y|X) < H(Y). The above result can be summarized under the fundamental conclusion, which has already been established: conditioning reduces entropy. Thus, given two sources X, Y, the information we obtain from source X given the prior knowledge of the information from Y is less than or equal to that available from X alone, meaning that entropy has been reduced by the fact of conditioning. The strict inequality applies in the case where the