 two sources have nonzero mutual information (H(X; Y) > 0). If the two sources are disjoint, or made of independent events, then the equality applies, and conditioning from Y has no effect on the information of X. Next, I shall introduce another definition, which is that of conditional relative entropy, given the joint distributions p and q over the space {X, Y}: (5.33) xeX yey noting that averaging is made through the joint distribution, p(x, y). This definition is similar to that of the relative entropy for joint distributions, Eq. (5.29), except that it applies here to conditional joint distributions. From Eqs. (5.27), (5.29), and (5.33), we can derive the following relation, or chain rule, between the relative entropies of two single-variate distributions p(x), q(x) and their two-variate or joint distributions p(x,y), q(x,y D[p(x,y)llq(x,y)]=D[p(x)llq(x)]+D[p(yx)llq(yx)]. (5.34) Mutual information and more entropies This chain rule can be memorized if one decomposes p(x,y) and q(x, y) through Bayes's theorem, i.e., p(x, y) = p(y|x)p(x) and q(x,y) = q(y|x)q(x), and rearranges the four factors into two distance terms D [*l/*]. A similar chain rule thus also applies with x and y being switched in the right-hand side. The usefulness of relative entropy and conditional-relative entropy can only be appre- ciated at a more advanced IT level, which is beyond the scope of these chapters. To satisfy a demanding student's curiosity, however, an illustrative example concerning the second law of thermodynamics is provided. For this, one needs to model the time evolution of probability distributions, which involves Markov chains. The concept of Markov chains and its application to the second law of thermodynamics are described in Appendix D, which is to be regarded as a tractable, bur somewhat advanced topic. Exercises 5.1 (B): Given all possible events a E A, and the conditional probability p(a)b) for any event b E B, demonstrate the summation property: p(ab)=1. (M): If two sources X, Y represent independent events, then 