 the fact that independent sources do not have any information in common. It takes a bit of painstaking but straightforward computation to show the following three equalities: H(X;Y)=H(X)-H(XY) = H(Y) - H(Y|X) (5.13) = H(X) + H(Y) H(X, Y). The first two equalities above confirm the observation derived from our previous numer- ical example. They can be interpreted according to the following statement: mutual information is the reduction of uncertainty in X that we get from the knowledge of Y (and the reverse). The last equality, as rewritten under the form H(X,Y)=H(X)+H(Y)-H(X;Y), (5.14) shows that the joint entropy of two sources is generally less than the sum of the source entropies. The difference is the mutual information that the sources have in common, which reduces the net uncertainty or joint entropy. Finally, we note from the three relations in Eq. (5.13) that the mutual information is symmetrical in the arguments, namely, H(X; Y) = H(Y; X), as is expected from its very meaning. The different entropy definitions introduced up to this point may seem a bit abstract and their different relations apparently not very practical to memorize! But the situation becomes different after we draw an analogy with the property of ensembles. Consider, indeed, two ensembles, called A and B. The two ensembles may be united to form a whole, which is noted F = AUB (A union B). The two ensembles may or may not have elements in common. The set of common elements is called G = ANB (A intersection B). The same definitions of union and intersection apply to any three ensembles A, B, and C. Figure 5.1 shows Venn diagram representations of such ensemble combinations. While Venn diagrams were introduced in Chapter 1, I shall provide further Mutual information and more entropies AnBn-C AUBUC An-Bn-0 AnBnC Figure 5.1 Venn diagrams representing two (A,B) or three (A, B, C) ensembles with their unions (AUB or AUBUC) and their intersections (An B or AnBnC). The intersections defined by An-B,Bn-A, andAN-Bn-C,ANBN-Care also shown. related 