itional entropy is given by the contributions of all the events belonging to U but not to V. This property is far from obvious, unless we can visualize it. Figure 5.2 illustrates all the above logical equivalences through Venn diagrams, using up to three sources. Considering the two-source case, we can immediately visualize from Fig. 5.2 to which subsets the differences H(X) - H(X;) and H(Y) - H(X;Y) actually correspond. Given the identities listed in Eq. (5.15), we can call these two subsets H(X|Y) and H(Y|X), respectively, which proves the previous point. We also observe from the Venn Mutual information and more entropies diagram that H(X|Y) < H(X) and H(Y|X) < H(Y), with equality if the sources are independent. The above property can be summarized by the statement according to which condi- tioning reduces entropy. A formal demonstration, using the concept of "relative entropy," is provided later. The three-source case, as illustrated in Fig. 5.2, is somewhat more tricky because it generates more complex entropy definitions with three arguments X, Y, and Z. Con- ceptually, defining joint or conditional entropies and mutual information with three (or more) sources is not this difficult. Considering the joint probability p(x, y, z) for the three sources, we can indeed generalize the previous two-source definitions according to the following: (5.16) XEX yey (5.17) p(x)p(y)p(z) xEX yey zeZ H(ZX,Y)=-2p(x,y,z)logp(zx,y), (5.18) XEX yey ZEZ (5.19) XEX yey zeZ These four definitions correspond to the joint entropy of the three sources X, Y, Z (Eq. (5.16)), the mutual information of the three sources X, Y, Z (Eq. (5.17)), the entropy of source Z given the known entropy of X, Y (Eq. (5.18)), and the joint entropy of X, Y given the known entropy of Z (Eq. (5.19)). The last two definitions are seen to involve conditional probabilities of higher orders, namely, p(z/x, y) and p(x, y|z), which are easily determined from the generalization of Bayes's theorem.5 Other entropies of the type H(X; Y|Z) and H(X); Z) are more tricky 