to determine from the above definitions. But we can resort in all confidence to the equivalence relations and the corresponding two-source or three-source Venn diagrams shown in Fig. 5.2. Indeed, a straightforward observation of the diagrams leads to the following correspondences: H(X;YZ)=H(X;Y)-H(Z), (5.20) H(XY;Z)=H(X)-H(Y;Z). (5.21) Finally, the Venn diagrams (with the help of Eq. (5.15) make it possible to establish the following properties for H(X, Y|Z) and H(X), Z). The first chain rule is H(X,Y|Z) = H(X|Z) + H(Y|X, Z), (5.22) 5 As we have p(x,y,z) = p(z|x,y)p(x)p(y) -> p(z/x,y = p(x, y, z)/[p(x)p(y)] and p(x,y,z) = p(x,y|z)p(z) -> p(x, y|z) == p(x,y,z)/p(z). 5.3 Relative entropy which is easy to memorize if a condition |z is applied to both sides of the definition of joint entropy, H(X, Y) = H(X) + H(Y|X). The second chain rule, H(X,Y|Z) = H(Y|Z) + H(X),2 (5.23) comes from the permutation in Eq. (5.22) of the sources X, Y, since the joint entropy H(X, Y) is symmetrical with respect to the arguments. The lesson learnt from using Venn diagrams is that there is, in fact, little to memorize, as long as we are allowed to make drawings! The only general rule to remember is: H(U|Z) is equal to the entropy H(U) defined by the source U (for instance, U = X, Y or U = X; Y) minus the entropy H(Z) defined by the source Z, the reverse being true for H(Z|U). But the use of Venn diagrams require us not to forget the unique correspondence between the ensemble or Boolean operators (un -) and the separators (, ; |) in the entropy-function arguments. Relative entropy In this section, I introduce the notion of distance between two event sources and the associated concept of relative entropy. The mathematical concept of distance between two real variables x,y is famil- iarly known as the quantity d = |x - y|. For two points A, B in the plane, with coordinates (XA,YA) and (XB,YB), respectively, the distance is defined as d = - XB)2 + (YA - More generally, any definition of distance d(X, Y) between two entities X, Y must obey fo