ur axiomatic principles: (a) Positivity, d(X,Y) 0; (b) Symmetry, d(X, Y) = d(Y, X); (c) Nullity for self, d(X, X) = 0; (d) Triangle inequality, d(X, Z) < d(Y, X) + d(Y, Z). Consider now the quantity D(X, Y), which we define as D(X,Y)=H(X,Y)-H(X;Y) (5.24) From the visual reference of the Venn diagrams in Fig. 5.2 (top), it is readily verified that D(X, Y) satisfies at least the first three above distance axioms (a), (b), and (c). The last axiom, (d), or the triangle inequality, can also be proven through the Venn diagrams when considering three ensembles X, Y, Z, which I leave here as an exercise. Therefore, D(X, Y) represents a distance between the two sources X, Y. It is straightforward to visualize from the Venn diagrams in Fig. 5.2 (top) that D(X,Y)=H(XY)+H(YX), (5.25) Mutual information and more entropies or using the definition for the conditional entropies, Eq. (5.6), and grouping them together, (5.26) We note from the above definition of distance that the weighted sum involves the joint distribution p(x, y). The concept of distance can now be further refined. I shall introduce the relative entropy between two PDFs, which is also called the Kullback-Leibler (KL) distance or the discrimination. Consider two PDFs, p(x) and q(x), where the argument x belongs to a single source X. The relative entropy, or KL distance, is noted D[p(x)llq(x)] and is defined as follows: (5.27) this definition, the continuity limit E log(s) = 0(8 0) applies, while, by convention, we must set E log(e/e') = 0(8, E' The relative entropy is not strictly a distance, since it is generally not symmetric (D(pl|q) #D(q\|p), as the averaging is based on the PDF p or q in the first argument) and, furthermore, it does not satisfy the triangle inequality. It is, however, zero for (D(q/|9) = 0), and it can be verified as an exercise that it is always nonnegative (D[p||q] > 0). An important case of relative entropy is where q(x) is a uniform distribution. If the source X has N events, the uniform PDF is thus defined as q(x) = 1/N. Replacing this 