issue is analyzed in Appendix E. The comparison consists in sampling, or "discretizing" a continuous source, and calculating the corresponding discrete entropy, H" The discrete entropy is then compared with the differential entropy, H'. As the Appendix shows, the two entropies are shifted from each other by a constant n = - log A, or H" = H' + n. The constant n corresponds to the number of extra bits required to discretize the continuous distribution with bins of size = 1/2" The key issue is that in the integral limit n 0, this constant is infinite! The conclusion is that an infinite discrete sum, corresponding to the integral limit A 0, is associated with an infinite number of degrees of freedom, and hence, with infinite entropy. The differential entropy defined in Eq. (6.1), as applying to any integration domain X, is always finite, however. This proof is left as an (advanced) exercise. In the experimental domain, calculations are always made in discrete steps. If the source is continuous, the experimentally calculated entropy is not H', but its discretized version H", which assumes a heuristic sampling bin A. The constant n = - log A should then be subtracted from the discrete entropy H" in order to obtain the differential entropy H', thus ensuring reconciliation between the two concepts. 3 For multivariate expressions, e.g., two-dimensional, this constant is where J(x/y) is the Jacobian matrix. 4 Excluding the case of a discrete distribution where p(xi) = Sij (Kronecker symbol). 6.1 Entropy of continuous sources The various relations and properties that were obtained in the discrete case between entropy, conditional entropy, joint entropy, relative entropy, and mutual information also apply to the continuous case. To recall, for convenience, these relations and properties are: H(X, Y) H(Y|X) + H(X) (6.9) H(X, Y) = H(X|Y) H(Y), H(X;Y)=H(X)-H(XY) - = =H(Y)-H(YX) (6.10) = H(X) H(Y) - H(X,Y), D(X, Y) = H(X, Y) - H(X; Y) = :H(XY)+H(Y)X, (6.11) D(pllq) > 0, (6.12) H(X;Y) = : D[p(x, y)||p(x)p(y)] (6.13) D[p(x, y)llq