EX" xjEX" )logp'(xj) xj€X" In Eq. (E6), the first term represents the Riemann integral of the function - p(x) log p'(x). This Riemann integral converges to the integral - I p(x) log p'(x)dx = H(X') as the bin size is made to vanish (A 0), which is the continuous entropy. But in such a limit, the second term in -log A becomes infinite! This divergence reflects the fact that what we have done is an n-bit quantization of the continuous function p'(x). In such a quantization, the number of bits is n = - log A (or A = 2-n). Thus, the result in Eq. (E6) shows that the relation between the discrete entropy H(X") and what we have defined to be the continuous entropy H(X) is actually H(X") = H(X') + n. The quantity H(X') + n can be interpreted as the number of bits required (on average) to describe the continuous random variable X' with n-bit accuracy. To understand what "n-bit accuracy" means, consider that A = 1, or n = 0, corresponds to integers. Then A = 1/2, or n = 1, corresponds to numbers increasing by steps of 0.5. It takes 1 bit of extra accuracy to specify whether x is rounded to an integer I or to I + 0.5. Then A = 1/4 or n = 2 corresponds to numbers increasing by steps of 0.25. It takes 2 bits of extra accuracy to specify the value of x out of the four rounded cases I, I + 0.25, I + 0.5, or I + 0.75. Thus, n-bit accuracy corresponds to the number of extra bits required to be accurate within any incremental power of 1/2. The divergence in - log A, which makes the discrete-case entropy infinite, reflects the fact that it takes an infinite number of bits to discretize a continuous set of variables accurately. Appendix F (Chapter 8) Kraft-McMillan inequality Assume a prefix code made of a set of N codewords C1, C2 CN of various lengths 11, l2...IN satisfying l < l2 < < If the code is binary, the Kraft-McMillan inequality is and for M-ary code: To prove this inequality, I shall use the demonstration of Proakis (2001), 1 one of several other variations to be found in the literature. For simplicity, we consider the c