tinuous sources, to which are associated PDFs of continuous variables. It is referred to as differential entropy, and we shall analyze here its properties as well as those of all of its above-listed variants. This will require the use of some integral calculus, but only at a relatively basic level. An interesting issue, which is often overlooked, concerns the comparison between discrete and differential entropies, which is nontrivial. We will review different examples of differential entropy, along with illustrative applications of the KL distance. Then we will address the issue of maximiz- ing differential entropy (finding the optimal PDF corresponding to the upper entropy bound under a set of constraints), as was done in Chapter 4 in the discrete-PDF case. Entropy of continuous sources A continuous source of random events is characterized by a real variable x and its associate PDF, p(x), which is a continuous "density function" over a certain domain X. Several examples of continuous PDFs were described in Chapter 2, including the uniform, exponential, and Gaussian (normal) PDFs. By analogy with the discrete case, one defines the Shannon entropy of a continuous source X according to: H(X)== g(p)x)xx. (6.1) As in the discrete case, the logarithm in the integrand is conventionally chosen in base two, the unit of entropy being bit/symbol. In some specific cases, the defini- tion could preferably involve the natural logarithm, which defines entropy in units of nat/symbol. The entropy of a continuous source, as defined above, is referred to as 6.1 Entropy of continuous sources differential entropy. 1 Similar definitions apply to multivariate PDFs, yielding the joint entropy, H(X, Y), the conditional entropy, H(X|Y), the mutual information, H(X; Y), and the relative entropy or KL distance, D(pllq) with following definitions: H(X,) y)dxdy, (6.2) (6.3) dxdy, (6.4) (6.5) dxdy. (6.6) The above definitions of differential entropies appear to come naturally as the generaliza- tion of the discrete-source case (Chapter 5). Ho