be maximized, and to which event would the maximum correspond? To answer these questions, I shall proceed from the simple to the general, then to the more complex. I consider first a system with two events, then with k events, then with an infinite number of discrete events. Finally, I introduce some constraints in the entropy maximization problem. Assume first two complementary events X1, X2 with probabilities p(x1) = q and p(x2) = 1 - q, respectively. By definition, the entropy of the source X = {x1,x2} is given by = -x1 (4.13) -qlogq-(1-q)log(1-q)=fq) 14 The entropy of the sample paragraph is H = 4.137 bit/symbol, which indicates that the sample is reasonably close to the reference (H(1982) = 4.185 bit/symbol), meaning that there is no parasitic effect due to the author's choice of the subject or his own use of words. Entropy Note the introduction of the function f(q): f(q) = -qlogq-(1-q)log(1-q)=f(q) (4.14) which will be used several times through these chapters. A first observation from the definition of H(X) = f(q) is that if one of the two events becomes "impossible," i.e., q = E -> 0 or 1 q=E-> 0, the entropy remains bounded. Indeed, the corresponding term vanishes, since E log 0 when E 0. In such a limit, however, the other term corresponding to the complementary event, which becomes "absolutely certain," also vanishes (since u log u 0 when U 1). Thus, in this limit the entropy also vanishes, or H(X) 0, meaning that the source's information is identical to zero as a statistical average. This situation of zero entropy corresponds to a fictitious system, which would be frozen in a state of either "impossibility" or "absolute certainty." We assume next that the two events are equiprobable, i.e., q = 1/2. We then obtain from Eq. (4.13): s)111811(415 The result is that the source's average information (its entropy) is exactly one bit. This means that it takes a single bit to define the system: either event X1 or event X2 is observed, with equal likelihood. The source information is, thus, given by a simple YE