 0.014 1.4144 2 0.020 0.008 0.028 3.4851 4 0.032 0.012 1.4829 2 0.016 0.002 0.009 4.4851 5 0.010 0.004 1.9084 2 0.004 0.002 0.009 4.4851 5 0.010 0.004 1.9084 2 0.004 0.001 0.005 4.9851 5 0.005 0.002 2.1211 3 0.003 0.001 0.005 4.9851 5 0.005 0.002 2.1211 3 0.003 1.000 Entropy H4 = 2.092 L** = 2.472 H26 = 0.890 L** = 1.226 quad/symbol quad/symbol 26-it/symbol 26-it/symbol (source) (code) (source) (code) Coding 84.63% 72.63% efficiency worse for any other M-ary codewords. This property17 17 can easily be explained through the definition of coding efficiency: 17 To my knowledge, this is not found in any textbooks. Information coding p(x)[-logMx+M(x)] (8.18) p(x)logmx+p(x)eM(x) = 1 + which gives, using the relation HM M = H2122, (8.19) It is seen that the coding efficiency decreases in inverse proportion to In M, which for- mally confirms our previous numerical results. However, this decrease is also a function of (EM)X, which represents the mean value of the truncations of - -logm p(x). Both depend on M and the distribution p(x), therefore, there is no general rule for estimating it. We may, however, conservatively assume that for most probability distributions of interest we should have 0.25 < (EM)X < 0.5, which is well satisfied in the example in Table 8.5,1 and which yields the coding efficiency boundaries: (8.20) This last result confirms that the efficiency of Shannon codes is highest for binary codewords and decreases as the reciprocal of the number of alphabet characters used for the codewords (In M/ In 2). I will show next that the Shannon code is generally optimal when considering individ- ual (and not average) codeword lengths, which we have noted 1** (x). Consider, indeed, any different code for which the individual codeword length is 1(x). A first theorem states that the probability P for which 1** (x) > 1(x) +n(n = number of excess bits between the two codes) satisfies P < 1/2n-1.19 Thus, the probability that the Shannon code yields an individual codeword length in excess of two bits (or more) is 1/2 at 