ssume next that the probability of any bit in the sequence A summary is provided in T.M. Cover and J.A. Thomas, Elements of Information Theory (New York: John Wiley & Sons, 1991), p. 191. The notation Xk refers to the extended source corresponding to k repeated observations or uses of the source 8 Such a property comes from the definition of joint entropy, assuming two sources with independent events: = p(x) p(y)] = H(X) H(Y). With the extended source X2, we obtain H(X2) = 2H(X), and consequently 13.2 Typical sequences and the typical set being one is p(x; = 1) = q. We, thus, expect that any sequence roughly contains kq bits equal to one, and k(1 - q) bits equal to zero. 9 This property becomes more accurately verified as the sequence length k is sufficiently large, as we shall see later. The probability of a sequence O containing exactly kq 1 bits and k(1 - q) 0 bits is (13.21) where m = CK is the number of possible O sequences. Taking the minus logarithm (base 2) of both sides in Eq. (13.21) yields log(1 (13.22) +kH(X) log - + H(X*). The result obtained in Eq. (13.22) shows that the probability of obtaining a sequence 0 (13.23) Thus, all O sequences are equiprobable, and each individual sequence in this set of size m has the probability p(x) = p(f)/m = 2-kH(X) = = 2-H(X*) We shall now (tentatively) call any 0 a typical sequence, and the set of such sequences A, the typical set. An example of a typical set (as tentatively defined) is provided in Fig. 13.4. In this example, the parameters are chosen to be k = 6 and q = 1/3. For each sequence O containing j bits equal to one (j = 0,1,...,6), the corresponding proba- bility was calculated according to p(x) = qqj(1-q)k-j. - The source entropy is calcu- lated to be H(X) = kH(X) = kf(q) = 6 (f(1/3) = 6x 0.918 = 5.509 bit. The typical set, thus, comprises m = 15 typical sequences with equal probabilities p(x) = = = The figure shows the log probabilities of the entire set of 26 = 64 possible sequences, along with the relative size of the typi- cal set and associated pro