tech/math/prime/mersenne.html#M32582657 http://primes.utm.edu/largest.html. For expediency, reduce the text source to only a few ten thousand digits. Compute the entropy of the source. Could the result have been guessed directly? (Computing method clue: see Chapter 4, note 11.) Mutual information and more entropies This chapter marks a key turning point in our journey in information-theory land. Heretofore, we have just covered some very basic notions of IT, which have led us, nonetheless, to grasp the subtle concepts of information and entropy. Here, we are going to make significant steps into the depths of Shannon's theory, and hopefully begin to appreciate its power and elegance. This chapter is going to be somewhat more mathematically demanding, but it is guaranteed to be not significantly more complex than the preceding materials. Let's say that there is more ink involved in the equations and the derivation of the key results. But this light investment will turn out well worth it to appreciate the forthcoming chapters! I will first introduce two more entropy definitions: joint and conditional entropies, just as there are joint and conditional probabilities. This leads to a new fundamental notion, that of mutual information, which is central to IT and the various Shannon's laws. Then I introduce relative entropy, based on the concept of "distance" between two PDFs. Relative entropy broadens the perspective beyond this chapter, in particular with an (optional) appendix exploration of the second law of thermodynamics, as analyzed in the light of information theory. Joint and conditional entropies So far, in this book, the notions of probability distribution and entropy have been associated with single, independent events X, as selected from a discrete source X = {x}. With the example of written language (Chapter 4), we have seen that the occurrence single characters from the alphabetical-character source X = {A, , Z} is bound to a quasi-exponential PDF, which varies according to language, context, and time. Con