sidering the fact that language is made out of words, and not single characters, we realize that the previous analysis is incomplete. For instance, we could make an inventory of all words made of two, three, or more characters, and derive a new set of statistics. We could then say how often in English the letter E appears next to the letter A, or what the probability is that a given English word of five characters simultaneously contains the letters A, T, and N, and, given the knowledge that the first two letters are TH, what the probability is that the third one is E, to form the ubiquitous word Mutual information and more entropies This observation leads us to defining the entropy associated with joint events, on one hand, or conditional events on the other hand. To do this, let us briefly recall the properties of joint and conditional probabilities, which have been outlined in Chapter 1, and give them further attention here. Consider two events x E X and y E Y, where X, Y are two random sources. The two events can occur simultaneously, regardless of any sequence order: we just observe x and y, or y and X, which conveys the same information. Alternatively, we can observe x then y, but for some reason we seem never to get y then X. Our observation could also be that whenever x happens, we are likely to see y happen as well. Alternatively, the fact that x happens could have no incidence on the outcome of y. For instance, define the following event sources: X = {quarterly results of a major telecom company}, Y = {stock market}, Z = {weather forecast}, W = {outbound city traffic}. It is safe to say (scientifically speaking!) that events x from X and have no impact whatsoever on events Z or w from Z or W. Thus events x and Z (or x and w) are independent. The probabilities p(x) that x = excellent, and p(z) that Z = cold and rainy also have different and unrelated PDFs. As we know from Chapter 1, the joint probability that we observe both events simultaneously is given by the mere product: p(x,y) = p(excellent, cold a