equal to Shannon's Entropy of (denoted as ). However, if we use a different probability distribution () when creating the entropy encoding scheme, then a larger number of bits will be used (on average) to identify an event from a set of possibilities. This new (larger) number is measured by the cross entropy between and . The cross entropy between two probability distributions ( and ) measures the average number of bits needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution , rather than the "true" distribution . The cross entropy for two distributions and over the same probability space is thus defined as follows. For explicit derivation of this, see the Motivation section above. Under this scenario, relative entropies (kl-divergence) can be interpreted as the extra number of bits, on average, that are needed (beyond ) for encoding the events because of using for constructing the encoding scheme instead of . Bayesian updating In Bayesian statistics, relative entropy can be used as a measure of the information gain in moving from a prior distribution to a posterior distribution: . If some new fact is discovered, it can be used to update the posterior distribution for from to a new posterior distribution using Bayes' theorem: This distribution has a new entropy: which may be less than or greater than the original entropy . However, from the standpoint of the new probability distribution one can estimate that to have used the original code based on instead of a new code based on would have added an expected number of bits: to the message length. This therefore represents the amount of useful information, or information gain, about , that has been learned by discovering . If a further piece of data, , subsequently comes in, the probability distribution for can be updated further, to give a new best guess . If one reinvestigates the information gain for using rather than , it turns out that it may be either greater or less than previously estimated: may be ≤ or > than and so the combined information gain does not obey the triangle inequality: may be <, = or > than All one can say is that on average, averaging using , the two sides will average out. Bayesian experimental design A common goal in Bayesian experimental design is to