in these formulae are usually taken to base 2 if information is measured in units of bits, or to base if information is measured in nats. Most formulas involving relative entropy hold regardless of the base of the logarithm. Various conventions exist for referring to in words. Often it is referred to as the divergence between and , but this fails to convey the fundamental asymmetry in the relation. Sometimes, as in this article, it may be described as the divergence of from or as the divergence from to . This reflects the asymmetry in Bayesian inference, which starts from a prior and updates to the posterior . Another common way to refer to is as the relative entropy of with respect to or the information gain from over . Basic example Kullback gives the following example (Table 2.1, Example 2.1). Let and be the distributions shown in the table and figure. is the distribution on the left side of the figure, a binomial distribution with and . is the distribution on the right side of the figure, a discrete uniform distribution with the three possible outcomes , , (i.e. ), each with probability . Relative entropies and are calculated as follows. This example uses the natural log with base e, designated to get results in nats (see units of information). Interpretations Statistics In the field of statistics the Neyman-Pearson lemma states that the most powerful way to distinguish between the two distributions and based on an observation (drawn from one of them) is through the log of the ratio of their likelihoods: . The KL divergence is the expected value of this statistic if is actually drawn from . Kullback motivated the statistic as an expected log likelihood ratio. Coding In the context of coding theory, can be constructed by measuring the expected number of extra bits required to code samples from using a code optimized for rather than the code optimized for . Inference In the context of machine learning, is often called the information gain achieved if would be used instead of which is currently used. By analogy with information theory, it is called the relative entropy of with respect to . Expressed in the language of Bayesian inference, is a measure of the information gained by revising one's beliefs from the prior probability distribution to the posterior probability distribution . In other words, it is the