is a dimensioned variable, and are also dimensioned, since e.g. is dimensionless. The argument of the logarithmic term is and remains dimensionless, as it must. It can therefore be seen as in some ways a more fundamental quantity than some other properties in information theory (such as self-information or Shannon entropy), which can become undefined or negative for non-discrete probabilities. Relative entropy is additive for independent distributions in much the same way as Shannon entropy. If are independent distributions, and , and likewise for independent distributions then Relative entropy is convex in the pair of probability measures , i.e. if and are two pairs of probability measures then Duality formula for variational inference The following result, due to Donsker and Varadhan, is known as Donsker and Varadhan's variational formula. For alternative proof using measure theory, see. Examples Multivariate normal distributions Suppose that we have two multivariate normal distributions, with means and with (non-singular) covariance matrices If the two distributions have the same dimension, , then the relative entropy between the distributions is as follows: The logarithm in the last term must be taken to base e since all terms apart from the last are base-e logarithms of expressions that are either factors of the density function or otherwise arise naturally. The equation therefore gives a result measured in nats. Dividing the entire expression above by yields the divergence in bits. In a numerical implementation, it is helpful to express the result in terms of the Cholesky decompositions such that and . Then with and solutions to the triangular linear systems , and , A special case, and a common quantity in variational inference, is the relative entropy between a diagonal multivariate normal, and a standard normal distribution (with zero mean and unit variance): For two univariate normal distributions p and q the above simplifies to In the case of co-centered normal distributions with , this simplifies to: Uniform distributions Consider two uniform distributions, with the support of one () enclosed within the other (). Then the information gain is: Intuitively, the information gain to a times narrower uniform distribution contains bits. This connects with the use of bits in computing, where bits would be needed to identify one element of a long stream. Relation to metrics While relative entropy is a statistical distance, it is not a metric on the space of probability distributions, but instead it is a divergence.