Gibbs sampling equations. This is the case even for many models that are conceptually quite simple, as is demonstrated below in the case of a basic non-hierarchical model with only two parameters and no latent variables. Mathematical derivation Problem In variational inference, the posterior distribution over a set of unobserved variables given some data is approximated by a so-called variational distribution, The distribution is restricted to belong to a family of distributions of simpler form than (e.g. a family of Gaussian distributions), selected with the intention of making similar to the true posterior, . The similarity (or dissimilarity) is measured in terms of a dissimilarity function and hence inference is performed by selecting the distribution that minimizes . KL divergence The most common type of variational Bayes uses the Kullback–Leibler divergence (KL-divergence) of Q from P as the choice of dissimilarity function. This choice makes this minimization tractable. The KL-divergence is defined as Note that Q and P are reversed from what one might expect. This use of reversed KL-divergence is conceptually similar to the expectation-maximization algorithm. (Using the KL-divergence in the other way produces the expectation propagation algorithm.) Intractability Variational techniques are typically used to form an approximation for: The marginalization over to calculate in the denominator is typically intractable, because, for example, the search space of is combinatorially large. Therefore, we seek an approximation, using . Evidence lower bound Given that , the KL-divergence above can also be written as Because is a constant with respect to and because is a distribution, we have which, according to the definition of expected value (for a discrete random variable), can be written as follows which can be rearranged to become As the log-evidence is fixed with respect to , maximizing the final term minimizes the KL divergence of from . By appropriate choice of , becomes tractable to compute and to maximize. Hence we have both an analytical approximation for the posterior , and a lower bound for the log-evidence (since the KL-divergence is non-negative). The lower bound is known as the (negative) variational free energy in analogy with thermodynamic free energy because it can also be expressed as a negative energy plus the entropy of . The term is also known as Evidence Lower Bound, abbreviated as ELBO, to emphasize that it is a lower bound on the log-evidence of the data. Proofs By the generalized Pythagorean theorem of