Bregman divergence, of which KL-divergence is a special case, it can be shown that: where is a convex set and the equality holds if: In this case, the global minimizer with can be found as follows: in which the normalizing constant is: The term is often called the evidence lower bound (ELBO) in practice, since , as shown above. By interchanging the roles of and we can iteratively compute the approximated and of the true model's marginals and respectively. Although this iterative scheme is guaranteed to converge monotonically, the converged is only a local minimizer of . If the constrained space is confined within independent space, i.e. the above iterative scheme will become the so-called mean field approximation as shown below. Mean field approximation The variational distribution is usually assumed to factorize over some partition of the latent variables, i.e. for some partition of the latent variables into , It can be shown using the calculus of variations (hence the name "variational Bayes") that the "best" distribution for each of the factors (in terms of the distribution minimizing the KL divergence, as described above) can be expressed as: where is the expectation of the logarithm of the joint probability of the data and latent variables, taken over all variables not in the partition: refer to for a derivation of the distribution . In practice, we usually work in terms of logarithms, i.e.: The constant in the above expression is related to the normalizing constant (the denominator in the expression above for ) and is usually reinstated by inspection, as the rest of the expression can usually be recognized as being a known type of distribution (e.g. Gaussian, gamma, etc.). Using the properties of expectations, the expression can usually be simplified into a function of the fixed hyperparameters of the prior distributions over the latent variables and of expectations (and sometimes higher moments such as the variance) of latent variables not in the current partition (i.e. latent variables not included in ). This creates circular dependencies between the parameters of the distributions over variables in one partition and the expectations of variables in the other partitions. This naturally suggests an iterative algorithm, much like EM (the expectation-maximization algorithm), in which the expectations (and possibly higher moments) of the latent variables are initialized in some fashion (perhaps randomly), and then the parameters of each distribution are computed in turn using the current