Question: Represent the scientific article to find a short TLDR

Question: The standard variational lower bounds used to train latent variable models produce biased estimates of most quantities of interest. We introduce an unbiased estimator of the log marginal likelihood and its gradients for latent variable models based on randomized truncation of infinite series. If parameterized by an encoder-decoder architecture, the parameters of the encoder can be optimized to minimize its variance of this estimator. We show that models trained using our estimator give better test-set likelihoods than a standard importance-sampling based approach for the same average computational cost. This estimator also allows use of latent variable models for tasks where unbiased estimators, rather than marginal likelihood lower bounds, are preferred, such as minimizing reverse KL divergences and estimating score functions. Latent variable models are powerful tools for constructing highly expressive data distributions and for understanding how high-dimensional observations might possess a simpler representation. Latent variable models are often framed as probabilistic graphical models, allowing these relationships to be expressed in terms of conditional independence. Mixture models, probabilistic principal component analysis , hidden Markov models, and latent Dirichlet allocation are all examples of powerful latent variable models. More recently there has been a surge of interest in probabilistic latent variable models that incorporate flexible nonlinear likelihoods via deep neural networks . These models can blend the advantages of highly structured probabilistic priors with the empirical successes of deep learning . Moreover, these explicit latent variable models can often yield relatively interpretable representations, in which simple interpolation in the latent space can lead to semantically-meaningful changes in high-dimensional observations (e.g.,). It can be challenging, however, to fit the parameters of a flexible latent variable model, since computing the marginal probability of the data requires integrating out the latent variables in order to maximize the likelihood with respect to the model parameters. Typical approaches to this problem include the celebrated expectation maximization algorithm , Markov chain Monte Carlo, and the Laplace approximation. Variational inference generalizes expectation maximization by forming a lower bound on the aforementioned (log) marginal likelihood, using a tractable approximation to the unmanageable posterior over latent variables. The maximization of this lower bound-rather than the true log marginal likelihood-is often relatively straightforward when using automatic differentiation and Monte Carlo sampling. However, a lower bound may be ill-suited for tasks such as posterior inference and other situations where there exists an entropy maximization objective; for example in entropy-regularized reinforcement learning (; ;) which requires minimizing the log probability of the samples under the model. While there is a long history in Bayesian statistics of estimating the marginal likelihood (e.g., ;), we often want high-quality estimates of the logarithm of the marginal likelihood, which is better behaved when the data is high dimensional; it is not as susceptible to underflow and it has gradients that are numerically sensible. However, the log transformation introduces some challenges: Monte Carlo estimation techniques such as importance sampling do not straightforwardly give unbiased estimates of this quantity. Nevertheless, there has been significant work to construct estimators of the log marginal likelihood in which it is possible to explicitly trade off between bias against computational cost (; ;). Unfortunately, while there are asymptotic regimes where the bias of these estimators approaches zero, it is always possible to optimize the parameters to increase this bias to infinity. In this work, we construct an unbiased estimator of the log marginal likelihood. Although there is no theoretical guarantee that this estimator has finite variance, we find that it can work well in practice. We show that this unbiased estimator can train latent variable models to achieve higher test log-likelihood than lower bound estimators at the same expected compute cost. More importantly, this unbiased estimator allows us to apply latent variable models in situations where these models were previously problematic to optimize with lower bound estimators. Such applications include latent variable modeling for posterior inference and for reinforcement learning in high-dimensional action spaces, where an ideal model is one that is highly expressive yet efficient to sample from. Latent variable models (LVMs) describe a distribution over data in terms of a mixture over unobserved quantities. Let p θ (x) be a family of probability density (mass) functions on a data space X, indexed by parameters θ. We will generally refer to this as a "density" for consistency, even when the data should be understood to be discrete; similarly we will use integrals even when the marginalization is over a discrete set. In a latent variable model, p θ (x) is defined via a space of latent variables Z, a family of mixing measures on this latent space with density denoted p θ (z), and a conditional distribution p θ (x | z). This conditional distribution is sometimes called an "observation model" or a conditional likelihood. We will take θ to parameterize both p θ (x | z) and p θ (z) in the service of determining the marginal p θ (x) via the mixture integral: This simple formalism allows for a large range of modeling approaches, in which complexity can be baked into the latent variables (as in traditional graphical models), into the conditional likelihood (as in variational autoencoders), or into both (as in structured VAEs). The downside of this mixing approach is that the integral may be intractable to compute, making it difficult to evaluate p θ (x)-a quantity often referred to in Bayesian statistics and machine learning as the marginal likelihood or evidence. Various Monte Carlo techniques have been developed to provide consistent and often unbiased estimators of p θ (x), but it is usually preferable to work with log p θ (x) and unbiased estimation of this quantity has, to our knowledge, not been previously studied. Fitting a parametric distribution to observed data is often framed as the minimization of a difference between the model distribution and the empirical distribution. The most common difference measure is the forward Kullback-Leibler (KL) divergence; if p data (x) is the empirical distribution and p θ (x) is a parametric family, then minimizing the KL divergence (D KL) with respect to θ is equivalent to Since expectations can be estimated in an unbiased manner using Monte Carlo procedures, simple subsampling of the data enables powerful stochastic optimization techniques, with stochastic gradient descent in particular forming the basis for learning the parameters of many nonlinear models. However, this requires unbiased estimates of ∇ θ log p θ (x), which are not available for latent variable models. Instead, a stochastic lower bound of log p θ (x) is often used and then differentiated for optimization. Though many lower bound estimators (; ;) are applicable, we focus on an importance-weighted evidence lower bound . This lower bound is constructed by introducing a proposal distribution q(z; x) and using it to form an importance sampling estimate of the marginal likelihood: If K samples are drawn from q(z; x) then this provides an unbiased estimate of p θ (x) and the biased "importance-weighted autoencoder" estimator IWAE K (x) of log p θ (x) is given by The special case of K = 1 generates an unbiased estimate of the evidence lower bound (ELBO), which is often used for performing variational inference by stochastic gradient descent. While the IWAE lower bound acts as a useful replacement of log p θ (x) in maximum likelihood training, it may not be suitable for other objectives such as those that involve entropy maximization. We discuss tasks for which a lower bound estimator would be ill-suited in Section 3.4. There are two properties of IWAE that will allow us to modify it to produce an unbiased estimator: First, it is consistent in the sense that as the number of samples K increases, the expectation of IWAE K (x) converges to log p θ (x). Second, it is also monotonically non-decreasing in expectation: These properties are sufficient to create an unbiased estimator using the Russian roulette estimator. In order to create an unbiased estimator of the log probability function, we employ the Russian roulette estimator . This estimator is used to estimate the sum of infinite series, where evaluating any term in the series almost surely requires only a finite amount of computation. Intuitively, the Russian roulette estimator relies on a randomized truncation and upweighting of each term to account for the possibility of not computing these terms. To illustrate the idea, let∆ k denote the k-th term of an infinite series. Assume the partial sum of the series ∞ k=1∆ k converges to some quantity we wish to obtain. We can construct a simple estimator by always computing the first term then flipping a coin b ∼ Bernoulli(q) to determine whether we stop or continue evaluating the remaining terms. With probability 1 − q, we compute the rest of the series. By reweighting the remaining future terms by 1 /(1−q), we obtain an unbiased estimator: To obtain the "Russian roulette" (RR) estimator , we repeatedly apply this trick to the remaining terms. In effect, we make the number of terms a random variable K, taking values in 1, 2,... to use in the summation (i.e., the number of successful coin flips) from some distribution with probability mass function p(K) = P(K = K) with support over the positive integers. With K drawn from p(K), the estimator takes the form: The equality on the right hand of equation 7 holds so long as (i) P(K ≥ k) > 0, ∀k > 0, and (ii) the series converges absolutely, i.e.,; Lemma 3). This condition ensures that the average of multiple samples will converge to the value of the infinite series by the law of large numbers. However, the variance of this estimator depends on the choice of p(K) and can potentially be very large or even infinite (; ; We can turn any absolutely convergent series into a telescoping series and apply the Russian roulette randomization to form an unbiased stochastic estimator. We focus here on the IWAE bound described in Section 2.2. converges absolutely, we apply equation 7 to construct our estimator, which we call SUMO (Stochastically Unbiased Marginalization Objective). The detailed derivation of SUMO is in Appendix A.1. The randomized truncation of the series using the Russian roulette estimator means that this is an unbiased estimator of the log marginal likelihood, regardless of the distribution p(K): where the expectation is taken over p(K) and q(z; x) (see Algorithm 1 for our exact sampling procedure). Furthermore, under some conditions, we have To efficiently optimize a limit, one should choose an estimator to minimize the product of the second moment of the gradient estimates and the expected compute cost per evaluation. The choice of p(K) effects both the variance and computation cost of our estimator. DenotingĜ:= ∇ θŶ and ∆, the Russian roulette estimator is optimal across a broad family of unbiased randomized truncation estimators if the ∆ g k are statistically independent, in which case it has second moment E||Ĝ|| 2/P(K≥k) . While the Algorithm 1 Computing SUMO, an unbiased estimator of log p(x). Input: are not in fact strictly independent with our sampling procedure (Algorithm 1), and other estimators within the family may perform better, we justify our choice by showing that E∆ i ∆ j for i = j converges to zero much faster than E∆ 2 k (Appendices A.2 & A.3). In the following, we assume independence of ∆ g k and choose p(K) to minimize the product of compute and variance. We first show that E||∆.5 ). This implies the optimal compute-variance product is given by 2 ). In our case, this gives P(K ≥ k) = 1 /k, which in an estimator with infinite expected computation and no finite bound on variance. In fact, any p(K) which gives rise to provably finite variance requires a heavier tail than P(K ≥ k) = 1 /k and so will have infinite expected computation. Though we could not theoretically show that our estimator and gradients have finite variance, we empirically find that gradient descent converges -even in the setting of minimizing log probability. We plot ||∆ k || 2 2 for the toy variational inference task used to assess signal to noise ratio in and Rainforth et al. (2018b), and find that they converge faster than 1 k 2 in practice (Appendix A.6). While this indicates the variance is better than the theoretical bound, an estimator having infinite expected computation cost will always be an issue as it indicates significant probability of sampling arbitrarily large K. We therefore modify the tail of the sampling distribution such that the estimator has finite expected computation: We typically choose α = 80, which gives an expected computation cost of approximately 5 terms. One way to improve the RR estimator is to construct it so that some minimum number of terms (denoted here as m) are always computed. This puts a lower bound on the computational cost, but can potentially lower variance, providing a design space for trading off estimator quality against computational cost. This corresponds to a choice of RR estimator in which This computes the sum out to m terms (effectively computing IWAE m) and then estimates the remaining difference with Russian roulette: In practice, instead of tuning parameters of p(K), we set m to achieve a given expected computation cost per estimator evaluation for fair comparison with IWAE and related estimators. The SUMO estimator does not require amortized variational inference, but the use of an "encoder" to produce an approximate posterior q(z; x) has been shown to be a highly effective way to perform rapid feedforward inference in neural latent variable models. We use φ to denote the parameters of the encoder q φ (z; x). However, the gradients of SUMO with respect to φ are in expectation zero precisely because SUMO is an unbiased estimator of log p θ (x), regardless of our choice of q φ (z; x). Nevertheless, we would expect the choice of q φ (z; x) significantly impacts the variance of our estimator. As such, we optimize q φ (z; x) to reduce the variance of the SUMO estimator. We can obtain unbiased gradients in the following way : Notably, the expectation of this estimator depends on the variance of SUMO, which we have not been able to bound. In practice, we observe gradients which are sometimes very large. We apply gradient clipping to the encoder to clip gradients which are excessively large in magnitude. This helps stabilize the training progress but introduces bias into the encoder gradients. Fortunately, the encoder itself is merely a tool for variance reduction, and biased gradients with respect to the encoder can still significantly help optimization. Here we list some applications where an unbiased log probability is useful. Using SUMO to replace existing lower bound estimates allows latent variable models to be used for new applications where a lower bound is inappropriate. As latent variable models can be both expressive and efficient to sample from, they are frequently useful in applications where the data is high-dimensional and samples from the model are needed. Minimizing log p θ (x). Some machine learning objectives include terms that seek to increase the entropy of the learned model. The "reverse KL" objective-often used for training models to perform approximate posterior inferences-minimizes E x∼p θ (x) [log p θ (x) − log π(x)] where π(x) is a target density that may only be known up a normalization constant. Local updates of this form are the basis of the expectation propagation procedure . This objective has also been used for distilling autoregressive models that are inefficient at sampling . Moreover, reverse KL is connected to the use of entropy-regularized objectives (; ; ;) in decision-making problems, where the goal is to encourage the decision maker toward exploration and prevent it from settling into a local minimum. Unbiased score function ∇ θ log p θ (x). The score function is the gradient of the log-likelihood with respect to the parameters and has uses in estimating the Fisher information matrix and performing stochastic gradient Langevin dynamics , among other applications. Of particular note, the REINFORCE gradient estimator -generally applicable for optimizing objectives of the form max θ E x∼p θ (x) [R(x)]-is estimated using the score function. This can be replaced with the gradient of SUMO which itself is an estimator of the score func- where the inner expectation is over the stochasticity of the SUMO estimator. Such estimators are often used for reward maximization in reinforcement learning where p θ (x) is a stochastic policy. There is a long history in Bayesian statistics of marginal likelihood estimation in the service of model selection. The harmonic mean estimator , for example, has a long (and notorious) history as a consistent estimator of the marginal likelihood that may have infinite variance and exhibits simulation psuedo-bias . The Chib estimator , the Laplace approximation, and nested sampling are alternative proposals that can often have better properties . Annealed importance sampling probably represents the gold standard for marginal likelihood estimation. These, however, turn into consistent estimators at best when estimating the log marginal probability (a). Bias removal schemes such as jackknife variational inference have been proposed to debias log-evidence estimation, IWAE in particular. Hierarchical IWAE uses a joint proposal to induce negative correlation among samples and connects the convergence of variance of the estimator and the convergence of the lower bound. Russian roulette also has a long history. It dates back to unpublished work from von Neumann and Ulam, who used it to debias Monte Carlo methods for matrix inversion and particle transport problems . It has gained popularity in statistical physics (; ;), for unbiased ray tracing in graphics and rendering , and for a number of estimation problems in the statistics community (; ; ; ;). It has also been independently rediscovered many times (; ; ;). The use of Russian roulette estimation in deep learning and generative modeling applications has been gaining traction in recent years. It has been used to solve short-term bias in optimization problems . Though we extend latent variable models to applications that require unbiased estimates of log probability and benefit from efficient sampling, an interesting family of models already fulfill these requirements. Normalizing flows offer exact log probability and certain models have been proven to be universal density estimators . However, these models often require restrictive architectural choices with no dimensionalityreduction capabilities, and make use of many more parameters to scale up than alternative generative models. Discrete variable versions of these models are still in their infancy and make use of biased gradients , whereas latent variable models naturally extend to discrete observations. We first compare the performance of SUMO when used as a replacement to IWAE with the same expected cost on density modeling tasks. We make use of two benchmark datasets: dynamically binarized MNIST and binarized OMNIGLOT . We use the same neural network architecture as IWAE . The prior p(z) is a 50-dimensional standard Gaussian distribution. The conditional distributions p(x i |z) are independent Bernoulli, with the decoder parameterized by two hidden layers, each with 200 tanh units. The approximate posterior q(z; x) is also a 50-dimensional Gaussian distribution with diagonal covariance, whose mean and variance are both parameterized by two hidden layers with 200 tanh units. We reimplemented and tuned IWAE, obtaining strong baseline which are better than those previously reported. We then used the same hyperparameters to train with the SUMO estimator. We find clipping very large gradients can help performance, as large gradients may be infrequently sampled. This introduces a small amount of bias into the gradients while reducing variance, but can nevertheless help achieve faster convergence and should still in a less-biased estimator. A posthoc study of the effect on final test performance as a function of this bias-variance tradeoff mechanism is discussed in Appendix A.7. We note that gradient clipping is only done for the density modeling experiments. The averaged test log-likelihoods and standard deviations over 3 runs are summarized in Table 1. To be consistent with existing literature, we evaluate our model using IWAE with 5000 samples. In all the cases, SUMO achieves slightly better performance than IWAE with the same expected cost. We also bold the that are statistically insignificant from the best performing model according to an unpaired t-test with significance level 0.05. However, we do see diminishing returns as we increase k, suggesting that as we increase compute, the variance of our estimator may impact performance more than the bias of IWAE. We move on to our first task for which a lower bound estimate of log probability would not suffice. The reverse KL objective is useful when we have access to a (possibly unnormalized) target distribution but no efficient sampling algorithm. A major problem with fitting latent variables models to this objective is the presence of an entropy maximization term, effectively a minimization of log p θ (x). Estimating this log marginal probability with a lower bound estimator could in optimizing θ to maximize the bias of the estimator instead of the true objective. Our experiments demonstrate that this causes IWAE to often fail to optimize the objective unless we use a large amount of computation. 85 Figure 1: We trained latent variable models for posterior inference, which requires minimizing log probability under the model. Training with IWAE leads to optimizing for the bias while leaving the true model in an unstable state, whereas training with SUMO-though noisy-leads to convergence. Modifying IWAE. The bias of the IWAE estimator can be interpreted as the KL between an importance-weighted approximate posterior q IW (z; x) implicitly defined by the encoder and the true posterior p(z|x) . Both the encoder and decoder parameters can therefore affect this bias. In practice, we find that the encoder optimization proceeds at a faster timescale than the decoder optimization: i.e., the encoder can match q IW (z; x) to the decoder's p(z|x) more quickly than the latter can match an objective. For this reason, we train the encoder to reduce bias and use a minimax training objective Though this is still a lower bound with unbounded bias, it makes for a stronger baseline than optimizing q(z; x) in the same direction as p(x, z). We find that this approach can work well in practice when k is set sufficiently high. We choose a "funnel" target distribution (Figure 1) similar to the distribution used as a benchmark for inference in , where p * has support in R 2 and is defined p * (x 1, x 2) = N (x 1 ; 0, 1.35 2)N (x 2 ; 0, e 2x1) We use neural networks with one hidden layer of 200 hidden units and tanh activations for both the encoder and decoder networks. We use 20 latent variables, with p(z), p θ (x|z), and q φ (z; x) all being Gaussian distributed. Figure 2 shows the learning curves when using IWAE and SUMO. Unless k is set very large, IWAE will at some point start optimizing the bias instead of the actual objective. The reverse KL is a non-negative quantity, so any estimate significantly below zero can be attributed to the unbounded bias. On the other hand, SUMO Figure 3: Latent variable policies allow faster exploration than autoregressive policy models, while being more expressive than an independent policy. SUMO works well with entropy regularization, whereas IWAE is unstable and converges to similar performance as the non-latent variable model. correctly optimizes for the objective even with a small expected cost. Increasing the expected cost k for SUMO reduces variance. For the same expected cost, SUMO can optimize the true objective but IWAE cannot. We also found that if k is set sufficiently large, then IWAE can work when we train using the minimax objective in equation 15, suggesting that a sufficiently debiased estimator can also work in practice. However, this requires much more compute and likely does not scale compared to SUMO. We also visualize the contours of the ing models in Figure 1. For IWAE, we visualize the model a few iterations before it reaches numerical instability. Let us now consider the problem of finding the maximum of a non-differentiable function, a special case of reinforcement learning without an interacting environment. Variational optimization can be used to reformulate this as the optimization of a parametric distribution, which is now a differentiable function with respect to the parameters θ, whose gradients can be estimated using a combination of the REINFORCE gradient estimator and the SUMO estimator (equation 13). Furthermore, entropy regularized reinforcement learning-where we maximize R(x) + λH(p θ) with H(p θ) being the entropy of p θ (x)-encourages exploration and is inherently related to minimizing a reverse KL objective with the target being an exponentiated reward . For concreteness, we focus on the problem of quadratic pseudo-Boolean optimization (QPBO) where the objective is to maximize where ∈ {0, 1} are binary variables. Without further assumptions, QPBO is NPhard . As there exist complex dependencies between the binary variables and optimization of equation 16 requires sampling from the policy distribution p θ (x), a model that is both expressive and allows efficient sampling would be ideal. For this reason, we motivate the use of latent variable models with independent conditional distributions, which we trained using the SUMO objective. Our baselines are an autoregressive policy, which captures dependencies but for which sampling must be performed sequentially, and an independent policy, which is easy to sample from but captures no dependencies. We note that also argued for latent variable policies in favor of learning diverse strategies but ultimately had to make use of normalizing flows which did not require marginalization. We constructed one problem instance for each d ∈ {100, 500}, which we note are already intractable for exact optimization. For each instance, we randomly sampled the weights w i and w ij uniformly from the interval [−1, 1]. Figure 3 shows the performance of each policy model. In general, the independent policy is quick to converge to a local minima and is unable to explore different regions, whereas more complex models have a better grasp of the "frontier" of reward distributions during optimization. The autoregressive works well overall but is much slower to train due to its sequential sampling procedure; with d = 500, it is 19.2× slower than training with SUMO. Surprisingly, we find that estimating the REINFORCE gradient with IWAE in decent performance when no entropy regularization is present. With entropy regularization, all policies improve significantly; however, training with IWAE in this setting in performance similar to the independent model. On the other hand, SUMO works with both REINFORCE gradient estimation and entropy regularization, albeit at the cost of slower convergence due to variance. We introduced SUMO, a new unbiased estimator of the log probability for latent variable models, and demonstrated tasks for which this estimator performs better than standard lower bounds. Specifically, we investigated applications involving entropy maximization where a lower bound performs poorly, but our unbiased estimator can train properly with relatively smaller amount of compute. In the future, we plan to investigate new families of gradient-based optimizers which can handle heavy-tailed stochastic gradients. It may also be fruitful to investigate the use of convex combination of consistent estimators within the SUMO approach, as any convex combination is unbiased, or to apply variance reduction methods to increase stability of training with SUMO. Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. PhD thesis, figshare, 2010. A APPENDIX where z 1,.., z k are sampled independently from q(z; x). And we define the k-th term of the infinite. Using the properties of IWAE in equation 6, we have∆ k (x) ≥ 0, and which means the series converges absolutely. This is a sufficient condition for finite expectation of the Russian roulette estimator (; Lemma 3). Applying equation 7 to the series: Let, Hence our estimator is constructed: And it can be easily seen from equation 22 and equation 23 that SUMO is an unbiased estimator of the log marginal likelihood: A.2 CONVERGENCE OF ∆ k We follow the analysis of JVI , which applied the delta method for moments to show the asymptotic on the bias and variance of IWAE k both at a rate of O(and we define Y k := 1 k k i=1 w i as the sample mean and we have E[ We note that we rely on ||Y k − µ|| < 1 for this power series to converge. This condition was implicitly assumed, but not explicitly noted, in . This condition will hold for sufficiently large k so long as the moments of w i exist: one could bound the probability ||Y k −µ|| ≥ 1 by Chebyshev's inequality or by the Central Limit Theorem. We use the central moments Expanding Eq. 28 to order two gives Since we use cumulative sum to compute Y k and Y k+1, we obtain We note that Without loss of generality, suppose j ≥ k + 1, For clarity, let C k = Y k − µ be the zero-mean random variable. gives the relations Expanding both the sums inside the brackets to order two: We will proceed by bounding each of the terms,,,. First, we decompose C j. Let We know that B k,j is independent of C k and Now we show that is zero: We now investigate: We now show that is zero: Finally, we investigate: Using the relation in equation 36, we have Assume that ∇ θ SUMO is bounded: it is sufficient that ∇ θ IWAE 1 is bounded and that the sampling probabilities are chosen such that the partial sums of by the dominated convergence theorem, as long as SUMO is everywhere differentiable, which is satisfied by all of our experiments. If ReLU neural networks are to be used, one may be able to show the same property using Theorem 5 of , assuming finite higher moments and Lipschitz constant. The IWAE log likelihood estimate is: The gradient of this with respect to λ, where λ is either θ or ψ, is We abbreviate w i:= p θ (x,zi) q ψ (zi|x), and ν i = dwi dλ. In both λ = ψ and λ = θ cases, it suffices to treat the w i and ν i as i.i.d. random variables with finite variance and expectation. Being a likelihood ratio, w i could be ill behaved when the importance sampling distribution q ψ (z i |x) is is particularly mismatched from the true posterior p(z i |x) = p θ (x,zi E z∼p(z) p θ (x,z). However, the analysis from IWAE requires assuming that the likelihood ratios w i = p θ (x,zi) q ψ (zi|x) are bounded, and we adopt this assumption. Reasoning about when this assumption holds, and the behavior of IWAE-like estimators when it does not, is an interesting area for future work. Consider the differences between two gradients: we label ∆ g as follows: We have: We again let Y k denote the kth sample mean 1 k i w i. Then: The sample means Y k andμ k have finite expectation and variance. The variance vanishes as k → ∞ (but the expectation does not change). The second term vanishes at a rate strictly faster than 1 k 2: the variance of φ k goes to zero as k → ∞. But the first term does not: φ k is a biased estimator of φ ∞ so Eφ k does change with k, but it does not necessarily go to zero: A.7 BIAS-VARIANCE TRADEOFF VIA GRADIENT CLIPPING While SUMO is unbiased, its variance is extremely high or potentially infinite. This property leads to poor performance compared to lower bound estimates such as IWAE when maximizing loglikelihood. In order to obtain models with competitive log-likelihood values, we can make use of gradient clipping. This allows us to ignore rare gradient samples with extremely large values due to the heavy-tailed nature of its distribution. Gradient clipping introduces bias in favor of reduced variance. Figure 6 shows how the performance changes as a function of the clipping value, and more importantly, the percentage of clipped gradients. As shown, neither full clipping and no clipping are desirable. We performed this experiment after reporting the in Table 1, so this grid search was not used to tune hyperparameter for our experiments. As bias is introduced, we do not use gradient clipping for entropy maximization or policy gradient (REINFORCE). Figure 6: Test negative log-likelihood against the gradient clipping norm and clipping percentage, when training with SUMO (k=15). In density modeling experiments, all the models are trained using a batch size of 100 and an Amsgrad optimizer with parameters lr = 0.001, β 1 = 0.9, β 2 = 0.999 and = 10 −4. The learning rate is reduced by factor 0.8 with a patience of 50 epochs. We use gradient norm scaling in both the inference and generative networks. We train SUMO using the same architecture and hyperparameters as IWAE except the gradient clipping norm. We set the gradient norm to 5000 for encoder and {20, 40, 60} for decoder in SUMO. For IWAE, the gradient norm is fixed to 10 in all the experiments. We report the performance of models with early stopping if no improvements have been observed for 300 epochs on the validation set. We add additional plots of the test NLL against the norm and percentage of gradients clipped for the decoder in Figure 6. The plot is based on MNIST with expected number of compute k = 15. Gradient clipping was not used in the other experiments except the density modeling ones, where it can be used as a tool to obtain a better bias-variance trade-off. A.8.1 REVERSE KL AND COMBINATORIAL OPTIMIZATION These two tasks use the same encoder and decoder architecture: one hidden layer with tanh nonlinearities and 200 hidden units. We set the latent state to be of size 20. The prior is a standard Gaussian with diagonal covariance, while the encoder distribution is a Gaussian with parameterized diagonal covariance. For reverse KL, we used independent Gaussian conditional likelihoods for p(x|z), while for combinatorial optimization we used independent Bernoulli conditional distribu-tions. We found it helps stablize training for both IWAE and SUMO to remove momentum and used RMSprop with learning rate 0.00005 and epsilon 1e-3 for fitting reverse KL. We used Adam with learning rate 0.001 and epsilon 1e-3, plus standard hyperparameters for the combinatorial optimization problems. SUMO used an expected compute of 15 terms, with m = 5 and the tail-modified telescoping Zeta distribution.

represent the natural language
------
For example, 'We propose a novel semi-supervised learning approach with SOTA performance on combating learning with noisy labels.' should be close to 'Deep neural networks are known to be annotation-hungry. Numerous efforts have been devoted to reducing the annotation cost when learning with deep networks. Two prominent directions include learning with noisy labels and semi-supervised learning by exploiting unlabeled data. In this work, we propose DivideMix, a novel framework for learning with noisy labels by leveraging semi-supervised learning techniques. In particular, DivideMix models the per-sample loss distribution with a mixture model to dynamically divide the training data into a labeled set with clean samples and an unlabeled set with noisy samples, and trains the model on both the labeled and unlabeled data in a semi-supervised manner. To avoid confirmation bias, we simultaneously train two diverged networks where each network uses the dataset division from the other network. During the semi-supervised training phase, we improve the MixMatch strategy by performing label co-refinement and label co-guessing on labeled and unlabeled samples, respectively. Experiments on multiple benchmark datasets demonstrate substantial improvements over state-of-the-art methods. Code is available at https://github.com/LiJunnan1992/DivideMix. The remarkable success in training deep neural networks (DNNs) is largely attributed to the collection of large datasets with human annotated labels. However, it is extremely expensive and time-consuming to label extensive data with high-quality annotations. On the other hand, there exist alternative and inexpensive methods for mining large-scale data with labels, such as querying commercial search engines (a), downloading social media images with tags , leveraging machine-generated labels , or using a single annotator to label each sample . These alternative methods inevitably yield samples with noisy labels. A recent study shows that DNNs can easily overfit to noisy labels and in poor generalization performance. Existing methods on learning with noisy labels (LNL) primarily take a loss correction approach. Some methods estimate the noise transition matrix and use it to correct the loss function . However, correctly estimating the noise transition matrix is challenging. Some methods leverage the predictions from DNNs to correct labels and modify the loss accordingly . These methods do not perform well under high noise ratio as the predictions from DNNs would dominate training and cause overfitting. To overcome this, adopt MixUp augmentation. Another approach selects or reweights samples so that noisy samples contribute less to the loss . A challenging issue is to design a reliable criteria to select clean samples. It has been shown that DNNs tend to learn simple patterns first before fitting label noise . Therefore, many methods treat samples with small loss as clean ones . Among those methods, Co-teaching and Co-teaching+ train two networks where each network selects small-loss samples in a mini-batch to train the other. Another active area of research that also aims to reduce annotation cost is semi-supervised learning (SSL). In SSL, the training data consists of unlabeled samples in addition to the labeled samples. Significant progress has been made in leveraging unlabeled samples by enforcing the model to produce low entropy predictions on unlabeled data or consistent predictions on perturbed input (; ;). propose MixMatch, which unifies several dominant SSL approaches in one framework and achieves state-of-the-art performance. Despite the individual advances in LNL and SSL, their connection has been underexplored. In this work, we propose DivideMix, which addresses learning with label noise in a semi-supervised manner. Different from most existing LNL approaches, DivideMix discards the sample labels that are highly likely to be noisy, and leverages the noisy samples as unlabeled data to regularize the model from overfitting and improve generalization performance. The key contributions of this work are: • We propose co-divide, which trains two networks simultaneously. For each network, we dynamically fit a Gaussian Mixture Model (GMM) on its per-sample loss distribution to divide the training samples into a labeled set and an unlabeled set. The divided data is then used to train the other network. Co-divide keeps the two networks diverged, so that they can filter different types of error and avoid confirmation bias in self-training. • During SSL phase, we improve MixMatch with label co-refinement and co-guessing to account for label noise. For labeled samples, we refine their ground-truth labels using the network's predictions guided by the GMM for the other network. For unlabeled samples, we use the ensemble of both networks to make reliable guesses for their labels. • We experimentally show that DivideMix significantly advances state-of-the-art on multiple benchmarks with different types and levels of label noise. We also provide extensive ablation study and qualitative to examine the effect of different components. 2 RELATED WORK Most existing methods for training DNNs with noisy labels seek to correct the loss function. The correction can be categorized in two types. The first type treats all samples equally and correct loss either explicitly or implicitly through relabeling the noisy samples. For relabeling methods, the noisy samples are modeled with directed graphical models , Conditional Random Fields , knowledge graph (b), or DNNs . However, they require access to a small set of clean samples. and propose iterative methods which relabel samples using network predictions. For explicit loss correction. propose a bootstrapping method which modifies the loss with model predictions, and improve the bootstrapping method by exploiting the dimensionality of feature subspaces. estimate the label corruption matrix for loss correction, and improve the corruption matrix by using a clean set of data. The second type of correction focuses on reweighting training samples or separating clean and noisy samples, which in correcting the loss function . A common method is to consider samples with smaller loss as clean ones . train a mentor network to guide a student network by assigning weights to samples. reweight samples based on their gradient directions. apply cross validation to identify clean samples. calculate sample weights by modeling per-sample loss with a mixture model. train two networks which select small-loss samples within each mini-batch to train each other, and improve it by updating the network on disagreement data to keep the two networks diverged. Contrary to all aforementioned methods, our method discards the labels that are highly likely to be noisy, and utilize the noisy samples as unlabeled data to regularize training in a SSL manner. and have shown that SSL method is effective in LNL. However, their methods do not perform well under high levels of noise, whereas our method can better distinguish and utilize noisy samples. Besides leveraging SSL, our method also introduces other advantages. Compared to self-training methods , our method can avoid the confirmation bias problem by training two networks to filter error for each other. Compared to Co-teaching and Co-teaching+, our method is more robust to noise by enabling the two networks to teach each other implicitly at each epoch (co-divide) and explicitly at each mini-batch (label co-refinement and co-guessing). its per-sample loss distribution with a GMM to divide the dataset into a labeled set (mostly clean) and an unlabeled set (mostly noisy), which is then used as training data for the other network (i.e. co-divide). At each mini-batch, a network performs semi-supervised training using an improved MixMatch method. We perform label co-refinement on the labeled samples and label co-guessing on the unlabeled samples. SSL methods aim to improve the model's performance by leveraging unlabeled data. Current state-of-the-art SSL methods mostly involve adding an additional loss term on unlabeled data to regularize training. The regularization falls into two classes: consistency regularization (; ;) enforces the model to produce consistent predictions on augmented input data; entropy minimization encourages the model to give high-confidence predictions on unlabeled data. propose MixMatch, which unifies consistency regularization, entropy minimization, and the MixUp regularization into one framework. In this section, we introduce DivideMix, our proposed method for learning with noisy labels. An overview of the method is shown in Figure 1. To avoid confirmation bias of self-training where the model would accumulate its errors, we simultaneously train two networks to filter errors for each other through epoch-level implicit teaching and batch-level explicit teaching. At each epoch, we perform co-divide, where one network divides the noisy training dataset into a clean labeled set (X) and a noisy unlabeled set (U), which are then used by the other network. At each mini-batch, one network utilizes both labeled and unlabeled samples to perform semi-supervised learning guided by the other network. Algorithm 1 delineates the full algorithm. Deep networks tend to learn clean samples faster than noisy samples , leading to lower loss for clean samples . , we aim to find the probability of a sample being clean by fitting a mixture model to the per-sample denote the training data, where x i is an image and yi ∈ {0, 1} C is the one-hot label over C classes. Given a model with parameters θ, the cross-entropy loss (θ) reflects how well the model fits the training samples: where p c model is the model's output softmax probability for class c. fit a two-component Beta Mixture Model (BMM) to the max-normalized loss to model the distribution of clean and noisy samples. However, we find that BMM tends to produce undesirable flat distributions and fails when the label noise is asymmetric. Instead, Gaussian Mixture Model (GMM) can better distinguish clean and noisy samples due to its flexibility in the sharpness of distribution. Therefore, we fit a two-component GMM to using the Expectation-Maximization algorithm. For each sample, its clean probability w i is the posterior probability p(g| i), where g is the Gaussian component with smaller mean (smaller loss). We divide the training data into a labeled set and an unlabeled set by setting a threshold τ on w i. However, training a model using the data divided by itself could lead to confirmation bias (i.e. the 1 Input: θ and θ, training dataset (X, Y), clean probability threshold τ, number of augmentations M, sharpening temperature T, unsupervised loss weight λu, Beta distribution parameter α for MixMatch. // standard training (with confidence penalty) 3 while e < MaxEpoch do ) // model per-sample loss with θ to obtain clean proabability for θ ) // model per-sample loss with θ to obtain clean proabability for θ 6 for k = 1, 2 do // train the two networks one by one // refine ground-truth label guided by the clean probability produced by the other network // apply temperature sharpening to the refined label ) // co-guessing: average the predictions from both networks across augmentations of u b // apply temperature sharpening to the guessed label model is prone to confirm its mistakes ), as noisy samples that are wrongly grouped into the labeled set would keep having lower loss due to the model overfitting to their labels. Therefore, we propose co-divide to avoid error accumulation. In co-divide, the GMM for one network is used to divide training data for the other network. The two networks are kept diverged from each other due to different (random) parameter initialization, different training data division, different (random) mini-batch sequence, and different training targets. Being diverged offers the two networks distinct abilities to filter different types of error, making the model more robust to noise. Confidence Penalty for Asymmetric Noise. For initial convergence of the algorithm, we need to "warm up" the model for a few epochs by training on all data using the standard cross-entropy loss. The warm up is effective for symmetric (i.e. uniformly random) label noise. However, for asymmetric (i.e. class-conditional) label noise, the network would quickly overfit to noise during warm up and produce over-confident (low entropy) predictions, which leads to most samples having near-zero normalized loss (see Figure 2a). In such cases, the GMM cannot effectively distinguish clean and noisy samples based on the loss distribution. To address this issue, we penalize confident predictions from the network by adding a negative entropy term, −H , to the cross-entropy loss during warm up. The entropy of a model's prediction for an input x is defined as: By maximizing the entropy, becomes more evenly distributed (see Figure 2b) and easier to be modeled by the GMM. Furthermore, in Figure 2c we show when the model is trained with DivideMix for 10 more epochs after warm up. The proposed method can significantly reduce the loss for clean samples while keeping the loss larger for most noisy samples. To account for label noise, we make two improvements to MixMatch which enable the two networks to teach each other. First, we perform label co-refinement for labeled samples by linearly combining the ground-truth label y b with the network's prediction p b (averaged across multiple augmentations of x b), guided by the clean probability w b produced by the other network: Then we apply a sharpening function on the refined label to reduce its temperature: Second, we use the ensemble of predictions from both networks to "co-guess" the labels for unlabeled samples (algorithm 1, line 20), which can produce more reliable guessed labels. Having acquiredX (andÛ) which consists of multiple augmentations of labeled (unlabeled) samples and their refined (guessed) labels, we follow MixMatch to "mix" the data, where each sample is interpolated with another sample randomly chosen from the combined mini-batch ofX andÛ. Specifically, for a pair of samples (x 1, x 2) and their corresponding labels (p 1, p 2), the mixed (x, p) is computed by: MixMatch transformsX andÛ into X and U. Equation 6 ensures that X are "closer" toX thanÛ. The loss on X is the cross-entropy loss and the loss on U is the mean squared error: Under high levels of noise, the network would be encouraged to predict the same class to minimize the loss. To prevent assigning all samples to a single class, we apply the regularization term used by and , which uses a uniform prior distribution π (i.e. π c = 1/C) to regularize the model's average output across all samples in the mini-batch: Finally, the total loss is: In our experiments, we set λ r as 1 and use λ u to control the strength of the unsupervised loss. We extensively validate our method on four benchmark datasets, namely CIFAR-10, CIFAR-100 , Clothing1M , and WebVision (a). Both CIFAR-10 and CIFAR-100 contain 50K training images and 10K test images of size 32 × 32. Following previous works , we experiment with two types of label noise: symmetric and asymmetric. Symmetric noise is generated by randomly replacing the labels for a percentage of the training data with all possible labels. Note that there is another criterion for symmetric label noise injection where the true labels cannot be maintained (;, for which we also report the (Table 6 in Appendix). Asymmetric noise is designed to mimic the structure of real-world label noise, where labels are only replaced by similar classes (e.g. deer→horse, dog↔cat). We use an 18-layer PreAct Resnet and train it using SGD with a momentum of 0.9, a weight decay of 0.0005, and a batch size of 128. The network is trained for 300 epochs. We set the initial learning rate as 0.02, and reduce it by a factor of 10 after 150 epochs. The warm up period is 10 epochs for CIFAR-10 and 30 epochs for CIFAR-100. We find that most hyperparameters introduced by DivideMix do not need to be heavily tuned. For all CIFAR experiments, we use the same hyperparameters M = 2, T = 0.5, and α = 4. τ is set as 0.5 except for 90% noise ratio when it is set as 0.6. We choose λ u from {0, 25, 50, 150} using a small validation set. Clothing1M and WebVision 1.0 are two large-scale datasets with real-world noisy labels. Clothing1M consists of 1 million training images collected from online shopping websites with labels generated from surrounding texts. We follow previous work and use ResNet-50 with ImageNet pretrained weights. WebVision contains 2.4 million images crawled from the web using the 1,000 concepts in ImageNet ILSVRC12. Following previous work , we compare baseline methods on the first 50 classes of the Google image subset using the inception-resnet v2 . The training details are delineated in Appendix B. We compare DivideMix with multiple baselines using the same network architecture. Here we introduce some of the most recent state-of-the-art methods: Meta-Learning proposes a gradient based method to find model parameters that are more noise-tolerant; Joint-Optim and P-correction jointly optimize the sample labels and the network parameters; M-correction Table 1: Comparison with state-of-the-art methods in test accuracy (%) on CIFAR-10 and CIFAR-100 with symmetric noise. Methods marked by * denote re-implementations based on public code. Note that none of these methods can consistently outperform others across different datasets. Mcorrection excels at symmetric noise, whereas Meta-Learning performs better for asymmetric noise. Table 1 shows the on CIFAR-10 and CIFAR-100 with different levels of symmetric label noise ranging from 20% to 90%. We report both the best test accuracy across all epochs and the averaged test accuracy over the last 10 epochs. DivideMix outperforms state-of-the-art methods by a large margin across all noise ratios. The improvement is substantial (∼10% in accuracy) for the more challenging CIFAR-100 with high noise ratios. Appendix A shows comparison with more methods in Table 6. The on CIFAR-10 with asymmetric noise is shown in Table 2. We use 40% because certain classes become theoretically indistinguishable for asymmetric noise larger than 50%. Cross-Entropy 85.0 72.3 F-correction 87.2 83.1 M-correction 87.4 86.3 Iterative-CV 88.6 88.0 P-correction 88.5 88.1 Joint-Optim 88.9 88.4 Meta-Learning 89.2 88.6 DivideMix 93.4 92.1 Table 2: Comparison with state-of-the-art methods in test accuracy (%) on CIFAR-10 with 40% asymmetric noise. We re-implement all methods under the same setting. Table 3 and Table 4 show the on Clothing1M and WebVision, respectively. DivideMix consistently outperforms state-of-the-art methods across all datasets with different types of label noise. For WebVision, we achieve more than 12% improvement in top-1 accuracy. Cross-Entropy 69.21 F-correction 69.84 M-correction 71.00 Joint-Optim 72.16 Meta-Cleaner 72.50 Meta-Learning 73.47 P-correction 73.49 DivideMix 74.76 62.68 84.00 57.80 81.36 MentorNet 63.00 81.40 57.80 79.92 Co-teaching 63.58 85.20 61.48 84.70 Iterative-CV 65 Table 5: Ablation study in terms of test accuracy (%) on CIFAR-10 and CIFAR-100. • To study the effect of model ensemble during test, we use the prediction from a single model θ instead of averaging the predictions from both networks as in DivideMix. Note that the training process remains unchanged. The decrease in accuracy suggests that the ensemble of two diverged networks consistently yields better performance during inference. • To study the effect of co-training, we train a single network using self-divide (i.e. divide the training data based on its own loss). The performance further decreases compared to θ. • We find that both label refinement and input augmentation are beneficial for DivideMix. • We combine self-divide with the original MixMatch as a naive baseline for using SLL in LNL. Appendix A also introduces more in-depth studies in examining the robustness of our method to label noise, including the AUC for clean/noisy sample classification on CIFAR-10 training data, qualitative examples from Clothing1M where our method can effectively identify the noisy samples and leverage them as unlabeled data, and visualization using t-SNE. In this paper, we propose DivideMix for learning with noisy labels by leveraging SSL. Our method trains two networks simultaneously and achieves robustness to noise through dataset co-divide, label co-refinement and co-guessing. Through extensive experiments across multiple datasets, we show that DivideMix consistently exhibits substantial performance improvements compared to state-of-the-art methods. For future work, we are interested in incorporating additional ideas from SSL to LNL, and vice versa. Furthermore, we are also interested in adapting DivideMix to other domains such as NLP. In In Figure 3, we show the Area Under a Curve (AUC) for clean/noisy sample classification on CIFAR-10 training data from one of the GMMs during the first 100 epochs. Our method can effectively separate clean and noisy samples as training proceeds, even for high noise ratio. In Figure 4, we show example images in Clothing1M identified by our method as noisy samples. Our method achieves noise filtering by discarding the noisy labels (shown in red) and using the co-guessed labels (shown in blue) to regularize training. In Figure 5, we visualize the features of training images using t-SNE . The model is trained using DivideMix for 200 epochs on CIFAR-10 with 80% label noise. The embeddings form 10 distinct clusters corresponding to the true class labels, not the noisy training labels, which demonstrates our method's robustness to label noise. For CIFAR experiments, the only hyperparameter that we tune on a per-experiment basis is the unsupervised loss weight λ u. Table 7 shows the value that we use. A larger λ u is required for stronger regularization under high noise ratios or with more classes. For both Clothing1M and WebVision, we use the same set of hyperparameters M = 2, T = 0.5, τ = 0.5, λ u = 0, α = 0.5, and train the network using SGD with a momentum of 0.9, a weight decay of 0.001, and a batch size of 32. The warm up period is 1 epoch. For Clothing1M, we train the network for 80 epochs. The initial learning rate is set as 0.002 and reduced by a factor of 10 after 40 epochs. For each epoch, we sample 1000 mini-batches from the training data while ensuring the labels (noisy) are balanced. For WebVision, we train the network for 100 epochs. The initial learning rate is set as 0.01 and reduced by a factor of 10 after 50 epochs. Table 7: Unsupervised loss weight λ u for CIFAR experiments. Higher noise ratio requires stronger regularization from unlabeled samples. Here we clarify some details for the baseline methods in the ablation study. First, DivideMix w/o co-training still has dataset division, label refinement and label guessing, but performed by the same model. Thus, the performance drop (especially for CIFAR-100 with high noise ratio) suggests the disadvantage of self-training. Second, label refinement is important for high noise ratio because more noisy samples would be mistakenly divided into the labeled set. Third, augmentation improves performance through both producing more reliable predictions and achieving consistency regularization. In addition, same as , we also find that temperature sharpening is essential for our method to perform well. We analyse the training time of DivideMix to understand its efficiency. In Table 8, we compare the total training time of DivideMix on CIFAR-10 with several state-of-the-art methods, using a single Nvidia V100 GPU. DivideMix is slower than Co-teaching+, but faster than P-correction and Meta-Learning which involve multiple training iterations. In Table 9, we also break down the computation time for each operation in DivideMix.'

We create an unbiased estimator for the log probability of latent variable models, extending such models to a larger scope of applications.

It's important to consider optimization in function space, not just parameter space. We introduce a learning rule that reduces distance traveled in function space, just like SGD limits distance traveled in parameter space.