Question: Represent the text to find a very short summary (1-2 sentences max)

Question: This paper investigates strategies that defend against adversarial-example attacks on image-classification systems by transforming the inputs before feeding them to the system. Specifically, we study applying image transformations such as bit-depth reduction, JPEG compression, total variance minimization, and image quilting before feeding the image to a convolutional network classifier. Our experiments on ImageNet show that total variance minimization and image quilting are very effective defenses in practice, in particular, when the network is trained on transformed images. The strength of those defenses lies in their non-differentiable nature and their inherent randomness, which makes it difficult for an adversary to circumvent the defenses. Our best defense eliminates 60% of strong gray-box and 90% of strong black-box attacks by a variety of major attack methods. As the use of machine intelligence increases in security-sensitive applications BID2 BID0, robustness has become a critical feature to guarantee the reliability of deployed machine-learning systems. Unfortunately, recent research has demonstrated that existing models are not robust to small, adversarially designed perturbations of the input BID1 BID31 BID14 BID20 BID6. Adversarially perturbed examples have been deployed to attack image classification services BID22, speech recognition systems BID6, and robot vision BID25. The existence of these adversarial examples has motivated proposals for approaches that increase the robustness of learning systems to such examples BID28 BID20 BID7.The robustness of machine learning models to adversarial examples depends both on the properties of the model (i.e., Lipschitzness) and on the nature of the problem considered, e.g., on the input dimensionality and the Bayes error of the problem BID11. Consequently, defenses that aim to increase robustness against adversarial examples fall in one of two main categories. The first category comprises model-specific strategies that enforce model properties such as invariance and smoothness via the learning algorithm or regularization scheme BID30 BID20 BID7, potentially exploiting knowledge about the adversary's attack strategy BID14. The second category of defenses are model-agnostic: they try to remove adversarial perturbations from the input. For example, in the context of image classification, adversarial perturbations can be partly removed via JPEG compression BID9 or image re-scaling BID23. Hitherto, none of these defenses has been shown to be very effective. Specifically, model-agnostic defenses appear too simple to sufficiently remove adversarial perturbations from input images. By contrast, model-specific defenses make strong assumptions about the nature of the adversary (e.g., on the norm that the adversary minimizes or on the number of iterations it uses to generate the perturbation). Consequently, they do not satisfy BID18 principle: the adversary can alter its attack to circumvent such model-specific defenses. In this paper, we focus on increasing the effectiveness of model-agnostic defense strategies by developing approaches that remove the adversarial perturbations from input images, maintain sufficient information in input images to correctly classify them, and are still effective in settings in which the adversary has information on the defense strategy being used. We explore transformations based on image cropping and rescaling BID15, bit-depth reduction ), JPEG compression (, total variance minimization BID29, and image quilting BID10 . We show that these defenses can be surprisingly effective against existing attacks, in particular, when the convolutional network is trained on images that are transformed in a similar way. The image transformations are good at countering the (iterative) fast gradient sign method BID20 ), Deepfool (, and the BID5 attack, even in gray-box settings in which the model architecture and parameters are public. Our strongest defenses are based on total variation minimization and image quilting: these defenses are non-differentiable and inherently random, which makes it difficult for an adversary to get around them. Our best defenses eliminate 60% of gray-box attacks and 90% of black-box attacks by four major attack methods that perturb pixel values by 8% on average. We study defenses against non-targeted adversarial examples for image-recognition systems. Let DISPLAYFORM0 H×W ×C be the image space. Given an image classifier h(·) and a source image x ∈ X, a non-targeted 1 adversarial example of x is a perturbed image x ∈ X such that h(x) = h(x) and d(x, x) ≤ ρ for some dissimilarity function d(·, ·) and ρ ≥ 0. Ideally, d(·, ·) measures the perceptual difference between x and x but, in practice, the Euclidean distance d(x, x) = x−x 2 or the Chebyshev distance d(x, x) = x − x ∞ is most commonly used. Given a set of N images {x 1, . . ., x N} and a target classifier h(·), an adversarial attack aims to generate {x 1, . . ., x N} such that each x n is an adversarial example for x n. The success rate of an attack is measured by the proportion of predictions that was altered by an attack: DISPLAYFORM1 The success rate is generally measured as a function of the magnitude of the perturbations performed by the attack, using the normalized L 2 -dissimilarity: DISPLAYFORM2 A strong adversarial attack has a high success rate whilst its normalized L 2 -dissimilarity is low. In most practical settings, an adversary does not have direct access to the model h(·) and has to do a black-box attack. However, prior work has shown successful attacks by transferring adversarial examples generated for a separately-trained model to an unknown target model BID22. Therefore, we investigate both the black-box and a more difficult gray-box attack setting: in our gray-box setting, the adversary has access to the model architecture and the model parameters, but is unaware of the defense strategy that is being used. A defense is an approach that aims make the prediction on an adversarial example h(x) equal to the prediction on the corresponding clean example h(x). In this study, we focus on imagetransformation defenses g(x) that perform prediction via h(g(x)). Ideally, g(·) is a complex, nondifferentiable, and potentially stochastic function: this makes it difficult for an adversary to attack the prediction model h(g(x)) even when the adversary knows both h(·) and g(·). One of the first successful attack methods is the fast gradient sign method (FGSM; BID14). Let (·, ·) be the differentiable loss function that was used to train the classifier h(·), e.g., the cross-entropy loss. The FGSM adversarial example corresponding to a source input x and true label y is: DISPLAYFORM0 for some > 0 that governs the perturbation magnitude. A stronger variant of this attack, called iterative FGSM (I-FGSM; BID21), iteratively applies the FGSM update: Alternative attacks aim to minimize the Euclidean distance between the input and the adversarial example instead. For instance, assuming h(·) is a binary classifier, DeepFool projects x onto a linearization of the decision boundary defined by h(·) for M iterations: DISPLAYFORM1 DISPLAYFORM2 where x and x are defined as in I-FGSM. The multi-class variant of DeepFool performs the projection onto the nearest class boundaries. The linearization performed in DeepFool is particularly well suited for ReLU-networks, as these represent piecewise linear class boundaries. Carlini-Wagner's L 2 attack (CW-L2; BID5) is an optimization-based attack that combines a differentiable surrogate for the model's classification accuracy with an L 2 -penalty term. Let Z(x) be the operation that computes the logit vector (i.e., the output before the softmax layer) for an input x, and Z(x) k be the logit value corresponding to class k. The untargeted variant of CW-L2 finds a solution to the unconstrained optimization problem min DISPLAYFORM3 where κ denotes a margin parameter, and where the parameter λ f trades off the perturbation norm and the hinge loss of predicting a different class. We perform the minimization over x using the Adam optimizer BID19 for 100 iterations with an initial learning rate of 0.001.All of the aforementioned attacks enforce that x ∈ X by clipping values between 0 and 1. FIG0 shows adversarial images produced by all four attacks at five normalized L 2 -dissimilarity levels. Adversarial attacks alter particular statistics of the input image in order to change the model prediction. Indeed, adversarial perturbations x−x have a particular structure, as illustrated by FIG0. We design and experiment with image transformations that alter the structure of these perturbations, and investigate whether the alterations undo the effects of the adversarial attack. We investigate five image transformations: image cropping and rescaling, bit-depth reduction, JPEG compression, total variance minimization, and image quilting. Figure 2: Illustration of total variance minimization and image quilting applied to an original and an adversarial image (produced using I-FGSM with = 0.03, corresponding to a normalized L 2 -dissimilarity of 0.075). From left to right, the columns correspond to: no transformation, total variance minimization, and image quilting. From top to bottom, rows correspond to: the original image, the corresponding adversarial image produced by I-FGSM, and the absolute difference between the two images above. Difference images were multiplied by a constant scaling factor to increase visibility. We first introduce three simple image transformations: image cropping-rescaling BID15, bit-depth reduction, and JPEG compression and decompression BID9. Image croppingrescaling has the effect of altering the spatial positioning of the adversarial perturbation, which is important in making attacks successful. Following BID16, we crop and rescale images at training time as part of the data augmentation. At test time, we average predictions over random image crops. Bitdepth reduction ) perform a simple type of quantization that can removes small (adversarial) variations in pixel values from an image; we reduce images to 3 bits in our experiments. JPEG compression removes small perturbations in a similar way; we perform compression at quality level 75 (out of 100). An alternative way of removing adversarial perturbations is via a compressed sensing approach that combines pixel dropout with total variation minimization BID29. This approach randomly selects a small set of pixels, and reconstructs the "simplest" image that is consistent with the selected pixels. The reconstructed image does not contain the adversarial perturbations because these perturbations tend to be small and localized. Specifically, we first select a random set of pixels by sampling a Bernoulli random variable X(i, j, k) for each pixel location (i, j, k); we maintain a pixel when X(i, j, k) = 1. Next, we use total variation minimization to constructs an image z that is similar to the (perturbed) input image x for the selected set of pixels, whilst also being "simple" in terms of total variation by solving: DISPLAYFORM0 Herein, denotes element-wise multiplication, and TV p (z) represents the L p -total variation of z: DISPLAYFORM1 The total variation (TV) measures the amount of fine-scale variation in the image z, as a of which TV minimization encourages removal of small (adversarial) perturbations in the image. The objective function is convex in z, which makes solving for z straightforward. In our implementation, we set p = 2 and employ a special-purpose solver based on the split Bregman method BID13 ) to perform total variance minimization efficiently. The effectiveness of TV minimization is illustrated by the images in the middle column of Figure 2: in particular, note that the adversarial perturbations that were present in the for the nontransformed image (see bottom-left image) have nearly completely disappeared in the TV-minimized adversarial image (bottom-center image). As expected, TV minimization also changes image structure in non-homogeneous regions of the image, but as these perturbations were not adversarially designed we expect the negative effect of these changes to be limited. Image quilting BID10 ) is a non-parametric technique that synthesizes images by piecing together small patches that are taken from a database of image patches. The algorithm places appropriate patches in the database for a predefined set of grid points, and computes minimum graph cuts BID3 ) in all overlapping boundary regions to remove edge artifacts. Image quilting can be used to remove adversarial perturbations by constructing a patch database that only contains patches from "clean" images (without adversarial perturbations); the patches used to create the synthesized image are selected by finding the K nearest neighbors (in pixel space) of the corresponding patch from the adversarial image in the patch database, and picking one of these neighbors uniformly at random. The motivation for this defense is that the ing image only consists of pixels that were not modified by the adversary -the database of real patches is unlikely to contain the structures that appear in adversarial images. The right-most column of Figure 2 illustrates the effect of image quilting on adversarial images. Whilst interpretation of these images is more complicated due to the quantization errors that image quilting introduces, it is interesting to note that the absolute differences between quilted original and the quilted adversarial image appear to be smaller in non-homogeneous regions of the image. This suggests that TV minimization and image quilting lead to inherently different defenses. We performed five experiments to test the efficacy of our defenses. The experiment in Section 5.2 considers gray-box attacks: it applies the defenses on adversarial images before using them as input into a convolutional network trained to classify "clean" images. In this setting, the adversary has access to the model architecture and parameters but is unaware of the defense strategy. The experiment in Section 5.3 focuses on a black-box setting: it replaces the convolutional network by networks that were trained on images with a particular input-transformation. The experiment in Section 5.4 combines our defenses with ensembling and model transfer. The experiment in Section 5.5 investigates to what extent networks trained on image-transformations can be attacked in a gray-box setting. The experiment in Section 5.6 compares our defenses with prior work. The setup of our gray-box and black-box experiments is illustrated in FIG1. Code to reproduce our is available at https://github.com/facebookresearch/adversarial_image_defenses. We performed experiments on the ImageNet image classification dataset. The dataset comprises 1.2 million training images and 50, 000 test images that correspond to one of 1, 000 classes. Our adversarial images are produced by attacking a ResNet-50 model BID16. We evaluate our defense strategies against the four adversarial attacks presented in Section 3. We measure the strength of an adversary in terms of its normalized L 2 -dissimilarity and report classification accu- racies as a function of the normalized L 2 -dissimilarity. To produce adversarial images like those in FIG0, we set the normalized L 2 -dissimilarity for each of the attacks as follows:• FGSM. Increasing the step size increases the normalized L 2 -dissimilarity.• I-FGSM. We fix M = 10, and increase to increase the normalized L 2 -dissimilarity.• DeepFool. We fix M = 5, and increase to increase the normalized L 2 -dissimilarity.• CW-L2. We fix κ = 0 and λ f = 10, and multiply the ing perturbation by an appropriately chosen ≥ 1 to alter the normalized L 2 -dissimilarity. We fixed the hyperparameters of our defenses in all experiments: specifically, we set pixel dropout probability p = 0.5 and the regularization parameter of the total variation minimizer λ TV = 0.03. We use a quilting patch size of 5×5 and a database of 1, 000, 000 patches that were randomly selected from the ImageNet training set. We use the nearest neighbor patch (i.e., K = 1) for experiments in Sections 5.2 and 5.3, and randomly select a patch from one of K = 10 nearest neighbors in all other experiments. In the cropping defense, we sample 30 crops of size 90×90 from the 224×224 input image, rescale the crops to 224×224, and average the model predictions over all crops. FIG2 shows the top-1 accuracy of a ResNet-50 tested on transformed adversarial images as a function of the adversary strength for each of the four attacks. Each plot shows for five different transformations we apply to the images at test time (viz., image cropping-rescaling, bitdepth reduction, JPEG compression, total variation minimization, and image quilting). The dotted line shows the classification error of the ResNet-50 model on images that are not adversarially perturbed, i.e., it gives an upper bound on the accuracy that defenses can achieve. In line with the reported in the literature, the four adversaries successfully attack the ResNet-50 model in nearly all cases (FGSM has a slightly lower favorable attack rate of 80−90%) when the input images are not transformed. The also show that the proposed image transformations are capable of partly eliminating the effect of the attacks. In particular, ensembling 30 predictions over different, random image crops is very efficient: these predictions are correct for 40−60% of the images (note that 76% is the highest accuracy that one can expect to achieve). This suggests that adversarial examples are susceptible to changes in the location and scale of the adversarial perturbations. While not as effective, image transformations based on total variation minimization and image quilting also successfully defend against adversarial examples from all four attacks: applying these transformations allows us to classify 30−40% of the images correctly. This suggests that total variation minimization and image quilting can successfully remove part of the perturbations from adversarial images. In particular, the accuracy of the image-quilting defense hardly deteriorates as the strength of the adversary increases. However, the quilting transformation does severely impact the model's accuracy on non-adversarial images. The high relative performance of image cropping-rescaling in 5.2 may be partly explained by the fact that the convolutional network was trained on randomly cropped-rescaled images 2, but not on any of the other transformations. This implies that independent of whether an image is adversarial or not, the network is more robust to image cropping-rescaling than it is to those transformations. The in FIG2 suggest that this negatively affects the effectiveness of these defenses, even if the defenses are successful in removing the adversarial perturbation. To investigate this, we trained ResNet-50 models on transformed ImageNet training images. We adopt the standard data augmentation from BID16, but apply bit-depth reduction, JPEG compression, TV minimization, or image quilting on the resized image crop before feeding it to the network. We measure the classification accuracy of the ing networks on the same adversarial images as before. Note that this implies that we assume a black-box setting in this experiment. Table 1: Top-1 classification accuracy of ensemble and model transfer defenses (columns) against four black-box attacks (rows). The four networks we use to classify images are ResNet-50 (RN50), ResNet-101 (RN101), DenseNet-169 (DN169), and Inception-v4 (Iv4). Adversarial images are generated by running attacks against the ResNet-50 model, aiming for an average normalized L 2 -dissimilarity of 0.06. Higher is better. The best defense against each attack is typeset in boldface. We present the of these experiments in FIG3. Training convolutional networks on images that are transformed in the same way as at test time, indeed, dramatically improves the effectiveness of all transformation defenses. In our experiments, the image-quilting defense is particularly effective against strong attacks: it successfully defends against 80−90% of all four attacks, even when the normalized L 2 -dissimilarity of the attack approaches 0.08. We evaluate the efficacy of ensembling different defenses and "transferring" attacks to different network architectures (in a black-box setting). Specifically, we measured the accuracy of four networks using ensembles of defenses on adversarial images generated to attack a ResNet-50; the four networks we consider are ResNet-50, ResNet-101, DenseNet-169 BID17, and Inception-v4 BID33. To ensemble the image quilting and TVM defenses, we average the image-quilting prediction (using a weight of 0.5) with model predictions for 10 different TVM reconstructions (with a weight of 0.05 each), re-sampling the pixels used to measure the reconstruction error each time. To combine cropping with other transformations, we first apply those transformations and average predictions over 10 random crops from the transformed images. The of our ensembling experiments are presented in Table 1. The show that gains of 1 − 2% in classification accuracy can be achieved by ensembling different defenses, whereas transferring attacks to different convolutional network architectures can lead to an improvement of 2−3%. Inception-v4 performs best in our experiments, but this may be partly due to that network having a higher accuracy even in non-adversarial settings. Our best black-box defense achieves an accuracy of about 71% against all four defenses: the attacks deteriorate the accuracy of our best classifier (which combines cropping, TVM, image quilting, and model transfer) by at most 6%. The previous experiments demonstrated the effectiveness of image transformations against adversarial images, in particular, when convolutional networks are re-trained to be robust to those image transformations. In this experiment, we investigate to what extent the ing networks can be attacked in a gray-box setting in which the adversary has access to those networks (but does not have access to the input transformations applied at test time). We use the four attack methods to generate novel adversarial images against the transformation-robust networks trained in 5.3, and measure the accuracy of the networks on these novel adversarial images in FIG4.The show that bit-depth reduction and JPEG compression are weak defenses in such a graybox setting. Whilst their relative ordering varies between attack methods, image cropping and rescaling, total variation minimization, and image quilting are fairly robust defenses in the white-box setting. Specifically, networks using these defenses classify up to 50% of adversarial images correctly. In our final set of experiments, we compare our defenses with the state-of-the-art ensemble adversarial training approach proposed by BID34. Ensemble adversarial training fits the parameters of a convolutional network on adversarial examples that were generated to attack an ensemble of pre-trained models. These adversarial examples are very diverse, which makes the convolutional network being trained robust to a variety of adversarial perturbation. In our experiments, we used the model released by BID34: an Inception-Resnet-v2 BID32 trained on adversarial examples generated by FGSM against Inception-Resnet-v2 and Inception-v3 models. We compare the model to our ResNet-50 models with image cropping, total variance minimization, and image quilting defenses. We note that there are two small differences in terms of the assumptions that ensemble adversarial training makes and the assumptions our defenses make: in contrast to ensemble adversarial training, our defenses assume that part of the defense strategy (viz., the input transformation) is unknown to the adversary, and in contrast to ensemble adversarial training, our defenses assume no prior knowledge of the attacks being used. The former difference is advantageous to our defenses, whereas the latter difference gives our defenses a disadvantage compared to ensemble adversarial training. Table 2 compares the classification accuracies of the defense strategief on adversarial examples with a normalized L 2 -dissimilarity of 0.06. The show that ensemble adversarial training works better on FGSM attacks (which it uses at training time), but is outperformed by each of the transformation-based defenses all other attacks. Input transformations particularly outperform ensemble adversarial training against the iterative attacks: our defense are are 18−24× more robust than ensemble adversarial training against DeepFool attacks. Combining cropping, TVM, and quilting increases the accuracy of our defenses against DeepFool gray-box attacks to 51.51% (compared to 1.84% for ensemble adversarial training). The from this study suggest there exists a range of image transformations that have the potential to remove adversarial perturbations while preserving the visual content of the image: one merely has to train the convolutional network on images that were transformed in the same way. A critical property that governs which image transformations are most effective in practice is whether Table 2: Top-1 classification accuracy on images perturbed using attacks against ResNet-50 models trained on input-transformed images, and an Inception-v4 model trained using ensemble adversarial. Adversarial images are generated by running attacks against the models, aiming for an average normalized L 2 -dissimilarity of 0.06. The best defense against each attack is typeset in boldface.an adversary can incorporate the transformation in its attack. For instance, median filtering likely is a weak remedy because one can backpropagate through the median filter, which is sufficient to perform any of the attacks described in Section 3. A strong input-transformation defense should, therefore, be non-differentiable and randomized, a strategy has been previously shown to be effective BID35 b). Two of our top defenses possess both properties:1. Both total variation minimization and image quilting are difficult to differentiate through. Specifically, total variation minimization involves solving a complex minimization of a function that is inherently random. Image quilting involves a discrete variable that selects the patch from the database, which is a non-differentiable operation, and the graph-cut optimization complicates the use of differentiable approximations BID24.2. Both total variation minimization and image quilting give rise to randomized defenses. Total variation minimization randomly selects the pixels it uses to measure reconstruction error on when creating the denoised image. Image quilting randomly selects one of the K nearest neighbors uniformly at random. The inherent randomness of our defenses makes it difficult to attack the model: it implies the adversary has to find a perturbation that alters the prediction for the entire distribution of images that could be used as input, which is harder than perturbing a single image BID27.Our with gray-box attacks suggest that randomness is particularly important in developing strong defenses. Therefore, we surmise that total variation minimization, image quilting, and related methods BID8 are stronger defenses than deterministic denoising procedures such as bit-depth reduction, JPEG compression, or non-local means BID4. Defenses based on total variation minimization and image quilting also have an advantage over adversarial-training approaches BID20: an adversarially trained network is differentiable, which implies that it can be attacked using the methods in Section 3. An additional disadvantage of adversarial training is that it focuses on a particular attack; by contrast, transformation-based defenses generalize well across attack methods because they are model-agnostic. While our study focuses exclusively on image classification, we expect similar defenses to be useful in other domains for which successful attacks have been developed, such as semantic segmentation and speech recognition BID6 BID38. In speech recognition, for example, total variance minimization can be used to remove perturbations from waveforms, and one could develop "spectrogram quilting" techniques that reconstruct a spectrogram by concatenating "spectrogram patches" along the temporal dimension. We leave such extensions to future work. In future work, we also intend to study combinations of our input-transformation defenses with ensemble adversarial training BID34, and we intend to investigate new attack methods that are specifically designed to circumvent our input-transformation defenses.

Represent the TLDR for finding the corresponding scientific article

We apply a model-agnostic defense strategy against adversarial examples and achieve 60% white-box accuracy and 90% black-box accuracy against major attack algorithms.

Represent the short summary for finding the corresponding scientific article (long)


Question: Represent.

Question: In this paper, we propose the Asynchronous Accelerated Nonuniform Randomized Block Coordinate Descent algorithm (A2BCD). We prove A2BCD converges linearly to a solution of the convex minimization problem at the same rate as NU_ACDM, so long as the maximum delay is not too large. This is the first asynchronous Nesterov-accelerated algorithm that attains any provable speedup. Moreover, we then prove that these algorithms both have optimal complexity. Asynchronous algorithms complete much faster iterations, and A2BCD has optimal complexity. Hence we observe in experiments that A2BCD is the top-performing coordinate descent algorithm, converging up to 4-5x faster than NU_ACDM on some data sets in terms of wall-clock time. To motivate our theory and proof techniques, we also derive and analyze a continuous-time analog of our algorithm and prove it converges at the same rate. In this paper, we propose and prove the convergence of the Asynchronous Accelerated Nonuniform Randomized Block Coordinate Descent algorithm (A2BCD), the first asynchronous Nesterovaccelerated algorithm that achieves optimal complexity. No previous attempts have been able to prove a speedup for asynchronous Nesterov acceleration. We aim to find the minimizer x * of the unconstrained minimization problem: DISPLAYFORM0 f (x) = f x,..., x (n) (FORMULA52 where f is σ-strongly convex for σ > 0 with L-Lipschitz gradient ∇f = (∇ 1 f, . . ., ∇ n f). x ∈ R d is composed of coordinate blocks x,..., x (n). The coordinate blocks of the gradient ∇ i f are assumed L i -Lipschitz with respect to the ith block. That is, ∀x, h ∈ R d: DISPLAYFORM1 where P i is the projection onto the ith block of R d. LetL 1 n n i=1 L i be the average block Lipschitz constant. These conditions on f are assumed throughout this whole paper. Our algorithm can also be applied to non-strongly convex objectives (σ = 0) or non-smooth objectives using the black box reduction techniques proposed in BID1. Hence we consider only the coordinate smooth, strongly-convex case. Our algorithm can also be applied to the convex regularized ERM problem via the standard dual transformation (see for instance): DISPLAYFORM2 Hence A2BCD can be used as an asynchronous Nesterov-accelerated finite-sum algorithm. Coordinate descent methods, in which a chosen coordinate block i k is updated at every iteration, are a popular way to solve equation 1.1. Randomized block coordinate descent updates a uniformly randomly chosen coordinate block i k with a gradient-descent-like step: DISPLAYFORM3 The complexity K of an algorithm is defined as the number of iterations required to decrease the error E(f (x k) − f (x *)) to less than (f (x 0) − f (x *)). Randomized coordinate descent has a complexity of K = O(n(L/σ) ln(1/)).Using a series of averaging and extrapolation steps, accelerated improves RBCD's iteration complexity K to O(n L /σ ln(1/)), which leads to much faster convergence whenL σ is large. This rate is optimal when all L i are equal. Finally, using a special probability distribution for the random block index i k, the non-uniform accelerated coordinate descent method BID2 (NU_ACDM) can further decrease the complexity to O(DISPLAYFORM4 L i /σ ln(1/)), which can be up to √ n times faster than accelerated RBCD, since some L i can be significantly smaller than L. NU_ACDM is the current state-of-the-art coordinate descent algorithm for solving equation 1.1.Our A2BCD algorithm generalizes NU_ACDM to the asynchronous-parallel case. We solve equation 1.1 with a collection of p computing nodes that continually read a shared-access solution vector y into local memory then compute a block gradient ∇ i f, which is used to update shared solution vectors (x, y, v). Proving convergence in the asynchronous case requires extensive new technical machinery. A traditional synchronous-parallel implementation is organized into rounds of computation: Every computing node must complete an update in order for the next iteration to begin. However, this synchronization process can be extremely costly, since the lateness of a single node can halt the entire system. This becomes increasingly problematic with scale, as differences in node computing speeds, load balancing, random network delays, and bandwidth constraints mean that a synchronous-parallel solver may spend more time waiting than computing a solution. Computing nodes in an asynchronous solver do not wait for others to complete and share their updates before starting the next iteration. They simply continue to update the solution vectors with the most recent information available, without any central coordination. This eliminates costly idle time, meaning that asynchronous algorithms can be much faster than traditional ones, since they have much faster iterations. For instance, random network delays cause asynchronous algorithms to complete iterations Ω(ln(p)) time faster than synchronous algorithms at scale. This and other factors that influence the speed of iterations are discussed in Hannah & Yin (2017a). However, since many iterations may occur between the time that a node reads the solution vector, and the time that its computed update is applied, effectively the solution vector is being updated with outdated information. At iteration k, the block gradient ∇ i k f is computed at a delayed iterateŷ k defined as 1: DISPLAYFORM5 for delay parameters j(k, 1),..., j(k, n) ∈ N. Here j(k, i) denotes how many iterations out of date coordinate block i is at iteration k. Different blocks may be out of date by different amounts, which is known as an inconsistent read. We assume 2 that j(k, i) ≤ τ for some constant τ < ∞.Asynchronous algorithms were proposed in to solve linear systems. General convergence and theory were developed later in BID5;;; Luo & Tseng (1992; 1993); There is also a rich body of work on asynchronous SGD. In the distributed setting, showed global convergence for stochastic variationally coherent problems even when the delays grow at a polynomial rate. , an asynchronous decentralized SGD was proposed with the same optimal sublinear convergence rate as SGD and linear speedup with respect to the number of workers. , authors obtained an asymptotic rate of convergence for asynchronous momentum SGD on streaming PCA, which provides insight into the tradeoff between asynchrony and momentum. , authors prove convergence for asynchronous SGD that highlight the tradeoff between faster iterations and iteration complexity. Further related work is discussed in Section 4. In this paper, we prove that A2BCD attains NU_ACDM's state-of-the-art iteration complexity to highest order for solving equation 1.1, so long as delays are not too large (see Section 2). The proof is very different from that of BID2, and involves significant technical innovations and complexity related to the analysis of asynchronicity. We also prove that A2BCD (and hence NU_ACDM) has optimal complexity to within a constant factor over a fairly general class of randomized block coordinate descent algorithms (see Section 2.1). This extends in to asynchronous algorithms with L i not all equal. Since asynchronous algorithms complete faster iterations, and A2BCD has optimal complexity, we expect A2BCD to be faster than all existing coordinate descent algorithms. We confirm with numerical experiments that A2BCD is the current fastest coordinate descent algorithm (see Section 5).We are only aware of one previous and one contemporaneous attempt at proving convergence for asynchronous Nesterov-accelerated algorithms. However, the first is not accelerated and relies on extreme assumptions, and the second obtains no speedup. Therefore, we claim that our are the first-ever analysis of asynchronous Nesterov-accelerated algorithms that attains a speedup. Moreover, our speedup is optimal for delays not too large 3.The work of Meng et al. claims to obtain square-root speedup for an asynchronous accelerated SVRG.In the case where all component functions have the same Lipschitz constant L, the complexity they obtain reduces to (n + κ) ln(1/) for κ = O τ n 2 (Corollary 4.4). Hence authors do not even obtain accelerated rates. Their convergence condition is τ < In a contemporaneous preprint, authors in skillfully devised accelerated schemes for asynchronous coordinate descent and SVRG using momentum compensation techniques. Although their complexity have the improved √ κ dependence on the condition number, they do not prove any speedup. Their complexity is τ times larger than the serial complexity. Since τ is necessarily greater than p, their imply that adding more computing nodes will increase running time. The authors claim that they can extend their to linear speedup for asynchronous, accelerated SVRG under sparsity assumptions. And while we think this is quite likely, they have not yet provided proof. We also derive a second-order ordinary differential equation (ODE), which is the continuous-time limit of A2BCD (see Section 3). This extends the ODE found in to an asynchronous accelerated algorithm minimizing a strongly convex function. We prove this ODE linearly converges to a solution with the same rate as A2BCD's, without needing to resort to the restarting techniques. The ODE analysis motivates and clarifies the our proof strategy of the main . We should consider functions f where it is efficient to calculate blocks of the gradient, so that coordinate-wise parallelization is efficient. That is, the function should be "coordinate friendly" Peng et al. (2016b). This is a very wide class that includes regularized linear regression, logistic regression, etc. The L 2 -regularized empirical risk minimization problem is not coordinate friendly in general, however the equivalent dual problem is, and hence can be solved efficiently by A2BCD (see , and Section 5).To calculate the k + 1'th iteration of the algorithm from iteration k, we use only one block of the gradient ∇ i k f. We assume that the delays j(k, i) are independent of the block sequence i k, but otherwise arbitrary (This is a standard assumption found in the vast majority of papers, but can be relaxed ; ;). Definition 1. Asynchronous Accelerated Randomized Block Coordinate Descent (A2BCD). Let f be σ-strongly convex, and let its gradient ∇f be L-Lipschitz with block coordinate Lipschitz parameters L i as in equation 1.2. We define the condition number κ = L/σ, and let L = min i L i. Using these parameters, we sample i k in an independent and identically distributed (IID) fashion according to DISPLAYFORM0 Let τ be the maximum asynchronous delay. We define the dimensionless asynchronicity parameter ψ, which is proportional to τ, and quantifies how strongly asynchronicity will affect convergence: DISPLAYFORM1 We use the above system parameters and ψ to define the coefficients α, β, and γ via eqs. (2.3) to (2.5). Hence A2BCD algorithm is defined via the iterations: eqs. (2.6) to (2.8). DISPLAYFORM2 See Section A for a discussion of why it is practical and natural to have the gradient DISPLAYFORM3 DISPLAYFORM4 Here we define y k = y 0 for all k < 0. The determination of the coefficients c i is in general a very involved process of trial and error, intuition, and balancing competing requirements. The algorithm doesn't depend on the coefficients, however; they are only an analytical tool. We define DISPLAYFORM5 To simplify notation 4, we assume that the minimizer x * = 0, and that f (x *) = 0 with no loss in generality. We define the Lyapunov function: DISPLAYFORM6 We now present this paper's first main contribution. Theorem 1. Let f be σ-strongly convex with a gradient ∇f that is L-Lipschitz with block Lipschitz constants DISPLAYFORM7 ). Then for A2BCD we have: DISPLAYFORM8 To obtain E[ρ k] ≤ ρ 0, it takes K A2BCD iterations for: 13) where O(·) is asymptotic with respect to σ −1/2 S → ∞, and uniformly bounded. DISPLAYFORM9 This is proven in Section B. A stronger for L i ≡ L can be proven, but this adds to the complexity of the proof; see Section E for a discussion. In practice, asynchronous algorithms are far more resilient to delays than the theory predicts. τ can be much larger without negatively affecting the convergence rate and complexity. This is perhaps because we are limited to a worst-case analysis, which is not representative of the average-case performance. Allen-Zhu et al. FORMULA15 (Theorem 5.1) shows a linear convergence rate of 1 − 2/ 1 + 2σ −1/2 S for NU_ACDM, which leads to the corresponding iteration complexity of DISPLAYFORM10. Hence, we have: DISPLAYFORM11 We can assume x * = 0 with no loss in generality since we may translate the coordinate system so that x * is at the origin. We can assume f (x *) = 0 with no loss in generality, since we can replace f (x) with f (x)−f (x *). Without this assumption, the Lyapunov function simply becomes: DISPLAYFORM12 Published as a conference paper at ICLR 2019 DISPLAYFORM13, the complexity of A2BCD asymptotically matches that of NU_ACDM. Hence A2BCD combines state-of-the-art complexity with the faster iterations and superior scaling that asynchronous iterations allow. We now present some special cases of the conditions on the maximum delay τ required for good complexity. Remark 1. Reduction to synchronous case. Notice that when τ = 0, we have ψ = 0, c i ≡ 0 and hence A k ≡ 0. Thus A2BCD becomes equivalent to NU_ACDM, the Lyapunov function 5 ρ k becomes equivalent to one found in BID2 (pg. 9), and Theorem 1 yields the same complexity. The maximum delay τ will be a function τ (p) of p, number of computing nodes. Clearly τ ≥ p, and experimentally it has been observed that τ = O(p). Let gradient complexity K(, τ) be the number of gradients required for an asynchronous algorithm with maximum delay τ to attain suboptimality. τ = 0, since with only 1 computing node there can be no delay. This corresponds to the serial complexity. We say that an asynchronous algorithm attains a complexity speedup if DISPLAYFORM0 is increasing in p. We say it attains linear complexity speedup if DISPLAYFORM1 In Theorem 1, we obtain a linear complexity speedup (for p not too large), whereas no other prior attempt can attain even a complexity speedup with Nesterov acceleration. In the ideal scenario where the rate at which gradients are calculated increases linearly with p, algorithms that have linear complexity speedup will have a linear decrease in wall-clock time. However in practice, when the number of computing nodes is sufficiently large, the rate at which gradients are calculated will no longer be linear. This is due to many parallel overhead factors including too many nodes sharing the same memory read/write bandwidth, and network bandwidth. However we note that even with these issues, we obtain much faster convergence than the synchronous counterpart experimentally. NU_ACDM and hence A2BCD are in fact optimal in some sense. That is, among a fairly wide class of coordinate descent algorithms A, they have the best-possible worst-case complexity to highest order. We extend the work in to encompass algorithms are asynchronous and have unequal L i. For a subset S ∈ R d, we let IC(S) (inconsistent read) denote the set of vectors v whose components are a combination of components of vectors in the set S. DISPLAYFORM0 Definition 4. Asynchronous Randomized Incremental Algorithms. Consider the unconstrained minimization problem equation 1.1 for function f satisfying the conditions stated in Section 1. We define the class A as algorithms G on this problem such that: DISPLAYFORM1 This is a rather general class: x k+1 can be constructed from any inconsistent reading of past iterates IC(X k), and any past gradient of an inconsistent read ∇ ij f (IC(X j)). DISPLAYFORM2 Hence A has a complexity lower bound: DISPLAYFORM3 Our proof in Section D follows very similar lines to;. In this section we present and analyze an ODE which is the continuous-time limit of A2BCD. This ODE is a strongly convex, and asynchronous version of the ODE found in. For simplicity, assume L i = L, ∀i. We rescale (I.e. we replace f (x) with 1 σ f.) f so that σ = 1, and hence κ = L/σ = L. Taking the discrete limit of synchronous A2BCD (i.e. accelerated RBCD), we can derive the following ODE 6 (see Section equation C.1): DISPLAYFORM0 We define the parameter η nκ 1/2, and the energy: DISPLAYFORM1. This is very similar to the Lyapunov function discussed in equation 2.11, with DISPLAYFORM2 the role of v k 2, and A k = 0 (since there is no delay yet). Much like the traditional analysis in the proof of Theorem 1, we can derive a linear convergence with a similar rate. See Section C.2. We may also analyze an asynchronous version of equation 3.1 to motivate the proof of our main theorem. HereŶ (t) is a delayed version of Y (t) with the delay bounded by τ. DISPLAYFORM0 Unfortunately, this energy satisfies (see Section equation C.4, equation C.7): DISPLAYFORM1 Hence this energy E(t) may not be decreasing in general. But, we may add a continuous-time asynchronicity error (see), much like in Definition 2, to create a decreasing energy. Let c 0 ≥ 0 and r > 0 be arbitrary constants that will be set later. Define: DISPLAYFORM2 Lemma 6. When rτ ≤ 1 2, the asynchronicity error A(t) satisfies: DISPLAYFORM3 DISPLAYFORM4 Hence f (Y (t)) convergence linearly to f (x *) with rate O exp −t/(nκ 1/2) Notice how this convergence condition is similar to Corollary 3, but a little looser. The convergence condition in Theorem 1 can actually be improved to approximately match this (see Section E).Proof. DISPLAYFORM5 The preceding should hopefully elucidate the logic and general strategy of the proof of Theorem 1. We now discuss related work that was not addressed in Section 1. Nesterov acceleration is a method for improving an algorithm's iteration complexity's dependence the condition number κ. FORMULA15 showed that many of the assumptions used in prior work (such as bounded delay τ < ∞) were unrealistic and unnecessary in general. In Hannah & Yin (2017a) the authors showed that asynchronous iterations will complete far more iterations per second, and that a wide class of asynchronous algorithms, including asynchronous RBCD, have the same iteration complexity as their synchronous counterparts. Hence certain asynchronous algorithms can be expected to significantly outperform traditional ones. authors propose a novel asynchronous catalyst-accelerated BID6 primal-dual algorithmic framework to solve regularized ERM problems. They structure the parallel updates so that the data that an update depends on is up to date (though the rest of the data may not be). However catalyst acceleration incurs a log(κ) penalty over Nesterov acceleration in general. In BID0, the author argues that the inner iterations of catalyst acceleration are hard to tune, making it less practical than Nesterov acceleration. To investigate the performance of A2BCD, we solve the ridge regression problem. Consider the following primal and corresponding dual objective (see for instance): DISPLAYFORM0 where A ∈ R d×n is a matrix of n samples and d features, and l is a label vector. We let A = [A 1, . . ., A m] where A i are the column blocks of A. We compare A2BCD (which is asynchronous accelerated), synchronous NU_ACDM (which is synchronous accelerated), and asynchronous RBCD (which is asynchronous non-accelerated). Nodes randomly select a coordinate block according to equation 2.1, calculate the corresponding block gradient, and use it to apply an update to the shared solution vectors. synchronous NU_ACDM is implemented in a batch fashion, with batch size p (1 block per computing node). Nodes in synchronous NU_ACDM implementation must wait until all nodes apply their computed gradients before they can start the next iteration, but the asynchronous algorithms simply compute with the most up-to-date information available. We use the datasets w1a (47272 samples, 300 features), wxa which combines the data from from w1a to w8a (293201 samples, 300 features), and aloi (108000 samples, 128 features) from. The algorithm is implemented in a multi-threaded fashion using C++11 and GNU Scientific Library with a shared memory architecture. We use 40 threads on two 2.5GHz 10-core Intel Xeon E5-2670v2 processors. See Section A.1 for a discussion of parameter tuning and estimation. The parameters for each algorithm are tuned to give the fastest performance, so that a fair comparison is possible. A critical ingredient in the efficient implementation of A2BCD and NU_ACDM for this problem is the efficient update scheme discussed in Lee & Sidford (2013b; a). In linear regression applications such as this, it is essential to be able to efficiently maintain or recover Ay. This is because calculating block gradients requires the vector A T i Ay, and without an efficient way to recover Ay, block gradient evaluations are essentially 50% as expensive as full-gradient calculations. Unfortunately, every accelerated iteration in dense updates to y k because of the averaging step in equation 2.6. Hence Ay must be recalculated from scratch. However Lee & Sidford (2013a) introduces a linear transformation that allows for an equivalent iteration that in sparse updates to new iteration variables p and q. The original purpose of this transformation was to ensure that the averaging steps (e.g. equation 2.6) do not dominate the computational cost for sparse problems. However we find a more important secondary use which applies to both sparse and dense problems. Since the updates to p and q are sparse coordinate-block updates, the vectors Ap, and Aq can be efficiently maintained, and therefore block gradients can be efficiently calculated. The specifics of this efficient implementation are discussed in Section A.2.In Table 5, we plot the sub-optimality vs. time for decreasing values of λ, which corresponds to increasingly large condition numbers κ. When κ is small, acceleration doesn't in a significantly better convergence rate, and hence A2BCD and async-RBCD both outperform sync-NU_ACDM since they complete faster iterations at similar complexity. Acceleration for low κ has unnecessary overhead, which means async-RBCD can be quite competitive. When κ becomes large, async-RBCD is no longer competitive, since it has a poor convergence rate. We observe that A2BCD and sync-NU_ACDM have essentially the same convergence rate, but A2BCD is up to 4 − 5× faster than sync-NU_ACDM because it completes much faster iterations. We observe this advantage despite the fact that we are in an ideal environment for synchronous computation: A small, homogeneous, high-bandwidth, low-latency cluster. In large-scale heterogeneous systems with greater synchronization overhead, bandwidth constraints, and latency, we expect A2BCD's advantage to be much larger. TAB4: Sub-optimality f (y k) − f (x *) (y-axis) vs time in seconds (x-axis) for A2BCD, synchronous NU_ACDM, and asynchronous RBCD for data sets w1a, wxa and aloi for various values of λ. An efficient implementation will have coordinate blocks of size greater than 1. This to ensure the efficiency of linear algebra subroutines. Especially because of this, the bulk of the computation for each iteration is computing ∇ i k f (ŷ k), and not the averaging steps. Hence the computing nodes only need a local copy of y k in order to do the bulk of an iteration's computation. Given this gradient ∇ i k f (ŷ k), updating y k and v k is extremely fast (x k can simply be eliminated). Hence it is natural to simply store y k and v k centrally, and update them when the delayed gradients ∇ i k f (ŷ k). Given the above, a write mutex over (y, v) has minuscule overhead (which we confirm with experiments), and makes the labeling of iterates unambiguous. This also ensures that v k and y k are always up to date when (y, v) are being updated. Whereas the gradient ∇ i k f (ŷ k) may at the same time be out of date, since it has been calculated with an outdated version of y k. However a write mutex is not necessary in practice, and does not appear to affect convergence rates or computation time. Also it is possible to prove convergence under more general asynchronicity. When defining the coefficients, σ may be underestimated, and L, L 1,..., L n may be overestimated if exact values are unavailable. Notice that x k can be eliminated from the above iteration, and the block gradient ∇ i k f (ŷ k) only needs to be calculated once per iteration. A larger (or overestimated) maximum delay τ will cause a larger asynchronicity parameter ψ, which leads to more conservative step sizes to compensate. To estimate ψ, one can first performed a dry run with all coefficient set to 0 to estimate τ. All function parameters can be calculated exactly for this problem in terms of the data matrix and λ. We can then use these parameters and this tau to calculate ψ. ψ and τ merely change the parameters, and do not change execution patterns of the processors. Hence their parameter specification doesn't affect the observed delay. Through simple tuning though, we found that ψ = 0.25 ed in good performance. In tuning for general problems, there are theoretical reasons why it is difficult to attain acceleration without some prior knowledge of σ, the strong convexity modulus BID3. Ideally σ is pre-specified for instance in a regularization term. If the Lipschitz constants L i cannot be calculated directly (which is rarely the case for the classic dual problem of empirical risk minimization objectives), the line-search method discussed in Section 4 can be used. As mentioned in Section 5, authors in Lee & Sidford (2013a) proposed a linear transformation of an accelerated RBCD scheme that in sparse coordinate updates. Our proposed algorithm can be given a similar efficient implementation. We may eliminate x k from A2BCD, and derive the equivalent iteration below: DISPLAYFORM0 where C and Q k are defined in the obvious way. Hence we define auxiliary variables p k, q k defined via: DISPLAYFORM0 These clearly follow the iteration: DISPLAYFORM1 Since the vector Q k is sparse, we can evolve variables p k, and q k in a sparse manner, and recover the original iteration variables at the end of the algorithm via A.1.The gradient of the dual function is given by: DISPLAYFORM2 As mentioned before, it is necessary to maintain or recover Ay k to calculate block gradients. Since Ay k can be recovered via the linear relation in equation A.1, and the gradient is an affine function, we maintain the auxiliary vectors Ap k and Aq k instead. Hence we propose the following efficient implementation in Algorithm 1. We used this to generate the in Table 5. We also note also that it can improve performance to periodically recover v k and y k, reset the values of p k, q k, and C to v k, y k, and I respectively, and restarting the scheme (which can be done cheaply in time O(d)).We let B ∈ R 2×2 represent C k, and b represent B −1. ⊗ is the Kronecker product. Each computing node has local outdated versions of p, q, Ap, Aq which we denotep,q,Âp,Âq respectively. We also find it convenient to define: DISPLAYFORM3 Algorithm 1 Shared-memory implementation of A2BCD Randomly select block i via equation 2.1. Read shared data into local memory:p ← p,q ← q,Âp ← Ap,Âq ← Aq,B ← B. Compute block gradient: DISPLAYFORM0 10: DISPLAYFORM1 11: DISPLAYFORM0 12: DISPLAYFORM1 Increase iteration count: k ← k + 1 14: end while 15: Recover original iteration variables: DISPLAYFORM2 We first recall a couple of inequalities for convex functions. Lemma 7. Let f be σ-strongly convex with L-Lipschitz gradient. Then we have: DISPLAYFORM0 We also find it convenient to define the norm: DISPLAYFORM1 B.1 Starting point First notice that using the definition equation 2.8 of v k+1 we have: DISPLAYFORM0 We have the following general identity: DISPLAYFORM1 It can also easily be verified from equation 2.6 that we have: DISPLAYFORM2 DISPLAYFORM3 This inequality is our starting point. We analyze the terms on the second line in the next section. To analyze these terms, we need a small lemma. This lemma is fundamental in allowing us to deal with asynchronicity. Lemma 8. Let χ, A > 0. Let the delay be bounded by τ. Then: DISPLAYFORM0 Proof. See Hannah & Yin (2017a). We have: DISPLAYFORM0 The terms in bold in equation B.8 and equation B.9 are a of the asynchronicity, and are identically 0 in its absence. Our strategy is to separately analyze terms that appear in the traditional analysis of Nesterov FORMULA15, and the terms that from asynchronicity. We first prove equation B.8: FIG5 equation B.10 follows from strong convexity (equation B.2 with x = y k and y = x *), and the fact that ∇f is L-Lipschitz. The term due to asynchronicity becomes: DISPLAYFORM0 DISPLAYFORM1 using Lemma 8 with χ = κψ −1, A = y k. Combining this with equation B.10 completes the proof of equation B.8.We now prove equation B.9: DISPLAYFORM2 Here the last line follows from Lemma 8 with χ = κψ DISPLAYFORM3 We can complete the proof using the following identity that can be easily obtained from equation 2.6: DISPLAYFORM4 Much like , we need a f (x k) term in the Lyapunov function (see the middle of page 357). However we additionally need to consider asynchronicity when analyzing the growth of this term. Again terms due to asynchronicity are emboldened. Lemma 10. We have: DISPLAYFORM0 Proof. From the definition equation 2.7 of x k+1, we can see that x k+1 − y k is supported on block i k. Since each gradient block ∇ i f is L i Lipschitz with respect to changes to block i, we can use equation B.1 to obtain: DISPLAYFORM1 Here the last line followed from the definition equation B.3 of the norm · * 1/2. We now analyze the middle term: DISPLAYFORM2 We then apply Lemma 8 to this with χ = 2h DISPLAYFORM3 Finally to complete the proof, we combine equation B.11, with equation B.12. The previous inequalities produced difference terms of the form y k+1−j − y k−j 2. The following lemma shows how these errors can be incorporated into a Lyapunov function. Lemma 11. Let 0 < r < 1 and consider the asynchronicity error and corresponding coefficients: DISPLAYFORM0 Remark 2. Interpretation. This means that an asynchronicity error term A k can negate a series of difference terms − ∞ j=1 s j y k+1−j − y k−j 2 at the cost of producing an additional error c 1 E k y k+1 − y k 2, while maintaining a convergence rate of r. This essentially converts difference terms, which are hard to deal with, into a y k+1 − y k 2 term which can be negated by other terms in the Lyapunov function. The proof is straightforward. Proof. DISPLAYFORM0 Noting the following completes the proof: DISPLAYFORM1 Given that A k allows us to negate difference terms, we now analyze the cost c 1 E k y k+1 − y k 2 of this negation. We have: DISPLAYFORM0 Proof. DISPLAYFORM1 Here equation B.13 following from equation 2.8, the definition of v k+1. equation B.14 follows from the inequality x + y 2 ≤ 2 x 2 + 2 y 2. The rest is simple algebraic manipulation. DISPLAYFORM2 (definitions of h and α: equation 2.3, and equation 2.5) = 1 DISPLAYFORM3 Rearranging the definition of ψ, we have: DISPLAYFORM4 Using this on equation B.15, we have: DISPLAYFORM5 This completes the proof. We are finally in a position to bring together all the all the previous together into a master inequality for the Lyapunov function ρ k (defined in equation 2.11). After this lemma is proven, we will prove that the right hand size is negative, which will imply that ρ k linearly converges to 0 with rate β. Lemma 13. Master inequality. We have: DISPLAYFORM0 Proof. DISPLAYFORM1 We now collect and organize the similar terms of this inequality. DISPLAYFORM2 Now finally, we add the function-value and asynchronicity terms to our analysis. We use Lemma 11 is with r = 1 − σ 1/2 S −1, and DISPLAYFORM3 Notice that this choice of s i will recover the coefficient formula given in equation 2.9. Hence we have: DISPLAYFORM4 (Lemmas 11 and 12) + c 1 2α In the next section, we will prove that every coefficient on the right hand side of equation B.16 is 0 or less, which will complete the proof of Theorem 1. DISPLAYFORM5 DISPLAYFORM6 Here the last line followed since ψ ≤ 1 2 and σ 1/2 S −1 ≤ 1. We now analyze the coefficient of DISPLAYFORM7 Proof. DISPLAYFORM8 in Lemma 13 is non-positive. Proof. We first need to bound c 1.(equation B.18 and equation 2.9) c 1 = s DISPLAYFORM9 It can be easily verified that if x ≤ 1 2 and y ≥ 0, then (1 − x) −y ≤ exp(2xy). Using this fact with x = σ 1/2 S −1 and y = τ, we have: DISPLAYFORM10 (since ψ ≤ 3/7 and hence τ σ DISPLAYFORM11 We now analyze the coefficient of ∇f (ŷ k) DISPLAYFORM12 Proof. DISPLAYFORM13 Here the last inequality follows since β ≤ 1 and α ≤ σ 1/2 S −1. We now rearrange the definition of ψ to yield the identity: DISPLAYFORM14 Using this, we have: DISPLAYFORM15 Here the last line followed since L ≤ L, ψ ≤ 3 7, and τ ≥ 1. Hence the proof is complete. Proof of Theorem 1. Using the master inequality 13 in combination with the previous Lemmas 14, 15, 16, and 17, we have: DISPLAYFORM16 When we have: DISPLAYFORM17 then the Lyapunov function ρ k has decreased below ρ 0 in expectation. Hence the complexity K satisfies: DISPLAYFORM18 Now it can be shown that for 0 < x ≤ 1 2, we have: DISPLAYFORM19 Since n ≥ 2, we have σ 1/2 S −1 ≤ 1 2. Hence: DISPLAYFORM20 An expression for K NU_ACDM , the complexity of NU_ACDM follows by similar reasoning. DISPLAYFORM21 Finally we have: DISPLAYFORM22 which completes the proof. C.1 Derivation of ODE for synchronous A2BCDIf we take expectations with respect to E k, then synchronous (no delay) A2BCD becomes: DISPLAYFORM0 We find it convenient to define η = nκ 1/2. Inspired by this, we consider the following iteration: DISPLAYFORM1 C.1 Derivation of ODE for synchronous A2BCDfor coefficients: DISPLAYFORM2 s is a discretization scale parameter that will be sent to 0 to obtain an ODE analogue of synchronous A2BCD. We first use equation DISPLAYFORM3 The proof of convergence is completed in Section 3. For parameter set σ, L 1,..., L n, n, we construct a block-separable function f on the space R As mentioned, a stronger than Theorem 1 is possible. In the case when L i = L for all i, we can consider a slight modification of the coefficients: DISPLAYFORM0 (E.1) DISPLAYFORM1 for the asynchronicity parameter: DISPLAYFORM2 This leads to complexity: DISPLAYFORM3 Here there is no restriction on ψ as in Theorem 1, and hence there is no restriction on τ. Assuming ψ ≤ 1 gives optimal complexity to within a constant factor. Notice then that the ing condition of τ τ ≤ 1 6 nκ −1/2 (E.9) now essentially matches the one in Theorem 3 in Section 3. While this is stronger, it increases the complexity of the proof substantially. So in the interests of space and simplicity, we do not prove this stronger .

represent this text to find a much much longer article about it with say hundreds-thousands of words

We prove the first-ever convergence proof of an asynchronous accelerated algorithm that attains a speedup.

General analysis of sign-based methods (e.g. signSGD) for non-convex optimization, built on intuitive bounds on success probabilities.


Question: Represent the scientific article to find a short TLDR

Question: A framework for efficient Bayesian inference in probabilistic programs is introduced by embedding a sampler inside a variational posterior approximation. Its strength lies in both ease of implementation and automatically tuning sampler parameters to speed up mixing time. Several strategies to approximate the evidence lower bound (ELBO) computation are introduced, including a rewriting of the ELBO objective. Experimental evidence is shown by performing experiments on an unconditional VAE on density estimation tasks; solving an influence diagram in a high-dimensional space with a conditional variational autoencoder (cVAE) as a deep Bayes classifier; and state-space models for time-series data. We consider a probabilistic program (PP) to define a distribution p(x, z), where x are observations and z, both latent variables and parameters, and ask queries involving the posterior p(z|x). This distribution is typically intractable but, conveniently, probabilistic programming languages (PPLs) provide inference engines to approximate it using Monte Carlo methods (e.g. particle Markov Chain Monte Carlo (MCMC) or Hamiltonian Monte Carlo (HMC) ) or variational approximations (e.g. Automatic Differentiation Variational Inference (ADVI) ). Whereas the latter are biased and underestimate uncertainty, the former may be exceedingly slow depending on the target distribution. For such reason, over the recent years, there has been an increasing interest in developing more efficient posterior approximations (; ;). It is known that the performance of a sampling method depends on the parameters used . Here we propose a framework to automatically adapt the posterior shape and tune the parameters of a posterior sampler with the aim of boosting Bayesian inference in PPs. Our framework constitutes a principled way to enhance the flexibility of the variational posterior approximation, yet can be seen also as a procedure to tune the parameters of an MCMC sampler. Our contributions are a new flexible and unbiased variational approximation to the posterior, which improves an initial variational approximation with a (learnable via automatic differentiation) stochastic process. Appendix A discusses related work. In standard VI, the variational approximation q φ (z|x) is analytically tractable and typically chosen as a factorized Gaussian distribution. We propose to use a more flexible approximating posterior by embedding a sampler through: q φ,η (z|x) = Q η,T (z|z 0)q 0,φ (z 0 |x)dz 0, where q 0,φ (z|x) is the initial and tractable density (i.e., the starting state for the sampler). We refer to q φ,η (z|x) as the refined variational approximation. The distribution Q η,T (z|z 0) refers to a stochastic process parameterized by η used to evolve the original density q 0,φ (z|x) and achieve greater flexibility; we describe below particular forms of it. When T = 0, no refinement steps are performed, and the refined variational approximation coincides with the original one, q φ,η (z|x) = q 0,φ (z|x). As T increases, the variational approximation will be closer to the exact posterior, provided that Q η,T is a valid MCMC sampler. Next, we maximize a refined ELBO objective, ELBO(q) = E q φ,η (z|x) [log p(x, z) − log q φ,η (z|x)] to optimize the divergence KL(q φ,η (z|x)||p(z|x)). The first term of ELBO only requires sampling from q φ,η (z|x); however the second term, −E q φ,η (z|x) [log q φ,η (z|x)] requires also evaluating the evolving density. Regarding Q η,T (z|z 0), we consider the following families of sampling algorithms. When the latent variables z are continuous (z ∈ R d), we evolve the original variational density q 0,φ (z|x) through a stochastic diffusion process. To make it tractable, we discretize the Langevin dynamics using the Euler-Maruyama scheme, arriving at the stochastic gradient Langevin dynamics (SGLD) sampler. We then follow the process Q η,T (z|z 0) (representing T iterations of an MCMC sampler). As an example, for the SGLD sampler z i = z i−1 + η∇ log p(x, z i−1) + ξ i, where i iterates from 1 to T; in this case, the only parameter of the SGLD sampler is the learning rate η. The noise for the SGLD is ξ i ∼ N (0, 2ηI). The initial variational distribution q 0,φ (z|x) is a Gaussian parameterized by a deep neural network (NN). Then, T iterations of a sampler Q parameterized by η are applied leading to q φ,η. An alternative may be given by ignoring the noise vector ξ , thus refining the initial variational approximation with just stochastic gradient descent (SGD). Moreover, we can use Stein variational gradient descent (SVGD) or a stochastic version to apply repulsion between particles and promote a more extensive exploration of the latent space. We propose a set of guidelines for the ELBO optimization using the refined variational approximation. Particle approximation We can consider the flow Q η,T (z|z 0) as a mixture of Dirac deltas (i.e., we approximate it with a finite set of particles). That is, we sample z 1,..., z K ∼ Q η,T (z|z 0) and useQ η,T (z|z 0) = T i=1 q η (z i |z i−1)q 0,φ (z 0 |x). The entropy for each factor can be straightforwardly computed, i.e. for the case of SGLD, q η (z i |z i−1) = N (z i−1 + η∇ log p(x, z i−1), 2ηI). This approximation keeps track of a better estimate of the entropy than the particle approximation. Deterministic flows If using a deterministic flow (such as SGD or SVGD), we can keep track of the change in entropy at each iteration using the change of variable formula as done in. However, this requires a costly Jacobian computation, making it unfeasible to combine with our backpropagation through the sampler scheme (Sec. 2.3) for moderately complex problems. In standard VI, the variational approximation q(z|x; φ) is parameterized by φ. The parameters are learned using SGD or variants such as Adam , using ∇ φ ELBO(q). Since we have shown how to embed a sampler inside the variational guide, it is also possible to compute a gradient of the objective with respect to the sampler parameters η. For instance, we can compute a gradient with respect to the learning rate η from the SGLD or SGD process from Section 2.1, ∇ η ELBO(q), to search for an optimal step size at every VI iteration. This is an additional step apart from using the gradient ∇ φ ELBO(q) employed to learn a good initial sampling distribution. See Appendix D.3 for a discussion on two modes of automatic differentiation that can be used. Code is released at https://github.com/vicgalle/vis. The VIS framework was implemented using Pytorch , though we also release a notebook for the first experiment using Jax to highlight its simple implementation. Appendix B contains additional experiments; Appendix C, implementation details. Funnel density As a preliminary experiment, we test the VIS framework on a synthetic yet complex target distribution. The target, bi-dimensional density is defined through: As a variational approximation we take the usual diagonal Gaussian. For the VIS case, we refine it for T = 1 steps using SGLD. Results are in Figure 1. Clearly, our refined version achieves a tighter bound, the VIS variant is placed nearer to the mean of the true distribution and is more disperse than the original variational approximation, confirming that the refinement step helps in attaining more flexible posterior approximations. State-space model (DLM) We now test the VIS framework on the Mauna Loa monthly CO 2 time series data . As the training set, we take the first 10 years, and we evaluate over the next 2 years. We use a dynamic linear model (DLM) composed of a local linear trend plus a seasonality block of periodicity 12. Full model specification can be checked in Appendix C.1. As a preprocessing step, we standardize the time series to zero mean and unitary deviation. To guarantee the same computational budget time, the model without refining is run for 10 epochs, whereas the model with refinement is run for 4 epochs. We use the particle approximation from Sec. 2.2. We report mean absolute error (MAE) and predictive entropy in Table 1. In addition, we compute the interval score as defined in , a strictly proper scoring rule. As can be seen, for similar wall-clock times, the refined model not only achieves lower MAE, but also its predictive intervals are narrower than the non-refined counterpart. Variational Autoencoder We aim to check whether VIS is competitive with respect to other recent algorithms. We test our approach in a Variational Autoencoder (VAE) model , which is the building block of more complex models and tasks (b;). The VAE defines a conditional distribution p θ (x|z), generating an observation x from a latent variable z. We are interested in modelling two 28 × 28 image distributions, MNIST and fashion-MNIST. To perform inference (learn parameters θ), the VAE introduces a variational approximation q φ (z|x). In the standard setting, this is Gaussian; we instead use the refined variational approximation with various values of T. We used the MC approximation, though achieved similar using the Gaussian one. As experimental setup, we reproduce the setting from. Results are reported in Table 2. To guarantee a fair comparison, we trained the VIS-5-10 variant for 10 epochs, whereas all the other variants were trained for 15 (fMNIST) or 20 epochs (MNIST), so that the VAE performance is comparable to that in. Although VIS is trained for less epochs, by increasing the number of MCMC iterations T, we dramatically improve on test log-likelihood. In terms of computational complexity, the average time per epoch using T = 5 is 10.46s, whereas with no refinement (T = 0) is 6.10s (hence our decision to train the refined variant for less epochs): a moderate increase in computing time compensates the dramatic increase in log-likelihood while not introducing new parameters, except for the learning rate η. We also compare our with the contrastive divergence approach . Figure 2 displays ten random samples of reconstructed digit images as visual check. Discussion We have proposed a flexible and efficient framework to perform inference in probabilistic programs defining wide classes of models. Our framework can be seen as a general way of tuning SG-MCMC sampler parameters, adapting the initial distributions and the learning rate. Key to the success and applicability of the VIS framework are the approximations introduced for the intractable parts of the refined variational approximations, which are computationally cheap but convenient. The idea of preconditioning the posterior distribution to speed up the mixing time of an MCMC sampler has recently been explored in and , where a reparameterization is learned before performing the sampling via HMC. Both papers extend seminal work of by learning an efficient and expressive deep, non-linear transformation instead of a polynomial regression. However, they do not account for tuning the parameters of the sampler as we introduce in Section 2, where a fully, end to end differentiable sampling scheme is proposed. The work of introduced a general framework for constructing more flexible variational distributions, called normalizing flows. These transformations are one of the main techniques to improve the flexibility of current VI approaches and have recently pervaded the literature of approximate Bayesian inference with current developments such as continuous-time normalizing flows (a) which extend an initial simple variational posterior with a discretization of Langevin dynamics. However, they require a generative adversarial network (GAN) to learn the posterior, which can be unstable in high-dimensional spaces. We overcome this issue with the novel formulation stated in Section 2. Our framework is also compatible with different optimizers, not only those derived from Langevin dynamics. Other recent proposals to create more flexible variational posteriors are based on implicit approaches, which typically require a GAN (Huszár, 2017) or implicit schema such as UIVI or SIVI . Our variational approximation is also implicit, but we use a sampling algorithm to drive the evolution of the density, combined with a Dirac delta approximation to derive an efficient variational approximation as we report on the extensive experiments in the Section 3. Closely related to our framework is the work of , where a VAE is learned using HMC. We use a similar compound distribution as the variational approximation, though our framework allows for any SG-MCMC sampler (via the entropy approximation strategies introduced) and also the tuning of sampler parameters via gradient descent. Our work is also related to the recent idea of amortization of samplers . A common problem with these approaches is that they incur in an additional error, the amortization gap . We alleviate this by evolving a set of particles z i with a stochastic process in the latent space after learning a good initial distribution. Hence, the bias generated by the initial approximation is significantly reduced after several iterations of the process. A recent article related to our paper is , who define a compound distribution similar to our framework. However, we focus on an efficient approximation using the reverse KL divergence, the standard and well understood divergence used in variational inference, which allows for tuning sampler parameters and achieving competitive . With the final experiments we show that the VIS framework can deal with more general probabilistic graphical models. Influence diagrams are one of the most familiar representations of a decision analysis problem. There is a long history on bridging the gap between influence diagrams and probabilistic graphical models (see , for instance), so developing better tools for Bayesian inference can be automatically used to solve influence diagrams. We showcase the flexibility of the proposed scheme to solve inference problems in an experiment with a classification task in a high-dimensional setting. As dataset, the MNIST handwritten digit classification task is chosen, in which grey-scale 28 × 28 images have to be classified in one of the ten classes Y = {0, 1, . . ., 9}. More concretely, we extend the VAE model to condition it on a discrete variable y, leading to the conditional VAE (cVAE). A cVAE defines a decoder distribution p θ (x|z, y) on an input space x ∈ R D given class label y ∈ Y and latent variable z ∈ R d. To perform inference, a variational posterior is learned as an encoder q φ (z|x, y) from a prior p(z) ∼ N (0, I). Leveraging the conditional structure on y, we use the generative model as a classifier using Bayes rule: where we use K Monte Carlo samples z (k) ∼ q φ (z|x, y). In the experiments we set K = 5. Given a test sample x, the labelŷ with highest probability p(y|x) is predicted. Figure 5 in Appendix depicts the corresponding influence diagram. Additional details regarding the model architecture and hyperparameters can be found in Appendix C. For comparison purposes, we perform various experiments changing T for the transition distribution Q η,T in the refined variational approximation. Results are in Table 3. We report the test accuracy achieved at the end of training. Note we are comparing different values of T depending on being on the training or testing phases (in the latter, where the model and variational parameters are kept frozen). The model with T tr = 5 was trained for 10 epochs, whereas the other settings for 15 epochs, in order to give all settings similar training times. Results are averaged from 3 runs with different random seeds. From the it is clear that the effect of using the refined variational approximation (the cases when T > 0) is crucially beneficial to achieve higher accuracy. The effect of learning a good initial distribution and inner learning rate by using the gradients ∇ φ ELBO(q) and ∇ η ELBO(q) has a highly positive impact in the accuracy obtained. On a final note, we have not included the case when only using a SGD or SGLD sampler (i.e., without learning an initial distribution q 0,φ (z|x)) since the were much worse than the ones in Table 3, for a comparable computational budget. This strongly suggests that for We test our variational approximation on two state-space models, one for discrete data and the other for continuous observations. All the experiments in this subsection use the Fast AD version from Section D.3 since it was not necessary to further tune the sampler parameters to have competitive . The model equations are given by where each conditional is a Categorical distribution which takes 5 different classes and the prior p(θ) = p(θ em)p(θ tr) are two Dirichlet distributions that sample the emission and transition probabilities, respectively. We perform inference on the parameters θ. The model equations are the same as in the HMM case, though the conditional distributions are now Gaussian and the parameters θ refer to the emission and transition variances. As before, we perform inference over θ. The full model implementations can be checked in Appendix C.1, based on funsor 1, a PPL on top of the Pytorch autodiff framework. For each model, we generate a synthetic dataset, and use the refined variational approximation with T = 0, 1, 2. As the original variational approximation to the parameters θ we use a Dirac Delta. Performing VI with this approximation corresponds to MAP estimation using the Kalman filter in the DLM case and the Baum-Welch algorithm in the HMM case , since we marginalize out the latent variables z 1:τ. Model details are given in Appendix C.1.1. Figure 3 shows the . The first row reports the experiments related to the HMM; the second one to the DLM. While in all graphs we report the evolution of the loglikelihood during inference, in the first column we report the number of ELBO iterations, whereas in the second column we measure wall-clock time as the optimization takes place. We confirm that VIS (T > 0) achieve better than regular optimization with VI (T = 0) for a similar amount of time. With the aim of assessing whether ELBO optimization helps in attaining better auxiliary scores, we also report on a prediction task. We generate a synthetic time series of alternating 0 and 1 for τ = 105 timesteps. We train the HMM model from before on the first 100 points, and report in Table 4 the accuracy of the predictive distribution p(y t) averaged over the last 5 time-steps. We also report the predictive entropy since it helps in assessing the confidence of the model in its forecast and is a strictly proper scoring rule . To guarantee the same computational budget time and a fair comparison, the model without refining is run with 50 epochs, whereas the model with refinement is run for 20 epochs. We see that the refined model achieves higher accuracy than its counterpart; in addition it is correctly more confident in its predictions. x t ∼ N (3.0z t + 0.5, σ em). with z 0 = 0.0. The DLM model is comprised of a linear trend component plus a seasonal block of period 12. The trend is specified as With respect to the seasonal component, the main idea is to cycle the state: suppose θ t ∈ R p, with p being the seasonal period. Then, at each timestep, the model focuses on the first component of the state vector: Thus, we can specify the seasonal component via: where F is a p−dimensional vector and G is a p × p matrix such that def encode(self, x): h1_mu = F.relu(self.fc1_mu(x)) h1_cov = F.relu(self.fc1_cov(x)) h1_mu = F.relu(self.fc12_mu(h1_mu)) h1_cov = F.relu(self.fc12_cov(h1_cov)) # we work in the logvar-domain return self.fc2_mu(h1_mu), torch.log(F.softplus(self.fc2_cov(h1_cov))) def decode(self, z): h3 = F.relu(self.fc3(z)) h3 = F.relu(self.fc32(h3)) return torch.sigmoid(self.fc4(h3)) The VAE model is implemented with PyTorch . The prior distribution p(z) for the latent variables z ∈ R 10 is a standard factorized Gaussian. The decoder distribution p θ (x|z) and the encoder distribution (initial variational approximation) q 0,φ (z|x) are parameterized by two feed-forward neural networks whose details can be checked in Figure 4. The optimizer Adam is used in all experiments, with a learning rate λ = 0.001. We also set η = 0.001. We train for 15 epochs (fMNIST) and 20 epochs (MNIST), in order to achieve similar performance to the explicit VAE case in . For the VIS-5-10 setting, we train for only 10 epochs, to allow for a fair computational comparison (similar computing times). The cVAE model is implemented with PyTorch . The prior distribution p(z) for the latent variables z ∈ R 10 is a standard factorized Gaussian. The decoder distribution p θ (x|y, z) and the encoder distribution (initial variational approximation) q 0,φ (z|x, y) are parameterized by two feed-forward neural networks whose details can be checked in Figure 6. The integral is approximated with 1 MC sample from the variational approximation in all experimental settings. The optimizer Adam is used in all the experiments, with a learning rate λ = 0.01. We set the initial η = 5e − 5. In this Section we study in detail key properties of the proposed VIS framework. Performing variational inference with the refined variational approximation can be regarded as using the original variational guide while optimizing an alternative, tighter ELBO. Note that for a refined guide of the form q(z|z 0)q(z 0 |x), the objective function can be written as However, using the Dirac Delta approximation for q(z|z 0) and noting that z = z 0 + η∇ log p(x, z 0) when using SGD with T = 1, we arrive at the modified objective: which is equivalent to the refined ELBO introduced in. Since we are perturbing the latent variables in the steepest ascent direction, it is straightforward to show that, for moderate η, the previous bound is tighter than the one, for the original variational guide q(z 0 |x), E q(z 0 |x) [log p(x, z 0) − log q(z 0 |x)]. This reformulation of ELBO is also convenient since it provides a clear way of implementing our refined variational inference framework in any PPL supporting algorithmic differentiation. From the in subsection D.1, we can further restrict to the case when the original variational approximation is also a Dirac point mass. Then, the original ELBO optimization resorts to the standard maximum likelihood estimation, i.e., max z log p(x, z). Within the VIS framework, we optimize instead max z log p(x, z + ∆z), where ∆z is one iteration of the sampler, i.e., ∆z = η∇ log p(x, z) in the SGD case. For notational clarity we resort to the case T = 1, but a similar analysis can be straightforwardly done if more refinement steps are performed. We may now perform a first-order Taylor expansion of the refined objective as log p(x, z + ∆z) ≈ log p(x, z) + (∆z) ∇ log p(x, z). Taking gradients of the first order approximation w.r.t. the latent variables z we arrive at ∇ z log p(x, z) + η∇ z log p(x, z) ∇ 2 z log p(x, z), where we have not computed the gradient through the ∆z term. That is, the refined gradient can be deemed as the original gradient plus a second order correction. Instead of being modulated by a constant learning rate, this correction is adapted by the chosen sampler. In the experiments in Section B.1 we show that this is beneficial for the optimization as it can take less iterations to achieve lower losses. By further taking gradients through the ∆z term, we may tune the sampler parameters such as the learning rate as described in Section 2.3. Consequently, the next subsection describes both modes of differentiation. Here we describe how to implement two variants of the ELBO objective. First, we define a stop gradient operator 2 ⊥ that sets the gradient of its operand to zero, i.e., ∇ x ⊥(x) = 0 whereas in the forward pass it acts as the identity function, that is, ⊥(x) = x. Then, the two variants of the ELBO objective are E q [log p(x, z + ∆z) − log q(z + ∆z|x)] (Full AD) and E q [log p(x, z + ⊥(∆z)) − log q(z + ⊥(∆z)|x)]. (Fast AD) The Full AD ELBO makes it possible to further compute a gradient wrt sampler parameters inside ∆z at the cost of a slight increase in the computational burden. Note that for a large class of models (including HMMs and DLMs) we can marginalize out z 1:τ and have reduced variance iterating with: θ ← θ + ∇ θ log p(x 1:τ |θ) + ξ, where the latent variables z 1:τ have been marginalized out using the sum-product algorithm. For linear-Gaussian models we can also compute the exact form of the refined posterior, since all terms in Eq. 5 are linear wrt the latent variables θ. However, inference in these linear models is exact by using conjugate distributions, so the proposed framework is more fit to the case of state-space models containing non-linear (or non-conjugate) components. For these families of models, we resort to use just a gradient estimator of the entropy or the Delta approximation in Section 2.1.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We embed SG-MCMC samplers inside a variational approximation

Represent the short summary for finding the corresponding scientific article (long).

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.