Question: Represent the article for finding a single-sentence TLDR

Question: Generative models that can model and predict sequences of future events can, in principle, learn to capture complex real-world phenomena, such as physical interactions. However, a central challenge in video prediction is that the future is highly uncertain: a sequence of past observations of events can imply many possible futures. Although a number of recent works have studied probabilistic models that can represent uncertain futures, such models are either extremely expensive computationally as in the case of pixel-level autoregressive models, or do not directly optimize the likelihood of the data. To our knowledge, our work is the first to propose multi-frame video prediction with normalizing flows, which allows for direct optimization of the data likelihood, and produces high-quality stochastic predictions. We describe an approach for modeling the latent space dynamics, and demonstrate that flow-based generative models offer a viable and competitive approach to generative modeling of video. Exponential progress in the capabilities of computational hardware, paired with a relentless effort towards greater insights and better methods, has pushed the field of machine learning from relative obscurity into the mainstream. Progress in the field has translated to improvements in various capabilities, such as classification of images , machine translation and super-human game-playing agents , among others. However, the application of machine learning technology has been largely constrained to situations where large amounts of supervision is available, such as in image classification or machine translation, or where highly accurate simulations of the environment are available to the learning agent, such as in game-playing agents. An appealing alternative to supervised learning is to utilize large unlabeled datasets, combined with predictive generative models. In order for a complex generative model to be able to effectively predict future events, it must build up an internal representation of the world. For example, a predictive generative model that can predict future frames in a video would need to model complex real-world phenomena, such as physical interactions. This provides an appealing mechanism for building models that have a rich understanding of the physical world, without any labeled examples. Videos of real-world interactions are plentiful and readily available, and a large generative model can be trained on large unlabeled datasets containing many video sequences, thereby learning about a wide range of real-world phenoma. Such a model could be useful for learning representations for further downstream tasks , or could even be used directly in applications where predicting the future enables effective decision making and control, such as robotics . A central challenge in video prediction is that the future is highly uncertain: a short sequence of observations of the present can imply many possible futures. Although a number of recent works have studied probabilistic models that can represent uncertain futures, such models are either extremely expensive computationally (as in the case of pixel-level autoregressive models), or do not directly optimize the likelihood of the data. In this paper, we study the problem of stochastic prediction, focusing specifically on the case of conditional video prediction: synthesizing raw RGB video frames conditioned on a short context of past observations (; ; ; ;). Specifically, we propose a new class of video prediction models that can provide exact likelihoods, generate diverse stochastic futures, and accurately synthesize realistic and high-quality video frames. The main idea behind our approach is to extend flow-based generative models (; into the setting of conditional video prediction. To our knowledge, flow-based models have been applied only to generation of non-temporal data, such as images , and to audio sequences . Conditional generation of videos presents its own unique challenges: the high dimensionality of video sequences makes them difficult to model as individual datapoints. Instead, we learn a latent dynamical system model that predicts future values of the flow model's latent state. This induces Markovian dynamics on the latent state of the system, replacing the standard unconditional prior distribution. We further describe a practically applicable architecture for flow-based video prediction models, inspired by the Glow model for image generation , which we call VideoFlow. Our empirical show that VideoFlow achieves that are competitive with the state-ofthe-art in stochastic video prediction on the action-free BAIR dataset, with quantitative that rival the best VAE-based models. VideoFlow also produces excellent qualitative , and avoids many of the common artifacts of models that use pixel-level mean-squared-error for training (e.g., blurry predictions), without the challenges associated with training adversarial models. Compared to models based on pixel-level autoregressive prediction, VideoFlow achieves substantially faster test-time image synthesis 1, making it much more practical for applications that require real-time prediction, such as robotic control. Finally, since VideoFlow directly optimizes the likelihood of training videos, without relying on a variational lower bound, we can evaluate its performance directly in terms of likelihood values. Early work on prediction of future video frames focused on deterministic predictive models (; ; ; ;). Much of this research on deterministic models focused on architectural changes, such as predicting high-level structure Villegas et al. (2017b), incorporating pixel transformations (; ;) and predictive coding architectures , as well as different generation objectives (; ;) and disentangling representations (a;). With models that can successfully model many deterministic environments, the next key challenge is to address stochastic environments by building models that can effectively reason over uncertain futures. Real-world videos are always somewhat stochastic, either due to events that are inherently random, or events that are caused by unobserved or partially observable factors, such as off-screen events, humans and animals with unknown intentions, and objects with unknown physical properties. In such cases, since deterministic models can only generate one future, these models either disregard potential futures or produce blurry predictions that are the superposition or averages of possible futures. A variety of methods have sought to overcome this challenge by incorporating stochasticity, via three types of approaches: models based on variational auto-encoders (VAEs) , generative adversarial networks , and autoregressive models (; ; van den b; c;). Among these models, techniques based on variational autoencoders which optimize an evidence lower bound on the log-likelihood have been explored most widely;;; ). To our knowledge, the only prior class of video prediction models that directly maximize the log-likelihood of the data are auto-regressive models (; ; van den b; c;), that generate the video one pixel at a time. However, synthesis with such models is typically inherently sequential, making synthesis substantially inefficient on modern parallel hardware. Prior work has aimed to speed up training and synthesis with such auto-regressive models . However, show that the predictions from these models are sharp but noisy and that the proposed VAE model produces substantially better predictions, especially for longer horizons. In contrast to autoregressive models, we find that our proposed method exhibits faster sampling, while still directly optimizing the log-likelihood and producing high-quality long-term predictions.... Figure 1: Left: Multi-scale prior The flow model uses a multi-scale architecture using several levels of stochastic variables. Right: Autoregressive latent-dynamic prior The input at each timestep xt is encoded into multiple levels of stochastic variables (z t). We model those levels through a sequential process ). Flow-based generative models (; have a unique set of advantages: exact latentvariable inference, exact log-likelihood evaluation, and parallel sampling. In flow-based generative models (;, we infer the latent variable z corresponding to a datapoint x, by transforming x through a composition of invertible We assume a tractable prior p θ (z) over latent variable z, for eg. a Logistic or a Gaussian distribution. By constraining the transformations to be invertible, we can compute the log-likelihood of x exactly using the change of variables rule. Formally, where is transformed to h i by f i. We learn the parameters of f 1... f K by maximizing the log-likelihood, i.e Equation, over a training set. Given g = f −1, we can now generate a samplex from the data distribution, by sampling z ∼ p θ (z) and computingx = g(z). We propose a generative flow for video, using the standard multi-scale flow architecture in as a building block. In our model, we break up the latent space z into separate latent variables per timestep: z = {z t} T t=1. The latent variable z t at timestep t is an invertible transformation of a corresponding frame of video: x t = g θ (z t). Furthermore, like in , we use a multi-scale architecture for g θ (z t) (Fig. 1): the latent variable z t is composed of a stack of multiple levels: where each level l encodes information about frame x t at a particular scale: We first briefly describe the invertible transformations used in the multi-scale architecture to infer {z = f θ (x t) and refer to for more details. For convenience, we omit the subscript t in this subsection. We choose invertible transformations whose Jacobian determinant in Equation 1 is simple to compute, that is a triangular matrix, diagonal matrix or a permutation matrix as explored in prior work . For permutation matrices, the Jacobian determinant is one and for triangular and diagonal Jacobian matrices, the determinant is simply the product of diagonal terms. • Actnorm: We apply a learnable per-channel scale and shift with data-dependent initialization. • Coupling: We split the input y equally across channels to obtain y 1 and y 2. We compute z 2 = f (y 1) * y 2 + g(y 1) where f and g are deep networks. We concat y 1 and z 2 across channels. • SoftPermute: We apply a 1x1 convolution that preserves the number of channels. • Squeeze: We reshape the input from H × W × C to H/2 × W/2 × 4C which allows the flow to operate on a larger receptive field. We infer the latent variable z (l) at level l using: where N is the number of steps of flow. In Equation, via Split, we split the output of Flow equally across channels into h (>l), the input to Flow (l+1) and z (l), the latent variable at level l. We, thus enable the flows at higher levels to operate on a lower number of dimensions and larger scales. When l = 1, h (>l−1) is just the input frame x and for l = L we omit the Split operation. Finally, our multi-scale architecture f θ (x t) is a composition of the flows at multiple levels from l = 1... L from which we obtain our latent variables i.e {z We use the multi-scale architecture described above to infer the set of corresponding latent variables for each individual frame of the video: Figure 1 for an illustration. As in Equation, we need to choose a form of latent prior p θ (z). We use the following autoregressive factorization for the latent prior: where z <t denotes the latent variables of frames prior to the t-th timestep: {z 1, ..., z t−1}. We specify the conditional prior p θ (z t |z <t) as having the following factorization: where <t is the set of latent variables at previous timesteps and at the same level l, while z (>l) t is the set of latent variables at the same timestep and at higher levels. See Figure 1 for a graphical illustration of the dependencies. ) be a conditionally factorized Gaussian density: where where N N θ is a deep 3-D residual network augmented with dilations and gated activation units and modified to predict the mean and log-scale. We describe the architecture and our ablations of the architecture in Section B and C of the appendix. In summary, the log-likelhood objective of Equation has two parts. The invertible multi-scale architecture contributes | via the sum of the log Jacobian determinants of the invertible transformations mapping the video {x t} T t=1 to {z t} T t=1; the latent dynamics model contributes log p θ (z), i.e Equation. We jointly learn the parameters of the multi-scale architecture and latent dynamics model by maximizing this objective. Note that in our architecture we have chosen to let the prior p θ (z), as described in eq., model temporal dependencies in the data, while constraining the flow g θ to act on separate frames of video. Fooling rate SAVP-VAE 16.4 % VideoFlow 31.8 % SV2P 17.5 % Table 1: We compare the realism of the generated trajectories using a real-vs-fake 2AFC Amazon Mechanical Turk with SAVP-VAE and SV2P. Figure 2: We condition the VideoFlow model with the frame at t = 1 and display generated trajectories at t = 2 and t = 3 for three different shapes. We have experimented with using 3-D convolutional flows, but found this to be computationally overly expensive compared to an autoregressive prior; in terms of both number of operations and number of parameters. Further, due to memory limits, we found it only feasible to perform SGD with a small number of sequential frames per gradient step. In case of 3-D convolutions, this would make the temporal dimension considerably smaller during training than during synthesis; this would change the model's input distribution between training and synthesis, which often leads to various temporal artifacts. Using 2-D convolutions in our flow f θ with autoregressive priors, allows us to synthesize arbitrarily long sequences without introducing such artifacts. All our generated videos and qualitative can be viewed at this website. In the generated videos, a border of blue represents the conditioning frame, while a border of red represents the generated frames. We use VideoFlow to model the Stochastic Movement Dataset used in. The first frame of every video consists of a shape placed near the center of a 64x64x3 resolution gray with its type, size and color randomly sampled. The shape then randomly moves in one of eight directions with constant speed. show that conditioned on the first frame, a deterministic model averages out all eight possible directions in pixel space. Since the shape moves with a uniform speed, we should be able to model the position of the shape at the (t + 1) th step using only the position of the shape at the t th step. Using this insight, we extract random temporal patches of 2 frames from each video of 3 frames. We then use VideoFlow to maximize the loglikelihood of the second frame given the first, i.e the model looks back at just one frame. We observe that the bits-per-pixel on the holdout set reduces to a very low 0.04 bits-per-pixel for this model. On generating videos conditioned on the first frame, we observe that the model consistently predicts the future trajectory of the shape to be one of the eight random directions. We compare our model with two state-of-the-art stochastic video generation models SV2P and ) using their Tensor2Tensor implementation . We assess the quality of the generated videos using a real vs fake Amazon Mechanical Turk test. In the test, we inform the rater that a "real" trajectory is one in which the shape is consistent in color and congruent throughout the video. We show that VideoFlow outperforms the baselines in terms of fooling rate in Table 1 consistently generating plausible "real" trajectories at a greater rate. We use the action-free version of the BAIR robot pushing dataset that contain videos of a Sawyer robotic arm with resolution 64x64. In the absence of actions, the task of video generation is completely unsupervised with multiple plausible trajectories due to the partial observability of the environment and stochasticity of the robot actions. We train the baseline models, SAVP-VAE, SV2P and SVG-LP to generate 10 target frames, conditioned on 3 input frames. We extract random temporal patches of 4 frames, and train VideoFlow to maximize the log-likelihood of Bits-per-pixel VideoFlow 1.87 SAVP-VAE ≤ 6.73 SV2P ≤ 6.78 Table 2: Left: We report the average bits-per-pixel across 10 target frames with 3 conditioning frames for the BAIR action-free dataset. Figure 3: We measure realism using a 2AFC test and diversity using mean pairwise cosine distance between generated samples in VGG perceptual space. the 4th frame given a context of 3 past frames. We, thus ensure that all models have seen a total of 13 frames during training. We estimated the variational bound of the bits-per-pixel on the test set, via importance sampling, from the posteriors for the SAVP-VAE and SV2P models. We find that VideoFlow outperforms these models on bits-per-pixel and report these values in Table 2. We attribute the high values of bits-per-pixel of the baselines to their optimization objective. They do not optimize the variational bound on the log-likelihood directly due to the presence of a β = 1 term in their objective and scheduled sampling . Figure 4: For a given set of conditioning frames on the BAIR action-free we sample 100 videos from each of the stochastic video generation models. We choose the video closest to the ground-truth on the basis of PSNR, SSIM and VGG perceptual metrics and report the best possible value for each of these metrics. All the models were trained using ten target frames but are tested to generate 27 frames. For all the reported metrics, higher is better. Accuracy of the best sample: The BAIR robot-pushing dataset is highly stochastic and the number of plausible futures are high. Each generated video can be super realistic, can represent a plausible future in theory but can be far from the single ground truth video perceptually. To partially overcome this, we follow the metrics proposed in prior work; ) to evaluate our model. For a given set of conditioning frames in the BAIR action-free test-set, we generate 100 videos from each of the stochastic models. We then compute the closest of these generated videos to the ground truth according to three different metrics, PSNR (Peak Signal to Noise Ratio), SSIM (Structural Similarity) and cosine similarity using features obtained from a pretrained VGG network and report our findings in Figure 4. This metric helps us understand if the true future lies in the set of all plausible futures according to the video model. In prior work, effectively tune the pixel-level variance as a hyperparameter and sample from a deterministic decoder. They obtain training stabiltiy and improve sample quality by removing pixel-level noise using this procedure. We can remove pixel-level noise in our VideoFlow model ing in higher quality videos at the cost of diversity by sampling videos at a lower temperature, analogous to the procedure in . For a network trained with additive coupling layers, we can sample the t th frame x t from P (x t |x <t) with a temperature T simply by scaling the standard deviation of the latent gaussian distribution P (z t |z <t) by a factor of T. We report with both a temperature of 1.0 and the optimal temperature tuned on the validation set using VGG similarity metrics in Figure 4. Additionally, we also applied low-temperature sampling to the latent gaussian priors of SV2P and SAVP-VAE and empirically found it to hurt performance. We report these in Figure 10 For SAVP-VAE, we notice that the hyperparameters that perform the best on these metrics are the ones that have disappearing arms. For completeness, we report these numbers as well as the numbers for the best performing SAVP models that do not have disappearing arms. Our model with optimal temperature performs better or as well as the SAVP-VAE and SVG-LP models on the VGG-based similarity metrics, which correlate well with human perception and SSIM. Our model with temperature T = 1.0 is also competent with state-of-the-art video generation models on these metrics. PSNR is explicitly a pixel-level metric, which the VAE models incorporate as part of its optimization objective. VideoFlow on the other-hand models the conditional probability of the joint distribution of frames, hence as expected it underperforms on PSNR. Diversity and quality in generated samples: For each set of conditioning frames in the test set, we generate 10 videos and compute the mean distance in VGG perceptual space across these 45 different pairs. We average this across the test-set for T = 1.0 and T = 0.6 and report these numbers in Figure 3. We also assess the quality of the generated videos at T = 1.0 and T = 0.6, using a real vs fake Amazon Mechanical Turk test and report fooling rates. We observe that VideoFlow outperforms diversity values reported in prior work while being competitive in the realism axis. We also find that VideoFlow at T = 0.6 has the highest fooling rate while being competent with state-of-the-art VAE models in diversity. On inspection of the generated videos, we find that at lower temperatures, the arm exhibits less random behaviour with the objects remaining static and clear achieving higher realism scores. At higher temperatures, the motion of arm is much more stochastic, achieving high diversity scores with the objects becoming much noisier leading to a drop in realism. Figure 6: Left: We display interpolations between a) a small blue rectangle and a large yellow rectangle b) a small blue circle and a large yellow circle. Right: We display interpolations between the first input frame and the last target frame of two test videos in the BAIR robot pushing dataset. BAIR robot pushing dataset: We encode the first input frame and the last target frame into the latent space using our trained VideoFlow encoder and perform interpolations. We find that the motion of the arm is interpolated in a temporally cohesive fashion between the initial and final position. Further, we use the multi-level latent representation to interpolate representations at a particular level while keeping the representations at other levels fixed. We find that the bottom level interpolates the motion of objects which are at a smaller scale while the top level interpolates the arm motion. We encode two different shapes with their type fixed but a different size and color into the latent space. We observe that the size of the shape gets smoothly interpolated. During training, we sample the colors of the shapes from a uniform discrete distribution which is reflected in our experiments. We observe that all the colors in the interpolated space lie in the set of colors in the training set. Figure 7: Left: We generate 100 frames into the future with a temperature of 0.5. The top and bottom row correspond to generated videos in the absence and presence of occlusions respectively. Right: We use VideoFlow to detect the plausibility of a temporally inconsistent frame to occur in the immediate future. We generate 100 frames into the future using our model trained on 13 frames with a temperature of 0.5 and display our in Figure 7. On the top, even 100 frames into the future, the generated frames remain in the image manifold maintaining temporal consistency. In the presence of occlusions, the arm remains super-sharp but the objects become noisier and blurrier. Our VideoFlow model has a bijection between the z t and x t meaning that the latent state z t cannot store information other than that present in the frame x t. This, in combination with the Markovian assumption in our latent dynamics means that the model can forget objects if they have been occluded for a few frames. In future work, we would address this by incorporating longer memory in our VideoFlow model; for example by parameterizing N N θ as a recurrent neural network in our autoregressive prior (eq. 8) or using more memory-efficient backpropagation algorithms for invertible neural networks. We use our trained VideoFlow model, conditioned on 3 frames as explained in Section 5.2, to detect the plausibility of a temporally inconsistent frame to occur in the immediate future. We condition the model on the first three frames of a test-set video X <4 to obtain a distribution P (X 4 |X <4) over its 4th frame X 4. We then compute the likelihood of the t th frame X t of the same video to occur as the 4th time-step using this distribution. i.e, P(X 4 = X t |X <4) for t = 4... 13. We average the corresponding bits-per-pixel values across the test set and report our findings in Figure 7. We find that our model assigns a monotonically decreasing log-likelihood to frames that are more far out in the future and hence less likely to occur in the 4th time-step. We describe a practically applicable architecture for flow-based video prediction models, inspired by the Glow model for image generation , which we call VideoFlow. We introduce a latent dynamical system model that predicts future values of the flow model's latent state replacing the standard unconditional prior distribution. Our empirical show that VideoFlow achieves that are competitive with the state-of-the-art VAE models in stochastic video prediction. Finally, our model optimizes log-likelihood directly making it easy to evaluate while achieving faster synthesis compared to pixel-level autoregressive video models, making our model suitable for practical purposes. In future work, we plan to incorporate memory in VideoFlow to model arbitrary long-range dependencies and apply the model to challenging downstream tasks. be our dataset of i.i.d. observations of a random variable x with an unknown true distribution p * (x). Our data consist of 8-bit videos, with each dimension rescaled to the domain . We add a small amount of uniform noise to the data, u ∼ U(0, 1/256.), matching its discretization level . Let q(x) be the ing empirical distribution corresponding to this scaling and addition of noise. Note that additive noise is required to prevent q(x) from having infinite densities at the datapoints, which can in ill-behaved optimization of the log-likelihood; it also allows us to recast maximization of the log-likelihood as minimization of a KL divergence. We repeat our evaluations described in Figure 4 applying low temperature to the latent gaussian priors of SV2P and SAVP-VAE. We empirically find that decreasing temperature from 1.0 to 0.0 monotonically decreases the performance of the VAE models. Our insight is that the VideoFlow model gains by low-temperature sampling due to the following reason. At lower T, we obtain a tradeoff between a performance gain by noise removal from the and a performance hit due to reduced stochasticity of the robot arm. On the other hand, the VAE models have a clear but slightly blurry throughout from T = 1.0 to T = 0.0. Reducing T in this case, solely reduces the stochasticity of the arm motion thus hurting performance. We show correlation between training progression (measured in bits per pixel) and quality of the generated videos in Figure 11. We display the videos generated by conditioning on frames from the test set for three different values of bits-per-pixel on the test-set. As we approach lower bits-per-pixel, our VideoFlow model learns to model the structure of the arm with high quality as well as its motion ing in high quality video. To report bits-per-pixel we use the following set of hyperparameters. We use a learning rate schedule of linear warmup for the first 10000 steps and apply a linear-decay schedule for the last 150000 steps. We train all our baseline models for 300K steps using the Adam optimizer. Our models were tuned using the maximum VGG cosine similarity metric with the ground-truth across 100 decodes. We use three values of latent loss multiplier 1e-3, 1e-4 and 1e-5. For the SAVP-VAE model, we additionally apply linear decay on the learning rate for the last 100K steps. SAVP-GAN: We tune the gan loss multiplier and the learning rate on a logscale from 1e-2 to 1e-4 and 1e-3 to 1e-5 respectively. Figure 12: We compare P(X4 = Xt|X<4) and VGG cosine similarity between X4 and Xt for t = 4... 13 We plot correlation between cosine similarity using a pretrained VGG network and bits-per-pixel using our trained VideoFlow model. We compare P(X 4 = X t |X <4) as done in Section 5.5 and the VGG cosine similarity between X 4 and X t for t = 4... 13. We report our for every video in the test set in Figure 13. We notice a weak correlation between VGG perceptual metrics and bits-per-pixel with a correlation factor of −0.51. I VIDEOFLOW: LOW PARAMETER REGIME We repeated our evaluations described in Figure 4, with a smaller version of our VideoFlow model with 4x parameter reduction. Our model remains competetive with SVG-LP on the VGG perceptual metrics. Figure 13: We repeat our evaluations described in Figure 4 with a smaller version of our VideoFlow model.

represent the short summary for finding the corresponding scientific article (long).

We demonstrate that flow-based generative models offer a viable and competitive approach to generative modeling of video.

Represent the TLDR for finding the corresponding scientific article:

We develop VAEs where the encoder takes a model parameter vector as input, so we can do rapid inference for many models


Question: Represent the scientific article to find a short TLDR

Question: Understanding the behavior of stochastic gradient descent (SGD) in the context of deep neural networks has raised lots of concerns recently. Along this line, we theoretically study a general form of gradient based optimization dynamics with unbiased noise, which unifies SGD and standard Langevin dynamics. Through investigating this general optimization dynamics, we analyze the behavior of SGD on escaping from minima and its regularization effects. A novel indicator is derived to characterize the efficiency of escaping from minima through measuring the alignment of noise covariance and the curvature of loss function. Based on this indicator, two conditions are established to show which type of noise structure is superior to isotropic noise in term of escaping efficiency. We further show that the anisotropic noise in SGD satisfies the two conditions, and thus helps to escape from sharp and poor minima effectively, towards more stable and flat minima that typically generalize well. We verify our understanding through comparing this anisotropic diffusion with full gradient descent plus isotropic diffusion (i.e. Langevin dynamics) and other types of position-dependent noise. As a successful learning algorithm, stochastic gradient descent (SGD) was originally adopted for dealing with the computational bottleneck of training neural networks with large-scale datasets BID0. Its empirical efficiency and effectiveness have attracted lots of attention. And thus, SGD and its variants have become standard workhorse for learning deep models. Besides the aspect of empirical efficiency, recently, researchers started to analyze the optimization behaviors of SGD and its impacts on generalization. The optimization properties of SGD have been studied from various perspectives. The convergence behaviors of SGD for simple one hidden layer neural networks were investigated in BID13 BID1. In non-convex settings, the characterization of how SGD escapes from stationary points, including saddle points and local minima, was analyzed in BID3 BID10 BID8.On the other hand, in the context of deep learning, researchers realized that the noise introduced by SGD impacts the generalization, thanks to the research on the phenomenon that training with a large batch could cause a significant drop of test accuracy BID11. Particularly, several works attempted to investigate how the magnitude of the noise influences the generalization during the process of SGD optimization, including the batch size and learning rate BID7 BID5 BID2 BID9. Another line of research interpreted SGD from a Bayesian perspective. In BID14 BID2, SGD was interpreted as performing variational inference, where certain entropic regularization involves to prevent overfitting. And the work BID21 tried to provide an understanding based on model evidence. These explanations are compatible with the flat/sharp minima argument BID6 BID11, since Bayesian inference tends to targeting the region with large probability mass, corresponding to the flat minima. However, when analyzing the optimization behavior and regularization effects of SGD, most of existing works only assume the noise covariance of SGD is constant or upper bounded by some constant, and what role the noise structure of stochastic gradient plays in optimization and generalization was rarely discussed in literature. In this work, we theoretically study a general form of gradient-based optimization dynamics with unbiased noise, which unifies SGD and standard Langevin dynamics. By investigating this general dynamics, we analyze how the noise structure of SGD influences the escaping behavior from minima and its regularization effects. Several novel theoretical and empirical justifications are made.1. We derive a key indicator to characterize the efficiency of escaping from minima through measuring the alignment of noise covariance and the curvature of loss function. Based on this indicator, two conditions are established to show which type of noise structure is superior to isotropic noise in term of escaping efficiency;2. We further justify that SGD in the context of deep neural networks satisfies these two conditions, and thus provide a plausible explanation why SGD can escape from sharp minima more efficiently, converging to flat minima with a higher probability. Moreover, these flat minima typically generalize well according to various works BID6 BID11 BID16 BID22. We also show that Langevin dynamics with well tuned isotropic noise cannot beat SGD, which further confirms the importance of noise structure of SGD; 3. A large number of experiments are designed systematically to justify our understanding on the behavior of the anisotropic diffusion of SGD. We compare SGD with full gradient descent with different types of diffusion noise, including isotropic and positiondependent/independent noise. All these comparisons demonstrate the effectiveness of anisotropic diffusion for good generalization in training deep networks. The remaining of the paper is organized as follows. In Section 2, we introduce the of SGD and a general form of optimization dynamics of interest. We then theoretically study the behaviors of escaping from minima in Ornstein-Uhlenbeck process in Section 3, and establish two conditions for characterizing the noise structure that affects the escaping efficiency. In Section 4, we show that the noise of SGD in the context of deep learning meets the two conditions, and thus explains its superior efficiency of escaping from sharp minima over other dynamics with isotropic noise. Various experiments are conducted for verifying our understanding in Section 5, and we conclude the paper in Section 6. In general, supervised learning usually involves an optimization process of minimizing an empirical loss over training data, DISPLAYFORM0 denotes the training set with N i.i.d. samples, the prediction function f is often parameterized by θ ∈ R D, such as deep neural networks. And (·, ·) is the loss function, such as mean squared error and cross entropy, typically corresponding to certain negative log likelihood. Due to the over parameterization and non-convexity of the loss function in deep networks, there exist multiple global minima, exhibiting diverse generalization performance. We call those solutions generalizing well good solutions or minima, and vice versa. Gradient descent and its stochastic variants A typical approach to minimize the loss function is gradient descent (GD), the dynamics of which in each iteration t is, θ t+1 = θ t − η t g 0 (θ t), where g 0 (θ t) = ∇ θ L(θ t) denotes the full gradient and η t denotes the learning rate. In non-convex optimization, a more useful kind of gradient based optimizers act like GD with an unbiased noise, including gradient Langevin dynamics (GLD), θ t+1 = θ t − η t g 0 (θ t) + σ t t, t ∼ N (0, I), and stochastic gradient descent (SGD), during each iteration t of which, a minibatch of training samples with size m are randomly selected, with index set B t ⊂ {1, 2, . . ., N}, and a stochastic gradient is evaluated based on the chosen minibatch,g(θ t) = i∈Bt ∇ θ (f (x i ; θ t), y i )/m, which is an unbiased estimator of the full gradient g 0 (θ t). Then, the parameters are updated with some learning rate η t as θ t+1 = θ t − η tg (θ t). Denote g(θ) = ∇ θ ((f (x; θ), y), the gradient for loss with a single data point (x, y), and assume that the size of minibatch is large enough for the central limit theorem to hold, and thusg(θ t) follows a Gaussian distribution BID14, DISPLAYFORM1 Note that the covariance matrix Σ depends on the model architecture, dataset and the current parameter θ t. Now we can rewrite the update of SGD as, DISPLAYFORM2 Inspired by GLD and SGD, we may consider a general kind of optimization dynamics, namely, gradient descent with unbiased noise, DISPLAYFORM3 For small enough constant learning rate η t = η, the above iteration in Eq. can be treated as the numerical discretization of the following stochastic differential equation BID9 BID2, DISPLAYFORM4 Considering ησ 2 t Σ t as the coefficient of noise term, existing works BID7 BID9 studied the influence of noise magnitude of SGD on generalization, i.e. ησ 2 t = η/m. In this work, we focus on studying the benefits of anisotropic structure of Σ t in SGD helping escape from minima by bridging the covariance matrix with the Hessian of the loss surface, and its implicit regularization effects on generalization, especially in deep learning context. For the purpose of eliminating the influence of the noise magnitude, we constrain it to be a constant when studying different structures of noise covariance. The noise magnitude could be evaluated as the expectation of the squared norm of the noise vector, DISPLAYFORM5 Thus, we introduce the following constraint, DISPLAYFORM6 From the statistical physics point of view, Tr(ησ 2 t Σ t) characterizes the kinetic energy (Gardiner), thus it is natural to force the energy to be unchanging, otherwise it is trivial that the higher the energy is, the less stable the system is. For simplicity, we absorb ησ 2 t into Σ t, denoting ησ 2 t Σ t as Σ t. If not pointed out, the subscript t of matrix Σ t is omitted to emphasize that we are fixing t and discussing the varying structure of Σ. For a general loss function L(θ) = E X X (θ) (the expectation could be either population or empirical), where X denotes data example and θ denoted parameters to be optimized, under suitable smoothness assumptions, the SDE associated with the gradient variant optimizer as shown in Eq. can be written as follows BID9 BID2 BID8, with little abuse of notation, DISPLAYFORM0 Let L 0 = L(θ 0) be one of the minimal values of L(θ), then for a fixed t small enough (such that DISPLAYFORM1 It is natural to measure the escaping efficiency using E[L t − L 0] since it characterizes the increase of the potential, i.e., the increase of the loss L. And also note that L t − L 0 ≥ 0, for any δ > 0, the escaping probability DISPLAYFORM2 where H t denotes the Hessian of L(θ t) at θ t.We provide the proof in Appendix, and the same for the other propositions. The escaping efficiency for general processes is hard to analyze due to the intractableness of the integral in Eq.. However, we may consider the second-order approximation locally near the minima θ 0, where DISPLAYFORM3. Without losing generality, we suppose θ 0 = 0. Further, suppose that H is a positive definite matrix and the diffusion covariance Σ t = Σ is constant for t. Then the SDE FORMULA7 becomes an Ornstein-Uhlenbeck process, DISPLAYFORM4 Proposition 2 (Escaping efficiency of Ornstein-Uhlenbeck process). For Ornstein-Uhlenbeck process, with t small enough, the escaping efficiency from minimum θ 0 = 0 is, DISPLAYFORM5 Inspired by Proposition 1 and Proposition 2, we propose Tr (HΣ) as an empirical indicator measuring the efficiency for a stochastic process escaping from minima. Now we turn to analysis which kind of noise covariance structure Σ will benefit escaping sharp minima, under the constraint Eq..Firstly, for the isotropic loss surface, i.e., DISPLAYFORM6 Tr Σ, which is invariant under the constraint that Tr Σ is constant (Eq. FORMULA6). Thus it is only nontrivial to study the impact of noise structure when the Hessian of loss surface is anisotropic. Secondly, H and Σ being semi-positive definite, to achieve the maximum of Tr(HΣ) under constraint, Σ should be DISPLAYFORM7, where λ 1, u 1 are the maximal eigenvalue and corresponding unit eigenvector of H. Note that the rank-1 matrix Σ * is highly anisotropic. More generally, the following Proposition 3 characterizes one kind of anisotropic noise significantly outperforming isotropic noise in order of number of parameters D, given H is ill-conditioned. Proposition 3 (The benefits of anisotropic noise). With semi-positive definite H and Σ, assume DISPLAYFORM8 DISPLAYFORM9 Σ is "aligned" with H. Let u i be the corresponding unit eigenvector of eigenvalue λ i, for some projection coefficient a > 0, DISPLAYFORM10 then we have the benefit of the anisotropic noise over the isotropic one in term of escaping efficiency, which can be characterized by the follow ratio, DISPLAYFORM11 whereΣ = Tr Σ D I denotes the covariance of isotropic noise, to meet the constraint Eq..To give some geometric intuitions on the left hand side of Eq. FORMULA2, let the maximal eigenvalue and its corresponding unit eigenvector of Σ be γ 1, v 1, then the right hand side has a lower bound as u DISPLAYFORM12 Thus if the maximal eigenvalues of H and Σ are aligned in proportion, γ 1 / Tr Σ ≥ a 1 λ 1 / Tr H, and the angle of their corresponding unit eigenvectors is close to zero, u 1, v 1 ≥ a 2, the second condition Eq. in Proposition 3 holds for a = a 1 a 2.Typically, in the scenario of modern deep neural networks, due to the over-parameterization, Hessian and the gradient covariance are usually ill-conditioned and anistropic near minima, as shown by BID19 and BID2. Thus the first condition in Eq. usually holds for deep neural networks, and we further justify it by experiments in Section 5.3. Therefore, in the following section, we turn to focus on how the gradient covariance, i.e. the covariance of SGD noise meets the second condition of Proposition 3 in the context of deep neural networks. In this section, we mainly investigate the anisotropic structure of gradient covariance in SGD, and explore its connection with the Hessian of loss surface. Around the true parameter According to the classic statistical theory (, Chap. 8), for population loss L(θ) = E X (θ), with being the negative log likelihood, when evaluating at the true parameter θ *, there is the exact equivalence between the Hessian H of the population loss and Fisher information matrix F, DISPLAYFORM0 In practice, with the assumptions that the sample size N is large enough (i.e. indicating asymptotic behavior) and suitable smoothness conditions, when the current parameter θ t is not far from the ground truth, Fisher is close to Hessian. Thus we can obtain the following approximate equality between gradient covariance and Hessian, DISPLAYFORM1 The first approximation is due to the dominance of noise over the mean of gradient in the later stage of SGD optimization, which has been shown in BID20. A similar experiment as BID20 has been conducted to demonstrate this observation, which is left in Appendix due to the limit of space. In the following, we theoretically characterize the closeness between Σ and H in the context of one hidden layer neural networks; and show that the gradient covariance introduced by SGD indeed has more benefits than isotropic one in term of escaping from minima, provided some assumptions. One hidden layer neural network with fixed output layer parameters For binary classification neural network with one hidden layer in classic setups (with softmax and cross-entropy loss), we have following to globally bound Fisher and Hessian with each other. Proposition 4 (The relationship between Fisher and Hessian in one hidden layer neural network). Consider the binary classification problem with data {(x i, y i)} i∈I, y ∈ {0, 1}, and typical (either population or empirical) loss as DISPLAYFORM2, where f denotes the output of neural network, and φ denotes the cross-entropy loss with softmax, DISPLAYFORM3 If: the neural network f is with one hidden layer and piece-wise linear activation. And the parameters of output layer are fixed during training; the optimization happens on a set U such that, f (x; θ) ∈ (−C, C), ∀θ ∈ U, ∀x, i.e., the output of the classifier is bounded during optimization. Then, we have the following relationship between (either population or empirical) Fisher F and Hessian H almost everywhere: DISPLAYFORM4 A B means that (B − A) is semi-positive definite. There are a few remarks on Proposition 4. Firstly, as shown in BID1, the considered neural networks in Proposition 4 are non-convex and have multiple minima, and thus it is still nontrivial to consider the escaping from minima. Secondly, the Proposition 4 holds in both population and empirical sense, since the proof does not distinguish the two circumstances. Thirdly, the bound between F and H holds "globally" in the set U where the output f is bounded, rather than merely around the true global minima as discussed previously. By Proposition 4, the following relationship between gradient covariance and Hessian could be derived. Proposition 5 (The relationship between gradient covariance and Hessian in one hidden layer neural network). Assume the conditions in Proposition 4 hold, then for some small δ > 0 and for θ close enough to minima θ * (local or global), DISPLAYFORM5 holds for any positive eigenvalue λ and its corresponding unit eigenvector u of Hessian H.As a direct corollary of Proposition 5, for such neural networks, the second condition Eq. in Proposition 3 holds in a very loose sense. Therefore, based on the discussion on population loss around the true parameters and one hidden layer neural network with fixed output layer parameters, given the ill-conditioning of H due to the over-parameterization of modern deep networks, according to Proposition 3, we can conclude the noise structure of SGD helps to escape from sharp minima much faster than the dynamics with isotropic noise, and converge to flatter solutions with a high probability. These flat minima typically generalize well BID6 BID11 BID16 BID22. Thus, we attribute such properties of SGD on its better generalization performance comparing to GD, GLD and other dynamics with isotropic noise BID7 BID5 BID11.In the following, we conduct a series of experiments systematically to verify our understanding on the behavior of escaping from minima and its regularization effects for different optimization dynamics. To better understanding the behavior of anisotropic noise different from isotropic ones, we introduce dynamics with different kinds of noise structure to empirical study with, as shown on TAB0. DISPLAYFORM0 σt is adjusted to make σt t share the same expected norm as that of SGD DISPLAYFORM1 The covariance diag(Σt) is the diagonal of the covariance of SGD noise. DISPLAYFORM2. γi, vi are the first k leading eigenvalues and corresponding eigenvalues of the covariance of SGD noise, respectively. (A low rank approximation of Σ sgd t) GLD Hessian t ∼ N 0,Ht H t is a low rank approximation of the Hessian matrix of loss L(θ) by its the first k leading eigenvalues and corresponding eigenvalues. GLD 1st eigven (H) t ∼ N 0, λ1u1u DISPLAYFORM3 λ1, u1 are the maximal eigenvalue and its corresponding unit eigenvector of the Hessian matrix of loss L(θt). We design a 2-D toy example L(w 1, w 2) with two basins, a small one and a large one, corresponding to a sharp and flat minima, and (−1, −1), respectively, both of which are global minima. Please refer to Appendix for the detailed constructions. We initialize the dynamics of interest with the sharp minimum (w 1, w 2) =, and run them to study their behaviors escaping from this sharp minimum. To explicitly control the noise magnitude, we only conduct experiments on GD, GLD const, GLD diag, GLD leading (with k = 2 = D in TAB0, or in other words, the exactly covariance of SGD noise), GLD Hessian (k = 2) and GLD 1st eigven(H). And we adjust σ t in each dynamics to force their noise to share the same expected squared norm as defined in Eq.. Figure 1(a) shows the trajectories of the dynamics escaping from the sharp minimum towards the flat one (−1, −1), while Figure 1(b) presents the success rate of escaping for each dynamic during 100 repeated experiments. As shown in Figure 1, GLD 1st eigvec(H) achieves the highest success rate, indicating the fastest escaping speed from the sharp minimum. The dynamics with anisotropic noise aligned with Hessian well, including GLD 1st eigvec(H), GLD Hessian and GLD leading, greatly outperform GD, GLD const with isotropic noise, and GLD diag with noise poorly aligned with Hessian. These experiments are consistent with our theoretical analysis on Ornstein-Uhlenbeck process shown Proposition 2 and 3, demonstrating the benefits of anisotropic noise for escaping from sharp minima. We empirically show that in one hidden layer neural network with fixed output layer parameters, the anisotropic noise induced by SGD indeed helps escape from sharp minima more efficiently than isotropic noise. Three networks are trained to binary classify 1, 000 linearly separable two-dimensional points. The number of hidden nodes for each network varies in {20, 200, 2000}. We plot the empirical indicator Tr (HΣ) in FIG3. We can easily observe that as the increase of the number of hidden nodes, the ratio is enlarged significantly, which is consistent with the Eq. described in Proposition 3. In this part, we conduct a series of experiments in real deep learning scenarios to demonstrate the behavior of SGD noise and its implicit regularization effects. We construct a noisy training set based on FashionMNIST dataset 1. Concretely, the training set consist of 1000 images with correct labels, and another 200 images with random labels. All the test data are with clean labels. A small LeNet-like network is utilized such that the spectrum decomposition over gradient covariance matrix and Hessian matrix are computationally feasible. The network consists of two convolutional layers and two fully-connected layers, with 11, 330 parameters in total. We firstly run the standard gradient decent for 3000 iterations to arrive at the parameters θ * GD near the global minima with near zero training loss and 100% training accuracy, which are typically sharp minima that generalize poorly BID16. And then all other compared methods are initialized with θ * GD and run for optimization with the same learning rate η t = 0.07 and same batch size m = 20 (if needed) for fair comparison 2.Verification of SGD noise satisfying the conditions in Proposition 3 To see whether the noise of SGD in real deep learning circumstance satisfies the two conditions in Proposition 3, we run SGD optimizer initialized from θ * GD, i.e. the sharp minima found by GD. FIG4 shows the first 400 eigenvalues of Hessian at θ * GD, from which we see that the 140th eigenvalue has already decayed to about 1% of the first eigenvalue. Note that Hessian H ∈ R D×D, D = 11330, thus H around θ * GD approximately meets the ill-conditioning requirement in Proposition 3. FIG4 shows the projection coefficient estimated byâ = u T 1 Σu1 Tr H λ1 Tr Σ along the trajectory of SGD. The plot indicates that the projection coefficient is in a descent scale comparing to D 2d−1, thus satisfying the second condition in Proposition 3. Therefore, Proposition 3 ensures that SGD would escape from minima θ * GD faster than GLD in order of O(D 2d−1), as shown in FIG4 (c). An interesting observation is that in the later stage of SGD optimization, Tr(HΣ) becomes significantly (10 7 times) smaller than in the beginning stage, implying that SGD has already converged to minima being almost impossible to escape from. This phenomenon demonstrates the reasonability to employ Tr(HΣ) as an empirical indicator for escaping efficiency. Behaviors of different dynamics escaping from minima and its generalization effects To compare the different dynamics on escaping behaviors and generalization performance, we run dynamics initialized from the sharp minima θ * GD found by GD. The settings for each compared method are as follows. The hyperparameter σ 2 for GLD const has already been tuned as optimal (σ = 0.001) by grid search. For GLD leading, we set k = 20 for comprising the computational cost and approximation accuracy. As for GLD Hessian, to reduce the expensive evaluation of such a huge Hessian in each iteration, we set k = 20 and update the Hessian every 10 iterations. We adjust σ t in GLD dynamic, GLD Hessian and GLD 1st eigvec(H) to guarantee that they share the same expected squred noise norm defined in Eq. as that of SGD. And we measure the expected sharpness of different minima as E ν∼N (0,δ 2 I) L(θ + ν) − L(θ), as defined in BID16, Eq. FORMULA7 ). The are shown in FIG5.As shown in FIG5, SGD, GLD 1st eigvec(H), GLD leading and GLD Hessian successfully escape from the sharp minima found by GD, while GLD, GLD dynamic and GLD diag are trapped in the minima. This demonstrates that the methods with anisotropic noise "aligned" with loss curvature can help to find flatter minima that generalize well. We also provide experiments on standard CIFAR-10 with VGG11 in Appendix. DISPLAYFORM0, and δ = 0.01, the expectation is computed by average on 1000 times sampling. We theoretically investigate a general optimization dynamics with unbiased noise, which unifies various existing optimization methods, including SGD. We provide some novel on the behaviors of escaping from minima and its regularization effects. A novel indicator is derived for characterizing the escaping efficiency. Based on this indicator, two conditions are constructed for showing what type of noise structure is superior to isotropic noise in term of escaping. We then analyze the noise structure of SGD in deep learning and find that it indeed satisfies the two conditions, thus explaining the widely know observation that SGD can escape from sharp minima efficiently toward flat minina that generalize well. Various experimental evidence supports our arguments on the behavior of SGD and its effects on generalization. Our study also shows that isotropic noise helps little for escaping from sharp minima, due to the highly anisotropic nature of landscape. This indicates that it is not sufficient to analyze SGD by treating it as an isotropic diffusion over landscape (; BID15 . A better understanding of this out-of-equilibrium behavior BID2) is on demand. Taking expectation with respect to the distribution of θ t, DISPLAYFORM0 for the expectation of Brownian motion is zero. Thus the solution of EY t is, DISPLAYFORM1 Proof. Without losing generality, we assume that L 0 = 0.For multivariate Ornstein-Uhlenbeck process, when θ 0 = 0 is an constant, θ t follows a multivariate Gaussian distribution (Øksendal, 2003).Consider change of variables θ → φ(θ, t) = e Ht θ t. Here, for symmetric matrix A, DISPLAYFORM0 where λ 1,..., λ n and U are the eigenvalues and eigenvector matrix of A. Note that with this notation, DISPLAYFORM1 Applying Ito's lemma, we have DISPLAYFORM2 which we can integrate form 0 to t to get DISPLAYFORM3 The expectation of θ t is zero. And by Ito's isometry (Øksendal, 2003), the covariance of θ t is, DISPLAYFORM4 DISPLAYFORM5 The proof is finished. Proof. Firstly compute the gradients and Hessian of φ, DISPLAYFORM0 And note the Gauss-Newton decomposition for functions with the form of L = φ • f, DISPLAYFORM1 Since the output layer parameters for f is fixed and the activation functions are piece-wise linear, f (x; θ) is a piece-wise linear function on its parameters θ. Therefore ∂ 2 f ∂θ 2 = 0, a.e., and H = E (x,y) DISPLAYFORM2 It is easy to check that e DISPLAYFORM3. Thus, DISPLAYFORM4 A.5 PROOF OF PROPOSITION 5Proof. For simplicity, we define g:= ∇, g 0:= ∇L = E∇.The gradient covariance and Fisher has the following relationship, DISPLAYFORM5 Hence, DISPLAYFORM6 Therefore, with the condition DISPLAYFORM7, we have DISPLAYFORM8 Tr F e −2δ, for δ small enough. On the other hand, Proposition 4 indicates that e −C F H e C F, which means, DISPLAYFORM9. Therefore, for λ, u being a positive eigenvalue and the corresponding unit eigenvector of H, we have DISPLAYFORM10 B ADDITIONAL EXPERIMENTS B.1 DOMINANCE OF NOISE OVER GRADIENT FIG6 shows the comparison of gradient mean and the expected norm of noise during training using SGD. The dataset and model are same as the experiments of FashionMNIST in main paper, or as in Section C.2. From FIG6, we see that in the later stage of SGD optimization, noise indeed dominates gradient. These experiments are implemented by TensorFlow 1.5.0. FIG7 shows the first 50 iterations of FashionMNIST experiments in main paper. We observe that SGD, GLD 1st eigvec(H), GLD Hessian and GLD leading successfully escape from the sharp minima found by GD, while GLD diag, GLD dynamic, GLD const and GD do not. These experiments are implemented by TensorFlow 1.5.0. Dataset Standard CIFAR-10 dataset without data augmentation. Model Standard VGG11 network without any regularizations including dropout, batch normalization, weight decay, etc. The total number of parameters of this network is 9, 750, 922.Training details Learning rates η t = 0.05 are fixed for all optimizers, which is tuned for the best generalization performance of GD. The batch size of SGD is m = 100. The noise std of GLD constant is σ = 10 −3, which is tuned to best. Due to computational limitation, we only conduct experiments on GD, GLD const, GLD dynamic, GLD diag and SGD.Estimation of Sharpness The sharpness are estimated by DISPLAYFORM0 with M = 100 and δ = 0.01.Experiments Similar experiments are conducted as in main paper for CIFAR-10 and VGG11, as shown in FIG8. The observations and consist with main paper. These experiments are implemented by PyTorch 0.3.0. Note that Σ is the inverse of the Hessian of the quadric form generalizeing the sharp minima. And the 3-dimensional plot of the loss surface is shown in FIG9.Hyperparameters All learning rates are equal to 0.005. All dynamics concerned are tuned to share the same expected square norm, 0.01. The number of iteration during one run is 500.These experiments are implemented by PyTorch 0.3.0. Dataset Our training set consists of 1200 examples randomly sampled from original FashionM-NIST training set, and we further specify 200 of them with randomly wrong labels. The test set is same as the original FashionMNIST test set. Model Network architecture: input ⇒ conv1 ⇒ max_pool ⇒ ReLU ⇒ conv2 ⇒ max_pool ⇒ ReLU ⇒ fc1 ⇒ ReLU ⇒ fc2 ⇒ output. Both two convolutional layers use 5 × 5 kernels with 10 channels and no padding. The number of hidden units between fully connected layers are 50. The total number of parameters of this network are 11, 330. • GD: Learning rate η = 0.1. We tuned the learning rate (in diffusion stage) in a wide range of {0.5, 0.2, 0.15, 0.1, 0.09, 0.08, . . ., 0.01} and no improvement on generalization.• GLD constant: Learning rate η = 0.07, noise std σ = 10 −3. We tuned the noise std in range of {10 −1, 10 −2, 10 −3, 10 −4, 10 −5} and no improvement on generalization.• GLD dynamic: Learning rate η = 0.07.• GLD diagnoal: Learning rate η = 0.07.• GLD leading: Learning rate η = 0.07, number of leading eigenvalues k = 20, batchsize m = 20. We first randomly divide the training set into 60 mini batches containing 20 examples, and then use those minibatches to estimate covariance matrix.• GLD Hessian: Learning rate η = 0.07, number of leading eigenvalues = 20, update frequence f = 10. Do to the limit of computational resources, we only update Hessian matrix every 10 iterations. But add Hessian generated noise every iteration. And to the same reason, we simplily set the coefficent of Hessian noise to Tr H/m Tr Σ, to avoid extensively tuning of hyperparameter. • GLD 1st eigvec(H): Learning rate η = 0.07, as for GLD Hessian, and we set the coefficient of noise to λ 1 /m Tr Σ, where λ 1 is the first eigenvalue of H. These experiments are implemented by TensorFlow 1.5.0.

Represent the TLDR for finding the corresponding scientific article

We provide theoretical and empirical analysis on the role of anisotropic noise introduced by stochastic gradient on escaping from minima.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Current model-based reinforcement learning approaches use the model simply as a learned black-box simulator to augment the data for policy optimization or value function learning. In this paper, we show how to make more effective use of the model by exploiting its differentiability. We construct a policy optimization algorithm that uses the pathwise derivative of the learned model and policy across future timesteps. Instabilities of learning across many timesteps are prevented by using a terminal value function, learning the policy in an actor-critic fashion. Furthermore, we present a derivation on the monotonic improvement of our objective in terms of the gradient error in the model and value function. We show that our approach (i) is consistently more sample efficient than existing state-of-the-art model-based algorithms, (ii) matches the asymptotic performance of model-free algorithms, and (iii) scales to long horizons, a regime where typically past model-based approaches have struggled. Model-based reinforcement learning (RL) offers the potential to be a general-purpose tool for learning complex policies while being sample efficient. When learning in real-world physical systems, data collection can be an arduous process. Contrary to model-free methods, model-based approaches are appealing due to their comparatively fast learning. By first learning the dynamics of the system in a supervised learning way, it can exploit off-policy data. Then, model-based methods use the model to derive controllers from it either parametric controllers (; ;) or non-parametric controllers . Current model-based methods learn with an order of magnitude less data than their model-free counterparts while achieving the same asymptotic convergence. Tools like ensembles, probabilistic models, planning over shorter horizons, and meta-learning have been used to achieved such performance (; ; . However, the model usage in all of these methods is the same: simple data augmentation. They use the learned model as a black-box simulator generating samples from it. In high-dimensional environments or environments that require longer planning, substantial sampling is needed to provide meaningful signal for the policy. Can we further exploit our learned models? In this work, we propose to estimate the policy gradient by backpropagating its gradient through the model using the pathwise derivative estimator. Since the learned model is differentiable, one can link together the model, reward function, and policy to obtain an analytic expression for the gradient of the returns with respect to the policy. By computing the gradient in this manner, we obtain an expressive signal that allows rapid policy learning. We avoid the instabilities that often from back-propagating through long horizons by using a terminal Q-function. This scheme fully exploits the learned model without harming the learning stability seen in previous approaches (; . The horizon at which we apply the terminal Q-function acts as a hyperparameter between model-free (when fully relying on the Q-function) and model-based (when using a longer horizon) of our algorithm. The main contribution of this work is a model-based method that significantly reduces the sample complexity compared to state-of-the-art model-based algorithms . For instance, we achieve a 10k return in the half-cheetah environment in just 50 trajectories. We theoretically justify our optimization objective and derive the monotonic improvement of our learned policy in terms of the Q-function and the model error. Furtermore, we experimentally analyze the theoretical derivations. Finally, we pinpoint the importance of our objective by ablating all the components of our algorithm. The are reported in four model-based benchmarking environments ). The low sample complexity and high performance of our method carry high promise towards learning directly on real robots. Model-Based Reinforcement Learning. Learned dynamics models offer the possibility to reduce sample complexity while maintaining the asymptotic performance. For instance, the models can act as a learned simulator on which a model-free policy is trained on (; ;). The model can also be used to improve the target value estimates or to provide additional context to a policy . Contrary to these methods, our approach uses the model in a different way: we exploit the fact that the learned simulator is differentiable and optimize the policy with the analytical gradient. Long term predictions suffer from a compounding error effect in the model, ing in unrealistic predictions. In such cases, the policy tends to overfit to the deficiencies of the model, which translates to poor performance in the real environment; this problem is known as model-bias . The model-bias problem has motivated work that uses meta-learning, interpolation between different horizon predictions , and interpolating between model and real data . To prevent model-bias, we exploit the model for a short horizon and use a terminal value function to model the rest of the trajectory. Finally, since our approach returns a stochastic policy, dynamics model, and value function could use model-predictive control (MPC) for better performance at test time, similar to . MPC methods have shown to be very effective when the uncertainty of the dynamics is modelled (; . Differentable Planning. Previous work has used backpropagate through learned models to obtain the optimal sequences of actions. For instance, learn linear local models and obtain the optimal sequences of actions, which is then distilled into a neural network policy. The planning can be incorporated into the neural network architecture (; ; ;) or formulated as a differentiable function . Planning sequences of actions, even when doing model-predictive control (MPC), does not scale well to high-dimensional, complex domains. Our method, instead learns a neural network policy in an actor-critic fashion aided with a learned model. In our study, we evaluate the benefit of carrying out MPC on top of our learned policy at test time, Section 5.4. The suggest that the policy captures the optimal sequence of action, and re-planning does not in significant benefits. Policy Gradient Estimation. The reinforcement learning objective involves computing the gradient of an expectation (a). By using Gaussian processes , it is possible to compute the expectation analytically. However, when learning expressive parametric non-linear dynamical models and policies, such closed form solutions do not exist. The gradient is then estimated using Monte-Carlo methods . In the context of model-based RL, previous approaches mostly made use of the score-function, or REINFORCE estimator . However, this estimator has high variance and extensive sampling is needed, which hampers its applicability in high-dimensional environments. In this work, we make use of the pathwise derivative estimator . Similar to our approach, uses this estimator in the context of model-based RL. However, they just make use of real-world trajectories that introduces the need of a likelihood ratio term for the model predictions, which in turn increases the variance of the gradient estimate. Instead, we entirely rely on the predictions of the model, removing the need of likelihood ratio terms. Actor-Critic Methods. Actor-critic methods alternate between policy evaluation, computing the value function for the policy; and policy improvement using such value function . Actor-critic methods can be classified between on-policy and off-policy. On-policy methods tend to be more stable, but at the cost of sample efficiency . On the other hand, off-policy methods offer better sample complexity. Recent work has significantly stabilized and improved the performance of off-policy methods using maximum-entropy objectives (a) and multiple value functions . Our method combines the benefit of both. By using the learned model we can have a learning that resembles an on-policy method while still being off-policy. In this section, we present the reinforcement learning problem, two different lines of algorithms that tackle it, and a summary on Monte-Carlo gradient estimators. A discrete-time finite Markov decision process (MDP) M is defined by the tuple (S, A, f, r, γ, p 0, T). Here, S is the set of states, A the action space, s t+1 ∼ f (s t, a t) the transition distribution, r: S × A → R is a reward function, p 0: S → R + represents the initial state distribution, γ the discount factor, and T is the horizon of the process. We define the return as the sum of rewards r(s t, a t) along a trajectory τ:= (s 0, a 0, ..., s T −1, a T −1, s T). The goal of reinforcement learning is to find a policy π θ: S × A → R + that maximizes the expected return, i.e., max Actor-Critic. In actor-critic methods, we learn a functionQ (critic) that approximates the expected return conditioned on a state s and action a, Then, the learned Q-function is used to optimize a policy π (actor). Usually, the Q-function is learned by iteratively minimizing the Bellman residual: The above method is referred as one-step Q-learning, and while a naive implementation often in unstable behaviour, recent methods have succeeded in stabilizing the Q-function training . The actor then can be trained to maximize the learnedQ function The benefit of this form of actor-critic method is that it can be applied in an off-policy fashion, sampling random mini-batches of transitions from an experience replay buffer . Model-Based RL. Model-based methods, contrary to model-free RL, learn the transition distribution from experience. Typically, this is carried out by learning a parametric function approximatorf φ, known as a dynamics model. We define the state predicted by the dynamics model asŝ t+1, i.e.,ŝ t+1 ∼f φ (s t, a t). The models are trained via maximum likelihood: In order to optimize the reinforcement learning objective, it is needed to take the gradient of an expectation. In general, it is not possible to compute the exact expectation so Monte-Carlo gradient estimators are used instead. These are mainly categorized into three classes: the pathwise, score function, and measure-valued gradient estimator . In this work, we use the pathwise gradient estimator, which is also known as the re-parameterization trick . This estimator is derived from the law of the unconscious statistician (LOTUS) Here, we have stated that we can compute the expectation of a random variable x without knowing its distribution, if we know its corresponding sampling path and base distribution. A common case, and the one used in this manuscript, θ parameterizes a Gaussian distribution:, which is equivalent to x = µ θ + σ θ for ∼ N. Exploiting the full capability of learned models has the potential to enable complex and highdimensional real robotics tasks while maintaining low sample complexity. Our approach, modelaugmented actor-critic (MAAC), exploits the learned model by computing the analytic gradient of the returns with respect to the policy. In contrast to sample-based methods, which one can think of as providing directional derivatives in trajectory space, MAAC computes the full gradient, providing a strong learning signal for policy learning, which further decreases the sample complexity. In the following, we present our policy optimization scheme and describe the full algorithm. Among model-free methods, actor-critic methods have shown superior performance in terms of sample efficiency and asymptotic performance (a). However, their sample efficiency remains worse than modelbased approaches, and fully off-policy methods still show instabilities comparing to on-policy algorithms . Here, we propose a modification of the Q-function parametrization by using the model predictions on the first time-steps after the action is taken. Specifically, we do policy optimization by maximizing the following objective: whereby, s t+1 ∼f (s t, a t) and a t ∼ π θ (s t). Note that under the true dynamics and Q-function, this objective is the same as the RL objective. Contrary to previous reinforcement learning methods, we optimize this objective by back-propagation through time. Since the learned dynamics model and policy are parameterized as Gaussian distributions, we can make use of the pathwise derivative estimator to compute the gradient, ing in an objective that captures uncertainty while presenting low variance. The computational graph of the proposed objective is shown in Figure 1. While the proposed objective resembles n-step bootstrap , our model usage fundamentally differs from previous approaches. First, we do not compromise between being offpolicy and stability. Typically, n-step bootstrap is either on-policy, which harms the sample complexity, or its gradient estimation uses likelihood ratios, which presents large variance and in unstable learning. Second, we obtain a strong learning signal by backpropagating the gradient of the policy across multiple steps using the pathwise derivative estimator, instead of the REINFORCE estimator . And finally, we prevent the exploding and vanishing gradients effect inherent to back-propagation through time by the means of the terminal Q-function . The horizon H in our proposed objective allows us to trade off between the accuracy of our learned model and the accuracy of our learned Q-function. Hence, it controls the degree to which our algorithm is model-based or well model-free. If we were not to trust our model at all (H = 0), we would end up with a model-free update; for H = ∞, the objective in a shooting objective. Note that we will perform policy optimization by taking derivatives of the objective, hence we require accuracy on the derivatives of the objective and not on its value. The following lemma provides a bound on the gradient error in terms of the error on the derivatives of the model, the Q-function, and the horizon H. Lemma 4.1 (Gradient Error). Letf andQ be the learned approximation of the dynamics f and Q-function Q, respectively. Assume that Q andQ have L q /2-Lipschitz continuous gradient and f and f have L f /2-Lipschitz continuous gradient. Let f = max t ∇f (ŝ t,â t) − ∇f (s t, a t) 2 be the error on the model derivatives and Q = ∇Q(ŝ H,â H) − ∇Q(s H, a H) 2 the error on the Q-function derivative. Then the error on the gradient between the learned objective and the true objective can be bounded by: The in Lemma 4.1 stipulates the error of the policy gradient in terms of the maximum error in the model derivatives and the error in the Q derivatives. The functions c 1 and c 2 are functions of the horizon and depend on the Lipschitz constants of the model and the Q-function. Note that we are just interested in the relation between both sources of error, since the gradient magnitude will be scaled by the learning rate, or by the optimizer, when applying it to the weights. In the previous section, we presented our objective and the error it incurs in the policy gradient with respect to approximation error in the model and the Q function. However, the error on the gradient is not indicative of the effect of the desired metric: the average return. Here, we quantify the effect of the modeling error on the return. First, we will bound the KL-divergence between the policies ing from taking the gradient with the true objective and the approximated one. Then we will bound the performance in terms of the KL. Lemma 4.2 (Total Variation Bound). Under the assumptions of the Lemma 4.1, let θ = θ o + α∇ θ J π be the parameters ing from taking a gradient step on the exact objective, andθ = θ o + α∇ θĴπ the parameters ing from taking a gradient step on approximated objective, where α ∈ R +. Then the following bound on the total variation distance holds Proof. See Appendix. The previous lemma in a bound on the distance between the policies originated from taking a gradient step using the true dynamics and Q-function, and using its learned counterparts. Now, we can derive a similar from to bound the difference in average returns. Theorem 4.1 (Monotonic Improvement). Under the assumptions of the Lemma 4.1, be θ andθ as defined in Lemma 4.2, and assuming that the reward is bounded by r max. Then the average return of the πθ satisfies Proof. See Appendix. Hence, we can provide explicit lower bounds of improvement in terms of model error and function error. Theorem 4.1 extends previous work of monotonic improvement for model-free policies (b;), to the model-based and actor critic set up by taking the error on the learned functions into account. From this bound one could, in principle, derive the optimal horizon H that minimizes the gradient error. However, in practice, approximation errors are hard to determine and we treat H as an extra hyper-parameter. In section 5.2, we experimentally analyze the error on the gradient for different estimators and values of H. Based on the previous sections, we develop a new algorithm that explicitly optimizes the modelaugmented actor-critic (MAAC) objective. The overall algorithm is divided into three main steps: model learning, policy optimization, and Q-function learning. Model learning. In order to prevent overfitting and overcome model-bias , we use a bootstrap ensemble of dynamics models {f φ1, ...,f φ M}. Each of the dynamics models parameterizes the mean and the covariance of a Gaussian distribution with diagonal covariance. The bootstrap ensemble captures the epistemic uncertainty, uncertainty due to the limited capacity or data, while the probabilistic models are able to capture the aleatoric uncertainty , inherent uncertainty of the environment. We denote by f φ the transitions dynamics ing from φ U, where U ∼ U[M] is uniform random variable on {1, ..., M}. The dynamics models are trained via maximum likelihood with early stopping on a validation set. Sample trajectories from the real environment with policy π θ. Add them to D env. 4: for i = 1... G 2 do 10: end for 13: until the policy performs well in the real environment 14: return Optimal parameters θ * Policy Optimization. We extend the MAAC objective with an entropy bonus (b), and perform policy learning by maximizing We learn the policy by using the pathwise derivative of the model through H steps and the Q-function by sampling multiple trajectories from the sameŝ 0. Hence, we learn a maximum entropy policy using pathwise derivative of the model through H steps and the Q-function. We compute the expectation by sampling multiple actions and states from the policy and learned dynamics, respectively. Q-function Learning. In practice, we train two Q-functions since it has been experimentally proven to yield better . We train both Q functions by minimizing the Bellman error (Section 3.1): Similar to , we minimize the Bellman residual on states previously visited and imagined states obtained from unrolling the learned model. Finally, the value targets are obtained in the same fashion the Stochastic Ensemble Value Expansion , using H as a horizon for the expansion. In doing so, we maximally make use of the model by not only using it for the policy gradient step, but also for training the Q-function. Our method, MAAC, iterates between collecting samples from the environment, model training, policy optimization, and Q-function learning. A practical implementation of our method is described in Algorithm 1. First, we obtain trajectories from the real environment using the latest policy available. Those samples are appended to a replay buffer D env, on which the dynamics models are trained until convergence. The third step is to collect imaginary data from the models: we collect k-step transitions by unrolling the latest policy from a randomly sampled state on the replay buffer. The imaginary data constitutes the D model, which together with the replay buffer, is used to learn the Q-function and train the policy. Our algorithm consolidates the insights built through the course of this paper, while at the same time making maximal use of recently developed actor-critic and model-based methods. All in all, it consistently outperforms previous model-based and actor-critic methods. Our experimental evaluation aims to examine the following questions: 1) How does MAAC compares against state-of-the-art model-based and model-free methods? 2) Does the gradient error correlate with the derived bound?, 3) Which are the key components of its performance?, and 4) Does it benefit from planning at test time? In order to answer the posed questions, we evaluate our approach on model-based continuous control benchmark tasks in the MuJoCo simulator (; . We compare our method on sample complexity and asymptotic performance against state-of-the-art model-free (MF) and model-based (MB) baselines. Specifically, we compare against the model-free soft actor-critic (SAC) (a), which is an off-policy algorithm that has been proven to be sample efficient and performant; as well as two state-of-the-art model-based baselines: modelbased policy-optimization (MBPO) and stochastic ensemble value expansion (STEVE) . The original STEVE algorithm builds on top of the model-free algorithm DDPG, however this algorithm is outperformed by SAC. In order to remove confounding effects of the underlying model-free algorithm, we have implemented the STEVE algorithm on top of SAC. We also add SVG to comparison, which similar to our method uses the derivative of dynamic models to learn the policy. The , shown in Fig. 2, highlight the strength of MAAC in terms of performance and sample complexity. MAAC scales to higher dimensional tasks while maintaining its sample efficiency and asymptotic performance. In all the four environments, our method learns faster than previous MB and MF methods. We are able to learn near-optimal policies in the half-cheetah environment in just over 50 rollouts, while previous model-based methods need at least the double amount of data. Furthermore, in complex environments, such as ant, MAAC achieves near-optimal performance within 150 rollouts while other take orders of magnitudes more data. Here, we investigate how the bounds obtained relate to the empirical performance. In particular, we study the effect of the horizon of the model predictions on the gradient error. In order to do so, we construct a double integrator environment; since the transitions are linear and the cost is quadratic for a linear policy, we can obtain the analytic gradient of the expect return. Figure 3: L1 error on the policy gradient when using the proposed objective for different values of the horizon H as well as the error obtained when using the true dynamics. The correlate with the assumption that the error in the learned dynamics is lower than the error in the Q-function, as well as they correlate with the derived bounds. Figure 3 depicts the L1 error of the MAAC objective for different values of the horizon H as well as what would be the error using the true dynamics. As expected, using the true dynamics yields to lower gradient error since the only source comes from the learned Q-function that is weighted down by γ H. The using learned dynamics correlate with our assumptions and the derived bounds: the error from the learned dynamics is lower than the one in the Q-funtion, but it scales poorly with the horizon. For short horizons the error decreases as we increase the horizon. However, large horizons is detrimental since it magnifies the error on the models. In order to investigate the importance of each of the components of our overall algorithm, we carry out an ablation test. Specifically, we test three different components: 1) not using the model to train the policy, i.e., set H = 0, 2) not using the STEVE targets for training the critic, and 3) using a single sample estimate of the path-wise derivative. The ablation test is shown in Figure 4. The test underpins the importance of backpropagating through the model: setting H to be 0 inflicts a severe drop in the algorithm performance. On the other hand, using the STEVE targets in slightly more stable training, but it does not have a significant effect. Finally, while single sample estimates can be used in simple environments, they are not accurate enough in higher dimensional environments such as ant. Figure 4: Ablation test of our method. We test the importance of several components of our method: not using the model to train the policy (H = 0), not using the STEVE targets for training the Q-function (-STEVE), and using a single sample estimate of the pathwise derivative. Using the model is the component that affects the most the performance, highlighting the importance of our derived estimator. One of the key benefits of methods that combine model-based reinforcement learning and actor-critic methods is that the optimization procedure in a stochastic policy, a dynamics model and a Q-function. Hence, we have all the components for, at test time, refine the action selection by the means of model predictive control (MPC). Here, we investigate the improvement in performance of planning at test time. Specifically, we use the cross-entropy method with our stochastic policy as our initial distributions. The , shown in Table 2, show benefits in online planning in complex domains; however, its improvement gains are more timid in easier domains, showing that the learned policy has already interiorized the optimal behaviour. HalfCheetahEnv HopperEnv Walker2dEnv MAAC+MPC 3.97e3 ± 1.48e3 1.09e4 ± 94.5 2.8e3 ± 11 1.76e3 ± 78 MAAC 3.06e3 ± 1.45e3 1.07e4 ± 253 2.77e3 ± 3.31 1.61e3 ± 404 Table 1: Performance at test time with (maac+mpc) and without (maac) planning of the converged policy using the MAAC objective. In this work, we present model-augmented actor-critic, MAAC, a reinforcement learning algorithm that makes use of a learned model by using the pathwise derivative across future timesteps. We prevent instabilities arisen from backpropagation through time by the means of a terminal value function. The objective is theoretically analyzed in terms of the model and value error, and we derive a policy improvement expression with respect to those terms. Our algorithm that builds on top of MAAC is able to achieve superior performance and sample efficiency than state-of-the-art model-based and model-free reinforcement learning algorithms. For future work, it would be enticing to deploy the presented algorithm on a real-robotic agent. Then, the error in the gradient in the previous term is bounded by In order to bound the model term we need first to bound the rewards since Similar to the previous bounds, we can bound now each reward term by With this we can bound the total error in models Then, the gradient error has the form A.2 PROOF OF LEMMA 4.2 The total variation distance can be bounded by the KL-divergence using the Pinsker's inequality Then if we assume third order smoothness on our policy, by the Fisher information metric theorem then Given that θ −θ 2 = α ∇ θ J π − ∇ θĴπ 2, for a small enough step the following inequality holds Given the bound on the total variation distance, we can now make use of the monotonic improvement theorem to establish an improvement bound in terms of the gradient error. Let J π (θ) and J π (θ) be the expected return of the policy π θ and πθ under the true dynamics. Let ρ andρ be the discounted state marginal for the policy π θ and πθ, respectively Then, combining the from Lemma 4.2 we obtain the desired bound. In order to show the significance of each component of MAAC, we conducted more ablation studies. The are shown in Figure 5. Here, we analyze the effect of training the Q-function with data coming from just the real environment, not learning a maximum entropy policy, and increasing the batch size instead of increasing the amount of samples to estimate the value function. Figure 5: We further test the significance of some components of our method: not use the dynamics to generate data, and only use real data sampled from environments to train policy and Q-functions (real_data), remove entropy from optimization objects (no_entropy), and using a single sample estimate of the pathwise derivative but increase the batch size accordingly (5x batch size). Considering entropy and using dynamic models to augment data set are both very important. A.5 EXECUTION TIME COMPARISON

represent the tldr for finding the corresponding scientific article!

Policy gradient through backpropagation through time using learned models and Q-functions. SOTA results in reinforcement learning benchmark environments.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We augment the Q-value estimates with a count-based bonus that ensures optimism during action selection and bootstrapping, even if the Q-value estimates are pessimistic.