Question: represent the text to find a very short summary (1-2 sentences max)

Question: Deep neural networks have achieved impressive performance in handling complicated semantics in natural language, while mostly treated as black boxes. To explain how the model handles compositional semantics of words and phrases, we study the hierarchical explanation problem. We highlight the key challenge is to compute non-additive and context-independent importance for individual words and phrases. We show some prior efforts on hierarchical explanations, e.g. contextual decomposition, do not satisfy the desired properties mathematically, leading to inconsistent explanation quality in different models. In this paper, we propose a formal way to quantify the importance of each word or phrase to generate hierarchical explanations. We modify contextual decomposition algorithms according to our formulation, and propose a model-agnostic explanation algorithm with competitive performance. Human evaluation and automatic metrics evaluation on both LSTM models and fine-tuned BERT Transformer models on multiple datasets show that our algorithms robustly outperform prior works on hierarchical explanations. We show our algorithms help explain compositionality of semantics, extract classification rules, and improve human trust of models. Recent advances in deep neural networks have led to impressive on a range of natural language processing (NLP) tasks, by learning latent, compositional vector representations of text data (; ; b). However, interpretability of the predictions given by these complex, "black box" models has always been a limiting factor for use cases that require explanations of the features involved in modeling (e.g., words and phrases) . Prior efforts on enhancing model interpretability have focused on either constructing models with intrinsically interpretable structures (; a), or developing post-hoc explanation algorithms which can explain model predictions without elucidating the mechanisms by which model works . Among these work, post-hoc explanation has come to the fore as they can operate over a variety of trained models while not affecting predictive performance of models. Towards post-hoc explanation, a major line of work, additive feature attribution methods (; ; ;), explain a model prediction by assigning importance scores to individual input variables. However, these methods may not work for explaining compositional semantics in natural language (e.g., phrases or clauses), as the importance of a phrase often is non-linear combination of the importance of the words in the phrase. Contextual decomposition (CD) and its hierarchical extension go beyond the additive assumption and compute the contribution solely made by a word/phrase to the model prediction (i.e., individual contribution), by decomposing the output variables of the neural network at each layer. Using the individual contribution scores so derived, these algorithms generate hierarchical explanation on how the model captures compositional semantics (e.g., stress or negation) in making predictions (see Figure 1 for example). (a) Input occlusion assigns a negative score for the word "interesting", as the sentiment of the phrase becomes less negative after removing "interesting" from the original sentence. (b) Additive attributions assign importance scores for words "not" and "interesting" by linearly distributing contribution score of "not interesting", exemplified with Shapley Values . Intuitively, only (c) Hierarchical explanations highlight the negative compositional effect between the words "not" and "interesting". However, despite contextual decomposition methods have achieved good in practice, what reveals extra importance that emerge from combining two phrases has not been well studied. As a , prior lines of work on contextual decomposition have focused on exploring model-specific decompositions based on their performance on visualizations. We identify the extra importance from combining two phrases can be quantified by studying how the importance of the combined phrase differs from the sum of the importance of the two component phrases on its own. Similar strategies have been studied in game theory for quantifying the surplus from combining two groups of players . Following the definition above, the key challenge is to formulate the importance of a phrase on it own, i.e., context independent importance of a phrase. However, while contextual decomposition algorithms try to decompose the individual contributions from given phrases for explanation, we show neither of them satisfy this context independence property mathematically. To this end, we propose a formal way to quantify the importance of each individual word/phrase, and develop effective algorithms for generating hierarchical explanations based on the new formulation. To mathematically formalize and efficiently approximate context independent importance, we formulate N -context independent importance of a phrase, defined as the difference of model output after masking out the phrase, marginalized over all possible N words surrounding the phrase in the sentence. We propose two explanation algorithms according to our formulation, namely the Sampling and Contextual Decomposition algorithm (SCD), which overcomes the weakness of contextual decomposition algorithms, and the Sampling and OCclusion algorithm (SOC), which is simple, model-agnostic, and performs competitively against prior lines of algorithms. We experiment with both LSTM and fine-tuned Transformer models to evaluate the proposed methods. Quantitative studies involving automatic metrics and human evaluation on sentiment analysis and relation extraction tasks show that our algorithms consistently outperform competitors in the quality of explanations. Our algorithms manage to provide hierarchical visualization of compositional semantics captured by models, extract classification rules from models, and help users to trust neural networks predictions. In summary, our work makes the following contributions: we identify the key challenges in generating post-hoc hierarchical explanations and propose a mathematically sound way to quantify context independent importance of words and phrases for generating hierarchical explanations; we extend previous post-hoc explanation algorithm based on the new formulation of N -context independent importance and develop two effective hierarchical explanation algorithms; and both experiments using automatic evaluation metrics and human evaluation demonstrate that the proposed explanation algorithms consistently outperform the compared methods (with both LSTM and Transformer as base models) over several datasets. We consider a sequence of low-dimensional word embeddings x 1:T:= (x 1, x 2, ..., x T), or denoted as x for brevity, as the input to a neural sequence model, such as standard RNNs, LSTM and Transformers . These neural models extract latent, compositional representations h 1:T (i.e., hidden states) from the input sequence x, and feed these hidden state vectors to a prediction layer to generate output in the label space (e.g., sentiment polarity of a sentence). For LSTM, we use the last hidden state h T to give unnormalized prediction scores s(x) ∈ R dc over d c label classes as follows. where W l ∈ R dc×d h is a trainable weight matrix. For Transformers, the representation corresponding to the " [CLS] " token at the final layer is fed to the prediction layer to generate scores s(x). Towards post-hoc explanation of s(x), a notable line of work, additive feature attribution methods (; ;), measure word-level importance to the model prediction s(x) by attributing a importance score φ(x i, x) to each word in the input sequence x i ∈ x. Such additive attribution methods are related to Shapley Values and thus can be proven to enjoy good properties, including that it has unique solution of a "fair" attribution . However, the additive assumption hinders these methods from explaining the complex interactions between words and compositional semantics in a sentence (e.g., modeling negation, transition, and emphasis in sentiment classification), as shown in Figure 1. To caputure non-linear compositional semantics, the line of work on contextual decomposition (CD) designs non-additive measures of importance from individual words/phrases to the model predictions, and further extend to agglomerative contextual decomposition (ACD) algorithm for generating hierarchical explanations. Given a phrase p = x i:j in the input sequence x, contextual decomposition (CD) attributes a score φ(p, x) as the contribution solely from p to the model's prediction s(x). Note that φ(p, x) does not equal to the sum of the scores of each word in the phrase, i.e., φ(p, x) = xi∈p φ(x i, x). Starting from the input layer, CD iteratively decomposes each hidden state h of the model into the contribution solely made by p, denoted as β, and the contributions involving the words outside the phrase p, denoted as γ, with the relation h = β + γ holds. Note that the algorithm also keeps the contribution from the bias term, denoted as ζ, temporally before element-wise multiplication. For a linear layer h = W i x t + b i with input x t, the contribution solely from p to h is defined as β = W i x t when x t is part of the phrase (i.e., x t ∈ p), and the contribution involving other words in the sentences (denoted as x\p) is defined as γ = 0. The contribution of the bias term ζ is thus b i. When x t lies outside of the phrase in the sentence (i.e., x t ∈ p), γ is quantified as W i x t and β is 0. In the cases when CD encounters element-wise multiplication operations h = h a * h b (e.g., in LSTMs), it eliminates the multiplicative interaction terms which involve the information outside of the phrase p. Specifically, suppose that h a and h b have been decomposed as When dealing with non-linear activation h = σ(h), CD computes the contribution solely from the phrase p as the average activation differences caused by β supposing γ is present or absent, Following the three strategies introduced above, CD decomposes all the intermediate outputs starting from the input layer, until reaching the final output of the model h T = β + γ. The logit score W l β is treated as the contribution of the given phrase p to the final prediction s(x). As a follow-up study, extends CD algorithm to other families of neural network architectures, and proposes agglomerative contextual decomposition algorithm (ACD). The decomposition of activation functions is modified as β = σ(β). For the linear layer h = W h + b with its decomposition h = β + γ, the bias term b is decomposed proportionally and merged into the β term of h, based on β = W β + |W β|/(|W β| + |W γ|) · b. In this section, we start by identifying desired properties of phrase-level importance score attribution for hierarchical explanations. We propose a measure of context-independent importance of a phrase and introduce two explanation algorithms instantiated from the proposed formulation. x ⇠ p(x |x) Despite the empirical success of CD and ACD, no prior works analyze what common properties a score attribution mechanism should satisfy to generate hierarchical explanations that reveal compositional semantics formed between phrases. Here we identify two properties that an attribution method should satisfy to generate informative hierarchical explanations. Non-additivity Importance of a phrase φ(p, x) should be quantified by a non-linear function over the importance scores of all the component words, in contrast to the family of additive feature attribution methods. Context Independence For deep neural networks, when two phrases combine, their importance to predicting a class may greatly change. The surplus by combining two phrases can be quantified by the difference between the importance of the combined phrase and the sum of the importance of two phrases on its own. It follows how the surplus of combining two groups of players can be quantified in the game theory . According to the definition, the importance of two component phrases should be evaluated independently of each other. Formally, if we are interested in how combining two phrases p 1 and p 2 contribute to a specific prediction for an input x, we expect for input sentencesx where only p 2 is replaced to another phrase, the importance attribution for p 1 remains the same, i.e., φ(p 1, x) = φ(p 1,x). In the hierarchical explanation setting, we are interested in how combining a phrase and any other contextual words or phrases in the input x changes the prediction for the input x. Therefore, we expect φ(p, x) = φ(p,x) given the phrase p in two different contexts x andx. Limitations of CD and ACD Unfortunately, while CD and ACD try to construct decomposition so that β terms represent the contributions solely from a given a phrase, the assigned importance scores by these algorithms do not satisfy the context independence property mathematically. For CD, we see the computation of β involves the γ term of a specific input sentence in Eq. 2 (see Figure 2 (a) for visualization). Similarly, for ACD, the decomposition of the bias term involves the γ terms of a specific input sentence. As a , the β terms computed by both algorithms depend on the context of the phrase p. Regarding the decomposition of activation functions, the decomposition β = σ(β) in ACD seems plausible, which does not involve γ for computing β terms. However, in case every activation is decomposed in this way and suppose the bias terms are merged into β, the algorithm is equivalent to feeding only the phrase p into the classifier with all other input masked as zero. Empirical show unreliable explanation quality of both algorithms in some models. Given the limitation of prior works, we start by formulating a importance measure of phrases that satisfies both non-additivity and context independence property. Given a phrase p:= x i:j appearing in a specific input x 1:T, we first relax our setting and define the importance of a phrase independent of all the possible N -word contexts adjacent to it. The N-context independent importance is defined as the output difference after masking out the phrase p, marginalized over all the possible N -word contexts, denoted as x δ, around p in the input x. For an intuitive example, to evaluate the context independent importance up to one word of very in the sentence The film is very interesting in a sentiment analysis model, we sample some possible adjacent words before and after the word very, and observe the prediction difference after some practice of masking the word very. In Figure 2 (Right), we illustrated an example for the sampling and masking steps. The process of evaluating context independent importance is formally written as, where x −δ denotes the ing sequence after masking out an N -word context surrounding the phrase p from the input x. Here, x δ is a N -word sequence sampled from a distribution p(x δ |x −δ), which is conditioned on the phrase p as well as other words in the sentence x. Details on the sampling process will be elaborated later in the section. Accordingly, we use s(x −δ ; x δ) to denote the model prediction score after replacing the masked-out context x −δ with a sampled N -word sequence x δ. We use x\p to denote the operation of masking out the phrase p from the input sentence x. The specific implementation of this masking out operation varies across different explanation algorithms and is instantiated from their formulation. Following the notion of N -context independent importance, we define context-independent importance of a phrase p by increasing the size of the context N to sufficiently large (e.g., length of the sentence). The context independent importance can be equivalently written as follows. Computing Eqs. 3 and 4 are intractable as it requires integrating over a large number of variants of x δ as replacements (i.e., number of variants for x δ is exponential to the size of N). While it is possible to approximate the expectations in Eqs. 3 and 4 by sampling from the training text corpus, we find it common that a phrase occurs sparsely in the corpus. Therefore, we approximate the expectation by sampling from a language model pre-trained using the training corpus. The language model helps model a smoothed distribution of p(x δ |x −δ). In practice, all our explanation algorithms implements N -context independent importance following Eq. 3, where the size of the neighborhood N is a parameter to be specified. In contextual decomposition algorithm, the desirable context independence property is compromised when computing decomposition of activation functions, as discussed in Section 3.1. Following the new formulation on context-independent importance introduced in Section 3.2, we present a simple modification of the contextual decomposition algorithm, and develop a new sampling and contextual decomposition (SCD) algorithm for effective generation of hierarchical explanations. SCD only modifies the way to decompose activation functions in CD. Specifically, given the output h = s (l) (x) at an intermediate layer l with the decomposition h = β + γ, we decompose the activation value σ(h) into β + γ, with the following definition: i.e., β is defined as the expected difference between the activation values when the β term is present or absent. h is computed for different input sequences x with the contexts of the phrase p sampled from the distribution p(x δ |x −δ). Eq. 5 is a layer wise application of Eq. 4, where the masking operation is implemented with calculating σ(h − β). Figure 2 (b) provides a visualization for the decomposition. To perform sampling, we first pretrain a LSTM language model from two directions on the training data. For sampling, we mask the words that are not conditioned in p(x δ |x −δ). Some other sampling options include performing Gibbs sampling from a masked language model . The algorithm then obtain a set of samples S by sampling with the trained language model. For each sample in S, the algorithm records the input of the i-th non-linear activation function to obtain a sample set S h. During the explanation, the decomposition of the i-th non-linear activation function is calculated as, Some neural models such as Transformers involve operations that normalize over different dimensions of a vectors, e.g. softmax functions and layer normalization operations. We observe improved performance by not decomposing the normalizer of these terms when the phrase p is shorter than a threshold, assuming that the impact of p to the normalizer can be ignored. Besides, for element-wise multiplication in LSTM models, we treat them in the same way as other nonlinear operations and decompose them as Eq. 5, where the decomposition of h 1 h 2 is written as We show it is possible to fit input occlusion algorithms into our formulation. Input occlusion algorithms calculate the importance of p specific to an input example x by observing the prediction difference after replacing the phrase p with padding tokens, noted as 0 p, It is obvious that importance score by the input occlusion algorithm is dependent on the all the context words of p in x. To eliminate the dependence, we perform sampling around the phrase p. This leads to the Sampling and Occlusion (SOC) algorithm, which computes the importance of phrases as the average prediction difference after masking the phrase for each replacement of neighboring words in the input example. Similar to SCD, SOC samples neighboring words x δ from a trained language model p(x δ |x −δ) and obtain a set of neighboring word replacement S. For each replacement x δ ∈ S, the algorithm computes the model prediction differences after replacing the phrase p with padding tokens. The importance φ(p, x) is then calculated as the average prediction differences. Formally, the algorithm calculates, We evaluate explanation algorithms on both shallow LSTM models and deep fine-tuned BERT Transformer models. We use two sentiment analysis datasets, namely the Stanford Sentiment Treebank-2 (SST-2) dataset and the Yelp Sentiment Polarity dataset , as well as a relation extraction dataset, namely the TACRED dataset . The two tasks are modeled as binary and multi-class classification tasks respectively. For the SST-2 dataset, while it provides sentiment polarity scores for all the phrases on the nodes of the constituency parsing trees, we do not train our model on these phrases, and use these scores as the evaluation for the phrase level explanations. Our Transformer model is fine-tuned from pretrained BERT model. See Appendix A for model details. We compare our explanation algorithm with following baselines: Input occlusion and Integrated Gradient+SHAP (GradSHAP) ; two algorithms applied for hierarchical explanations, namely Contextual Decomposition (CD) , and Agglomerative Contextual Decomposition (ACD) . We also compare with a naive however neglected baseline in prior literature, which directly feed the given We generate explanations for all the phrases on the truncated constituency parsing tree, where positive sentiments are colored red and negative sentiments are colored blue. We see our method identify positive segments in the overall negative sentence, such as "a breath of fresh air" phrase to the model and take the prediction score as the importance of the phrase, noted as Direct Feed. For our algorithms, we list the performance of corpus statistic based approach (Statistic) for approximating context independent importance in Eq. 3, Sampling and Contextual Decomposition (SCD), and Sampling and Occlusion (SOC) algorithm. We verify the performance of our algorithms in identifying important words and phrases captured by models. We follow the quantitative evaluation protocol proposed in CD algorithm for evaluating word-level explanations, which computes Pearson correlation between the coefficients learned by a linear bag-of-words model and the importance scores attributed by explanation methods, also noted as the word ρ. When the linear model is accurate, its coefficients could stand for general importance of words. For evaluating phrase level explanations, we notice the SST-2 dataset provides human annotated real-valued sentiment polarity for each phrase on constituency parsing trees. We generate explanations for each phrase on the parsing tree and evaluate the Pearson correlation between the ground truth scores and the importance scores assigned for phrases, also noted as the phrase ρ. This evaluation assume that annotators consider the polarity of incomplete phrases by considering there effects in possible contexts for annotations. We draw K = 20 samples from N = 10 words adjacent to a phrase to be explained at the sampling step in our SOC and SCD algorithms. The parameter setting is trade-off between the efficiency and performance. Table 1 shows word ρ and phrase ρ achieved by our algorithms and competitors. Generally, explanation algorithms that follow our formulations achieve highest word ρ and phrase ρ for all the datasets and models. SOC and SCD perform robustly on the deep Transformer model, achieving higher word ρ and phrase ρ than input occlusion and contextual decomposition algorithms by a large margin. We see the simple Direct Feed method provide promising on shallow LSTM networks, but fail in deeper Transformer models. The statistic based approximation of the context independent impor- tance, which do not employ a trained sampler, yields competitive words ρ, but it is not competitive for phrase ρ, pushing the phrase ρ towards that of the input occlusion algorithm. We find it common that a long phrase does not exist in previously seen examples. Qualitative study also shows that our explanation visualize complicated compositional semantics captured by models, such as positive segments in the negative example, and adversative conjunctions connected with "but". We present an example explanation provided by SOC algorithm in Figure 3 and Appendix. We show our explanation algorithm is a nature fit for extracting phrase level classification rules from neural classifiers. With the agglomerative clustering algorithm in , our explanation effectively identify phrase-level classification patterns without evaluating all possible phrases in the sentence even when a predefined hierarchy does not exist. Figure 4 show an example of automatically constructed hierarchy and extracted classification rules in an example in the TACRED dataset. We follow the human evaluation protocol in and study whether our explanations help subjects to better trust model predictions. We ask subjects to rank the provided visualizations based on how they would like to trust the model. For the SST-2 dataset, we show subjects the predictions of the fine-tuned BERT model, and the explanations generated by SOC, SCD, ACD and GradSHAP algorithms for phrases. The phrase polarities are visualized in a hierarchy with the provided parsing tree of each sentence in the dataset. For the TACRED dataset, we show the explanations provided by SOC, SCD, CD and Direct Feed algorithms on the LSTM model. We binarilize the importance of a phrase by calculating the difference between its importance to the predicted class and the its top importance to other classes, and the hierarchies are constructed automatically with agglomerative clustering . Figure 5 shows average ranking of explanations, where 4 notes the best, and 1 notes the worst. On the SST-2 dataset, SOC achieve significantly higher ranking than ACD and GradSHAP, showing a p-value less than 0.05 and 0.001 respectively. On the TACRED dataset, SCD achieve the best ranking, showing significantly better ranking than CD and Direct Feed with p-value less than 10 −6. Both SOC and SCD algorithms require specifying the size of the context region N and the number of samples K. In Figure 6 (also Figure 7 in Appendix) we show the impact of these parameters. We also plot the performance curves when we pad the contexts instead of sampling. We see sampling the context achieves much better performance than padding the context. We also see word ρ and phrase ρ increase as the number of samples K increases. The overall performance also increases as the size of the context region N increases at the early stage, and saturates when N grows large, which implies words or phrases usually do not interact with the words that are far away them in the input. The saturation also implies the performance of trained language models can be a bottleneck of the performance. Interpretability of neural networks has been studied with vairous techniques, including probing learned features with auxiliary tasks , or designing models with inherent interpretability . A major line of work, local explanation algorithms, explains predictions by assigning importance scores for input features. This line of work include input occlusion (Kádár et al., 2017), gradient based algorithms (; ;), additive feature attribution methods (; ;), among which Shapley value based approaches have been studied intensively because of its good mathematical properties. Researchers also study how to efficiently marginalize over alternative input features to be explained for input occlusion algorithms, while our research show extra focus could be placed on marginalizing over contexts. Regarding explanations of models with structured inputs, propose L-Shapley and C-Shapley for efficient approximation of Shapley values, with a similar hypothesis with us that the importance of a word is usually strongly dependent on its neighboring contexts. propose a feature selection based approach for explanation in an information theoretic perspective. On the other hand, global explanation algorithms have also been studied for identifying generally important features, such as Feature Importance Ranking Measure , Accumulated Local Effects (. We note that the context independence property in our proposed methods implies we study hierarchical explanation as a global explanation problem . Compared with local explanation algorithms, global explanation algorithms are less studied for explaining individual predictions , because they reveal the average behavior of models. However, with a hierarchical organization, we show global explanations are also powerful at explaining individual predictions, achieving better human evaluation scores and could explain compositional semantics where local explanation algorithms such as additive feature attribution algorithms totally fail. Moreover, we note that the use of explanation algorithms is not exclusive; we may apply explanation algorithms of different categories to make a more holistic explanation of model predictions. In this work, we identify two desirable properties for informative hierarchical explanations of predictions, namely the non-additivity and context-independence. We propose a formulation to quantify context independent importance of words and phrases that satisfies the properties above. We revisit the prior line of works on contextual decomposition algorithms, and propose Sampling and Contextual Decomposition (SCD) algorithm. We also propose a simple and model agnostic explanation algorithm, namely the Sampling and Occlusion algorithm (SOC). Experiments on multiple datasets and models show that our explanation algorithms generate informative hierarchical explanations, help to extract classification rules from models, and enhance human trust of models. Table 2: Phrase-level classification patterns extracted from models. We show the of SCD and SOC respectively for the SST-2 and the TACRED dataset. Our LSTM classifiers use 1 layer unidirectional LSTM and the number of hidden units is set to 128, 500, and 300 for SST-2, Yelp, and TACRED dataset respectively. For all models, we load the pretrained 300-dimensional Glove word vectors . The language model sampler is also built on LSTM and have the same parameter settings as the classifiers. Our Transformer models are fine-tuned from pretrained BERT models , which have 12 layers and 768 hidden units of per representation. On three datasets, LSTM models achieve 82% accuracy, 95% accuracy, and 0.64 F1 score on average. The fine-tuned BERT models achieve 92% accuracy, 96% accuracy, and 0.68 F1 score on average. We use the same parameter settings between LSTM classifiers and language models on three datasets. For computing context independent importance of a phrase, an intuitive and simple alternative approach, which is nevertheless neglected in prior literature, is to only feed the input to the model and treat the prediction score as the explanation. In Table 1, while the score of the Direct Feed is lower than that of the best performing algorithms, the score is rather competitive. The potential risk of this explanation is that it assumes model performs reasonably on incomplete sentence fragments that are significantly out of the data distribution. As a , the explanation of short phrases can be misleading. To simulate the situation, we train a LSTM model on inversed labels on isolate words, in addition to the origin training sentences. The model could achieve the same accuracy as the original LSTM model. However, the word ρ and the phrase ρ of Direct Feed drop by a large margin, showing a word ρ of -0.38 and 0.09. SOC and SCD are still robust on the adverse LSTM model, both showing a word ρ and phrase ρ of more than 0.60 and 0.55. The masking operation could also cause performance drop because the masked sentence can be out of data distribution when explaining long phrases. For SOC, the risk can be resolved by implementing the masking operation of the phrase p by another round of sampling from a language model conditioned on its context x −p, but we do not find empirical evidence showing that it improves performance.

Represent the TLDR for finding the corresponding scientific article

We propose measurement of phrase importance and algorithms for hierarchical explanation of neural sequence model predictions

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent text.

Question: Stochastic gradient descent (SGD) with stochastic momentum is popular in nonconvex stochastic optimization and particularly for the training of deep neural networks. In standard SGD, parameters are updated by improving along the path of the gradient at the current iterate on a batch of examples, where the addition of a ``momentum'' term biases the update in the direction of the previous change in parameters. In non-stochastic convex optimization one can show that a momentum adjustment provably reduces convergence time in many settings, yet such have been elusive in the stochastic and non-convex settings. At the same time, a widely-observed empirical phenomenon is that in training deep networks stochastic momentum appears to significantly improve convergence time, variants of it have flourished in the development of other popular update methods, e.g. ADAM, AMSGrad, etc. Yet theoretical justification for the use of stochastic momentum has remained a significant open question. In this paper we propose an answer: stochastic momentum improves deep network training because it modifies SGD to escape saddle points faster and, consequently, to more quickly find a second order stationary point. Our theoretical also shed light on the related question of how to choose the ideal momentum parameter--our analysis suggests that $\beta \in$ should be large (close to 1), which comports with empirical findings. We also provide experimental findings that further validate these . SGD with stochastic momentum has been a de facto algorithm in nonconvex optimization and deep learning. It has been widely adopted for training machine learning models in various applications. Modern techniques in computer vision (e.g. ; ; ;), speech recognition (e.g.), natural language processing (e.g.), and reinforcement learning (e.g.) use SGD with stochastic momentum to train models. The advantage of SGD with stochastic momentum has been widely observed (; ;). demonstrate that training deep neural nets by SGD with stochastic momentum helps achieving in faster convergence compared with the standard SGD (i.e. without momentum). The success of momentum makes it a necessary tool for designing new optimization algorithms in optimization and deep learning. For example, all the popular variants of adaptive stochastic gradient methods like Adam or AMSGrad (Reddi et al. (2018b) ) include the use of momentum. Despite the wide use of stochastic momentum (Algorithm 1) in practice, justification for the clear empirical improvements has remained elusive, as has any mathematical guidelines for actually setting the momentum parameter-it has been observed that large values (e.g. β = 0.9) work well in practice. It should be noted that Algorithm 1 is the default momentum-method in popular software packages such as PyTorch and Tensorflow. 1 In this paper we provide a theoretical analysis for SGD with 1: Required: Step size parameter η and momentum parameter β. 2: Init: w0 ∈ R d and m−1 = 0 ∈ R d. 3: for t = 0 to T do 4: Given current iterate wt, obtain stochastic gradient gt:= ∇f (wt; ξt). Update stochastic momentum mt:= βmt−1 + gt. Update iterate wt+1:= wt − ηmt. 7: end for momentum. We identify some mild conditions that guarantees SGD with stochastic momentum will provably escape saddle points faster than the standard SGD, which provides clear evidence for the benefit of using stochastic momentum. For stochastic heavy ball momentum, a weighted average of stochastic gradients at the visited points is maintained. The new update is computed as the current update minus a step in the direction of the momentum. Our analysis shows that these updates can amplify a component in an escape direction of the saddle points. In this paper, we focus on finding a second-order stationary point for smooth non-convex optimization by SGD with stochastic heavy ball momentum. Specifically, we consider the stochastic nonconvex optimization problem, min w∈R d f (w):= E ξ∼D [f (w; ξ)], where we overload the notation so that f (w; ξ) represents a stochastic function induced by the randomness ξ while f (w) is the expectation of the stochastic functions. An (,)-second-order stationary point w satisfies ∇f (w) ≤ and ∇ 2 f (w) − I. Obtaining a second order guarantee has emerged as a desired goal in the nonconvex optimization community. Since finding a global minimum or even a local minimum in general nonconvex optimization can be NP hard (; ; ;), most of the papers in nonconvex optimization target at reaching an approximate second-order stationary point with additional assumptions like Lipschitzness in the gradients and the Hessian (e.g. ; et al. ). 2 We follow these related works for the goal and aim at showing the benefit of the use of the momentum in reaching an (,)-second-order stationary point. We introduce a required condition, akin to a model assumption made in , that ensures the dynamic procedure in Algorithm 2 produces updates with suitable correlation with the negative curvature directions of the function f. Definition 1. Assume, at some time t, that the Hessian H t = ∇ 2 f (w t) has some eigenvalue smaller than − and ∇f (w t) ≤. Let v t be the eigenvector corresponding to the smallest eigenvalue of ∇ 2 f (w t). The stochastic momentum m t satisfies Correlated Negative Curvature (CNC) at t with parameter γ > 0 if As we will show, the recursive dynamics of SGD with heavy ball momentum helps in amplifying the escape signal γ, which allows it to escape saddle points faster. We show that, under CNC assumption and some minor constraints that upper-bound parameter β, if SGD with momentum has properties called Almost Positively Aligned with Gradient (APAG), Almost Positively Correlated with Gradient (APCG), and Gradient Alignment or Curvature Exploitation (GrACE), defined in the later section, then it takes T = O((1 − β) log(1/(1 − β) ) −10 ) iterations to return an (,) second order stationary point. Alternatively, one can obtain an (, √) second order stationary point in T = O((1 − β) log(1/(1 − β) ) −5 ) iterations. Our theoretical demonstrates that a larger momentum parameter β can help in escaping saddle points faster. As saddle points are pervasive in the loss landscape of optimization and deep learning , the sheds light on explaining why SGD with momentum enables training faster in optimization and deep learning. Notation: In this paper we use E t [·] to represent conditional expectation E[·|w 1, w 2, . . ., w t], which is about fixing the randomness upto but not including t and notice that w t was determined at t − 1. 2.1 A THOUGHT EXPERIMENT. Let us provide some high-level intuition about the benefit of stochastic momentum with respect to avoiding saddle points. In an iterative update scheme, at some time t 0 the parameters w t0 can enter a saddle point region, that is a place where Hessian ∇ 2 f (w t0) has a non-trivial negative eigenvalue, say λ min (∇ 2 f (w t0)) ≤ −, and the gradient ∇f (w t0) is small in norm, say ∇f (w t0) ≤. The challenge here is that gradient updates may drift only very slowly away from the saddle point, and may not escape this region; see for additional details. On the other hand, if the iterates were to move in one particular direction, namely along v t0 the direction of the smallest eigenvector of ∇ 2 f (w t0), then a fast escape is guaranteed under certain constraints on the step size η; see e.g. ). While the negative eigenvector could be computed directly, this 2nd-order method is prohibitively expensive and hence we typically aim to rely on gradient methods. With this in mind, , who study non-momentum SGD, make an assumption akin to our CNC property described above that each stochastic gradient g t0 is strongly non-orthogonal to v t0 the direction of large negative curvature. This suffices to drive the updates out of the saddle point region. In the present paper we study stochastic momentum, and our CNC property requires that the update direction m t0 is strongly non-orthogonal to v t0; more precisely, E t0 [m t0, v t0 2] ≥ γ > 0. We are able to take advantage of the analysis of to establish that updates begin to escape a saddle point region for similar reasons. Further, this effect is amplified in successive iterations through the momentum update when β is close to 1. Assume that at some w t0 we have m t0 which possesses significant correlation with the negative curvature direction v t0, then on successive rounds m t0+1 is quite close to βm t0, m t0+2 is quite close to β 2 m t0, and so forth; see Figure 1 for an example. This provides an intuitive perspective on how momentum might help accelerate the escape process. Yet one might ask does this procedure provably contribute to the escape process and, if so, what is the aggregate performance improvement of the momentum? We answer the first question in the affirmative, and we answer the second question essentially by showing that momentum can help speed up saddle-point escape by a multiplicative factor of 1 − β. On the negative side, we also show that β is constrained and may not be chosen arbitrarily close to 1. Let us now establish, empirically, the clear benefit of stochastic momentum on the problem of saddle-point escape. We construct two stochastic optimization tasks, and each exhibits at least one significant saddle point. The two objectives are as follows. Problem of these was considered by (; Reddi et al. (2018a) ) and represents a very straightforward non-convex optimization challenge, with an embedded saddle given by the matrix H:= diag([1, −0.1]), and stochastic gaussian perturbations given by b i ∼ N (0, diag([0.1, 0.001])); the small variance in the second component provides lower noise in the escape direction. Here we have set n = 10. Observe that the origin is in the neighborhood of saddle points and has objective We plot convergence in function value f (·) given in. Initialization is always set as w0 = 0. All the algorithms use the same step size η = 5 × 10 −5. Fig. 4b: We plot convergence in relative distance to the true model w *, defined as min(wt − w *, wt + w *)/ w *, which more appropriately captures progress as the global sign of the objective is unrecoverable. All the algorithms are initialized at the same point w0 ∼ N (0, I d /(10000d)) and use the same step size η = 5 × 10 −4. value zero. SGD and SGD with momentum are initialized at the origin in the experiment so that they have to escape saddle points before the convergence. The second objective appears in the phase retrieval problem, that has real applications in physical sciences (Candés et al.; ). In phase retrieval 3, one wants to find an unknown w * ∈ R d with access to but a few samples y i = (a i w *) 2; the design vector a i is known a priori. Here we have sampled The empirical findings, displayed in Figure 2, are quite stark: for both objectives, convergence is significantly accelerated by larger choices of β. In the first objective (Figure 4a), we see each optimization trajectory entering a saddle point region, apparent from the "flat" progress, yet we observe that large-momentum trajectories escape the saddle much more quickly than those with smaller momentum. A similar affect appears in Figure 4b. To the best of our knowledge, this is the first reported empirical finding that establishes the dramatic speed up of stochastic momentum for finding an optimal solution in phase retrieval. Heavy ball method: The heavy ball method was originally proposed by. It has been observed that this algorithm, even in the deterministic setting, provides no convergence speedup over standard gradient descent, except in some highly structure cases such as convex quadratic objectives where an "accelerated" rate is possible . We provide a comprehensive survey of the related works about heavy ball method in Appendix A. Reaching a second order stationary point: As we mentioned earlier, there are many works aim at reaching a second order stationary point. We classify them into two categories: specialized algorithms and simple GD/SGD variants. Specialized algorithms are those designed to exploit the negative curvature explicitly and escape saddle points faster than the ones without the explicit exploitation). Our work belongs to this category. In this category, perhaps the pioneer works are and . show that explicitly adding isotropic noise in each iteration guarantees that GD escapes saddle points and finds a second order stationary 3 It is known that phase retrieval is nonconvex and has the so-called strict saddle property: every local minimizer {w *, −w *} is global up to phase, each saddle exhibits negative curvature (see e.g. (Sun et al. (2015; ;)) point with high probability. Following , assume that stochastic gradient inherently has a component to escape. Specifically, they make assumption of the Correlated Negative Curvature (CNC) for stochastic gradient g t so that E t [g t, v t 2] ≥ γ > 0. The assumption allows the algorithm to avoid the procedure of perturbing the updates by adding isotropic noise. Our work is motivated by but assumes CNC for the stochastic momentum m t instead. In Appendix A, we compare the of our work with the related works. We assume that the gradient ∇f is L-Lipschitz; that is, f is L-smooth. Further, we assume that the Hessian ∇ 2 f is ρ-Lipschitz. These two properties ensure that ∇f w−w 3, ∀w, w. Furthermore, we assume that the stochastic gradient has bounded noise ∇f (w) − ∇f (w; ξ) 2 ≤ σ 2 and that the norm of stochastic momentum is bounded so that m t ≤ c m. We denote Π i M i as the matrix product of matrices {M i} and we use to denote the spectral norm of the matrix M. Our analysis of stochastic momentum relies on three properties of the stochastic momentum dynamic. These properties are somewhat unusual, but we argue they should hold in natural settings, and later we aim to demonstrate that they hold empirically in a couple of standard problems of interest. Definition 2. We say that SGD with stochastic momentum satisfies Almost Positively Aligned with Gradient (APAG) 4 if we have We say that SGD with stochastic momentum satisfies Almost Positively Correlated with Gradient (APCG) with parameter τ if ∃c > 0 such that, where the PSD matrix M t is defined as for any integer 1 ≤ k ≤ τ − 1, and η is any step size chosen that guarantees each G s,t is PSD. Definition 3. We say that the SGD with momentum exhibits Gradient Alignment or Curvature Exploitation (GrACE) if ∃c h ≥ 0 such that APAG requires that the momentum term m t must, in expectation, not be significantly misaligned with the gradient ∇f (w t). This is a very natural condition when one sees that the momentum term is acting as a biased estimate of the gradient of the deterministic f. APAG demands that the bias can not be too large relative to the size of ∇f (w t). Indeed this property is only needed in our analysis when the gradient is large (i.e. ∇f (w t) ≥ ) as it guarantees that the algorithm makes progress; our analysis does not require APAG holds when gradient is small. APCG is a related property, but requires that the current momentum term m t is almost positively correlated with the the gradient ∇f (w t), but measured in the Mahalanobis norm induced by M t. It may appear to be an unusual object, but one can view the PSD matrix M t as measuring something about the local curvature of the function with respect to the trajectory of the SGD with momentum dynamic. We will show that this property holds empirically on two natural problems for a reasonable constant c. APCG is only needed in our analysis when the update is in a saddle region with significant 2 versus iterations. For (b), we only report them when the gradient is large (∇f (wt) ≥ 0.02). It shows that the value is large than −0.5 except the transition. For (e), we observe that the value is almost always nonnegative. Sub-figures (c) and (f): We plot the value of Gs,t) and we only report the values when the update is in the region of saddle points. For (f), we let Mt = (Π 500 s=1 Gs,t)(Π 500 s=1 Gs,t) and we observe that the value is almost always nonnegative. The figures implies that SGD with momentum has APAG and APCG properties in the experiments. Furthermore, an interesting observation is that, for the phase retrieval problem, the expected values might actually be nonnegative. negative curvature, ∇f (w) ≤ and λ min (∇ 2 f (w)) ≤ −. Our analysis does not require APCG holds when the gradient is large or the update is at an (,)-second order stationary point. For GrACE, the first term on l.h.s of measures the alignment between stochastic momentum m t and the gradient ∇f (w t), while the second term on l.h.s measures the curvature exploitation. The first term is small (or even negative) when the stochastic momentum m t is aligned with the gradient ∇f (w t), while the second term is small (or even negative) when the stochastic momentum m t can exploit a negative curvature (i.e. the subspace of eigenvectors that corresponds to the negative eigenvalues of the Hessian ∇ 2 f (w t) if exists). Overall, a small sum of the two terms (and, consequently, a small c h) allows one to bound the function value of the next iterate (see Lemma 8). On Figure 3, we report some quantities related to APAG and APCG as well as the gradient norm when solving the previously discussed problems and using SGD with momentum. We also report a quantity regarding GrACE on Figure 4 in the appendix. The high level idea of our analysis follows as a similar template to (; ;). Our proof is structured into three cases: either (a) ∇f (w) ≥, or (b) ∇f (w) ≤ and λ min (∇ 2 f (w)) ≤ −, or otherwise (c) ∇f (w) ≤ and λ min (∇ 2 f (w)) ≥ −, meaning we have arrived in a second-order stationary region. The precise algorithm we analyze is Algorithm 2, which identical to Algorithm 1 except that we boost the step size to a larger value r on 4 Note that our analysis still go through if one replaces 1 2 on r.h.s. of with any larger number c < 1; the ed iteration complexity would be only a constant multiple worse. Step size parameters r and η, momentum parameter β, and period parameter T thred. 2: Init: Get stochastic gradient gt at wt, and set stochastic momentum mt:= βmt−1 + gt. Set learning rate:η:= η unless (t mod T thred) = 0 in which caseη:= r 6: wt+1 = wt −ηmt. 7: end for occasion. We will show that the algorithm makes progress in cases (a) and (b). In case (c), when the goal has already been met, further execution of the algorithm only weakly hurts progress. Ultimately, we prove that a second order stationary point is arrived at with high probability. While our proof borrows tools from , much of the momentum analysis is entirely novel to our knowledge. Theorem 1. Assume that the stochastic momentum satisfies CNC. Set, and ). If SGD with momentum (Algorithm 2) has APAG property when gradient is large (∇f (w) ≥ ), APCG T thred property when it enters a region of saddle points that exhibits a negative curvature (∇f (w) ≤ and λ min (∇ 2 f (w)) ≤ − ), and GrACE property throughout the iterations, then it reaches an (,) second order stationary point in ) iterations with high probability 1 − δ. The theorem implies the advantage of using stochastic momentum for SGD. Higher β leads to reaching a second order stationary point faster. As we will show in the following, this is due to that higher β enables escaping the saddle points faster. In Subsection 3.2.1, we provide some key details of the proof of Theorem 1. The interested reader can read a high-level sketch of the proof, as well as the detailed version, in Appendix G. Remark 1: (constraints on β) We also need some minor constraints on β so that β cannot be too close to 1. They are 1) On the other hand, for CNC-SGD, based on Table 3 in their paper, is T thred =Õ 1 η. One can clearly see that T thred of our has a dependency 1 − β, which makes it smaller than that of for any same η and consequently demonstrates escaping saddle point faster with momentum. Remark 3: (finding a second order stationary point) Denote a number such that ∀t, g t ≤. In Appendix G.3, we show that in the high momentum regime where, Algorithm 2 is strictly better than CNC-SGD of , which means that a higher momentum can help find a second order stationary point faster. Empirically, we find out that c ≈ 0 (Figure 3) and c h ≈ 0 (Figure 4) in the phase retrieval problem, so the condition is easily satisfied for a wide range of β. In this subsection, we analyze the process of escaping saddle points by SGD with momentum. Denote t 0 any time such that (t 0 mod T thred) = 0. Suppose that it enters the region exhibiting a small gradient but a large negative eigenvalue of the Hessian (i.e. ∇f (w t0) ≤ and λ min (∇ 2 f (w t0)) ≤ − ). We want to show that it takes at most T thred iterations to escape the region and whenever it escapes, the function value decreases at least by F thred = O on expectation, where the precise expression of F thred will be determined later in Appendix E. The technique that we use is proving by contradiction. Assume that the function value on expectation does not decrease at least F thred in T thred iterations. Then, we get an upper bound of the expected distance E t0 [w t0+T thred − w t0 2] ≤ C upper. Yet, by leveraging the negative curvature, we also show a lower bound of the form E t0 [w t0+T thred − w t0 2] ≥ C lower. The analysis will show that the lower bound is larger than the upper bound (namely, C lower > C upper), which leads to the contradiction and concludes that the function value must decrease at least F thred in T thred iterations on expectation. Since 6 ), the dependency on β suggests that larger β can leads to smaller T thred, which implies that larger momentum helps in escaping saddle points faster. Lemma 1 below provides an upper bound of the expected distance. The proof is in Appendix C. Lemma 1. Denote t 0 any time such that We see that C upper,t in Lemma 1 is monotone increasing with t, so we can define C upper:= C upper,T thred. Now let us switch to obtaining the lower bound of E t0 [w t0+T thred − w t0 2]. The key to get the lower bound comes from the recursive dynamics of SGD with momentum. Lemma 2. Denote t 0 any time such that (t 0 mod T thred) = 0. Let us define a quadratic ap- Then we can write w t0+t − w t0 exactly using the following decomposition. The proof of Lemma 2 is in Appendix D. Furthermore, we will use the quantities q v,t−1, q m,t−1, q q,t−1, q w,t−1, q ξ,t−1 as defined above throughout the analysis. Lemma 3. Following the notations of Lemma 2, we have that We are going to show that the dominant term in the lower bound of, which is the critical component for ensuring that the lower bound is larger than the upper bound of the expected distance. Following the conditions and notations in Lemma 1 and Lemma 2, we have that Proof. We know that λ min (H) ≤ − < 0. Let v be the eigenvector of the Hessian H with unit norm that corresponds to λ min (H) so that Hv = λ min (H)v. We have where (a) is because v is with unit norm, (b) is by Cauchy-Schwarz inequality, (c), (d) are by the definitions, and (e) is by the CNC assumption so that Observe that the lower bound in is monotone increasing with t and the momentum parameter β. Moreover, it actually grows exponentially in t. To get the contradiction, we have to show that the lower bound is larger than the upper bound. By Lemma 1 and Lemma 3, it suffices to prove the following lemma. We provide its proof in Appendix E. Lemma 5. Let F thred = O and η 2 T thred ≤ r 2. By following the conditions and notations in Theorem 1, Lemma 1 and Lemma 2, we conclude that if SGD with momentum (Algorithm 2) has the APCG property, then we have that In this paper, we identify three properties that guarantee SGD with momentum in reaching a secondorder stationary point faster by a higher momentum, which justifies the practice of using a large value of momentum parameter β. We show that a greater momentum leads to escaping strict saddle points faster due to that SGD with momentum recursively enlarges the projection to an escape direction. However, how to make sure that SGD with momentum has the three properties is not very clear. It would be interesting to identify conditions that guarantee SGD with momentum to have the properties. Perhaps a good starting point is understanding why the properties hold in phase retrieval. We believe that our shed light on understanding the recent success of SGD with momentum in non-convex optimization and deep learning. Heavy ball method: The heavy ball method was originally proposed by. It has been observed that this algorithm, even in the deterministic setting, provides no convergence speedup over standard gradient descent, except in some highly structure cases such as convex quadratic objectives where an "accelerated" rate is possible . In recent years, some works make some efforts in analyzing heavy ball method for other classes of optimization problems besides the quadratic functions. provide a unified analysis of stochastic heavy ball momentum and Nesterov's momentum for smooth non-convex objective functions. They show that the expected gradient norm converges at rate O(1/ √ t). Yet, the rate is not better than that of the standard SGD. We are also aware of the works (Ghadimi & Lan (2016; 2013) ), which propose some variants of stochastic accelerated algorithms with first order stationary point guarantees. Yet, the framework in (Ghadimi & Lan (2016; 2013) ) does not capture the stochastic heavy ball momentum used in practice. There is also a negative about the heavy ball momentum. show that for a specific strongly convex and strongly smooth problem, SGD with heavy ball momentum fails to achieving the best convergence rate while some algorithms can. Reaching a second order stationary point: As we mentioned earlier, there are many works aim at reaching a second order stationary point. We classify them into two categories: specialized algorithms and simple GD/SGD variants. Specialized algorithms are those designed to exploit the negative curvature explicitly and escape saddle points faster than the ones without the explicit exploitation), in which the gradient g t is multiplied by a preconditioning matrix G t and the update is w t+1 = w t − G −1/2 t g t. The work shows that the algorithm can help in escaping saddle points faster compared to the standard SGD under certain conditions. propose average-SGD, in which a suffix averaging scheme is conducted for the updates. They also assume an inherent property of stochastic gradients that allows SGD to escape saddle points. We summarize the iteration complexity of the related works for simple SGD variants on Table 1. 6 The readers can see that the iteration complexity of and our . So, we want to explain the and clarify the differences. First, we focus on explaining why the popular algorithm, SGD with heavy ball momentum, works well in practice, which is without the suffix averaging scheme used in and is without the explicit perturbation used in . Specifically, we focus on studying the effect of stochastic heavy ball momentum and showing the advantage of using it. Furthermore, our analysis framework is built on the work of . We believe that, based on the insight in our work, one can also show the advantage of stochastic momentum by modifying the assumptions and algorithms in or and consequently get a better dependency on. B LEMMA 6, 7, AND 8 In the following, Lemma 7 says that under the APAG property, when the gradient norm is large, on expectation SGD with momentum decreases the function value by a constant and consequently makes progress. On the other hand, Lemma 8 upper-bounds the increase of function value of the next iterate (if happens) by leveraging the GrACE property. Lemma 6. If SGD with momentum has the APAG property, then, considering the update step w t+1 = w t − ηm t, we have that Proof. By the L-smoothness assumption, Taking the expectation on both sides. We have where we use the APAG property in the last inequality. Lemma 7. Assume that the step size η satisfies η ≤ 2 8Lc 2 m. If SGD with momentum has the APAG property, then, considering the update step w t+1 = w t − ηm t, we have that 2, where the last inequality is due to the constraint of η. Lemma 8. If SGD with momentum has the GrACE property, then, considering the update step w t+1 = w t − ηm t, we have that Proof. Consider the update rule w t+1 = w t − ηm t, where m t represents the stochastic momentum and η is the step size. By ρ-Lipschitzness of Hessian, we have Taking the conditional expectation, one has C PROOF OF LEMMA 1 Lemma 1 Denote t 0 any time such that (t 0 mod T thred) = 0. Suppose that Proof. Recall that the update is w t0+1 = w t0 − rm t0, and w t0+t = w t0+t−1 − ηm t0+t−1, for t > 1. We have that where the first inequality is by the triangle inequality and the second one is due to the assumption that m t ≤ c m for any t. Now let us denote s=1 α s and let us rewrite g t = ∇f (w t) + ξ t, where ξ t is the zero-mean noise. We have that To proceed, we need to upper bound E t0 [4η We have that where (a) is because E t0 [ξ t0+i ξ t0+j] = 0 for i = j, (b) is by that ξ t 2 ≤ σ 2 and max t α t ≤ 1 1−β. Combining,,,, Now we need to bound E t0 [. By using ρ-Lipschitzness of Hessian, we have that By adding η ∇f (w t0+s−1), g t0+s−1 on both sides, we have η ∇f (w t0+s−1), g t0+s−1 ≤ f (w t0+s−1) − f (w t0+s) + η ∇f (w t0+s−1), g t0+s−1 − m t0+s−1 Taking conditional expectation on both sides leads to D PROOF OF LEMMA 2 AND LEMMA 3 Lemma 2 Denote t 0 any time such that (t 0 mod T thred) = 0. Let us define a quadratic approximation at w t0, Q(w):= f (w t0) + w − w t0, ∇f (w t0) + 1 2 (w − w t0) H(w − w t0), where • qv,t−1:= Π t−1 j=1 Gj − rmt 0. • qm,t−1:= − t−1 s=1 Π t−1 j=s+1 Gj β s mt 0. • qq,t−1: • qw,t−1: • q ξ,t−1: Then, w t0+t − w t0 = q v,t−1 + ηq m,t−1 + ηq q,t−1 + ηq w,t−1 + ηq ξ,t−1. Denote t 0 any time such that (t 0 mod T thred) = 0. Let us define a quadratic approximation at w t0, where H:= ∇ 2 f (w t0). Also, we denote Proof. First, we rewrite m t0+j for any j ≥ 1 as follows. We have that where (a) is by using with j = t − 1, (b) is by subtracting and adding back the same term, and (c) is by ∇Q(w t0+t−1) = ∇f (w t0) + H(w t0+t−1 − w t0). To continue, by using the nations in, we can rewrite as Recursively expanding leads to where (a) we use the notation that Π t−1 and the notation that Π t−1 j=t G j = 1 and (b) is by the update rule. By using the definitions of {q,t−1} in the lemma statement, we complete the proof. Lemma 3 Following the notations of Lemma 2, we have that Proof. Following the proof of Lemma 2, we have Therefore, by using a + b 2 ≥ a 2 + 2 a, b, E PROOF OF LEMMA 5 Lemma 5 Let F thred = O and η 2 T thred ≤ r 2. By following the conditions and notations in Theorem 1, Lemma 1 and Lemma 2, we conclude that if SGD with momentum (Algorithm 2) has the APCG property, then we have that, cr ≤,,, from " ) from W.l.o.g, we assume that c m, L, σ 2, c, c h, and ρ are not less than one and that ≤ 1. E.1 SOME CONSTRAINTS ON β. We require that parameter β is not too close to 1 so that the following holds, • 2) σ 2 (1 − β) 3 > 1. • 3) c (1 − β) 2 > 1. • The constraints upper-bound the value of β. That is, β cannot be too close to 1. We note that the β dependence on L, σ, and c are only artificial. We use these constraints in our proofs but they are mostly artefacts of the analysis. For example, if a function is L-smooth, and L < 1, then it is also 1-smooth, so we can assume without loss of generality that L > 1. Similarly, the dependence on σ is not highly relevant, since we can always increase the variance of the stochastic gradient, for example by adding an O gaussian perturbation. To prove Lemma 5, we need a series of lemmas with the choices of parameters on Table 3. Upper bounding E t0 [q q,t−1]: Lemma 9. Following the conditions in Lemma 1 and Lemma 2, we have Proof. where (a), (c), (d) is by triangle inequality, (b) is by the fact that Ax 2 ≤ A 2 x 2 for any matrix A and vector x. Now that we have an upper bound of ∇f (w t0+k) − ∇f (w t0+s), where (a) is by the assumption of L-Lipschitz gradient and (b) is by applying the triangle inequality (s − k) times and that w t − w t−1 ≤ η m t−1 ≤ ηc m, for any t. We can also derive an upper bound of E t0 [∇f (w t0+s) − ∇Q(w t0+s) ], Above, (a) is by the fact that if a function f (·) has ρ Lipschitz Hessian, then (c.f. Lemma 1.2.4 in ) and using the definition that Combing,,, we have that where on the last line we use the notation that To continue, let us analyze Π t−1 j=s+1 G j 2 first. Above, we use the notation that θ j:= j k=1 β j−k. For (a), it is due to that λ:= −λ min (H), λ max (H) ≤ L, and the choice of η so that 1 ≥ ηL 1−β, or equivalently, For (b), it is due to that θ j ≥ 1 for any j and λ ≥. Therefore, we can upper-bound the first term on r.h.s of as where (a) is by that fact that is by using, and (c) is by using that η. Now let us switch to bound where (a) is by the fact that, (c) is by using that (1−z) 2 for any |z| ≤ 1 and substituting z = 1 1+η, which leads to in which the last inequality is by chosen the step size η so that η ≤ 1. By combining,, and, we have that Proof. where the last inequality is because η is chosen so that 1 ≥ ηL 1−β and the fact that λ max (H) ≤ L. Lower bounding E t0 [2η q v,t−1, q q,t−1]: Lemma 11. Following the conditions in Lemma 1 and Lemma 2, we have Proof. By the of Lemma 9 and Lemma 10 Under review as a conference paper at ICLR 2020 Lower bounding E t0 [2η q v,t−1, q ξ,t−1]: Lemma 12. Following the conditions in Lemma 1 and Lemma 2, we have Proof. where (a) holds for some coefficients α k, (b) is by the tower rule, (c) is because q v,t−1 is measureable with t 0, and (d) is by the zero mean assumption of ξ's. Lower bounding E t0 [2η q v,t−1, q m,t−1]: Lemma 13. Following the conditions in Lemma 1 and Lemma 2, we have Proof. where (a) is by defining the matrix B: For (b), notice that the matrix B is symmetric positive semidefinite. To see that the matrix B is symmetric positive semidefinite, observe that each G j:= (I − η j k=1 β j−k H) can be written in the form of G j = U D j U for some orthonormal matrix U and a diagonal matrix D j. Therefore, the matrix product U is symmetric positive semidefinite as long as each G j is. So, (b) is by the property of a matrix being symmetric positive semidefinite. Lower bounding 2ηE t0 [q v,t−1, q w,t−1]: Lemma 14. Following the conditions in Lemma 1 and Lemma 2, if SGD with momentum has the APCG property, then Proof. Define where (a) is by the APCG property. We also have that where (a) and (b) is by. Substituting the back to, we get Using the fact that ∇f (w t0) ≤ completes the proof. Recall that the strategy is proving by contradiction. Assume that the function value does not decrease at least F thred in T thred iterations on expectation. Then, we can get an upper bound of the expected distance E t0 [w t0+T thred − w t0 2] ≤ C upper but, by leveraging the negative curvature, we can also show a lower bound of the form E t0 [w t0+T thred − w t0 2] ≥ C lower. The strategy is showing that the lower bound is larger than the upper bound, which leads to the contradiction and concludes that the function value must decrease at least F thred in T thred iterations on expectation. To get the contradiction, according to Lemma 1 and Lemma 3, we need to show that Yet, by Lemma 13 and Lemma 12, we have that ηE t0 [q v,T thred −1, q m,T thred −1] ≥ 0 and ηE t0 [q v,T thred −1, q ξ,T thred −1] = 0. So, it suffices to prove that and it suffices to show that Under review as a conference paper at ICLR 2020 as guaranteed by the constraint of η. So, where (a) is by using the inequality log(1 + x) ≥ x 2 with x = ηθ j ≤ 1 and (b) is by making (1−β) 2, which is equivalent to the condition that Now let us substitute the of back to. We have that which is what we need to show. By choosing T thred large enough, for some constant c > 0, we can guarantee that the above inequality holds. Lemma 15 Let us define the event Set T = 2T thred f (w 0) − min w f (w) /(δ∆). We return w uniformly randomly from w 0, w T thred, w 2T thred,..., w kT thred,..., w KT thred, where K:= T /T thred. Then, with probability at least 1 − δ, we will have chosen a w k where Υ k did not occur. Proof. Let P k be the probability that Υ k occurs. Summing over all K, we have −β) ) −6 ). If SGD with momentum (Algorithm 2) has APAG property when gradient is large (∇f (w) ≥ ), APCG T thred property when it enters a region of saddle points that exhibits a negative curvature (∇f (w) ≤ and λ min (∇ 2 f (w)) ≤ − ), and GrACE property throughout the iterations, then it reaches an (,) second order stationary point in T = O((1 − β) log(Lcmσ 2 ρc c h (1−β)δγ ) −10 ) iterations with high probability 1 − δ. In this subsection, we provide a sketch of the proof of Theorem 1. The complete proof is available in Appendix G. Our proof uses a lemma in , which is Lemma 15 below. The lemma guarantees that uniformly sampling a w from {w kT thred}, k = 0, 1, 2,..., T /T thred gives an (,)-second order stationary point with high probability. We replicate the proof of Lemma 15 in Appendix F. We return w uniformly randomly from w 0, w T thred, w 2T thred,..., w kT thred,..., w KT thred, where K:= T /T thred. Then, with probability at least 1 − δ, we will have chosen a w k where Υ k did not occur. To use the of Lemma 15, we need to let the conditions in be satisfied. We can bound E[f (w (k+1)T thred ) − f (w kT thred)|Υ k ] ≤ −F thred, based on the analysis of the large gradient norm regime (Lemma 7) and the analysis for the scenario when the update is with small gradient norm but a large negative curvature is available (Subsection 3.2.1). For the other condition, E[f (w (k+1)T thred ) − f (w kT thred)|Υ, it requires that the expected amortized increase of function value due to taking the large step size r is limited (i.e. bounded by δ) when w kT thred is a second order stationary point. By having the conditions satisfied, we can apply Lemma 15 and finish the proof of the theorem. Proof. Our proof is based on Lemma 15. So, let us consider the events in Lemma 15, Υ k:= {∇f (w kT thred) ≥ or λ min (∇ 2 f (w kT thred)) ≤ − }. We first show that E[f (w (k+1)T thred ) − f (w kT thred)|Υ k ] ≤ F thred. When ∇f (w kT thred) ≥: Now we are ready to use Lemma 15, since both the conditions are satisfied. According to the lemma and the choices of parameters value on Table 3, we can set T = 2T thred f (w 0) − min w f (w) /(δF thred) = O((1 − β) log(Lcmσ 2 ρc c h (1−β)δγ ) −10 ), which will return a w that is an (,) second order stationary point. Thus, we have completed the proof. (1−β)δγ ) −10 ) for Algorithm 2. Before making a comparison, we note that their does not have a dependency on the variance of stochastic gradient (i.e. σ 2), which is because they assume that the variance is also bounded by the constant (can be seen from, Algorithm 2 is strictly better than that of , which means that a higher momentum can help to find a second order stationary point faster.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Higher momentum parameter $\beta$ helps for escaping saddle points faster

represent the tldr for finding the corresponding scientific article

We investigate the convergence of popular optimization algorithms like Adam , RMSProp and propose new variants of these methods which provably converge to optimal solution in convex  settings.