Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The complex world around us is inherently multimodal and sequential (continuous). Information is scattered across different modalities and requires multiple continuous sensors to be captured. As machine learning leaps towards better generalization to real world, multimodal sequential learning becomes a fundamental research area. Arguably, modeling arbitrarily distributed spatio-temporal dynamics within and across modalities is the biggest challenge in this research area. In this paper, we present a new transformer model, called the Factorized Multimodal Transformer (FMT) for multimodal sequential learning. FMT inherently models the intramodal and intermodal (involving two or more modalities) dynamics within its multimodal input in a factorized manner. The proposed factorization allows for increasing the number of self-attentions to better model the multimodal phenomena at hand; without encountering difficulties during training (e.g. overfitting) even on relatively low-resource setups. All the attention mechanisms within FMT have a full time-domain receptive field which allows them to asynchronously capture long-range multimodal dynamics. In our experiments we focus on datasets that contain the three commonly studied modalities of language, vision and acoustic. We perform a wide range of experiments, spanning across 3 well-studied datasets and 21 distinct labels. FMT shows superior performance over previously proposed models, setting new state of the art in the studied datasets. In many naturally occurring scenarios, our perception of the world is multimodal. For example, consider multimodal language (face-to-face communication), where modalities of language, vision and acoustic are seamlessly used together for communicative intent . Such scenarios are widespread in everyday life, where continuous sensory perceptions form multimodal sequential data. Each modality within multimodal data exhibits exclusive intramodal dynamics, and presents a unique source of information. Modalities are not fully independent of each other. Relations across two (bimodal) or more (trimodal, . . .) of them form intermodal dynamics; often asynchronous spatio-temporal dynamics which bind modalities together. Learning from multimodal sequential data has been an active, yet challenging research area within the field of machine learning (Baltrušaitis et al., 2018). Various approaches relying on graphical models or RNNs have been proposed for multimodal sequential learning. Transformer models are a new class of neural models that rely on a carefully designed non-recurrent architecture for sequential modeling . Their superior performance is attributed to a self-attention mechanism, which is uniquely capable of highlighting related information across a sequence. This self-attention is a particularly appealing mechanism for multimodal sequential learning, as it can be modified into a strong neural component for finding relations between different modalities (the cornerstone of this paper). In practice, numerous such relations may simultaneously exist within multimodal data, which would require increasing the number of attention units (i.e. heads). Increasing the number of attentions in an efficient and semantically meaningful way inside a transformer model, can boost the performance in modeling multimodal sequential data. In this paper, we present a new transformer model for multimodal sequential learning, called Factorized Multimodal Transformer (FMT). FMT is capable of modeling asynchronous intramodal and intermodal dynamics in an efficient manner, within one single transformer network. It does so by specifically accounting for possible sets of interactions between modalities (i.e. factorizing based on combinations) in a Factorized Multimodal Self-attention (FMS) unit. We evaluate the performance of FMT on multimodal language: a challenging type of multimodal data which exhibits idiosyncratic and asynchronous spatio-temporal relations across language, vision and acoustic modalities. FMT is compared to previously proposed approaches for multimodal sequential learning over multimodal sentiment analysis (CMU-MOSI) , multimodal emotion recognition (IEMOCAP) , and multimodal personality traits recognition (POM) . The related works to studies in this paper fall into two main areas. Modeling multimodal sequential data is among the core research areas within the field of machine learning. In this area, previous work can be classified into two main categories. The first category of models, and arguably the simplest, are models that use early or late fusion. Early fusion uses feature concatenation of all modalities into a single modality. Subsequently, the multimodal sequential learning task is treated as a unimodal one and tackled using unimodal sequential models such as Hidden Markov Models (HMMs) , Hidden Conditional Random Fields (HCRFs), and RNNs (e.g.). While such models are often successful for real unimodal data (i.e. not feature concatenated multimodal data), they lack the necessary components to deal with multimodal data often causes suboptimal performance . Contrary to early fusion which concatenates modalities at input level, late fusion models have relied on learning ensembles of weak classifiers from different modalities (; ;). Hybrid methods have also been used to combine early and late fusion together (; ; ;). The second category of models comprise of models specifically designed for multimodal data. Multimodal variations of graphical models have been proposed, including Multi-view HCRFs where the potentials of the HCRF are changed to facilitate multiple modalities Song et al. (2012; . Multimodal models based on LSTMs include , Memory Fusion Network (a) with its recurrent and graph variants c), as well as Multi-attention Recurrent Networks (b). Studies have also proposed generic fusion techniques that can be used in various models including Tensor Fusion and its approximate variants (Liang et al.;, as well as Compact Bilinear Pooling (; ;). Many of these models, from both first and second categories, are used as baselines in this paper. Transformer is a non-recurrent neural architecture designed for modeling sequential data . It has shown superior performance across multiple NLP tasks when compared to RNN-based or convolutional architectures . This superior performance of Transformer model is largely credited to a self-attention; a neural component that allows for efficiently extracting both short and long-range dependencies within its input sequence space. Transformer models have been successfully applied to various areas within machine learning including NLP and computer vision (; ;). Extending transformer to multimodal domains, specially for structured multimodal sequences is relatively understudied; with the previous works mainly focusing on using transformer models for modality alignment using cross-modal links between single transformers for each modality . In this section, we outline the proposed Factorized Multimodal Transformer 1 (FMT). Figure 1 shows the overall structure of the FMT model. The input first goes through an embedding layer, followed by multiple Multimodal Transformer Layers (MTL). Each MTL consists of multiple Factorized Multimodal Self-attentions (FMS). FMS explicitly accounts for intramodal and intermodal factors within its multimodal input. S1 and S2 are two summarization networks. They are necessary components of FMT which allow for increasing the number of attentions efficiently, without overparameterization of the FMT. Layer (MTL) Consider a multimodal sequential dataset with constituent modalities of language, vision and acoustic. The modalities are denoted as {L, V, A} from hereon for abbreviation. After resampling using a reference clock, modalities can follow the same frequency. Essentially, this resampling is often based on word timestamps (i.e. word alignment). Subsequently, the dataset can be denoted as: x i ∈ R Ti×dx, y i ∈ R dy are the inputs and labels. is a triplet of language, visual and audio inputs for timestamp t in i-th datapoint. N is the total number of samples within the dataset, and T i the total number of timestamps within i-th datapoint. Zero paddings (on the left) can be used to unify the length of all sequences to a desired fixed length A denotes the dimensionality of input at each timestep, which in turn is equal to the sum of dimensionality of each modality. d y denotes the dimensionality of the associated labels of a sequence. At the first step within the FMT model, each modality is passed to a unimodal embedding layer with the operation Positional embeddings are also added to the input at this stage. The output of the embeddings collectively form We denote the dimensionality of this output as e x = e L + e V + e A. After the initial embedding, FMT now consists of a stack of Multimodal Transformer Layers (MTL). MTL 1) captures factorized dynamics within multimodal data in parallel, and 2) aligns the timeasynchronous information both within and across modalities. Both of these are achieved using multiple Factorized Multimodal Self-attentions (FMS), each of which has multiple specialized selfattentions inside. The high dimensionality of the intermediate attention outputs within MTL and FMS is controlled using two distinct summarization networks. The continuation of this section provides detailed explanation of the inner-operations of MTL. ·,i) denote the input to the k-th MTL. We assume a total of K MTLs in a FMT (indexed 0 . . . K − 1), with k = 0 being the output of the embedding layer (input to k = 0 MTL). The input of MTL, immediately goes through one/multiple 2 Factorized 1 Code: github.com/removed-for-blind-review, Public Data: https://github.com/A2Zadeh/CMUMultimodalSDK 2 Multiple FMS have the same time-domain receptive field, which is equal to the length of the input. This is contrary to the implementations of the transformer model that split the sequence based on number of attention heads. Output (The grayed areas are for demonstration purposes, and not a part of the implementation. Figure 2 . For 3 modalities 3, there exist 7 distinct attentions inside a single FMS unit. Each attention has a unique receptive field with respect to modalities f ∈ F = {L,V,A,LV,LA,VA,LVA}; essentially denoting the modalities visible to the attention. Using this factorization, FMS explicitly accounts for possible unimodal, bimodal and trimodal interactions existing within the multimodal input space. All attentions within a FMS extend to the length of the sequence, and therefore can extract asynchronous relations within and across modalities. For f ∈ F, each attention within a single FMS unit is controlled by the Key K f, Query Q f, and Value V f all with dimensionality R T ×T; parameterized respectively using affine maps W K f, W Q f, and W V f. After the attention is applied using Key, Query and Value operations , the output of each of the attentions goes through a residual addition with its perceived input (input in the attention receptive field), followed by a normalization. The output of the FMS contains the aligned and extracted information from the unimodal, bimodal and trimodal factors. This output is high-dimensional; essentially R 4×T ×ex (each dimension within input of shape T ×e x is present in 4 factors). Our goal is to reduce this high-dimensional data using a mapping from R 4×T ×ex → R T ×ex. Without overparameterizing the FMS, in practice, we observed this mapping can be efficiently done using a simple 1D convolutional network S1 M∈{L,V,A} (·); R 4 → R. Internally, S1(·) maps its input to multiple layers of higher dimensions and subsequently to R. Using language as an example, S1 L moves across language modality dimensions e L for t = 1... T and summarizes the information across all the factors. The output of this summarization applied on all modality dimensions and timesteps, is the output of FMS, which has the dimensionality R T ×ex. In practice, there can be various possible unimodal, bimodal or trimodal interactions within a multimodal input. For example, consider multiple sets of important interactions between L and V (e.g. smile + positive word, as well as eyebrows up + excited phrase), all of which need to be highlighted and extracted. A single FMS may not be able to highlight all these interactions without diluting its intrinsic attentions. Multiple FMS can be used inside a MTL to efficiently extract diverse multimodal interactions existing in the input data 4. Consider a total of U FMS units inside a MTL. The output of each FMS goes through a feedforward network (for each timestamp t of the FMS output). 73.9/-73.4/-1.040 0.633 MARN (b) 77.1/-77.0/-0.968 0.625 MFN (a) 77.4/-77.3/-0.965 0.632 RMFN 78.4/-78.0/-0.922 0.681 RAVEN 78.0/--/-0.915 0.691 MulT -/83.0 -/82.8 0.87 0.698 FMT (ours) 81.5/83.5 81.4/83.5 0.837 0.744 Table 1: FMT achieves superior performance over baseline models for CMU-MOSI dataset (multimodal sentiment analysis). We report BA (binary accuracy) and F1 (both higher is better), MAE (Mean-absolute Error, lower is better), and Corr (Pearson Correlation Coefficient, higher is better). For BA and F1, we report two numbers: the number on the left side of "/" is calculated based on approach taken by Zadeh et al. (2018b), and the right side is by. The output of this feedfoward network is residually added with its input, and subsequently normalized. The feedforward network is the same across all U FMS units and timestamps t. Subsequently, the dimensionality of the output of the normalizations collectively is R U ×T ×ex. Similar to operations performed by S1, a secondary summarization network S2 M∈{L,V,A} (·); R U → R can be used here. S2 is also a 1D convolutional network that moves across modality dimensions and different timesteps to map R U ×T ×ex to R T ×ex. The output of the secondary summarization network is the final output of MTL, and denoted asx be the output of last MTL in the stack. For supervision, we feed this input one timestamp at a time as input to a Gated Recurrent Unit (GRU) . The prediction is conditioned on output at timestamp t = T of the GRU, using an affine map to d y. In this section, we discuss the experimental methodology including tasks, datasets, computational descriptors, and comparison baselines. The following inherently multimodal tasks (and accompanied datasets) are studied in this paper. All the tasks are related to multimodal language: a complex and idiosyncratic sequential multimodal signal, where semantics are arbitrarily scattered across modalities . The first benchmark in our experiments is multimodal sentiment analysis, where the goal is to identify a speaker's sentiment based on the speaker's display of verbal and nonverbal behaviors. We use the well-studied CMU-MOSI (CMU Multimodal Opinion Sentiment Intensity) dataset for this purpose . There are a total of 2199 data points (opinion utterances) within CMU-MOSI dataset. The dataset has real-valued sentiment intensity annotations in the range [−3, +3]. It is considered a challenging dataset due to speaker diversity (1 video per distinct speaker), topic variations and low-resource setup. The second benchmark in our experiments is multimodal emotion recognition, where the goal is to identify a speaker's emotions based on the speaker's verbal and nonverbal behaviors. We use the well-studied IEMOCAP dataset . IEMOCAP consists of 151 sessions of recorded dialogues, of which there are 2 speaker's per session for a total of 302 videos across the dataset. We perform experiments for discrete emotions of Happy, Sad, Angry and Neutral (no emotions) -similar to previous works (; 24.1 31.0 31.5 34.5 24.6 25.6 27.6 29.1 MARN (b) 29.1 33.0 --31.5 ---MFN (a) 34.5 35.5 37.4 41.9 34.5 36.9 36.0 37.9 RMFN 37.4 38.4 37.4 -37.4 38.9 38.9 -MulT 34.5 34.5 36.5 38.9 37.4 36.9 37. 30.5 38.9 35.5 37.4 33.0 42.4 27.6 33.0 MARN (b) 36.9 -52.2 --47.3 31.0 44.8 MFN (a) 38. Table 3: FMT achieves superior performance over baseline models in POM dataset (multimodal personality traits recognition). For label abbreviations please refer to Section 4.3. MA denotes multi-class accuracy for-class personality labels (higher is better). The third benchmark in our experiments is speaker trait recognition based on communicative behavior of a speaker. It is a particularly difficult task, with 16 different speaker traits in total. We study the POM dataset which contains 1,000 movie review videos . Each video is annotated for various personality and speaker traits, specifically: Confident (Con), Passionate (Pas), Voice Pleasant (Voi), Dominant (Dom), Credible (Cre), Vivid (Viv), Expertise (Exp), Entertaining (Ent), Reserved (Res), Trusting (Tru), Relaxed (Rel), Outgoing (Out), Thorough (Tho), Nervous (Ner), Persuasive (Per) and Humorous (Hum). The short form of these speaker traits is indicated inside the parentheses and used for the rest of this paper. The following computational descriptors are used by FMT and baselines (all the baselines use the same descriptors in their original respective papers). Language: P2FA forced alignment model is used to align the text and audio at word level. From the forced alignment, the timing of words and sentences are extracted. Word-level alignment is used to unify the modality frequencies. GloVe embeddings are subsequently used for word representation. Visual: For the visual modality, the Emotient FACET (iMotions, 2017) is used to extract a set of visual features including Facial Action Units , visual indicators of emotions, and sparse facial landmarks. Acoustic: COVAREP is used to extract the following features: fundamental frequency, quasi open quotient , normalized amplitude quotient, glottal source parameters (H1H2, Rd, Rd conf) , Voiced/Unvoiced segmenting features (VUV) , maxima dispersion quotient (MDQ), the first 3 formants, parabolic spectral parameter (PSP), harmonic model and phase distortion mean (HMPDM 0-24) and deviations (HMPDD 0-12), spectral tilt/slope of wavelet responses (peak/slope), Mel Cepstral Coefficients (MCEP 0-24). The following strong baselines are compared to FMT: MV-LSTM . There are fundamental distinctions between FMT and MulT, chief among them: 1) MulT consists of 6 transformers, 3 cross-modal transformers and 3 unimodal. Naturally this increases the overall model size substantially. FMT consists of only one transformer, with components to avoid overparameterization. 2) FMT sees interactions as undirected (unlike MulT which has L → V and V → L), and therefore semantically combines two attentions in one. 3) MulT has no trimodal factors (which are important according to Section 5). 4) MulT has no direct unimodal path (e.g. only L), as input to unimodal transformers are outputs of cross-modal transformers. 5) All FMT attentions have full time-domain receptive field, while MulT splits the input based on the heads. In their original publication, all the models report 6 the performance over the datasets in Section 4.1, using the same descriptors discussed in Section 4.2. The models in this paper are compared using the following performance measures (depending on the dataset): (BA) denotes binary accuracyhigher is better, (MA5,MA7) are 5 and 7 multiclass accuracy -higher is better, (F1) denotes F1 score -higher is better, (MAE) denotes the Mean-Absolute Error -lower is better, (Corr) is Pearson Correlation Coefficient -higher is better. The hyperparameter space search for FMT (and baselines if retrained) is discussed in Appendix A.1. The of sentiment analysis experiments on CMU-MOSI dataset are presented in Table 1. FMT achieves superior performance than the previously proposed models for multimodal sentiment analysis. We use two approaches for calculating BA and F1 based on negative vs. non-negative sentiment (b) on the left side of /, and negative vs. positive on the right side. MAE and Corr are also reported. For multimodal emotion recognition, experiments on IEMOCAP are reported in Table 2. The performance of FMT is superior than other baselines for multimodal emotion recognition (with the exception of Happy emotion). The of experiments for personality traits recognition on POM dataset are reported in Table 3. We report MA5 and MA7, depending on the label. FMT outperforms baselines across all personality traits. We study the importance of the factorization in FMT. We first remove the unimodal, bimodal and trimodal attentions from the FMT model, ing in 3 alternative implementations of FMT. demonstrates the of this ablation experiment over CMU-MOSI dataset. Furthermore, we use only one modality as input for FMT, to understand the importance of each modality (all other factors removed). We also replace the summarization networks with simple vector addition operation. All factors, modalities, and summarization components are needed for achieving best performance. We also perform experiments to understand the effect of number of FMT units within each MTL. Table 5 shows the performance trend for different number of FMT units. The model with 6 number of FMS (42 attentions in total) achieves the highest performance (6 is also the highest number we experimented with). reports the best performance for CMU-MOSI dataset is achieved when using 40 attentions per cross-modal transformer (3 of each, therefore 120 attention, without counting the subsequent unimodal transformers). FMT uses fewer number of attentions than MulT, yet achieves better performance. We also experiment with number of heads for original transformer model and compare to FMT (Appendix A.3). In this paper, we presented the Factorized Multimodal Transformer (FMT) model for multimodal sequential learning. Using a Factorized Multimodal Self-attention (FMS) within each Multimodal Transformer Layer (MTL), FMT is able to model the intra-model and inter-modal dynamics within asynchronous multimodal sequences. We compared the performance of FMT to baselines approaches over 3 publicly available datasets for multimodal sentiment analysis (CMU-MOSI, 1 label), emotion recognition (IEMOCAP, 4 labels) and personality traits recognition (POM, 16 labels). Overall, FMT achieved superior performance than previously proposed models across the studied datasets. A APPENDIX The hyperparameters of FMT include the Adam learning rate ({0.001, 0.0001}), structure of summarization network (randomly picked 5 architectures from {1, 2, 3} layers of conv, with kernel shapes of {2, 5, 10, 15, 20}), number of MTL layers ({4, 6, 8} except for ablation experiments which was 2... 8), number of FMT units ({4, 6}, except for ablation experiment which was 1... 6), e M ∈{L,V,A} ({20, 40}), dropout (0, 0.1). The same parameters (when applicable) are used for training MulT for POM dataset (e.g. num encoder layers same as number of MTL). Furthermore, for MulT specific hyperparameters, we use similar values as Table 5 in the original paper. All models are trained for a maximum of 200 epochs. The hyperparameter validation is similar to Zadeh et al. (2018b). We study the effect of number of MTL on FMT performance. Table 6 shows the of this experiment. The best performance is achieved using 8 MTL layers (which was also the maximum layers we tried in our hyperparameter search). In this section, we discuss the effect of increasing the number of heads on the original transformer model . Please note that we implement the OTF to allow for all attention heads to have full input receptive field (from 1 . . . T), similar to FMT. We increase the attention heads from 1 to 35 (after 35 does not fit on a Tesla-V100 GPU with batchsize of 20). Table 7 shows the of increasing number of attention heads for both models. We observe that achieving superior performance is not a matter of increasing the attention heads. Even using 1 FMS unit, which leads to 7 total attention, FMT achieves higher performance than counterpart OTF. In many scenarios in nature, as well as what is currently pursued in machine learning, the number of modalities goes as high as 3 (mostly language, vision and acoustic, as studied in this paper). This leads to 7 attentions within each FMS, well manageable for successful training of FMT as demonstrated in this paper. However, as the number of modalities increases, the underlying multimodal phenomena becomes more challenging to model. This causes complexities for any competitive multimodal model, regardless of their internal design. While studying these cases are beyond the scope of this paper, due to rare nature of having more than 3 main modalities modalities, for FMT, the complexity can be managed due to the factorization in FMS. We propose two approaches: 1) for high number of modalities, the involved factors can be reduced based on domain knowledge, the nature of the problem, and the assumed dependencies between modalities (e.g. removing factors between modalities that are deemed weakly related). Alternatively, without making assumptions about inter-modality dependencies, a greedy approach may be taken for adding factors; an approach similar to stepwise regression , iteratively adding the next most important factor. Using these two methods, the model can cope with higher number of modalities with a controllable compromise between performance and overparameterization.

A multimodal transformer for multimodal sequential learning, with strong empirical results on multimodal language metrics such as multimodal sentiment analysis, emotion recognition and personality traits recognition.

represent this text to find a much much longer article about it with say hundreds-thousands of words

------

Given We find movement in function space is not proportional to movement in parameter space during optimization. We propose a new natural-gradient style optimizer to address this., a positive would be To optimize a neural network one often thinks of optimizing its parameters, but it is ultimately a matter of optimizing the function that maps inputs to outputs. Since a change in the parameters might serve as a poor proxy for the change in the function, it is of some concern that primacy is given to parameters but that the correspondence has not been tested. Here, we show that it is simple and computationally feasible to calculate distances between functions in a $L^2$ Hilbert space. We examine how typical networks behave in this space, and compare how parameter $\ell^2$ distances compare to function $L^2$ distances between various points of an optimization trajectory. We find that the two distances are nontrivially related. In particular, the $L^2/\ell^2$ ratio decreases throughout optimization, reaching a steady value around when test error plateaus. We then investigate how the $L^2$ distance could be applied directly to optimization. We first propose that in multitask learning, one can avoid catastrophic forgetting by directly limiting how much the input/output function changes between tasks. Secondly, we propose a new learning rule that constrains the distance a network can travel through $L^2$-space in any one update. This allows new examples to be learned in a way that minimally interferes with what has previously been learned. These applications demonstrate how one can measure and regularize function distances directly, without relying on parameters or local approximations like loss curvature. A neural network's parameters collectively encode a function that maps inputs to outputs. The goal of learning is to converge upon a good input/output function. In analysis, then, a researcher should ideally consider how a network's input/output function changes relative to the space of possible functions. However, since this space is not often considered tractable, most techniques and analyses consider the parameters of neural networks. Most regularization techniques, for example, act directly on the parameters (e.g. weight decay, or the implicit constraints stochastic gradient descent (SGD) places upon movement). These techniques are valuable to the extent that parameter space can be taken as a proxy for function space. Since the two might not always be easily related, and since we ultimately care most about the input/output function, it is important to develop metrics that are directly applicable in function space. In this work we show that it is relatively straightforward to measure the distance between two networks in function space, at least if one chooses the right space. Here we examine L 2 -space, which is a Hilbert space. Distance in L 2 space is simply the expected 2 distance between the outputs of two functions when given the same inputs. This computation relies only on function inference. Using this idea of function space, we first focus on characterizing how networks move in function space during optimization with SGD. Do random initializations track similar trajectories? What happens in the overfitting regime? We are particularly interested in the relationship between trajectories in function space and parameter space. If the two are tightly coupled, then parameter change can be taken as a proxy for function change. This common assumption (e.g. Lipschitz bounds) might not always be the case. Next, we demonstrate two possibilities as to how a function space metric could assist optimization. In the first setting we consider multitask learning, and the phenomenon of catastrophic forgetting that makes it difficult. Many well-known methods prevent forgetting by regularizing how much the parameters are allowed to shift due to retraining (usually scaled by a precision matrix calculated on previous tasks). We show that one can instead directly regularize changes in the input/output function of early tasks. Though this requires a "working memory" of earlier examples, this scheme turns out to be quite data-efficient (and more so than actually retraining on examples from old tasks).In the second setting we propose a learning rule for supervised learning that constrains how much a network's function can change any one update. This rule, which we call Hilbert-constrained gradient descent (HCGD), penalizes each step of SGD to reduce the magnitude of the ing step in L 2 -space. This learning rule thus changes the course of learning to track a shorter path in function space. If SGD generalizes in part because large changes to the function are prohibited, then this rule will have advantages over SGD. Interestingly, HCGD is conceptually related to the natural gradient. As we derive in §3.2.1, the natural gradient can be viewed as ing from constrains changes in a function space measured by the Kullbeck-Leibler divergence. We propose to examine the trajectories of networks in the space of functions defined by the inner product f, g = X f (x)g(x)dµ(x), which yields the following norm: DISPLAYFORM0 Here µ is a measure and corresponds to the probability density of the input distribution X. Note that this norm is over an empirical distribution of data and not over the uniform distribution of all possible inputs. The | · | 2 operator refers to the 2-norm and can apply to vector-valued functions. While we refer to this space as a Hilbert space, we make no use of an inner product and can also speak of this as any normed vector space, e.g. a Banach space. This norm leads to a notion of distance between two functions f and g given by DISPLAYFORM1 Since µ is a density, X dµ = 1, and we can write DISPLAYFORM2 The expectation can be approximated as an empirical expectation over a batch of examples drawn from the input distribution: DISPLAYFORM3 The quality of the empirical distance, of course, will depend on the shape and variance of the distribution of data as well as the form of f and g. In section 2.3, we empirically investigate the quality of this estimator for reasonably sample sizes N. We wish to compare at high level how networks move through parameter and function space. Our first approach is to compare a low-dimensional embedding of the trajectories through these spaces. In FIG2, we take a convolutional neural network and train three random initializations on a 5000-image subset of CIFAR-10. By saving the parameters of the network at each epoch as well as the output on a single large validation batch, we can later compute the 2 parameter distance and the FIG2: Visualization of the trajectories of three random initializations of a network through function space, left, and parameter space, right. The network is a convolutional network trained on a 5,000 image subset of CIFAR-10. At each epoch, we compute the L 2 and 2 distances between all previous epochs, forming two distance matrices, and then recompute the 2D embedding from these matrices using multidimensional scaling. Each point on the plots represents the network at a new epoch of training. The black arrows represent the direction of movement. 2 function distance between the snapshots of network at each epoch. The ing distance matrix is then visualized as a two-dimensional embedding. In parameter space, the networks are initialized at very different points and proceed to diverge yet further from these points. Despite this divergence, each trajectory yields a network that has learned the training data perfectly and generalizes with ∼ 50% accuracy to a test set. This illustrates the wide range of parameter settings that can be used to represent a given neural network function. The behavior of the same initializations in function space is quite different. First, note that all three initializations begin at approximately the same point in function space. This is an intriguing property of random initializations that, rather than encoding entirely random functions, random sets of parameters lead on average to the same function (for related work, see e.g. BID5). The initializations then largely follow an identical path for the initial stage of learning. Different initializations thus learn in similar manners, even if the distance between their parameters diverges. During late-stage optimization, random initializations turn from a shared trajectory and begin to diverge in L 2 space. These differences underlie the general principle that L 2 distances behave differently than 2 distances, and that functional regularization could assist training and reduce overfitting. How well do parameter distances reflect function distances? The answer to this question is relevant for any method that directly considers the behavior of parameters. Certain theoretical analyses, furthermore, desire bounds on function distances but instead find bounds on parameter distances and relate the two with a Lipschitz constant (e.g. BID6). Thus, for theoretical analyses and optimization methods alike, it is important to empirically evaluate how well parameter distances correspond to function distances in typical situations. We can compare these two situations by plotting a change in parameters ∆θ against the corresponding change in the function f θ − f θ+∆θ. In FIG0 we display this relation for several relevant distance during the optimization of a CNN on CIFAR-10. There are three scales: the distance between individual updates, the distance between epochs, and the distance from initialization. What changes strikingly throughout optimization is the relationship between parameter and function space. There is a qualitative difference in the ratio of parameter distances to function distances that visible at all three distance scales. Early epochs generally see larger changes in L 2 space for a given change in parameters. Intriguingly, the ratio of the two distances appears to converge to a single value at late optimization, after test error saturates. This is not because the network ceases to move, as noted above. Rather, the loss landscape shifts such that this ratio become constant. It is also clear from these plots that there is not a consistent positive correlation between the parameter and function distances between any two points on the optimization trajectory. For example, the parameter distance between successive epochs is negatively correlated with the L 2 distance for most of optimization FIG0. The distance from initialization shows a clean and positive relationship, but the relationship changes during optimization. Between successive batches, L 2 distance correlates with parameter distance at late epochs, but less so early in optimization when learning is quickest. Thus, at different stages of optimization, the L 2 / 2 ratio is often quite different. The usage of Batch Normalization (BN) and weight decay in this analysis somewhat affects the trends in the L 2 / 2 ratio. In Appendix A we reproduce these plots for networks trained without BN and without weight decay. The overall message that the L 2 / 2 ratio changes during optimization is unchanged. However, these methods both change the scale of updates, and appear to do so differently throughout optimization, and thus some trends are different. In Appendix B, we also isolate the effect of training data, by reproducing these plots for a CNN trained on MNIST and find similar trends. Overall, the correspondence between parameter and function distances depends strongly on the context. It might be worried that since function space is of infinite dimension, one would require prohibitively many examples to estimate a function distance. However, we find that one can compute a distance between two functions with a relatively small amount of examples. FIG4 shows how the estimated L 2 distance converges with an increasing number examples. In general, we find that only a few hundred examples are necessary to converge to an estimation within a few percent. If, after having been trained on a task, a neural network is retrained on a new task, it often forgets the first task. This phenomenon is termed'catastrophic forgetting'. It is the central difficulty of multitask training as well as applications requiring that learning be done online (especially in non-IID situations). Essentially, new information must be encoded in the network, but the the information pertinent to the previous task must not be overwritten. Most efforts to combat catastrophic forgetting rely on restricting how much parameters can change between tasks. Elastic Weight Consolidation (EWC; BID10), for example, adds a penalty to the loss on a new task B that is the distance from the weights after learning on an earlier FIG4: The variance of the the L 2 estimator is small enough that it can be reasonably estimated from a few hundred examples. In panels A and D, we reproduced L 2 distances seen in the panels of FIG0. As we increase the number of validation examples these distances are computed over, the estimations become more accurate. Panels B and E show the 95% confidence bounds for the estimation; on 95% of batches, the value will lie bewteen these bounds. These bounds can be obtained from the standard deviation of the L 2 distance on single examples. In panel C we show that the standard deviation scales linearly with the L 2 distance when measured between updates, meaning that a fixed batch size will often give similar percentage errors. This is not true for the distance from initialization, in panel F; early optimization has higher variance relative to magnitude, meaning that more examples are needed for the same uncertainty. In the Appendix, we also display the convergence of the L 2 distance estimator between epochs.task A, multiplied by the diagonal of the Fisher information matrix F (calculated on task A): DISPLAYFORM0 This idea is closely related to well-studied approaches to Bayesian online learning, if F is interpreted as a precision matrix BID7, BID15 ). Other similar approaches include that of BID21, who use a more accurate approximation of the Fisher, and Synaptic Intelligence (SI; BID28), which discounts parameter change via a diagonal matrix in which each entry reflects the sum contribution of that parameter to the loss. Each of these method discourages catastrophic forgetting by restricting movement in parameter space between tasks, scaled by a (perhaps diagonal) precision matrix calculated on previous tasks. Using a function space metric, it is not hard to ensure that the network's output function on previous tasks does not change during learning. In this case, the loss for a new task B is modified to be: DISPLAYFORM1 The regularization term is the L 2 distance between the current function f θ B and the function after training on task A, f θ A. Since our function space metric is defined over a domain of examples, we will store a small set of previously seen examples in a working memory, as well as the output on those examples. This memory set will be used to calculate the L 2 distance between the current iteration and the snapshot after training. This is a simple scheme, but novel, and we are not aware of direct precedence in the literature. A working memory approach is employed in related work BID12 BID20 ). Note, however, that storing old examples violates the rules of strict online learning. Nevertheless, for large networks it will be more memory-efficient. EWC, for example, requires storing a snapshot of each parameter at the end of the previous task, as well as a diagonal precision matrix with as many entries as parameters. For the 2 hidden layer network with 400 nodes each that was used in the MNIST task in BID10, this is 1,148,820 new parameters, or as many pixels as 1,465 MNIST images. When each layer has as many as 2,000 nodes, as in FIG4 of BID10, the extra stored parameters are comparable to 15,489 MNIST images. The working memory approach that is required to regularize function change from old tasks is thus comparable or cheaper in memory. We compared the performance of our approach at the benchmark task of permuted MNIST. This task requires a single MLP to learn to classify a sequence of MNIST tasks in which the pixels have been randomly permuted differently on each task. We trained an MLP with 2 hidden layers, 400 nodes each, for 10 epochs on each of 8 such permuted datasets. In FIG5, we display how the test accuracy on the first of 8 tasks degrades with subsequent learning. To build the working memory, we keep 1024 examples from previous tasks, making sure that the number of examples from each task is equal. We also remember the predictions on those examples at the end of training on their originating tasks. To calculate the L 2 distance, we simply re-infer on the examples in working memory, and regularize the distance from the current outputs to the remembered outputs. We chose λ = 1.3 as the regularizing hyperparameter from a logarithmic grid search. In FIG5, we compare this method to four comparison methods. The "ADAM" method is ADAM with a learning rate of 0.001, which nearly forgets the first task completely at the end of the 8 tasks. The "ADAM+retrain" method is augmented with a working memory of 1024 examples that are stored from previous tasks. Every n iterations (we found n = 10 to be best), a step is taken to decrease the loss on the memory cache. This method serves as a control for the working memory concept. We also include EWC and SI as comparisons, using the hyperparameters used in their publications (λ = 500, = c = 0.1). Overall, we found that regularizing the L 2 distance on a working memory cache was more successful than simply retraining on the same cache. It also outperformed EWC, but not SI. Note that these methods store diagonal matrices and the old parameters, and in this circumstance these were larger in memory than the memory cache. Figure 4: Regularizing the L 2 distance from old tasks (calculated over a working memory cache of size 1024) can successfully prevent catastrophic forgetting. Here we display the test performance on the first task as 7 subsequent tasks are learned. Our method outperforms simply retraining on the same cache (ADAM+retrain), which potentially overfits to the cache. Also displayed are ADAM without modifications, EWC, and SI. In this section we propose that the L 2 distance can be used for regularization in a single supervised task. In the space of parameters, SGD is a strongly local update rule and large jumps are generally prohibited. SGD is thus more likely to find solutions that are close to the initialization, and furthermore to trace a path of limited length. This discourages the sampling a large volume of parameter space during optimization. If the mapping between parameter and function space is not already very tight, and locality is important for generalization, then additionally constricting changes in function space should help. On the basis of this logic, we propose a learning rule that directly constrains the path length of the optimization trajectory L 2 space. If a network would have been trained to adjust the parameters θ to minimize some cost C 0, we will instead minimize at each step t a new cost given by: DISPLAYFORM0 Like all regularization terms, this can also be viewed as a Langrangian that satisfies a constraint. Here, this constraint ensures that the change in L 2 -space does not exceed some constant value. To evaluate Equation 1, we can approximate the norm with an empirical expectation over X: DISPLAYFORM1 This cost function imposes a penalty upon the difference between the output of the current network at time t and the proposed network at t + 1. The data x i may derive from some validation batch but must pull from the same distribution X. It would also be possible to use unlabeled data. We can write an update rule to minimize Equation 1 that is a modification of gradient descent. We call the rule Hilbert-constrained gradient descent (HCGD). It minimizes C in Equation 1 via an inner loop of gradient descent. To optimize C via gradient descent, we first replace C 0 with its first order approximation J T ∆θ, where J is the Jacobian. Thus we seek to converge to a ∆θ at each update step, where DISPLAYFORM2 Minimization of the proper ∆θ can be performed in an inner loop by a first order method. We first propose some ∆θ 0 = − J = − ∇ θ C 0 (for learning rate) and then iteratively correct this proposal by gradient descent towards ∆θ. If only one correction is performed, we simply add the derivative of the Hilbert-constraining term after ∆θ 0 has been proposed. We found empirically that a single correction was often sufficient. In Appendix C, we demonstrate that this algorithm does actually decrease the distance traveled in function space, as expected. This algorithm is shown in Algorithm 1. DISPLAYFORM3 Calculate correction 9: DISPLAYFORM4 10: DISPLAYFORM5 return θ t Note that the "proposed update" is presented as an SGD step, but could be a step of another optimizer (e.g. ADAM). In the Appendix, we display an extended version of this algorithm. This version allows for multiple corrective iterations in each step. It also allows for a form of momentum. In standard momentum for SGD, one follows a "velocity" term v which is adjusted at each step with the rule v ← βv + J (e.g. see BID26). For HCGD, we also keep a velocity term but update it with the final Hilbert-constrained update ∆θ rather than J. The velocity is used to propose the initial ∆θ 0 in the next update step. We found that this modification of momentum both quickened optimization and lowered generalization error. The natural gradient turns out to carry a similar interpretation as HCGD, in that the natural gradient also regularizes the change in functions' output distributions. Specifically, the natural gradient can be derived from a penalty upon the change in a network's output distribution as measured by the Kullbeck-Leibler divergence (rather than the L 2 distance).To show this, we start with a similar goal of function regularization and will come upon the natural gradient. Let us seek to regularize the change in a network's output distribution P θ throughout optimization of the parameters θ, choosing the Kullbeck-Leibler (KL) divergence as a measure of similarity between any two distributions. To ensure the output distribution changes little throughout optimization, we define a new cost function DISPLAYFORM0 where C 0 is the original cost function and λ is a hyperparameter that controls the importance of this regularization term. Optimization would be performed with respect to the proposed update θ t+1.Evaluating the KL divergence directly is problematic because it is infeasible to define the output density P θ everywhere. One can obtain a more calculable form by expanding D KL (P θt+1 P θt) around θ t to second order with respect to θ. The Hessian of the KL divergence is the Fisher information metric F. With ∆θ ≡ (θ t+1 − θ t), we can rewrite our regularized cost function as DISPLAYFORM1 To optimize C via gradient descent we first replace C 0 with its first order approximation. DISPLAYFORM2 At each evaluation, J is evaluated before any step is made, and we seek the value of ∆θ that minimizes Equation FORMULA15. By setting the derivative with respect to ∆θ to be zero, we can see that this value is DISPLAYFORM3 When λ = 1 this update is equal to the natural gradient. Thus, the natural gradient emerges as the optimal update when one regularizes the change in the output distribution during learning. In Appendix E, we show how one can approximate the natural gradient with an inner first-order optimization loop, like in HCGD. We note that HCGD is computationally cheaper than the exact natural gradient. It does not require any matrix inversions, nor the calculation of separate per-example gradients. When the validation batch X V is drawn anew for each of n corrective iterations (step 8 in Algorithm 1), HCGD requires an additional two forward passes and one backwards pass for each correction, for a total of 2 + 3n passes each outer step. In addition to being seen as a regularizer of functional change, it in an interesting aside to note that variants of the natural gradient have appeared with many justifications. These include data efficiency, minimizing a regret bound during learning, speeding optimization, and the benefits of whitened gradients. Amari originally developed the natural gradient in the light of information geometry and efficiency BID1; ). If some directions in parameter space are more informative of the network's outputs than others, then updates should be scaled by each dimension's informativeness. Equivalently, if not all examples carry equal information about a distribution, then the update step should be modified to make use of highly informative examples. That is, we wish to find a Fisherefficient algorithm (see BID2). The natural gradient uses the Fisher information matrix to scale the update by parameters' informativeness. There is also a connection between the natural gradient (and thus HCGD) and techniques that normalize and whiten gradients. The term F −1 J, after all, simply ensures that steps are made in a parameter space that is whitened by the covariance of the gradients. Whitening the gradients thus has the effect that SGD becomes more similar to the natural gradient. It appears that many approaches to normalize and whiten activations or gradients have been forwarded in the literature BID19; BID25; BID24; BID3 BID27; BID11; BID23; BID22 ). A similar effect is able to be learned with Batch Normalization, as well BID9 ). By normalizing and whitening the gradients, or by proxy, the activations, these various methods ensure that parameter space is a better proxy for function space. We compared HCGD and SGD on feedforward and recurrent architectures. If it is important that SGD limits changes in function space, and parameter and function space are loosely coupled, then HCGD should improve upon SGD. In all tests, we used a tuned learning rate for SGD, and then used the same learning rate for HCGD. We use values of λ = 0.5 and η = 0.02, generally about 10 times less than the principal learning rate. (For the n = 1 version, λ can be folded into the inner learning rate η. Values were chosen so that λη = 0.01.) We chose the batch size for the "validation" batch to be 256. While the examples in each "validation" batch were different than the training batch, they were also drawn from the train set. All models were implemented in PyTorch BID17 ).We tested HCGD as applied to the CIFAR-10 image classification problem. For reproducibility, we trained a Squeezenet v1.1, a convolutional neural network model with batch normalization optimized for parameter efficiency BID8 ). Overall HCGD does not outperform SGD in the final learning stage when trained with the same learning rate as SGD (initial = 0.1), though it does perform better in the early stage while the learning rate is high (FIG1). When we increase the initial learning rate to = 0.3 (red trace), the training accuracy decreases but the test accuracy is still marginally higher than SGD. Given the difference in relative performance between the high and low learning rate stages, it is possible that HCGD requires a different learning rate schedule to achieve the same level of gradient noise. HCGD thus decreases the test error at a given learning rate, but needs to be trained at a higher learning rate to achieve the same level of gradient noise. We next tested the performance of HCGD on a recurrent task. We trained an LSTM on the sequential MNIST task, in which pixels are input one at a time. The order of the pixels was permuted to further complicate the task. We found that HCGD outperformed SGD (Figure 6 . We used 1 correction step, as before, but found that using more correction steps yielded even better performance. However, HCGD underperformed ADAM. While not the ideal optimizer for this task, the fact that SGD can be improved indicates that SGD does not move as locally in function space as it should. Parameter space thus a poor proxy for function space in recurrent networks. HCGD first proposes an update by SGD, and then corrects it, but the first update step can also be other optimizers. Since Adam worked well for the sequential MNIST task, we tested if Adam could also be improved by taking a step to penalize the change in function space. We found that this is indeed the case, and show the as well in Figure 6 . To differentiate the SGD-and Adam-based methods, we refer to in the figure as SGD+HC and Adam+HC. This combination of Adam and L 2 functional regularization could help to achieve state-of-the-art performance on recurrent tasks. Neural networks encode functions, and it is important that analyses discuss the empirical relationship between function space and the more direct parameter space. Here, we argued that the L 2 Hilbert space defined over an input distribution is a tractable and useful space for analysis. We found that networks traverse this function space qualitatively differently than they do parameter space. Depending on the situation, a distance of parameters cannot be taken to represent a proportional distance between functions. We proposed two possibilities for how the L 2 distance could be used directly in applications. The first addresses multitask learning. By remembering enough examples in a working memory to accurately Figure 6 : Results of a singlelayer LSTM with 128 hidden units trained on the sequential MNIST task with permuted pixels. Shown are the traces for SGD and Adam (both with learning rate 0.01). We then take variants of the HCGD algorithm in which the first proposed step is taken to be an SGD step (SGD+HC) or an Adam step (Adam+HC). For SGD+HC we also show the effect of introducing more iterations n in the SGD+HC step.estimate an L 2 distance, we can ensure that the function (as defined on old tasks) does not change as a new task is learned. This regularization term is agnostic to the architecture or parameterization of the network. We found that this scheme outperforms simply retraining on the same number of stored examples. For large networks with millions of parameters, this approach may be more appealing than comparable methods like EWC and SI, which require storing large diagonal matrices. We also proposed a learning rule that reduces movement in function space during single-task optimization. Hilbert-constrained gradient descent (HCGD) constrains the change in L 2 space between successive updates. This approach limits the movement of the encoded function in a similar way as gradient descent limits movement of the parameters. It also carries a similar intuition as the forgetting application: to learn from current examples only in ways that will not affect what has already been learned from other examples. HCGD can increase test performance at image classification in recurrent situations, indicating both that the locality of function movement is important to SGD and that it can be improved upon. However, HCGD did not always improve , indicating either that SGD is stable in those regimes or that other principles are more important to generalization. This is by no means the only possibility for using an L 2 norm to improve optimization. It may be possible, for example, to use the norm to regularize the confidence of the output function (e.g. BID18). We are particularly interested in exploring if more implicit, architectural methods, like normalization layers, could be designed with the L 2 norm in mind. It interesting to ask if there is support in neuroscience for learning rules that diminish the size of changes when that change would have a large effect on other tasks. One otherwise perplexing finding is that behavioral learning rates in motor tasks are dependent on the direction of an error but independent of the magnitude of that error BID4. This is not expected by most models of gradient descent, but would be expected if the size of the change in the output distribution (i.e. behavior) were regulated to be constant. Regularization upon behavioral change (rather than synaptic change) would predict that neurons central to many actions, like neurons in motor pools of the spinal cord, would learn very slowly after early development, despite the fact that their gradient to the error on any one task (if indeed it is calculated) is likely to be quite large. Given our general resistance to overfitting during learning, and the great variety of roles of neurons, it is likely that some type of regularization of behavioral and perceptual change is at play. Figure A.5: Same as above, but for a network trained without Batch Normalization and also without weight decay. Weight decay has a strong effect. The main effect is that decreases the 2 distance traveled at all three scales (from last update, last epoch, and initialization), especially at late optimization. This explains the left column, and some of the middle and right columns. (It is helpful to look at the "white point" on the color scale, which indicates the point halfway through training. Note that parameter distances continue to change after the white point when WD is not used). An additional and counterintuitive property is that the L 2 distance from the last epoch increases in scale during optimization when WD is not used, but decreases if it is. These comparisons show that WD has a strong effect on the L 2 / 2 ratio, but that this ratio still changes considerable throughout training. This is in line with this paper's motivation to consider L 2 distances directly. Figure B.6: Here we reproduce the of FIG0 and FIG4 for the MNIST task, again using a CNN with batch normalization trained with SGD with momentum. It can be seen first that the majority of function space movement occurs very early in optimization, mostly within the first epoch. The standard deviation of the L 2 estimator, which sets the number of examples needed to accurately estimate a consistent value, is somewhat higher than for CIFAR-10. Finally, at right, it can be seen that the relationship between parameter distance traveled and function distance is similar to that of a CNN on CIFAR-10, include the qualitative change after test error converges (which here is around epoch 1). 2 SPACE Figure C.7: The HCGD algorithm is designed to reduce motion through L2-space. To confirm this, here we plot the cumulative squared distance traveled during optimization for a simple MLP trained on MNIST. This is calculated by the simple cumulative sum of the squared distances between consecutive updates. (The squared distance is nice because Brownian motion will present as a linear increase in its cumulative sum). It can be seen that SGD continues to drift in L2-space during the overfitting regime (around epoch 15, which is when test error saturates), while HCGD plateaus. This indicates that the function has converged to a single location; it ceases to change. With SGD, on the other hand, the network continues to cahnge even long after test error saturates. It is interesting to note that HCGD allows the parameters to continue to drift even though the function has generally converged. This version of the algorithm includes momentum. It also allows for multiple corrections. J ← ∇ θ C 0 (X) Calculate gradients 8: DISPLAYFORM0 ∆θ 0 ← −v Proposed update 10: DISPLAYFORM1 12: DISPLAYFORM2 Update buffer 14:for 1 < j < n do Optional additional corrections 15: ∆θ j ← ∆θ j−1 − η(g L 2) DISPLAYFORM3 18: DISPLAYFORM4 19:θ t ← θ t−1 + ∆θ 20:return θ t In order to better compare the natural gradient to the Hilbert-constrained gradient, we propose a natural gradient algorithm of a similar style. Previous work on the natural gradient has aimed to approximate F −1 as best and as cheaply as possible. This is equivalent to minimizing Equation 2 (i.e. J∆θ + λ 2 ∆θ T F ∆θ) with a single iteration of a second-order optimizer. For very large neural networks, however, it is much cheaper to calculate matrix-vector products than to approximately invert a large matrix. It is possible that the natural gradient may be more accessible via an inner gradient descent, which would be performed during each update step as an inner loop. We describe this idea at high level in Algorithm 2. After an update step is proposed by a standard optimizer, the algorithm iteratively corrects this update step towards the natural gradient. To start with a good initial proposed update, it is better to use a fast diagonal approximation of the natural gradient (such as Adagrad or RMSprop) as the main optimizer. Each additional correction requires just one matrix-vector product after the gradients are calculated. Depending on the quality of the proposed update, the number of iterations required is likely to be small, and even a small number of iterations will improve the update. ∆θ i+1 = ∆θ i − η(J + λF ∆θ i)Step towards DISPLAYFORM0 θ ← θ + ∆θ 8:return θ tSince the Fisher matrix F can be calculated from the covariance of gradients, it never needs to be fully stored. Instead, for an array of gradients G of size (# parameters, # examples), we can write DISPLAYFORM1 The choice of G is an important one. It cannot be a vector of aggregated gradients (i.e. J), as that would destroy covariance structure and would in a rank-1 Fisher matrix. Thus, we must calculate the gradients on a per-example basis. To compute G efficiently it is required that a deep learning framework implement forward-mode differentiation, which is currently not supported in popular frameworks. If we choose G to be the array of per-example gradients on the minibatch, F is known as the'empirical Fisher'. As explained in BID13 and in BID16, the proper method is to calculate G from the predictive (output) distribution of the network, P θ (y|x). This can be done as in BID14 by sampling randomly from the output distribution and re-running backpropagation on these fictitious targets, using (by necessity) the activations from the minibatch. Alternatively, as done in BID16, one may also use unlabeled or validation data to calculate G on each batch.

We compare graph RNNs and graph ConvNets, and we consider the most generic class of graph ConvNets with residuality.


Question: represent the scientific article to find a short tldr

Question: We develop a novel and efficient algorithm for optimizing neural networks inspired by a recently proposed geodesic optimization algorithm. Our algorithm, which we call Stochastic Geodesic Optimization (SGeO), utilizes an adaptive coefficient on top of Polyak's Heavy Ball method effectively controlling the amount of weight put on the previous update to the parameters based on the change of direction in the optimization path. Experimental on strongly convex functions with Lipschitz gradients and deep Autoencoder benchmarks show that SGeO reaches lower errors than established first-order methods and competes well with lower or similar errors to a recent second-order method called K-FAC (Kronecker-Factored Approximate Curvature). We also incorporate Nesterov style lookahead gradient into our algorithm (SGeO-N) and observe notable improvements. First order methods such as Stochastic Gradient Descent (SGD) with Momentum and their variants are the methods of choice for optimizing neural networks. While there has been extensive work on developing second-order methods such as Hessian-Free optimization and Natural Gradients , they have not been successful in replacing them due to their large per-iteration costs, in particular, time and memory. Although Nesterov's accelerated gradient and its modifications have been very effective in deep neural network optimization , some research have shown that Nesterov's method might perform suboptimal for strongly convex functions without looking at local geometry of the objective function. Further, in order to get the best of both worlds, search for optimization methods which combine the the efficiency of first-order methods and the effectiveness of second-order updates is still underway. In this work, we introduce an adaptive coefficient for the momentum term in the Heavy Ball method as an effort to combine first-order and second-order methods. We call our algorithm Geodesic Optimization (GeO) and Stochastic Geodesic Optimization (SGeO) (for the stochastic case) since it is inspired by a geodesic optimization algorithm proposed recently . The adaptive coefficient effectively weights the momentum term based on the change in direction on the loss surface in the optimization process. The change in direction can contribute as implicit local curvature information without resorting to the expensive second-order information such as the Hessian or the Fisher Information Matrix. Our experiments show the effectiveness of the adaptive coefficient on both strongly-convex functions with Lipschitz gradients and general non-convex problems, in our case, deep Autoencoders. GeO can speed up the convergence process significantly in convex problems and SGeO can deal with illconditioned curvature such as local minima effectively as shown in our deep autoencoder benchmark experiments. SGeO has similar time-efficiency as first-order methods (e.g. Heavy Ball, Nesterov) while reaching lower reconstruction error. Compared to second-order methods (e.g., K-FAC), SGeO has better or similar reconstruction errors while consuming less memory. The structure of the paper is as follows: In section 2, we give a brief on the original geodesic and contour optimization introduced in , neural network optimization methods and the conjugate gradient method. In section 3, we introduce our adaptive coefficient specifically designed for strongly-convex problems and then modify it for general non-convex cases. In section 4, we discuss some of the related work in the literature. Section 5 illustrates the algorithm's performance on convex and non-convex benchmarks. More details and insights regarding the algorithm and the experiments can be found in the Appendix. The goal is to solve the optimization problem min θ∈R f (θ) where f: R D → R is a differentiable function. approach the problem by following the geodesics on the loss surface guided by the gradient. In order to solve the geodesic equation iteratively, the authors approximate it using a quadratic. In the neighbourhood of θ t, the solution of the geodesic equation can be approximated as: Clearly, one can see the conjugate gradient method as a momentum method where t is the learning rate and t γ t−1 is the momentum parameter: We added the prime notation to avoid confusion with d t = θ t+1 − θ t throughout the paper). To avoid calculating the Hessian ∇ 2 f which can be very expensive in terms of computation and memory, is usually determined using a line search, i.e. by approximately calculating t = arg min f (θ t + d t) and several approximations to γ have been proposed. For example, have proposed the following: Note that γ F R (with an exact line search) is equivalent to the original conjugate gradient algorithm in the quadratic case. The adaptive coefficient that appears before the unit tangent vector in equation 2 has an intuitive geometric interpretation: where φ t is the angle between the previous update d t = θ t − θ t−1 and the negative of the current gradient −g t. Since 0 ≤ φ ≤ π, thus −1 ≤ cos (π − φ t) ≤ 1. The adaptive coefficient embeds a notion of direction change on the path of the algorithm which can be interpreted as implicit second-order information. The change in direction at the current point tells us how much the current gradient's direction is different from the previous gradients which is similar to what second-order information (e.g. the Hessian) provide. For more details on the adaptive coefficient we refer the reader to Appendix C. We propose to apply this implicit second-order information to the Heavy Ball method of as an adaptive coefficient for the momentum term such that, in strongly-convex functions with Lipschitz gradients, we reinforce the effect of the previous update when the directions align, i.e. in the extreme case: φ = 0 and decrease when they don't, i.e. the other extreme case: φ = π. Thus, we write the coefficient as γ C t = 1 −ḡ t ·d t with C indicating "convex". It's obvious that 0 ≤ γ C t ≤ 2. Note that we will use the bar notation (e.g.d) throughout the paper indicating normalization by magnitude. Aμ-strongly convex function f with L-Lipschitz gradients has the following properties: Applying the proposed coefficient to the Heavy Ball method, we have the following algorithm which we call GeO (Geodesic Optimization): Algorithm 1: GEO (STRONGLY CONVEX AND LIPSCHITZ) Calculate the gradient g t = ∇f (θ t) Calculate adaptive coefficient γ where T is total number of iterations and α is a tunable parameter set based on the function being optimized and is the learning rate. Incorporating Nesterov's momentum We can easily incorporate Nesterov's lookahead gradient into GeO by modifying line 4 to g t = ∇f (θ t + µd t) which we call GeO-N. In GeO-N the gradient is taken at a further point θ t + µd t where µ is a tunable parameter usually set to a value close to 1. However, the algorithm proposed in the previous section would be problematic for non-convex functions such as the loss function when optimizing neural networks. Even if the gradient information was not partial (due to minibatching), the current direction of the gradient cannot be trusted because of non-convexity and poor curvature (such as local minima, saddle points, etc). To overcome this issue, we propose to alter the adaptive coefficient to with N C indicating "non-convex". By applying this small change we are reinforcing the previous direction when the directions do not agree thus avoiding sudden and unexpected changes of direction (i.e. gradient). In other words, we choose to trust the previous history of the path already taken more, thus acting more conservatively. To increase efficiency, we use minibatches, calling the following algorithm SGeO (Stochastic Geodesic Optimization): Draw minibatch from training set Calculate the gradient g t = ∇f (θ t) Calculate adaptive coefficient γ Further we found that using the unit vectors for the gradientḡ and the previous updated, when calculating the next update in the non-convex case makes the algorithm more stable. In other words, the algorithm behaves more robustly when we ignore the magnitude of the gradient and the momentum term and only pay attention to their directions. Thus, the magnitudes of the updates are solely determined by the corresponding step sizes, which are in our case, the learning rate and the adaptive geodesic coefficient. Same as the strongly convex case, we can integrate Nesterov's lookahead gradient into SGeO by replacing line 5 with g t = ∇f (θ t + µd t) which we call SGeO-N. There has been extensive work on large-scale optimization techniques for neural networks in recent years. A good overview can be found in. Here, we discuss some of the work more related to ours in three parts. Adagrad is an optimization technique that extends gradient descent and adapts the learning rate according to the parameters. Adadelta and RMSprop improve upon Adagrad by reducing its aggressive deduction of the learning rate. Adam improves upon the previous methods by keeping an additional average of the past gradients which is similar to what momentum does. Adaptive Restart proposes to reset the momentum whenever rippling behaviour is observed in accelerated gradient schemes. AggMo keeps several velocity vectors with distinct parameters in order to damp oscillations. AMSGrad on the other hand, keeps a longer memory of the past gradients to overcome the suboptimality of the previous algorithms on simple convex problems. We note that these techniques are orthogonal to our approach and can be adapted to our geodesic update to further improve performance. Several recent works have been focusing on acceleration for gradient descent methods. propose an adaptive method to accelerate Nesterov's algorithm in order to close a small gap in its convergence rate for strongly convex functions with Lipschitz gradients adding a possibility of more than one gradient call per iteration. , the authors propose a differential equation for modeling Nesterov inspired by the continuous version of gradient descent, a.k.a. gradient flow. take this further and suggest that all accelerated methods have a continuous time equivalent defined by a Lagrangian functional, which they call the Bregman Lagrangian. They also show that acceleration in continuous time corresponds to traveling on the same curve in spacetime at different speeds. It would be of great interest to study the differential equation of geodesics in the same way. In a recent work, proposes a differential geometric interpretation of Nesterov's method for strongly-convex functions with links to continuous time differential equations mentioned earlier and their Euler discretization. Second-order methods are desirable because of their fine convergence properties due to dealing with bad-conditioned curvature by using local second-order information. Hessian-Free optimization is based on the truncated-Newton approach where the conjugate gradient algorithm is used to optimize the quadratic approximation of the objective function. The natural gradient method reformulates the gradient descent in the space of the prediction functions instead of the parameters. This space is then studied using concepts in differential geometry. K-FAC approximates the Fisher information matrix which is based on the natural gradient method. Our method is different since we are not using explicit second-order information but rather implicitly deriving curvature information using the change in direction. We evaluated SGeO on strongly convex functions with Lipschitz gradients and benchmark deep autoencoder problems and compared with the Heavy-Ball and Nesterov's algorithms and K-FAC. We borrow these three minimization problems from where they try to accelerate Nesterov's method by using adaptive step sizes. The problems are Anisotropic Bowl, Ridge Regression and Smooth-BPDN. The learning rate for all methods is set to 1 L except for Nesterov which is set to 4 3L+μ and the momentum parameter µ for Heavy Ball, Nesterov and GeO-N is set to the following: where L is the Lipschitz parameter andμ is the strong-convexity parameter. The adaptive parameter γ t for Fletcher-Reeves is set to γ and for GeO and GeO-N is γ C t = 1 −ḡ t ·d t. The functionspecific parameter α is set to 1, 0.5 and 0.9 in that order for the following problems. It's important to note that the approximate conjugate gradient method is only exact when an exact line search is used, which is not the case in our experiments with a quadratic function (Ridge Regression). Anisotropic Bowl The Anisotropic Bowl is a bowl-shaped function with a constraint to get Lipschitz continuous gradients: As in , we set n = 500, τ = 4 and and µ = 1. Figure 1 shows the convergence for our algorithms and the baselines. The algorithms terminate when f (θ) − f * < 10 −12. GeO-N and GeO take only 82 and 205 iterations to converge, while the closest is that of Heavy-Ball and Fletcher-Reeves which take approximately 2500 and 3000 iterations respectively. Ridge Regression The Ridge Regression problem is a linear least squares function with Tikhonov regularization: where A ∈ R m×n is a measurement matrix, b ∈ R m is the response vector and γ > 0 is the ridge parameter. The function f (θ) is a positive definite quadratic function with the unique solution of 2 + λ and strong convexity parameterμ = λ. , m = 1200, n = 2000 and λ = 1. A is generated from U ΣV T where U ∈ R m×m and V ∈ R n×m are random orthonormal matrices and Σ ∈ R m×m is diagonal with entries linearly distanced in while b = randn(m, 1) is drawn (i.i.d) from the standard normal distribution. Thusμ = 1 and L ≈ 1001. Figure 2 shows the where Fletcher-Reeves, which is a conjugate gradient algorithm, performs better than other methods but we observe similar performances overall except for gradient descent. The tolerance is set to f (θ) − f * < 10 −13. Smooth-BPDN Smooth-BPDN is a smooth and strongly convex version of the BPDN (basis pursuit denoising) problem: where Since we cannot find the solution analytically, Nesterov's method is used as an approximation to the solution (f * N) and the tolerance is set to f (θ) − f * N < 10 −12. Figure 3 shows the for the algorithms. GeO-N and GeO converge in 308 and 414 iterations respectively, outperforming all other methods. Closest to these two is Fletcher-Reeves with 569 iterations and Nesterov and Heavy Ball converge similarly in 788 iterations. To evaluate the performance of SGeO, we apply it to 3 benchmark deep autoencoder problems first introduced in which use three datasets, MNIST, FACES and CURVES. Due to the difficulty of training these networks, they have become standard benchmarks for neural network optimization. To be consistent with previous literature (; ;), we use the same network architectures as in and also report the reconstruction error instead of the log-likelihood objec- Our baselines are the Heavy Ball algorithm (SGD-HB) , SGD with Nesterov's Momentum (SGD-N) and K-FAC , a second-order method utilizing natural gradients using an approximation of the Fisher information matrix. Both the baselines and SGeO were implemented using MATLAB on GPU with single precision on a single machine with a 3.6 GHz Intel CPU and an NVIDIA GeForce GTX 1080 Ti GPU with 11 GBs of memory. The are shown in Figures 4 to 6. Since we are mainly interested in optimization and not in generalization, we only report the training error, although we have included the test set performances in the Appendix B. We report the reconstruction relative to the computation time to be able to compare with K-FAC, since each iteration of K-FAC takes orders of magnitude longer than SGD and SGeO. The per-iteration graphs can be found in the Appendix A. All methods use the same parameter initialization scheme known as "sparse initialization" introduced in. The experiments for the Heavy Ball algorithm and SGD with Nesterov's momentum follow which were tuned to maximize performance for these problems. For SGeO, we chose a fixed momentum parameter and used a simple multiplicative schedule for the learning rate: where the initial value was chosen from {0.1,0.15,0.2,0.3,0.4,0.5} and is decayed (K) every 2000 iterations (parameter updates). The decay parameter (β) was set to 0.95. For the momentum parameter µ, we did a search in {0.999,0.995,0.99}. The minibatch size was set to 500 for all methods except K-FAC which uses an exponentially increasing schedule for the minibatch size. For K-FAC we used the official code provided 1 by the authors with default parameters to reproduce the . The version of K-FAC we ran was the Blk-Tri-Diag approach which achieves the best in all three cases. To do a fair comparison with other methods, we disabled iterate averaging for K-FAC. It is also worth noting that K-FAC uses a form of momentum . In all three experiments, SGeO-N is able to outperform the baselines (in terms of reconstruction error) and performs similarly as (if not better than) K-FAC. We can see the effect of the adaptive coefficient on the Heavy Ball method, i.e. SGeO, which also outperforms SGD with Nesterov's momentum in two of the experiments, MNIST and FACES, and also outperforms K-FAC in the MNIST experiment. Use of Nesterov style lookahead gradient significantly accelerates training for the MNIST and CURVES dataset, while we see this to a lesser extent in the FACES dataset. This is also the case for the other baselines . Further, we notice an interesting phenomena for the MNIST dataset (Figure 4). Both SGeO and SGeO-N reach very low error rates, after only 900 seconds of training, SGeO and SGeO-N arrive at an error of 0.004 and 0.0002 respectively. We proposed a novel and efficient algorithm based on adaptive coefficients for the Heavy Ball method inspired by a geodesic optimization algorithm. We compared SGeO against SGD with Nesterov's Momentum and regular momentum (Heavy Ball) and a recently proposed second-order method, K-FAC, on three deep autoencoder optimization benchmarks and three strongly convex functions with Lipschitz gradients. We saw that SGeO is able to outperform all first-order methods that we compared to, by a notable margin. SGeO is easy to implement and the computational overhead it has over the first-order methods, which is calculating the dot product, is marginal. It can also perform as effectively as or better than second-order methods (here, K-FAC) without the need for expensive higher-order operations in terms of time and memory. We believe that SGeO opens new and promising directions in high dimensional optimization research and in particular, neural network optimization. We are working on applying SGeO to other machine learning paradigms such as CNNs, RNNs and Reinforcement Learning. It remains to analyse the theoretical properties of SGeO such as its convergence rate in convex and non-convex cases which we leave for future work. Here we include the per-iteration for the autoencoder experiments in Figures 7 to 9. We reported the reconstruction error vs. running time in the main text to make it easier to compare to K-FAC. K-FAC, which is a second-order algorithm, converges in fewer iterations but has a high per-iteration cost. All other methods are first-order and have similar per-iteration costs. We include generalization experiments on the test set here. However, as mentioned before, our focus is optimization and not generalization, we are aware that the choice of optimizer can have a significant effect on the performance of a trained model in practise. Results are shown in Figures 10 to 12. SGeO-N shows a significant better predictive performance than SGeO on the CURVES data set and both perform similarly on the two other datasets.. Note that the algorithms are tuned for best performance on the training set. Overfitting can be dealt with in various ways such as using appropriate regularization during training and using a small validation set to tune the parameters. C ADAPTIVE COEFFICIENT BEHAVIOUR C.1 GEOMETRIC INTERPRETATION Figure 13 (b) shows the dot product valueḡ ·d which is equivalent to cos (π − φ) (where φ is the angle between the previous update and the negative of the current gradient) for different values of φ. Figure 13 (a) shows the adaptive coefficient (γ) behaviour for different values of φ for both convex and non-convex cases. Recall that the adaptive coefficient is used on top the Heavy Ball method. For strongly convex function with Lipschitz gradients we set γ C = 1 −ḡ ·d and for non-convex cases γ N C = 1 +ḡ ·d. Here we include the values of the adaptive coefficient during optimization from our experiments. where w i = 1 + θi−1 4 for i = 1, 2. We initialize all three methods at. Scaled Goldstein-Price Function The scaled Godstein-Price function (Surjanovic & Bingham;) features several local minima, ravines and plateaus which can be representative whereθ i = 4θ i − 2 for i = 1, 2. We initialize all methods at (1.5, 1.5). Details The momentum parameter µ for both Nesterov and Geodesic-N was set to 0.9. The learning rate for all methods is fixed and is tuned for best performance. The from both experiments indicate that Geodesic is able to effectively escape local minima and recover from basins of attraction, while Nesterov's method gets stuck at local minima in both cases. We can also observe the effect of lookahead gradient on our method where the path taken by Geodesic-N is much smoother than Geodesic.

Represent the short summary for finding the corresponding scientific article (long)!

We utilize an adaptive coefficient on top of regular momentum inspired by geodesic optimization which significantly speeds up training in both convex and non-convex functions.

Represent the TLDR for finding the corresponding scientific article

We propose a hypothesis for why gradient descent generalizes based on how per-example gradients interact with each other.