Question: represent the article for finding a single-sentence tldr

Question: Recent trends of incorporating attention mechanisms in vision have led researchers to reconsider the supremacy of convolutional layers as a primary building block. Beyond helping CNNs to handle long-range dependencies, showed that attention can completely replace convolution and achieve state-of-the-art performance on vision tasks. This raises the question: do learned attention layers operate similarly to convolutional layers? This work provides evidence that attention layers can perform convolution and, indeed, they often learn to do so in practice. Specifically, we prove that a multi-head self-attention layer with sufficient number of heads is at least as expressive as any convolutional layer. Our numerical experiments then show that self-attention layers attend to pixel-grid patterns similarly to CNN layers, corroborating our analysis. Our code is publicly available. Recent advances in Natural Language Processing (NLP) are largely attributed to the rise of the transformer . Pre-trained to solve an unsupervised task on large corpora of text, transformer-based architectures, such as GPT-2 , BERT and Transformer-XL, seem to possess the capacity to learn the underlying structure of text and, as a consequence, to learn representations that generalize across tasks. The key difference between transformers and previous methods, such as recurrent neural networks and convolutional neural networks (CNN), is that the former can simultaneously attend to every word of their input sequence. This is made possible thanks to the attention mechanism-originally introduced in Neural Machine Translation to better handle long-range dependencies . With self-attention in particular, the similarity of two words in a sequence is captured by an attention score measuring the distance of their representations. The representation of each word is then updated based on those words whose attention score is highest. Inspired by its capacity to learn meaningful inter-dependencies between words, researchers have recently considered utilizing self-attention in vision tasks. Self-attention was first added to CNN by either using channel-based attention or non-local relationships across the image . More recently, augmented CNNs by replacing some convolutional layers with self-attention layers, leading to improvements on image classification and object detection tasks. noticed that, even though state-of-the art are reached when attention and convolutional features are combined, under same computation and model size constraints, self-attention-only architectures also reach competitive image classification accuracy. These findings raise the question, do self-attention layers process images in a similar manner to convolutional layers? From a theoretical perspective, one could argue that transfomers have the capacity to simulate any function-including a CNN. Indeed, Pérez et al. showed that a multilayer attention-based architecture with additive positional encodings is Turing complete under some strong theoretical assumptions, such as unbounded precision arithmetic. Unfortunately, universality do not reveal how a machine solves a task, only that it has the capacity to do so. Thus, the question of how self-attention layers actually process images remains open. We here recall the mathematical formulation of self-attention layers and emphasize the role of positional encodings. Let X ∈ R T ×Din be an input matrix consisting of T tokens in of D in dimensions each. While in NLP each token corresponds to a word in a sentence, the same formalism can be applied to any sequence of T discrete objects, e.g. pixels. A self-attention layer maps any query token t ∈ [T] from D in to D out dimensions as follows: Self-Attention(X) t,::= softmax (A t,:) XW val, where we refer to the elements of the T × T matrix A:= XW qry W key X as attention scores and the softmax output 3 as attention probabilities. The layer is parametrized by a query matrix W qry ∈ R Din×D k, a key matrix W key ∈ R Din×D k and a value matrix W val ∈ R Din×Dout.For simplicity, we exclude any residual connections, batch normalization and constant factors. A key property of the self-attention model described above is that it is equivariant to reordering, that is, it gives the same output independently of how the T input tokens are shuffled. This is problematic for cases we expect the order of things to matter. To alleviate the limitation, a positional encoding is learned for each token in the sequence (or pixel in an image), and added to the representation of the token itself before applying self-attention where P ∈ R T ×Din contains the embedding vectors for each position. More generally, P may be substituted by any function that returns a vector representation of the position. It has been found beneficial in practice to replicate this self-attention mechanism into multiple heads, each being able to focus on different parts of the input by using different query, key and value matrices. In multi-head self-attention, the output of the N h heads of output dimension D h are concatenated and projected to dimension D out as follows: and two new parameters are introduced: the projection matrix W out ∈ R N h D h ×Dout and a bias term b out ∈ R Dout. Convolutional layers are the de facto choice for building neural networks that operate on images. We recall that, given an image tensor X ∈ R W ×H×Din of width W, height H and D in channels, the output of a convolutional layer for pixel (i, j) is given by where W is the K × K × D in × D out weight tensor 4, b ∈ R Dout is the bias vector and the set contains all possible shifts appearing when convolving the image with a K × K kernel. In the following, we review how self-attention can be adapted from 1D sequences to images. With images, rather than tokens, we have query and key pixels q, k. Accordingly, the input is a tensor X of dimension W × H × D in and each attention score associates a query and a key pixel. To keep the formulas consistent with the 1D case, we abuse notation and slice tensors by using a 2D index vector: if p = (i, j), we write X p,: and A p,: to mean X i,j,: and A i,j,:,:, respectively. With this notation in place, the multi-head self attention layer output at pixel q can be expressed as follows: and accordingly for the multi-head case. There are two types of positional encoding that has been used in transformer-based architectures: the absolute and relative encoding (see also Table 3 in the Appendix). With absolute encodings, a (fixed or learned) vector P p,: is assigned to each pixel p. The computation of the attention scores we saw in eq. can then be decomposed as follows: A abs q,k = (X q,: + P q,:)W qry W key (X k,: + P k,:) = X q,: W qry W key X k,: + X q,: W qry W key P k,: + P q,: W qry W key X k,: + P q,: W qry W key P k,: where q and k correspond to the query and key pixels, respectively. The relative positional encoding was introduced by. The main idea is to only consider the position difference between the query pixel (pixel we compute the representation of) and the key pixel (pixel we attend) instead of the absolute position of the key pixel: In this manner, the attention scores only depend on the shift δ:= k − q. Above, the learnable vectors u and v are unique for each head, whereas for every shift δ the relative positional encoding r δ ∈ R Dp is shared by all layers and heads. Moreover, now the key weights are split into two types: W key pertain to the input and W key to the relative position of pixels. This section derives sufficient conditions such that a multi-head self-attention layer can simulate a convolutional layer. Our main is the following: Theorem 1. A multi-head self-attention layer with N h heads of dimension D h, output dimension D out and a relative positional encoding of dimension D p ≥ 3 can express any convolutional layer of kernel size The theorem is proven constructively by selecting the parameters of the multi-head self-attention layer so that the latter acts like a convolutional layer. In the proposed construction, the attention scores of each self-attention head should attend to a different relative shift within the set ∆ ∆ K = {− K/2, . . ., K/2} 2 of all pixel shifts in a K × K kernel. The exact condition can be found in the statement of Lemma 1. Then, Lemma 2 shows that the aforementioned condition is satisfied for the relative positional encoding that we refer to as the quadratic encoding: The learned parameters 2 ) and α (h) determine the center and width of attention of each head, respectively. On the other hand, δ = (δ 1, δ 2) is fixed and expresses the relative shift between query and key pixels. It is important to stress that the above encoding is not the only one for which the conditions of Lemma 1 are satisfied. In fact, in our experiments, the relative encoding learned by the neural network also matched the conditions of the lemma (despite being different from the quadratic encoding). Nevertheless, the encoding defined above is very efficient in terms of size, as only D p = 3 dimensions suffice to encode the relative position of pixels, while also reaching similar or better empirical performance (than the learned one). The theorem covers the general convolution operator as defined in eq.. However, machine learning practitioners using differential programming frameworks might question if the theorem holds for all hyper-parameters of 2D convolutional layers: • Padding: a multi-head self-attention layer uses by default the "SAME" padding while a convolutional layer would decrease the image size by K − 1 pixels. The correct way to alleviate these boundary effects is to pad the input image with K/2 zeros on each side. In this case, the cropped output of a MHSA and a convolutional layer are the same. • Stride: a strided convolution can be seen as a convolution followed by a fixed pooling operation-with computational optimizations. Theorem 1 is defined for stride 1, but a fixed pooling layer could be appended to the Self-Attention layer to simulate any stride. • Dilation: a multi-head self-attention layer can express any dilated convolution as each head can attend a value at any pixel shift and form a (dilated) grid pattern. Remark for the 1D case. Convolutional layers acting on sequences are commonly used in the literature for text , as well as audio (van den) and time series . Theorem 1 can be straightforwardly extended to show that multi-head self-attention with N h heads can also simulate a 1D convolutional layer with a kernel of size K = N h with min(D h, D out) output channels using a positional encoding of dimension D p ≥ 2. Since we have not tested empirically if the preceding construction matches the behavior of 1D self-attention in practice, we cannot claim that it actually learns to convolve an input sequence-only that it has the capacity to do so. The proof follows directly from Lemmas 1 and 2 stated below: be a bijective mapping of heads onto shifts. Further, suppose that for every head the following holds: Then, for any convolutional layer with a K × K kernel and D out output channels, there exists {W Attention maps for pixel val . We show attention maps computed for a query pixel at position q. Proof. Our first step will be to rework the expression of the Multi-Head Self-Attention operator from equation and equation such that the effect of the multiple heads becomes more transparent: Note that each head's value matrix W (h) val ∈ R Din×D h and each block of the projection matrix W out of dimension D h × D out are learned. Assuming that D h ≥ D out, we can replace each pair of matrices by a learned matrix W (h) for each head. We consider one output pixel of the multi-head self-attention: Due to the conditions of the Lemma, for the h-th attention head the attention probability is one when k = q − f (h) and zero otherwise. The layer's output at pixel q is thus equal to For K = √ N h, the above can be seen to be equivalent to a convolutional layer expressed in eq. 17: there is a one to one mapping (implied by map f) between the matrices 2. Remark about D h and D out. It is frequent in transformer-based architectures to set In that case, W (h) can be seen to be of rank D out − D h, which does not suffice to express every convolutional layer with D out channels. Nevertheless, it can be seen that any D h out of D out outputs of MHSA(X) can express the output of any convolutional layer with D h output channels. To cover both cases, in the statement of the main theorem we assert that the output channels of the convolutional layer should be min(D h, D out). In practice, we advise to concatenate heads of dimension D h = D out instead of splitting the D out dimensions among heads to have exact re-parametrization and no "unused" channels. Lemma 2. There exists a relative encoding scheme {r δ ∈ R Dp} δ∈Z 2 with D p ≥ 3 and parame- Proof. We show by construction the existence of a D p = 3 dimensional relative encoding scheme yielding the required attention probabilities. As the attention probabilities are independent of the input tensor X, we set W key = W qry = 0 which leaves only the last term of eq.. Setting W key ∈ R D k ×Dp to the identity matrix (with appropriate row padding), yields A q,k = v r δ where δ:= k − q. Above, we have assumed that D p ≤ D k such that no information from r δ is lost. Now, suppose that we could write: for some constant c. In the above expression, the maximum attention score over A q,: is −αc and it is reached for A q,k with δ = ∆. On the other hand, the α coefficient can be used to scale arbitrarily the difference between A q,∆ and the other attention scores. In this way, for δ = ∆, we have and for δ = ∆, the equation becomes lim α→∞ softmax(A q,:) k = 0, exactly as needed to satisfy the lemma statement. What remains is to prove that there exist v and {r δ} δ∈Z 2 for which eq. holds. Expanding the RHS of the equation, we have which matches eq. with c = − ∆ 2 and the proof is concluded. Remark on the magnitude of α. The exact representation of one pixel requires α (or the matrices W qry and W key) to be arbitrary large, despite the fact that the attention probabilities of all other pixels converge exponentially to 0 as α grows. Nevertheless, practical implementations always rely on finite precision arithmetic for which a constant α suffices to satisfy our construction. For instance, since the smallest positive float32 scalar is approximately 10 −45, setting α = 46 would suffice to obtain hard attention. The aim of this section is to validate the applicability of our theoretical -which state that self-attention can perform convolution-and to examine whether self-attention layers in practice do actually learn to operate like convolutional layers when trained on standard image classification tasks. In particular, we study the relationship between self-attention and convolution with quadratic and learned relative positional encodings. We find that, for both cases, the attention probabilities learned tend to respect the conditions of Lemma 1, supporting our hypothesis. We study a fully attentional model consisting of six multi-head self-attention layers. As it has already been shown by that combining attention features with convolutional features improves performance on Cifar-100 and ImageNet, we do not focus on attaining state-of-the-art performance. Nevertheless, to validate that our model learns a meaningful classifier, we compare it to the standard ResNet18 on the CIFAR-10 dataset (Krizhevsky et al.). In all experiments, we use a 2 × 2 invertible down-sampling on the input to reduce the size of the image. As the size of the attention coefficient tensors (stored during forward) scales quadratically with the size of the input image, full attention cannot be applied to bigger images. The fixed size representation of the input image is computed as the average pooling of the last layer representations and given to a linear classifier. We used the PyTorch library and based our implementation on PyTorch Transformers 5. We release our code on Github 6 and hyper-parameters are listed in Table 2 (Appendix). Remark on accuracy. To verify that our self-attention models perform reasonably well, we display in Figure 6 the evolution of the test accuracy on CIFAR-10 over the 300 epochs of training for our self-attention models against a small ResNet (Table 1). The ResNet is faster to converge, but we cannot ascertain whether this corresponds to an inherent property of the architecture or an artifact of the adopted optimization procedures. Our implementation could be optimized to exploit the locality of Gaussian attention probabilities and reduce significantly the number of FLOPS. We observed that learned embeddings with content-based attention were harder to train probably due to their increased number of parameters. We believe that the performance gap can be bridged to match the ResNet performance, but this is not the focus of this work. As a first step, we aim to verify that, with the relative position encoding introduced in equation, attention layers learn to behave like convolutional layers. We train nine attention heads at each layer to be on par with the 3 × 3 kernels used predominantly by the ResNet architecture. The center of attention of each head h is initialized to ∆ (h) ∼ N (0, 2I 2). Figure 3 shows how the initial positions of the heads (different colors) at layer 4 changed during training. We can see that after optimization, the heads attend on specific pixel of the image forming a grid around the query pixel. Our intuition that Self-Attention applied to images learns convolutional filters around the queried pixel is confirmed. Figure 4 displays all attention head at each layer of the model at the end of the training. It can be seen that in the first few layers the heads tend to focus on local patterns (layers 1 and 2), while deeper layers (layers 3-6) also attend to larger patterns by positioning the center of attention further from the queried pixel position. We also include in the Appendix a plot of the attention positions for a higher number of heads (N h = 16). Figure 14 displays both local patterns similar to CNN and long range dependencies. Interestingly, attention heads do not overlap and seem to take an arrangement maximizing the coverage of the input space. Figure 4: Centers of attention of each attention head (different colors) for the 6 self-attention layers using quadratic positional encoding. The central black square is the query pixel, whereas solid and dotted circles represent the 50% and 90% percentiles of each Gaussian, respectively. We move on to study the positional encoding used in practice by fully-attentional models on images. We implemented the 2D relative positional encoding scheme used by (; : we learn a D p /2 position encoding vector for each row and each column pixel shift. Hence, the relative positional encoding of a key pixel at position k with a query pixel at position q is the concatenation of the row shift embedding δ 1 and the column shift embedding δ 2 (where δ = k − q). We chose D p = D out = 400 in the experiment. We differ from their (unpublished) implementation in the following points: (i) we do not use convolution stem and ResNet bottlenecks for downsampling, but only a 2 × 2 invertible downsampling layer At first, we discard the input data and compute the attention scores solely as the last term of eq.. The attention probabilities of each head at each layer are displayed on Figure 5. The figure confirms our hypothesis for the first two layers and partially for the third: even when left to learn the positional encoding scheme from randomly initialized vectors, certain self-attention heads (depicted on the left) learn to attend to individual pixels, closely matching the condition of Lemma 1 and thus Theorem 1. At the same time, other heads pay attention to horizontally-symmetric but non-localized patterns, as well as to long-range pixel inter-dependencies. We move on to a more realistic setting where the attention scores are computed using both positional and content-based attention (i.e., q k + q r in ) which corresponds to a full-blown standalone self-attention model. The attention probabilities of each head at each layer are displayed in Figure 6. We average the attention probabilities over a batch of 100 test images to outline the focus of each head and remove the dependency on the input image. Our hypothesis is confirmed for some heads of layer 2 and 3: even when left to learn the encoding from the data, certain self-attention heads only exploit positionbased attention to attend to distinct pixels at a fixed shift from the query pixel reproducing the receptive field of a convolutional kernel. Other heads use more content-based attention (see Figures 8 to 10 in Appendix for non-averaged probabilities) leveraging the advantage of Self-Attention over CNN which does not contradict our theory. In practice, it was shown by that combining CNN and self-attention features outperforms each taken separately. Our experiments shows that such combination is learned when optimizing an unconstrained fully-attentional model. The similarity between convolution and multi-head self-attention is striking when the query pixel is slid over the image: the localized attention patterns visible in Figure 6 follow the query pixel. This characteristic behavior materializes when comparing Figure 6 with the attention probabilities at a different query pixel (see Figure 7 in Appendix). Attention patterns in layers 2 and 3 are not only localized but stand at a constant shift from the query pixel, similarly to convolving the receptive field of a convolutional kernel over an image. This phenomenon is made evident on our interactive website 7. This tool is designed to explore different components of attention for diverse images with or without content-based attention. We believe that it is a useful instrument to further understand how MHSA learns to process images. Figure 6: Attention probabilities for a model with 6 layers (rows) and 9 heads (columns) using learned relative positional encoding and content-content based attention. Attention maps are averaged over 100 test images to display head behavior and remove the dependence on the input content. The black square is the query pixel. More examples are presented in Appendix A. In this section, we review the known differences and similarities between CNNs and transformers. The use of CNN networks for text-at word level or character level -is more seldom than transformers (or RNN). Transformers and convolutional models have been extensively compared empirically on tasks of Natural Language Processing and Neural Machine Translation. It was observed that transformers have a competitive advantage over convolutional model applied to text . It is only recently that; used transformers on images and showed that they achieve similar accuracy as ResNets. However, their comparison only covers performance and number of parameters and FLOPS but not expressive power. Beyond performance and computational-cost comparisons of transformers and CNN, the study of expressiveness of these architectures has focused on their ability to capture long-term dependencies. Another interesting line of research has demonstrated that transformers are Turingcomplete (; Pérez et al., 2019), which is an important theoretical but is not informative for practitioners. To the best of our knowledge, we are the first to show that the class of functions expressed by a layer of self-attention encloses all convolutional filters. The closest work in bridging the gap between attention and convolution is due to. They cast attention and convolution into a unified framework leveraging tensor outerproduct. In this framework, the receptive field of a convolution is represented by a "basis" tensor A ∈ R K×K×H×W ×H×W. For instance, the receptive field of a classical K × K convolutional kernel would be encoded by A ∆,q,k = 1{k − q = ∆} for ∆ ∈ ∆ ∆ K. The author distinguishes this index-based convolution with content-based convolution where A is computed from the value of the input, e.g., using a key/query dot-product attention. Our work moves further and presents sufficient conditions for relative positional encoding injected into the input content (as done in practice) to allow content-based convolution to express any index-based convolution. We further show experimentally that such behavior is learned in practice. We showed that self-attention layers applied to images can express any convolutional layer (given sufficiently many heads) and that fully-attentional models learn to combine local behavior (similar to convolution) and global attention based on input content. More generally, fully-attentional models seem to learn a generalization of CNNs where the kernel pattern is learned at the same time as the filters-similar to deformable convolutions . Interesting directions for future work include translating existing insights from the rich CNNs literature back to transformers on various data modalities, including images, text and time series. Jean-Baptiste Cordonnier is thankful to the Swiss Data Science Center (SDSC) for funding this work. Andreas Loukas was supported by the Swiss National Science Foundation (project "Deep Learning for Graph Structured Data", grant number PZ00P2 179981). We present more examples of attention probabilities computed by self-attention model. Figure 7 shows average attention at a different query pixel than Figure 6. Figures 8 to 10 display attention for single images. Figure 7: Attention probabilities for a model with 6 layers (rows) and 9 heads (columns) using learned relative positional encoding and content-content attention. We present the average of 100 test images. The black square is the query pixel. Figure 8: Attention probabilities for a model with 6 layers (rows) and 9 heads (columns) using learned relative positional encoding and content-content based attention. The query pixel (black square) is on the frog head. Figure 9: Attention probabilities for a model with 6 layers (rows) and 9 heads (columns) using learned relative positional encoding and content-content based attention. The query pixel (black square) is on the horse head. Figure 10: Attention probabilities for a model with 6 layers (rows) and 9 heads (columns) using learned relative positional encoding and content-content based attention. The query pixel (black square) is on the building in the . Proof. Our first step will be to rework the expression of the Multi-Head Self-Attention operator from equation and equation such that the effect of the multiple heads becomes more transparent: Note that each head's value matrix W (h) val ∈ R Din×D h and each block of the projection matrix W out of dimension D h × D out are learned. Assuming that D h ≥ D out, we can replace each pair of matrices by a learned matrix W (h) for each head. We consider one output pixel of the multi-head self-attention and drop the bias term for simplicity: with a q,: ) k. We rewrite the output of a convolution at pixel q in the same manner: Equality between equations and has a restricted support: only the columns associated with a pixel shift ∆ ∈ ∆ ∆ K in the receptive field of pixel q can be non-zero. This leads to the factorization Figure 11 where W conv ∈ R Necessary. Assume there exists x ∈ R HW such that x ∈ row(E q) and x ∈ row(A q) and set x to be a row of V We noticed the similarity of the attention probabilities in the quadratic positional encoding (Section 3) to isotropic bivariate Gaussian distributions with bounded support: Building on this observation, we further extended our attention mechanism to non-isotropic Gaussian distribution over pixel positions. Each head is parametrized by a center of attention ∆ and a covariance matrix Σ to obtain the following attention scores, where, once more, δ = k − q. The last term can be discarded because the softmax is shift invariant and we rewrite the attention coefficient as a dot product between the head target vector v and the relative position encoding r δ (consisting of the first and second order combinations of the shift in pixels δ): Evaluation. We trained our model using this generalized quadratic relative position encoding. We were curious to see if, using the above encoding the self-attention model would learn to attend to non-isotropic groups of pixels-thus forming unseen patterns in CNNs. Each head was parametrized by ∆ ∈ R 2 and Σ −1/2 ∈ R 2×2 to ensure that the covariance matrix remained positive semi-definite. We initialized the center of attention to ∆ (h) ∼ N (0, 2I 2) and Σ −1/2 = I 2 + N (0, 0.01I 2) so that initial attention probabilities were close to an isotropic Gaussian. Figure 12 shows that the network did learn non-isotropic attention probability patterns, especially in high layers. Nevertheless, the fact that we do not obtain any performance improvement seems to suggest that attention non-isotropy is not particularly helpful in practice-the quadratic positional encoding suffices. Figure 12: Centers of attention of each attention head (different colors) for the 6 self-attention layers using non-isotropic Gaussian parametrization. The central black square is the query pixel, whereas solid and dotted circles represent the 50% and 90% percentiles of each Gaussian, respectively. Pruning degenerated heads. Some non-isotropic attention heads attend on "non-intuitive" patches of pixels: either attending a very thin stripe of pixels, when Σ −1 was almost singular, or attending all pixels uniformly, when Σ −1 was close to 0 (i.e. constant attention scores). We asked ourselves, are such attention patterns indeed useful for the model or are these heads degenerated and unused? To find out, we pruned all heads having largest eigen-values smaller than 10 −5 or condition number (ratio of the biggest and smallest eigen-values) greater than 10 5. Specifically in our model with 6-layer and 9-heads each, we pruned heads from the first to the last layer. This means that these layers cannot express a 3 × 3 kernel anymore. As shown in yellow on fig. 2, this ablation initially hurts a bit the performance, probably due to off biases, but after a few epochs of continued training with a smaller learning rate (divided by 10) the accuracy recovers its unpruned value. Hence, without sacrificing performance, we reduce the size of the parameters and the number of FLOPS by a fourth. For completeness, we also tested increasing the number of heads of our architecture from 9 to 16. Similar to Figure 4, we see that the network distinguishes two main types of attention patterns. Localized heads (i.e., those that attend to nearly individual pixels) appear more frequently in the first few layers. The self-attention layer uses these heads to act in a manner similar to how convolutional layers do. Heads with less-localized attention become more common at higher layers.

A self-attention layer can perform convolution and often learns to do so in practice.

Represent the short summary for finding the corresponding scientific article (long)

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We introduce a “learning-based” algorithm for the low-rank decomposition problem: given an $n \times d$ matrix $A$, and a parameter $k$, compute a rank-$k$ matrix $A'$ that minimizes the approximation loss $||A- A'||_F$. The algorithm uses a training set of input matrices in order to optimize its performance. Specifically, some of the most efficient approximate algorithms for computing low-rank approximations proceed by computing a projection $SA$, where $S$ is a sparse random $m \times n$ “sketching matrix”, and then performing the singular value decomposition of $SA$. We show how to replace the random matrix $S$ with a “learned” matrix of the same sparsity to reduce the error. Our experiments show that, for multiple types of data sets, a learned sketch matrix can substantially reduce the approximation loss compared to a random matrix $S$, sometimes by one order of magnitude. We also study mixed matrices where only some of the rows are trained and the remaining ones are random, and show that matrices still offer improved performance while retaining worst-case guarantees. The success of modern machine learning made it applicable to problems that lie outside of the scope of "classic AI". In particular, there has been a growing interest in using machine learning to improve the performance of "standard" algorithms, by fine-tuning their behavior to adapt to the properties of the input distribution, see e.g.,. This "learning-based" approach to algorithm design has attracted a considerable attention over the last few years, due to its potential to significantly improve the efficiency of some of the most widely used algorithmic tasks. Many applications involve processing streams of data (video, data logs, customer activity etc) by executing the same algorithm on an hourly, daily or weekly basis. These data sets are typically not "random" or "worst-case"; instead, they come from some distribution which does not change rapidly from execution to execution. This makes it possible to design better algorithms tailored to the specific data distribution, trained on past instances of the problem. The method has been particularly successful in the context of compressed sensing. In the latter framework, the goal is to recover an approximation to an n-dimensional vector x, given its "linear measurement" of the form Sx, where S is an m × n matrix. Theoretical show that, if the matrix S is selected at random, it is possible to recover the k largest coefficients of x with high probability using a matrix S with m = O(k log n) rows. This guarantee is general and applies to arbitrary vectors x. However, if vectors x are selected from some natural distribution (e.g., they represent images), recent works show that one can use samples from that distribution to compute matrices S that improve over a completely random matrix in terms of the recovery error. Compressed sensing is an example of a broader class of problems which can be solved using random projections. Another well-studied problem of this type is low-rank decomposition: given an n × d matrix A, and a parameter k, compute a rank-k matrix Low-rank approximation is one of the most widely used tools in massive data analysis, machine learning and statistics, and has been a subject of many algorithmic studies. In particular, multiple algorithms developed over the last decade use the "sketching" approach, see e.g.,. Its idea is to use efficiently computable random projections (a.k.a., "sketches") to reduce the problem size before performing low-rank decomposition, which makes the computation more space and time efficient. For example, show that if S is a random matrix of size m × n chosen from an appropriate distribution, for m depending on, then one can recover a rank-k matrix A such that by performing an SVD on SA ∈ R m×d followed by some post-processing. Typically the sketch length m is small, so the matrix SA can be stored using little space (in the context of streaming algorithms) or efficiently communicated (in the context of distributed algorithms). Furthermore, the SVD of SA can be computed efficiently, especially after another round of sketching, reducing the overall computation time. See the survey for an overview of these developments. In light of the aforementioned work on learning-based compressive sensing, it is natural to ask whether similar improvements in performance could be obtained for other sketch-based algorithms, notably for low-rank decompositions. In particular, reducing the sketch length m while preserving its accuracy would make sketch-based algorithms more efficient. Alternatively, one could make sketches more accurate for the same values of m. This is the problem we address in this paper. Our Results. Our main finding is that learned sketch matrices can indeed yield (much) more accurate low-rank decompositions than purely random matrices. We focus our study on a streaming algorithm for low-rank decomposition due to, described in more detail in Section 2. Specifically, suppose we have a training set of matrices Tr = {A 1, . . ., A N} sampled from some distribution D. Based on this training set, we compute a matrix S * that (locally) minimizes the empirical loss where SCW(S *, A i) denotes the output of the aforementioned Sarlos-Clarkson-Woodruff streaming low-rank decomposition algorithm on matrix A i using the sketch matrix S *. Once the the sketch matrix S * is computed, it can be used instead of a random sketch matrix in all future executions of the SCW algorithm. We demonstrate empirically that, for multiple types of data sets, an optimized sketch matrix S * can substantially reduce the approximation loss compared to a random matrix S, sometimes by one order of magnitude (see Figure 1). Equivalently, the optimized sketch matrix can achieve the same approximation loss for lower values of m. A possible disadvantage of learned sketch matrices is that an algorithm that uses them no longer offers worst-case guarantees. As a , if such an algorithm is applied to an input matrix that does not conform to the training distribution, the might be worse than if random matrices were used. To alleviate this issue, we also study mixed sketch matrices, where (say) half of the rows are trained and the other half are random. We observe that if such matrices are used in conjunction with the SCW algorithm, its are no worse than if only the random part of the matrix was used 2. Thus, the ing algorithm inherits the worst-case performance guarantees of the random part of the sketching matrix. At the same time, we show that mixed matrices still substantially reduce the approximation loss compared to random ones, in some cases nearly matching the performance of "pure" learned matrices with the same number of rows. Thus, mixed random matrices offer "the best of both worlds": improved performance for matrices from the training distribution, and worst-case guarantees otherwise. Notation. Consider a distribution D on matrices A ∈ R n×d. We define the training set as {A 1, · · ·, A N} sampled from D. For matrix A, its singular value decomposition (SVD) can be written as A = U ΣV such that both U and V have orthonormal columns and Σ = diag{λ 1, · · ·, λ d} is a diagonal matrix with nonnegative entries. In many applications it is quicker and more economical to Algorithm 1 Rank-k approximation of a matrix A using a sketch matrix S (from Section 4.1.1 of How sketching works. We start by describing the SCW algorithm (Algorithm 1) for low-rank approximation. The algorithm computes the SVD(SA):= U ΣV, and compute the best rank-k approximation of AV. Finally it outputs [AV] k V as a rank-k approximation of A. Note that if m is much smaller than d and n, the space bound of this algorithm is significantly better than when computing a rank-k approximation for A in the naïve way. Thus, minimizing m automatically reduces the space usage of the algorithm. Sketching matrix. We use matrix S that is sparse. Specifically, each column of S has exactly one non-zero entry, which is either +1 or −1. This means that the fraction of non-zero entries in S is 1/m. Therefore, one can use a vector to represent S, which is very memory efficient. It is worth noting, however, after multiplying S with other matrices, the ing matrix is in general not sparse. In this section, we describe our learning-based algorithm for computing a data dependent sketch S. The main idea is to use backpropagation algorithm to compute the stochastic gradient of S with respect to the rank-k approximation loss in Equation 1, where the initial value of S is the same random sparse matrix used in SCW. Once we have the stochastic gradient, we can run stochastic gradient descent (SGD) algorithm to optimize S, in order to improve the loss. Our algorithm maintains the sparse structure of S, and only optimizes the values of the n non-zero entries (initially +1 or −1). However, the standard SVD implementation (step 2 in Algorithm 1) is not differentiable, which means we cannot get the gradient in the straightforward way. To make SVD implementation differentiable, we use the fact that the SVD procedure can be represented as m individual top singular value decompositions (see e.g. ), and that every top singular value decomposition can be computed using the power method. The full description is deferred to the long version of this paper. Due to the extremely long computational chain, it is infeasible to write down the explicit form of loss function or the gradients. However, just like how modern deep neural networks compute their gradients, we used the autograd feature in PyTorch to numerically compute the gradient with respect to the sketching matrix S. We emphasize again that our method is only optimizing S for the training phase. After S is fully trained, we still call Algorithm 1 for low rank approximation, which has exactly the same running time as the SCW algorithm, but with better performance. The main question considered in this paper is whether, for natural matrix datasets, optimizing the sketch matrix S can improve the performance of the sketching algorithm for the low-rank decomposition problem. To answer this question, we implemented and compared the following methods for computing S ∈ R m×n. • Sparse Random. Sketching matrices are generated at random as in. Specifically, we select a random hash function h:, and for all i ∈ [n], S h[i],i is selected to be either +1 or −1 with equal probability. All other entries in S are set to 0. • Dense Random. All entries in the sketching matrices are sampled from Gaussian distribution. • Learned. Using the sparse random matrix as the initialization, we optimize the sketching matrix using the training set, and return the optimized matrix. • Mixed (J). We first generate two sparse random matrices S 1, S 2 ∈ R m 2 ×n (assuming m is even), and define S to be their combination. We then optimize S using the training set, but only S 1 will be updated, while S 2 is fixed. Therefore, S is a mixture of learned matrix and random matrix, and the first matrix is trained jointly with the second one. • Mixed (S). We first compute a learned matrix S 1 ∈ R m 2 ×n using the training set, and then append another sparse random matrix S 2 to get S ∈ R m×n. Therefore, S is a mixture of learned matrix and random matrix, but the learned matrix is trained separately. Datasets. We used a variety of datasets to test the performance of our methods: • Videos 3: Logo, Friends, Eagle. We downloaded three high resolution videos from Youtube, including logo video, Friends TV show, and eagle nest cam. From each video, we collect 500 frames of size 1920 × 1080 × 3 pixels, and use 400 matrices as the training (test) set. For each frame, we resize it as a 5760 × 1080 matrix. • Hyper. We use matrices from HS-SOD, a dataset for hyperspectral images from natural scenes. Each matrix has 1024 × 768 pixels, and we use 400 matrices as the training (test) set. • Tech. We use matrices from TechTC-300, a dataset for text categorization. Each matrix has 835, 422 rows, but on average only 25, 389 of the rows contain non-zero entries. On average each matrix has 195 columns. We use 200 matrices as the training (test) set. To evaluate the quality of a sketching matrix S, it suffices to evaluate the output of Algorithm 1 using the sketching matrix S on different input matrices A. For a collection of matrices Te, we define the error of the sketch S as Err(Te, S) where the second term denotes the optimal approximation loss on Te. In our datasets, some of the matrices have much larger singular values than the others. To avoid imbalance in the dataset, we normalize the matrices so that their top singular values are all equal. We first test all methods on different datasets, with various combination of k, m. See Figure 1 for the when k = 10, m = 20. As we can see, for video datasets, learned sketching matrices can get 20× better test error than the sparse random or dense random sketching matrices. For other datasets, learned sketching matrices are still more than 2× better. We also include the test error in Table 1 for the case when k = 20, 30. In Table 2, we investigate the performance of the mixed sketching matrices by comparing them with random and learned sketching matrices. In all scenarios, mixed sketching matrices yield much better than random sketching matrices, and sometimes the are comparable to those of learned sketching matrices. This means, in most cases it suffices to train one half of the sketching matrix to obtain good empirical , and at the same time, we can use the remaining random half of the sketch matrix to obtain worst-case guarantees.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Learning-based algorithms can improve upon the performance of classical algorithms for the low-rank approximation problem while retaining the worst-case guarantee.

Represent the TLDR for finding the corresponding scientific article

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the article for finding a single-sentence TLDR

Question: Neural conversational models are widely used in applications like personal assistants and chat bots. These models seem to give better performance when operating on word level. However, for fusion languages like French, Russian and Polish vocabulary size sometimes become infeasible since most of the words have lots of word forms. We propose a neural network architecture for transforming normalized text into a grammatically correct one. Our model efficiently employs correspondence between normalized and target words and significantly outperforms character-level models while being 2x faster in training and 20\% faster at evaluation. We also propose a new pipeline for building conversational models: first generate a normalized answer and then transform it into a grammatically correct one using our network. The proposed pipeline gives better performance than character-level conversational models according to assessor testing. Neural conversational models BID18 are used in a large number of applications: from technical support and chat bots to personal assistants. While being a powerful framework, they often suffer from high computational costs. The main computational and memory bottleneck occurs at the vocabulary part of the model. Vocabulary is used to map a sequence of input tokens to embedding vectors: one embedding vector is stored for each word in vocabulary. English is de-facto a standard language for training conversational models, mostly for a large number of speakers and simple grammar. In english, words usually have only a few word forms. For example, verbs may occur in present and past tenses, nouns can have singular and plural forms. For many other languages, however, some words may have tens of word forms. This is the case for Polish, Russian, French and many other languages. For these languages storing all forms of frequent words in a vocabulary significantly increase computational costs. To reduce vocabulary size, we propose to normalize input and output sentences by putting them into a standard form. Generated texts can then be converted into grammatically correct ones by solving morphological agreement task. This can be efficiently done by a model proposed in this work. Our contribution is two-fold:• We propose a neural network architecture for performing morphological agreement in fusion languages such as French, Polish and Russian (Section 2).• We introduce a new approach to building conversational models: generating normalized text and then performing morphological agreement with proposed model (Section 3); In this section we propose a neural network architecture for solving morphological agreement problem. We start by formally defining the morphological agreement task. Consider a grammatically correct sentence with words [a 1, a 2, . . ., a K]. Let S(a) be a function that maps any word to its standard form. For example, S("went") = "go". Goal of morphological agreement task is to learn a mapping from normalized sentence [S(a 1), S(a 2),..., S(a K)] = [a n 1,,a n 2, . . ., a n K] to initial sentence [a 1, a 2, . . ., a K]. Interestingly, reverse mapping may be performed for each word independently using specialized dictionaries. Original task, however, needs to consider dependencies between words in sequence in order to output a coherent text. An important property of this task is that the number of words, their order and meaning are explicitly contained in input sequence. To employ this knowledge, we propose a specific neural network architecture illustrated in FIG0. Network operates as follows: first all normalized words are embedded using the same character-level LSTM encoder. The goal of the next step is to incorporate global information about other words to embedding of each word. To do so we pass word embedding sequence through a bidirectional LSTM BID7. This allows new embeddings to get information from all other words: information about previous words is brought by forward LSTM and information about subsequent words is taken from backward LSTM. Finally, new embeddings are decoded with character-level LSTM decoders. At this stage we also added attention BID1 over input characters of corresponding words for better performance. High-level overview of this model resembles sequence-to-sequence BID17 network: model learns to map input characters to output characters for each word using encoder-decoder sceme. The difference is that bidirectional neural network is used to distribute information between different words. Unlike simple character-level sequence-to-sequence model, our architecture allows for much faster evaluation, since encoding and decoding phases can be done in parallel for each word. Also, one of the main advantages of our approach is that information paths from inputs to outputs are much shorter which leads to slower performance degradation as input length increases (see Section 5.3). As discussed above, we propose a two stage approach to build a neural conversational model: first generate normalized answer using normalized question and then apply Concorde model to obtain grammatically correct response. In this section we discuss a modification of Concorde model for conditioning its output on question's morphological features. A modification of Concorde model that we call Q-Concorde uses two sources of input: question and normalized answer. Question is first embedded into a single vector with character-level RNN. This vector may carry important morphological information such as time, case and plurality of questions. Question embedding is then mixed with answer embeddings using linear mapping. The final model is shown in FIG2. DISPLAYFORM0 Most frequently used models for sequence to sequence mapping rely on generating embedding vector that contains all information about input sequence. Reconstruction solely from this vector usually in worse performance as length of the output sequence increases. Attention BID19, BID14, BID1 ) partially fixes this problem, though information bottleneck of embedding vector is still high. Encoder-decoder models have mostly been applied to tasks like speech recognition BID6, BID7, BID5 ), machine translation BID1, BID17, BID2 ) and neural conversational models BID18, BID16 ).Some works have tried to perform decomposition of input sequence to obtain shorter information paths. In input is first processed character-wise and then embeddings that correspond to word endings are used for sequence-to-sequence model. This modification makes input-output information paths shorter which leads to better performance. Word inflection is a most similar task to ours, but in this task model is asked to generate a specific word form while we want our model to automatically select desired word forms. BID3 proposed a supervised approach to predicting the set of all word forms by generating transformation rules from known inflection tables. They also propose to use Conditional Random Fields for unseen base forms. Some authors have also tried to apply neural networks for this problem. BID0 and BID4 propose to use bidirectional LSTM to encode the word. Then BID4 uses different decoders for different word forms, while BID0 suggests to have one decoder and to attach morphological features to its input. Besides recurrent networks, there has been an attempt to use convolutional networks. BID15 based his work on BID4 and proposed to first pass raw data through convolutional layers and then to pass them through recurrent encoder. BID9 BID11 library. We leave first 20 words from each sentence to reduce computational costs. Concorde and Q-Concorde models consist of 2-layer LSTM encoder and decoder with hidden size 512. We compare our model to three baselines: unigram charRNN, bigram charRNN and hierarchical model. Unigram and bigram models are standard sequence-to-sequence models with attention BID14 that operate with characters or pairs of characters as tokens. We use 2-layer LSTM as an encoder. Decoder consists of 2-layer LSTM followed by attention layer and another recurrent layer. The third baseline is a hierarchical model motivated by: we first embed each word using recurrent encoder and then compute sentence embedding by running word-level encoder on these embeddings. For baselines we use layer size of 768 which in a comparable number of parameters for all models. We train models with Adam BID10 optimizer in batch size 16 with learning rate 0.0002 that halves after each 50k updates. We terminate training after 300k updates which is enough for all models to converge. To evaluate our model we used French, Russian an Polish corpuses from OpenSubtitles 2 database. We performed morphological agreement for each subtitle line independently. We estimated potential vocabulary size reduction from normalization by selecting words that appeared more than 10 times in first 10M examples. This lead to 2.5 times reduction for Polish language, 2.4 for Russian, and 1.8 for French. We evaluated our model in two metrics: word and sentence accuracies. Word accuracy shows a fraction of words that were correctly generated. Sentence accuracy corresponds to the fraction of sentences that were transformed without any mistake. Results are reported in TAB1. From four models that we compared, our model gave the best performance among all datasets, while second best model was hierarchical model. We inspected our model to show some examples where it was able to infer plural form and gender for unseen words TAB2 ). For Russian language we found out that the model was able to learn some rare rules like changing the letter я to й when going to plural form in some words: один заяц, два зайца (one rabbit, two rabbits).We can also see that our model can infer gender from words. To show that we chose feminine, masculine and neuter words and asked the model to perform agreement with word "one". This word changes its spelling for different genders in French, Polish, and Russian. Results presented in TAB3 suggest that model can indeed correctly solve this task by setting correct gender to the numeral. TAB4 we show on full sentences. Interestingly, on quite a complex Russian example our model was able to perform agreement. To select the correct form of word соседнем (neighbouring), network had to use multiple markers from different parts of a sentence: gender from подъезд (entrance) and case from в (in). As a motivation for our model we argued that making shorter input-output paths may reduce information load of the embedding vector. To check this hypothesis we computed average sentence accuracy for different input lengths and reported in FIG2.We can clearly see that all baseline models perform worse as the input length increases. However, this is not the case for our model -while character-level models perform with almost 0% accuracy when input is 100 characters long, our model still gives similar performance as for short sentences. This can be explained by the way in which models use embedding vectors. Baseline models have to share embedding capacity between all words in a sentence. Our model, however, has a separate embedding for each word and does not require the whole sentence to be squeezed into one vector. It is also clear that character-level models perform better for short sequences (about 33% of the test set). This may be the case since the capacity of the embedding vector is not fully used for them. Despite being worse for short inputs, our model can still handles many important cases quite well including those discussed in Section 5.2. To evaluate our conversation model we constructed a corpus of question-answer pairs from web site Otvet.mail.ru -general topic Russian service for questions and answers (analogue of Quora.com). The uniqueness of this corpus is that it contains general knowledge questions that allow the trained model to answer questions about movies, capitals, etc. This requires many rare entity-related words to be in the vocabulary which makes it extremely large without normalization. First we compared Q-Concorde and Concorde models to show that Q-Concorde can indeed grasp important morphological features form a context. We also trained baseline models with context concatenated to input sentence (with a special delimiter in between). word and sentence accuracies are reported in TAB5. Again, Concorde model was able to outperform baselines even though it didn't have access to the context. Also, Q-Concorde model was able to improve Concorde's performance. We inspected cases on which Q-Concorde model showed better performance than Concorde TAB6 ). In example 1 of this table, question was asked about a single object. Q-Concorde model used singular form, while Concorde used plural. Q-Concorde was also able to successfully carry correct case (example 2) and time (example 3) from the question. Some mistakes made by Q-Concorde model are shown in TAB7. In example 1, Q-Concorde wasn't able to decide whether to use polite form or not and used one word in a less polite form than another. An important property of Q-Concorde model is that it can generate different texts depending on lexical features of a question. For example, in TAB8 we changed question's tense from present simple to past simple ("what do you do?" and "what did you do?"). The model correctly generated answer in corresponding tense. We also tried to change gender of a word "did" in a question: from masculine делал ) to feminine далала. Our model used the correct gender to generate the answer. While model generated grammatically correct answers in all three cases, in the third case (past simple, masculine) model answered in less common form with a meaning that differs from expected one. Finally, we apply Q-Concorde model to a proposed pipeline for training conversational models. We compare our model with a 3-layer character-level sequence-to-sequence model which was trained on grammatically correct sentences. For generating diverse answers we train two models: one to predict answer given question and the other to predict question given answer, as suggested in BID16. This allows us to discard answers that are too general. To compare two models we set an experiment environment where assessors were asked to select one of two possible answers to the given question: one was generated by character-wise model and another was generated by our pipeline. Assessors did not know the order in which cases were shown and so they did not know which model generated the text. In 62.1% cases assessors selected the proposed model, and in the remaining 37.9% assessors preferred character-wise model. We noticed that time for processing one batch is much higher for character-level models since they need to process longer sequences sequentially. In TAB9 we report time for forward and backward pass of one batch (16 objects) and other important computational characteristics. We measured this time on GeForce GTX TITAN X graphic card. It turns out that proposed models have comparable evaluation time, but train faster than unigram and hierarchical models. In this paper we proposed a neural network model that can efficiently employ relationship between input and output words in morphological agreement task. We also proposed a modification for this model that uses context sentence. We apply this model for neural conversational model in a new pipeline: we use normalized question to generate normalized answer and then apply proposed model to obtain grammatically correct response. This model showed better performance than character level neural conversational model based on assessors responses. We achieved significant improvement comparing to character-level, bigram and hierarchical sequenceto-sequence models on morphological agreement task for Russian, French and Polish languages. Trained models seem to understand main grammatical rules and notions such as tenses, cases and pluralities.

represent the short summary for finding the corresponding scientific article (long)

Proposed architecture to solve morphological agreement task

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.


Question: Represent this text

Question: This paper proposes the use of spectral element methods \citep{canuto_spectral_1988} for fast and accurate training of Neural Ordinary Differential Equations (ODE-Nets; \citealp{Chen2018NeuralOD}) for system identification. This is achieved by expressing their dynamics as a truncated series of Legendre polynomials. The series coefficients, as well as the network weights, are computed by minimizing the weighted sum of the loss function and the violation of the ODE-Net dynamics. The problem is solved by coordinate descent that alternately minimizes, with respect to the coefficients and the weights, two unconstrained sub-problems using standard backpropagation and gradient methods. The ing optimization scheme is fully time-parallel and in a low memory footprint. Experimental comparison to standard methods, such as backpropagation through explicit solvers and the adjoint technique \citep{Chen2018NeuralOD}, on training surrogate models of small and medium-scale dynamical systems shows that it is at least one order of magnitude faster at reaching a comparable value of the loss function. The corresponding testing MSE is one order of magnitude smaller as well, suggesting generalization capabilities increase. Neural Ordinary Differential Equations (ODE-Nets;) can learn latent models from observations that are sparse in time. This property has the potential to enhance the performance of neural network predictive models in applications where information is sparse in time and it is important to account for exact arrival times and delays. In complex control systems and model-based reinforcement learning, planning over a long horizon is often needed, while high frequency feedback is necessary for maintaining stability . Discrete-time models, including RNNs , often struggle to fully meet the needs of such applications due to the fixed time resolution. ODE-Nets have been shown to provide superior performance with respect to classic RNNs on time series forecasting with sparse training data. However, learning their parameters can be computationally intensive. In particular, ODE-Nets are memory efficient but time inefficient. In this paper, we address this bottleneck and propose a novel alternative strategy for system identification. We propose SNODE, a compact representation of ODE-Nets for system identification with full state information that makes use of a higher-order approximation of its states by means of Legendre polynomials. This is outlined in Section 4. In order to find the optimal polynomial coefficients and network parameters, we develop a novel optimization scheme, which does not require to solve an ODE at each iteration. The ing algorithm is detailed in Section 3 and is based on backpropagation (; ;) and automatic differentiation . The proposed method is fully parallel with respect to time and its approximation error reduces exponentially with the Legendre polynomial order . Summary of numerical experiments. In Section 5, our method is tested on a 6-state vehicle problem, where it is at least one order or magnitude faster in each optimizer iteration than explicit and adjoint methods, while convergence is achieved in a third of the iterations. At test time, the MSE is reduced by one order of magnitude. In Section 6, the method is used for a 30-state system consisting of identical vehicles, coupled via a known collision avoidance policy. Again, our method converges in a third of the iterations required by backpropagation thourgh a solver and each iteration is 50x faster than the fastest explicit scheme. The minimization of a scalar-valued loss function that depends on the output of an ODE-Net can be formulated as a general constrained optimization problem: where x(t) ∈ X is the state, u(t) ∈ U is the input, the loss and ODE functions L and f are given, and the parameters θ have to be learned. The spaces X and U are typically Sobolev (e.g. Hilbert) spaces expressing the smoothness of x(t) and u(t) (see Section 8). Equation can be used to represent several inverse problems, for instance in machine learning, estimation, and optimal control (; ;). Problem can be solved using gradient-based optimization through several time-stepping schemes for solving the ODE. have proposed to use the adjoint method when f is a neural network. These methods are typically relying on explicit time-stepping schemes . Limitations of these approaches are briefly summarized: Limitations of backpropagation through an ODE solver. The standard approach for solving this problem is to compute the gradients ∂L/∂θ using backpropagation through a discrete approximation of the constraints, such as Runge-Kutta methods (Runge, 1895;) or multistep solvers. This ensures that the solution remains feasible (within a numerical tolerance) at each iteration of a gradient descent method. However, it has several drawbacks: 1) the memory cost of storing intermediate quantities during backpropagation can be significant, 2) the application of implicit methods would require solving a nonlinear equation at each step, 3) the numerical error can significantly affect the solution, and 4) the problem topology can be unsuitable for optimization . Limitations of adjoint methods. ODE-Nets solve using the adjoint method, which consists of simulating a dynamical system defined by an appropriate augmented Hamiltonian , with an additional state referred to as the adjoint. In the backward pass the adjoint ODE is solved numerically to provide the gradients of the loss function. This means that intermediate states of the forward pass do not need to be stored. An additional step of the ODE solver is needed for the backward pass. This suffers from a few drawbacks: 1) the dynamics of either the hidden state or the adjoint might be unstable, due to the symplectic structure of the underlying Hamiltonian system, referred to as the curse of sensitivity in ; 2) the procedure requires solving a differential algebraic equation and a boundary value problem which is complex, time consuming, and might not have a solution . Limitations of hybrid methods. ANODE splits the problem into time batches, where the adjoint is used, storing in memory only few intermediate states from the forward pass. This allows to improve the robustness and generalization of the adjoint method. A similar improvement could be obtained using reversible integrators. However, its computational cost is of the same order of the adjoint method and it does not offer further opportunities for parallelization. Our algorithm is based on two ingredients: i) the discretization of the problem using spectral elements leading to SNODE, detailed in Section 4, and ii) the relaxation of the ODE constraint from, enabling efficient training through backpropagation. The latter can be applied directly at the continuous level and significantly reduces the difficulty of the optimization, as shown in our examples. The problem in is split into two smaller subproblems: one finds the trajectory x(t) that minimizes an unconstrained relaxation of. The other trains the network weights θ such that the trajectory becomes a solution of the ODE. Both are addressed using standard gradient descent and backpropagation. In particular, a fixed number of ADAM or SGD steps is performed for each problem in an alternate fashion, until convergence. In the following, the details of each subproblem are discussed. Step 0: Initial trajectory. The initial trajectory x(t) is chosen by solving the problem If this problem does not have a unique solution, a regularization term is added. For a quadratic loss, a closed-form solution is readily available. Otherwise, a prescribed number of SGD iterations is used. Step 1: Coordinate descent on residual. Once the initial trajectory x (t) is found, θ is computed by solving the unconstrained problem: If the value of the residual at the optimum θ * is smaller than a prescribed tolerance, then the algorithms stops. Otherwise, steps 1 and 2 are iterated until convergence. Step 2: Coordinate descent on relaxation. Once the candidate parameters θ are found, the trajectory is updated by minimizing the relaxed objective: Discussion. The proposed algorithm can be seen as an alternating coordinate gradient descent on the relaxed functional used in problem, i.e., by alternating a minimization with respect to x(t) and θ. If γ = 0, multiple minima can exist, since each choice of the parameters θ would induce a different dynamics x(t), solution of the original constraint. For γ = 0, the loss function in trades-off the ODE solution residual for the data fitting, providing a unique solution. The choice of γ implicitly introduces a satisfaction tolerance (γ), i.e., similar to regularized regression , implying that ẋ(t) − f (t, x(t); θ) ≤ (γ). Concurrently, problem reduces the residual. In order to numerically solve the problems presented in the previous section, a discretization of x(t) is needed. Rather than updating the values at time points t i from the past to the future, we introduce a compact representation of the complete discrete trajectory by means of the spectral element method. Spectral approximation. We start by representing the scalar unknown trajectory, x(t), and the known input, u(t), as truncated series: where x i, u i ∈ R and ψ i (t), ζ i (t) are sets of given basis functions that span the spaces X h ⊂ X and U h ⊂ U. In this work, we use orthogonal Legendre polynomials of order p for, where p is a hyperparameter, and the cosine Fourier basis for ζ i (t), where z is fixed. Collocation and quadrature. In order to compute the coefficients x i of, we enforce the equation at a discrete set Q of collocation points t q. Here, we choose p + 1 Gauss-Lobatto nodes, which include t = t 0. This directly enforces the initial condition. Other choices are also possible . Introducing the vectors of series coefficients and of evaluations at quadrature points x(t Q) = {x(t q)} q∈Q, the collocation problem can be solved in matrix form as We approximate the integral as a sum of residual evaluations over Q. Assuming that x = x 0, the integrand at all quadrature points t Q can be computed as a component-wise norm Fitting the input data. For the case when problem admits a unique a solution, we propose a new direct training scheme, δ-SNODE, which is summarized in Algorithm 1. In general, a least-squares approach must be used instead. This entails computing the integral in, which can be done by evaluating the loss function L at quadrature points t q. If the input data is not available at t q, we approximate the integral by evaluating L at the available time points. The corresponding alternating coordinate descent scheme α-SNODE is presented in Algorithm 2. In the next sections, we study the consequences of a low-data scenario on this approach. We use fixed numbers N t and N x of updates for, respectively, θ and x(t). Both are performed with standard routines, such as SGD. In our experiments, we use ADAM to optimize the parameters and an interpolation order p = 14, but any other orders and solvers are possible. Input: M, D from-. ] end for end while Output: θ * Ease of time parallelization. If R(t q) = 0 is enforced explicitly at q ∈ Q, then the ing discrete system can be seen as an implicit time-stepping method of order p. However, while ODE integrators can only be made parallel across the different components of x(t), the assembly of the residual can be done in parallel also across time. This massively increases the parallelization capabilities of the proposed schemes compared to standard training routines. Memory cost. If an ODE admits a regular solution, with regularity r > p, in the sense of Hilbert spaces, i.e., of the number of square-integrable derivatives, then the approximation error of the SNODE converges exponentially with p . Hence, it produces a very compact representation of an ODE-Net. Thanks to this property, p is typically much lower than the equivalent number of time steps of explicit or implicit schemes with a fixed order. This greatly reduces the complexity and the memory requirements of the proposed method, which can be evaluated at any t via by only storing few x i coefficients. Stability and vanishing gradients. The forward Euler method is known to have a small region of convergence. In other words, integrating very fast dynamics requires a very small time step, dt, in order to provide accurate . In particular, for the solver error to be bounded, the eigenvalues of the state Jacobian of the ODE need to lie into the circle of the complex plane centered at (−1, 0) with radius 1/dt . Higher-order explicit methods, such as Runge-Kutta (Runge, 1895), have larger but still limited convergence regions. Our algorithms on the other hand are implicit methods, which have a larger region of convergence than recursive (explicit) methods . We claim that this in a more stable and robust training. This claim is supported by our experiments. Reducing the time step can improve the Euler accuracy but it can still lead to vanishing or exploding gradients . In Appendix C, we show that our methods do not suffer from this problem. Experiments setup and hyperparameters. For all experiments, a common setup was employed and no optimization of hyperparameters was performed. Time horizon T = 10s and batch size of 100 were used. Learning rates were set to 10 −2 for ADAM (for all methods) and 10 −3 for SGD (for α-SNODE). For the α-SNODE method, γ = 3 and 10 iterations were used for the SGD and ADAM algorithms at each epoch, as outlined in Algorithm 2. The initial trajectory was perturbed as x 0 = x 0 + ξ, ξ ∼ U (−0.1, 0.1). This perturbation prevents the exact convergence of Algorithm 1 during initialization, allowing to perform the alternating coordinate descent algorithm. Let us consider the systemη where η, v ∈ R 3 are the states, u = (F x, 0, τ xy) is the control, C(v) is the Coriolis matrix, d(v) is the (linear) damping force, and J(η) encodes the coordinate transformation from the body to the world frame . A gray-box model is built using a neural network for each matrix Each network consists of two layers, the first with a tanh activation. Bias is excluded for f C and f d. For f J, sin(φ) and cos(φ) are used as input features, where φ is the vehicle orientation. When inserted in, these discrete networks produce an ODE-Net that is a surrogate model of the physical system. The trajectories of the system and the learning curves are shown in Appendix A. Comparison of methods in the high-data regime. In the case of δ-SNODE and α-SNODE, only p + 1 points are needed for the accurate integration of the loss function, if such points coincide with the Gauss-Lobatto quadrature points. We found that 100 equally-spaced points produce a comparable . Therefore, the training performance of the novel and traditional training methods were compared by sampling the trajectories at 100 equally-spaced time points. Table 1a shows that δ-SNODE outperforms BKPR-DoPr5 by a factor of 50, while producing a significantly improved generalization. The speedup reduces to 20 for α-SNODE, which however yields a further reduction of the testing MSE by a factor of 10, as can be seen in Figure 1. Comparison in the low-data regime. The performance of the methods was compared using fewer time points, randomly sampled from a uniform distribution. For the baselines, evenly-spaced points were used. Table 1b shows that α-SNODE preserves a good testing MSE, at the price of an increased number of iterations. With only 25% of data, α-SNODE is 10x faster than BKPR-DoPr5. Moreover, its test MSE is 1/7 than BKPR-DoPr5 and up to 1/70 than BKPR-Euler, showing that the adaptive time step of DoPr5 improves significantly the baseline but it is unable to match the accuracty of the proposed methods. The adjoint method produced the same as the backprop (±2%). Consider the multi-agent system consisting of N a kinematic vehicles: where η ∈ R 3Na are the states (planar position and orientation), J(η i) is the coordinate transform from the body to world frame, common to all agents. The agents velocities are determined by known arbitrary control and collision avoidance policies, respectively, K c and K o plus some additional high frequency measurable signal w = w(t), shared by all vehicles. The control laws are non-linear and are described in detail in Appendix B. We wish to learn their kinematics matrix by means of a neural network as in Section 5. The task is simpler here, but the ing ODE has 3N a states, coupled by K 0. We simulate N a = 10 agents in series. Comparison of methods with full and sparse data. The learning curves for high-data regime are in Figure 3. For method α-SNODE, training was terminated when the loss in is less than γL +R, withL = 0.11 andR = 0.01. For the case of 20% data, we setL = 0.01. Table 2 summarizes . δ-SNODE is the fastest method, followed by α-SNODE which is the best performing. Iteration time of BKPR-Euler is 50x slower, with 14x worse test MSE. ADJ-Euler is the slowest but its test MSE is in between BKPR-Euler and our methods. Random down-sampling of the data by 50% and 20% (evenly-spaced for the baselines) makes ADJ-Euler fall back the most. BKPR-DoPr5 failed to find a time step meeting the tolerances, therefore they were increased to rtol= 10 −5, atol= 10 −7. Since the loss continued to increase, training was terminated at 200 epochs. ADJ-DoPr5 failed to compute gradients. Test trajectories are in Figure 2. Additional details are in Appendix B. Robustness of the methods. The use of a high order variable-step method (DoPr5), providing an accurate ODE solution, does not however lead to good training . In particular, the loss function continued to increase over the iterations. On the other hand, despite being nearly 50 times slower than our methods, the fixed-step forward Euler solver was successfully used for learning the dynamics of a 30-state system in the training configuration described in Appendix B. One should however note that, in this configuration, the gains for the collision avoidance policy K o (which couples the ODE) were set to small values. This makes the system simpler and more stable than having a larger gain. As a , if one attempts to train with the test configuration from Appendix B, where the gains are increased and the system is more unstable, then backpropagating trough Euler simply fails. Comparing Figures 3 and 4, it can be seen that the learning curves of our methods are unaffected by the change in the gains, while BKPR-Euler and ADJ-Euler fail to decrease the loss. RNN training pathologies. One of the first RNNs to be trained successfully were LSTMs , due to their particular architecture. Training an arbitrary RNN effectively is generally difficult as standard RNN dynamics can become unstable or chaotic during training and this can cause the gradients to explode and SGD to fail . When RNNs consist of discretised ODEs, then stability of SGD is intrinsically related to the size of the convergence region of the solver . Since higher-order and implicit solvers have larger convergence region , following it can be argued that our method has the potential to mitigate instabilities and hence to make the learning more efficient. This is supported by our . Unrolled architectures. In , an RNN has been used with a stopping criterion, for iterative estimation with adaptive computation time. Highway and residual networks have been studied in as unrolled estimators. In this context, treated residual networks as autonomous discrete-ODEs and investigated their stability. Finally, in a discrete-time non-autonomous ODE based on residual networks has been made explicitly stable and convergent to an input-dependant equilibrium, then used for adaptive computation. Training stable ODEs. In , ODE stability conditions where used to train unrolled recurrent residual networks. Similarly, when using our method on ODE stability can be enforced by projecting the state weight matrices, A, into the Hurwitz stable space: i.e. A ≺ 0. At test time, overall stability will also depend on the solver . Therefore, a high order variable step method (e.g. DoPr5) should be used at test time in order to minimize the approximation error. Dynamics and machine learning. A physics prior on a neural network was used by in the form of a consistency loss with data from a simulation. In , a differentiable physics framework was introduced for point mass planar models with contact dynamics. looked at Partial Differential Equations (PDEs) to analyze neural networks, while ) used Gaussian Processes (GP) to model PDEs. The solution of a linear ODE was used in in conjunction with a structured multi-output GP to model patients outcome of continuous treatment observed at random times. predicted the divergence rate of a chaotic system with RNNs. Test time and cross-validation At test time, since the future outputs are unknown, an explicit integrator is needed. For cross-validation, the loss needs instead to be evaluated on a different dataset. In order to do so, one needs to solve the ODE forward in time. However, since the output data is available during cross-validation, a corresponding polynomial representation of the form can be found and the relaxed loss can be evaluated efficiently. Nonsmooth dynamics. We have assumed that the ODE-Net dynamics has a regularity r > p in order to take advantage of the exponential convergence of spectral methods, i.e., that their approximation error reduces as O(h p), where is h is the size of the window used to discretize the interval. However, this might not be true in general. In these cases, the optimal choice would be to use a hp-spectral approach , where h is reduced locally only near the discontinuities. This is very closely related to adaptive time-stepping for ODE solvers. Topological properties, convergence, and better generalization. There are few theoretical open questions stemming from this work. We argue that one reason for the performance improvement shown by our algorithms is the fact that the set of functions generated by a fixed neural network topology does not posses favorable topological properties for optimization, as discussed in . Therefore, the constraint relaxation proposed in this work may improve the properties of the optimization space. This is similar to interior point methods and can help with accelerating the convergence but also with preventing local minima. One further explanation is the fact that the proposed method does not suffer from vanishing nor exploding gradients, as shown in Appendinx C. Moreover, our approach very closely resembles the MAC scheme, for which theoretical convergence are available . Multiple ODEs: Synchronous vs Asynchronous. The proposed method can be used for an arbitrary cascade of dynamical systems as they can be expressed as a single ODE. When only the final state of one ODE (or its trajectory) is fed into the next block, e.g. as in , the method could be extended by means of 2M smaller optimizations, where M is the number of ODEs. Hidden states. Latent states do not appear in the loss, so training and particularly initializing the polynomial coefficients is more difficult. A hybrid approach is to warm-start the optimizer using few iterations of backpropagation. We plan to investigate a full spectral approach in the future. The model is formulated in a concentrated parameter form . We follow the notation of . Recall the system definition: where η, v ∈ R 3 are the states, namely, the x, and y coordinates in a fixed (world) frame, the vehicle orientation with respect this this frame, φ, and the body-frame velocities, v x, v y, and angular rate, ω. The input is a set of torques in the body-frame, u = (F x, 0, τ xy). The Kinematic matrix is with backpropagation. First three columns: comparison of true trajectories (red) with the prediction from the surrogate (black) at different iterations of the optimization. Last column: loss function at each iteration. δ-SNODE has faster convergence than Euler. The multi-agent simulation consists of N a kinematic vehicles: where η ∈ R 3Na are the states for each vehicle, namely, the x i, y i positions and the orientation φ i of vehicle i in the world frame, while v i ∈ R 2Na are the controls signals, in the form of linear and angular velocities, ν i, ω i. The kinematics matrix is J(η i) = cos(φ i) 0 sin(φ i) 0 0 1.

This paper proposes the use of spectral element methods for fast and accurate training of Neural Ordinary Differential Equations for system identification.

represent the tldr for finding the corresponding scientific article

For complex constraints in which it is not easy to estimate the gradient, we use the discounted penalty as a guiding signal. We prove that under certain assumptions it converges to a feasible solution.