Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Neural networks powered with external memory simulate computer behaviors. These models, which use the memory to store data for a neural controller, can learn algorithms and other complex tasks. In this paper, we introduce a new memory to store weights for the controller, analogous to the stored-program memory in modern computer architectures. The proposed model, dubbed Neural Stored-program Memory, augments current memory-augmented neural networks, creating differentiable machines that can switch programs through time, adapt to variable contexts and thus fully resemble the Universal Turing Machine. A wide range of experiments demonstrate that the ing machines not only excel in classical algorithmic problems, but also have potential for compositional, continual, few-shot learning and question-answering tasks. Recurrent Neural Networks (RNNs) are Turing-complete . However, in practice RNNs struggle to learn simple procedures as they lack explicit memory . These findings have sparked a new research direction called Memory Augmented Neural Networks (MANNs) that emulate modern computer behavior by detaching memorization from computation via memory and controller network, respectively. MANNs have demonstrated significant improvements over memory-less RNNs in various sequential learning tasks a; ). Nonetheless, MANNs have barely simulated general-purpose computers. Current MANNs miss a key concept in computer design: stored-program memory. The concept has emerged from the idea of Universal Turing Machine (UTM) and further developed in Harvard Architecture , Von Neumann Architecture (von . In UTM, both data and programs that manipulate the data are stored in memory. A control unit then reads the programs from the memory and executes them with the data. This mechanism allows flexibility to perform universal computations. Unfortunately, current MANNs such as Neural Turing Machine (NTM) , Differentiable Neural Computer (DNC) and Least Recently Used Access (LRUA) only support memory for data and embed a single program into the controller network, which goes against the stored-program memory principle. Our goal is to advance a step further towards UTM by coupling a MANN with an external program memory. The program memory co-exists with the data memory in the MANN, providing more flexibility, reuseability and modularity in learning complicated tasks. The program memory stores the weights of the MANN's controller network, which are retrieved quickly via a key-value attention mechanism across timesteps yet updated slowly via backpropagation. By introducing a meta network to moderate the operations of the program memory, our model, henceforth referred to as Neural Stored-program Memory (NSM), can learn to switch the programs/weights in the controller network appropriately, adapting to different functionalities aligning with different parts of a sequential task, or different tasks in continual and few-shot learning. To validate our proposal, the NTM armed with NSM, namely Neural Universal Turing Machine (NUTM), is tested on a variety of synthetic tasks including algorithmic tasks from , composition of algorithmic tasks and continual procedure learning. For these algorithmic problems, we demonstrate clear improvements of NUTM over NTM. Further, we investigate NUTM in few-shot learning by using LRUA as the MANN and achieve notably better . Finally, we expand NUTM application to linguistic problems by equipping NUTM with DNC core and achieve competitive performances against stateof-the-arts in the bAbI task. Taken together, our study advances neural network simulation of Turing Machines to neural architecture for Universal Turing Machines. This develops a new class of MANNs that can store and query both the weights and data of their own controllers, thereby following the stored-program principle. A set of five diverse experiments demonstrate the computational universality of the approach. In this section, we briefly review MANN and its relations to Turing Machines. A MANN consists of a controller network and an external memory M ∈ R N ×M, which is a collection of N M -dimensional vectors. The controller network is responsible for accessing the memory, updating its state and optionally producing output at each timestep. The first two functions are executed by an interface network and a state network 1, respectively. Usually, the interface network is a Feedforward neural network whose input is c t -the output of the state network implemented as RNNs. Let W c denote the weight of the interface network, then the state update and memory control are as follows, where x t and r t−1 are data from current input and the previous memory read, respectively. The interface vector ξ t then is used to read from and write to the memory M. We use a generic notation memory (ξ t, M) to represent these memory operations that either update or retrieve read value r t from the memory. To support multiple memory accesses per step, the interface network may produce multiple interfaces, also known as control heads. Readers are referred to App. F and Graves et al. (2014; ; for details of memory read/write examples. A deterministic one-tape Turing Machine can be defined by 4-tuple (Q, Γ, δ, q 0), in which Q is finite set of states, q 0 ∈ Q is an initial state, Γ is finite set of symbol stored in the tape (the data) and δ is the transition function (the program), δ: Q × Γ → Γ × {−1, 1} × Q. At each step, the machine performs the transition function, which takes the current state and the read value from the tape as inputs and outputs actions including writing new values, moving tape head to new location (left/right) and jumping to another state. Roughly mapping to current MANNs, Q, Γ and δ map to the set of the controller states, the read values and the controller network, respectively. Further, the function δ can be factorized into two sub functions: Q × Γ → Γ × {−1, 1} and Q × Γ → Q, which correspond to the interface and state networks, respectively. By encoding a Turing Machine into the tape, one can build a UTM that simulates the encoded machine . The transition function δ u of the UTM queries the encoded Turing Machine that solves the considering task. Amongst 4 tuples, δ is the most important and hence uses most of the encoding bits. In other words, if we assume that the space of Q, Γ and q 0 are shared amongst Turing Machines, we can simulate any Turing Machine by encoding only its transition function δ. Translating to neural language, if we can store the controller network into a queriable memory and make use of it, we can build a Neural Universal Turing Machine. Using NSM is a simple way to achieve this goal, which we introduce in the subsequent section. A Neural Stored-program Memory (NSM) is a key-value memory M p ∈ R P ×(K+S), whose values are the basis weights of another neural network−the programs. P, K, and S are the number of programs, the key space dimension and the program size, respectively. This concept is a hybrid between the traditional slow-weight and fast-weight . Like slow-weight, the keys and values in NSM are updated gradually by backpropagation. However, the values are dynamically interpolated to produce the working weight on-the-fly during the processing of a sequence, which resembles fast-weight computation. Let us denote M p (i).k and M p (i).v as the key and the program of the i-th memory slot. At timestep t, given a query key k p t, the working program is retrieved as follows, where D (·) is cosine similarity and β p t is the scalar program strength parameter. The vector working program p t is then reshaped to its matrix form and ready to be used as the weight of other neural networks. The key-value design is essential for convenient memory access as the size of the program stored in M p can be millions of dimensions and thus, direct content-based addressing as in Graves et al. (2014; ; is infeasible. More importantly, we can inject external control on the behavior of the memory by imposing constraints on the key space. For example, program collapse will happen when the keys stored in the memory stay close to each other. When this happens, p t is a balanced mixture of all programs regardless of the query key and thus having multiple programs is useless. We can avoid this phenomenon by minimizing a regularization loss defined as the following, It turns out that the combination of MANN and NSM approximates a Universal Turing Machine (Sec. 2). At each timestep, the controller in MANN reads its state and memory to generate control signal to the memory via the interface network W c, then updates its state using the state network RN N. Since the parameters of RN N and W c represent the encoding of δ, we should store both into NSM to completely encode an MANN. For simplicity, in this paper, we only use NSM to store W c, which is equivalent to the Universal Turing Machine that can simulate any one-state Turing Machine. In traditional MANN, W c is constant across timesteps and only updated slowly during training, typically through backpropagation. In our design, we compute W c t from NSM for every timestep and thus, we need a program interface network−the meta network P I −that generates an interface vector for the program memory: ξ That is, only the meta-network learns the mapping from context c t to program. When it falls into some local-minima (generating suboptimal w p t), the metanetwork struggles to escape. In our proposal, together with the meta-network, the memory keys are learnable. When the memory keys are slowly updated, the meta-network will shift its query key generation to match the new memory keys and possibly escape from the local-minima. For the case of multi-head NTM, we implement one NSM per control head and name this model Neural Universal Turing Machine (NUTM). One NSM per head is to ensure programs for one head do not interfere with other heads and thus, encourage functionality separation amongst heads. Each control head will read from (for read head) or write to (for write head) the data memory M via memory (ξ t, M) as described in. It should be noted that using multiple heads is unlike using multiple controllers per head. The former increases the number of accesses to the data memory at each timestep and employs a fixed controller to compute multiple heads, which may improve capacity yet does not enable adaptability. On the contrary, the latter varies the property of each memory access across timesteps by switching the controllers and thus potential for adaptation. Other MANNs such as DNC and LRUA can be armed with NSM in this manner. We also employ the regularization loss l p to prevent the programs from collapsing, ing in a final loss as follows, where Loss pred is the prediction loss and η t is annealing factor, reducing as the training step increases. The details of NUTM operations are presented in Algorithm 1. Learning to access memory is a multi-dimensional regression problem. Given the input c t, which is derived from the state h t of the controller, the aim is to generate a correct interface vector ξ t via optimizing the interface network. Instead of searching for one transformation that maps the whole space of c t to the optimal space of ξ t, NSM first partitions the space of c t into subspaces, then finds multiple transformations, each of which covers subspace of Require: a sequence x = {x t} T t=1, a data memory M and R program memories {M p,n} R n=1 corresponding to R control heads 1: Initilize h 0, r 0 2: for t = 1, T do 3: RN N can be replaced by GRU/LSTM 4: for n = 1, R do Compute the program interface ξ p t,n ← P I,n (c t) 6: Compute the data interface ξ t,n ← c t W c t,n 8: Read r t,n from memory M (if read head) or update memory M (if write head) using memory n (ξ t,n, M) c t. The program interface network P I is a meta learner that routes c t to the appropriate transformation, which then maps c t to the ξ t space. This is analogous to multilevel regression in statistics . Practical studies have shown that multilevel regression is better than ordinary regression if the input is clustered . RNNs have the capacity to learn to perform finite state computations (; Tiňo et al., 1998). The states of a RNN must be grouped into partitions representing the states of the generating automaton. As Turing Machines are finite state automata augmented with an external memory tape, we expect MANN, if learnt well, will organize its state space clustered in a way to reflect the states of the emulated Turing Machine. That is, h t as well as c t should be clustered. We realize that NSM helps NTM learn better clusterization over this space (see App. A), thereby improving NTM's performances. In this section, we investigate the performance of NUTM on algorithmic tasks introduced in doubles the length of training sequences in the Copy task. In these tasks, the model will be fed a sequence of input items and is required to infer a sequence of output items. Each item is represented by a binary vector. In the experiment, we compare two models: NTM 2 and NUTM with two programs. Although the tasks are atomic, we argue that there should be at least two memory manipulation schemes across timesteps, one for encoding the inputs to the memory and another for decoding the output from the memory. The two models are trained with cross-entropy objective function under the same setting as in. For fair comparison, the controller hidden dimension of NUTM is set smaller to make the total number of parameters of NUTM equivalent to that of NTM. The number of memory heads for both models are always equal and set to the same value as in the original paper (details in App. C). We run each experiments five times and report the mean with error bars of training losses for NTM tasks in Fig. 2 (a). Except for the Copy task, which is too simple, other tasks observe convergence speed improvement of NUTM over that of NTM, thereby validating the benefit of using two programs across timesteps even for the single task setting. NUTM requires fewer training samples to converge and it generalizes better to unseen sequences that are longer than training sequences. Table 1 reports the test of the best models chosen after five runs and confirms the outperformance of NUTM over NTM for generalization. To illustrate the program usage, we plot NUTM's program distributions across timesteps for Repeat Copy and Priority Sort in Fig. 3 (a) and (b), respectively. Examining the read head for Repeat Copy, we observe two program usage patterns corresponding to the encoding and decoding phases. As there is no reading in encoding, NUTM assigns the "no-read" strategy mainly to the "orange program". In decoding, the sequential reading is mostly done by the "blue program" with some contributions from the "orange program" when resetting reading head. Similar behaviors can be found in the write head for Priority Sort. While the encoding "fitting writing" (see for explanation on the strategy) is often executed by the "blue program", the decoding writing is completely taken by the "orange" program (more visualizations in App. B). In this section, we conduct an ablation study on Associative Recall (AR) to validate the benefit of proposed components that constitute NSM. We run the task with three additional baselines: NUTM using direct attention (DA), NUTM using key-value without regularization (KV), NUTM using fixed, uniform program distribution (UP) and a vanilla NTM with 2 memory heads (h = 2). The meta-network P I in DA generates the attention weight w p t directly. The KV employs key-value attention yet excludes the regularization loss presented in Eq.. The training curves over 5 runs are plotted in Fig. 2 (b). The demonstrate that DA exhibits fast yet shallow convergence. It tends to fall into local minima, which finally fails to reach zero loss. Key-value attention helps NUTM converge completely with fewer iterations. The performance is further improved with the proposed regularization loss. UP underperforms NUTM as it lacks dynamic programs. The NTM with 2 heads shows slightly better convergence compared to the NTM, yet obviously underperforms NUTM (p = 2) with 1 head and fewer parameters. This validates our argument on the difference between using multiple heads and multiple programs (Sec. 3.2). In neuroscience, sequencing tasks test the ability to remember a series of tasks and switch tasks alternatively . A dysfunctional brain may have difficulty in changing from one task to the next and get stuck in its preferred task (perseveration phenomenon). To analyze this problem in NTM, we propose a new set of experiments in which a task is generated by sequencing a list of subtasks. The set of subtasks is chosen from the NTM single tasks (excluding Dynamic N-grams for format discrepancy) and the order of subtasks in the sequence is dictated by an indicator vector put at the beginning of the sequence. Amongst possible combinations of subtasks, we choose {Copy, Repeat Copy}(C+RC), {Copy, Associative Recall} (C+AR), {Copy, Priority Sort} (C+PS) and all (C+RC+AC+PS) 3. The learner observes the order indicator followed by a sequence of subtasks' input items and is requested to consecutively produce the output items of each subtasks. As shown in Fig. 4, some tasks such as Copy and Associative Recall, which are easy to solve if trained separately, become unsolvable by NTM when sequenced together. One reason is NTM fails to change the memory access behavior (perseveration). For examples, NTM keeps following repeat copy reading strategy for all timesteps in C+RC task (Fig. 3 (d) ). Meanwhile, NUTM can learn to change program distribution when a new subtask appears in the sequence and thus ensure different accessing strategy per subtask (Fig. 3 (c) ). In continual learning, catastrophic forgetting happens when a neural network quickly forgets previously acquired skills upon learning new skills . In this section, we prove the versatility of NSM by showing that a naive application of NSM without much modification can help NTM to mitigate catastrophic forgetting. We design an experiment similar to the Split MNIST to investigate whether NSM can improve NTM's performance. In our experiment, we let the models see the training data from the while freezing others, we force "hard" attention over the programs by replacing the softmax function in Eq. 5 with the Gumbel-softmax . Also, to ignore catastrophic forgetting in the state network, we use Feedforward controllers in the two baselines. After finishing one task, we evaluate the bit accuracy −measured by 1−(bit error per sequence/total bits per sequence) over 4 tasks. As shown in in Fig. 5, NUTM outperforms NTM by a moderate margin (10-40% per task). Although NUTM also experiences catastrophic forgetting, it somehow preserves some memories of previous tasks. Especially, NUTM keeps performing perfectly on Copy even after it learns Repeat Copy. For other dissimilar task transitions, the performance drops significantly, which requires more effort to bring NSM to continual learning. Few-shot learning or meta learning tests the ability to rapidly adapt within a task while gradually capturing the way the task structure varies . By storing sampleclass bindings, MANNs are capable of classifying new data after seeing only few samples . As NSM gives flexible memory controls, it makes MANN more adaptive to changes and thus perform better in this setting. To verify that, we apply NSM to the LRUA memory and follow the experiments introduced in , using the Omniglot dataset to measure few-shot classification accuracy. The dataset includes images of 1623 characters, with 20 examples of each character. During training, a sequence (episode) of images are randomly selected from C classes of characters in the training set (1200 characters), where C = 5, 10 corresponding to sequence length of 50, 75, respectively. Each class is assigned a random label which shuffles between episodes and is revealed to the models after each prediction. After 100,000 episodes of training, the models are tested with unseen images from the testing set (423 characters). The two baselines are MANN and NUTM (both use LRUA core). For NUTM, we only tune p and pick the best values: p = 2 and p = 3 for 5 classes and 10 classes, respectively. Table 2: Test-set classification accuracy (%) on the Omniglot dataset after 100,000 episodes of training. * denotes available from . Model Error DNC 16.7 ± 7.6 SDNC 6.4 ± 2.5 ADNC 6.3 ± 2.7 DNC-MD 9.5 ± 1.6 NUTM (DNC core, p=1) 9.7 ± 3.5 NUTM (DNC core, p=2) 7.5 ± 1.6 NUTM (DNC core, p=4) 5.6 ± 1.9 persistent memory mode, which demands fast forgetting old experiences in previous episodes, NUTM 4. Readers are referred to App. D for more details on learning curves and more of the models. Reading comprehension typically involves an iterative process of multiple actions such as reading the story, reading the question, outputting the answers and other implicit reasoning steps. We apply NUTM to the question answering domain by replacing the NTM core with DNC. Compared to NTM's sequential addressing, dynamic memory addressing in DNC is more powerful and thus suitable for NSM integration to solve non-algorithmic problems such as question answering. Following previous works of DNC, we use bAbI dataset to measure the performance of the NUTM with DNC core (three variants p = 1, p = 2 and p = 4). In the dataset, each story is followed by a series of questions and the network reads all word by word, then predicts the answers. Although synthetically generated, bAbI is a good benchmark that tests 20 aspects of natural language reasoning including complex skills such as induction and counting, We found that increasing number of programs helps NUTM improve performance. In particular, NUTM with 4 programs, after 50 epochs jointly trained on all 20 question types, can achieve a mean test error rate of 3.3% and manages to solve 19/20 tasks (a task is considered solved if its error <5%). The mean and s.d. across 10 runs are also compared with other reported by recent works (see Table 3). Excluding baselines under different setups, our is the best reported mean on bAbI that we are aware of. More details are described in App. E. Previous investigations into MANNs mostly revolve around memory access mechanisms. The works in Graves et al. (2014; introduce content-based, location-based and dynamic memory reading/writing. scales to bigger memory by sparse access; optimizes memory operations with uniform writing; and MANNs with extra memory have been proposed (b). However, these works keep using memory for storing data rather than the weights of the network and thus parallel to our approach. Other DNC modifications are also orthogonal to our work. Another line of related work involves modularization of neural networks, which is designed for visual question answering. In module networks (b; a), the modules are manually aligned with predefined concepts and the order of execution is decided by the question. Although the module in these works resembles the program in NSM, our model is more generic and flexible with soft-attention over programs and thus fully differentiable. Further, the motivation of NSM does not limit to a specific application. Rather, NSM aims to help MANN reach general-purpose computability. If we view NSM network as a dynamic weight generator, the program in NSM can be linked to fast weight (von der ; ; b). These papers share the idea of using different weights across timesteps to enable dynamic adaptation. Using outer-product is a common way to implement fast-weight (a;). These fast weights are directly generated and thus different from our programs, which are interpolated from a set of slow weights. Tensor/Multiplicative RNN and Hypernetwork are also relevant related works. These methods attempt to make the working weight of RNNs dependent on the input to enable quick adaption through time. Nevertheless, they do not support modularity. In particular, Hypernetwork generates scaling factors for the single weight of the main RNN. It does not aim to use multiple slow-weights (programs) and thus, different from our approach. Tensor RNN is closer to our idea when the authors propose to store M slow-weights, where M is the number of input dimension, which is acknowledged impractical. Unlike our approach, they do not use a meta-network to generate convex combinations amongst weights. Instead, they propose Multiplicative RNN that factorizes the working weight to product of three matrices, which looses modularity. On the contrary, we explicitly model the working weight as an interpolation of multiple programs and use a meta-network to generate the coefficients. This design facilitates modularity because each program is trained towards some functionality and can be switched or combined with each other to perform the current task. Last but not least, while the related works focus on improving RNN with fast-weight, we aim to reach a neural simulation of Universal Turing Machine, in which fast-weight is a way to implement stored-program principle. This paper introduces the Neural Stored-program Memory (NSM), a new type of external memory for neural networks. The memory, which takes inspirations from the stored-program memory in computer architecture, gives memory-augmented neural networks (MANNs) flexibility to change their control programs through time while maintaining differentiability. The mechanism simulates modern computer behavior, potential making MANNs truly neural computers. Our experiments demonstrated that when coupled with our model, the Neural Turing Machine learns algorithms better and adapts faster to new tasks at both sequence and sample levels. When used in few-shot learning, our method helps MANN as well. We also applied the NSM to the Differentiable Neural Computer and observed a significant improvement, reaching the state-of-the-arts in the bAbI task. Although this paper limits to MANN integration, other neural networks can also reap benefits from our proposed model, which will be explored in future works. Table 9: Task settings (continual procedure learning tasks). We use similar hyper-parameters as in , which are reported in Tab Table 11: Test-set classification accuracy (%) on the Omniglot dataset after 100,000 episodes of training. * denotes available from (some are estimated from plotted figures). We train the models using RMSprop optimizer with fixed learning rate of 10 −4 and momentum of 0.9. The batch size is 32 and we adopt layer normalization to DNC's layers. practice, we also remove temporal linkage for faster training. The details of hyper-parameters are listed in Table 12. Full NUTM (p = 4) are reported in 3.3 5.6 ± 1.9 Failed (Err. >5%) 1 3 ± 1.2 Table 13: NUTM (p = 4) bAbI best and mean errors (%). 9 When p = 1, the model converges to layer-normed DNC For all tasks, η t is fixed to 0.1, reducing with decay rate of 0.9. Ablation study's learning losses with mean and error bar are plotted in Fig. 23.

represent the tldr for finding the corresponding scientific article

A neural simulation of Universal Turing Machine

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question: Represent the article for finding a single-sentence TLDR

Question: It is common practice to decay the learning rate. Here we show one can usually obtain the same learning curve on both training and test sets by instead increasing the batch size during training. This procedure is successful for stochastic gradient descent (SGD), SGD with momentum, Nesterov momentum, and Adam. It reaches equivalent test accuracies after the same number of training epochs, but with fewer parameter updates, leading to greater parallelism and shorter training times. We can further reduce the number of parameter updates by increasing the learning rate $\epsilon$ and scaling the batch size $B \propto \epsilon$. Finally, one can increase the momentum coefficient $m$ and scale $B \propto 1/(1-m)$, although this tends to slightly reduce the test accuracy. Crucially, our techniques allow us to repurpose existing training schedules for large batch training with no hyper-parameter tuning. We train ResNet-50 on ImageNet to 76.1% validation accuracy in under 30 minutes. Stochastic gradient descent (SGD) remains the dominant optimization algorithm of deep learning. However while SGD finds minima that generalize well BID30 BID26, each parameter update only takes a small step towards the objective. Increasing interest has focused on large batch training BID8 BID10 BID27, in an attempt to increase the step size and reduce the number of parameter updates required to train a model. Large batches can be parallelized across many machines, reducing training time. Unfortunately, when we increase the batch size the test set accuracy often falls BID12 BID8.To understand this surprising observation, BID23 argued one should interpret SGD as integrating a stochastic differential equation. They showed that the scale of random fluctuations in the SGD dynamics, g = (N B − 1), where is the learning rate, N training set size and B batch size. Furthermore, they found that there is an optimum fluctuation scale g which maximizes the test set accuracy (at constant learning rate), and this introduces an optimal batch size proportional to the learning rate when B N. BID8 already observed this scaling rule empirically and exploited it to train ResNet-50 to 76.3% ImageNet validation accuracy in one hour. Here we show,• When one decays the learning rate, one simultaneously decays the scale of random fluctuations g in the SGD dynamics. Decaying the learning rate is simulated annealing. We propose an alternative procedure; instead of decaying the learning rate, we increase the batch size during training. This strategy achieves near-identical model performance on the test set with the same number of training epochs but significantly fewer parameter updates. Our proposal does not require any fine-tuning as we follow pre-existing training schedules; when the learning rate drops by a factor of α, we instead increase the batch size by α.• As shown previously, we can further reduce the number of parameter updates by increasing the learning rate and scaling B ∝. One can also increase the momentum coefficient and scale B ∝ 1/(1 − m), although this slightly reduces the test accuracy. We train InceptionResNet-V2 on ImageNet in under 2500 parameter updates, using batches of 65536 images, and reach a validation set accuracy of 77%. We also replicate the setup of BID8 on TPU and train ResNet-50 on ImageNet to 76.1% accuracy in under 30 minutes. We note that a number of recent works have discussed increasing the batch size during training BID7 BID3 BID1 BID2 BID5, but to our knowledge no paper has shown empirically that increasing the batch size and decaying the learning rate are quantitatively equivalent. A key contribution of our work is to demonstrate that decaying learning rate schedules can be directly converted into increasing batch size schedules, and vice versa; providing a straightforward pathway towards large batch training. In section 2 we discuss the convergence criteria for SGD in strongly convex minima, in section 3 we interpret decaying learning rates as simulated annealing, and in section 4 we discuss the difficulties of training with large momentum coefficients. Finally in section 5 we present conclusive experimental evidence that the empirical benefits of decaying learning rates in deep learning can be obtained by instead increasing the batch size during training. We exploit this observation and other tricks to achieve efficient large batch training on CIFAR-10 and ImageNet. SGD is a computationally-efficient alternative to full-batch training, but it introduces noise into the gradient, which can obstruct optimization. It is often stated that to reach the minimum of a strongly convex function we should decay the learning rate, such that BID22: DISPLAYFORM0 DISPLAYFORM1 i denotes the learning rate at the i th gradient update. Intuitively, equation 1 ensures we can reach the minimum, no matter how far away our parameters are initialized, while equation 2 ensures that the learning rate decays sufficiently quickly that we converge to the minimum, rather than bouncing around it due to gradient noise BID25. However, although these equations appear to imply that the learning rate must decay during training, equation 2 holds only if the batch size is constant.1 To consider how to proceed when the batch size can vary, we follow recent work by BID23 and interpret SGD as integrating the stochastic differential equation below, DISPLAYFORM2 C represents the cost function (summed over all training examples), and ω represents the parameters, which evolve in continuous "time" t towards their final values. Meanwhile η(t) represents Gaussian random noise, which models the consequences of estimating the gradient on a mini-batch. They showed that the mean η(t) = 0 and variance η(t)η(t) = gF (ω)δ(t − t), where F (ω) describes the covariances in gradient fluctuations between different parameters. They also proved that the "noise scale" g = (DISPLAYFORM3, where is the learning rate, N the training set size and B the batch size. This noise scale controls the magnitude of the random fluctuations in the training dynamics. Usually B N, and so we may approximate g ≈ N/B. When we decay the learning rate, the noise scale falls, enabling us to converge to the minimum of the cost function (this is the origin of equation 2 above). However we can achieve the same reduction in noise scale at constant learning rate by increasing the batch size. The main contribution of this work is to show that it is possible to make efficient use of vast training batches, if one increases the batch size during training at constant learning rate until B ∼ N/10. After this point, we revert to the use of decaying learning rates. To the surprise of many researchers, it is now increasingly accepted that small batch training often generalizes better to the test set than large batch training. This "generalization gap" was explored extensively by BID12. BID23 observed an optimal batch size B opt which maximized the test set accuracy at constant learning rate. They argued that this optimal batch size arises when the noise scale g ≈ N/B is also optimal, and supported this claim by demonstrating empirically that B opt ∝ N. Earlier, BID8 exploited a linear scaling rule between batch size and learning rate to train ResNet-50 on ImageNet in one hour with batches of 8192 images. These indicate that gradient noise can be beneficial, especially in non-convex optimization. It has been proposed that noise helps SGD escape "sharp minima" which generalize poorly BID9 BID4 BID12 BID23. Given these , it is unclear to the present authors whether equations 1 and 2 are relevant in deep learning. Supporting this view, we note that most researchers employ early stopping BID20, whereby we intentionally prevent the network from reaching a minimum. Nonetheless, decaying learning rates are empirically successful. To understand this, we note that introducing random fluctuations whose scale falls during training is also a well established technique in non-convex optimization; simulated annealing. The initial noisy optimization phase allows us to explore a larger fraction of the parameter space without becoming trapped in local minima. Once we have located a promising region of parameter space, we reduce the noise to fine-tune the parameters. Finally, we note that this interpretation may explain why conventional learning rate decay schedules like square roots or exponential decay have become less popular in deep learning in recent years. Increasingly, researchers favor sharper decay schedules like cosine decay BID16 or step-function drops BID29. To interpret this shift, we note that it is well known in the physical sciences that slowly annealing the temperature (noise scale) helps the system to converge to the global minimum, which may be sharp. Meanwhile annealing the temperature in a series of discrete steps can trap the system in a "robust" minimum whose cost may be higher but whose curvature is lower. We suspect a similar intuition may hold in deep learning. Many researchers no longer use vanilla SGD, instead preferring SGD with momentum. extended their analysis of SGD to include momentum, and found that the "noise scale", DISPLAYFORM0 This reduces to the noise scale of vanilla SGD when the momentum coefficient m → 0. Intuitively, ef f = /(1 − m) is the effective learning rate. They proposed to reduce the number of parameter updates required to train a model by increasing the learning rate and momentum coefficient, while simultaneously scaling B ∝ /(1 − m). We find that increasing the learning rate and scaling B ∝ performs well. However increasing the momentum coefficient while scaling B ∝ 1/(1−m) slightly reduces the test accuracy. To analyze this observation, consider the momentum update equations, DISPLAYFORM1 DISPLAYFORM2 A is the "accumulation", while dĈ dω is the mean gradient per training example, estimated on a batch of size B. In Appendix A we analyze the growth of the accumulation at the start of training. This variable tracks the exponentially decaying average of gradient estimates, but initially it is initialized to zero. We find that the accumulation grows in exponentially towards its steady state value over a "timescale" of approximately B/(N (1 − m)) training epochs. During this time, the magnitude of the parameter updates ∆ω is suppressed, reducing the rate of convergence. Consequently when training at high momentum one must introduce additional epochs to allow the dynamics to catch up. Furthermore, when we increase the momentum coefficient we increase the timescale required for the accumulation to forget old gradients (this timescale is also ∼ B/(N (1 − m))). Once this timescale becomes several epochs long, the accumulation cannot adapt to changes in the loss landscape, impeding training. This is likely to be particularly problematic at points where the noise scale decays. BID13 proposed initialization bias correction, whereby the learning rate is increased at early times to compensate the suppressed initial value of the accumulation. However when the batch size is large, we found that this often causes instabilities during the early stages of training. We note that BID8 recommended a reduced learning rate for the first few epochs. In section 5.1, we demonstrate that decreasing the learning rate and increasing the batch size during training are equivalent. In section 5.2, we show we can further reduce the number of parameter updates by increasing the effective learning rate and scaling the batch size. In section 5.3 we apply our insights to train Inception-ResNet-V2 on ImageNet, using vast batches of up to 65536 images. Finally in section 5.4, we train ResNet-50 to 76.1% ImageNet validation accuracy within 30 minutes. Our first experiments are performed on CIFAR-10, using a "16-4" wide ResNet architecture, following the implementation of BID29. We use ghost batch norm BID10, with a ghost batch size of 128. This ensures the mean gradient is independent of batch size, as required by the analysis of BID23. To demonstrate the equivalence between decreasing the learning rate and increasing the batch size, we consider three different training schedules, as shown in figure 1. "Decaying learning rate" follows the original implementation; the batch size is constant, while the learning rate repeatedly decays by a factor of 5 at a sequence of "steps". "Hybrid" holds the learning rate constant at the first step, instead increasing the batch size by a factor of 5. However after this first step, the batch size is constant and the learning rate decays by a factor of 5 at each subsequent step. This schedule mimics how one might proceed if hardware imposes a limit on the maximum achievable batch size. In "Increasing batch size", we hold the learning rate constant throughout training, and increase the batch size by a factor of 5 at every step. If the learning rate itself must decay during training, then these schedules should show different learning curves (as a function of the number of training epochs) and reach different final test set accuracies. Meanwhile if it is the noise scale which should decay, all three schedules should be indistinguishable. We plot the evolution of the training set cross entropy in figure 2a, where we train using SGD with momentum and a momentum parameter of 0.9. The three training curves are almost identical, despite showing marked drops as we pass through the first two steps (where the noise scale is reduced). These suggest that it is the noise scale which is relevant, not the learning rate. To emphasize the potential benefits of increasing the batch size, we replot the training cross-entropy in figure 2b, but as a function of the number of parameter updates rather than the number of epochs. While all three schedules match up to the first "step", after this point increasing the batch size dramatically reduces the number of parameter updates required to train the model. Finally, to confirm that our alternative learning schedules generalize equally well to the test set, in figure 3a we exhibit the test set accuracy, as a function of the number of epochs (so each curve can be directly compared). Once again, the three schedules are almost identical. We conclude that we can achieve all of the benefits of decaying the learning rate in these experiments by instead increasing the batch size. We present additional to establish that our proposal holds for a range of optimizers, all using the schedules presented in FIG0. In figure 3b, we present the test set accuracy, when training with Nesterov momentum BID19 and momentum parameter 0.9, observing three nearidentical curves. In FIG2, we repeat the same experiment with vanilla SGD, again obtaining three highly similar curves (In this case, there is no clear benefit of decaying the learning rate after the first step). Finally in FIG2 we repeat the experiment with Adam BID13. We Test accuracy as a function of the number of parameter updates. " Increasing batch size" replaces learning rate decay by batch size increases. "Increased initial learning rate" additionally increases the initial learning rate from 0.1 to 0.5. Finally "Increased momentum coefficient" also increases the momentum coefficient from 0.9 to 0.98.use the default parameter settings of TensorFlow, such that the initial base learning rate here was 10 −3, β 1 = 0.9 and β 2 = 0.999. Thus the learning rate schedule is obtained by dividing figure 1a by 10 −2. Remarkably, even here the three curves closely track each other. We now focus on our secondary objective; minimizing the number of parameter updates required to train a model. As shown above, the first step is to replace decaying learning rates by increasing batch sizes. We show here that we can also increase the effective learning rate ef f = /(1 − m) at the start of training, while scaling the initial batch size B ∝ ef f. All experiments are conducted using SGD with momentum. There are 50000 images in the CIFAR-10 training set, and since the scaling rules only hold when B N, we decided to set a maximum batch size B max = 5120.We consider four training schedules, all of which decay the noise scale by a factor of five in a series of three steps. " Original training schedule" follows the implementation of BID29, using an initial learning rate of 0.1 which decays by a factor of 5 at each step, a momentum coefficient of 0.9, and a batch size of 128. "Increasing batch size" also uses a learning rate of 0.1, initial batch size of 128 and momentum coefficient of 0.9, but the batch size increases by a factor of 5 at each step. These schedules are identical to "Decaying learning rate" and "Increasing batch size" in section 5.1 above. "Increased initial learning rate" also uses increasing batch sizes during training, but additionally uses an initial learning rate of 0.5 and an initial batch size of 640. Finally "Increased momentum coefficient" combines increasing batch sizes during training and the increased initial learning rate of 0.5, with an increased momentum coefficient of 0.98, and an initial batch size of 3200. Note that we only increase the batch size until it reaches B max, after this point we achieve subsequent decays in noise scale by decreasing the learning rate. We emphasize that, as in the previous section, all four schedules require the same number of training epochs. We plot the evolution of the test set accuracy in FIG3, as a function of the number of parameter updates. Our implementation of the original training schedule requires ∼80000 updates, and reaches a final test accuracy of 94.3% (the original paper reports 95% accuracy, which we have not been able to replicate). " Increasing batch size" requires ∼29000 updates, reaching a final accuracy of 94.4%. "Increased initial learning rate" requires under 6500 updates, reaching a final accuracy of 94.5%. Finally, "Increased momentum coefficient" requires less than 2500 parameter updates, but reaches a lower test accuracy of 93.3%. Across five additional training runs for each schedule, the median accuracies were 94.3%, 94.2%, 94.2% and 93.5% respectively. We discussed a potential explanation for the performance drop when training with large momentum coefficients in section 4. We provide additional in appendix B, varying the initial learning rate between 0.1 and 3.2 while holding the batch size constant. We find that the test accuracy falls for initial learning rates larger than ∼0.4.(a) (b) Figure 6: Inception-ResNet-V2 on ImageNet. Increasing the batch size during training achieves similar to decaying the learning rate, but it reduces the number of parameter updates from just over 14000 to below 6000. We run each experiment twice to illustrate the variance. We now apply our insights to reduce the number of parameter updates required to train ImageNet. BID8 trained a ResNet-50 on ImageNet in one hour, reaching 76.3% validation accuracy. To achieve this, they used batches of 8192, with an initial learning rate of 3.2 and a momentum coefficient of 0.9. They completed 90 training epochs, decaying the learning rate by a factor of ten at the 30th, 60th and 80th epoch. ImageNet contains around 1.28 million images, so this corresponds to ∼14000 parameter updates. They also introduced a warm-up phase at the start of training, in which the learning rate and batch size was gradually increased. We also train for 90 epochs and follow the same schedule, decaying the noise scale by a factor of ten at the 30th, 60th and 80th epoch. However we did not include a warm-up phase. To set a stronger baseline, we replaced ResNet-50 by Inception-ResNet-V2 BID24. Initially we used a ghost batch size of 32. In figure 6, we train with a learning rate of 3.0 and a momentum coefficient of 0.9. The initial batch size was 8192. For "Decaying learning rate", we hold the batch size fixed and decay the learning rate, while in "Increasing batch size" we increase the batch size to 81920 at the first step, but decay the learning rate at the following two steps. We repeat each schedule twice, and find that all four runs exhibit a very similar evolution of the test set accuracy during training. The final accuracies of the two "Decaying learning rate" runs are 78.7% and 77.8%, while the final accuracy of the two "Increasing batch size" runs are 78.1% and 76.8%. Although there is a slight drop, the difference in final test accuracies is similar to the variance between training runs. Increasing the batch size reduces the number of parameter updates during training from just over 14000 to below 6000. Note that the training curves appear unusually noisy because we reduced the number of test set evaluations to reduce the model training time. BID8 already increased the learning rate close to its maximum stable value. To further reduce the number of parameter updates we must increase the momentum coefficient. We introduce a maximum batch size, B max = 2 16 = 65536. This ensures B N, and it also improved the stability of our distributed training. We also increased the ghost batch size to 64, matching the batch size of our GPUs and reducing the training time. We compare three different schedules, all of which have the same base schedule, decaying the noise scale by a factor of ten at the 30th, 60th and 80th epoch. We use an initial learning rate of 3 throughout. " Momentum 0.9" uses an initial batch size of 8192, "Momentum 0.975" uses an initial batch size of 16384, and "Momentum 0.9875" uses an initial batch size of 32768. For all schedules, we decay the noise scale by increasing the batch size until reaching B max, and then decay the learning rate. We plot the test set accuracy in figure 7. "Momentum 0.9" achieves a final accuracy of 78.8% in just under 6000 updates. We performed two runs of "Momentum 0.95", achieving final accuracies of 78.1% and 77.8% in under 3500 updates. Finally "Momentum 0.975" achieves final accuracies of 77.5% and 76.8% in under 2500 updates. Figure 7: Inception-ResNet-V2 on ImageNet. Increasing the momentum parameter reduces the number of parameter updates required, but it also leads to a small drop in final test accuracy. To confirm that increasing the batch size during training can reduce model training times, we replicated the set-up described by BID8 on a half TPU pod, comprising 256 tensorcores BID11. Using tensorFlow, we first train ResNet-50 for 90 epochs to 76.1% validation set accuracy in under 45 minutes, utilising batches of 8192 images. To utilise the full TPU pod, we then increase the batch size after the first 30 epochs to 16384 images, and achieve the same validation accuracy of 76.1% in under 30 minutes. The last 60 epochs and the first 30 epochs both take just under 15 minutes, demonstrating near-perfect scaling efficiency across the pod, such that the number of parameter updates provides a meaningful measure of the training time. To our knowledge, this is the first procedure which has reduced the training time of BID8 without sacrificing final validation accuracy BID28 BID0. By contrast, doubling the initial learning rate and using batches of 16384 images throughout training achieves a lower validation set accuracy of 75.0% in 22 minutes, demonstrating that increasing the batch size during training is crucial to the performance gains above. These show that the ideas presented in this paper will become increasingly important as new hardware for large-batch training becomes available. This paper extends the analysis of SGD in BID23 to include decaying learning rates. BID17 also interpreted SGD as a stochastic differential equation, in order to discuss how SGD could be modified to perform approximate Bayesian posterior sampling. However they state that their analysis holds only in the neighborhood of a minimum, while BID12 showed that the beneficial effects of noise are most pronounced at the start of training. BID15 proposed the use of control theory to set the learning rate and momentum coefficient. BID8 observed a linear scaling rule between batch size and learning rate, B ∝, and used this rule to reduce the time required to train ResNet-50 on ImageNet to one hour. To our knowledge, this scaling rule was fist adopted by BID14. BID2 (section 4.2) demonstrated that SGD converges to strongly convex minima in similar numbers of training epochs if B ∝. BID10 proposed an alternative scaling rule, B ∝ √. BID27 proposed Layer-wise Adaptive Rate Scaling (LARS), which applies different learning rates to different parameters in the network, and used it to train ImageNet in 14 minutes BID28, albeit to a lower final accuracy of 74.9%. K-FAC BID18 is also gaining popularity as an efficient alternative to SGD. BID26 argued that adaptive optimization methods tend to generalize less well than SGD and SGD with momentum (although they did not include K-FAC in their study), while our work reduces the gap in convergence speed. Asynchronous-SGD is another popular strategy, which enables the use of multiple GPUs even when batch sizes are small BID21 BID6. We do not consider asynchronous-SGD in this work, since the scaling rules enabled us to use batch sizes on the order of the training set size. We can often achieve the benefits of decaying the learning rate by instead increasing the batch size during training. We support this claim with experiments on CIFAR-10 and ImageNet, and with a range of optimizers including SGD, Momentum and Adam. Our findings enable the efficient use of vast batch sizes, significantly reducing the number of parameter updates required to train a model. This has the potential to dramatically reduce model training times. We further increase the batch size B by increasing the learning rate and momentum parameter m, while scaling B ∝ /(1 − m). Combining these strategies, we train Inception-ResNet-V2 on ImageNet to 77% validation accuracy in under 2500 parameter updates, using batches of 65536 images. We also exploit increasing batch sizes to train ResNet-50 to 76.1% ImageNet validation set accuracy on TPU in under 30 minutes. Most strikingly, we achieve this without any hyper-parameter tuning, since our scaling rules enable us to directly convert existing hyper-parameter choices from the literature for large batch training. The update equations for SGD with momentum, DISPLAYFORM0 DISPLAYFORM1 A is the "accumulation" variable, while dĈ dω is the mean gradient per training example, estimated on a batch of size B. We initialize the accumulation to zero, and it takes a number of updates for the magnitude of the accumulation to "grow in". During this time, the size of the parameter updates ∆ω is suppressed, reducing the effective learning rate. We can model the growth of the accumulation by assuming that the gradient at the start of training is approximately constant, such that N epochs denotes the number of training epochs performed. The accumulation variable grows in exponentially, and consequently we can estimate the effective number of "lost" training epochs, DISPLAYFORM2 Since the batch size B ∝ /(1 − m), we find N lost ∝ /(N (1 − m) 2 ). We must either introduce additional training epochs to compensate, or ensure that the number of lost training epochs is negligible, when compared to the total number of training epochs performed before the decaying the noise scale. Note that N lost rises most rapidly when one increases the momentum coefficient. We exhibit the test accuracy of our "16-4" wide ResNet implementation on CIFAR10 in FIG7, as a function of the initial learning rate. For learning rate = 0.1, the batch size B = 128 is constant throughout training. This matches the "Original training schedule" of section 5.2 of the main text. When we increase the learning rate we scale B ∝ and perform the same number of training epochs.

Represent the short summary for finding the corresponding scientific article (long)

Decaying the learning rate and increasing the batch size during training are equivalent.

Represent the TLDR for finding the corresponding scientific article

We propose a hypothesis for why gradient descent generalizes based on how per-example gradients interact with each other.


Question: Represent the scientific article to find a short TLDR

Question: Traditional models for question answering optimize using cross entropy loss, which encourages exact answers at the cost of penalizing nearby or overlapping answers that are sometimes equally accurate. We propose a mixed objective that combines cross entropy loss with self-critical policy learning, using rewards derived from word overlap to solve the misalignment between evaluation metric and optimization objective. In addition to the mixed objective, we introduce a deep residual coattention encoder that is inspired by recent work in deep self-attention and residual networks. Our proposals improve model performance across question types and input lengths, especially for long questions that requires the ability to capture long-term dependencies. On the Stanford Question Answering Dataset, our model achieves state of the art with 75.1% exact match accuracy and 83.1% F1, while the ensemble obtains 78.9% exact match accuracy and 86.0% F1. Existing state-of-the-art question answering models are trained to produce exact answer spans for a question and a document. In this setting, a ground truth answer used to supervise the model is defined as a start and an end position within the document. Existing training approaches optimize using cross entropy loss over the two positions. However, this suffers from a fundamental disconnect between the optimization, which is tied to the position of a particular ground truth answer span, and the evaluation, which is based on the textual content of the answer. This disconnect is especially harmful in cases where answers that are textually similar to, but distinct in positions from, the ground truth are penalized in the same fashion as answers that are textually dissimilar. For example, suppose we are given the sentence "Some believe that the Golden State Warriors team of 2017 is one of the greatest teams in NBA history", the question "which team is considered to be one of the greatest teams in NBA history", and a ground truth answer of "the Golden State Warriors team of 2017". The span "Warriors" is also a correct answer, but from the perspective of traditional cross entropy based training it is no better than the span "history".To address this problem, we propose a mixed objective that combines traditional cross entropy loss over positions with a measure of word overlap trained with reinforcement learning. We obtain the latter objective using self-critical policy learning in which the reward is based on word overlap between the proposed answer and the ground truth answer. Our mixed objective brings two benefits: (i) the reinforcement learning objective encourages answers that are textually similar to the ground truth answer and discourages those that are not; (ii) the cross entropy objective significantly facilitates policy learning by encouraging trajectories that are known to be correct. The ing objective is one that is both faithful to the evaluation metric and converges quickly in practice. In addition to our mixed training objective, we extend the Dynamic Coattention Network (DCN) by with a deep residual coattention encoder. This allows the network to build richer representations of the input by enabling each input sequence to attend to previous attention contexts. BID26 show that the stacking of attention layers helps model long-range DISPLAYFORM0 Figure 1: Deep residual coattention encoder.dependencies. We merge coattention outputs from each layer by means of residual connections to reduce the length of signal paths. BID6 show that skip layer connections facilitate signal propagation and alleviate gradient degradation. The combination of the deep residual coattention encoder and the mixed objective leads to higher performance across question types, question lengths, and answer lengths on the Stanford Question Answering Dataset (SQuAD) BID20 compared to our DCN baseline. The improvement is especially apparent on long questions, which require the model to capture long-range dependencies between the document and the question. Our model, which we call DCN+, achieves state-of-the-art on SQuAD, with 75.1% exact match accuracy and 83.1% F1. When ensembled, the DCN+ obtains 78.9% exact match accuracy and 86.0% F1. We consider the question answering task in which we are given a document and a question, and are asked to find the answer in the document. Our model is based on the DCN by, which consists of a coattention encoder and a dynamic decoder. The encoder first encodes the question and the document separately, then builds a codependent representation through coattention. The decoder then produces a start and end point estimate given the coattention. The DCN decoder is dynamic in the sense that it iteratively estimates the start and end positions, stopping when estimates between iterations converge to the same positions or when a predefined maximum number of iterations is reached. We make two significant changes to the DCN by introducing a deep residual coattention encoder and a mixed training objective that combines cross entropy loss from maximum likelihood estimation and reinforcement learning rewards from self-critical policy learning. Because it only has a single-layer coattention encoder, the DCN is limited in its ability to compose complex input representations. BID26 proposed stacked self-attention modules to facilitate signal traversal. They also showed that the network's ability to model long-range dependencies can be improved by reducing the length of signal paths. We propose two modifications to the coattention encoder to leverage these findings. First, we extend the coattention encoder with self-attention by stacking coattention layers. This allows the network to build richer representations over the input. Second, we merge coattention outputs from each layer with residual connections. This reduces the length of signal paths. Our encoder is shown in Figure 1.Suppose we are given a document of m words and a question of n words. Let L D ∈ R e×m and L Q ∈ R e×n respectively denote the word embeddings for the document and the question, where e is the dimension of the word embeddings. We obtain document encodings E D 1 and question DISPLAYFORM0 DISPLAYFORM1 Here, h denotes the hidden state size and the +1 indicates the presence of an additional sentinel word which allows the coattention to not focus on any part of the input. Like the original DCN, we add a non-linear transform to the question encoding. We compute the affinity matrix between the document and the question as DISPLAYFORM2. Let softmax (X) denote the softmax operation over the matrix X that normalizes X column-wise. The document summary vectors and question summary vectors are computed as DISPLAYFORM3 DISPLAYFORM4 We define the document coattention context as follows. Note that we drop the dimension corresponding to the sentinel vector -it has already been used during the summary computation and is not a potential position candidate for the decoder. DISPLAYFORM5 We further encode the summaries using another bidirectional LSTM. DISPLAYFORM6 Equation FORMULA4 to equation 5 describe a single coattention layer. We compute the second coattention layer in a similar fashion. Namely, let coattn denote a multi-valued mapping whose inputs are the two input sequences E D 1 and E Q 1. We have DISPLAYFORM7 The output of our encoder is then obtained as DISPLAYFORM8 where concat (A, B) denotes the concatenation between the matrices A and B along the first dimension. This encoder is different than the original DCN in its depth and its use of residual connections. We use not only the output of the deep coattention network C. This is akin to transformer networks BID26, which achieved stateof-the-art on machine translation using deep self-attention layers to help model long-range dependencies, and residual networks BID6, which achieved state-of-the-art in image classification through the addition of skip layer connections to facilitate signal propagation and alleviate gradient degradation. The DCN produces a distribution over the start position of the answer and a distribution over the end position of the answer. Let s and e denote the respective start and end points of the ground truth answer. Because the decoder of the DCN is dynamic, we denote the start and end distributions produced at the tth decoding step by p start t ∈ R m and p end t ∈ R m. For convenience, we denote the greedy estimate of the start and end positions by the model at the tth decoding step by s t and e t. Moreover, let Θ denote the parameters of the model. Similar to other question answering models, the DCN is supervised using the cross entropy loss on the start position distribution and the end position distribution: DISPLAYFORM9 Equation FORMULA10 states that the model accumulates a cross entropy loss over each position during each decoding step given previous estimates of the start and end positions. The question answering task consists of two evaluation metrics. The first, exact match, is a binary score that denotes whether the answer span produced by the model has exact string match with the ground truth answer span. The second, F1, computes the degree of word overlap between the answer span produced by the model and the ground truth answer span. We note that there is a disconnect between the cross entropy optimization objective and the evaluation metrics. For example, suppose we are given the answer estimates A and B, neither of which match the ground truth positions. However, A has an exact string match with the ground truth answer whereas B does not. The cross entropy objective penalizes A and B equally, despite the former being correct under both evaluation metrics. In the less extreme case where A does not have exact match but has some degree of word overlap with the ground truth, the F1 metric still prefers A over B despite its wrongly predicted positions. We encode this preference using reinforcement learning, using the F1 score as the reward function. DISPLAYFORM10 denote the sampled start and end positions from the estimated distributions at decoding step t. We define a trajectoryτ as a sequence of sampled start and end pointsŝ t andê t through all T decoder time steps. The reinforcement learning objective is then the negative expected rewards R over trajectories. DISPLAYFORM11 We use F 1 to denote the F1 scoring function and ans (s, e) to denote the answer span retrieved using the start point s and end point e. In equation 13, instead of using only the F1 word overlap as the reward, we subtract from it a baseline. BID4 show that a good baseline reduces the variance of gradient estimates and facilitates convergence. In our case, we employ a self-critic BID10 ) that uses the F1 score produced by the current model during greedy inference without teacher forcing. For ease of notation, we abbreviate R (s, e,ŝ T,ê T ; Θ) as R. As per BID25 and BID21, the expected gradient of a non-differentiable reward function can be computed as In equation 16, we approximate the expected gradient using a single Monte-Carlo sample τ drawn from p τ. This sample trajectory τ contains the start and end positionsŝ t andê t sampled during all decoding steps. DISPLAYFORM12 DISPLAYFORM13 One of the key problems in applying RL to natural language processing is the discontinuous and discrete space the agent must explore in order to find a good policy. For problems with large exploration space, RL approaches tend to be applied as fine-tuning steps after a maximum likelihood model has already been trained BID18 BID30. The ing model is constrained in its exploration during fine-tuning because it is biased by heavy pretraining. We instead treat the optimization problem as a multi-task learning problem. The first task is to optimize for positional match with the ground truth answer using the the cross entropy objective. The second task is to optimize for word overlap with the ground truth answer with the self-critical reinforcement learning objective. In a similar fashion to BID8, we combine the two losses using homoscedastic uncertainty as task-dependent weightings. DISPLAYFORM14 Here, σ ce and σ rl are learned parameters. The gradient of the cross entropy objective can be derived using straight-forward backpropagation. The gradient of the self-critical reinforcement learning objective is shown in equation 16. FIG0 illustrates how the mixed objective is computed. In practice, we find that adding the cross entropy task significantly facilitates policy learning by pruning the space of candidate trajectories -without the former, it is very difficult for policy learning to converge due to the large space of potential answers, documents, and questions. We train and evaluate our model on the Stanford Question Answering Dataset (SQuAD). We show our test performance of our model against other published models, and demonstrate the importance of our proposals via ablation studies on the development set. To preprocess the corpus, we use the reversible tokenizer from Stanford CoreNLP. For word embeddings, we use GloVe embeddings pretrained on the 840B Common Crawl corpus BID19 as well as character ngram embeddings by BID5. In addition, we concatenate these embeddings with context vectors (CoVe) trained on WMT BID15. For out of vocabulary words, we set the embeddings and context vectors to zero. We perform word dropout on the document which zeros a word embedding with probability 0.075. In addition, we swap the first maxout layer of the highway maxout network in the DCN decoder with a sparse mixture of experts layer. This layer is similar to the maxout layer, except instead of taking the top scoring expert, we take the top k = 2 expert. The model is trained using ADAM with default hyperparameters. Hyperparameters of our model are identical to the DCN. We implement our model using PyTorch. BID12, BiDAF BID22, DCN w/ CoVe BID15, ReasoNet BID24, Document Reader, FastQA BID29, DCN. The CoVe authors did not submit their model, which we use as our baseline, for SQuAD test evaluation. The performance of our model is shown in BID15. This model is identical to the DCN by, except that it augments the word representations with context vectors trained on WMT16.Comparison to baseline DCN with CoVe. DCN+ outperforms the baseline by 3.2% exact match accuracy and 3.2% F1 on the SQuAD development set. FIG2 shows the consistent performance gain of DCN+ over the baseline across question types, question lengths, and answer lengths. In particular, DCN+ provides a significant advantage for long questions. Ablation study. The contributions of each part of our model are shown in Table 2. We note that the deep residual coattention yielded the highest contribution to model performance, followed by the mixed objective. The sparse mixture of experts layer in the decoder added minor improvements to the model performance. 71.3% -3.2% 79.9% -3.2% Table 2: Ablation study on the development set of SQuAD. Figure 4: Training curve of DCN+ with and without reinforcement learning. In the latter case, only the cross entropy objective is used. The mixed objective initially performs worse as it begins policy learning from scratch, but quickly outperforms the cross entropy model. Mixed objective convergence. The training curves for DCN+ with reinforcement learning and DCN+ without reinforcement learning are shown in Figure 4 to illustrate the effectiveness of our proposed mixed objective. In particular, we note that without mixing in the cross entropy loss, it is extremely difficult to learn the policy. When we combine the cross entropy loss with the reinforcement learning objective, we find that the model initially performs worse early on as it begins policy learning from scratch (shown in Figure 4b). However, with the addition of cross entropy loss, the model quickly learns a reasonable policy and subsequently outperforms the purely cross entropy model (shown in Figure 4a).Sample predictions. FIG4 compares predictions by DCN+ and by the baseline on the development set. Both models retrieve answers that have sensible entity types. For example, the second example asks for "what game" and both models retrieve an American football game; the third example asks for "type of Turing machine" and both models retrieve a type of turing machine. We find, however, that DCN+ consistently make less mistakes on finding the correct entity. This is especially apparent in the examples we show, which contain several entities or candidate answers of the correct type. In the first example, Gasquet wrote about the plague and called it "Great Pestilence". While he likely did think of the plague as a "great pestilence", the phrase "suggested that it would appear to be some form of ordinary Eastern or bubonic plague" provides evidence for the correct answer -"some form of ordinary Eastern or bubonic plague". Similarly, the second example states that Thomas Davis was injured in the "NFC Championship Game", but the game he insisted on playing in is the "Super Bowl". Finally, "multi-tape" and "single-tape" both appear in the sentence that provides provenance for the answer to the question. However, it is the "single-tape" Turing machine that implies quadratic time. In these examples, DCN+ finds the correct entity out of ones that have the right type whereas the baseline does not. Neural models for question answering. Current state-of-the-art approaches for question answering over unstructured text tend to be neural approaches. BID28 proposed one of the first conditional attention mechanisms in the Match-LSTM encoder. Coattention, bidirectional attention flow BID22, and self-matching attention all build codependent representations of the question and the document. These approaches of conditionally encoding two sequences are widely used inThe historian Francis Aidan Gasquet wrote about the'Great Pestilence' in 1893 and suggested that "it would appear to be some form of the ordinary Eastern or bubonic plague". He was able to adopt the epidemiology of the bubonic plague for the Black Death for the second edition in 1908, implicating rats and fleas in the process, and his interpretation was widely accepted for other ancient and medieval epidemics, such as the Justinian plague that was prevalent in the Eastern Roman Empire from 541 to 700 CE.What did Gasquet think the plague was?Carolina suffered a major setback when Thomas Davis, an 11-year veteran who had already overcome three ACL tears in his career, went down with a broken arm in the NFC Championship Game. Despite this, he insisted he would still find a way to play in the Super Bowl. His prediction turned out to be accurate. What game did Thomas Davis say he would play in, despite breaking a bone earlier on?But bounding the computation time above by some concrete function f(n) often yields complexity classes that depend on the chosen machine model. For instance, the language {xx | x is any binary string} can be solved in linear time on a multi-tape Turing machine, but necessarily requires quadratic time in the model of single-tape Turing machines. If we allow polynomial variations in running time, Cobham-Edmonds thesis states that "the time complexities in any two reasonable and general models of computation are polynomially related" (Goldreich 2008, Chapter 1.2). This forms the basis for the complexity class P, which is the set of decision problems solvable by a deterministic Turing machine within polynomial time. The corresponding set of function problems is FP.A language solved in quadratic time implies the use of what type of Turing machine? question answering. After building codependent encodings, most models predict the answer by generating the start position and the end position corresponding to the estimated answer span. The generation process utilizes a pointer network BID27 over the positions in the document. also introduced the dynamic decoder, which iteratively proposes answers by alternating between start position and end position estimates, and in some cases is able to recover from initially erroneous predictions. Neural attention models. Neural attention models saw early adoption in machine translation BID0 and has since become to de-facto architecture for neural machine translation models. Self-attention, or intra-attention, has been applied to language modeling, sentiment analysis, natural language inference, and abstractive text summarization BID18. BID26 extended this idea to a deep self-attentional network which obtained state-of-the-art in machine translation. Coattention, which builds codependent representations of multiple inputs, has been applied to visual question answering BID13. introduced coattention for question answering. Bidirectional attention flow BID22 and self-matching attention also build codependent representations between the question and the document. Reinforcement learning in NLP. Many tasks in natural language processing have evaluation metrics that are not differentiable. BID3 proposed a hierarchical reinforcement learning technique for generating text in a simulated way-finding domain. BID17 applied deep Q-networks to learn policies for text-based games using game rewards as feedback. BID11 introduced a neural conversational model trained using policy gradient methods, whose reward function consisted of heuristics for ease of answering, information flow, and semantic coherence. BID1 proposed a general actor-critic temporal-difference method for sequence prediction, performing metric optimization on language modeling and machine translation. Direct word overlap metric optimization has also been applied to summarization BID18, and machine translation BID30. We introduced DCN+, an state-of-the-art question answering model with deep residual coattention trained using a mixed objective that combines cross entropy supervision with self-critical policy learning. We showed that our proposals improve model performance across question types, question lengths, and answer lengths on the Stanford Question Answering Dataset (SQuAD). On SQuAD, the DCN+ achieves 75.1% exact match accuracy and 83.1% F1. When ensembled, the DCN+ obtains 78.9% exact match accuracy and 86.0% F1.

Represent the short summary for finding the corresponding scientific article (long):

We introduce the DCN+ with deep residual coattention and mixed-objective RL, which achieves state of the art performance on the Stanford Question Answering Dataset.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.