Question: Represent the text to find a very short summary (1-2 sentences max).

Question: This paper introduces a framework for solving combinatorial optimization problems by learning from input-output examples of optimization problems. We introduce a new memory augmented neural model in which the memory is not resettable (i.e the information stored in the memory after processing an input example is kept for the next seen examples). We used deep reinforcement learning to train a memory controller agent to store useful memories. Our model was able to outperform hand-crafted solver on Binary Linear Programming (Binary LP). The proposed model is tested on different Binary LP instances with large number of variables (up to 1000 variables) and constrains (up to 700 constrains). An intelligent agent with a long-term memory processes raw data (as images, speech and natural language sentences) and then transfer these data streams into knowledge. The knowledge stored in the long-term memory can be used later in inference either by retrieving segments of memory during recalling, matching stored concepts with new raw data (e.g. image classification tasks) or solving more complex mathematical problems that require memorizing either the method of solving a problem or simple steps during solving. For example, the addition of long-digit numbers requires memorizing both the addition algorithm and the carries produced from the addition operations BID28.In neural models, the weights connecting the layers are considered long term memories encoding the algorithm that transform inputs to outputs. Other neural models as recurrent neural networks (RNNs) introduce a short-term memory encoded as hidden states of previous inputs BID12 BID7.In memory augmented neural networks (MANNs), a controller writes memories projected from its hidden state to a memory bank (usually in the form of a matrix), the controller then reads from the memory using some addressing mechanisms and generates a read vector which will be fed to the controller in the next time step BID6. The memory will contain information about each of the input sequence tokens and the controller enriches its memory capacity by using the read vector form the previous time step. Unfortunately, In MANNs the memory is not a long-term memory and is re-settable when new examples are processed, making it unable to capture general knowledge about the inputs domain. In context of natural language processing, one will need general knowledge to answer open-ended questions that do not rely on temporal information only but also on general knowledge from previous input streams. In long-digits multiplication, it will be easier to store some intermediate multiplication steps as digit by digit multiplications and use them later when solving other instances than doing the entire multiplication digit by digit each time from scratch. Neural networks have a large capacity of memorizing, a long-term persistent memory will even increase the network capacity to memorize but will decrease the need for learning coarse features of the inputs that requires more depth. Storing features of the inputs will create shortcut paths for the network to learn the correct targets. Such a network will no longer need to depend on depth to learn good features of the inputs but instead will depend on stored memory features. In other words a long-term memory can provide intermediate answers to the network. Unlike regular MANNs and RNNs, a long-term memory can provide shortcut connections to both inputs features and previous time steps inputs. Consider when the memory contains the output of previous examples, the network would cheat from the memory to provide answers. Training such a network will focus on two stages: Learning to find similarities between memory vectors and current input data, learning to transform memory vectors into meaningful representations for producing the final output. The No Free Lunch Theorem of optimization BID25 states that: any two algorithms are equivalent when their performance is averaged across all possible problems, this means that an algorithm that solve certain classes of problems efficiently will be incompetent in other problems. In the setting of combinatorial optimization, there is no algorithm able to do better than a random strategy in expectation. The only way an algorithm outperforms another is to be specialized to a certain class of optimization problems BID0. Learning optimization algorithms from scratch using pairs of input-output examples is a way to outperform other algorithms on certain classes. It is further interesting to investigate the ability of learned models to generate better solutions than hand crafted solvers. The focus of this paper is on designing neural models to solve Binary Linear Programming (or 0-1 Integer Programming) which is a special case of Integer Linear Programming problems where all decision variables are binary. The 0-1 integer programming is one of Krap's 21 NP-complete problems introduced in BID9. The goal of Binary LP is to optimize a linear function under certain constrains. It is proved by BID3 that Binary LP expresses the complexity class NP (i.e any problem in the complexity class NP can be modeled as Binary LP instance).The standard form of a Binary LP problem is: DISPLAYFORM0 where c and b are vectors and A is a matrix. We propose a general framework for long-term memory neural models that uses reinforcement learning to store memories from a neural network. A long-term memory is not resettable and may or may not store hidden states from individual time steps. Instead a long term memory stores information that is considered to be useful for solving similar instances. The controller that decides to write memories follows a policy function that properly constructs the memory contents. We train and test this framework on synthetic data set of Binary LP instances. We analyze the model capability of generalization to more complex instances beyond the training data set. A related line of work is learning to learn and meta-learning, argued that it is an important building block in artificial intelligence. BID26 and BID0 used recurrent neural networks to act as a gradient descent procedure to train other networks. The application of neural networks to combinatorial optimization has a long distinguished history. Most common approach is using Hopfield networks BID8 for solving Travelling Salesman Problem (TSP). Other neural approaches applied to Travelling Salesman Problem include Elastic nets BID4 and Self-organized maps BID5.Pointer networks BID24 an architecture similar to sequence-to-sequence neural network BID20 to solve combinatorial problems as TSP. Pointer networks solve class of problems where the number of target classes in each step of the output depends on the input length which is variable. Recently, BID2 introduced a pointer network for solving combinatorial problems which is optimized using policy gradient methods. This pointer network was applied to TSP and Knapsack. Our work is different from this recent approaches, we applied our model to a general mathematical formulation of combinatorial optimization problems, instead of learning solutions from problem dependent data. We introduce a memory neural network where the coupled memory is persistent and is not resettable. The model is able to learn and store inputs features as neural memories and utilizes them in solving Binary LP instances. The model can learn from minimal amount of data and yet generate better solutions than handcrafted solver.3 LONG-TERM MEMORY NETWORK FOR BINARY LP Figure 1: The operation of an LTMN processing one Binary LP instance. First the encoder receives a sequence of constrains coefficients and produces a memory vector via a linear projection of encoder final hidden state. The memory controller receives this linear projection and decides whether to store it in memory M or delete the current slot it points to. A control signal (sequence of costs) is then passed to the decoder to produce un-normalized weights w t over the memory locations. A dot product operation between w t and M Finally the output o t at each time step is generated using the memory vector (or encoder final state), the decoder hidden state and the read vector. A sequence-to-sequence approach to solve Binary LP requires formulating the entire problem inputs (costs vector c T, bounds vector b and constrains matrix A) as one sequence and mapping it to an output sequence. A naive linear programming solver constructs the set of feasible solutions from the constrains matrix A, we denote this set as S F, then it iterates over S F using the cost function to find the optimal solution. For a Binary LP instance, the number of possible feasible solutions is 2 N where N is the number of variables. The model we describe works in in a similar way to the naive solver, an encoder encodes the entire constrains matrix into one fixed vector v a, this vector is a continuous representation of the set S F. The vector v a is then passed as the initial state of a decoder along with each cost c t that is used as a control signal to read from memory. Figure 1 describes the operation of an LTMN for a single Binary LP instance. A general Long-Term Memory Network (LTMN) consists of three basic modules: a core network, a memory bank in the form of a matrix and a memory controller. The core network has three main functions: providing memory fragments that are useful information from input signals to be stored in the memory, providing a reading mechanism to read from the memory, and finally producing an output. The memory bank is a matrix R × C, were each row is addressed by a pointer P T R which is a value in range [0, R − 1]. The pointer is provided to the memory controller to know the last accessed memory slot. Clearing the memory contents is not allowed by the memory controller. A memory controller is another module that uses reinforcement learning as a hard attention mechanism to store memory fragments from the core network as in BID27. Upon processing of each example, the controller basically chooses either to store or discard the memory fragments. Writing to the memory in LTMN is totally a discrete process, the final computational graph will not be fully differentiable. Thus it is essential to pass the memories to the output layer for backpropagation to work. The final output depends on both memory contents representation (a read vector) and inputs embedding, this is similar to end-to-end memory networks BID19 that is used in question answering tasks. In end-to-end memory networks the final answer depends on both the output memory representation and question embedding. For non-sequential data the core network can be a regular feed forward neural network and the memories can be the hidden vectors from each layer, while for sequential data the core network can be a recurrent neural network and memories can be its hidden states at each time step. The attention mechanism BID1 used over to attend to memory locations will depend either on a control signal or previous output. Let the sequence A = {a 11, a 12, ..., a 1N, SEP ARAT OR, b i, SEP ARAT OR, ..., a M N} be the sequence describing the constrains set, a ij represents the coefficient of the j th variable in the i th constrain. The SEP ARAT OR token separates between the left and right hand sides of the constrains inequality, the same token is used to separate between constrains in the sequence. Finally b i is the bound for the i th constrain. The encoder module encodes the entire sequence A in to a fixed vector. The encoder then produces a linear projection of this vector to be stored in memory. Equations and describe the constrains encoder operation. DISPLAYFORM0 where W hv and b h are weight matrix and bias vector respectively. The input to the decoder is a sequence of costs C = {c 1, ...c N}. The decoder uses the fixed vector v a as initial hidden state. At each time step the decoder reads a cost c t of the costs vector C and produces a hidden state vector: DISPLAYFORM0 The reader uses the decoder hidden state h d t to produce weights that are used to address the memory: DISPLAYFORM1 A simple dot product operation between the memory and the weight vector produces a read vector: DISPLAYFORM2 where M (i) is the memory produced by the memory controller after processing instance i in the data set. The linear projection h a, the cost embedding w t and the read vector r t are used to generate a final output: DISPLAYFORM3 The final output layer is a soft-max layer, and the whole model can be thought of estimating the probability distribution P (o t | h a, w t, r t). Each cost is embedded via a linear layer first, then the embedding is passed to recurrent layers. The reader is implemented as a recurrent layer, in this sense the weights produced will depend on both the cost at time step t (the control signal) and the weight from the previous time step w t−1.The reader produces un-normalized weights over memory slots N in range [-1,1]. When the weight for a slot i is 0 this means that the memory slot does not contribute at all to the read vector, when it is 1 the whole memory slot is used. The interesting case is when the weight is −1 which means the inverse of the current memory slot contents, when the weight is between between 0 and 1 the information from the memory slot is preserved to a certain extent, instead a −1 weight transforms the entire slot into a new representation. Hopefully, the reader weights learned by backpropgation through the whole network will not only act as attention weights over memory slots but also transform the memory contents in to useful representations. A typical memory controller is provided a memory fragment at a discrete time t and makes a decision whether to store or discard this memory. Learning to store useful memories is descirbed as an reinforcement learning (RL) framework. An agent senses the environment s, takes an action a from an action space A using a policy function π and receives an immediate reward r. The total future reward from point t is expressed by: DISPLAYFORM0 where γ is the discount factor. The environment is the memory bank at discrete time step t along with the current pointer P T R t and a window of previous memories W t that the controller has been subjected to. A window is similar to a short term memory with limited size τ and the controller chooses to store one of the contents of that memory. In all our experiments, the controller stores the last memory vector in the window, in this sense the window gives insights to the controller about what memories have been produced by the core network. The basic actions a controller may perform are storing the current memory we refer to this as a ST R, and not to store any thing at all we refer to this as a N O−OP. We include three more actions: delete the current slot a DELET E, increment the pointer then store a ST RIN C, and decrement the pointer then store a ST RDEC. An action taken by the controller is evaluated through a reward function. One can evaluate the controller the same way we evaluate the core network, through common metrics as final loss or accuracy, for each example the model solves correctly the controller gets a reward. In this way, the rewards received will only account for how better the whole model gets on solving tasks. Instead we want the controller to store useful memories that will be used in the next processed examples. The reward function is a simple question: Did the current memory in a better response than empty memory?. The reward function is: DISPLAYFORM0 where M Zero is the memory with zero entries, M t is the memory produced by memory controller and eval is an evaluation function for the core network. When the entries of a read vector are all zeros the output will depend only on information coming from the control signal (costs c t in case of Binary LP). The memory controller receives a reward only when the non-empty memory in solving an example correctly and empty memory in incorrect response. An optimal training procedure will make the core network depends only on non-empty memory to minimize a loss function, while an optimal training procedure for the memory controller will in useful memories only. At the heart of the memory controller is an RL agent which interacts with the environment. The environment is basically the memories received per each example the LTMN solves. For each instance the LTMN processes, the RL agent receives a state s t and selects an action a t following a policy π(a t |s t) and receives a reward r t and the next state s t+1 BID21.To train the memory controller agent, we used deep Q-learning algorithm with experience replay BID15 as described in algorithm 1. A neural network is used to estimate the actionvalue function Q(s, a) BID23 BID18 BID17. We choose to implement the action-value neural network as a stacked LSTM which receives the memory contents M t, one slot each time step, followed by window contents W t and the pointer P T R.We draw the similarity between using deep Q-learning algorithm for playing Atari games and using it for storing long-term memories. During each episode the core network is given a sequence of instances form a training dataset X controller and the memory agent should take proper actions on memory to construct useful memory entries that helps solving theses instances. One can consider the agent is learning to construct the memory as constructing a puzzle from pieces, where the pieces come from a bucket. There will be no final state for the agent to reach, because of the fact that one can not know exactly how the memory should be constructed. In the setting of Atari games the agent will reach the final state (game over) quickly in the first few epochs as the agent will still be struggling to learn an optimal action-value function. In the case of the memory agent, we have to determine an episode length T that both simulates the agent failure in the earlier training phase and the agent reaching a final state. The final state of the agent will be reached when the LTMN processes the last example in an episode. Thus, there will be an infinite number of final states. We can keep the episode length constant so the agent have a limited time to succeed in accumulating rewards. Both the reader and the memory controller agent should be trained jointly together. This procedure is critical since both the reader and the memory controller depend on each other, the reader learns to generate proper weights over memory slots that depend on a control signal and the memory controller learns how to construct memories that are useful for the reader. For each action the memory agent perform on the memory, one should train the core network. We sample a batch of instances from training data X train, run the memory agent using this batch for one session (memory is not cleared between instances) and perform the backward pass on the core network. A typical learning algorithm for neural networks as stochastic gradient descent, updates the network parameters using data batches. The number of iterations (one forward pass followed by one backward pass) is equal to N/m where N is the size of dataset and m is the batch size. The number of iterations can get large as the size of the dataset increases. To effectively train the LTMN core network, the episode length should be as the same as the number of iterations. In our experiments, we keep the episode length T as small as possible and then increase it after K epochs. To avoid similar episode lengths and simulate various solving sessions (where the memory controller agent has to store memories), each episode T is chosen randomly between [k, N/m], where k is the minimum length of an episode. We generate two separate data sets: one for training the core network and the other for training the memory controller. We generate 14k Binary LP instances as the core network training set, 3k instances as the memory controller training set and 3k as a validation/test set. We use mixed curriculum learning strategy as suggested in BID28. For each instance, we generate two random numbers n and m, where n is the number of variables and m is the number of constrains for the instance. The maximum number of variables in an instance is N and maximum number of constrains is M. In our experiments, n is chosen uniformly between [3,N] and m is between [1,M]. For our training dataset N is 10 and M is 5.All the coefficients of the objective function and the constrains matrix A is generated between [-99,99]. To ensure that the constrains matrix is sparse we generate a random number SL called sparsity level, for each constrain in the problem we ensure that at most 1/3 of the coefficients are zeros. To generate supervised targets for the problems we used a python interface to the COIN-OR branch and cut solver BID13 called PuLP BID14. We ensure that all the generated problems have a feasible solution. We use denote the COIN-OR branch and cut solver as the baseline solver and compare the solver with the LTMN . Select randomly an episode length T 5:for t = 1,T do 6:Select randomly an example x i from X controller 7:Solve x i using Encoder to get memory vector h ai 8:Store h ai in to window W t With probability select a random action a t 10:otherwise select a t = max a Q * (s t, a; θ)11:Execute action a t on memory M t observe the next memory state M t+1 and reward r t 12: DISPLAYFORM0 Sample random minibatch of transitions (s t, a t, r t, s t+1) from D end for 21: end for We implement both the encoder and decoder as a recurrent neural network with gated recurrent units (GRU). We use three GRU layers each with 256 units followed by a linear layer with 128 units to project the memory vector of the constrains sequence. As suggested by BID2, it is harder to learn from input-output examples of optimization problems due to subtle features that model cannot figure out. We suggest a technique to learn more features by connecting the 1, 2,..., L − 1 layers to layer L instead of connecting only the previous layer L − 1. In this way an encoder can learn more features using combined features from the previous layers. The decoder is implemented as two stacked GRU layers, the reader is implemented as one GRU layer. The costs are first embedded using a linear layer of 64 units. The rest of the decoder has 256 units. All the weights are initialized uniformly in [-0.08,0,08]. We set the dropout ratio to 0.2 in all GRU layers. We constrain the norm of the gradients to be no greater than 5 (gradient clipping). We set the learning rate initially to 0.001 and drop it by factor 0.5 after each 10 epochs. The memory controller agent is implemented as a stacked LSTM of 2 layers each with 200 units. We used batch normalization over the inputs to normalize the memory contents and window contents to zero-mean and unit variance. We used Adam BID10 optimizer to train the agent. The window size τ is 5 in all our experiments. We used generalization loss as an early stopping criteria for the core network as suggested by BID16. We then allow the memory controller agent to train until a reasonable average total reward per episode is reached. We compare the performance of LTMN to the baseline solver and the sequence-to-sequence approach. We trained a network of GRU units with comparable number of parameters (seq-to-seq has 1.3M parameters and LTMN has 1.9M parameters). The seq-to-seq network receives the input as one long sequence. We train the LTMN core network and seq-to-seq model using RMSprop algorithm BID22 with cross entropy as a loss function. We define a metric to test the quality of solutions produced by the model, we define average costs over N Binary LP instances: DISPLAYFORM0 All the instances in training data and testing data are maximization problems, so the higher the average costs the better. We sample only feasible solutions using the output probability distribution with temperatures [0.6, 1.0, 1.2, 1.5, 2.2].First we evaluate the LTMN model against the baseline solver, we generate a test set of 1000 instances where all instances have 10 variables and the number of constrains is between 1 and 5. Table 1 shows at different sampling temperatures the average costs and the number of instances where the model outperformed the solver and found a better solution. The LTMN model outperformed the baseline solver by large margin (average costs are nearly 51.7% higher).To validate that the sampling technique is effective, we used an initial untrained model to sample solutions. TAB1 shows that the untrained model performed poorly on the same test set, and hence the random sampling is not generating solutions by chance and the trained model learned to generate solutions effectively. We also evaluate the LTMN against the seq-to-seq GRU model which fails to generate any feasible solutions. While the number of instances is small (only 14K instances), other similar models as pointer networks BID24 used a training data of 1M instances for Travelling Salesman Problem, however our model did not require large training dataset. We test the generalization ability of LTMN to longer sequences beyond those in training set. FIG2 shows the on test sets of Binary LP instances (each test set contains 1000 instances) where the number of variables is incremented while the maximum number of constrains is 5. LTMN outperformed the baseline solver by large margin even when the number of variables is larger than 100, the LTMN was still able to generate better solutions. We test LTMN generalization on very large instances. TAB2 shows that LTMN outperformed the baseline solver even for very large instances with 1000 variables and 700 constrains. To understand the effectiveness of using an augmented long-term memory network for solving Binary LP instances, we conduct tests to prove that long-term memory improves model . We calculate the average costs for the same test set (10 variables and constrains between 1 and 5) but we reset the memory each time a new example is processed. TAB3 shows that the average costs is slightly dropped when the memory is reset between examples. We conduct a per-example test where we identify whether memory helped in generating good solutions or not. Given two problems identified by their cost vectors and constrains matrices as [c 1, A 1] and [c 2, A 2]. Let A 1 = A 2 but both of them construct the same set of feasible solutions S F. The encoder produces two memory vectors h A1 and h A2 for constrains A 1 and A 2 respectively. To ensure that both h A1 and h A2 represent the same set of feasible solutions S F, we measure the cosine similarity between these two memory vectors such that CosSim(h A1, h A2) ≥ S, where S is set to 0.8. We then enforce the controller to store h A1, and generate a solution for [c 2, A 2] We define three metrics:Memory Faults: number of examples where non-empty memory in worse solution than empty memory. Memory Trues: number of examples where non-empty memory in better solution than empty memory. Memory Equals: number of examples where both non-empty memory and empty memory in the same solution. We generate 10K instances (with 10 variables and constrains in range of ), we compare the feasible solutions of each two consecutive instances and calculate the cosine similarity between the two memory vectors. We found 181 instances with the same feasible solutions. We record the solutions of these instances using both empty memory and non-empty memory containing h A1. TAB4 shows high memory trues where non-empty memory helped generating better solutions than empty memory. The table also shows high percentage of memory equals, for these 28 instances the non-empty memory did not help much in generating a better solution, in fact both memory faults and memory equals are similar metrics and can be thought of as the number of instances where non-empty memory failed to generate a better solution. A good memory controller agent should maximize the memory trues and minimize both the memory faults and equals at the same time. We conclude that a long-term memory is quite effective in generating better solutions and the memory controller learns effectively how to store input features useful for longer interaction with a model. This paper introduced a long term memory coupled with a neural network, that is able to memorize useful input features to solve similar instances. We applied LTMN model to solve Binary LP instances. The LTMN was able to learn from supervised targets provided by a handcrafted solver, and generate better solutions than the solver. The LTMN model was able to generalize to more complex instances beyond those in the training set.

We propose a memory network model to solve Binary LP instances where the memory information is perseved for long-term use.

Represent the TLDR for finding the corresponding scientific article

We use a hypernetwork to predict optimal weights given hyperparameters, and jointly train everything together.


Question: Represent the scientific article to find a short TLDR

Question: Sequence-to-sequence (Seq2Seq) models with attention have excelled at tasks which involve generating natural language sentences such as machine translation, image captioning and speech recognition. Performance has further been improved by leveraging unlabeled data, often in the form of a language model. In this work, we present the Cold Fusion method, which leverages a pre-trained language model during training, and show its effectiveness on the speech recognition task. We show that Seq2Seq models with Cold Fusion are able to better utilize language information enjoying i) faster convergence and better generalization, and ii) almost complete transfer to a new domain while using less than 10% of the labeled training data. Sequence-to-sequence (Seq2Seq) BID1 models have achieved state-of-the-art on many natural language processing problems including automatic speech recognition BID2 BID4, neural machine translation, conversational modeling and many more. These models learn to generate a variable-length sequence of tokens (e.g. texts) from a variable-length sequence of input data (e.g. speech or the same texts in another language). With a sufficiently large labeled dataset, vanilla Seq2Seq can model sequential mapping well, but it is often augmented with a language model to further improve the fluency of the generated text. Because language models can be trained from abundantly available unsupervised text corpora which can have as many as one billion tokens BID13 BID19, leveraging the rich linguistic information of the label domain can considerably improve Seq2Seq's performance. A standard way to integrate language models is to linearly combine the score of the task-specific Seq2Seq model with that of an auxiliary langauge model to guide beam search BID5 BID20. BID10 proposed an improved algorithm called Deep Fusion that learns to fuse the hidden states of the Seq2Seq decoder and a neural language model with a gating mechanism, after the two models are trained independently. While this approach has been shown to improve performance over the baseline, it has a few limitations. First, because the Seq2Seq model is trained to output complete label sequences without a language model, its decoder learns an implicit language model from the training labels, taking up a significant portion of the decoder capacity to learn redundant information. Second, the residual language model baked into the Seq2Seq decoder is biased towards the training labels of the parallel corpus. For example, if a Seq2Seq model fully trained on legal documents is later fused with a medical language model, the decoder still has an inherent tendency to follow the linguistic structure found in legal text. Thus, in order to adapt to novel domains, Deep Fusion must first learn to discount the implicit knowledge of the language. In this work, we introduce Cold Fusion to overcome both these limitations. Cold Fusion encourages the Seq2Seq decoder to learn to use the external language model during training. This means that Seq2Seq can naturally leverage potentially limitless unsupervised text data, making it particularly proficient at adapting to a new domain. The latter is especially important in practice as the domain from which the model is trained can be different from the real world use case for which it is deployed. In our experiments, Cold Fusion can almost completely transfer to a new domain for the speech recognition task with 10 times less data. Additionally, the decoder only needs to learn task relevant information, and thus trains faster. The paper is organized as follows: Section 2 outlines the and related work. Section 3 presents the Cold Fusion method. Section 4 details experiments on the speech recognition task that demonstrate Cold Fusion's generalization and domain adaptation capabilities.2 AND RELATED WORK 2.1 SEQUENCE-TO-SEQUENCE MODELS A basic Seq2Seq model comprises an encoder that maps an input sequence x = (x 1, . . ., x T) into an intermediate representation h, and a decoder that in turn generates an output sequence y = (y 1, . . ., y K) from h BID21. The decoder can also attend to a certain part of the encoder states with an attention mechanism. The attention mechanism is called hybrid attention BID7, if it uses both the content and the previous context to compute the next context. It is soft if it computes the expectation over the encoder states BID1 as opposed to selecting a slice out of the encoder states. For the automatic speech recognition (ASR) task, the Seq2Seq model is called an acoustic model (AM) and maps a sequence of spectrogram features extracted from a speech signal to characters. During inference, we aim to compute the most likely sequenceŷ: DISPLAYFORM0 Here, p(y|x) is the probability that the task-specific Seq2Seq model assigns to sequence y given input sequence x. The argmax operation is intractable in practice so we use a left-to-right beam search algorithm similar to the one presented in BID20. We maintain a beam of K partial hypothesis starting with the start symbol s. At each time-step, the beam is extended by one additional character and only the top K hypotheses are kept. Decoding continues until the stop symbol /s is emitted, at which point the hypothesis is added to the set of completed hypotheses. A standard way to integrate the language model with the Seq2Seq decoder is to change the inference task to:ŷ = argmax DISPLAYFORM1 where p LM (y) is the language model probability assigned to the label sequence y.; describe several heuristics that can be used to improve this basic algorithm. We refer to all of these methods collectively as Shallow Fusion, since p LM is only used during inference. BID10 proposed Deep Fusion for machine translation that tightens the connection between the decoder and the language model by combining their states with a parametric gating: DISPLAYFORM2 where s t, s In Deep Fusion, the Seq2Seq model and the language model are first trained independently and later combined as in Equation. The parameters v and b are trained on a small amount of data keeping the rest of the model fixed, and allow the gate to decide how important each of the models are for the current time step. Ground Truth where's the sport in that greer snorts and leaps greer hits the dirt hard and rolls Plain Seq2Seqwhere is the sport and that through snorks and leaps clear its the dirt card and rules Deep Fusion where is the sport and that there is north some beliefs through its the dirt card and rules Cold Fusion where's the sport in that greer snorts and leaps greer hits the dirt hard and rolls Cold Fusion (Fine-tuned) where's the sport in that greer snorts and leaps greer hits the dirt hard and rollsGround Truth jack sniffs the air and speaks in a low voice Plain Seq2Seqjacksonice the air and speech in a logos Deep Fusion jacksonice the air and speech in a logos Cold Fusion jack sniffs the air and speaks in a low voice Cold Fusion (Fine-tuned) jack sniffs the air and speaks in a low voice Ground Truth skipper leads her to the dance floor he hesitates looking deeply into her eyes Plain Seq2Seq skip er leadure to the dance floor he is it takes looking deeply into her eyes Deep Fusion skip er leadure to the dance floor he has it takes looking deeply into her eyes Cold Fusion skipper leads you to the dance floor he has a tates looking deeply into her eyes Cold Fusion (Fine-tuned) skipper leads her to the dance floor he hesitates looking deeply into her eyes Table 1: Some examples of predictions by the Deep Fusion and Cold Fusion models. The biggest disadvantage with Deep Fusion is that the task-specific model is trained independently from the language model. This means that the Seq2Seq decoder needs to learn a language model from the training data labels, which can be rather parsimonious compared to the large text corpora available for language model training. So, the fused output layer of should learn to overcome this bias in order to incorporate the new language information. This also means that a considerable portion of the decoder capacity is wasted. A few methods have been proposed for leveraging unlabeled text corpora in the target domain, for both better generalization and domain transfer. BID18 proposed backtranslation as a way of using unlabeled data for machine translation. Backtranslation improves the BLEU score by increasing the parallel training corpus of the neural machine translation model by automatically translating the unlabeled target domain text. However, this technique does not apply well to other tasks where backtranslation is infeasible or of very low quality (like image captioning or speech recogntion). BID17 proposed warm starting the Seq2Seq model from language models trained on source and target domains separately. Unsupervised pre-training shows improvements in the BLEU scores. BID17 also show that this improvement is from improved generalization, and not only better optimization. While this is a promising approach, the method is potentially difficult to leverage for the transfer task since training on the parallel corpus could end up effectively erasing the knowledge of the language models. Both back-translation and unsupervised pre-training are simple methods that require no change in the architecture. Our proposed Cold Fusion method is largely motivated from the Deep Fusion idea but with some important differences. The biggest difference is that in Cold Fusion, the Seq2Seq model is trained from scratch together with a fixed pre-trained language model. Because the Seq2Seq model is aware of the language model throughout training, it learns to use the language model for language specific information and capture only the relevant information conducive to mapping from the source to the target sequence. This disentanglement can increase the effective capacity of the model significantly. This effect is demonstrated empirically in Section 4 where Cold Fusion models perform well even with a very small decoder. We also improve on some of the modeling choices of the fusion mechanism.1. First, both the Seq2Seq hidden state s t and the language model hidden state s LM t can be used as inputs to the gate computation. The task-specific model's embedding contains information about the encoder states which allows the fused layer to decide its reliance on the language model in case of input uncertainty. For example, when the input speech is noisy or a token unseen by the Seq2Seq model is presented, the fusion mechanism learns to pay more attention to the language model. 2. Second, we employ fine-grained (FG) gating mechanism as introduced in BID24. By using a different gate value for each hidden node of the language model's state, we allow for greater flexibility in integrating the language model because the fusion algorithm can choose which aspects of the language model it needs to emphasize more at each time step. 3. Third, we replace the language model's hidden state with the language model probability. The distribution and dynamics of s LM t can vary considerably across different language models and data. As a concrete example, any fusion mechanism that uses the LM state is not invariant to the permutation of state hidden nodes. This limits the ability to generalize to new LMs. By projecting the token distribution onto a common embedding space, LMs that model novel uses of the language can still be integrated without state discrepancy issues. This also means that we can train with or swap on n-gram LMs during inference. The Cold Fusion layer (Figure 1) works as follows: DISPLAYFORM0 is the logit output of the language model, s t is the state of the task specific model, and s CF t is the final fused state used to generate the output. Since logits can have arbitrary offsets, the maximum value is subtracted off before feeding into the layer. In (4a), (4d), the DNN can be a deep neural network with any number of layers. In our experiments, we found a single affine layer, with ReLU activation prior to softmax, to be helpful. For our experiments, we tested the Cold Fusion method on the speech recognition task. The are compared using the character error rate (CER) and word error rate (WER) on the evaluation sets. For all models which were trained on the source domain, the source CER and WER indicate in-domain performance and the target CER and WER indicate out-of-domain performance. We collected two data sets: one based on search queries which served as our source domain, and another based on movie transcripts which served as our target domain. For each dataset, we used Amazon Mechanical Turk to collect audio recordings of speakers reading out the text. We gave identical instructions to all the turkers in order to ensure that the two datasets only differed in the text domain. The source dataset contains 411,000 utterances (about 650 hours of audio), and the target dataset contains 345,000 utterances (about 676 hours of audio). We held out 2048 utterances from each domain for evaluation. The text of the two datasets differ significantly. Table 2 shows of training character-based recurrent neural network language models BID15 on each of the datasets and evaluating on both datasets. Language models very easily overfit the training distribution, so models trained on one corpus will perform poorly on a different distribution. We see this effect in Table 2 Table 2: Dev set perplexities for character RNN language models trained on different datasets on source and target domain. Note that i) the model trained on source domain does poorly on target domain and vice-versa indicating that the two domains are very different, and ii) the best model on both domains is a larger model trained on a superset of both corpuses. We use the model trained on the full dataset (which contains the source and target datasets along with some additional text) for all of the LM integration experiments. The language model described in the final row of Table 2 was trained on about 25 million words. This model contains three layers of gated recurrent units (GRU) BID8 ) with a hidden state dimension of 1024. The model was trained to minimize the cross-entropy of predicting the next character given the previous characters. We used the Adam optimizer BID14 with a batch size of 512. The model gets a perplexity of 2.49 on the source data and 2.325 on the target data. TAB4: Results from models trained on the publicly available Librispeech data. All of the acoustic models were trained on Librispeech training data and evaluated on librispeech test-clean, WSJ test-92 and our proprietary target domain data. Results from the Wav2Letter model BID9 ) are presented for referenceFor the acoustic models, we used the Seq2Seq architecture with soft attention based on BID2. The encoder consists of 6 bidirectional LSTM (BLSTM) BID12 layers each with a dimension of 480. We also use max pooling layers with a stride of 2 along the time dimension after the first two BLSTM layers, and add residual connections BID11 for each of the BLSTM layers to help speed up the training process. The decoder consisted of a single layer of 960 dimensional Gated Recurrent Unit (GRU) with a hybrid attention BID7.The final Cold Fusion mechanism had one dense layer of 256 units followed by ReLU before softmax. The input sequence consisted of 40 mel-scale filter bank features. We expanded the datasets with noise augmentation; a random noise is added with a 40% probability at a uniform random SNR between 0 and 15 dB. We did not use any other form of regularization. We trained the entire system end-to-end with Adam BID14 ) with a batch size of 64. The learning rates were tuned separately for each model using random search. To stabilize training early on, the training examples were sorted by increasing input sequence length in the first epoch BID0. During inference, we used beam search with a fixed beam size of 128 for all of our experiments. We also used scheduled sampling with a sampling rate of 0.2 which was kept fixed throughout training. Scheduled sampling helped reduce the effect of exposure bias due to the difference in the training and inference mechanisms. Leveraging a language model that has a better perplexity on the distribution of interest should directly mean an improved WER for the ASR task. In this section, we compare how the different fusion methods fare in achieving this effect. Swapping the language model is not possible with Deep Fusion because of the state discrepancy issue motivated in Section 3. All fusion models were therefore trained and evaluated with the same language model that achieved a low perplexity on both the source and target domains (See Table 2). This way, we can measure improvements in transfer capability over Deep Fusion due to the training and architectural changes. TAB2 compares the performance of Deep Fusion and Cold Fusion on the source and target held-out sets. Clearly, Cold Fusion consistently outperforms on both metrics on both domains than the baselines. For the task of predicting in-domain, the baseline model gets a word error of 14.68%, while our best model gets a relative improvement of more than 21% over that number. Even compared to the recently proposed Deep Fusion model BID10, the best Cold Fusion model gets a relative improvement of 15%.We get even bigger improvements in out-of-domain . The baseline attention model, when trained on the source domain but evaluated on the target domain gets, 43.5% WER. This is significantly worse than the 17.6% that we can get by training the same model on the target dataset. The goal of domain adaptation is to bridge the gap between these numbers. The final column in TAB2 shows the remaining gap as a fraction of the difference for each model. We present additional on the publicly available Librispeech dataset BID16 ) in TAB4. Librispeech contains 1000 hours of read audio speech and comes with train, development and test splits. We used similar architectures for the Seq2Seq model and the language model as in the previous experiments, but we trained the Seq2Seq model only on the 960 hour Librispeech training dataset. We test whether cold fusion does indeed relieve the decoder of learning a language model. We do so by checking how a decrease in the decoder capacity affected the error rates. As evidenced in TAB3, the performance of the Cold Fusion models degrades gradually as the decoder cell size is decreased whereas the performance of the attention models deteriorates abruptly beyond a point. It is remarkable that the Cold Fusion decoder still outperforms the full attentional decoder with 4× fewer number of parameters. Also, we find that training is accelerated by a factor of 3 (see Figure 2). Attention models typically need hundreds of thousands of iterations to converge BID6. Most of the training time is spent in learning the attention mechanism. One can observe this behavior by plotting the attention context over time and seeing that the diagonal alignment pattern emerges in later iterations. Because the pretrained, fixed language model infuses the model with lower level language features like the likely spelling of a word, error signals propagate more directly into the attention context. In the presence of limited data from the target distribution, fine tuning a model for domain transfer is often a promising approach. We test how much labeled data from the target distribution is required for Cold Fusion models to effectively close the domain adaptation gap. The same language model from Section 4.4 trained on both the source and target domains was used for all fine-tuning experiments. The learning rate was restored to its initial value. Then, we finetuned only the fusion mechanism of the best Cold Fusion model from TAB2 on various amounts of the labeled target dataset. Results are presented in TAB5. With just 0.6% of labeled data, the domain gap decreases from 38.2% to 21.3%. With less than 10% of the data, this gap is down to only 8%. Note that because we keep the Seq2Seq parameters fixed during the fine-tuning stage, all of the improvements from fine-tuning come from combining the acoustic and the language model better. It's possible that we can see bigger gains by fine-tuning all the parameters. We do not do this in our experiments because we are only interested in studying the effects of language model fusion in the Seq2Seq decoder. Some examples are presented in Table 1. Recall that all models are trained on the source domain consisting of the read speech of search queries and evaluated on the read speech of movie scripts to measure out-of-domain performance. Because search queries tend to be sentence fragments, we see that the main mode of error for vanilla attention and Deep Fusion is due to weak grammar knowledge. Cold Fusion on the other hand demonstrates a better grasp of grammar and is able to complete sentences. In this work, we presented a new general Seq2Seq model architecture where the decoder is trained together with a pre-trained language model. We study and identify architectural changes that are vital for the model to fully leverage information from the language model, and use this to generalize better; by leveraging the RNN language model, Cold Fusion reduces word error rates by up to 18% compared to Deep Fusion. Additionally, we show that Cold Fusion models can transfer more easily to new domains, and with only 10% of labeled data nearly fully transfer to the new domain.

Represent the TLDR for finding the corresponding scientific article!

We introduce a novel method to train Seq2Seq models with language models that converge faster, generalize better and can almost completely transfer to a new domain using less than 10% of labeled data.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce and analyze the phenomenon of "hallucinations" in NMT, or spurious translations unrelated to source text, and propose methods to reduce its frequency.


Question: A central capability of intelligent systems is the ability to continuously build upon previous experiences to speed up and enhance learning of new tasks. Two distinct research paradigms have studied this question. Meta-learning views this problem as learning a prior over model parameters that is amenable for fast adaptation on a new task, but typically assumes the set of tasks are available together as a batch. In contrast, online (regret based) learning considers a sequential setting in which problems are revealed one after the other, but conventionally train only a single model without any task-specific adaptation. This work introduces an online meta-learning setting, which merges ideas from both the aforementioned paradigms to better capture the spirit and practice of continual lifelong learning. We propose the follow the meta leader (FTML) algorithm which extends the MAML algorithm to this setting. Theoretically, this work provides an O(logT) regret guarantee for the FTML algorithm. Our experimental evaluation on three different large-scale tasks suggest that the proposed algorithm significantly outperforms alternatives based on traditional online learning approaches. Two distinct research paradigms have studied how prior tasks or experiences can be used by an agent to inform future learning. Meta-learning casts this as the problem of learning to learn, where past experience is used to acquire a prior over model parameters or a learning procedure. Such an approach, where we draw upon related past tasks and form associated priors, is particularly crucial to effectively learn when data is scarce or expensive for each task. However, meta-learning typically studies a setting where a set of meta-training tasks are made available together upfront as a batch. In contrast, online learning considers a sequential setting where tasks are revealed one after another, but aims to attain zero-shot generalization without any task-specific adaptation. We argue that neither setting is ideal for studying continual lifelong learning. Metalearning deals with learning to learn, but neglects the sequential and non-stationary nature of the world. Online learning offers an appealing theoretical framework, but does not generally consider how past experience can accelerate adaptation to a new task. In this work, we motivate and present the online meta-learning problem setting, where the agent simultaneously uses past experiences in a sequential setting to learn good priors, and also adapt quickly to the current task at hand. Our contributions: In this work, we first formulate the online meta-learning problem setting. Subsequently, we present the follow the meta-leader (FTML) algorithm which extends MAML to this setting. FTML is analogous to follow the leader in online learning. We analyze FTML and show that it enjoys a O(log T) regret guarantee when competing with the best metalearner in hindsight. In this endeavor, we also provide the first set of (under any assumptions) where MAML-like objective functions can be provably and efficiently optimized. We also develop a practical form of FTML that can be used effectively with deep neural networks on large scale tasks, and show that it significantly outperforms prior methods in terms of learning efficiency on vision-based sequential learning problems with the MNIST, CIFAR, and PASCAL 3D+ datasets. Due to space constraints, we review the foundations of our work (meta-learning and online learning) in Appendix A. We consider a general sequential setting where an agent faces tasks one after another. Each task corresponds to a round, denoted by t. In each round, the goal of the learner is to determine model parameters w t that perform well for the corresponding task at that round. This is monitored by f t: w ∈ W → R, which we would like to be minimized. Crucially, we consider a setting where the agent can perform some local task-specific updates to the model before it is deployed and evaluated T t=1 be the sequence of models generated by the algorithm. Then, the regret we consider is: DISPLAYFORM0 Notice that we allow the comparator to adapt locally to each task at hand; thus the comparator has strictly more capabilities than the learning agent, since it is presented with all the task functions in batch mode. Achieving sublinear regret suggests that the agent is improving over time and is competitive with the best meta-learner in hindsight (since Regret T /T → 0 as T → ∞). In the batch setting, meta-learning has been observed to perform better than jointly training a single model to work on all the tasks (; ; ; ;). Thus, we may hope that learning sequentially, but still being competitive with the best meta-learner in hindsight, provides a significant leap in continual learning. Our algorithmic approach, follow the meta leader (FTML), takes inspiration from follow the leader (FTL) and adapts it to the online meta learning setting. FTML chooses the parameters according to: DISPLAYFORM0 This can be interpreted as the agent playing the best meta-learner in hindsight if the learning process were to stop at round t. In practice, we may not have full access to f k (·), such as when it is the population risk and we only have a finite size dataset. In such cases, we will draw upon stochastic approximation algorithms to solve the optimization problem in Eq..We concentrate on the case where the update procedure is 1 step of stochastic gradient descent, as in the case of MAML, i.e. U t (w) = w − α∇f i (w). We assume that each loss function, {f t,f t} ∀t, is C 2 −smooth (i.e. G−Lipschitz, β−smooth, and ρ−Lipschitz Hessian) and µ−strongly convex. See Appendix B for more details and implications of these assumptions, and connections to standard online learning setting. Importantly, these assumptions do not trivialize the meta-learning setting. There is a clear separation between meta-learning and joint training even for linear regression (simplest strongly convex problem). See Appendix E for an example illustration. Theorem 1. Suppose f andf: R d → R satisfy the stated assumptions. Letf be the function evaluated after the gradient update procedure, i.e.f (w):= f w − α∇f (w). If the step size is selected as α ≤ min{1 2β, µ 8ρG}, thenf isβ = 9β/8 smooth andμ = µ/8 strongly convex. Since the objective function is convex, we may expect first-order optimization methods to be effective, since gradients can be efficiently computed with standard automatic differentiation libraries (as discussed in). In fact, this work provides the first set of (under any assumptions) under which MAML-like objective function can be provably and efficiently optimized. An immediate corollary of our main theorem is that FTML now enjoys the same regret guarantees (up to constant factors) as FTL does in the comparable setting (with strongly convex losses). Corollary 1. (inherited regret bound for FTML) Suppose that for all t, f t andf t satisfy assumptions 1 and 2. Suppose that the update procedure in FTML (Eq. 2) is chosen as U t (w) = w − α∇f t (w) with α ≤ min{1 2β, µ 8ρG}. Then, FTML enjoys the following regret guarantee DISPLAYFORM1 More generally, our main theorem implies that there exists a large family of online meta-learning algorithms that enjoy sub-linear regret, based on the inherited smoothness and strong convexity of f (· DISPLAYFORM2 Here, ν t (·) denotes a sampling distribution for the previously seen tasks (we use a uniform distribution in our experiments). L(D, w) is the loss function (e.g. cross-entropy) averaged over the datapoints (x, y) ∈ D for the model with parameters w. While U t in Eq. includes only one gradient step, we observed that it is beneficial to take multiple gradient steps in the inner loop (i.e., in U t), which is consistent with prior works (; ; BID3 .The overall algorithmic procedure proceeds as follows. We first initialize a task buffer B = []. When presented with a new task at round t, we add task T t to B and initialize a task-specific dataset D t = [], which is appended to as data incrementally arrives for task T t. As new data arrives for task T t, we iteratively compute and apply the gradient in Eq., which uses data from all tasks seen so far. Once all of the data (finite-size) has arrived for T t, we move on to task T t+1. Our experimental evaluation studies the practical FTML algorithm (Section 2.1) in the context of vision-based online learning problems. These problems include synthetic modifications of the MNIST dataset, pose detection with synthetic images based on PASCAL3D+ models , and realistic online image classification experiments with the CIFAR-100 dataset. The aim of our experimental evaluation is to study the following questions: can online meta-learning (and specifically FTML) be successfully applied to multiple non-stationary learning problems? and does online meta-learning (FTML) provide empirical benefits over prior methods?To this end, we compare to the following algorithms: (a) Train on everything (TOE) trains on all available data so far (including D t at round t) and trains a single predictive model. This model is directly tested without any specific adaptation since it has already been trained on D t. (b) Train from scratch, which initializes w t randomly, and finetunes it using D t. (c) Joint training with fine-tuning, which at round t, trains on all the data jointly till round t − 1, and then finetunes it specifically to round t using only D t. This corresponds to the standard online learning approach where FTL is used (without any meta-learning objective), followed by task-specific fine-tuning. We compare the algorithms on two metrics: task performance (e.g. classification accuracy) for each task in the sequence; and learning efficiency or amount of data needed to reach a proficiency threshold (e.g. 90% classification accuracy). We note that TOE is a very strong point of comparison, capable of reusing representations across tasks, as has been proposed in a number of prior continual learning works (; BID1). However, unlike FTML, TOE does not explicitly learn the structure across tasks. Thus, it may not be able to fully utilize the information present in the data, and will likely not be able to learn new tasks with only a few examples. Further, the model might incur negative transfer if the new task differs substantially from previously seen ones, as has been observed in prior work . FTL with fine-tuning represents a natural online learning comparison, which in principle should combine the best parts of learning from scratch and TOE, since this approach adapts specifically to each task and benefits from prior data. However, in contrast to FTML, this method does not explicitly meta-learn and hence may not fully utilize any structure in the tasks, and may also overfit in the fine-tuning stage. In the rainbow MNIST experiment, we transform the digits in a number of ways to create different tasks, such as 7 different colored s, 2 scales (half size and original size), and 4 rotations of 90 degree intervals. A task involves correctly classifying digits with a randomly sampled , scale, and rotation. As seen in the curves in FIG0, FTML learns tasks more and more quickly with each new task. We also observe that FTML substantially outperforms the baselines in both efficiency and end performance. FTL is better than TOE since it performs task-specific adaptation, but still worse than FTML. We hypothesize that, while TOE and FTL improve in efficiency over the course of learning as they see more tasks, they struggle to prevent negative transfer on each new task. Our last observation is that training independent models does not learn efficiently, compared to models that incorporate data from other tasks; but, their asymptotic performance with a large data size is similar. Our next experiment studies a 3D pose prediction problem. Each task involves learning to predict the global position and orientation of an object in an image. We construct a dataset of synthetic images using 50 object models from 9 different object classes in the PASCAL3D+ dataset , rendering the objects on a table using the renderer accompanying the MuJoCo physics engine . To place an object on the table, we select a random 2D location, as well as a random azimuthal angle. Each task corresponds to a different object with a randomly sampled camera angle. For the loss functions, we use mean-squared error, and set the proficiency threshold to an error of 0.05. We show the of this experiment in FIG0. The demonstrate that meta-learning can significantly improve both efficiency and performance of new tasks over the course of learning, solving many of the tasks with only 10 datapoints. Unlike the previous settings, TOE substantially outperforms the independent task models, indicating that it can effectively make use of the previous data from other tasks (likely due to greater structural similarity in this task). However, the efficiency and performance of online meta-learning demonstrates that even better transfer can be accomplished by explicitly optimizing for the ability to quickly and effectively learn new tasks. Appendix C presents a more detailed evaluation and on the CIFAR task. In this paper, we introduced the online meta-learning problem statement, with the aim of connecting the fields of meta-learning and online learning. Online meta-learning provides, in some sense, a more natural perspective on the ideal real-world learning procedure. An intelligent agent interacting with a constantly changing environment should utilize streaming experience to both master the task at hand, and become more proficient at learning new tasks in the future. We summarize prior work related to our setting in Appendix D. For the online meta-learning setting, we proposed the FTML algorithm and showed that it enjoys logarithmic regret. We then illustrated how FTML can be adapted to a practical algorithm. Our experimental evaluations demonstrated that the proposed practical variant outperforms prior methods. Here, we summarize the foundations of our online meta-learning formulation. In particular, we concentrate on model agnostic meta-learning and online (i.e. regret based) learning. To illustrate the differences in setting and algorithms, we will use the running example of few-shot learning, which we describe below first. We emphasize that online learning, MAML, and the online meta-learning formulations have a broader scope than few-shot supervised learning. We use the few-shot supervised learning example primarily for illustration. In the few-shot supervised learning setting , we are interested in a family of tasks, where each task T is associated with a notional and infinite-size population of input-output pairs. In the few-shot learning, the goal is to learn a task while accessing only a small, finite-size labeled dataset D i:= {x i, y i} corresponding to task T i. If we have a predictive model, h(·; w), with parameters w, the population risk of the model is f i (w):= E (x,y)∼Ti [(x, y, w)], where the expectation is over the task population and (·) is a loss function, such as the square loss or crossentropy between the model prediction and the correct label. For example, in th e case of square loss we have, (x, y, w) = ||y − h(x; w)|| 2. Let L(D i, w) represent the average loss on the dataset D i.Being able to effectively minimize f i (w) is likely hard if we rely only on D i due to the small size of the dataset. However, we are exposed to many such tasks from the family -either in sequence or as a batch, depending on the setting. By being able to draw upon the multiplicity of tasks, we may hope to perform better, as for example demonstrated in the meta-learning literature. A.2 META-LEARNING AND MAML Meta-learning, or learning to learn , aims to effectively bootstrap from a set of tasks to learn faster on a new task. It is assumed that tasks are drawn from a fixed distribution, DISPLAYFORM0 are drawn from this distribution and datasets corresponding to them are made available to the agent. At deployment time, we are faced with a new test task T j ∼ P(T), for which we are again presented with a small labeled dataset D j:= {x j, y j}. Meta-learning algorithms attempt to find a model using the M training tasks, such that when D j is revealed from the test task, the model can be quickly updated to minimize f j (w). MAML does this by learning an initial set of parameters w MAML, such that at meta-test time, performing a few steps of gradient descent from w MAML using D j minimizes f j (·). To get such an initialization, at meta-training time, MAML solves the optimization problem: DISPLAYFORM1 The inner gradient, ∇f i (w), is based on a mini-batch of data from D i. Hence, MAML optimizes for few-shot generalization. Note that the optimization problem is subtle: we have a gradient descent step embedded in the actual objective function. show that gradient-based methods can be used on this optimization objective with existing automatic differentiation libraries. Stochastic optimization techniques are used to solve the optimization problem in Eq. since the population risk is not known directly. At meta-test time, the solution to Eq. 4 is fine-tuned as: w j ← w MAML − α∇f j (w MAML) with the gradient obtained using D j.MAML and other meta-learning algorithms are not directly applicable to sequential settings for two reasons. First, they have two distinct phases: meta-training and meta-testing or deployment. We would like the algorithms to work in a continuous learning fashion. Second, meta-learning methods generally assume that the tasks come from some fixed distribution, whereas we would like methods that work for non-stationary task distributions. In the online learning setting, an agent faces a sequence of loss functions {f t} ∞ t=1, one in each round t. These functions need not be drawn from a fixed distribution, and could even be chosen adversarially over time. The goal for the learner is to sequentially decide on model parameters {w t} ∞ t=1 that perform well on the loss sequence. In particular, the standard objective is to minimize some notion of regret defined as the difference between our learner's loss, T t=1 f t (w t), and the best performance achievable by some family of methods (comparator class). The most standard notion of regret is to compare to the cumulative loss of the best fixed model in hindsight: DISPLAYFORM0 The goal in online learning is to design algorithms such that this regret grows with T as slowly as possible. In particular, an agent (algorithm) whose regret grows sub-linearly in T is non-trivially learning and adapting. One of the simplest algorithms in this setting is follow the leader (FTL) , which updates the parameters as: DISPLAYFORM1 FTL enjoys strong performance guarantees depending on the properties of the loss function, and some variants use additional regularization to improve stability. For the few-shot supervised learning example, FTL would consolidate all the data from the prior stream of tasks into a single large dataset and fit a single model to this dataset. As observed in the meta-learning literature, such a "joint training" approach may not learn effective models. To overcome this issue, we may desire a more adaptive notion of a comparator class, and algorithms that have low regret against such a comparator, as done in the online meta-learning formulation. In this section, outline the assumptions and proofs. We make the following assumptions about each loss function {f t,f t} ∀t in the learning problem. Let θ and φ represent two arbitrary choices of model parameters. Assumption 1. (C 2 -smoothness) 1. (Lipschitz in function value) f has gradients bounded by G, i.e. ||∇f (θ)|| ≤ G ∀ θ. Assumption 2. (Strong convexity) Suppose that f is convex. Furthermore, suppose f is µ−strongly convex, i.e. ||∇f (θ) − ∇f (φ)|| ≥ µ||θ − φ||.These assumptions are largely standard in online learning, in various settings Cesa-Bianchi & Lugosi FORMULA1, except 1.3. Examples where these assumptions hold include logistic regression and L2 regression over a bounded domain. Assumption 1.3 is a statement about the higher order smoothness of functions which is common in non-convex analysis Nesterov & Polyak FORMULA1;. In our setting, it allows us to characterize the landscape of the MAML-like function which has a gradient update step embedded within it. Importantly, these assumptions do not trivialize the meta-learning setting. A clear difference in performance between meta-learning and joint training can be observed even in the case where f (·) are quadratic functions, which correspond to the simplest strongly convex setting. See Appendix E for an example illustration. We analyze the FTML algorithm when the update procedure is a single step of gradient descent, as in the formulation of MAML. Concretely, the update procedure we consider is U t (w) = w − α∇f t (w). We restate our main theorem below for completeness. Theorem. Suppose f andf: R d → R satisfy assumptions 1 and 2. Letf be the function evaluated after a one step gradient update procedure, i.e. DISPLAYFORM0 If the step size is selected as α ≤ min{1 2β, µ 8ρG}, thenf is convex. Furthermore, it is alsoβ = 9β/8 smooth andμ = µ/8 strongly convex. Proof. First, the smoothness and strong convexity of f andf implies µ ≤ ||∇ 2f (θ)|| ≤ β ∀θ. Thus, DISPLAYFORM1 Also recall the earlier notationθ = U (θ) = θ − α∇f (θ). For α < 1/β, we have the following bounds: DISPLAYFORM2 since we have U (θ) − U (φ) = I − α∇ 2f (ψ) (θ − φ) for some ψ that connects θ and φ due to the mean value theorem on ∇f. Using the chain rule and our definitions, DISPLAYFORM3 Taking the norm on both sides, for the specified α, we have: DISPLAYFORM4 Similarly, we obtain the following lower bound DISPLAYFORM5 which completes the proof. The following corollary is now immediate. satisfy assumptions 1 and 2, then the MAML optimization problem, DISPLAYFORM0 } is convex. Furthermore, it is 9β/8-smooth and µ/8-strongly convex. Since the objective function is convex, we may expect first-order optimization methods to be effective, since gradients can be efficiently computed with standard automatic differentiation libraries (as discussed in). In fact, this work provides the first set of (under any assumptions) under which MAML-like objective function can be provably and efficiently optimized. Another immediate corollary of our main theorem is that FTML now enjoys the same regret guarantees (up to constant factors) as FTL does in the comparable setting (with strongly convex losses).Corollary. (inherited regret bound for FTML) Suppose that for all t, f t andf t satisfy assumptions 1 and 2. Suppose that the update procedure in FTML (Eq. 2) is chosen as U t (w) = w − α∇f t (w) with α ≤ min{1 2β, µ 8ρG}. Then, FTML enjoys the following regret guarantee DISPLAYFORM1 Proof. From Theorem 1, we have that each functionf t (w) = f t (U t (w)) isμ = µ/8 strongly convex. The FTML algorithm is identical to FTL on the sequence of loss functions {f t} T t=1, which has a O(performance after 100 datapoints on the current task. Right: The task performance after all 900 datapoints for the current task have been received. Lower is better for all plots. FTML can learn new tasks more and more efficiently as each new task is received, demonstrating effective forward transfer. In this appendix, we present the extended experimental evaluation that is introduced in Section 3. In this experiment, we create a sequence of tasks based on the MNIST character recognition dataset. We transform the digits in a number of ways to create different tasks, such as 7 different colored s, 2 scales (half size and original size), and 4 rotations of 90 degree intervals. As illustrated in FIG1, a task involves correctly classifying digits with a randomly sampled , scale, and rotation. This leads to 56 total tasks. We partitioned the MNIST training dataset into 56 batches of examples, each with 900 images and applied the corresponding task transformation to each batch of images. The ordering of tasks was selected at random and we set 90% classification accuracy as the proficiency threshold. The learning curves in FIG3 show that FTML learns tasks more and more quickly, with each new task added. We also observe that FTML substantially outperforms the alternative approaches in both efficiency and final performance. FTL performance better than TOE since it performs task-specific adaptation, but its performance is still inferior to FTML. We hypothesize that, while the prior methods improve in efficiency over the course of learning as they see more tasks, they struggle to prevent negative transfer on each new task. Our last observation is that training independent models does not learn efficiently, compared to models that incorporate data from other tasks; but, their final performance with 900 data points is similar. In this experiment, we create a sequence of 5-way classification tasks based on the CIFAR-100 dataset, which contains more challenging and realistic RGB images than MNIST. Each classification problem involves a newly-introduced class from the 100 classes in CIFAR-100. Thus, different tasks correspond to different labels spaces. The ordering of tasks is selected at random, and we measure performance using classification accuracy. Since it is less clear what the proficiency threshold should Figure 4: Online CIFAR-100 , evaluating task performance after 50, 250, and 2000 datapoints have been received for a given task. We see that FTML learns each task much more efficiently than models trained from scratch, while both achieve similar asymptotic performance after 2000 datapoints. We also observe that FTML benefits from adapting all layers rather than learning a shared feature space across tasks while adapting only the last layer. On the left, we observe that online meta-learning generally leads to faster learning as more and more tasks are introduced, learning with only 10 datapoints for many of the tasks. In the center and right, we see that meta-learning enables transfer not just for faster learning but also for more effective performance when 60 and 400 datapoints of each task are available. Note that the order of tasks is randomized, hence leading to spikes when more difficult tasks are introduced.be for this task, we evaluate the accuracy on each task after varying numbers of datapoints have been seen. Since these tasks are mutually exclusive (as label space is changing), it makes sense to train the TOE model with a different final layer for each task. An extremely similar approach to this is to use our meta-learning approach but to only allow the final layer parameters to be adapted to each task. Further, such a meta-learning approach is a more direct comparison to our full FTML method, and the comparison can provide insight into whether online meta-learning is simply learning features and performing training on the last layer, or if it is adapting the features to each task. Thus, we compare to this last layer online meta-learning approach instead of TOE with multiple heads. The (see Figure 4) indicate that FTML learns more efficiently than independent models and a model with a shared feature space. The on the right indicate that training from scratch achieves good performance with 2000 datapoints, reaching similar performance to FTML. However, the last layer variant of FTML seems to not have the capacity to reach good performance on all tasks. In our final experiment, we study a 3D pose prediction problem. Each task involves learning to predict the global position and orientation of an object in an image. We construct a dataset of synthetic images using 50 object models from 9 different object classes in the PASCAL3D+ dataset , rendering the objects on a table using the renderer accompanying the MuJoCo physics engine (see FIG1 . To place an object on the table, we select a random 2D location, as well as a random azimuthal angle. Each task corresponds to a different object with a randomly sampled camera angle. We place a red dot on one corner of the table to provide a global reference point for the position. Using this setup, we construct 90 tasks (with an average of about 2 camera viewpoints per object), with 1000 datapoints per task. All models are trained to regress to the global 2D position and the sine and cosine of the azimuthal angle (the angle of rotation along the z-axis). For the loss functions, we use mean-squared error, and set the proficiency threshold to an error of 0.05. We show the of this experiment in FIG4. The demonstrate that meta-learning can improve both efficiency and performance of new tasks over the course of learning, solving many of the tasks with only 10 datapoints. Unlike the previous settings, TOE substantially outperforms training from scratch, indicating that it can effectively make use of the previous data from other tasks, likely due to the greater structural similarity between the pose detection tasks. However, the performance of FTML suggests that even better transfer can be accomplished by explicitly optimizing for the ability to quickly and effectively learn new tasks. Finally, we find that FTL performs comparably or worse than TOE, indicating that task-specific fine-tuning can lead to overfitting when the model is not explicitly trained for the ability to fine-tune effectively. Our work proposes to use meta-learning or learning to learn;; , in the context of online (regret-based) learning. We reviewed the foundations of these approaches in Section A, and we summarize additional related work along different axis. FORMULA0, nearly all prior meta-learning algorithms assume that the meta-training tasks come from a stationary distribution. Furthermore, most prior work has not evaluated versions of meta-learning algorithms when presented with a continuous stream of tasks. Recent work has considered handling non-stationary task distributions in meta-learning using Dirichlet process mixture models over meta-learned parameters. Unlike this prior work, we introduce a simple extension onto the MAML algorithm without mixtures over parameters, and provide theoretical guarantees. Continual learning: Our problem setting is related to (but distinct from) continual, or lifelong learning; In this paper, we sidestep the problem of catastrophic forgetting by maintaining a buffer of all the observed data. In future work, we hope to understand the interplay between limited memory and catastrophic forgetting for variants of the FTML algorithm. Here, we instead focuses on the problem of forward transfer -maximizing the efficiency of learning new tasks within a non-stationary learning setting. Prior works have also considered settings that combine joint training across tasks with task-specific adaptation FORMULA0, we also focus on the setting where there are several tens or hundreds of tasks. This setting is interesting since there is significantly more information that can be transferred from previous tasks and we can employ more sophisticated techniques such as meta-learning for transfer, enabling the agent to move towards few-shot learning after experiencing a large number of tasks. Online learning: Similar to continual learning, online learning deals with a sequential setting with streaming tasks. It is well known in online learning that FTL has good regret guarantees, but is often computationally expensive. Thus, there is a large body of work on developing computationally cheaper algorithms;;;. Again, in this work, we sidestep the computational considerations to first study if the meta-learning analog of FTL can provide performance gains. For this, we derived the FTML algorithm which has low regret when compared to a powerful adaptive comparator class that performs task-specific adaptation. We leave the design of more computationally efficient versions of FTML to future work. To avoid the pitfalls associated with a single best model in hindsight, online learning literature has also studied alternate notions of regret, with the closest settings being dynamic regret and adaptive or tracking regret. In the dynamic regret setting (; ;), the performance of the online learner's model sequence is compared against the sequence of optimal solutions corresponding to each loss function in the sequence. Unfortunately, lower-bounds suggest that the comparator class is too powerful and may not provide for any non-trivial learning in the general case. To overcome these limitations, prior work has placed restrictions on how quickly the loss functions or the comparator model can change (; ;). In contrast, we consider a different notion of adaptive regret, where the learner and comparator both have access to an update procedure. The update procedures allow the comparator to produce different models for different loss functions, thereby serving as a powerful comparator class (in comparison to a fixed model in hindsight). For this setting, we derived sublinear regret algorithms without placing any restrictions on the sequence of loss functions. We believe that this setting captures the spirit and practice of continual lifelong learning, and also leads to promising empirical .

represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce the online meta learning problem setting to better capture the spirit and practice of continual lifelong learning.

represent the tldr for finding the corresponding scientific article:

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the scientific article to find a short TLDR

Question: We investigate the extent to which individual attention heads in pretrained transformer language models, such as BERT and RoBERTa, implicitly capture syntactic dependency relations. We employ two methods—taking the maximum attention weight and computing the maximum spanning tree—to extract implicit dependency relations from the attention weights of each layer/head, and compare them to the ground-truth Universal Dependency (UD) trees. We show that, for some UD relation types, there exist heads that can recover the dependency type significantly better than baselines on parsed English text, suggesting that some self-attention heads act as a proxy for syntactic structure. We also analyze BERT fine-tuned on two datasets—the syntax-oriented CoLA and the semantics-oriented MNLI—to investigate whether fine-tuning affects the patterns of their self-attention, but we do not observe substantial differences in the overall dependency relations extracted using our methods. Our suggest that these models have some specialist attention heads that track individual dependency types, but no generalist head that performs holistic parsing significantly better than a trivial baseline, and that analyzing attention weights directly may not reveal much of the syntactic knowledge that BERT-style models are known to learn. Pretrained Transformer models like OpenAI GPT BID9 and BERT BID1 have shown stellar performance on language understanding tasks. BERT and BERTbased models significantly improve the state-ofthe-art on many tasks such as constituency parsing BID5, question answering BID11, and have attained top positions on the GLUE leaderboard. As BERT becomes a staple component of many NLP models, many researchers have attempted to analyze the linguistic knowledge that BERT has learned by analyzing the BERT model BID3 or training probing classifiers on the contextualized embeddings of BERT BID12.BERT, as a Transformer-based language model, computes the hidden representation at each layer for each token by attending to all the tokens in an input sentence. The attention heads of Transformer have been claimed to capture the syntactic structure of the sentences BID13. Intuitively, for a given token, some specific tokens in the sentence would be more linguistically related to it than the others, and therefore the selfattention mechanism should be expected to allocate more weight to the linguistically related tokens in computing the hidden state of the given token. In this work, we aim to investigate the hypothesis that syntax is implicitly encoded by BERT's self-attention heads. We use two relation extraction methods to extract dependency relations from all the self-attention heads of BERT. We analyze the ing dependency relations to investigate whether the attention heads of BERT implicitly track syntactic dependencies significantly better than chance, and what type of dependency relations BERT learn. We extract the dependency relations from the self-attention heads instead of the contextualized embeddings of BERT. In contrast to probing models, our dependency extraction methods require no further training. Our experiments suggest that the attention heads of BERT encode most dependency relation types with substantially higher accuracy than our baselines-a randomly initialized Transformer and relative positional baselines. Finetuning BERT on the syntax-oriented CoLA does not appear to impact the accuracy of extracted dependency relations. However, when fine-tuned on the semantics-oriented MNLI dataset, there is a slight improvement in accuracy for longer-term clausal relations and a slight loss in accuracy for shorter-term relations. Overall, while BERT models obtain non-trivial accuracy for some dependency types such as nsubj, obj, nmod, aux, and conj, they do not substantially outperform the trivial right-branching trees in terms of undirected unlabeled attachment scores (UUAS). Therefore, although the attention heads of BERT reflect a small number of dependency relation types, it does not reflect the full extent of the significant amount of syntactic knowledge BERT is shown to learn by the previous probing work. There has been substantial work so far on extracting syntactic trees from the attention heads of Transformer-based neural machine translation (NMT) models. BID6 aggregate the attention weights across the self-attention layers and heads to form a single attention weight matrix. Using this matrix, they propose a method to extract constituency and (undirected) dependency trees by recursively splitting and constructing the maximum spanning tree respectively. In contrast, BID10 train Transformer-based machine translation model on different language-pairs and extract the dependency trees using the maximum spanning tree algorithm on the attention weights of the encoder for each layer and head individually. In work concurrent with ours, BID14 focus on finding confident attention heads of the Transformer encoder based on a heuristic of the concentration of attention weights on single tokens. They identify that these heads appear to serve three specific functions: attending to relative positions, syntactic relations, and rare words. Prior work on the analysis of the contextualized embeddings of BERT has shown that BERT learns significant knowledge of syntax BID3. BID12 introduce a probingstyle method for evaluating syntactic knowledge in BERT and show that BERT encodes syntax more than semantics. BID4 train a structural probing model that maps the hidden representations of each token to an innerproduct space that corresponds to syntax tree distance. They show that the learned spaces of strong models such as BERT and ELMo BID7 are better able to reconstruct dependency trees compared to baselines that can encode features for training a parser but aren't capable of parsing themselves. BERT BID1 ) is a Transformer-based masked language model pretrained on BooksCorpus BID21 and English Wikipedia that has attained stellar performance on a variety of downstream NLP tasks. We run our experiments on the pretrained cased and uncased versions of the BERT-large model, which is a Transformer model consisting of 24 self-attention layers with 16 heads each. For a given dataset, we feed each input sentence through BERT and capture the attention weights for each individual head and layer. BID8 report that they achieve performance gains on the GLUE benchmark by supplementing pre-trained BERT with data-rich supervised tasks such as the Multi-Genre Natural Language Inference dataset (MNLI; BID18 . Therefore, we also run experiments on the uncased BERT-large model fine-tuned on the Corpus of Linguistic Acceptability (CoLA; BID16 and MNLI, to investigate the impact of fine-tuning on a syntax-related task (CoLA) and a semantic-related task (MNLI) on the structure of attention weights and ant extracted dependency relations. We refer to these fine-tuned models as CoLA-BERT and MNLI-BERT. As a baseline, we apply the same relation extraction methods to the BERT-large model with randomly initialized weights (which we refer to as random BERT) as the previous work has shown that randomly initialized sentence encoders perform surprisingly well on a suite of NLP tasks BID20 BID17. We aim to test the hypothesize that the attention heads of BERT learn syntactic relations implicitly, and that self-attention between two words encodes information about their dependency relation. We use two methods for extracting relations from the attention weights in BERT. Both methods operate on the weight matrix W ∈ T ×T for a given head at a given layer, where T is the number of tokens in the sequence, and the rows and columns correspond to the attending and attended tokens respectively (such that each row sums to 1). We exclude [CLS] and [SEP] tokens from the attention matrices, which allows us to focus on inter-word attention. Where the tokenization of our parsed corpus does not match the BERT tokenization, we merge the non-matching tokens until they are mutually compatible, and sum the attention weights for the corresponding columns and rows. We then apply either of the two extraction methods to the attention matrix. To handle the subtokens within the merged tokens, we set all subtokens except for the first to depend on the first subtoken. This approach is largely similar to that in BID4. We use the English Parallel Universal Dependencies (PUD) treebank from the CoNLL 2017 shared task BID19 as gold standard for our evaluation. We assign a relation (w i, w j) between word w i and w j if j = argmax W [i] for each row i in attention matrix W. Based on this simple method, we extract relations for all sentences in our evaluation datasets. The relations extracted using this method need not form a valid tree, or even be fully connected. The ing edge directions may or may not match the canonical directions in a tree, so we evaluate the ing arcs as undirected. Maximum Spanning Tree To extract valid dependency trees from the attention weights for a given layer and head, we follow the approach of BID10 and treat the matrix of attention weight tokens as a complete weighted directed graph, with the edges pointing from the output token to each attended token. As in Raganato and Tiedemann, we take the root of the gold dependency tree as the starting node and apply the Chu-Liu-Edmonds algorithm BID0 BID2 to compute the maximum spanning tree. The ing tree is a valid undirected dependency tree. Relative position baselines Many dependency relations tend to occur in specific positions relative to the parent word. For example, nsubj mostly occurs between a verb and the adjacent word before verb. As an example, FIG0 shows the distribution of relative positions for four major UD rela-, we compute the most common positional offset between a parent and child word for a given dependency relation, and formulate a baseline based on that most common relative positional offset. FIG1 and Table 1 describes the accuracy for nsubj, obj, advmod, and amod and the 10 most frequent relation types in the dataset using relations extracted based on the maximum attention weight method. We also include advcl and csubj in Table 1 as it shows the behavior of MNLI-BERT that tends to track longer-term clasual dependencies better than BERT and CoLA-BERT. Additionally, FIG2 shows the accuracy for nsubj, obj, advmod, and amod relations extracted based on the maximum spanning tree algorithm. The pre-trained and fine-tuned BERT models outperform random BERT substantially for all dependency types. They also outperform the relative position baselines for more than 75% of relation types. They outperform all baselines by a large margin for nsubj and obj, but only slightly better for advmod and amod. These suggest that the self-attention weights in trained BERT models implicitly encode certain dependency relations. Moreover, we do not observe very substantial changes in accuracy by fine-tuning on CoLA and MNLI. However, both BERT and CoLA-BERT have similar or slightly better performance than MNLI-BERT, except for clausal dependencies such as advcl (adverbial clause modifier) and csubj (clausal subject) where MNLI-BERT outperforms BERT and CoLA-BERT by more than 5 absolute points in accuracy. This suggests that semantic-oriented fine-tuning task encourages effective long-distance dependencies. FIG3 describes the maximum undirected unlabeled attachment scores (UUAS) across each Table 1: Highest accuracy for the most frequent dependency types (excluding nsubj, obj, advmod, and amod). We include advcl and csubj although they are not among the ten most frequent relation types as MNLI-BERT outperform other models for these dependency types. Bold marks the highest accuracy for each dependency type. Italics marks accuracies that outperform our trivial baselines. layer. BERT, CoLA-BERT, and MNLI-BERT achieve significantly higher UUAS than the random BERT. Although BERT models perform better than the right-branching baseline in most cases, the performance gap is not very large. However, as the performance gap between BERT models and the trivial right-branching baseline is not substantial, we cannot confidently conclude that the attention heads of BERT track syntactic dependencies. Additionally, the performance of BERT and the fine-tuned BERTs are similar, as fine-tuning on CoLA and MNLI does not have a large impact on UUAS. In this work, we investigate whether the attention heads of BERT exhibit the implicit syntax depen- dency by extracting and analyzing the dependency relations from the attention heads of BERT at all layers. We use two simple dependency relation extraction methods that require no additional training, and observe that there are attention heads of BERT that track more than 75% of the dependency types with higher accuracy than our baselines. However, the hypothesis that the attention heads of BERT track the dependency syntax is not well-supported as the linguistically uninformed baselines outperform BERT on nearly 25% of the dependency types. Additionally, BERT's performance in terms of UUAS is only slightly higher than that of the trivial right-branching trees, suggesting that the dependency syntax learned by the attention heads is trivial. Additionally, we observe that fine-tuning on the CoLA and MNLI does not affect the pattern of self-attention, although the fine-tuned models shows different performance from BERT on the GLUE benchmark.

represent the tldr for finding the corresponding scientific article.

Attention weights don't fully expose what BERT knows about syntax.

Represent the short summary for finding the corresponding scientific article (long)

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.