Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We outline new approaches to incorporate ideas from deep learning into wave-based least-squares imaging. The aim, and main contribution of this work, is the combination of handcrafted constraints with deep convolutional neural networks, as a way to harness their remarkable ease of generating natural images. The mathematical basis underlying our method is the expectation-maximization framework, where data are divided in batches and coupled to additional "latent" unknowns. These unknowns are pairs of elements from the original unknown space (but now coupled to a specific data batch) and network inputs. In this setting, the neural network controls the similarity between these additional parameters, acting as a "center" variable. The ing problem amounts to a maximum-likelihood estimation of the network parameters when the augmented data model is marginalized over the latent variables. In least-squares imaging, we are interested in inverting the following inconsistent ill-conditioned linear inverse problem: In this expression, the unknown vector x represents the image, y i, i = 1,..., N the observed data from N source experiments and A i the discretized linearized forward operator for the ith source experiment. Despite being overdetermined, the above least-squares imaging problem is challenging. The linear systems A i are large, expensive to evaluate, and inconsistent because of noise and/or linearization errors. As in many inverse problems, solutions of problem 1 benefit from adding prior information in the form of penalties or preferentially in the form of constraints, yielding with C representing a single or multiple (convex) constraint set(s). This approach offers the flexibility to include multiple handcrafted constraints. Several key issues remain, namely; (i) we can not afford to work with all N experiments when computing gradients for the above data-misfit objective; (ii) constrained optimization problems converge slowly; (iii) handcrafted priors may not capture complexities of natural images; (iv) it is non-trivial to obtain uncertainty quantification information. To meet the computational challenges that come with solving problem 2 for non-differentiable structure promoting constraints, such as the 1 -norm, we solve problem 2 with Bregman iterations for a batch size of one. The kth iteration reads with A k the adjoint of A k, where, and being the projection onto the (convex) set and 2 the dynamic steplength. Contrary to the Iterative Shrinkage Thresholding Algorithm (ISTA), we iterate on the dual variablex. Moreover, to handle more general situations and to ensure we are for every iteration feasible (= in the constraint set) we replace sparsity-promoting thresholding with projections that ensure that each model iterate remains in the constraint set. As reported in Witte et al., iterations 3 are known to converge fast for pairs {y k, A k} that are randomly drawn, with replacement, from iteration to iteration. As such, Equation 3 can be interpreted as stochastic gradient descent on the dual variable. Handcrafted priors, encoded in the constraint set C, in combination with stochastic optimization, where we randomly draw a different source experiment for each iteration of Equation 3, allow us to create high-fidelity images by only working with random subsets of the data. While encouraging, this approach relies on handcrafted priors encoded in the constraint set C. Motivated by recent successes in machine learning and deep convolutional networks (CNNs) in particular, we follow Van Veen et al., Dittmer et al. and Wu and McMechan and propose to incorporate CNNs as deep priors on the model. Compared to handcrafted priors, deep priors defined by CNNs are less biased since they only require the model to be in the range of the CNN, which includes natural images and excludes images with unnatural noise. In its most basic form, this involves solving problems of the following type: In this expression, g(z, w) is a deep CNN parameterized by unknown weights w and z ∼ N is a fixed random vector in the latent space. In this formulation, we replaced the unknown model by a neural net. This makes this formulation suitable for situations where we do not have access to data-image training pairs but where we are looking for natural images that are in the range of the CNN. In recent work by Van Veen et al., it is shown that solving problem 5 can lead to good estimates for x via the CNN g(z, w) where w is the minimizer of problem 5 highly suitable for situations where we only have access to data. In this approach, the parameterization of the network by w for a fixed z plays the role of a non-linear redundant transform. While using neural nets as strong constraints may offer certain advantages, there are no guarantees that the model iterates remain physically feasible, which is a prerequisite if we want to solve non-linear imaging problems that include physical parameters. Unless we pre-train the network, early iterations while solving problem 5 will be unfeasible. Moreover, as mentioned by Van Veen et al., from solving inverse problems with deep priors may benefit from additional types of regularization. We accomplish this by combining hard handcrafted constraints with a weak constraint for the deep prior ing in a reformulation of the problem 5 into In this expression, the deep prior appears as a penalty term weighted by the trade-off parameter λ > 0. In this weak formulation, x is a slack variable, which by virtue of the hard constraint will be feasible throughout the iterations. The above formulation offers flexibility to impose constraints on the model that can be relaxed during the iterations as the network is gradually "trained". We can do this by either relaxing the constraint set (eg. by increasing the size of the TV-norm ball) or by increasing the trade-off parameter λ. So far, we used the neural network to regularize inverse problems deterministically by selecting a single latent variable z and optimizing over the network weights initialized by white noise. While this approach may remove bias related to handcrafted priors, it does not fully exploit documented capabilities of generative neural nets, which are capable of generating realizations from a learned distribution. Herein lies both an opportunity and a challenge when inverse problems are concerned where the objects of interest are generally not known a priori. Basically, this leaves us with two options. Either we assume to have access to an oracle, which in reality means that we have a training set of images obtained from some (expensive) often unknown imaging procedure, or we make necessary assumptions on the statistics of real images. In both cases, the learned priors and inferred posteriors will be biased by our (limited) understanding of the inversion process, including its regularization, or by our (limited) understanding of statistical properties of the unknown e.g. geostatistics. The latter may lead to perhaps unreasonable simplifications of the geology while the former may suffer from remnant imprint of the nullspace of the forward operator and/or poor choices for the handcrafted and deep priors. Contrary to approaches that have appeared in the literature, where the authors assume to have access to a geological oracle to train a GAN as a prior, we opt to learn the posterior through inversion deriving from the above combination of hard handcrafted constraints and weak deep priors with the purpose to train a network to generate realizations from the posterior. Our approach is motivated by Han et al. who use the Expectation Maximization (EM) technique to train a generative model on sample images. We propose to do the same but now for seismic data collected from one and the same Earth model. To arrive at this formulation, we consider each of the N source experiments with data y k as separate datasets from which images x k can in principle be inverted. In other words, contrary to problem 1, we make no assumptions that the y k come from one and the same x but rather we consider n N different batches each with their own x k. Using the these y k, we solve an unsupervised training problem during which • n minibatches of observed data, latent, and slack variables are paired into tuples with the latent variables z i's initialized as zero-centered white Gaussian noise, z i ∼ N (0, I). The slack variables x i's are computed by the numerically expensive Bregman iterations, which during each iteration work on the randomized source experiment of each minibatch. • latent variables z i's are sampled from p(z i |x i, w) by running l iterations of Stochastic Gradient Langevin Dynamics (SGLD, Welling and Teh ) (Equation 7), where w is the current estimate of network weights, and x i's are computed with Bregman iterations (Equation 8). These iterations for the latent variables are warm-started while keeping the network weights w fixed. This corresponds to an unsupervised inference step where training are created. Uncertainly in the z i's is accounted for by SGLD iterations. • the network weights are updated using {x i, z i} n i=1 with a supervised learning procedure. During this learning step, the network weights are updated by sample averaging the gradients w.r.t. w for all z i' s. As stated by Han et al., we actually compute a Monte Carlo average from these samples. By following this semi-supervised learning procedure, we expose the generative model to uncertainties in the latent variables by drawing samples from the posterior via Langevin dynamics that involve the following iterations for the pairs with ε the steplength. Compared to ordinary gradient descent, 7 contains an additional noise term that under certain conditions allows us to sample from the posterior distribution, p(z i |x i, w). The training samples x i came from the following Bregman iterations in the outer loop After sampling the latent variables, we update the network weights via for the z i's fixed with η steplength for network weights. Conceptually, the above training procedure corresponds to carrying out n different inversions for each data set y i separately. We train the weights of the network as we converge to the different solutions of the Bregman iterations for each dataset. As during Elastic-Averaging Stochastic Gradient Descent [10, Chaudhari et al. ], x i's have room to deviate from each other when λ is not too large. Our approach differs in the sense that we replaced the center variable by a generative network. We numerically conduct a survey where the source experiments contain severe incoherent noise and coherent linearization errors:, where A k = ∇F k is the Jacobian and F k (m) is the nonlinear forward operator with m the known smooth model and δm the unknown perturbation (image). The signal-to-noise ratio of the observed data is −11.37 dB. The of this experiment are included in Figure 1 from which we make the following observations. First, as expected the models generated from g(z, w) are smoother than the primal Bregman variable. Second, there are clearly variations amongst the different g(z, w)'s and these variations average out in the mean, which has fewer imaging artifacts. Because we were able to train the g(z, w) as a "byproduct" of the inversion, we are able to compute statistical information from the trained generative model that may give us information about the "uncertainty". In Figure 2, we included a plot of the pointwise standard deviation, computed with 3200 random realizations of g(z, w), z ∼ p z (z), and two examples of sample "prior" (before training) and "posterior" distribution. As expected, the pointwise standard deviations shows a reasonable sharpening of the probabilities before and after training through inversion. We also argue that the areas of high pointwise standard deviation coincide with regions that are difficult to image because of the linearization error and noise. In this work, we tested an inverse problem framework which includes hard constraints and deep priors. Hard constraints are necessary in many problems, such as seismic imaging, where the unknowns must belong to a feasible set in order to ensure the numerical stability of the forward problem. Deep priors, enforced through adherence to the range of a neural network, provide an additional, implicit type of regularization, as demonstrated by recent work [2, Dittmer et al. ], and corroborated by our numerical . The ing algorithm can be mathematically interpreted in light of expectation maximization methods. Furthermore, connections to elastic averaging SGD highlight potential computational benefits of a parallel (synchronous or asynchronous) implementation. On a speculative note, we argue that the presented method, which combines stochastic optimization on the dual variable with on-the-fly estimation of the generative model's weights using Langevin dynamics, reaps information on the "posterior" distribution leveraging multiplicity in the data and the fact that the data is acquired over one and the same Earth model. Our preliminary seem consistent with a behavior to be expected from a "posterior" distribution. b,c) sample "prior" (before training) and "posterior" distribution functions for two points in the model.

represent the short summary for finding the corresponding scientific article (long)

We combine hard handcrafted constraints with a deep prior weak constraint to perform seismic imaging and reap information on the "posterior" distribution leveraging multiplicity in the data.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent the text to find a very short summary (1-2 sentences max):

Question: When translating natural language questions into SQL queries to answer questions from a database, contemporary semantic parsing models struggle to generalize to unseen database schemas. The generalization challenge lies in (a) encoding the database relations in an accessible way for the semantic parser, and (b) modeling alignment between database columns and their mentions in a given query. We present a unified framework, based on the relation-aware self-attention mechanism,to address schema encoding, schema linking, and feature representation within a text-to-SQL encoder. On the challenging Spider dataset this framework boosts the exact match accuracy to 53.7%, compared to 47.4% for the previous state-of-the-art model unaugmented with BERT embeddings. In addition, we observe qualitative improvements in the model’s understanding of schema linking and alignment. The ability to effectively query databases with natural language has the potential to unlock the power of large datasets to the vast majority of users who are not proficient in query languages. As such, a large body of research has focused on the task of translating natural language questions into queries that existing database software can execute. The release of large annotated datasets containing questions and the corresponding database SQL queries has catalyzed progress in the field, by enabling the training of supervised learning models for the task. In contrast to prior semantic parsing datasets , new tasks such as WikiSQL and Spider (b) pose the real-life challenge of generalization to unseen database schemas. Every query is conditioned on a multi-table database schema, and the databases do not overlap between the train and test sets. Schema generalization is challenging for three interconnected reasons. First, any text-to-SQL semantic parsing model must encode a given schema into column and table representations suitable for decoding a SQL query that might involve any of the given columns or tables. Second, these representations should encode all the information about the schema, including its column types, foreign key relations, and primary keys used for database joins. Finally, the model must recognize natural language used to refer to database columns and tables, which might differ from the referential language seen in training. The latter challenge is known as schema linking -aligning column/table references in the question to the corresponding schema columns/tables. While the question of schema encoding has been studied in recent literature (b), schema linking has been relatively less explored. Consider the example in Figure 1. It illustrates the challenge of ambiguity in linking: while "model" in the question refers to car_names.model rather than model_list.model, "cars" actually refers to both cars_data and car_names (but not car_makers) for the purpose of table joining. To resolve the column/table references properly, the semantic parser must take into account both the known schema relations (e.g. foreign keys) and the question context. Prior work (b) addressed the schema representation problem by encoding the directed graph of foreign key relations among the columns with a graph neural network. While effective, this approach has two important shortcomings. First, it does not contextualize schema encoding with the question, thus making it difficult for the model to reason about schema linking after both the column representations and question word representations have been built. Second, it limits information propagation during schema encoding to predefined relations in the schema such as foreign keys. The advent of self-attentional mechanisms in natural language processing shows that global reasoning is crucial to building effective representations of relational structures. However, we would like any global reasoning to also take into account the aforementioned predefined schema relations. In this work, we present a unified framework, called RAT-SQL, 1 for encoding relational structure in the database schema and a given question. It uses relation-aware self-attention to combine global reasoning over the schema entities and question words with structured reasoning over predefined schema relations. We then apply RAT-SQL to the problems of schema encoding and schema linking. As a , we obtain 53.7% exact match accuracy on the Spider test set. At the time of writing, this is the state of the art among models unaugmented with pretrained BERT embeddings. In addition, we experimentally demonstrate that RAT-SQL enables the model to build more accurate internal representations of the question's true alignment with schema columns and tables. Semantic parsing of natural language to SQL queries recently surged in popularity thanks to the creation of two new multi-table datasets with the challenge of schema generalization -WikiSQL and Spider (b). Schema encoding is not as challenging in WikiSQL as in Spider thanks to the lack of multi-table relations. Schema linking is relevant for both tasks but also more challenging in Spider due to the richer natural language expressiveness and less restricted SQL grammar observed in it. Indeed, the state of the art semantic parser on WikiSQL achieves a test set accuracy of 91.8%, significantly higher than the state of the art on Spider. The recent state-of-the-art models evaluated on Spider use various attentional architectures for question/schema encoding and AST-based structural architectures for query decoding. IRNet encodes the question and schema separately with LSTM and self-attention respectively, augmenting them with custom type vectors for schema linking. They further use the AST-based decoder of to decode a query in an intermediate representation (IR) that exhibits higher-level abstraction structure than SQL. Bogin et al. (2019b) encode the schema with a graph neural network and a similar grammar-based decoder. Both approaches highlight the importance of schema encoding and schema linking, but design separate feature engineering techniques to augment word vectors (as opposed to relations between words and columns) to resolve it. In contrast, the relational framework of RAT-SQL provides a unified way to encode arbitrary relational information among the inputs. Concurrently with this work, Bogin et al. (2019a) published Global-GNN, a different approach to schema linking for Spider which applies global reasoning between question words and schema columns/tables. Global reasoning is implemented by gating the graph neural network that computes the representation of schema elements using question token representations. This conceptually differs from RAT-SQL in two important ways: (a) question word representations influence the schema representations but not vice versa, and (b) like in other GNN-based encoding approaches, message propagation is limited to the schema-induced edges such as foreign key relations. In contrast, our Table 1 to reduce clutter. relation-aware transformer mechanism allows encoding arbitrary relations between question words and schema elements explicitly, and these representations are computed jointly using self-attention. We use the same formulation of relation-aware self-attention as. However, that work only applied it to sequences of words in the context of machine translation, and as such, their set of relation types only encoded the relative distance between two words. We extend their work and show that relation-aware self-attention can effectively encode more complex relationships that exist within an unordered sets of elements (in this case, columns and tables within a database schema as well as relations between the schema and the question). To the best of our knowledge, this is the first application of relation-aware self-attention to joint representation learning with both predefined and softly induced relations in the input structure. We now describe the RAT-SQL framework and its application to the problems of schema encoding and linking. First, we formally define the text-to-SQL semantic parsing problem and its components. Then, we introduce the relation-aware self-attention mechanism, our framework for jointly encoding relational structure between the question and the schema. Finally, we present our implementation of schema linking in the RAT-SQL framework. Given a natural language question Q and a schema S = C, T for a relational database, our goal is to generate the corresponding SQL P. Here the question Q = q 1... q |Q| is a sequence of words, and the schema consists of columns C = {c 1, . . ., c |C|} and tables T = t 1,..., t |T |. Each column name c i contains words c i,1,..., c i,|ci| and each table name t i contains words t i,1,..., t i,|ti|. The desired program P is represented as an abstract syntax tree T in the context-free grammar of SQL. Some columns in the schema are primary keys, used for uniquely indexing the corresponding table, and some are foreign keys, used to reference a primary key column in a different table. As described in Section 1, we would like to softly bias our schema encoding mechanism toward these predefined relations. In addition, each column has a type τ such as number or text. Schema linking aims at finding the alignment between question words and mentioned columns or tables. It's a crucial step for a parser to generate the right columns and tables in SQL. We model the latent alignment explicitly using an alignment matrix (Section 3.6), which is softly biased towards some string-match based relations, as inspired by. To support reasoning about relationships between schema elements in the encoder, we begin by representing the database schema using a directed graph G, where each node and edge has a label. We represent each table and column in the schema as a node in this graph, labeled with the words in the name; for columns, we prepend the type of the column to the label. For each pair of nodes x and y in the graph, Table 1 describes when there exists an edge from x to y and the label it should have. Figure 2 illustrates an example graph (although not all edges and labels are shown). Tree-structured decoder Self-attention layers (c) The decoder, choosing a column (Section 3.7) Figure 3: Overview of the stages of our approach. We now obtain an initial representation for each of the nodes in the graph, as well as for the words in the input question. For the graph nodes, we use a bidirectional LSTM (BiLSTM) over the words contained in the label. We concatenate the output of the initial and final time steps of this LSTM to form the embedding for the node. For the question, we also use a bidirectional LSTM over the words: where each of the BiLSTM functions first lookup word embeddings for each of the input tokens. The LSTMs do not share any parameters. At this point, we have representations c init i, t init i, and q init i. Similar to encoders used in some previous papers, these initial representations are independent of each other (uninfluenced by which other columns or tables are present). Now, we would like to imbue these representations with the information in the schema graph. We use a form of self-attention that is relation-aware to achieve this goal. In one step of relation-aware self-attention, we begin with an input x of n elements (where x i ∈ R dx) and transform each x i into y i ∈ R dx. We follow the formulation described in: where FC is a fully-connected layer, 1 ≤ h ≤ H, and W The r ij terms encode the relationship between the two elements x i and x j in the input. We explain how we obtain r ij in the next part. Application Within Our Encoder At the start, we construct the input x of |c| + |t| + |q| elements using c init i, t init i, and q init i: We then apply a stack of N relation-aware self-attention layers, where N is a hyperparameter. The weights of the encoder layers are not tied; each layer has its own set of weights. After processing through the stack of N encoder layers, we obtain Description of edge types present in the directed graph created to represent the schema. An edge exists from source node x ∈ S to target node y ∈ S if the pair fulfills one of the descriptions listed in the table, with the corresponding label. Otherwise, no edge exists from x to y. Column Column SAME-TABLE x and y belong to the same table. FOREIGN-KEY-COL-F x is a foreign key for y. FOREIGN-KEY-COL-R y is a foreign key for x. Table PRIMARY-KEY-F x is the primary key of y. x is a column of y (but not the primary key). Table Column PRIMARY-KEY-R y is the primary key of x. BELONGS-TO-R y is a column of x (but not the primary key). Table Table FOREIGN -KEY-TAB-F Table x has a foreign key column in y. FOREIGN-KEY-TAB-R Same as above, but x and y are reversed. FOREIGN-KEY-TAB-B x and y have foreign keys in both directions. We use c We define a discrete set of possible relation types, and map each type to an embedding to obtain r V ij and r K ij. We need a value of r ij for every pair of elements in x. In the subsequent sections, we describe the set of relation types we used. If x i and x j both correspond to nodes in G (i.e. each is either a column or table) with an edge from x i to x j, then we use the label on that edge (possibilities listed in Table 1) for r ij. However, this is not sufficient to obtain r ij for every pair of i and j. The graph G has no nodes corresponding to the question words, not every pair of schema nodes has an edge between them, and there is no self-edges (for when i = j). As such, we add more types beyond what is defined in Table 1: • If i = j, then COLUMN-IDENTITY or TABLE-IDENTITY. • • x i ∈ question, x j ∈ column ∪ table; or x i ∈ column ∪ table, x j ∈ question: see Section 3.6. • Otherwise, one of COLUMN-COLUMN, COLUMN -TABLE, TABLE-COLUMN, or TABLE-TABLE. 3.6 SCHEMA LINKING To aid the model with aligning column/table references in the question to the corresponding schema columns/tables, we furthermore define relation types which indicate when parts of the question textually match the names of the columns and tables. Specifically, for all n-grams of length 1 to 5 in the question, we determine whether it exactly matches the name of a column/table (exact match); or whether the n-gram is a subsequence of the name of a column/table (partial match). Therefore, for the case where x i ∈ question, x j ∈ column ∪ table; or x i ∈ column ∪ table, x j ∈ question, we set r ij to QUESTION-COLUMN-M, QUESTION-TABLE-M, COLUMN-QUESTION-M or TABLE-QUESTION-M depending on the type of x i and x j. M is one of EXACTMATCH, PARTIALMATCH, or NOMATCH. In the end, we add 2 + 5 + (4 × 3) + 4 types (one term per bullet in Section 3.5) beyond the 10 in Table 1, for a total of 33 types. Memory-Schema Alignment Matrix Our intuition suggests that the columns and tables which occur in the SQL P will generally have a corresponding reference in the natural language question (for example, "cars" and "cylinders" in Figure 1). To capture this intuition in the model, we apply relation-aware attention as a pointer mechanism between every memory element in y and all the columns/tables to compute explicit alignment matrices L col ∈ R |y|×|C| and L tab ∈ R |y|×|T |: The memory-schema alignment matrix is expected to resemble the real discrete alignments, therefore should respect certain constraints like sparsity. For example, the question word "model" in Figure 1 should be aligned with car_names.model rather than model_list.model or model_-list.model_id. To further bias the soft alignment towards the real discrete structures, we add an auxiliary loss to encourage sparsity of the alignment matrix. Specifically, for a column/table that is mentioned in the SQL query, we treat the model's current belief of the best alignment as the ground truth. Then we use a cross-entropy loss, referred as alignment loss, to strengthen the model's belief: where Rel(C) and Rel(T) denote the set of relevant columns and tables that appear in the SQL P. Once we have obtained an encoding of the input, we used the decoder from to generate the SQL P. The decoder generates P as an abstract syntax tree in depth-first traversal order, by using an LSTM to output a sequence of decoder actions that (i) expand the last generated node in the tree according to the grammar, called APPLYRULE; or when necessary to complete the last node, (ii) chooses a column or table from the schema, called SELECTCOLUMN and SELECTTABLE. Formally, we have the following: where y is the final encoding of the question and schema from the previous section, and a <t are all previous actions. We update the LSTM's state in the following way: m t, h t = f LSTM ([a t−1 z t h pt a pt n ft], m t−1, h t−1 ) where m t is the LSTM cell state, h t is the LSTM output at step t, a t−1 is the embedding of the previous action, p t is the step corresponding to expanding the parent AST node of the current node, and n ft is the embedding of the current node type. We obtain z t using multi-head attention (with 8 heads) on h t−1 over y. is a 2-layer MLP with a tanh non-linearity. For SELECTCOLUMN, we computẽ and similarly for SELECTTABLE. We implemented our model using PyTorch . During preprocessing, the input of questions, column names and table names are tokenized and lemmatized with the StandfordNLP toolkit. Within the encoder, we use GloVe word embeddings, held fixed in training except for the 50 most common words in the training set. All word embeddings have dimension 300. The bidirectional LSTMs have hidden size 128 per direction, and use the recurrent dropout method of with rate 0.2. We stack 8 relation-aware self-attention layers on top of the bidirectional LSTMs. Within the relation-aware self-attention layers, we set d x = d z = 256, H = 8, and use dropout with rate 0.1. The position-wise feed-forward network has inner layer dimension 1024. Inside the decoder, we use rule embeddings of size 128, node type embeddings of size 64, and a hidden size of 512 inside the LSTM with dropout rate 0.21. We used the Adam optimizer with β 1 = 0.9, β 2 = 0.999, and = 10 −9, which are defaults in PyTorch. During the first warmup_steps = max_steps/20 steps of training, we linearly increase the learning rate from 0 to 7.4 × 10 −4. Afterwards, the learning rate is annealed to 0, with formula 10 −3 (1 − step−warmup_steps max_steps−warmup_steps) −0.5. For all parameters, we used the default initialization method in PyTorch. We use a batch size of 20 and train for up to 40,000 steps. We use the Spider dataset (b) for all our experiments. As described by Yu et al. (2018b), the training data contains 8,659 examples, including 1,659 examples (questions and queries, with the accompanying schemas) from the Restaurants , GeoQuery , Scholar , Academic , Yelp and IMDB datasets. As Yu et al. (2018b) make the test set accessible only through an evaluation server, we perform most evaluations (other than the final accuracy measurement) using the development set. It contains 1,034 examples, with databases and schemas distinct from those in the training set. We report using the same metrics as Yu et al. (2018a): exact match accuracy on all examples, as well as divided by difficulty levels specified in the dataset. As in previous work, these metrics do not measure the model's performance on generating values within the queries. In Table 2a we show accuracy on the (hidden) test set for RAT-SQL and compare to all other approaches that are at or near state-of-the-art (according to the official dataset leaderboard). RAT-SQL outperforms all other methods that, like RAT-SQL, are not augmented with BERT embeddings. It even comes within 1.3% of beating the best BERT-augmented model. Since the typical improvement achieved by BERT augmentation is about 7% for all models, we are hopeful that adding such augmentation to RAT-SQL will also lead to state-of-the-art performance among BERT models. We also provide a breakdown of the accuracy by difficulty in Table 2b. As expected, performance drops with increasing difficulty. The overall generalization gap between development and test was strongly affected by the significant drop in accuracy (15%) on the extra hard questions. Table 2c shows an ablation study without RAT-based schema linking relations. Schema linking makes a statistically significant improvement to accuracy (p<0.001). The full Figure 4: Alignment between the question "For the cars with 4 cylinders, which model has the largest horsepower" and the database car_1 schema (columns and tables). model accuracy here differs from Table 2a because the latter shows the best single model from a hyper-parameter sweep (submitted for test evaluation) and the former gives the mean over ten runs. Alignment Recall from Section 3 that we explicitly represent the alignment between question words and table columns which is used during decoding for column selection. The existence of the alignment matrix provides a mechanism for the model to align words to columns, but the additional terms in the loss encourage it to actually act like an alignment. In our final model, the alignment loss terms do not make a difference in overall accuracy. This is surprising to us because in earlier development, the alignment loss did improve the model (statistically significantly, from 53.0% to 55.4%). We hypothesize that hyper-parameter tuning that caused us to increase encoding depth also eliminated the need for explicit supervision of alignment. An accurate alignment representation has other benefits as well, such as identifying question words to copy when a constant is needed (not part of the Spider dataset evaluation). In Figure 4 we show the alignment generated by our model on an example from the development set. 3 For the three key words that reference columns ("cylinders", "model", "horsepower"), the alignment matrix correctly identifies their corresponding column (cylinders, model, horsepower) and the table (cars_data) except it mistakenly aligns "model" to cars_data also instead of to car_names. The word "cars" aligns to the primary key of the cars_data table. Despite the abundance of research in semantic parsing of text to SQL, many contemporary models struggle to learn good representations for a given database schema as well as to properly link column/table references in the question. These problems are related: to encode & use columns/tables from the schema, the model must reason about their role in the context of a given question. In this work, we present a unified framework for addressing the schema encoding and linking challenges. Thanks to relation-aware self-attention, it jointly learns schema and question word representations based on their alignment with each other and predefined schema relations. Empirically, the RAT framework allows us to gain significant state of the art improvement on textto-SQL parsing. Qualitatively, it provides a way to combine predefined hard schema relations and inferred soft self-attended relations in the same encoder architecture. We foresee this joint representation learning being beneficial in many learning tasks beyond text-to-SQL, as long as the input has predefined structure. A THE NEED FOR SCHEMA LINKING One natural question is how often does the decoder fail to select the correct column, even with the schema encoding and linking improvements we have made. To answer this, we conducted an oracle experiment (see Table 3). For "oracle sketch", at every grammar nonterminal the decoder is forced to make the correct choice so the final SQL sketch exactly matches that of the correct answer. The rest of the decoding proceeds as if the decoder had made the choice on its own. Similarly, "oracle cols" forces the decoder to output the correct column or table at terminal productions. With both oracles, we see an accuracy of 99.4% which just verifies that our grammar is sufficient to answer nearly every question in the data set. With just "oracle sketch", the accuracy is only 70.9%, which means 73.5% of the questions that RAT-SQL gets wrong and could get right have incorrect column or table selection. Similarly, with just "oracle cols", the accuracy is 67.6%, which means that 82.0% of the questions that RAT-SQL gets wrong have incorrect structure. In other words, most questions have both column and structure wrong, so both problems will continue to be important to work on for the future.

State of the art in complex text-to-SQL parsing by combining hard and soft relational reasoning in schema/question encoding.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the article for finding a single-sentence TLDR

Question: As our experience shows, humans can learn and deploy a myriad of different skills to tackle the situations they encounter daily. Neural networks, in contrast, have a fixed memory capacity that prevents them from learning more than a few sets of skills before starting to forget them. In this work, we make a step to bridge neural networks with human-like learning capabilities. For this, we propose a model with a growing and open-bounded memory capacity that can be accessed based on the model’s current demands. To test this system, we introduce a continual learning task based on language modelling where the model is exposed to multiple languages and domains in sequence, without providing any explicit signal on the type of input it is currently dealing with. The proposed system exhibits improved adaptation skills in that it can recover faster than comparable baselines after a switch in the input language or domain. In a classic cartoon by Gary Larson, a student raises his hand to ask the teacher: "Mr. Osborne, may I be excused? My brain is full." . We laugh at this situation because we know it is absurd. Human brains don't just get full. Instead, they seem to be able to keep in their long-term memory massive amounts of information encoding well-acquired knowledge and skills. Furthermore, the information stored in memory is not necessarily relevant at all times. For instance, a person may have a phone call in French in the morning, then go about her daily errands in German, and later write an email in English. Different linguistic knowledge will be required for each of these situations, and context alone, rather than some explicit signal, will dictate what is needed at each given moment. Vanilla neural network models have been successfully deployed in various applications in the past. However, they rely on fixed sized memories and suffer from the problem known as "catastrophic forgetting" , which refers to the fact that previously acquired information is quickly forgotten as novel skills need to be mastered. Earlier work attempted to correct this problem by looking for available capacity on a fixed-sized network that would allow encoding a new solution without affecting previously learned tasks (; ; Serrà et al., 2018; ; ;). The problem with this approach is that eventually, the system will run out of available capacity. Instead, here we argue for developing models that can grow their internal capacity. While some work has also relied on growing the model to face catastrophic forgetting (; ;), they all rely, to the best of our knowledge, on an explicit signal identifying the task that the system is currently solving. Indeed, most work dealing with catastrophic forgetting has evaluated the models on settings often making unrealistic assumptions. Not only they typically provided the model with an explicit identifier for the task at hand, but also tasks featured unnatural properties, such as scrambled pixels, or categories that were incrementally added, but presented sequentially on blocks once and for all, and never encountered again during training. Only recently, some work has started tackling continual learning in a more realistic task-agnostic way . Yet, there are no standard publicly available datasets that can help the evaluation of continual learning systems on more natural settings. In this paper, we make a two-fold contribution towards task agnostic continual learning. First, we introduce a recurrent neural network that can grow its memory by creating new modules as training progresses. Rather than using all modules simultaneously, or indexing them based on a task identification signal, our model learns to weight their contributions to adapt to the current context. Second, we introduce to the community a multilingual/multidomain language modelling task with switching domains that we hope can fit this bill. We propose two variants of it. The first is a character-based language modelling benchmark with text written in 5 different languages that randomly switch between one another. The second one is a word-based language modelling task, where the text oscillates between 4 different domains. No segmentation signal is given when there is a switch, making the models having to discover it autonomously while they are evaluated for their adaptation skills. Our experimental show that our system can switch between different domains faster than comparable neural networks. Furthermore, our model is very general because it does not make any assumption about the type of underlying neural network architecture and thus, it can easily be adopted for tackling other tasks in conjunction with any other neural network system. Growth in neural networks has been explored with different perspectives. Here, we present a discussion of the possible avenues for developing neural networks with unbounded memory. 1. Growth of layers: Early work on Neural Networks used this method to reduce the amount of computational complexity or for escaping local minima . The goal, back then, was using the smallest possible number of hidden units. Here, instead, we are interested in allowing neural networks to grow for endowing them with larger memory capacity. In this sense, this strategy seems limited because all units remain fully connected at all time, forcing the network to access all memories simultaneously. 2. Growth of architecture: A different type of growth could be the one dictated by a different model that decides the characteristics of the learning system, including how many units to put in it. Neural architecture search and, particularly, neuro-evolution provide good examples of this. Note, however, that this type of growth is different from the main problem that we are dealing with here, in which a model needs to be able to extend itself. 3. Learned, structured growth: Models, like the Stack-RNNs permit the model to create new units, which are placed on a stack data structure, allowing it thus to have a flexible memory to process different problem instances of varying sizes. The model itself learns how many computational resources to use, but so far this has been demonstrated only on toy problems like sequence memorization. Moreover, Stack-RNNs are also unable to quickly recover "memories" from distant past because it would imply cycling through the whole stack. 4. Sparse growth: This is the strategy that we focus on in this paper. The network is grown by blocks or modules. One potential drawback with this strategy is the linear increase of time complexity as the network grows. To prevent this, here we simply limit the maximum number of modules that are kept alive at any given time. Other, more sophisticated, options could employ a Hierarchical Softmax operation over the modules or Winner-Takes-All types of rules , essentially searching for just the right memories to answer the current situation. The proposed Growing Long-Term Memory Network (GLTMN) is composed of modules operating in concert to compute the network's output by means of a weighted combination of their predictions. As such, it belongs to a family of architectures that, depending on whether the predictions are additively or multiplicatively combined, are referred as Mixture-of-Experts or Product-of-Experts . Before combining them, all the module's predictions are weighted by a vector of coefficients that can be produced by another module that is jointly trained. Or system differs in the following ways. First, the modules in our system are subdivided into two main blocks: the short-term memory (STM) and long-term memory (LTM). These two components differ in the following ways. First, while the STM has a fixed number of modules, the LTM grows incrementally, only being limited in size by the hosting computer memory capacity. Second, while predictions are computed as a standard MoE architecture, using both LTM and STM modules, only the latter ones gets trained on incoming experience. Mixture weights that encompass both LTM and STEM are kept as a separate parameter of the system that is continually trained based on recent experience. Modules in the STM are consolidated into LTM whenever a trigger point is reached (such as, after a given number of examples have been processed) choosing the module that has been contributing the most to the output according to its corresponding mixture weight recent history. At this point, the module is removed from STM and frozen into LTM. Similarly, LTM modules can be reinstated back into STM by picking the module with the highest contribution weight and copying back into STM. When a maximum size is reached, the LTM module that was reinstated into STM is removed, thus keeping the overall size of the model constant (see Figure 1 for a general sketch). Learning More formally, our model is composed of a set of modules M = {M 1, . . ., M n}, where M 1,..., M l are the modules in the LTM, and {M l+1, . . ., M n} are the ones in STM. At the beginning of training all modules belong to STM, thus l = 0. The system computes its predictions as follows. When an input x (with target y) is observed, it is fed to all modules M 1...n, obtaining log-linear output vectorsŷ. An additional vector of mixture weights w ∈ R n is used to linearly combine them. The output of the full model y is computed as a linear combination of the individual modules outputs weighted by the parameters w i: Note that since we are combining the model unnormalized predictions before the application of the softmax, we are effectively computing a geometric combination of each individual module's unnormalized probabilities:. Thus, this model can be seen as a Product of Experts. Compared to a Mixture of Experts, this approach does not require to normalize the output of each individual model, thus being much more efficient to compute. The predictions are then used to compute the cross-entropy loss L(ŷ, y), which is then used to backpropagate both into the STM modules and the mixture weights, and then optimized independently through gradient descent. In particular, to swiftly adapt the weights, we repeat the training update for w multiple (k = 100) times, whereas we only do it once for the modules. Note that in order to compute the gradients of the loss with respect to the weights after each update there is no need to recompute each module's output and thus, each step is not expensive to compute. Memory management Every T processed examples, the system consolidates a STM module into LTM. For this, it picks the module that has been most active in recent experience, as measured by the absolute mean weight value over the past examples (we use the last 20 batches). To limit the amount of computational power needed, we restricted the maximum total number of modules to n = 30. When this limit is reached, another module is removed from LTM and reinstated back into STM for further training. We pick the module with the highest mixture weight in absolute value. That is, while STM modules are selected for consolidation based on their past importance so they can be preserved for future use, LTM modules are reinstated based on their present relevance, so they can be further adapted. Note that, despite that, in practice the model has a memory limit, its potential to allocate new memory is unbounded, not unlike modern computer systems where physical memory limits do not affect the way programs are written. In the future, the memory efficiency could be improved by incorporating mechanisms such as distillation to compress the information stored across different modules into a single one. Last, but not least, note that the above-described model does not make any assumption about the type of architecture used. In our following experiments, we evaluate the model using an LSTM architecture, but there is nothing preventing it to be applied to feed-forward, convolutional or other types of networks; or even a mixture thereof. In this work we instantiate the GLTMN for an online language modelling task. For this we adopt double-layered LSTM networks as modules. Each of these modules observe text as small sequences of tokens, as is standard when applying the backpropagation through time algorithm, and has to predict each of the upcoming tokens by acting in concert with all the other modules 1. When the system is not allowed to use the LTM, our model reduces to a Products of Experts (PoE) where the mixture coefficients are given by an independent weight vector parameter. To make this architecture comparable to previously proposed neural mixture models (e.g.), we consider as a baseline a PoE model where the weights are computed by another network. In particular, we used another LSTM network that looks at the current batch and produces the weights. Our experimental setup aims to establish whether our model is able to adapt to a continuous stream of circumstances in which it needs to develop, improve and use a wide range of skills. While most previous work interested in continual learning considered sequences of tasks that were unambiguously identified by a marker given as an extra input to the model, here we are interested in a more realistic setup where only context can dictate which skills are required at a given time. For this, we introduce two lifelong language modelling tasks, where the model is continually exposed to a novel linguistic stream that switches between different languages or domains without any explicit marker signalling the change. More specifically, we propose two language modelling benchmarks: One is wordlevel and multi-domain whereas the other is character-level and multilingual. Both benchmarks feature conflicting learning signals when moving between domains or languages, making the network susceptible to catastrophic forgetting. A good model should be very good at transitioning between languages or domains, while still maintaining good overall performance for (online) language modelling. We are interested in modelling the continual adaptation to incoming non-i.i.d. data, a situation that is closer to the learning experience of any human being. Therefore, the traditional train-test split approach is not adequate here. Instead, we adopt an online learning paradigm. This means that at each time step the model receives an instance x t and makes a predictionŷ t. Then, the true target y t will be observed, with the model incurring in a loss L(ŷ t, y t). After reporting this loss, the model is trained, possibly for more than a single iteration, on the just observed example. The goal is minimizing the cumulative loss English, French, Spanish, German and Czech because they all have similar character sets, while also showing interesting linguistic variability thanks to belonging to three different Indo-European branches: Romance (French and Spanish), Germanic (English and German) and Slavic (Czech). Compared to earlier multilingual corpora , our dataset was carefully constructed to include only linguistically valid character sets, in order to prevent non-linguistic noise from interfering with our experiments. For this, we removed all lines from the input that containing characters appearing less than 100 times on the full orpus. The ing character vocabulary is no bigger than 215 characters. The second dataset is an English multi-domain dataset. For this, we used four different source corpora: news (same as above), europarl , the Toronto Book Corpus and Wikipedia . We kept in the vocabulary the top 25K words for each corpus, which after merging yielded a vocabulary size of 58K words. We then split the corpus in fragments coming from different languages or domains with lengths randomly sampled from a (truncated) exponential distribution. Thanks to the memorylessness property of this distribution, it is virtually impossible to estimate when the next switch is going to happen. For the multilingual dataset, we extracted 1M and 10M-characters-long randomly alternating combinations of 100 sequences, 10 for each language, with lengths sampled from a (truncated) exponential distribution with means λ = 10k and λ = 100k characters, respectively, and a 10M-characters-long one with 1000 sequences with mean length of 10k characters. For the multi-domain dataset we followed the same procedure, extracting 100 alternating sequences with mean lengths of λ = 10k and λ = 20k, for a total of 1M and 2M words. We used a smaller corpus in this last case because to allow for faster experimentation as the models have now to predict over a larger vocabulary, and thus they require more training time. We set the maximum size of the GLTMN to 30 LSTM modules having two layers and 200 hidden units each. We first compared it to the performance of a single equally-sized LSTM network, allowing us to measure the advantage of any mixture model with respect to any given single module. Second, we included as a further baseline another double-layered LSTM with 1300 units which has the same amount of parameters as our fully grown model on the multilingual task. As reference points, we also trained independent LSTMs, one for each domain or language (thus, using for diagnostic purposes a "forbidden" domain supervision signal), enabling us to compare the performance of our model to a situation where there is no forgetting from conflicting learning signals, but also where there is no possibility of transferring learned representations across possibly related domains. Finally, we evaluated a PoE, as described in Section 3.1. This is a model whose mixture coefficients are produced by a simple LSTM module with 10 hidden units. We also experimented with passing the weights through a softmax layer, thus enforcing a convex rather than a linear combination of the modules, but this did not prove useful neither for the GLTMN nor the PoE. We tuned the hyperparameters of all the models on a development set for each corpus. Among the ones that we considered for the GLTMN was the size of the STM choosing between 10, 15, 20 or all 30 modules. In the latter case, the model reduces to a PoE with a weights parametrized by a trainable vector of coefficients. Using 20 modules for the STM proved to be the best-performing strategy. Details of the hyperparamter search for the models are included in Appendix A.1. We are interested in measuring whether the growing model brings in any advantage at recovering information that the network had learned before, while remaining competitive in terms of overall performance. To measure these aspects, we propose the following metrics: Online perplexity This is the general perplexity over the data measured during model training. Note that since the task is framed as an online learning one, the training loss serves as a test measure because no example is seen twice. Post-switch confusion: When a distribution shift is experienced, a neural network that suffers from forgetting typically produces a spike. With this measure we aim at capturing how large was this spike. Let L avg be the average cross-entropy loss of the model between a switch occurring at t = 0 and the following one, and let t avg be the time step at which the model touches this level for the first time. Then, we define confusion as: That is, confusion computes the number of time steps weighted by the relative perplexity increase during which the model remains above the average loss for that sequence. We also complement this measure with plots that illustrate this process. In order to observe the asymptotic behaviour of the models, we restrict our analysis by reporting measures pertaining only to the second half of the data. We report our experimental for both the multilingual task and for the multi-domain data in Table 1. Results disaggregated by domain or language are reported in Appendix A.2. There are several aspects that are worth pointing out in these . First, we can see that the in overall online perplexity are mixed. The PoE with LSTM weights scores the biggest number of wins (two), followed by ours and a plain LSTM, winning one each. In general, we can conclude that there is a slight advantage for the PoE model, which may be due to its higher learning capacity during stable non-switching periods, but the GLTMN performs almost on-par with it. Moreover, when looking at the surprisal measures, we can see that the GLTMN excels at recovering after changes in the distribution. This observation can be graphically confirmed by looking at Figure 2. As it can be seen, the GLTMN recovers much faster than the stand-alone LSTM and the PoE models. It does spike, however, on the first batch, although this is completely expected. Recall that weights of the model are optimized according to each new batch. Thus, before the fist batch of the new data was observed it was not possible for it to adapt. We also note that while on the multilingual data, a LSTM trained independently on each different language (first row in the table) exhibits the lowest perplexity, this does not hold in the multi-domain corpus. This shows that while there is limited room for transferring knowledge in the multilingual case, in consonance with previous . In contrast, the multi-domain setting provides plenty of opportunities for transferring knowledge across each domain, and thus task-agnostic systems can benefit them. The main takeaways are that enhancing the neural network with a modular structure (e.g. PoE, GLTMN) brings an improvement to the general performance, but the LTM component helps the network to recover faster after the task switch. To get some further insights into the workings of the model we analyzed its weight vectors in the multilingual and the multi-domain tasks (λ = 10k) for the last seven exposures to each language or domain. We then averaged all the weight vectors on every batch for each of these seven linguistic sequences. In Figure 3 we show all these vectors sorted by the corresponding type of sequence, while weight indices were sorted by means of a clustering scheme to highlight modules that are working in tandem. Note that any of these modules could be in LTM or STM at any given time. In the multilingual case, we can see that there are a large number of modules allocated both to Czech and German. This may be explained by the fact that these are the two hardest domains as measured by the disaggregated perplexities (see Appendix for details). While there is some transfer of modules between languages, this remains quite limited, consistently with our observation above that models trained independently on each linguistic stream reach the lowest online perplexity. In contrast, for the multi-domain corpus, the clustering is much less clear. While the weights for the book corpus and the europarl domains seem to be mostly anti correlated and quite idiosyncratic, the modules acting on the news and wiki domains seem to be more distributed. This may also be explaining our findings that knowledge transfer helps in this task. Our work is related to the efforts aimed at solving the problem of catastrophic forgetting in neural networks (; ; ;), which have received considerable attention in the machine learning community. These approaches can be categorized into mainly two different branches: Those that keep the neural network size fixed and attempt to correct the learning procedure to avoid forgetting (; ; Serrà et al., 2018; ; ;), and those that allow the system to grow new modules to account for novel tasks (; ; ; d'). Our work is closer to the second stem. Also, close to our approach are mixture of experts systems (; ; ; ; ;), in particular the product of experts approach . Other models with unbounded memory were proposed in the past (; ; ;), although not all of them were studied in the context of continual learning, as we are doing here, and those who were assumed training tasks to be properly identified, as previously noted. Similar to our work are the models enahnced with a memory component, such as: memory networks , stack RNNs and neural turing machines which show that having a structured memory helps with learning longer dependencies and remembering. While our approach has some similarities, the proposed model saves fully connected modules which can save into the memory not only data but also the algorithms learned by the modules. The interaction between recent and remote memory has been extensively studied in the neuroscientific literature . We do not claim any direct connection between our model and how the human brain works, but we borrow the terms associated with consolidation and reinstatement of memories, as they fit quite neatly into our context. Finally, our problem formulation is an instance of neural-network-assisted language modelling and character level language modeling (; ; ;). Some models conceived for language modeling can extend their memories to support fast-changing statistics from the recent past, as in cache models . Also, some other work has extended these models towards the multilingual setting (Östling &). Here, we adapt these problems to a life-long learning setup where different languages can be conceived as different tasks. Differently form cache models, context switching implies retrieving a vast set of skills from a relatively distant past. We believe that developing more flexible forms of artificial intelligence will probably require flexible memory capabilities that can only be delivered by models capable of growth. Here we have proposed a method based on growing full-fledged modules over time. We explored a particular instantiation of this architecture in which modules are grown at a constant rate and consolidated into a long-term memory (LTM). Once the model has reached a maximum size, memories can be still be consolidated into LTM by reinstating LTM modules back into STM (see Figure 1). Furthermore, we introduced to the community two lifelong language modelling tasks. One, characterbased and multilingual, and other, word-based on multiple domains. Our experiments confirm the efficacy of our Growing LTM model, showing that it can learn to adapt much faster than comparable baselines without suffering in terms of its overall performance. The proposed system is very flexible, allowing it to be used with any neural network architecture. While here we have studied it in the lifelong language modeling setting, we believe that the system will also show promising in other domains with similar requirements, such as robotics -where the model can learn to deal with different kinds of terrains-or image recognition -where it can learn different kinds of visual information depending on the contextual requirements . In the future, mechanisms that exploit the structure of the input data for associating it with the relevant sets of models can be explored. Furthermore, we plan to study mechanisms that would allow the model to decide when to grow, rather than keeping a constant schedule. In the long term, the model should be capable of deciding how to structure its long-term memory and whether or not to grow it, as Stack-RNNs do to grow the working memory. Moreover, we are interested in exploring how communication between memories can be enabled through a central routing mechanism, in a similar fashion to the model proposed by. To conclude, in this work we have given a step -and we hope that more will follow-in providing neural networks with flexible memory structures. We expect that further pursuing this goal will pave the way towards developing more general learning systems and, fundamentally, that in the future neural networks will no longer need to be excused from class just because their weights are full.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce a continual learning setup based on language modelling where no explicit task segmentation signal is given and propose a neural network model with growing long term memory to tackle it.

Represent the TLDR for finding the corresponding scientific article

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.