Question: Represent the article for finding a single-sentence TLDR

Question: While counter machines have received little attention in theoretical computer science since the 1960s, they have recently achieved a newfound relevance to the field of natural language processing (NLP). Recent work has suggested that some strong-performing recurrent neural networks utilize their memory as counters. Thus, one potential way to understand the sucess of these networks is to revisit the theory of counter computation. Therefore, we choose to study the abilities of real-time counter machines as formal grammars. We first show that several variants of the counter machine converge to express the same class of formal languages. We also prove that counter languages are closed under complement, union, intersection, and many other common set operations. Next, we show that counter machines cannot evaluate boolean expressions, even though they can weakly validate their syntax. This has implications for the interpretability and evaluation of neural network systems: successfully matching syntactic patterns does not guarantee that a counter-like model accurately represents underlying semantic structures. Finally, we consider the question of whether counter languages are semilinear. This work makes general contributions to the theory of formal languages that are of particular interest for the interpretability of recurrent neural networks. It is often taken for granted that modeling natural language syntax well requires a hierarchically structured grammar formalism. Early work in linguistics established that finite-state models are insufficient for describing the dependencies in natural language data . Instead, a formalism capable of expressing the relations in terms of hierarchical constituents ought to be necessary. Recent advances in deep learning and NLP, however, challenge this long-held belief. Neural network formalisms like the long short-term memory network (LSTM) have been shown to perform well on tasks requiring structure sensitivity , even though it is not obvious that such models have the capacity to represent hierarchical structure. This mismatch raises interesting questions for both linguists and practitioners of NLP. It is unclear what about the LSTM's structure lends itself towards good linguistic representations, and under what conditions these representations might fall short of grasping the structure and meaning of language. Recent work has suggested that the expressive capacity of LSTMs resembles that of counter machines (; ;). studied LSTMs with fully saturated weights (i.e. the activation functions evaluate to their asymptotic values instead of intermediate rational values) and showed that such models can express simplified counter languages. , on the other hand, showed that the general counter languages are an upper bound on the expressive capacity of saturated LSTMs. Thus, there seems to be a strong theoretical connection between LSTMs and the counter automata.;; also all report experimental suggesting that some class of counter languages matches the learnable capacity of LSTMs trained by gradient descent. Taking the counter machine as a simplified formal model of the LSTM, we study the formal properties of counter machines as grammars. We do this with the hope of understanding to what degree counter machines, and LSTMs by extension, have computational properties well-suited for representing the structure of natural language. The contributions of this paper are as follows: • We prove that general counter machines, incremental counter machines, and stateless counter machines have equivalent expressive capacity, whereas simplified counter machines are strictly weaker than the general class. • We demonstrate that counter languages are closed under complement, union, intersection, and many other common operations. • We show that counter machines are incapable of representing the deep syntactic structure or semantics of boolean expressions, even though they can validate whether a boolean expression is well-formed. • We prove that a certain subclass of the counter languages are semilinear, and conjecture that this holds for all counter languages. Informally, we can think of counter automata as finite-state automata that have been augmented by a finite number of integer-valued counters. While processing a string, the machine can update the values of the counters, and the counters can in turn inform the machine's state transitions. Early in theoretical computer science established that a 2-counter machine with unbounded computation time is Turing-complete . However, restricting computation to be realtime (i.e. one iteration of computation per input) severely limits the counter machine's computational capacity . A similar fact holds for recurrent neural networks like LSTMs . We study the capabilities of several types of real-time counter automata. The first counter automaton we introduce is the general counter machine. This machine can manipulate the counters by adding or subtracting from them. The other variants that we will go on to define are special cases of this general machine. For m ∈ Z, we write +m to denote the function λx.x + m. By ×0, we denote the constant zero function λx.0. Given an input string x and a counter machine, we perform computation by processing x one token at a time. For each token, we use u to update the counters and δ to update the state according to the current input token, the current state, and a finite mask of the current counter values. We formalize this in Definition 2.2. As a preliminary remark on notation, we use z(x) to denote the zero check function Given a vector x, we use z(x) to represent this function broadcasted over each element of the vector. Definition 2.2 (Counter machine computation). Let q, c ∈ Q × Z k be a configuration of machine M. Upon reading input x t ∈ Σ, we define the transition q, c → xt δ(x t, q, z(c)), u(x t, q, z(c))(c). Definition 2.3 (Real-time acceptance). For any string x ∈ Σ * with length n, a counter machine accepts x if there exist states q 1,.., q n and counter configurations c 1,.., c n such that Definition 2.4 (Real-time language acceptance). A counter machines accepts a language L if, for each x ∈ Σ *, it accepts x if and only if x ∈ L. We denote the set of languages that are acceptable in real time by a general counter machine as CL. We will use the terms "accept" and "decide" interchangeably, as accepting and deciding a language are equivalent for real-time automata. Now, we can can consider various restrictions of the general counter machine, and the corresponding classes of languages acceptable by such automata. First, we present the simplified counter machine discussed by. The counter update function in the simplified counter machine has two important constraints compared to the general machine. First, it can only be conditioned by the input symbol at each time step. Second, it can only increment or decrement its counters instead of being able to add or subtract arbitrary constants. Definition 2.5 (Simplified counter machine). A counter machine is simplified if u has the form Another variant that we consider is the incremental counter machine. This machine also is constrained to have only increments and decrements on its counters, but the counter update function is allowed to depend on the state and counter value. Definition 2.6 (Incremental counter machine). An counter machine is incremental if u has the form Finally, we define a stateless variant of the counter machine. Removing state from the counter machine is equivalent to allowing it to only have one state q 0. Definition 2.7 (Stateless counter machine). A counter machine is stateless if Our first relating counter classes is to show that the simplified counter languages are a proper subset of the general counter languages. The weakness of the simplified machine is that the update function is conditioned only by the input symbol. Thus, languages like a m b 2m, which require switching counting behavior, cannot be decided correctly. We formalize this in Theorem 3.1. Theorem 3.1 (Weakness of SCL). Let SCL be the set of languages acceptable in real time by a simplified counter machine. Then, SCL ⊂ CL. Proof. Consider the language a m b 2m. This is trivially acceptable by a 1-counter machine that adds 2 for each a and subtracts 1 for each b. On the other hand, we shall show that it cannot be accepted by any simplified machine. Assume by way of contradiction that such a simplified machine M exists. Tracking the ratio between a's and b's requires infinite state. Thus, the counters of M, as opposed to the finite state, must encode whether 2m = l for strings of the form a m b l. Let c be the value of some counter in M. We can decompose c into the update contributed by a's and the the update contributed by b's as follows: Exhausting all the possible functions that c can compute, we get We ignore the first four options for z(c), as they clearly do not relate m to l. The final option checks whether the ratio is 1, not 2. Thus, z(c) cannot distinguish whether 2m = l. Note that this argument breaks down if we allow the counter update to depend on the state. In that case, we can build a machine that has two counters and two states: q 0 adds 1 to the first counter while it reads a, and then decrements the first counter and increments the second counter when it reads b. When the first counter is empty and the second counter is not empty, q 0 transitions to q 1, which decrements the second counter. We accept if and only if both counters are 0 after x n. Unlike the simplified counter machine, the incremental machine has the same linguistic capacity as the general machine. We can simulate each counter on a general machine with a finite amount of overhead. This provides a reduction from general to incremental machines. Theorem 3.2 (Generality of ICL). Let ICL be the set of languages acceptable in real time by an incremental counter machine. Then, ICL = CL. Proof. Let d be the maximum that is ever added or subtracted from a counter c in M. We simulate c in M using a counter c and a value q ∈ Z mod d encoded in finite state. We will implement a "ring-counter" encoding of c such that To simulate a ×0 update on c, we apply ×0 to c, and transition state such that q:= 0. To simulate a +m update on c for some m ∈ Z, we first change state such that q:= (q + m) mod d. Next, we apply the following update to c: We can compute z(c) by checking whether z(c) = 0 and q = 0. Similarly, restricting a counter machine to be stateless does not weaken its expressive capacity. We show how to reduce an arbitrary stateful machine to a stateless machine that has been augmented with additional counters. The key idea here is that we can use the additional counters as a one-hot vector that tracks the state of the original machine. At initialization, q encodes the initial state since q = 0 = ω. Furthermore, we define the invariant that, at any given time, q = ω(i) for some state i. Thus, the additional counters now encode the current state. Let x y denote the concatenation of vectors x and y. We define the acceptance mask in M as An analogous transformation allows us to update the counters inherited from M. The last step is to properly update the new counters q. For each transition δ(x t, q i, b) = q j in M, we update q by adding −ω(i) + ω(j). This ensures that the updated value of q is one-hot since The general counter machine, incremental counter machine, and stateless counter machine all converge to the same linguistic capacity, which we call CL. The simplified counter machine defined by , however, has a linguistic capacity SCL that is strictly weaker than CL. Another way to understand the counter languages is through their closure properties. It turns out that the real-time counter languages are closed under a wide array of common operations, including complement, intersection, union, set difference, and symmetric set difference. The general in Theorem 4.1 implies these closure properties, as well as many others. Theorem 4.1 (General set operation closure). Let P be an m-ary operation over languages. If there exists an m-ary boolean function p such that then CL and SCL are both closed under P. Proof. First, we construct counter machines M 1,.., M m that decide the counter languages L 1,.., L m. We define a new machine M that, on input x, simulates M 1,.., M m in parallel, and accepts if and only if Let Λ be a placeholder for either CL or SCL. Let L 1, L 2 ∈ Λ. By Theorem 4.1, Λ is closed under the following operations: We now study the ability of counter machines to represent the language L m (Definition 5.1). Like natural language, L m has a deep structure recursively composed from hierarchical constituents. Definition 5.1 (L m ;). For any m, let L m be the language generated by: <exp> -> <VALUE> <exp> -> <UNARY> <exp> <exp> -> <BINARY> <exp> <exp>.. <exp> -> <m-ARY> <exp>.. <exp>, shows that, by implementing Algorithm 1, even a 1-counter machines can decide L m in real time. Algorithm 1 uses a counter to keep track of the depth at any given index. If the depth counter reaches −1 at the end of the string, the machine has verified that the string is well-formed. We define the arity of a <VALUE> as 0, and the arity of an <m-ARY> operation as m. Algorithm 1 Deciding L m 1: procedure DECIDE (x) 2: for each x t ∈ x do 4: return c = −1 While Algorithm 1 decides L m, we observe that it is agnostic to the deep structure of the input in that it does not represent the dependencies between tokens. This means that it could not be used to evaluate these expressions, for example. Based on this observation, we prove that no counter machine can evaluate boolean expressions due to the deep structural sensitivity that semantic evaluation (as opposed to syntactic acceptance) requires. We view boolean evaluation as a simpler formal analogy to evaluating the compositional semantics of natural language. To be more formal, consider an instance of L 2 with values {0, 1} and binary operations {∧, ∨}. We assign the following semantics to the terminals: Furthermore, our semantics evaluates each nonterminal by applying the denotation of each syntactic argument to the semantic arguments of the operation. For example, We also define semantics for non-constituent prefixes via function composition. For example, Finally, we define the language B as the set of expressions x where [[x]] = 1 under these semantics. Theorem 5.1 (Weak evaluation). For any k, a real-time k-counter machine cannot decide B. Proof. Assume by way of contradiction that such an evaluation can be performed. We consider an input x that contains a prefix of p operators followed by a suffix of p + 1 values. For the machine to evaluate x correctly, the configuration after x p must encode which boolean function x p specifies. However, a counter machine with k counters only has O(p k) configurations after reading p characters. We show by induction over p that an p-length prefix of operators can encode ≥ 2 p boolean functions. Since the counter machine does not have enough configurations to encode all the possibilities, we reach a contradiction. Base Case With p = 0, we have a null prefix followed by one value that determines [[x] ]. Therefore, we can represent exactly 1 (2 0) function, which is the identity. Inductive Case The expression has a prefix of operators x 1:p+1 followed by values x p+2:2p+3. We decompose the semantics of the full expression to To do this, consider the minimal sequence of values that will satisfy them according to a right-to-left ordering of the sequences. For f ∧, this minimal sequence ends in 1, whereas for f ∨ it must end in a 0. Therefore, f can have 2 unique values for each value of [[x 2:2p+2]]. Thus, a p + 1-length sequence of prefixes can encode ≥ 2 · 2 p = 2 p+1 boolean functions. Theorem 5.1 shows how counter machines cannot represent certain hierarchical dependencies, even when the generated language is within the counter machine's weak expressive capacity. This is analogous to how CFGs can weakly generate Dutch center embedding , even though they cannot assign the correct cross-serial dependencies between subjects and verbs . Semilinearity is a condition that has been proposed as a desired property for any formalism of natural language syntax . Intuitively, semilinearity ensures that the set of string lengths in a language is not unnaturally sparse. Regular, context-free, and a variety of mildly context-sensitive languages are known to be semilinear . The semilinearity of CL is an interesting open question if we aim to understand the abilities of counter machines as grammars. We first define semilinearity over sets of vectors before considering languages. To start, we introduce the notion of a linear set: Definition 6.1 (Linear set). A set S ⊆ N k is linear if there exist W ∈ N k×m and b ∈ N k such that Semilinearity, then, is a weaker condition that specifies that a set is made up of a finite number of linear components: Definition 6.2 (Semilinear set). A set S ⊆ N k is semilinear if it is the finite union of linear sets. To apply this definition to a language L, we translate each sentence x ∈ L into a vector by taking Ψ(x), the Parikh mapping of x. The Parikh mapping of a sentence is, in more familiar machine learning terms, just its bag of tokens representation. For example, the Parikh mapping of abaa with respect to Σ = {a, b} is 3, 1. Definition 6.3 (Semilinear language). A language L is semilinear if {Ψ(x) | x ∈ L} is semilinear. While we do not prove that the general counter languages are semilinear, we do prove it for a dramatically restricted subclass of the counter languages. We defineQSCL as the set of language acceptable by a counter machine that is both simplified (Definition 2.5) and stateless (Definition 2.7), and show that this class is indeed semilinear. Proof. We express L as Since semilinear languages are closed under finite union and intersection, the problem reduces to showing that {x | c i (x) = b i } is semilinear. We apply the following trick: where Z is the set of all tokens that set counter i to 0, and L(b, i) is the set of suffixes after the last occurence of some token in Z, for ever string in L. Since semilinear languages are closed under concatenation, and Σ * and the finite language Z are trivially semilinear, we just need to show that L(b, i) is semilinear. Counter i cannot be set to zero on strings of L(b, i), so we can write where # σ (x) is the number of occurrences of σ in x, and u i denotes the vector of possible updates to counter i where each index corresponds to a different σ ∈ Σ. So, L(b, i) is the linear language Although the proof of Theorem 6.1 is nontrivial, it should be noted thatQSCL is quite a weak class. Such languages have limited ability to even detect the relative order of tokens in a string. We hope this argument might be extended to show that SCL or CL is semilinear. We have shown that many variants of the counter machine converge to express the same class of formal languages, which supports that CL is a robustly defined class. We also proved that real-time counter languages are closed under a large number of common set operations. This provides tools for future work investigating real-time counter automata. We also showed that counter automata are incapable of evaluating boolean expressions, even though they are capable of verifying that boolean expressions are syntactically well-formed. This has a clear parallel in the domain of natural language, where deciding whether a sentence is grammatical is a different task than representing its deep syntactic or semantic structure. A general take-away from our is that just because a counter machine (or LSTM) is sensitive to surface patterns in linguistic data does not mean it can build correct semantic representations. Counter memory can be exploited to weakly match patterns in language, which might provide the wrong kinds of inductive bias for achieving sophisticated natural language understanding. Finally, we asked whether counter languages are semilinear as another way of studying their linguistic capacity. We concluded only that a quite weak subclass of the counter languages are semilinear, and encourage future work to address the general case.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We study the class of formal languages acceptable by real-time counter automata, a model of computation related to some types of recurrent neural networks.

Represent the short summary for finding the corresponding scientific article (long):

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the article for finding a single-sentence TLDR

Question: Attentional, RNN-based encoder-decoder models for abstractive summarization have achieved good performance on short input and output sequences. For longer documents and summaries however these models often include repetitive and incoherent phrases. We introduce a neural network model with a novel intra-attention that attends over the input and continuously generated output separately, and a new training method that combines standard supervised word prediction and reinforcement learning (RL). Models trained only with supervised learning often exhibit "exposure bias" - they assume ground truth is provided at each step during training. However, when standard word prediction is combined with the global sequence prediction training of RL the ing summaries become more readable. We evaluate this model on the CNN/Daily Mail and New York Times datasets. Our model obtains a 41.16 ROUGE-1 score on the CNN/Daily Mail dataset, an improvement over previous state-of-the-art models. Human evaluation also shows that our model produces higher quality summaries. Text summarization is the process of automatically generating natural language summaries from an input document while retaining the important points. By condensing large quantities of information into short, informative summaries, summarization can aid many downstream applications such as creating news digests, search, and report generation. There are two prominent types of summarization algorithms. First, extractive summarization systems form summaries by copying parts of the input BID5 BID22. Second, abstractive summarization systems generate new phrases, possibly rephrasing or using words that were not in the original text BID4.Neural network models based on the attentional encoder-decoder model for machine translation BID0 were able to generate abstractive summaries with high ROUGE scores. However, these systems have typically been used for summarizing short input sequences (one or two sentences) to generate even shorter summaries. For example, the summaries on the DUC-2004 dataset generated by the state-of-the-art system by BID40 are limited to 75 characters. also applied their abstractive summarization model on the CNN/Daily Mail dataset BID8, which contains input sequences of up to 800 tokens and multisentence summaries of up to 100 tokens. But their analysis illustrates a key problem with attentional encoder-decoder models: they often generate unnatural summaries consisting of repeated phrases. We present a new abstractive summarization model that achieves state-of-the-art on the CNN/Daily Mail and similarly good on the New York Times dataset (NYT) BID30. To our knowledge, this is the first end-to-end model for abstractive summarization on the NYT dataset. We introduce a key attention mechanism and a new learning objective to address the FIG2: Illustration of the encoder and decoder attention functions combined. The two context vectors (marked "C") are computed from attending over the encoder hidden states and decoder hidden states. Using these two contexts and the current decoder hidden state ("H"), a new word is generated and added to the output sequence. repeating phrase problem: (i) we use an intra-temporal attention in the encoder that records previous attention weights for each of the input tokens while a sequential intra-attention model in the decoder takes into account which words have already been generated by the decoder. (ii) we propose a new objective function by combining the maximum-likelihood cross-entropy loss used in prior work with rewards from policy gradient reinforcement learning to reduce exposure bias. Our model achieves 41.16 ROUGE-1 on the CNN/Daily Mail dataset. Moreover, we show, through human evaluation of generated outputs, that our model generates more readable summaries compared to other abstractive approaches. In this section, we present our intra-attention model based on the encoder-decoder network BID33. In all our equations, x = {x 1, x 2, . . ., x n} represents the sequence of input (article) tokens, y = {y 1, y 2, . . ., y n} the sequence of output (summary) tokens, and denotes the vector concatenation operator. DISPLAYFORM0 At each decoding step t, we use an intra-temporal attention function to attend over specific parts of the encoded input sequence in addition to the decoder's own hidden state and the previouslygenerated word BID31. This kind of attention prevents the model from attending over the sames parts of the input on different decoding steps. have shown that such an intra-temporal attention can reduce the amount of repetitions when attending over long documents. We define e ti as the attention score of the hidden input state h e i at decoding time step t: DISPLAYFORM0 where f can be any function returning a scalar e ti from the h d t and h e i vectors. While some attention models use functions as simple as the dot-product between the two vectors, we choose to use a bilinear function: DISPLAYFORM1 We normalize the attention weights with the following temporal attention function, penalizing input tokens that have obtained high attention scores in past decoding steps. We define new temporal scores e ti: DISPLAYFORM2 Finally, we compute the normalized attention scores α e ti across the inputs and use these weights to obtain the input context vector c While this intra-temporal attention function ensures that different parts of the encoded input sequence are used, our decoder can still generate repeated phrases based on its own hidden states, especially when generating long sequences. To prevent that, we can incorporate more information about the previously decoded sequence into the decoder. Looking back at previous decoding steps will allow our model to make more structured predictions and avoid repeating the same information, even if that information was generated many steps away. To achieve this, we introduce an intradecoder attention mechanism. This mechanism is not present in existing encoder-decoder models for abstractive summarization. For each decoding step t, our model computes a new decoder context vector c d t. We set c d 1 to a vector of zeros since the generated sequence is empty on the first decoding step. For t > 1, we use the following equations: A closely-related intra-RNN attention function has been introduced by BID3 but their implementation works by modifying the underlying LSTM function, and they do not apply it to long sequence generation problems. This is a major difference with our method, which makes no assumptions about the type of decoder RNN, thus is more simple and widely applicable to other types of recurrent networks. DISPLAYFORM0 To generate a token, our decoder uses either a token-generation softmax layer or a pointer mechanism to copy rare or unseen from the input sequence. We use a switch function that decides at each decoding step whether to use the token generation or the pointer BID7. We define u t as a binary value, equal to 1 if the pointer mechanism is used to output y t, and 0 otherwise. In the following equations, all probabilities are conditioned on y 1,..., y t−1, x, even when not explicitly stated. Our token-generation layer generates the following probability distribution: DISPLAYFORM0 On the other hand, the pointer mechanism uses the temporal attention weights α e ti as the probability distribution to copy the input token x i. DISPLAYFORM1 We also compute the probability of using the copy mechanism for the decoding step t: DISPLAYFORM2 where σ is the sigmoid activation function. Putting Equations 9, 10 and 11 together, we obtain our final probability distribution for the output token y t: p(y t) = p(u t = 1)p(y t |u t = 1) + p(u t = 0)p(y t |u t = 0).The ground-truth value for u t and the corresponding i index of the target input token when u t = 1 are provided at every decoding step during training. We set u t = 1 either when y t is an out-ofvocabulary token or when it is a pre-defined named entity (see Section 5). In addition to using the same embedding matrix W emb for the encoder and the decoder sequences, we introduce some weight-sharing between this embedding matrix and the W out matrix of the tokengeneration layer, similarly to Inan et al. FORMULA1 and BID26. This allows the tokengeneration function to use syntactic and semantic information contained in the embedding matrix. DISPLAYFORM0 2.5 REPETITION AVOIDANCE AT TEST TIME Another way to avoid repetitions comes from our observation that in both the CNN/Daily Mail and NYT datasets, ground-truth summaries almost never contain the same trigram twice. Based on this observation, we force our decoder to never output the same trigram more than once during testing. We do this by setting p(y t) = 0 during beam search, when outputting y t would create a trigram that already exists in the previously decoded sequence of the current beam. In this section, we explore different ways of training our encoder-decoder model. In particular, we propose reinforcement learning-based algorithms and their application to our summarization task. The most widely used method to train a decoder RNN for sequence generation, called the teacher forcing" algorithm BID37, minimizes a maximum-likelihood loss at each decoding step. We define y * = {y * 1, y * 2, . . ., y * n} as the ground-truth output sequence for a given input sequence x. The maximum-likelihood training objective is the minimization of the following loss: DISPLAYFORM0 However, minimizing L ml does not always produce the best on discrete evaluation metrics such as ROUGE BID15. This phenomenon has been observed with similar sequence generation tasks like image captioning with CIDEr BID28 and machine translation with BLEU. There are two main reasons for this discrepancy. The first one, called exposure bias BID27, comes from the fact that the network has knowledge of the ground truth sequence up to the next token during training but does not have such supervision when testing, hence accumulating errors as it predicts the sequence. The second reason is due to the large number of potentially valid summaries, since there are more ways to arrange tokens to produce paraphrases or different sentence orders. The ROUGE metrics take some of this flexibility into account, but the maximum-likelihood objective does not. One way to remedy this is to learn a policy that maximizes a specific discrete metric instead of minimizing the maximum-likelihood loss, which is made possible with reinforcement learning. In our model, we use the self-critical policy gradient training algorithm BID28.For this training algorithm, we produce two separate output sequences at each training iteration: y s, which is obtained by sampling from the p(y s t |y s 1, . . ., y s t−1, x) probability distribution at each decoding time step, andŷ, the baseline output, obtained by maximizing the output probability distribution at each time step, essentially performing a greedy search. We define r(y) as the reward function for an output sequence y, comparing it with the ground truth sequence y * with the evaluation metric of our choice. DISPLAYFORM0 We can see that minimizing L rl is equivalent to maximizing the conditional likelihood of the sampled sequence y s if it obtains a higher reward than the baselineŷ, thus increasing the reward expectation of our model. One potential issue of this reinforcement training objective is that optimizing for a specific discrete metric like ROUGE does not guarantee an increase in quality and readability of the output. It is possible to game such discrete metrics and increase their score without an actual increase in readability or relevance BID16. While ROUGE measures the n-gram overlap between our generated summary and a reference sequence, human-readability is better captured by a language model, which is usually measured by perplexity. Since our maximum-likelihood training objective (Equation 14) is essentially a conditional language model, calculating the probability of a token y t based on the previously predicted sequence {y 1, . . ., y t−1} and the input sequence x, we hypothesize that it can assist our policy learning algorithm to generate more natural summaries. This motivates us to define a mixed learning objective function that combines equations 14 and 15: DISPLAYFORM0 where γ is a scaling factor accounting for the difference in magnitude between L rl and L ml. A similar mixed-objective learning function has been used by for machine translation on short sequences, but this is its first use in combination with self-critical policy learning for long summarization to explicitly improve readability in addition to evaluation metrics. Neural encoder-decoder models are widely used in NLP applications such as machine translation BID33, summarization BID4, and question answering BID8. These models use recurrent neural networks (RNN), such as long-short term memory network (LSTM) BID9 to encode an input sentence into a fixed vector, and create a new output sequence from that vector using another RNN.To apply this sequence-to-sequence approach to natural language, word embeddings BID20 BID25 are used to convert language tokens to vectors that can be used as inputs for these networks. Attention mechanisms BID0 ) make these models more performant and scalable, allowing them to look back at parts of the encoded input sequence while the output is generated. These models often use a fixed input and output vocabulary, which prevents them from learning representations for new words. One way to fix this is to allow the decoder network to point back to some specific words or sub-sequences of the input and copy them onto the output sequence BID35. BID7 and BID18 combine this pointer mechanism with the original word generation layer in the decoder to allow the model to use either method at each decoding step. Reinforcement learning (RL) is a way of training an agent to interact with a given environment in order to maximize a reward. RL has been used to solve a wide variety of problems, usually when an agent has to perform discrete actions before obtaining a reward, or when the metric to optimize is not differentiable and traditional supervised learning methods cannot be used. This is applicable to sequence generation tasks, because many of the metrics used to evaluate these tasks (like BLEU, ROUGE or METEOR) are not differentiable. In order to optimize that metric directly, BID27 have applied the REINFORCE algorithm BID36 to train various RNN-based models for sequence generation tasks, leading to significant improvements compared to previous supervised learning methods. BID1 also use a different kind of reinforcement learning algorithm on machine to optimize BLEU scores in machine translation tasks. While both these methods require an additional neural network, called a critic model, to predict the expected reward and stabilize the objective function gradients, Rennie et al. FORMULA1 designed a self-critical sequence training method that does not require this critic model and lead to further improvements on image captioning tasks. Most summarization models studied in the past are extractive in nature BID5 BID22 BID6, which usually work by identifying the most important phrases of an input document and re-arranging them into a new summary sequence. The more recent abstractive summarization models have more degrees of freedom and can create more novel sequences. Many abstractive models such as Rush et al. FORMULA1, BID4 and are all based on the neural encoder-decoder architecture (Section 4.1). BID19 extend the encoder-decoder architecture with a variational auto-encoder, and use REINFORCE to train it as well. A well-studied set of summarization tasks is the Document Understanding Conference (DUC) 1. These summarization tasks are varied, including short summaries of a single document and long summaries of multiple documents categorized by subject. Most abstractive summarization models have been evaluated on the DUC-2004 dataset, and outperform extractive models on that task BID5. However, models trained on the DUC-2004 task can only generate very short summaries up to 75 characters, and are usually used with one or two input sentences. applied different kinds of attention mechanisms for summarization on the CNN dataset, and Nallapati et al. FORMULA1 used different attention and pointer functions on the CNN and Daily Mail datasets combined. In parallel of our work, BID32 also developed an abstractive summarization model on this dataset with an extra loss term to increase temporal coverage of the encoder attention function. We evaluate our model on a modified version of the CNN/Daily Mail dataset BID8, following the same pre-processing steps described in. We refer the reader to that paper for a detailed description. The final dataset contains 287,113 training examples, 13,368 validation examples and 11,490 testing examples. After limiting the input length to 800 tokens and output length to 100 tokens, the average input and output lengths are respectively 632 and 53 tokens. The New York Times (NYT) dataset BID30 ) is a large collection of articles published between 1996 and 2007. Even though this dataset has been used to train extractive summarization systems BID6 BID10 or closely-related models for predicting the importance of a phrase in an article BID39 BID24 BID11, we are the first group to run an end-to-end abstractive summarization model on the article-abstract pairs of this dataset. While CNN/Daily Mail summaries have a similar wording to their corresponding articles, NYT abstracts are more varied, are shorter and can use a higher level of abstraction and paraphrase. Because of these differences, these two formats are a good complement to each other for abstractive summarization models. We describe the dataset preprocessing and pointer supervision in Section A of the Appendix. We evaluate the intra-decoder attention mechanism and the mixed-objective learning by running the following experiments on both datasets. We first run maximum-likelihood (ML) training with and without intra-decoder attention (removing c d t from Equations 9 and 11 to disable intraattention) and select the best performing architecture. Next, we initialize our model with the best ML parameters and we compare reinforcement learning (RL) with our mixed-objective learning (ML+RL), following our objective functions in Equation 15 and 16. The hyperparameters and other implementation details are described in the Appendix. We report the full-length F-1 score of the ROUGE-1, ROUGE-2 and ROUGE-L metrics with the Porter stemmer option. For RL and ML+RL training, we use the ROUGE-L score as a reinforcement reward. We also tried ROUGE-2 but we found that it created summaries that almost always reached the maximum length, often ending sentences abruptly. Our for the CNN/Daily Mail dataset are shown in Table 1, and for the NYT dataset in TAB2. We observe that the intra-decoder attention function helps our model achieve better ROUGE scores on the CNN/Daily Mail but not on the NYT dataset. Further analysis on the CNN/Daily Mail test set shows that intra-attention increases the ROUGE-1 score of examples with a long ground truth summary, while decreasing the score of shorter summaries, as illustrated in FIG3. This confirms our assumption that intra-attention improves performance on longer output sequences, and explains why intra-attention doesnt improve performance on the NYT dataset, which has shorter summaries on average. In addition, we can see that on all datasets, both the RL and ML+RL models obtain much higher scores than the ML model. In particular, these methods clearly surpass the state-of-the-art model from on the CNN/Daily Mail dataset, as well as the lead-3 extractive baseline (taking the first 3 sentences of the article as the summary) and the SummaRuNNer extractive model BID22. BID32 also reported their on a closely-related abstractive model the CNN/DailyMail but used a different dataset preprocessing pipeline, which makes direct comparison with our numbers difficult. However, their best model has lower ROUGE scores than their lead-3 baseline, while our ML+RL model beats the lead-3 baseline as shown in Table 1. Thus, we conclude that our mixedobjective model obtains a higher ROUGE performance than theirs. We also compare our model against extractive baselines (either lead sentences or lead words) and the extractive summarization model built by BID6, which was trained using a smaller version of the NYT dataset that is 6 times smaller than ours but contains longer summaries. We trained our ML+RL model on their dataset and show the on TAB4. Similarly to BID6, we report the limited-length ROUGE recall scores instead of full-length F-scores. For BID22 39.2 15.7 35.5 SummaRuNNer BID22 39.6 16.2 35.3 words-lvt2k-temp-att Table 5: Comparison of human readability scores on a random subset of the CNN/Daily Mail test dataset. All models are with intra-decoder attention. DISPLAYFORM0 each example, we limit the generated summary length or the baseline length to the ground truth summary length. Our show that our mixed-objective model has higher ROUGE scores than their extractive model and the extractive baselines. We perform human evaluation to ensure that our increase in ROUGE scores is also followed by an increase in human readability and quality. In particular, we want to know whether the ML+RL training objective did improve readability compared to RL.Evaluation setup: To perform this evaluation, we randomly select 100 test examples from the CNN/Daily Mail dataset. For each example, we show the original article, the ground truth summary as well as summaries generated by different models side by side to a human evaluator. The human evaluator does not know which summaries come from which model or which one is the ground truth. Two scores from 1 to 10 are then assigned to each summary, one for relevance (how well does the summary capture the important parts of the article) and one for readability (how well-written the summary is). Each summary is rated by 5 different human evaluators on Amazon Mechanical Turk and the are averaged across all examples and evaluators. Results: Our human evaluation are shown in Table 5. Even though RL has the highest ROUGE-1 and ROUGE-L scores, it produces the least readable summaries among our experiments. The most common readability issue observed in our RL , as shown in the example of TAB3, is the presence of short and truncated sentences towards the end of sequences. This confirms that optimizing for single discrete evaluation metric such as ROUGE with RL can be detrimental to the model quality. On the other hand, our RL+ML summaries obtain the highest readability and relevance scores among our models, hence solving the readability issues of the RL model while also having a higher ROUGE score than ML. This shows the value of the RL+ML training method. We also report perplexity scores in Table 5. Even though the ML model has the lowest perplexity, it doesn't have the highest readability. This indicate that perplexity measurements cannot replace human judgment for readability evaluation. We presented a new model and training procedure that obtains state-of-the-art in text summarization for the CNN/Daily Mail, improves the readability of the generated summaries and is better suited to long output sequences. We also run our abstractive model on the NYT dataset for the first time. We saw that despite their common use for evaluation, ROUGE scores have their shortcomings and should not be the only metric to optimize on summarization model for long sequences. Our intra-attention decoder and combined training objective could be applied to other sequence-tosequence tasks with long inputs and outputs, which is an interesting direction for further research. A NYT DATASET We remove all documents that do not have a full article text, abstract or headline. We concatenate the headline, byline and full article text, separated by special tokens, to produce a single input sequence for each example. We tokenize the input and abstract pairs with the Stanford tokenizer. We convert all tokens to lower-case and replace all numbers with "0", remove "(s)" and "(m)" marks in the abstracts and all occurrences of the following words, singular or plural, if they are surrounded by semicolons or at the end of the abstract: "photo", "graph", "chart", "map", "table" and "drawing". Since the NYT abstracts almost never contain periods, we consider them multisentence summaries if we split sentences based on semicolons. This allows us to make the summary format and evaluation procedure similar to the CNN/Daily Mail dataset. These pre-processing steps give us an average of 549 input tokens and 40 output tokens per example, after limiting the input and output lengths to 800 and 100 tokens. We created our own training, validation, and testing splits for this dataset. Instead of producing random splits, we sorted the documents by their publication date in chronological order and used the first 90% (589,284 examples) for training, the next 5% for validation, and the remaining 5% for testing. This makes our dataset splits easily reproducible and follows the intuition that if used in a production environment, such a summarization model would be used on recent articles rather than random ones. We run each input and abstract sequence through the Stanford named entity recognizer (NER). For all named entity tokens in the abstract if the type "PERSON", "LOCATION", "ORGANIZATION" or "MISC", we find their first occurrence in the input sequence. We use this information to supervise p(u t) (Equation 11) and α e ti (Equation 4) during training. Note that the NER tagger is only used to create the dataset and is no longer needed during testing, thus we're not adding any dependencies to our model. We also add pointer supervision for out-of-vocabulary output tokens if they are present in the input. For ML training, we use the teacher forcing algorithm with the only difference that at each decoding step, we choose with a 25% probability the previously generated token instead of the ground-truth token as the decoder input token y t−1, which reduces exposure bias BID34. We use a γ = 0.9984 for the ML+RL loss function. We use two 200-dimensional LSTMs for the bidirectional encoder and one 400-dimensional LSTM for the decoder. We limit the input vocabulary size to 150,000 tokens, and the output vocabulary to 50,000 tokens by selecting the most frequent tokens in the training set. Input word embeddings are 100-dimensional and are initialized with GloVe BID25. Based on these dimensions and sizes, our final model has 16.9M trainable parameters, 15M of which are word embeddings. We train all our models with Adam BID13 ) with a batch size of 50 and a learning rate α of 0.001 for ML training and 0.0001 for RL and ML+RL training. At test time, we use beam search of width 5 on all our models to generate our final predictions.

A summarization model combining a new intra-attention and reinforcement learning method to increase summary ROUGE scores and quality for long sequences.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Language generation using seq2seq models which produce word embeddings instead of a softmax based distribution over the vocabulary at each step enabling much faster training while maintaining generation quality


Question: represent the article for finding a single-sentence tldr

Question: Knowledge Distillation (KD) is a common method for transferring the ``knowledge'' learned by one machine learning model (the teacher) into another model (the student), where typically, the teacher has a greater capacity (e.g., more parameters or higher bit-widths). To our knowledge, existing methods overlook the fact that although the student absorbs extra knowledge from the teacher, both models share the same input data -- and this data is the only medium by which the teacher's knowledge can be demonstrated. Due to the difference in model capacities, the student may not benefit fully from the same data points on which the teacher is trained. On the other hand, a human teacher may demonstrate a piece of knowledge with individualized examples adapted to a particular student, for instance, in terms of her cultural and interests. Inspired by this behavior, we design data augmentation agents with distinct roles to facilitate knowledge distillation. Our data augmentation agents generate distinct training data for the teacher and student, respectively. We focus specifically on KD when the teacher network has greater precision (bit-width) than the student network. We find empirically that specially tailored data points enable the teacher's knowledge to be demonstrated more effectively to the student. We compare our approach with existing KD methods on training popular neural architectures and demonstrate that role-wise data augmentation improves the effectiveness of KD over strong prior approaches. The code for reproducing our will be made publicly available. Background and Motivation. In the educational psychology literature, it is generally considered beneficial if teachers can adapt curricula based upon students' prior experiences (; ; ;). These vary widely depending on students' cultural s, previous educational experiences, interests, and motivations. To help students with different prior experiences to comprehend, memorise, and consolidate a piece of knowledge, teachers may provide extra and customized teaching material during their teaching processes. For instance, when teaching the concept of the color pink, a teacher may choose flamingos, sakura (cherry blossoms), or ice cream cones as the example, depending on a student's . Knowledge distillation (KD) is a common framework for training machine learning models. It works by transferring knowledge from a higher-capacity teacher model to a lower-capacity student model. Most KD methods can be categorized by how they define the knowledge stored in the teacher (i.e., the "soft targets" of training as defined in existing literature). For instance, originally proposed KD for neural networks, and they define the output class probabilities (i.e., soft labels) generated by the teacher as the targets for assisting the training of students. In a follow up work, defined the soft targets via the feature maps in the teacher model's hidden layers. To train a student network with KD effectively, it is important to distill as much knowledge from the teacher as possible. However, previous methods overlook the importance of the medium by which the teacher's knowledge is demonstrated: the training data points. We conjecture that there exist examples, not necessarily seen and ingested by the teacher, that might make it easier for the student to absorb the teacher's knowledge. Blindly adding more training examples may not be beneficial because it may slow down training and introduce unnecessary biases . The analogy with how human teachers adjust their teaching to their students' particular situations (e.g., with the feedback gathered from the students during teaching) suggests that a reasonable yet uninvestigated approach might be to augment the training data for both the teacher and student according to distinct policies. In this paper, we study whether and how adaptive data augmentation and knowledge distillation can be leveraged synergistically to better train student networks. We propose a two-stage, rolewise data augmentation process for KD. This process consists of: training a teacher network till convergence while learning a schedule of policies to augment the training data specifically for the teacher; distilling the knowledge from the teacher into a student network while learning another schedule of policies to augment the training data specifically for the student. It is worth noting that this two-stage framework is orthogonal to existing methods for KD, which focus on how the knowledge to be distilled is defined; thus, our approach can be combined with previous methods straighforwardly. Although our proposed method can in principle be applied to any models trained via KD, we focus specifically on how to use it to transfer the knowledge from a full-precision teacher network into a student network with lower bit-width. Network quantization is crucial when deploying trained models on embedded devices, or in data centers to reduce energy consumption . KD-based quantization jointly trains a full-precision model, which acts as the teacher, alongside a low-precision model, which acts as the student. Previous work has shown that distilling a full-precision teacher's knowledge into a low-precision student, followed by fine-tuning, incurs noticeable performance degradation, especially when the bit-widths are below four . We show that it is advantageous to use adaptive data augmentation to generate more training data for the low-precision network based on its specific weaknesses. For example, low-precision networks may have difficulties learning rotationrelated patterns, 1 and the data augmentation agent should be aware of this and generate more such data points. One positive side-effect for demonstrating the effectiveness of our method is that the improvement brought by our proposed method is more significant compared to the experiments on all full-precision models. Knowledge distillation. KD is initially proposed for model compression, where a powerful wide/deep teacher distills knowledge to a narrow/shallow student to improve its performance . In terms of the definition of knowledge to be distilled from the teacher, existing models typically use teacher's class probabilities and/or intermediate features (; ; ;). Among those KD methods that utilize intermediate feature maps, Relational KD (RKD) considers the intra-relationship in the same feature map, while Multi-Head KD (MHKD) and KD using SVD (KD-SVD) utilize the inter-relationship across feature maps. By contrast, we propose to incorporate both the intra-and inter-relationships within and across feature maps. Automated data augmentation. Manually applying data augmentation rules such as random rotating, flipping, and scaling are common practices for training neural models on image classification tasks (; . Several recent works attempt to automate the data augmentation process. Generative adversarial networks and Bayesian optimization have been used for this process. augment training data in the learned feature space by injecting noise and interpolation. learn how to combine pairs of images for data augmentation. AutoAugment searches for the optimal data augmentation policies (e.g., how to rotate) based on reinforcement learning. However, the search process is computationally expensive. Population-based augmentation (PBA) uses an evolution-based algorithm to automatically augmenting data in an efficient way. In contrast to previous approaches, we study the effect of the training data for KD and propose to use automatic data augmentation to train the student better from her teacher. PBA , as an evolutionary search algorithm, learns a dynamic per-epoch schedule of augmentation policies, denoted as A. Since this schedule is epoch-based, it will re-create the augmented dataset every epoch. More concretely, PBA begins with a population of models that are trained in parallel on a small subset of the original training data. The weights of the worst performing models in the population are replaced by those from better performing models (i.e., exploitation), and the policies are mutated to new ones within the pre-defined policy search space (i.e., exploration). After training, PBA usually keeps the learned augmentation schedule of policies but discards the elementary parameters of the models. A different model (e.g., a larger one) can then use the learned schedule to improve its training on the same task. Following the notations in , a KD method aims to minimize the objective function where λ is a hyper-parameter to balance the impact of the KD loss term. In this paper, for classification tasks,, y truth ), where X refers to training sample space, y truth ∈ Y are the ground-truth labels, F S (·) is the student network, and H(·) denotes the cross-entropy. The KD term can be defined as where F(·) is the function of the network and l(·) is a loss function to compute the difference between the teacher network and the student network. For KD methods that use soft labels , the objective can be defined as where F final (x i) is the feature map of the final layer. There exist some KD methods that utilize the intermediate feature maps in complementary ways. For example, Relational KD considers the intra-relationships. That is, given the feature map of layer j, the KD loss can be formulated as: where Φ(·) refers to the potential function measuring the pairwise relationship inside a feature map from student network or teacher network and F j (x i) is the feature map of layer j (which may include the final logits layer). Therefore, this feature-based KD method includes the benefits of using soft labels. On the other hand, some works consider the interrelationships, where the KD term can be formulated as: Here, ϕ(·) measures the inter-relationship between feature maps of different layers, i.e. k = j. In this work, we use DoReFa 2 to quantize both weights and activations. The quantization function Q(·) is defined as: where r is the full-precision value, r q indicates the quantized value, n bits refers to the number of bits to represent this value. With this quantization function, the quantization on weights w is defined as: The back-propagation is approximated by the straight-through estimator and the partial gradient ∂l ∂r w.r.t. the loss l is computed as: 4 THE PROPOSED METHOD Our proposed method has two stages, which will be described in the following subsections. In the first stage, we train a teacher network, denoted as N T, with the help of PBA-based augmentation. In the second stage, we further distill the knowledge from N T (pre-trained in the first stage) to the student network, denoted as N S, while learning another augmentation schedule to augment the training data for N S. In general, a teacher can provide better training signals for the student if the teacher's performance increases . As shown in Fig. 1, we apply PBA to learn a dynamic per-epoch schedule of augmentation policies, A T, for N T on a small subset of training data. That is, the augmentation agent's training signal is defined as the feedback of N T's accuracy on a subset of the dataset. After this, we use the discovered schedule A T to augment the whole training dataset and re-train N T on it till convergence. KD methods have shown to be effective at improving the performance of lower-capacity networks using the knowledge from higher-capacity networks. In order to take advantage of this functionality, we apply the KD methods together with data augmentation in stage-β, as shown in Fig. 2. More concretely, we first use PBA to learn an epoch-based augmentation schedule A S for N S on a subset of the dataset. Different from the schedule A T learned in stage-α, A S is learned based on the feedback (i.e., accuracy) from N S, which is trained with KD. In other words, N S receives additional training signals from N T that is pre-trained in stage-α. We then use the learned A S to augment the whole training dataset, and re-train N S on it with the distilled knowledge from N T. Note that, because the learned schedule is epoch-based, we do not use the discovered schedule A T from stage-α to augment the training data as initialization. Figure 2: Concept diagram (stage-β) to augment training datapoints for both N T and N S. The N T has been pre-trained using the method shown in Section 4.1, and is fixed during training. The augmentation agent in stage-β is designed to learn schedules of polices that are different from those learned in stage-α, and thus the agent only receives the feedback from N S. When N S is a low-precision network, following , we share the same network architecture 3 between N T and N S. When N S is a full-precision network, it will have fewer layers compared to N T. We evaluate our approach on two benchmark datasets: CIFAR-10 and CIFAR-100. CIFAR-10 consists of 60,000 32×32 color images in 10 classes, with 6000 images per class and CIFAR-100 has 100 classes with 600 images per class. Both have 50,000 training images and 10,000 test images. We search over a "reduced" CIFAR-10/CIFAR-100 with 4,000 training images and 36,000 validation images, which is the same as in . All the data augmentation models are run with 16 total trials to generate augmentation schedules. Following PBA, in stage-α, we run PBA to create schedules over separate models and then transfer the CIFAR-10 policy to CIFAR-100. However, for student network training in stage-β, we empirically use the respective "reduced" dataset. The data augmentation approaches for the baselines include random crop and horizontal flipping operations. Following , our policy search space has a total of 15 operations, each having two magnitude and discrete probability values. We use discrete probability values from 0% to 100%, in increments of 10%. Magnitudes range from 0 to 9. The models we evaluate on include AlexNet and ResNet18. The number of epochs is 200 and the batch size is set to 128. For full-precision network N T, the learning rate starts from 0.1 and is decayed by 0.1 after every 30% of the total epochs. we use SGD with a Nesterov momentum optimizer. The weight decay is set to 5 × 10 −4. For quantization, the learning rate is set to 10 −3 and is divided by 10 every 30% of the total epochs. We use the pre-trained teacher network model as the initial point of student network. We use a smaller weight decay 10 −5 assuming that less regularization is needed by the lower-precision networks. Following DoReFa , the first layer and last layer are not quantized. , during training, we gradually transit the student from learning based on the teacher to training based on the ground-truth labels. This heuristic provides the student with more rich training signals in the early stage but does not force the student to strictly mimic the teacher's behaviors. As for the implementation, we decay the balancing hyper-parameter λ in the KD loss by 0.5 every 60 epochs. As mentioned in Section 3.2, there exist complementary KD methods considering both intra-and inter-relationships within and across feature maps. A natural question is if it would be beneficial to combine them to further boost the performance together with data augmentation. Therefore, we propose a simple extension to these complementary KD methods, dubbed as II-KD, by incorporating intra-relationships inside the feature map and inter-relationships across different feature maps. We incorporate the two relationships into the final objective function as follows: where we only use a single balancing hyper-parameter λ between the original loss and the distillation loss, which does not introduce extra hyper-parameters. More precisely, our KD method incorporates components of three conventional KD methods: RKD , MHGD and KD-SVD . As shown in Eq., we add the three KD terms together with equal coefficients. We use the loss function l(·) following their approaches. For the back-propagation, we clip the gradient for KD loss as in KD-SVD, because this will smoothly post-processes the gradient to limit the impact of KD loss in training. For AlexNet we select the feature maps of ReLU layers after the convolution/max pooling layer. For ResNet18, we select the feature maps of the last ReLU layer of each residual block. We evaluate our proposed KD extension on CIFAR-100 with ResNet18 for different bit-width settings by comparing with various KD methods. For the baseline methods, we use their default settings with a fixed and pre-trained teacher network in the training stage and λ = 1 for the knowledge distillation loss. We set λ = 0.4 for II-KD in Eq., as we have two KD terms. Tab. 1 reports the on various augmented KD methods. We observe that our proposed methods clearly outperforms the other KD methods on all the settings, though the improvements over MHGD and KD-SVD are not huge. The also reveal that only relying soft labels is not as effective as utilizing multiple supervising signals from the teacher. In this subsection, we aim to answer this question: is our two-stage role-wise augmentation with KD effective for network quantization? We conduct experiments on CIFAR-10 and CIFAR-100 datasets under full-precision, 4-bit, and 2-bit settings. From Tab. 2, we can observe that training with learned data augmentation schedules does not improve the performance of low-precision networks too much. Similar to the obtained in , transferring knowledge from the full-precision to the low-precision student usually helps the training of students, which is especially obvious on the CIFAR-100 dataset. Tab. 2 also clearly shows that our proposed pipeline consistently improves the performance of the lowprecision student networks. For example, the 4-bit N S is comparable with full-precision reference without loss of accuracy for CIFAR-10 and with loss of accuracy within 1.0% on CIFAR-100. When decreasing the precision to 2-bit, the are still promising as compared with other baselines, even though there is a performance gap between the 2-bit and the full-precision models. For instance, our approach usually outperforms the strong baseline, only using II-KD, by more than 1.0%. Here we aim to answer this question: how effective is it if we use A T, learned based on the feedback from N T in stage-α, to dynamically augment the training dataset and train N S on it? Tab. 3 reports the accuracy comparison with different KD methods and augmentation schedules. We can clearly see that augmenting the training dataset for N S with A S consistently outperforms those using the transferred schedules A T among different KD methods. This observation is consistent with our assumption that N S has her own optimal augmentation schedule, A S, that is different from A T for N T. In particular, blindly applying the teacher augmentation schedule A T may negatively influence Table 2: Accuracy (%) on CIFAR-10 and CIFAR-100 datasets with different bit-widths. Vanilla Training for 4-bit and 2-bit refers to training a network based on DoReFa from scratch without learned data augmentation. Teacher after Stage-α refers to using learned schedules discovered by PBA to re-train N T as described in Section 4.1. Student with only II-KD refers to training N S using II-KD but without the learned data augmentation. Student after Stage-β refers to training N S using II-KD and the learned data augmentation. For Vanilla Training and Teacher after Stage-α, we report the accuracy of N T, and for the rest we report the accuracy of N S. the training of N S as compared to only using KD. For example, the learned schedule based on the teacher A T degrades the performance of N S by 0.58% for AlexNet on CIFAR-100 as compared to applying KD methods, as shown in Tab. 2. To analyze the difference on the discovered schedules between N T (i.e., full-precision ResNet18) and N S (i.e., 4-bit ResNet18), we report their augmented schedules quantitatively in terms of normalized probability and magnitude on CIFAR-100 in Fig. 3. We normalize the probability of each epoch by dividing the maximal summation of probabilities for all operations across all epochs. It can be seen that the discovered schedules A S for N S is quite different from A T for N T. In particular, for A T, there is an emphasis on Brightness, Posterize, Rotate, Sharpness and TranslateY, while A S cares more about Contrast, ShearX and TranslateY. Furthermore, we observe that the probability and magnitude increase as the epoch evolves. For A S, in the beginning, KD plays a more important role, and there is no augmentation operation before about epoch 50. As the training continues, the augmentation policies become more important. One possible reason is that, for lowprecision networks, KD methods can provide rich training signals such that data augmentation does not help in the early training phases. Furthermore, we observe that, compared to A T, the schedule for student A S evolves more smoothly in the sense that the policy updating frequency is lower. For example, the probability and magnitude values change about every 40 epochs for student, while the policies for teacher update about every 15 epochs. One possible reason is that for the low-precision N S, KD methods make the training process more smooth and it is not necessary to change the augmentation policies too frequently. This is consistent with the observations shown in Tab. 2 that KD can already provide useful training signals. Also, this validates our assumption that N S has her own optimal augmentation schedule A S that is different from A T. (a) Normalized plot of operation probability parameters over time for the teacher network NT. (b) Operation magnitude parameters over time for the teacher network NT. (c) Normalized plot of operation probability parameters over time for the student network NS. (d) Operation magnitude parameters over time for the student network NS. Figure 3: Evolution of magnitude and probability parameters in the learned schedules. Each operation appears in the parameter list twice, and we take the mean values of the parameter. This subsection aims to verify the effectiveness of our proposed methods on more conventional settings where both N T and N S are full-precision networks. We select ResNet18 as the teacher, and ResNet8 as the student to check how our proposed methods affect the student network. Tab. 4 shows that our proposed method outperforms the standard baseline training. The discovered augmentation schedule further boosts the performance of shallow N S based on II-KD, though the improvement is not that significant compared with the obtained when N S is low-precision network. This shows that our proposed method can be used for full-precision training tasks. Table 4: Accuracy (%) on CIFAR-100 with full-precision ResNet8 as N S and full-precision ResNet18 as N T under different settings. Vanilla Training refers to training a full-precision network from scratch. Re-Training with PBA refers to using learned schedules discovered by PBA to re-train N T as described in Section 4.1. Student with only II-KD refers to training N S using II-KD but without the learned data augmentation. Student after Stage-β refers to training N S using II-KD and the learned data augmentation. Previous literature on KD focuses on exploring the knowledge representation and the strategies for distillation. However, both the teacher and student learn from the same training data without adapting the different learning capabilities. To address this issue, we propose customizing distinct agents to automatically augment the training data for the teacher and student, respectively. We have extensively studied the effect of combining data augmentation and knowledge distillation. Furthermore, we propose a simple feature-based KD variant that incorporates both intra-and inter-relationships within and across feature maps. We have empirically observed that the student can learn better from the teacher with the proposed approach, especially in the challenging low-precision scenario, and the learned schedules are different for the teacher and student.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We study whether and how adaptive data augmentation and knowledge distillation can be leveraged simultaneously in a synergistic manner for better training student networks.

Represent the TLDR for finding the corresponding scientific article!

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.