Question: Represent the scientific article to find a short TLDR

Question: Human-computer conversation systems have attracted much attention in Natural Language Processing. Conversation systems can be roughly divided into two categories: retrieval-based and generation-based systems. Retrieval systems search a user-issued utterance (namely a query) in a large conversational repository and return a reply that best matches the query. Generative approaches synthesize new replies. Both ways have certain advantages but suffer from their own disadvantages. We propose a novel ensemble of retrieval-based and generation-based conversation system. The retrieved candidates, in addition to the original query, are fed to a reply generator via a neural network, so that the model is aware of more information. The generated reply together with the retrieved ones then participates in a re-ranking process to find the final reply to output. Experimental show that such an ensemble system outperforms each single module by a large margin. Automatic human-computer conversation systems have long served humans in domain-specific scenarios. A typical approach for such systems is built by human engineering, for example, using manually constructed ontologies ), natural language templates ), and even predefined dialogue state tracking BID29 ).Recently, researchers have paid increasing attention to open-domain, chatbot-style human-computer conversations such as XiaoIce 1 and Duer 2 due to their important commercial values. For opendomain conversations, rules and templates would probably fail since they hardly can handle the great diversity of conversation topics and flexible representations of natural language sentences. With the increasing popularity of on-line social media and community question-answering platforms, a huge number of human-human conversation utterances are available on the public Web BID32; BID13 ). Previous studies begin to develop data-oriented approaches, which can be roughly categorized into two groups: retrieval systems and generative systems. When a user issues an utterance (called a query), the retrieval-based conversation systems search a corresponding utterance (called a reply) that best matches the query in a pre-constructed conversational repository BID10; BID11 ). Owing to the abundant web resources, the retrieval mechanism will always find a candidate reply given a query using semantic matching. The retrieved replies usually have various expressions with rich information. However, the retrieved replies are limited by the capacity of the pre-constructed repository. Even the best matched reply from the conversational repository is not guaranteed to be a good response since most cases are not tailored for the issued query. To make a reply tailored appropriately for the query, a better way is to generate a new one accordingly. With the prosperity of neural networks powered by deep learning, generation-based conversation systems are developing fast. Generation-based conversation systems can synthesize a new sentence as the reply, and thus bring the of good flexibility and quality. A typical generationbased conversation model is seq2seq BID23; BID22; BID20 ), in which two recurrent neural networks (RNNs) are used as the encoder and the decoder. The encoder is to capture the semantics of the query with one or a few distributed and real-valued vectors (also known as embeddings); the decoder aims at decoding the query embeddings to a reply. Long short term memory (LSTM) BID8 Table 1: Characteristics of retrieved and generated replies in two different conversational systems.(GRUs) BID3 ) could further enhance the RNNs to model longer sentences. The advantage of generation-based conversation systems is that they can produce flexible and tailored replies. A well known problem for the generation conversation systems based on "Seq2Seq" is that they are prone to choose universal and common generations. These generated replies such as "I don't know" and "Me too" suit many queries BID20 ), but they contain insufficient semantics and information. Such insufficiency leads to non-informative conversations in real applications. Previously, the retrieval-based and generation-based systems with their own characteristics, as listed in Table 1, have been developed separately. We are seeking to absorb their merits. Hence, we propose an ensemble of retrieval-based and generation-based conversation systems. Specifically, given a query, we first apply the retrieval module to search for k candidate replies. We then propose a "multi sequence to sequence" (multi-seq2seq) model to integrate each retrieved reply into the Seq2Seq generation process so as to enrich the meaning of generated replies to respond the query. We generate a reply via the multi-seq2seq generator based on the query and k retrieved replies. Afterwards, we construct a re-ranker to re-evaluate the retrieved replies and the newly generated reply so that more meaningful replies with abundant information would stand out. The highest ranked candidate (either retrieved or generated) is returned to the user as the final reply. To the best of our knowledge, we are the first to build a bridge over retrieval-based and generation-based modules to work out a solution for an ensemble of conversation system. Experimental show that our ensemble system consistently outperforms each single component in terms of subjective and objective metrics, and both retrieval-based and generation-based methods contribute to the overall approach. This also confirms the rationale for building model ensembles for conversation systems. In early years, researchers mainly focus on domain-specific conversation systems, e.g., train routing BID0 ) and human tutoring BID6 ). Typically, a pre-constructed ontology defines a finite set of slots and values, for example, cuisine, location, and price range in a food service conversation system; during human-computer interaction, a state tracker fills plausible values to each slot from the user input, and recommend the restaurant that best meets the user's requirement BID30; BID17; ).In the open domain, however, such slot-filling approaches would probably fail because of the diversity of topics and natural language utterances. BID10 apply information retrieval techniques to search for related queries and replies. BID11 and BID32 use both shallow hand-crafted features and deep neural networks for matching. BID13 propose a random walk-style algorithm to rank candidate replies. In addition, their model can incorporate additional content (related entities in the conversation context) by searching a knowledge base when a stalemate occurs during human-computer conversations. Generative conversation systems have recently attracted increasing attention in the NLP community. BID18 formulate query-reply transformation as a phrase-based machine translation. BID36 use two RNNs in encoder and one RNN in decoder to translate a sentence into two different languages into another language. Since the last year, the renewed prosperity of neural networks witnesses an emerging trend in using RNN for conversation systems BID25; BID26; BID23; BID22; BID20 ). The prevalent structure is the seq2seq model BID25 ) which comprises of one encoder and one decoder. However, a known issue with RNN is that it prefers to generate short and meaningless utterances. Following the RNN-based approach, BID12 propose a mutual information objective in contrast to the conventional maximum likelihood criterion. BID16 and BID31 Figure 1: The overall architecture of our model ensemble. We combine retrieval-based and generation-based conversation systems with two mechanisms. The first ensemble is to enhance the generator with the retrieved candidates. The second is the re-ranking of both candidates.generation. BID7 uses knowledge base for answer generation in question answering task and BID14 investigates different attention strategies in multi-source generation. To the best of our knowledge, the two main streams namely retrieval-based and generation-based systems have developed independently, and we are the first to combine these two together. In the following section, we depict the whole picture of our ensemble framework, and describe how to integrate those two modules in detail.3 MODEL ENSEMBLE 3.1 OVERVIEW Figure 1 depicts the overview of our proposed conversation system that ensembles the retrieval-based and generation-based approaches. It consists of the following components. We briefly describe each component, then present the details in the following sub-sections.• Retrieval Module. We have a pre-constructed repository consisting millions of query-reply pairs q *, r *, collected from human conversations. When a user sends a query utterance q, our approach utilizes a state-of-the-practice information retrieval system to search for k best matched queries (q *), and return their associated replies r * as k candidates.• Generation Module. We propose the multi-seq2seq model, which takes the original query q and k retrieved candidate replies r * 1, r * 2,..., r * k as input, and generates a new reply r +. Thus the generation process could not only consider about the given query, but also take the advantage of the useful information from the retrieved replies. We call it the first ensemble in our framework.• Re-ranker. Finally, we develop a re-ranker to select the best reply r from the k + 1 candidates obtained from retrieval-based and generation-based modules. Through the ensemble of retrievalbased and generation-based conversation, the enlarged candidate set enhances the quality of the final . We call this procedure the second ensemble in our framework. The information retrieval-based conversation is based on the assumption that the appropriate reply to the user's query is contained by the pro-constructed conversation datasets. We collect huge amounts of conversational corpora from on-line chatting platforms, whose details will be described in the section of evaluation. Each utterance and its corresponding reply form a pair, denoted as q *, r *.Based on the pre-constructed dataset, the retrieval process can be performed using an the state-ofthe-practice information retrieval system. We use a Lucene 3 powered system for the retrieval implementation. We construct the inverted indexes for all the conversational pairs at the off-line stages. When a query is issued, the keyword search with the tf.idf weighting schema will be performed to retrieve several q * that match the user's query q. Given the retrieved q *, the associated r * will be returned as the output, in an indirect matching between the user's query q and the retrieved reply r *. The retrieval systems would provide more than one replies and score them according to the semantic matching degree, which is a traditional technic in information retrieval. As the top ranked one may not perfectly match the query, we keep top-k replies for further process. The information retrieval is a relatively mature technique, so the retrieval framework can be alternated by any systems built keep to the following principles. Figure 2: The multi-seq2seq model, which takes a query q and k retrieved candidate replies r * as the input and generate a new reply r + as the output. The technique of neural networks has become a popular approach to build end-to-end trainable conversation systems BID16 ). A generation-based conversation system is able to synthesize new utterances, which is complementary to retrieval-based methods. The seq2seq model BID25 ), considering the Recurrent Neural Network (RNNs) as the encoder and decoder to transfer source sentence to target sentence, has long been used for generation tasks. The objective function for the seq2seq model in our scenario is the log-likelihood of the generated reply r + given the query q: DISPLAYFORM0 Since the reply is generated on the conditional probabilities given the query, the universal replies which have relatively higher probabilities achieve higher rankings. However, these universal sentences contain less information, which impair the performance of generative systems. BID16 also observe that in open-domain conversation systems, if the query does not carry sufficient information, seq2seq tends to generate short and meaningless sentences. Different from the pipeline in seq2seq model, we propose the multi-seq2seq model (Figure 2), which synthesizes a tailored reply r + by using the information both from the query q and the retrieved r * 1, r * 2,..., r * k. multi-seq2seq employs k + 1 encoders, one for query and other k for retrieved r *. The decoder receives the outputs of all encoders, and remains the same with traditional seq2seq for sentence generation. multi-seq2seq model improves the quality of the generated reply in two ways. First, the newly generated reply conditions not only on the given query but also on the retrieved reply. So the probability of universal replies would decrease since we add an additional condition. The objective function can be written as: DISPLAYFORM1 Thus the r + would achieve higher score only if it has a high concurrency with both q and r * 1, r * 2,..., r * k. Second, the retrieved replies r * 1, r * 2,..., r * k are the human-produced utterances and probably contain more information, which could be used as the additional information for the generated reply r +. Hence, the generated reply can be fluent and tailored to the query, and be more meaningful due to the information from the retrieved candidates. To take advantage of retrieved replies, we propose to integrate attention and copy mechanisms into decoding process. Attention helps the decoder to decide which parts in each retrieved reply are useful for current generation step. Copy mechanism directly extracts proper words from encoders, namely both query and retrieved replies, and utilizes them as the output words during the decoding process.• Two-level Attention. multi-seq2seq conducts sentence-and character-level attention to make better use of the query and retrieved replies. As multiple replies are of uneven quality, we use sentence-level attention to assign different importance to each retrieved replies. Similarly, multiple words are of uneven quality in a sentence, we use character-level attention to measure different importance to each words in retrieved replies. Specifically, for the sentence-level, we use k + 1 vectors obtained from the encoders to capture the information of q and the k r *, denoted as q and r) to multi-source attention to introduce retrieved replies, given by DISPLAYFORM2 where c i is the context vector at each time step in decoding, which integrates query and all possible words in k retrieved replies. l is the length of query, h j is the hidden state of query, l m is the length of r * m, h m,j is the hidden state of r * m. s i is the hidden state of decoder at time step i, α i,m,j is the normalized attention weights for each word. e i,m,j is calculated by a bilinear matching function and M a is the parameter matrix.• Copy Mechanism. multi-seq2seq also uses copy mechanism to explicitly extract words from the retrieved replies. For each word y t in vocabulary V, the probability p(y t |s t) in decoding process is comprised of k + 1 parts. The first part p ori follows the original probability calculated by GRU/LSTM cells, and the following parts p r * m reflect the matching degree between the current state vector s t and the corresponding states of y t in encoders, given by, DISPLAYFORM3 DISPLAYFORM4 where h yt,m is the hidden states of retrieved reply r * m who responds y t in decoder, δ(·) is the sigmoid function, M c is the parameter for matching s t and h yt,m. If y t has not appeared in a retrieved replies r * m, the corresponding probabilities p r * m would be zero. Both attention and copy mechanism aim to enrich the generated reply r + via useful and informative words extracted from retrieved replies r * 1, r * 2,..., r * k. Figure 2 displays the design of multi-seq2seq model. We can see that the generated reply has the corresponding relation with the query, and absorbs the keywords from the retrieved replies. Now that we have k retrieved candidate replies r * as well as a generated one r +. As all the retrieved candidates are obtained via indirect matching, these replies need a further direct matching with the user-issued query. On the other hand, the generated reply set may contain the influent and meaningless utterances. Hence, we propose the second ensemble to derive the final ranking list by feeding all the candidates into a re-ranker. We deploy a Gradient Boosting Decision Tree (GBDT) BID34 ) classifier since it is believed to have ability to handle the replies with various traits. The GBDT classifier utilizes several highlevel features, as listed in the following. The first four are pairwise feature, and the last two are features based on the properties of replies.• Term similarity. The word overlap ratio captures the literal similarity between the query and reply. For both query and reply, we transform them into binary word vectors, in which each element indicates if a word appears in the corresponding sentence. Then we apply the cosine function to calculate the term overlap similarity of the query and the reply.• Entity similarity. Named entities in utterances are a special form of terms. We distinguish persons, locations and organizations from plain texts with the help of named entity recognition techniques. Then we maintain the vectors of recognized entities for both query and its reply and calculate the similarity (measured by cosine similarity) between two entity-based vector representations.• Topic similarity. "Topics" has long been regarded as the abstractive semantic representation (Hofmann FORMULA0). We apply Latent Dirichlet Allocation BID2 ) to discover the latent topics of the query and reply. The inferred topic representation is the probabilities for the piece of text belonging to each latent topic. By setting the topic number as 1000, which works efficiently in practice, we use the cosine similarity to calculate the topical score between vectors of latent topics.• Statistical Machine Translation. By treating queries and replies as different languages in the paradigm of machine translation, we train a translation model to "translate" the query into a reply based on the training corpora to get the translating word pairs (one word from a query and one word from its corresponding reply) with scores indicating their translating possibilities. To get the translation score for the query and reply, we sum over the translating scores of the word pairs extracted from these two sentences, and conduct normalization on the final score.• Length. Since too short replies are not preferred in practical conversation systems, we take the length of replies as a point-wise feature. We conduct a normalization to map the value to.• Fluency. Fluency is to examine whether two neighboring terms have large co-occurrence likelihood. We calculate the co-occurrence probability for the bi-grams of the candidate replies and then take the average value as the fluency. The confidence scores produced by the GBDT classifier are used to re-rank all the replies. The re-ranking mechanism can eliminate both meaningless short replies that are eventually generated by multi-seq2seq and less appropriate replies selected by the retrieval system. The re-ranker further ensures an optimized effect of model ensemble. Since our framework consists of learnable but independent components (i.e., multi-seq2seq and Re-ranker), the model training is constructed for each component separately. In multi-seq2seq, we use human-human utterance pairs q, r as data samples. k retrieved candidates r * are also provided as the input when we train the neural network. Standard crossentropy loss of all words in the reply is applied as the training objective, given by, DISPLAYFORM0 where J is the objective of trainning, T is the length of r and t (i) is the one-hot vector of the next target word in the reply, serving as the ground-truth, y j is the probability of a word obtained from the softmax function, and V is the vocabulary size. In the re-ranker part, the training samples are either q, r pairs or generated by negative sampling. We evaluate our ensemble model on our established conversation system in Chinese. Both retrieval-based and generation-based components require a large database of query-reply pairs, whose statistics is exhibited in TAB4. To construct a database for information retrieval, we collected human-human utterances from massive online forums, microblogs, and question-answering communities, including Sina Weibo 4 and Baidu Tieba. 5 In total, the database contains 7 million query-reply pairs for retrieval. For each query, corresponding to a question, we retrieve k replies (k = 2) for generation part and re-ranker. For the generation part, we use the dataset comprising 1,606,741 query-reply pairs originating from Baidu Tieba. Please note that q and r * are the input of multi-seq2seq, whose is supposed to should approximate the ground-truth. We randomly selected 1.5 million pairs for training and 100K pairs for validation. The left 6,741 pairs are used for testing both in generation part and the whole system. Notice that this corpus is different from the corpus used in the retrieval part so that the ground-truth of the test data are excluded in the retrieval module. The training-validation-testing split remains the same for all competing models. Human Score BLEU-1 BLEU-2 BLEU-3 BLEU-4 BID22 To train our neural models, we implement code based on dl4mt-tutorial 6, and follow BID22 for hyper-parameter settings as it generally works well in our model. We did not tune the hyperparameters, but are willing to explore their roles in conversation generation in future. All the embeddings are set to 620-dimension and the hidden states are set to 1000-dimension. We apply AdaDelta with a mini-batch size of 80. Chinese word segmentation is performed on all utterances. We keep the same set of 100k words for all encoders and 30K for the decoder due to efficiency concerns. The validation set is only used for early stop based on the perplexity measure. We compare our model ensemble with each individual component and provide a thorough ablation test. Listed below are the competing methods in our experiments. For each method, we keep one best reply as the final to be assessed. All competing methods are trained in the same way as our full model, when applicable, so that the comparison is fair.• Retrieval-1, Retrieval-2. The top and second ranked replies for the user-issued query from a state-of-the-practice conversation system ), which is a component of our model ensemble; it is also a strong baseline (proved in our experiments).• seq2seq. An encoder-decoder framework BID25 ), first introduced as neural responding machine by BID22.• multi-seq2seq −. Generation component, which only applies two-level attention strategies.• multi-seq2seq. Generation component, which applies two-level attention and copy strategy.• Ensemble(Retrieval-1,Retrieval-2, seq2seq). Ensemble with retrieval and seq2seq.• Ensemble(Retrieval-1, Retrieval-2, multi-seq2seq).Ensemble with retrieval and multi-seq2seq. This is the full proposed model ensemble. We evaluate our approach in terms of both subjective and objective metrics.• Subjective metric. Human evaluation, albeit time-and labor-consuming, conforms to the ultimate goal of open-domain conversation systems. We ask three educated volunteers to annotate the BID22; BID13; BID16 ). Annotators are asked to label either "0" (bad), "1" (borderline), or "2" (good) to a query-reply pair. The subjective evaluation is performed in a strictly random and blind fashion to rule out human bias.• Objective metric. We adopt BLEU 1-4 for the purpose of automatic evaluation. While further strongly argue that no existing automatic metric is appropriate for open-domain dialogs, they show a slight positive correlation between BLEU-2 and human evaluation in nontechnical Twitter domain, which is similar to our scenario. We nonetheless include BLEU scores as the expedient objective evaluation, serving as a supporting evidence. BLEUs are also used in BID12 for model comparison and in BID16 for model selection. Notice that, the automatic metrics were computed on the entire test set, whereas the subjective evaluation was based on 100 randomly chosen test samples due to the limitation of human resources. We present our main in TAB6. Table 4 presents two examples of our ensemble and its "base" models. As showed, the retrieval system, which our model ensemble is based on, achieves better performance than RNN-based sequence generation. This also verifies that the retrieval-based conversation system in our experiment is a strong baseline to compare with. Table 4: Examples of retrieved and generated ones. " √ " indicates the reply selected by the re-ranker. Combining the retrieval system, generative system multi-seq2seq and the re-ranker, our model leads to the bset performance in terms of both human evaluation and BLEU scores. Concretely, our model ensemble outperforms the state-of-the-practice retrieval system by +34.45% averaged human scores, which we believe is a large margin. Having verified that our model achieves the best performance, we are further curious how each gadget contributes to our final system. Specifically, we focus on the following research questions. What is the performance of multi-seq2seq (the First Ensemble in Figure 1) in comparison with traditional seq2seq?From BLEU scores in TAB6, we see both multi-seq2seq − and multi-seq2seq significantly outperform conventional seq2seq, and multi-seq2seq is slightly better than multi-seq2seq −. These imply the effectiveness of both two-level attention and copy mechanism. We can also see multi-seq2seq outperforms the second retrieval in BLEUs. In the retrieval and seq2seq ensemble, 72.84% retrieved and 27.16% generated ones are selected. In retrieval and multi-seq2seq ensemble, the percentage becomes 60.72% vs. 39.28%. The trend indicates that multi-seq2seq is better than seq2seq from the re-ranker's point of view. RQ2: How do the retrieval-and generation-based systems contribute to re-ranking (the Second Ensemble in Figure 1)? As the retrieval and generation module account for 60.72% and 39.28% in the final of retrieval and multi-seq2seq ensemble, they almost contribute equally to the whole framework. More importantly, we notice that retrieval-1 takes the largest proportion in two ensemble systems, and it may explain why most on-line chatting platforms choose retrieval methods to build their systems. multi-seq2seq decreases the proportion of retrieved one in the second ensemble systems, which indicates multi-seq2seq achieves better than retrieval-1 in some cases. Since the two ensembles are demonstrated to be useful already, can we obtain further gain by combining them together?We would also like to verify if the combination of multi-seq2seq and re-ranking mechanisms yields further gain in our ensemble. To test that, we compare the full model Ensemble(Retrieval, multi-seq2seq) with an ensemble that uses traditional seq2seq, i.e., Ensemble(Retrieval, seq2seq). As indicated in TAB6, even with the re-ranking mechanism, the ensemble with underlying multi-seq2seq still outperforms the one with seq2seq. Likewise, Ensemble(Retrieval, multi-seq2seq) outperforms both Retrieval and multi-seq2seq in terms of most metrics. Through the above ablation tests, we conclude that both components (first and second ensemble) play a role in our ensemble when we combine the retrieval-and generation-based systems. In this paper, we propose a novel ensemble of retrieval-based and generation-based open-domain conversation systems. The retrieval part searches the k best-matched candidate replies, which are, along with the original query, fed to an RNN-based multi-seq2seq reply generator. Then the generated replies and retrieved ones are re-evaluated by a re-ranker to find the final . Although traditional generation-based and retrieval-based conversation systems are isolated, we have designed a novel mechanism to connect both modules. The proposed ensemble model clearly outperforms state-of-the-art conversion systems in the constructed large-scale conversation dataset.

represent the tldr for finding the corresponding scientific article

Examples:

"Gaussian normalization performs a least-squares fit during back-propagation, which zero-centers and decorrelates partial derivatives from normalized activations." == "Batch Normalization (BN) and its variants have seen widespread adoption in the deep learning community because they improve the training of deep neural networks. Discussions of why this normalization works so well remain unsettled. We make explicit the relationship between ordinary least squares and partial derivatives computed when back-propagating through BN. We recast the back-propagation of BN as a least squares fit, which zero-centers and decorrelates partial derivatives from normalized activations. This view, which we term {\em gradient-least-squares}, is an extensible and arithmetically accurate description of BN. To further explore this perspective, we motivate, interpret, and evaluate two adjustments to BN. Training deep neural networks has become central to many machine learning tasks in computer vision, speech, and many other application areas. BID10 showed empirically that Batch Normalization (BN) enables deep networks to attain faster convergence and lower loss. Reasons for the effectiveness of BN remain an open question BID12. Existing work towards explaining this have focused on covariate shift; described how BN makes the loss function smoother. This work examines the details of the back-propagation of BN, and recasts it as a least squares fit. This gradient regression zero-centers and decorrelates partial derivatives from the normalized activations; it passes on a scaled residual during back-propagation. Our view provides novel insight into the effectiveness of BN and several existing alternative normalization approaches in the literature. Foremost, we draw an unexpected connection between least squares and the gradient computation of BN. This motivates a novel view that complements earlier investigations into why BN is so effective. Our view is consistent with recent empirical surprises regarding ordering of layers within ResNet residual maps BID5 and within shake-shake regularization branches BID6. Finally, to demonstrate the extensibility of our view, we motivate and evaluate two variants of BN from the perspective of gradient-least-squares. In the first variant, a least squares explanation motivates the serial chaining of BN and Layer Normalization (LN) BID0. In the second variant, regularization of the least-squares leads to a version of BN that performs better on batch size two. In both variants, we provide empirical support on CIFAR-10.In summary, our work presents a view, which we term gradient-least-squares, through which the back-propagation of BN and related work in a neural network can be recast as least squares regression. This regression decomposes gradients into an explained portion and a residual portion; BN back-propagation will be shown to remove the explained portion. Hopefully, gradient-least-squares will be broadly useful in the future design and understanding of neural network components. Figure 1 reviews normalization with batch statistics, and illustrates our main theorem. Gradient-least-squares relates quantities shown in hexagons Figure 1: The left figure reviews, for a single channel at a particular layer within a single batch, notable quantities computed during the forward pass and during back-propagation of BN. Let DISPLAYFORM0 Let L be a function dependent on the normalized activations z i defined for each j by z j = (x j − µ) σ This, along with partial derivatives, are shown in the left figure. Our work establishes a novel identity on the quantities shown in hexagons. The right figure illustrates our main in a scatter plot, in which each pair z i, ∂L ∂z i is shown as a data point in the regression. Consider any particular channel within which {x i} are activations to be normalized in BN moment calculations. BID10 defined BN as DISPLAYFORM0 where σ, µ are batch moments, but b and c are learned per-channel parameters persistent across batches. In BN, the batch dimension and spatial dimensions are marginalized out in the computation of batch moments. For clarity, we consider a simplified version of BN. We ignore the variables b and c in equation 1 responsible for a downstream channel-wise affine transformation. Ignoring b and c is done without loss of generality, since the main observation in this work will focus on the Gaussian normalization and remains agnostic to downstream computations. We also ignore a numerical stability hyperparameter.We examine back-propagation of partial derivatives through this normalization, where µ and σ are viewed as functions of x. Notably, µ and σ are functions of each x i, and thus the division by σ is not affine. We write the normalized output as DISPLAYFORM1 We review ordinary least squares of a single variable with intercept BID2.Let g j = α + βz j + j where α and β are parameters, z and g are observations. z j and g j are entries in z and g respectively. j are i.i.d. Gaussian residuals. We wish to fit α and β α,β = arg min DISPLAYFORM2 The least-squares problem in equation 3 is satisfied byβ = Cov(z, g) DISPLAYFORM3 When z are normalized activations and g are partial derivatives, then Ez = 0 and Var(z) = 1. In this special case, the solution simplifies intô DISPLAYFORM4 Theorem 1 (Normalization gradients are least-squares residuals). Let i ∈ {1 . . . N} be indices over some set of activations {x i}. Then the moment statistics are defined by µ = N i=1x i N and DISPLAYFORM5 Let L be a function dependent on the normalized activations z i defined for each j by z j = (x j − µ) σ. Then, the gradients of L satisfy, for all j ∈ {1, . . ., N}, the following: DISPLAYFORM6 where DISPLAYFORM7 Proof: Normalization gradients are least-squares residuals. The proof involves a derivation of partial derivatives by repeated applications of the chain rule and rules of total derivative. Because {z i} normalized over i has mean 0 and variance 1, the partial derivatives can be rearranged to satisfy the single variable ordinary least squares framework. Fix j. We expand ∂L ∂x j as a linear combination of DISPLAYFORM8 We state ∂z i ∂x j directly. Steps are in Appendix A under Lemma 1. DISPLAYFORM9 Through substitution of equations 10 into 9, we get DISPLAYFORM10 Noting that {z i} normalized over i has mean 0 and variance 1, we recoverβ andα, in the sense of equations 4 and 5, from equation 13. DISPLAYFORM11 Finally, we rearrange equations 15 and 14 into 13 to conclude, as desired, DISPLAYFORM12 During back-propagation of a single batch, the normalization function takes in partial derivatives ∂L ∂z (·), and removes that which can be explained by least squares of ∂L ∂z (·) against z (·). As illustrated in Figure 1, during back-propagation, the residual then divides away σ to become ∂L ∂x (·), the gradient for the unnormalized activations. CALCULATIONS BN aims to control its output to have mean near 0 and variance near 1, normalized over the dataset; this is related to the original explanation termed internal covariate shift BID10. Most existing work that improve or re-purpose BN have focused on describing the distribution of activations. Definition 1. In the context of normalization layers inside a neural network, activations are split into partitions, within which means and variances are computed. We refer to these partitions as normalization partitions. Definition 2. Within the context of a normalization partition, we refer to the moments calculated on the partitions as partition statistics. Theorem 1 shows that BN has least squares fitting built into the gradient computation. Gradients of the activations being normalized in each batch moment calculation are fit with a single-variable with-intercept least squares model, and only a rescaled residual is kept during back-propagation. We emphasize that the data on which the regression is trained and applied is a subset of empirical activations within a batch, corresponding to the normalization partitions of BN.To show extensibility, we recast several popular normalization techniques into the gradient-leastsquares view. We refer to activations arising from a single member of a particular batch as an item. BHW C refers to dimensions corresponding to items, height, width, and channels respectively. In non-image applications or fully connected layers, H and W are 1. BN marginalizes out the items and spatial dimensions, but statistics for each channel are kept separate. In the subsequent sections, we revisit several normalization methods from the perspective of the gradient. FIG1 reviews the normalization partitions of these methods, and places our main theorem about gradient-least-squares into context. BID0 introduced Layer Normalization (LN) in the context of large LSTM models and recurrent networks. Only the (H, W, C) dimensions are marginalized in LN, whereas BN marginalizes out the (B, H, W) dimensions. In our regression framework, the distinction can be understood as changing the data point partitions in which least squares are fit during back-propagation. LN marginalizes out the channels, but computes separate statistics for each batch item. To summarize, the regression setup in the back-propagation of LN is performed against other channels, rather than against other batch items. introduced Instance Normalization (IN) in the context of transferring styles across images. IN is is closely related to contrast normalization, an older technique used in image processing. IN emphasizes end-to-end training with derivatives passing through the moments. Only the (H, W) dimensions are marginalized in IN, whereas BN marginalizes (B, H, W) dimensions. In our framework, this can be understood as using fewer data points and a finer binning to fit the least squares during back-propagation, as each batch item now falls into its own normalization partition. introduced Group Normalization (GN) to improve performance on image-related tasks when memory constrains the batch size. Similar to LN, GN also marginalizes out the (H, W, C) dimensions in the moment computations. The partitions of GN are finer: the channels are grouped into disjoint sub-partitions, and the moments are computed for each sub-partition. When the number of groups is one, GN reduces to LN.In future normalization methods that involve normalizing with respect to different normalization partitions; such methods can pattern match with BN, LN, IN, or GN; the back-propagation can be formulated as a least-squares fit, in which the partial derivatives at normalized activations ∂L ∂z (·) are fitted against the normalized z (·), and then the residual of the fit is rescaled to become ∂L ∂x (·).Figure 2 summarize the normalization partitions for BN, LN, IN, and GN; the figure visualizes, as an example, a one-to-one correspondence between an activation in BN, and a data point in the gradient regression. Theorem 1 is agnostic to the precise nature of how activations are partitioned before being normalized; thus, equation 9 applies directly to any method that partitions activations and performs Gaussian normalization on each partition. DISPLAYFORM0 DISPLAYFORM1 The L2 normalization of weights in WN appears distinct from the Gaussian normalization of activations in BN; nevertheless, WN can also be recast as a least squares regression. BID4 and improved BID5 residual mappings in ResNets. Arrows point in the direction of the forward pass. Dotted lines indicate that gradients are zero-centered and decorrelated with respect to downstream activations in the residual mapping. The improved ordering has BN coming first, and thus constrains that gradients of the residual map must be decorrelated with respect to some normalized activations inside the residual mapping. An update to the popular ResNet architecture showed that the network's residual mappings can be dramatically improved with a new ordering BID5. The improvement moved BN operations into early positions and surprised the authors; we support the change from the perspective of gradient-least-squares. FIG2 reviews the precise ordering in the two versions. BID6 provides independent empirical support for the BN-early order, in shake-shake regularization BID3 architectures. We believe that the surprise arises from a perspective that views BN only as a way to control the distribution of activations; one would place BN after a sequence of convolution layers. In the gradient-least-squares perspective, the first layer of each residual mapping is also the final calculation for these gradients before they are added back into BID0 0.9102 0.3548 the main trunk. The improved residual branch constrains the gradients returning from the residual mappings to be zero-centered and decorrelated with respect to some activations inside the branch. We illustrate this idea in FIG2. Gradient-least-squares views back-propagation in deep neural networks as a solution to a regression problem. Thus, formulations and ideas from a regression perspective would motivate improvements and alternatives to BN. We pursue and evaluate two of these ideas. BN and LN are similar to each other, but they normalize over different partitioning of the activations; in back-propagation, the regressions occur respectively with respect to different partitions of the activations. Suppose that a BN and a LN layer are chained serially in either order. This in a two-step regression during back-propagation; in reversed order, the residual from the first regression is further explained by a second regression on a different partitioning. In principle, whether this helps would depend on the empirical characteristics of the gradients encountered during training. The second regression could further decorrelate partial gradients from activations. Empirically, we show improvement in a reference ResNet-34-v2 implementation on CIFAR-10 relative to BN with batch size 128. In all cases, only a single per-channel downstream affine transformation is applied, after both normalization layers, for consistency in the number of parameters. See table 1 for CIFAR-10 validation performances. We kept all default hyperparameters from the reference implementation: learning schedules, batch sizes, and optimizer settings. BN performs less well on small batches BID9. Gradient-least-squares interprets this as gradient regressions failing on correlated data, an issue typically addressed by regularization. We pursue this idea to recover some performance on small batches by use of regularization. Our regularization uses streaming estimates of past gradients to create virtual data in the regression. This performed better than standard BN on the same batch size, but we did not recover the performance of large batches; this is consistent with the idea that regularization could not in general compensate for having much less data. See Appendix C for CIFAR-10 validation performances. DISPLAYFORM0 to rescale the contributions to the batch mean for each normalization scheme. It uses an analogous set of parameters λ k and activations w k for variances. We sketch the back-propagation of a simplified version of SN in the perspective of gradient-least-squares. We ignore both the division and downstream affine z → c · z + b. The normalization calculation inside SwN can be written as: DISPLAYFORM1 where Ω = {BN, LN, IN}. There is potentially a unique mean and variance used for each activation. Equation 19 bears similarities to the setup in Theorem 1, but we leave unresolved whether there is a gradient-least-squares regression interpretation for SN. Decorrelated Batch Normalization (DBN) ) is a generalization of BN that performs Mahalanobis ZCA whitening to decorrelate the channels, using differentiable operations. On some level, the matrix gradient equation resemble the least squares formulation in Theorem 1.Spectral Normalization (SpN) BID15 is an approximate spectral generalization of WN. For DBN and SpN, the regression interpretations remain unresolved. BN has been instrumental in the training of deeper networks BID10. Subsequent work ed in Batch Renormalization BID9, and further emphasized the importance of passing gradients through the minibatch moments, instead of a gradient-free exponential running average. In gradient-least-squares, use of running accumulators in the training forward pass would stop the gradients from flowing through them during training, and there would be no least-squares. BID5 demonstrate empirically the unexpected advantages of placing BN early in residual mappings of ResNet. Santurkar et al. FORMULA1 showed that BN makes the loss landscape smoother, and gradients more predictable across stochastic gradient descent steps. BID1 found evidence that spatial correlation of gradients explains why ResNet outperforms earlier designs of deep neural networks. BID11 proved that BN accelerates convergence on least squares loss, but did not consider back-propagation of BN as a least squares residual. BID14 has recast BN as a stochastic process, ing in a novel treatment of regularization. This work makes explicit how BN back-propagation regresses partial derivatives against the normalized activations and keeps the residual. This view, in conjunction with the empirical success of BN, suggests an interpretation of BN as a gradient regression calculation. BN and its variants decorrelate and zero-center the gradients with respect to the normalized activations. Subjectively, this can be viewed as removing systematic errors from the gradients. Our view also support empirical in literature preferring early BN placement within neural network branches. Leveraging gradient-least-squares considerations, we ran two sets of normalization experiments, applicable to large batch and small batch settings. Placing a LN layer either before or after BN can be viewed as two-step regression that better explains the residual. We show empirically on CIFAR-10 that BN and LN together are better than either individually. In a second set of experiments, we address BN's performance degradation with small batch size. We regularize the gradient regression with streaming gradient statistics, which empirically recovers some performance on CIFAR-10 relative to basic BN, on batch size two. Why do empirical improvements in neural networks with BN keep the gradient-least-squares residuals and drop the explained portion? We propose two open approaches for investigating this in future work. A first approach focuses on how changes to the gradient regression in different formulations; the two empirical experiments in our work contribute to this. A second approach examines the empirical relationships between gradients of activations evaluated on the same parameter values; we can search for a shared noisy component arising from gradients in the same normalization partition. Suppose that the gradient noise correlates with the activations -this is plausible because the population of internal activations arise from using shared weights -then normalizations could be viewed as a layer that removes systematic noise during back-propagation. In DISPLAYFORM0 Then, the partial derivatives satisfy DISPLAYFORM1 Proof. In deriving ∂z j ∂x i, we will treat the cases of when j = i and when j = i separately. We start by examining intermediate quantities of interest as a matter of convenience for later use. We define helper quantities u i = x i − µ. Note that each u j depends on all of x i via µ. Next, we write out useful identities DISPLAYFORM2 We prepare to differentiate with rule of total derivative: DISPLAYFORM3 Making use of equations 21, 22, 23 and 25, We simplify ∂σ ∂x i for any i as follows. DISPLAYFORM4 We apply the quotient rule on ∂z j ∂x i when j = i, then substitute equation 33 DISPLAYFORM5 Similarly, when i = j, inputs in batch b. In our work, we keep track of am exponential running estimates across batches, DISPLAYFORM6 DISPLAYFORM7 DISPLAYFORM8 that marginalize the (B, H, W) dimensions into accumulators of shape C. The b subscript of the outer expectation is slightly abusive notation indicating thatα * andβ * are running averages across recent batches with momentum as a hyperparameter that determines the weighting. We regularize the gradient regression with virtual activations and virtual gradients, defined as follows. We append two virtual batch items, broadcast to an appropriate shape, x + = µ b + σ b and x − = µ b − σ b. Here, µ b and σ b are batch statistics of the real activations. The concatenated tensor undergoes standard BN, which outputs the usual {z i} for the real activations, but z + = 1 and z − = −1 for the virtual items. The z + and z − do not affect the feed forward calculations, but they receive virtual gradients during back-propagation: DISPLAYFORM9 Virtual data z +, ∂L ∂z + and z −, ∂L ∂z − regularizes the gradient-least-squares regression. ∂L ∂z + and ∂L ∂z − eventually modify the gradients received by the real x i activations. The virtual data can be weighted with hyperparameters. In our experiments, we see improvements, robust to a hyperparameter cross-product search over the weightings and the momentum forα * andβ *. The momentum forα * andβ * were in {.997, .5} and the virtual item weights were in {2 i−1} i∈{0,1,2,3}. The performance of larger batches are not recovered; regularized regression could not be reasonably expected to recover the performance of regressing with more data. See table 2 for final validation performances with a reference Tensorflow ResNet-34-v2 implementation on batch size of two. The baseline evaluation with identity (no normalization) experienced noticeable overfitting in terms of cross entropy but not accuracy. The base learning rate was multiplied by 1 64 relative to the baseline rate used in runs with batch size 128."

A novel ensemble of retrieval-based and generation-based for open-domain conversation systems.

represent this text to find a much much longer article about it with say hundreds-thousands of words.

We systematically examine why knowledge distillation is crucial to the training of non-autoregressive translation (NAT) models, and propose methods to further improve the distilled data to best match the capacity of an NAT model.


Question: Represent the article for finding a single-sentence TLDR

Question: Deep neural networks trained on a wide range of datasets demonstrate impressive transferability. Deep features appear general in that they are applicable to many datasets and tasks. Such property is in prevalent use in real-world applications. A neural network pretrained on large datasets, such as ImageNet, can significantly boost generalization and accelerate training if fine-tuned to a smaller target dataset. Despite its pervasiveness, few effort has been devoted to uncovering the reason of transferability in deep feature representations. This paper tries to understand transferability from the perspectives of improved generalization, optimization and the feasibility of transferability. We demonstrate that 1) Transferred models tend to find flatter minima, since their weight matrices stay close to the original flat region of pretrained parameters when transferred to a similar target dataset; 2) Transferred representations make the loss landscape more favorable with improved Lipschitzness, which accelerates and stabilizes training substantially. The improvement largely attributes to the fact that the principal component of gradient is suppressed in the pretrained parameters, thus stabilizing the magnitude of gradient in back-propagation. 3) The feasibility of transferability is related to the similarity of both input and label. And a surprising discovery is that the feasibility is also impacted by the training stages in that the transferability first increases during training, and then declines. We further provide a theoretical analysis to verify our observations. The last decade has witnessed the enormous success of deep neural networks in a wide range of applications. Deep learning has made unprecedented advances in many research fields, including computer vision, natural language processing, and robotics. Such great achievement largely attributes to several desirable properties of deep neural networks. One of the most prominent properties is the transferability of deep feature representations. Transferability is basically the desirable phenomenon that deep feature representations learned from one dataset can benefit optimization and generalization on different datasets or even different tasks, e.g. from real images to synthesized images, and from image recognition to object detection . This is essentially different from traditional learning techniques and is often regarded as one of the parallels between deep neural networks and human learning mechanisms. In real-world applications, practitioners harness transferability to overcome various difficulties. Deep networks pretrained on large datasets are in prevalent use as general-purpose feature extractors for downstream tasks . For small datasets, a standard practice is to fine-tune a model transferred from large-scale dataset such as ImageNet to avoid over-fitting. For complicated tasks such as object detection, semantic segmentation and landmark localization, ImageNet pretrained networks accelerate training process substantially . In the NLP field, advances in unsupervised pretrained representations have enabled remarkable improvement in downstream tasks . Despite its practical success, few efforts have been devoted to uncovering the underlying mechanism of transferability. Intuitively, deep neural networks are capable of preserving the knowledge learned on one dataset after training on another similar dataset (; b; 2019). This is even true for notably different datasets or apparently different tasks. Another line of works have observed several detailed phenomena in the transfer learning of deep networks , yet it remains unclear why and how the transferred representations are beneficial to the generalization and optimization perspectives of deep networks. The present study addresses this important problem from several new perspectives. We first probe into how pretrained knowledge benefits generalization. Results indicate that models fine-tuned on target datasets similar to the pretrained dataset tend to stay close to the transferred parameters. In this sense, transferring from a similar dataset makes fine-tuned parameters stay in the flat region around the pretrained parameters, leading to flatter minima than training from scratch. Another key to transferability is that transferred features make the optimization landscape significantly improved with better Lipschitzness, which eases optimization. Results show that the landscapes with transferred features are smoother and more predictable, fundamentally stabilizing and accelerating training especially at the early stages of training. This is further enhanced by the proper scaling of gradient in back-propagation. The principal component of gradient is suppressed in the transferred weight matrices, controlling the magnitude of gradient and smoothing the loss landscapes. We also investigate a common concern raised by practitioners: when is transfer learning helpful to target tasks? We test the transferability of pretrained networks with varying inputs and labels. Instead of the similarity between pretrained and target inputs, what really matters is the similarity between the pretrained and target tasks, i.e. both inputs and labels are required to be sufficiently similar. We also investigate the relationship between pretraining epoch and transferability. Surprisingly, although accuracy on the pretrained dataset increases throughout training, transferability first increases at the beginning and then decreases significantly as pretraining proceeds. Finally, this paper gives a theoretical analysis based on two-layer fully connected networks. Theoretical consistently justify our empirical discoveries. The analysis here also casts light on deeper networks. We believe the mechanism of transferability is the fundamental property of deep neural networks and the in-depth understanding presented here may stimulate further algorithmic advances. There exists extensive literature on transferring pretrained representations to learn an accurate model on a target dataset. employed a brand-new label predictor to classify features extracted by the pre-trained feature extractor at different layers of AlexNet . showed deep features can benefit object detection tasks despite the fact that they are trained for image classification. introduced a selective joint fine-tuning scheme for improving the performance of deep learning tasks under the scenario of insufficient training data. The enormous success of the transferability of deep networks in applications stimulates empirical studies on fine-tuning and transferability. observed the transferability of deep feature representations decreases as the discrepancy between pretrained task and target task increases and gets worse in higher layers. Another phenomenon of catastrophic forgetting as discovered by describes the loss of pretrained knowledge when fitting to distant tasks. delved into the influence of ImageNet pretrained features by pretraining on various subsets of the ImageNet dataset. further demonstrated that deep models with better ImageNet pretraining performance can transfer better to target tasks. As for the techniques used in our analysis, Li et al. (2018a) proposed the impact of the scaling of weight matrices on the visualization of loss landscapes. proposed to measure the variation of loss to demonstrate the stability of loss function. provided a powerful framework of analyzing two-layer over-parametrized neural networks, with elegant and no strong assumptions on input distributions, which is flexible for our extensions to transfer learning. A basic observation of transferability is that tasks on target datasets more similar to the pretrained dataset have better performance. We delve deeper into this phenomenon by experimenting on a variety of target datasets (Figure 1), carried out with two common settings: 1) train only the last layer by fixing the pretrained network as the feature extractor and 2) train the whole network by fine-tuning from the pretrained representations. Results in Table 1 clearly demonstrate that, for both settings and for all target datasets, the training error converges to nearly zero while the generalization error varies significantly. In particular, a network pretrained on more similar dataset tends to generalize better and converge faster on the target dataset. A natural implication is that the knowledge learned from the pretrained networks can only be preserved to different extents for different target datasets. We substantiate this implication with the following experiments. To analyze to what extent the knowledge learned from pretrained dataset is preserved, for the fixing setting, we compute the Frobenius norm of the deviation between fine-tuned weight W and pretrained weight W 0 as 1 √ n W − W 0 F, where n denotes the number of target examples (for the fine-tuning setting, we compute the sum of deviations in all layers Results are shown in Figure 2 . It is surprising that although accuracy may oscillate, (a) Figure 2: The deviation of the weight parameters from the pretrained ones in the transfer process to different target datasets. For all datasets, can be preserved on target datasets more similar to ImageNet, yielding smaller Why is preserving pretrained knowledge related to better generalization? From the experiments above, we can observe that models preserving more transferred knowledge (i.e. yielding smaller It is reasonable to hypothesize that 1 √ n W −W 0 F is implicitly bounded in the transfer process, and that the bound is related to the similarity between pretrained and target datasets (We will formally study this conjecture in the theoretical analysis). Intuitively, a neural network attempts to fit the training data by twisting itself from the initialization point. For similar datasets the twist will be mild, with the weight parameters staying closer to the pretrained parameters. Such property of staying near the pretrained weight is crucial for understanding the improvement of generalization. Since optimizing deep networks inevitably runs into local minima, a common belief of deep networks is that the optimization trajectories of weight parameters on different datasets will be essentially different, leading to distant local minima. To justify whether this is true, we compare the weight matrices of training from scratch and using ImageNet pretrained representations in Figure 4. Results are quite counterintuitive. The local minima of different datasets using ImageNet pretraining are closed to each other, all concentrating around ImageNet pretrained weight. However, the local minima of training from scratch and ImageNet pretraining are way distant, even on the same dataset.. Surprisingly, weight matrices on the same dataset may be distant at convergence when using different initializations. On the contrary, even for discrepant datasets, the weight matrices stay close to the initialization when using the same pretrained parameters. This provides us with a clear picture of how transferred representations improve generalization on target datasets. Rich studies have indicated that the properties of local minima are directly related to generalization . Using pretrained representations restricts weight matrices to stay near the pretrained weight. Since the pretrained dataset is usually sufficiently large and of high-quality, transferring their representations will lead to flatter minima located in large flat basins. On the contrary, training from scratch may find sharper minima. To observe this, we adopt filter normalization (a) as the visualization tool, and illustrate the loss landscapes around the minima in Figure 3. This observation concurs well with the experiments above. The weight matrices for datasets similar to pretrained ones deviate less from pretrained weights and stay in the flat region. On more different datasets, the weight matrices have to go further from pretrained weights to fit the data and may run out of the flat region. A common belief of modern deep networks is the improvement of loss landscapes with techniques such as BatchNorm and residual structures . Li et al. (2018a); validated this improvement when the model is close to convergence. However, it is often overlooked that loss landscapes can still be messy at the initialization point. To verify this conjecture, we visualize the loss landscapes centered at the initialization point of the 25th layer of ResNet-50 in Figure 5. (Visualizations of the other layers can be found in Appendix B.4.) ImageNet pretrained networks have much smoother landscape than networks trained with random initialization. The improvement of loss landscapes at the initialization point directly gives rise to the acceleration of training. Concretely, transferred features help ameliorate the chaos of loss landscape with improved Lipschitzness in the early stages of training. Thus, gradient-based optimization method can easily escape from the initial region where the loss is very large. The properties of loss landscapes influence the optimization fundamentally. In randomly initialized networks, going in the direction of gradient may lead to large variation in the loss function. On the contrary, ImageNet pretrained features make the geometry of loss landscape much more predictable, and a step in gradient direction will lead to mild decrease of loss function. To demonstrate the impact of transferred features on the stability of loss function, we further analyze the variation of loss in the direction of gradient in Figure 6. For each step in the training process, we compute the gradient of the loss and measure how the loss changes as we move the weight matrix in that direction. We can clearly observe that in contrast to networks with transferred features, randomly initialized networks have larger variation along the gradient, where a step along the gradient leads to drastic change in the loss. Why can transferred features control the magnitude of gradient and smooth the loss landscape? A natural explanation is that transferred weight matrices provide appropriate transform of gradient in each layer and help stabilize its magnitude. Note that in deep neural networks, the gradient w.r.t. each layer is computed through back-propagation by, where I k i denotes the activation of x i at layer k. The weight matrices W k function as the scaling factor of gradient in back-propagation. Basically, a randomly initialized weight matrix will multiply the magnitude of gradient by its norm. In pretrained weight matrices, situation is completely different. To delve into this, we decompose the gradient into singular vectors and measure the projections of weight matrices in these principal directions. Results are shown in Figure 7 (c). During pretraining, the singular vectors of the gradient with large singular values are shrunk in the weight matrices. Thus, the magnitude of gradient back-propagated through a pretrained layer is controlled. In this sense, pretrained weight matrices stabilize the magnitude of gradient especially in lower layers. We visualize the magnitude and scaling of gradient of different layers in ResNet-50 in Figure 7. The gradient of randomly initialized networks grows fast with layer numbers during back-propagation while the gradient of ImageNet pretrained networks remains stable. Note that ResNet-50 already incorporates BatchNorm and skip-connections to improve the gradient flow, and pretrained representations can stabilize the magnitude of gradient substantially even in these modern networks. We complete this analysis by visualizing the change of landscapes during back-propagation in Section B.4. Transferring from pretrained representations boosts performance in a wide range of applications. However, as discovered by; , there still exist cases when pretrained representations provide no help for target tasks or even downgrade test accuracy. Hence, the conditions on which transfer learning is feasible is an important open problem to be explored. In this section, we delve into the feasibility of transfer learning with extensive experiments, while the theoretical perspectives are presented in the next section. We hope our analysis will provide insights into how to adopt transfer learning by practitioners. As a common practice, people choose datasets similar to the target dataset for pretraining. However, how can we determine whether a dataset is sufficiently similar to a target dataset? We verify with experiments that the similarity depends on the nature of tasks, i.e. both inputs and labels matter. Varying input with fixed labels. We randomly sample 600 images from the original SVHN dataset, and fine-tune the MNIST pretrained LeNet to this SVHN subset. For comparison, we pretrain other two models on MNIST with images upside down and Fashion-MNIST , respectively. Note that for all three pretrained models, the dataset sizes, labels, and the number of images per class are kept exactly the same, and thus the only difference lies in the image pixels themselves. Results are shown in Figure 8 (a). Compared to training from scratch, MNIST pretrained features improve generalization significantly. Upside-down MNIST shows slightly worse generalization performance than the original one. In contrast, fine-tuning from Fashion-MNIST barely improves generalization. We also compute the deviation from pretrained weight of each layer. The weight matrices and convolutional kernel deviation of Fashion-MNIST pretraining show no improvement over training from scratch. A reasonable implication here is that choosing a model pretrained on a more similar dataset in the inputs yields a larger performance gain. Varying labels with fixed input. We train a ResNet-50 model on Caltech-101 and then fine-tune it to Webcam . We train another ResNet-50 to recognize the color of the upper part of Caltech-101 images and fine-tune it to Webcam. Results in Figure 8 (b) indicate that the latter one provides no improvement over training on Webcam from scratch, while pretraining on standard Caltech-101 significantly boosts performance. Models generalizing very well on similar images are not transferable to the target dataset with totally different labels. These experiments challenge the common perspective of similarity between datasets. The description of similarity using the input (images) themselves is just one point. Another key factor of similarity is the relationship between the nature of tasks (labels). This observation is further in line with our theoretical analysis in Section 6. Currently, people usually train a model on ImageNet until it converges and use it as the pretrained parameters. However, the final model do not necessarily have the highest transferability. To see this, we pretrain a ResNet-50 model on Food-101 and transfer it to CUB-200, with shown in Figure 9. During the early epochs, the transferability increases sharply. As we continue pretraining, although the test accuracy on the pretraining dataset continues increasing, the test accuracy on the target dataset starts to decline, indicating downgraded transferability. Intuitively, during the early epochs, the model learns general knowledge that is informative to many datasets. As training goes on, however, the model starts to fit the specific knowledge of the pretrained dataset and even fit noise. Such dataset-specific knowledge is usually detrimental to the transfer performance. This interesting finding implies a promising direction for improving the de facto pretraining method: Instead of seeking for a model with higher accuracy only on the pretraining dataset, a more transferable model can be pretrained with appropriate epochs such that the fine-tuning accuracies on a diverse set of target tasks are advantageous. Algorithms for pretraining should take this point into consideration. We have shown through extensive empirical analysis that transferred features exert a fundamental impact on generalization and optimization performance, and provided some insights for the feasibility of transfer learning. In this section, we analyze some of our empirical observations from a theoretical perspective. We base our analysis on two-layer fully connected networks with ReLU activation and sufficiently many hidden units. Our theoretical are in line with the experimental findings. Denote by σ(·) the ReLU activation function, σ (z) = max{z, 0}. I{A} is the indicator function, i.e. I{A} = 1 if A is true and 0 otherwise. [m] is the set of integers ranging from 1 to m. Consider a two-layer ReLU network of m hidden units and W = (w 1, · · ·, w m) ∈ R d×m as the weight matrix. We are provided with n Q samples {x Q,i, y Q,i} n Q i=1 drawn i.i.d. from the target distribution Q as the target dataset and a weight matrix W(P) pretrained on n P samples {x P,i, y P,i} n P i=1 drawn i.i.d. from pretrained distribution P. Suppose x 2 = 1 and |y| ≤ 1. Our goal is transferring the pretrained W(P) to learn an accurate model W(Q) for the target distribution Q. When training the model on the pretraining dataset, we initialize the weight as: w r ∼ N (0, κ 2 I), a r ∼unif ({−1, 1}), where ∀r ∈ [m] and κ is a constant. For both pretraining and fine-tuning, the objective function of the model is the squared loss Note that a is fixed throughout training and W is updated with gradient descent. The learning rate is set to η. We base our analysis on the theoretical framework of , since it provides elegant on convergence of two-layer ReLU networks without strong assumptions on the input distributions, facilitating our extension to the transfer learning scenarios. In our analysis, we use the Gram matrices H ∞ P ∈ R n P ×n P and H ∞ Q ∈ R n Q ×n Q to measure the quality of pretrained input and target input as To quantify the relationship between pretrained input and target input, we define the following Gram matrix H ∞ P Q ∈ R n P ×n Q across samples drawn from P and Q: Assume Gram matrices H ∞ P and H ∞ Q are invertible with smallest eigenvalue λ P and λ Q greater than zero. H ∞ P −1 y P characterizes the labeling function of pretrained tasks. y P →Q H ∞ P Q H ∞ P −1 y P further transforms the pretrained labeling function to the target labels. A critical point in our analysis is y Q − y P →Q, which measures the task similarity between target label and transformed label. To analyze the Lipschitzness of loss function, a reasonable objective is the magnitude of gradient, which is a direct manifestation of the Lipschitz constant. We analyze the gradient w.r.t. the activations. For the magnitude of gradient w.r.t. the activations, we show that the Lipschitz constant is significantly reduced when the pretrained and target datasets are similar in both inputs and labels. Theorem 1 (The effect of transferred features on the Lipschitzness of the loss). Denote by X 1 the activations in the target dataset. For a two-layer networks with sufficiently large number of hidden unit m defined in Section 6.1, if m ≥ poly(n P, n Q, δ −1, λ −1 with probability no less than 1 − δ over the random initialization, This provides us with theoretical explanation of experimental in Section 4. The control of Lipschitz constant relies on the similarity between tasks in both input and labels. If the original target label is similar to the label transformed from the pretrained label, i.e. y Q − y P →Q 2 2 is small, the Lipschitzness of loss function will be significantly improved. On the contrary, if the pretrained and target tasks are completely different, the transformed label will be discrepant with target label, ing in larger Lipschitz constant of the loss function and worse landscape in the fine-tuned model. Recall in Section 3 that we have investigated the weight change W(Q) − W(P) F during training and point out the role it plays in understanding the generalization. In this section, we show that W(Q)−W(P) F can be bounded with terms depicting the similarity between pretrained and target tasks. Note that the Rademacher complexity of the function class is bounded with W(Q)−W(P) F as shown in the seminal work , thus the generalization error is directly related to W(Q) − W(P) F. We still use the Gram matrices defined in Section 6.1. Theorem 2 (The effect of transferred features on the generalization error). For a two-layer networks with m ≥ poly(n P, n Q, δ −1, λ −1, with probability no less than 1 − δ over the random initialization, This is directly related to the generalization error and casts light on our experiments in Section 5.1. Note that when training on the target dataset from scratch, the upper bound of By fine-tuning from a similar pretrained dataset where the transformed label is close to target label, the generalization error of the function class is hopefully reduced. On the contrary, features pretrained on discrepant tasks do not transfer to classification task in spite of similar images since they have disparate labeling functions. Another example is fine-tuning to Food-101 as in the experiment of . Since it is a fine-grained dataset with many similar images, H ∞ Q will be more singular than common tasks, ing in a larger deviation from the pretrained weight. Hence even transferring from ImageNet, the performance on Food-101 is still far from satisfactory. Why are deep representations pretrained from modern neural networks generally transferable to novel tasks? When is transfer learning feasible enough to consistently improve the target task performance? These are the key questions in the way of understanding modern neural networks and applying them to a variety of real tasks. This paper performs the first in-depth analysis of the transferability of deep representations from both empirical and theoretical perspectives. The reveal that pretrained representations will improve both generalization and optimization performance of a target network provided that the pretrained and target datasets are sufficiently similar in both input and labels. With this paper, we show that transfer learning, as an initialization technique of neural networks, exerts implicit regularization to restrict the networks from escaping the flat region of pretrained landscape. In this section, we provide details of the architectures, setup, methods of visualizations in our analysis. The codes and visualizations are attached with the submission and will be made available online. We implement all models on PyTorch with 2080Ti GPUs. For object recognition and scene recognition tasks, we use standard ResNet-50 from torchvision. ImageNet pretrained models can be found in torchvision, and Places pretrained models are provided by . During fine-tuning we use a batch size of 32 and set the initial learning rate to 0.01 with 0.9 momentum following the protocol of (b). For fine-tuning, we train the model for 200 epochs. We decay the learning rate by 0.1 with the time of decay set by cross validation. In Figure 2 (a) where the pretrained ResNet-50 functions as feature extractor, the downstream classifier is a two-layer ReLU network with Batch-Norm and Leaky-ReLU non-linearity. The number of hidden unit is 512. For this task, the backbone ResNet-50 is fixed, with the downstream two-layer classifier trained with momentum SGD. The learning rate is set to 0.01 with 0.9 momentum, and remains constant throughout training. For digit recognition tasks, we use LeNet . The learning rate is also set to 0.01, with 5 × 10 −4 weight decay. The batch-size is set to 64. We train the model for 100 epochs. Fine-tuning. We follow the protocol of fine-tuning as in the previous paragraphs. In Tables 1 and 2, we run all the experiments for 3 times and report their mean and variance. For Table 2, the improvement of fine-tuing is calculated with the generalization error of fine-tuning divided by the generalization error of training from scratch. Visualization of loss landscapes. We use techniques similar to filter normalization to provide an accurate analysis of loss landscapes (a). Note that ReLU networks are invariant to the scaling of weight parameters. To remove this scaling effect, the direction used in visualization should be normalized in a filter-wise way. Concretely, the axes of each landscape figure are two random Gaussian orthogonal vectors normalized by the scale of each filter in the convolutional layers. Concretely, suppose the parameter of the center point is θ. θ i,j denotes the j-th filter of the i-th layer. Suppose the two unit orthogonal vectors are a and b. Then with filter normalization, a i,j ← ai,j ai,j θ i,j and b i,j ← bi,j bi,j θ i,j. For each point (i.e. pixel) (p, q) in the plot, the value is evaluated with g(p, q) = L(f (θ + η(pa + qb))), where L denotes the loss function, f denotes the neural networks. η is a parameter to control the scale of the plot. In all visualization images of ResNet-50, the resolution is 200 × 200, i.e. p = −100, −99, · · ·, 98, 99 and q = −100, −99, · · ·, 98, 99. For additional details of filter normalization, please refer to (a). η is set to 0.001, which is of the same order as 10 times the step size in training. This is a reasonable scale if we want to study the local loss landscape of model using SGD. For fair comparison between the pretrained landscapes and randomly initialized landscapes, the scale of loss variation in each plot is exactly the same. The difference of loss value between each contour is 0.05. When we compute the loss landscape of one layer, the parameters of other layers are fixed. The gradient is computed based on 256 fixed samples since the gradient w.r.t. full dataset requires too much computation. Figure 3 and Figure 10 are centered at the final weight parameters, while others are centered at the initialization point to show the situation when training just starts. We visualize the loss landscapes on CUB-200, Stanford Cars and Food-101 for multiple times and reach consistent . But due to space limitation, we only show the on one dataset for each experiment in the main paper. Other are deferred to Section B. Computing the eigenvalue of Hessian. We compute the eigenvalues of Hessian with Hessianvector product and power methods based on the autograd of PyTorch. A similar implementation is provided by. We only list top 20 eigenvalues in limited space. t-SNE embedding of model parameters. We put the weight matrices of ResNet-50 in one vector as input. For faster computation, we pre-compute the distance between parameters of every two models with PyTorch and then use the distance matrix to compute the t-SNE embedding with scikitlearn. Note that we use the same ImageNet model from torchvision and the same Places model from for fine-tuning. Variation of loss function in the direction of gradient. Based on the original trajectory of training, we take steps in the direction of gradient from parameters at different steps during training to calculate the maximum changes of loss in that direction. The step size is set to the size of gradient. We take 100 steps from the original trajectories to measure the local property of loss landscapes. We aim to quantify the stability of loss functions and directly show the magnitude of gradient with this experiments on different datasets. Results on CUB-200 are provided in the main paper, with additional further provided in Section B. Not that this experiment is inspired by. We use the similar protocol as Section 3.2 in. Another protocol is to fix the step size along the gradient and compute the maximum variation of loss. Results on Stanford Cars with this protocol are provided in Section B.2. Results for both scenarios are similar. Figure 11: Variation of the loss in ResNet-50 with ImageNet pretrained weight and random initialization. We compare the variation of loss function in the direction of gradient during the training process on Stanford Cars dataset. The variation of pretrained networks is substantially smaller than the randomly initialized one, implying a more desirable loss landscape and more stable optimization. To validate that the generalization error is indeed improved with pretraining, for each dataset, we list the generalization error and the norm of deviation from the pretrained parameters in Table 2. The decreased percentage is calculated by dividing the error reduced in fine-tuning with the error of training from scratch. Compared to the of fine-tuning, we observe that ImageNet pretraining improves the generalization performance of general coarse-grained classification tasks significantly, yet the performance boost is smaller for fine-grained tasks which are dissimilar in the sense of task with ImageNet. Note that, although Stanford Cars and CUB-200 are visually similar to ImageNet, what really matters is the similarity between the nature of tasks, i.e. both images and labels matter. We visualize the loss landscape of 25-48th layers in ResNet-50 on Food-101 dataset. We compare the landscapes centered at the initialization point of randomly initialized and ImageNet pretrained networks -see Figure 12 and Figure 13. Results are in line with our observations of the magnitude of gradient in Figure 7. At higher layers, the landscapes of random initialization and ImageNet pretraining are similar. However, as the gradient is back-propagated through lower layers, the landscapes of pretrained networks remain as smooth as the higher layers. In sharp contrast, the landscapes of randomly initialized networks worsen through the lower layers, indicating that the magnitude of gradient is substantially worsened in back-propagation. Figure 12: Landscapes centered at the initialization point of each layer in ResNet-50 using ImageNet pretrained weight. The smoothness of landscapes in each layer are nearly identical, indicating a proper scaling of gradient. Figure 13: Landscapes centered at the initialization point of each layer in ResNet-50 initialized randomly. At the higher layers, the landscapes tend to be smooth. However, as the gradient is propagated to lower layers, the landscapes are becoming full of ridges and trenches in spite of the presence of Batch-Norm and skip connections. To study how transferring pretrained knowledge helps target tasks, we first study the trajectories of weight matrices during pretraining and then analyze its effect as an initialization in target tasks. Our analysis is based on's framework for over-parametrized networks. For the weight matrix W, W denotes the random initialization. W P (k) denotes W at the kth step of pretraining. W(P) denotes the pretrained weight matrix after training K steps. W Q (k) denotes the weight matrix after K steps of fine-tuning from W(P). For other terms, the notation at each step is similar. We first analyze the pretraining process on the source datasets based on. Define a matrix Z P ∈ R md×n P which is crucial to analyzing the trajectories of the weight matrix during pretraining, where I P i,j = I{w i x P,j ≥ 0}. Z P (k) denotes the matrix corresponding to W P (k). Note that the gradient descent is carried out as where vec (·) denotes concatenating a column of a matrice into a single vector. Then in the K iterations of pretraining on the source dataset, The first term is the primary component in the pretrained matrix, while the second and third terms is small under the over-parametrized conditions. Now following , the magnitude of these terms can be bounded with probability no less than 1 − δ, Here we also provide lemmas from which are extensively used later., with probability at least 1 − δ over the random initialization we have Lemma 2. If w 1,..., w m are i.i.d. generated from N (0, I), then with probability at least 1 − δ, the following holds. For any set of weight vectors w 1,..., w m ∈ R d that satisfy for any r ∈ [m], w r − w r 2 ≤ cδλ0 n 2 R for some small positive constant c, then the matrix H ∈ R n×n defined by Now we start to analyze the influence of pretrained weight on target tasks. 1) We show that during pretraining, 2) Then we analyze u Q (P)−u Q with the properties of H ∞ P Q. 3) Standard calculation shows the magnitude of gradient relates closely to u Q (P) − u Q, and we are able to find out how is the magnitude of gradient improved. To start with, we analyze the properties of the matrix H ∞ P Q. We show that under over-parametrized conditions, H ∞ P Q is close to the randomly initialized Gram matrix Z P Z Q (P). Use H P Q to denote Z P Z Q, and H P Q (P) to denote Z P Z Q (P). Lemma 3. With the same condition as lemma 1, with probability no less than 1 − δ, where with a small c. Since w r is independent of x Q,i and x Q,i 2 = 1, the distribution of w r x Q,i and w r are the same Gaussian. Applying Markov's inequality, and noting that we have with probability no less than 1 − δ, Also note that E[H P Q,ij] = H ∞ P Q,ij. By Hoeffding's inequality, we have with probability at least 1 − δ, Combining equation 12 and equation 11, we have with probability at least 1 − δ, Denote by u Q (P), u Q the output on the target dataset using weight matrix W(P) and W 0 respectively. First, we compute the gradient with respect to the activations, It is obvious from equation 14 that u Q (P) − u Q should become the focus of our analysis. To calculate u Q (P) − u Q, we need to sort out how the activations change by initializing the target networks with W(P) instead of W. For each x Q,i, divide r into two sets to quantify the change of variation in activations on the target dataset. where For r in S i, we can estimate the size of S i. Note that, since the distribution of w r is Gaussian with mean 0 and covariance matrix κ 2 I. Therefore, taking sum over all i and m and using Markov inequality, with probability at least 1 − δ over the random initialization we have Thus, this part of activations is the same for W and W(P) on the target dataset. For each x Q,i, where The first term is the primary part, while we can show that the second and the third term can be bounded with where 1 and 2 correspond to each of the second term and third term in equation 18. Thus, using lemma 1 and the estimation of |S i |, with probability no less than 1 − δ, Now equipped with equation 6, equation 19, equation 20 and lemma 3, we are ready to calculate exactly how much pretrained wight matrix W(P) help reduce the magnitude of gradient over W,, and u Q 2 are all small values we have estimated above. Therefore, using Z P F ≤ √ n P and we can control the magnitude of the perturbation terms under over-parametrized conditions. Concretely, with probability at least 1 − δ over random initialization, Substituting these estimations into equation 21 completes the proof of Theorem 1. In this subsection, we analyze the impact of pretrained weight matrix on the generalization performance. First, we show that a model will converge if initialized with pretrained weight matrix. Based on this, we further investigate the trajectories during transfer learning and bound W − W(P) F with the relationship between source and target datasets. we set the number of hidden nodes m = Ω, and the learning then with probability at least 1 − δ over the random initialization we have for k = 0, 1, 2,... The following lemma is a direct corollary of Theorem 3 and lamma 1, and is crucial to analysis to follow. Lemma 4. Under the same conditions as Theorem 3, with probability at least 1 − δ over the random initialization we have ∀r ∈ [m], ∀k ≥ 0, We have the estimation of w Q,r (k) − w r 2 from lemma 1. From w Q,r (k) − w r 2 ≤ w Q,r (k) − w r (P) 2 + w r (P) − w r 2, we can proove lemma 4 by estimating w Q,r (k) − w r (P) 2., and u Q (P) − u Q = Z Q (Z PH ∞ P −1 y P + ) + 1 + 2. Substituting lemma 3, equation 6, and equation 22 into u Q (P) − u Q 2 completes the proof. Now we start to prove Theorem 3 by induction. We have the following corollary if condition 1 holds, Corollary 1. If condition 1 holds for k = 0,..., k, for every r ∈ [m], with probability at least 1 − δ, w Q,r (k) − w r 2 ≤ 4 √ n P y P − u P 2 √ mλ P + 4 √ n Q y Q − u Q (P) 2 √ mλ Q R. If k = 0, by definition Condition 1 holds. Suppose for k = 0,..., k, condition 1 holds and we want to show it still holds for k = k + 1. The strategy is similar to the proof of convergence on training from scratch. By classifying the change of activations into two categories, we are able to deal with the ReLU networks as a perturbed version of linear regression. We define the event A ir = w r x Q,i ≤ R, where R = where we notice max k>0 k 1 − The first term is the primary part, while the second and the third are considered perturbations and could be controlled using lemma 6 and equation 38. since Z Q (k) − Z Q F is bounded, the maximum eigenvalue of H ∞ Q is λ −1 which completes the proof.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

Understand transferability from the perspectives of improved generalization, optimization and the feasibility of transferability.

represent the tldr for finding the corresponding scientific article

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.