Question: Recent few-shot learning algorithms have enabled models to quickly adapt to new tasks based on only a few training samples. Previous few-shot learning works have mainly focused on classification and reinforcement learning. In this paper, we propose a few-shot meta-learning system that focuses exclusively on regression tasks. Our model is based on the idea that the degree of freedom of the unknown function can be significantly reduced if it is represented as a linear combination of a set of sparsifying basis functions. This enables a few labeled samples to approximate the function. We design a Basis Function Learner network to encode basis functions for a task distribution, and a Weights Generator network to generate the weight vector for a novel task. We show that our model outperforms the current state of the art meta-learning methods in various regression tasks. Regression deals with the problem of learning a model relating a set of inputs to a set of outputs. The learned model can be thought as function y = F (x) that gives a prediction y ∈ R dy given input x ∈ R dx where d y and d x are dimensions of the output and input respectively. Typically, a regression model is trained on a large number of data points to be able to provide accurate predictions for new inputs. Recently, there have been a surge in popularity on few-shot learning methods (; ;). Few-shot learning methods require only a few examples from each task to be able to quickly adapt and perform well on a new task. These few-shot learning methods in essence are learning to learn i.e. the model learns to quickly adapt itself to new tasks rather than just learning to give the correct prediction for a particular input sample. In this work, we propose a few shot learning model that targets few-shot regression tasks. Our model takes inspiration from the idea that the degree of freedom of F (x) can be significantly reduced when it is modeled a linear combination of sparsifying basis functions. Thus, with a few samples, we can estimate F (x). The two primary components of our model are (i) the Basis Function Learner network which encodes the basis functions for the distribution of tasks, and (ii) the Weights Generator network which produces the appropriate weights given a few labelled samples. We evaluate our model on the sinusoidal regression tasks and compare the performance to several meta-learning algorithms. We also evaluate our model on other regression tasks, namely the 1D heat equation tasks modeled by partial differential equations and the 2D Gaussian distribution tasks. Furthermore, we evaluate our model on image completion as a 2D regression problem on the MNIST and CelebA data-sets, using only a small subset of known pixel values. To summarize, our contributions for this paper are: • We propose to address few shot regression by linear combination of a set of sparsifying basis functions. • We propose to learn these (continuous) sparsifying basis functions from data. Traditionally, basis functions are hand-crafted (e.g. Fourier basis). • We perform experiments to evaluate our approach using sinusoidal, heat equation, 2D Gaussian tasks and MNIST/CelebA image completion tasks. An overview of our model as in meta-training. Our system learns the basis functions Φ that can in sparse representation for any task drawn from a certain task distribution. The basis functions are encoded in the Basis Function Learner network. The system produces predictions for a regression task by generating a weight vector, w for a novel task, using the Weights Generator network. The prediction is obtained by taking a dot-product between the weight vector and learned basis functions. Regression problems has long been a topic of study in the machine learning and signal processing community . Though similar to classification, regression estimates one or multiple scalar values and is usually thought of as a single task problem. A single model is trained to only perform regression on only one task. Our model instead reformulates the regression problem as a few-shot learning problem, allowing for our model to be able to perform regressions of tasks sampled from the same task distribution. The success achieved by deep neural networks heavily relies on a large amount of data, especially labelled ones. As labelling data is time-consuming and labor-intensive, learning from limited labelled data is drawing more and more attention. A prominent approach is meta learning. Meta learning, also referred as learning to learn, aims at learning an adaptive model across different tasks. Meta learning has shown potential in style transfer , visual navigation , etc. Meta learning has also been applied to few-shot learning problems, which concerns models that can learn from prior experiences to adapt to new tasks. proposed the one-shot classification problem and introduced the Omniglot data set as a few-shot classification data set, similar to MNIST for traditional classification. Since then, there has been a surge of meta learning methods striving to solve few-shot problems. Some meta learning approaches learn a similarity metric (; ;) between new test examples with few-shot training samples to make the prediction. The similarity metric used here can be Euclidean distance, cosine similarity or more expressive metric learned by relation networks . On the other hand, optimization-based approaches learn how to optimize the model directly. learned an optimal initialization of models for different tasks in the same distribution, which is able to achieve good performance by simple gradient descent. learned how to perform gradient descent in the latent space to adapt the model parameters more effectively. employed an LSTM to learn an optimization algorithm. Generative models are also proposed to overcome the limitations ed from few-shot setting; ). Few-shot regression tasks are used among various few-shot leaning methods (; ;). In most existing works, these experiment usually does not extend beyond the sinusoidal and linear regression tasks. A prominent family of algorithms that tackles a similar problem as few-shot regression is Neural Processes (b; a;). Neural Processes algorithms model the distributions of the outputs of regression functions using Deep Neural Networks given pairs of input-output pairs. Similar to Variational Autoencoders , Neural Processes employ a Bayesian approach in modelling the output distribution of regression function using an encoder-decoder architecture. Our model on the other hand employs a deterministic approach where we directly learn a set of basis functions to model the output distribution. Our model also does not produce any latent vectors but instead produces predictions via a dot product between the learned basis functions and weight vector. Our experiment show that our model (based on sparse linear combination of basis functions) compares favorably to Neural Processes (based on conditional stochastic processes). Our proposed sparse linear representation framework for few shot regression makes the few shot regression problem appears to be similar to another research problem called dictionary learning (DL) , which focuses on learning dictionaries of atoms that provide efficient representations of some class of signals. However the differences between DL and our problem are significant: Our problems are continuous rather than discrete as in DL, and we only observe a very small percentage of samples. Detailed comparison with DL is discussed in the appendix. 3 PROPOSED METHOD We first provide problem definition for few-shot regression. We aim at developing a model that can rapidly regress to a variety of equations and functions based on only a few training samples. We assume that each equation we would like to regress is a task T i sampled from a distribution p(T). We train our model on a set of training tasks, S train, and evaluate it on a separate set of testing tasks, S test. Unlike few-shot classification tasks, the tasks distribution p(T) is continuous for regression task in general. Each regression task is comprised of training samples D train and validation samples D val, for both the training set S train and testing set S test, D train is comprised of K training samples and labels D train = {(x Here we discuss our main idea. We would like to model the unknown function y = F (x) given only D train = {(x k t, y k t)|k = 1...K}. With small K, e.g. K = 10, this is an ill-posed task, as F (x) can take any form. As stated before, we assume that each function we would like to regress is a task T i drawn from an unknown distribution p(T). To simplify discussion, we assume scalar input and scalar output. Our idea is to learn sparse representation of the unknown function F (x), so that a few samples {(x k t, y k t)|k = 1...K} can provide adequate information to approximate the entire F (x). Specifically, we model the unknown function F (x) as a linear combination of a set of basis functions {φ i (x)}: Many handcrafted basis functions have been developed to expand F (x). For example, the Maclaurin series expansion (Taylor series expansion at x = 0) uses {φ i (x)} = {1, x, x 2, x 3, ...}: If F (x) is a polynomial, can be a sparse representation, i.e. only a few non-zero, significant w i, and most w i are zero or near zero. However, if F (x) is a sinusoid, it would require many terms to represent F (x) adequately, e.g.: In, M is large and M K. Given only K samples {(x k t, y k t)|k = 1...K}, it is not adequate to determine {w i} and model the unknown function. On the other hand, if we use the Fourier basis instead, i.e., {φ i (x)} = {1, sin(x), sin(2x),..., cos(x), cos(2x),...}, clearly, we can obtain a sparse representation: we can adequately approximate the sinusoid with only a few terms. Under Fourier basis, there are only a few non-zero significant weights w i, and K samples are sufficient to estimate the significant w i and approximate the function. Essentially, with a sparsifying basis {φ i (x)}, the degree of freedom of F (x) can be significantly reduced when it is modeled using, so that K samples can well estimate F (x). Our approach is to use the set of training tasks drawn from p(T) to learn {φ i (x)} that in sparse representation for any task drawn from p(T). The set of {φ i (x)} is encoded in the Basis Function Learner Network that takes in x and outputs T. In our framework, Φ(x) is the same for any task drawn from p(T), as it encodes the set of {φ i (x)} that can sparsely represent any task from p(T). We further learn a Weights Generator Network to map the K training samples of a novel task to a constant vector w = [w 1, w 2, ..., w M] T. The unknown function is modeled as w T Φ(x). An overview of our model is depicted in Figure 1. Given a regression task T with The model is then applied to make prediction for any input x. During meta-training, the validation set D val = {x n p, y n p |n = 1...N} for a task T is given. The prediction is produced by taking a dot product between task-specific weights vector, w and the set of learned basis functions: To train our model, we design a loss function L that consists of three terms. The first term is a mean-squared error between the validation set labels y n p ∈ D val and the predicted y n pred. We also add two penalty terms on the weights vector w generated for each task. The first penalty term is on the L1 norm of the generated weight vectors. This is to encourage the learned weight vectors to be sparse in order to approximate the unknown function with a few significant basis functions. The second penalty term is on the L2 norm of the generated weights vector. This is used to reduce the variance of the estimated weights as commonly used in regression . The full loss function L is as follows: where λ 1 and λ 2 represents the weightage of the L1 and L2 terms respectively. Note that, it turns out that our loss function for meta learning is is similar to that of the Elastic Net Regression with both L1 and L2 regularization terms. However, the difference is significant: Instead of focusing on a single regression task as in , we use this loss function to learn (i) the parameter θ for the Basis Function Learner network, which encodes the sparsifying basis functions for any task drawn from a task distribution, and (ii) the parameter ψ for the Weight Generator network, which produces the weights for any novel task drawn from the same task distribution. 1.13 ± 0.18 0.77 ± 0.11 0.48 ± 0.08 Meta-SGD 0.90 ± 0.16 0.53 ± 0.09 0.31 ± 0.05 EMAML (small) 0.885 ± 0.117 0.615 ± 0.091 0.371 ± 0.048 EMAML (large) 0.783 ± 0.101 0.537 ± 0.079 0.307 ± 0.040 BMAML (small) 0.927 ± 0.116 0.735 ± 0.104 0.459 ± 0.058 BMAML (large) 0.878 ± 0.108 0.675 ± 0.094 0.442 ± 0.055 NP (b) 0.640 ± 0.205 0.561 ± 0.234 0.421 ± 0.088 CNP (a) 0.910 ± 0.234 0.630 ± 0.222 0.393 ± 0.145 ANP 0.488 ± 0.188 0.216 ± 0.082 0.095 ± 0.068 Ours (small) 0.363 ± 0.018 0.169 ± 0.007 0.076 ± 0.004 Ours (large) 0.199 ± 0.010 0.062 ± 0.003 0.027 ± 0.002 In this section we describe the experiments we ran and introduce the types of regression task used to evaluate our method. For all of our experiments, we set the learning rate to 0.001 and use the Adam Optimizer as the optimization method to preform stochastic gradient decent on our model. We implement all our models using the Tensorflow library. In the following subsections, we decribe each of experiments in more detail. We include the experiments on the 1D Heat Equation and 2D Gaussian regression tasks in the appendix. For all 1D Regression tasks, the Basis Function Learner consists of two fully connected layers with 40 hidden units. For the loss function we set λ 1 = 0.001 and λ 2 = 0.0001. Sinusoidal Regression. We first evaluate our model on the sinusoidal regression task which is a few-shot regression task that is widely used by other few-shot learning methods as a few-shot learning task to evaluate their methods on (; ;). The target function is defined as y(x) = Asin(ωx + b), where amplitude A, phase b, frequency ω are the parameters of the function. We follow the setup exactly as in . We sample the each parameters uniformly from range. We train our model on tasks of batch size 4 and 60000 iterations for 5,10 and, 20 shot cases, where each training task contains K ∈ {5, 10, 20} training samples and 10 validation samples. We compare our method against recent few-shot learning methods including Meta-SGD , MAML , EMAML,BMAML and the Neural Processes family of methods including Neural Processes (b) Conditional Neural Processes (a) and Attentive Neural Processes . We use the officially released code for these three methods 1. We show the in Table 1. We provide two variants our model in this experimental setup. The two models differ only in the size of the Weights Generator. For the "small" model the Weights Generator consist of B = 1 self-attention blocks followed by a fully connected layer of 40 hidden units. The self-attention block consists of three parallel weight projections of 40 dimensions followed by fully connected layers of 80 and 40 hidden units respectively. The "large" model consists of B = 3 self-attention blocks also followed by a fully connected layer of 40 hidden units. Each self-attention block has weight projections of 64 dimensions followed by fully connected layers of 128 and 64 hidden units respectively. Both MAML and Meta-SGD uses an architecture of 2 fully connected layers with 40 hidden units which is similar to the architecture of the Basis Learner network, though both Meta-SGD and MAML both have additional optimization for individual tasks. The Neural Process family of methods uses encoder archtecture of 4 fully connected layers with 128 hidden units and decoder architecture of 2 fully connected layers of 128 hidden units respectively which is more similar in architecture our larger model. Similarly, we also compare our methods against two variants of EMAML and BMAML. The "small" model consist of 2 fully connected layers with 40 hidden units each while the "large" model consists of 5 fully connected layers with 40 hidden units each. This is to ensure fair comparison as both BMAML and EMAML lack a separate network to generate weight vectors but are ensemble methods that aggregate from M p number of model instances. We set the number of model instances in BMAML and EMAML to 10. Alternative Sinusoidal Regression We also evaluate our method on another version of the sinusoidal task as introduced by. The range of A remain the same while the range of b is increased to [0, 2π] and the range of ω is increased to [0.5, 2.0]. An extra noise term, is also added the function y(x). For noise, we sample it from distribution N ∼ (0, (0.01A) 2 ). We also fix the total number of our tasks used during training to 1000 as in . For this experimental setup we also include an ensemble version of our model where we train 10 separate instance of our model on the same 1000 tasks and aggregate their by taking a mean of the predictions. We evaluate our model for both 10 shot and 5 shot cases and show the mean-squared error in Table 2. For this experimental setup, we calculate the mean-squared error from 10 randomly points from 1000 advanced sinusoidal tasks. Our show that our method outperforms all recent few-shot regression methods in sinusoidal tasks. We also tested our method on more challenging image data, as done in (a; b;). We use MNIST and CelebA datasets here for qualitative and quantitative comparison. Each image can be regarded as a continuous function f: R 2 → R dy, where d y = 1 if the image is gray-scale or or d y = 3 if it is RGB. The input x ∈ R 2 to f is the normalized coordinates of pixels and the output y ∈ R dy is the normalized pixel value. The size of the images is 28 × 28 in MNIST and rescaled to 32 × 32 in CelebA. During meta-training, we randomly sample K points from 784 pixels in one image as D train and another K points as D val to form a regression task. In the meta-testing stage, the MSE is evaluated on 784−K pixels. 60,000 images are used for meta-training and 10,000 for meta-testing for MNIST(CelebA) dataset. We compare our methods with NP family: CNP (a), NP (b) and ANP for K = 50 and K = 100. Deeper network structure is adopted due to the complexity of regression on image data. Namely, we use 5 fully connected layers with 128 hidden units in Basis Function Learner and 3 attention blocks in Weight Generator for our method. The encoders and decoders in NP family are all MLPs including 4 fully connected layers with 128 hidden units. Thus, the comparison is fair in terms of network capacity. All the models are trained for 500 epochs with batch size 80. The MSE on 10,000 tasks from meta-testing set is reported with 95% confidence interval, shown in Table 3. The top are highlighted. It can be observed that our method outperforms two of three NP methods and achieves MSE very close to most recent ANP. The outputs of regression on CelebA image data are high-dimension predictions, which demonstrates the effectiveness of our method in such challenging tasks. Note that ANP significantly improves upon NP and CNP using cross-attention, which can potentially be applied to our method as well. In this subsection we provide some deeper analysis on the basis functions that are learned by our method. In particular, we provide some further evidence to our claim that our method learns a set of sparsifying basis functions that correspond to the regression tasks that we would like to model. To demonstrate the sparsity of basis functions, we take only the S largest weights in terms of |w| and their corresponding basis functions and illustrate the predicted regression function with the combination of only the S weights and basis functions. We conduct this experiment on both the sinusoidal regression task and the more difficult image completion task and show these S-weights predictions in Figures 3 and 2b respectively. The figures illustrate that our method is able to produce a good prediction of the regression function with only a fraction of the full set learned basis function (40 for the sinusoidal task, 128 for the MNIST image completion task). This demonstrates the sparsity of Φ(x) as most of the prediction is carried out by just a small number of basis functions. This also demonstrates that our method is able to force most of the information of F (x) to be contained in a few terms. Therefore, using K samples to estimate the weights of these few important terms could achieve a good approximation of F (x). In this subsection we detail some ablation studies on our model to test the validity of certain design choices of our model. In particular we focus on the effects of the addition of self-attention operations in the Weights Generator and also the effects of using different penalty terms on our loss function. To test out the effects of adding the selfattention operations to our model, we conduct a simple experiment where we replace the self attention operations in the self-attention block with just a single fully connected layer of equal dimensions as the self-attention weight projection. Essentially, this reduces the Weights Generator to be just a series of fully connected layers with residual connections and layer normalization. We compare the simpler model performance on the sinusoidal regression task as specified in Table 1 with our original model and show the in Table 4. The show that adding the self-attention operations do improve our methods performance on the 1D sinusoidal regression task. We also conducted experiments to test the effects of the different penalty terms on the the generated weights vector. In this ablation study, we compared our models trained using different variants of the. Similar to the previous study, we evaluate them on their performance on the sinusoidal regression task as specified in Table 1. The variants we tested out are: (i) Loss function with only the L1-norm penalty term; (ii) Loss function with only the L2-norm penalty term (iii) Loss function with both L1 and L2-norm penalty terms. To demonstrate the sparsity of the weights vectors of each variant, we also show the a breakdown of the magnitude of the learned weight vectors over 100 sinusoidal tasks. We group the weight vectors into three groups: |w| less than 0.02 to indicate weights that are near zero, |w| between 0.02 and 1 and weights with magnitude more than 1. We show the of the different variants in Table 5. We also present histograms of the magnitude of the learned weight vectors in Figure 4 The do show that the combination of both L1 and L2 penalty terms do ultimately give the best performance for the sinusoidal regression task. In terms of sparsity, the model trained with only the L1 loss term do gives the highest percentage of sparse weights though we found the model with both L1 and L2 terms do give a better performance while still maintaining a relatively high percentage of near zero weights. We propose a few-shot meta learning system that focuses exclusively on regression tasks. Our model is based on the idea of linear representation of basis functions. We design a Basis Function Learner network to encode the basis functions for the entire task distribution. We also design a Weight generator network to generate the weights from the K training samples of a novel task drawn from the same task distribution. We show that our model has competitive performance in in various few short regression tasks. In this section we illustrate all of the individual non-zero basis functions learned by our model for the sinusoidal task. These functions are shown Figure 5. Note that out of 40 of the basis functions, only 22 of the learned basis functions are non-zero functions, further demonstrating that indeed our method is forcing the model to learn a set of sparse functions to represent the tasks. Furthermore, it can be seen that the learned basis functions all correspond to the different "components" of sinusoidal function: most of the learned functions seem to represent possible peaks, or troughs if multiplied with a negative weight at various regions of the input range whereas the top four basis function seem to model the more complicated periodic nature of the sinusoidal functions. Adding on to the experiments in Section 4.3, we also illustrate what happens when do the exact opposite. We take the prediction using the full set of weight vectors/basis function and study the effect of the prediction when we remove certain basis function from the prediction. Similar to the previous experiment, we remove the basis function by order of magnitude starting with the basis function with the largest corresponding |w|. we conduct this experiment on the sinusoidal regression task and illustrate the in Figure 6. Similarly, this study also demonstrates the importance of certain basis functions as removing them caused the prediction the change drastically. In particular, notice that for sinusoidal task, removing just 4 of the most important basis functions ed in a less accurate prediction than using just 10 of the most important basis functions. Here we provide more details on the architecture of the Weights Generator Network. As mentioned previously in Section 3.3. The Weights Generator Network consists of a series of self attention blocks followed by a final fully connected layer. We define a self attention block as such: An attention block consists of a self attention operation on the input of self attention block. Following the self-attention operation, the ant embedding is further passed through two fully connected layers. A residual connection from the output of the self-attention operation to the output of the second fully connected layer. Finally, ant embedding of the residual connection is then passed though a a layer normalization operation . Note that the input of the first self attention block will always be the input to the Weights Generator network, (Φ(x k t), y k t ) whereas the inputs to subsequent attention blocks are the outputs of the previous attention block. For the self-attention operation, the input is transformed into query, key and value vectors though their respective weight projections. These query, key and value vectors, Q, K and V then go through a scaled dot-product self-attention operation as introduced by : We also evaluate our method on another 1D Regression task, the 1D heat Equation task, we define it as such: Consider a 1-dimensional rod of length L with both of its ends connected to heat sinks, i.e. the temperature of the ends will always be fixed at 0K unless a heat source is applied at the end. a constant point heat source is then applied to a random point s on the rod such the the heat point source will always have a temperature of 1.0K. We would like the model the temperature u(x, t) at each point of the rod a certain time t after applying the heat source until the temperature achieves equilibrium throughout the rod. The temperature at each point x after time t is given by the heat equation: ∂x 2 For our experiments, we set L to be 5 and randomly sample K points of range on the heat equation curve. We fix the total number of tasks used during training to 1000 and evaluate our model on both 10 shot and 5 shot cases, similar to the experimental setup for the Advanced Sinusoidal tasks. We also compare our to both EMAML and BMAML on this regrssion task and add an ensemble version of method for comparison. The of our evaluation is presented in Table 6. 2D Gaussian. We also evaluated our method on the for the 2D Gaussian regression tasks. For this task, we train our model to predict the probability distribution function of a two-dimensional Gaussian distribution. We train our model from Gaussian distribution task with mean ranging from (−2, −2) to and standard deviation of range [0.1, 2]. We fix the standard deviation to be of the same value in both directions. Similar to the heat equation, we use the same setup as the Advanced Sinusoidal task and compare our methods to EMAML and BMAML. We evaluate our model on 10, 20 and 50 shot case. The of our evaluation is presented in Table 7. Qualitative on CelebA datasets. We provide the qualitative on CelebA datasets in Figure 7. We note that the RGB images are complex 2D functions. We choose them to evaluate so that we can see the more directly, not to compare with image inpainting methods, which is also mentioned in (a). The in Figure 7a are consistent with Figure 2a. The regression from our method are visually better than NP and CNP. The predictions using first S largest weights are shown in Figure 7b. The 2D image function is usually predicted with less than 50 weights, which suggests that the information of the 2D function is kept in several terms. Our proposed sparse linear representation framework for few shot regression makes the few shot regression problem appears to be similar to another research problem called dictionary learning (DL), which focuses on learning dictionaries of atoms that provide efficient representations of some class of signals . However the differences between DL and our problem are significant: Our problems are continuous rather than discrete as in DL, and we only observe a very small percentage of samples. Specifically, for a given y ∈ R n, the goal of DL is to learn the dictionary (n by M) Φ for some sparse w: In typical DL, the entire y is given. Also, M > n for an overcomplete dictionary (Figure 8). In few shot regression, the goal is to predict the entire continuous function y = F (x). Therefore, viewing this as the setup in, n is infinite. Moreover, only a few (K) samples of y is given: y k t = F (x k t). The locations of the given samples (x k t) are different for different y (different task). Therefore, our problem is significantly different and more difficult than DL. Typical DL algorithms solve and return Φ, which is a n by M matrix of finite dimensions (the dictionary). In our setup, the basis matrix Φ has infinite entries, and Φ is encoded by the proposed Basis Function Learner network. GT NP CNP ANP Ours K=100

represent the tldr for finding the corresponding scientific article

We propose a method of doing few-shot regression by learning a set of basis functions to represent the function distribution.

represent the natural language

For example, 'A simple modification to low-rank factorization that improves performances (in both image and language tasks) while still being compact.' should be close to 'Modern deep neural networks have a large amount of weights, which make them difficult to deploy on computation constrained devices such as mobile phones. One common approach to reduce the model size and computational cost is to use low-rank factorization to approximate a weight matrix. However, performing standard low-rank factorization with a small rank can hurt the model expressiveness and significantly decrease the performance. In this work, we propose to use a mixture of multiple low-rank factorizations to model a large weight matrix, and the mixture coefficients are computed dynamically depending on its input. We demonstrate the effectiveness of the proposed approach on both language modeling and image classification tasks. Experiments show that our method not only improves the computation efficiency but also maintains (sometimes outperforms) its accuracy compared with the full-rank counterparts. Modern neural networks usually contain millions of parameters BID4 BID8, and they are difficult to be deployed on mobile devices with limited computation resources. To solve this problem, model compression techniques are proposed in recent years. Low-rank factorization is a popular way of reducing the matrix size. It has been extensively explored in the literature BID5 BID6 BID3 BID10. Mathematically, a large weight matrix W ∈ R m×n is factorized to two small rank-d matrices U ∈ R m×d, V ∈ R n×d with W = U V T. Since both U and V are dense, no sparsity support is required from specialized hardware. It naturally fits the general-purpose, off-the-shelf CPUs and GPUs. To significantly reduce the model size and computation, the rank d in the low-rank factorization needs to be small. However, a small rank can limit the expressiveness of the model BID9 and lead to worse performance. To understand the limitations, given a n-dim feature vector h, we observe that DISPLAYFORM0, is a linear projection from a high-dimensional space (n dims) to a low-dimensional space (d dims). This can lead to a significant loss of information. The conflict between the rank d and the model expressiveness prevents us from obtaining a both compact and accurate model. To address the dilemma, we propose to increase the expressiveness by learning an adaptive, inputdependent factorization, rather than performing a fixed factorization of a weight matrix. To do so, we use a mixture of multiple low-rank factorizations. The mixing weights are computed based on the input. This creates an adaptive linear projection from a high-dimensional space to a low-dimensional space. Compared to the conventional low-rank factorization, the proposed approach can significantly improve its performance while only introducing a small additional cost. DISPLAYFORM1 where z can be treated as the middle layer. Techniques like pooling can be applied to compute π to make it efficient. We propose to use an unnormalized learned mixture of low-rank factorizations whose mixing weights are computed adaptively based on the input. More specifically, denoting the input by h and the number of mixture components by K, we decompose a large weight matrix by DISPLAYFORM0 where π(·): R n → R K is the function which maps each input to its mixture coefficients. For example, π can be a small neural network. This introduces a small amount of extra parameters and computation. We will later discuss the details of efficient ways to implement the mixture function π. If π k, k = 1,..., K, is chosen to be constant (input independent), it can be absorbed into either DISPLAYFORM1. Thus, the proposed method reduces to the low-rank factorization. This is evidenced by rewriting DISPLAYFORM2. In other words, the conventional low-rank factorization can be considered as a special case of our method. FIG0 depicts the proposed framework. Adaptive mixing weights π(h). The mixing weights can encode important information that we can use to increase the expressiveness of the projected low-dimensional space. Under our framework, the generation of the mixing weights π(h) is flexible. A straight-forward approach is to use a non-linear transformation of the input to the weight matrix. For example, π(h) = σ(P h), where σ is a non-linear transformation, such as sigmoid or hyperbolic tangent function, and P ∈ R K×n is an extra weight matrix. This adds some extra parameters and computation to the model since the linear projection that we construct is R n → R K. To further reduce the parameter and computation in the mixing weights π, we propose the following strategies. Pooling before projection. We do not require the whole input to compute the mixture function π. Instead, we can apply pooling to the input h before projection. For example, a global average pooling can be applied if the input is a 3D tensor (for images); for a 1D vector, we can segment the vector and average each segmentations. By applying pooling, we can both save the computation and better capture the global information. Random projection. To reduce the number of parameters in the linear projection of h, we can use a random matrix P random in place of a fully adjustable P, i.e. π(h) = σ(P random h). Note that we can simply save a seed to recover the random matrix, but it still requires the same amount of memory and computation as the fully adjustable linear projection of h. Increased expressiveness. The adaptive mixing weights introduce a non-linear transformation into the high-to-low-dimensional projection that can be more expressive. Since each W (h) is a data-dependent low-rank matrix, there is no constant linear weight independent to the input (even a full-rank matrix) that can mimic the transformation W (h). Generating the whole weight matrices can be very expensive. Our method can be seen as a swift approach to generate the weights by adaptively adjusting mixing weights for the linear bottleneck. It assigns weights into groups and dynamically controls them at the group level. Recurrent neural networks for language modeling. Recurrent neural networks (RNNs) are widely used in language modeling, machine translation and sequence modeling in general. We adopt the same Long Short Term Memory (LSTM) models and follow the settings from a previous state-ofthe-art model BID11 for language modeling, and use Penn Tree Bank (PTB) as well as Text8 datasets. More specifically, we use the medium sized model introduced in BID11.We test three variants of the proposed model against regular low-rank factorization, each with different ways of computing mixing weights, namely MIX-ALL-PJ: direct linear projection of the input vector h, MIX-POOL-PJ: linear projection after segment-based mean pooling of the input vector h, and MIX-RND-PJ: use a random projection for the input vector h. Among these adaptive projection methods, MIX-ALL-PJ has a large amount of extra parameters, MIX-POOL-PJ has a small amount of extra parameters, and MIX-RND-PJ has no extra parameters. We compute the FLOPs of a single time-step of applying LSTM, and the perplexity associated to different settings. The are shown in FIG1. Firstly, with adaptive mixtures, the low-rank factorization model achieved 40% reduction in FLOPs, and even surpassed the performance of the full matrix baseline by decreasing the perplexity by 1.7 points. Secondly, the use of adaptive mixtures can significantly improve the performance compared with regular, non-adaptive low-rank factorization. Thirdly, using pooling before projection can be a good choice for computing the mixing weights π. It not only reduces the computation and parameter size, but can better capture the global information and achieve better accuracy. CNN for image recognition. We further demonstrate the effectiveness of the proposed approach on compressing CNN models on ImageNet BID1. We chose to use modern compact CNN models as the baseline (which are harder to compress), rather than using the bulky CNN models (which is easier to compress). Specifically, we choose to compress the point-wise convolution in depth-wise separable convolutions BID0, MobileNet BID2 BID7 in particular. TAB0 shows the comparison of different state-of-art compact convolutional models. We observed that compared to the regular low-rank factorization of MobileNet model (a.k.a. MobileNet V2), the proposed method achieves significantly better (2.5% and 2% for two different Low-rank MobileNet settings, respectively), while only adding negligible extra FLOPs (less than 1%).'

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent the scientific article to find a short TLDR!

Question: Large-scale pre-trained language model, such as BERT, has recently achieved great success in a wide range of language understanding tasks. However, it remains an open question how to utilize BERT for text generation tasks. In this paper, we present a novel approach to addressing this challenge in a generic sequence-to-sequence (Seq2Seq) setting. We first propose a new task, Conditional Masked Language Modeling (C-MLM), to enable fine-tuning of BERT on target text-generation dataset. The fine-tuned BERT (i.e., teacher) is then exploited as extra supervision to improve conventional Seq2Seq models (i.e., student) for text generation. By leveraging BERT's idiosyncratic bidirectional nature, distilling the knowledge learned from BERT can encourage auto-regressive Seq2Seq models to plan ahead, imposing global sequence-level supervision for coherent text generation. Experiments show that the proposed approach significantly outperforms strong baselines of Transformer on multiple text generation tasks, including machine translation (MT) and text summarization. Our proposed model also achieves new state-of-the-art on the IWSLT German-English and English-Vietnamese MT datasets. Large-scale pre-trained language model, such as ELMo , GPT and BERT , has become the de facto first encoding step for many natural language processing (NLP) tasks. For example, BERT, pre-trained with deep bidirectional Transformer via masked language modeling and next sentence prediction, has revolutionized the state of the art in many language understanding tasks, such as natural language inference and question answering . However, beyond common practice of fine-tuning BERT for language understanding, applying BERT to language generation still remains an open question. Text generation aims to generate natural language sentences conditioned on certain input, with applications ranging from machine translation , text summarization (; ;) ), to image captioning; ). In this paper, we study how to use BERT for better text generation, which to the best of our knowledge is still a relatively unexplored territory. Intuitively, as BERT is learned with a generative objective via Masked Language Modeling (MLM) during the pre-training stage, a natural assumption is that this training objective should have learned essential, bidirectional, contextual knowledge that can help enhance text generation. Unfortunately, this MLM objective is not auto-regressive, which encumbers its direct application to auto-regressive text generation in practice. In this paper, we tackle this challenge by proposing a novel and generalizable approach to distilling knowledge learned in BERT for text generation tasks. We first propose a new Conditional Masked Language Modeling (C-MLM) task, inspired by MLM but requiring additional conditional input, which enables fine-tuning pre-trained BERT on a target dataset. In order to extract knowledge from the fine-tuned BERT and apply it to a text generation model, we leverage the fine-tuned BERT as a teacher model that generates sequences of word probability logits for the training samples, and treat the text generation model as a student network, which can effectively learn from the teacher's outputs for imitation. The proposed approach improves text generation by providing a good estimation on the word probability distribution for each token in a sentence, consuming both the left and the right context, the exploitation of which encourages conventional text generation models to plan ahead. Text generation models are usually trained via Maximum Likelihood Estimation (MLE), or teacher forcing: at each time step, it maximizes the likelihood of the next word conditioned on its previous ground-truth words. This corresponds to optimizing one-step-ahead prediction. As there is no explicit signal towards global planning in the training objective, the generation model may incline to focusing on local structure rather than global coherence. With our proposed approach, BERT's looking into the future ability can act as an effective regularization method, capturing subtle long-term dependencies that ensure global coherence and in consequence boost model performance on text generation. An alternative way to leverage BERT for text generation is to initialize the parameters of the encoder or decoder of Seq2Seq with pre-trained BERT, and then fine-tuning on the target dataset. However, this approach requires the encoder/decoder to have the same size as BERT, inevitably making the final text generation model too large. Our approach, on the other hand, is modular and compatible to any text-generation model, and has no restriction on the model size (e.g., large or small) or model architecture (e.g., LSTM or Transformer). The main contributions of this work are three-fold. (i) We present a novel approach to utilizing BERT for text generation. The proposed method induces sequence-level knowledge into the conventional one-step-ahead and teacher-forcing training paradigm, by introducing an effective regularization term to MLE training loss. (ii) We conduct comprehensive evaluation on multiple text generation tasks, including machine translation, text summarization and image captioning. Experiments show that our proposed approach significantly outperforms strong Transformer baselines and is generalizable to different tasks. (iii) The proposed model achieves new state-of-the-art on both IWSLT14 German-English and IWSLT15 English-Vietnamese datasets. Pre-trained Language Models Prior to pre-trained language model, word embeddings (; ;) were widely used for NLP tasks. Recently, CoVe introduced (conditional) language models pre-trained on paired machine translation corpus. ELMo learned a contextual language model on a large corpus with bidirectional RNN. GPT used unidirectional Transformer to achieve better contextualized word representation. By fine-tuning pre-trained language models, ULMFit also achieved promising on text classification. In our study, we focus on BERT due to its superior performance on multiple language understanding tasks. However, different from previous work exploiting BERT for language understanding tasks, here we aim to apply BERT to text generation. To the best of our knowledge, this is still a relatively unexplored space. The proposed approach is also model-agnostic and can be applied to other pretrained language models as well. There has been some recent attempt on applying BERT to text generation. trained cross-lingual MLM and demonstrated promising for cross-lingual natural language inference and unsupervised neural machine translation (NMT). formulated BERT as a Markov Random Field LM and showed preliminary on unsupervised text generation with improved diversity. Zhang et al. (2019a) utilized an encoder with BERT and a two-stage decoder for text summarization. proposed Masked Seq2Seq (MASS) pre-training, demonstrating promising on unsupervised NMT, text summarization and conversational response generation. Concurrent with our work, proposed a similar conditional MLM for constant-time translation, and studied how to fine-tune BERT for NMT. Our approach is novel in the sense that we do not directly use the parameters of BERT in the Seq2Seq model. Instead, BERT acts as an effective regularization to the MLE training loss, by proactively injecting future information for predicting the present. Right-to-Left Generation Our work also shares a high-level intuition with those approaches that try to regularize left-to-right generative models with a right-to-left counterpart. Specifically, trained a separate reverse NMT and performed joint decoding at inference time to enforce agreement between forward and reverse models. Twin Networks used a backward RNN jointly trained with a forward RNN decoder by matching their hidden states. Zhang et al. (2019b) further extended the idea to Transformer with joint training, so that the forward and the backward models iteratively improve each other. Our proposed approach stems from a similar intuition. However, we focus on using pre-trained language model such as BERT to regularize an auto-regressive generation model. Knowledge Distillation Our method shares the same loss formulation as Knowledge Distillation (KD) proposed in;; , where a smaller student model is trained on soft labels provided by a larger teacher model. More recently, applied KD to multilingual proposed patient KD for BERT model compression. Compared with these previous studies, where both the teacher and the student are trained on the same task, our approach is different in the sense that the BERT teacher is not designed to perform the student's generation task. We focus on using KD to leverage the learned knowledge of BERT for text generation, while previous work mostly focused on model compression. In this section, we present our proposed approach to distilling the knowledge in BERT for text generation in generic sequence-to-sequence (Seq2Seq) setting. We first review Seq2Seq learning in Section 3.1, and then describe the proposed approach in Section 3.2 and 3.3. Seq2Seq learning aims to generate a sequence of discrete output Y = (y 1, . . ., y N) of length N, conditioned on a sequence of discrete input X = (x 1, . . ., x M) of length M. A Seq2Seq model learns parameters θ to estimate the conditional likelihood P θ (Y |X), typically trained via Maximum Likelihood Estimation (MLE), or equivalently, minimizing the cross-entropy loss as follows: where each conditional probability can be calculated via an attention-based recurrent neural network (RNN) (;, Transformer , or any other neural sequence-generation models. This generic Seq2Seq learning framework is the state of the art on a wide range of text generation tasks. Using modern deep neural networks, the conditional probabilities can be readily modeled as a sequence of classifications over the word vocabulary. However, during training, in order to generate the t-th token y t, the model only sees a partial sentence y 1:t−1 from the ground-truth training data. Intuitively, it is reasonable to assume that a bidirectional model can be more informative than a leftto-right generation model, since additional context from the right (or future) is also incorporated to predict the current word. Unfortunately, this additional information is not utilized in a standard Seq2Seq model, since it can only be trained in a left-to-right manner, where the future context is masked out to prevent each word from indirectly "seeing itself ". To compensate this singledirectional limitation of Seq2Seq setting, we propose a new conditional language model (C-MLM) to enable the fine-tuning of BERT on target generation task, in hope that the fine-tuned bidirectional BERT can be utilized for better text generation. BERT is a deep bidirectional Transformer trained via Masked Language Modeling (MLM). 2 In a similar setting, where the input is a sequence pair (X, Y), 3 15% of the tokens are randomly masked. Formally, we denote the masked token sets as X m and Y m, and the disjoint counterpart (i.e., the unmasked tokens) as X u and Y u, respectively. The trained BERT model aims to estimate the joint probability: where i and j denote the number of masked tokens in X and Y, respectively. Each x m ∈ X m, and each y m ∈ Y m. Eqn. can be trained with the standard word-level cross-entropy loss. We aim to marry MLM pre-training with Seq2Seq learning, to leverage bidirectional language model for text generation. To this end, we propose a conditional-MLM, a variant of MLM that allows further fine-tuning of pre-trained BERT on target dataset. For example, for machine translation, X and Y represent the source and the target sentence, respectively. We first concatenate them together and randomly mask 15% of the tokens only in Y, then train the network to model the joint probability The above C-MLM objective is similar to the conditional language modeling (LM) objective in Eqn., but conditional LM only permits predicting a word based on its left context. C-MLM is also related to Masked Seq2Seq (MASS) pre-training. However, in MASS, the encoder takes a sentence with randomly masked fragment (several consecutive tokens) as input, and the decoder tries to predict this masked fragment, which is different from our model design. The final goal is also different: MASS focuses on Seq2Seq pre-training, while we focus on leveraging BERT for text generation. In our experiments, we observe that the C-MLM task can obtain high accuracy and good generalization on word prediction. However, it is not feasible to generate sequential output directly from C-MLM. Instead, we use knowledge distillation to distill the knowledge learned from the fine-tuned BERT into a Seq2Seq model for direct text generation, which will be explained in the next sub-section. Our inspiration springs from the observation that the probability distribution of the masked word y m t is estimated using both y u 1:t−1 and y u t+1:N from Y u. In other words, the distribution for a given word P (y m t |X, Y u) contains information from both backward and forward contexts, which is a desirable benefit for providing sequence-level global guidance. This probability distribution can be considered as soft targets for a text generation model to mimic from, which potentially contains more useful and fine-grained information than the usual hard-assigned, one-hot label, therefore enhancing conventional left-to-right generation models to look into the future. In a knowledge distillation setting, the BERT model can be considered as a teacher, while the Seq2Seq model acts as a student. Specifically, the Seq2Seq model can be trained with the following objective function: where P φ (y t) is the soft target estimated by the fine-tuned BERT with learned parameters φ, and V denotes the output vocabulary. Note that φ is fixed during the distillation process. An illustration of this learning process is provided in Figure 1, which aims to match the word probability distribution P θ (y t) provided by the student with P φ (y t) provided by the teacher (i.e., distillation). To further improve the Seq2Seq student model, hard-assigned labels are also utilized. the final model is trained with the following compound objective: where α is a hyper-parameter for tuning the relative importance of the two training targets: soft estimation from fine-tuned BERT, and ground-truth hard label. Note that our proposed approach only has a minimal requirement on the architecture of the incorporated Seq2Seq model. As long as the model is trained to estimate word-level probability as in Eqn., it can be trained jointly with the proposed objective function Eqn.. At a higher level, the additional loss term L bidi can be interpreted as a sequence-level objective function. Our auto-regressive (or causal) model θ tries to predict the probability distribution that matches the estimation the bidirectional teacher model predicts, hence encouraging the planning of future (right context) for generation. In this section, we describe our experiments on two well-studied text generation tasks: machine translation, and abstractive text summarization. Machine Translation We consider two relatively small-scale datasets, IWSLT15 EnglishVietnamese (En-Vi, 113k training samples) and IWSLT14 German-English (De-En, 160k training samples), and one medium-scale dataset, WMT14 English-German (En-De, 4.5M training samples). For IWSLT15 En-Vi, we use the pre-processed dataset provided by. We use tst2012 as dev set and test on tst2013. For IWSLT14 De-En, we follow the pre-processing steps and the same train/dev/test split as in. For WMT14 En-De, we follow the preprocessing steps in for fair comparison. We use newstest2013 as the dev set and newstest2014 as the test set. We report BLEU scores for evaluation of MT performance following the Moses script. Abstractive Summarization For summarization, we conduct experiments on the Gigaword summarization dataset . Note that the original train/valid/test split of Gigaword is 3.8M/190k/2k. In our experiments, we observed severe distribution mismatch between the validation and test data. See Table 4, 5, and Sec. 4.3 for detailed discussion. Therefore, we further sampled 5k/5k dev/test-dev splits from the validation set and tuned hyper-parameters on the dev set only. We report ROUGE scores on test-dev for the evaluation of our proposed approach, and include on the standard test split for the comparison with prior work. Training and Hyper-parameters Our implementation is based on the PyTorch version of OpenNMT seq2seq toolkit. We use the'base' model of 6-layer Transformer with 512-hidden 8-head attention blocks and 2048-hidden feed-forward layer for all experiments, with label smoothing regularization (LSR) of 0.1. We batch examples with similar sequence length, and count batch size by the number of tokens. For MT we use the pre-trained BERT-base-multilingual-cased model, and for summarization we use BERTbase-uncased as the starting point of BERT fine-tuning. 5 We use the corresponding pre-trained byte-pair-encoding shipped together with the BERT model for tokenization. For all training methods of all Transformer models, the learning rate schedule is set to lr = η · d −0.5 model · min(step −0.5, step · warmup steps −1.5), where d model = 512 is the attention representation size . For all BERT fine-tuning, we follow use a triangular learning rate schedule with maximum learning rate η. The parameters are updated with the Adam optimizer . In the distillation stage, we pre-compute BERT's prediction logits of the training data using top-K distillation to reduce computation overhead and memory footprint, where K is set to 8 across all the experiments. We also tune the temperature T for the sof tmax applied at the teacher's logits. For the detailed values of the hyper-parameters for each experiment, please refer to the supplementary material. We found it necessary to train longer with L bidi, since it is still improving after the step at which the baseline Transformer starts to plateau. At inference time, we use beam search with beam size 4 and length penalty of 0.6 across all the models. All the hyperparameters are tuned on the development set. Note that we tuned our Transformer baseline to achieve higher scores than the reference implementation on each dataset with default hyper-parameters (in most cases comparable to the state-of-the-art). We first validate our proposed text generation approach on machine translation task. Experimental are summarized in Table 1, 2 and 3, which show that our model significantly improves over the strong Transformer baseline across all three datasets. Note that our baseline is the'base' model of Transformer, which has 44M trainable parameters, and the reference implementation by is Transformer (big) with 176M trainable parameters. For IWSLT German-English translation, our method improves over the Transformer baseline by 1.54 BLEU points, and achieves new state of the art. Our approach outperforms previously-reported such as ConvS2S+MRT, a convolutional-based model with minimum risk training, and Lightweight and Dynamic Convolution . Note that also tuned checkpoint averaging, which creates a soft ensemble effect. And their model has roughly the same amount of parameters as Transformer (big). For IWSLT English-Vietnamese translation, since most prior work experimented with RNN models, we also report RNN-based here. This also suggests that our method is model-agnostic. Our best model outperforms Seq2Seq-OT that utilizes optimal transport for sequencelevel training, as well as the ELMo and CVT reported in. 8 For WMT14 6 Different from the original KD, we do not apply the same temperature on the student. In our preliminary experiment we found high T of Seq2Seq in much worse performance. We hypothesize the low-entropy nature of conditioned text generation is not suitable for temperature scaling. 7 Parameter counts exclude word embedding and final linear projection, which mostly depends on the vocabulary size. BERT-base has 86M trainable parameters. 8 The CVT used a much larger RNN and CNN-based character embedding, as well as a customized structure. Therefore, we did not try to use RNN to match their . English-German translation, our method still improves over the well-tuned Transformer baseline. We also report the scores of Transformer (big) and state-of-the-art Dynamic Convolution model for reference. Table 4 and Table 5 show the of our approach on abstractive summarization task, where R-1, R-2, and R-L denote F 1 scores of ROUGE-1, ROUGE-2, and ROUGE-L, respectively. Our method shows improvement on all the metrics, as shown in Table 4. We observe that the performance on test set is much lower, which suggests that the distribution in the test set is very different from that in the validation set, as mentioned in Section 4.1. When we manually checked the test set data, we found many corrupted examples such as short input articles, meaningless text, and dominating unknown words. Given that the official test split contains only 1,951 noisy examples, we believe that our on the dev/test-dev sets are more reliable. R-1 R-2 R-L Dev On the test split, our best model is comparable to state-of-the-art models that use much more complex architectures specifically designed for summarization. CGU augmented convolutional gating units. FTSum g (b) leveraged extra information extraction and dependency parsing features. E2T cnn utilized entities provided by an external entity linking system. Re 3 Sum (a) carefully designed a retrieve-and-rerank pipeline with human-written soft templates. Despite that our model has no summarization-specific model design, we still achieve comparable performance to these models on all the metrics. There are several possible factors that could contribute to the performance gain: additional parameters of BERT, extra data (pretraining corpus) of BERT, and the bidirectional nature. To better understand the key contributions of our method, we conduct an ablation study described in the following. We finetune 2 extra teachers: BERT sm and BERT l2r. For BERT sm, we use a smaller BERT (6 layers) for C-MLM finetuning, which has approximately the same number of parameters as Transformer-base 9. For BERT l2r, we use the full BERT model but finetune it using left-to-right LM as in the conventional Seq2Seq model. Next, we apply the proposed KD method to train the Transformer on En-Vi and De-En MT tasks. Results are shown in Table 6. BERT sm still works well though the full BERT provides further improvement. On the other hand, BERT l2r slightly hurts the performance. We hypothesize that it generates noisy learning targets for the student, hence the performance drop. Empirically, we show that the bidirectional knowledge could be more important than the extra parameters, while the pre-trained weights remain useful for more stable C-MLM training. We next analyze the effect of our proposed approach on different output lengths. We plot the BLEU scores on MT w.r.t. different output generation lengths N on the development set. 10 Results are provided in Figure 2. For IWSLT German-English dataset (Figure 2 : Left), we can see a shared trend that the proposed L bidi objective gains higher BLEU points on longer translation pairs. For WMT English-German (Figure 2 : Middle), we can see that although the proposed method performs much worse when the output sentences are very short, it achieves relatively consistent improvement on longer cases, hence ing in overall BLEU improvement. For IWSLT English-Vietnamese (Figure 2 : Right), we see a similar trend when the length N > 24. In Table 7, we show some translation examples on IWSLT German-English dataset. In the first example, the baseline Transformer cannot recover from'with' and'of', which renders the full sentence not making much sense. " I started reading with..." would make sense from the left context; however, if the model also considers the right context "the age of two", the word'with' would be assigned with lower probability by the soft labels provided by the BERT teacher. Even though at test-time the model cannot'look ahead', the soft-targets at training-time prevents the over-confidence of the model on one-hot label; hence the better generalization at the test-time. Similarly, other examples show that our model can generate text more coherently w.r.t. the context on the right (underlined in Table 7), thus making more accurate and natural translation. In this work, we propose a novel and generic approach to utilizing pre-trained language models to improve text generation without explicit parameter sharing, feature extraction, or augmenting with auxiliary tasks. Our proposed Conditional MLM mechanism leverages unsupervised language models pre-trained on large corpus, and then adapts to supervised sequence-to-sequence tasks. Our distillation approach indirectly influences the text generation model by providing soft-label distributions only, hence is model-agnostic. Experiments show that our model improves over strong Transformer baselines on multiple text generation tasks such as machine translation and abstractive summarization, and achieves new state-of-the-art on some of the translation tasks. For future work, we will explore the extension of Conditional MLM to multimodal input such as image captioning. We run all experiments on single GPU of NVIDIA Titan RTX or V100 except for WMT En-De we use 4 V100s for training. Note that for large batch sizes that do not fit in GPU memory, we use the gradient accumulation tricks as in. Batch sizes are counted in number of tokens. Note that all the hyper-parameters are tuned on the development set only. IWSLT De-En For C-MLM fine-tuning, we train for 100k steps with 5k warmup steps, η = 5 · 10 −5, and batch size of 16k tokens. For baseline model, we train for 50k steps with 4k warmup steps and batch size of 6k tokens. The learning rate η is set to 1. For the proposed model, we train for 100k steps with 8k warmup steps and batch size of 6k tokens. The learning rate η is set to 2, α = 0.5, and T = 10. Seq2Seq model uses dropout of 0.3 in both cases. IWSLT En-Vi For C-MLM fine-tuning and baseline Transformer, the hyper-parameters are identical to that of IWSLT De-En. For the proposed model, we train for 100k steps with 8k warmup steps and batch size of 6k tokens. The learning rate η is set to 2, α = 0.1, and T = 5. Dropout is still 0.1. WMT En-De For C-MLM fine-tuning, we train for 100k steps with 5k warmup steps, η = 5 · 10 −5, and batch size of 512k tokens. For baseline model, we train for 30k steps with 4k warmup steps and batch size of 384k tokens. The learning rate η is set to 4. Since this is our largest dataset and training is slow, for the proposed model we use the baseline Transformer to initialize the Seq2Seq student. For the proposed model, we continue training for 50k steps with 4k warmup steps and batch size of 64k tokens. The learning rate η is set to 2, α = 0.1, and T = 5. Seq2Seq model uses dropout of 0.1 in both cases. Gigaword For C-MLM fine-tuning, we train for 100k steps with 5k warmup steps, η = 5·10 −5, and batch size of 64k tokens. For baseline model, we train for 50k steps with 4k warmup steps and batch size of 40k tokens. The learning rate η is set to 1. For the proposed model, we train for 70k steps with 4k warmup steps and batch size of 36k tokens. The learning rate η is set to 2, α = 0.1, and T = 10. Seq2Seq model uses dropout of 0.1 in both cases. We show Gigaword summarization examples in Table 9 and extra En-DE generation examples in Reference it would be immoral to leave these young people with a climate system spiraling out of control. Transformer it would be immoral to let these young people leave a climate system that was out of control. (44.6) Ours it would be immoral to leave these young people with a climate system out of control. Table 8: Qualitative examples from IWSLT German-English translation. Numbers inside the parenthesis are sentence-level BLEU scores. Red word is where the baseline Transformer makes a mistake without considering the possible future phrase and fails to recover. On the other hand, our model makes the right decision at the blue word, hence generates more coherent sentence. Please refer to Section 4.5 in the main paper for detailed explanation. Reference china offers tax exemptions for laid-off workers Transformer china encourages laid-off workers to seek employment Ours china offers tax exemptions to laid-off workers Reference swiss police arrest britons who allegedly ran rental car racket Transformer three britons arrested in swiss luxury hotel Ours swiss police arrest three britons in rental car racket case Reference south korea stocks extend declines as kia concerns intensify Transformer south korean stocks fall for #th time in # days; kia leads Ours south korean stocks fall as kia troubles intensify Table 9: Qualitative examples from the Gigaword summarization dataset. Baseline model suffers from early mistakes. Our model generates more coherent summaries.

We propose a model-agnostic way to leverage BERT for text generation and achieve improvements over Transformer on 2 tasks over 4 datasets.

Represent the short summary for finding the corresponding scientific article (long)!

We revisit self-training as a semi-supervised learning method for neural sequence generation problem, and show that self-training can be quite successful with injected noise.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: Humans have the remarkable ability to correctly classify images despite possible degradation. Many studies have suggested that this hallmark of human vision from the interaction between feedforward signals from bottom-up pathways of the visual cortex and feedback signals provided by top-down pathways. Motivated by such interaction, we propose a new neuro-inspired model, namely Convolutional Neural Networks with Feedback (CNN-F). CNN-F extends CNN with a feedback generative network, combining bottom-up and top-down inference to perform approximate loopy belief propagation. We show that CNN-F's iterative inference allows for disentanglement of latent variables across layers. We validate the advantages of CNN-F over the baseline CNN. Our experimental suggest that the CNN-F is more robust to image degradation such as pixel noise, occlusion, and blur. Furthermore, we show that the CNN-F is capable of restoring original images from the degraded ones with high reconstruction accuracy while introducing negligible artifacts. Convolutional neural networks (CNNs) have been widely adopted for image classification and achieved impressive prediction accuracy. While state-of-the-art CNNs can achieve near-or super-human classification performance, these networks are susceptible to accuracy drops in the presence of image degradation such as blur and noise, or adversarial attacks, to which human vision is much more robust. This weakness suggests that CNNs are not able to fully capture the complexity of human vision. Unlike the CNN, the human's visual cortex contains not only feedforward but also feedback connections which propagate the information from higher to lower order visual cortical areas as suggested by the predictive coding model. Additionally, recent studies suggest that recurrent circuits are crucial for core object recognition. A recently proposed model extends CNN with a feedback generative network, moving a step forward towards more brain-like CNNs. The inference of the model is carried out by the feedforward only CNN. We term convolutional neural networks with feedback whose inference uses no iterations as CNN-F (0 iterations). The generative feedback models the joint distribution of the data and latent variables. This methodology is similar to how human brain works: building an internal model of the world. Despite the success of CNN-F (0 iterations) in semi-supervised learning and out-of-distribution detection, the feedforward only CNN can be a noisy inference in practice and the power of the rendering top-down path is not fully utilized. A neuro-inspired model that carries out more accurate inference is therefore desired for robust vision. Our work is motivated by the interaction of feedforward and feedback signals in the brain, and our contributions are: We propose the Convolutional Neural Network with Feedback (CNN-F) with more accurate inference. We perform approximated loopy belief propagation to infer latent variables. We introduce recurrent structure into our network by feeding the generated image from the feedback process back into the feedforward process. We term the model with k-iteration inference as CNN-F (k iterations). In the context without confusion, we will use the name CNN-F for short in the rest of the paper. We demonstrate that the CNN-F is more robust to image degradation including noise, blur, and occlusion than the CNN. In particular, our experiments show that CNN-F experiences smaller accuracy drop compared to the corresponding CNN on degraded images. We verify that CNN-F is capable of restoring degraded images. When trained on clean data, the CNN-F can recover the original image from the degraded images at test time with high reconstruction accuracy. Convolutional Neural Network with Feedback (CNN-F) is a generative model that generates images by coarse-to-fine rendering using the features computed by the corresponding CNN. Latent variables in CNN-F account for the uncertainty of the rendering process. The prior distribution of those latent variables is designed to capture the dependencies between them across layers. Inference for the optimal latent variables given image x and label y matches a feedforward CNN in CNN-F (0 iterations) (see Fig. 1). We provide mathematical description of CNN-F (0 iterations) below. Let h be the generated image, y ∈ {1, ..., K} be object category. z = {t, s}, = 1,..., L are the latent variables at layer, where t defines translation of rendering templates based on the position of local maximum from Maxpool, and s decides whether to render a pixel or not based on whether it is activated (ReLU) in the feed-forward CNN. T (t) denotes the translation matrix corresponding to the translation latent variable t. W are rendering templates, where W is the weight matrix at layer in the corresponding CNN. h is the intermediate rendered image at layer. The generation process in CNN-F (0 iterations) is given by: The dependencies among latent variables {z} 1:L across different layers are captured by the structured prior π z|y Softmax η exp(η), and b corresponds the bias after convolutions in CNN. Under the assumption that the intermediate rendered images {h} 1:L are nonnegative, the joint maximum a posteriori (JMAP) inference of latent variable z in CNN-F (0 iterations) is a CNN. Convolutional Neural Networks with Feedback using k-iteration inference [CNN-F (k iterations)] performs approximated loopy belief propagation on CNN-F for k times (see Fig. 1). Inference of latent variables is performed by propagating along both directions of the model. In the following of this session, we will use CNN-F to denote CNN-F (k iterations) for short. Inheriting the notation for the formulation in the CNN-F (0 iterations), we formulate CNN-F as follows. The generation process of the top-down pathway in CNN-F is the same as in the CNN-F (0 iterations), i.e. h(− 1) = T (t)W (s h). Different from the CNN-F (0 iterations), the generated image h in the CNN-F is fed back to the bottomup pathway for approximated loopy belief propagation. In other words, the CNN-F performs bottom-up followed by top-down inference such that the information at later layers in the CNNs can be used to update the noisy estimations at the early layers in the same network. Specifically, the feedforward process in the CNN-F is g = W AdaPool{AdaRelu(g( − 1))} + b, where g denotes the network activations at layer. The top-down messages correct for the noisy bottom-up inference by the adaptive operators Input: Input image x. and object class y *. where W is the rendering template at layer, and b is the parameters of the structured prior π z|y at layer. T (t) is the translation matrix corresponding to the translation latent variable t. Repeat step 2 -3 until convergence or early stopping. (see Algorithm 1): We study the robustness and image restoration performance of CNN-F (10 iterations). Additionally, we observe the disentanglement of information stored in latent variables. In this section, we will refer to CNN-F (10 iterations) as CNN-F. We train a 4 layer CNN and CNN-F (10 iterations) of corresponding architecture on the clean MNIST train set. For the architecture, we use 3 convolutional layers followed by 1 fully connected layer. We use 5x5 convolutional kernel for each convolutional layer with 8 channels in the first layer followed by 16 channels in the second layer followed by 8 channels in the third layer. We use instance norm between layers to normalize the input. We test the models on degraded test set images. The CNN trained has test accuracy 99.1% while CNN-F has test accuracy 95.26%. Our experimental study shows that the iterative inference in CNN-F promotes disentanglement of latent factors across layers. In particular, we observe that the latent variables at each layer in CNN-F captures different essences of the reconstructed image. For example, in the case of MNIST digits, those essences are different strokes that form the digits. Those strokes differ from each other in their location, styles, or angles. In our experiment, we trained a CNN-F with 3 convolutional layers on MNIST. Then, we sent an MNIST image of digit 0 and an MNIST image of digit 1 into the trained networks and collected their corresponding sets of latent variables. We denote z k to be the estimated latent variables from the image of digit 1 at layer k = 1, 2, 3 in CNN-F. Figure 2 illustrates that each set of latent variables z k captures strokes at a particular location in digit 1. In the first column of Figure 2, we use latent variables z 3 at the top layer in CNN-F to reconstruct the image. Similarly, in the second column of Figure 2, in addition to z 3, we add z 2 into the reconstruction. We observe that the latent variables z 3 capture the center of the digit 1 while the latent variables z 2 try to extend the digits to both ends. Finally, we include z 1 into the reconstruction and observe that it completes the digit by filling in the two ends. This observation suggests that CNN-F and its iterative inference algorithm lead to effective disentanglement of latent factors across the layers. Robustness Table 1 shows the accuracy and percent accuracy drop on noisy, blurry and occluded input. The accuracy of CNN-F drops less compared to CNN of same architecture, indicating that CNN-F is more robust. Image Restoration Table 2 shows CNN-F's reconstruction of images with added gaussian noise, blur, and occlusion. CNN-F is able to denoise, deblur, and do some degree of inpainting in on the degraded images. We note that with more iterations of feedback, the reconstructed image becomes more clean. The ability of CNN-F to restore images is consistent with studies in neuroscience which suggest that feedback signals contribute to automatic sharpening of images. For example, Abdelhack and Kamitani showed that the neural representation of blurry images is more similar to the latent representation of the clean version from a deep neural network than the latent representation of the blurry image. CNN-F is able to sharpen blurry images, which is consistent with this study. to understand better the role of feedback in robust vision. To compare CNN-F with neuronal/psychological data, we will scale up the training to ImageNet. A more challenging scenario for robust vision is adversarial attack. We will study the robustness of the proposed CNN-F under various types of adversarial attacks. We also plan to measure the similarity between the latent representations of the CNN-F with neural activity recorded from the brain in order to access whether CNN-F is a good model for human vision. We propose the Convolutional Neural Networks with Feedback (CNN-F) which consists of both a classification pathway and a generation pathway similar to the feedforward and feedback connections in human vision. Our model uses approximate loopy belief propagation for inferring latent variables, allowing for messages to be propagated along both directions of the model. We also introduce recurrency by passing the reconstructed image and predicted label back into the network. We show that CNN-F is more robust than CNN on corrupted images such as noisy, blurry, and occluded images and is able to restore degraded images when trained only on clean images.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

CNN-F extends CNN with a feedback generative network for robust vision.

Represent the TLDR for finding the corresponding scientific article

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.