Question: Represent the scientific article to find a short TLDR.

Question: We propose an algorithm, guided variational autoencoder (Guided-VAE), that is able to learn a controllable generative model by performing latent representation disentanglement learning. The learning objective is achieved by providing signal to the latent encoding/embedding in VAE without changing its main backbone architecture, hence retaining the desirable properties of the VAE. We design an unsupervised and a supervised strategy in Guided-VAE and observe enhanced modeling and controlling capability over the vanilla VAE. In the unsupervised strategy, we guide the VAE learning by introducing a lightweight decoder that learns latent geometric transformation and principal components; in the supervised strategy, we use an adversarial excitation and inhibition mechanism to encourage the disentanglement of the latent variables. Guided-VAE enjoys its transparency and simplicity for the general representation learning task, as well as disentanglement learning. On a number of experiments for representation learning, improved synthesis/sampling, better disentanglement for classification, and reduced classification errors in meta learning have been observed. The resurgence of autoencoders (AE) (; ;) is an important component in the rapid development of modern deep learning. Autoencoders have been widely adopted for modeling signals and images . Its statistical counterpart, the variational autoencoder (VAE) , has led to a recent wave of development in generative modeling due to its two-in-one capability, both representation and statistical learning in a single framework. Another exploding direction in generative modeling includes generative adversarial networks (GAN) , but GANs focus on the generation process and are not aimed at representation learning (without an encoder at least in its vanilla version). Compared with classical dimensionality reduction methods like principal component analysis (PCA) and Laplacian eigenmaps , VAEs have demonstrated their unprecedented power in modeling high dimensional data of real-world complexity. However, there is still a large room to improve for VAEs to achieve a high quality reconstruction/synthesis. Additionally, it is desirable to make the VAE representation learning more transparent, interpretable, and controllable. In this paper, we attempt to learn a transparent representation by introducing guidance to the latent variables in a VAE. We design two strategies for our Guided-VAE, an unsupervised version (Fig. 1 .a) and a supervised version (Fig. 1.b). The main motivation behind Guided-VAE is to encourage the latent representation to be semantically interpretable, while maintaining the integrity of the basic VAE architecture. Guided-VAE is learned in a multi-task learning fashion. The objective is achieved by taking advantage of the modeling flexibility and the large solution space of the VAE under a lightweight target. Thus the two tasks, learning a good VAE and making the latent variables controllable, become companions rather than conflicts. In unsupervised Guided-VAE, in addition to the standard VAE backbone, we also explicitly force the latent variables to go through a lightweight encoder that learns a deformable PCA. As seen in Fig. 1.a, two decoders exist, both trying to reconstruct the input data x: Dec main. The main decoder, denoted as Dec main, functions regularly as in the standard VAE ; the secondary decoder, denoted as Dec sub, explicitly learns a geometric deformation together with a linear sub-space. In supervised Guided-VAE, we introduce a subtask for the VAE by forcing one latent variable to be discriminative (minimizing the classification error) while making the rest of the latent variable to be adversarially discriminative (maximizing the minimal classification error). This subtask is achieved using an adversarial excitation and inhibition formulation. Similar to the unsupervised Guided-VAE, the training process is carried out in an end-to-end multi-task learning manner. The is a regular generative model that keeps the original VAE properties intact, while having the specified latent variable semantically meaningful and capable of controlling/synthesizing a specific attribute. We apply Guided-VAE to the data modeling and few-shot learning problems and show favorable on the MNIST, CelebA, and Omniglot datasets. The contributions of our work can be summarized as follows: • We propose a new generative model disentanglement learning method by introducing latent variable guidance to variational autoencoders (VAE). Both unsupervised and supervised versions of Guided-VAE have been developed. • In unsupervised Guided-VAE, we introduce deformable PCA as a subtask to guide the general VAE learning process, making the latent variables interpretable and controllable. • In supervised Guided-VAE, we use an adversarial excitation and inhibition mechanism to encourage the disentanglement, informativeness, and controllability of the latent variables. Guided-VAE is able to keep the attractive properties of the VAE and it is easy to implement. It can be trained in an end-to-end fashion. It significantly improves the controllability of the vanilla VAE and is applicable to a range of problems for generative modeling and representation learning. Related work can be discussed along several directions. Generative model families such as generative adversarial networks (GAN) and variational autoencoder (VAE) have received a tremendous amount of attention lately. Although GAN produces higher quality synthesis than VAE, GAN is missing the encoder part and hence is not directly suited for representation learning. Here, we focus on disentanglement learning by making VAE more controllable and transparent. Disentanglement learning (; ; ;) recently becomes a popular topic in representation learning. Adversarial training has been adopted in approaches such as (; . Various methods (; ;) have imposed constraints/regularizations/supervisions to the latent variables but these existing approaches often involve an architectural change to the VAE backbone and the additional components in these approaches are not provided as secondary decoder for guiding the main encoder. A closely related work is the β-VAE approach in which a balancing term β is introduced to control the capacity and the independence prior. β-TCVAE further extends β-VAE by introducing a total correlation term. From a different angle, principal component analysis (PCA) family (; ; Candès et al., 2011) can also be viewed as representation learning. Connections between robust PCA (Candès et al., 2011) and VAE have been observed . Although being a widely adopted method, PCA nevertheless has limited modeling capability due to its linear subspace assumption. To alleviate the strong requirement for the input data being prealigned, RASL deals with unaligned data by estimating a hidden transformation to each input. Here, we take the advantage of the transparency of PCA and the modeling power of VAE by developing a sub-encoder (see Fig. 1 .a), deformable PCA, that guides the VAE training process in an integrated end-to-end manner. After training, the sub-encoder can be removed by keeping the main VAE backbone only. To achieve disentanglement learning in supervised Guided-VAE, we encourage one latent variable to directly correspond to an attribute while making the rest of the variables uncorrelated. This is analogous to the excitation-inhibition mechanism or the explaining-away phenomena. Existing approaches impose supervision as a conditional model for an image translation task, whereas our supervised Guided-VAE model targets the generic generative modeling task by using an adversarial excitation and inhibition formulation. This is achieved by minimizing the discriminative loss for the desired latent variable while maximizing the minimal classification error for the rest of the variables. Our formulation has connection to the domain-adversarial neural networks (DANN) but the two methods differ in purpose and classification formulation. Supervised Guided-VAE is also related to the adversarial autoencoder approach but the two methods differ in objective, formulation, network structure, and task domain. In , the domain invariant variational autoencoders method (DIVA) differs from ours by enforcing disjoint sectors to explain certain attributes. Our model also has connections to the deeply-supervised nets (DSN) where intermediate supervision is added to a standard CNN classifier. There are also approaches in which latent variables constraints are added but they have different formulations and objectives than Guided-VAE. Recent efforts in fairness disentanglement learning also bear some similarity but there is still with a large difference in formulation. In this section, we present the main formulations of our Guided-VAE models. The unsupervised Guided-VAE version is presented first, followed by introduction of the supervised version. Following the standard definition in variational autoencoder (VAE) , a set of input data is denoted as X = (x 1, ..., x n) where n denotes the number of total input samples. The latent variables are denoted by vector z. The encoder network includes network and variational parameters φ that produces variational probability model q φ (z|x). The decoder network is parameterized by θ to reconstruct samplex = f θ (z). The log likelihood log p(x) estimation is achieved by maximizing the Evidence Lower BOund (ELBO) : The first term in eq. corresponds to a reconstruction loss q φ (z|x) × ||x − f θ (z)|| 2 dz (the first term is the negative of reconstruction loss between input x and reconstructionx) under Gaussian parameterization of the output. The second term in eq. refers to the KL divergence between the variational distribution q φ (z|x) and the prior distribution p(z). The training process thus tries to find the optimal (θ, φ) * such that: In our unsupervised Guided-VAE, we introduce a deformable PCA as a secondary decoder to guide the VAE training. An illustration can be seen in Fig. 1.a. This secondary decoder is called Dec sub. Without loss of generality, we let z = (z def, z cont). z def decides a deformation/transformation field, e.g. an affine transformation denoted as τ (z def). z cont determines the content of a sample image for transformation. The PCA model consists of K basis B = (b 1, ..., b K). We define a deformable PCA loss as: where • defines a transformation (affine in our experiments) operator decided by τ (z def) and 2 can be optionally added to force the basis to be unit vectors. We follow the spirit of the PCA optimization and a general formulation for learning PCA, which can be found in (Candès et al., 2011). To keep the simplicity of the method we learn a fixed basis function B and one can also adopt a probabilistic PCA model . Thus, learning unsupervised Guided-VAE becomes: For training data X = (x 1, ..., x n), suppose there exists a total of T attributes with ground-truth labels. The t-th attribute, let z = (z t, z rst t) where z t defines a scalar variable deciding to decide the t-th attribute and z rst t represents remaining latent variables. Let y t (x i) be the ground-truth label for the t−th attribute of sample x i; y t (x i) ∈ {−1, +1}. For each attribute, we use an adversarial excitation and inhibition method with term: which is a hinge term. This is an excitation process since we want latent variable z t to directly correspond to the attribute label. Notice the − sign before the summation since this term will be combined with eq. for maximization. where C t (z rst t) refers to classifier making a prediction for the t-th attribute using the remaining latent variables z rst t. − log p Ct (y = y(x)|z rst t ) is a cross-entropy term for minimizing the classification error in eq.. This is an inhibition process since we want the remaining variables z rst t as independent as possible to the attribute label. Note that the term L Inhibition (φ, t) within eq. for maximization is an adversarial term to make z rst t as uninformative to attribute t as possible, by making the best possible classifier C t to be undiscriminative. The formulation of eq. bears certain similarity to that in domain-adversarial neural networks in which the label classification is minimized with the domain classifier being adversarially maximized. Here, however, we respectively encourage and discourage different parts of the features to make the same type of classification. In this section, we first present qualitative demonstrating our proposed unsupervised Guided-VAE (Figure 1a) capable of disentangling latent embedding in a more favourable way than VAE and previous disentangle methods on MNIST dataset . We also show that our learned latent representation can be later used to improve classification performance. Next, we extend this idea to a supervised guidance approach in an adversarial excitation and inhibition fashion, where a discriminative objective for certain image properties is given (Figure 1b) on the CelebA dataset . Further, we show that our method can be applied to the few-shot classification tasks, which achieves competitive performance on Omniglot dataset proposed by. 4.1 UNSUPERVISED GUIDED-VAE 4.1.1 QUALITATIVE EVALUATION We present qualitative on MNIST dataset by traversing latent variables received affine transformation guiding signal. Here, we applied the Guided-VAE with the bottleneck size of 10 (i.e. the latent variables z ∈ R 10). The first latent variable z 1 represents the rotation information and the second latent variable z 2 represents the scaling information. The rest of the latent variables z 3:10 represent the content information. Thus, the latent variables z ∈ R 10 are represented by z = (z def, z cont) = (z 1:2, z 3:10)., β-VAE with controlled capacity increase (CCβ-VAE), Joint-VAE and our Guided-VAE on the MNIST dataset. z1 and z2 in Guided-VAE are controlled. Figure 3: PCA basis learned by the secondary decoder in unsupervised Guided-VAE. In Figure 2, we show traversal of all latent variables on MNIST dataset for vanilla VAE , β-VAE , JointVAE and our guided VAE (β-VAE, JointVAE are adopted from ). While β-VAE cannot generate meaningful disentangled representations, even with controlled capacity increased, JointVAE is able to disentangle class type from continuous factors. Different from previous methods, our Guided-VAE disentangles geometry properties (z 1 and z 2) like rotation angle and stroke thickness from the rest content information z 3:10. In Figure 3, we visualize the basis B = (b 1, ..., b 8) in the PCA part of Dec sub. The basis primarily capture the content information. For a quantitative evaluation, we first compare the reconstruction error among different models on the MNIST dataset. In this experiment, we set the bottleneck size to 8 in Guided-VAE and use three settings for the deformation/transformation: Rotation, scaling, and both. In Guided-VAE (Rotation) or Guided-VAE (Scaling), we take the first latent variable z 1 to represent the rotation or the scaling information. In Guided-VAE (Rotation and Scaling), we use the first and second latent variables (z 1 and z 2) to represent rotation and scaling respectively. As Table 1 shows, our reconstruction loss is on par with vanilla VAE, whereas the previous disentangling method (β-VAE) has higher loss. Our proposed method is able to achieve added disentanglement while not sacrificing reconstruction capability over vanilla VAE. In addition, we perform classification tasks on latent embeddings of different models. Specifically, for each data point (x, y), we use the pre-trained VAE model to obtain the value of latent variable z given input image x. Here z is a d z -dim vector. We then train a linear classifier f (·) on the embedding-label pairs {(z, y)} in order to predict the class of digits. For the Guided-VAE, we disentangle the latent variables z into deformation variables z def and content variables z cont with same dimensions (i.e. d z def = d zcont) and use affine transformation as τ (z def). We compare the classification errors of different models under multiple choices of dimensions of the latent variables in Table 2. It shows that generally higher dimensional latent variables in lower classification errors. Our Guided-VAE method compares favourably over vanilla VAE and β-VAE. Moreover, we attempt to validate the effectiveness of disentanglement in Guided-VAE. We follow the same classification tasks above but use different parts of latent variables as input features for the classifier f (·): We may choose the deformation variables z def, the content variables z cont, or the whole latent variables z as the input feature vector. To reach a fair comparison, we keep the same dimensions for the deformation variables z def and the content variables z cont. Table 3 shows that the classification errors on z cont are significantly lower than the ones on z def, which indicates the success of disentanglement since the content variables should determine the class of digits while the deformation variables should be invariant to the class. In addition, when the dimensions of latent variables z are higher, the classification errors on z def increase while the ones on z cont decrease, indicating a better disentanglement between deformation and content. We first present qualitative on the CelebA dataset by traversing latent variables of attributes. We select three labeled attributes (emotion, gender and color) in the CelebA dataset as supervised guidance objectives. The bottleneck size is set to 16. We use the first three latent variables z 1, z 2, z 3 to represent the attribute information and the rest z 4:16 to represent the content information. During evaluation, we choose z t ∈ {z 1, z 2, z 3} while keeping the remaining latent variables z rst t fixed. Then we obtain a set of images through traversing from the image with t-th attribute to the image without t-th attribute (e.g. smiling to non-smiling) and compare them over methods. Figure 4 shows the traversal for β-VAE and our Guided-VAE. β-VAE performs decently for the controlled attribute change, but the individual z in β-VAE is not fully entangled or disentangled with the attribute. Guided-VAE has a better disentanglement for latent variables and is able to better isolate the attributes w.r.t. the corresponding latent variables. In supervised Guided-VAE, we train a classifier to predict the attributes by using the disentangled attribute latent variable z t or the rest of latent variables z rst t as input features. We perform adversarial excitation and inhibition by encouraging the target latent variable to best predict the corresponding t-th attribute and discouraging the rest of the variables for the prediction of that attribute. (left) shows that the classification errors on z t is significantly lower than the ones on z rst t, which indicates the effectiveness of disentanglement during the training procedure. fiers for attribute classification) prediction for being negatives on the generated images. We traverse z1 (gender) and z2 (smile) separately to generate images for the classification test. Each latent z is traversed from −3.0 to 3.0 with 0.1 as the stride length. Furthermore, we attempt to validate that the generated images from the supervised Guided-VAE can be actually controlled by the disentangled attribute variables. Thus, we pre-train an external binary classifier for t-th attribute on the CelebA training set and then use this classifier to test the generated images from Guided-VAE. Each test includes 10, 000 generated images randomly sampled on all latent variables except for the particular latent variable z t we decide to control. As Figure 5 (right) shows, we can draw the confidence-z curves of the t-th attribute where z = z t ∈ [−3.0, 3.0]. For the gender and the smile attributes, it can be seen that the corresponding z t is able to enable (z t < −1) and disable (z t > 1) the attribute of the generated image. Besides, for all the attributes, the probability monotonically decreases when z t increases, which shows the controlling ability of the t-th attribute by tuning the corresponding latent variable z t. Previously, we have shown that Guided-VAE can generate images and be used as representation to perform classification task. In this section, we will apply the proposed method to few-shot classification problem. Specifically, we use our adversarial excitation and inhibition method in the Neural Statistician by adding a supervised guidance network after the statistic network. The supervised guidance signal is the label of each input. We also apply the Mixup method in the supervised guidance network. However, we couldn't reproduce exact reported in the Neural Statistician, which is also indicated in. For comparison, we mainly consider the Matching Nets and Bruno . Yet it cannot outperform Matching Nets, our proposed Guided-VAE reaches equivalent performance as Bruno (discriminative), where a discriminative objective is fine-tuned to maximize the likelihood of correct labels. We conduct a series of ablation experiments to validate our proposed Guided-VAE model. In this part, we conduct an experiment by excluding the geometry-guided part from the unsupervised Guided-VAE. In this way, the nudging decoder is just a PCA-like decoder but not a deformable PCA. The setting of this experiment is exactly same as described in the unsupervised Guided-VAE section. The bottleneck size of our model is set to 10 of which the first two latent variables z 1, z 2 represent the rotation and scaling information separately. In the ablation part, we drop off the geometry-guided part so all 10 latent variables are controlled by the PCA-like light decoder. In this part, we conduct an experiment of using the adversarial excitation method. We design the experiment using the exact same setting described in the supervised Guided-VAE part. As Figure 7 shows, though the traversal still show the traversed on some latent variables. The from the adversarial excitation method outperforms the from the discriminative method. While traversing the latent variable controlling the smiling information, the left part (a) also changes in the smiling status but it's controlled by another latent variable. shows the traversed images from the supervised Guided-VAE without adversarial inhibition. The right part shows the traversed images from the supervised Guided-VAE using adversarial excitation and inhibition. Both images are traversed on the latent variable that is supposed to control the gender information. In this paper we have presented a new representation learning method, guided variational autoencoder (Guided-VAE), for disentanglement learning. Both versions of Guided-VAE utilize lightweight guidance to the latent variables to achieve better controllability and transparency. Improvements on disentanglement, image traversal, and meta-learning over the competing methods are observed. Guided-VAE maintains the backbone of VAE and can be applied to other generative modeling applications. A.1 PERCENTAGE OF DATA PARTICIPATING IN THE GUIDED SUB-NETWORK In this part, we design an experiment to show how the percentage of data participating in the guided sub-network can influence the final prediction. We conduct this ablation study on MNIST using unsupervised Guided-VAE. We change the percentage of data participating in the guided sub-network and then present the classification accuracy using the first half latent variables (represent geometry information) and the second half latent variables (represent content information) separately. From Figure 8, we observe consistent improvement for the last half latent variables when adding more samples to guide sub-network. This indicates adding more samples can improve disentanglement, which causes that more content information is represented in the second half latent variables. Similarity, the improvement of disentanglement leads the first half latent variables can represent more geometry information, which is indiscriminative for classes. We also observe accuracy improvement when large amount of samples are used to train sub-network. We hypothesize this is because geometry information is still partially affected by classes.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Learning a controllable generative model by performing latent representation disentanglement learning.

Inspired by trial-to-trial variability in the brain that can result from multiple noise sources, we introduce variability through noise in the knowledge distillation framework and studied their effect on generalization and robustness.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Neural language models (NLMs) are generative, and they model the distribution of grammatical sentences. Trained on huge corpus, NLMs are pushing the limit of modeling accuracy. Besides, they have also been applied to supervised learning tasks that decode text, e.g., automatic speech recognition (ASR). By re-scoring the n-best list, NLM can select grammatically more correct candidate among the list, and significantly reduce word/char error rate. However, the generative nature of NLM may not guarantee a discrimination between “good” and “bad” (in a task-specific sense) sentences, ing in suboptimal performance. This work proposes an approach to adapt a generative NLM to a discriminative one. Different from the commonly used maximum likelihood objective, the proposed method aims at enlarging the margin between the “good” and “bad” sentences. It is trained end-to-end and can be widely applied to tasks that involve the re-scoring of the decoded text. Significant gains are observed in both ASR and statistical machine translation (SMT) tasks. Language models (LMs) estimate the likelihood of a symbol sequence {s i} n i=0, based on the joint probability, p(s 0, . . ., s n) = p(s 0) n i=1 p(s i |s i−1, s i−2, . . ., s 0). works, BID7 and BID25, propose to predict the next symbol based on a fusion of the hidden states in the ASR/SMT and language model. A gating mechanism is jointly trained to determine how much the language model should contribute. The afore-discussed language models are generative in the sense that they merely model the joint distribution of a symbol sequence (Eq. ). While the research community is mostly focused on pushing the limit of modeling accuracy (lower PPL) (e.g., BID12, very limited attention has been paid to the discrimination ability of language models when they are applied to supervised learning tasks, such as ASR and SMT. Discriminative language modeling aims at enhancing the performance in supervised learning tasks. In specific, existing works BID23 BID10 BID21 often target at improving ASR accuracy. The key motivation underlying them is that the model should be able to discriminate between "good" and "bad" sentences in a task-specific sense, instead of just modeling grammatical ones. The common methodology is to build a binary classifier upon hand-crafted features extracted from the sentences. However, it is not obvious how these methods can utilize large unsupervised corpus, which is often easily available, and the hand-crafted features are also ad hoc and may in suboptimal performance. In this work, we study how to improve the discrimination ability of a neural language model. The proposed method enlarges the difference between the log-likelihoods of "good" and "bad" sentences. In contrast to the existing works BID23 BID10 BID21, our method does not rely on hand-crafted features. It is trained in end-to-end manner and able to take advantage of large external text corpus. We apply the proposed large margin language model to ASR and SMT tasks. It reduces word error rate (WER) and increases bilingual evaluation understudy (BLEU) scores significantly, showing notable advantage over several alternative methods that are well adopted.2 RELATED WORK BID23 BID10 and BID21 proposed to train discriminative language models based on hand crafted features. They essentially build linear classifiers that give high scores on "good" sentences but low scores on "bad" ones. These methods all rely on ad hoc choice of features, e.g., counts of n-grams where n varies in a small range (e.g., 1 ∼ 3). Moreover, it is also not clear how these methods would take advantage of an existing language model (trained on large unsupervised corpus). BID28 tries to overcome the above issues by adapting an NLM on the transcriptions of a speech dataset. Although the setup is more similar to ours, their objective is not well-behaved and difficult to optimize when there are multiple beam candidates. An in-depth discussion will be given in Section 3.1. BID15 designed another approach to train a discriminative language model, which is based on bi-grams. Similar to our method, the objective there aims at increasing the difference between the scores of the best candidate and ground-truth. However, since the language model is not end-to-end, there are several issues complicating the training, e.g., handling back-off weight. Our proposed method is based on comparisons between pairs of sentences. Its implementation resembles siamese network architecture BID3 BID29, first proposed for face verification tasks. Recently, siamese network has also been applied to learning similarities on sequences BID19 BID20. In spite of solving different problems, the common methodology is to extract a pair of hidden representations for a pair of input samples (through a shared network). It then manipulates the distance between the hidden representations based on whether the two samples are considered similar or not. Our work also draws some inspirations from information retrieval (IR) BID16. As a representative IR method, ranking SVM BID8 assumes a linear scoring function, and imposes a hinge loss on the difference between the scores of sample pairs. We explicitly define LM score as the log-likelihood of a sentence estimated by an NLM. Existing works on NLM often train to maximize the score on a corpus that are assumed to be grammatical. However, they do not utilize any negative examples, where the "negative" means incompetence for a specific task. For example, in ASR BID0, negative samples may have spelling or grammar errors. In a conversational model BID30, negative samples are noninformative replies like "I don't know". An NLM trained in the maximum-likelihood fashion is not aware of the specific task. It therefore cannot guarantee a larger score for a positive sample than a negative one. This fact may handicap applications that need LMs to distinguish between positive and negative candidates. Examples include automatic speech recognition (ASR) and statistical machine translation (SMT). We aim at enhancing the LM's discrimination in these applications. Interestingly, negative samples are easy to obtain in the aforementioned applications. A beam search decoder can often yield abundant negative sentences (suboptimal beam candidates) that differ from ground-truth text (considered as positive). We motivate our method with an ASR example. A CTCbased BID6 ASR model is trained on Wall Street Journal (WSJ) dataset. We then input an audio utterance whose ground-truth transcription is given in the first row of TAB0. Four extracted beam candidates are listed in the following rows, from the best to the worst. Except beam 0, beam 1 to 3 all make some mistakes compared with the ground-truth. In this case, a language model is supposed to give a high score for beam 0 whereas low scores for beam 1 through 3. However, we observe that the scores of beam 2 and 3 are not sufficiently smaller than beam 0. We denote the i-th ground-truth sentence as x i where i = 1,..., N; correspondingly, the j-th beam candidate as x i,j, where j = 1,..., B and B is the beam size. Without loss of generality, we assume that these B candidates all differ from the ground-truth by some mistakes/incompetences. The NLM is desired to assign big log-likelihoods for the x i's but small ones for the x i,j's. A straightforward way is to adopt the following objective: DISPLAYFORM0 Similar formulation is also seen in BID28, where they only utilize one beam candidate, i.e., B = 1. The idea is to maximize the likelihood on the positive samples, and at the same time, minimize the likelihood on the negative samples. Optimization can be carried out by mini-batch stochastic gradient descent (SGD). Each iteration, SGD randomly samples a batch of i's and j's, computes stochastic gradient w.r.t. θ, and takes an update step. However, a potential problem with this formulation is that the second term (corresponding to the negative samples) may dominate the optimization. Specifically, the training is almost always driven by the negative x i,j's, but does not effectively enhance the discrimination. We illustrate this fact in the following experiment. Using the aforementioned ASR system, we extract 256 beam candidates for every training sample in the WSJ dataset. As a baseline for beam rescoring, a conventional NLM is trained on a large corpus, i.e. common-crawl 1. From the pre-trained baseline NLM, we warm start the training and apply SGD to optimize the objective in Eq., with a mini-batch size of 128. The training loss is shown in Figure 1a. We observe that the learning dynamic is very unstable. Using the trained model, we want the ground-truth sentences to have larger scores than the beam candidates. Therefore, we inspect log p θ (x i) − log p θ (x i,j), the margin between the scores of a ground-truth and a candidate. In FIG0, we histogram the margins for all the i, j's in a dev set. The distribution appears to be symmetric around zero, which indicates poor discrimination ability. Given these facts, we conclude that the straightforward formulation in Eq. FORMULA0 is not effective. To effectively utilize all the negative beam candidates, we propose the following objective, DISPLAYFORM0 where log p θ (x i) − log θ (x i,j) is the margin between the scores of a ground-truth x i and a negative candidate x i,j. The hinge loss on the margin encourages the log-likelihood of the ground-truth to be at least τ larger than that of the "bad" candidate. We call the above formulation Large Margin Language Model (LMLM).We repeat the same experiment in section 3.1, but change the objective function to Eq.. We fix τ = 1 across the paper. Figure 1b shows the training loss, which steadily decreases and approaches zero rapidly. Compared with the learning curve of naive formulation (figure 1a), the large margin based training is much more stable. In FIG0, we also examine the histogram of log p θ (x i) − log p θ (x i,j), where p θ (·) is the language model learned by LMLM. Compared with the histogram by the baseline NLM, LMLM significantly moves the distribution to the positive side, indicating more discrimination. In most cases, all beam candidates are imperfect. It would be beneficial to exploit the information that some candidates are relatively better than the others. We consider ranking them according to some metrics w.r.t. the ground-truth sentences. For ASR, the metric is edit distance, and for SMT, the metric is BLEU score. We define x i,0 x i and assume that the candidates {x i,j} B j=1 in the beam are sorted such that DISPLAYFORM0 for ASR, and DISPLAYFORM1 for SMT. In other words, x i,j−1 has better quality than x i,j.We then enforce the "better" sentences to have a score at least τ larger than those "worse" ones. This leads to the following formulation, DISPLAYFORM2 Compared with LMLM formulation Eq., the above introduces more comparisons among the candidates, and hence more computational cost during training. We call this formulation rank-LMLM. In this section, we study LMLM and rank-LMLM through extensive experiments on ASR and SMT. We demonstrate that both LMLM and rank-LMLM significantly outperform a baseline language model (baseline-LM) and two other domain adapted models. The baseline-LM architecture, shown in FIG1, starts with a 2048 dimensional embedding layer, followed by two LSTM layers, each with 2048 nodes. The LSTM hidden states are then projected down to dimension 512. Finally, a softmax layer with 400K dimensional output is appended to produce a distribution over the vocabulary. The huge vocabulary size incurs a large computational cost, and we use sampled softmax technique BID11 to accelerate training. This type of neural architecture has been shown effective in conventional language modeling tasks BID12. We trained the baseline-LM on the common-crawl corpus. Common-crawl corpus has a vocabulary size about 400K. This large vocabulary ensures a small out of vocabulary (OOV) rate for the used ASR and SMT datasets, details of which are summarized in TAB1. The baseline-LM achieves a reasonably good perplexity of 110 on a dev set with 400K sentences, significantly outperforming a 5-gram model, which has a dev perplexity of about 300. The experimental setup is as follows. First we train an ASR/SMT model on the training set. Then we extract B beam candidates for every training sample. This beam set, together with the corresponding ground-truth text, are used as the training data for LMLM and rank-LMLM. We then re-score the beams by linearly combining ASR/SMT and language model scores. The combination weight is found by optimizing the WER/BLEU on the dev set. Finally, WER/BLEU on the test set are reported. For comparison, we also include two other approaches that adapt the baseline-LM to the text in the specific task. One way is to fine-tune the baseline NLM using the task-specific text but still under the minimum-perplexity objective BID13. We call it refine-LM. The other way is to train a smaller NLM from scratch on the task-specific data, and then linearly interpolate with the baseline-LM BID10. We call it interp-LM. In all the experiments with LMLM and rank-LMLM, we set τ = 1. For rank-LMLM, since the total number of pairs is huge, we randomly sample 20% of them. An advantage of LMLM (and rank-LMLM) is being able to utilize huge unsupervised data. This is achieved by warm-starting the LMLM with a conventional language model trained on any unsupervised corpus. However, one would doubt why it is necessary to warm-start, since LMLM might easily learn to make binary decisions on pairs of "good" and "bad" sentences. Interestingly, we show that warm-starting is very important. Without warm-starting, the LMLM training will be stuck to "bad" local minimal and cannot generalize well. Earlier works in various applications BID9 BID5 have observed similar behavior and suggested using unsupervised model to warm-start supervised training. The observation on LMLM is yet another supportive evidence, elaborated in the following experiment. Using the same ASR system in Section 3.1, we extract 64 beam candidates for every training utterance in the WSJ dataset. We then train LMLM either with or without warm-starting from the baseline-LM. FIG2 compares the learning curves for both cases. With warm starting, training and dev losses start to decay from smaller values, approaching zero at convergence. The stronger generalization ability is further illustrated in FIG2. FIG2 shows the histogram of the margins between the scores for the ground-truths and beam candidates in the dev set. The more positive the margin is, the more the separation between positive and negative examples. In the warm-started case, the distribution is shifted rightwards, indicating more discrimination. We revisit the example in TAB0, in order to understand why LMLM and rank-LMLM can work. We estimate the language model scores using LMLM and rank-LMLM. Scores are listed in TAB2, in comparison with those by the baseline-LM, which we have seen in TAB0. Numbers in the brackets are the margins. A large positive margin indicates effective identification of the erroneous sentence. Overall, with LMLM and rank-LMLM, the margins become significantly more positive. More interestingly, rank-LMLM is able to assign larger score for beam 2 than beam 3, showing more selectivity than LMLM.We also notice that all the scores by LMLM and rank-LMLM are smaller than those by the baseline-LM, since the proposed methods are not guided by the conventional max-likelihood objective. Compared with LMLM, rank-LMLM scores are even smaller. This is due to more pairwise constraints imposed in training, which makes rank-LMLM deviate even more from the max-likelihood objective. However, we argue that the max-likelihood training is not well aligned with beam re-scoring purpose, which we shall see in the reported WERs soon. In this section, we apply the proposed methods for ASR tasks, and report WERs and CERs on test sets. The datasets we used are WSJ and Fisher, whose statistics are summarized in the 3rd and 4th columns of TAB1. Note that for Fisher task, we follow the standard setup BID22, which evaluates on the hub5 set, including two subsets (SWBD and CallHome). In total, there are 4,458 utterances for test. The ASR models for WSJ has one convolutional layer, followed by 5 layers of RNNs. The ASR model for Fisher has two convolutional layers, followed by 6 layers of GRUs. The final hidden representations are input into fully connected layers and then the models are trained using CTC loss BID6. Note that the ASR models are trained using only in-domain data. That is, the training utterances of WSJ or Fisher respectively. Using a language model during decoding may significantly improve the performance. Therefore, during decoding, we also applied an n-gram model learned from the in-domain training text. The beam search decoder has a beam width of 2000. The top-1 beam candidates give us strong baseline WERs, e.g., for WSJ task, the test WER is 7.58. The extracted beams candidates are then re-scored using the baseline-LM, refine-LM, interp-LM, LMLM and rank-LMLM. Training of these language models follow the experimental protocol in section 4.1. We report the WERs of the rescored top-1 candidates on the test set. The same exper-iment is repeated on WSJ and Fisher tasks. Results are listed in TAB3 respectively. Bold numbers are the best and italics are runner-ups. Among all the language models for re-scoring, rank-LMLM achieves the best WER and CER, and LMLM is also very competitive with rank-LMLM. They both significantly outperform the other methods, all of which are generative language models. Based on these observations, we argue that a language model has to be adapted in a way that is suitable for re-scoring purpose in supervised tasks, rather than just maximizing likelihood. LMLM and rank-LMLM are general in the sense that they can be applied to any re-scoring problem on text data. In this section, we show a proof-of-concept example on an SMT task. We experiment with IWSLT15 Vietnamese-to-English dataset 2. We use "tst2012" as dev set and "tst2013" as test set. The dataset is simplified by decapitalizing all the words and removing punctuations. This in a vocabulary size of about 44K, and only a tiny fraction of them are out of the vocabulary of common crawl. Details of the cleaned dataset are listed in the last column of TAB1.We train an SMT model based on the attention mechanism in BID17. The encoder and decoder both have two layers of LSTMs, each with 128 activations. We follow the experimental protocol outlined in Section 4.1. TAB5 reports the BLEU scores on the test set. LMLM and rank-LMLM both significantly outperform the other methods. In contrast, all the generative language models (3rd to 5th column in TAB5) do not improve upon the case without re-scoring. In fact, for these three cases, we found that the best weight to combine the language model is zero, meaning that the language model is not providing any complementary information to the SMT model. A key point to understand this phenomenon is that the decoder in a seq-to-seq model implicitly works as a conventional language model. This in very grammatical beam candidates, but their qualities differ in some semantic sense. A generative language model acts more or less like a grammar checker, and has no power of discrimination in the semantic field. Conventional language models are guided by minimizing perplexity, and they are generative models. This work proposes an approach to enhance the discrimination ability of language models. It is trained end-to-end by maximizing the margin between "good" and "bad" (in a task-specific sense) sentences. The method is general and can be applied to various tasks that require re-scoring of text data. Experiments on ASR and SMT have shown a consistent gain over several baselines. These facts argue that min-perplexity is not necessarily an appropriate guideline when we want to apply language models in some supervised learning problems. A future direction is to apply the proposed method to conversation generation. The goal is to discriminate between boring (e.g., "I don't know") and informative replies, thus deprecating the former. Another interesting future work is to apply the LMLM/rank-LMLM to lattices during decoding.

Enhance the language model for supervised learning task

represent the short summary for finding the corresponding scientific article (long):

We introduce and analyze the phenomenon of "hallucinations" in NMT, or spurious translations unrelated to source text, and propose methods to reduce its frequency.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Conventionally, convolutional neural networks (CNNs) process different images with the same set of filters. However, the variations in images pose a challenge to this fashion. In this paper, we propose to generate sample-specific filters for convolutional layers in the forward pass. Since the filters are generated on-the-fly, the model becomes more flexible and can better fit the training data compared to traditional CNNs. In order to obtain sample-specific features, we extract the intermediate feature maps from an autoencoder. As filters are usually high dimensional, we propose to learn a set of coefficients instead of a set of filters. These coefficients are used to linearly combine the base filters from a filter repository to generate the final filters for a CNN. The proposed method is evaluated on MNIST, MTFL and CIFAR10 datasets. Experiment demonstrate that the classification accuracy of the baseline model can be improved by using the proposed filter generation method. Variations exist widely in images. For example, in face images, faces present with different head poses and different illuminations which are challenges to most face recognition models. In the conventional training process of CNNs, filters are optimized to deal with different variations. The number of filters increases if more variations are added to the input data. However, for a test image, only a small number of the neurons in the network are activated which indicates inefficient computation BID13 ).Unlike CNNs with fixed filters, CNNs with dynamically generated sample-specific filters are more flexible since each input image is associated with a unique set of filters. Therefore, it provides possibility for the model to deal with variations without increasing model size. However, there are two challenges for training CNNs with dynamic filter generation. The first challenge is how to learn sample-specific features for filter generation. Intuitively, filter sets should correspond to variations in images. If the factors of variations are restricted to some known factors such as face pose or illumination, we can use the prior knowledge to train a network to represent the variation as a feature vector. The main difficulty is that besides the factors of variations that we have already known, there are also a number of them that we are not aware of. Therefore, it is difficult to enumerate all the factors of variations and learn the mapping in a supervised manner. The second challenge is that how to map a feature vector to a set of new filters. Due to the high dimension of the filters, a direct mapping needs a large number of parameters which can be infeasible in real applications. In response, we propose to use an autoencoder for variation representation leaning. Since the objective of an autoencoder is to reconstruct the input images from internal feature representations, each layer of the encoder contains sufficient information about the input image. Therefore, we extract features from each layer in the encoder as sample-specific features. For the generation of filters, given a sample-specific feature vector, we firstly construct a filter repository. Then we learn a matrix that maps the feature vector to a set of coefficients which will be used to linearly combine the base filters in the repository to generate new filters. Our model has several elements of interest. Firstly, our model bridges the gap between the autoencoder network and the prediction network by mapping the autoencoder features to the filters in the prediction network. Therefore, we embed the knowledge from unsupervised learning to supervised learning. Secondly, instead of generating new filters directly from a feature vector, we facilitate the generation with a filter repository which stores a small number of base filters. Thirdly, we use linear combination of the base filters in the repository to generate new filters. It can be easily implemented as a convolution operation so that the whole pipeline is differentiable with respect to the model parameters. The essential part of the proposed method is the dynamical change of the parameters of a CNN. In general, there are two ways to achieve the goal including dynamically changing the connection and dynamically generating the weights, both of which are related to our work. In this section, we will give a brief review of the works from these two aspects. There are several works in which only a subset of the connections in a CNN are activated in a forward pass. We term this kind of strategy dynamic connection. Since the activation of connections depends on input images, researchers try to find an efficient way to select subsets of the connections. The benefit of using dynamical connection is the reduction in computation cost. BID13 propose a conditional convlutional neural network to handle multimodal face recognition. They incorporate decision trees to dynamically select the connections so that images from different modalities activate different routes. BID6 present deep neural decision forests that unify classification trees with representation learning functionality. Each node of the tree performs routing decisions via a decision function. For each route, the input images are passed through a specific set of convolutional layers. BID5 and BID0 also propose similar frameworks for combining decision forests and deep CNNs. Those hybrid models fuse the high representation learning capability of CNNs and the computation efficiency of decision trees. We refer to weights that are dynamically generated as dynamic weights. Furthermore, since the weights are the parameters of a CNN, learning to generate those weights can also be viewed as a meta learning approach. BID1 propose to use dynamic weights in the scenario of one-shot learning. They construct a learnet to generate the weights of another deep model from a single exemplar. A number of factorizations of the parameters are proposed to reduce the learning difficulty. BID4 present hypernetworks which can also generate weights for another network, especially a deep convolutional network or a long recurrent network. The hypernetworks can generate non-shared weights for LSTM and improve its capability of sequence modelling. There are several other similar architectures BID11 BID3 ).Results from those works demonstrate that dynamical weights help learn feature representation more effectively. The work that most resembles ours is the work of De BID3. However, our work is different in the following aspects. (i) The feature vectors we used for filter generation are extracted from the feature maps of an autoencoder network. (ii) New filters are generated by the linear combination of base filters in a filter repository. The rest of the paper is structured as follows. Section 3 presents the details of the proposed method. Section 4 shows the experiment and Section 5 concludes the paper. The framework of the proposed method is illustrated in FIG0. The description of our model will be divided into three parts, i.e. sample-specific feature learning, filter generation, and final prediction. The framework of the proposed method. The autoencoder network in the first row is used to extract features from the input image. The obtained feature maps are fed to a dimension reduction module to reduce the dimension of the feature maps. Then the reduced features are used to generate new filters in the filter generation module. Finally, the prediction network takes in the same input image and the generated filters to make the final prediction for high level tasks such as detection, classification and so on. "*" indicates the convolution operation. It is difficult to quantify variations in an image sample. Thus, we adopt an autoencoder to learn sample-specific features. Typically, an autoender consists of an encoder and a decoder. The encoder extracts features from the input data layer by layer while the decoder plays the role of image reconstruction. Therefore, we use the features from each layer of the encoder as representations of the input image. Since the feature maps from the encoder are three-dimensional, we use dimension reduction modules to reduce the dimension of the feature maps. For each dimension reduction module, there are several convolutional layers with stride larger than 1 to reduce the spatial size of the feature maps to 1 × 1. After dimension reduction, we obtained the sample-specific features at different levels. The loss function for the autoencoder network is the binary cross entropy loss DISPLAYFORM0 N pix is the number of pixels in the image. o i is the value of the ith element in the reconstructed image and t i is the value of the ith element in the input image. Both the input image and the output image are normalized to. The filter generation process is shown in FIG1. The input to the filter generation module is the sample-specific feature vector and the output is a set of the generated filters. If we ignore the bias term, a filter can be flatten to a vector. Given an input feature vector, the naive way to generate filters is to use a fully connected layer to directly map the input vector to the filters. However, it is infeasible when the number of filters is large. Let the length of each filter be L k and the length of the feature vector be L f. If we need to generate N filter vectors from the feature vector. We need DISPLAYFORM0 In order to tackle the problem, we refactor each filter vector k i as DISPLAYFORM1 w j is the coefficient of the base filter b j which is from a filter repository. M is the number of filters in the filter repository. Equation 2 assumes that each filter vector can be generated by a set of base filters. The assumption holds true if M = L K and those base filters are orthogonal. However, in real applications of CNNs, each convolutional layer has limited number of filters which indicates that compared to the large dimension of the filter vector space, only a small subspace is used in the final trained model. Based on this observation, we set M << L k in this work. The total number of parameters in the transformation matrix is N ×L f ×M which is much smaller than the original size. The filters in the repositories are orthogonally initialized and optimized during the training process. The prediction network is the network for the high level task, such as image classification, recognition, detection and so on. The filters used in the prediction network are provided by the filter generation module while the weights of the classifier in the prediction network are learned during the training process. Loss functions for high level tasks are task-dependent. In this work, we will use classification task for demonstration and the loss is the negative log likelihood loss DISPLAYFORM0 where t is the image label and p t is the softmax probability of the tth label. Therefore, the entire loss function for training our model is DISPLAYFORM1 The proposed method aims for generating dynamic filters to deal with variations and improve the performance of a baseline network. In the following experiments, we evaluate our method on three tasks, i.e. digit classification on MNIST dataset(Section 4.1), facial landmark detection on MTFL dataset (Section 4.1) and image classification on CIFAR10 dataset (Section 4.3). The number of the base filters in each filter repository is the same as the number of the filters in each layer of the prediction network if not specified. We will also present further analysis on the generated filters in Section 4.4. Details of all network structures are given in Appendix A.1. To begin our evaluation, we firstly set up a simple experiment on digit classification using MNIST dataset BID8 ). We will show the accuracy improvement brought by our dynamic filters by comparing the performance difference of a baseline network with and without our dynamic filters. We will also analyze how the size of the encoder network and the size of the filter repository (the number of filters in the repository) effect the accuracy of digit classification. The baseline model used in this experiment is a small network with two convolutional layers followed by a fully connected layer that outputs ten dimensions. For simplicity, we only use five filters in each convolutional layer. Details of the network structures are shown in Appendix A.1.1. To evaluate the effect of the size of the encoder network, we compare the classification accuracy obtained when the encoder network has different numbers of filters in each layer. Let n enc be the number of filters in each layer of the encoder network. We choose n enc from {5, 10, 20}. We also choose different repository size s from {2, 5, 10}. In the evaluation of the effect of s, we fix n enc = 20 and we fix s = 5 to evaluate the effect of n enc. We train this baseline model with and without filter generation for 20 epochs respectively. We show the classification accuracy on the test set in TAB0. The first row shows the test accuracy after training the network for only one epoch and the second row shows the final test accuracy. From both tables, we can find that the final test accuracy of the baseline model using our dynamically generated filters is higher than that using fixed filters. The highest accuracy obtained by our generated filters is 99.1% while the accuracy of the fixed filters is 98.25%.Interestingly, the test accuracies after the first epoch (first row in TAB0) show that our dynamically generated filters help the network fit the data better than the original baseline model. It could be attribute to the flexibility of the generated filters. Though there are only a small number of base filters in the repository, linear combination of those base filters can provide filters that efficiently extract discriminative features from input images. In TAB0, when s = 5, the classification accuracy increases as encoder network has more filters. It is straightforward because with more filters, the encoder network can better capture the variation in the input image. So it can provide more information for the generation of filters. Based on the observation from TAB1, it seems that the final classification accuracy is less dependent on the repository size given n enc = 20. In this section, we apply our filter generation to the task of facial landmark detection. To give a more straightforward understanding of the usefulness of our filter generation, we firstly investigate the performance difference of a baseline model before and after some explicit variations are added to the dataset. Then we show the detection performance improvement with respect to the size of the detection network. Dataset. MTFL dataset BID15 ) contains 10,000 face images with ground truth landmark locations. In order to compare the performance difference of baseline models with respect to variations, we construct two datasets from the original MTFL dataset. Rotation variation is used here since it can be easily introduced to the images by manually rotating the face images. Dataset D-Align. We follow BID14 to aligned all face images and crop the face region to the size of 64 × 64.Dataset D-Rot. This dataset is constructed based on D-Align. we randomly rotate all face images within [−45 DISPLAYFORM0 Some image samples for both datasets are shown in Appendix A.2 Figure 6 .We split both datasets into the training dataset containing 9,000 images and the test dataset containing 1,000 images. Note that the train-test splits in D-Align and D-Rot are identical. Models Here we train two baseline models based on UNet BID10). The baseline models are M odel 32 with 32 filters in each convolutional layer and M odel 64 with 64 filters in each convolutional layer. M odel 32 and M odel 64 share the same architecture. Details of the network structures are shown in Appendix A.1.2.We firstly trained M odel 32 and M odel 64 on D-Align and D-Rot without our filter generation module. Then we train them on D-Rot with our filter generation module. For evaluation, we use two metrics here. One is the mean error which is defined as the average landmark distance to groundtruth, normalized as percentages with respect to interocular distance (Burgos-Artizzu et al. FORMULA0). The other is the maximal normalized landmark distance. Since there are more rotation variations in D-Rot than D-Align, we can consider landmark detection task on D-Rot is more challenging than that on D-Align. This is also proved by the increase in detection error when the dataset is switched from D-Align to D-Rot as shown in FIG2 and TAB2. However, when we train the same baseline model with our generated filters, the detection error decreases, compared to the same model trained on D-Rot. There is also a large error drop in maximal detection error. These indicate that using filters conditioned on the input image can reduces the effect of variations in the dataset. Comparing the averaged mean error in FIG2 and FIG2, we find that the performance gain brought by filter generation is larger on M odel 32 than that on M odel 64. It could be explained by the capacity of the baseline models. The capacity of M odel 64 is larger than that of M odel 32. So M odel 64 can handle more variations than M odel 32, so the performance gain on M odel 64 is smaller. FORMULA2 ) dataset consists of natural images with more variations. We evaluate models on this dataset to show the effectiveness of our dynamic filter generation in this challenging scenario. We construct a small baseline network with only four convolutional layers followed by a fully connected layer. We train this model on CIFAR10 firstly without filter generation and then with filter generation. We also train a VGG11 model BID12 ) on this dataset. The are shown in FIG3. From the training accuracy curves, we observe that the baseline model trained without filter generation doesn't fit the data as well as other models. This is because there are only five layers in the network which limits the network's capacity. When the baseline model is trained with filter generation, the model can fit the data well, reaching more than 98% training accuracy. VGG11 also achieves high training accuracy which is not supervising since there are more layers (eleven layers) in the models. The test accuracy curves also show the benefit of adopting our dynamic filter generation. The baseline classification accuracy is improved by ∼1% by using filter generation and the test accuray is comparable to VGG11.Based on the above evaluations on different datasets, we claim that dynamically generated filters can help improve the performance of the baseline models. Using linear combination of base filters from filter repositories can generate effective filters for high level tasks. In this section, we visualize the distributions of the coefficients, the generated filters and the feature maps using MNIST dataset. Then we conduct another experiment on CIFAR10 dataset to demonstrate that the generated filters are sample-specific. DISPLAYFORM0 Figure 5: Visualization of the distributions of the generated coefficients, filters, and feature maps from the first (top row) and the second (bottom row) convolutional layer. The model we used for visualization is the baseline model trained in MNIST experiment with n ae = 20 and s = 5. TSNE BID9 ) is applied to project the high dimensional features into a two-dimensional space. The visualization are shown in Figure 5. In the first row, we show the distributions of the coefficients, the filters, and the feature maps from the first convolutional layer of the model. We observe that the generated filters are shared by certain categories but not all the categories. It is clear in Figure 5a that the coefficients generated by some digits are far away from those by other digits. Nevertheless, the feature maps from the first convolution layer show some separability. In the second row, the generated coefficients and the generated filters forms into clusters which means digits from different categories activate different filters. This behavior makes the final feature maps more separable. We further analyze the generated filters to show that those filters are sample-specific. We take CI-FAR10 dataset as an example. The model used here is the same trained model used in the CIFAR10 experiment (Section 4.3). In this experiment, we feed a test image A to the classification network and another different image B to generate filters. In other words, the filters that will be used in the classification network are not generated from A but from B. This time the classification accuracy of the classification network falls to 15.24%, which is nearly the random guess. This accuracy drop demonstrates that the generated filters are sample-specific. Filters generated from one image doesn't work on the other image. In this paper, we propose to learn to generate filters for convolutional neural networks. The filter generation module transforms features from an autoencoder network to sets of coefficients which are then used to linearly combine base filters in filter repositories. Dynamic filters increase model capacity so that a small model with dynamic filters can also be competitive to a deep model. Evaluation on three tasks show the accuracy improvement brought by our filter generation. In this section, we show the details of the network structures used in our experiments. When we extract sample-specific features, we directly take the convolution feature maps (before LReLU layer) from the autoencoder network as input and feed them to the dimension reduction network. The entire process of sample-specific feature extraction is split into the autoencoder network and the dimension reduction network for the purpose of plain and straightforward illustration. The networks used in the MNIST experiment are shown from TAB3 to TAB6. The networks used in the MTFL experiment are shown from TAB0. The networks used in the CIFAR10 experiment are shown from TAB0 A.2 IMAGE SAMPLES FROM DATASET D-Align AND DATASET D-Rot

Represent the TLDR for finding the corresponding scientific article.

dynamically generate filters conditioned on the input image for CNNs in each forward pass

Represent the short summary for finding the corresponding scientific article (long)

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question: represent the scientific article to find a short tldr:

Question: We propose a new anytime neural network which allows partial evaluation by subnetworks with different widths as well as depths. Compared to conventional anytime networks only with the depth controllability, the increased architectural diversity leads to higher resource utilization and consequent performance improvement under various and dynamic resource budgets. We highlight architectural features to make our scheme feasible as well as efficient, and show its effectiveness in image classification tasks. When we deploy deep neural network models on resource-constrained mobile devices or autonomous vehicles with a strict real-time latency requirement, it is essential to develop a model which makes the best of available resources. Although many network compaction techniques including distillation, pruning and quantization have been proposed BID7 BID12 BID4 BID3 BID9 BID6 BID13, this goal is still challenging because the resource availabilities are continuously changing over time while these resources are being shared with other program instances BID1, and multiple resources with different characteristics (e.g. computational capacity, memory usage) should be considered together. Anytime machine learning algorithms have addressed the first issue, how to get the optimal performance under dynamic resource budgets, by allowing us to activate only a part of the model with graceful output quality degradation BID14 BID2. Most anytime algorithms based on deep neural networks appear in the form of early termination of forward processing according to the current resource budget or the difficulty of the given task BID8 BID0 BID10. In other words, the conventional anytime networks are trained to embed many potential sub-networks with different effective depths so that the best one can be chosen according to the current budget. In this work, we propose a new type of the anytime neural network, doubly nested network, to solve the other issue, more efficient utilization of multiple heterogeneous resources. The proposed network can be sliced along the width as well as the depth to generate more diverse sub-networks than the conventional anytime networks allowing the sub-network extraction only along the depth-wise direction. As depicted in FIG0, the increased degree of freedom enables us to get higher resource utilization in the devices constrained by dynamically changing resource budget with multiple criteria. Causal convolution It is straightforward to form a sub-network along the depth by appending a separate output generation stage to the final convolution layer of the sub-network. Since one specific layer's output does not depend on the following (upper) layers' outputs, the jointly trained subnetwork does not suffer from the performance degradation even after the extraction. However, this approach does not work along the width because of the interdependence between two nodes at different horizontal locations (e.g. different channels). To address this issue, we propose a channelcausal convolution where i-th channel group in one layer is calculated only with activation values from the channel groups from the first to i-th channel group in the previous layer as shown in the right of FIG1. The circle indicates the feature map while the square indicates the classifier. Color refers each channel. Our network based on the causal convolution allows us to extract the sub-network easily along any directions by making both horizontal and vertical data flow unidirectionally. Output generation stage sharing fully-connected layers Rather than having a separate fullyconnected (FC) layer for one sub-network to generate the final output (e.g. a predicted class given image input), our network is designed to have the FC layers each of which takes only a part of activations from the preceding convolution layers and produce the final output for one sub-network by averaging multiple FC layers' outputs as depicted in FIG1. Sharing the FC layers between the sub-networks at the same depth helps us to have similar computational and memory costs of the FC layers in the depth-controllable anytime network BID8 even with much more possible output locations. We can obtain a loss function for each sub-network: DISPLAYFORM0 where L and C refer to the number of possible vertical and horizontal partitions and N is the number of classes. y i is a target label of class i andŷ DISPLAYFORM1 Experimental setup We evaluated the proposed method on the CIFAR-10 and the SVHN datasets. Similarly to the ResNet-32 model BID5, our full network architecture consists of one convolution layer fed by external input, the following 15 residual blocks and fully-connected layers for the final output generation. The network has 16 possible output locations along the depth from the first convolution layer and all residual blocks, and 22 locations along the width. Thus, we can extract 16×22 sub-networks with different widths and depths from the base network. Resource usages of the sub-networks As the selected sub-network gets deeper or wider, all computational and memory requirements such as the number of MAC (multiply-accumulate) operations, the number of parameters and the size of the largest feature map increase. However, their rates of the increase are different from each other as shown in FIG3. This means that our scheme can benefit from the larger diversity of resource usage compared to the conventional anytime methods. Comparison with other methods One of the key advantages of the proposed architecture is nontrivial nesting of sub-networks along the width direction. FIG4 shows that our scheme outperforms two straightforward vertical slicing schemes (Brute-force slicing, Fine-tuning) that can generate sub-networks with different widths to a large extent without significant performance degradation compared to the upper bound (Full training). We revealed that resource-constrained devices could benefit from the architectural diversity enriched by our anytime prediction scheme. Our future works include adding adaptive conditioning BID11 which modulates intermediate activations or weight parameters depending on the current sub-network configuration to improve the performance only with a small increase of conditioning parameters.

Represent the short summary for finding the corresponding scientific article (long)

We propose a new anytime neural network which allows partial evaluation by subnetworks with different widths as well as depths.

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.