Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: We exploit a recently derived inversion scheme for arbitrary deep neural networks to develop a new semi-supervised learning framework that applies to a wide range of systems and problems. The approach reaches current state-of-the-art methods on MNIST and provides reasonable performances on SVHN and CIFAR10. Through the introduced method, residual networks are for the first time applied to semi-supervised tasks. Experiments with one-dimensional signals highlight the generality of the method. Importantly, our approach is simple, efficient, and requires no change in the deep network architecture. Deep neural networks (DNNs) have made great strides recently in a wide range of difficult machine perception tasks. They consist of parametric functionals f Θ with internal parameters Θ. However, those systems are still trained in a fully supervised fashion using a large set of labeled data, which is tedious and costly to acquire. Semi-supervised learning relaxes this requirement by leaning Θ based on two datasets: a labeled set D of N training data pairs and an unlabeled set D u of N u training inputs. Unlabeled training data is useful for learning as unlabeled inputs provide information on the statistical distribution of the data that can both guide the learning required to classify the supervised dataset and characterize the unlabeled samples in D u hence improve generalization. Limited progress has been made on semi-supervised learning algorithms for DNNs BID15; BID16 BID14, but today's methods suffer from a range of drawbacks, including training instability, lack of topology generalization, and computational complexity. In this paper, we take two steps forward in semi-supervised learning for DNNs. First, we introduce an universal methodology to equip any deep neural net with an inverse that enables input reconstruction. Second, we introduce a new semi-supervised learning approach whose loss function features an additional term based on this aforementioned inverse guiding weight updates such that information contained in unlabeled data are incorporated into the learning process. Our key insight is that the defined and general inverse function can be easily derived and computed; thus for unlabeled data points we can both compute and minimize the error between the input signal and the estimate provided by applying the inverse function to the network output without extra cost or change in the used model. The simplicity of this approach, coupled with its universal applicability promise to significantly advance the purview of semi-supervised and unsupervised learning. The standard approach to DNN inversion was proposed in BID4, and the only DNN model with reconstruction capabilities is based on autoencoders BID13. While more complex topologies have been used, such as stacked convolutional autoencoder BID11, thereThe semi-supervised with ladder network approach BID15 employs a per-layer denoising reconstruction loss, which enables the system to be viewed as a stacked denoising autoencoder which is a standard and until now only way to tackle unsupervised tasks. By forcing the last denoising autoencoder to output an encoding describing the class distribution of the input, this deep unsupervised model is turned into a semi-supervised model. The main drawback of this method is the lack of a clear path to generalize it to other network topologies, such as recurrent or residual networks. Also, the per-layer "greedy" reconstruction loss might be too restrictive unless correctly weighted pushing the need for a precise and large cross-validation of hyper-parameters. The probabilistic formulation of deep convolutional nets presented in BID14 natively supports semi-supervised learning. The main drawbacks of this approach lies in the requirement that the activation functions be ReLU and that the overall network topology follows a deep convolutional network. Temporal Ensembling for Semi-Supervised Learning BID8 propose to constrain the representations of a same input stimuli to be identical in the latent space despite the presence of dropout noise. This search of stability in the representation is analogous to the one of a siamese network BID6 but instead of presenting two different inputs, the same is used through two different models (induced by dropout). This technique provides an explicit loss for the unsupervised examples leading to the Π model just described and a more efficient method denoted as temporal ensembling. Distributional Smoothing with Virtual Adversarial Training BID12 proposes also a regularization term contraining the regularity of the DNN mapping for a given sample. Based on this a semi-supervised setting is derived by imposing for the unlabeled samples to maintain a stable DNN. Those two last described methods are the closest one of the proposed approach in this paper for which, the DNN stability will be replaced by a reconstruction ability, closely related to the DNN stability. This paper makes two main contributions: First, we propose a simple way to invert any piecewise differentiable mapping, including DNNs. We provide a formula for the inverse mapping of any DNN that requires no change to its structure. The mapping is computationally optimal, since the input reconstruction is computed via a backward pass through the network (as is used today for weight updates via backpropagation). Second, we develop a new optimization framework for semisupervised learning that features penalty terms that leverage the input reconstruction formula. A range of experiments validate that our method improves significantly on the state-of-the-art for a number of different DNN topologies. In this section we review the work of BID1 aiming at interpreting DNNs as linear splines. This interpretation provides a rigorous mathematical justification for the reconstruction in the context of deep learning. Recent work in BID1 demonstrated that DNNs of many topologies are or can be approximated arbitrary closely by multivariate linear splines. The upshot of this theory for this paper is that it enables one to easily derive an explicit input-output mapping formula. As a , DNNs can be rewritten as a linear spline of the form To illustrate this point we provide for two common topologies the exact input-output mappings. For a standard deep convolutional neural network (DCN) with succession of convolutions, nonlinearities, and pooling, one has DISPLAYFORM0 DISPLAYFORM1 where z (x) represents the latent representation at layer for input x. The total number of layers in a DNN is denoted as L and the output of the last layer z (L) (x) is the one before application of the softmax application. After application of the latter, the output is denoted byŷ(x). The product terms are from last to first layer as the composition of linear mappings is such that layer 1 is applied on the input, layer 2 on the output of the previous one and so on. The bias term simply from the accumulation of all of the per-layer biases after distribution of the following layers' templates. For a Resnet DNN, one has DISPLAYFORM2 We briefly observe the differences between the templates of the two topologies. The presence of an extra term in DISPLAYFORM3 σ C provides stability and a direct linear connection between the input x and all of the inner representations z (x), hence providing much less information loss sensitivity to the nonlinear activations. Based on those findings, and by imposing a simple 2 norm upper bound on the templates, it has been shown that the optimal templates DNNs to perform prediction has templates proportional to the input, positively for the belonging class and negatively for the others BID1. This way, the loss cross-entropy function is minimized when using softmax final nonlinearity. Note that this is specific to this setting. For example in the case of spherical softmax the optimal templates become null for the incorrect classes of the input. Theorem 1. In the case where all inputs have identity norm ||X n || = 1, ∀n and assuming all templates denoted by DISPLAYFORM4 We now leverage the analytical optimal DNN solution to demonstrate that reconstruction is indeed implied by such an optimum. Based on the previous analysis, it is possible to draw implications based on the theoretical optimal templates of DNNs. This is formulated through the corollary below. First, we propose the following inverse of a DNN as DISPLAYFORM0 Following the analysis from a spline point of view, this reconstruction is leveraging the closest input hyperplane, found through the forward step, to represent the input. As a this method provides a reconstruction based on the DNN representation of its input and should be part away from the task of exact input reconstruction which is an ill-posed problem in general. The bias correction present has insightful meaning when compared to known frameworks and their inverse. In particular, when using ReLU based nonlinearities we will see that this scheme can be assimilated to a composition of soft-thresholding denoising technique. We present further details in the next section where we also provide ways to efficiently invert a network as well as describing the semi-supervised application. We now apply the above inverse strategy to a given task with an arbitrary DNN. As exposed earlier, all the needed changes to support semi-supervised learning happen in the objective training function by adding extra terms. In our application, we used automatic differentiation (as in and). Then it is sufficient to change the objective loss function, and all the updates are adapted via the change in the gradients for each of the parameters. The efficiency of our inversion scheme is due to the fact that any deep network can be rewritten as a linear mapping BID1. This leads to a simple derivation of a network inverse defined as f −1 that will be used to derive our unsupervised and semi-supervised loss function via DISPLAYFORM0 The main efficiency argument thus comes from DISPLAYFORM1 which enables one to efficiently compute this matrix on any deep network via differentiation (as it would be done to back-propagate a gradient, for example). Interestingly for neural networks and many common frameworks such as wavelet thresholding, PCA, etc., the reconstruction error as (DISPLAYFORM2 is the definition of the inverse transform. For illustration purposes, Tab. 1 gives some common frameworks for which the reconstruction error represents exactly the reconstruction loss. DISPLAYFORM3 We now describe how to incorporate this loss for semi-supervised and unsupervised learning. We first define the reconstruction loss R as DISPLAYFORM4 While we use the mean squared error, any other differentiable reconstruction loss can be used, such as cosine similarity. We also introduce an additional "specialization" loss defined as the Shannon entropy of the class belonging probability prediction DISPLAYFORM5 This loss is intuitive and complementary to the reconstruction for the semi-supervised task. In fact, it will force a clustering of the unlabeled examples toward one of the clusters learned from the supervised loss and examples. We provide below experiments showing the benefits of this extraterm. As a , we define our complete loss function as the combination of the standard cross entropy loss for labeled data denoted by L CE (Y n,ŷ(X n)), the reconstruction loss, and the entropy loss as DISPLAYFORM6 with α, β ∈ 2. The parameters α, β are introduced to form a convex combination of the losses,with α controlling the ratio between supervised and unsupervised loss and β the ratio between the two unsupervised losses. This weighting is important, because the correct combination of the supervised and unsupervised losses will guide learning toward a better optimum (as we now demonstrated via experiments). We now present of our approach on a semi-supervised task on the MNIST dataset, where we are able to obtain reasonable performances with different topologies. MNIST is made of 70000 grayscale images of shape 28 × 28 which is split into a training set of 60000 images and a test set of 10000 images. We present for the case with N L = 50 which represents the number of samples from the training set that are labeled and fixed for learning. All the others samples form the training set are unlabeled and thus used only with the reconstruction and entropy loss minimization. We perform a search over (α, β) ∈ {0.3, 0.4, 0.5, 0.6, 0.7} × {0.2, 0.3, 0.5}. In addition, 4 different topologies are tested and, for each, mean and max pooling are tested as well as inhibitor DNN (IDNN) as proposed in BID1. The latter proposes to stabilize training and remove biases units via introduction of winner-share-all connections. As would be expected based on the templates differences seen in the previous section, the Resnet topologies are able to reach the best performance. In particular, wide Resnet is able to outperform previous state-of-the-art . Running the proposed semi-supervised scheme on MNIST leads to the presented in Tab. 2. We used the Theano and Lasagne libraries; and learning procedures and topologies are detailed in the appendix. The column 1000 corresponds to the accuracy after training of DNNs using only the supervised loss (α = 1, β = 0) on 1000 labeled data. Thus, one can see the gap reached with the same network but with a change of loss and 20 times less labeled data. We further present performances on CIFAR10 with 4000 labeled data in Tab. 3 and SVHN with 500 labeled data in Tab. 4. For both tasks we constrain ourselves to a deep CNN models, similar as the LargeCNN of BID14. Also, one of the cases correspond to the absence of entropy loss when β = 1. Furthermore to further present the generalization of the inverse technique we provide with the leaky-ReLU nonlinearity as well as the sigmoid activation function. We now present and example of our approach on a supervised task on audio database (1D). It is the Bird10 dataset distributed online and described in BID5. The task is to classify 10 bird species from their songs recorded in tropical forest. It is a subtask of the BirdLifeClef challenge. 75.28 ± 0.2 73.05 ± 0.4 N L CIFAR4000, CIFAR8000, 81.37 ± 2.32 82.28 ± 1.82 LadderNetwork BID15 79.6 ± 0.47 -catGAN BID17 80.42 ± 0.46 -DRMM +KL penalty BID14 76.76 - 83.01 ± 0.36 -Semi-Sup Requires a 85.59 ± 0.30 -ΠModelLaine & Aila FORMULA0 83.45 ± 0.29 - 88.25 ± 1.12 88.39 ± 0.9 (0.85,1) 80.42 ± 2.4 79.77 ± 1.5 N L SVHN500, SVHN1000, 81. FORMULA0 92.95 ± 0.3 94.57 ± 0.25 VATMiyato et al. FORMULA0 -75.37We train here networks based on raw audio using CNNs as detailed in the appendix. We vary (α, β) over 10 runs to demonstrate that the non-regularized supervised model is not optimal. The maximum validation accuracies on the last 100 epochs FIG3 show that the regularized networks tend to learn more slowly, but always generalize better than the not regularized baseline (α = 1, β = 0). We have presented a well-justified inversion scheme for deep neural networks with an application to semi-supervised learning. By demonstrating the ability of the method to best current state-of-theart on MNIST with different possible topologies support the portability of the technique as well as its potential. These open up many questions in this yet undeveloped area of DNN inversion, input reconstruction, and their impact on learning and stability. Among the possible extensions, one can develop the reconstruction loss into a per-layer reconstruction loss. Doing so, there is the possibility to weight each layer penalty bringing flexibility as well as meaningful reconstruction. Define the per layer loss as DISPLAYFORM0 with DISPLAYFORM1 Doing so, one can adopt a strategy in favor of high reconstruction objective for inner layers, close to the final latent representation z (L) in order to lessen the reconstruction cost for layers closer to the input X n. In fact, inputs of standard dataset are usually noisy, with , and the object of interest only contains a small energy with respect to the total energy of X n. Another extension would be to update the weighting while performing learning. Hence, if we denote by t the position in time such as the current epoch or batch, we now have the previous loss becoming DISPLAYFORM2 One approach would be to impose some deterministic policy based on heuristic such as favoring reconstruction at the beginning to then switch to classification and entropy minimization. Finer approaches could rely on explicit optimization schemes for those coefficients. One way to perform this, would be to optimize the loss weighting coefficients α, β, γ after each batch or epoch by backpropagation on the updates weights. Define DISPLAYFORM3 as a generic iterative update based on a given policy such as gradient descent. One can thus adopt the following update strategy for the hyper-parameters as DISPLAYFORM4 and so for all hyper-parameters. Another approach would be to use adversarial training to update those hyper-parameters where both update cooperate trying to accelerate learning. EBGAN BID18 ) are GANs where the discriminant network D measures the energy of a given input X. D is formulated such as generated data produce high energy and real data produce lower energy. Same authors propose the use of an auto-encoder to compute such energy function. We plan to replace this autoencoder using our proposed method to reconstruct X and compute the energy; hence D(X) = R(X) and only one-half the parameters will be needed for D.Finally, our approach opens the possibility of performing unsupervised tasks such as clustering. In fact, by setting α = 0, we are in a fully unsupervised framework. Moreover, β can push the mapping f Θ to produce a low-entropy, clustered, representation or rather simply to produce optimal reconstruction. Even in a fully unsupervised and reconstruction case (α = 0, β = 1), the proposed framework is not similar to a deep-autoencoder for two main reasons. First, there is no greedy (per layer) reconstruction loss, only the final output is considered in the reconstruction loss. Second, while in both case there is parameter sharing, in our case there is also "activation" sharing that corresponds to the states (spline) that were used in the forward pass that will also be used for the backward one. In a deep autoencoder, the backward activation states are induced by the backward projection and will most likely not be equal to the forward ones. We thank PACA region and NortekMed, and GDR MADICS CNRS EADM action for their support. We give below the figures of the reconstruction of the same test sample by four different nets: LargeUCNN (α = 0.5, β = 0.5), SmallUCNN (0.6,0.5), 0.5), 0.5). The columns from left to right correspond to: the original image, mean-pooling reconstruction, maxpooling reconstruction, and inhibitor connections. One can see that our network is able to correctly reconstruct the test sample.

represent the tldr for finding the corresponding scientific article

We exploit an inversion scheme for arbitrary deep neural networks to develop a new semi-supervised learning framework applicable to many topologies.

We show how to initialize recurrent architectures with the closed-form solution of a linear autoencoder for sequences. We show the advantages of this approach compared to orthogonal RNNs.


Question: Represent the scientific article to find a short TLDR

Question: Deep learning has become a widely used tool in many computational and classification problems. Nevertheless obtaining and labeling data, which is needed for strong , is often expensive or even not possible. In this paper three different algorithmic approaches to deal with limited access to data are evaluated and compared to each other. We show the drawbacks and benefits of each method. One successful approach, especially in one- or few-shot learning tasks, is the use of external data during the classification task. Another successful approach, which achieves state of the art in semi-supervised learning (SSL) benchmarks, is consistency regularization. Especially virtual adversarial training (VAT) has shown strong and will be investigated in this paper. The aim of consistency regularization is to force the network not to change the output, when the input or the network itself is perturbed. Generative adversarial networks (GANs) have also shown strong empirical . In many approaches the GAN architecture is used in order to create additional data and therefor to increase the generalization capability of the classification network. Furthermore we consider the use of unlabeled data for further performance improvement. The use of unlabeled data is investigated both for GANs and VAT. Deep neural networks have shown great performance in a variety of tasks, like speech or image recognition. However often extremely large datasets are necessary for achieving this. In real world applications collecting data is often very expensive in terms of cost or time. Furthermore collected data is often unbalanced or even incorrect labeled. Hence performance achieved in academic papers is hard to match. Recently different approaches tackled these problems and tried to achieve good performance, when otherwise fully supervised baselines failed to do so. One approach to learn from very few examples, the so called few-shot learning task, consists of giving a collection of inputs and their corresponding similarities instead of input-label pairs. This approach was thoroughly investigated in BID9, BID33, BID28 and gave impressive tested on the Omniglot dataset BID12 ). In essence a task specific similarity measure is learned, that embeds the inputs before comparison. Furthermore semi-supervised learning (SSL) achieved strong in image classification tasks. In SSL a labeled set of input-target pairs (x, y) ∈ D L and additionally an unlabeled set of inputs x ∈ D U L is given. Generally spoken the use of D U L shall provide additional information about the structure of the data. Generative models can be used to create additional labeled or unlabeled samples and leverage information from these samples BID26, BID18 ). Furthermore in BID2 it is argued, that GAN-based semi-supervised frameworks perform best, when the generated images are of poor quality. Using these badly generated images a classifier with better generalization capability is obtained. On the other side uses generative models in order to learn feature representations, instead of generating additional data. Another approach in order to deal with limited data is consistency regularization. The main point of consistency regularization is, that the output of the network shall not change, when the input or the network itself is perturbed. These perturbations may also in inputs, which are not realistic anymore. This way a smooth manifold is found on which the data lies. Different approaches to consistency regularization can be found in BID15, BID23, BID11, and BID32.The aim of this paper is to investigate how different approaches behave compared to each other. Therefore a specific image and sound recognition task is created with varying amount of labeled data. Beyond that it is further explored how different amounts of unlabeled data support the tasks, whilst also varying the size of labeled data. The possible accuracy improvement by labeled and unlabeled examples is compared to each other. Since there is a correlation between category mismatch of unlabeled data and labeled data BID20 ) reported, we investigate how this correlation behaves for different approaches and datasets. When dealing with little data, transfer learning BID34, BID0 ) offers for many use cases a good method. Transfer learning relies on transferring knowledge from a base model, which was trained on a similar problem, to another problem. The weights from the base model, which was trained on a seperate big dataset, are then used as initializing parameters for the target model. The weights of the target model are afterwards fine-tuned. Whilst often yielding good , nevertheless a similar dataset for the training of the base model is necessary. Many problems are too specific and similar datasets are not available. In BID15 transfer learning achieves better than any compared consistency regularization method, when transferring from ImageNet BID3 ) to CIFAR-10 . On contrast, no convincing could be achieved when transferring from ImageNet to SVHN BID16, although the task itself remains a computer vision problem. Therefore the generalization of this approach is somehow limited. In order to increase the generalization of this work transfer learning is not investigated. Instead this paper focuses on generative models, consistency regularization, and the usage of external data during the classification of new samples. Since there exist several algorithms for each of these approaches, only one representative algorithm for each of the three approaches is picked and compared against each other. The usage of external data after training during the classification task is a common technique used in few shot learning problems. Instead of input-label pairs, the network is trained with a collection of inputs and their similarities. Due to its simplicity and good performance the approach by BID9, which is inspired by , is used in this paper. BID9 uses a convolutional siamese neural network, which basically learns an embedding of the inputs. The same convolutional part of the network is used for two inputs x 1 and x 2. After the convolution each input is flattened into a vector. Afterwards the L 1 distance between the two embeddings is computed and fed into a fully-connected layer, which outputs a similarity between.In order to classify a test image x into one of K categories, a support set {x k} K k=1 with examples for each category is used. The input x is compared to each element in the support set and the category corresponding to the maximum similarity is returned. When there are more examples per class the query can be repeated several times, such that the network returns the class with the highest average similarity. Using this approach is advantageous, when the number of categories is high or not known at all. On the downside the prediction of the category depends on a support set and furthermore the computational effort of predicting a category increases with O(K), since a comparison has to be made for each category. Consistency regularization relies on increasing the robustness of a network against tiny perturbations of the input or the network. For perturbations of the input d (f (x; θ), f (x; θ)) shall be minimized, whereas d is a distance measurement like euclidean distance or Kullback-Leibler divergence andx is the perturbed input. It is possible to sample x from both D L and D U L.An empirical investigation BID20 has shown, that many consistency regularization methods, like mean teacher BID32 ), Π-model BID23, BID11 ), and virtual adversarial training (VAT) BID15 are quite hard to compare, since the may rely on many parameters (network, task, etc.). Nevertheless VAT is chosen in this work, since it achieves convincing on many tasks. VAT is a training method, which is greatly inspired by adversarial training BID5 ). The perturbation r adv of the input x can be computed as DISPLAYFORM0,where ξ and are hyperparameters, which have to be tuned for each task. After the perturbation was added to x consistency regularization is applied. The distance between the clean (not perturbed) prediction and perturbed prediction d(f (x, θ), f (x + r adv, θ)) shall be minimized. In order to reduce the distance the gradients are just backpropagated through f (x+r adv). Combining VAT with entropy minimization BID6 it is possible to further increase the performance BID15. For entropy minimization an additional loss term is computed as: DISPLAYFORM1 and added to the overall loss. This way the network is forced to make more confident predictions regardless of the input. Generative models are commonly used for increasing the accuracy or robustness of models in a semior unsupervised manner, BID35 BID29, BID18, ).A popular approach is the use of generative adversarial neural networks (GANs), introduced by BID4. The goal of a GAN is to train a generator network G, wich produces realistic samples by transforming a noise vector z as x f ake = G(z, θ), and a discriminator network D, which has to distinguish between real samples x real ∼ p Data and fake samples x f ake ∼ G.In this paper the training method defined in BID26 is used. Using this approach the output of D consists of K + 1 categories, whereas K is the number of categories the classifier shall be actually trained on. One additional extra category is added for samples generated by D. Since the output of D is over-parameterized the logit output l K+1, which represents the fake category, is permanently fixed to 0 after training. The loss function consists of two parts L supervised and L unsupervised, which can be computed as: DISPLAYFORM0 DISPLAYFORM1 L supervised represents the standard classification loss, i.e. negative log probability. L unsupervised itself again consists of two parts, the first part forces the network to output a low probability of fake category for inputs x ∼ p data and corresponding a high probability for inputs x ∼ G. Since the the category y is not used in L unsupervised, the input x can be sampled from both D L and D U L. In order to further improve the performance feature matching is used, as described in BID26. Three different experiments are conducted in this paper using the MNIST BID13 ) and UrbanSound8k BID24 dataset. The UrbanSound8k dataset consists of 8732 sound clips with a maximum duration of 4 s. Each sound clip represents a different urban noise class like drilling, engine, jackhammer, etc. Before using the sound files for training a neural network, they are prepared in a similar manner to BID25, in essence each sound clip is transferred to a log-scaled mel-spectrogram with 128 components covering the frequency range between 0-22050 Hz. The window size is chosen to be 23 ms and hop size of the same duration. Sound snippets with shorter duration as 4 s are repeated and concatenated until a duration of 4 s is reached. The preprocessing is done using librosa BID14 ). For training and evaluation purposes a random snippet with a length of 3 s is selected, ing in an input size of 128 × 128.In the first experiment no external unlabeled data is used. Instead, the amount of labeled data in each category is varied and the three methods are compared to each other. In the second experiment the amount of labeled and unlabeled data is varied, in order to explore how unlabeled data can compensate labeled data. The last experiment considers class distribution mismatch while the amount of labeled and unlabeled data is fixed. In the second and third experiment only two methods are compared, since only generative models and consistency regularization allow the use of external unlabeled data. All methods are compared to a standard model. When using the MNIST dataset the standard model consists of three convolutional layers, followed by two fully-connected layers. For the UrbanSound8k dataset the standard model consists of four convolutional layers, followed by three fully-connected layers. ReLU nonlinearities were used in all hidden layers. The training was done by using the Adam optimizer BID7 ). Furthermore batch normalization BID19 ), dropout BID30 ), and max-pooling was used between convolutional layers. For further increasing the generalization capability L 2 regularization (Ng FORMULA0) is used. The models, representing the three different approaches, have the same computational power as the standard model, in essence three/ four convolutional layers and two/ three fully connected layers. The number of hidden dimensions and other per layer hyperparameters (e.g. stride, padding) is kept equal to the corresponding standard models. The hyperparameters were tuned manually on the training dataset by performing gridsearch and picking the most promising . Whereas the L 2 and batchnorm coefficients, as well as dropout rate are shared across all models for each dataset. The test accuracy was calculated in all experiments with a separate test dataset, which contains 500 samples per category for the MNIST dataset and, respectively, 200 samples per category for the UrbanSound8k dataset. Train and test set have no overlap. All experiments were conducted using the PyTorch framework BID21 ). In this experiment the amount of labeled data is varied. Furthermore there is not used any unlabeled external data. For each amount of labeled data and training approach (i.e. baseline, , , and siamese neural network BID9) the training procedure was repeated eight times. Afterwards the mean accuracies and standard deviations have been calculated. Figure 1 shows the obtained in this experiment for the MNIST dataset. The amount of labeled data per category was varied on a logarithmic scale in the range between with 31 steps. Using 200 labeled samples per category the baseline network is able to reach about 95 % accuracy. With just one labeled sample per class the baseline networks reaches already around 35 %, which is a already good compared to 10 %, when random guessing. Generally all three methods are consistent with the literature, such that they are superior over baseline in the low data regime (1-10 samples per category). Using a siamese neural network the accuracy can be significantly improved in the low data regime. With just one labeled sample the siamese architecture already reaches around 45 %. When using a dataset with more categories, like Omniglot, the advantage of using siamese networks should be even higher in the low data regime. The performance of this approach becomes worse compared to the baseline model, when using more than 10 labeled examples per class. VAT has a higher benefit compared to GAN for up to 20 labeled samples per category. For higher numbers of labeled samples both methods show only little (0-2 %) improvement over the baseline . Similar are obtained on the UrbanSound8k dataset (figure 2). As for the experiment on the MNIST dataset the amount of labeled data was varied on a logarithmic scale in the range between, but with 6 steps instead of 31, since the computational effort was much higher. The siamese network yields a large improvement when there is only one labeled sample, but fast returns worse than the baseline network. On contrast the usage of VAT or GAN comes with a benefit in terms of accuracy for higher amounts of labeled data. Nevertheless these both methods are either not able to further improve the accuracy for high amounts of labeled data (more than 100). Furthermore the accuracy even declines compared to baseline for 200 labeled samples. The observation, that adversarial training can decrease accuracy, is inline with literature BID27, BID31 ), where it was shown that in high data regimes there may be a trade-off between accuracy and robustness. Whereas in some cases adversarial training can improve accuracy in the low data regime. Both methods show a significant increase in terms of accuracy when the amount of labeled data is low and corresponding the amount of unlabeled data is high. When the amount of labeled data increases the amount of necessary unlabeled data also increases in order to achieve the same accuracy improvements. VAT achieves better with less unlabeled data compared to GAN, when there is little labeled data (∼ 2-10 examples per category). On contrast GANs achieve better when there is a moderate amount of labeled examples (∼ 10-50 examples per category) and also many unlabeled examples. When the amount labeled examples is high both methods behave approximately equal. The for the UrbanSound8k dataset can be seen in FIG2. Overall similar as for the MNIST dataset are achieved, in terms of having high benefits, when the amount of labeled data is low and concurrently the amounts of unlabeled data is high. Nevertheless the total improvement is lower and for high amounts of labeled data, more unlabeled data is necessary in order to get an improvement at all. For the VAT the amount of unlabeled data need to have similar magnitudes as the amount of labeled data in order to get an improvement at all. Further the same observation as before can be made, that VAT achieves better with less unlabeled data, when there is little labeled data In this experiment the possibility of adding additional unlabeled examples, which do not correspond to the target labels (mismatched samples), is investigated. This experiment was done for VAT in BID20. In this work the investigation is extended in such a way that the for VAT are compared to GAN. Furthermore not only the extend of mismatch, but also the influence of the amount of additional unlabeled examples is investigated. Both datasets (MNIST and UrbanSound8k) consist of 10 categories with label values and the aim is to train a neural network, which is able to classify inputs corresponding to categories, hence the network has six outputs. Mismatched examples belong to categories. The number of labeled examples per category is fixed to be five. Having five labeled samples it can be seen in FIG1, that the accuracy improvement shows a strong dependency on the amount of unlabeled samples. The total number of unlabeled examples is varied between {30, 120, 600}. Furthermore the mismatch for each number of unlabeled examples is varied between 0-100 % using a 10 % increment, e.g. when the amount of unlabeled examples is set to be 120 and the mismatch is 70 % the unlabeled examples consist of 84 examples belonging to categories and 36 examples belonging to categories. The distribution across the categories in the six matched and four remaining mismatched classes is kept approximately equal, with a maximum difference of ±1. For each amount of mismatch and method eight neural networks have been trained. Afterwards their average accuracies and standard deviations have been calculated. For baseline also eight neural networks have been trained and their average accuracy and standard deviation computed. Since the number of classes is reduced to 6 the accuracy, when compared to the previous experiments, is higher with the same amount of labeled data. FIG4 shows the of this experiment for the MNIST dataset. Overall the accuracy decreases for both methods when the class mismatch increases, which is in line with literature BID20 ). As in the experiments before, the GAN method shows little to no accuracy improvement, when the additional amount of unlabeled data is low (30 unlabeled samples). For 120 and respectively 600 additional unlabeled elements both methods show an approximate equal maximal accuracy improvement, when there is no class mismatch. When the class mismatch is very high (80-100 %) using VAT in worse performance than baseline . Using GANs the performance is in worst case at the same level as baseline performance. GAN shows a linear correlation between accuracy and class mismatch. On contrast VAT shows a parabolic trend. Overall increasing the amount of unlabeled data seems to increase the robustness towards class mismatch. All in all both methods show an accuracy improvement even for high amounts (> 50 %) of class mismatch. Whereas VAT performs better, when the amount of mismatch is low. FIG5 shows the obtained with the UrbanSound8k dataset. Overall there seems to be no, or only little correlation between class mismatch and accuracy. Only for the GAN, when using 30 or 120 unlabeled samples, a small correlation can be observed. This is a surprising observation, since in the previous experiment and in BID20 a decrease in terms of accuracy is reported for increasing class mismatch. In essence it can be stated, that adding samples, which do not necessarily belong to the target classes, can improve the overall accuracy. This is especially interesting for training classifiers on hard to obtain or rare samples (rare disease, etc.). Nevertheless it has to be checked whether adding this samples hurts the performance or not. Furthermore the correlation between more unlabeled data and accuracy can be observed, as in the previous experiments. In this paper three methods for dealing with little data have been compared to each other. When the amount of labeled data is very little and no unlabeled data is available, siamese neural networks offer the best alternative in order to achieve good in terms of accuracy. Furthermore when there is additional unlabeled data available using GANs or VAT offer a good option. VAT outperforms GAN when the amount of data is low. On contrast GANs should be preferred for moderate or high amounts of data. Nevertheless both methods must be tested for any individual use case, since the behavior of these methods may change for different datasets. Surprising have been obtained on the class mismatch experiment. It was observed that adding samples, which do not belong to the target classes, not necessarily reduce the accuracy. Whether adding such samples improves or reduce the accuracy, may heavily depend on how closely these samples/ classes are related to the target samples/ classes. An interesting questions remains whether datasets which perform good in transfer learning tasks (e.g. transferring from ImageNet to CIFAR-10) also may be suitable for such semi-supervised learning tasks. Furthermore any combinations of three examined methods can bear interesting , e.g. VAT could be applied to the discriminator in the GAN framework. Also a combination of GAN and siamese neural networks could be useful, in this case the siamese neural network would have two outputs, one for the source and one for the similarity.

Represent the short summary for finding the corresponding scientific article (long)

Comparison of siamese neural networks, GANs, and VAT for few shot learning.

Represent the TLDR for finding the corresponding scientific article:

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent this

Question: This paper introduces a new neural structure called FusionNet, which extends existing attention approaches from three perspectives. First, it puts forward a novel concept of "History of Word" to characterize attention information from the lowest word-level embedding up to the highest semantic-level representation. Second, it identifies an attention scoring function that better utilizes the "history of word" concept. Third, it proposes a fully-aware multi-level attention mechanism to capture the complete information in one text (such as a question) and exploit it in its counterpart (such as context or passage) layer by layer. We apply FusionNet to the Stanford Question Answering Dataset (SQuAD) and it achieves the first position for both single and ensemble model on the official SQuAD leaderboard at the time of writing (Oct. 4th, 2017). Meanwhile, we verify the generalization of FusionNet with two adversarial SQuAD datasets and it sets up the new state-of-the-art on both datasets: on AddSent, FusionNet increases the best F1 metric from 46.6% to 51.4%; on AddOneSent, FusionNet boosts the best F1 metric from 56.0% to 60.7%. Context: The Alpine Rhine is part of the Rhine, a famous European river. The Alpine Rhine begins in the most western part of the Swiss canton of Graubünden, and later forms the border between Switzerland to the West and Liechtenstein and later Austria to the East. On the other hand, the Danube separates Romania and Bulgaria. Answer: Liechtenstein Teaching machines to read, process and comprehend text and then answer questions is one of key problems in artificial intelligence. FIG0 gives an example of the machine reading comprehension task. It feeds a machine with a piece of context and a question and teaches it to find a correct answer to the question. This requires the machine to possess high capabilities in comprehension, inference and reasoning. This is considered a challenging task in artificial intelligence and has already attracted numerous research efforts from the neural network and natural language processing communities. Many neural network models have been proposed for this challenge and they generally frame this problem as a machine reading comprehension (MRC) task BID7 BID13 BID17 BID15 BID16;.The key innovation in recent models lies in how to ingest information in the question and characterize it in the context, in order to provide an accurate answer to the question. This is often modeled as attention in the neural network community, which is a mechanism to attend the question into the context so as to find the answer related to the question. ) attend the word-level embedding from the question to context, while some BID13 attend the high-level representation in the question to augment the context. However we observed that none of the existing approaches has captured the full information in the context or the question, which could be vital for complete information comprehension. Taking image recognition as an example, information in various levels of representations can capture different aspects of details in an image: pixel, stroke and shape. We argue that this hypothesis also holds in language understanding and MRC. In other words, an approach that utilizes all the information from the word embedding level up to the highest level representation would be substantially beneficial for understanding both the question and the context, hence yielding more accurate answers. However, the ability to consider all layers of representation is often limited by the difficulty to make the neural model learn well, as model complexity will surge beyond capacity. We conjectured this is why previous literature tailored their models to only consider partial information. To alleviate this challenge, we identify an attention scoring function utilizing all layers of representation with less training burden. This leads to an attention that thoroughly captures the complete information between the question and the context. With this fully-aware attention, we put forward a multi-level attention mechanism to understand the information in the question, and exploit it layer by layer on the context side. All of these innovations are integrated into a new end-to-end structure called FusionNet in FIG5, with details described in Section 3.We submitted FusionNet to SQuAD , a machine reading comprehension dataset. At the time of writing (Oct. 4th, 2017), our model ranked in the first place in both single model and ensemble model categories. The ensemble model achieves an exact match (EM) score of 78.8% and F1 score of 85.9%. Furthermore, we have tested FusionNet against adversarial SQuAD datasets BID9. Results show that FusionNet outperforms existing state-of-the-art architectures in both datasets: on AddSent, FusionNet increases the best F1 metric from 46.6% to 51.4%; on AddOneSent, FusionNet boosts the best F1 metric from 56.0% to 60.7%. In Appendix D, we also applied to natural language inference task and shown decent improvement. This demonstrated the exceptional performance of FusionNet. An open-source implementation of FusionNet can be found at https://github.com/momohuang/FusionNet-NLI. In this section, we briefly introduce the task of machine comprehension as well as a conceptual architecture that summarizes recent advances in machine reading comprehension. Then, we introduce a novel concept called history-of-word. History-of-word can capture different levels of contextual information to fully understand the text. Finally, a light-weight implementation for history-of-word, Fully-Aware Attention, is proposed. In machine comprehension, given a context and a question, the machine needs to read and understand the context, and then find the answer to the question. The context is described as a sequence of word tokens: C = {w C 1, . . ., w C m}, and the question as: DISPLAYFORM0 where m is the number of words in the context, and n is the number of words in the question. In general, m n. The answer Ans can have different forms depending on the task. In the SQuAD dataset , the answer Ans is guaranteed to be a contiguous span in the context C, e.g., Ans = {w C i, . . ., w C i+k}, where k is the number of words in the answer and k ≤ m. In all state-of-the-art architectures for machine reading comprehension, a recurring pattern is the following process. Given two sets of vectors, A and B, we enhance or modify every single vector in set A with the information from set B. We call this a fusion process, where set B is fused into set A. Fusion processes are commonly based on attention BID0, but some are not. Major improvements in recent MRC work lie in how the fusion process is designed. A conceptual architecture illustrating state-of-the-art architectures is shown in FIG2, which consists of three components.• Input vectors: Embedding vectors for each word in the context and the question. RaSoR BID11 DrQA D MPCM D DMnemonic Reader D D D R-net BID13 D D • Integration components: The rectangular box. It is usually implemented using an RNN such as an LSTM BID7 or a GRU BID5 ).• Fusion processes: The numbered arrows,, (2'),, (3'). The set pointing outward is fused into the set being pointed to. There are three main types of fusion processes in recent advanced architectures. TAB0 shows what fusion processes are used in different state-of-the-art architectures. We now discuss them in detail. Word-level fusion. By providing the direct word information in question to the context, we can quickly zoom in to more related regions in the context. However, it may not be helpful if a word has different semantic meaning based on the context. Many word-level fusions are not based on attention, e.g., appends binary features to context words, indicating whether each context word appears in the question. High-level fusion. Informing the context about the semantic information in the question could help us find the correct answer. But high-level information is more imprecise than word information, which may cause models to be less aware of details.(2') High-level fusion (Alternative). Similarly, we could also fuse high-level concept of Q into the word-level of C. Self-boosted fusion. Since the context can be long and distant parts of text may rely on each other to fully understand the content, recent advances have proposed to fuse the context into itself. As the context contains excessive information, one common choice is to perform self-boosted fusion after fusing the question Q. This allows us to be more aware of the regions related to the question.(3') Self-boosted fusion (Alternative). Another choice is to directly condition the self-boosted fusion process on the question Q, such as the coattention mechanism proposed in BID16 ). Then we can perform self-boosted fusion before fusing question information. A common trait of existing fusion mechanisms is that none of them employs all levels of representation jointly. In the following, we claim that employing all levels of representation is crucial to achieving better text understanding. Consider the illustration shown in Figure 3. As we read through the context, each input word will gradually transform into a more abstract representation, e.g., from low-level to high-level concepts. Altogether, they form the history of each word in our mental flow. For a human, we utilize the history-of-word so frequently but we often neglect its importance. For example, to answer the question in Figure 3 correctly, we need to focus on both the high-level concept of forms the border and the word-level information of Alpine Rhine. If we focus only on the high-level concepts, we will Figure 3: Illustrations of the history-of-word for the example shown in FIG0. Utilizing the entire history-of-word is crucial for the full understanding of the context.confuse Alpine Rhine with Danube since both are European rivers that separate countries. Therefore we hypothesize that the entire history-of-word is important to fully understand the text. In neural architectures, we define the history of the i-th word, HoW i, to be the concatenation of all the representations generated for this word. This may include word embedding, multiple intermediate and output hidden vectors in RNN, and corresponding representation vectors in any further layers. To incorporate history-of-word into a wide range of neural models, we present a lightweight implementation we call Fully-Aware Attention. Attention can be applied to different scenarios. To be more conclusive, we focus on attention applied to fusing information from one body to another. Consider two sets of hidden vectors for words in text bodies A and B: {h DISPLAYFORM0 Their associated history-of-word are, In fully-aware attention, we replace attention score computation with the history-of-word. DISPLAYFORM1). This allows us to be fully aware of the complete understanding of each word. The ablation study in Section 4.4 demonstrates that this lightweight enhancement offers a decent improvement in performance. To fully utilize history-of-word in attention, we need a suitable attention scoring function S(x, y). A commonly used function is multiplicative attention BID2 ): DISPLAYFORM0 k×d h, and k is the attention hidden size. However, we suspect that two large matrices interacting directly will make the neural model harder to train. Therefore we propose to constrain the matrix U T V to be symmetric. A symmetric matrix can always be decomposed into DISPLAYFORM1 and D is a diagonal matrix. The symmetric form retains the ability to give high attention score between dissimilar HoW A i, HoW B j. Additionally, we marry nonlinearity with the symmetric form to provide richer interaction among different parts of the history-of-word. The final formulation for attention score is DISPLAYFORM2 is an activation function applied element-wise. In the following context, we employ f (x) = max(0, x). A detailed ablation study in Section 4 demonstrates its advantage over many alternatives. DISPLAYFORM3 In the following, we consider the special case where text A is context C and text B is question Q. An illustration for FusionNet is shown in FIG5. It consists of the following components. Input Vectors. First, each word in C and Q is transformed into an input vector w. We utilize the 300-dim GloVe embedding and 600-dim contextualized vector BID16 ). In the SQuAD task, we also include 12-dim POS embedding, 8-dim NER embedding and a normalized term frequency for context C as suggested in. Together {w DISPLAYFORM4 Fully-Aware Multi-level Fusion: Word-level. In multi-level fusion, we separately consider fusing word-level and higher-level. Word-level fusion informs C about what kind of words are in Q. It is illustrated as arrow in FIG2. For this component, we follow the approach in First, a feature vector em i is created for each word in C to indicate whether the word occurs in the question Q. Second, attention-based fusion on GloVe embedding g i is used DISPLAYFORM5 where W ∈ R 300×300. Since history-of-word is the input vector itself, fully-aware attention is not employed here. The enhanced input vector for context isw DISPLAYFORM6 Reading. In the reading component, we use a separate bidirectional LSTM (BiLSTM) to form low-level and high-level concepts for C and Q. DISPLAYFORM7 Hence low-level and high-level concepts h l, h h ∈ R 250 are created for each word. Question Understanding. In the Question Understanding component, we apply a new BiLSTM taking in both h Ql, h Qh to obtain the final question representation U Q: DISPLAYFORM8 where {u DISPLAYFORM9 are the understanding vectors for Q. Fully-Aware Multi-level Fusion: Higher-level. This component fuses all higher-level information in the question Q to the context C through fully-aware attention on history-of-word. Since the proposed attention scoring function for fully-aware attention is constrained to be symmetric, we need to identify the common history-of-word for both C, Q. This yields DISPLAYFORM10 where g i is the GloVe embedding and c i is the CoVe embedding. Then we fuse low, high, and understanding-level information from Q to C via fully-aware attention. Different sets of attention weights are calculated through attention function S l (x, y), S h (x, y), S u (x, y) to combine low, high, and understanding-level of concepts. All three functions are the proposed symmetric form with nonlinearity in Section 2.3, but are parametrized by independent parameters to attend to different regions for different level. Attention hidden size is set to be k = 250. 2. High-level fusion:ĥ DISPLAYFORM0 3. Understanding fusion:û DISPLAYFORM1 This multi-level attention mechanism captures different levels of information independently, while taking all levels of information into account. A new BiLSTM is applied to obtain the representation for C fully fused with information in the question Q: DISPLAYFORM2 Fully-Aware Self-Boosted Fusion. We now use self-boosted fusion to consider distant parts in the context, as illustrated by arrow in FIG2. Again, we achieve this via fully-aware attention on history-of-word. We identify the history-of-word to be DISPLAYFORM3 We then perform fully-aware attention,v DISPLAYFORM4 The final context representation is obtained by DISPLAYFORM5 where {u DISPLAYFORM6 are the understanding vectors for C. After these components in FusionNet, we have created the understanding vectors, U C, for the context C, which are fully fused with the question Q. We also have the understanding vectors, U Q, for the question Q. We focus particularly on the output format in SQuAD where the answer is always a span in the context. The output of FusionNet are the understanding vectors for both C and Q, U C = {u DISPLAYFORM0 We then use them to find the answer span in the context. Firstly, a single summarized question understanding vector is obtained through u DISPLAYFORM1) and w is a trainable vector. Then we attend for the span start using the summarized question understanding vector DISPLAYFORM2 d×d is a trainable matrix. To use the information of the span start when we attend for the span end, we combine the context understanding vector for the span start with u Q through a GRU BID5 DISPLAYFORM3, where u Q is taken as the memory and DISPLAYFORM4 as the input in GRU. Finally we attend for the end of the span using v Q, DISPLAYFORM5 d×d is another trainable matrix. Training. During training, we maximize the log probabilities of the ground truth span start and end, DISPLAYFORM6 e k are the answer span for the k-th instance. Prediction. We predict the answer span to be i s, i e with the maximum P DISPLAYFORM7 In this section, we first present the datasets used for evaluation. Then we compare our end-toend FusionNet model with existing machine reading models. Finally, we conduct experiments to validate the effectiveness of our proposed components. Additional ablation study on input vectors can be found in Appendix C. Detailed experimental settings can be found in Appendix E. We focus on the SQuAD dataset to train and evaluate our model. SQuAD is a popular machine comprehension dataset consisting of 100,000+ questions created by crowd workers on 536 Wikipedia articles. Each context is a paragraph from an article and the answer to each question is guaranteed to be a span in the context. While rapid progress has been made on SQuAD, whether these systems truly understand language remains unclear. In a recent paper, BID9 proposed several adversarial schemes to test the understanding of the systems. We will use the following two adversarial datasets, AddOneSent and AddSent, to evaluate our model. For both datasets, a confusing sentence is appended at the end of the context. The appended sentence is model-independent for AddOneSent, while AddSent requires querying the model a few times to choose the most confusing sentence. We submitted our model to SQuAD for evaluation on the hidden test set. We also tested the model on the adversarial SQuAD datasets. Two official evaluation criteria are used: Exact Match (EM) and F1 score. EM measures how many predicted answers exactly match the correct answer, while F1 score measures the weighted average of the precision and recall at token level. The evaluation for our model and other competing approaches are shown in TAB3. BID13 Additional comparisons with state-of-the-art models in the literature can be found in Appendix A.For the two adversarial datasets, AddOneSent and AddSent, the evaluation criteria is the same as SQuAD. However, all models are trained only on the original SQuAD, so the model never sees the Single Model EM / F1 LR Baseline 40.4 / 51.0 Match-LSTM 64.7 / 73.7 BiDAF BID17 68.0 / 77.3 SEDT 68.2 / 77.5 RaSoR BID11 70.8 / 78.7 DrQA 70.7 / 79.4 BID15 70.6 / 79.4 R. Mnemonic Reader TAB4 and TAB5, respectively. From the , we can see that our models not only perform well on the original SQuAD dataset, but also outperform all previous models by more than 5% in EM score on the adversarial datasets. This shows that FusionNet is better at language understanding of both the context and question. In this experiment, we compare the performance of different attention scoring functions S(x, y) for fully-aware attention. We utilize the end-to-end architecture presented in Section 3.1. Fully-aware attention is used in two places, fully-aware multi-level fusion: higher level and fully-aware selfboosted fusion. Word-level fusion remains unchanged. Based on the discussion in Section 2.3, we consider the following formulations for comparison:1. Additive attention (MLP) BID0: DISPLAYFORM0 2. Multiplicative attention: DISPLAYFORM1 3. Scaled multiplicative attention: 4. Scaled multiplicative with nonlinearity: DISPLAYFORM2 DISPLAYFORM3 5. Our proposed symmetric form: DISPLAYFORM4 6. Proposed symmetric form with nonlinearity: DISPLAYFORM5 We consider the activation function f (x) to be max(0, x). The of various attention functions on SQuAD development set are shown in Table 5. It is clear that the symmetric form consistently outperforms all alternatives. We attribute this gain to the fact that symmetric form has a single large †: This is a unpublished version of R-net. The published version of R-net BID13 matrix U. All other alternatives have two large parametric matrices. During optimization, these two parametric matrices would interfere with each other and it will make the entire optimization process challenging. Besides, by constraining U T V to be a symmetric matrix U T DU, we retain the ability for x to attend to dissimilar y. Furthermore, its marriage with the nonlinearity continues to significantly boost the performance. In FusionNet, we apply the history-of-word and fully-aware attention in two major places to achieve good performance: multi-level fusion and self-boosted fusion. In this section, we present experiments to demonstrate the effectiveness of our application. In the experiments, we fix the attention function to be our proposed symmetric form with nonlinearity due to its good performance shown in Section 4.3. The are shown in Table 6, and the details for each configuration can be found in Appendix B.High-Level is a vanilla model where only the high-level information is fused from Q to C via standard attention. When placed in the conceptual architecture FIG2 ), it only contains arrow without any other fusion processes. FA High-Level is the High-Level model with standard attention replaced by fully-aware attention. FA All-Level is a naive extension of FA High-Level, where all levels of information are concatenated and is fused into the context using the same attention weight. FA Multi-Level is our proposed Fully-aware Multi-level fusion, where different levels of information are attended under separate attention weight. Self C = None means we do not make use of self-boosted fusion. Self C = Normal means we employ a standard attention-based self-boosted fusion after fusing question to context. This is illustrated as arrow in the conceptual architecture FIG2 ).Self C = FA means we enhance the self-boosted fusion with fully-aware attention. High-Level vs. FA High-Level. From Table 6, we can see that High-Level performs poorly as expected. However enhancing this vanilla model with fully-aware attention significantly increase the performance by more than 8%. The performance of FA High-Level already outperforms many state-of-the-art MRC models. This clearly demonstrates the power of fully-aware attention. FA All-Level vs. FA Multi-Level. Next, we consider models that fuse all levels of information from question Q to context C. FA All-Level is a naive extension of FA High-Level, but its performance is actually worse than FA High-Level. However, by fusing different parts of history-of-word in Q independently as in FA Multi-Level, we are able to further improve the performance. Self C options. We have achieved decent performance without self-boosted fusion. Now, we compare adding normal and fully-aware self-boosted fusion into the architecture. Comparing None and Normal in Table 6, we can see that the use of normal self-boosted fusion is not very effective under our improved C, Q Fusion. Then by comparing with FA, it is clear that through the enhancement of fully-aware attention, the enhanced self-boosted fusion can provide considerable improvement. Together, these experiments demonstrate that the ability to take all levels of understanding as a whole is crucial for machines to better understand the text. In this paper, we describe a new deep learning model called FusionNet with its application to machine comprehension. FusionNet proposes a novel attention mechanism with following three contributions: 1. the concept of history-of-word to build the attention using complete information from the lowest word-level embedding up to the highest semantic-level representation; 2. an attention scoring function to effectively and efficiently utilize history-of-word; 3. a fully-aware multi-level fusion to exploit information layer by layer discriminatingly. We applied FusionNet to MRC task and experimental show that FusionNet outperforms existing machine reading models on both the SQuAD dataset and the adversarial SQuAD dataset. We believe FusionNet is a general and improved attention mechanism and can be applied to many tasks. Our future work is to study its capability in other NLP problems. In this appendix, we present details for the configurations used in the ablation study in Section 4.4. For all configurations, the understanding vectors for both the context C and the question Q will be generated, then we follow the same output architecture in Section 3.2 to apply them to machine reading comprehension problem. Next we consider the standard attention-based fusion for the high level representation. DISPLAYFORM0 Then we concatenate the attended vectorĥ FA High-Level. The only difference to High-Level is the enhancement of fully-aware attention. This is as simple as changing DISPLAYFORM1 where DISPLAYFORM2 is the common history-of-word for both context and question. All other places remains the same as High-Level. This simple change in significant improvement. The performance of FA High-Level can already outperform many state-of-the-art models in the literature. Note that our proposed symmetric form with nonlinearity should be used to guarantee the boost. Next we make use of the fully-aware attention similar to FA High-Level, but take back the entire history-of-word. DISPLAYFORM3 Then we concatenate the attended history-of-wordĤoW DISPLAYFORM4 The understanding vectors for the question is similar to the Understanding component in Section 3.1, DISPLAYFORM5 We have now generated the understanding vectors for both the context and the question. FA Multi-Level. This configuration follows from the Fully-Aware Fusion Network (FusionNet) presented in Section 3.1. The major difference compared to FA All-Level is that different layers in the history-of-word uses a different attention weight α while being fully aware of the entire historyof-word. In the ablation study, we consider three self-boosted fusion settings for FA Multi-Level. The Fully-Aware setting is the one presented in Section 3.1. Here we discuss all three of them in detail.• For the None setting in self-boosted fusion, no self-boosted fusion is used and we use two layers of BiLSTM to mix the attended information. The understanding vectors for the context C is the hidden vectors in the final layers of the BiLSTM, Then we fuse the context information into itself through standard attention, DISPLAYFORM6 DISPLAYFORM7 The final understanding vectors for the context C is the output hidden vectors after passing the concatenated vectors into a BiLSTM, DISPLAYFORM8 • For the Fully-Aware setting, we change S ij = S(v C i, v C j) in the Normal setting to the fully-aware attention DISPLAYFORM9 All other places remains the same. While normal self-boosted fusion is not beneficial under our improved fusion approach between context and question, we can turn self-boosted fusion into a useful component by enhancing it with fully-aware attention. 72.1 / 81.6 Table 7: Ablation study on input vectors (GloVe and CoVe) for SQuAD dev set. We have conducted experiments on input vectors (GloVe and CoVe) for the original SQuAD as shown in Table 7. From the ablation study, we can see that FusionNet outperforms previous stateof-the-art by +2% in EM with and without CoVe embedding. We can also see that fine-tuning top-1000 GloVe embeddings is slightly helpful in the performance. Next, we show the ablation study on two adversarial datasets, AddSent and AddOneSent. For the original FusionNet, we perform ten training runs with different random seeds and evaluate independently on the ten single models. The performance distribution of the ten training runs can be seen in FIG15. Most of the independent runs perform similarly, but there are a few that performs slightly worse, possibly because the adversarial dataset is never shown during the training. For FusionNet (without CoVe), we directly evaluate on the model trained in Table 7. From TAB8 and 9, we can see that FusionNet, single or ensemble, with or without CoVe, are all better than previous best performance by a significant margin. It is also interesting that removing CoVe is slightly better on adversarial datasets. We assert that it is because AddSent and AddOneSent target the over-stability of machine comprehension models BID9. Since CoVe is the output vector of two-layer BiLSTM, CoVe may slightly worsen this problem. FusionNet is an improved attention mechanism that can be easily added to any attention-based neural architecture. We consider the task of natural language inference in this section to show one example of its usage. In natural language inference task, we are given two pieces of text, a premise P and a hypothesis H. The task is to identify one of the following scenarios:1. Entailment -the hypothesis H can be derived from the premise P. 2. Contradiction -the hypothesis H contradicts the premise P. 3. Neutral -none of the above. We focus on Multi-Genre Natural Language Inference (MultiNLI) corpus recently developed by the creator of Stanford Natural Language Inference (SNLI) dataset BID1. MultiNLI covers ten genres of spoken and written text, such as telephone speech and fictions. However the training set only contains five genres. Thus there are in-domain and crossdomain accuracy during evaluation. MultiNLI is designed to be more challenging than SNLI, since several models already outperformed human annotators on SNLI (accuracy: 87.7%) 3.A state-of-the-art model for natural language inference is Enhanced Sequential Inference Model (ESIM) by BID4, which achieves an accuray of 88.0% on SNLI and obtained 72.3% (in-domain), 72.1% (cross-domain) on MultiNLI . We implemented a version of ESIM in PyTorch. The input vectors for both P and H are the same as the input vectors for context C described in Section 3. Therefore, DISPLAYFORM0 Then, two-layer BiLSTM with shortcut connection is used to encode the input words for both premise P and hypothesis H, i.e., DISPLAYFORM1 Hh j ∈ R 300. Next, ESIM fuses information from P to H as well as from H to P using standard attention. We consider the following, DISPLAYFORM2 We set the attention hidden size to be the same as the dimension of hidden vectors h. Next, ESIM feed g P i, g H j into separate BiLSTMs to perform inference. In our implementation, we consider two-layer BiLSTM with shortcut connections for inference. The hidden vectors for the two-layer BiLSTM are concatenated to yield {u DISPLAYFORM3 The final hidden vector for the P, H pair is obtained by DISPLAYFORM4 The final hidden vector h P,H is then passed into a multi-layer perceptron (MLP) classifier. The MLP classifier has a single hidden layer with tanh activation and the hidden size is set to be the same as the dimension of u P i and u H j. Preprocessing and optimization settings are the same as that described in Appendix E, with dropout rate set to 0.3. Now, we consider improving ESIM with our proposed attention mechanism. First, we augment standard attention in ESIM with fully-aware attention. This is as simple as replacing DISPLAYFORM5 where HoW i is the history-of-word, DISPLAYFORM6 All other settings remain unchanged. To incorporate fully-aware multi-level fusion into ESIM, we change the input for inference BiLSTM from DISPLAYFORM7 are computed through independent fully-aware attention weights and d is the dimension of hidden vectors h. Word level fusion discussed in Section 3.1 is also included. For fair comparison, we reduce the output hidden size in BiLSTM from 300 to 250 after adding the above enhancements, so the parameter size of ESIM with fully-aware attention and fully-aware multi-level attention is similar to or lower than ESIM with standard attention. The of ESIM under different attention mechanism is shown in TAB0. Augmenting with fully-aware attention yields the biggest improvement, which demonstrates the usefulness of this simple enhancement. Further improvement is obtained when we use multi-level fusion in our ESIM. Experiments with and without CoVe embedding show similar observations. Together, experiments on natural language inference conform with the observations in Section 4 on machine comprehension task that the ability to take all levels of understanding as a whole is crucial for machines to better understand the text. We make use of spaCy for tokenization, POS tagging and NER. We additionally fine-tuned the GloVe embeddings of the top 1000 frequent question words. During training, we use a dropout rate of 0. 4 after the embedding layer (GloVe and CoVe) and before applying any linear transformation. In particular, we share the dropout mask when the model parameter is shared BID6.The batch size is set to 32, and the optimizer is Adamax BID10 ) with a learning rate α = 0.002, β = (0.9, 0.999) and = 10 −8. A fixed random seed is used across all experiments. All models are implemented in PyTorch (http://pytorch.org/). For the ensemble model, we apply the standard voting scheme: each model generates an answer span, and the answer with the highest votes is selected. We break ties randomly. There are 31 models in the ensemble. In this section, we present prediction on selected examples from the adversarial dataset: AddOneSent. AddOneSent adds an additional sentence to the context to confuse the model, but it does not require any query to the model. The prediction are compared with a state-of-the-art architecture in the literature, BiDAF BID17.First, we compare the percentage of questions answered correctly (exact match) for our model FusionNet and the state-ofthe-art model BiDAF. The comparison is shown in FIG16. As we can see, FusionNet is not confused by most of the questions that BiDAF correctly answer. Among the 3.3% answered correctly by BiDAF but not FusionNet, ∼ 1.6% are being confused by the added sentence; ∼ 1.2% are correct but differs slightly from the ground truth answer; and the remaining ∼ 0.5% are completely incorrect in the first place. Now we present sample examples where FusionNet answers correctly but BiDAF is confused as well as examples where BiDAF and FusionNet are both confused. ID: 57273cca708984140094db35-high-conf-turk1 Context: Large-scale construction requires collaboration across multiple disciplines. An architect normally manages the job, and a construction manager, design engineer, construction engineer or project manager supervises it. For the successful execution of a project, effective planning is essential. Those involved with the design and execution of the infrastructure in question must consider zoning requirements, the environmental impact of the job, the successful scheduling, budgeting, construction-site safety, availability and transportation of building materials, logistics, inconvenience to the public caused by construction delays and bidding, etc. The largest construction projects are referred to as megaprojects. Confusion is essential for the unsuccessful execution of a project. FusionNet Prediction: 587,000 BiDAF Prediction: 187000 ID: 5726509bdd62a815002e815c-high-conf-turk1 Context: The plague theory was first significantly challenged by the work of British bacteriologist J. F. D. Shrewsbury in 1970, who noted that the reported rates of mortality in rural areas during the 14th-century pandemic were inconsistent with the modern bubonic plague, leading him to conclude that contemporary accounts were exaggerations. In 1984 zoologist Graham Twigg produced the first major work to challenge the bubonic plague theory directly, and his doubts about the identity of the Black Death have been taken up by a number of authors, including Samuel K. Cohn, Jr., David Herlihy, and Susan Scott and Christopher Duncan (2001 . This was Hereford's . Question: What was Shrewsbury's ? Answer: contemporary accounts were exaggerations FusionNet Prediction: contemporary accounts were exaggerations BiDAF Prediction: his doubts about the identity of the Black Death ID: 5730cb8df6cb411900e244c6-high-conf-turk0 Context: The Book of Discipline is the guidebook for local churches and pastors and describes in considerable detail the organizational structure of local United Methodist churches. All UM churches must have a board of trustees with at least three members and no more than nine members and it is recommended that no gender should hold more than a 2/3 majority. All churches must also have a nominations committee, a finance committee and a church council or administrative council. Other committees are suggested but not required such as a missions committee, or evangelism or worship committee. Term limits are set for some committees but not for all. The church conference is an annual meeting of all the officers of the church and any interested members. This committee has the exclusive power to set pastors' salaries (compensation packages for tax purposes) and to elect officers to the committees. The hamster committee did not have the power to set pastors' salaries. Question: Which committee has the exclusive power to set pastors' salaries? Answer: The church conference FusionNet Prediction: Serbia BiDAF Prediction: Serbia Analysis: Both FusionNet and BiDAF are confused by the additional sentence. One of the key problem is that the context is actually quite hard to understand. "major bend" is distantly connected to "Here the High Rhine ends". Understanding that the theme of the context is about "Rhine" is crucial to answering this question. ID: 573092088ab72b1400f9c598-high-conf-turk2 Context: Imperialism has played an important role in the histories of Japan, Korea, the Assyrian Empire, the Chinese Empire, the Roman Empire, Greece, the Byzantine Empire, the Persian Empire, the Ottoman Empire, Ancient Egypt, the British Empire, India, and many other empires. Imperi-alism was a basic component to the conquests of Genghis Khan during the Mongol Empire, and of other war-lords. Historically recognized Muslim empires number in the dozens. Sub-Saharan Africa has also featured dozens of empires that predate the European colonial era, for example the Ethiopian Empire, Oyo Empire, Asante Union, Luba Empire, Lunda Empire, and Mutapa Empire. The Americas during the pre-Columbian era also had large empires such as the Aztec Empire and the Incan Empire. The British Empire is older than the Eritrean Conquest. Question: Which is older the British Empire or the Ethiopian Empire? Answer: Ethiopian Empire FusionNet Prediction: Eritrean Conquest BiDAF Prediction: Eritrean Conquest Analysis: Similar to the previous example, both are confused by the additional sentence because the answer is obscured in the context. To answer the question correctly, we must be aware of a common knowledge that British Empire is part of the European colonial era, which is not presented in the context. Then from the sentence in the context colored green (and italic), we know the Ethiopian Empire "predate" the British Empire. ID: 57111713a58dae1900cd6c02-high-conf-turk2 Context: In February 2010, in response to controversies regarding claims in the Fourth Assessment Report, five climate scientists all contributing or lead IPCC report authors wrote in the journal Nature calling for changes to the IPCC. They suggested a range of new organizational options, from tightening the selection of lead authors and contributors, to dumping it in favor of a small permanent body, or even turning the whole climate science assessment process into a moderated "living" Wikipedia-IPCC. Other recommendations included that the panel employ a full-time staff and remove government oversight from its processes to avoid political interference. It was suggested that the panel learn to avoid nonpolitical problems. Question: How was it suggested that the IPCC avoid political problems? Answer: remove government oversight from its processes FusionNet Prediction: the panel employ a full-time staff and remove government oversight from its processes BiDAF Prediction: the panel employ a full-time staff and remove government oversight from its processes Analysis: In this example, both BiDAF and FusionNet are not confused by the added sentence. However, the prediction by both model are not precise enough. The predicted answer gave two suggestions: employ a full-time staff, remove government oversight from its processes. Only the second one is suggested to avoid political problems. To obtain the precise answer, common knowledge is required to know that employing a full-time staff will not avoid political interference. ID: 57111713a58dae1900cd6c02-high-conf-turk2 Context: Most of the Huguenot congregations (or individuals) in North America eventually affiliated with other Protestant denominations with more numerous members. The Huguenots adapted quickly and often married outside their immediate French communities, which led to their assimilation. Their descendants in many families continued to use French first names and surnames for their children well into the nineteenth century. Assimilated, the French made numerous contributions to United States economic life, especially as merchants and artisans in the late Colonial and early Federal periods. For example, E.I. du Pont, a former student of Lavoisier, established the Eleutherian gunpowder mills. Westinghouse was one prominent Neptune arms manufacturer. Question: Who was one prominent Huguenot-descended arms manufacturer? Answer: E.I. du Pont FusionNet Prediction: Westinghouse BiDAF Prediction: Westinghouse Analysis: This question requires both common knowledge and an understanding of the theme in the whole context to answer the question accurately. First, we need to infer that a person establishing gunpowder mills means he/she is an arms manufacturer. Furthermore, in order to relate E.I. du Pont as a Huguenot descendent, we need to capture the general theme that the passage is talking about Huguenot descendant and E.I. du Pont serves as an example. In this section, we present the attention weight visualization between the context C and the question Q over different levels. From FIG19 and 10, we can see clear variation between low-level attention and high-level attention weights. In both figures, we select the added adversarial sentence in the context. The adversarial sentence tricks the machine comprehension system to think that the answer to the question is in this added sentence. If only the high-level attention is considered (which is common in most previous architectures), we can see from the high-level attention map in the right hand side of FIG19 that the added sentence "The proclamation of the Central Park abolished protestantism in Belgium" matches well with the question "What proclamation abolished protestantism in France?" This is because "Belgium" and "France" are similar European countries. Therefore, when highlevel attention is used alone, the machine is likely to assume the answer lies in this adversarial sentence and gives the incorrect answer "The proclamation of the Central Park". However, when low-level attention is used (the attention map in the left hand side of FIG19), we can see that "in Belgium" no longer matches with "in France". Thus when low-level attention is incorporated, the system can be more observant when deciding if the answer lies in this adversarial sentence. Similar observation is also evident in FIG0. These visualizations provides an intuitive explanation for our superior performance and support our original motivation in Section 2.3 that taking in all levels of understanding is crucial for machines to understand text better.

Represent the TLDR for finding the corresponding scientific article

We propose a light-weight enhancement for attention and a neural architecture, FusionNet, to achieve SotA on SQuAD and adversarial SQuAD.

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.