Question: Represent the scientific article to find a short TLDR

Question: We introduce a novel framework for generative models based on Restricted Kernel Machines (RKMs) with multi-view generation and uncorrelated feature learning capabilities, called Gen-RKM. To incorporate multi-view generation, this mechanism uses a shared representation of data from various views. The mechanism is flexible to incorporate both kernel-based, (deep) neural network and convolutional based models within the same setting. To update the parameters of the network, we propose a novel training procedure which jointly learns the features and shared representation. Experiments demonstrate the potential of the framework through qualitative evaluation of generated samples. In the past decade, interest in generative models has grown tremendously, finding applications in multiple fields such as, generated art, on-demand video, image denoising , exploration in reinforcement learning , collaborative filtering , inpainting and many more. Some examples of graphical models based on a probabilistic framework with latent variables are Variational Auto-Encoders and Restricted Boltzmann Machines (RBMs) . More recently proposed models are based on adversarial training such as Generative Adversarial Networks (GANs) and its many variants. Furthermore, auto-regressive models such as Pixel Recurrent Neural Networks (PixelRNNs) model the conditional distribution of every individual pixel given previous pixels. All these approaches have their own advantages and disadvantages. For example, RBMs perform both learning and Bayesian inference in graphical models with latent variables. However, such probabilistic models must be properly normalized, which requires evaluating intractable integrals over the space of all possible variable configurations . Currently GANs are considered as the state-of-the-art for generative modeling tasks, producing high-quality images but are more difficult to train due to unstable training dynamics, unless more sophisticated variants are applied. Many datasets are comprised of different representations of the data, or views. Views can correspond to different modalities such as sounds, images, videos, sequences of previous frames, etc. Although each view could individually be used for learning tasks, exploiting information from all views together could improve the learning quality (; ;). Also, it is among the goals of the latent variable modelling to model the description of data in terms of uncorrelated or independent components. Some classical examples are Independent Component Analysis; Hidden Markov models ; Probabilistic Principal Component Analysis (PCA) ; Gaussian-Process Latent variable model and factor analysis. Hence, when learning a latent space in generative models, it becomes interesting to find a disentangled representation. Disentangled variables are generally considered to contain interpretable information and reflect separate factors of variation in the data for e.g. lighting conditions, style, colors, etc. The definition of disentanglement in the literature is not precise, however many believe that a representation with statistically independent variables is a good starting point . Such representations extract information into a compact form which makes it possible to generate samples with specific characteristics (; ; ;). Additionally, these representations have been found to generalize better and be more robust against adversarial attacks . In this work, we propose an alternative generative mechanism based on the framework of Restricted Kernel Machines (RKMs) , called Generative RKM (Gen-RKM). RKMs yield a representation of kernel methods with visible and hidden units establishing links between Kernel PCA, Least-Squares Support Vector Machines (LS-SVM) and RBMs. This framework has a similar energy form as RBMs, though there is a non-probabilistic training procedure where the eigenvalue decomposition plays the role of normalization. used this framework to develop tensor-based multi-view classification models and showed how kernel PCA fits into this framework. Contributions. 1) A novel multi-view generative model based on the RKM framework where multiple views of the data can be generated simultaneously. 2) Two methods are proposed for computing the pre-image of the feature vectors: with the feature map explicitly known or unknown. We show that the mechanism is flexible to incorporate both kernel-based, (deep) convolutional neural network based models within the same setting. 3) When using explicit feature maps, we propose a training algorithm that jointly performs the feature-selection and learns the common-subspace representation in the same procedure. 4) Qualitative and quantitative experiments demonstrate that the model is capable of generating good quality images of natural objects. Further experiments on multi-view datasets exhibit the potential of the model. Thanks to the orthogonality of eigenvectors of the kernel matrix, the learned latent variables are uncorrelated. This resembles a disentangled representation, which makes it possible to generate data with specific characteristics. This paper is organized as follows. In Section 2, we discuss the Gen-RKM training and generation mechanism when multiple data sources are available. In Section 3, we explain how the model incorporates both kernel methods and neural networks through the use of implicit and explicit feature maps respectively. When the feature maps are defined by neural networks, the Gen-RKM algorithm is explained in Section 4. In Section 5, we show experimental of our model applied on various public datasets. Section 6 concludes the paper along with directions towards the future work. Additional supplementary materials are given in the Appendix A. The proposed Gen-RKM framework consists of two phases: a training phase and a generation phase which occurs one after another. Similar to Energy-Based Models (EBMs, see for details), the RKM objective function captures dependencies between variables by associating a scalar energy to each configuration of the variables. Learning consists of finding an energy function in which the observed configurations of the variables are given lower energies than unobserved ones. Note that the schematic representation, as shown in Figure 1 is similar to Discriminative RBMs and the objective function J t (defined below) has an energy form similar to RBMs with additional regularization terms. The latent space dimension in the RKM setting has a similar interpretation as the number of hidden units in a restricted Boltzmann machine, where in the specific case of the RKM these hidden units are uncorrelated. We assume a dataset, with x i ∈ R d, y i ∈ R p comprising of N data points. Here y i may represent an additional view of x i, e.g., an additional image from a different angle, the caption of an image or a class label. Starting from the RKM interpretation of Kernel PCA, which gives an upper bound on the equality constrained Least-Squares Kernel PCA objective function , and applying the feature-maps φ 1: Figure 1: Gen-RKM schematic representation modeling a common subspace H between two data sources X and Y. The φ 1, φ 2 are the feature maps (F x and F y represent the feature-spaces) corresponding to the two data sources. While ψ 1, ψ 2 represent the pre-image maps. The interconnection matrices U, V model dependencies between latent variables and the mapped data sources. the training objective function J t for generative RKM is given by 1: where U ∈ R d f ×s and V ∈ R p f ×s are the unknown interaction matrices, and h i ∈ R s are the latent variables modeling a common subspace H between the two input spaces X and Y (see Figure 1). The derivation of this objective function is given in the Appendix A.1. Given η 1 > 0 and η 2 > 0 as regularization parameters, the stationary points of J t are given by: Substituting U and V in the first equation above, denoting Λ = diag{λ 1, . . ., λ s} ∈ R s×s with s ≤ N, yields the following eigenvalue problem: where H = h 1,..., h N ∈ R s×N with s ≤ N is the number of selected principal components and K 1, K 2 ∈ R N ×N are the kernel matrices corresponding to data sources 2. Based on Mercer's theorem , positive-definite kernel functions k 1:.., N forms the elements of corresponding kernel matrices. The feature maps φ 1 and φ 2, mapping the input data to the high-dimensional feature space (possibly infinite) are implicitly defined by kernel functions. Typical examples of such kernels are given by the Gaussian RBF kernel − xi−xj 2/σ just to name a few . However, one can also define explicit feature maps, still preserving the positive-definiteness of the kernel function by construction . In this section, we derive the equations for the generative mechanism. RKMs resembling energybased models, the inference consists in clamping the value of observed variables and finding configurations of the remaining variables that minimizes the energy . Given the Otherwise, a centered kernel matrix could be obtained using Eq. 17 (Appendix A.4). 2 While in the above section we have assumed that only two data sources (namely X and Y) are available for learning, the above procedure could be extended to multiple data-sources. For the M views or data-sources, this yields the training problem: learned interconnection matrices U and V, and a given latent variable h, consider the following objective function: with an additional regularization term on data sources. Here J g denotes the objective function for generation. The given latent variable h can be the corresponding latent code of a training point, a newly sampled hidden unit or a specifically determined one. Above cases correspond to generating the reconstructed visible unit, generating a random new visible unit or exploring the latent space by carefully selecting hidden units respectively. The stationary points of J g are characterized by: Using U and V from Eq. 2, we obtain the generated feature vectors: To obtain the generated data, one now needs to compute the inverse images of the feature maps φ 1 (·) and φ 2 (·) in the respective input spaces, i.e., solve the pre-image problem. We seek to find the functions where φ 1 (x) and φ 2 (y) are calculated using Eq. 6. When using kernel methods, explicit feature maps are not necessarily known. Commonly used kernels such as the radial-basis function and polynomial kernels map the input data to a very high dimensional feature space. Hence finding the pre-image, in general, is known to be an ill-conditioned problem . However, various approximation techniques have been proposed (; ; ;) which could be used to obtain the approximate pre-imagex of φ 1 (x). In section 3.1, we employ one such technique to demonstrate the applicability in our model, and consequently generate the multi-view data. One could also define explicit pre-image maps. In section 3.2, we define parametric pre-image maps and learn the parameters by minimizing the appropriately defined objective function. The next section describes the above two pre-image methods for both cases, i.e., when the feature map is explicitly known or unknown, in greater detail. As noted in the previous section, since x may not exist, we find an approximationx. A possible technique is shown by. Left multiplying Eq. 6 by φ 1 (x i) and φ 2 (y i), ∀i = 1,..., N, we obtain: where, represents the similarities between φ 1 (x) and training data points in the feature space, and K 1 ∈ R N ×N represents the centered kernel matrix of X. Similar conventions follow for Y respectively. Using the kernel-smoother method , the pre-images are given by: wherek 1 (x i, x) andk 2 (y i, y) are the scaled similarities (see Eq. 8) between 0 and 1 and n r the number of closest points based on the similarity defined by kernelsk 1 andk 2. While using an explicit feature map, Mercer's theorem is still applicable due to the positive semidefiniteness of the kernel function by construction, thereby allowing the derivation of Eq. 3. In the experiments, we use a set of (convolutional) neural networks as the feature maps φ θ (·). Another (transposed convolutional) neural network is used for the pre-image map ψ ζ (·) . The network parameters {θ, ζ} are learned by minimizing the reconstruction errors defined by L 1 (x, ψ 1 ζ 1 (φ 1 θ 1 (x))) and L 2 (y, ψ 2 ζ 2 (φ 2 θ 2 (y))). In our experiments, we use the mean-squared, however, in principle, one can use any other loss appropriate to the dataset. Here φ 1 θ 1 (x i) and φ 2 θ 2 (y i) are computed from Eq. 6, i.e., the generated points in feature space from the subspace H. Adding the loss function directly into the objective function J t is not suitable for minimization. Instead, we use the stabilized objective function defined as is the regularization constant . This tends to push the objective function J t towards zero, which is also the case when substituting the solutions λ i, h i back into J t (see Appendix A.3 for details). The combined training objective is given by: where c acc ∈ R + is a regularization constant to control the stability with reconstruction accuracy. In this way, we combine feature-selection and subspace learning within the same training procedure. There is also an intuitive connection between Gen-RKM and autoencoders. Namely, the properties of kernel PCA resemble the objectives of the 3 variations of an autoencoder: standard , VAE and β-VAE. 1) Similar to an autoencoder, Gen-RKM minimizes the reconstruction error in the loss function (see Eq. 9), where kernel PCA which acts as a denoiser (the information is compressed in the principal components). 2) By interpreting kernel PCA within the LS-SVM setting , the PCA analysis can take the interpretation of a one-class modeling problem with zero target value around which one maximizes the variance . When choosing a good feature map, one expects the latent variables to be normally distributed around zero. This property resembles the added regularization term in the objective of the VAE , which is expressed as the Kullback-Leibler divergence between the encoder's distribution and a unit Gaussian as a prior on the latent variables. 3) Kernel PCA gives uncorrelated components in feature space. While it was already shown that PCA does not give a good disentangled representation for images (; . Hence by designing a good kernel (through appropriate feature-maps) and doing kernel PCA, it is possible to get a disentangled representation for images as we show on the example in Figure 5. The uncorrelated components enhances the interpretation of the model. Based on the previous analysis, we propose a novel algorithm, called the Gen-RKM algorithm, combining kernel learning and generative models. We show that this procedure is efficient to train and evaluate. It is also scalable to large datasets when using explicit feature maps. The training procedure simultaneously involves feature selection, common-subspace learning and pre-image map learning. This is achieved via an optimization procedure where one iteration involves an eigendecomposition of the kernel matrix which is composed of the features from various views (see Eq. 3). The latent variables are given by the eigenvectors, which are then passed via a pre-image map to reconstruct the sample. Figure 1 shows a schematic representation of the algorithm when two data sources are available. Thanks to training in m mini-batches, this procedure is scalable to large datasets (sample size N) with training time scaling super-linearly with T m = c. While using neural networks as feature maps, d f and p f correspond to the number of neurons in the output layer, which are chosen as hyperparameters by the practitioner. Eigendecomposition of this smaller covariance matrix would yield U and V as eigenvectors (see Eq. 10 and Appendix A.2 for detailed derivation), where computing the h i involves only matrix-multiplication which is readily parallelizable on modern GPUs: Algorithm 1 Gen-RKM, η1, η2, feature map φj(·) -explicit or implicit via kernels kj(·, ·), for j ∈ {1, 2} Output: Generated data x, y 1: procedure TRAIN 2: if φj(·) = Implicit then 3: Hyperparameters: kernel specific 4: Solve Eq. 3 5: Select s principal components 6: else if φj(·) = Explicit then 7: while not converged do 8: {x, y} ← {Get mini-batch} 9: φ1(x) ← x; φ2(y) ← y 10: do steps 4-5 11: {φ1(x), φ2(y)} ← h (Eq. 6) 12: {x, y} ← {ψ1(φ1(x)), ψ2(φ2(y))} 13: ∆θ1 ∝ −∇ θ 1 Jc; ∆θ2 ∝ −∇ θ 2 Jc 14: if φj(·) = Implicit then 4: Hyperparameter: nr 5: Compute kx *, ky * (Eq. 7) 6: Getx,ŷ (Eq. 8) 7: else if φj(·) = Explicit then 8: do steps 11-12 9: end if 10: end procedure To demonstrate the applicability of the proposed framework and algorithm, we trained the Gen-RKM model on a variety of datasets commonly used to evaluate generative models: MNIST , Fashion-MNIST , CIFAR-10 , CelebA , Dsprites and Teapot . The experiments were performed using both the implicit feature map defined by a Gaussian kernel and parametric explicit feature maps defined by deep neural networks, either convolutional or fully connected. As explained in Section 2, in case of kernel methods, training only involves constructing the kernel matrix and solving the eigenvalue problem in Eq. 3. In our experiments, we fit a Gaussian mixture model (GMM) with l components to the latent variables of the training set, and randomly sample a new point h for generating views using a kernel smoother. In case of explicit feature maps, we define φ 1 θ 1 and ψ 1 ζ 1 as convolution and transposed-convolution neural networks, respectively ; and φ 2 θ 2 and ψ 1 ζ 2 as fully-connected networks. The particular architecture details are outlined in Table 3 in the Appendix. The training procedure in case of explicitly defined maps consists of minimizing J c using the Adam optimizer to update the weights and biases. To speed-up learning, we subdivided the datasets into m mini-batches, and within each iteration of the optimizer, Eq. 3 is solved to update the value of H. Information on the datasets and hyperparameters used for the experiments is given in Table 4 in the Appendix. Qualitative examples: Figure 2 shows the generated images using a convolutional neural network and transposed-convolutional neural network as the feature map and pre-image map respectively. The first column in yellow-boxes shows the training samples and the second column on the right shows the reconstructed samples. The other images shown are generated by random sampling from a GMM over the learned latent variables. Notice that the reconstructed samples are of better quality visually than the other images generated by random sampling. To elucidate that the model has not merely memorized the training examples, we show the generated images via bilinear-interpolations in the latent space in 2e and 2f. Comparison: We compare the proposed model with the standard VAE . For a fair comparison, the models have the same encoder/decoder architecture, optimization parameters and are trained until convergence, where the details are given in Table 3. We evaluate the performance qualitatively by comparing reconstruction and random sampling, the are shown in Figure 8 in the Appendix. In order to quantitatively assess the quality of the randomly generated samples, we use the Fréchet Inception Distance (FID) introduced by. The are reported in Table 1. Experiments were repeated for different latent-space dimensions (h dim), and we observe empirically that FID scores are better for the Gen-RKM. This is confirmed by the qualitative evaluation in Table 8, where the VAE generates smoother images. An interesting trend could be noted that as the dimension of latent-space is increased, VAE gets better at generating images whereas the performance of Gen-RKM decreases slightly. This is attributed to the eigendecomposition of the kernel matrix whose eigenvalue spectrum decreases rapidly depicting that most information is captured in few principal components, while the rest is noise. The presence of noise hinders the convergence of the model. It is therefore important to select the number of latent variables proportionally to the size of the mini-batch and the corresponding spectrum of the kernel matrix (the diversity within a mini-batch affects the eigenvalue spectrum of the kernel matrix). Multi-view Generation: Figures 3 & 4 demonstrate the multi-view generative capabilities of the model. In these datasets, labels or attributes are seen as another view of the image that provides extra information. One-hot encoding of the labels was used to train the model. Figure 4a shows the generated images and labels when feature maps are only implicitly known i.e. through a Gaussian kernel. Figures 4b, 4c shows the same when using fully-connected networks as parametric functions to encode and decode labels. We can see that both the generated image and the generated label matches in most cases, albeit not all. Qualitative examples: The latent variables are uncorrelated, which gives an indication that the model could resemble a disentangled representation. This is confirmed by the empirical evidence on Figure 5, where we explore the uncorrelated features learned by the models on the Dsprites and celebA dataset. In our experiments, the Dsprites training dataset comprised of 32 × 32 positions of oval and heart-shaped objects. The number of principal components chosen were 2 and the goal was to findout whether traversing along the eigenvectors, corresponds to traversing the generated im-age in one particular direction while preserving the shape of the object. Rows 1 and 2 of Figure 5 show the reconstructed images of an oval while moving along first and second principal component respectively. Notice that the first and second components correspond to the y and x positions respectively. Rows 3 and 4 show the same for hearts. On the celebA dataset, we train the Gen-RKM with 15 components. Rows 5 and 6 shows the reconstructed images while traversing along the principal components. When moving along the first component from left-to-right, the hair-color of the women changes, while preserving the face structure. Whereas traversal along the second component, transforms a man to woman while preserving the orientation. When the number of principal components were 2 while training, the brightness and light-source corresponds to the two largest variances in the dataset. Also notice that, the reconstructed images are more blurry due to the selection of less number of components to model H. Comparison: To quantitatively assess disentanglement performance, we compare Gen-RKM with VAE and beta-VAE on the Dsprites and Teapot datasets . The models have the same encoder/decoder architecture, optimization parameters and are trained until convergence, where the details are given in Table 3. The performance is measured using the proposed framework 3 of , which gives 3 measures: disentanglement, completeness and informativeness. The are depicted in Table 2. Gen-RKM has good performance on the Dsprites dataset when the latent space dimension is equal to 2. This is expected as the number of disentangled generating factors in the dataset is also equal to 2, hence there are no noisy components in the kernel PCA hindering the convergence. The opposite happens in the case h dim = 10, where noisy component are present. The above is confirmed by the Relative Importance Matrix on Figure 6 in the Appendix, where the 2 generating factors are well separated in the latent space of the Gen-RKM. For the Teapot dataset, Gen-RKM has good performance when h dim = 10. More components are needed to capture all variations in the dataset, where the number of generating factors is now equal to 5. In the other cases, Gen-RKM has a performance comparable to the others. The paper proposes a novel framework, called Gen-RKM, for generative models based on RKMs with extensions to multi-view generation and learning uncorrelated representations. This allows for a mechanism where the feature map can be implicitly defined using kernel functions or explicitly by (deep) neural network based methods. When using kernel functions, the training consists of only solving an eigenvalue problem. In the case of a (convolutional) neural network based explicit feature map, we used (transposed) networks as the pre-image functions. Consequently, a training procedure was proposed which involves joint feature-selection and subspace learning. Thanks to training in mini-batches and capability of working with covariance matrices, the training is scalable to large datasets. Experiments on benchmark datasets illustrate the merit of the proposed framework for generation quality as well as disentanglement. Extensions of this work consists of adapting the model to more advanced multi-view datatsets involving speech, images and texts; further analysis on other feature maps, pre-image methods, loss-functions and uncorrelated feature learning. Finally, this paper has demonstrated the applicability of the Gen-RKM framework, suggesting new research directions to be worth exploring., where ) for the two data sources can be written as: where U ∈ R d×s and V ∈ R p×s are the interconnection matrices. Using the notion of conjugate feature duality introduced in , the error variables e i are conjugated to latent variables h i using: which is also known as the Fenchel-Young inequality for the case of quadratic functions . By eliminating the variables e i from Eq. 11 and using Eq. 12, we obtain the Gen-RKM training objective function: A.2 KERNEL PCA IN THE PRIMAL From Eq. 2, eliminating the variables h i yields the following: Denote.., λ s } ∈ R s×s with s ≤ N. Now, composing the above equations in matrix form, we get the following eigen-decomposition problem: Here the size of the covariance matrix is The latent variables h i can be computed using Eq. 2, which simply involves matrix multiplications. A.3 STABILIZING THE OBJECTIVE FUNCTION Proposition 1. All stationary solutions for H,Λ in Eq. 3 of J t lead to J t = 0. Proof. Let λ i, h i are given by Eq. 3. Using Eq. 2 to substitute V and U in Eq. 1 yields: From Eq. 3, we get: Proposition 2. Let J(x): R N − → R be a smooth function, for all x ∈ R N and for c ∈ R >0, definē 2. Assuming (1 + cJ(x)) = 0, then x is the stationary points ofJ(x) iff x is the stationary point for J(x). Proof. Let x be a stationary point of J(x), meaning that ∇J(x) = 0. The stationary points for J(x) can be obtained from: It is easy to see from Eq. 2 that if x = x *, ∇J(x *) = 0, we have that dJ dx x * = 0, meaning that all the stationary points of J(x) are stationary points ofJ(x). To show the other way, let x be stationary point ofJ(x) i.e. ∇J(x) = 0. Assuming (1 + cJ(x)) = 0, then from Eq. 16 for all c ∈ R >0, we have Based on the above propositions, we stabilize our original objective function Eq. 1 to keep it bounded and hence is suitable for minimization with Gradient-descent methods. Without the reconstruction errors, the stabilized objective function is Since the derivatives of J t are given by Eq. 2, the stationary points of J are: assuming 1 + c stab J t = 0. Elimination of V and U yields 1 η1 K 1 + 1 η2 K 2 H = H Λ, which is indeed the same solution for c stab = 0 in Eq. 1 and Eq. 3. Centering of the kernel matrix is done by the following equation: where 1 denotes an N -dimensional vector of ones and K is either K 1 or K 2. See Table 3 and 4 for details on model architectures, datasets and hyperparameters used in this paper. The PyTorch library in Python was used as the programming language with a 8GB NVIDIA QUADRO P4000 GPU. Random Generation CelebA Figure 8: Comparing Gen-RKM and standard VAE for reconstruction and generation quality. In reconstruction MNIST and reconstruction CelebA, uneven columns correspond to the original image, even columns to the reconstructed image.

Represent the short summary for finding the corresponding scientific article (long)!

Gen-RKM: a novel framework for generative models using Restricted Kernel Machines with multi-view generation and uncorrelated feature learning.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We investigate a variant of variational autoencoders where there is a superstructure of discrete latent variables on top of the latent features.


Question: Represent the scientific article to find a short TLDR

Question: Work on the problem of contextualized word representation—the development of reusable neural network components for sentence understanding—has recently seen a surge of progress centered on the unsupervised pretraining task of language modeling with methods like ELMo . This paper contributes the first large-scale systematic study comparing different pretraining tasks in this context, both as complements to language modeling and as potential alternatives. The primary of the study support the use of language modeling as a pretraining task and set a new state of the art among comparable models using multitask learning with language models. However, a closer look at these reveals worryingly strong baselines and strikingly varied across target tasks, suggesting that the widely-used paradigm of pretraining and freezing sentence encoders may not be an ideal platform for further work. InputFigure 1: Our common model design: During pretraining, we train the shared encoder and the task-specific model for each pretraining task. We then freeze the shared encoder and train the task-specific model anew for each target evaluation task. Tasks may involve more than one sentence. State-of-the-art models for natural language processing (NLP) tasks like translation, question answering, and parsing include components intended to extract representations for the meaning and contents of each input sentence. These sentence encoder components are typically trained directly for the target task at hand. This approach can be effective on data rich tasks and yields human performance on some narrowly-defined benchmarks BID35 BID13, but it is tenable only for the few NLP tasks with millions of examples of training data. This has prompted interest in pretraining for sentence encoding: There is good reason to believe it should be possible to exploit outside data and training signals to effectively pretrain these encoders, both because they are intended to primarily capture sentence meaning rather than any task-specific skill, and because we have seen dramatic successes with pretraining in the related domains of word embeddings and image encoders BID46.More concretely, four recent papers show that pretrained sentence encoders can yield very strong performance on NLP tasks. show that a BiLSTM encoder from a neural machine translation (MT) system can be effectively reused elsewhere. BID16,, and BID33 show that various kinds of encoder pretrained in an unsupervised fashion through generative language modeling (LM) are effective as well. Each paper uses its own evaluation methods, though, making it unclear which pretraining task is most effective or whether multiple pretraining tasks can be productively combined; in the related setting of sentence-to-vector encoding, multitask learning with multiple labeled datasets has yielded a robust state of the art BID39. This paper attempts to systematically address these questions. We train reusable sentence encoders on 17 different pretraining tasks, several simple baselines, and several combinations of these tasks, all using a single model architecture and procedure for pretraining and transfer, inspired by ELMo. We then evaluate each of these encoders on the nine target language understanding tasks in the GLUE benchmark BID41, yielding a total of 40 sentence encoders and 360 total trained models. We then measure correlation in performance across target tasks and plot learning curves evaluating the effect of training data volume on each pretraining and target tasks. Looking to the of this experiment, we find that language modeling is the most effective single pretraining task we study, and that multitask learning during pretraining can offer further gains and a new state-of-the-art among fixed sentence encoders. We also, however, find reasons to worry that ELMo-style pretraining, in which we pretrain a model and use it on target tasks with no further fine-tuning, is brittle and seriously limiting: (i) Trivial baseline representations do nearly as well as the best pretrained encoders, and the margins between substantially different pretraining tasks can be extremely small. (ii) Different target tasks differ dramatically on what kinds of pretraining they benefit most from, and multitask pretraining is not sufficient to circumvent this problem and offer general-purpose pretrained encoders. Work toward learning reusable sentence encoders can be traced back at least as far as the multitask model of BID7, but has seen a recent surge in progress with the successes of CoVe BID25, ULMFit BID16, ELMo, and the Transformer LM BID33. However, each uses a different model and dataset from the others, so while these works serve as existence proofs that effective reusable sentence encoders are possible, they do not address the question of what task or tasks should be used to create them. The revival of interest in sentence encoder pretraining is recent enough that relatively little has been done to understand the relative merits of these models, though two exceptions stand out. In unpublished work, offer an analysis of the relative strengths of translation and language modeling using a single architecture and training dataset. They find that encoders trained as language models reliably uncover the most syntactic structure, even when they are trained on a strict subset of the data used for a comparable translation model. Peters et al. offer a deeper investigation of model design issues for ELMo, showing that all of the standard architectures for sentence encoding can be effectively pretrained with broadly similar performance, and that all learn reasonably good representations of the morphological and syntactic properties of sentences. There has been a great deal of work on sentence-to-vector encoding, a setting in which the pretrained encoder produces a fixed-size vector representation for each input sentence BID10 BID20 BID14 BID8 BID45. These vectors are potentially useful for tasks that require fast similarity-based matching of sentences, but using them to replace sentence encoders trained in the conventional way on a given target text classification task does not reliably yield state-of-the art performance on that task BID39.Multitask representation learning in NLP in general has been well studied, and again can be traced back at least as far as BID7. For example, BID23 show promising from the combination of translation and parsing, BID39 show the benefits of multitask learning in sentence-to-vector encoding, and BID0 and BID4 offer studies of when multitask learning is helpful for lower-level NLP tasks. Our main experiment compares encoders pretrained on a large number of tasks and task combinations, where a task is a dataset-objective function pair. This section lists these tasks, which we select either to serve as baselines or because they have shown promise in outside prior work, especially prior work on sentence-to-vector encoding. Appendix A includes additional details on how we implemented some of these tasks, and names tasks we evaluated but left out. Random Encoder Our primary baseline is equivalent to pretraining on a task with zero examples. Here, we randomly initialize a sentence encoder and use it directly with no further training. This baseline works well, yielding scores far above those of a bag-of-words encoder.1 This surprising matches seen recently with ELMo-like models by and earlier work on Reservoir Computing. This baseline is especially strong because our model contains a skip connection from the input of the shared encoder to its output, allowing the task-specific model to directly see our word representations, or, in experiments where we use a pretrained ELMo model as our input layer, ELMo's contextual word representations. We use the nine tasks included with GLUE as pretraining tasks: acceptability classification with CoLA BID42; binary sentiment classification with SST BID38; semantic similarity with the MSR Paraphrase Corpus (MRPC; BID11, the Quora Question Pairs 2 (QQP), and STS-Benchmark (STS; BID3 ; and textual entailment with the Multi-Genre NLI Corpus (MNLI BID44, RTE 1, 2, 3, and 5 (RTE; , et seq.), and data from SQuAD (QNLI, BID34 and the Winograd Schema Challenge (WNLI, BID21 recast as entailment in the style of BID43 . MNLI is the only task with substantial prior work in this area, as it was found to be highly effective as a pretraining strategy by BID8 and BID39 . Other tasks are included to represent a broad sample of labeling schemes commonly used in NLP. We train language models on two datasets: WikiText-103 (WP, BID26 and 1 Billion Word Language Model Benchmark (BWB, BID5, which are used by ULMFit BID16 and ELMo respectively. Translation We train MT models on two datasets: WMT14 English-German BID1 and WMT17 English-Russian BID2 . SkipThought Our SkipThought model BID20 BID40) is a sequence-tosequence model that reads a sentence from WikiText-103 running text and attempts to decode the following sentence from that text. We train our DisSent model BID17 BID29 to read two separate clauses that appear in WikiText-103 connected by a discourse marker such as and, but, or so and predict the identity of the discourse marker. Reddit These models reconstruct comment threads from reddit.com using a dataset of about 18M comment-response pairs collected from 2008-2011 by BID45. We consider two settings: A classification task in which the model makes a binary prediction about whether a candidate response is the actual response to a given comment, and a sequence-to-sequence task in the model attempts to generate the true response to a comment. We implement our models using the AllenNLP toolkit BID12, aiming to build the simplest architecture that could be reasonably expected to perform well on the target tasks under study. 3 The design of the models roughly follows that used in the GLUE baselines and ELMo. The core of our model is a two-layer 1024D bidirectional LSTM. We feed the word representations to the biLSTM and take the sequence of hidden states from the top-level LSTM as the contextual representation. The downstream task-specific model sees both the top-layer hidden states of this model and, through a skip connection, the input representations for each word. All of our models use the pretrained character-level convolutional neural network (CNN) word encoder from ELMo. This encoder acts as a standard input layer which uses no information beyond the word, and allows us to avoid potentially the difficult issues surrounding unknown word handling in transfer learning. In some experiments, we use the full pretrained ELMo model as an input handler, yielding a form of multitask learning in which the lower layers of the overall model (ELMo) are pretrained on language modeling, and the higher layers (our shared encoder) are pretrained on some additional task or tasks. We choose to use this pretrained model because it represents a larger model with more extensive tuning than we have the resources to produce ourselves. We compare pretraining tasks in this setting to understand how well they complement large-scale language model pretraining, and we additionally train our own language models to directly compare between language modeling and other pretraining methods. We follow the standard practice of training a set of scalar weights of ELMo's three layers. We use one set of weights to supply input to the shared encoder, and an additional set for each target task to use in the skip connection. We use only ELMo and not the similarly-situated CoVe, as BID41 showed CoVe to be less effective on the GLUE tasks. Evaluation and Per-Task Models The GLUE benchmark BID41 ) is an open-ended shared task competition and evaluation toolkit for reusable sentence encoders, and we use it as our primary vehicle for evaluation. GLUE is a set of nine classification or regression tasks over sentences and sentence pairs spanning a range of dataset sizes, paired with private test data and an online leaderboard. GLUE offers a larger set of tasks than evaluated by ELMo or CoVe while omitting more expensive paragraph-level tasks, allowing us to evaluate a substantially larger number of experiments with available compute resources. To evaluate the shared encoder, we use the following procedure: We freeze the pretrained encoder and, for each of the nine tasks in the GLUE benchmark, separately train a target-task model on the representations produced by the encoder. We then evaluate each of these models on the validation or test set of the corresponding task using the standard metric(s) for that task, and report the ing scores and the overall average GLUE scores, which weight each task equally. For single-sentence target tasks (CoLA, SST) and sentence-pair tasks with smaller training datasets (MRPC, RTE, WNLI) we train a linear projection over the output states of the shared encoder, max-pool over those projected states, and feed the to a one-hidden-layer classifier MLP. For smaller sentence pair-tasks, we perform these steps on both sentences and use the heuristic matching feature vector [h 1 ; h 2 ; h 1 · h 2 ; h 1 − h 2] in the MLP, following BID28.For the remaining sentence-pair tasks (MNLI, QNLI, QQP, STS), we use an attention mechanism between all pairs of words, followed by a 512D ×2 BiLSTM with max-pooling over time, following the basic mechanism used in BiDAF BID37. This is followed by heuristic matching and a final MLP, as above. Appendices A and B present additional details on the task specific models. Pretraining Task Models For pretraining on GLUE tasks, we use the architecture described above, except that we do not use an attention mechanism, as early indicated that this hurt cross-task transfer performance. For consistency with other experiments when pretraining on a GLUE task, we reinitialize the task-specific parameters between pretraining and target-task training. Several of the outside (non-GLUE) pretraining tasks involve sentence pair classification. For these, we use the same non-attentive architecture as for the larger GLUE tasks. For LM, to prevent information leakage across directions and LSTM layers, we follow the broad strategy used by ELMo: We train separate forward and backward two-layer LSTM language models, and concatenate the outputs during target task training. For sequence-to-sequence pretraining tasks (MT, SkipThought, Reddit), we use an LSTM decoder with a single layer. We also investigate three sets of tasks for multitask pretraining: all GLUE tasks, all outside (non-GLUE) pretraining tasks, and all pretraining tasks. Because ELMo representations are computed with the full context and so cannot be used as the input to downstream unidirectional language models, we exclude language modeling from multitask runs that use ELMo. At each update during multitask learning, we randomly sample a single task with probability proportional to its training data size raised to the power of 0.75. This sampling rate is meant to balance the risks of overfitting small-data tasks and underfitting large ones, and performed best in early exper-iments. More extensive experiments with methods like this are shown in Appendix C. We perform early stopping based on an unweighted average of the pretraining tasks' validation metrics. For validation metrics like perplexity that decrease from high starting values during training, we include the transformed metric 1 − m 250 in our average, tuning the constant 250 in early experiments. Optimization We train our models with the AMSGrad optimizer -a variant of Adam BID19. We perform early stopping at pretraining time and target task training time using the respective dev set performances. Typical experiments, including pretraining one encoder and training the nine associated target-task models, take 1-5 days to complete on an NVIDIA P100 GPU. See Appendix B for more details. Hyperparameter Tuning Appendix B describes our chosen hyperparameter values. As our primary experiment required more than 100 GPU-days on NVIDIA P100 GPUs to run-not counting debugging or learning curves-we did not have the resources for extensive hyperparameter tuning. Instead of carefully tuning our shared and task-specific models on a single pretraining task in a way that might bias toward that task, we simply chose commonly-used values for most hyperparameters. The choice not to tune limits our ability to diagnose the causes of poor performance when it occurs, and we invite readers to further refine our models using the public code.5 TAB0 shows on the GLUE dev set for all our pretrained encoders, each with and without the pretrained ELMo BiLSTM layers (E). The N/A baselines are untrained encoders with random intialization. The Single-Task baselines are aggregations of from nine GLUE runs: The in this row for a given GLUE task uses the encoder pretrained on only that task. For consistency with other runs, we treat the pretraining task and the target task as two separate tasks in all cases (including here) and give them separate task-specific parameters, despite the fact that they use identical data. We use S and C to distinguish the sequence-to-sequence and classification versions of the Reddit task, respectively. To comply with GLUE's limits on test set access, we evaluated only three of our pretrained encoders on test data. These reflect our best models with and without the use of the pretrained ELMo encoder, and with and without the use of GLUE data during pretraining. For discussion of our limited hyperparameter tuning, see above. For roughly-comparable GLUE in prior work, see BID41 or https://www.gluebenchmark.com; we omit them here in the interest of space. The limited size of a US Letter page prevent us from including these baselines in this table. As of writing, the best test using a comparable frozen pretrained encoder is 68.9 from BID41 for a model similar to our GLUE E multitask model, and the best overall is 72.8 from BID33 with a model that is fine-tuned in its entirety for each target task. While not feasible to run each setting multiple times, we estimate the variance of the GLUE score by re-running the random encoder and MNLI pretraining setups with and without ELMo with different random seeds. Across five runs, we recorded σ = 0.4 for the random encoder (N/A in table), and σ = 0.2 for MNLI E. This variation is substantial but not so high as to render meaningless. For the explicitly adversarial WNLI dataset (based on the Winograd Schema Challenge; BID21, only one of our models reached even the most frequent class performance of 56.3. In computing average and test set performances, we replace model predictions with the most frequent label to simulate the better performance achievable by choosing not to model that task. Looking to other target tasks, the grammar-related CoLA task benefits dramatically from ELMo pretraining: The best without language model pretraining is less than half the achieved with such pretraining. In contrast, the meaning-oriented textual similarity benchmark STS sees good with several kinds of pretraining, but does not benefit substantially from the use of ELMo. Comparing pretraining tasks in isolation without ELMo, language modeling performs best, followed by MNLI. The remaining pretraining tasks yield performance near that of the random baseline. Even when training directly on each target task (Single-Task in table), we get less than a one point gain over this simple baseline. Adding ELMo yielded improvements in performance across all pretraining tasks. MNLI and English-German translation perform best in this setting, with SkipThought, Reddit classification, and DisSent also outperforming the ELMo-augmented random baseline. With ELMo, a multitask model performs best, but without it, all three multitask models are tied or outperformed by models trained on one of their constituent tasks, suggesting that our approach to multitask learning is not reliably able to produce models that productively use the knowledge taught by each training task. However, of the two non-ELMo models that perform best on the development data, the multitask model generalizes better than the single-task model on test data for tasks like STS where the test set contains new out-of-domain data. TAB1 presents an alternative view of the of the main experiment TAB0: The table shows the correlations between pairs of tasks over the space of pretrained encoders. These reflect the degree to which knowing the performance of one target task with some encoder will allow us to predict the performance of the other target task with that same encoder. Many correlations are low, suggesting that different tasks benefit from different forms of pretraining to a substantial degree, and mirroring the observation that no one pretraining task yields good performance on all target tasks. As noted above, the models that tended to perform best overall also overfit the WNLI training set most, leading to a negative correlation between WNLI and overall GLUE score. STS also shows a negative correlation, likely due to the observation that it does not benefit from ELMo pretraining. In contrast, CoLA shows a strong 0.93 correlation with the overall GLUE scores, but has weak or negative correlations with many tasks-the use of ELMo or LM pretraining dramatically improves CoLA performance, but most other forms of pretraining have little effect. FIG1 shows two types of learning curves. The first set measures performance on the overall GLUE metric for encoders trained to convergence on each pretraining task with varying amounts of data. The second set focuses on three pretrained encoders and measures performance on each GLUE target task separately with varying amounts of target task data. Looking at pretraining tasks in isolation (top left), most tasks improve slightly as the amount of pretraining data increases, with the LM and MT tasks showing the most promising combination of slope and maximum performance. Combining these pretraining tasks with ELMo (top right) yields a less interpretable : the relationship between training data volume and performance becomes weaker, and some of the best reported in this paper are achieved by models that combine pretrained ELMo with restricted-data versions of other pretraining tasks like MNLI and QQP.Looking at target task performance as target task training data volume varies, we see that all tasks benefit from increasing data quantities, with no obvious diminishing returns, and that most tasks see a constant improvement in performance across data volumes from the use of pretraining, either with ELMo (center) or with multitask learning (right). Results on the GLUE Diagnostic Set From GLUE's auxiliary diagnostic analysis dataset, we find that ELMo and other forms of unsupervised pretraining helps on examples that involve world knowledge and lexical-semantic knowledge, and less so on examples that highlight complex sentence structures. See TAB5 in Appendix D for more details. This paper presents a systematic comparison of tasks and task-combinations for the pretraining of sentence-level BiLSTM encoders like those seen in ELMo and CoVe. With 40 pretraining tasks and task combinations (not counting many more ruled out early) and nine target tasks, this represents a far more comprehensive study than any seen on this problem to date. Our chief positive are perhaps unsurprising: Language modeling works well as a pretraining task, and no other single task is consistently better. Multitask pretraining can produce better than any single task can, and sets a new state-of-the-art among comparable models. Target task performance continues to improve with the addition of more language model data, even at large scales, suggesting that further work scaling up language model pretraining is warranted. However, a closer look at our suggests that the pretrain-and-freeze paradigm that underlies ELMo and CoVe might not be a sound platform for future work: Some trivial baselines do strikingly well, the margins between pretraining tasks are small, and some pretraining configurations (such as MNLI E) yield better performance with less data. This suggests that we may be nearing an upper bound on the performance that can be reached with methods like these. In addition, different tasks benefit from different forms of pretraining to a striking degree-with correlations between target tasks often low or negative-and multitask pretraining tasks fail to reliably produce models better than their best individual components. This suggests that if truly generalpurpose sentence encoders are possible, our current methods cannot produce them. While further work on language modeling seems straightforward and worthwhile, the author(s) of this paper believe that the future of this line of work will require a better understanding of the ways in which neural network target task models can benefit from outside knowledge and data, and new methods for pretraining and transfer learning to allow them to do so. DisSent To extract discourse model examples from the WikiText-103 corpus BID26, we follow the procedure described in BID29 by extracting clause-pairs that follow specific dependency relationships within the corpus (see Figure 4 in BID29 . We use the Stanford Parser BID6 distributed in Stanford CoreNLP version 3.9.1 to identify the relevant dependency arcs. Reddit Response Prediction The Reddit classification task requires a model to select which of two candidate replies to a comment is correct. Since the dataset from BID45 contains only real comment-reply pairs, we select an incorrect distractor reply for each correct reply by permuting each minibatch. Alternative Tasks Any large-scale comparison like the one attempted in this paper is inevitably incomplete. Among the thousands of publicly available NLP datasets, we also performed initial trial experiments on several datasets for which we were not able to reach development-set performance above that of the random encoder baseline in any setting. These include image-caption matching with MSCOCO BID22, following BID18 ; the small-to-medium-data textunderstanding tasks collected in NLI format by BID32 ; ordinal common sense inference ; POS tagging on the Penn Treebank BID24; and supertagging on CCGBank BID15. See Section 4 for general comments on hyperparameter tuning. Validation We evaluate on the validation set for the current training task or tasks every 1,000 steps, except where noted otherwise for small-data target tasks. During multitask learning, we multiply this interval by the number of tasks, evaluating every 9,000 steps during GLUE multitask training, for example. Optimizer We use AMSGrad BID36. During pretraining, we use a learning rate of 1e-4 for classification and regression tasks, and 1e-3 for text generation tasks. During target-task training, we use a learning rate of 3e-4 for all tasks. Learning Rate Decay We multiply the learning rate by 0.5 whenever validation performance fails to improve for more than 4 validation checks. We stop training if the learning rate falls below 1e-6.Early Stopping We maintain a saved checkpoint reflecting the best validation seen so far. We stop training if we see no improvement after more than 20 validation checks. After training, we use the last saved checkpoint. Regularization We apply dropout with a drop rate of 0.2 after the input layer (the character CNN or ELMo), after each LSTM layer, and after each MLP layer in the task-specific classifier or regressor. For small-data target tasks, we increase MLP dropout to 0.4 during target-task training. Preprocessing We use Moses tokenizer for encoder inputs, and set a maximum sequence length of 40 tokens. There is no input vocabulary, as we use ELMo's character-based input layer. For English text generation tasks, we use the Moses tokenizer to tokenize our data, but use a wordlevel output vocabulary of 20,000 types for tasks that require text generation. For translation tasks, we use BPE tokenization with a vocabulary of 20,000 types. For all sequence-to-sequence tasks we train word embeddings on the decoder side. Target-Task-Specific Parameters To ensure that baseline performance for each target task is competitive, we find it necessary to use slightly different models and training regimes for larger and smaller target tasks. We used partially-heuristic tuning to separate GLUE tasks into big-, mediumand small-data groups, giving each group its own heuristically chosen task-specific model specifications. Exact values are shown in Table 3. Table 3: Hyperparameter settings for target-task models and target-task training. Attention is always disabled when pretraining on GLUE tasks. STS has a relatively small training set, but consistently patterns with the larger tasks in its behavior. Sequence-to-Sequence Models We found attention to be helpful for the SkipThought and Reddit pretraining tasks but not for machine translation, and report for these configurations. We use the max-pooled output of the encoder to initialize the hidden state of the decoder, and the size of this hidden state is equal to the size of the output of our shared encoder. We reduce the dimension of the output of the decoder by half via a linear projection before the output softmax layer. Our multitask learning experiments have three somewhat distinctive properties: (i) We mix tasks with very different amounts of training data-at the extreme, under 1,000 examples for WNLI, and over 1,000,000,000 examples from LM BWB. (ii) Our goal is to optimize the quality of the shared encoder, not the performance of any one of the tasks in the multitask mix. (iii) We mix a relatively large number of tasks, up to eighteen at once in some conditions. These conditions make it challenging but important to avoid overfitting or underfitting any of our tasks. Relatively little work has been done on this problem, so we conduct a small experiment here. All our experiments use the basic paradigm of randomly sampling a new task to train on at each step, and we experiment with two hyperparameters that can be used to control over-and underfitting: The probability with which we sample each task and the weight with which we scale the loss for each task. Our experiments follow the setup in Appendix B, and do not use the ELMo BiLSTM.Task Sampling We consider several approaches to determine the probability with which to sample a task during training, generally making this probability a function of the amount of data available for the task. For task i with training set size N i, the probability is DISPLAYFORM0 where a is a constant. Loss Scaling At each update, we scale the loss of a task with weight DISPLAYFORM1 Experiments For task sampling, we run experiments with multitask learning on the full set of nine GLUE tasks, as well as three subsets: single sentence tasks (S1: SST, CoLA), similarity and paraphrase tasks (S2: MRPC, STS, QQP), and inference tasks (S3: WNLI, QNLI, MNLI, RTE). The are shown in TAB3.We also experiment with several combinations of task sampling and loss scaling methods, using only the full set of GLUE tasks. The are shown in TAB4.While no combination of methods consistently offers dramatically better performance than any other, we observe that it is generally better to apply only one of non-uniform sampling and nonuniform loss scaling at a time rather than apply both simultaneously, as they provide roughly the same effect. Following encouraging from earlier pilot experiments, we use power 0.75 task sampling and uniform loss scaling in the multitask learning experiments shown in TAB0. TAB5, below, shows on the four coarse-grained categories of the GLUE diagnostic set for all our pretraining experiments. This set consists of about 1000 expert-constructed examples in NLI While no model achieves near-human performance, the use of ELMo and other forms of unsupervised pretraining appears to be helpful on examples that highlight world knowledge and lexicalsemantic knowledge, and less so on examples that highlight complex logical reasoning patterns or alternations in sentence structure. This relative weakness on sentence structure is somewhat surprising given the finding in that language model pretraining is helpful for tasks involving sentence structure.

Represent the short summary for finding the corresponding scientific article (long)

We compare many tasks and task combinations for pretraining sentence-level BiLSTMs for NLP tasks. Language modeling is the best single pretraining task, but simple baselines also do well.

Meta-learning methods used for vision, directly applied to NLP, perform worse than nearest neighbors on new classes; we can do better with distributional signatures.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In this paper, we study the adversarial attack and defence problem in deep learning from the perspective of Fourier analysis. We first explicitly compute the Fourier transform of deep ReLU neural networks and show that there exist decaying but non-zero high frequency components in the Fourier spectrum of neural networks. We then demonstrate that the vulnerability of neural networks towards adversarial samples can be attributed to these insignificant but non-zero high frequency components. Based on this analysis, we propose to use a simple post-averaging technique to smooth out these high frequency components to improve the robustness of neural networks against adversarial attacks. Experimental on the ImageNet and the CIFAR-10 datasets have shown that our proposed method is universally effective to defend many existing adversarial attacking methods proposed in the literature, including FGSM, PGD, DeepFool and C&W attacks. Our post-averaging method is simple since it does not require any re-training, and meanwhile it can successfully defend over 80-96% of the adversarial samples generated by these methods without introducing significant performance degradation (less than 2%) on the original clean images. Although deep neural networks (DNN) have shown to be powerful in many machine learning tasks, found that they are vulnerable to adversarial samples. Adversarial samples are subtly altered inputs that can fool the trained model to produce erroneous outputs. They are more commonly seen in image classification task and typically the perturbations to the original images are so small that they are imperceptible to human eye. Research in adversarial attacks and defences is highly active in recent years. In the attack side, many attacking methods have been proposed (; ; a; ; ; ; a; ; ;, with various ways to generate effective adversarial samples to circumvent new proposed defence methods. However, since different attacks usually are effective to different defences or datasets, there is no consensus on which attack is the strongest. Hence for the sake of simplicity, in this work, we will evaluate our proposed defence approach against four popular attacks for empirical analysis. In the defence side, various defence mechanisms have also been proposed, including adversarial training (; ; Tramèr et al., 2017;), network distillation (b), gradient masking , adversarial detection and adding modifications to neural networks . Nonetheless, many of them were quickly defeated by new types of attacks (; 2017b; c; a;). tried to provide a theoretical security guarantee for adversarial training by a min-max loss formulation, but the difficulties in non-convex optimization and in finding the ultimate adversarial samples for training may loosen this robustness guarantee. As a , so far there is no defence that is universally robust to all adversarial attacks. Along the line of researches, there were also investigations into the properties and existence of adversarial samples. first observed the transferability of adversarial samples across models trained with different hyper-parameters and across different training sets. They also attributed the adversarial samples to the low-probability blind spots in the manifold. In , the authors explained adversarial samples as "a of models being too linear, rather than too nonlinear." In , the authors showed the transferability occurs across models with different structures and even different machine learning techniques in addition to neural networks. In summary, the general existence and transferability of adversarial samples are well known but the reason of adversarial vulnerability still needs further investigation. Generally speaking, when we view neural network as a multivariate function f (x) of input x, if a small imperceptible perturbation ∆x leads to a huge fluctuation ∆f (x), the large quantity ∆f (x)/∆x essentially corresponds to high frequency components in the Fourier spectrum of f (x). In this paper, we will start with the Fourier analysis of neural networks and elucidate why there always exist some decaying but nonzero high frequency response components in neural networks. Based on this analysis, we show that neural networks are inherently vulnerable to adversarial samples due to the underlying model structure. Next, we propose a simple post-averaging method to tackle this problem. Our proposed method is fairly simple since it works as a post-processing stage of any given neural network models and it does not require re-training the networks at all. Furthermore, we have evaluated the post-averaging method against four popular adversarial attacking methods and our method is shown to be universally effective in defending all examined attacks. Experimental on the ImageNet and the CIFAR-10 datasets have shown that our simple post-averaging method can successfully defend over 80-96% of the adversarial samples generated by these attacks with little performance degradation (less than 2%) on the original clean images. In order to understand the behaviour of adversarial samples, it is essential to find the Fourier transform of neural networks. Fortunately, for some widely used neural networks, namely fully-connected neural networks using ReLU activation functions, we may explicitly derive their Fourier transform under some minor conditions. As we will show, these theoretical will shed light on how adversarial samples happen in neural networks. As we know, any fully-connected ReLU neural networks (prior to the softmax layer) essentially form piece-wise linear functions in input space. Due to space limit, we will only present the main in this section and the proofs and more details may be found in Appendix. Definition 2.1. A piece-wise linear function is a continuous function f: R n − → R such that there are some hyperplanes passing through origin and dividing R n into M pairwise disjoint regions R m, (m = 1, 2, ..., M), on each of which f is linear: Composition of a piece-wise linear function with a ReLU activation function is also a piece-wise linear function. Theorem 2.3. The output of any hidden unit in an unbiased fully-connected ReLU neural network is a piece-wise linear function. This is straightforward because the input to any hidden node is a linear combination of piece-wise linear functions and this input is composed with the ReLU activation function to yield the output, which is also piece-wise linear. However, each region R m is the intersection of a different number of half-spaces, enclosed by various hyperplanes in R n. In general, these regions R m (m = 1, · · ·, M) do not have simple shapes. For the purpose of mathematical analysis, we need to decompose each region into a union of some well-defined shapes having a uniform form, which is called infinite simplex. Definition 2.4. Let V = {v 1, v 2, ..., v n} be a set of n linearly independent vectors in R n. An infinite simplex, R + V, is defined as the region linearly spanned by V using only positive weights: Theorem 2.5. Each piece-wise linear function f (x) can be formulated as a summation of some simpler functions:, each of which is linear and non-zero only in an infinite simplex as follows: where V l is a set of n linearly independent vectors, and w l is a weight vector. In practice, we can always assume that the input to neural networks, x, is bounded. As a , for computational convenience, we may normalize all inputs x into the unit hyper-cube, U n = n. Obviously, this assumption can be easily incorporated into the above analysis by multiplying each is the Heaviside step function. Alternatively, we may simplify this term by adding n 2 additional hyperplanes to further split the input space to ensure all the elements of x do not change signs within each region R + Vq. In this case, within each region R + Vq, the largest absolute value among all elements of x is always achieved by a specific element, which is denoted as r q. In other words, the dimension x rq achieves the largest absolute value inside R + Vq. Similarly, the normalized piece-wise linear function may be represented as a summation of some functions: f (x) = Q q=1 g q (x), where each g q (x) (q = 1, 2, · · ·, Q) has the following form: For every V q, there exists an n × n invertible matrix A q to linearly transform all vectors of V q into standard basis vectors e i in R n. As a , each function g q (x) may be represented in terms of standard bases V * = {e 1, · · ·, e n} as follows: q. Lemma 2.6. Fourier transform of the following function: 0 otherwise may be presented as: where ω r is the r-th component of frequency vector ω (r = 1, · · ·, n), and ω 0 = 0. Finally we derive the Fourier transform of fully-connected ReLU neural networks as follows. Theorem 2.7. The Fourier transform of the output of any hidden node in a fully-connected unbiased 1 ReLU neural network may be represented as Obviously, neural networks are the so-called approximated bandlimited models as defined in , which have decaying high frequency components in Fourier spectrum. Theorem 2.7 further suggests that the matrices A −1 q may contribute to the high frequency components when the corresponding region R + Vq are too small. This is clear because the determinant of A q is proportional to the volume of R + Vq in R n. In summary, the high frequency components of neural networks are mostly attributed to these tiny regions in the input space. As we will show later, these small regions may be explicitly exploited to generate adversarial samples for neural networks. As shown in Theorem 2.3, neural network may be viewed as a sequential division of the input space into many small regions, as illustrated in Figure 1. Each layer is a further division of the existing regions from the previous layers, with each region being divided differently. Hence a neural network with multiple layers would in a tremendous amount of sub-regions in the input space. For example, when cutting an n-dimensional space using N hyperplanes, the maximum number of regions may be computed as For a hidden layer of N = 1000 nodes and input dimension is n = 200, the maximum number of regions is roughly equal to 10 200. In other words, even a middle-sized neural network can partition input space into a huge number of sub-regions, which can easily exceed the total number of atoms in the universe. When we learn a neural network, we can not expect there is at least one training sample inside each region. For those regions that do not have any training sample, the ant linear functions in them may be arbitrary since they do not contribute to the training objective function at all. Of course, most of these regions are extremely small in size. When we measure the expected loss function over the entire space, their contributions are negligible since the chance for a randomly sampled point to fall into these tiny regions is extremely small. However, adversarial attack is imposing a new challenge since adversarial samples are not naturally sampled. Given that the total number of regions is huge, those tiny regions are almost everywhere in the input space. For any data point in the input space, we almost surely can find such a tiny region in proximity where the linear function is arbitrary. If a point inside this tiny region is selected, the output of the neural network may be unexpected. We believe that these tiny unlearned regions may be a major reason why neural networks are vulnerable to adversarial samples. In layered deep neural networks, the linear functions in all regions are not totally independent. If we use v (l) to denote the weight matrix in layer l, the ant linear weight w k in eq. is actually the sum of all concatenated v (l) along all active paths. When we make a small perturbation ∆x to any input x, the fluctuation in the output of any hidden node can be approximated represented as: where N denotes the total number of hyperplanes to be crossed when moving x to x + ∆x. In any practical neural network, we normally have at least tens of thousands of hyperplanes crossing the hypercube U n = n. In other words, for any input x in a high-dimensional space, a small perturbation can always easily cross a large number of hyperplanes to enter a tiny unlearned region. When N is fairly large, the above equation indicates that the output of a neural network can still fluctuate dramatically even after all weight vectors are regularized by L 1 or L 2 norm. As a reference, we have verified this on some ImageNet data using a VGG16 model. When PGD is used to generate adversarial samples with average perturbation ||∆x|| 2 ≤ 0.35, which is extremely small perturbation since x has over a hundred thousand dimensions on ImageNet, we have observed that in average about N = 5278 hyperplanes are crossed per layer even after such a small perturbation is added. At last, since the ubiquitous existence of unlearned tiny regions is an intrinsic property of neural networks given its current model structure, we believe that adversarial training strategies will not be sufficient to completely get rid of adversarial samples. In principle, neural networks must be strictly bandlimited to filter out those decaying high frequency components in order to completely eliminate all adversarial samples. We definitely need more research efforts to figure out how to do this effectively and efficiently for neural networks. 3 THE PROPOSED DEFENCE APPROACH: POST-AVERAGING 3.1 POST-AVERAGING In this paper, we propose a simple post-processing method to smooth out those high frequency components as much as possible, which relies on a simple idea similar to moving-average in onedimensional sequential data. Instead of generating prediction merely from one data point, we use the averaged value within a small neighborhood around the data point, which is called post-averaging here. Mathematically, the post-averaging is computed as an integral over a small neighborhood centered at the input: where x is the input and f (x) represents the output of the neural network, and C denotes a small neighborhood centered at the origin and V C denotes its volume. When we choose C to be an n-sphere in R n of radius r, we may simply derive the Fourier transform of f C (x) as follows: where J n 2 (·) is the first kind Bessel function of order n/2. Since the Bessel functions, J ν (ω), decay with rate 1/ √ ω as |ω| → ∞ , we have as |ω| → ∞. Therefore, if r is chosen properly, the post-averaging operation can significantly bandlimit neural networks by smoothing out high frequency components. Note that the similar ideas have been used in to improve robustness in speech recognition. However, it is intractable to compute the above integral for any meaningful neural network used in practical applications. In this work, we propose to use a simple numerical method to approximate it. For any input x, we select K points in the neighborhood C centered at x, i.e. {x 1, x 2, · · ·, x K}, to approximately compute the integral as Obviously, in order to defend against adversarial samples, it is important to have samples outside the current unlearned tiny region. In the following, we use a simple sampling method based on directional vectors. To generate a relatively even set of samples for eq., we first determine some directional vectorsv, and then move the input x along these directions using several step sizes within the sphere of radius r:, ±r], andv is a selected unit-length directional vector. For each selected direction, we generate six samples within C along both the positive and the negative directions to ensure efficiency and even sampling. We use this implementation for the convenience to extend with different types of sampling strategies. We tried several direction sampling strategies, including using the directions towards the closest region boundaries, and found that the simple random direction sampling gives the best performance. In this sampling method, we fill the directional vectors with random numbers generated from a standard normal distribution, and then normalize them to have unit length. In this section, we evaluate the above post-averaging method on defending against several popular adversarial attacking methods. • Dataset: We evaluated our method on both the ImageNet and CIFAR-10 datasets. Since our proposed post-averaging method does not need to re-train neural networks, we do not need to use any training data in our experiments. For evaluation purpose, we use the validation set of the ImageNet dataset. The validation set consists of 50000 images labelled into 1000 categories. For computational efficiency, we randomly choose 5000 images from the ImageNet validation set and evaluate our model on these 5000 images. For the CIFAR-10 dataset, we use the full test set, which consists of 10000 images labelled into 10 categories. • Target model: For model on ImageNet, we use a pre-trained ResNet-152 network that is available from PyTorch, while for CIFAR-10, we use a pre-trained ResNet-110 network from Yerlan Idelbayev 2. In our experiments, we directly use these pre-trained models without any modification. • Source of adversarial attacking methods: We use Foolbox, an open source tool box to generate adversarial samples using different adversarial attacking methods. In this work, we tested our method against four popular attacking methods in the literature: Fast Gradient Sign method (FGSM) , Projected Gradient Descent (PGD) method , DeepFool (DF) attack method and Carlini & Wagner (C&W) L2 attack method (a). We used these attack methods in their default settings. • Threat model: In our experiments, we use an l ∞ norm to constrain the allowed perturbation distance. For each experiment, we define: • Clean set: The dataset that consists of the original images from ImageNet or CIFAR-10. • Attacked set: For every correctly classified image in the Clean set, if an adversarial sample is successfully generated under the attacking criteria, the original sample is replaced with the adversarial sample; if no adversarial sample is found, the original sample is kept in the dataset. Meanwhile, all the misclassified images are kept in the dataset without any change. Therefore the Attacked set has the same number of images as the clean set. In our experiments, we evaluate the original network and the network defended by post-averaging on both the Clean and the Attacked sets. The performance is measured in terms of: • Accuracy: number of correctly classified images over the whole dataset. • Defence rate: number of successfully defended adversarial samples over the total number of adversarial samples in the Attacked set. By "successfully defended", it refers to the case where an adversarial sample is correctly classified after the original model is defended by the post-averaging approach. Table 1 shows the performance of our defence approach against different attacking methods. In this table, the samples for post-averaging are selected within an n-sphere of radius r as in eq., with K = 15 different directions. Thus in a total of 15 × 2 × 3 + 1 = 91 samples (including the input) for each input image to be used in eq.. Moreover, all the adversarial samples generated are restricted to be within the perturbation range = 8 /255. We show the top-1 accuracy of the original model and the defended model on both the Clean and the Attacked set respectively, as well as the defence rate of the defended model. Besides, we also show the number of adversarial samples successfully generated by each attacking method in the last column. From Table 1, we can see that our proposed defence approach is universally robust to all of the attacking methods we have examined. It has achieved above 80-96% defence rates in all the experiments with only a minor performance degradation in the Clean set (less than 2%). Especially on the ImageNet dataset, our method is able to defend about 95% of the adversarial samples. However, an interesting observation from the experimental is that the defence rate in the CIFAR-10 dataset is lower than the usually more challenging ImageNet dataset. We think this may be because data points are sparser in the ImageNet space than in the CIFAR-10 space, as ImageNet has a much larger dimensionality. Generally, using a larger sampling radius r can increase the chance of moving out of the unlearned regions as we desired, but it will also introduce more noise that can harm the prediction accuracy; On the other hand, using a smaller sampling radius r can reduce the performance degradation but it may not be sufficient to defend against adversarial samples. The optimal value for r varies with different datasets due to their dimensionality and data sparsity. In experiments, we found that r = 30 for ImageNet and r = 6 for CIFAR-10 achieved relatively better performance. Figure 2 shows how the model defence rate on ImageNet varies with different r. As shown in the figure, the optimal value for r also varies in different attacking methods, but the performance variations are small. In general, our model retains high defence rate throughout the r range. We also tested the effect of K, the number of sampling directions used, on the model performance. From Table 2, we can see that our model performance is not very sensitive to K. It is able to achieve a good defence rate with only K = 6, that is, 37 samples used for each input image. In implementation, these samples can be easily packed into a mini-batch for fast computation in GPUs. When running on the same machine, we measured the averaged inference time for a single input image on the original network as 0.04 seconds, while the inference time for our models with different K are shown in Table 2. By comparison, we can know that the inference time after adding post-averaging is roughly 2 3 K of the original inference time. At last, we evaluated our post-averaging defence approach against attacks with different allowed perturbation ranges. The are shown in Figure 3. As we can see, our model retains very good attack defence rate up to = 32 /255. Note that the defence rate against PGD and C&W doesn't change much along the variation of, this is because PGD and C&W have already successfully generated adversarial samples for most of the correctly classified inputs when is small. Hence their generated adversarial samples will not change much when using larger. For FGSM, our method yields lower defending performance. The possible reason is that FGSM tends to generate much larger perturbations than other three stronger attacking methods under the same setting. A large perturbation is more likely to move samples across class-specific decision boundaries to generate much more confusing samples. In our opinion, this is a general phenomenon in pattern classification, not particular to adversarial attacks. In this paper, we have presented some theoretical by Fourier analysis of ReLU neural networks. These are useful for us to understand why neural networks are vulnerable to adversarial samples. Based on the , we hypothesize that the inevitable and ubiquitous existence of tiny unlearned regions in the model function mapping may be a major reason for adversarial vulnerability. As a possible defence strategy, we have proposed a simple post-averaging method. Experimental on the ImageNet and the CIFAR-10 datasets have demonstrated that our simple defence technique turns out to be very effective against many popular attack methods in the literature. Finally, it will be interesting to see whether our post-averaging method will be still robust against any new attack methods in the future. Definition B.1. A piece-wise linear function is a continuous function f: R n − → R such that there are some hyperplanes passing through origin and dividing R n into M pairwise disjoint regions R m, (m = 1, 2, ..., M), on each of which f is linear: Proof. This proposition immediately follows lemma B.2. Definition B.4. Let V = {v 1, v 2, ..., v n} be a set of n independent vectors in R n. An infinite simplex, R + V, is defined as the region linearly spanned by V using only positive weights: Theorem B.5. Each piece-wise linear function f (x) can be formulated as a summation of some functions:, each of which is linear and non-zero only in an infinite simplex as follows: where V k is a set of n independent vectors, and w k is a weight vector. Proof. Each region R p of a piece-wise linear function, f (x), which describes the behavior of a ReLU node if intersects with an affine hyper-plane in a convex polytope. This convex polytope can be triangulated into some simplices. Define V k, (k = 1, 2, ..., K), sets of vertexes of these simplices. The infinite simplexes created by these vector sets will have the desired property and f (x) can be written as: As explained earlier in the original article by adding n 2 hyper-planes to those defining the piece-wise linear function, the output of a ReLU node may be represented as f (x) = Q q=1 g q (x). These hyper-planes are those perpendicular to standard basis vectors and subtraction of one of these vectors from another one. That is, e i (i = 1, . . ., n) and e i − e j (1 ≤ i < j ≤ n). Given this representation, the final step to achieve the Fourier transform is the following lemma: Lemma B.6. Fourier transform of the following function: 0 otherwise may be presented as: where ω r is the rth component of frequency vector ω (r = 1, · · ·, n), and ω 0 = 0. Proof. Alternatively, s(x) may be represented as: Therefore, we need to compute Fourier transform of h(x)h(1 − x): By taking the inverse Fourier transform of the function: where δ n is n-dimensional Dirac Delta function, it can be shown that it is the Fourier transform of Now we can find the Fourier transform of s(x) where Ω = {ω 0, ..., ω n}, σ B is the summation over elements of B and A r = If B does not contain ω r and have at least 2 elements then the terms for B and B ∪ {ω r} will cancel each other out. Also, sign(|B| − 1) will vanish if B has only one element. Therefore, there only remains empty set and sets with two elements one of them being ω r. Given the fact that A r = 0, the of the integral will be: or equivalently: Therefore: whereω q = ωA −1 q. As for the Fourier transform computed in section 3.1, it should be mentioned that the integral in equation 6 is the Fourier transform of: which can be derived utilizing the property of the Fourier transforms for radially symmetric functions : Given this transform:

Represent this text to find a much much longer article about it with say hundreds-thousands of words

An insight into the reason of adversarial vulnerability, an effective defense method against adversarial attacks.

Represent the TLDR for finding the corresponding scientific article

This paper proves that skinny neural networks cannot approximate certain functions, no matter how deep they are.