Question: Represent the article for finding a single-sentence TLDR

Question: Generative Adversarial Networks (GANs) have become the gold standard when it comes to learning generative models for high-dimensional distributions. Since their advent, numerous variations of GANs have been introduced in the literature, primarily focusing on utilization of novel loss functions, optimization/regularization strategies and network architectures. In this paper, we turn our attention to the generator and investigate the use of high-order polynomials as an alternative class of universal function approximators. Concretely, we propose PolyGAN, where we model the data generator by means of a high-order polynomial whose unknown parameters are naturally represented by high-order tensors. We introduce two tensor decompositions that significantly reduce the number of parameters and show how they can be efficiently implemented by hierarchical neural networks that only employ linear/convolutional blocks. We exhibit for the first time that by using our approach a GAN generator can approximate the data distribution without using any activation functions. Thorough experimental evaluation on both synthetic and real data (images and 3D point clouds) demonstrates the merits of PolyGAN against the state of the art. Generative Adversarial Networks (GANs) are currently one of the most popular lines of research in machine learning. Research on GANs mainly revolves around: (a) how to achieve faster and/or more accurate convergence (e.g., by studying different loss functions (; ;) or regularization schemes (; ;) ), and (b) how to design different hierarchical neural networks architectures composed of linear and non-linear operators that can effectively model high-dimensional distributions (e.g., by progressively training large networks or by utilizing deep ResNet type of networks as generators ). Even though hierarchical deep networks are efficient universal approximators for the class of continuous compositional functions , the non-linear activation functions pose difficulties in their theoretical analysis, understanding, and interpretation. For instance, as illustrated in , element-wise non-linearities pose a challenge on proving convergence, especially in an adversarial learning setting . Consequently, several methods, e.g.,;;; , focus only on linear models (with respect to the weights) in order to be able to rigorously analyze the neural network dynamics, the residual design principle, local extrema and generalization error, respectively. Moreover, as stated in the recent in-depth comparison of many different GAN training schemes , the improvements may mainly arise from a higher computational budget and tuning and not from fundamental architectural choices. In this paper, we depart from the choice of hierarchical neural networks that involve activation functions and investigate for the first time in the literature of GANs the use of high-order polynomials as an alternative class of universal function approximators for data generator functions. This choice is motivated by the strong evidence provided by the Stone-Weierstrass theorem , which states that every continuous function defined on a closed interval can be uniformly approximated as closely as desired by a polynomial function. Hence, we propose to model the vector-valued generator function Gpzq: R d Ñ R o by a high-order multivariate polynomial of the latent vector z, whose unknown parameters are naturally represented by high-order tensors. However, the number of parameters required to accommodate all higher-order correlations of the latent vector explodes with the desired order of the polynomial and the dimension of the latent vector. To alleviate this issue and at the same time capture interactions of parameters across different orders of approximation in a hierarchical manner, we cast polynomial parameters estimation as a coupled tensor factorization that jointly factorizes all the polynomial parameters tensors. To this end, we introduce two specifically tailored coupled canonical polyadic (CP)-type of decompositions with shared factors. The proposed coupled decompositions of the parameters tensors into two different hierarchical structures (i.e., architectures of neural network decoders) that do not involve any activation function, providing an intuitive way of generating samples with an increasing level of detail. This is pictorially shown in Figure 1. The of the proposed PolyGAN using a fourth-order polynomial approximator is shown in Figure 1 (a), while Figure 1 (b) shows the corresponding generation when removing the fourth-order power from the generator. Our contributions are summarized as follows: • We model the data generator with a high-order polynomial. Core to our approach is to cast polynomial parameters estimation as a coupled tensor factorization with shared factors. To this end, we develop two coupled tensor decompositions and demonstrate how those two derivations in different neural network architectures involving only linear (e.g., convolution) units. This approach reveals links between high-order polynomials, coupled tensor decompositions and network architectures. • We experimentally verify that the ing networks can learn to approximate functions with analytic expressions. • We show how the proposed networks can be used with linear blocks, i.e., without utilizing activation functions, to synthesize high-order intricate signals, such as images. • We demonstrate that by incorporating activation functions to the derived polynomial-based architectures, PolyGAN improves upon three different GAN architectures, namely DC-GAN , SNGAN and SAGAN . (a) (b) Figure 1: Generated samples by an instance of the proposed PolyGAN. (a) Generated samples using a fourth-order polynomial and (b) the corresponding generated samples when removing the terms that correspond to the fourth-order. As evidenced, by extending the polynomial terms, PolyGAN generates samples with an increasing level of detail. In this Section, we investigate the use of a polynomial expansion as a function approximator for the data generator in the context of GANs. To begin with, we introduce the notation in Section 2.1. In Section 2.2, we introduce two different polynomials models along with specifically tailored coupled tensor factorizations for the efficient estimation of their parameters. Matrices (vectors) are denoted by uppercase (lowercase) boldface letters e.g., X, (x). Tensors are denoted by calligraphic letters, e.g., X. The order of a tensor is the number of indices needed to address its elements. Consequently, each element of an M th-order tensor X is addressed by M indices, i.e., pX q i1,i2,...,i M. The mode-m unfolding of a tensor X P R I1ˆI2ˆ¨¨¨ˆI M maps X to a matrix X pmq P R ImˆĪm with I k such that the tensor element x i1,i2,...,i M is mapped to the matrix element x im,j where j " 1`ř n"1 n‰m I n. The mode-m vector product of X with a vector u P R Im, denoted by Xˆn u P R I1ˆI2ˆ¨¨¨ˆIn´1ˆIn`1ˆ¨¨¨ˆI N, in a tensor of order M´1: The Khatri-Rao product (i.e., column-wise Kronecker product) of matrices A P R IˆN and B P R JˆN is denoted by A d B and yields a matrix of dimensions pIJqˆN. The Hadamard product of A P R IˆN and B P R IˆN is defined as A˚B and is equal to A pi,jq B pi,jq for the pi, jq element. The CP decomposition factorizes a tensor into a sum of component rank-one tensors. An M th-order tensor X P R I1ˆI2ˆ¨¨¨ˆI M has rank-1, when it is decomposed as the outer product of M vectors tu, where˝denotes for the vector outer product. Consequently, the rank-R CP decomposition of an M th-order tensor X is written as: where the factor matrices U rms " ru collect the vectors from the rank-one components. By considering the mode-1 unfolding of X, the CP decomposition can be written in matrix form as : More details on tensors and multilinear operators can be found in;. GANs typically consist of two deep networks, namely a generator G and a discriminator D. G is a decoder (i.e., a function approximator of the sampler of the target distribution) which receives as input a random noise vector z P R d and outputs a sample x " Gpzq P R o. D receives as input both Gpzq and real samples and tries to differentiate the fake and the real samples. During training, both G and D compete against each other till they reach an "equilibrium" . In practice, both the generator and the discriminator are modeled as deep neural networks, involving composition of linear and non-linear operators . In this paper, we focus on the generator. Instead of modeling the generator as a composition of linear and non-linear functions, we assume that each generated pixel x i " pGpzqq i may be expanded as a N th order polynomial 1 in z. That is, 1 With an N th order polynomial we can approximate any smooth function . Under review as a conference paper at ICLR 2020 where the scalar β i, and the set of tensors W are the parameters of the polynomial expansion associated to each output of the generator, e.g., pixel. Clearly, when n " 1, the weights are d-dimensional vectors; when n " 2, the weights, i.e., W r2s i, form a dˆd matrix. For higher orders of approximation, i.e., when n ě 3, the weights are n th order tensors. By stacking the parameters for all pixels, we define the parameters β.. Consequently, the vector-valued generator function is expressed as: Intuitively, is an expansion which allows the N th order interactions between the elements of the noise latent vector z. Furthermore, resembles the functional form of a truncated Maclaurin expansion of vector-valued functions. In the case of a Maclaurin expansion, W rns represent the n th order partial derivatives of a known function. However, in our case the generator function is unknown and hence all the parameters need to be estimated from training samples. The number of the unknown parameters in is pd, which grows exponentially with the order of the approximation. Consequently, the model of is prone to overfitting and its training is computationally demanding. A natural approach to reduce the number of parameters is to assume that the weights exhibit redundancy and hence the parameter tensors are of low-rank. To this end, several low-rank tensor decompositions can be employed . For instance, let the parameter tensors W rns admit a CP decompostion of mutilinear rank-k, namely, tW rns " rrU rns,1, U rns,2,..., U rns,pn`1q ssu N n"1, with U rns,1 P R oˆk, and U rns,m P R dˆk, for m " 2,..., n`1. Then, is expressed as which has significantly less parameters than, especially when k! d. However, a set of different factor matrices for each level of approximation are required in equation 6, and hence the hierarchical nature of images is not taken into account. To promote compositional structures and capture interactions among parameters in different orders of approximation we introduce next two coupled CP decompositions with shared factors. Model 1: Coupled CP decomposition: Instead of factorizing each parameters tensor individually we propose to jointly factorize all the parameter tensors using a coupled CP decomposition with a specific pattern of factor sharing. To illustrate the factorization, we assume a third order approximation (N " 3), however in the appendix a generalization to N -th order approximation is provided. Let us assume that the parameters tensors admit the following coupled CP decomposition with the factors corresponding to lower-order levels of approximation being shared across all parameters tensors. That is: • Let W r1s " CU T r1s, be the parameters for first level of approximation. • Let assume W r2s being a superposition of of two weights tensors, namely W r2s " W r2s 1:2Ẁ r2s 1:3, with W r2s i:j denoting parameters associated with the second order interactions across the i-th and j-th order of approximation. By enforcing the CP decomposition of the above tensors to share the factor with tensors corresponding to lower-order of approximation we obtain in matrix form: • Similarly, we enforce the third-order parameters tensor to admit the following CP decomposition (in matrix form) Note that all but the U r3s factor matrices are shared in the factorization of tensors capturing polynomial parameters for the first and second order of approximation. The parameters are C P R oˆk, U rms P R dˆk for m " 1, 2, 3. Then, for N " 3 is written as: The third order approximation of can be implemented as a neural network with the structure of Figure 2 (proved in section B, Claim 1 of the appendix). It is worth noting that the structure of the proposed network allows for incremental network growth. Model 2: Coupled nested CP decomposition: Instead of explicitly separating the interactions between layers, we can utilize a joint hierarchical decomposition on the polynomial parameters. Let us first introduce learnable hyper-parameters b rns P R ω (N n"1, which act as scaling factors for each parameter tensor. Therefore, we modify to: with. For illustration purposes, we consider a third order function approximation (N " 3). That is, To estimate its parameters we jointly factorize all parameters tensors by employing nested CP detecomposion with parameter sharing as follows (in matrix form) • First order parameters: W r1s p1q " CpA r3s d B r3s q T. • Second order parametes: • Third order parameters: ) ˙S r3s * T with C P R oˆk, A rns P R dˆk, S rns P R kˆk, B rns P R ωˆk for n " 1,..., N. Altogether, is written as: As we prove in the appendix (section B, Claim 3), can be implemented in a hierarchical manner with a three-layer neural network as shown in Figure 3. Comparison between the two models: Both models are based on the polynomial expansion, however there are few differences between those. The Coupled CP decomposition has a simpler expression, however the Coupled nested CP decomposition relates to standard architectures using hierarchical composition that has recently yielded promising in GANs (see Section 3). In the remainder of the paper, we use the Coupled nested CP decomposition by default; in Section G, we include an experimental comparison of the two models. The experimental comparison demonstrates that neither model outperforms the other in all datasets; they perform similarly. Figure 3: Schematic illustration of the Coupled nested CP decomposition (for third order approximation). Symbol˚refers to the Hadamard product. The literature on GANs is vast; we focus only on the works most closely related to ours. The interested reader can find further information in a recent survey . Despite the propagation of the noise z to successive layers, the aforementioned works have substantial differences from ours. We introduce a well-motivated and mathematically elaborate method to achieve a more precise approximation with a polynomial expansion. In contrast to the previously mentioned works, we also do not concatenate the noise with the feature representations, but rather perform multiplication of the noise with the feature representations, which we mathematically justify. The work that is most closely related to ours is the recently proposed StyleGAN , which is an improvement over the Progressive Growing of GANs (ProGAN) . As ProGAN, StyleGAN is a highly-engineered network that achieves compelling on synthesized 2D images. In order to provide an explanation on the improvements of StyleGAN over ProGAN, the authors adopt arguments from the style transfer literature . Nevertheless, the idea of style transfer proposes to use features from images for conditional image translation, which is very different to unsupervised samples (image) generation. We believe that these improvements can be better explained under the light of our proposed polynomial function approximation. That is, as we show in Figure 1, the Hadamard products build a hierachical decomposition with increasing level of detail (rather than different styles). In addition, the improvements in StyleGAN are demonstrated by using a well-tuned model. In this paper we showcase that without any complicated engineering process the polynomial generation can be applied into several architectures (or any other type of decoders) and consistently improves the performance. A sequence of experiments in both synthetic data (2D and 3D data manifolds) and higher-dimensional signals are conducted to assess the empirical performance of the proposed polynomial expansion. The first experiments are conducted on a 2D manifolds that are analytically known (Section 4.1). Further experiments on three 3D manifolds are deferred to the appendix (Section D). In Section 4.2, the polynomial expansion is used for synthesizing digits. Experiments on images beyond digits are conducted in Section E; more specifically, we experiment with images of faces and natural scenes. The experiments with such images demonstrate how polynomial expansion can be used for learning highly complex distributions by using a single activation function in the generator. Lastly, we augment our polynomial-based generator with non-linearities and show that this generator is at least as powerful as contemporary architectures. Apart from the polynomial-based generators, we implemented two variations that are considered baselines: (a)'Concat': we replace the Hadamard operator with concatenation (used frequently in recent methods, such as in), (b)' Orig': the Hadamard products are ditched, while use b r1s Ð z, i.e., there is a composition of linear layers that transform the noise z. Sinusoidal: We assess the polynomial-based generator on a sinusoidal function in the bounded domain r0, 2πs. Only linear blocks, i.e., no activation functions, are used in the generator. That is, all the element-wise non-linearities (such as ReLU's, tanh) are ditched. The distribution we want to match is a sin x signal. The input to the generator is z P R and the output is rx, sin xs with x P r0, 2πs. We assume a 12 th order approximation where each S ris, A ris is a fully-connected layer and B ris is an identity matrix. Each fully-connected layer has width 15. In Figure 4, 2, 000 random samples are synthesized. We indeed verify that in low-dimensional distributions, such as the univariate sinusoidal, PolyGAN indeed approximates the data distribution quite accurately without using any non-linear activation functions. The linear generator of the previous section is extended to greyscale images, in which an analytic expression of the ground-truth distribution remains elusive. To our knowledge, there has not been a generation of greyscale images based on polynomial expansion in the past. We capitalize on the expressivity of the recent resnet-based generator , to devise a new polynomial generator Gpzq: R 128 Ñ R 32x32. We consider a fourth-order approximation (as derived in) where B ris is the identity matrix, S ris is a residual block with two convolutions for i " 1,..., 4. We emphasize that the residual block as well as all layers are linear, i.e., there are no activation functions. We only add a tanh in the output of the generator for normalization purposes. The discriminator and the optimization procedure are the same as in SNGAN; the only difference is that we run one discriminator step per generator step (n dis " 1). Note that the'Orig' resnet-based generator resembles the generator of in this case. We perform digit generation (trained on MNIST ). In Figure 5, random samples are visualized for the three compared methods. Note that the two baselines have practically collapsed into a single number each, whereas PolyGAN does synthesize plausible digits. To further assist the generation process, we utilize the labels and train a conditional GAN. That is, the class labels are used for conditional batch normalization. As illustrated in Figure 6, the samples synthesized are improved over the unsupervised setting.' Orig' and'Concat' still suffer from severe mode collapse, while PolyGAN synthesizes digits that have different thickness (e.g. 9), style (e.g. 2) and rotation (e.g. 1). Figure 6: Conditional digit generation. Note that both'Orig' and'Concat' suffer from severe mode collapse (details in section 4.2). On the contrary, PolyGAN synthesizes digits that have different thickness (e.g. 9), style (e.g. 2) and rotation (e.g. 1). We express data generation as a polynomial expansion task. We model the high-order polynomials with tensorial factors. We introduce two tailored coupled decompositions and show how the polynomial parameters can be implemented by hierarchical neural networks, e.g. as generators in a GAN setting. We exhibit how such polynomial-based generators can be used to synthesize images by utilizing only linear blocks. In addition, we empirically demonstrate that our polynomial expansion can be used with non-linear activation functions to improve the performance of standard state-of-the-art architectures. Finally, it is worth mentioning that our approach reveals links between high-order polynomials, coupled tensor decompositions and network architectures. Algorithm 1: PolyGAN (model 1). % Perform the Hadamard product for the n th layer. Algorithm 2: PolyGAN (model 2). for n=2:N do 6 % Multiply with the current layer weight S rns and perform the Hadamard product. κ "´S rns κ`pB rns q T b rns¯˚´p A rns q T v7 end 8 x " β`Cκ. The appendix is organized as: • Section B provides the Lemmas and their proofs required for our derivations. • Section C generalizes the Coupled CP decomposition for N th order expansion. • Section D extends the experiments to 3D manifolds. • In Section E, additional experiments on image generation with linear blocks are conducted. • Comparisons with popular GAN architectures are conducted in Section F. Specifically, we utilize three popular generator architectures and devise their polynomial equivalent and perform comparisons on image generation. We also conduct an ablation study indicating how standard engineering techniques affect the image generation of the polynomial generator. • In Section G, a comparison between the two proposed decompositions is conducted on data distributions from the previous Sections. For a set of matrices tX m P R ImˆN u N m"1 the Khatri-Rao product is denoted by: In this section, we prove the following identity connecting the sets of matrices tA ν R IνˆK u N ν"1 and To demonstrate the simple case with two matrices, we prove first the special case with N " 2. Lemma 1. It holds that Proof. Initially, both sides of the equation have dimensions of KˆL, i.e., they match. The pi, jq element of the matrix product of pA Then the pi, jq element of the right hand side (rhs) of is: A 2,pk2,iq B 2,pk2,jq q " pA 1,pk1,iq A 2,pk2,iq qpB p1,k1,jq B 2,pk2,jq q From the definition of Khatri-Rao, it is straightforward to obtain the pρ, iq element with ρ " pk 1´1 qI 2`k2, (i.e. ρ P r1, I 1 I 2 s) of A 1 d A 2 as A 1,pk1,iq A 2,pk2,iq. Similarly, the pρ, jq element of B 1 d B 2 is B 1,pk1,jq B 2,pk2,jq. The respective pi, jq element of the left hand side (lhs) of is: In the last equation, we replace the sum in ρ (ρ P r1, I 1 I 2 s) with the equivalent sums in k 1, k 2. In a similar manner, we generalize the identity to the case of N ą 2 terms below. Lemma 2. It holds that Proof. The rhs includes the Hadamard products of the matrices A T ν¨Bν. Each matrix multiplication (A T ν¨Bν) in a matrix of KˆL dimensions. Thus, the rhs is a matrix of KˆL dimensions. The lhs is a matrix multiplication of two Khatri-Rao products. The first Khatri-Rao product has dimensions Kˆp ś ν I ν q, while the second p ś ν I ν qˆL. Altogether, the lhs has KˆL dimensions. Similarly to the previous Lemma, the pi, jq element of the rhs is: To proceed with the lhs, it is straightforward to derive that where s 1 " i and s ν is a recursive function of the s ν´1. However, the recursive definition of s ν is summed in the multiplication and we obtain: Below, we prove that (main paper) is equivalent to the three-layer neural network as shown in Figure 2. z. Then, the form of is equal to: Proof. Applying Lemma 2 on, we obtain: The last equation is the same as. In Claim 2 and Claim 3, we prove that (main paper) is equivalent to the three-layer neural network as shown in Figure 3. Claim 2. Let Proof. We will prove the equivalence starting from and transform it into. From: where in the last equation, we have applied Lemma 1. Applying the Lemma once more in the last term of, we obtain. with ω as in Claim 2. Then, it holds for Gpzq of that Gpzq " λ. Proof. Transforming into: To simplify the notation, we define M 1 " " ´A r1s dB r1s¯Sr2s ı * and M 2 "´A r2s dB r2s¯. The last term of becomes: Replacing into, we obtain. Note that the λ in Claim 3 is the equation behind Figure 3. By proving the claim, we have illustrated how the polynomial generator can be transformed into a network architecture for third-order approximation. In this Section, we will show how the Coupled CP decomposition generalizes to the N th order approximation. It suffices to find the decomposition that converts the N th order polynomial into a network structure (see Alg. 1). As done in Section 2.2, we capture the n th order interactions by decomposing the parameter tensor W rns (with 2 ď n ď N) as:... The term W rns 1:jn´1:...:j1 denotes the interactions across the layers 1, j n´1,..., j 1. The N th order approximation becomes:... By considering the mode-1 unfoding of Coupled CP decomposition (like in Section 2.2), we obtain:... where we use x N as an abbreviation of the sums. In the last equation, we have used Lemma 2 (Section B). Claim 4. The N th order approximation of can be implemented with a neural network as described in Alg. 1. Proof. We will use induction to prove the Claim. For N " 2, it trivially holds, while the proof for N " 3 is provided in Claim 1. Suppose it holds for N th order approximation; we prove below that it holds for N`1 th order approximation. Let us denote the approximation of as G N pzq. The pN`1q th order approximation from is:......... In the last equation, the first term in the sums is x N; for the rest two terms we apply Lemma 2:... The term λ is equal to the κ " pn´1q th order of, while there is only a single term for n " N. Therefore, is transformed into: which is exactly the form described by Alg. 1. This concludes the induction proof. Astroid: We implement a superellipse with parametric expression rα cos 3 t, α sin 3 ts for t P r´α, αs. This has a more complex distribution and four sharp edges. The random samples are visualized in Figure 7. PolyGAN models the data distribution accurately in contrast to the two baselines. We conduct three experiments in which the data distribution is analytically derived. The experiments are: Sin3D: The data manifold is an extension over the 2D manifold of the sinusoidal experiment (Section 4.1). The function we want to learn is Gpzq: R 2 Ñ R 3 with the data manifold described by the vector rx, y, sinp10˚ax 2`y2 qs for x, y P r´0.5, 0.5s. In Figure 8, 20, 000 samples are sampled from the generators and visualized. PolyGAN captures the data distribution, while'Orig' and'Concat' fail. Swiss roll: The three dimensional vector rt¨sin t, y, t¨cos ts`0.05¨s for t, y P r0, 1s and s " N p0, 1q forms the data manifold 2. In Figure 9, 20, 000 samples are visualized. Gabriel's Horn: The three dimensional vector rx, α¨c os t x, α¨s in t x s for t P r0, 160πs and x P r1, 4s forms the data manifold. The dependence on both sinusoidal and the function 1 x makes this curve challenging for a polynomial expansion. In Figure 10, the synthesized samples are plotted. PolyGAN learns how to generate samples on the manifold despite the fraction in the parametric form. Apart from digit generation (Section 4.2), we conduct two experiments on image generation of face and natural scenes. Since both distributions are harder than the digits one, we extend the approximation followed on Section 4.2 by one order, i.e., we assume a fifth-order approximation. We emphasize that each block is a residual block with no activation functions. Faces: In the experiment with faces, we utilize as the training samples the YaleB dataset. The dataset includes greyscale images of faces under extreme illuminations. We rescale all of the images into 64ˆ64 for our analysis. Random samples are illustrated in Figure 11. Our method generates diverse images and captures the case of illuminating either half part of the face, while'Orig' and'Concat' generate images that have a dark side only on the left and right side, respectively. The difference becomes profound in the finer details of the face (please zoom in), where both baselines fail to synthesize realistic semantic parts of the face. et al., 2001) ) for a generator with linear blocks and a single activation function only on the output (i.e., tan h). Notice that our method can illuminate either the left or right part of the face, in contrast to'Orig' (and 'Concat') which generate images that have a dark side only on the left (respectively right) side. In addition, both'Orig' and'Concat' fail to capture the fine details of the facial structure (please zoom in for the details). Natural scenes: We further evaluate the generation of natural images, specifically by training on CIFAR10 . CIFAR10 includes 50, 000 training images of 32ˆ32ˆ3 resolution. In Table 3, we evaluate the standard metrics of Inception Score (IS) and Frechet Inception Distance (FID) (see more details for the metrics in section F). Our model outperforms both'Orig' and'Concat' by a considerable margin. In Figure 12, some random synthesized samples are presented. To demonstrate the flexibility of the PolyGAN, we utilize three different popular generators. The three acrhitectures chosen are DCGAN , SNGAN , and SAGAN . Each original generator is converted into a polynomial expansion, while we use the non-linearities to boost the performance of the polynomial generator. The hyperparameters are kept the same as the corresponding baseline. Algorithms 3 and 4 succinctly present the key differences of our approach compared to the traditional one (in the case of SNGAN, similarly for other architectures). In addition to the baseline, we implement the most closely related alternative to our framework, namely instead of using the Hadamard operator as in Figure 3, we concatenate the noise with the feature representations at that block. The latter approach is frequently used in the literature (referred as "Concat" in the paper). The number of the trainable parameters of the generators are reported in Table 13. Our method has only a minimal increase of the parameters, while the concatenation increases the number of parameters substantially. To reduce the variance often observed during GAN training , each reported score is averaged over 10 runs utilizing different seeds. The metrics we utilize are Inception Score (IS) and Frechet Inception Distance (FID) . Below, we perform an ablation study on Section F.2, and then present the experiments on unsupervised (Section F.3) and conditional image generation (Section F.4) respectively. Datasets: We use CIFAR10 and Imagenet as the two most widely used baselines for GANs: • CIFAR10 includes 60, 000 images of 32ˆ32 resolution. We use 50, 000 images for training and the rest for testing. • Imagenet is a large scale dataset that includes over one million training images and 50, 000 validation images. We reshape the images to 128ˆ128 resolution. Baseline architectures: The architectures employed are: • DCGAN , as implemented in https://github.com/pytorch/ examples/tree/master/dcgan. This is a widely used baseline. • SNGAN , as implemented in https://github.com/ pfnet-research/sngan_projection. SNGAN is a strong performing GAN that introduced a spectral normalization in the discriminator. • SAGAN , as implemented in https://github.com/voletiv/ self-attention-GAN-pytorch. This is a recent network architecture that utilizes the notion of self-attention in a GAN setting, achieving impressive on Imagenet . The default hyper-parameters are left unchanged. The aforementioned codes are used for reporting the of both the baseline and our method to avoid any discrepancies, e.g. different frameworks ing in unfair comparisons. The source code will be released to enable the reproduction of our . Evaluation metrics: The popular Inception Score (IS) and Frechet Inception Distance (FID) are used for the quantitative evaluation. Both scores extract feature representations from a pretrained classifier (in practice the Inception network ). Despite their shortcomings, IS and FID are widely used , since alternative metrics fail for generative models . The Inception Score is defined as where x is a generated sample and ppy|xq is the conditional distribution for labels y. The distribution ppyq over the labels is approximated by 1 M ř M n"1 ppy|x n q for x n generated samples. Following the methods in the literature , we compute the inception score for M " 5, 000 generated samples per run (10 splits for each run). The Frechet Inception Distance (FID) utilizes feature representations from a pretrained network and assumes that the distributions of these representations are Gaussian. Denoting the representations of real images as N pµ r, C r q and the generated (fake) as N pµ f, C f q, FID is: In the experiments, we use M " 10, 000 to compute the mean and covariance of the real images and M " 10, 000 synthesized samples for µ f, C r. For both scores the original tensorflow inception network weights are used; the routines of tensorflow.contrib.gan.eval are called for the metric evaluation. We experimentally define that a (series of) affine transformation(s) on the input noise z are beneficial before using the transformed z for the Hadamard products. 3 These affine transformations are henceforth mentioned as global transformations on z. The implementation details for each network are the following: • DCGAN: We use a global transformation followed by a RELU non-linearity. WThe rest details remain the same as the baseline model. • SNGAN: Similarly to DCGAN, we use a global transformation with a RELU non-linearity. We consider each residual block as one order of approximation and compute the Hadamard product after each block (see algorithm 4). We conduct an ablation study based on SNGAN architecture (or our variant of SNGAN-poly), since most recent methods are based on similar generators;. Unless explicitly mentioned otherwise, the SNGAN is trained on CIFAR10 for unsupervised image generation. We add a global transformation on z, i.e. a fully-connected layer and use the transformed noise as input to the generator. In the first experiment, we evaluate whether to add a non-linear activation to the global transformation. The two alternatives are: i) with linear global transformation ('Ours-linear-global'), i.e. no non-linearity, and ii) with global transformation followed by a RELU non-linearity ('Ours-RELU-global'). The first two in Table 5 demonstrate that both metrics marginally improve when using a non-linear activation function. We add this global transformation with RELU on the original SNGAN. The are reported in the last two rows of Table 5 (where the original is mentioned as 'Orig', while the alternative of adding a global transformation as 'Original-RELU-global'). Split z into chunks: The recent BigGAN of performs hierarchical synthesis of images by splitting the latent vector z into one chunk per resolution (block). Each chunk is then concatenated into the respective resolution. We scrutinize this splitting against our method; we split the noise z into pk`1q non-overlapping chunks of equal size for performing k injections. The injection with splitting is mentioned as'Injectsplit' below. Our splitting deteriorates the scores on the task as reported in Table 6. It is possible that more elaborate splitting techniques, such as those in We scrutinize a feature normalization on the baseline of'Ours-RELU-global'. For each layer i we divide the A ris z vector with its standard deviation. The variant with global transformation followed by RELU and normalization before the Hadamard product is called'Ours-norm'. The in Table 7 illustrate that normalization improves the metrics. In Table 8, we use'Ours-RELU-global' as baseline against the model with the skip connection ('Ours-skip'). Since we use SNGAN both for unsupervised/conditional image generation, we verify the aforementioned in the conditional setting, i.e. when the class information is also provided to the generator and the discriminator. Normalization before Hadamard product: Similarly to the experiment above, for each layer i we divide the A ris z vector with its standard deviation. The quantitative in Table 9 improve the IS score, but the FID deteriorates. Skip the Hadamard product: Similarly to the aforementioned unsupervised case, we assess the performance if we add a skip connection in the Hadamard. In Table 10, the quantitative comparing the baseline and the skip case are presented. In this experiment, we study the image generation problem without any labels or class information for the images. The architectures of DCGAN and resnet-based SNGAN are used for image generation in CIFAR10 . Table 11 summarizes the of the IS/FID scores of the compared methods. In all of the experiments, PolyGAN outperforms the compared methods. Frequently class information is available. We can utilize the labels, e.g. use conditional batch normalization or class embeddings, to synthesize images conditioned on a class. We train two networks, et al., 2015). SAGAN uses self-attention blocks to improve the resnet-based generator. Despite our best efforts to show that our method is both architecture and database agnostic, the recent methods are run for hundreds of thousands or even million iterations till "convergence". In SAGAN the authors report that for each training multiple GPUs need to be utilized for weeks to reach the final reported Inception Score. We report the metrics for networks that are run with batch size 64 (i.e., four times less than the original 256) to fit in a single 16GB NVIDIA V100 GPU. Following the current practice in ML, due to the lack of computational budget , we run SAGAN for 400, 000 iterations (see Figure 3 of the original paper for the IS during training) 4. Each such experiment takes roughly 6 days to train. The FID/IS scores of our approach compared against the baseline method can be found in Table 12. In both cases, our proposed method yields a higher Inception Score and a lower FID. An experimental comparison of the two models described in Section 2 is conducted below. Unless explicitly mentioned otherwise, the networks used below do not include any non-linear activation functions, they are polynomial expansions with linear blocks. We use the following four experiments: Sinusoidal on 2D: The data distribution is described by rx, sinpxqs with x P r0, 2πs (see Section 4.1 for further details). We assume 8 th order approximation for Coupled CP decomposition and 12 th order for Coupled nested CP decomposition. Both have width 15 units. The comparison between the two models in Figure 13 demonstrates that they can both capture the data manifold. Impressively, the Coupled CP decomposition does not synthesize a single point that is outside of the manifold. Astroid: The data distribution is described on Section D. The samples comparing the two models are visualized in Figure 15. Sin3D: The data distribution is described on Section D. In Figure 15 the samples from the two models are illustrated. Swiss roll: The data distribution is described on Section D. In Figure 16 the samples from the two models are illustrated. We conduct an experiment on images to verify that both architectures can learn higher-dimensional distributions. We select the digit images as described in Section 4.2. In this case, Coupled CP decomposition is implemented as follows: each U ris is a series of linear convolutions with stride 2 for i " 1,..., 4, while C is a linear residual block. We emphasize that in both models all the activation functions are removed and there is a single tanh in the output of the generator for normalization purposes.

We model the data generator (in GAN) by means of a high-order polynomial represented by high-order tensors.

we define the filter-level pruning problem for binary neural networks for the first time and propose method to solve it.


Question: Represent the scientific article to find a short TLDR

Question: Deep neural networks trained on large supervised datasets have led to impressive in recent years. However, since well-annotated datasets can be prohibitively expensive and time-consuming to collect, recent work has explored the use of larger but noisy datasets that can be more easily obtained. In this paper, we investigate the behavior of deep neural networks on training sets with massively noisy labels. We show on multiple datasets such as MINST, CIFAR-10 and ImageNet that successful learning is possible even with an essentially arbitrary amount of noise. For example, on MNIST we find that accuracy of above 90 percent is still attainable even when the dataset has been diluted with 100 noisy examples for each clean example. Such behavior holds across multiple patterns of label noise, even when noisy labels are biased towards confusing classes. Further, we show how the required dataset size for successful training increases with higher label noise. Finally, we present simple actionable techniques for improving learning in the regime of high label noise. Deep learning has proven to be powerful for a wide range of problems, from image classification to machine translation. Typically, deep neural networks are trained using supervised learning on large, carefully annotated datasets. However, the need for such datasets restricts the space of problems that can be addressed. This has led to a proliferation of deep learning on the same tasks using the same well-known datasets. Carefully annotated data is difficult to obtain, especially for classification tasks with large numbers of classes (requiring extensive annotation) or with fine-grained classes (requiring skilled annotation). Thus, annotation can be expensive and, for tasks requiring expert knowledge, may simply be unattainable at scale. To address this limitation, other training paradigms have been investigated to alleviate the need for expensive annotations, such as unsupervised learning BID11, self-supervised learning BID16 BID23 and learning from noisy annotations (; BID15 BID22 . Very large datasets (e.g., BID7 ; BID19) can often be attained, for example from web sources, with partial or unreliable annotation. This can allow neural networks to be trained on a much wider variety of tasks or classes and with less manual effort. The good performance obtained from these large noisy datasets indicates that deep learning approaches can tolerate modest amounts of noise in the training set. In this work, we take this trend to an extreme, and consider the performance of deep neural networks under extremely low label reliability, only slightly above chance. We envision a future in which arbitrarily large amounts of data will easily be obtained, but in which labels come without any guarantee of validity and may merely be biased towards the correct distribution. The key takeaways from this paper may be summarized as follows:• Deep neural networks are able to learn from data that has been diluted by an arbitrary amount of noise. We demonstrate that standard deep neural networks still perform well even on training sets in which label accuracy is as low as 1 percent above chance. On MNIST, for example, performance still exceeds 90 percent even with this level of label noise (see Figure 1). This behavior holds, to varying extents, across datasets as well as patterns of label noise, including when noisy labels are biased towards confused classes.• A sufficiently large training set can accommodate a wide range of noise levels. We find that the minimum dataset size required for effective training increases with the noise level. A large enough training set can accommodate a wide range of noise levels. Increasing the dataset size further, however, does not appreciably increase accuracy.• Adjusting batch size and learning rate can allow conventional neural networks to operate in the regime of very high label noise. We find that label noise reduces the effective batch size, as noisy labels roughly cancel out and only a small learning signal remains. We show that dataset noise can be partly compensated for by larger batch sizes and by scaling the learning rate with the effective batch size. Learning from noisy data. Several studies have investigated the impact of noisy datasets on machine classifiers. Approaches to learn from noisy data can generally be categorized into two groups:In the first group, approaches aim to learn directly from noisy labels and focus on noise-robust algorithms, e.g., BID0 BID3; BID8; BID13; BID14; BID20. The second group comprises mostly label-cleansing methods that aim to remove or correct mislabeled data, e.g., BID1. Methods in this group frequently face the challenge of disambiguating between mislabeled and hard training examples. To address this challenge, they often use semi-supervised approaches by combining noisy data with a small set of clean labels BID27. Some approaches model the label noise as conditionally independent from the input image BID15 BID17 and some propose image-conditional noise models BID22 BID24. Our work differs from these approaches in that we do not aim to clean the training dataset or propose new noise-robust training algorithms. Instead, we study the behavior of standard neural network training procedures in settings with massive label noise. We show that even without explicit cleaning or noise-robust algorithms, neural networks can learn from data that has been diluted by an arbitrary amount of label noise. Analyzing the robustness of neural networks. Several investigative studies aim to improve our understanding of convolutional neural networks. One particular stream of research in this space seeks to investigate neural networks by analyzing their robustness. For example, show that network architectures with residual connections have a high redundancy in terms of parameters and are robust to the deletion of multiple complete layers during test time. Further, BID18 investigate the robustness of neural networks to adversarial examples. They show that even for fully trained networks, small changes in the input can lead to large changes in the output and thus misclassification. In contrast, we are focusing on non-adversarial noise during training time. Within this stream of research, closest to our work are studies that focus on the impact of noisy training datasets on classification performance (e.g., BID17 BID20 ; BID26). In these studies an increase in noise is assumed to decrease not only the proportion of correct examples, but also their absolute number. In contrast to these studies, we separate the effects and show in §4 that a decrease in the number of correct examples is more destructive to learning than an increase in the number of noisy labels. In this work, we are concerned with scenarios of abundant data of very poor label quality, i.e., the regime in which falsely labeled training examples vastly outnumber correctly labeled examples. In particular, our experiments involve observing the performance of deep neural networks on multiclass classification tasks as label noise is increased. To formalize the problem, we denote the number of original training examples by n. To model the amount of noise, we dilute the dataset by adding α noisy examples to the training set for each original training example. Thus, the total number of noisy labels in the training set is αn. Note that by varying the noise level α, we do not change the available number of original examples. Thus, even in the presence of high noise, there is still appreciable data to learn from, if we are able to pick it out. This is in contrast to previous work (e.g., BID17 ; BID20 ; BID26), in which an increase in noise also implies a decrease in the absolute number Figure 1: Performance on MNIST as different amounts of noisy labels are added to a fixed training set of clean labels. We compare a perceptron, MLPs with 1, 2, and 4 hidden layers, and a 4-layer ConvNet. Even with 100 noisy labels for every clean label the ConvNet still attains a performance of 91%. Performance on CIFAR-10 as different amounts of noisy labels are added to a fixed training set of clean labels. We tested ConvNets with 4 and 6 layers, and a ResNet with 101 layers. Even with 10 noisy labels for every clean label the ResNet still attains a performance of 85%. of correct examples. In the following experiments we investigate three different types of noise: uniform label-swapping, structured label-swapping, and out-of-vocabulary examples. A key assumption in this paper is that unreliable labels are better modeled by an unknown stochastic process rather than by the output of an adversary. This is a natural assumption for data that is pulled from the environment, in which antagonism is not to be expected in the noisy annotation process. Deep neural networks have been shown to be exceedingly brittle to adversarial noise patterns BID18. In this work, we demonstrate that even massive amounts of non-adversarial noise present far less of an impediment to learning. As a first experiment, we will show that common training procedures for neural networks are resilient even to settings where correct labels are outnumbered by labels sampled uniformly at random at a ratio of 100 to 1. For this experiment we focus on the task of image classification and work with three commonly used datasets, MNIST , CIFAR-10 and ImageNet BID2 ).In FIG1 we show the classification performance with varying levels of label noise. For MNIST, we vary the ratio α of randomly labeled examples to cleanly labeled examples from 0 (no noise) to 100 (only 11 out of 101 labels are correct, as compared with 10.1 for pure chance). For the more challenging dataset CIFAR-10, we vary α from 0 to 10. For the most challenging dataset ImageNet, we let α range from 0 to 5. We compare various architectures of neural networks: multilayer perceptrons with different numbers of hidden layers, convolutional networks (ConvNets) with different numbers of convolutional layers, and residual networks (ResNets) with different numbers of layers BID4. We evaluate performance after training on a test dataset that is free from noisy labels. Full details of our experimental setup are provided in §3.4.Our show that, remarkably, it is possible to attain over 90 percent accuracy on MNIST, even when there are 100 randomly labeled images for every cleanly labeled example, to attain over 85 percent accuracy on CIFAR-10 with 10 random labels for every clean label, and to attain over 70 percent top-5 accuracy on ImageNet with 5 random labels for every clean label. Thus, in this highnoise regime, deep networks are able not merely to perform above chance, but to attain accuracies that would be respectable even without noise. Further, we observe from Figures 1 and 2 that larger neural network architectures tend also to be more robust to label noise. On MNIST, the performance of a perceptron decays rapidly with in- Figure 4: Illustration of uniform and structured noise models. In the case of structured noise, the order of false labels is important; we tested decreasing order of confusion, increasing order of confusion, and random order. The parameter δ parameterizes the degree of structure in the noise. It defines how much more likely the second most likely class is over chance.creasing noise (though it still attains 40 percent accuracy, well above chance, at α = 100). The performance of a multilayer perceptron drops off more slowly, and the ConvNet is even more robust. Likewise, for CIFAR-10, the accuracy of the residual network drops more slowly than that of the smaller ConvNets. This observation provides further support for the effectiveness of ConvNets and ResNets in particular for applications where noise tolerance may be important. We have seen that neural networks are extremely robust to uniform label noise. However, label noise in datasets gathered from a natural environment is unlikely to follow a perfectly uniform distribution. In this experiment, we investigate the effects of various forms of structured noise on the performance of neural networks. Figure 4 illustrates the procedure used to model noise structure. In the uniform noise setting, as illustrated on the left side of Figure 4, correct labels are more likely than any individual false label. However, overall false labels vastly outnumber correct labels. We denote the likelihood over chance for a label to be correct as. Note that = 1/(1 + α), where α is the ratio of noisy labels to certainly correct labels. To induce structure in the noise, we bias noisy labels to certain classes. We introduce the parameter δ to parameterize the degree of structure in the noise. It defines how much more likely the second most likely class is over chance. With δ = 0 the noise is uniform, whereas for δ = 1 the second most likely class is equally likely as the correct class. The likelihood for the remaining classes is scaled linearly, as illustrated in Figure 4 on the right. We investigate three different setups for structured noise: labels biased towards easily confused classes, towards hardly confused classes and towards random classes. FIG4 shows the on MNIST for the three different types of structured noise, as δ varies from 0 to 1. In this experiment, we train 4-layer ConvNets on a dataset that is diluted with 20 noisy labels for each clean label. We vary the order of false labels so that, besides the correct class, labels are assigned most frequently to those most often confused with the correct class, those least often confused with it, and in a random order. We determine commonly confused labels by training the network repeatedly on a small subset of MNIST and observing the errors it makes on a test set. The show that deep neural nets are robust even to structured noise, as long as the correct label remains the most likely by at least a small margin. Generally, we do not observe large differences between the different models of noise structure, only that bias towards random classes seems to hurt the performance a little more than bias towards confused classes. This might help explain why we often observe quite good from real world noisy datasets, where label noise is more likely to be biased towards related and confusing classes. In the preceding experiments, we diluted the training sets with noisy examples drawn from the same dataset; i.e., falsely labeled examples were images from within other categories of the dataset. In "confusing order" (highest probability for the most confusing label), "reverse confusing order", and random order. We interpolate between uniform noise, δ = 0, and noise so highly skewed that the most common false label is as likely as the correct label, δ = 1. Except for δ ≈ 1, performance is similar to uniform noise.: Performance on CIFAR-10 for varying amounts of noisy labels. Noisy training examples are drawn from CIFAR-10 itself, but mislabeled uniformly at random, CIFAR-100, with uniformly random labels, and white noise with mean and variance chosen to match those of CIFAR-10. Noise drawn from CIFAR-100 ed in only half the drop in performance observed with noise from CIFAR-10 itself, while white noise examples did not appreciable affect performance.natural scenarios, however, noisy examples likely also include categories not included in the dataset that have erroneously been assigned labels within the dataset. Thus, we now consider two alternative sources for noisy training examples. First, we dilute the training set with examples that are drawn from a similar but different dataset. In particular, we use CIFAR-10 as our training dataset and dilute it with examples from CIFAR-100, assigning each image a category from CIFAR-10 at random. Second, we also consider a dilution of the training set with "examples" that are simply white noise; in this case, we match the mean and variance of pixels within CIFAR-10 and again assign labels uniformly at random. FIG5 shows the obtained by a six-layer ConvNet on the different noise sources for varying levels of noise. We observe that both alternative sources of noise lead to better performance than the noise originating from the same dataset. For noisy examples drawn from CIFAR-100, performance drops only about half as much as when noise originates from CIFAR-10 itself. This trend is consistent across noise levels. For white noise, performance does not drop regardless of noise level; this is in line with prior work that has shown that neural networks are able to fit random input BID26. This indicates the scenarios considered in Experiments 1 and 2 represent in some sense a worst case. In natural scenarios, we may expect massively noisy datasets to fall somewhere in between the cases exemplified by CIFAR-10 and CIFAR-100. That is, some examples will be relevant but mislabeled. However, it is likely that many examples will not be from any classes under consideration and therefore will influence training less negatively. In fact, it is possible that such examples might increase accuracy, if the erroneous labels reflect underlying similarity between the examples in question. All models are trained with AdaDelta as optimizer and a batch size of 128. For each level of label noise we train separate models with different learning rates ranging from 0.01 to 1 and pick the learning rate that in the best performance. Generally, we observe that the higher the label noise, the lower the optimal learning rate. We investigate this trend in detail in §5. There seems to be a critical amount of clean training data required to successfully train the networks. This threshold increases as the noise level rises. For example, at α = 10, 2,000 clean labels are needed to attain 90% performance, while at α = 50, 10,000 clean labels are needed. In Experiments 1 and 2, noisy labels are drawn from the same dataset as the labels guaranteed to be correct. This involves drawing the same example many times from the dataset, giving it the correct label once, and in every other instance picking a random label according to the noise distribution in question. We show in Figure 7 that performance would have been comparable had we been able to draw noisy labels from an extended dataset, instead of repeating images. Specifically, we train a convolutional network on a subset of MNIST, with 2,500 certainly correct labels and with noisy labels drawn either with repetition from this set of 2,500 or without repetition from the remaining examples in the MNIST dataset. The are essentially identical between repeated and unique examples, supporting our setup in the preceding experiments. Underlying the ability of deep networks to learn from massively noisy data is the size of the data in question. It is well-established, see e.g., BID2, that traditional deep learning relies upon large datasets. We will now see how this is particularly true of noisy datasets. In Figure 8, we compare the performance of a ConvNet on MNIST as the size of the training set varies. We also show the performance of the same ConvNet trained on MNIST diluted with noisy labels sampled uniformly. We show how the performance of the ConvNet varies with the number of cleanly labeled training examples. For example, for the blue curve of α = 10 and 1,000 clean labels, the network is trained on 11,000 examples: 1,000 cleanly labeled examples and 10,000 with random labels. Generally, we observe that independent of the noise level the networks benefit from more data and that, given sufficient data, the networks reach similar . Further, the indicate that there seems to be a critical amount of clean training data that is required to successfully train the networks. This critical amount of clean data depends on the noise level; in particular, it increases as the noise level rises. Since performance rapidly levels off past the critical threshold the main requirement for the clean training set is to be of sufficient size. It is because of the critical amount of required clean data that we have not attempted to train networks for α 100. The number of correct examples needed to train such a network might rise above the 60,000 provided in the MNIST dataset. In a real-world dataset, the amount of (noisy) data available for training is likely not to be the limiting factor. Rather, considerations such as training time and learning rate may play a more important role, as we discuss in the following section. In the preceding sections, our were obtained by training neural networks with fixed batch size and running a parameter search to pick the optimal learning rate. We now look in more detail into how the choice of hyperparameters affects learning on noisy datasets. First, we investigate the effect of the batch size on the noise robustness of neural network training. In Figure 9, we compare the performance of a simple 2-layer ConvNet on MNIST with increasing noise, as batch size varies from 32 to 256. We observe that increasing the batch size provides greater robustness to noisy labels. One reason for this behavior could be that, within a batch, gradient updates from randomly sampled noisy labels cancel out, while gradients from correct examples that are marginally more frequent sum together and contribute to learning. By this logic, large batch sizes would be more robust to noise since the mean gradient over a larger batch is closer to the gradient for correct labels. All other experiments in this paper are performed with a fixed batch size of 128.We may also consider the theoretical case of infinite batch size, in which gradients are averaged over the entire space of possible inputs at each training step. While this is often impossible to perform in practice, we can simulate such behavior by an auxiliary loss function. In classification tasks, we are given an input x and aim to predict the class f (x) ∈ {1, 2, . . ., m}. The value f (x) is encoded within a neural network by the 1-hot vector y(x) such that DISPLAYFORM0 for 1 ≤ k ≤ m. Then, the standard cross-entropy loss over a batch X is given by: DISPLAYFORM1 whereŷ is the predicted vector and · X denotes the expected value over the batch X. We assume thatŷ is normalized (e.g. by the softmax function) so that the entries sum to 1.For a training set with noisy labels, we may consider the label f (x) given in the training set to be merely an approximation to the true label f 0 (x). Consider the case of n training examples, and αn noisy labels that are sampled uniformly at random from the set {1, 2, . . ., m}. Then, f (x) = f 0 (x) with probability 1 1+α, and otherwise it is 1, 2,..., m, each with probability α m(1+α). As batch size increases, the expected value over the batch X is approximated more closely by these probabilities. In the limit of infinite batch size, equation FORMULA1 takes the form of a noisy loss function H α: DISPLAYFORM2 We can therefore compare training using the cross-entropy loss with αn noisy labels to training using the noisy loss function H α without noisy labels. The term on the right-hand side of represents the noise contribution, and is clearly minimized whereŷ k are all equal. As α increases, this contribution is weighted more heavily against − logŷ f0(x) X, which is minimized atŷ(x) = y(x).We show in Figure 9 the of training our 2-layer ConvNet on MNIST with the noisy loss function H α, simulating αn noisy labels with infinite batch size. We can observe that the network's accuracy does not decrease as α increases. This can be explained by the observation that an increasing α is merely decreasing the magnitude of the true gradient, rather than altering its direction. Our observations indicate that increasing noise in the training set reduces the effective batch size, as noisy signals roughly cancel out and only small learning signal remains. We show that increasing the batch size is a simple practical means to mitigate the effect of noisy training labels. It has become common practice in training deep neural networks to scale the learning rate with the batch size. In particular, it has been shown that the smaller the batch size, the lower the optimal learning rate BID9. In our experiments, we have observed that noisy labels reduce the effective batch size. As such, we would expect that lower learning rates perform better than large learning rates as noise increases. Figure 10 shows the performance of a 4-layer ConvNet trained with different learning rates on CIFAR-10 for varying label noise. As expected, we observe that the optimal learning rate decreases as noise increases. For example, the optimal learning rate for the clean dataset is 1, while, with the introduction of noise, this learning rate becomes unstable. To sum up, we observe that increasing label noise reduces the effective batch size. We have shown that the effect of label noise can be partly counterbalanced for by a larger training batch size. Now, we see that one can additionally scale the learning rate to compensate for any remaining change in effective batch size induced by noisy labels. In this paper, we have considered the behavior of deep neural networks on training sets with very noisy labels. In a series of experiments, we have demonstrated that learning is robust to an essentially arbitrary amount of label noise, provided that the number of clean labels is sufficiently large. We have further shown that the threshold required for clean labels increases as the noise level does. Finally, we have observed that noisy labels reduce the effective batch size, an effect that can be mitigated by larger batch sizes and downscaling the learning rate. It is worthy of note that although deep networks appear robust to even high degrees of label noise, clean labels still always perform better than noisy labels, given the same quantity of training data. Further, one still requires expert-vetted test sets for evaluation. Lastly, it is important to reiterate that our studies focus on non-adversarial noise. Our work suggests numerous directions for future investigation. For example, we are interested in how label-cleaning and semi-supervised methods affect the performance of networks in a high-noise regime. Are such approaches able to lower the threshold for training set size? Finally, it remains to translate the we present into an actionable trade-off between data annotation and acquisition costs, which can be utilized in real world training pipelines for deep networks on massive noisy data.

Represent the short summary for finding the corresponding scientific article (long)

We show that deep neural networks are able to learn from data that has been diluted by an arbitrary amount of noise.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce a statistical approach to assessing neural network robustness that provides an informative notion of how robust a network is, rather than just the conventional binary assertion of whether or not of property is violated.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In this paper, we propose to extend the recently introduced model-agnostic meta-learning algorithm for low resource neural machine translation (NMT). We frame low-resource translation as a meta-learning problem, and we learn to adapt to low-resource languages based on multilingual high-resource language tasks. We use the universal lexical representation (b) to overcome the input-output mismatch across different languages. We evaluate the proposed meta-learning strategy using eighteen European languages (Bg, Cs, Da, De, El, Es, Et, Fr, Hu, It, Lt, Nl, Pl, Pt, Sk, Sl, Sv and Ru) as source tasks and five diverse languages (Ro, Lv, Fi, Tr, and Ko) as target tasks. We show that the proposed approach significantly outperforms the multilingual, transfer learning based approach and enables us to train a competitive NMT system with only a fraction of training examples. For instance, the proposed approach can achieve as high as 22.04 BLEU on Romanian-English WMT’16 by seeing only 16,000 translated words (~600 parallel sentences). Despite the massive success brought by neural machine translation (NMT, BID36 BID4 BID37, it has been noticed that the vanilla NMT often lags behind conventional machine translation systems, such as statistical phrase-based translation systems (PBMT, BID24, for low-resource language pairs (see, e.g., BID23 . In the past few years, various approaches have been proposed to address this issue. The first attempts at tackling this problem exploited the availability of monolingual corpora BID17 BID32 BID40 . It was later followed by approaches based on multilingual translation, in which the goal was to exploit knowledge from high-resource language pairs by training a single NMT system on a mix of high-resource and low-resource language pairs (a,b; BID27 BID21 BID19 . Its variant, transfer learning, was also proposed by BID42, in which an NMT system is pretrained on a high-resource language pair before being finetuned on a target low-resource language pair. In this paper, we follow up on these latest approaches based on multilingual NMT and propose a meta-learning algorithm for low-resource neural machine translation. We start by arguing that the recently proposed model-agnostic meta-learning algorithm could be applied to low-resource machine translation by viewing language pairs as separate tasks. This view enables us to use MAML to find the initialization of model parameters that facilitate fast adaptation for a new language pair with a minimal amount of training examples (§3). Furthermore, the vanilla MAML however cannot handle tasks with mismatched input and output. We overcome this limitation by incorporating the universal lexical representation BID15 and adapting it for the meta-learning scenario (§3.3).We extensively evaluate the effectiveness and generalizing ability of the proposed meta-learning algorithm on low-resource neural machine translation. We utilize 17 languages from Europarl and Russian from WMT as the source tasks and test the meta-learned parameter initialization against five target languages (Ro, Lv, Fi, Tr and Ko), in all cases translating to English. Our experiments using only up to 160k tokens in each of the target task reveal that the proposed meta-learning approach outperforms the multilingual translation approach across all the target language pairs, and the gap grows as the number of training examples 2 Background Neural Machine Translation (NMT) Given a source sentence X = {x 1, ..., x T}, a neural machine translation model factors the distribution over possible output sentences Y = {y 1, ..., y T} into a chain of conditional probabilities with a leftto-right causal structure: DISPLAYFORM0 where special tokens y 0 (bos) and y T +1 (eos) are used to represent the beginning and the end of a target sentence. These conditional probabilities are parameterized using a neural network. Typically, an encoder-decoder architecture BID36 BID9 BID4 with a RNN-based decoder is used. More recently, architectures without any recurrent structures BID13 BID37 have been proposed and shown to speedup training while achieving state-of-the-art performance. Low Resource Translation NMT is known to easily over-fit and in an inferior performance when the training data is limited BID23. In general, there are two ways for handling the problem of low resource translation: utilizing the resource of unlabeled monolingual data, and sharing the knowledge between low-and high-resource language pairs. Many research efforts have been spent on incorporating the monolingual corpora into machine translation, such as multi-task learning BID17 ), back-translation , dual learning BID20 and unsupervised machine translation with monolingual corpora only for both sides BID3 BID26. For the second approach, prior researches have worked on methods to exploit the knowledge of auxiliary translations, or even auxiliary tasks. For instance, BID8; BID28 investigate the use of a pivot to build a translation path between two languages even without any directed resource. The pivot can be a third language or even an image in multimodal domains. When pivots are not easy to obtain, BID11; BID27; BID21 have shown that the structure of NMT is suitable for multilingual machine translation. BID15 also showed that such a multilingual NMT system could improve the performance of low resource translation by using a universal lexical representation to share embedding information across languages. All the previous work for multilingual NMT assume the joint training of multiple high-resource languages naturally in a universal space (for both the input representation and the model) which, however, is not necessarily true, especially for very low resource cases. Meta Learning In the machine learning community, meta-learning, or learning-to-learn, has recently received interests. Meta-learning tries to solve the problem of "fast adaptation on new training data." One of the most successful applications of meta-learning has been on few-shot (or oneshot) learning BID25, where a neural network is trained to readily learn to classify inputs based on only one or a few training examples. There are two categories of meta-learning:1. learning a meta-policy for updating model parameters (see, e.g., BID1 BID18 BID30 2. learning a good parameter initialization for fast adaptation (see, e.g., BID10 BID38 BID35 .In this paper, we propose to use a meta-learning algorithm for low-resource neural machine translation based on the second category. More specifically, we extend the idea of model-agnostic metalearning in the multilingual scenario. The underlying idea of MAML is to use a set of source tasks T 1,..., T K to find the initialization of parameters θ 0 from which learning a target task T 0 would require only a small number of training examples. In the context of machine translation, this amounts to using many high-resource language pairs to find good initial parameters and training a new translation model on a low-resource language starting from the found initial parameters. This process can be understood as That is, we meta-learn the initialization from auxiliary tasks and continue to learn the target task. DISPLAYFORM0 We refer the proposed meta-learning method for NMT to MetaNMT. See FIG0 for the overall illustration. Given any initial parameters θ 0 (which can be either random or meta-learned), the prior distribution of the parameters of a desired NMT model can be defined as an isotropic Guassian: DISPLAYFORM0 where 1/β is a variance. With this prior distribution, we formulate the language-specific learning process Learn(D T ; θ 0) as maximizing the logposterior of the model parameters given data D T: DISPLAYFORM1 where we assume p(X|θ) to be uniform. The first term above corresponds to the maximum likelihood criterion often used for training a usual NMT system. The second term discourages the newly learned model from deviating too much from the initial parameters, alleviating the issue of overfitting when there is not enough training data. In practice, we solve the problem above by maximizing the first term with gradient-based optimization and early-stopping after only a few update steps. Thus, in the low-resource scenario, finding a good initialization θ 0 strongly correlates the final performance of the ing model. We find the initialization θ 0 by repeatedly simulating low-resource translation scenarios using auxiliary, high-resource language pairs. Following Finn et al. FORMULA0 we achieve this goal by defining the meta-objective function as DISPLAYFORM0 where k ∼ U({1, . . ., K}) refers to one metalearning episode, and D T, D T follow the uniform distribution over T's data. We maximize the meta-objective function using stochastic approximation BID31 with gradient descent. For each episode, we uniformly sample one source task at random, T k. We then sample two subsets of training examples independently from the chosen task, D T k and D T k. We use the former to simulate languagespecific learning and the latter to evaluate its outcome. Assuming a single gradient step is taken only the with learning rate η, the simulation is: DISPLAYFORM1 Once the simulation of learning is done, we evaluate the updated parameters θ k on D T k, The gradient computed from this evaluation, which we refer to as meta-gradient, is used to update the meta model θ. It is possible to aggregate multiple episodes of source tasks before updating θ: where η is the meta learning rate. Unlike a usual learning scenario, the ing model θ 0 from this meta-learning procedure is not necessarily a good model on its own. It is however a good starting point for training a good model using only a few steps of learning. In the context of machine translation, this procedure can be understood as finding the initialization of a neural machine translation system that could quickly adapt to a new language pair by simulating such a fast adaptation scenario using many high-resource language pairs. DISPLAYFORM2 We use the following approximation property DISPLAYFORM0 where ν is a small constant and DISPLAYFORM1 In practice, we find that it is also possible to ignore the second-order term, ending up with the following simplified update rule: DISPLAYFORM2 Related Work: Multilingual Transfer Learning The proposed MetaNMT differs from the existing framework of multilingual translation BID27 BID21 BID15 or transfer learning BID42. The latter can be thought of as solving the following problem: DISPLAYFORM3 We omit the subscript k for simplicity.where D k is the training set of the k-th task, or language pair. The target low-resource language pair could either be a part of joint training or be trained separately starting from the solution θ 0 found from solving the above problem. The major difference between the proposed MetaNMT and these multilingual transfer approaches is that the latter do not consider how learning happens with the target, low-resource language pair. The former explicitly incorporates the learning process within the framework by simulating it repeatedly in Eq.. As we will see later in the experiments, this in a substantial gap in the final performance on the low-resource task. Illustration In Fig. 2, we contrast transfer learning, multilingual learning and meta-learning using three source language pairs (Fr-En, Es-En and Pt-En) and two target pairs (Ro-En and Lv-En). Transfer learning trains an NMT system specifically for a source language pair (Es-En) and finetunes the system for each target language pair (RoEn, Lv-En). Multilingual learning often trains a single NMT system that can handle many different language pairs (Fr-En, Pt-En, Es-En), which may or may not include the target pairs (Ro-En, LvEn). If not, it finetunes the system for each target pair, similarly to transfer learning. Both of these however aim at directly solving the source tasks. On the other hand, meta-learning trains the NMT system to be useful for fine-tuning on various tasks including the source and target tasks. This is done by repeatedly simulating the learning process on low-resource languages using many high-resource language pairs (Fr-En, Pt-En, Es-En). I/O mismatch across language pairs One major challenge that limits applying meta-learning for low resource machine translation is that the approach outlined above assumes the input and output spaces are shared across all the source and target tasks. This, however, does not apply to ma-chine translation in general due to the vocabulary mismatch across different languages. In multilingual translation, this issue has been tackled by using a vocabulary of sub-words BID32 or characters BID27 shared across multiple languages. This surface-level sharing is however limited, as it cannot be applied to languages exhibiting distinct orthography (e.g., IndoEuroepan languages vs. Korean.)Universal Lexical Representation (ULR) We tackle this issue by dynamically building a vocabulary specific to each language using a keyvalue memory network (; BID16, as was done successfully for low-resource machine translation recently by BID15 . We start with multilingual word embedding matrices k query ∈ R |V k |×d pretrained on large monolingual corpora, where V k is the vocabulary of the k-th language. These embedding vectors can be obtained with small dictionaries of seed word pairs BID2 BID34 or in a fully unsupervised manner BID41 BID0 . We take one of these languages k to build universal lexical representation consisting of a universal embedding matrix u ∈ R M ×d and a corresponding key matrix key ∈ R M ×d, where M < |V k |. Both k query and key are fixed during meta-learning. We then compute the language-specific embedding of token x from the language k as the convex sum of the universal embedding vectors by DISPLAYFORM0 DISPLAYFORM1 and τ is set to 0.05. This approach allows us to handle languages with different vocabularies using a fixed number of shared parameters ( u, key and A.)Learning of ULR It is not desirable to update the universal embedding matrix u when finetuning on a small corpus which contains a limited set of unique tokens in the target language, as it could adversely influence the other tokens' embedding vectors. We thus estimate the change to each embedding vector induced by languagespecific learning by a separate parameter ∆ k [x]: DISPLAYFORM2 During language-specific learning, the ULR Preprocessing and ULR Initialization As described in §3.3, we initialize the query embedding vectors k query of all the languages. For each language, we use the monolingual corpora built from Wikipedia 7 and the parallel corpus. The concatenated corpus is first tokenized and segmented using byte-pair encoding (BPE, BID33, ing in 40, 000 subwords for each language. We then estimate word vectors using fastText BID5 and align them across all the languages in an unsupervised way using MUSE BID0 to get multilingual word vectors. We use the multilingual word vectors of the 20,000 most frequent words in English to form the universal embedding matrix u . Model We utilize the recently proposed Transformer BID37 as an underlying NMT system. We implement Transformer in this paper based on BID14 8 and mod-ify it to use the universal lexical representation from §3.3. We use the default set of hyperparameters (d model = d hidden = 512, n layer = 6, n head = 8, n batch = 4000, t warmup = 16000) for all the language pairs and across all the experimental settings. We refer the readers to BID37 BID14 for the details of the model. However, since the proposed metalearning method is model-agnostic, it can be easily extended to any other NMT architectures, e.g. RNN-based sequence-to-sequence models with attention BID4.Learning We meta-learn using various sets of source languages to investigate the effect of source task choice. For each episode, by default, we use a single gradient step of language-specific learning with Adam BID22 per computing the meta-gradient, which is computed by the first-order approximation in Eq..For each target task, we sample training examples to form a low-resource task. We build tasks of 4k, 16k, 40k and 160k English tokens for each language. We randomly sample the training set five times for each experiment and report the average score and its standard deviation. Each fine-tuning Ro is done on a training set, early-stopped on a validation set and evaluated on a test set.-En Lv-En Fi-En Tr-En Ko-EnFine-tuning Strategies The transformer consists of three modules; embedding, encoder and decoder. We update all three modules during metalearning, but during fine-tuning, we can selectively tune only a subset of these modules. Following BID42, we consider three fine-tuning strategies; fine-tuning all the modules (all), fine-tuning the embedding and encoder, but freezing the parameters of the decoder (emb+enc) and fine-tuning the embedding only (emb). We metalearn the initial models on all the source tasks using either Ro-En or Lv-En as a validation task. We also train the initial models to be multilingual translation systems. We fine-tune them using the four target tasks (Ro-En, Lv-En, Fi-En and Tr-En; 16k tokens each) and compare the proposed meta-learning strategy and the multilingual, transfer learning strategy. As presented in FIG2, the proposed learning approach significantly outperforms the multilingual, transfer learning strategy across all the target tasks regardless of which target task was used for early stopping. We also notice that the emb+enc strategy is most effective for both meta-learning and transfer learning approaches. With the proposed meta-learning and emb+enc fine-tuning, the final NMT systems trained using only a fraction of all available training examples achieve 2/3 (Ro-En) and 1/2 (Lv-En, Fi-En and Tr-En) of the BLEU score achieved by the models trained with full training sets. Similarly to training any other neural network, meta-learning still requires early-stopping to avoid overfitting to a specific set of source tasks. In doing so, we observe that the choice of a validation task has nonnegligible impact on the final performance. For instance, as shown in FIG2, Fi-En benefits more when Ro-En is used for validation, while the opposite happens with Tr-En. The relationship between the task similarity and the impact of a validation task must be investigated further in the future. Training Set Size We vary the size of the target task's training set and compare the proposed meta-learning strategy and multilingual, transfer learning strategy. We use the emb+enc fine-tuning on Ro-En and Fi-En. FIG3 demonstrates that the meta-learning approach is more robust to the drop in the size of the target task's training set. The gap between the meta-learning and transfer learning grows as the size shrinks, confirming the effectiveness of the proposed approach on extremely lowresource language pairs. TAB2, we present the on all five target tasks obtained while varying the source task set. We first see that it is Source (Tr) google mülteciler için 11 milyon dolar toplamaküzere bagış eşleştirme kampanyasını başlattı. Target google launches donation-matching campaign to raise $ 11 million for refugees. Meta-0 google refugee fund for usd 11 million has launched a campaign for donation. Meta-16k google has launched a campaign to collect $ 11 million for refugees. always beneficial to use more source tasks. Although the impact of adding more source tasks varies from one language to another, there is up to 2× improvement going from one source task to 18 source tasks (Lv-En, Fi-En, Tr-En and Ko-En). The same trend can be observed even without any fine-tuning (i.e., unsupervised translation, BID26 BID3). In addition, the choice of source languages has different implications for different target languages. For instance, Ro-En benefits more from {Es, Fr, It, Pt} than from {De, Ru}, while the opposite effect is observed with all the other target tasks. Training Curves The benefit of meta-learning over multilingual translation is clearly demonstrated when we look at the training curves in FIG4. With the multilingual, transfer learning approach, we observe that training rapidly saturates and eventually degrades, as the model overfits to the source tasks. MetaNMT on the other hand continues to improve and never degrades, as the metaobjective ensures that the model is adequate for fine-tuning on target tasks rather than for solving the source tasks. Sample Translations We present some sample translations from the tested models in TAB4.Inspecting these examples provides the insight into the proposed meta-learning algorithm. For instance, we observe that the meta-learned model without any fine-tuning produces a word-by-word translation in the first example (Tr-En), which is due to the successful use of the universal lexcial representation and the meta-learned initialization. The system however cannot reorder tokens from Turkish to English, as it has not seen any training example of Tr-En. After seeing around 600 sentence pairs (16K English tokens), the model rapidly learns to correctly reorder tokens to form a better translation. A similar phenomenon is observed in the Ko-En example. These cases could be found across different language pairs. In this paper, we proposed a meta-learning algorithm for low-resource neural machine translation that exploits the availability of high-resource languages pairs. We based the proposed algorithm on the recently proposed model-agnostic metalearning and adapted it to work with multiple languages that do not share a common vocabulary using the technique of universal lexcal representation, ing in MetaNMT. Our extensive evaluation, using 18 high-resource source tasks and 5 low-resource target tasks, has shown that the proposed MetaNMT significantly outperforms the existing approach of multilingual, transfer learning in low-resource neural machine translation across all the language pairs considered. The proposed approach opens new opportunities for neural machine translation. First, it is a principled framework for incorporating various extra sources of data, such as source-and targetside monolingual corpora. Second, it is a generic framework that can easily accommodate existing and future neural machine translation systems.

we propose a meta-learning approach for low-resource neural machine translation that can rapidly learn to translate on a new language

represent the tldr for finding the corresponding scientific article

We revisit self-training as a semi-supervised learning method for neural sequence generation problem, and show that self-training can be quite successful with injected noise.