Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Low bit-width weights and activations are an effective way of combating the increasing need for both memory and compute power of Deep Neural Networks. In this work, we present a probabilistic training method for Neural Network with both binary weights and activations, called PBNet. By embracing stochasticity during training, we circumvent the need to approximate the gradient of functions for which the derivative is zero almost always, such as $\textrm{sign}(\cdot)$, while still obtaining a fully Binary Neural Network at test time. Moreover, it allows for anytime ensemble predictions for improved performance and uncertainty estimates by sampling from the weight distribution. Since all operations in a layer of the PBNet operate on random variables, we introduce stochastic versions of Batch Normalization and max pooling, which transfer well to a deterministic network at test time. We evaluate two related training methods for the PBNet: one in which activation distributions are propagated throughout the network, and one in which binary activations are sampled in each layer. Our experiments indicate that sampling the binary activations is an important element for stochastic training of binary Neural Networks. Deep Neural Networks are notorious for having vast memory and computation requirements, both during training and test/prediction time. As such, Deep Neural Networks may be unfeasible in various environments such as battery powered devices, embedded devices (because of memory requirement), on body devices (due to heat dissipation), or environments in which constrains may be imposed by a limited economical budget. Hence, there is a clear need for Neural Networks that can operate in these resource limited environments. One method for reducing the memory and computational requirements for Neural Networks is to reduce the bit-width of the parameters and activations of the Neural Network. This can be achieved either during training (e.g., BID15 ; BID0) or using post-training mechanisms (e.g., BID15, BID5). By taking the reduction of the bit-width for weights and activations to the extreme, i.e., a single bit, one obtains a Binary Neural Network. Binary Neural Networks have several advantageous properties, i.e., a 32× reduction in memory requirements and the forward pass can be implemented using XNOR operations and bit-counting, which in a 58× speedup on CPU BID20. Moreover, Binary Neural Networks are more robust to adversarial examples BID2. BID21 introduced a probabilistic training method for Neural Networks with binary weights, but allow for full precision activations. In this paper, we propose a probabilistic training method for Neural Networks with both binary weights and binary activations, which are even more memory and computation efficient. In short, obtain a closed form forward pass for probabilistic neural networks if we constrain the input and weights to binary (random) variables. The output of the Multiply and Accumulate (MAC) operations, or pre-activation, is approximated using a factorized Normal distribution. Subsequently, we introduce stochastic versions of Max-Pooling and Batch Normalization that allow us to propagate the pre-activatoins throughout a single layer. By applying the sign(·) activation function to the random pre-activation, we not only obtain a distribution over binary activations, it also allows for backpropagation through the sign(·) operation. This is especially convenient as this in a deterministic Neural Network all gradient information is zeroed out when using sign as activation. We explore two different methods for training this probabilistic binary neural network: In the first method the activation distribution of layer l is propagated to layer (l + 1), which means the MAC operation is performed on two binary random variables. In the second method the binary activation is sampled as the last operation in a layer using the concrete relaxation BID16. This can be thought of as a form of local reparametrization BID11. We call the networks obtained using these methods PBNet and PBNet-S, respectively. At test time, we obtain a single deterministic Binary Neural Network, an ensemble of Binary Neural Networks by sampling from the parameter distribution, or a Ternary Neural Network based on the Binary weight distribution. An advantage of our method is that we can take samples from the parameter distribution indefinitely-without retraining. Hence, this method allows for anytime ensemble predictions and uncertainty estimates. Note that while in this work we only consider the binary case, our method supports any discrete distribution over weights and activations. Algorithm 1: Pseudo code for forward pass of single layer in PBNet(-S). a l−1 denotes the activation of the previous layer, B the random binary weight matrix, τ is the temperature used for the concrete distribution, f (·, ·) the linear transformation used in the layer, > 0 a small constant for numerical stability, D the dimensionality of the inner product in f, and γ & β are the parameters for batch normalization. DISPLAYFORM0 // Max pooling if max pooling required then n ∼ N (0, I); s = µ + σ n; ι = max-pooling-indices(s); µ, σ 2 = select-at-indices(µ, σ 2, ι); end // Binarization and sampling DISPLAYFORM1 In this section we introduce the probabilistic setting of the PBNet. Moreover, the approximation of the distribution on the pre-activations is introduced. For an explanation of the other operations in the PBNet, see Section 2.1 for the activation, Section 2.1.1 for the sampling of activations, and Section 2.2 for Pooling and Normalization. We aim to train a probabilistic Binary Neural Network. As such, we pose a binary distribution over the weights of the network and optimize the parameters of this distribution instead of the parameters directly. This way, we obtain a distribution over parameters, but also deal with the inherent discreteness of a Binary Neural Network. Given an objective function L(·), this approach can be thought of in terms of the variational optimization framework BID23. Specifically, by optimizing the parameters of the weight distributions, we optimize a bound on the actual loss: min DISPLAYFORM2 where B are the binary weights of the network and q θ (B) is a distribution over the binary weights. For q θ (B) a slight reparametrization of the Bernoulli distribution is used, which we will refer to as the Binary distribution. This distribution is parameterized by θ ∈ [−1, 1] and is defined by: DISPLAYFORM3 For the properties of this distribution, please refer to Appendix A.We will now consider using the Binary distribution for both the weights and the activations in a Neural Network. Since the pre-activations in a Neural Network are computed using MAC operations, which are the same for each value in the pre-activation, we will only consider a single value in our discussion here. Let w ∼ Binary(θ) and h ∼ Binary(φ) be the weight and input random variable for a given layer. As such, the innerproduct between the weights and input is distributed according to a translated and scaled Poisson binomial distribution: DISPLAYFORM4 Where D is the dimensionality of h and w and denotes element-wise multiplication. See the picket fence on the top in FIG0 for an illustration of the PMF of a Poisson binomial distribution. Although the scaled and translated Poisson binomial distribution is the exact solution for the inner product between the weight and activation random variables, it is hard to work with in subsequent layers. For this reason, and the fact that the Poisson binomial distribution is well approximated by a Normal distribution , we use a Normal approximation to the Poisson binomial distribution, which allows for easier manipulations. Using the properties of the Binary distribution and the Poisson binomial distribution, the approximation for the pre-activation a is given by: DISPLAYFORM5 Note that, this is exactly the approximation one would obtain by using the Lyapunov Central Limit Theorem (CLT), which was used by BID21. This allows us to obtain a close approximation to the pre-activation distribution, which we can propagate through the layer and/or network. So far, only the MAC operation in a given layer is discussed, in Section 2.1 application of the binary activation is discussed and in Section 2.1. The stochastic versions of Batch Normalization and Max Pooling are introduced in Section 2.2. For specifics on sampling the binary activation, see Section 2.1.1. The full forward pass for a single layer is given in detail in Algorithms 1. Since the output of a linear operation using binary inputs is not restricted to be binary, it is required to apply a binarization operation to the pre-activation in order to obtain binary activations. Various works -e.g., BID7 and BID20 -use either deterministic or stochastic binarization functions, i.e., DISPLAYFORM0 +1 with probability p = sigmoid(a) −1 with probability 1 − p.In our case the pre-activations are random variables. Hence, applying a deterministic binarization function to a random pre-activations in a stochastic binary activation. Specifically, let a i ∼ N (µ i, σ 2 i) be a random pre-ctivation obtained using the normal approximation, as introduced in the previous section, then the activation (after binarization) is again given as a Binary random variable". Interestingly, the Binary probability can be computed in closed form by evaluating the probability density that lies above the binarization threshold: DISPLAYFORM1 where Φ(·|µ, σ 2) denotes the CDF of N (µ, σ 2). Applying the binarization function to a random pre-activation has two advantages. First, the derivatives ∂q i /∂µ i and ∂q i /∂σ i are not zero almost everywhere, in contrast to the derivatives of b det and b stoch when applied to a deterministic input. Second, the distribution over h i reflects the true uncertainty about the sign of the activation, given the stochastic weights, whereas b stoch uses the magnitude of the pre-activation as a substitute. For example, a pre-activation with a high positive magnitude and high variance will be deterministically mapped to 1 by b stoch. In contrast, our method takes the variance into account and correctly assigns some probability mass to −1. See FIG0 for a graphical depiction of the stochastic binary activation. So far, we have discussed propagating distributions throughout the network. Alternatively, the binary activations can be sampled using the Concrete distribution BID16 during training. specifically, we use the hard sample method as discussed by BID9. By sampling the activations, the input for subsequent layers will match the input that is observed at test time more closely. As a consequence of sampling the activation, the input to a layer is no longer a distribution but a h ∈ {−1, +1} D vector instead. As such, the normal approximation to the pre-activation is computed slightly different. From the Lyapunov CLT it follows that the approximation to the distribution of the pre-activation is given by: DISPLAYFORM0 where w ∼ Binary(θ) is a random weight. Similarly, the pre-activation of the input layer is also computed using this approximation-given a real-valued input vector. We will refer to a PBNet that uses activation sampling as PBNet-S. Other than a linear operation and an (non-linear) activation function, Batch Normalization BID8 and pooling are two popular building blocks for Convolutional Neural Networks. For Binary Neural Networks, applying Batch Normalization to a binarized activation will in a non-binary . Moreover, the application of max pooling on a binary activation will in a feature map containing mostly +1s. Hence, both operations must be applied before binarization. However, in the PBNet, the binarization operation is applied before sampling. As a consequence, the Batch Normalization and pooling operations can only be applied on random pre-activations. For this reason, we define these methods for random variables. Although there are various ways to define these operation in a stochastic fashion, our guiding principle is to only leverage stochasticity during training, i.e., at test time, the stochastic operations are replaced by their conventional implementations and parameters learned in the stochastic setting must be transferred to their deterministic counterparts. Batch Normalization (BN) BID8 -including an affine transformation -is defined as follows: DISPLAYFORM0 where a i denotes the pre-activation before BN,â the pre-activation after BN, and m & v denote the sample mean and variance of DISPLAYFORM1, for an M -dimensional pre-activation. In essence, BN translates and scales the pre-activations such that they have approximately zero mean and unit variance, followed by an affine transformation. Hence, in the stochastic case, our aim is that samples from the pre-activation distribution after BN also have approximately zero mean and unit variance-to ensure that the stochastic batch normalization can be transfered to a deterministic binary neural network. This is achieved by subtracting the population mean from each pre-activation random variable and by dividing by the population variance. However, since a i is a random variable in the PBNet, simply using the population mean and variance equations will in non-standardized output. Instead, to ensure a standardized distribution over activations, we compute the expected population mean and variance under the pre-activation distribution: DISPLAYFORM2 where M is the total number of activations and a i ∼ N (µ i, σ i) are the random pre-activations. By substituting m and v in Equation 8 by Equation 9 and 10, we obtain the following batch normalized Gaussian distributions for the pre-activations: DISPLAYFORM3 Note that this assumes a single channel, but is easily extended to 2d batch norm in a similar fashion as conventional Batch Normalization. At test time, Batch Normalization in a Binary Neural Network can be reduced to an addition and sign flip of the activation, see Appendix B for more details. In general, pooling applies an aggregation operation to a set of (spatially oriented) pre-activations. Here we discuss max pooling for stochastic pre-activations, however, similar considerations apply for other types of aggregation functions. In the case of max-pooling, given a spatial region containing stochastic pre-activations a 1,..., a K, we aim to stochastically select one of the a i. Note that, although the distribution of max(a 1, . . ., a K) is well-defined BID18, its distribution is not Gaussian and thus does not match one of the input distributions. Instead, we sample one of the input random variables in every spatial region according to the probability of that variable being greater than all other variables, i.e., ρ i = p(a i > z \i), where z \i = max({a j} j =i ). ρ i could be obtained by evaluating the CDF of (z \i − a i) at 0, but to our knowledge this has no analytical form. Alternatively, we can use Monte-Carlo integration to obtain ρ: DISPLAYFORM0 where one-hot(i) returns a K-dimensional one-hot vector with the ith elements set to one. The pooling index ι is then sampled from Cat(ρ). However, more efficiently, we can sample s ∼ p(a 1, . . ., a K) and select the index of the maximum in s, which is equivalent sampling from Cat(ρ). Hence, for a given max pooling region, it is sufficient to obtain a single sample from each normal distribution associated with each pre-activation and keep the random variable for which this sample is maximum. A graphical overview of this is given in Figure 2.Other forms of stochastic or probabilistic max pooling were introduced by BID13 , however, in both cases a single activation is sampled based on the magnitude of the activations. In contrast, in our procedure we stochastically propagate one of the input distributions over activations. For the PBNet the parameters θ for q θ (B) are initialized from a uniform U (−1, 1) distribution. Although the final parameter distribution more closely follows a Beta(α, α) distribution, for α < 1, we did not observe any significant impact choosing another initialization method for the PBNet. In the case of the PBNet-S, we observed a significant improvement in training speed and performance by initializing the parameters based on the parameters of a pre-trained full precission Neural Network. This initializes the convolutional filters with more structure than a random initialization. This is desirable as in order to flip the value of a weight, the parameter governing the weight has to pass through a high variance regime, which can slow down convergence considerably. Select maximum per region 2 Sample from input distributions 1Keep maximum distribution for each region 3Figure 2: Max pooling for random variables is performed by taking a single sample from each of the input distributions. The output random variable for each pooling region is the random variable that is associated with the maximum sample. For the PBNet-S, We use the weight transfer method introduced by BID21 in which the parameters of the weight distribution for each layer are initialized such that the expected value of the random weights equals the full precision weight divided by the standard deviation of the weights in the given layer. Since not all rescaled weights lay in the [−1, 1] range, all binary weight parameters are clipped between [−0.9, 0.9]. This transfer method transfers the structure present in the filters of the full precision network and ensures that a significant part of the parameter distributions is initialized with low variance. In our training procedure, a stochastic neural network is trained. However, at test time (or on hardware) we want to leverage all the advantages of a full binary Neural Network. Therefore, we obtain a deterministic binary Neural Network from the parameter distribution q θ (B) at test time. We consider three approaches for obtaining a deterministic network: a deterministic network based on the mode of q θ (B) called PBNET-MAP, an ensemble of binary Neural Networks sampled from q θ (B) named PBNET-x, and a ternary Neural Network (PBNET-TERNARY), in which a single parameter W i may be set to zero based on q θ, i.e.: DISPLAYFORM0 The ternary network can also be viewed as a sparse PBNet, however, sparse memory look-ups may slow down inference. Note that, even when using multiple binary neural networks in an ensemble, the ensemble is still more efficient in terms of computation and memory when compared to a full precision alternative. Moreover, it allows for anytime ensemble predictions for improved performance and uncertainty estimates by sampling from the weight distribution. Since the trained weight distribution is not fully deterministic, the sampling of individual weight instantiations will in a shift of the batch statistics. As a consequence, the learned batch norm statistics no longer closely match the true statistics. This is alleviated by re-estimating the batch norm statistics based on (a subset of) the training set after weight sampling using a moving mean and variance estimator. We observed competitive using as little as 20 batches from the training set. Binary and low precision neural networks have received significant interest in recent years. Most similar to our work, in terms of the final neural network, is the work on Binarized Neural Networks by BID7. in this work a real-valued shadow weight is used and binary weights are obtained by binarizing the shadow weights. Similarly the pre-activations are binarized using the same binarization function. In order to back-propagate through the binarization operation the straightthrough estimator BID6 ) is used. Several extensions to Binarized Neural Networks have been proposed which -more or less -qualify as binary neural networks: XNOR-net BID20 in which the real-valued parameter tensor and activation tensor is approximated by a binary tensor and a scaling factor per channel. ABC-nets take this approach one step further and approximate the weight tensor by a linear combination of binary tensors. Both of these approaches perform the linear operations in the forward pass using binary weights and/or binary activations, followed by a scaling or linear combination of the pre-activations. , similar methods to BID7 are used to binarize a wide resnet to obtain on ImageNet very close to the full precision performance. Another method for training binary neural networks is Expectation Backpropagation BID22 in which the central limit theorem and online expectation propagation is used to find an approximate posterior. This method is similar in spirit to ours, but the training method is completely different. Most related to our work is the work by BID21 which use the local reparametrization trick to train a Neural Network with binary weights and the work by BID1 which also discuss a binary Neural Network in which the activation distribution are propagated through the network. Moreover, in the CLT was used to approximate dropout noise during training in order to speed up training, however, there is no aim to learn binary (or discrete) weights or use binary activations in this work. We evaluate the PBNet on the MNIST and CIFAR-10 benchmarks and compare the to Binarized Neural Networks BID7, since the architectures of the deterministic networks obtained by training the PBNet are equivalent. The PBNets are trained using either a cross-entropy (CE) loss or a binary cross entropy for each class (BCE). For the CE loss there is no binarization step in the final layer, instead the mean of the Gaussian approximation is used as the input to a softmax layer. For BCE, there is a binarization step, and we treat the probability of the ith output being +1 as the probability of the input belonging to the ith class. Specifically, for an output vector p ∈ C for C classes and the true class y, the BCE loss for a single sample is defined as DISPLAYFORM0 The weights for the PBNet-S networks are initialized using the transfer method described in Section 2.3 and the PBNets are initialized using a uniform initialization scheme. All models are optimized using Adam and a validation loss plateau learning rate decay scheme. We keep the temperature for the binary concrete distribution static at 1.0 during training. For all settings, we optimize model parameters until convergence, after which the best model is selected based on a validation set. Our code is implemented using PyTorch BID19.For Binarized Neural Networks we use the training procedure described by BID7, i.e., a squared hinge loss and layer specific learning rates that are determined based on the Glorot initialization method BID4.Experimental details specific to datasets are given in Appendix C and the are presented in TAB0. We report both test set accuracy obtained after binarizing the network as well as the the test set accuracy obtained by the stochastic network during training (i.e., by propagating activation distributions). As presented in TAB0 the accuracy improves when using an ensemble. Moreover, the predictions of the ensemble members can be used to obtain an estimate of the certainty of the ensemble as a whole. BID7, PBNet, and a full precission network (FPNet). PBNet-map refers to a deterministic PBNet using the map estimate, PBNet-Ternary is a ternary deterministic network obtained from q θ, and PBNet-X refers to an ensemble of X networks, each sampled from the same weight distribution. For the ensemble both mean and standard deviation are presented. The propagate column contains obtained using the stochastic network whereas in the binarized column are obtained using a deterministic binary Neural Network. To evaluate this, we plot an error-coverage curve BID3 in FIG2. This curve is obtained by sorting the samples according to a statistic and computing the error percentage in the top x% of the samples -according to the statistic. For the Binarized Neural Network and PBNet-MAP the highest softmax score is used, whereas for the ensembles the variance in the prediction of the top class is used. The figure suggests that the ensemble variance is a better estimator of network certainty, and moreover, the estimation improves as the ensemble sizes increases. As discussed in Section 2.4, after sampling the parameters of a deterministic network the batch statistics used by Batch Normalization must be re-estimated. FIG2 shows the obtained using a various number of batches from the training set to re-estimate the statistics. This shows that even a small number of samples is sufficient to estimate the statistics. We perform an ablation study on both the use of (stochastic) Batch Normalization and the use of weight transfer for the PBNet-S on CIFAR-10. For Batch Normalization, we removed all batch normalization layers from the PBNet-S and retrained the model on CIFAR-10. This ed in a test set accuracy of 79.21%. For the weight initialization experiment, the PBNet-S weights are initialized using a uniform initialization scheme and is trained on CIFAR-10, ing in a test set accuracy of 83.61%. Moreover, the accuracy on the validation set during training is presented in FIG2. Note that these numbers are obtained without sampling a binarized network from the weight distribution, i.e., local reparametrization and binary activation samples are used. The PBNet-S that uses both weight transfer and stochastic Batch Normalization in a significant performance improvement, indicating that both stochastic Batch Normalization and weight transfer are necessary components for the PBNet-S. The of our experiments show that, following our training procedure, sampling of the binary activations is a necessary component. Although the stochastic PBNet generalizes well to unseen data, there is a significant drop in test accuracy when a binary Neural Network is obtained from the stochastic PBNet. In contrast, this performance drop is not observed for PBNet-S. A potential explanation of this phenomenon is that by sampling the binary activation during training, the network is forced to become more robust to the inherent binarization noise that is present at test time of the binarized Neural Network. If this is the case, then sampling the binary activation can be thought of as a regularization strategy that prepares the network for a more noisy binary setting. However, other regularization strategies may also exist. We have presented a stochastic method for training Binary Neural Networks. The method is evaluated on multiple standardized benchmarks and reached competitive . The PBNet has various advantageous properties as a of the training method. The weight distribution allows one to generate ensembles online which in improved accuracy and better uncertainty estimations. Moreover, the Bayesian formulation of the PBNet allows for further pruning of the network, which we leave as future work. A BINARY DISTRIBUTION For convenience, we have introduced the Binary distribution in this paper. In this appendix we list some of the properties used in the paper, which all follow direcly from the properties of the Bernoulli distribution. The Binary distribution is a reparametrization of the Bernoulli distribution such that: DISPLAYFORM0 This gives the following probability mass function: DISPLAYFORM1 where a ∈ {−1, +1} and θ ∈ [−1, 1]. From this, the mean and variance are easily computed: DISPLAYFORM2 Finally, let b ∼ Binary(φ), then ab ∼ Binary(θφ). During training the PBNet is trained using stochastic Batch Normalization. At test time, the parameters learned using stochastic Batch Normalization can be transferred to a conventional Batch Normalization implementation. Alternatively, Batch Normalization can be reduced to an (integer) addition and multiplication by ±1 after applying the sign activation function. Given a pre-activation a, the application of Batch Normalization followed by a sign binarization function can be rewritten as: DISPLAYFORM0 DISPLAYFORM1 when a ∈ Z, which is the case for all but the first layer Note that we have used sign = b det = +1 here, as we have used everywhere in order to use sign as a binarization function. The MNIST dataset consists of of 60K training and 10K test 28×28 grayscale handwritten digit images, divided over 10 classes. The images are pre-processed by subtracting the global pixel mean and dividing by the global pixel standard deviation. No other form of pre-processing or data augmentation is used. For MNIST, we use the following architecture: DISPLAYFORM0 where XC3 denotes a binary convolutional layer using 3 × 3 filters and X output channels, Y FC denotes a fully connected layer with Y output neurons, SM10 denotes a softmax layer with 10 outputs, and MP2 denotes 2 × 2 (stochastic) max pooling with stride 2. Note that if a convolutional layer is followed by a max pooling layer, the binarization is only performed after max pooling. All layers are followed by (stochastic) batch normalization and binarization of the activations. We use a batchsize of 128 and an initial learning rate of 10 −2 Results are reported in TAB0. The CIFAR-10 (BID12) dataset consists of 50K training and 10K test 32 × 32 RGB images divided over 10 classes. The last 5,000 images from the training set are used as validation set. Tthe images are only pre-processed by subtracting the channel-wise mean and dividing by the standard deviation. We use the following architecture for our CIFAR-10 experiment (following BID21): DISPLAYFORM0 where we use the same notation as in the previous section. The Binarized Neural Network baseline uses the same architecture, except for one extra 1024 neuron fully connected layer. During training, the training set is augmented using random 0px to 4px translations and random horizontal fl Results are reported in TAB0.

We introduce a stochastic training method for training Binary Neural Network with both binary weights and activations.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: The goal of generative models is to model the underlying data distribution of a sample based dataset. Our intuition is that an accurate model should in principle also include the sample based dataset as part of its induced probability distribution. To investigate this, we look at fully trained generative models using the Generative Adversarial Networks (GAN) framework and analyze the ing generator on its ability to memorize the dataset. Further, we show that the size of the initial latent space is paramount to allow for an accurate reconstruction of the training data. This gives us a link to compression theory, where Autoencoders (AE) are used to lower bound the reconstruction capabilities of our generative model. Here, we observe similar to the perception-distortion tradeoff . Given a small latent space, the AE produces low quality and the GAN produces high quality outputs from a perceptual viewpoint. In contrast, the distortion error is smaller for the AE. By increasing the dimensionality of the latent space the distortion decreases for both models, but the perceptual quality only increases for the AE. Generative Adversarial Networks (GANs) were introduced by for the purpose of generative modelling. Since then this framework has been successfully applied to works in style transfer by , superresolution by and semi-supervised learning by , but what GANs actually learn is still poorly understood as has been noted by. Recently, GANs have been used to solve inverse problems, where it was tried to use the generated manifold to solve an auxiliary task like image completion , MRI reconstruction or anomaly detection . For those applications, it is necessary to know if the generator NN actually describes the distribution well. Related works have shown that faithfully reconstructing the images from a generator network is not trivial . The original convergence proof by assumes that the generator and discriminator Neural Networks (NNs) have infinite capacity and they showed that the discriminator network models the Jensen-Shannon divergence between the probability distribution induced by the generator and the real data distribution. Others have adapted this paradigm and devised loss functions which have been shown to converge to other divergences or distances on the underlying probability distributions;; ). Regularization techniques like Gradient Penalty and Spectral Norm did improve the stability of the training process ) but it is still unclear how well this NNs actually approximate such distances even for trivial problems . Additionally, it is not at all clear how the generated distribution or the actual target distribution look like. used the birthday paradox to empirically gauge the size of the support of the generated distribution. GANs are used to transform a well understood low dimensional distribution (in practice either gaussian or uniform) to a high dimensional unknown distribution by playing a min-max game between two NNs. This paper is based around the intuition, that an estimated probability distribution from a dataset X has high precision if a high percentage of the actual data samples are included in the estimated distribution. To have a sense of what an adequate capacity for a generator network is, we use AE to reconstruct the dataset first. This work relies on the assumption that it is easier to reconstruct the data samples alone, than to reconstruct the entire manifold and Section 5 shows empirical evidence for this. Based on our intuition the manifold consists of the data samples and imposes additional structure between the data samples. In contrast by just reproducing the data samples, no such additional restrictions are given, making the problem strictly simpler. AEs can be trained rapidly and have been researched in detail for a long time . In contrast, trying to do a hyperparameter search on the GAN networks themselves gives rise to all kinds of problems, like instabilities in the training process, random failures and dependence on random seeds for their performance ). Hence, our contributions are as follows: • An investigation of the impact of the dimensionality of the latent space on the generated manifold. We showcase that the fit of the data depends heavily on the latent space. We also show similar thereby to the perception-distortion tradeoff , where with a small dimension for the latent space, the GAN optimizes for perception and the AE optimizes for distortion. • Relating the GAN problem to a compression task and furthermore using compression tools via deep learning to produce a lower bound for a dataset dependent suitable dimensionality of the latent space. • An investigation of the generated manifold and the limitations thereof to produce shifted or noisy images and how this relates to the size of the latent space and overfitting of the generative model. The remainder of this paper is organized as follows. Section 2 shows the related work. Then in Section 3 we revisit the theory behind pseudo inverting NNs and we explain our methodology in Section 4. In Section 5 the are shown. Section 6 draws based on those . GANs have been invented by for the task of image synthesis and semisupervised learning. The initial formulation has been shown to suffer from mode collapse and vanishing gradient problems and multiple other loss functions have been proposed (; ; . We choose the Wasserstein GAN, due to it using the L2-distance as its transport cost function to judge the quality of the generated manifold, which is similar to the Mean Squared Error (MSE) used in the classical AE . To satisfy the Lipschitz constraint we use , due to its good performance in review works; ). For evaluation purposes we need a quality metrics for GAN and the most common ones rely on the usage of classification models, which are trained on ImageNet . Two commonly used metrics are the Inception Score and the Fréchet Inception Distance (FID) . To combat the bias of having trained on proposed to train a new classifier to evaluate the GAN model, but this is time consuming and prone to errors by not having a standardized metric. We opted for the FID score, due to its prevalence in related works; ). AEs have been studied extensively in the past and initially have been proposed by. They have found usage in compression literature (Cheng et al. (2018a; b); ). Their first usage in a GAN framework has been shown in the Energy based GAN by. Since then they have been used as preprocessing to reduce the dimensionality of the data , as regularizers or as generative models themselves . The inversion of generator networks has been initially used by and there have been multiple works on how to improve their method by adding heuristics like stochastic clipping on top of that. We opted against stochastic clipping, because we already project back to the likely set of candidates. Stochastic clipping would add even more randomness to the . Generalization of GANs through the detection of overfitting has been proposed recently by. Therein the authors are mostly concerned (f) CelebA: dim(z) = 5000 Figure 1: Histogram of the values in the weight matrix and the convolutional layers of a trained WGAN-GP with a latent space specific dimensionality for CIFAR-10 and CelebA datasets. Notice, that the weights share the same characteristics as if they were drawn from a normal distribution. with detecting memorisation of the training set in the manifold and avoid said memorisation to generalize. In contrast we make the assumption that an estimated probability distribution based on samples should include those samples and overfitting is detected based on a hold out set as well as the ability to reconstruct that hold out set as well as the training set. Additionally, we show that the ability of the GAN to memorize depends mostly on the dimensionality of the latent space and the capacity of the generator network and can be estimated using compression techniques. In this work, we want to show the fit of the distribution of trained GAN newtworks using inversions of such networks. This motivates the question of " When are NNs invertible?". Due to the nonconvex and non-bijective nature of NNs using ReLU activations invertibility is not given in the general case. The function induced by the NN is in general neither injective nor surjective and it is trivial to construct such cases. However, if the weights are gaussian distributed, proofs exist, which guarantee that the NN is invertible with a high probability. For this, we revisit the inversion theory of NNs largely pioneered by. They empirically showed that the weights of trained NNs actually exhibit random like properties, enabling the usage of Random Matrix Theory (RMT) to investigate such networks. Theorem 1 in proves that we can invert 2-layer NNs with high probability. In their work they empirically showed that this also holds for multi layer networks. In Figure 1 we show that the same holds true for trained GAN networks. Note, that this property holds across datasets and latent spaces used to train those networks. Glorot initialization was used to initialize these networks. Therefore, the standard deviation used for the initialization process differs for each layer. However, as is demonstrated in Fig. 1, all layers converged to a similar gaussian distribution. Another indicator, that the optimization process is likely to succeed is given by a trick proposed by for classification networks. It allows us to visualize the loss surface of the optimization problem for one sample at a time. For a particular data point z *, we chose two orthogonal vectors α, β and plot the following 2D function: We chose α = z * − z 0 to model the optimization path towards the original data point. For this experiment we do not need to run an optimization, because the point is to show that it is possible There is a clear optimization path from z 0 = 0 to a point z * even in this simplified plot, empirically validating the claims of , that an accurate reconstruction of the generator manifold is possible works using first order methods. to recover points from the manifold. To ensure that β is an orthogonal vector, we simply chose this vector by setting β = 0 and then. Then α, β are orthogonal and we can use them as a basis for the plots shown in Fig. 1. For visualization purposes, we scaled β to have the same norm as α; e.g. ||β|| 2 = ||α|| 2. We vary a, b ∈ [−2, 2]. In our experiments this always ed in benign structures as is shown in Fig. 2. If it is trivial to recover points from the manifold and impossible to recover the training images, than this implies that the manifold does not include the training images. Currently, designing new GAN networks works by running a huge hyperparameter search on a plausible set of combinations; ). noted that the progress in GANs has actually been quite minimal and vastly overstated, and depends on the hyperparameters more than on the improvements in the methodology, while did notice that non-saturating loss in combiniation with spectral norm produced consistently good , but never the best ones. The goal of the GAN framework is to encapsulate the high dimensional image distribution in a lower dimensional noise distribution, whereas a generator network maps the lower dimensional distribution to the former high dimensional image distribution . This implies that we compress the larger distribution onto the simple lower dimensional distribution and use the generator network as an approximation for the decompression. We rely on the assumption, that less capacity is needed to reconstruct the training set, than to reconstruct the entire manifold. Therefore, to gain some insight into the necessary capacity of the generator network, the usage of an AE is proposed to gauge the number of parameters and the size of the latent space to enable suitable approximation of the training set. In theory, a latent dimension of 1 would be sufficient to reconstruct the entire dataset, by mapping a curve to the datapoints. A simple smooth function that does this for a dataset {x 1, .., x n} = X ∈ R n×d and {p 1, ..., p n} = P ∈ R n×1 looks like this: This function will output the corresponding image x i for every p i. In practice, this behavior is not observed. AEs with small latent spaces tend to output blurry approximations of the training images as is shown in Section 5. Once a suitable AE for a dataset is found, the decoder part of the AE is used as the generative model of choice. The AE is trained using the L2-loss, which also is the transport cost function used in WGANs ) to gauge the quality of the generated manifold. Therefore, the objective function of the AE using a decoder d and an encoder e is defined as follows: ||d(e(x)) − x|| 2 We do not opt for any regularization, because we want to fit the data as closely as possible to determine if the generator capacity is adequate for reconstructing the dataset. In this work we show empirically in Sec. 5 that if the AE already fails to produce crisp , then also the GAN algorithm fails at having the dataset included in the generated manifold. One thing to note is that, while the encoder part is used as our discriminator, it has a completely different objectives and it is unclear, if a better encoder is also a better discriminator. The key part of this work is to analyse the ability of the GAN to memorise the training dataset, following the assumption, that a probablity distribution based on a sample dataset has to include that dataset. To our best knowledge there is no analytical way to invert generative NNs. Therefore this ill posed problem is stated as an inverse optimzation problem for a function g: This is solved using a standard first order gradient method. However, as is shown in Section 5 the ing latent vectors are unlikely to stem from the targeted noise distribution, if there are no constraints added. Therefore, the search space for z is restricted to the space of plausible latent vectors. Plausibility in this context means that the probability of the latent vector originating from this space is larger than 99.7% (3-sigma-rule). In the standard multivariate gaussian distribution the norm of the vector z corresponds inversely to the likelihood of z ∈ N (0, I). Therefore we restrict the norm to be ||z|| ≤ E[||z||] + 3 V AR [||z||]. For the multivariate gaussian distribution this is easy to calculate and the final objective function is given as follows: This is solved using a projected ADAM optimizer . Our experiments are done on medium sized datasets (CIFAR-10 and CelebA (64 × 64 resolution)) and the proof of concept on MNIST is relegated to the supplementary material. The NN adapted for this task is the standard DCGAN from. Convolutional layers and resize operations are used instead of transposed convolutions, to avoid checkerboard artifacts . The full hyperparameter setting is shown in the supplementary material. The visual quality measured by the FID of the ing samples is on par with the survey work by. This work is not aimed at improving the visual quality of the samples generated by GAN algorithms. For CIFAR-10 the test set is used as our validation set and for CelebA we randomly split off 20% of the dataset to be used as validation set. In principle, getting from an AE to a GAN is just a rearrangement of the NNs. For the GAN training, the default hyperparameter are used as in. The AE NNs are optimized using the ADAM optimizer with default hyperparameters for all the datasets, always on a single GPU. A comparison of the ing reconstruction errors is shown in Fig. 3. Therein the different AE and GAN models only differ in their latent space sizes. Increasing the latent space dimension also increases the number of parameters. However, for the sake of simplicity this was neglected. The AE network acts as a lower bound for the GAN algorithm, therefore validating our intuition that the AE complexity lower-bounds the GAN complexity. The visual quality of the generated images is evaluated using the FID versus the reconstructed images by an AE for the same latent space sizes. The experimental are shown in Fig. 4 for the CIFAR and CelebA datasets and demonstrate, that the visual quality of the GAN generated images barely changes with different latent dimensionality. In contrast, the AE reconstructions gain quality and sharpness as the latent space increases. This reflects the output of those models shown in Fig. 5, where the ing images are clearly an approximation of the actual training images. On the other hand, the GAN reconstructions have similar visual quality independent of the actual latent space. However, the ing images are only remotely similar to the actual image for insufficiently large latent spaces. This phenomenon has been observed in other works, where a similar looking person to an actual training image is found by the inversion procedure . However, we show in Fig. 5, that this phenomenon depends mostly on the size of the latent space. curve for different latent spaces (lower is better). Increasing or decreasing the latent space does not in a noticeable difference in the quality metric for the WGAN-GP, but it does for the AE. While it is possible to reconstruct the dataset accurately using a large enough latent space, in this section we look at what else those models can reconstruct. In these experiments, our baseline model The first columns show the original images. As we increase the latent space of the autoencoder the image quality increasing. The GAN reconstructions are of high perceptual quality even for low dimensional spaces, but correspond to other people as has been observed by. is a WGAN-GP trained on CelebA. The first experiment shown in Fig. 6 uses translated training images. This is especially challenging for this dataset, due to the pre-aligned face images used in the CelebA dataset. As is shown in Fig. 6 the error goes up and the visual quality of the reconstructions deteriorates even for small pixel shifts as long as the latent space is small enough. Figure 6: Impact of translation on the reconstruction ability of GAN networks. Notice, that by increasing latent space dimension, the generator network is able to reconstruct translations with a similar precision as the original images. The second experiment is on reconstructing the CIFAR-10 dataset using the CelebA trained model. To provide the same resolution, the CIFAR-10 images were upscaled to 64 × 64 using bilinear interpolation. If the latent space is large enough, we can reconstruct the images well, as shown in Fig. 7. The reconstruction of the CelebA images is always worse as measured by the MSE (Fig. 7). The final experiment is on reconstructing noise images. For this a dataset of uniform noise images with size 64 × 64 is used. The are shown in Fig. 8. As is suggested by the , it is in fact not possible to reconstruct everything. One thing to note though is that with lower latent spaces some face structure is visible in the reconstructed images, showing that those models have a strong prior on the face structure. We noticed and the corresponding experiment is shown in the supplementary material, that the norm of the latent vector is smaller than of random vectors. Our hypothesis is that those images are the base face features, which are refined with larger latent vectors, but this guess needs further validation work. In this work, we show that by reducing the problem to a compression task, we can give a lower bound on the required capacity and latent space dimensionality of the generator network for the distribution estimation task. Relating these two different algorithms to each other, the literature surrounding AEs for invertability and dimensionality reduction, as well as the corresponding theoretical advancements are used. While in this work the encoder and the discriminator NNs use the same architecture, we have not discovered any relation between them. Still, the same architecture works well empirically for both task. Using our framework we show various properties of generator networks. The perceptual image quality appears to be independent of the actual size of the latent space, which is in contrast to AE, where the visual quality improves if the dimensionality of the latent space is increased. However, the ability to reconstruct the training set correctly does depend on the initial latent space. Also the ability of the generator to reconstruct deviations from the original dataset, like a validation set or shifted images depends just as much on the initial latent space. However, the same cannot be said for reconstructing arbitrary noise images. Here the reconstruction ability is independent of the initial latent space unless it is chosen very large, suggesting that the generator has learned realistic natural image features. Here for smaller latent spaces we can still observe face like features. Our hypothesis is that the implicit bias induced by the generator architecture lends itself to generating natural images and GANs are skilled at that by learning primitives which can be combined to construct arbitrary images. In future works we want to use our setup to search towards better and more reliable generators for images. For our experiments a standard CNN was used as is described in Table 1. All the convolutions use 5 × 5 kernels. For the CIFAR-10 experiments we removed one of the up and downlayers and started with 1024 feature maps. We used the ADAM optimizer with learning rate 1E − 4 and β 1 = 0. and β 2 = 0.9 similar to the original WGAN-GP as described by. We trained for 80.000 generator iterations and 5 discriminator iterations per generator update for all of our networks. We also trained the AE for 80.000 iterations with ADAM using the default hyperparameters. The inversion of the generator networks was done using 10.000 ADAM steps using default hyperparameters. We found that the changes afterwards were negligible. The initial experiments on MNIST validating the legitimacy of the approach are shown in Fig. 9. Instead of using a visual quality metric to determine if the ing images are similar enough, we used the reconstructed training images of the GAN for training a LeNet network to determine if the images are descriptive. The manifold of MNIST has been faithfully learned before by. In contrast to their work, we show that this only works given a large enough latent space and that the reconstruction error of the AE and the WGAN-GP converge for larger latent spaces. In this section the main question is if the found reconstructions likely to appear in the generative model. For this reason the inverse optimization procedure is run without the projection onto the feasible set for dim(z) = 1000 on CIFAR-10. The ing latent vector norm of the reconstructions is shown in Fig 10. The corresponding set of vectors, which is likely to appear in the training is shown in blue, while the training set reconstructions are shown in red and the test set reconstructions are shown in green. As is apparent therein, the overlap is virtually non-existant and this is the reason why we used a projected gradient method. We also investigated the difference between the ability of GAN to reconstruct images of CIFAR-10 while trained on CelebA compared to AE. The are shown in Fig. 11 and as is already shown in Fig. 7 the GAN algorithm is better at reconstructing CIFAR-10 images than CelebA images. However, this is not the case for the AE. And as a final experiment we show visual on CIFAR-10 images produced by an inverted GAN and by an AE in Fig. 12. As in the main paper we can the same behavior again. The AE produces initially blurry and then sharp ones and the GAN produces sharp of unrelated images and then produces images which are closer to the actual images. The first column are the actual images. As we increase the latent space of the autoencoder we can see the image quality increasing. The same cannot be said about the GAN reconstructions. Those are of high quality even for low dimensional space, but correspond to other people.

Represent the short summary for finding the corresponding scientific article (long)

We analyze the impact of the latent space of fully trained generators by pseudo inverting them.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We address the problem of open-set authorship verification, a classification task that consists of attributing texts of unknown authorship to a given author when the unknown documents in the test set are excluded from the training set. We present an end-to-end model-building process that is universally applicable to a wide variety of corpora with little to no modification or fine-tuning. It relies on transfer learning of a deep language model and uses a generative adversarial network and a number of text augmentation techniques to improve the model's generalization ability. The language model encodes documents of known and unknown authorship into a domain-invariant space, aligning document pairs as input to the classifier, while keeping them separate. The ing embeddings are used to train to an ensemble of recurrent and quasi-recurrent neural networks. The entire pipeline is bidirectional; forward and backward pass are averaged. We perform experiments on four traditional authorship verification datasets, a collection of machine learning papers mined from the web, and a large Amazon-Reviews dataset. Experimental surpass baseline and current state-of-the-art techniques, validating the proposed approach. We investigate the applicability of transfer learning techniques to Authorship Verification (AV) problems, and propose a a method that uses some of the most recent advances in deep learning to achieve state of the art on a variety of datasets. AV seeks to determine whether two or more text documents have been written by the same author. Some applications of AV include plagiarism analysis, sock-puppet detection, blackmailing, and email spoofing prevention BID7. Traditionally, studies on AV consider a closed and limited set of authors, and a closed set of documents written by such authors. During the training step, some of these documents (sometimes as long as a novel) are used. The goal can be formulated as to successfully identify whether the authors of a pair of documents are identical BID14 BID19 BID11. This type of AV tasks assumes access to the writing samples of all possible authors during the training step, which is not realistic. Recently, the AV problem has changed to reflect realistic -and more challenging-scenarios. The goal is no longer to individually learn the writing style of the authors (like in traditional AV methods), but to learn what differentiates two different authors within a corpus. This task involves predicting authorship of documents that may not have been previously encountered within the training set; in fact, the presence of the authors in the training data is not guaranteed either. That is, the test set may contain out of training sample data; given a set of authors of unknown papers contained within the training data, A unknown train, and a set of authors of unknown papers in the test data, A unknown test, it is neither unreasonable nor unexpected to find that A unknown train ∩A unknown test = ∅. Some other challenges arise in modern AV tasks, making authorship verification of a given pair of documents hard to infer. One is the lack of training data, which can manifest itself in any one or more of the following: the training set may be small, samples of available writings may be limited, or the length of the given documents may be insufficient. Another is the test and train documents belonging to different genre and/or topics, both within their respective sets as well as between the train and the test set -implying they were likely drawn from different distributions. The challenge is to ensure robustness in a multitude of possible scenarios. Regardless of the AV problem specifics, generally we assume a training dataset made of sets of triples: DISPLAYFORM0 with x i X known, x j X unknown a realization from random variables X known and X unknown, and the label y i,j Y is drawn from a random variable Y, producing a total of P sets of realizations, each potentially by a different author, thus forming up to P source domains, because it can be argued that a collection of literary works by one author forms a latent domain of it's own. The goal is to learn a prediction function f: X → Y that can generalize well and make accurate predictions regarding documents written by authors both inside and outside of the training set, even if those documents were not seen in training. Less formally, in AV the task is composed of multiple sub-problems: for each given sub-set of texts, we are provided one or more documents that need to be verified and one or more that are known to be of identical authorship. We approach the AV problem by designing a straightforward deep document classification model that relies on transfer learning a deep language model, ensembles, an adversary, differential learning rates, and data augmentation. In order to ensure the design's versatility and robustness, we perform authorship verification on a collection of datasets that have little in common in terms of size, distribution, origins, and manner they were designed. For evaluation, we consider standard AV corpora with minimal amount of training data, PAN-2013 BID12, PAN-2014E and PAN-2014N BID27, PAN-2015 BID28, a collection of scientific papers mined from the web BID2, and Amazon Reviews dataset BID8. The proposed approach performs well in all scenarios with no specific modifications and minimal fine-tuning, defeating all baselines, PAN competition winners, as well as the recent Transformation Encoder and PRNN models that were recently shown to perform well on AV tasks. BID8. Our method consists of three major components: augmentation, transfer learning, and the training/testing process itself. At a high level, we augment the data, fine-tune a deep LSTM-based language model (LM) known as ULMFit BID9 on the augmented training set, train an ensemble of RNN and QRNN classifiers with the encoding produced by the LM forward and backward, and evaluate the test data while performing test-time data augmentation. We utilize various data augmentation techniques in order to improve model generalization TAB0. They broadly fall into two categories, document manipulation and adversarial noise injection with the LM. In addition, most of these techniques can be applied to the test set documents during evaluation; however, some do more harm than good when used in such manner. Noise injection is performed by a 5-layer LSTM model that was pre-trained on Wikipedia and fine-tuned on our data. In our setup, it acts as a generator with a 3-layer RNN classifier working as a critic. Adversarial loss function is a weighted average of the two losses DISPLAYFORM0 where g is the LM, f is an RNN and h is the linear classifier trained on RNN's average then max pooled and flattened 2 top layers. We use a weighted average because the nature of loss functions is very different. To improve quality of augmentation, we devised the following approach (Algorithm 1). Given a training set consisting of a number of problems, with each problem containing one or more documents known to be written by the same author and a single document of unknown authorship, we cycle through each problem in the training set. If for a given problem the ground truth answer is positive, we train on all documents and try injecting noise. If the critic can tell the fake, it means our new document is most likely too different from actual ones by this author to be of any use; we then try training some more, and inject shorter sentences and less of them. The process continues until critic is fooled or generator diverges -an unlikely event because critic is not hard to fool. We hypothesize that documents form latent domains of their own based on various linguistic characteristics, making it beneficial to transform the pairs of documents into a domain-invariant space. Documents forming latent domains means that authorship verification is a separate but similar task for each domain. We cannot exploit the similarity between tasks directly because the data distributions are different, and not accounting for that while building a model would violate basic principles of machine learning BID21. Domain Adaptation (DA), a subset of Transfer learning, addresses such Algorithm 1: Noise injection algorithm using language model with an adversary problems by establishing knowledge transfer from a labeled source domain to an unlabeled (or partially labeled) target domain, by exploring domain-invariant features or invariants which transfer across domains BID22 BID21 BID5 BID29, or by embedding the data into domain-invariant subspace. Another issue that we must address comes from the nature of the data. As the documents come in pairs, they are not readily suitable for standard classifiers. A naive approach of concatenation produces poor , and various distance function schema suitable for most linear models are not very suitable for RNNs. To address these problems we utilize a deep language model that produces an encoder capable of producing an embedding representing a pair of documents. It also alleviates the need for data by being pre-trained on a large set of Wikipedia articles BID9. The domain discrepancy issue is in part mitigated too, because the ing embedding subspace features are more invariant. In a gist, our model (FIG0) is a bi-directional pipeline of recurrent neural networks. It is built on top of a pre-trained 5-layer LSTM model and takes it's last 3 (2 intermediate hidden ones and the final embedding output) layers as inputs by pooling them together. We use an ensemble of sequence classifiers, one based on an RNN and the other using a QRNN BID1, a recent addition to the RNN family that combines some properties of recurrent and convolutional networks. Both are 3-layer models with the last 2 layers average then max pooled and passed through a ReLU non-linearity and then to logit units. We output probabilities rather than labels. The predictions made by RNN and QRNN are averaged. In taking advantage of improved generalization through making the model bi-directional, we faced two challenges. First, the pre-trained LSTM model we used is uni-directional. Second, QRNN design used in this paper does not support bi-directional training, either. We circumvented the issue by tokenizing and numericalizing the text data and first training in regular fashion on a normal pre-trained Wikipedia model, then loading the numericalized tokens backwards and using a model that was trained on Wikipedia backwards, as well. At test time, we reversed each document and gave the normal ones to the forward model and backward ones the backward version, then averaged the of two runs, effectively reaping the benefits of using a bi-directional RNN without actually doing so. We call our design 2WD-UAV in reference to ensembling of two versions of RNN for authorship verification and because of it's ULMFit heritage. The architecture is implemented in PyTorch with elements of fast.ai library BID10. PAN We use all available authorship identification datasets released by PAN 1 (TAB2 . Each PAN dataset consists of a training and test corpus and each corpus has a various number of distinct problems. Each problem is composed of one to five writings by a single person (implicitly disjoint For PAN2014 and PAN2015 and explicitly disjoint for PAN2013), and one piece of writing of unknown authorship. In the other words, we are given up to five pairs of documents where one document's authorship is known and the other one's is not. Two documents of a pair might be from significantly different genres Table 3. Similarity functions. x, y: document feature vectors, n: # of features in x and y Chi2 kernel exp(−γ i [ DISPLAYFORM0]) Cosine similarity xy T /(||x||||y||) DISPLAYFORM1 and topics. The length of a document changes from a few hundred to a few thousand words. PAN2014 includes two datasets: Essays and Novels. The paired documents in PAN datasets are used for our experiments. For a problem P = (S, T), S (source) is the first document and T (target) is the second document of a PAN problem BID8. Amazon Reviews We use a dataset made by selecting 300 authors with at least 40 reviews to make the positive and negative candidate sets. Then, for each author, the positive candidate set is all possible and unique combinations of the author's reviews. A positive class consists of 4500 review pairs from this positive candidate set at random. The negative candidate set is made of all unique and possible combinations of review pairs having different authors. For this dataset, the negative class of equal size with the positive class was created by random selection from the negative candidate set. In prior work, 5-fold cross validation was used for this data. We do the same in order for our to be comparable. BID8. MLPA* This schema was created using MPLA-400 dataset that contains 20 articles by each of the top-20 authors by citation in Machine Learning BID2. In MLPA*, only publications from MPLA-400 that are written by a single author and have no co-authors are used BID8. To keep the distribution of authors and classes balanced, MPLA* contains an equal number of single-authorship articles from all existing 20 authors. The positive class consists of the pairs which are made up of all possible combinations of same-authorship articles (20 × 9 2 = 720). The negative class includes the pairs that are randomly selected from the set of all unique combinations of articles of different authorship and is of the same size as the positive class. Like Amazon Reviews, MLPA* dataset authors recommend using 5-fold cross validation BID8. We compare our method with the top methods of PAN AV competition between 2013 and 2015 (TAB2). The of each method for one year of the competition are available and we report them here. Our comparisons are not impacted by different parameter settings and implementation details of these methods as long as we keep the test and training sets the same as theirs. We choose several classifiers widely used in the area with the seven similarity measures to set strong baselines (Table 3). Since each example in our underlying dataset structure comprises two documents, we need to adapt it to the structure of an ordinary classifier input by converting them to one single entity. A simple direct way is to concatenate their feature vectors. However, our experiments show it provides weak mostly equal to the random label assignment. So, we define the summary vector as a single unit representative of each example/problem P = (D S, D T) by utilizing several similarity measures. The summary vector comprises a class of several metrics each measuring one aspect of the closeness of the two documents (D S and D T) of the pair for all underlying feature sets. For any two feature vector documents x, y their summary vector is sum(x, y) = [sim j i (x, y)] where sim j i (x, y) 1≤i≤M,1≤j≤F computes the ith similarity metric of M metrics in Table 3 under jth of F = 7 feature sets (Section 3.2) between x, y. Then, we use a suite of classifiers including SVM, Gaussian Naive Bayes (GNB), K-Nearest Neighbor (KNN), Logistic Regression (LR), Decision Tree (DT) and MultiLayer Perception (MLP) to predict the class label. All baselines are implemented by the scikit-learn library BID23. 2WD-UAV For our model, a number of important parameters are set. Most importantly, to achieve our , we make use of recent work on alternating learning rates, as well as one-cycle learning policy BID25 BID26. The basic approach to training is as follows:-Contract learning rate lr for one cycle -Freeze it and save -Give the learning rate on next layer a very large value -Freeze it and save unfreeze the previous one -Assign a very small value to the next layer -Continue cycling until gradients explode -Return the last saved checkpoint -this is the global minimumWe also use a range of momentum across layers, as well as different learning rates for each. For the optimizer we choose AdamW BID18, an improved version of Adam BID13 with better weight decay regularization. We begin with weight decay of 0.03 and regularize by adjusting it as training progresses. 2. Gaussian distribution is chosen for Naive Bayes. For K-Nearest Neighbor we set K=3. The L-2 regularization is used for Logistic Regression. For document expansion, we set the size of the sliding window to l = 10. On average it expands one document into 30 smaller documents for PAN datasets. All other parameters are selected based on pilot experiments. We report accuracy, the Area Under Receiver Operating Characteristic (ROC) curve BID3 (AUC). The higher AUC and Score indicate more effective classification. We compare our proposed model 2WD-UAV with several relevant baselines. Table 4 evaluates our model with PAN datasets for different years and also the best performing model in the relevant competition years for PAN. Results show that 2WD-UAV consistently outperforms all baselines and all best-reported models in PAN competitions for all years in the Score metric. The Score metric is essentially Accuracy × ROC thereby measuring joint performance gains as both ROC and accuracy are important. 2WD-UAV outperforms in Accuracy for all competitors in PAN14Essay and PAN13 dataset. It is the second best in PAN15 just offset by one decimal point. While it is not performing the best in accuracy for PAN14Novels, it yields competitive performances of accuracy and outperforms all others in ROC metric. 2WD-UAV also outperforms all other models in the ROC metric for PAN15. For PAN14E and PAN 13, it outperforms several baselines and offers stellar performance in ROC metric, just to be second to MLP and CNG respectively. While it is true that the proposed approach is not always the best performing on PAN data in every metric except Score, we believe one reason is due to the inherently smaller data sizes (both total words of data per author to train upon and also the total number of authors to scale up training) that make the approach a little weak. Hence, we further explored larger datasets of Amazon Reviews BID8 and MLPA* BID8 BID2 in Table 5 which shows significant performance gains in accuracy, defeating a variety of baselines. All in all, we do find stable and consistent performance gains with Table 5. Accuracy using 5-fold cross-validation on MLPA* and Amazon Reviews. Domain Adaptation Documents forming latent domains means that authorship verification is a separate but similar task for each domain. We cannot exploit the similarity between tasks directly because the data distributions are different, and not accounting for that while building a model would violate basic principles of machine learning BID21. Domain Adaptation (DA) addresses such problems by establishing knowledge transfer from a labeled source domain to an unlabeled (or partially labeled) target domain, and by exploring domain-invariant features or invariants which transfer across domains BID22 BID21 BID5 BID29. Authorship Verification In vast majority of the AV approaches, the writing style of a questioned author is known to us as we are given some scripts of the author and the task is to determine whether a piece of work is written by the same person. The depth of difference between two sets of documents is measured using the unmasking technique while ignoring the negative examples BID14. This one-class technique achieves high accuracy for 21 considerably large books (ebook above 500K). A simple feed forward three-layer neural network, an auto-encoder, is used for AV considering it a one-class classification problem BID19. They observe the behavior of the neural network for documents by different authors and build a classifier for each author. Their idea originates from one of the first applications of auto-encoder in classification as a novelty detector BID11. AV is also studied for detecting sock-puppets who deliberately change their writing styles to pass the filters and provide opinion Spam. A spy induction method is proposed to leverage the test data in training step under "out-of-training" setting BID7 where a questioned author is from a closed set of candidates while appearing unknown to the verifier. However, in a more realistic case we have no specified writing samples of a questioned author and there is no closed candidate set of authors. Since 2013, a surge of interest arose for this type of AV problem. BID24 investigate whether a document is one of the outliers in a corpus by generalizing the Many-Candidate method by BID15. The best method of PAN 2014 for Essays dataset optimizes a decision tree. Its method is enriched by adopting variety of features and similarity measures BID4. However, for the Novels dataset, the other dataset of that year, the best are achieved by an author verifier using fuzzy C-Means clustering BID20. In an alternative approach, BID16 generate a set of impostor documents and apply iterative feature randomization to compute the similarity distance between pairs of documents. One of the more interesting and powerful approaches investigates the language model of all authors using a shared recurrent layer and builds a classifier for each author BID0. Parallel recurrent neural network and transformation auto-encoder approaches were recently shown to produce excellent far a variety of AV problems BID8. The AV problem is also studied by a non Machine Learning model comprising of a compression algorithm, a dissimilarity method and a threshold. When evaluated on PAN datasets, this approach stands at first ranking position for the two out of four PAN datasets BID6. Recently, Linguistic traits of sock-puppets are deeply studied to verify the authorship of a pair of accounts in online discussion communities BID17. Authorship verification has always been a challenging problem. It can be even more difficult when no writing samples of questioned author/authors is given. In this paper, we explore the possibility of a more general approach to the problem, one that does not rely on having most of the authors within the training set. To this end, we use transfer and adversarial learning learning, data augmentation, ensemble methods, and cutting edge developments in training deep models to produce an architecture that is to the best of our knowledge novel at least to problem setting. Our design exhibits a high degree of robustness and stability when dealing with out-of-sample (previously unseen) authors and lack of training data and delivers state-of-the-art performance.

Represent the short summary for finding the corresponding scientific article (long)!

We propose and end-to-end model-building process that is universally applicable to a wide variety of authorship verification corpora and outperforms state-of-the-art with little to no modification or fine-tuning.

represent the short summary for finding the corresponding scientific article (long)

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.