Question: represent the scientific article to find a short tldr

Question: This paper proposes ASAL, a new pool based active learning method that generates high entropy samples. Instead of directly annotating the synthetic samples, ASAL searches similar samples from the pool and includes them for training. Hence, the quality of new samples is high and annotations are reliable. ASAL is particularly suitable for large data sets because it achieves a better run-time complexity (sub-linear) for sample selection than traditional uncertainty sampling (linear). We present a comprehensive set of experiments on two data sets and show that ASAL outperforms similar methods and clearly exceeds the established baseline (random sampling). In the discussion section we analyze in which situations ASAL performs best and why it is sometimes hard to outperform random sample selection. To the best of our knowledge this is the first adversarial active learning technique that is applied for multiple class problems using deep convolutional classifiers and demonstrates superior performance than random sample selection. The goal of active learning (AL) algorithms is to train a model most efficiently, i.e. achieving the best performance with as few labelled samples as possible. Typical AL algorithms operate in an iterative fashion, where in each AL-cycle a query strategy selects samples that the oracle should annotate. These samples are expected to improve the model most effectively when added to the training set. This procedure continues until a predefined stopping criteria is met. In this paper we will mainly focus on pool based active learning, because a pool of unlabelled samples is often available beforehand or can easily be build. Furthermore, annotating all pool samples serves as an ideal evaluation environment for active learning algorithms. It enables to train a fullysupervised model that establishes a performance upper bound on this data set. Similarly, randomly selecting instead of actively choosing samples establishes a lower bound. Then, the goal of an active learning algorithm is to approximate the performance of the fully supervised model with as few labelled samples as possible, while exceeding the performance of random sampling. Uncertainty sampling is an effective query strategy that identifies samples that are more informative than random ones. The heuristic is, that samples for which the model is most uncertain contain new information and improve the model. However, to identify such samples an exhaustive search over the full pool is required and the uncertainty score needs to be recomputed as soon as the model is updated (each AL cycle). Thus, uncertainty sampling has a linear run-time complexity such that scanning very large unlabelled data sets is impractical even for inexpensive score functions. Our contributions are as follows:• We propose Adversarial Sampling for Active Learning (ASAL) that allows to approximate the performance of uncertainty sampling with a sub-linear run-time complexity.• We conduct an extensive set of experiments using four different benchmarks (two and ten classes) and discuss the limitations of ASAL and how to overcome them.• We demonstrate ASAL with different CNN based classifiers and three different feature sets to compare samples: raw pixel values, compressed representations of an auto-encoder and the features used to discriminate between real and fake samples in GANs. We review related work on active learning especially on pool based uncertainty sampling and methods attempting to improve the run-time complexity of these active learning methods. Pool-based active learning methods select new training samples from a predefined unlabelled data set BID6;; ). A common query strategy to identify new samples is uncertainty sampling . and BID2 use minimum-distance sampling to train Support Vector Machines (SVMs). Minimum distance sampling is a well known uncertainty sampling strategy, it assumes that the classifier is uncertain about samples in the vicinity of the separating hyper-plane. This strategy is mainly used for two class but can be extended to multiple class problems by using the SVM in one vs. one or one vs. all settings (Jain et al. FORMULA1). use information entropy to measure the uncertainty of the classifier for a particular sample. Computing uncertainty with information entropy is equally suitable for two or multiple classes. propose two hashing based method to accelerate minimum distance sampling by selecting new samples in sub-linear time. These methods are designed to select the closest point (approximately) to a hyper-plane in a k-dimensional feature space, where the positions of the data points are fixed but the hyper-plane is allowed to move. Thus, these methods are limited to SVMs with fixed feature maps, because, if the feature map changes, the position of the samples become obsolete and need to be recomputed. Hence, the run time complexity is sub-linear for constant feature maps and linear otherwise. Unfortunately, CNN based methods update their feature maps during training. Thus, their methods are as efficient as exhaustive uncertainty sampling if CNNs are involved. propose Generative Adversarial Active Learning (GAAL) that uses a Generative Adversarial Network (GAN), that is trained on the pool samples, to generate synthetic samples in each AL cycle. Generating instead of selecting uncertain samples leads to a constant run-time complexity because producing a new sample is independent of the pool size. use the traditional minimal distance optimization problem (see Eq. 1) but replace the variable x (denoting a pool sample) with the trained generator. Then, they use gradient descent to minimize the objective. The latent variable minimizing the objective in a synthetic image close to the separating hyper-plane. They annotate the synthetic sample and use it for training. demonstrate GAAL on subsets of MNIST and CIFAR-10 (two classes) using linear SVMs and DCGANs (; BID3). However, GAAL performs worse than random sampling on both data sets, because it suffers from sampling bias and annotating is arbitrarily hard caused by sometimes poor quality of the synthetic uncertain samples. Note, that the GAAL requires visually distinct classes (horse & automobile) to enable manual annotations at all. We propose ASAL that reuses the sample generation idea of but we use information entropy as uncertainty score and directly extend it to multiple classes. Additionally, ASAL uses CNN based classifiers instead of linear SVMs. For the generator we train Wasserstein GANs beforehand ). We avoid annotating synthetic images by selecting the most similar ones from the pool with a newly developed sample matching method. We propose three different feature maps that we compute for each pool sample to fit a fast nearest neighbour model beforehand. During active learning, we compute the feature map of the synthetic sample and retrieve the most similar one from the pool in sub-linear time. In this section we introduce the two uncertainty query strategies: minimum distance and maximum entropy sampling. We use the following notation: The set describing the pool is denoted by P, the classifier at each AL cycle k is denoted by θ k. For uncertainty sampling where the model consists of a SVM the query strategy is based on the assumption that the model is least certain for samples that are in the vicinity of the separating hyper- DISPLAYFORM0 with (x, y) is the training set at cycle k, θ is the classifier, z the latent variable, G the generator,x the synthetic samples, F the feature extractor, f the features, P the pool and NN the nearest neighbour method.plane. Thus, newly selected samples are close to the decision boundary, are ideally support vectors that improve the decision boundary. Minimal distance sampling using SVM reads as DISPLAYFORM1 where w and b define the separating hyper-plane and φ(·) is a feature map, e.g. induced by a SVM kernel or a neural network. Instead of considering the distance to the separating hyper-plane, information entropy computes the information content in each sample for the current classifier. Thus, the classifier is uncertain for samples with a high entropy and these samples have a high information content for the task of improving the classifier. Maximum entropy sampling read as follows: DISPLAYFORM2 where DISPLAYFORM3 and m is the number of categories. Solving the optimization problems requires an exhaustive search over the whole pool P that requires computing the uncertainty score for each sample. Furthermore, we need to recompute the uncertainty score in each AL cycle because updating the classifier invalidates the previous score. Thus, classical uncertainty sampling has a linear run time complexity O(|P|) with respect to the pool size |P|. ASAL adapts the sample generation idea of to pool based active learning using multiple classes and information entropy to measure uncertainty. Fig. 1 shows the main components of the proposed ASAL. We use a labelled data set (X k, Y k) to train the classifier θ k. Then, we use the trained classifier θ k and the generator G to produce uncertain samplesx. The feature extractor F computes features that the nearest neighbour model uses to retrieve the most similar real samples from the pool. Finally, an oracle annotates the new samples and adds them to the training set. Then, a new AL cycle starts. In the remainder of this section, we introduce the adversarial sample generation and the sample matching method. Instead of selecting uncertain samples from the pool, we follow Figure 2: The rows show either generated or matched samples using different feature sets for MNIST -ten classes. The brackets denote (label id / sample id).drawn from the generator G are indistinguishable from real samples. At convergence, the generator produces the function G: R n → X that maps the latent space variable z ∼ N (0 n, I n) to the image domain X. Including the generator G(·) in Eq. equation 2 leads to the following optimization problem with respect to x DISPLAYFORM0 Removing the constraint x ∈ P by including the generator simplifies the problem but changes its solution. New samples are no longer selected from the pool but are visually indistinguishable from these samples. We solve the optimization problem in two steps: (i) we use the chain rule and gradient descent to minimize the objective with respect to z and (ii) we use G to recover a synthetic sample x from z. Thus, solving problem equation 3 has a constant run-time complexity O because it is independent of the pool size. The goal of the sample matching method is retrieving the most similar sample from the pool for a given synthetic sample. Thus, we need (i) representative features for comparison, (ii) a distance measure and (iii) a fast nearest neighbour method. The ideal features would group the samples with similar entropy in features space close together. This guarantees that the nearest real neighbour of a synthetic sample with high entropy has a high entropy as well. However, updating the model, changes the entropy of each sample in the pool and destroys the previous structure in feature space. Thus, keeping a similar grouping in feature space, requires updating the features and recomputing the position of each sample. This leads to a linear run-time complexity. Hence, for a more efficient method we require fixed features for sample matching. To design such features, we use the fact that they are not required to structure the samples according to their entropy. Indeed it is sufficient that the features identify one sample in the pool that is very similar to the synthetic sample. Then, the two samples will not only share properties the classifier is comfortable with, but also the features that lead to high entropy. Thus, the features should be representative for the data set, be diverse and allow to discriminate the main properties of different samples. The raw pixel values are a simple representation that allows to differentiate between different samples but is close for images with similar scene. Auto-encoders extract more representative features for a specific data set than the raw pixel values and lead to a compressed set of core features representing the images. Additionally, we study the features extracted from the discriminator that was used in the training of the GAN. We expect that the features used to differentiate between real and synthetic samples allow to compute representative sample properties. We use the Euclidean distance measure to compute the similarity between two samples in feature space. Furthermore, we use a multidimensional binary search tree (k-d tree) BID1 ) for efficient nearest neighbour selection. The run-time complexity to search a nearest neighbour is sub-linear O(log(|P|) with respect to the pool size |P|. Additionally, we use Principal Component Analysis (PCA) to reduce the number of dimensions of the feature space to achieve a small absolute run-time and to ensure similar run-times when using different features set with different number of dimensions. For the experiments we use two different dataset: MNIST and CIFAR-10 (Krizhevsky FORMULA2). The MNIST data set contains ten different digits 0 to 9 unevenly distributed. Each image has a resolution of 28 × 28 gray-scale pixels. The data set consists of 50k training, 10k validation and 10k testing samples. The CIFAR-10 consists of 50k training and 10k validation 32 × 32 color images with uniformly distributed label categories. We use the validation set for testing. For close comparison we follow and construct two class data sets, consisting of the MNIST digits 5 & 7 and the CIFAR-10 classes automobile & horse. First, we produce different references to assess the performance of ASAL. The classification accuracy for the fully supervised model establishes a performance upper bound that any active learning strategy attempt to approximate with as few training samples as possible. Furthermore, random sampling establishes the baseline that we want to exceed or at least perform equally. Additionally, we report the performance of traditional pool-based maximum entropy sampling that ASAL tries to approximate with sub-linear run-time complexity. We examine three different versions of ASAL using the previously introduced set of features: ASALGray/RGB, ASAL-Autoencoder, and ASAL-Discriminator. We reduce the dimension of the feature space to 50 using PCA. We experimentally verified that larger dimensions only increase the runtime but do not lead to better classification accuracy. To synthesize new samples we use the Adam optimizer and apply 100 gradient steps to minimize the negative entropy with respect to the latent space variable (see Eq. equation 3). Note, that we directly optimize for multiple latent space variables at the same time, embedding them in one batch with random initialization. We always draw samples from the pool without replacement. We do not use data augmentation for any experiment and train all models from scratch in each AL cycle. We run all experiments for five different runs with different random seeds except the computationally demanding experiments on CIFAR-10 with ten classes that we run for three random seeds. We report the training iterations for the GANs on each data sets in Tab. 3 in the appendix. We use the default values for all other parameters given by BID5 and Wei et al. (2018a) in the papers and code BID4 Wei et al. (2018b) ). We describe the different architectures and training settings of the auto-encoders in Sec. B in the appendix. Additionally, we report further insights such as label distribution, entropy of newly added samples and additional experiments using other GANs or uncertainty scores in the appendix. For binary digit classification we train a linear model with cross entropy loss. We train the model for 10 epochs using the Adam optimizer with a batch size of 10 and learning rate of 0.001. We train the Wasserstein GAN BID5 ) with gradient penalty to synthesize only the digits 5 & 7. FIG0 shows that for a budget of 500 samples only the aggressive active learner reaches the performance of the fully supervised model. However, ASAL performs clearly superior to random sampling and converges faster to the accuracy of the fully supervised model. We want to emphasize, that all ASAL strategies outperform random sampling. Furthermore, FIG0 verifies that the entropy of newly added samples to the training set is higher for ASAL than for random sampling. On average, all versions of ASAL select samples with 63% higher entropy than randomly selected samples. report worse performance than random sampling when training and testing on MNIST -two classes. For comparison we re-implement their GAAL with DCGAN. For a fairer comparison, we use an improved version of GAAL using the same Wasserstein GAN as ASAL and additionally test ASAL with the DCGAN. FIG0 shows that our implementation of GAAL performs similarly as random sampling and suffers less from sampling bias than reported by. Possible reasons are the different human annotators or slightly different design choices. Furthermore, we observe that both methods outperform random sampling when using Wasserstein GANs and perform worse when using DCGAN. However, ASAL exceeds the performance of GAAL especially when using the Wasserstein GAN. Furthermore, using Wasserstein GAN leads to less stable performance of GAAL than ASAL(higher variance between 250 and 300 training samples for GAAL, note the red spikes in negative direction). FIG0 shows only the for ASAL-Gray, for additional , see Fig. 10 in the appendix. For binary classification on CIFAR-10, we reuse the experimental setup presented in Sec. 5.3 but change the batch size to 50. We run the active learning strategies until the budget of 1000 samples is reached. Again we use a Wasserstein GAN BID5 ) with gradient penalty that synthesizes only two classes (automobile & horse). FIG1 shows that especially ASAL-Autoencoder exceeds the performance of random sampling and achieves similar or slightly better than exhaustive uncertainty sampling, for a training set containing more than 500 samples. For ten digit classification we use LeNet with cross entropy. We train the model for 10 epochs using the Adam optimizer with a batch size of 50 and learning rate of 0.001. For active learning, we start with an initial data set containing 10 samples for each class and add 50 samples each AL cycle. We run our experiments until the training set contains 10k samples. We synthesize samples for all ten classes using a Wasserstein GAN with gradient penalty BID5. FIG1 shows that the proposed ASAL strategies also tackle multiple class problems and exceed the quality of random sampling. Using all classes of CIFAR-10 complicates the classification task and we require a deep model to achieve close to state-of-the-art . Therefore, we use the All-CNN model proposed by with a reported classification error of 9.08%. We use the proposed architectures and training strategies and use stochastic gradient descent with constant momentum of 0.9 and a learning rate of 0.01 that we decay by a factor of 10 at the 130th and the 140th epoch. We train the model for 150 epochs with a batch size of 128 without data augmentation and report a classification Figure 5: The rows show either generated or matched samples using different feature sets for CIFAR-10 -ten classes. The brackets denote (label id / sample id).error of 11.8%. The All-CNN contains ∼1.4 million different parameters. Hence, we require larger initial training sets than for the previous models. Thus we include 100 randomly selected images per class. We add 1000 samples to the data set every AL cycle until the budget of 30k samples is reached. We generate ten times a batch containing 100 samples because optimizing for all samples at the same time is unfeasible. In contrast to the previous experiments we use a residual Wasserstein GAN (Wei et al. (2018a) ) with gradient penalty and soft consistency term. We observed an Inception score of 7.8 without and 8.3 with adding the soft consistency term. We use the publicly available TensorFlow implementation of Wei et al. (2018b). FIG1 shows the for different ASALs using the promising residual GAN, that achieves the highest Inception score. Unfortunately, FIG1 shows that the performance of ASAL follows random sampling or is slightly worse, whereas maximum entropy sampling converges to the quality of the fully supervised model but uses 60% of all pool samples. Figs. 26 and 27 in the appendix show the label distribution for each AL cycle. It reveals that maximum entropy sampling selects most frequently cat exactly one of the classes that are least frequent in most of the training set of ASAL. Furthermore, FIG1 in the appendix reports the same experiments using different GANs but none of them leads to superior performance than random sampling. Our experiments and show that ASAL clearly outperforms random sampling and approximates exhaustive uncertainty sampling on three out of four benchmarks. Compared to GAAL, ASAL outperforms random sampling, enables annotating real samples, handles multiple class problems and uses CNN based classifiers. ASAL allows to update the feature maps of a classifier in each AL cycle and still achieves sub-linear run-time complexity whereas the hashing based methods of Jain et al. FORMULA1 has a linear run-time complexity if the feature maps are updated. Updating the classifier and keeping the features for matching fixed, leads to sub-linear run-times but without guaranteeing that newly added samples have the highest entropy of all samples available in the pool. To achieve a sub-linear run-time complexity, ASAL requires to train a GAN and potentially an autoencoder beforehand. Nonetheless, this initial cost pays off for extremely large data sets. Although, it might be impractical to consider each sample during training of the GAN, it can generate representative samples and ASAL allows to select samples from the pool that were not used to train the GAN. Thus, ASAL favours large data sets with similar samples, where it is only possible to train the GAN for a fixed number of iterations but contains a close match for any synthetic sample. Conversely, small data sets with diverse samples allow to train the GANs for many epochs such that it is align to the data distribution. However, real samples are sparsely distributed in feature space such that even the closest matches of a synthetic sample are significantly different. We observed in FIG1 that ASAL performs similar as random sampling. Although ASAL enables to generate uncertain samples, it fails to select similar samples from the pool that have high entropy. One explanation is the aforementioned situation, where the images are diverse but the data set is comparatively small. Note, that CIFAR-10 is clearly more diverse than MNIST but has the same amount of samples. Furthermore, the top row in Fig. 5 shows that synthetic images still look unrealistic and identifying a similar real sample is a challenging problem. Another reason for poor performance is using low level features to compare different samples. To achieve state-of-the-art on CIFAR-10, we had to use a much deeper network than for all other experiments but kept the architectures of the feature extractors almost identical. This can lead to a mismatch where the entropy of a sample mainly depends on high-level features but the matching method uses only low-level features to compare samples. FIG2 in the appendix shows for example that exhaustive uncertainty sampling selects most frequently images with the category cat exactly a class that ASAL selects least frequently. This is a sign that ASAL considers low-level features to find similar samples instead of more complex properties that characterize class information. Fig. 5 provides again such an indication. The last column shows a synthetic image with a white horse on a gray and ASAL proposes matches with white object on a gray but contain either a ship or an airplane. This means, that the classifier requires samples of a specific class it is uncertain about, ASAL generates these samples but fails to retrieve matches showing theses categories. On CIFAR-10 -two classes we reported for ASAL-Autoencoder similar or slightly higher accuracy than for exhaustive uncertainty sampling. Although we consider uncertainty sampling as a performance reference that we try to approximate, it is always possible to exceed its performance. Note, entropy is one particular property that can identify informative samples. Nonetheless, it is possible that samples with lower entropy are more effective for training the classifier. We proposed and evaluated a new pool-based active learning method that uses sample generation and matching. However, the sub-linear run-time complexity requires relaxing the guarantee, that selected samples have the highest entropy of all pool samples. We showed, that the success of ASAL depends on different factors: the structure of the data set, the quality of the trained GAN and the relevance of the feature used to compare samples. A poor GAN can generate high entropy samples but poor quality samples are impractical to match. Small data sets that contain very different samples complicate both, training GANs and finding similar matches. Less representative features might not contain the properties needed to find similar samples, where both have a high entropy. Nonetheless, we demonstrated that ASAL outperforms random sample selection and approximates exhaustive uncertainty sampling in three out of four cases. Furthermore, the sub-linear run-time complexity makes ASAL suitable for large data set. We pointed out that ASAL uses low-level feature but there are signs that high-level features might be more suitable to match samples. Thus, one particular direction of future research includes identifying such high-level features. Possible candidates are VGG (Simonyan & Zisserman FORMULA1) or AlexNet (Krizhevsky et al. FORMULA1 features. Training the model on the unlabelled pool and the small initial data set might lead to features covering the needed properties. In addition, sample generation allows adding other scores beside information entropy. Thus, an interesting direction of future research is designing other scores that will be used during sample generation i.e. measuring sample diversity (Zhu et al. FORMULA2 We keep the suggested splitting into training, validation and testing for each benchmark. We train a Wasserstein GAN with gradient penalty and an auto-encoder for ASAL beforehand (see Tabs. 1 and 2 for the architectures). We use a Nvidia GeForce GTX TITAN X GPU to train the models. The 100k training iterations for the GAN take roughly 25 h and the 50k iterations of the auto encoder 1.6 h. We use for both the Adam optimizer with a learning rate of 0.0001 and batch size 64. The number of compressed features of the auto-encoder is 128. For sample matching we decrease the number of features to 50 using PCA.For classification we use the CNN presented in Tab. 1. We use the Adam optimizer with a learning rate of 0.001 and a batch size of 50 and train for 30 epochs. We start active learning with 100 labelled samples, where the number of samples per class corresponds to the data distribution. We select and label ten new samples in each active learning cycle until we exhaust the budget of 2000 samples. We run all experiments for three different random seeds. For sample generation we apply 100 gradient descent steps using the Adam optimizer with a step size of 0.01. We optimize for all ten samples at the same time. A.3 AND TIMINGS FIG2 shows the test accuracy of random sampling, maximum entropy sampling and ASAL for three different benchmarks on CelebA. We conclude, that ASAL clearly outperforms random sampling and approaches the accuracy of maximum entropy sampling but requires less time for sample selection, see FIG4. Thus, the proposed idea of sample generation and matching is simple but effective. FIG4 reports the time required to select ten new samples in each active learning cycle with respect to different data set sizes. We randomly augmented the data set containing 160k samples to report the timings of maximal entropy sampling and ASAL (only nearest-neighbor search depends on data set size) for data sets with up to 16M samples. Whereas it it possible to keep all images in memory for 160k (1.98GB) this is hardly possible for 16M images (198GB). Nonetheless, we did not add additional I/O-time for reading the images from disk to the maximum entropy timings in each AL cycle. The sample matching proposed in ASAL reduces the memory consumption because it requires to keep only 50 features per images (32MB for 160k and 3.2GB for 16M images). This saving allows to keep the features in memory and to build the nearest neighbor model even for huge data sets. We use a Nvidia GeForce GTX TITAN X GPU to report all timings. ASAL has a sub-linear run-time complexity to select new samples. However, it requires several pre-processing steps such as training the GAN (∼ 25h), training the auto-encode (∼ 1.6h), extracting the features (∼ 32s per 160k samples) and fitting the nearest-neighbor model (∼ 5min for 16M samples). Note, that the number of iterations depends mainly on the difficulty of the data set than on its size. As sample selection for ASAL is almost negligible (44s for 16M samples per AL cycle) compared to the pre-processing time, the transition point is approximately then, when uncertainty sampling takes more time than preparing ASAL. Thus, maximal uncertainty sampling is more efficient when using small data sets or running active learning only for a few samples. Nonetheless, for all settings and data set sizes there exists a transition point, see Fig. 8. For example, ASAL is already more efficient after only 30 cycles (300 added samples) than maximum entropy sampling for a data set containing 16 million samples for the given setup. Figure 8: ASAL is more efficient for selecting new samples than maximal entropy sampling. However, it requires pre-processing time. These diagrams show the transition point (number of AL cycles when ASAL gets more efficient than maximum entropy sampling) with respect to the data set size. For MNIST -two classes we use the following auto-encoder settings: the encoder consists of three convolution layers with a stride of two, each followed by an activation leading to 64 compressed features. The decoder uses three deconvolution layers each with an activation. We train the autoencoder for 40 epochs with a batch size of 100 using the Adam optimizer with a learning rate of 0.001. For MNIST -ten classes we reuse the same settings but with three times more channels ing in 192 compressed features.64 × 64 × 3 3 × 3 conv: 16 linear: 128 × 4096 5 × 5 conv: 128, stride=2 ReLU, Maxpool 2 × 2 Batch norm, ReLU leakyReLU 3 × 3 conv: 32 5 × 5 deconv: 256 5 × 5 conv: 256, stride=2 ReLU, Maxpool 2 × 2 Batch norm, ReLU leakyReLU 3 × 3 conv: 64 5 × 5 deconv: 128 5 × 5 conv: 512, stride=2 ReLU, Maxpool 2 × 2 Batch norm, ReLU leakyReLU linear: 4096 × 1024 5 × 5 deconv: 64 5 × 5 conv: 512, stride=2 ReLU, Dropout 0.5 Batch norm, ReLU leakyReLU linear: 1024 × 1 5 × 5 deconv: 3 linear: 8192 × 1 TanhThe Encoder for CIFAR-10 consists of three layers, each with a convolution followed by batch normalization BID7 ), activation and max pooling (stride of two and window size 2 × 2). The number of compressed features is 256. The decoder uses first a layer consisting of a convolution, batch normalization and activation followed by three deconvolution layers each with batch normalization and activation. We train the auto-encoder for 100 epochs with a batch size of 128 using the Adam optimizer with a learning rate of 0.001. We use the same settings for CIFAR-10 -two classes and CIFAR-10 -ten classes. Label distribution for uncertainty sampling using maximum entropy and random sampling for MNIST -two classes using different uncertainty measures and loss functions. The tick on the right show the true label distribution in the pool. The label distribution of the training set, assembled with random sampling (top), converges to the true label distribution of the pool. Conversely, uncertainty sampling leads to a training set that contains more frequently the label 5 than 7 compared to the pool that contains 7 more frequently. Apparently, images with the digit 5 lead to higher uncertainty of the used classifier. Figure 10: Test accuracy on MNIST -two classes of a fully supervised model, for random sampling, uncertainty sampling and different ASALs using different GANs, uncertainty measures and loss functions. ASAL with WGAN-GP (bottom) clearly exceed the performance of ASAL using DCGAN (top). Maximum entropy sampling and using the cross entropy loss lead to the setup (10d) that approaches the fully-supervised model with the fewest samples and reaches the smallest gap for all ASAL using 500 labelled samples. Figure 11: Label distribution for active learning using different matching strategies, uncertainty measures and GANs for MNIST -two classes. The ticks on the right show the true label distribution in the pool. ASAL using WGAN-GP (third and fourth row) reaches a label distribution of the training data that is similar to the true label distribution in the pool. Conversely, ASAL using DCGAN (first and second row) leads to a training set that contains almost three times as many images with the digit 7 than digit 5. Most likely, the DCGAN is responsible for this behaviour because we already observed that it produces the digit 7 more frequently than the digit 5, see FIG0. Figure 12: Average entropy of images that are selected and added to the training set for MNIST -two classes using different GANs, uncertainty measures and loss functions. All figures show that ASAL selects samples from the pool that have a higher entropy than randomly sampled images. However, maximum entropy sampling and WGAN-GP (12d) lead to the largest entropy gap between selected and randomly sampled images. Maximum entropy sampling (right column) in smaller average entropy of the classifier than minimum distance sampling (left column) because we use the cross-entropy loss that directly optimizes for small entropy, opposed to the hinge loss that minimizes the distance to the separating hyper-plane. Instead of manually annotating images we propose to select similar images from the pool and ask for labels of these images. Similar images might show an object of the same class, have similar surrounding, colors, size or share other features. Thus, we compare the agreement of the manual class annotations of the generated images with the matched images, using the three different strategies. We use 1300 generated samples for each GAN, annotate the images manually and retrieve the closest match with the corresponding label from the pool. We assume that the final model will be measured on an almost evenly distributed test set similar to MNIST and USPS. However, the test set for this experiment contains the generated samples with manual annotations and the GAN may generate samples with unevenly distributed label frequency. Thus, we compute the accuracy for each class independently and average these values subsequently to obtain the final score. FIG0 shows that the agreement is higher for ASAL strategies using WGAN-GP than DCGAN. Furthermore, we observe that the matching based on gray values achieves the highest agreement. Similarly, Figs. 10a and 10b show best performance for ASAL-Gray. The matching strategies employed in ASAL allow to select similar images from the pool and compare these labels to the manual annotations. For MNIST -two classes the agreement for WGAN-GP is higher than for DCGAN. Figure 15: Average entropy of images that are selected and added to the training set for MNIST -ten classes using different GANs. Both figures show that at the beginning ASAL selects images with higher entropy than random sampling. In average WGAN-GP leads to a larger gap than DCGAN. However, this gap rapidly shrinks when increasing the training set.: Label distribution for uncertainty sampling using maximum entropy, random sampling and active learning using different matching strategies and GANs for MNIST -ten classes. The tick on the right show the true label distribution in the pool. Note the different scaling of the yaxis. Random sampling converges to the true label distribution in the pool and maximum entropy sampling leads to a training set with a higher ration of certain digits or lower than the pool. Similarly, ASAL using WGAN-GP (bottom row) selects certain digits more frequently than others. Conversely, ASAL using DCGAN (top row) leads to a training set that contains 30% images with the digit 1. Most likely, the DCGAN is responsible for this behaviour because we already observed that it produces the digit 1 more frequently than any other digit, see FIG0. E report the accuracy of GAAL when trained on MNIST and tested on USPS for two classes. They report best performance on USPS and outperform the fully supervised model. However, it is unclear how they up-sample the 16 × 16 USPS images to test on the 28 × 28 model trained on MNIST. We redo the experiments using ASAL and up-sample the USPS images as follows: padding the images with three pixels at each side, up-sampling the images to 30 and cropping the images to 28 × 28 to remove boundary artifacts. Following this strategy, we report an average test accuracy of 0.91 for the fully supervised model compared to 0.70 reported by. FIG4 shows that ASAL outperforms aggressive uncertainty and random sampling. We repeat the experiment for MNIST -ten classes using DCGAN and WGAN-GP. This time, uncertainty sampling clearly outperforms all other strategies and the fully supervised model. Nonetheless, ASAL-Auto and ASAL-Disc lead to a better training performance than passive learning for WGAN-GP. Figure 17: Test accuracy on USPS -two classes but trained on MNIST -two classes of a fully supervised model, for random sampling, uncertainty sampling and different ASALs using different GANs, uncertainty measures and loss functions. Uncertainty sampling performs worse than any other strategy because it aggressively trains the classifier for the samples present in the pool and generalizes less. Random sampling and ASAL tend to generalize better by respecting the true data distribution either through random sampling or using a pretrained GAN on the data set to find new samples. Figure 18: Test accuracy on USPS -ten classes but trained on MNIST -ten classes of a fully supervised model, for random sampling, uncertainty sampling and different ASALs using two different GANs. Maximum entropy sampling for ten classes exceeds the quality of any other method compared to binary classification where it performed worst, see FIG4. The more elaborate LeNet and using more classes and samples to train lead to a classifier that generalizes well. The active learning strategies using WGAN-GP exceed the quality of random sampling. ASAL-Disc. even outperforms the fully supervised mode. ASAL using DCGAN performs comparable to random sampling. For CIFAR-10, we do not indicate the true label distribution by a tick because the validation set contains the same number of samples for each class. Figure 19: Label distribution for uncertainty sampling using maximum entropy and random sampling for CIFAR-10 -two classes using different uncertainty measures and loss functions. The label distribution of the training set of all strategies converges to the true label distribution of the pool. However, in average over all active learning iterations the training set of the uncertainty sampling strategies most frequently contained the images with the label horse.: Average entropy of images that are selected and added to the training set for CIFAR-10 -two classes using different GANs. The mean entropy of the random sampling and the proposed method show hardly any difference. However, for maximum entropy sampling at least at the beginning ASAL selects images with higher entropy than random sampling. For CIFAR-10, we do not indicate the true label distribution by a tick because the validation set contains the same number of samples for each class. Figure 25: Average entropy of images that are selected and added to the training set for CIFAR-10 -ten classes using different GANs. There is hardly any difference for random sampling and ASAL in the entropy of newly added samples. Only at the beginning, random sampling retrieves samples with slightly higher entropy. Figure 26: Label distribution for uncertainty sampling using maximum entropy and random sampling for CIFAR-10 -ten classes. Random sampling converges to the true label distribution in the pool. Maximum entropy sampling selects most frequently cat, dog, bird, deer and least frequently automobile, ship, truck to exceed the classification quality of random sampling.: Label distribution for active learning using different matching strategies, uncertainty measures and GANs for CIFAR-10 -ten classes. Exactly the classes cat, dog that are most common in the training set of uncertainty sampling are less common in the data sets of most setups. Conversely, frog is for many setups the most common class but is not particularly frequent in the uncertainty sampling data set. FIG0: The rows show generated and matched images for CIFAR-10 -ten classes using WGAN-GP. Most of the generated images achieve only a moderate quality and even the closest samples from the pool have a high perceptual visual distance or assign images that show non matching classes, see last column where the images have a similar appearance but an appropriate label for the generated images would be horse but the selected samples show airplane and ship. To produce the images displayed in Figs. 32, 33, 34 and 35 we trained the classifier using the initial training set. Then we used maximum entropy sample generation to produce samples with a high entropy. Figure 32: Comparison of random and uncertain MNIST -two classes. The samples are generated using different GANs. The random samples are visually more appealing and identifying the label is easier than for the uncertain samples. WGAN-GP generate images for both digits equally likely, whereas DCGAN most frequently generates images showing the digit 7. Figure 33: Comparison of random and uncertain MNIST -ten classes samples. The samples are generated using different GANs. The random samples are visually more appealing and identifying the label is easier than for the uncertain samples. WGAN-GP uniformly generates images for all digits, whereas DCGAN mainly generates images showing the digit 1. Comparison of random and uncertain samples for CIFAR-10 -two classes using maximum entropy. The samples are generated using different GANs. The residual GANs (bottom row) produce more visually appealing samples than the other GANs. For most of these images it would be possible to identify whether the image shows a horse or automobile. Comparison of random and uncertain samples for CIFAR-10 -ten classes using maximum entropy. The samples are generated using different GANs. The residual GANs (bottom row) produce more visually appealing samples than the other GANs. Although, the quality of the random images is higher than of the uncertain images, annotating with high confidence is still very difficult.

ASAL is a pool based active learning method that generates high entropy samples and retrieves matching samples from the pool in sub-linear time.

represent the tldr for finding the corresponding scientific article.

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent the article for finding a single-sentence TLDR

Question: We point out important problems with the common practice of using the best single model performance for comparing deep learning architectures, and we propose a method that corrects these flaws. Each time a model is trained, one gets a different due to random factors in the training process, which include random parameter initialization and random data shuffling. Reporting the best single model performance does not appropriately address this stochasticity. We propose a normalized expected best-out-of-n performance (Boo_n) as a way to correct these problems. Replicating in deep learning research is often hard. This harms their usefulness to industry, leads to a waste of effort by other researchers, and limits the scientific value of such . One reason is that many papers provide information insufficient for replication. Details of the experimental setup can significantly influence the BID13 BID10 BID23, so the details should be provided at least in appendices, ideally alongside the source code, as was strongly emphasized e.g. by BID17.However, an important second factor hinders replicability: most deep learning training methods are inherently stochastic. This randomness usually comes from random data ordering in stochastic gradient descent and from random parameter initialization, though there can be additional sources of randomness such as dropout or gradient noise. Consequently, even if we fix the model architecture and the experimental setup (including the hyperparameters), we obtain a different each time we run an experiment. Statistical techniques are needed to handle this variability. However, in deep learning research, they are heavily underused. What is usually done instead?Most empirical deep learning papers simply report the performance of the best single model (sometimes calling it just "single model" performance). We will later show this is the case at least for some sub-domains. Given the stochasticity, such method is statistically flawed. The best model performance is not robust under experiment replication, and its expected value improves with an increasing number of experiments performed, among other problems. Since many deep learning publications largely ignore these issues, we dedicate the first part of this article to explaining them in some detail, and later run experiments to quantify them. Appropriate statistical techniques are hence necessary for evaluating (and comparing) the performance of machine learning (ML) architectures. Some well-developed methods exist for such comparisons (a great introduction is given for instance by BID5). However, most existing methods focus on comparing the mean performance. This may be one of the reasons why statistical methods are being underused, since mean may be unattractive to researchers in certain situations. There are multiple possible reasons for this. The one that we do consider sound 1 is that when deploying models in practice, it is often natural to train multiple instances of a model and then deploy the best one to production based on a validation set evaluation.2 Underperforming models can be discarded, so the final deployed model does come from 1 Other reasons why researchers resort to the best performance as opposed to the mean may come from the current highly competitive atmosphere in the field with (possibly excessive) focus on performance on standard datasets (see BID4 or BID26 for further discussion), which may motivate researchers to publish only their best . Also, statistically sound estimation of performance does require repeatedly re-running experiments, which does incur additional cost, which researchers may prefer to invest in additional model tuning, especially in the present situation where reviewers seem not to require statistically sound evaluation of models and on the other hand may favour high-performing models. Of course, these motives should instead give place to effort to do good science, as opposed to a race on standard benchmarks. 2 In some applications there is focus on speed of training and on reducing computational costs -there it does make sense to focus on the performance of the typical model as opposed to the best out of n, so the use of mean or median is appropriate. the higher tier of the model performance population, and the use of mean may be inappropriate. Hence, rather than to completely abandon reporting the performance of the best model, we propose a way to fix its flaws. We do this by estimating the expected best-out-of-n (Boo n) performance by running more than n experiments, which gives the estimate statistical validity if a sufficient number of experiments are run. We discuss how this measure relates to the performance distribution of the model, and we also give a method to empirically estimate Boo n.The paper proceeds as follows: First, we give a high-level explanation of why reporting performance of the best single model is problematic. We also give some evidence that it is widely used in the deep learning community, which is why this explanation may be needed. We proceed by presenting Boo n as a way to fix the above problems. We then give some experimental evidence for the flaws of best-singlemodel reporting and show that Boo n does not suffer from them. We wrap up by discussing the place of Boo n in a ML researcher's toolbox alongside traditional measures such as mean or median. In articles presenting new deep learning architectures, the performance is often reported as the score of the "best single model" or simply "single model". In practice, this usually means that the researchers train multiple instances of the proposed architecture (often with different sets of hyperparameters), evaluate these instances on some validation set, and select the best-performing model. This best model is evaluated on a test set, and the ing test score is then reported as the metric characterizing the architecture and used for comparing it to previous models. If the score is better than those reported in previous work, the architecture is presented as superior. This practice in several issues:Population variance Since of experiments are stochastic, the performance of a single model is just a single instance drawn from a possibly disparate population. If others train the model on their own, they get another sample from the architecture's performance distribution, which may substantially differ from the one listed in the original paper. Such paper thus gives insufficient information about what to expect from the new architecture, which should be one of the article's main purposes. One may object that the published in the paper is not chosen from the population at random -it is selected using a validation . However, the correlation between the validation and test is generally imperfect; in fact, in some of our experiments, it is almost zero, as we show in Section 4. Furthermore, if we indeed do have a strong correlation, we get another problem:Expectation of best increases with the number of experiments Simply put, the more samples from a population we draw, the more extreme the best among them is likely to be. In other words, the expected value of the best depends on the number of experiments that the researchers run. There are three closely related problems with this: Firstly, this makes the number of experiments run an important explanatory variable; however, this variable is usually unreported, which is a severe methodological flaw in itself. It also leads to the second problem: since each research team runs a different number of experiments, the are not directly comparable. Thirdly, this motivates researchers to run more experiments and gives an advantage to those who are able to do so. This pushes publishing quantitative towards a race in computational power rather than a fair comparison of architectures themselves. Best model performance is not a meaningful characteristic of the performance distribution Even if we knew the underlying theoretical performance distribution -that is, if we had perfect information about the architecture's performance -it would not be clear what we would mean by "best model performance" without specifying the size of the pool from which we are choosing the best model. Imagine some architecture having a Gaussian performance distribution. Asking what is the best possible performance does not make sense in such a case, since the support of the distribution is unbounded. Even for capped metrics such as accuracy, where the performance distribution necessarily has bounded support, the best (possible) model 3 may be so unlikely, that it would be of no practical importance. Hence, best model performance is not a meaningful characteristic of the performance distribution. Generality / Falsifiability Finally, there is the question of what the authors are trying to express. Using "best single model performance", they are essentially claiming: "There once existed an instance of our model that once achieved a X on dataset Y". Such fact is not of that much interest to the scientific community, which would rather need to know how the architecture behaves generally. Relatedly, a frequently given characteristic of science is falsifiability of theories BID22. A theory claiming that there are invisible unicorns running among us is not science, since we cannot think of any potential empirical evidence that could prove the theory false. Similarly, any number of replication experiments that produce substantially worse cannot prove the above performance claim wrong. If, for instance, a confidence interval were given, replications could very quickly show the published at least extremely implausible, if not false. We will quantify the former two problems for two concrete architectures in Section 4. Despite all these problems, reporting the performance of the best model is still the main way of reporting in some areas of ML, especially in empirically oriented deep learning papers, and, alarmingly, such practice seems to be tolerated by reviewers even at prime conferences. For instance, what concerns models published on the popular Children's Book Test dataset for reading comprehension (on which we run experiments later), none of the (more than ten) papers used any form of statistical testing or confidence intervals, and most reported only performance of the best single model without even mentioning the total number of experiments run. These include papers published at NIPS BID14, ICLR BID15 BID21, ACL BID3 BID7 BID6, or EMNLP BID28 ).The same is true for the recently popular SQuAD dataset: for instance, none of the four papers BID32 BID30 BID27 BID31 that published on this dataset at ICLR 2017 has used any statistical testing or confidence intervals nor published mean (or otherwise aggregated) across multiple runs. Let us look more generally at the example of ICLR 2017 (chosen as a deep-learning-focused conference featuring many empirical -as a rough guide, 174 out of 194 ICLR papers have "experiment" in a (sub-)section heading). Only 11 papers mention terms related to hypothesis testing 4, and 11 contain the string "confidence interval". Further details can be found in Appendix B.While this is a rough and limited survey, it does suggest that while deep learning research is to a large extent an empirical science, statistical methods are often underused.3. Expected Best-out-of-n (Boo n) PerformanceThe issues outlined above point to desiderata for a more suitable method of reporting an architecture's performance. It should provide information about general behaviour of the architecture under specified conditions, well characterizing the associated random performance distribution. It should also be invariant under the number of experiments run and be as robust as possible to random noise. Given these requirements, traditional statistical measures, such as mean or median, probably come to mind of many readers. They do indeed fix the above issues; still, they express only the performance of a typical member of the population. However, in many ML applications, it may be the best model from a pool that is of interest. When practitioners are choosing a model for deployment, they train multiple models and deploy the best-performing one 5. This gives some justification to reporting the performance of the best model and gives us a reason to attempt to fix its problems rather than completely dismiss it. Such corrected best-model measure would be more informative than mean or median in these outlined situations. A natural way to improve comparability between models, each evaluated in a different number of experiments, is to normalize the to the expected if the number of experiments were the same, say n, which can be easily estimated if we run m experiments, m ≥ n. The greater the number of experiments m, the more robust the estimate of the expected best, which also helps us eliminate the problem of statistical robustness. We are proposing the expected best-out-of-n performance, Boo n, to be used where the performance of the best model from a pool seems as an appropriate measure. Let us first examine how the expected best-out-of-n (Boo n) performance relates to the (theoretical) performance distribution we are trying to characterize; we will then proceed with empirical estimation, which is of value in practice. The calculations are not particularly innovative from statistical point of view and are close to many standard from the field of Order Statistics (see for instance BID1 for more context). Showing how to calculate Boo n of a known theoretical probability distribution will serve two purposes: Firstly, since we are proposing Boo n as a way to characterize the performance distribution, this will make the relation between Boo n and the performance distribution explicit. Secondly, in some cases we may be able to make an assumption about the family to which the theoretical distribution belongs (e.g. 5 This would usually be the case when a model is trained once and then deployed for longer-term usage, which may be the case for instance for Machine Translation systems. In other cases, when it is practical to train only as single model instance due to hardware constraints (either because training is extremely costly, or because it needs to be done repeatedly, e.g. for individual customers), we may indeed be interested in a typical model and hence in mean or median performance.we could assume it is approximately Gaussian). The analytic calculation below will allow us to leverage this information when empirically estimating Boo n by deducing a parametric estimator, which may be especially useful when our sample size m is small thanks to its lower variance due to added prior information. Let us first look at the simpler case of validation performance (that is, the case where we are choosing the best model with respect to the metric we are reporting) as it is easier to grasp: How do we calculate an expected best Boo n (P) 6 of independent identically distributed (i.i.d.) random variables X 1,..., X n with probability distribution P (the performance distribution of an architecture) with a probability density function (p.d.f.) f and a cumulative distribution function (c.d.f.) F? In the case where best means maximal (minimum can be calculated similarly), the maximum max{X 1, ..., X n} has a c.d.f. equal to DISPLAYFORM0 using the independence of the X i s in the last step. In the case of a continuous distribution, we can obtain the p.d.f. of the maximum by simply differentiating with respect to x: DISPLAYFORM1 Using the p.d.f., we can now calculate the expected value of the maximum as DISPLAYFORM2 We can get a precise numerical estimate of the above integrals in any major numerical computation package such as numpy. For illustration, for the standard normal distribution we have Boo 5 (N) ≈ 1.163, Boo 10 (N) ≈ 1.539. More generally, Boo n N (µ, σ 2) can then be expressed as µ + σBoo n (N). Thanks to this form we can get numerical estimates of Boo n N (µ, σ 2) just by estimating the two usual parameters of the Gaussian, Boo n (N) becoming just a constant coefficient if we fix n. The full details of calculation for the Gaussian distribution can be found in Appendix A.In the case of a discrete performance distribution, which will be useful for empirical estimation below, we get a prob-ability mass function DISPLAYFORM3 so if p j is the probability weight associated with value x j, i.e. P[X i = x j] = p j for all i, this gives us DISPLAYFORM4 In the previous part, we were choosing the best model with respect to the metric whose expectation we were calculating. Hence, that method can be used to calculate the expected best validation performance of n models. In practice, the best model is usually chosen with respect to the validation performance, while the primary interest is in the corresponding test performance. To calculate the expected test performance of the best-validation model, we need to substitute the direct value of x in Equations 2 and 4, with the expectation of the test performance X test conditional on the validation performance x val, DISPLAYFORM0 yielding an expression for the expected test performance of the best-validation model chosen from a pool of size n where P best val is the marginal probability distribution of the best-out-of-n validation performance. Similar simple substitution can be done in the discrete case. DISPLAYFORM1 Expanding the expression for the bivariate Gaussian distribution having marginal test performance with mean µ test, standard deviation σ test, and test-validation correlation ρ as in Appendix A.2 yields a convenient expression DISPLAYFORM2 which can again be used for parametric estimation. We usually do not know the exact performance distribution of the model; we only have samples from this distributionthe of our experiments. In such case, we can estimate the expected maximum empirically, and in fact it is the empirical estimates that are likely to be used in practice to compare models. To get a non-parametric estimator, for which we do not make any assumption about the family of the performance distribution, we take the empirical distribution arising from our sample as an approximation of the architecture's true performance distribution, similarly to Bootstrap methods. The empirical performance distribution P assigns a probability weight of 1 m to each of our m samples. We approximate Boo n of the true performance distribution by Boo n of this empirical distribution. For the uniform empirical distribution, all the p i in Equation 4 are equal to 1 m. Hence, if we rank our samples from worst-validation to best-validation DISPLAYFORM0 This is in fact a weighted average of the test . In case of a tie in validation , i.e. if x DISPLAYFORM1, one should assign an equal weight of This estimator does not make any assumption about the performance distribution from which our observations are drawn. If we do use such an assumption (e.g. we know that the performance distribution of our architecture usually belongs to a certain family, e.g. Gaussian), we can add information to our estimator and possibly get an even better estimate (i.e. one with lower sampling error). For the Gaussian distribution, we can use the standard estimators of the parameters in Equation 6 to get a parametric estimator DISPLAYFORM2 where µ test, ρ and σ test are standard estimators of mean, correlation, and standard deviation respectively. A similar parametric estimator could be calculated for other distributions. Boo n eliminates the problem of dependence on the number of experiments run, m. However, we still need to choose n, the number of experiments to which we normalize. This is similar to the choice one is facing when using a quantileshould one use the 75% one, the 95% one, or some other?The choice of n most useful to a reader is when n is the number of candidate models that a practitioner would train 7 You can get an optimized Python implementation of the nonparametric estimator via pip install boon. before choosing the best one for some target application. Such number will differ domain to domain and will heavily depend on the computational cost of training the specific architecture on the specific domain. The n of interest may differ for each reader -ideally, researchers should characterize the architecture's performance distribution as fully as possible and the readers may be able to deduce the value of Boo n for whichever n they choose (up to a limit).Leaving an additional degree of freedom in the choice of metric creates a risk of cherry picking. However, in many areas of machine learning, there already are many available metrics. Still, the main reporting metric seems to quickly converge on each task. The first published paper makes a choice; the subsequent ones usually follow suit (else they risk a suspicion that the architecture is not competitive on the previous metric). We believe similar convergence is likely for Boo n on each task. In our experiments, we decided to use n = 5 -the AS Reader model which we use for our experiments takes about 2 hours to train on a single GPU, so someone replicating the Boo 5 performance could expect to achieve it overnight, which seems a to be a reasonable requirement. Even Boo n is just a single number whose estimate can be noisy. Hence, with Boo n, as well as with mean and other ways of aggregating of a wider population, we should always use appropriate statistical methods when trying to compare the quantitative performance of a new model against a baseline. This can be done using significance testing (such as the t-test), or with the help of confidence intervals, which seems to be the method preferred by a significant part of the scientific community (e.g. BID11 or BID2), since it allows us to disentangle the effect size from the uncertainty associated with noise and sample size. For some theoretical distributions, there exist ways to calculate the hypothesis test or confidence interval analytically (e.g. using the t-test or standard normal quantiles for the Gaussian). However, in cases where the family of the performance distribution or of the estimator is not known, we need to resort to computational methods -usually Monte Carlo (if we do know at least the family of the performance distribution) or the Bootstrap BID8 ) (if we do not). A brief description of how to calculate confidence intervals using the Bootstrap is provided in the Appendix. Note: The data and code for their analysis can be found at http://gitlab.com/obajgar/boon, along with Python functions you can use to calculate Boo n. We have run several experiments to quantify the scope of the problems outlined in Section 2. We just briefly summarize the main here for illustration; a more detailed description of the experiments and analysis in the form of an iPython notebook can be found in the Gitlab repository or in Appendix C.Performance variation To estimate the random variation of , we repeatedly 8 trained models from two domains of deep learning: the ResNet BID12 on the CIFAR-100 dataset BID20 ) to represent Image Recognition and the Attention Sum Reader (AS Reader) BID19 on the Children's Book Test Common Nouns (CBT CN) BID15 to represent Reading Comprehension. Each of these trainings generated a pair of a validation and test performances. The ing empirical performance distributions are illustrated in FIG2.If we fix all hyperparameters, the interquartile ranges of the models' accuracies are 0.98(±0.09)% 9 and 1.20(±0.12)% (absolute). This is comparable to the median differences between published on these datasets: 0.86% and 1.15% respectively 10. Hence, random variation in performance cannot be considered negligible as is now often done. Furthermore, if we allow the hyperparameters to vary (in our case by random search), the variance further increases, which further amplifies the outlined effects. In the 8 Specifically, 74 times for Resnet, 370 times for the AS Reader with fixed hyperparameters, and 197 times for the AS Reader with random hyperparameters.9 ± standard deviation computed using 99999 Bootstrap samples. 10 We looked at of successive architectures evaluated on the two tasks as listed in BID16 BID21. We sorted the with respect to the test performance and then calculated the differences between successive models. From these we calculated the median. Full details can be found in the Gitlab repository.case of the AS Reader the interquartile range increased to 2.9(±0.3)% when we randomly picked hyperparameters from a range applicable to training the model on a single GPU. However, note that the problem of incomensurability due to hyperparameter optimization is not the focus of this work. The method that we present here is still applicable to the problem in the case of random hyperparameter sampling for which we include , however we aim to compensate mainly for randomness due to parameter initialization and data shuffling -which is significant in itself, as we have just demonstrated. Several other articles confirm significant variation in model performance due to different random seeds: e.g van den BID29 in Speech Recognition, BID13 in Deep Reinforcement Learning, or BID24 in Named Entity Recognition. They all agree that reporting performance scores of single models is insufficient to characterize architecture performance. FIG3 shows the 95% confidence intervals of best single model compared to the Boo 5 performance for a range of -pool sizes m. This is shown for the cases of both strong and weak test-validation correlation. In both cases Boo 5 is significantly less noisy than the best-single-model . In fact in the case of random hyperparameter search, Boo n shows even smaller variation than the mean (due to the negative skew of the performance distribution). Best-model performance improves with the number of experiments We also mentioned that if only the performance of the best model is reported, the more experiments are run, the better the expected . FIG3 illustrates that this effect can indeed be fairly strong, if the validation performance is a good predictor of the test performance, as is the case of the AS Reader with random hyperparameter search, where the expectation of the best single model performance increases from 61.3% if we train it once, to 63.3% if we train it 5 times, to 63.5% for 20 times. This effect is nicely explained in more detail e.g. by BID18. It gives a further argument for refraining from using this method and certainly also for publishing the number of experiments run, which is often not done. Boo n is not subject to this effect. Validation-test correlation However, note that the assumption that validation performance is a good predictor of test performance is sometimes not true. In the two cases with fixed hyperparameters that we looked at, the Spearman correlation between validation and test was only 0.10 and 0.18 respectively for the two models. The correlation significantly increases if we allow the hyperparameters to vary -to 0.83 for the AS Reader. These are also illustrated in FIG2. Larger validation sets are also likely to improve this correlation, which can be understood as the degree of generalization from validation to test. Note that the problem of increasing expected performance mentioned above is relevant only in the case of higher correlation between validation and test . The effect becomes very strong in the case where the performance we are reporting is also used for choosing the best model, which emphasizes the need for honest separation of validation and test data.11 While Boon and mean could be sampled using vanilla Bootstrap, best-validation is influenced only by a single value from the sample and hence uses only few values from the upper tier of our pool, which makes our pool size insufficient. Hence we use Gaussian kernel smoothing BID25 to expand our pool. Boo n does fix the main flaws of reporting the best single model performance. However, let us have a look at some of its limitations. Hyperparameter tuning This work does not fully compensate for improved expected due to hyperparameter tuning, nor was it its primary aim. Boo n is appropriate in the case of random hyperparameter sampling, where the performances in different runs are independent. However, this is not the case for more advanced hyperparameter optimization methods. The primary focus of this work was on tackling variability due to random initialization, data shuffling, and similar sources, which we have shown to be significant in itself. Compensation for more advanced hyperparameter tuning (and ensuring the comparability of models in that case) is certainly a worthwhile area for future research. Mean, median, and other alternatives We do not claim our method to be strictly superior to traditional ways of aggregating , such as mean or quantiles. However, we have outlined a case where using Boo n is justified -situations where a final model to be deployed can be chosen from a pool of trained candidates. In such case, Boo n is easily interpretable and more informative than a performance of a typical model, expressed by mean or median. Hence, we think Boo n is a useful addition to the methodological toolbox along existing methods. Just a single number Boo n is still just a single number whose ability to characterize the performance distribution is limited by its single dimension. Paper authors should try to characterise the performance distribution as fully as possible, which may involve a histogram, mean, standard deviation, ideally along a dataset containing the of all experiments, from which an interested reader may be able to deduce whichever characteristic she finds interesting. Unfortunately, such characterization is usually lacking. However, alongside this detailed characterization, describing an architecture's performance by a single number still has its appeal, especially for the purpose of comparison among architectures and choosing the best one according to some criterion (in fact, each quantitative score can be understood as a proxy for ordering architectures with respect to such criterion of interest, such as expected performance of the best model out of n). We have explained why, in some cases, Boo n may be useful for such purpose. Computational cost Some may deem Boo n impractical due to its requirement to train architectures many times, which may be very expensive in some cases. However, stochasticity needs to be addressed to produce reliable , and it is hard to imagine a general method to do so without repeated evaluation 12. Researchers should focus on architectures which they can evaluate properly given their resources. However, the main target of our criticism is not projects whose resources are stretched by a single training; it is projects that do have the necessary resources for multiple evaluations but use them to produce better-looking rather than that are more informative and robust. Reporting just the best single model performance is not statistically sound. This practice in machine learning research needs to change if the research is to have lasting value. Reviewers can play an important role in bringing this change. Still, asking for the performance of a best model out of n can have valid reasons. For the situations where the best-model performance is indeed a good metric, we are suggesting Boo n as a way to evaluate it properly. DISPLAYFORM0 where Φ is the c.d.f. of a standard normal random variable. DISPLAYFORM1 (the first integrand has the form of the p.d.f. found above and hence integrates to one) so the expected maximum is neatly expressed in terms of a maximum of a standard normal and is linearly proportional to both the mean and the standard deviation. Once n is fixed for comparison purposes, Boo n (N) is just a constant, e.g. Boo 5 FIG2 ) ≈ 1.163, Boo 10 (N) ≈ 1.539. Let us turn to the case of reporting the expected test set performance of a best-validation model. If we model the validation and test performances by a Bivariate Normal Distribution with valid-test correlation ρ, means µ val, µ test, and variances σ 2 val, σ 2 test, then given a validation performance x val, the test performance is distributed normally with conditional expectation DISPLAYFORM0 Using the same two tricks as above, this can be simplified to DISPLAYFORM1 where Boo n FIG2 ) is the single-evaluation expected maximum of the standard normal distribution as defined above. We downloaded the pdfs of all papers accepted to ICLR 2017 13, extracted text from them using the OpenSource Xpdf package 14 and then searched the ing text documents using the grep command as follows. Firstly, to roughly estimate the usage of experiments in the papers, we searched for the capitalized string "EXPERI-MENT" in the documents, since all (sub-)section headings are capitalized in the ICLR format. This was matched in 174 documents. Further 6 contained the string "EVALUA-TION" yielding a total of 180 out of 194 papers containing one of the two strings, which suggests that many ICLR papers indeed have an empirical component, though our rough method is only very approximate. We then searched for the string "confidence interval", which was matched in only 11 papers, and further 11 documents matched one of expressions related to hypothesis testing (curiously, a set completely disjoint from the "confidence interval" set). These terms were: "hypothesis test", "p-value", "t-test" "confidence level", "significance level", "ANOVA", "analysis of variance", "Wilcoxon", and "sign test". This may actually be only an upper bound since mentioning the term somewhere in the paper does not necessarily mean that the method was employed in the experimental procedure. Note: The data and code for their analysis are available at http://gitlab.com/obajgar/boon.Here we provide further details of our experiments quantifying the extent of stochasticity and the ing effects. To run our experiments we have chosen Open Source implementations 15 of models from two popular domains of deep learning, namely ResNet BID12 on the CIFAR-100 dataset BID20 for Image Classification and the AS Reader BID19 on the CBT CN dataset BID15 for Reading Comprehension. We believe these two models are representative of models in their respective areas -Resnet is based on a deep convolutional network architecture as most recent models in machine vision, while the AS Reader is based on a bidirectional GRU network with attention, as is the case for many models in natural language processing. To collect the data for our experiments, we repeatedly trained the two models. Each training instance had a different random parameter initialization and random data shuffling. We saved the model at least once per epoch and we then included the validation and test from the best-validation evaluation as a single data point in our metadataset. All training was done on Ubuntu 14.04 on a single GPU per training, either Nvidia Tesla K80 or GTX 1080. Resnet was trained with a single set of hyperparameters, the default ones for the above Open Source implementation. That means 5 residual units ing in a 32-layer Resnet. The model was trained using the 0.9 momentum optimizer, with batch size 128, initial learning rate of 0.1 lowered to 0.01 after 40,000 steps and to 0.001 after 60,000 steps. Data augmentation included padding to 36x36 and then random cropping, horizontal flipping and per-image whitening. L2 regularization weight was set 0.002. The training ran for 300 epochs. Training was done using Tensorflow 1.3. The AS Reader was trained in two different settings. Firstly 370 times with hyperparameters fixed to embedding dimension of 128 and 384 hidden dimensions in the GRU units, with all other hyperparameters as used in the original AS Reader paper BID19.In the second setting, the hyperparameters for each training instance were chosen randomly from the following ranges: The batch size was chosen from the range, and the embedding size and hidden state size were each chosen from the range with the log 2 value of the parameter being distributed uniformly in the interval. The upper bounds of these intervals matched maximum training size feasible on our hardware using this implementation. Training was done using Theano 0.9.0 and Blocks 0.2.0. FIG2 plots the histograms of test performances of the evaluated models. The mean test accuracy for Resnet was 68.41% with standard deviation of 0.67% (absolute), the range was 67.31% − 69.41%. For AS reader with fixed hyperparameters the mean was 63.16% with standard deviation 0.94% and range of 61.52% − 64.60%. In the case of random hyperparameter search the mean was 61.26%, standard deviation 2.48%, and values ranged from 56.61% to 64.01%. In both cases with fixed hyperparameters the collected are consistent with coming from a Gaussian distribution according to the Anderson-Darling test 16 BID0; the histograms also make it appear plausible that the performance distribution is approximately Gaussian. This is not the case for the random hyperparameter search where the distribution has a clear negative skew. To put the above numbers into context, we also examined the margin of improvement of successive architectures published on the corresponding datasets, as listed in BID21 BID16. We sorted the with respect to the test performance and then calculated the differences between successive models. The median difference was for 0.86% for CIFAR-100 and 1.15% for CBT CN.Note that the median differences are smaller than two standard deviations for each model. Two standard deviations from the mean approximately give the 95% confidence interval for a Gaussian distribution -hence we could typically fit three successive published within the width of one such confidence interval. The magnitude of the performance variation due to random initialization and data shuffling is therefore not negligible compared to the improvements in performance, which often hold an important place within articles in which they are presented. We hence think it is 16 That is, despite the relatively large sample sizes, gaussianity cannot be ruled out at 0.05 significance level based on collected evidence.inappropriate to completely ignore this random variation in evaluation protocols, which is currently the usual practice. The best model is usually selected using validation performance 17. This practice is based on the assumption that the validation accuracy is a reasonably good predictor of test accuracy. The of our experiments, illustrated also in FIG2, suggest that this assumption holds for performance variation due to hyperparameter choice. However, if we fix the hyperparameters, the correlation almost disappears. To some extent, this implies that selecting the best validation model means we are picking randomly with respect to the test performance. Since we are picking from a random test performance distribution, this further calls for better characterization of the distribution than a single instance drawn from it. On the other hand if the correlation is strong, as seems to be the case if we do perform hyperparameter search, we face the second problem with reporting the best-validation performance: If the validation performance is a good predictor of the test performance, then the more models we train the better the best-validation model is likely to be even on the test set since we are able to select models high up the right tail of the performance distribution. This effect has been described in more detail in BID18, though with focus on induction algorithms; here we present an estimate of its effect in the case of Resnet and AS Reader. To test this effect we took the pool of trained models. For each m in the range from 1 to 50 (or 100 for the AS Reader), we randomly sampled 100, 000 samples of size m from the pool, and selected the best-validation model from each sample. The mean test performance across the 100, 000 samples for each m is plotted in FIG3.The show that when there is suitable correlation between validation and test performances, increasing the number of experiments does increase the expected performance of the best-validation model. This makes the number of experiments an important explanatory variable, which, however, usually goes unreported. Furthermore, it makes reported by different research teams not directly comparable. Finally, it gives an advantage to those that can run more experiments. We believe that this again makes the practice of reporting the performance of the best single model unsuitable. 17 Or at least should be. If an estimator characterizing a performance distribution, say Boo n or average, is calculated from experimental observations, it is subject to random variation, so if another research team tries to reproduce the experiments, they generally get a different estimate. The more observations are collected, the more precise the estimate generally is. Confidence intervals provide a natural way to express this uncertainty. Their usage also gives a sense whether the number of performed experiments was sufficient to reduce the uncertainly to a reasonable level, which is again not frequently addressed in machine learning papers. The construction of the confidence interval would be trivial if we knew the distribution from which our estimate was drawn (as opposed to the distribution of the performance!) -it is simply the interval between the appropriate quantiles, e.g. the 2.5th and 97.5th quantiles in the case of the 95% confidence interval. Such distribution has been studied extensively for instance in the case of a mean of Gaussian random variables. However, in other cases, it is not known. If we know at least the distribution from which the individual observations were drawn, we can use Monte Carlo methods to precisely estimate the confidence interval; however, if we are not able to make an assumption about the underlying distribution, we need to use only what we have: our samples from the distribution. In such case the variability of our estimator can be approximated using the Bootstrap BID8 or similar methods. The Bootstrap consists of repeatedly sampling with replacement m random observations from our pool of m observations, say B times. Each such sample is then used to calculate an estimate of our quantity of interest, say Boo n or mean. This creates a sample of B values of the estimator. The confidence interval can then be easily estimated taking the appropriate quantiles from this ing Bootstrap distribution of the estimator, which should be approximating the unknown underlying sampling distribution. The Bootstrap distribution has been shown to converge to the true underlying performance distribution. If we know the underlying distribution (up to some parameters), we can estimate its parameters and then generate a simulated Monte Carlo sample from the distribution, which can be used to calculate a sample of the estimator and the corresponding confidence interval in a similar way as above with the advantage of the distribution being smoother. Beside estimating the confidence interval for the value of Boo n or mean itself, either re-sampling method can be used to construct a confidence interval for the relative improvement of the newly proposed architecture compared to a baseline. The improvement can then be considered significant if zero is not included in the confidence interval. More details on constructing Bootstrap confidence intervals can be found in many standard texts on computational statistics, for instance in BID9.For illustration, we calculated the Bootstrap confidence interval for several sample sizes m for Resnet and the AS Reader. Each was constructed using B = 100, 000. The are plotted in FIG3. FIG4 shows the comparison of the non-parametric and Gaussian parametric estimators of Boo n, both introduced in Section 3.2, in terms of their variance for various sample sizes. The parametric estimator shows a somewhat lower variance. This is an advantage if the performance distribution is indeed approximately Gaussian, which is the case for both cases with fixed hyperparameters that we tested in our experiments. However, this can introduce bias if the true performance distribution differs from the theoretical distribution assumed by a parametric estimator, so one should be prudent to use it.

represent the tldr for finding the corresponding scientific article

We point out important problems with the common practice of using the best single model performance for comparing deep learning architectures, and we propose a method that corrects these flaws.

Represent the TLDR for finding the corresponding scientific article

We introduce a statistical approach to assessing neural network robustness that provides an informative notion of how robust a network is, rather than just the conventional binary assertion of whether or not of property is violated.