Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Deep neural networks (DNN) are widely used in many applications. However, their deployment on edge devices has been difficult because they are resource hungry. Binary neural networks (BNN) help to alleviate the prohibitive resource requirements of DNN, where both activations and weights are limited to 1-bit. We propose an improved binary training method (BNN+), by introducing a regularization function that encourages training weights around binary values. In addition to this, to enhance model performance we add trainable scaling factors to our regularization functions. Furthermore, we use an improved approximation of the derivative of the sign activation function in the backward computation. These additions are based on linear operations that are easily implementable into the binary training framework. We show experimental on CIFAR-10 obtaining an accuracy of 86.5%, on AlexNet and 91.3% with VGG network. On ImageNet, our method also outperforms the traditional BNN method and XNOR-net, using AlexNet by a margin of 4% and 2% top-1 accuracy respectively. Deep neural networks (DNNs) have demonstrated success for many supervised learning tasks ranging from voice recognition to object detection BID26 BID11. The focus has been on increasing accuracy, in particular for image tasks, where deep convolutional neural networks (CNNs) are widely used. However, their increasing complexity poses a new challenge, and has become an impediment to widespread deployment in many applications; specifically when trying to deploy such models to resource constrained and lower-power devices. A typical DNN architecture contains tens to thousands of layers, ing in millions of parameters. As an example, AlexNet BID16 requires 200MB of memory, VGGNet BID26 requires 500MB memory. Large model sizes are further exacerbated by their computational cost, requiring GPU implementation to allow real-time inference. Such requirements evidently cannot be accustomed by edge devices as they have limited memory, computation power, and battery. This motivated the community to investigate methods for compressing and reducing computation cost of DNNs. To make DNNs compatible with the resource constraints of low power devices, there have been several approaches developed, such as network pruning BID17, architecture design BID25, and quantization BID0 BID4. In particular, weight compression using quantization can achieve very large savings in memory, where binary (1-bit), and ternary (2-bit) approaches have been shown to obtain competitive accuracy BID10 BID31 BID29. Using such schemes reduces model sizes by 8x to 32x depending on the bit resolution used for computations. In addition to this, the speed by quantizing the activation layers. In this way, both the weights and activations are quantized so that one can replace the expensive dot products and activation function evaluations with bitwise operations. This reduction in bit-width benefits hardware accelerators such as FPGAs and neural network chips. An issue with using low-bit DNNs is the drastic drop in accuracy compared to its full precision counterpart, and this is made even more severe upon quantizing the activations. This problem is largely due to noise and lack of precision in the training objective of the networks during back-propagation BID19. Although, quantizing the weights and activations have been attracting large interests thanks to their computational benefits, closing the gap in accuracy between the full precision and the quantized version remains a challenge. Indeed, quantizing weights cause drastic information loss and make neural networks harder to train due to a large number of sign fluctuations in the weights. Therefore, how to control the stability of this training procedure is of high importance. In theory, it is infeasible to back-propagate in a quantized setting as the weights and activations employed are discontinuous and discrete. Instead, heuristics and approximations are proposed to match the forward and backward passes. Often weights at different layers of DNNs follow a certain structure. How to quantize the weights locally, and maintaining a global structure to minimize a common cost function is important BID18.Our contribution consists of three ideas that can be easily implemented in the binary training framework presented by BID10 to improve convergence and generalization accuracy of binary networks. First, the activation function is modified to better approximate the sign function in the backward pass, second we propose two regularization functions that encourage training weights around binary values, and lastly a scaling factor is introduced in both the regularization term as well as network building blocks to mitigate accuracy drop due to hard binarization. Our method is evaluated on CIFAR-10 and ImageNet datasets and compared to other binary methods. We show accuracy gains to traditional binary training. We focus on challenges present in training binary networks. The training procedure emulates binary operations by restricting the weights and activations to single-bit so that computations of neural networks can be implemented using arithmetic logic units (ALU) using XNOR and popcount operations. More specifically, XNOR and popcount instructions are readily available on most CPU and GPU processing units. Custom hardware would have to be implemented to take advantage of operations with higher bits such as 2 to 4 bits. The goal of this binary training is to reduce the model size and gain inference speedups without performance degradation. Primary work done by BID0 (BinaryConnect) trains deep neural networks with binary weights {−1, +1}. They propose to quantize real values using the sign function. The propagated gradient applies update to weights |w| ≤ 1. Once the weights are outside of this region they are no longer updated, this is done by clipping weights between {−1, +1}. In that work, they did not consider binarizing the activation functions. BNN BID10 ) is the first purely binary network quantizing both the weights and activations. They achieve comparable accuracy to their prior work on BinaryConnect, and achieve significantly close performance to full-precision, by using large and deep networks. Although, they performed poorly on large datasets like ImageNet BID24. The ing network presented in their work obtains 32× compression rate and approximately 7× increase in inference speed. To alleviate the accuracy drop of BNN on larger datasets, BID23 proposed XNORNet, where they strike a trade-off between compression and accuracy through the use of scaling factors for both weights and activation functions. In their work, they show performance gains compared to BNN on ImageNet classification. The scaling factors for both the weights and activations are computed dynamically, which slows down training performance. Further, they introduced an additional complexity in implementing the convolution operations on the hardware, slightly reducing compression rate and speed up gains. DoReFa-Net BID30 further improves XNOR-Net by approximating the activations with more bits. The proposed rounding mechanism allows for low bit back-propagation as well. Although they perform multi-bit quantization, their model still suffers from large accuracy drop upon quantizing the last layer. Later in ABC-Net, BID29 propose several strategies, adjusting the learning rate for larger datasets. They show BNN achieve similar accuracy as XNOR-Net without the scaling overhead by adding a regularizer term which allows binary networks to generalize better. They also suggest a modified BNN, where they adopted the strategy of increasing the number of filters, to compensate for accuracy loss similar to wide reduced-precision networks BID21. More recently, developed a second-order approximation to the sign activation function for a more accurate backward update. In addition to this, they pre-train the network in which they want to binarize in full precision using the hard tangent hyperbolic (htanh) activation, see FIG0. They use the pre-trained network weights as an initialization for the binary network to obtain state of the art performance. Training a binary neural network faces two major challenges: on weights, and on activation functions. As both weights and activations are binary, the traditional continuous optimization methods such as SGD cannot be directly applied. Instead, a continuous approximation is used for the sign activation during the backward pass. Further, the gradient of the loss with respect to the weights are small. So as training progresses weight sign remains unchanged. These are both addressed in our proposed method. In this section, we present our approach to training 1-bit CNNs in detail. We quickly revisit quantization through binary training as first presented by BID0. In BID10, the weights are quantized by using the sign function which is +1 if w > 0 and −1 otherwise. In the forward pass, the real-valued weights are binarized to w b, and the ing loss is computed using binary weights throughout the network. For hidden units, the sign function non-linearity is used to obtain binary activations. Prior to binarizing, the real weights are stored in a temporary variable w. The variables w are stored because one cannot back-propagate through the sign operation as its gradient is zero everywhere, and hence disturbs learning. To alleviate this problem the authors suggest using a straight through estimator BID7 for the gradient of the sign function. This method is a heuristic way of approximating the gradient of a neuron, DISPLAYFORM0 where L is the loss function and 1 is the indicator function. The gradients in the backward pass are then applied to weights that are within [−1, +1]. The training process is summarized in Figure 1. As weights undergo gradient updates, they are eventually pushed out of the center region and instead make two modes, one at −1 and another at +1. This progression is also shown in Figure 1. DISPLAYFORM1 Figure 1: Binary training, where arrows indicate operands flowing into operation or block. Reproduced from (left). A convolutional layer depicting weight histogram progression during the popular binary training. The initial weight distribution is a standard Gaussian (right). Our first modification is on closing the discrepancy between the forward pass and backward pass. Originally, the sign derivative is approximated using the htanh(x) activation, as in FIG0. Instead, we modify the Swish-like activation BID22 BID1 BID6, where it has shown to outperform other activations on various tasks. The modifications are performed by taking its derivative and centering it around 0 DISPLAYFORM0 where σ(z) is the sigmoid function and the scale β > 0 controls how fast the activation function asymptotes to −1 and +1. The β parameter can be learned by the network or be hand-tuned as a hyperparameter. As opposed to the Swish function, where it is unbounded on the right side, the modification makes it bounded and a valid approximator of the sign function. As a , we call this activation SignSwish, and its gradient is DISPLAYFORM1 which is a closer approximation function compared to the htanh activation. Comparisons are made in FIG0. BID10 noted that the STE fails to learn weights near the borders of −1 and +1. As depicted in FIG0, our proposed SignSwish activation alleviates this, as it remains differentiable near −1 and +1 allowing weights to change signs during training if necessary. Note that the derivative d dx SS β (x) is zero at two points, controlled by β. Indeed, it is simple to show that the derivative is zero for x ≈ ±2.4/β. By adjusting this parameter beta, it is possible to adjust the location at which the gradients start saturating. In contrast to the STE estimators, where it is fixed. Thus, the larger β is, the closer the approximation is to the derivative of the sign function. DISPLAYFORM2 In general, a regularization term is added to a model to prevent over-fitting and to obtain robust generalization. The two most commonly used regularization terms are L 1 and L 2 norms. If one were to embed these regularization functions in binary training, it would encourage the weights to be near zero; though this does not align with the objective of a binary network. Instead, it is important to define a function that encourages the weights around −1 and +1. Further, in BID23 they present a scale to enhance the performance of binary networks. This scale is computed dynamically during training, using the statistics of the weights. To make the regularization term more general we introduce scaling factors α, ing in a symmetric regularization function with two minimums, one at −α and another at +α. As these scales are introduced in the regularization function and are embedded into the layers of the network they can be learned using backpropagation. The Manhattan regularization function is defined as DISPLAYFORM0 whereas the Euclidean version is defined as DISPLAYFORM1 where α > 0 is the scaling factor. As depicted in Figure 3, in the case of α = 1 the weights are penalized at varying degrees upon moving away from the objective quantization values, in this case, {−1, +1}.The proposed regularizing terms are inline with the wisdom of the regularization function R(w) = (1 − w 2)1 {|w|≤1} as introduced in BID29. A primary difference are in introducing a trainable scaling factor, and formulating it such that the gradients capture appropriate sign updates to the weights. Further, the regularization introduced in BID29 does not penalize weights that are outside of [−1, +1]. One can re-define their function to include a scaling factor as R(w) = (α − w 2)1 {|w|≤α}. In Figure 3, we depict the different regularization terms to help with intuition.−3 −2 −1 1 2 3 −3 −2 −1 1 2 3 Figure 3: R 1 (w) (left) and R 2 (w) (right) regularization functions for α = 0.5 (solid line) and α = 1 (dashed line). The scaling factor α is trainable, as a the regularization functions can adapt accordingly. Combining both the regularization and activation ideas, we modify the training procedure by replacing the sign backward approximation with that of the derivative of SS β activation. During training, the real weights are no longer clipped as in BNN training, as the network can back-propagate through the SS β activation and update the weights correspondingly. Additional scales are introduced to the network, which multiplies into the weights of the layers. The regularization terms introduced are then added to the total loss function, DISPLAYFORM0 where L(W, b) is the cost function, W and b are the sets of all weights and biases in the network, W l is the set weights at layer l and α l is the corresponding scaling factor. Here, R is the regularization function or. Further, λ controls the effect of the regularization term. To introduce meaningful scales, they are added to the basic blocks composing a typical convolutional neural network. For example, for convolutions, the scale is multiplied into the quantized weights prior to the convolution operation. Similarly, in a linear layer, the scales are multiplied into the quantized weights prior to the dot product operation. This is made more clear in the training algorithm 1.The scale α is a single scalar per layer, or as proposed in BID23 is a scalar for each filter in a convolutional layer. For example, given a CNN block with weight dimensionality (C in, C out, H, W), where C in is the number of input channels, C out is the number of output channels, and H, W, the height and width of the filter respectively, then the scale parameter would be a vector of dimension C out, that factors into each filter. As the scales are learned jointly with the network through backpropagation, it is important to initialize them appropriately. In the case of the Manhattan penalizing term, given a scale factor α and weight filter then the objective is to minimize DISPLAYFORM1 The minimum of the above is obtained when DISPLAYFORM2 Similarly, in the case of the Euclidean penalty the minimum is obtained when α * = mean(|W |) The scales are initialized with the corresponding optimal values after weights have been initialized first. One may notice the similarity of these optimal values with that defined by BID23, whereas in their case the optimal value for the weight filters and activations better matches the R 2 (w) goal. A difference is on how these approximations are computed, in our case they are updated on the backward pass, as opposed to computing the values dynamically. The final ing BNN+ training method is defined in Algorithm 1. In the following section, we present our experimental and important training details. Algorithm 1 BNN+ training. L is the unregularized loss function.λ and R 1 are the regularization terms we introduced. SS β is the SignSwish function we introduced and (SS β) is its derivative. N is the number of layers.• indicates element-wise multiplication. BatchNorm specifies how to batchnormalize the activation and BackBatchNorm how to back-propagate through the normalization. ADAM specifies how to update the parameters when their gradients are known. Require: a minibatch of inputs and targets (x 0, x *), previous weights W, previous weights' scaling factors α, and previous BatchNorm parameters θ. Ensure: updated weights W t+1, updated weights' scaling factors α t+1 and updated BatchNorm parameters θ t+1. {1. Forward propagation:} s 0 ← x 0 W 0 {We do not quantize the first layer.} DISPLAYFORM3 {We use our modified straight-through estimator to back-propagate through sign: DISPLAYFORM4 We evaluate our proposed method with the accuracy performance of training using BNN+ scheme versus other proposed binary networks, BID10 ; BID23 ; BID29 . We run our method on CIFAR-10 and ImageNet datasets and show accuracy gains. They are discussed in their respective sections below. The CIFAR-10 data BID15) consists of 50,000 train images and a test set of 10,000. For pre-processing the images are padded by 4 pixels on each side and a random crop is taken. We train both, AlexNet BID16, and VGG BID26 using the ADAM optimizer. The architecture used for VGG is conv → conv → conv → conv → conv → conv → fc → fc FORMULA2 where conv(·) is a convolutional layer, and fc(·) is a fully connected layer. The standard 3 × 3 filters are used in each layer. We also add a batch normalization layer BID12 prior to activation. For AlexNet, the architecture from BID14 is used, and batch normalization layers are added prior to activations. We use a batch size of 256 for training. Many learning rates were experimented with such as 0.1, 0.03, 0.001, etc, and the initial learning rate for AlexNet was set to 10 −3, and 3 × 10 −3 for VGG. The learning rates are correspondingly reduced by a factor 10 every 10 epoch for 50 epochs. We set the regularization parameter λ to 10 −6, and use the regularization term as defined in. In these experiments weights are initialized using BID2 initialization. Further, the scales are introduced for each convolution filter, and are initialized by sorting the absolute values of the weights for each filter and choosing the 75 th percentile value. The are summarized in TAB0. The ILSVRC-2012 dataset consists of ∼ 1.2M training images, and 1000 classes. For pre-processing the dataset we follow the typical augmentation: the images are resized to 256 × 256, then are randomly cropped to 224 × 224 and the data is normalized using the mean and standard deviation statistics of the train inputs; no additional augmentation is done. At inference time, the images are first scaled to 256 × 256, center cropped to 224 × 224 and then normalized. We evaluate the performance of our training method on two architectures AlexNet and Resnet-18. Following previous work, we used batch-normalization before each activation function. Additionally, we keep the first and last layers to be in full precision, as we lose 2-3% accuracy otherwise. This approach is followed by other binary methods that we compare to BID10 BID23 BID29. The are summarized in TAB1. In all the experiments involving R 1 regularization we set the λ to 10 −7 and R 2 regularization to 10 −6. Also, in every network, the scales are introduced per filter in convolutional layers, and per column in fully connected layers. The weights are initialized using a pre-trained model with htan activation function as done in. Then the learning rate for AlexNet is set to 2.33 × 10 − 3 and multiplied by 0.1 at the 12 th, 18 th epoch for a total of 25 epochs trained. For the 18-layer ResNet the learning rate is started from 0.01 and multiplied by 0.1 at 10 th, 20 th, 30 th epoch. We proposed two regularization terms and and an activation term with a trainable parameter β. We run several experiments to better understand the effect of the different modifications to the training method, especially using different regularization and asymptote parameters β. The parameter β is trainable and would add one equation through back-propagation. However, we fixed β throughout our experiments to explicit values. The are summarized in TAB1.Through our experiments, we found that adding regularizing term with heavy penalization degrades the networks ability to converge, as the term would in total loss be largely due to the regu- larizing term and not the target cross entropy loss. Similarly, the regularizing term was set to small values in BID29. As a , we set λ with a reasonably small value 10 −5 − 10 −7, so that the scales move slowly as the weights gradually converge to stable values. Some preliminary experimentation was to gradually increase the regularization with respect to batch iterations updates done in training, though this approach requires careful tuning and was not pursued further. From TAB1, and referring to networks without regularization, we see the benefit of using SwishSign approximation versus the STE. This was also noted in, where their second approximation provided better . There is not much difference between using R 1 versus R 2 towards model generalization although since the loss metric used was the cross-entropy loss, the order of R 1 better matches the loss metric. Lastly, it seems moderate values of β is better than small or large values. Intuitively, this happens because for small values of β, the gradient approximation is not good enough and as β increases the gradients become too large, hence small noise could cause large fluctuations in the sign of the weights. We did not compare our network with that of as they introduce a shortcut connection that proves to help even the full precision network. As a final remark, we note that the learning rate is of great importance and properly tuning this is required to achieve convergence. Table 3 summarizes the best of the ablation study and compares with BinaryNet, XNOR-Net, and ABC-Net. Table 3: Comparison of top-1 and top-5 accuracies of our method BNN+ with BinaryNet, XNORNet and ABC-Net on ImageNet, summarized from TAB1. The of BNN, XNOR, & ABC-Net are reported from the corresponding papers BID23 BID10 BID29. Results for ABC-NET on AlexNet were not available, and so is not reported. To summarize we propose three incremental ideas that help binary training: i) adding a regularizer to the objective function of the network, ii) trainable scale factors that are embedded in the regularizing term and iii) an improved approximation to the derivative of the sign activation function. We obtain competitive by training AlexNet and Resnet-18 on the ImageNet dataset. For future work, we plan on extending these to efficient models such as CondenseNet BID9, MobileNets BID8, MnasNet BID28 and on object recognition tasks.

represent this text to find a much much longer article about it with say hundreds-thousands of words

The paper presents an improved training mechanism for obtaining binary networks with smaller accuracy drop that helps close the gap with it's full precision counterpart

Represent the short summary for finding the corresponding scientific article (long)


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Clustering is a fundamental machine learning method. The quality of its is dependent on the data distribution. For this reason, deep neural networks can be used for learning better representations of the data. In this paper, we propose a systematic taxonomy for clustering with deep learning, in addition to a review of methods from the field. Based on our taxonomy, creating new methods is more straightforward. We also propose a new approach which is built on the taxonomy and surpasses some of the limitations of some previous work. Our experimental evaluation on image datasets shows that the method approaches state-of-the-art clustering quality, and performs better in some cases. Clustering is one of the most fundamental unsupervised machine learning problems. Its main goal is to separate data into clusters of similar data points. Besides having its own applications, it is beneficial for multiple other fundamental tasks. For instance, it can serve for automatic data labeling for supervised learning and as a pre-processing step for data visualization and analysis. However, the performance of clustering algorithms is dependent on the type of the input data, such that different problems and datasets could require different similarity measures and different separation techniques. As a , dimensionality reduction and representation learning have been extensively used alongside clustering, in order to map the input data into a feature space where separation is easier with respect to the problem's context. Using deep neural networks (DNNs), it is possible to learn non-linear mappings allowing to transform the data into more clustering-friendly representations. In the past, dimensionality reduction (or representation learning) and clustering have been treated separately, and sequentially applied on the data BID3 BID22 BID23. However, recent research has shown that jointly optimizing for both problems can achieve decent BID20 BID28 BID29 BID13.One of our main contributions is the formulation of a taxonomy of methods that use deep learning for clustering. Our taxonomy facilitates the overview of existing methods and the creation of new ones by using the best properties of the existing ones in a modular manner. Based on the taxonomy, we propose a new method that combines advantageous properties of some existing methods. We use an autoencoder-based method for learning better representations of the data which are clustering-friendly, with a state-of-the-art training procedure. The training has two phases, the first one being standard autoencoder training with the mean squared error reconstruction loss, and the second one is based on a loss function combining the reconstruction loss and a clustering-specific loss. Moreover, in the second phase, we alternate between optimizing the network model, and updating the clustering assignments. The rest of the paper is organized as follows: the taxonomy of clustering with deep learning and the corresponding building blocks is described in Section 2. In Section 3, several related methods are briefly described and compared based on the taxonomy. Subsequently, in Section 4, a new method is proposed and discussed based on the building blocks of the taxonomy. Results of the proposed method are shown in Section 5, followed by in Section 6. The most successful methods for clustering with deep neural networks all work following the same principle: representation learning using DNNs and using these representations as input for a specific clustering method. Every method consists of the following parts, for each of which there are several options to choose from:• Neural network training procedure, consisting of the following:-Main neural network branch and its usage * Architecture of main neural network branch, described in Section 2.1 * Set of deep features used for clustering, described in Section 2.2 -Neural network losses:* Non-clustering loss, described in Section 2.3 * Clustering loss, described in Section 2.4 * Method to combine the two losses, described in Section 2.5 -Cluster updates, described in Section 2.6• After the network training: re-run clustering (optional), described in Section 2.7 In most deep learning methods for clustering, the "main branch" of the neural network (apart from side branches towards non-clustering losses, see Section 2.3) is used to transform the inputs into a latent representation that is used for clustering. The following neural network architectures have previously been used for this purpose:• Multilayer perceptron (MLP): Feedforward network, consisting of several layers of neurons, such that the output of every hidden layer is the input to next one.• Convolutional neural network (CNN): Inspired by biology, more precisely by the organization of the animal visual cortex. Useful for applications to regular-grid data such as images, if locality and shift-equivariance/invariance of feature extraction is desired.• Deep belief network (DBN): Generative graphical model, consisting of several layers of latent variables. It is composed of several shallow networks such as restricted Boltzmann machines, such that the hidden layer of each sub-network serves as the visible layer of the next sub-network. DNNs serve for clustering as mappings to better representations. The features of these representations can be drawn from different layers of the network or even from several ones. It is possible to separate this choice into two categories:• One layer: Refers to the general case where only the output of the last layer of the network is used. This approach benefits from the low dimensionality of the representation.• Several layers: Refers to the case where the representation is a combination of the outputs of several layers. Based on that, the representation is richer and allows the embedded space to represent more complex semantic representations, which might enhance the separation process and help in the similarity computation BID19. The non-clustering loss is independent of the clustering algorithm and usually enforces a desired constraint on the learned model. The following are possible options for non-clustering loss functions:• No non-clustering loss: No additional non-clustering loss functions are used. In such cases, the network model is only constrained by the clustering loss requirements. For most clustering losses, no non-clustering loss can have a danger of worse representations/, or theoretically even collapsing clusters BID29, but the latter rarely occurs in practice.• Autoencoder reconstruction loss: The autoencoder consists of two parts: an encoder and a decoder. The encoder maps its input x to a representation z in a latent space Z. During training, the decoder tries to reconstruct x from z, making sure that useful information has not been lost by the encoding phase. In the context of clustering methods, once the training is done the decoder part is no longer used, and the encoder is left for mapping its input to the latent space Z. By applying this procedure, autoencoders can successfully learn useful representations in the cases where the output's dimensionality is different from the input's or when random noise is injected to the input BID25. Additionally, they can also be used for dimensionality reduction goals BID6. Generally the reconstruction loss is a distance measure d AE (x i, f (x i)) between the input x i to the autoencoder and the corresponding reconstruction f (x i). One particular formulation of it is using the mean squared error of the two variables: DISPLAYFORM0 where x i is the input and f (x i) is the autoencoder reconstruction. This loss function guarantees that the learned representation preserves important information from the initial one, which is why reconstruction is possible.• Other tasks: Additional information about training samples that is available in the form of targets, even if not perfectly suitable to dictate clustering, can be used in a (multi-task) non-clustering loss to encourage meaningful feature extraction. The second type of functions is specific to the clustering method and the clustering-friendliness of the learned representations, therefore such functions are called clustering loss functions. The following are options for clustering loss functions:• No clustering loss: Even if a neural network has only non-clustering losses (Section 2.3), the features it extracts can be used for clustering after training (Sections 2.6-2.7). The neural network serves in this case for changing the representation of the input, for instance changing its dimensionality. Such a transformation could be beneficial for the clustering sometimes, but using a clustering loss usually yields better BID28 BID29 ).• k-Means loss: Assures that the new representation is k-means-friendly BID29, i.e. data points are evenly distributed around the cluster centers. In order to obtain such a distribution a neural network is trained with the following loss function: DISPLAYFORM0 where z i is an embedded data point, µ k is a cluster center and s ik is a boolean variable for assigning z i with µ k. Minimizing this loss with respect to the network parameters assures that the distance between each data point and its assigned cluster center is small. Having that, applying k-means would in better clustering quality.• Cluster assignment hardening: Requires using soft assignments of data points to clusters. For instance, Student's t-distribution can be used as the kernel to measure the similarity (van der BID24 between points and centroids. This distribution Q is formulated as follows: DISPLAYFORM1 where z i is an embedded data point, µ j is the j th cluster centroid, and ν is a constant, e.g. ν = 1. These normalized similarities between points and centroids can be considered as soft cluster assignments. The cluster assignment hardening loss then enforces making these soft assignment probabilities stricter. It does so by letting cluster assignment probability distribution Q approach an auxiliary (target) distribution P which guarantees this constraint. BID28 propose the following auxiliary distribution: DISPLAYFORM2 By squaring the original distribution and then normalizing it, the auxiliary distribution P forces assignments to have stricter probabilities (closer to 0 and 1). It aims to improve cluster purity, put emphasis on data points assigned with high confidence and to prevent large clusters from distorting the hidden feature space BID28. One way to formulate the divergence between the two probability distributions is using the Kullback-Leibler divergence BID11. It is formulated as follows: DISPLAYFORM3 which is minimized for the aforementioned Q and P via neural network training.• Balanced assignments loss: This loss has been used alongside other losses such as the previous one BID4. Its goal is to enforce having balanced cluster assignments. It is formulated as follows: DISPLAYFORM4 where U is the uniform distribution and G is the probability distribution of assigning a point to each cluster: DISPLAYFORM5 By minimizing equation 6, the probability of assigning each data point to a certain cluster is uniform across all possible clusters BID4. It is important to note that this property (uniform assignment) is not always desired. Thus, in case any prior is known it is still possible to replace the uniform distribution by the known prior one.• Locality-preserving loss: This loss aims to preserve the locality of the clusters by pushing nearby data points together BID8. Mathematically, it is formulated as follows: DISPLAYFORM6 where N k (i) is the set of k nearest neighbors of the data point x i, and s(x i, x j) is a similarity measure between the points x i and x j.• Group sparsity loss: It is inspired by spectral clustering where block diagonal similarity matrix is exploited for representation learning BID17. Group sparsity is itself an effective feature selection method. In BID8, the hidden units were divided into G groups, where G is the assumed number of clusters. When given a data point x i the obtained representation has the form {φ DISPLAYFORM7 . Thus the loss can be defined as follows: DISPLAYFORM8 where DISPLAYFORM9 are the weights to sparsity groups, defined as DISPLAYFORM10 where n g is the group size and λ is a constant.• Cluster classification loss: Cluster assignments obtained during cluster updates (Section 2.6) can be used as "mock" class labels for a classification loss in an additional network branch, in order to encourage meaningful feature extraction in all network layers BID7 ).• Agglomerative clustering loss: Agglomerative clustering merges two clusters with maximum affinity (or similarity) in each step until some stopping criterion is fulfilled. A neural network loss inspired by agglomerative clustering BID30 ) is computed in several steps. First, the cluster update step (Section 2.6) merges several pairs [correct?] of clusters by selecting the pairs with the best affinity (some predefined measure of similarity between clusters). Then network training retrospectively even further optimizes the affinity of the already merged clusters (it can do so because the affinity is measured in the latent space to which the network maps). After the next cluster update step, the network training switches to retrospectively optimizing the affinity of the newest set of newly merged cluster pairs. In this way, cluster merging and retrospective latent space adjustments go hand in hand. Optimizing the network parameters with this loss function would in a clustering space more suitable for (agglomerative) clustering. In the case where a clustering and a non-clustering loss function are used, they are combined as follows: DISPLAYFORM0 where L c (θ) is the clustering loss, L n (θ) is the non-clustering loss, and α ∈ [0; 1] is a constant specifying the weighting between both functions. It is an additional hyperparameter for the network training. It can also be changed during training following some schedule. The following are methods to assign and schedule the values of α:• Pre-training, fine-tuning: First, α is set to 0, i.e. the network is trained using the nonclustering loss only. Subsequently, α is set to 1, i.e. the non-clustering network branches (e.g. autoencoder's decoder) are removed and the clustering loss is used to train (fine-tune) the obtained network. The constraint forced by the reconstruction loss could be lost after training the network long enough for clustering only. In some cases, losing such constraints may lead to worse (see TAB0).• Joint training: 0 < α < 1, for example α = 0.5, i.e. the network training is affected by both loss functions.• Variable schedule: α is varied during the training dependent on a chosen schedule. For instance, start with a low value for α and gradually increase it in every phase of the training. In phases with α = 1, no non-clustering loss is imposed, with potential disadvantages (see No nonclustering loss in Section 2.3). Similarly, in phases with α = 0, no clustering loss is imposed, with potential disadvantages (see No clustering loss in Section 2.4). Clustering methods can be broadly categorized into hierarchical and partitional (centroid-based) approaches BID10. Hierarchical clustering combines methods which aim to build a hierarchy of clusters and data points. On the other hand, partitional (centroid-based) clustering groups methods which create cluster centers and use metric relations to assign each of the data points into the cluster with the most similar center. In the context of deep learning for clustering, the two most dominant methods of each of these categories have been used. Agglomerative clustering, which is a hierarchical clustering method, has been used with deep learning BID30. The algorithm has been briefly discussed in Section 2.4. In addition, k-means, which falls into the category of centroid-based clustering, was extensively used BID28 BID29 BID13 BID7.During the network training, cluster assignments and centers (if a centroid-based method is used) are updated. Updating cluster assignments can have one of the two following forms:• Jointly updated with the network model: Cluster assignments are formulated as probabilities, therefore have continuous values between 0 and 1. In this case, they can be included as parameters of the network and optimized via back-propagation.• Alternatingly updated with the network model: Clustering assignments are strict and updated in a different step than the one where the network model is updated. In this case, several scenarios are possible, dependent on two main factors:-Number of iterations: Number of iterations of the chosen clustering algorithm, that are executed at every cluster update step. For instance, in BID28, at each cluster update step, the algorithm runs until a fixed percentage of points change assignments between two consecutive iterations. -Frequency of updates: How often are cluster updates started. For instance in BID30, for every P network model update steps, one cluster updates step happens. Once the training converges, the network should have learned a mapping from the input space to a more clustering-friendly space with respect to the dataset it was trained on. In other words, if the training was performed on digit images of N × N pixel size, the network should be able to map a set of N × N images to a space where clustering is easier. With such a mapping, it makes sense to run a clustering algorithm on a desired dataset. However, the majority of the presented methods performs clustering during the training and obtain their clustering from their last training iteration. Therefore the following are reasons for re-running the clustering after the training is done:• Clustering a similar dataset: The general and the most trivial case is to reuse the learned features representation mapping on another dataset which is similar to the one that has been used but has different data.• Obtaining better : Under certain circumstances, it is possible that the of clustering after the training are better than the ones obtained during the learning procedure. For instance, in BID30, such a behavior is reported. One possible reason for this to happen is that the cluster update step during the training doesn't go all the way till the end (see Number of iterations in Section 2.6) meaning that older steps used older representations that might be worse. Therefore, some of the cluster merging steps (agglomerative clustering) were performed on a less optimal feature representation, which is why clustering after the training performed better. Clustering has been extensively studied and researched. Its application with deep neural networks has gained additional interest in the last few years, due to the success of supervised deep learning. However, in most cases, clustering is handled in an unsupervised fashion, making its application with deep learning less trivial and requiring more modeling effort and theoretical analysis. Therefore, several approaches have been presented over the last years, trying to use the representational power of DNNs for preprocessing clustering inputs. Each of these approaches used different network architectures, structures, loss functions and training methods in order to achieve their and to improve the clustering quality. The following are some of the interesting methods that have been previously introduced. DEC is one of the most promising approaches in the field. It is based on autoencoders as network architecture and initialization method, and uses k-means for clustering BID28. As for training the neural network, the method first pretrains the model using a standard input reconstruction loss function. Secondly, the network's model is fine-tuned using the cluster assignment hardening loss and the clustering centers are updated. The clusters are iteratively refined by learning from their high confidence assignments with the help of the auxiliary target distribution. As a consequence, the method showed decent and has later been used as a reference to compare new methods performances. DCN is another autoencoder-based method that uses k-means for clustering BID29. Similar to DEC, in the first phase, the network is pretrained using the autoencoder reconstruction loss. However, the second phase is different. In contrast to DEC, the network is jointly trained using a mathematical combination of the autoencoder reconstruction loss and the k-means clustering loss function. Thus, due to the fact that strict cluster assignments were used during the training (instead of probabilities such as in DEC) the method required an alternation process between the network training and the cluster updates. The method performed well and even led to better than DEC on the MNIST dataset. With respect to the presented taxonomy, the approach in DBC BID13 is almost identical to DEC except for using convolutional autoencoders. Namely, it also uses k-means for clustering and the same training method: pretraining with autoencoder reconstruction loss and fine tuning using the cluster assignment hardening loss. Additionally, the same advantages and disadvantages are shared by both methods. Thus, due to the fact that DBC uses convolutional layers, it outperformed DEC's clustering quality on image datasets which was obviously expected. JULE uses a convolutional neural network for representation learning. For clustering, a hierarchical approach is used, specifically, the agglomerative clustering method is employed. Concerning the training, the method only uses a clustering loss, specifically, the agglomerative loss. Additionally, the method has a period hyper-parameter, by which the training behavior is altered. Namely, this hyper-parameter specifies the number of model updates that should be applied before the clustering algorithm executes a clustering iteration, for instance, ten learning sessions followed by fusing two clusters into one. In experiments, the method showed great , for example on MNIST, it performed better than all the other methods. However, the disadvantages of the lack of any nonclustering loss (see No non-clustering loss in Section 2.3) may be particularly pronounced, at least in theory BID29. CCNN uses a clustering CNN BID7 ) to achieve joint clustering and representation learning. One of the internal layers of the CCNN forms the feature space. At the same time, the CCNN's softmax layer predicts the cluster labels. Initially, features from k random images from the dataset are used to initialize the cluster centers. k-Means is performed on the features extracted from the input dataset to get corresponding cluster labels. Based on the assigned labels, and the labels predicted by the softmax layer, the CCNN parameters can be updated using the clustering classification loss discussed in section 2.4. The extracted features of the minibatch are then further used to update the corresponding cluster centroids. Besides the described methods, multiple attempts have been made in the field of clustering with deep learning. An interesting work is by BID19 where a standard autoencoder was used without additional clustering loss functions. However, the outputs of several layers of the network beside the last one are used as the final feature representation. This layer concatenation led to superior even when compared with methods which included a clustering-specific loss function. Moreover, in BID8, joint training was performed with a combination of an autoencoder reconstruction loss, a locality-preserving loss, and a group sparsity loss. Another work is by BID4, it is very similar to DEC, except for adding an additional term to the clustering loss which is a balanced assignments loss. By this addition, they alleviate the danger of obtaining degenerate solutions, but introduce again the need for alternating between network training and clustering updates. In addition to the mentioned methods, multiple others exist BID18 BID5 BID31 BID1 BID15 BID27 BID2.Rather than directly using a neural network to extract high-level features of samples, infinite ensemble clustering BID14 uses neural networks to generate infinite ensemble partitions and to fuse them into a consensus partition to obtain the final clustering. After identifying a taxonomy of clustering with deep learning (Section 2) and comparing methods in the field based on it TAB0, creating new improved methods became more straightforward. For instance, by looking at TAB0, one could notice that some combinations of method properties could lead to new methods. In some cases, such combinations could also surpass the limitations of the previous approaches and lead to better . This procedure was followed during this work. Namely, we picked an interesting combination of taxonomy features and came up with a new method FIG0.Our method uses a convolutional architecture, since our target clustering datasets are image datasets. Additionally, the network training has two phases. The first one is pretraining with an autoencoder reconstruction loss. In the second phase, the autoencoder loss and the cluster assignment hardening are jointly optimized. This second phase is different from DEC and DBC, which only use the cluster assignment hardening loss at this level. Omitting the reconstruction loss during one phase of the network training could lead to worse representations and solutions (see No non-clustering loss in Section 2.3). Therefore, combining the reconstruction loss with the cluster assignment hardening loss makes a lot more sense. This phase is also different from DCN, which has the joint training property, but uses the k-means loss. The k-means loss forces to alternate between joint training and clustering updates due to the hard cluster assignments. Using the cluster assignment hardening loss, this alternation procedure is no longer needed in our approach since this loss uses soft assignments which can be jointly updated with the network updates. Once both training phases are done, the network should be able to map its input into a more clustering-friendly space. Based on this assumption, we use the output of the network as the input to the k-means method which produces the final clustering . In this section we evaluate our model on real-world data and compare the against the methods previously discussed in section 3.Validation Metrics For evaluation, we use the clustering accuracy (ACC) and normalized mutual information (NMI) metrics BID21 BID26 BID0. These metrics lie in the range, with 1 being the perfect clustering, and 0 being the worst. Experimental Setup Training the network involved trying out several architectures and network sizes. In addition, it required tuning the learning hyper-parameters, such as the learning rate, initialization parameters, mini-batch size and others. In particular, we use a learning rate of 0.01 with a momentum of 0.9, in addition to batch normalization BID9 and L2 regularization. The presented are the best obtained ones during the experimentation phase. Datasets The experiments were performed on several publicly available datasets:• MNIST: Consists of 70000 images of hand-written digits of 28 × 28 pixel size. The digits are centered and size is normalized .• COIL20: Contains 1440, 32 × 32 gray scale images of 20 objects (72 images per object). The images of each object were taken 5 degrees apart BID16.Performance TAB0 shows the clustering performance in terms of accuracy and NMI for various clustering DNN approaches. The for all the methods are borrowed from their respective publications. From the table it can be seen the proposed algorithm performs comparable, if not better than a lot of state of the art approaches. Figure 2 and 3 show the clustering spaces at different stages of training the proposed network, with true cluster labels shown using different colors. The clustering spaces are 120-dimensional and 320-dimensional for MNIST and COIL20, respectively. It can be seen from the visualizations that the proposed method in much more clustering-friendly spaces than the original image space and the autoencoder space. In this work, we present a taxonomy for clustering with deep learning, identifying the general framework, and discussing different building blocks and possible options. In addition, a summary of methods in the field and their specific use of the taxonomy is presented alongside a general comparison of many of these methods. Using this taxonomy and the summary of previous methods, generating new methods is clearer and easier and can be done by creating new combinations of the taxonomy's building blocks. Moreover, we present a new method to the field, which is based on such a new combination. Our method overcomes the limitations of several previous ones, approaches state-ofthe-art performance and performs better in some cases.

Represent the TLDR for finding the corresponding scientific article.

Unifying framework to perform clustering using deep neural networks

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Generative models often use human evaluations to determine and justify progress. Unfortunately, existing human evaluation methods are ad-hoc: there is currently no standardized, validated evaluation that: measures perceptual fidelity, is reliable, separates models into clear rank order, and ensures high-quality measurement without intractable cost. In response, we construct Human-eYe Perceptual Evaluation (HYPE), a human metric that is grounded in psychophysics research in perception, reliable across different sets of randomly sampled outputs from a model, in separable model performances, and efficient in cost and time. We introduce two methods. The first, HYPE-Time, measures visual perception under adaptive time constraints to determine the minimum length of time (e.g., 250ms) that model output such as a generated face needs to be visible for people to distinguish it as real or fake. The second, HYPE-Infinity, measures human error rate on fake and real images with no time constraints, maintaining stability and drastically reducing time and cost. We test HYPE across four state-of-the-art generative adversarial networks (GANs) on unconditional image generation using two datasets, the popular CelebA and the newer higher-resolution FFHQ, and two sampling techniques of model outputs. By simulating HYPE's evaluation multiple times, we demonstrate consistent ranking of different models, identifying StyleGAN with truncation trick sampling (27.6% HYPE-Infinity deception rate, with roughly one quarter of images being misclassified by humans) as superior to StyleGAN without truncation (19.0%) on FFHQ. Historically, likelihood-based estimation techniques served as the de-facto evaluation metric for generative models BID18 BID5. But recently, with the application of generative models to complex tasks such as image and text generation BID14 BID34, likelihood or density estimation grew no longer tractable BID46. Moreover, for high-dimensional problems, even likelihood-based evaluation has been called into question BID46. Consequently, most generative tasks today resort to analyzing model outputs BID41 BID43 BID11 BID21 BID7 BID37. These output evaluation metrics consist of either automatic algorithms that do not reach the ideals of likelihood-based estimation, or ad-hoc human-derived methods that are unreliable and inconsistent BID41 BID11.Consider the well-examined and popular computer vision task of realistic face generation BID14. Automatic algorithms used for this task include Inception Score (IS) BID43 and Fréchet Inception Distance (FID) BID17. Both have been discredited for evaluation on non-ImageNet datasets such as faces BID2 BID40 BID6 BID38. They are also much more sensitive to visual corruptions such as salt and pepper noise than to semantic distortions such as swirled images BID17. So, while automatic metrics are consistent and standardized, they cannot fully capture the semantic side of perceptual fidelity BID6.Realizing the constraints of the available automatic metrics, many generative modeling challenges resort to summative assessments that are completely human BID41 BID43 BID11. These human measures are ad-hoc, each executed in idiosyncrasy without proof of reliability or grounding to theory, and high variance in their estimates BID43 BID11 BID33. These characteristics combine to a lack of reliability, and downstream, a lack of clear separability between models. Theoretically, given sufficiently large sample sizes of human evaluators and model outputs, the law of large numbers would smooth out the variance and reach eventual convergence; but this would occur at a high cost and a long delay. In this paper, we present HYPE (HUMAN EYE PERCEPTUAL EVALUATION) that addresses these criteria in turn. It: measures the perceptual fidelity of generative model outputs via a grounded method inspired by psychophysics methods in perceptual psychology, is a reliable and consistent estimator, is statistically separable to enable a comparative ranking, and ensures a cost and time efficient method through modern crowdsourcing techniques such as training and aggregation. We present two methods of evaluation. The first, called HYPE time, is drawn directly from psychophysics literature BID22 ) and displays images using adaptive time constraints to determine the time-limited perceptual threshold a person needs to distinguish real from fake BID9. The HYPE time score is understood as the minimum time, in milliseconds, that a person needs to see the model's output before they can distinguish it as real or fake. Small HYPE time scores indicate that model outputs can be identified even at a glance; large scores suggest that people need to dedicate substantial time and attention. The second method, called HYPE ∞, is derived from the first to make it simpler, faster, and cheaper while maintaining reliability. It measures human deception from fake images with no time constraints. The HYPE ∞ score is interpretable as the rate at which people mistake fake images and real images, given unlimited time to make their decisions. We demonstrate HYPE's performance on unconditional generation of human faces using generative adversarial networks (GANs) BID14. We evaluate four state-of-the-art GANs: WGAN-GP BID16, BEGAN BID4, ProGAN BID20, and the most recent StyleGAN BID21. First, we track progress across the years on the popular CelebA dataset BID28. We derive a ranking based on perception (HYPE time, in milliseconds) and error rate (HYPE ∞, as a percentage) as follows: StyleGAN (439.4ms, 50.7%), ProGAN (363.7ms, 40.3%), BEGAN (111.1ms, 10.0%), WGAN-GP (100.0ms, 3.8%). A score of 500ms on HYPE time indicates that outputs from the model become indistinguishable from real, when shown for 500ms or less, but any more would start to reveal notable differences. A score of 50% on HYPE ∞ represents indistinguishable from real, conditioned on the real training set, while a score above 50% through 100% represents hyper-realism in which generated images appear more real than real ones when drawn from a mixed pool of both. Next, we test StyleGAN trained on the newer FFHQ dataset BID21, comparing between outputs generated when sampled with and without the truncation trick, a technique used to prune low-fidelity generated images BID7 BID21. We find that outputs generated with the truncation trick (363.2ms, 27.6%) significantly outperforms those without it (240.7ms, 19.0%), which runs counter to scores reported by FID.HYPE indicates that GANs have clear, measurable perceptual differences between them. HYPE produces identical rankings between HYPE time and HYPE ∞. We also find that even the best eval- Images on the right exhibit the highest HYPE scores, the highest human perceptual fidelity. uated model, StyleGAN trained on FFHQ and sampled with the truncation trick, only performs at 27.6% HYPE ∞, suggesting substantial opportunity for improvement. Finally, we show that we can reliably reproduce these with 95% confidence intervals using 30 human evaluators at $60 in a task that takes 10 minutes. While important measures, we do not focus on diversity, overfitting, entanglement, training stability, and computational and sample efficiency of the model BID6 BID29 and instead aim to construct the gold standard for human perceptual fidelity. We deploy HYPE as a rapid solution for researchers to measure their generative models, requiring just a single click to produce reliable scores and measure progress. We deploy HYPE at https://hype.stanford.edu, where researchers can upload a model and retrieve a HYPE score in 10 minutes for $60. Future work would extend HYPE to adapt to other generative tasks such as text generation or abstractive summarization. Model creators can choose to perform two different evaluations and receive two different scores: the HYPE time score, which gathers time-limited perceptual thresholds to measure the psychometric function and report the minimum time people need to make accurate classifications, and the HYPE ∞ score, a simplified approach which assesses people's error rate under no time constraint. HYPE displays a series of images one by one to crowdsourced evaluators on Amazon Mechanical Turk and asks the evaluators to assess whether each image is real or fake. Half of the images are drawn from the model's training set (e.g., FFHQ or CelebA), which constitute the real images. The other half are drawn from the model's output. We use modern crowdsourcing training and quality control techniques to ensure high quality labels BID31. Our first method, HYPE time, measures time-limited perceptual thresholds. It is rooted in psychophysics literature, a field devoted to the study of how humans perceive stimuli, to evaluate human time thresholds upon perceiving an image. Our evaluation protocol follows the procedure known as the adaptive staircase method (see FIG1). An image is flashed for a limited length of time, after which the evaluator is asked to judge whether it is real or fake. If the evaluator consistently answers correctly, the staircase descends and flashes the next image with less time. If the evaluator is incorrect, the staircase ascends and provides more time. This process requires sufficient iterations to converge on the minimum time needed for each evaluator to sustain correct guesses in a sample-efficient manner BID9, producing what is known as the psychometric function BID51, the relationship of timed stimulus exposure to accuracy. For example, for an easily distinguishable set of generated images, a human evaluator would immediately drop to the lowest millisecond exposure. However, for a harder set, it takes longer to converge and the person would remain at a longer exposure level in order to complete the task accurately. The modal time value is the evaluator's perceptual threshold: the shortest exposure time at which they can maintain effective performance BID9 BID15 ). HYPE time displays three blocks of staircases for each evaluator. An image evaluation begins with a 3-2-1 countdown clock, each number displaying for 500 ms. The sampled image is then displayed for the current exposure time. Immediately after each image, four perceptual mask images are rapidly displayed for 30ms each. These noise masks are distorted to prevent visual afterimages and further sensory processing on the image afterwards BID15 ). We generate masks from the test images, using an existing texture-synthesis algorithm BID35. Upon each submission, HYPE time reveals to the evaluator whether they were correct. Image exposure times fall in the range [100ms, 1000ms], which we derive from the perception literature BID13. All blocks begin at 500ms and last for 150 images (50% generated, 50% real), values empirically tuned from prior work BID9 BID10. Exposure times are raised at 10ms increments and reduced at 30ms decrements, following the 3-up/1-down adaptive staircase approach. This 3-up/1-down approach theoretically leads to a 75% accuracy threshold that approximates the human perceptual threshold BID27 BID15 BID9.Every evaluator completes multiple staircases, called blocks, on different sets of images. As a , we observe multiple measures for the model. We employ three blocks, to balance quality estimates against evaluators' fatigue (BID25 BID42 . We average the modal exposure times across blocks to calculate a final value for each evaluator. Higher scores indicate a better model, whose outputs take longer time exposures to discern from real. Building on the previous method, we introduce HYPE ∞ : a simpler, faster, and cheaper method after ablating HYPE time to optimize for speed, cost, and ease of interpretation. HYPE ∞ shifts from a measure of perceptual time to a measure of human deception rate, given infinite evaluation time. The HYPE ∞ score gauges total error on the task, enabling the measure to capture errors on both fake and real images, and effects of hyperrealistic generation when fake images look even more realistic than real images. HYPE ∞ requires fewer images than HYPE time to find a stable value, at a 6x reduction in time and cost (10 minutes per evaluator instead of 60 minutes, at the same rate of $12 per hour). Higher scores are better, like HYPE time: a HYPE ∞ value of 10% indicates that only 10% of images deceive people, whereas 50% indicates that people are mistaking real and fake images at chance, rendering fake images indistinguishable from real. Scores above 50% suggest hyperrealistic images, as evaluators mistake images at a rate greater than chance, on average mistaking more fake images to be real than real ones and vice versa. HYPE ∞ shows each evaluator a total of 100 images: 50 real and 50 fake. We calculate the proportion of images that were judged incorrectly, and aggregate the judgments over the n evaluators on k images to produce the final score for a given model. To ensure that our reported scores are consistent and reliable, we need to sample sufficient model outputs, select suitable real images for comparison, and hire, qualify, and appropriately pay enough evaluators. To ensure a wide coverage of images, we randomly select the fake and real images provided to workers from a pool of 5000 images (see Sampling sufficient model outputs, below).Comparing between single evaluators can be problematic. To ensure HYPE is reliable, we must use a sufficiently large number of evaluators, n, which can be treated as a hyperparameter. To determine a suitable number, we use our experimental (further discussed in the Results section) to compute bootstrapped 95% confidence intervals (CI) across various values of n evaluators. To obtain a high-quality pool of evaluators, each is required to pass a qualification task. Such a pre-task filtering approach, sometimes referred to as a person-oriented strategy, is known to outperform process-oriented strategies that perform post-task data filtering or processing BID31. Our qualification task displays 100 images (50 real and 50 fake) with no time limits. Evaluators pass if they correctly classify 65% of both real and fake images. This threshold should be treated as a hyperparameter and may change depending upon the GANs used in the tutorial and the desired discernment ability of the chosen evaluators. We choose 65% based on the cumulative binomial probability of 65 binary choice answers out of 100 total answers: there is only a one in one-thousand chance that an evaluator will qualify by random guessing. Unlike in the staircase task itself, fake qualification images are drawn equally from multiple different GANs. This is to ensure an equitable qualification across all GANs, as to avoid a qualification that is biased towards evaluators who are particularly good at detecting one type of GAN. The qualification is designed to be taken occasionally, such that a pool of evaluators can assess new models on demand. Payment. Evaluators are paid a base rate of $1 for working on the qualification task. To incentivize evaluators to remained engaged throughout the task, all further pay after the qualification comes from a bonus of $0.02 per correctly labeled image. This pay rate typically in a wage of approximately $12 per hour, which is above a minimum wage in our local state. Sampling sufficient model outputs. The selection of K images to evaluate from a particular model is a critical component of a fair and useful evaluation. We must sample a large enough number of images that fully capture a model's generative diversity, yet balance that against tractable costs in the evaluation. We follow existing work on evaluating generative output by sampling K = 5000 generated images from each model BID43 BID32 BID49 and K = 5000 real images from the training set. From these samples, we randomly select images to give to each evaluator. Datasets. We evaluate on two datasets of human faces:1. CelebA-64 BID28 is popular dataset for unconditional image generation, used since 2015. CelebA-64 includes 202,599 images of human faces, which we align and crop to be 64 × 64 pixel images using a standard mechanism. We train all models without using attributes.2. FFHQ-1024 BID21 ) is a newer dataset released in 2018 with StyleGAN and includes 70,000 images of size 1024 × 1024 pixels. Architectures. We evaluate on four state-of-the-art models trained on CelebA-64: StyleGAN , ProGAN BID20, BEGAN BID4, and WGAN-GP BID16. We also evaluate on two types of sampling from StyleGAN trained on FFHQ-1024: with and without the truncation trick, which we denote StyleGAN trunc and StyleGAN no-trunc respectively. For parity on our best models across datasets, StyleGAN trained on CelebA-64 is sampled with the truncation trick. We train StyleGAN, ProGAN, BEGAN, and WGAN-GP on CelebA-64 using 8 Tesla V100GPUs for approximately 5 days. We use the official released pretrained StyleGAN model on FFHQ-1024 BID21.We sample noise vectors from the d-dimensional spherical Gaussian noise prior z ∈ R d ∼ N (0, I) during training and test times. We specifically opted to use the same standard noise prior for comparison, yet are aware of other priors that optimize for FID and IS scores BID7. We select training hyperparameters published in the corresponding papers for each model. We evaluate all models for each task with the two HYPE methods: HYPE time and HYPE ∞.Evaluator recruitment. We recruit 360 total human evaluators across our 12 evaluations, each of which included 30 evaluators, from Amazon Mechanical Turk. Each completed a single evaluation in {CelebA-64, FFHQ-1024} × {HYPE time, HYPE ∞}. To maintain a between subjects study in this evaluation, we did not allow duplicate evaluators across tasks or methods. In total, we recorded (4 CelebA-64 + 2 FFHQ-1024) models × 30 evaluators × 550 responses = 99, 000 total responses for our HYPE time evaluation and (4 CelebA-64 + 2 FFHQ-1024) models × 30 evaluators × 100 responses = 18, 000 total responses for our HYPE ∞ evaluation. Metrics. For HYPE time, we report the modal perceptual threshold in milliseconds. For HYPE ∞, we report the error rate as a percentage of images, as well as the breakdown of this rate on real and fake images individually. To show that our for each model are separable, we report a oneway ANOVA with Tukey pairwise post-hoc tests to compare all models within each {CelebA-64, FFHQ-1024} × {HYPE time, HYPE ∞} combination. As mentioned previously, reliability is a critical component of HYPE, as an evaluation is not useful if a researcher can re-run it and get a different answer. To show the reliability of HYPE, we use bootstrap BID12, a form of simulation, to simulate what the would be if we resample with replacement from this set of labels. Our goal is to see how much variation we may get in the outcome. We therefore report evaluator 95% bootstrapped confidence intervals, along with standard deviation of the bootstrap sample distribution. Confidence intervals (CIs) are defined as the region that captures where the modal exposure might be estimated to be if the same sampling procedure were repeated many times. For this and all following , bootstrapped confidence intervals were calculated by randomly sampling 30 evaluators with replacement from the original set of evaluators across 10, 000 iterations. Note that bootstrapped CIs do not represent that there necessarily exists substantial uncertainty-our reported modal exposure (for HYPE time) or detection rate (for HYPE ∞) is still the best point estimate of the value. We discuss bootstrapped CIs for other numbers of evaluators later on in the Cost Tradeoffs section. First, we report using the above datasets, models and metrics using HYPE time. Next, we demonstrate the HYPE ∞'s approximates the ones from HYPE time at a fraction of the cost and time. Next, we trade off the accuracy of our scores with time. We end with comparisons to FID. CelebA-64. We find that StyleGAN trunc ed in the highest HYPE time score (modal exposure time), at a mean of 439.3ms, indicating that evaluators required nearly a half-second of exposure to accurately classify StyleGAN trunc images (Table 1). StyleGAN trunc is followed by ProGAN at 363.7ms, a 17% drop in time. BEGAN and WGAN-GP are both easily identifiable as fake, so they are tied in third place around the minimum possible exposure time available of 100ms. Both BEGAN and WGAN-GP exhibit a bottoming out effect -reaching our minimum time exposure of 100ms quickly and consistently 1. This means that humans can detect fake generated images at 100ms and possibly lower. Thus, their scores are identical and indistinguishable. To demonstrate separability between StyleGAN trunc, ProGAN, BEGAN, and WGAN-GP together, we report from a one-way analysis of variance (ANOVA) test between all four models, where each model's input is the list of modes from each model's 30 evaluators. The ANOVA confirm that there is a statistically significant omnibus difference (F = 83.5, p < 0.0001). Pairwise post-hoc analysis using Tukey tests confirms that all pairs of models are separable (all p < 0.05), with the exception of BEGAN and WGAN-GP (n.s.).FFHQ-1024. We find that StyleGAN trunc ed in a higher exposure time than StyleGAN no-trunc, at 363.2ms and 240.7ms, respectively (Table 2). While the 95% confidence intervals that represent a very conservative overlap of 2.7ms, an unpaired t-test confirms that the difference between the two models is significant (t = 2.3, p = 0.02). Table 3: HYPE ∞ on four GANs trained on CelebA-64. Evaluators were deceived most often by StyleGAN trunc images, followed by ProGAN, BEGAN, and WGAN-GP. We also display the breakdown of the deception rate on real and fake images individually; counterintuitively, real errors increase with the errors on fake images, because evaluators become more confused and distinguishing factors between the two distributions become harder to discern. We observe a consistently separable difference between StyleGAN trunc and StyleGAN no-trunc and clear delineations between models TAB2. HYPE ∞ ranks StyleGAN trunc (27.6%) above StyleGAN no-trunc (19.0%) with no overlapping CIs. Separability is confirmed by an unpaired t-test (t = 8.3, p < 0.001). One of HYPE's goals is to be cost and time efficient. When running HYPE, there is an inherent tradeoff between accuracy and time, as well as between accuracy and cost. This is driven by the law of large numbers: recruiting additional evaluators in a crowdsourcing task often produces more consistent , but at a higher cost (as each evaluator is paid for their work) and a longer amount of time until completion (as more evaluators must be recruited and they must complete their work).To manage this tradeoff, we run an experiment with HYPE ∞ on StyleGAN trunc. We completed an additional evaluation with 60 evaluators, and compute 95% bootstrapped confidence intervals, choosing from 10 to 120 evaluators (Figure 4). We see that the CI begins to converge around 30 evaluators, our recommended number of evaluators to recruit and the default that we build into our system. As FID is one of the most frequently used evaluation methods for unconditional image generation, it is imperative to compare HYPE against FID on the same models (Table 5). We show through Spearman rank-order correlation coefficients that FID is correlated with neither human judgment measure, not HYPE time (ρ = −0.0286) nor with HYPE ∞ (ρ = −0.0857), where a Spearman correlation of -1.0 is ideal because lower FID and higher HYPE scores indicate stronger models. Meanwhile, HYPE time and HYPE ∞ exhibit strong correlation (ρ = 0.9429), where 1.0 is ideal because they are directly related. We calculate FID across the standard protocol of evaluating 50K generated and 50K real images for both CelebA-64 and FFHQ-1024, reproducing scores for StyleGAN no-trunc. Table 5: HYPE scores compared to FID. We put an asterisk on the most realistic GAN for each score (lower the better for FID, higher the better for HYPE). FID scores do not correlate fully with the human evaluation scores of HYPE ∞ on both CelebA-64 and FFHQ-1024 tasks. FID scores were calculated using 50K real (CelebA-64 or FFHQ-1024) and 50K generated images for each model. Cognitive psychology. We leverage decades of cognitive psychology to motivate how we use stimulus timing to gauge the perceptual realism of generated images. It takes an average of 150ms of focused visual attention for people to process and interpret an image, but only 120ms to respond to faces because our inferotemporal cortex has dedicated neural resources for face detection BID39 BID8. Perceptual masks are placed between a person's response to a stimulus and their perception of it to eliminate post-processing of the stimuli after the desired time exposure BID44. Prior work in determining human perceptual thresholds BID15 generates masks from their test images using the texture-synthesis algorithm BID35. We leverage this literature to establish feasible lower bounds on the exposure time of images, the time between images, and the use of noise masks. Success of automatic metrics. Common generative modeling tasks include realistic image generation BID14, machine translation BID0, image captioning BID48, and abstract summarization BID30, among others. These tasks often resort to automatic metrics like the Inception Score (IS) BID43 and Fréchet Inception Distance (FID) BID17 to evaluate images and BLEU BID34, CIDEr BID47 and METEOR BID1 scores to evaluate text. While we focus on how realistic generated content appears, other automatic metrics also measure diversity of output, overfitting, entanglement, training stability, and computational and sample efficiency of the model BID6 BID29 BID2. Our metric may also capture one aspect of output diversity, insofar as human evaluators can detect similarities or patterns across images. Our evaluation is not meant to replace existing methods but to complement them. Limitations of automatic metrics. Prior work has asserted that there exists coarse correlation of human judgment to FID BID17 and IS BID43, leading to their widespread adoption. Both metrics depend on the Inception v3 Network BID45, a pretrained ImageNet model, to calculate statistics on the generated output (for IS) and on the real and generated distributions (for FID). The validity of these metrics when applied to other datasets has been repeatedly called into question BID2 BID40 BID6 BID38. Perturbations imperceptible to humans alter their values, similar to the behavior of adversarial examples BID26. Finally, similar to our metric, FID depends on a set of real examples and a set of generated examples to compute high-level differences between the distributions, and there is inherent variance to the metric depending on the number of images and which images were chosen-in fact, there exists a correlation between accuracy and budget (cost of computation) in improving FID scores, because spending a longer time and thus higher cost on compute will yield better FID scores BID29. Nevertheless, this cost is still lower than paid human annotators per image. Human evaluations. Many human-based evaluations have been attempted to varying degrees of success in prior work, either to evaluate models directly BID11 BID33 or to motivate using automated metrics BID43 BID17. Prior work also used people to evaluate GAN outputs on CIFAR-10 and MNIST and even provided immediate feedback after every judgment BID43. They found that generated MNIST samples have saturated human performance-that is, people cannot distinguish generated numbers from real MNIST numbers, while still finding 21.3% error rate on CIFAR-10 with the same model BID43. This suggests that different datasets will have different levels of complexity for crossing realistic or hyper-realistic thresholds. The closest recent work to ours compares models using a tournament of discriminators BID33 BID24. The design would likely affect humans' absolute thresholds, as cognitive load may be of consideration; the number of humans required per task may require significant increase if evaluating fairly across all possible categories. Practically, the most valuable direction for the community to pursue with HYPE is likely one that includes the most difficult categories, especially when progress on those is hard to measure using automatic metrics. In the case of text generation (translation, caption generation), HYPE time may require much longer and much higher range adjustments to the perceptual time thresholds for text comprehensibility than those used in visual perception BID24.Future Work. We plan to extend HYPE to different imaging datasets and imaging tasks such as conditional image generation, as well as to text and video, such as translation BID34 and video captioning BID23. Future work would also explore budget-optimal estimation of HYPE scores and adaptive estimation of evaluator quality BID19. Additional improvements involve identifying images that require more evaluators BID50. We also aim to build in faster time exposures under 100ms -ideally down to 13ms, the minimum time exposure of human perception -for tasks that require that level of granularity. Doing so requires careful engineering solution, since 100ms appears to be the minimum time that is trustable before we are throttled by JavaScript paint and rendering times on modern browsers. We will investigate the ecological validity of our methods -that is, whether HYPE's evaluation is representative of how a person would perceive a GAN in everyday life. For instance, HYPE shows evaluators whether they classified an image correctly immediately after they answer. While this is standard practice in the psychophysics literature for staircase tasks, it likely does not reflect how one might encounter generated content in everyday life. Notably, in pilot studies, we found that without such feedback, evaluators were far less consistent and our metric would not be stable. Finally, we plan to investigate whether the reliability of HYPE may be impacted by the month or year at which it is run, as the population of available crowdsourced workers may differ across these factors. Anecdotally, we have found HYPE to be reliable regardless of the time of day.7 HYPE provides researchers with two human evaluation methods for GANs that are grounded in psychopisics to measure human perceptual fidelity directly, provide task designs that in consistent and reliable , distinguishes between different model performances through separable , is cost and time efficient. We report two metrics: HYPE time and HYPE ∞. HYPE time uses time perceptual thresholds where longer time constraints are more difficult to achieve because they give humans more time to interpret the generated content and observe artifacts. HYPE ∞ reports the error rate under unlimited time, where higher rates indicate a more realistic set of outputs. We demonstrate the efficacy of our approach on unconditional image generation across four GANs {StyleGAN, ProGAN, BEGAN, WGAN-GP} and two datasets of human faces {CelebA-64, FFHQ-1024}, with two types of output sampling on StyleGAN {with the truncation trick, without the truncation trick}. To encourage progress of generative models towards human-level visual fidelity, we deploy our evaluation system at https://hype.stanford.edu, so anyone can upload and evaluate their models based on HYPE at the click of a button. A. CONFIDENCE INTERVALS

represent the short summary for finding the corresponding scientific article (long):

HYPE is a reliable human evaluation metric for scoring generative models, starting with human face generation across 4 GANs.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Contrary to previous beliefs, the training performance of deep networks, when measured appropriately, is predictive of test performance, consistent with classical machine learning theory.