Question: Represent the scientific article to find a short TLDR

Question: Learning rules for neural networks necessarily include some form of regularization. Most regularization techniques are conceptualized and implemented in the space of parameters. However, it is also possible to regularize in the space of functions. Here, we propose to measure networks in an $L^2$ Hilbert space, and test a learning rule that regularizes the distance a network can travel through $L^2$-space each update. This approach is inspired by the slow movement of gradient descent through parameter space as well as by the natural gradient, which can be derived from a regularization term upon functional change. The ing learning rule, which we call Hilbert-constrained gradient descent (HCGD), is thus closely related to the natural gradient but regularizes a different and more calculable metric over the space of functions. Experiments show that the HCGD is efficient and leads to considerably better generalization. Large neural networks can overfit to training data, but we desire networks that instead learn general aspects that apply to new data. A learning rule can encourage generalization through regularization, which may be implicit to the learning rule or explicitly added to the cost function. Many regularization techniques introduce penalties in the space of network parameters. Canonical examples include weight decay and the constraints upon parameter movement inherent to Stochastic Gradient Descent (SGD). Ultimately, however, it is the complexity of the parameterized function that we seek to regularize, not pf the parameters themselves. A well-known example of this more direct approach is the natural gradient, which constrains how much the output distribution of a network can change due to an update. Here, we introduce a new learning rule that, like the natural gradient, limits how much the output function can change during learning. However, we use a different and more calculable metric over function space: the expected L 2 norm. Since the L 2 -space is a Hilbert space, we term the rule Hilbert-constrained gradient descent (HCGD).The interpretation of the natural gradient as ing from regularization competes in the literature with many other interpretations and justifications. In order to establish a foundation for the Hilbert constraint as analogous to the natural gradient, we begin by reviewing and discussing the natural gradient. In simplest mathematical terms, one performs the natural gradient by computing an expectation of the covariance of gradients E P θ [JJ T], and then multiplying the current gradients by its inverse. This covariance matrix is known as the Fisher information metric F, and the natural gradient step is in the direction F −1 J. In addition to being seen as a regularizer of functional change, as we make precise below, variants of the natural gradient have appeared with no less than four justifications. These are data efficiency, minimizing a regret bound during learning, speeding optimization, and the benefits of whitened gradients. We review these disparate methods to show that they are equivalent, and to emphasize how widespread the natural gradient is for the optimization of neural networks. Amari originally developed the natural gradient in the light of information geometry and efficiency BID2; ). If some directions in parameter space are more informative of the network's outputs than others, then we might wish to scale updates by each dimension's informativeness. In the terminology of Amari, this is to say that we want to find a learning rule that works when the loss geometry is not Euclidean but Riemannian . We can equivalently speak of the informativeness of new examples. If not all examples carry equal information about a distribution, then the update step should be modified to make use of highly informative examples. That is, we wish to find a Fisher-efficient algorithm (see BID3). The natural gradient uses the Fisher information matrix to scale the update by parameters' informativeness. The'adaptive' family of learning rules derive from a framework closely related to data efficiency. The original Adagrad paper showed that the F −1 J update reduces the bound on the regret relative to gradient descent BID5 ). There, however, F is referred to as the Malahanobis norm and is computed over the same examples as J. (Note that since the Fisher is properly taken in expectation over the output distribution, the F computed over the training batch is referred to elsewhere as the 'empirical Fisher'.) The number of parameters in neural networks typically prohibits calculating and storing a full gradient covariance matrix, let alone inverting one. Adagrad is thus most commonly implemented using a diagonal approximation of the empirical Fisher. Related adaptive methods like Adadelta and RMSprop can be also be seen as employing related diagonal approximations of the empirical Fisher . These update rules are in wide use in neural network optimization but not commonly referred to as approximations of the natural gradient. There is a strong connection between the natural gradient and techniques that normalize and whiten gradients. The term F −1 J, after all, simply ensures that steps are made in a parameter space that is whitened by the covariance of the gradients. Whitening the gradients thus has the effect that SGD becomes more similar to the natural gradient. Activation whitening methods are also known to speed convergence. It appears that many approaches to normalize and whiten activations or gradients have been forwarded in the literature BID15; BID18; BID17; BID4 BID19 BID10; BID16. A similar effect is at play for Batch Normalization, as well BID9 ). By normalizing and whitening the gradients, or by proxy, the activations, these various methods ensure that parameter space is a better proxy for function space. A good review of the thought behind recent natural gradient approximations can be found in BID12 and BID11. K-FAC is likely the most accurate scalable approximation of the natural gradient to date, reported to converge in 14x fewer iterations than SGD with momentum (and in less real time). Additionally, Pascanu and Bengio provide a clear exposition of the natural gradient as taking constant steps in the space of output distributions BID13 ), as measured by the Kullbeck-Leibler (KL) divergence. We present a version of this argument below but with emphasis on the interpretation as regularization. Here we show that the natural gradient is the optimal update when one penalizes the incremental change in a network's output distribution as measured by the KL divergence. A simple way to characterize a function is to examine the function's outputs for a set of inputs. Given an input distribution X, we will write the distribution of the network's outputs as P θ, where θ is the set of the network's parameters. Let us plan to regularize the change in the output distribution P θ throughout optimization of the parameters θ. For this, we need a measure of similarity between two distributions. One such measure is the Kullbeck-Leibler (KL) divergence. The KL divergence from P θt to P θt+1 is defined as DISPLAYFORM0 P θ is the density of P θ. To ensure the output distribution changes little throughout optimization, we define a new cost function DISPLAYFORM1 where C 0 is the original cost function and λ is a hyperparameter that controls the importance of this regularization term. Optimization would be performed with respect to the proposed update θ t+1.In information theory, the KL divergence from P θt to P θt+1 is called the relative entropy of P θt+1 with respect to P θt. We can thus equivalently speak of regularizing the change in the entropy of the output distribution throughout learning. If we assume that the initial parameterization leads to an output distribution with maximal entropy (as would be the case if, reasonably, the inputs are treated nearly equally by the initial parameterization), and the average update decreases output entropy, then regularizing the relative entropy between optimization steps will lead to a solution with higher entropy. Evaluating the KL divergence directly is problematic because it is infeasible to define the output density P θ everywhere. One can obtain a more calculable form by expanding D KL (P θt+1 P θt) around θ t to second order with respect to θ. The Hessian of the KL divergence is the Fisher information metric F. If J is the the Jacobian J = (∇ θ C 0), the Fisher metric is defined as E P θ [JJ T], i.e. the expectation (over the output distribution) of the covariance matrix of the gradients. With ∆θ ≡ (θ t+1 − θ t), we can rewrite our regularized cost function as DISPLAYFORM2 To optimize C via gradient descent we first replace C 0 with its first order approximation. DISPLAYFORM3 At each evaluation, J is evaluated before any step is made, and we seek the value of ∆θ that minimizes Equation 2. By setting the derivative with respect to ∆θ zero, we can see that this value is DISPLAYFORM4 When λ = 1 this update is exactly equal to the natural gradient. Thus, the natural gradient emerges as the optimal update when one explicitly regularizes the change in the output distribution during learning. The Fisher metric is not the only way to define (and regularize) a space of functions. We propose instead to use the L 2 space with norm: DISPLAYFORM0 Here µ is a measure and corresponds to the probability density of the input distribution X. The | · | 2 operator refers to the 2-norm to account for vector-valued functions. This norm leads to a notion of distance between two functions f and g given by DISPLAYFORM1 Since µ is a density, X dµ = 1, and we can write DISPLAYFORM2 We will regularize change of a network's output function as measured by this notion of distance. If a network would have been trained to adjust the parameters θ to minimize some cost C 0, we will instead minimize at each step t a new cost given by: DISPLAYFORM3 Like all regularization terms, this can also be viewed as a Langrangian that satisfies a constraint. Here, this constraint is upon gradient descent and ensures that change in L 2 -space does not exceed some constant value. To evaluate Equation 6, we can approximate the norm with an empirical expectation over X. DISPLAYFORM4 Here, the data x i may derive from some validation batch but must pull from the same distribution X. This cost function imposes a penalty upon the difference between the output of the current network at time t and the proposed network at t + 1. This cost implies a learning rule, which we call Hilbert-constrained gradient descent (HCGD).We can write an update rule to minimize Equation 6 that is a modification of gradient descent. Our implementation, displayed as Algorithm 1, takes some lessons from the natural gradient. Just as the natural gradient is the optimal solution to Equation 4 at each step, here we seek the optimal solution to Equation 6. Thus we seek to converge to a ∆θ at each update step, where DISPLAYFORM5 Minimization can be performed in an inner loop by a first order method. We first propose some ∆θ 0 = − J = − ∇ θ C 0 (for learning rate) and then iteratively correct this proposal by gradient descent towards ∆θ. If only one correction is performed, we can simply add the derivative of the Hilbert-constraining term after ∆θ 0 has been proposed. However it is possible that solving equation 7 to high precision is beneficial, so we include the possibility of multiple iterations in the algorithm. We found empirically that a single correction was often sufficient. In the appendix, we tighten the analogy between HCGD and the natural gradient by discussing how one can approximate the natural gradient with an inner first-order optimization loop. We discuss there that evaluating the Fisher and the gradient on the same batch of data causes poor behavior (see also BID13 BID11). Failure to use a different batch will in a type of overfitting, as gradients become, in a sense, judges of their own trustworthiness on the test set. We thus evaluate the empirical expectation of the L 2 distance on a validation batch (or, at least, a different batch than used for the initial proposed update). It would also be possible to use unlabeled data. Using a different batch of data for the inner loop ensures that the update does not overfit to the training set at the expense of its previous behavior. DISPLAYFORM6 ∆θ 0 ← −v Obtain proposed update via SGD with momentum 10: DISPLAYFORM7 First correction towards ∆θ 13: DISPLAYFORM8 Correct momentum buffer 14:for 1 < j < n do Optional additional corrections 15: DISPLAYFORM9 16: DISPLAYFORM10 17: DISPLAYFORM11 18: DISPLAYFORM12 return θ t SGD is commonly improved with momentum. Instead of following the instantaneous gradient J, the momentum procedure for SGD follows a'velocity' term v which is adjusted at each step with the rule v ← βv + J. To implement momentum for HCGD, we also keep a velocity term but update it with the final Hilbert-constrained update ∆θ rather than J. The velocity is used to propose the initial ∆θ 0 in the next update step. We found that this procedure both quickened optimization and lowered generalization error. Hilbert-constrained gradient descent is computationally cheaper than the exact natural gradient. We are not required to approximately invert any large matrix F, nor are we required to calculate the any per-example gradients separately. When the validation batch X V is drawn anew for each corrective iteration (step 8 in Algorithm 2), HCGD requires an additional two forward passes and one backwards pass for each correction i < n, for a total of 2 + 3n passes each outer step. This can be reduced by 1 pass for i ≥ 1 if X V is drawn anew just for i = 0. We demonstrate our method on training networks at the task of MNIST digit classification, with two network architectures, and on image classification in the CIFAR10 dataset. In all tests, we use a tuned learning rate for SGD, and then use the same learning rate for HCGD. We use values of λ = 0.5 and η = 0.02. (For the n = 1 version, λ can be folded into the inner learning rate η. Values were chosen so that λη = 0.01.) We chose the batch size for the "validation" batch to be 256. While the examples in each "validation" batch were different than the training batch, they were also drawn from the train set. All models were implemented in PyTorch BID14 ).We focus first on the clean example of training a dense multilayer perceptron without any modifications. We employ an 850 − 90 − 50 − 10 architecture with ReLU activations, and do not use dropout or batch normalization. The output is a softmax activation and the cost is the cross-entropy. As can be seen in Figure 1, HCGD notably improves performance on the test set. This is ture for both algorithms with momentum (1b) and without momentum (1c). We use = 0.04 with momentum and = 0.1 without momentum. The versions in which the gradient is corrected towards the ideal Hilbert-constrained update just once (n = 1) or many times (n = 10) behaved similarly. We use only the n = 1 version in future tests. Figure 1: Test and train accuracy on MNIST digit classification for a dense multilayer perceptron without dropout or normalization. Traces and envelopes represent the mean ± standard deviation of the traces of 10 runs. HCGD converges faster than SGD and generalizes better both for n = 1 and n = 10 corrections to the gradient. Both SGD and HCGD employ momentum (β) in (a) and (b), but use no momentum in (c).While HCGD converges more quickly than SGD, it requires a larger number of passes through the computational graph. In FIG0, we plot the same as Figure 1a,b but choose the total number of passes as the x-axis. It can be seen that when the number of inner-loop corrections n = 1, HCGD converges in similar compute time as SGD (but generalizes better).Next, we test the various optimizers at training a 3-layer convolutional neural network with Batch Normalization (BN) on MNIST digit classification. HCGD still outperforms standard SGD and ADAM, but by a much slimmer margin FIG1 ). Batch normalization has the effect that the activations, and thus the gradients, become normalized and whitened. For a BN network, then, Finally, we test the performance of HCGD as applied to the CIFAR10 image classification problem. We train a Squeezenet v1.1, a convolutional neural network model optimized for parameter efficiency BID8. As is common for training large models, we train at a large learning rate and then decrease the learning rate by a factor of 10 for fine tuning. When HCGD is trained with the same learning rate as SGD (initial = 0.1), it outperforms SGD while the learning rate is high, but performs worse than SGD once the learning rate is decreased (Figure 5). We suspect that this is because HCGD effectively reduces the learning rate, removing some of the positive annealing effects of a high initial learning rate. Note that we also decrease the inner learning rate η by a factor of 10. When we increase the initial learning rate such that the test error of HCGD matches that of SGD, the final train error decreases below that of SGD. Averaging the final 40 epochs to reduce noise, SGD achieves an mean error percentage of 8.00.01, while HCGD at = 0.3 achieved an error of 7.80.025 (with indicating the standard error of the mean). HCGD thus generally decreases the test error at a given learning rate, but needs to be trained at a higher learning rate to a achieve a given level of gradient noise. Figure 4: Results of a Squeezenet v1.1 trained on CIFAR10. The learning rate is decreased by a factor of 10 at epoch 150. HCGD requires a higher learning rate to a achieve a similar level of gradient noise, which is important to decrease overfitting. A central theme of this work is that regularization and optimization should occur in the space of functions, not merely the space of parameters. In this section we investigate this difference. In the space of parameters, SGD is a strongly local update rule. Large jumps are generally prohibited. SGD is thus more likely to find solutions that are close to the initialization, and furthermore to trace a path of limited length. This discourages the sampling a large volume of parameter space during optimization, which may lead to overfitting. This locality may partly explain the unexpected generalization abilities of SGD (e.g. BID21). Early stopping, which lowers generalization error, also limits great exploration. If SGD is successful due to its movement though parameter space, then it moves similarly in function space only to the extent to which there is a reasonably smooth mapping between these two spaces. Distances in these two spaces may still be qualitatively different. We can partially examine the difference between parameter and function space by examining how a network moves differently through them. In figure 5, we have plotted the cumulative squared distance traveled during optimization for an example network trained on MNIST. Figure 5a,c display the cumulative squared distance traveled (CSDT) in L 2 function space, while Figure 5b,d display the CSDT in parameter space. We first examine the behavior of SGD. While SGD moves more more slowly over time through parameter space (5c), reflecting a decreasing gradient, the CSDT of SGD in L 2 -space grows linearly for a large portion of optimization. Ideally, a network would cease to change before entering the overfitting regime. However, the network continues to drift through L 2 -space even after the test error saturates at around epoch 15 (see Figure 1). Note that SGD with momentum differs from plain SGD in that the scale of the total distance traveled is significantly reduced (Figure 5b,d). This effect could be due to a decreased level of noise in the update. Network drift in parameter space and in L 2 -space thus display qualitatively different behavior for SGD.Figure 5: The cumulative squared distance traveled through L 2 -space, top row, and through parameter space, bottom row, for an MLP trained on MNIST. It can be seen that SGD continues to drift in L 2 -space during the overfitting regime, while HCGD plateaus. This is true for networks trained with momentum, left column, and without momentum, right column. Note that momentum significantly decreases the scale of distance traveled. Individual traces represent random seeds. We measure distance in L 2 -space in the same manner as in the HCGD algorithm: by registering the Euclidean distance between the network's outputs on a single validation batch before and after an update. The HCGD algorithm is designed to reduce motion through L 2 -space. Figure 5a,b show that HCGD indeed greatly reduces motion through L 2 -space whether or not momentum is used. The plateau of the CSDT indicates that the function has converged to a single location; it ceases to change. SGD, on the other hand, does not converge to a single function even long after test error saturates. It is interesting to note that HCGD allows the parameters to continue to drift (5c,d) even though the function has generally converged. Neural networks encode functions, and it is important to consider the behavior of optimizers through the space of possible functions. The L 2 Hilbert space defined over distribution of input examples is a tractable and useful space for analysis. In this paper we propose to regularize the change in L 2 space between successive updates. The idea is to limit the movement of the function, just as gradient descent limits movement of the parameters. Our ing learning rule, Hilbert-constrained gradient descent (HCGD), increases test performance on standard image classification architectures. We hope that this work inspires more thought and analysis of behavior in L 2 -space. A alternative explanation of our algorithm is that it penalizes directions that are very sensitive controls of the outputs, similar to the natural gradient, while still allowing learning. In addition, since we evaluate the change in L 2 -space and the gradient on different data, HCGD asks the model to learn from current examples only in ways that will not affect what has already been learned from other examples. These intuitions are equivalent to the idea of limiting changes in L 2 -space. Given these empirical , it would be desirable to theoretically prove better generalization bounds for a method regularized in L 2 -space. One promising framework is stability analysis, which has recently been applied to establish some bounds on the generalization error of SGD itself BID7 ). It can be shown that generalization error is bounded by the stability of an algorithm, defined as the expected difference of the loss when two networks are trained on datasets that are identical except for one example. BID7 analyzes the stability of SGD in parameter space, then uses a Lipschitz condition to move to function space and bound the stability of the error. We expect that bounding the movement through L 2 -space leads to increased error stability compared to bounding movement through parameter space (as is done by SGD), simply by removing reliance on the assumed Lipschitz condition. We leave a proof of this idea to later work. It interesting to ask if there is support in neuroscience for learning rules that diminish the size of changes when that change would have a large effect on other tasks. It is unlikely that the nervous system performs precisely the natural gradient or HCGD, but there is some evidence that some analog is in play. One otherwise perplexing finding is that behavioral learning rates in motor tasks are dependent on the direction of an error but independent of the magnitude of that error BID6 ). This is not expected by most models of gradient descent, but would be expected if the size of the change in the output distribution (i.e. behavior) were regulated to be constant. Regularization upon behavior change (rather than synaptic change) would predict that neurons that are central to many actions, like neurons in motor pools of the spinal cord, would learn very slowly after early development, despite the fact that their gradient to the error on any one task (if indeed it is calculated) is likely to be quite large. Given our general resistance to overfitting during learning, and the great variety of roles of neurons, it is likely that some type of regularization of behavioral and perceptual change is at play. In order to better compare the natural gradient to the Hilbert-constrained gradient, we propose a natural gradient algorithm of a similar style. Previous work on the natural gradient has aimed to approximate F −1 as best and as cheaply as possible. This is equivalent to minimizing Equation 2 (i.e. J∆θ + λ 2 ∆θ T F ∆θ) with a single iteration of a second-order optimizer. For very large neural networks, however, it is much cheaper to calculate matrix-vector products than to approximately invert a large matrix. It is possible that the natural gradient may be more accessible via an inner gradient descent, which would be performed during each update step as an inner loop. We describe this idea at high level in Algorithm 2. After an update step is proposed by a standard optimizer, the algorithm iteratively corrects this update step towards the natural gradient. To start with a good initial proposed update, it is better to use a fast diagonal approximation of the natural gradient (such as Adagrad or RMSprop) as the main optimizer. Each additional correction requires just one matrix-vector product after the gradients are calculated. Depending on the quality of the proposed update, the number of iterations required is likely to be small, and even a small number of iterations will improve the update. Algorithm 2 Natural gradient by gradient descent. This algorithm can be paired with any optimizer to increase its similarity to the natural gradient. Require: n Number of corrective steps. ∆θ i+1 = ∆θ i − η(J + λF ∆θ i)Step towards θ ← θ + ∆θ 8:return θ tSince the Fisher matrix F can be calculated from the covariance of gradients, it never needs to be fully stored. Instead, for an array of gradients G of size (# parameters, # examples), we can write DISPLAYFORM0 The choice of G is an important one. It cannot be a vector of aggregated gradients (i.e. J), as that would destroy covariance structure and would in a rank-1 Fisher matrix. Thus, we must calculate the gradients on a per-example basis. To compute G efficiently it is required that a deep learning framework implement forward-mode differentiation, which is currently not supported in popular frameworks. If we choose G to be the array of per-example gradients on the minibatch, F is known as the'empirical Fisher'. As explained in BID11 and in BID13, the proper method is to calculate G from the predictive (output) distribution of the network, P θ (y|x). This can be done as in BID12 by sampling randomly from the output distribution and re-running backpropagation on these fictitious targets, using (by necessity) the activations from the minibatch. Alternatively, as done in BID13, one may also use unlabeled or validation data to calculate G on each batch.

It's important to consider optimization in function space, not just parameter space. We introduce a learning rule that reduces distance traveled in function space, just like SGD limits distance traveled in parameter space.

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO


Question: represent the article for finding a single-sentence tldr

Question: Stochastic gradient descent (SGD), which dates back to the 1950s, is one of the most popular and effective approaches for performing stochastic optimization. Research on SGD resurged recently in machine learning for optimizing convex loss functions and training nonconvex deep neural networks. The theory assumes that one can easily compute an unbiased gradient estimator, which is usually the case due to the sample average nature of empirical risk minimization. There exist, however, many scenarios (e.g., graphs) where an unbiased estimator may be as expensive to compute as the full gradient because training examples are interconnected. proposed using a consistent gradient estimator as an economic alternative. Encouraged by empirical success, we show, in a general setting, that consistent estimators in the same convergence behavior as do unbiased ones. Our analysis covers strongly convex, convex, and nonconvex objectives. We verify the with illustrative experiments on synthetic and real-world data. This work opens several new research directions, including the development of more efficient SGD updates with consistent estimators and the design of efficient training algorithms for large-scale graphs. Consider the standard setting of supervised learning. There exists a joint probability distribution P (x, y) of data x and associated label y and the task is to train a predictive model, parameterized by w, that minimizes the expected loss between the prediction and the ground truth y. Let us organize the random variables as ξ = (x, y) and use the notation (w; ξ) for the loss. If ξ i = (x i, y i), i = 1,..., n, are iid training examples drawn from P, then the objective function is either one of the following well-known forms: expected risk f (w) = E[(w; ξ)]; empirical risk f (w) = 1 n n i=1 (w; ξ i). Stochastic gradient descent (SGD), which dates back to the seminal work of , has become the de-facto optimization method for solving these problems in machine learning. In SGD, the model parameter is updated until convergence with the rule where γ k is a step size and g k is an unbiased estimator of the gradient ∇f (w k). Compared with the full gradient (as is used in deterministic gradient descent), an unbiased estimator involves only one or a few training examples ξ i and is usually much more efficient to compute. This scenario, however, does not cover all learning settings. A representative example that leads to costly computation of the unbiased gradient estimator ∇ (w, ξ i) is graph nodes. Informally speaking, a graph node ξ i needs to aggregate information from its neighbors. If information is aggregated across neighborhoods, ξ i must request information from its neighbors recursively, which in inquiring a large portion of the graph. In this case, the sample loss for ξ i involves not only ξ i, but also all training examples within its multihop neighborhood. The worst case scenario is that computing ∇ (w, ξ i) costs O(n) (e.g., for a complete graph or small-world graph), as opposed to O in the usual learning setting because only the single example ξ i is involved. In a recent work, proposed a consistent gradient estimator as an economic alternative to an unbiased one for training graph convolutional neural networks, offering substantial evidence of empirical success. A summary of the derivation is presented in Section 2. The subject of this paper is to provide a thorough analysis of the convergence behavior of SGD when g k in is a consistent estimator of ∇f (w k). We show that using this estimator in the same convergence behavior as does using unbiased ones. Definition 1. An estimator g N of h, where N denotes the sample size, is consistent if g N converges to h in probability: plim N →∞ g N = h. That is, for any > 0, lim N →∞ Pr(g N − h >) = 0. It is important to note that unbiased and consistent estimators are not subsuming concepts (one does not imply the other), even in the limit. This distinction renders the departure of our convergence , in the form of probabilistic bounds on the error, from the usual SGD that bound instead the expectation of the error. In what follows, we present examples to illustrate the distinctions between unbiasedness and consistency. To this end, we introduce asymptotic unbiasedness, which captures the idea that the bias of an estimator may vanish in the limit. Definition 2. An estimator g N of h, where N denotes the sample size, is asymptotically unbiased An estimator can be (asymptotically) unbiased but inconsistent. Consider estimating the mean h = µ of the normal distribution N (µ, σ 2) by using N independent samples X 1,..., X N. The estimator g N = X 1 (i.e., always use X 1 regardless of the sample size N) is clearly unbiased because E[X 1] = µ; but it is inconsistent because the distribution of X 1 does not concentrate around µ. Moreover, the estimator is trivially asymptotically unbiased. An estimator can be consistent but biased. Consider estimating the variance h = σ 2 of the normal distribution N (µ, σ 2) by using N independent samples X 1,..., X N. The estimator Hence, it is consistent owing to a straightforward invocation of the Chebyshev inequality, by noting that the mean approaches σ 2 and the variance approaches zero. However, the estimator admits a nonzero bias σ 2 /N for any finite N. An estimator can be consistent but biased even asymptotically. In the preceding example, the bias σ 2 /N approaches zero and hence the estimator is asymptotically unbiased. Other examples exist for the estimator to be biased even asymptotically. Consider estimating the quantity h = 0 with an estimator g N that takes the value 0 with probability (N − 1)/N and the value N with probability 1/N. Then, the probability that g N departs from zero approaches zero and hence it is consistent. However, E[g N] = 1 and thus the bias does not vanish as N increases. To the best of our knowledge, this is the first work that studies the convergence behavior of SGD with consistent gradient estimators, which from a real-world graph learning scenario that will be elaborated in the next section. With the emergence of graph deep learning models (; ; ; ; ; ; Velicković et al., 2018), the scalability bottleneck caused by the expensive computation of the sample gradient becomes a pressing challenge for training (as well as inference) with large graphs. We believe that this work underpins the theoretical foundation of the efficient training of a series of graph neural networks. The theory reassures practitioners of doubts on the convergence of their optimization solvers. Encouragingly, consistent estimators in a similar convergence behavior as do unbiased ones. The obtained here, including the proof strategy, offer convenience for further in-depth analysis under the same problem setting. This work opens the opportunity of improving the analysis, in a manner similar to the proliferation of SGD work, from the angles of relaxing assumptions, refining convergence rates, and designing acceleration techniques. We again emphasize that unbiasedness and consistency are two separate concepts; neither subsumes the other. One may trace that we intend to write the error bounds for consistent gradient estimators in a manner similar to the expectation bounds in standard SGD . Such a resemblance (e.g., in convergence rates) consolidates the foundation of stochastic optimization built so far. For a motivating application, consider the graph convolutional network model, GCN , that learns embedding representations of graph nodes. The l-th layer of the network is compactly written as where A is a normalization of the graph adjacency matrix, W (l) is a parameter matrix, and σ is a nonlinear activation function. The matrix H (l) contains for each row the embedding of a graph node input to the l-th layer, and similarly for the output matrix H (l+1). With L layers, the network transforms an initial feature input matrix H to the output embedding matrix H (L). For a node v, the embedding H (L) (v, :) may be fed into a classifier for prediction. Clearly, in order to compute the gradient of the loss for v, one needs the corresponding row of H (L), the rows of H (L−1) corresponding to the neighbors of v, and further recursive neighbors across each layer, all the way down to H. The computational cost of the unbiased gradient estimator is rather high. In the worst case, all rows of H are involved. To resolve the inefficiency, proposed an alternative gradient estimator that is biased but consistent. The simple and effective idea is to sample a constant number of nodes in each layer to restrict the size of the multihop neighborhood. For notational clarity, the approach may be easier to explain for a network with a single layer; theoretical for more layers straightforwardly follow that of Theorem 1 below, through induction. The approach generalizes the setting from a finite graph to an infinite graph, such that the matrix expression becomes an integral transform. In particular, the input feature vector H (u, :) for a node u is generalized to a feature function X(u), and the output embedding vector H (v, :) for a node v is generalized to an embedding function Z(v), where the random variables u and v in two sides of the layer reside in different probability spaces, with probability measures P (u) and P (v), respectively. Furthermore, the matrix A is generalized into a bivariate kernel A(v, u) and the loss is written as a function of the output Z(v). Then, and become Such a functional generalization facilitates sampling on all network layers for defining a gradient estimator. In particular, defining B(v) = A(v, u)X(u) dP (u), simple calculation reveals that the gradient with respect to the parameter matrix W is Then, one may use t iid samples of u in the input and s iid samples of v in the output to define an estimator of G: The gradient estimator G st so defined is consistent; see a proof in the supplementary material. Theorem 1. If q is continuous and f is finite, then plim s,t→∞ G st = G. We now settle the notations for SGD. We are interested in the (constrained) optimization problem where the feasible region S is convex. This setting includes the unconstrained case S = R d. We assume that the objective function f: R d → R is subdifferentiable; and use ∂f (w) to denote the subdifferential at w. When it is necessary to refer to an element of this set, we use the notation h. If f is differentiable, then clearly, ∂f (w) = {∇f (w)}. The standard update rule for SGD is w k+1 = Π S (w k − γ k g k), where g k is the negative search direction at step k, γ k is the step size, and Π S is the projection onto the feasible region: Π S (w):= argmin u∈S w − u. For unconstrained problems, the projection is clearly omitted: Denote by w * the global minimum. We assume that w * is an interior point of S, so that the subdifferential of f at w * contains zero. For differentiable f, this assumption simply means that ∇f (w *) = 0. Typical convergence are concerned with how fast the iterate w k approaches w *, or the function value f (w k) approaches f (w *). Sometimes, the analysis is made convenient through a convexity assumption on f, such that the average of historical function values f (w i), i = 1,..., k, is lowered bounded by f (w k), with w k being the cumulative moving average The following definitions are frequently referenced. Definition 3. We say that f is l-strongly convex (with l > 0) if for all w, u ∈ R d and h u ∈ ∂f (u), Recall that an estimator g N of h is consistent if for any > 0, In our setting, h corresponds to an element of the subdifferential at step k; i.e., h k ∈ ∂f (w k), g N corresponds to the negative search direction g k, and N corresponds to the sample size N k. That g converges to h k in probability does not imply that g N k k is unbiased. Hence, a natural question asks what convergence guarantees exist when using g N k k as the gradient estimator. This section answers that question. First, note that the sample size N k is associated with not only g We omit the superscript N k in these vectors to improve readability. Similar to the analysis of standard SGD, which is built on the premise of the unbiasedness of g k and the boundedness of the gradient, in the following subsection we elaborate the parallel assumptions in this work. They are stated only once and will not be repeated in the theorems that follow, to avoid verbosity. The convergence of the estimator does not characterize how fast it approaches the truth. One common assumption is that the probability in decreases exponentially with respect to the sample size. That is, we assume that there exists a step-dependent constant C k > 0 and a nonnegative function τ (δ) on the positive axis such that for all k > 1 and δ > 0. A similar assumption is adopted by that studied stochastic optimization through sample average approximation. In this case, the exponential tail occurs when the individual moment generating functions exist, a simple application of the Chernoff bound. For the motivating application GCN, the tail is indeed exponential as evidenced by Figure 3. Note the conditioning on the history g 1,..., g k−1 in. The reason is that h k (i.e., the gradient ∇f (w k) if f is differentiable) is by itself a random variable dependent on history. In fact, a more rigorous notation for the history should be filtration, but we omit the introduction of unnecessary additional definitions here, as using the notion g 1,..., g k−1 is sufficiently clear. Assumption 1. The gradient estimator g k is consistent and obeys. The use of a tail bound assumption, such as, is to reverse-engineer the required sample size given the desired probability that some event happens. In this particular case, consider the setting where T SGD updates are run. For any δ ∈, define the event Given and any ∈, one easily calculates that if the sample sizes satisfy for all k, then, Hence, all in this section are established under the event E δ that occurs with probability at least 1 −, a sufficient condition of which is. The sole purpose of the tail bound assumption is to establish the relation between the required sample sizes (as a function of δ and) and the event E δ, on which convergence in this work are based. One may replace the assumption by using other tail bounds as appropriate. It is out of the scope of this work to quantify the rate of convergence of the gradient estimator for a particular use case. For GCN, the exponential tail that agrees with is illustrated in Section 5.4. Additionally, parallel to the bounded-gradient condition for standard SGD analysis, we impose the following assumption. Assumption 2. There exists a finite G > 0 such that h ≤ G for all h ∈ ∂f (w) and w ∈ S. Let us begin with the strongly convex case. For standard SGD with unbiased gradient estimators, ample exist that indicate O(1/T) convergence 2 for the expected error, where T is the number of updates; see, e.g., (2.9)-(2.10) of and Section 3.1 of. We derive similar for consistent gradient estimators, as stated in the following Theorem 2. Different from the unbiased case, it is the error, rather than the expected error, to be bounded. The tradeoff is the introduction of the relative gradient estimator error δ, which relates to the sample sizes as in for guaranteeing satisfaction of the bound with high probability. Theorem 2. Let f be l-strongly convex with l ≤ G/ w 1 − w *. Assume that T updates are run, with diminishing step size γ k = [(l − δ)k] −1 for k = 1, 2,..., T, where δ = ρ/T and ρ < l is an arbitrary constant independent of T. Then, for any such ρ, any ∈, and sufficiently large sample sizes satisfying, with probability at least 1 −, we have and Note the assumption on l in Theorem 2. This assumption is mild since if f is l-strongly convex, it is also l -strongly convex for all l < l. The assumption is needed in the induction proof of when establishing the base case w 1 − w *. One may remove this assumption at the cost of a cumbersome right-hand side of, over which we favor a neater expression in the current form. With an additional smoothness assumption, we may eliminate the logarithmic factor in and obtain a for the iterate w T rather than the running average w T. The is a straightforward consequence of. Theorem 3. Under the conditions of Theorem 2, additionally let f be L-smooth. Then, for any ρ satisfying the conditions, any ∈, and sufficiently large sample sizes satisfying, with probability at least 1 −, we have In addition to O(1/T) convergence, it is also possible to establish linear convergence (however) to a non-vanishing right-hand side, as the following indicates. To obtain such a , we use a constant step size. show a similar for the function value with an additional smoothness assumption in a different setting; we give one for the iterate error without the smoothness assumption using consistent gradients. Theorem 4. Under the conditions of Theorem 2, except that one sets a constant step size γ k = c with 0 < c < (2l − δ) −1 for all k, for any ρ satisfying the conditions, any ∈, and sufficiently large sample sizes satisfying, with probability at least 1 −, we have Compare with in Theorem 2. The former indicates that in the limit, the squared iterate error is upper bounded by a positive term proportional to G 2; the remaining part of this upper bound decreases at a linear speed. The latter, on the other hand, indicates that the squared iterate error in fact will vanish, although it does so at a sublinear speed O(1/T). For convex (but not strongly convex) f, typically O(1/ √ T) convergence is asserted for unbiased gradient estimators; see., e.g., Theorem 2 of. These are often derived based on an additional assumption that the feasible region is compact. Such an assumption is not restrictive, because even if the problem is unconstrained, one can always confine the search to a bounded region (e.g., an Euclidean ball). Under this condition, we obtain a similar for consistent gradient estimators. Theorem 5. Let f be convex and the feasible region S have finite diameter D > 0; that is, sup w,u∈S w − u = D. Assume that T updates are run, with diminishing step size γ k = c/ √ k for k = 1, 2,..., T and for some c > 0. Let δ = ρ/ √ T where ρ > 0 is an arbitrary constant independent of T. Then, for any such ρ, any ∈, and sufficiently large sample sizes satisfying, with probability at least 1 −, we have One may obtain a of the same convergence rate by using a constant step size. In the case of unbiased gradient estimators, see Theorem 14.8 of. For such a , one assumes that the step size is inversely proportional to √ T. Such choice of the step size is common and is also used in the next setting. For the general (nonconvex) case, convergence is typically gauged with the gradient norm. One again obtains O(1/ √ T) convergence for unbiased gradient estimators; see, e.g., Theorem 1 of (which is a simplified consequence of the theory presented in). We derive a similar for consistent gradient estimators. Theorem 6. Let f be L-smooth and S = R d. Assume that T updates are run, with constant step size is an arbitrary constant. Then, for any such δ, any ∈, and sufficiently large sample sizes satisfying, with probability at least 1 −, we have All the in the preceding subsection assert convergence for SGD with the use of a consistent gradient estimator. As with the use of an unbiased one, the convergence for the strongly convex case is O(1/T), or linear if one tolerates a non-vanishing upper bound, and the convex and nonconvex cases O(1/ √ T). These theoretical , however, are based on assumptions of the sample size N k and the step size γ k that are practically challenging to verify. Hence, in a real-life machine learning setting, the sample size and the learning rate (the initial step size) are treated as hyperparameters to be tuned against a validation set. Nevertheless, these establish a qualitative relationship between the sample size and the optimization error. Naturally, to maintain the same failure probability, the relative gradient estimator error δ decreases inversely with the sample size N k. This intuition holds true in the tail bound condition with, when τ (δ) is a monomial or a positive combination of monomials with different degrees. With this assumption, the larger is N k, the smaller is δ (and also ρ, the auxiliary quantity defined in the theorems); hence, the smaller are the error bounds-. Theorem 4 presents a linear convergence for the strongly convex case, with a non-vanishing right-hand side. In fact, it is possible to obtain a with the same convergence rate but a vanishing right-hand side, if one is willing to additionally assume L-smoothness. The following theorem departs from the set of theorems in Section 4.2 on the assumption of the sufficient sample size N k and the gradient error δ. Theorem 7. Let f be l-strongly convex and L-smooth with l < L. Assume that T updates are run with constant step size γ k = 1/L for k = 1, 2,..., T. Let δ k, k ≥ 1 be a sequence where lim k→∞ δ k+1 /δ k ≤ 1. Then, for any positive η < l/L, ∈, and sample sizes with probability at least 1 −, we have where Here, δ k is the step-dependent gradient error. If it decreases to zero, then so does E T. Theorem 7 is adapted from , who studied unbiased gradients as well as noisy gradients. We separate Theorem 7 from those in Section 4.2 only for the sake of presentation clarity. The spirit, however, remains the same. Namely, consistent estimators in the same convergence behavior (i.e., rate) as do unbiased ones. All require an assumption on sufficient sample size owing to the probabilistic convergence of the gradient estimator. In this section, we report several experiments to illustrate the convergence behavior of SGD by using consistent gradient estimators. We base the experiments on the training of the GCN model motivated earlier (cf. Section 2). The code repository will be revealed upon paper acceptance. We use three data sets for illustration, one synthetic and two real-world benchmarks. The purpose of a synthetic data set is to avoid the regularity in the sampling of training/validation/test examples. The data set, called "Mixture," is a mixture of three overlapping Gaussians. The points are randomly connected, with a higher probability for those within the same component than the ones straddling across components. See the supplementary material for details of the construction. Because of the significant overlap, a classifier trained with independent data points unlikely predicts well the component label, but a graph-based method is more likely to be successful. Additionally, we use two benchmark data sets, Cora and Pubmed, often seen in the literature. These graphs are citation networks and the task is to predict the topics of the publications. We follow the split used in. See the supplementary material for a summary of all data sets. The GCN model is hyperparameterized by the number of layers. Without any intermediate layer, the model can be considered a generalized linear model and thus the cross-entropy loss function is convex. Moreover, with the use of an L 2 regularization, the loss becomes strongly convex. The predictive model reads P = softmax(AXW ), where X is the input feature matrix and P is the output probability matrix, both row-wise. One easily sees that the only difference between this model and logistic regression P = softmax(XW ) is the neighborhood aggregation AX. Standard batched training in SGD samples a batch (denoted by the index set I 1) from the training set and evaluates the gradient of the loss of softmax(A(I 1, :)XW ). In the analyzed consistentgradient training, we additionally uniformly sample the input layer with another index set I 0 and evaluate instead the gradient of the loss of softmax(Figure 1 shows the convergence curves as the iteration progresses. The plotted quantity is the overall loss on all training examples, rather than the batch loss for only the current batch. Hence, not surprisingly the curves are generally quite smooth. We compare standard SGD with the use of consistent gradient estimators, with varying sample size |I 0 |. Additionally, we compare with the Adam training algorithm , which is a stochastic optimization approach predominantly used in practice for training deep neural networks. One sees that for all data sets, Adam converges faster than does standard SGD. Moreover, as the sample size increases, the loss curve with consistent gradients approaches that with an unbiased one (i.e., standard SGD). This phenomenon qualitatively agrees with the theoretical ; namely, larger sample size improves the error bound. Note that all curves in the same plot from the same parameter initialization; and all SGD variants apply the same learning rate. It is important to note that the training loss is only a surrogate measure of the model performance; and often early termination of the optimization acts as a healthy regularization against over-fitting. In our setting, a small sample size may not satisfy the assumptions of the theoretical , but it proves to be practically useful. In Table 1 (left), we report the test accuracy attained by different training algorithms at the epoch where validation accuracy peaks. One sees that Adam and standard SGD achieves similar accuracies, and that SGD with consistent gradient sometimes surpasses these accuracies. For Cora, a sample size 400 already yields an accuracy noticeably higher than do Adam and standard SGD., and a GCN with more layers is analogous. We repeat the experiments in the preceding subsection. The are reported in Figure 2 and Table 1 (right). The observation of the loss curve follows the same as that in the convex case. Namely, Adam converges faster than does unbiased SGD; and the convergence curve with a consistent gradient approaches that with an unbiased one. On the other hand, compared with 1-layer GCN, 2-layer GCN yields substantially higher test accuracy for the data set Mixture, better accuracy for Cora, and very similar accuracy for Pubmed. Within each data set, the performances of different training algorithms are on par. In particular, a small sample size (e.g., 400) suffices for achieving comparable to the state of the art (cf.). The nature of a consistent estimator necessitates a characterization of the speed of probability convergence for building further , such as the ones in this paper. The speed, however, depends on the neural network architecture and it is out of the scope of this work to quantify it for a particular use case. Nevertheless, for GCN we demonstrate empirical findings that agree with the exponential tail assumption. In Figure 3 (solid curves), we plot the tail probability as a function of the sample size N at different levels of estimator error δ, for the initial gradient step in 1-layer GCN. For each N, 10,000 random gradient estimates were simulated for estimating the probability. Because the probability is plotted in the logarithmic scale, the fact that the curves bend down indicates that the convergence may be faster than exponential. Additionally, the case of 2-layer GCN is demonstrated by the dashed curves in Figure 3. The curves tend to be straight lines in the limit, which indicates an exponential convergence. To the best of our knowledge, this is the first work that studies the convergence behavior of SGD with consistent gradient estimators, and one among few studies of first-order methods that employ biased (d';) or noisy (; ;) estimators. The motivation originates from learning with large graphs and the main message is that the convergence behavior is well-maintained with respect to the unbiased case. While we analyze the classic SGD update formula, this work points to several immediate extensions. One direction is the design of more efficient update formulas resembling the variance reduction technique for unbiased estimators (; ;). Another direction is the development of more computation-and memory-efficient training algorithms for neural networks for large graphs. GCN is only one member of a broad family of message passing neural networks that suffer from the same limitation of neighborhood aggregation. Learning in these cases inevitably faces the costly computation of the sample gradient. Hence, a consistent estimator appears to be a promising alternative, whose construction is awaiting more innovative proposals. We are grateful to an anonymous reviewer who inspired us of an interesting use case (other than GCN). Learning to rank is a machine learning application that constructs ranking models for information retrieval systems. In representative methods such as RankNet and subsequent improvements , s i is the ranking function for document i and the learning amounts to minimizing the loss where the summation ranges over all pairs of documents such that i is ranked higher than j. The pairwise information may be organized as a graph and the loss function may be similarly generalized as a double integral analogous to. Because of nonlinearity, Monte Carlo sampling of each integral will in a biased but consistent estimator. Therefore, a new training algorithm is to sample i and j separately (forming a consistent gradient) and apply SGD. The theory developed in this work offers guarantees of training convergence. A.9 PROOF OF THEOREM 7 Theorem 2.2 of states that when the gradient error inequality holds. It remains to show that the probability that happens is at least 1 −. The assumption on the sample size N k means that Then, substituting δ k = δ h k into assumption yields Hence, the probability that happens is (1 − /T) ≥ 1 −, which concludes the proof. B EXPERIMENT DETAILS, and σ 3 = 0.25 are equally weighted but significantly overlap with each other. Random connections are made between every pair of points. For points in the same component, the probability that they are connected is p intra = 1e-3; for points straddle across components, the probability is p inter = 2e-4. See Figure 4 (a) for an illustration of the Gaussian mixture and Figure 4 (b) for the graph adjacency matrix. Table 2 for a summary of the data sets used in this work. Table 3 for the hyperparameters used in the experiments. For parameter initialization, we use the Glorot uniform initializer . B.4 RUN TIME See Table 4 for the run time (per epoch). As expected, a smaller sample size is more computationally efficient. SGD with consistent gradients runs faster than the standard SGD and Adam, both of which admit approximately the same computational cost.

Represent the TLDR for finding the corresponding scientific article

Convergence theory for biased (but consistent) gradient estimators in stochastic optimization and application to graph convolutional networks

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We consider the problem of uncertainty estimation in the context of (non-Bayesian) deep neural classification. In this context, all known methods are based on extracting uncertainty signals from a trained network optimized to solve the classification problem at hand. We demonstrate that such techniques tend to introduce biased estimates for instances whose predictions are supposed to be highly confident. We argue that this deficiency is an artifact of the dynamics of training with SGD-like optimizers, and it has some properties similar to overfitting. Based on this observation, we develop an uncertainty estimation algorithm that selectively estimates the uncertainty of highly confident points, using earlier snapshots of the trained model, before their estimates are jittered (and way before they are ready for actual classification). We present extensive experiments indicating that the proposed algorithm provides uncertainty estimates that are consistently better than all known methods. The deployment of deep learning models in applications with demanding decision-making components such as autonomous driving or medical diagnosis hinges on our ability to monitor and control their statistical uncertainties. Conceivably, the Bayesian framework offers a principled approach to infer uncertainties from a model; however, there are computational hurdles in implementing it for deep neural networks BID9. Presently, practically feasible (say, for image classification) uncertainty estimation methods for deep learning are based on signals emerging from standard (non Bayesian) networks that were trained in a standard manner. The most common signals used for uncertainty estimation are the raw softmax response BID4, some functions of softmax values (e.g., entropy), signals emerging from embedding layers BID20, and the MC-dropout method BID9 ) that proxies a Bayesian inference using dropout sampling applied at test time. These methods can be quite effective, but no conclusive evidence on their relative performance has been reported. A recent NIPS paper provides documentation that an ensemble of softmax response values of several networks performs better than the other approaches BID17.In this paper, we present a method of confidence estimation that can consistently improve all the above methods, including the ensemble approach of BID17. Given a trained classifier and a confidence score function (e.g., generated by softmax response activations), our algorithm will learn an improved confidence score function for the same classifier. Our approach is based on the observation that confidence score functions extracted from ordinary deep classifiers tend to wrongly estimate confidence, especially for highly confident instances. Such erroneous estimates constitute a kind of artifact of the training process with an stochastic gradient descent (SGD) based optimizers. During this process, the confidence in "easy" instances (for which we expect prediction with high confidence) is quickly and reliably assessed during the early SGD epochs. Later on, when the optimization is focused on the "hard" points (whose loss is still large), the confidence estimates of the easy points become impaired. Uncertainty estimates are ultimately provided in terms of probabilities. Nevertheless, as previously suggested BID10 BID20 BID17, in a non-Bayesian setting (as we consider here) it is productive to decouple uncertainty estimation into two separate tasks: ordinal ranking according to uncertainty, and probability calibration. Noting that calibration (of ordinal confidence ranking) already has many effective solutions BID21 BID23 BID27 BID11, our main focus here is on the core task of ranking uncertainties. We thus adopt the setting of BID17, and others BID10 BID20, and consider uncertainty estimation for classification as the following problem. Given labeled data, the goal is to learn a pair (f, κ), where f (x) is a classifier and κ(x) is a confidence score function. Intuitively, κ should assign lower confidence values to points that are misclassified by f, relative to correct classifications (see Section 2 for details).We propose two methods that can boost known confidence scoring functions for deep neural networks (DNNs). Our first method devises a selection mechanism that assigns for each instance an appropriate early stopped model, which improves that instance's uncertainty estimation. The mechanism selects the early-stopped model for each individual instance from among snapshots of the network's weights that were saved during the training process. This method requires an auxiliary training set to train the selection mechanism, and is quite computationally intensive to train. The second method approximates the first without any additional examples. Since there is no consensus on the appropriate performance measure for scoring functions, we formulate such a measure based on concepts from selective prediction BID10 BID26. We report on extensive experiments with four baseline methods (including all those mentioned above) and four image datasets. The proposed approach consistently improves all baselines, often by a wide margin. For completeness, we also validate our using probably-calibrated uncertainty estimates of our method that are calibrated with the well-known Platt scaling technique BID23 and measured with the negative log-likelihood and Brier score. In this work we consider uncertainty estimation for a standard supervised multi-class classification problem. We note that in our context uncertainty can be viewed as negative confidence and vice versa. We use these terms interchangeably. Let X be some feature space (e.g., raw image pixels) and Y = {1, 2, 3, . . ., k}, a label set for the classes. Let P (X, Y) be an unknown source distribution over X × Y. A classifier f is a function f: X → Y whose true risk w.r.t. DISPLAYFORM0 + is a given loss function, for example, the 0/1 error. Given a labeled set DISPLAYFORM1 We consider deep neural classification models that utilize a standard softmax (last) layer for multi-class classification. Thus, for each input x ∈ X, the vector f (x) = (f (x) 1,..., f (x) k ) ∈ R k is the softmax activations of the last layer. The model's predicted classŷ =ŷ f (x) = argmax i∈Y f (x) i.Consider the training process of a deep model f through T epochs using any mini-batch SGD optimization variant. For each 1 ≤ i ≤ T, we denote by f[i] a snapshot of the partially trained model immediately after epoch i. For a multi-class model f, we would like to define a confidence score function, κ(x, i, |f), where x ∈ X, and i ∈ Y. The function κ should quantify confidence in predicting that x is from class i, based on signals extracted from f. A κ-score function should induce a partial order over points in X, and thus is not required to distinguish between points with the same score. For example, for any softmax classifier f, the vanilla confidence score function is κ(x, i|f) = ∆ f (x) i (i.e., the softmax response values themselves). Perhaps due to the natural probabilistic interpretation of the softmax function (all values are non-negative and sum to 1), this vanilla κ has long been used as a confidence estimator. Note, however, that we are not concerned with the standard probabilistic interpretation (which needs to be calibrated to properly quantify probabilities BID11).An optimal κ (for f) should reflect true loss monotonicity in the sense that for every two labeled instances (x 1, y 1) ∼ P (X, Y), and (DISPLAYFORM2 In the domain of (deep) uncertainty estimation there is currently no consensus on how to measure performance (of ordinal estimators). For example, BID17 used the Brier score and the negative-log-likelihood to asses their , while treating κ values as absolute scores. In BID20 the area under the ROC curve was used for measuring performance. In this section we propose a meaningful and unitless performance measure for κ functions, which borrows elements from other known approaches. In order to define a performance measure for κ functions, we require a few concepts from selective classification BID7 BID25. As noted in , any κ function can be utilized to construct a selective classifier (i.e., a classifier with a reject option). Thus, selective classification is a natural application of confidence score functions based on which it is convenient and meaningful to assess performance. The structure of this section is as follows. We first introduce the (well known) terms selective classifier, selective risk and coverage. Then we introduce the risk-coverage curve. We propose to measure the performance of a κ function as the area under the risk-coverage curve (AURC) of a selective classifier induced by κ. The proposed measure is a normalization of AURC where we subtract the AURC of the best κ in hindsight. The benefit of the proposed normalization is that it allows for meaningful comparisons accross problems. We term the this normalized metric "excess AURC" (E-ARUC) and it will be used throughout the paper for performance evaluation of κ functions. A selective classifier is a pair (f, g), where f is a classifier, and g: X → {0, 1} is a selection function, which serves as a binary qualifier for f as follows, DISPLAYFORM0 The performance of a selective classifier is quantified using coverage and risk. Coverage, defined to DISPLAYFORM1, is the probability mass of the non-rejected region in X. The selective risk of DISPLAYFORM2 These two measures can be empirically evaluated over any finite labeled set S m (not necessarily the training set) in a straightforward manner. Thus, the empirical selective risk is, DISPLAYFORM3 whereφ is the empirical coverage,φ(f, DISPLAYFORM4 The overall performance profile of a family of selective classifiers (optimized for various coverage rates) can be measured using the risk-coverage curve (RC-curve), defined to be the selective risk as a function of coverage. Given a classifier f and confidence score function κ defined for f, we define an empirical performance measure for κ using an independent set V n of n labeled points. The performance measure is defined in terms of the following selective classifier (f, g) (where f is our given classifier), and the selection functions g is defined as a threshold over κ values, DISPLAYFORM5 Let Θ be the set of all κ values of points in V n, Θ = ∆ {κ(x,ŷ f (x)|f ): (x, y) ∈ V n }; for now we assume that Θ contains n unique points, and later we note how to deal with duplicate values. The performance of κ is defined to be the area under the (empirical) RC-curve (AURC) of the pair DISPLAYFORM6 Intuitively, a better κ will induce a better selective classifier that will tend to reject first the points that are misclassified by f. Accordingly, the associated RC-curve will decrease faster (with decreasing coverage) and the AURC will be smaller. For example, in Figure 1 we show (in blue) the RC-curve of classifier f obtained by training a DNN trained over the CIFAR-100 dataset. The κ induced by f is the softmax response confidence score, κ(x) = max i f (x) i. The RC-curve in the figure is calculated w.r.t. to an independent labeled set V n of n = 10, 000 points from CIFAR-100. Each point on the curve is the empirical selective risk of a selective classifier (f, g θ) such that θ ∈ Θ. As can be seen, the selective risk is monotonically increasing with coverage. For instance, at full coverage = 1, the risk is approximately 0.29. This risk corresponds to a standard classifier (that always predicts and does not reject anything). The risk corresponding to coverage = 0.5 is approximately 0.06 and corresponds to a selective classifier that rejects half of the points (those whose confidence is least). Not surprisingly, its selective risk is significantly lower than the risk obtained at full coverage. Figure 1: RC-curve for the CIFAR100 dataset with softmax response confidence score. Blue: the RC curve based on softmax response; black: the optimal curve that can be achieved in hindsight. An optimal in hindsight confidence score function for f, denoted by κ *, will yield the optimal risk coverage curve. This optimal function rates all misclassified points (by f) lower than all correctly classified points. The selective risk associated with κ * is thus zero at all coverage rates below 1 −r(f |V n). The reason is that the optimal function rejects all misclassified points at such rates. For example, in Figure 1 we show the RC-curve (black) obtained by relying on κ *, which reaches zero at coverage of 1 − 0.29 = 0.71 (red dot); note that the selective risk at full coverage is 0.29.Since the AURC of all RC-curves for f induced by any confidence scoring function will be larger than the AURC of κ *, we normalize by AURC(κ *) to obtain a unitless performance measure. To compute the AURC of κ *, we compute the discrete integral ofr (w.r.t. κ *) from the coverage level of 1 −r(f |V n) (0 errors) to 1 (nr errors). Thus, DISPLAYFORM7 We approximate using the following integral: DISPLAYFORM8 For example, the gray area in Figure 1 is the AURC of κ *, which equals 0.04802 (and approximated by 0.04800 using the integral).To conclude this section, we define the Excess-AURC (E-AURC) as E-AURC(κ, f |V n) = AURC(κ, f |V n) − AURC(κ *, f |V n). E-AURC is a unitless measure in, and the optimal κ will have E-AURC = 0. E-AURC is used as our main performance measure. The area of uncertainty estimation is huge, and way beyond our scope. Here we focus only on non-Bayesian methods in the context of deep neural classification. Motivated by a Bayesian approach, BID9 proposed the Monte-Carlo dropout (MC-dropout) technique for estimating uncertainty in DNNs. MC-dropout estimates uncertainty at test time using the variance statistics extracted from several dropout-enabled forward passes. The most common, and well-known approach for obtaining confidence scores for DNNs is by measuring the classification margin. When softmax is in use at the last layer, its values correspond to the distance from the decision boundary, where large values tend to reflect high confidence levels. This concept is widely used in the context of classification with a reject option in linear models and in particular, in SVMs BID0 BID3 BID8. In the context of neural networks, BID4 BID5 were the first to propose this approach and, for DNNs, it has been recently shown to outperform the MC-dropout on ImageNet BID10.A K-nearest-neighbors (KNN) algorithm applied in the embedding space of a DNN was recently proposed by BID20. The KNN-distances are used as a proxy for classconditional probabilities. To the best of our knowledge, this is the first non-Bayesian method that estimates neural network uncertainties using activations from non-final layers. A new ensemble-based uncertainty score for DNNs was proposed by BID17. It is well known that ensemble methods can improve predictive performance BID1. Their ensemble consists of several trained DNN models, and confidence estimates were obtained by averaging softmax responses of ensemble members. While this method exhibits a significant improvement over all known methods (and is presently state-of-the-art), it requires substantially large computing resources for training. When considering works that leverage information from the network's training process, the literature is quite sparse. BID14 proposed to construct an ensemble, composed of several snapshots during training to improve predictive performance with the cost of training only one model. However, due to the use of cyclic learning rate schedules, the snapshots that are averaged are fully converged models and produce a that is both conceptually and quantitatively different from our use of snapshots before convergence. BID15 similarly proposed to average the weights across SGD iterations, but here again the averaging was done on fully converged models that have been only fine-tuned after full training processes. Thus both these ensemble methods are superficially similar to our averaging technique but are different than our method that utilizes "premature" ensemble members (in terms of their classification performance). In this section we present an example that motivates our algorithms. Consider a deep classification model f that has been trained over the set S m through T epochs. Denote by f [i] the model trained at the ith epoch; thus, f = f [T]. Take an independent validation set V n of n labeled points. We monitor the quality of the softmax response generated from f (and its intermediate variants f [i] ), through the training process, as measured on points in V n. The use of V n allows us to make meaningful statements about the quality of softmax response values (or any other confidence estimation method) for unseen test points. We construct the example by considering two groups of instances in V n defined by confidence assessment assigned using the softmax response values f gives to points in V n. The green group contains the highest (99%-100%) percentile of most confident points in V n, and the red group contains the lowest (0%-1%) percentile of least confident points. Although the softmax response is rightfully criticized in its ability to proxy confidence BID9, it is reasonable to assume it is quite accurate in ranking green vs. red points (i.e., a prediction by f regarding a red point is likely to be less accurate than its prediction about a green point).We observe that the prediction of the green points' labels is learned earlier during training of f, compared to a prediction of any red point. This fact is evident in FIG0 where we see the training of f over CIFAR-100. Specifically, we see that the softmax response values of green points stabilize at their maximal values around Epoch 80. We also note that the green points in this top percentile are already correctly classified very early, near Epoch 25 (not shown in the figure). In contrast, red points continue to improve their confidence scores throughout. This observation indicates that green points can be predicted very well by an intermediate model such as f 130. Can we say that f 130 can estimate the confidence of green points correctly? Recall from Section 3 that a useful method for assessing the quality of a confidence function is the E-AURC measure (applied over an independent validation set). We now measure the quality of the softmax response of all intermediate classifiers shows the E-AURC of the red points. We see that for the green points, the confidence estimation quality improves (almost) monotonically and then degrades (almost) monotonically. The best confidence estimation is obtained by intermediate classifiers such as f 130. Surprisingly, the final model f [T] is one of the worst estimators for green points! In sharp contrast, the confidence estimates for the red points monotonically improves as training continues. The best estimator for red points is the final model f [T]. This behavior can be observed in all the datasets we considered (not reported). DISPLAYFORM0 The above dynamics indicates that the learning of uncertainty estimators for easy instances conceptually resembles overfitting in the sense that the assessment of higher confidence points in the test set degrades as training continues after a certain point. To overcome this deficiency we propose an algorithm that uses the concept of early stopping in a pointwise fashion, where for each sample (or set of samples) we find the best intermediate snapshot for uncertainty estimation. In this section, first we present a supervised algorithm that learns an improved scoring function for a given pair (f, κ), where f is a trained deep neural classifier, and κ is a confidence scoring function for f's predictions. In principle, κ: X → R, where κ(x) can be defined as any mapping from the activations of f applied on x to R. All the confidence estimation methods we described above comply with this definition.1 Our algorithm requires a labeled training sample. The second algorithm we present is an approximated version of the first algorithm, which does not rely on additional training examples. Let f be a neural classifier that has been trained using any (mini-batch) SGD variant for T epochs, and let F = ∆ {f [i]: 1 ≤ i ≤ T } be the set of intermediate models obtained during training (f [i] is the model generated at epoch i). We assume that f, the snapshots set F, and a confidence score function for f, κ(·, ·|f): X →), are given.2 Let V n be an independent training set. The Pointwise Early Stopping (PES) algorithm for confidence scores (see pseudo-code in Algorithm 1) operates as follows. The pseudo-code contains both the training and inference procedures. At each iteration of the training main loop (lines 3-11), we extract from V (which is initialized as a clone of the set V n) a set of the q most uncertain points. We abbreviate this set by S (the "layer"). The size of the layer is determined by the hyperparameter q. We then find the best model in F using the DISPLAYFORM0 for i = 0 to n/q do 4: DISPLAYFORM1 indicates the qth order statistic of r 6: DISPLAYFORM2 10: DISPLAYFORM3 end for 12:return K,Θ 13: end function 14: function ESTIMATE CONFIDENCE(x,f,K,Θ) 15: DISPLAYFORM4 DISPLAYFORM5 end function E-AURC measure with respect to S. This model, denoted f [j], is found by solving DISPLAYFORM6 The best performing confidence score over S, and the threshold over the confidence level, θ, are saved for test time (lines 8-9) and used to associate points with their layers. We iterate and remove layer after layer until V is empty. Our algorithm produces a partition of X comprising layers from least to highest confidence. For each layer we find the best performing κ function based on models from F.To infer the confidence rate for given point x at test time, we search for the minimal i that satisfies DISPLAYFORM7, where κ i and θ i are the i'th elements of K and Θ respectively. DISPLAYFORM8, where i is added to enforce full order on the confidence score between layers, recall that κ ∈. As we saw in Section 6.1, the computational complexity of the PES algorithm is quite intensive. Moreover, the algorithm requires an additional set of labeled examples, which may not always be available. The Averaged Early Stopping (AES) is a simple approximation of the PES motivated by the observation that "easy" points are learned earlier during training as shown in FIG0. By summing the area under the learning curve (a curve that is similar to 2(a)) we leverage this property and avoid some inaccurate confidence assessments generated by the last model alone. We approximate the area under the curve by averaging k evenly spaced points on that curve. Let F be a set of k intermediate models saved during the training of f, DISPLAYFORM0 where linspace(t, T, k) is a set of k evenly spaced integers between t and T (including t and T). We define the output κ as the average of all κs associated with models in F, DISPLAYFORM1 As we show in Section 7, AES works surprisingly well. In fact, due to the computational burden of running the PES algorithm, we use AES in most of our experiments below. We now present of our AES algorithm applied over the four known confidence scores: softmax response, NN-distance BID20, MC-dropout BID9 ans Ensemble BID17 ) (see Section 4). For implementation details for these methods, see Appendix A. We evaluate the performance of these methods and our AES algorithm that uses them as its core κ. In all cases we ran the AES algorithm with k ∈ {10, 30, 50}, and t = 0.4T. We experiment with four standard image datasets: CIFAR-10, CIFAR-100, SVHN, and Imagenet (see Appendix A for details).Our are reported in Table 4. The table contains four blocks, one for each dataset. Within each block we have four rows, one for each baseline method. To explain the structure of this table, consider for example the 4th row, which shows the corresponding to the softmax response for CIFAR-10. In the 2nd column we see the E-AURC (×10 3) of the softmax response itself (4.78). In the 3rd column, the of AES applied over the softmax response with k = 10 (reaching E-AURC of 4.81). In the 4th column we specify percent of the improvement of AES over the baseline, in this case -0.7% (i.e., in this case AES degraded performance). For the imagenet dataset, we only present for the softmax response and ensemble. Applying the other methods on this large dataset was computationally prohibitive. Let us now analyze these . Before considering the relative performance of the baseline methods compares to ours, it is interesting to see that the E-AURC measure nicely quantifies the difficulty level of the learning problems. Indeed, CIFAR-10 and SVHN are known as relatively easy problems and the E-AURC ranges we see in the table for most of the methods is quite small and similar. CIFAR-100 is considered harder, which is reflected by significantly larger E-AURC values recorded for the various methods. Finally, Imagenet has the largest E-AURC values and is considered to be the hardest problem. This observation supports the usefulness of E-AURC. A non-unitless measure such as AUC, the standard measure, would not be useful in such comparisons. DISPLAYFORM0 It is striking that among all 42 experiments, our method improved the baseline method in 39 cases. Moreover, when applying AES with k = 30, it always reduced the E-AURC of the baseline method. For each dataset, the ensemble estimation approach of BID17 is the best among the baselines, and is currently state-of-the-art. It follows that for all of these datasets, the application of AES improves the state-of-the-art. While the ensemble method (and its improvement by AES) achieve the best on these datasets, these methods are computationally intensive. It is, therefore, interesting to identify top performing baselines, which are based on a single classifier. In CIFAR-10, the best (single-classifier) method is softmax response, whose E-AURC is improved 6% by AES (ing in the best single-classifier performance). Interestingly, in this dataset, NN-distance incurs a markedly bad E-AURC (35.1), which is reduced (to 4.58) by AES, making it on par with the best methods for this dataset. Turning to CIFAR-100, we see that the (single-classifier) top method is NN-distance, with an E-AURC of 45.56, which is improved by 22% using AES. Table 2: NLL and Brier score of AES method applied with Platt scaling on CIFAR-10, CIFAR-100, SVHN and Imagenet compared to the baseline method (calibrated as well). Next we examine AES applied together with probability calibration. We calibrate the of the AES algorithm using the Platt scaling technique; see BID23 for details. Platt scaling is applied on the of the AES algorithm with k = 30, and compared to the independently scaled underlying measure without AES. Performance is evaluated using both negative log-likelihood (NLL) and the Brier score BID2. For further implementation details of this experiment see Appendix A. The appear in Table 2. As can easily be seen, the probability scaling are remarkably consistent with our raw uncertainty estimates (measured with the E-AURC) over all datasets and underlying uncertainty methods. We conclude that AES also improves calibrated probabilities of the underlying uncertainty measures, and the E-AURC can serve as a reliable proxy also for calibrated probabilities. We implemented the PES algorithm only over the softmax response method (SR) for several datasets. To generate an independent training set, which is required by PES, we randomly split the original validation set (in each dataset) into two parts and took a random 70% of the set for training our algorithm, using the remaining 30% for validation. Table 3: E-AURC and % improvement for the Pointwise Early Stopping algorithm (PES) compared to the softmax response (SR) on CIFAR-10, CIFAR-100 and SVHN. All E-AURC values are multiplied by 10 3 for clarity.applying PES over NN-distance, the time complexity is nmT k DISPLAYFORM0, where k is the number of neighbours and C f (S m) is the time complexity of running a forward pass of m samples using the classifier f. Similarly, the complexity of PES when the underlying scores are from MC-dropout is O(dT C f (V n) where d is the number of dropout iterations (forward passes) of the MC-dropout algorithm. Thus, when n = 7000, T = 250 (the parameters used for applying PES over CIFAR-100), and with d = 100 (as recommended in BID9), this amounts to 175,000,000 forward passes. We set q = n/3. We repeated the experiment over 10 random training-validation splits and report the average and the standard errors in Table 3.As seen, PES reduced the E-AURC of softmax on all datasets by a significant rate. The best improvement was achieved on CIFAR-100 (E-AURC reduced by 18%).Our difficulties when applying the PES algorithm on many of the underlying confidence methods, and the outstanding of the AES motivate further research that should lead to improving the algorithm and making it more efficient. We presented novel uncertainty estimation algorithms, which are motivated by an observation regarding the training process of DNNs using SGD. In this process, reliable estimates generated in early epochs are later on deformed. This phenomenon somewhat resembles the well-known overfitting effect in DNNs. The PES algorithm we presented requires an additional labeled set and expensive computational resources for training. The approximated version (AES) is simple and scalable. The ing confidence scores our methods generate systematically improve all existing estimation techniques on all the evaluated datasets. Both PES and AES overcome confidence score deformations by utilizing available snapshot models that are generated anyway during training. It would be interesting to develop a loss function that will explicitly prevent confidence deformations by design while maintaining high classification performance. In addition, the uncertainty estimation of each instance currently requires several forward passes through the network. Instead it would be interesting to consider incorporating distillation BID13 so as to reduce inference time. Another direction to mitigate the computational effort at inference time is to approximate PES using a single model per instance based on an early stopping criterion similar to the one proposed by BID19. Softmax Response: For the softmax response method (SR) we simply take the relevant softmax value of the sample, κ(x, i|f) = f (x) i. We implemented the NN-distance method using k = 500 for the nearest neighbors parameter. We didn't implemented the two proposed extensions (embedding regularization, and adversarial training), this add-on will degrade the performance of f for better uncertainty estimation, which we are not interested in. Moreover, running the NN-distance with this add-on will require to add it to all other methods to manage a proper comparison. The MC-dropout implemented with p = 0.5 for the dropout rate, and 100 feed-forward iterations for each sample. Ensemble: The Ensemble method is implemented as an average of softmax values across ensemble of 5 DNNs. Platt scaling BID23: The Platt scaling is applied as follows. Given a confidence measure κ and a validation set V, the scaling is the solution of the logistic regression from κ(x,ŷ f (x)|f ) to κ * (x,ŷ f (x)|f ), where κ * (x,ŷ f (x)|f ) is defined as 0 when x =ŷ f (x) and 1 otherwise. We train the logistic regression models based on all points in V. To validate the training of this calibration we randomly split the original test set to a training and test subsets. The calibration is learned over the training subset and evaluated on the test subset. The performance of the ing scaled probabilities has been evaluated using both negative log likelihood (NLL) and the Brier score BID2, which is simply the average L 2 distance between the predicted and the true probabilities. We provide here the table of the experiments of AES for softmax response and NN-distance now with standard errors. Due to computational complexity the standard error for all other methods has not been computed. Table 4: E-AURC and % improvement for AES method on CIFAR-10, CIFAR-100, SVHN and ImageNET for various k values compared to the baseline method. All E-AURC values are multiplied by 10 3 for clarity. In Section 5 we motivated our method by dividing the domain X to "easy points" (green) and "hard points" (red). We demonstrated that the "easy points" have a phenomenon similar to overfitting, where at some point during training the E-AURC measured for "easy points" start degrading. This observation strongly motivates our strategy that extracts information from early stages of the training process that helps to recover uncertainty estimates of the easy points. Here, we extend this demonstration that previously was presented done with respect to the softmax κ function. In Figures 3 and 4 we show plots similar to FIG0 (b,c) for the MC-dropout and NN-distance, respectively. It is evident that the overfitting occurs in all cases. but to a much lesser extent in the case of MC-dropout. This is consistent with the of the AES algorithm where E-AURC improvement over the MC-dropout was smaller compared to the improvements achieved for the other two methods. In the case of NN-distance a slight overfitting also affects the easy points, but the hard instances are affected much more severely. Thus, from this perspective in all three cases the proposed correction stratgey is potentially useful.(a) (b) Figure 3: The E-AURC of MC-dropout on CIFAR-100 along training for 5000 points with highest confidence (a), and 5000 points with lowest confidence (b).(a) (b) Figure 4: The E-AURC of NN-distance on CIFAR-100 along training for 5000 points with highest confidence (a), and 5000 points with lowest confidence (b).

We use snapshots from the training process to improve any uncertainty estimation method of a DNN classifier.

represent the short summary for finding the corresponding scientific article (long)

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.