Question: Represent the article for finding a single-sentence TLDR

Question: We study the properties of common loss surfaces through their Hessian matrix. In particular, in the context of deep learning, we empirically show that the spectrum of the Hessian is composed of two parts: the bulk centered near zero, and outliers away from the bulk. We present numerical evidence and mathematical justifications to the following conjectures laid out by Sagun et. al.: Fixing data, increasing the number of parameters merely scales the bulk of the spectrum; fixing the dimension and changing the data (for instance adding more clusters or making the data less separable) only affects the outliers. We believe that our observations have striking implications for non-convex optimization in high dimensions. First, the *flatness* of such landscapes (which can be measured by the singularity of the Hessian) implies that classical notions of basins of attraction may be quite misleading. And that the discussion of wide/narrow basins may be in need of a new perspective around over-parametrization and redundancy that are able to create *large* connected components at the bottom of the landscape. Second, the dependence of a small number of large eigenvalues to the data distribution can be linked to the spectrum of the covariance matrix of gradients of model outputs. With this in mind, we may reevaluate the connections within the data-architecture-algorithm framework of a model, hoping that it would shed light on the geometry of high-dimensional and non-convex spaces in modern applications. In particular, we present a case that links the two observations: small and large batch gradient descent appear to converge to different basins of attraction but we show that they are in fact connected through their flat region and so belong to the same basin. In this paper, we study the geometry of the loss surface of supervised learning problems through the lens of their second order properties. To introduce the framework, suppose we are given data in the form of input-label pairs, D = {(x i, y i)} N i=1 where x ∈ R d and y ∈ R that are sampled i.i.d. from a possibly unknown distribution ν, a model that is parametrized by w ∈ R M; so that the number of examples is N and the number of parameters of the system is M. Suppose also that there is a predictor f (w, x). The supervised learning process aims to solve for w so that f (w, x) ≈ y. To make the'≈' precise, we use a non-negative loss function that measures how close the predictor is to the true label, (f (w, x), y). We wish to find a parameter w * such that w * = arg min L(w) where, DISPLAYFORM0 (f (w, x i), y i ).In particular, one is curious about the relationship between L(w) andL(w):= d(ν). By the law of large numbers, at a given point w, L w →L w almost surely as N → ∞ for fixed M. However in modern applications, especially in deep learning, the number of parameters M is comparable to the number of examples N (if not much larger). And the behaviour of the two quantities may be drastically different (for a recent analysis on provable estimates see BID18).A classical algorithm to find w * is gradient descent (GD), in which the optimization process is carried out using the gradient of L. A new parameter is found iteratively by taking a step in the direction of the negative gradient whose size is scaled with a constant step size η that is chosen from line-search minimization. Two problems emerge: Gradient computation can be expensive, Line-search can be expensive. More involved algorithms, such as Newton-type methods, make use of second-order information BID19. Under sufficient regularity conditions we may observe: L(w + ∆w) ≈ L(w) + ∆w∇L(w) + ∆w T ∇ 2 L(w)∆w. A third problem emerges beyond an even more expansive computational cost of the Hessian: Most methods require the Hessian to be non-degenerate to a certain extent. When the gradients are computationally expensive, one can alternatively use its stochastic version (SGD) that replaces the above gradient with the gradient of averages of losses over subsets (such a subset will be called the mini-batch) of D (see BID5 for a classical reference). The benefit of SGD on real-life time limits is obvious, and GD may be impractical for practical purposes in many problems. In any case, the stochastic gradient can be seen as an approximation to the true gradient, and hence it is important to understand how the two directions are related to one another. Therefore, the discussion around the geometry of the loss surface can be enlightening in the comparison of the two algorithms: Does SGD locate solutions of a different nature than GD? Do they follow different paths? If so, which one is better in terms of generalization performance?For the second problem of expensive line-search, there are two classical solutions: using a small, constant step size, or scheduling the step size according to a certain rule. In practice, in the context of deep learning, the values for both approaches are determined heuristically, by trial and error. More involved optimal step size choices involve some kind of second-order information that can be obtained from the Hessian of the loss function BID24. From a computational point of view, obtaining the Hessian is extremely expensive, however obtaining some of its largest and smallest eigenvalues and eigenvectors are not that expensive. Is it enough to know only those eigenvalues and eigenvectors that are large in magnitude? How do they change through training? Would such a method work in SGD as well as it would on GD?For the third problem, let's look at the Hessian a little closer. A critical point is defined by w such that ||∇L(w)|| = 0 and the nature of it can be determined by looking at the signs of its Hessian matrix. If all eigenvalues are positive the point is called a local minimum, if r of them are negative and the rest are positive, then it is called a saddle point with index r. At the critical point, the eigenvectors indicate the directions in which the value of the function locally changes. Moreover, the changes are proportional to the corresponding -signed-eigenvalue. Under sufficient regularity conditions, it is rather straightforward to show that gradient-based methods converge to points where the gradient is zero. Recently BID15 showed that they indeed converge to minimizers. However, a significant and untested assumption to establish these convergence is that the Hessian of the loss is non-degenerate. A relaxation of the above convergence to the case of non-isolated critical points can be found in BID20. What about the critical points of machine learning loss functions? Do they satisfy the non-degeneracy assumptions? If they don't, can we still apply the of provable theorems to gain intuition? One of the first instances of the comparison of GD and SGD in the context of neural networks dates back to the late eighties and early nineties. BID4 points out that large eigenvalues of the Hessian of the loss can create the illusion of the existence of local minima and GD can get stuck there, it further claims that the help of the inherent noise in SGD may help to get out of this obstacle. The origin of this observation is due to BID6, as well as numerical justifications. However, given the computational limits of the time, these experiments relied on low-dimensional neural networks with few hidden units. The picture may be drastically different in higher dimensions. In fact, provable in statistical physics tell us that, in certain real-valued non-convex functions, the local minima concentrate at an error level near that of the global minima. A theoretical review on this can be found in BID0, while BID22 and BID2 provide an experimental simulation as well as a numerical study for neural networks. They notably find that high error local minima traps do not appear when the model is over-parametrized. These concentration can help explain why we find that the solutions attained by different optimizers like GD and SGD often have comparable training accuracies. However, while these methods find comparable solutions in terms of training error there is no guarantee they generalize equally. A recent work in this direction compares the generalization performance of small batch and large batch methods BID13. They demonstrate that the large batch methods always generalize a little bit worse even when they have similar training accuracies. The paper further makes the observation that the basins found by small batch methods are wider, thereby contributing to the claim that wide basins, as opposed to narrow ones, generalize better. The final part of the historical account is devoted to the observation of flatness of the landscape in neural networks and its consequences through the lens of the Hessian. In the early nineties, BID11 remarks that there are parts of the landscape in which the weights can be perturbed without significantly changing the loss value. Such regions at the bottom of the landscape are called the flat minima, which can be considered as another way of saying a very wide minima. It is further noted that such minima have better generalization properties and a new loss function that makes use of the Hessian of the loss function has been proposed that targets the flat minima. The computational complexity issues have been attempted to be resolved using the R-operator of. However, the new loss requires all the entries of the Hessian, and even with the R-operator, it is unimaginably slow for today's large networks. More recently, an exact numerical calculation of the Hessian has been carried out by. It turns out that the Hessian can have near zero eigenvalues even at a given random initial point, and that the spectrum of it is composed of two parts: the bulk, and the outliers. The bulk is mostly full of zero eigenvalues with a fast decaying tail, and the outliers are only a handful which appears to depend on the data. This implies that, locally, most directions in the weight space are flat, and leads to little or no change in the loss value, except for the directions of eigenvectors that correspond to the large eigenvalues of the Hessian. In this work, we present a phenomenological study in which we provide various observations on the local geometry at the bottom of the landscape and discuss their implications on certain features of solution spaces: Connectedness of basins found by large and small batch methods.1. Flatness at the bottom of the landscape: At the bottom, most of the eigenvalues in the spectrum of the Hessian are near zero, except for a small number of relatively larger ones.2. A possible explanation through over-parametrization: The decomposition of the Hessian as a sum of two matrices, where the first one is the sample covariance matrix of the gradients of model outputs and the second one is the Hessian of the function that describes the model outputs. We argue that the second term can be ignored as training progress which leaves us with the covariance term which leads to degeneracy in the Hessian when there are more parameters than samples. Dependence of eigenvalues to model-data-algorithm: We empirically examine the spectrum to uncover intricate dependencies within the data-architecture-algorithm triangle: more complex data produce more outliers, increasing the network size doesn't affect the density of large eigenvalues, large batch methods produce the same number of outliers that are larger in magnitude, and finally there are negative eigenvalues even after the training process appears to show no further progress but their magnitude is much smaller than the outliers. BID12, appear to draw the big-picture of isolated basins at the bottom of the landscape. One tool that is commonly used to demonstrate such claims is to evaluate the loss on a line that connects two solutions found by different methods. First of all, the notion of the basin itself can be misleading given the negative eigenvalues pointed out in the previous item. Moreover, we claim that this idea of isolated basins may be misleading based on the above observations of the dramatic level of flatness of the local geometry. In particular, we show that two solutions with different qualities can be shown to be in the same basin even with a loss evaluation on a straight line connecting the two. Remark 1: Throughout this paper we use the notions of data complexity and over-parametrization vaguely. The complexity of data can be defined in various ways and further research is required to determine the precise notion that is required that would link complexity to the spectrum. Overparametrization, similarly, can be defined in various ways: M >> N, M → ∞ for fixed N, or M/N → c for a certain constant, etc... However, more realistic notions of both complexity of the data and the over-parametrization should take the architecture into account, and detailed treatment of this should follow another work. Remark 2: The notion of the basin, also, can be defined precisely. However, it is unclear whether any algorithm used in practice actually locates the bottom of a basin described in classical ways. For instance, the norm of the gradients are small but not at the machine precision, and the eigenvalues of the Hessian still has a negative part even after SGD continues a long while without a meaningful decrease in the loss value. This is presumably the fault of the algorithm itself, however, it requires a further study, and hence such notions of sharp vs. wide minima in various recent work should be taken with a grain of salt. Remark 3: Even when one has a way to measure the width of a'basin', such ways of measuring the approximate width are all relative. In a recent study, BID8 shows how'sharp minima' can still generalize with proper modifications to the loss function. We note that it takes a non-linear transformation to deform relative widths of basins. And, in this work, we focus on relative values as opposed to absolute values to get a consistent comparison across different setups.2 SOME PROPERTIES OF THE HESSIAN We begin by an exact calculation of the spectrum of the Hessian at a random initial point, and at the end of training. Note that the plots are arranged in a way that they show the eigenvalues in the y-axis, and indicate the order of the eigenvalue in the x-axis. This choice is necessary to indicate the scale of the degeneracy while still showing all the eigenvalues in the same plot. FIG2 shows the full spectrum of the Hessian at the random initial point of training and after the final training point. The model of this example is a two hidden layer network with a total of 5K parameters that is trained using gradient descent. Also note that, throughout the paper, the exact full Hessian is computed via the Hessian-vector products BID21 up to the machine precision.: Ordered eigenvalues at a random initial point, and at the final point of GD. x-axis is the rank of the eigenvalue, and y-axis the value of the eigenvalue. In order to study its spectrum, we will describe how the Hessian can be decomposed into two meaningful matrices BID14 BID17. Suppose the loss function is given as a composition of two functions, the model function f •: R M −→ R is the real-valued output of a network that depends on the parameters; and the loss function •: R −→ R + is a convex function. Here, • refers to the given example. Examples include the regression: the mean-square loss composed with a real-valued output function, and classification: the negative log-likelihood loss composed with the dot product of the output of a softmax layer with the label vector. For ease of reading, we indicate the dependencies of functions and f to data by the index of the example, or omit it altogether in case it is not necessary, and unless noted otherwise the gradients are taken with respect to w. The gradient and the Hessian of the loss for a given example are given by DISPLAYFORM0 DISPLAYFORM1 where • T denotes the transpose operation (here the gradient is a column-vector). Note that since is convex (s) ≥ 0 and we can take its square root which allows us to rewrite the Hessian of the loss as follows: DISPLAYFORM2 In general, it isn't straightforward to discuss the spectrum of the sums of matrices by looking at the individual ones. Nevertheless, looking at the decomposition, we can still infer what we should expect. At a point close to a local minimum, the average gradient is close to zero. However, this doesn't necessarily imply that the gradients for individual samples are also zero. However, if (f (ŵ)) and ∇ 2 f (ŵ) are not correlated, then we can ignore the second term. And so the Hessian can be approximated by the first term: DISPLAYFORM3 Here, Equation 5 is the sum of rank one matrices (via the outer products of gradients of f multiplied by some non-negative number), therefore, the sum can be written as a product of an M × N matrix with its transpose where the columns of the matrix are formed by the scaled gradients of f. Immediately, this implies that there are at least M − N many trivial eigenvalues of the right-hand side in Equation FORMULA5.From a theoretical point of view, the tool that is required for the above problem should be a mapping of eigenvalues of the population matrix to the sample covariance matrix. Recent provable on this can be found in BID3 (please refer to the appendix for a review of this approach). We emphasize that this require independent inputs, and extensions to correlated data appear to be unavailable to the best of our knowledge. In this section, we leave the decomposition behind and focus on experimental of the spectrum of the full Hessian through the exact Hessian-vector products. We discuss how data, model, and algorithm affect the spectrum of the Hessian of the loss. In many cases of practical interest, the data contains redundancies. In such cases, the number of non-trivial eigenvalues could be even smaller than N. For instance, if one deals with a classification problem where the training data has k classes with relatively small deviation among each of them, it is reasonable to expect that there will be an order of k many non-trivial eigenvalues of the first term of the above decomposition of the Hessian in Equation 5. Then, if the second term is small (for instance when all the gradients per example are zero), we would expect to see k many outliers in the spectrum of the Hessian of the loss. To test this idea, we used a feed-forward neural network with a 100-dimensional input layer, two hidden layers each of which with 30 hidden units, and a k dimensional output layer that is combined with softmax for k-class classification. We randomly sampled from k Gaussian clusters in the input space and normalized the data globally. Then we carried out the training using SGD on ReLU network for the following number of clusters: k: {2, 5, 10, 20, 50}. The number of large eigenvalues that are above the gap in FIG3 match exactly the number of classes in the dataset. This experiment is repeated for various different setups. Please refer to Table 1 in the appendix for more experiments on this. We test the effect of growing the size of the network when data, architecture, and algorithm are fixed. In some sense, we make the system more and more over-parametrized. Based on the intutions developed above we should not observe a change in the size and number of the large eigenvalues of the Hessian at the bottom. To test this, we sample 1K examples from the MNIST dataset and fix them as the training set. Then we form four different networks each of which has a different number of nodes in its hidden layer (n hidden ∈ {10, 30, 50, 70}). All four networks are trained with the same step size and the same number of iterations and the exact Hessian is computed at the end. Figure 3 shows the largest 120 eigenvalues of each of the four Hessians ranked in an increasing order. For the right edge of the spectrum (that is, for the large positive eigenvalues), the shape of the plot remains invariant as the number of parameters increase (Figure 3). Finally, we turn to describing the nature of solutions found by the large and small batch methods for the training landscape. We train a convnet that is composed of 2 convolution layers with relu-maxpool followed by two fully-connected layers. The training set is a subsampled MNIST with 1K training examples. The small batch method uses a mini-batch size of 10, and the large-batch one uses 512. A learning rate for which both algorithms converge is fixed for both LB and SB. Note that the stochastic gradients are averaged over the mini-batch. Therefore, fixing the learning rate allows the algorithms to take steps whose lengths are proportional to the norms of the corresponding stochastic gradients averaged over the mini-batch. This way we ensure that both methods are compared fairly when we look at them after a fixed number of iterations. We train until the same number of iterations have been reached. Then, we calculate the Hessian and the spectrum of the Hessian in FIG0. The large batch method locates points with larger positive eigenvalues. This observation is consistent with BID13 as the way they measure flatness takes local rates of increase in a neighborhood of the solution into account which is intimately linked to the size of the large eigenvalues. Lastly, we observe that the negative eigenvalues at the end of the training are orders of magnitude smaller than the large ones. The very existence of the negative eigenvalues indicates that the algorithm didn't locate a local minimum, yet. Note that the stopping criterion in most practical cases is arbitrary. Training is stopped after there is no meaningful decrease in the loss value or increase in the test accuracy. In our experiments, the training ran well beyond the point of this saturation. In this time-scale, the loss decays in much smaller values. And we may expect convergence to a localminimum at large (possibly exponentially long) time-scales. However, from a practical point of view, it appears that the properties of the landscape at this fine-grained scale is less relevant in terms of its test performance. Anyhow, we observe that there are a number of negative eigenvalues but their magnitude is much smaller compared to the positive eigenvalues. The reason we look at the ranked negative eigenvalues in percentages rather than the order is the of the experiment in Section 3.2. Adding more weights to the system scale the small scale eigenvalues proportionally, the number of outliers remain unchanged whereas the ratio of negative x-axis indicates the order of the eigenvalue in percentages, y-axis indicates the eigenvalues.eigenvalues remain the same. Moreover, they appear to be converging to have the same shape (Figure 6). Also, note that the negative eigenvalues can only come from the second term of the decomposition in Equation 5. Unless the second term contributes to the spectrum in an asymmetrical way, the observation that the negative eigenvalues are small confirms our previous suspicions that the effect of the ignored term in the decomposition is small. Finally, we revisit the issue through the lens of the following question: What does overparametrization imply on the discussion around GD vs. SGD (or large batch vs small batch) especially for their generalization properties? In this final section, we will argue that, contrary to what is believed in BID13 and BID7 the two algorithms do not have to be falling into different basins. As noted in the introduction, for a while the common sense explanation on why SGD works well (in fact better) than GD (or large batch methods) was that the non-convex landscape had local minima at high energies which would trap large-batch or full-batch methods. Something that SGD with small batch shouldn't suffer due to the inherent noise in the algorithm. However, there are various experiments that have been carried out in the past that show that, for reasonable large systems, this is not the case. For instance, BID22 demonstrate that a two hidden layer fully connected network on MNIST can be trained by GD to reach at the same level of loss values as SGD 1. In fact, when the step size is fixed to the same value for both of the algorithms, they reach the same loss value at the same number of iterations. The training accuracy for both algorithms are the same, and the gap between test accuracies diminish as the size of the network increase with GD falling ever so slightly behind. It is also shown in BID13 that training accuracies for both large and small batch methods are comparably good. Furthermore, BID26 demonstrates that training landscape is easy to optimize even when there is no clear notion of generalization. Such observations are consistent with our observations: over-parametrization (due to the architecture of the model) leads to flatness at the bottom of the landscape which is easy to optimize. When we turn our attention to generalization, BID13 note that LB methods find a basin that is different than the one found by SB methods, and they are characterized by how wide the basin is. As noted in FIG0, indeed the large eigenvalues are larger in LB than in SB, but is it enough to justify that they are in different basins, especially given the fact that the number of flat directions are enormous. The observation that LB converges to sharper basins that are separated by wells from the wider basins found by SB has been an observation that triggered attention. In this section, we present two solutions with different qualities as measured by the generalization error and we show that they are in fact in the same'basin' by showing that the evaluation of the loss doesn't go through a barrier between the two solutions. We start by two common pitfalls that one may fall into in testing this:The problem with epoch based time scales: A common way to plot training profiles in larger scale neural networks is to stop every epoch to reserve extra computational power to calculate various statistics of the model at its current position. This becomes problematic when one compares training with different batch sizes, primarily because the larger batch model takes fewer steps in a given epoch. Recall that the overall loss is averaged, therefore, for a fixed point in the weight space, the empirical average of the gradients is an unbiased estimator for the expected gradient. Hence it is reasonable to expect that the norms of the large batch methods match to the ones of the small batch. And for a fair comparison, one should use the same learning rate for both training procedures. This suggests that a better comparison between GD and SGD (or LB and SB) should be scaled with the number of steps, so that, on average both algorithms are able to take similar number of steps of comparable sizes. The experiments we present use the number of iterations as the time-scale. The problem with line interpolations in the weight space: The architecture of typical neural networks have many internal symmetries, one of which is the flip symmetry: when one swaps two nodes (along with the weights connected to it) at a given layer, the ing network is identical to the original one. Therefore, when one trains two systems to compare, it may well be possible that the two fall into different flip symmetrical configurations that may look more similar when they are reordered. Therefore, training two systems with levels of randomness (seed, batch-size, choice of the initial point, etc.) may in two points in the weight space that present a barrier only because of such symmetries. In an attempt to partially alleviate this problem we switch dynamics of an already trained system. 1. Part I: Train full CIFAR10 data for a bare AlexNet (bare meaning: no momentum, no dropout, and no batch normalization) with a batch-size of 1, 000. Record every 100 steps for 250 times. 2. Part II: Continue training from the endpoint of the previous step with a smaller batch-size of 32. Everything else, including the constant learning rate is kept the same. And train another 250 periods each of which with 100 steps. The key observation is the jump in the training and test losses, and a drop in the corresponding accuracies (Figure 8). Toward the end of Part II the small batch reaches to a slightly better accuracy (about ∼ 1%). And this looks in line with the observations in BID13, in that, it appears that the LB solution and SB solutions are separated by a barrier and that the latter of which generalizes better. Moreover, the line interpolations extending away from either endpoint appear to be confirming the sharpness of LB solution. However, we find the straight line interpolation connecting the endpoints of Part I and Part II turns out to not contain any barriers (Figure 9). This suggests that while the Part I and Part II converge to two solutions with different properties, these solutions have been in the same basin all along. This raises the striking possibility that those other seemingly different solutions may be similarly connected by a flat region to form a larger basin (modulo internal symmetries).Another interpretation of this experiment, also, goes through the Gauss-Newton decomposition introduced in Equation 5. When we decrease the batch size, we increase the noise in the covariance of the gradients, and hence the first term starts to dominate. Even when the weight space has large flat regions, the fluctuations of the stochastic noise should be precisely in the directions of the large eigenvalues. Therefore, at the beginning of Part II, the loss increases because the point fluctuates in the directions corresponding to the large eigenvalues, and eventually settles at a point that lies at the interior of the same level set, essentially staying in the same basin.: Loss and accuracy evaluation on the straight line that contains LB and SB solutions. The accuracy of the SB solution is ∼ 1% better than the LB solution, but there is no barrier between the two points. One of the most striking implications of flatness may be the connected structure of the solution space. We may wonder whether two given solutions can be connected by a continuous path of solutions. This question has been explored in a recent work: in BID9 it is shown that for one hidden layer rectified neural networks the solution space is connected which is consistent with the flatness of the landscape. The classical notion of basins of attractions may not be the suitable objects to study for neural networks. Rather, we may look at the exploration of interiors of level sets of the landscape. We may be tempted to speculate that such an exploration may indeed in point that generalizes better. However, the flat space itself is very high dimensional which comes with its own computational issues. The training curve can be seen as composed of two parts: high gain part where the norm of the gradients are large, noise of the gradients is larger relative to the size of the stochastic gradients (see BID25 for a recent reference). We speculate that the first part is relatively easy and even a large batch method can locate a large level set that contains points that generalize better than what's initially found. From a practical point of view, using larger batches with larger step sizes can, in fact, accelerate training. An example of this can be found in BID10, where training Imagenet with a minibatch size of 8192 can match small batch performance. On a final note for further consideration, we remark that we used standard pre-processing and initialization methods that are commonly used in practice. Fixing these two aspects, we modified the data, model, and algorithm in order to study their relative effects. However, the effects of pre-processing and initialization on the Hessian is highly non-trivial and deserves a separate attention. We have shown that the level of the singularity of the Hessian cannot be ignored from theoretical considerations. Furthermore, we use the generalized Gauss-Newton decomposition of the Hessian to argue the cluster of zero eigenvalues are to be expected in practical applications. This allows us to reconsider the division between initial fast decay and final slow progress of training. We see that even large batch methods are able to get to the same basin where small batch methods go. As opposed to the common intuition, the observed generalization gap between the two is not due to small batch finding a different, better, wider basin. Instead, the two solutions appear to be in the same basin. This lack of a barrier between solutions is demonstrated by finding paths between the two points that lie in the same level set. To conclude, we propose a major shift in perspective on considerations of the energy landscape in deep learning problems. In the subsequent experiments, we used a feed-forward neural network with a 100 dimensional input layer, two hidden layers each of which with 30 hidden units, and a k dimensional output layer that is combined with softmax for k-class classification. We sampled random k Gaussian clusters in the input space and normalized the data globally. Then we carried out the training for the following sets of parameters: k: {2, 5, 10, 20, 50}, algorithm: {GD, SGD}, non-linearity: {tanh, ReLU}, initial multipler for the covariance of the input distribution: {1, 10}. Then we counted the number of large eigenvalues according to three different cutoff methods: largest consecutive gap, largest consecutive ratio, and a heuristic method of determining the threshold by searching for the elbow in the scree plot (see FIG3 . In Table 1 In this section, we will show that the spectrum of the Generalized Gauss-Newton matrix can be characterized theoretically under some conditions. Suppose that we can express the scaled gradient DISPLAYFORM0) as g = T x with the matrix T ∈ M × d depending only on the parameters w -which is the case for linear models. Then we can write: DISPLAYFORM1 Furthermore, without loss of generality, we assume that the examples are normalized such that the entries of X are independent with zero mean and unit variance. One of the first steps in studying G goes through understanding its principle components. In particular, we would like to understand how the eigenvalues and eigenvectors of G are related to the ones of Σ where DISPLAYFORM2 In the simplest case, we have Σ = Id so that the gradients are uncorrelated and the eigenvalues of G are distributed according to the Marčenko-Pastur law in the limit where N, M → ∞ and α:= M N. The dates back to sixties and can be found in BID16. Note that if M > N then there are M − N trivial eigenvalues of G at zero. Also, the width of the nontrivial distribution essentially depends on the ratio α. Clearly, setting the expected covariance to identity is very limiting. One of the earliest relaxations appear in BID1. They prove a phase transition for the largest eigenvalues of the sample covariance matrix which has been known as the BBP phase transition. A case that may be useful for our setup is as follows:Theorem 1 BID1. If Σ = diag(, 1, . . ., 1), > 1, and M, N → ∞ with M N = α ≥ 1. Let c = 1 + √ α, and let's call the top eigenvalue of the sample covariance matrix as λ max then:• If 1 ≤ < c then λ max is at the right edge of the spectrum with Tracy-Widom fluctuations.• If c < then λ max is an outlier that is away from bulk centered at (1 + α −1) with Gaussian fluctuations. Typically, due to the correlations in the problem we don't have Σ to be the identity matrix or a diagonal matrix with spikes. This makes the analysis of their spectrum a lot more difficult. A solution for this slightly more general case with non-trivial correlations has been provided only recently by BID3. We will briefly review these here see how they are related to the first term of the above decomposition. Theorem 2 BID3. If d = M, Σ − Id has bounded rank, log N is comparable to log M, and entries of X are independent with mean zero and variance one, then the spectrum of Σ can be precisely mapped to the one of G as M, N → ∞ for fixed α = M N. Let K = min{M, N}, and the decomposition of the spectrum can be described as follows:• Zeros: M − K many eigenvalues located at zero (if M > N).• Bulk: Order K many eigenvalues are distributed according to Marčenko-Pastur law.• Right outliers: All eigenvalues of Σ that exceed a certain value produce large-positive outlier eigenvalues to the right of the spectrum of G.• Left outliers: All eigenvalues of Σ that are close to zero produce small outlier eigenvalues between 0 and the left edge of the bulk of G.Moreover, the eigenvectors of outliers of G are close to the corresponding ones of Σ.This theorem essentially describes the way in which one obtains outlier eigenvalues in the sample covariance matrix assuming the population covariance is known. Here is an example: Figure 10: Spectrum of the logistic regression loss with tanh unit: when data has a single Gaussian blob (left), when data has two Gaussian blobs (right). In the latter case, the spectrum has outlier eigenvalues at 454.4, 819.5, and 92.7 for alpha = 1, 2, 0.02, respectively. Example 1 (Logistic regression). Consider the log-loss (s, y) = −y log 1 1+e −s − (1 − y) log(1 − 1 1+e −s) and a single neuron with the sigmoid non-linearity. Note that (s, y) is convex in s for fixed y, and we can apply the decomposition using and f (w, x) = w, x. In this case we have M = d, also, note that the second part of the Hessian in Equation 4 is zero since ∇ 2 f (w, x) = 0. So the Hessian of the loss is just the first term. It is straightforward to calculate that the gradient per sample is of the form g = c(w, x)Id M x for a positive constant c = c(w, x) that doesn't depend on y. This case falls into the classical Marčhenko-Pastur law (left pane of FIG2 . Example 2. Once we have more than one class this picture fails to hold. For, (s) = − log(s), and f (w; (x, y)) = exp − wy,x k exp − w y k,x the spectrum changes. It turns out that in that case the weights have one large outlier eigenvalue, and a bulk that's close to zero (right pane of FIG2).

The loss surface is *very* degenerate, and there are no barriers between large batch and small batch solutions.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question:
represent the scientific article to find a short tldr

E.g.
'We propose learning to transfer learn (L2TL) to improve transfer learning on a target dataset by judicious extraction of information from a source dataset. L2TL considers joint optimization of vastly-shared weights between models for source and target tasks, and employs adaptive weights for scaling of constituent losses. The adaptation of the weights is based on reinforcement learning, guided with a performance metric on the target validation set. We demonstrate state-of-the-art performance of L2TL given fixed models, consistently outperforming fine-tuning baselines on various datasets. In the regimes of small-scale target datasets and significant label mismatch between source and target datasets, L2TL outperforms previous work by an even larger margin.' == 'We propose learning to transfer learn (L2TL) to improve transfer learning on a target dataset by judicious extraction of information from a source dataset.' != 'An empirical study that provides a novel perspective on few-shot learning, in which a fine-tuning method shows comparable accuracy to more complex state-of-the-art methods in several classification tasks.'

Question: The allocation of computation resources in the backbone is a crucial issue in object detection. However, classification allocation pattern is usually adopted directly to object detector, which is proved to be sub-optimal. In order to reallocate the engaged computation resources in a more efficient way, we present CR-NAS (Computation Reallocation Neural Architecture Search) that can learn computation reallocation strategies across different feature resolution and spatial position diectly on the target detection dataset. A two-level reallocation space is proposed for both stage and spatial reallocation. A novel hierarchical search procedure is adopted to cope with the complex search space. We apply CR-NAS to multiple backbones and achieve consistent improvements. Our CR-ResNet50 and CR-MobileNetV2 outperforms the baseline by 1.9% and 1.7% COCO AP respectively without any additional computation budget. The models discovered by CR-NAS can be equiped to other powerful detection neck/head and be easily transferred to other dataset, e.g. PASCAL VOC, and other vision tasks, e.g. instance segmentation. Our CR-NAS can be used as a plugin to improve the performance of various networks, which is demanding. Object detection is one of the fundamental tasks in computer vision. The backbone feature extractor is usually taken directly from classification literature (; ; a;). However, comparing with classification, object detection aims to know not only what but also where the object is. Directly taking the backbone of classification network for object detectors is sub-optimal, which has been observed in. To address this issue, there are many approaches either manually or automatically modify the backbone network. proposes a neural architecture search (NAS) framework for detection backbone to avoid expert efforts and design trails. However, previous works rely on the prior knowledge for classification task, either inheriting the backbone for classification, or designing search space similar to NAS on classification. This raises a natural question: How to design an effective backbone dedicated to detection tasks? To answer this question, we first draw a link between the Effective Receptive Field (ERF) and the computation allocation of backbone. The ERF is only small Gaussian-like factor of theoretical receptive field (TRF), but it dominates the output . The ERF of image classification task can be easily fulfilled, e.g. the input size is 224×224 for the ImageNet data, while the ERF of object detection task need more capacities to handle scale variance across the instances, e.g. the input size is 800×1333 and the sizes of objects vary from 32 to 800 for the COCO dataset. Lin et al. (2017a) allocates objects of different scales into different feature resolutions to capture the appropriate ERF in each stage. Here we conduct an experiment to study the differences between the ERF of several FPN features. As shown in Figure 1, we notice the allocation of computation across different resolutions has a great impact on the ERF. Furthermore, appropriate computation allocation across spacial position boost the performance of detector by affecting the ERF. Figure 1: Following the instructions in , we draw the ERF of FPN in different resolution features. The size of base plate is 512×512, with respective anchor boxes ({64, 128, 256} for {p 3, p 4, p 5}) drawn in. The classification CNNs ResNet50 tends to have redundant ERF for high resolution features p 3 and limited ERF for low resolution features p 5. After stage reallocation, our SCR-ResNet50 has more balanced ERF across all resolutions which leads to a high performance. Based on the above observation, in this paper, we aim to automatically design the computation allocation of backbone for object detectors. Different from existing detection NAS works which achieve accuracy improvement by introducing higher computation complexity, we reallocate the engaged computation cost in a more efficient way. We propose computation reallocation NAS (CR-NAS) to search the allocation strategy directly on the detection task. A two-level reallocation space is conducted to reallocate the computation across different resolution and spatial position. In stage level, we search for the best strategy to distribute the computation among different resolution. In operation level, we reallocate the computation by introducing a powerful search space designed specially for object detection. The details about search space can be found in Sec. 3.2. We propose a hierarchical search algorithm to cope with the complex search space. Typically in stage reallocation, we exploit a reusable search space to reduce stage-level searching cost and adapt different computational requirements. Extensive experiments show the effectiveness of our approach. Our CR-NAS offers improvements for both fast mobile model and accurate model, such as ResNet , MobileNetV2 , ResNeXt . On the COCO dataset, our CR-ResNet50 and CR-MobileNetV2 can achieve 38.3% and 33.9% AP, outperforming the baseline by 1.9% and 1.7% respectively without any additional computation budget. Furthermore, we transfer our CR-ResNet and CR-MobileNetV2 into the another ERF-sensitive task, instance segmentation, by using the Mask RCNN framework. Our CR-ResNet50 and CR-MobileNetV2 yields 1.3% and 1.2% COCO segmentation AP improvement over baseline. To summarize, the contributions of our paper are three-fold: • We propose computation reallocation NAS(CR-NAS) to reallocate engaged computation resources. To our knowledge, we are the first to dig inside the computation allocation across different resolution. • We develop a two-level reallocation space and hierarchical search paradigm to cope with the complex search space. Typically in stage reallocation, we exploit a reusable model to reduce stage-level searching cost and adapt different computational requirements. • Our CR-NAS offers significant improvements for various types of networks. The discovered models show great transferablity over other detection neck/head, e.g. NAS-FPN , other dataset, e.g. PASCAL VOC and other vision tasks, e.g. instance segmentation. Neural Architecture Search(NAS) Neural architecture search focus on automating the network architecture design which requires great expert knowledge and tremendous trails. Early NAS approaches (; are computational expensive due to the evaluating of each candidate. Recently, weight sharing strategy is proposed to reduce searing cost. One-shot NAS method (; ;) build a directed acyclic graph G (a.k.a. supernet) to subsume all architectures in the search space and decouple the weights training with architectures searching. NAS works only search for operation in the certain layer. our work is different from them by searching for the computation allocation across different resolution. Computation allocation across feature resolutions is an obvious issue that has not been studied by NAS. We carefully design a search space that facilitates the use of existing search for finding good solution. NAS on object detection. There are some work use NAS methods on object detection task; ). search for scalable feature pyramid architectures and search for feature pyramid network and the prediction heads together by fixing the architecture of backbone CNN. These two work both introduce additional computation budget. The search space of is directly inherited from the classification task which is suboptimal for object detection task. search for dilated rate on channel level in the CNN backbone. These two approaches assume the fixed number of blocks in each resolution, while we search the number of blocks in each stage that is important for object detection and complementary to these approaches. Our search method is based on the Faster RCNN with FPN (a) for its excellent performance. We only reallocate the computation within the backbone, while fix other components for fair comparison. For more efficient search, we adopt the idea of one-shot NAS method (; ;). In one-shot NAS, a directed acyclic graph G (a.k.a. supernet) is built to subsume all architectures in the search space and is trained only once. Each architecture g is a subgraph of G and can inherit weights from the trained supernet. For a specific subgraph g ∈ G, its corresponding network can be denoted as N (g, w) with network weights w. We propose Computation Reallocation NAS (CR-NAS) to distribute the computation resources in two dimensions: stage allocation in different resolution, convolution allocation in spatial position. The backbone aims to generate intermediate-level features C with increasing downsampling rates 4×, 8×, 16×, and 32×, which can be regarded as 4 stages. The blocks in the same stage share the same spatial resolution. Note that the FLOPs of a single block in two adjacent spatial resolutions remain the same because a downsampling/pooling layer doubles the number of channels. So given the number of total blocks of a backbone N, we can reallocate the number of blocks for each stage while keep the total FLOPs the same. Figure 2 shows our stage reallocation space. In this search space, each stage contains several branches, and each branch has certain number of blocks. The numbers of blocks in different branches are different, corresponding to different computational budget for the stage. For example, there are 5 branches for the stage 1 in Figure 2, the numbers of blocks for these 5 branches are, respectively, 1, 2, 3, 4, and 5. We consider the whole network as a supernet T = {T 1, T 2, T 3, T 4}, where T i at the ith stage has K i branches, i.e. T i = {t Then an allocation strategy can be represented as τ = [τ 1, τ 2, τ 3, τ 4], where τ i denote the number of blocks in the ith branch. All blocks in the same stage have the same structure. residual blocks. We make a constraint that each stage at least has one convolutional block. We would like to find the best allocation strategy of ResNet101 is among the 32 3 possible choices. Since validating a single detection architecture requires hundreds of GPU-hours, it not realist to find the optimal architecture by human trails. On the other hand, we would like to learn stage reallocation strategy for different computation budgets simultaneously. Different applications require CNNs of different numbers of layers for achieving different latency requirements. This is why we have ReseNet18, ReseNet50, ReseNet101, etc. We build a search space to cover all the candidate instances in a certain series, e.g. ResNet series. After considering the trade off between granularity and range, we set the numbers of blocks for T 1 and T 2 as {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, and set the numbers of blocks for T 3 as {2, 3, 5, 6, 9, 11, 14, 17, 20, 23}, for T 4 as {2, 3, 4, 6, 7, 9, 11, 13, 15, 17} for the ResNet series. The stage reallocation space of MobileNetV2 and ResNeXt can be found in Appendix A.2. To reallocate the computation across spatial position, we utilize dilated convolution,. Dilated convolution effects the ERF by performing convolution at sparsely sampled locations. Another good feature of dilated convolution is that dilation introduce no extra parameter and computation. We define a choice block to be a basic unit which consists of multiple dilations and search for the best computation allocation. For ResNet Bottleneck, we modify the center 3 × 3 convolution. For ResNet BasicBlock, we only modify the second 3 × 3 convolution to reduce search space and searching time. We have three candidates in our operation set O: {dilated convolution 3 × 3 with dilation rate i|i = 1, 2, 3}. Across the entire ResNet50 search space, there are therefore 3 16 ≈ 4 × 10 7 possible architectures. We propose a hierarchical search procedure to cope with the complex reallocation space. Firstly, the stage space is explored to find the best computation allocation for different resolution. Then, the operation space is explored to further improve the architecture with better spatial allocation. To reduce the side effect of weights coupling, we adopt the uniform sampling in supernet training(a.k.a single-path one-shot) . After the supernet training, we can validate the allocation strategies τ ∈ T directly on the task detection task. Model accuracy(COCO AP) is defined as AP val (N (τ, w)). We set the block number constraint N. We can find the best allocation strategy in the following equation: Operations to be Sampled Figure 3: Evaluation of a choice in block operation search approach. As shown in figure, we have partial architecture of block 1 and block 2, and now we need to evaluate the performance of convolution with dilated rate 3 in the third block. We uniform sample the operation of rest blocks to generate a temporary architecture and then evaluate the choice through several temporary architectures. By introducing the operation allocation space as in Sec. 3.2.2, we can reallocate the computation across spatial position. Same as stage reallocation search, we train an operation supernet adopting random sampling in each choice block . For architecture search process, previous one-shot works use random search or evolutionary search . In our approach, We propose a greedy algorithm to make sequential decisions to obtain the final . We decode network architecture o as a sequential of choices In each choice step, the top K partial architectures are maintained to shrink the search space. We evaluate each candidate operation from the first choice block to the last. The greedy operation search algorithm is shown in Algorithm 1. The hyper-parameter K is set equal to 3 in our experiment. We first extend the partial architecture in the first block choice which contains three partial architectures in p extend. Then we expand the top 3 partial architectures into the whole length B, which means that there are 3 × 3 = 9 partial architectures in other block choice. For a specific partial architecture arch, we sample the operation of the unselected blocks uniformly for c architectures where c denotes mini batch number of D val. We validate each architecture on a mini batch and combine the to generate evaluate(arch). We finally choose the best architecture to obtain o *. Dataset We evaluate our method on the challenging MS COCO benchmark . We split the 135K training images trainval135 into 130K images archtrain and 5K images archval. First, we train the supernet using archtrain and evaluate the architecture using archval. After the architecture is obtained, we follow other standard detectors (; a) on using ImageNet for pre-training the weights of this architecture. The final model is fine-tuned on the whole COCO trainval135 and validated on COCO minival. Another detection dataset VOC is also used. We use VOC trainval2007+trainval2012 as our training dataset and VOC test2007 as our vaildation dataset. The supernet training setting details can be found in Appendix A.1. For the training of our searched models, the input images are resized to have a short side of 800 pixels or a long side of 1333 pixels. We use stochastic gradient descent (SGD) as optimizer with 0.9 momentum and 0.0001 weight decay. For fair comparison, all our models are trained for 13 epochs, known as 1× schedule . We use multi-GPU training over 8 1080TI GPUs with total batch size 16. The initial learning rate is 0.00125 per image and is divided by 10 at 8 and 11 epochs. Warm-up and synchronized BatchNorm (SyncBN) ) are adopted for both baselines and our searched models. We denote the architecture using our computation reallocation by prefix'CR-', e.g. CR-ResNet50. Our final architectures have the almost the same FLOPs as the original network(the negligible difference in FLOPs is from the BatchNorm layer and activation layer). As shown in Table 1, our CR-ResNet50 and CR-ResNet101 outperforms the baseline by 1.9% and 1.6% respectively. It is worth mentioning that many mile-stone backbone improvements also only has around 1.5% gain. For example, the gain is 1.5% from ResNet50 to ResNeXt50-32x4d as indicated in Table 4. In addition, we run the baselines and searched models under longer 2× setting ( shown in Appendix A.4). It can be concluded that the improvement from our approach is consistent. Our CR-ResNet50 and CR-ResNet101 are especially effective for large objects(3.5%, 4.8% improvement for AP l). To understand these improvements, we depict the architecture sketches in Figure 4. We can find in the stage-level, our Stage CR-ResNet50 reallocate more capacity in deep stage. It reveals the fact that the budget in shallow stage is redundant while the resources in deep stage is limited. This pattern is consistent with ERF as in Figure 1. In operation-level, dilated convolution with large rates tends to appear in the deep stage. We explain the shallow stage needs more dense sampling to gather exact information while deep stage aims to recognize large object by more sparse sampling. The dilated convolutions in deep stage further explore the network potential to detect large objects, it is an adaptive way to balance the ERF. For light backbone, our CR-ResNet18 and CR-MobileNetV2 both improves 1.7% AP over the baselines with all-round AP s to AP l improvements. For light network, it is a more efficient way to allocate the limited capacity in the deep stage for the discriminative feature captured in the deep stage can benefit the shallow small object by the FPN top-down pathway. Different dataset We transfer our searched model to another object detection dataset VOC . Training details can be found in Appendix A.3. We denote the VOC metric mAP@0.5 as AP 50 for consistency. As shown in Table 2, our CR-ResNet50 and CR-ResNet101 achieves AP 50 improvement 1.0% and 0.7% comparing with the already high baseline. Different task Segmentation is another task that is highly sensitive to the ERF . Therefore, we transfer our computation reallocation network into the instance segmentation task by using the Mask RCNN framework. The experimental on COCO are shown in Table 3. The instance segmentation AP of our CR-MobileNetV2, CR-ResNet50 and CR-ResNet101 outperform the baseline respectively by 1.2%, 1.3% and 1.1% absolute AP. We also achieve bounding box AP improvement by 1.5%, 1.5% and 1.8% respectively. Different head/neck Our work is orthogonal to other improvements on object detection. We exploit the SOTA detector Cascade Mask RCNN for further verification. The detector equipped with our CR-Res101 can achieve 44.5% AP, better than the regular Res101 43.3% baseline by a significant 1.2% gain. Additionally, we evaluate replacing the original FPN with a searched NAS-FPN neck to strength our . The Res50 with NAS-FPN neck can achieve 39.6% AP while our CR-Res50 with NAS-FPN can achieve 41.0% AP using the same 1× setting. More detailed can be found in Appendix A.4. Our design includes two parts, stage reallocation search and block operation search. In this section, we analyse the effectiveness of stage reallocation search alone. Table 4 shows the performance comparison between the baseline and the baseline with our stage reallocation search. From light MobileNetV2 model to heavy ResNeXt101, our stage reallocation brings a solid average 1.0% AP improvement. Figure 5 shows that our Stage-CR network series yield overall improvements over baselines with negligible difference in computation. The stage reallocation for more models are shown in Appendix A.2. There is a trend to reallocate the computation from shallow stage to deep stage. The intuitive explanation is that reallocating more capacity in deep stage in a balanced ERF as Figure 1 shows and can enhance the ability to detect medium and large object. Often, a large AP increase could be obtained by simply replacing backbone with stronger network, e.g. from ResNet50 to ResNet101 and then to ResNeXt101. The assumption is that strong network can perform well on both classification and detection tasks. We further explore the performance correlation between these two tasks by a lot of experiments. We draw ImageNet top1 accuracy versus COCO AP correlation in Figure 6 for different architectures of the same FLOPS. Each dot is a single network architecture. We can easily find that although the performance correlation between these two tasks is basically positive, better classification accuracy may not always lead to better detection accuracy. This study further shows the gap between these two tasks. In this paper, we present CR-NAS (Computation Reallocation Neural Architecture Search) that can learn computation reallocation strategies across different resolution and spatial position. We design a two-level reallocation space and a novel hierarchical search procedure to cope with the complex search space. Extensive experiments show the effectiveness of our approach. The discovered model has great transfer-ability to other detection neck/head, other dataset and other vision tasks. Our CR-NAS can be used as a plugin to other detection backbones to further booster the performance under certain computation resources. A.1 SUPERNET TRAINING Both stage and operation supernets use exactly the same setting. The supernet training process adopt the'pre-training and fine-tuning' paradigm. For ResNet and ResNeXt, the supernet channel distribution is. Supernet pre-training. We use ImageNet-1k for supernet pre-training. We use stochastic gradient descent (SGD) as optimizer with 0.9 momentum and 0.0001 weight decay. The supnet are trained for 150 epochs with the batch size 1024. To smooth the jittering in the training process, we adopt the cosine learning rate decay with the initial learning rate 0.4. Warming up and synchronized-BN ) are adopted to help convergence. Supernet fine-tuning. We fine tune the pretrained supernet on archtrain. The input images are resized to have a short side of 800 pixels or a long side of 1333 pixels. We use stochastic gradient descent (SGD) as optimizer with 0.9 momentum and 0.0001 weight decay. Supernet is trained for 25 epochs (known as 2× schedule ). We use multi-GPU training over 8 1080TI GPUs with total batch size 16. The initial learning rate is 0.00125 per image and is divided by 10 at 16 and 22 epochs. Warm-up and synchronized BatchNorm (SyncBN) ) are adopted to help convergence. stage allocation space For ResNeXt, the stage allocation space is exactly the same as ResNet series. For MobileNetV2, original block numbers in is defined by n=. We build our allocation space on the the bottleneck operator by fixing stem and tail components. A architecture is represented as Then we search for the spatial allocation by adopting the dilated convolution with different rates. the operation code as. we denote our final model as Our final model can be represnted as a series of allocation codes. We use the VOC trainval2007+trainval2012 to server as our whole training set. We conduct our on the VOC test2007. The pretrained model is apoted. The input images are resized to have a short side of 600 pixels or a long side of 1000 pixels. We use stochastic gradient descent (SGD) as optimizer with 0.9 momentum and 0.0001 weight decay. We train for 18 whole epochs for all models. We use multi-GPU training over 8 1080TI GPUs with total batch size 16. The initial learning rate is 0.00125 per image and is divided by 10 at 15 and 17 epochs. Warm-up and synchronized BatchNorm (SyncBN) ) are adopted to help convergence. longer schedule 2× schedule means training totally 25 epochs as indicated in. The initial learning rate is 0.00125 per image and is divided by 10 at 16 and 22 epochs. Other training settings is exactly the same as in 1×. Powerful detector The Cascade Mask RCNN is a SOTA multi-stage object detector. The detector is trained for 20 epochs. The initial learning rate is 0.00125 per image and is divided by 10 at 16 and 19 epochs. Warming up and synchronized-BN ) are adopted to help convergence. Powerful searched neck NAS-FPN is a powerful scalable feature pyramid architecture searched for object detection. We reimplement NAS-FPN (7 @ 384) in Faster RCNN (The original paper is implemented in RetinaNet (b) ). The detector is training under 1× setting as described in 4.1.

Represent the TLDR for finding the corresponding scientific article

We propose CR-NAS to reallocate engaged  computation resources in different resolution and spatial position.

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: Uncertainty is a very important feature of the intelligence and helps the brain become a flexible, creative and powerful intelligent system. The crossbar-based neuromorphic computing chips, in which the computing is mainly performed by analog circuits, have the uncertainty and can be used to imitate the brain. However, most of the current deep neural networks have not taken the uncertainty of the neuromorphic computing chip into consideration. Therefore, their performances on the neuromorphic computing chips are not as good as on the original platforms (CPUs/GPUs). In this work, we proposed the uncertainty adaptation training scheme (UATS) that tells the uncertainty to the neural network in the training process. The experimental show that the neural networks can achieve comparable inference performances on the uncertain neuromorphic computing chip compared to the on the original platforms, and much better than the performances without this training scheme. Uncertainty reasoning is the essence of human thinking activities and a key aspect of the intelligence. There are two kind of uncertainties in intelligent systems. One is the fuzziness, the other is the stochasticity. The fuzziness helps the brain deal with the real world efficiently by ignoring the enormous redundant information. When we try to distinguish a cat or a dog, we do not need to know the expressions and the number of the legs. Although such information can be easily captured by our visual system with a glance, it will be ignored for efficiency. The stochasticity endows the brain the creativity and enables us not always failed in an unfamiliar field. Our decisions may change when we do not sure. These characteristics are not available in most existing artificial intelligence (AI) systems, such as a classifier based on a deep neural network (DNN). The 32-bit or 64-bit floating numbers are used to describe the weights and activations. While some researchers found that the 8-bit integer is enough for many applications;. Moreover, after the training procedure, the will be the same no matter how many times it performs, although the margin is very small and the answer is wrong. There are some methods to address these issues, such as the network quantization and the Bayesian network. In addition, the neuromorphic computing chip has provide a hardware approach to supplement the missing uncertainty in DNN. In recent years, the emerging nanotechnology device and crossbar structure based neuromorphic computing chips have developed a lot;;. The Ohms law and Kirchhoffs law make the crossbar structure very efficient when doing the vectormatrix multiplication (VMM), and the emerging nanoscale nonvolatile memory (NVM) device at each cross point provides additional storage capability (Figure 1). The crossbar holds the devices conductances as memory in peacetime, and performs the computing function when applied voltages. The so-called computing in memory (CIM) architecture can relieve the memory bottleneck, which is the most serious problem in the von Neumann architecture, and make the neuromorphic computing chips more energy and area efficiency. Therefore, the neuromorphic computing has become a promising approach to realize the AI applications, which is full of VMMs and great memory requirement. Besides the energy and area efficiency, the uncertainty is also an important and intrinsic feature of the neuromorphic computing chips and is not well utilized. Figure 1: The crossbar structure. V is the applied voltage that correspond to the input x, G is the conductance of devices that correspond to the weight W, I is the output current, which can indicates the output y according to the Ohms law and Kirchhoffs law. The uncertainty in the neuromorphic computing chips comes from two aspects. The fuzziness is mainly caused by the analog to digital converters (ADCs) and the stochasticity is mainly induced by the NVM devices. According to the Kirchhoffs law, the VMM is indicated as the summarization of the currents, which is an analog output. It is necessary to use the ADC to convert the analog currents to digital voltages for data transferring. The function of ADC is similar as the activation quantization in the network. The stochasticity of the NVM device is due to the intrinsic physical mechanism;. The random movement of the particles in the device makes the conductance varied. The output current will be different even applying the same voltage. The stochasticity of the device is usually simulated as a non-ideal factor that makes the network perform worse;;. In this work, we proposed a training scheme that utilizes the stochasticity to improve the performance of the neuromorphic computing chips. There are several varieties of NVM devices including phase change memories , filamentary migrating oxide devices , ferroelectric tunnel junction synapses , and so on. The stochasticity of each types of devices is different due to the different intrinsic physical mechanisms. Without loss of generality, we used the Gaussian distribution to model the device stochasticity. Although the Gaussian distribution may not fit the stochasticity of each types of devices well, it has the basic characteristics, that is, a stable state and the farther away from the stable state, the probability is lower. The mean of the Gaussian distribution is the conductance value of the stable state. The variance of the Gaussian distribution is usually corresponded to the mean according to experimental . It is also hard to use a singular model the relations between the mean and the variance of various devices. For simplify and without loss of generality, we assumed that the standard deviation of the distribution is linearly and positively correlated to the mean. Furthermore, the real conductance of device is a positive value. Therefore, the conductances sampled from the distribution which are less than the given value G min will be cut off to G min, where G min > 0 denotes the minimum conductance that all devices could reach. The model of the devices stochasticity is as follow: where G s denotes the stochastic conductance, G denotes the value sampled from the Gaussian distribution N (µ, σ 2) with the a mean of µ and a variance of σ 2, G 0 denotes the conductance of the device when it is in the stable state, and α > 0 denotes the level of the devices stochasticity. The device fuzziness is a by-product of the writing process. Writing the conductance of each device in the crossbar is an essential step when using neuromorphic computing chip to realize an AI application. Before writing, we need to determine the target conductance of each device according to the weights of the neural network. That is what the mapping process do. Besides scaling the weights into the working range of devices conductance [G low, G high], we used the difference of two devices conductances G pos − G neg to express one weight w, which can be positive or negative. In order to achieve higher energy efficiency, it is better to use lower conductances to express the same weight. Moreover, in order to make full use of the entire conductance working range, the mapping algorithm is as follow: where |w| denotes the absolute value of a singular w, max w |w| denotes the maximum of all the |w|, G pos and G neg are the target conductances that we want to write. However, the conductance cannot be written accurately due to the stochasticity of the device and the fuzziness of the circuit. There is a variation when manipulate the conductance of the device and the measurement of the conductance is not accurate. The conductance value is obtained by dividing the read current by the applied voltage, which is obviously affected by the stochasticity of the device and the precision of the ADC. Therefore, we need a model to describe the fuzziness. Without loss of the generality, we also used the Gaussian distribution, which is as follow: where G f denotes fuzzy target conductance, G denotes the value sampled from the Gaussian distribution N µ, σ 2 with the a mean of µ and a variance of σ 2, G target denotes the target conductance determined in the mapping process, and β denotes the level of devices fuzziness, which is correspond to the level of the devices stochasticity, the precision of the ADC, and the writing strategy. For the sake of simplicity, we assumed β a constant in this work. If we program a well-trained DNN into a neuromorphic computing chip directly, the uncertainty will make the performance of the DNN worse, such as decrease the classification accuracy (, Fig. 2). However, the decrease of accuracy can be alleviated by using the proposed uncertainty adaptation training scheme (UATS), and in some cases the accuracy will improved. The core idea of UATS is to tell the uncertainty to the neural networks during training process and guide them to learn how to deal with this situation. The stochasticity model is introduced in every feed forward (FF) process. When a weight w participates in the calculation of the FF process, we use a sample of random variable w s to do the calculation instead of w. w s is defined as: Both G ps and G ns are obtained by the stochasticity model, and the conductances of the stable states G p0 and G n0 are calculated by and according to w, respectively. Actually, the w s can be approximated as if G is significantly larger than G min. where c = 2α The fuzziness model is introduced during the training process. After every k epochs of training, every weight w are replaced by a sample of random variable w f, which is defined as: Both G pf and G nf are obtained by the fuzziness model, and the target conductances G ptarget and G ntarget are calculated by and according to w, respectively. Actually, the w f can be approximated as if G is significantly larger than G min. Besides making the network train in an uncertain way, we also tried to teach it a better way to evaluate the networks performance when there is uncertainty, that is, how to calculate the loss function. The loss function is not calculated by the output of one FF process, but the average output of n FF processes with the same input batch. We evaluated the ideas of UATS on multiple models and datasets. We first investigated the effect of the uncertainty without UATS on MNIST dataset. Two multilayer perceptron (MLP) models (28 × 28-100-10 and 28 × 28-300-10) and a convolutional neural network (CNN) models (LeNet-5) are used. There are 60, 000 images in the training set of the MNIST dataset. We randomly selected 50, 000 images for training, and the other 10, 000 images for validation. The 10, 000 images in the test set were used to calculate the test error. G min = 1µS, G low = 5µS, G high = 50µS were used in the experiments. The models were trained in a normal way and then tested with different level of the uncertainty. The fuzziness model was first used to generate the weights that is actually written in the chip, and then the stochasticity model was used to simulate the read variation. We ran 20 trials for every model with every uncertainty level. The average test error and the standard deviation were reported. As shown in Figure 2, without using the UATS, the uncertainty makes the test errors of both MLP and CNN models higher. The higher level of the uncertainty, the higher the test error. The CNN model (LeNet-5) has the best performance without the uncertainty, but it is also the most affected by the uncertainty (t-test, p < 0.01). It is because the average width of LeNet-5 is smaller than the two MLP models, as the'mlp2' model is more robust to the uncertainty than the'mlp1'. Then we validated the power of UATS. We used UATS to tune the weight of the pre-trained model and also retrained the models from the initial. k=5,n=5 were used in the fine-tuning experiment. The number of epochs is 25. k = 10, n = 5 were used in the retraining experiment and the number of epochs is 100. As Figure 3 shown, UATS can significantly improve the accuracies with the same level of uncertainty in both retraining and fine-tuning experiments. Most of the retraining is better than the finetuning us UATS. When the uncertainty level is small (α = 0.1, β = 0.1), the UATS achieved a comparable to the ideal case. We also validated the power of UATS on CIFAR-10 dataset with a more complicated DNN model ResNet-44. All the training setting was same as the previous work except we used the UATS from the beginning. α = 0.1, β = 0.1, k = 10 were used in these experiments. The shows that UATS can even achieve a lower error rate than the ideal cases with proper hyper-parameters (Figure 4). The UATS performs better when the neural network has more layers. It can be seen as a regularization methods that make the training of DNN easier. The uncertainty is very important in the intelligent system. The Bayesian network is a very useful method to build an uncertain neural network. However, it usually requires that the distribution of each weight is controllable. This is hard to be realized by the neuromorphic computing chip due to the distribution is determined by the devices. Although there may be some methods to manipulate the conductance distribution of the device, it is not as convenient as UATS, which has no additional circuit required. We have tried a series of distributions to model the device stochasticity besides the Gaussian distribution, such as the Laplacian distribution, the uniform distribution, and the asymmetrical distributions, such as the lognormal distribution, the asymmetric Laplacian distribution, and the Bernoulli distribution for devices that have two stable states or the random telegraph noise (RTN). Although the modeled behavior of the device from different distributions is significantly different, the performance of network using each type of distribution with the same mean and variance is similar. It is because the VMM transform the individual distribution of each device to a summarization of a large number of random parameters.p The computation intension of UATS may be a little strong due to the requirement of a large number of random numbers. There are some methods to reduce the requirement of random numbers. Such as samples the weight for every input or every batch instead of the every VMM and using the uncertainty model of VMM instead of the weights. The simulation speed can be accelerated and achieve similar .

Represent the short summary for finding the corresponding scientific article (long)

A training method that can make deep learning algorithms work better on neuromorphic computing chips with uncertainty

represent the tldr for finding the corresponding scientific article

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.