Question: Represent the article for finding a single-sentence TLDR

Question: For a long time, designing neural architectures that exhibit high performance was considered a dark art that required expert hand-tuning. One of the few well-known guidelines for architecture design is the avoidance of exploding or vanishing gradients. However, even this guideline has remained relatively vague and circumstantial, because there exists no well-defined, gradient-based metric that can be computed {\it before} training begins and can robustly predict the performance of the network {\it after} training is complete. We introduce what is, to the best of our knowledge, the first such metric: the nonlinearity coefficient (NLC). Via an extensive empirical study, we show that the NLC, computed in the network's randomly initialized state, is a powerful predictor of test error and that attaining a right-sized NLC is essential for attaining an optimal test error, at least in fully-connected feedforward networks. The NLC is also conceptually simple, cheap to compute, and is robust to a range of confounders and architectural design choices that comparable metrics are not necessarily robust to. Hence, we argue the NLC is an important tool for architecture search and design, as it can robustly predict poor training outcomes before training even begins. Designing neural architectures that perform well can be a difficult process. In particular, the exploding / vanishing gradient problem has been a major challenge for building very deep neural networks at least since the advent of gradient-based parameter learning (; ; BID4 . However, there is still no consensus about which metric should be used for determining the presence of pathological exploding or vanishing gradients. Should we care about the length of the gradient vector , or about the size of individual components of the gradient vector (; ;), or about the eigenvalues of the Jacobian (; ;)? Depending on the metric used, different strategies arise for combating exploding and vanishing gradients. For example, manipulating the width of layers as suggested by e.g.; can greatly impact the size of gradient vector components but tends to leave the length of the entire gradient vector relatively unchanged. The popular He initialization for ReLU networks is designed to stabilize gradient vector length, wheareas the popular Xavier initialization for tanh networks is designed to stabilize the size of gradient vector components. While the papers cited above provide much evidence that gradient explosion / vanishing when defined according to some metrics is associated with poor performance when certain architectures are paired with certain optimization algorithms, it is often unclear how general those are. We make the following core contributions.1. We introduce the nonlinearity coefficient (NLC), a gradient-based measurement of the degree of nonlinearity of a neural network (section 3).2. We show that the NLC, computed in the networks randomly initialized state, is a powerful predictor of test error and that attaining a right-sized NLC is essential for achieving an optimal test error, at least in fully-connected feedforward networks (section 4).The NLC (defined at the top of page 3) combines the Frobenius norm of the Jacobian of a neural network with the global variability of the input data and the global variability of the network's outputs into a single metric. Despite its simplicity, it is tied to many important properties of the network. It is a remarkably accurate predictor of the network's nonlinearity as measured by the relative diameter of the regions in input space that can be well-approximated by a linear function (section 3 and figure 1). It is closely related to the nonlinearity of the individual activation functions used in the network and the degree to which they can be approximated by a linear function (section 5). It is tied to the susceptibility of the network's output to small random input perturbations. We define a neural network f as a function of the input x. Both x and the output f (x) are vectors of fixed dimensionality d in and d out respectively. We assume a prediction framework, where the output is considered to be the prediction and the goal is to minimize the value of the'error' e over this prediction and the label y, in expectation over some data distribution D. I.e., we wish to minimize E (x,y)∼D [e(f (x), y)]. In this paper, e is always classification accuracy. During training, we replace D with the training set and e with the surrogate loss function, which in this paper is always softmax plus cross-entropy. Let J (x):= df (x) dx be the Jacobian of the output with respect to the input. x(i) and f (x, j) denote the i'th component of the input vector and the j'th component of the output vector respectively, where 1 ≤ i ≤ d in and 1 ≤ j ≤ d out. Denote the component-wise standard deviation of a (possibly multivariate) random variable X as S [X]. Finally, let the'quadratic expectation' Q of a random variable X be defined as Q[X] = (E[X 2]), i.e. the generalization of the quadratic mean to random variables. Consider an arbitrary neural network f and assume it has a well-defined bounded domain D and bounded co-domain F. Then the Jacobian J taken at some x ∈ D defines a local linear approximation of the function f around x, i.e. f (x + δ) ≈ f (x) + J (x)δ for sufficiently small δ. The key insight behind the NLC and ultimately behind this paper is that there is a simple criterion for determining whether the approximation can be accurate for a given δ: If f (x) + J (x)δ is far away from F, then because f (x + δ) ∈ F, f (x) + J (x)δ is far away from f (x + δ) and thus it is inaccurate. We can use this insight to establish an approximate bound for the size of δ by referencing a well-known property of the Frobenius norm. Proposition 1. Let A be an m × n matrix and u a random n-dimensional vector of fixed length and uniformly random orientation. Then ||A|| F √ n ||u|| 2 = Q u ||Au|| 2. (See section F for the short proof.)As we also have f (x) ∈ F, we deduce that we must have DISPLAYFORM0 ||δ|| 2 diameter(F) for the local linear approximation at x to be accurate for a randomly oriented δ. In fact, an approximate bound for the diameter of the linearly approximable region at x in a random direction is DISPLAYFORM1. However, the size of this diameter by itself is not sufficient to judge the nonlinearity of f, because it does not take into account the size of the domain D. If D fits entirely within the linearly approximable region, then f is close to a linear function. If D is large compared to the linearly approximable region, we consider f highly nonlinear at x. Hence, we consider instead the approximate bound for the relative diameter of the linearly approximable region in a random direction, DISPLAYFORM2 ||J (x)|| F diameter (D). We posit the inverse of this value, DISPLAYFORM3, as our nonlinearity metric for f at x. Of course, in practice, a network might not have a well-defined, bounded domain and co-domain. Here we make a final modeling assumption. We proxy the diameter with the quadratic expectation of the distance of two random points from the data distribution. Thus our nonlinearity estimate becomes DISPLAYFORM4. And as we further show in section F, the quadratic expectation of this equals the NLC as defined below. Definition 1. The nonlinearity coefficient (NLC) of a network f and inputs x ∼ D is DISPLAYFORM5.The terms S x x(i) and S x f (x, j) denote the standard deviation of the activation of the i'th input neuron and j'th output neuron under the data distribution respectively. Q i (S x x(i)) and Q j (S x f (x, j)) denote the quadratic expectation of these values across all input / output neurons respectively. We DISPLAYFORM6 dout, where C x and C f are the covariance matrices of x and f (x) under the data distribution. Hence, the NLC can be re-written as DISPLAYFORM7. As a sanity check, let's look at the NLC of a linear network. In that case,f (x) = Ax + b for some A and b. Then the NLC becomes DISPLAYFORM8. This value equals 1, for example, if C x is a multiple of the identity or A is orthogonal. We further conjecture that this value is close to 1 for large, random matrices A as they occur in practice in randomly initialized neural networks, though this analysis goes beyond the scope of this paper. Note that an alternative interpretation of the NLC is that it represents the expected sensitivity of the network output with respect to small, randomly oriented changes to the input, normalized by the global variability of the input, output and √ d out. Finally, we refer readers interested in a pictorial illustration of the NLC to section A in the appendix. Computing the NLC It is worth noting that the NLC is cheap to (approximately) compute. Throughout this study, we compute Q i (S x x(i)) and Q j (S x f (x, j)) exactly via a single pass over the training set. Q x ||J (x)|| F can be computed stochastically by backpropagating Gaussian random vectors from the output layer. See section G for details. Now we investigate the empirical properties of the NLC through a large-scale study. Architectures used We sampled architectures at random by varying the depth of the network, the scale of initial weights, scale of initial bias, activation function, normalization method, presence of skip connections, location of skip connections and strength of skip connections. We chose from a set of 8 activation functions (table 1), which were further modified by random dilation, lateral shift and debiasing. For now, we only considered fully-connected feedforward networks, as is (perhaps unfortunately) common in analytical studies of deep networks (e.g. ; BID3). We have no reasons to suspect our will not generalize to CNNs, and we plan to investigate this point in future work. See section C for the full details of our architecture sampling scheme. Datasets used We studied three datasets: MNIST, CIFAR10 and waveform-noise. All our were highly consistent across these datasets. waveform-noise is from the UCI repository of datasets popular for evaluating fully-connected networks . See section D for further details on dataset selection and preprocessing. We sampled 250 architectures per dataset, a total of 750. DISPLAYFORM0 Points shown in blue correspond to architectures that have skip connections. Inset graphs in the bottom right are magnifications of the region 0.8 < N LC < 100. See section E.2 for details. Training protocol We trained each architecture with SGD with 40 different starting learning rates and selected the optimal one via a held-out validation set, independently for each architecture. During each run, we reduced the learning rate 10 times by a factor of 3. All training runs were conducted in 64 bit floating point. See section E for further experimental details and section B.1 for an analysis of how the learning rate search and numerical precision contributed to the outcome of our study. The of this study are shown in figures 1, 2, 3, 4, 7, 8, 9 and 10. All figures except figure 7 are scatter plots where each point corresponds to a single neural architecture. In most graphs, we show the correlation and its statistical significance between the quantities on the x and y axis at the top. Note that if any quantity is plotted in log-scale, the correlation is also using the log of that quantity. For each architecture, we only studied a single random initialization. Given a limited computational budget, we preferred studying a larger number of architectures instead. Note that all values depicted that were computed after training, such as test error or'NLC after training' in FIG2, are based on the training run with the lowest validation error, as described above. The NLC measures nonlinearity First, we verify that the NLC is indeed an accurate measure of nonlinearity. In FIG0, we plot the relative diameter of the linearly approximable regions of the network as discussed in section 3 and defined in section E.1 against the NLC in the randomly initialized state. We find a remarkably close match between both quantities. This shows empirically that our informal derivation of the NLC in section 3 leads to remarkably accurate nonlinearity estimates. The NLC predicts test error In figure 2, we plot the NLC computed in the randomly initialized state, before training, against the test error after training. We find that for all three datasets, the test error is highly related to the NLC. Further, figure 2 indicates that one must start with an NLC in a narrow range, approximately between 1 and 3, to achieve optimal test error, and the further one deviates from that range, the worse the achievable test error becomes. It is worth noting that some architectures, despite having an NLC in or close to this range, performed badly. One cause of this, high output bias, is explored later in this section. To verify that our were not dependent on using the SGD optimizer, we re-trained all 250 waveform-noise architectures with Adam using the same training protocol. In FIG2, we find that the closely match those of SGD from FIG1. A caveat is that we do not currently have a way to detect the ideal NLC range for a given dataset, except through observation, though we find this range to be consistent across our three datasets. FIG2, we plot the value of the NLC before training versus after training. Both values were computed on the training set. We find that for the vast majority of architectures, the value of the NLC decreases. However, if the NLC is very large in the beginning, it remains so. Overall, the before-training NLC significantly predicts the after-training NLC. In FIG2, we plot the after-training NLC versus test error. We find that unless the NLC lies in a narrow range, test error is close to random. Interestingly, the after-training NLC has a significantly lower correlation with test error than the before-training NLC. NLC predicts generalization, not trainability In FIG2, we show the training error achieved for 50 randomly selected architectures for waveform-noise. We re-trained those architectures without using early stopping based on the validation error and considered an even larger range of starting learning rates. We depict the lowest training classification error that was achieved across all learning rates. Points are shown in red for visibility. We find that independently of the NLC, the vast majority of architectures achieve a zero or near-zero training error. This finding is the opposite of that of and , who claim that networks which are highly sensitive to small input perturbations are untrainable. The reason we were able to train these architectures was our extensive learning rate search as well as our decision to train with 64 bit precision. In fact, we found that architectures with high NLC often require very small learning rates and very small parameter updates to train successfully. One architecture required a learning rate as small as 5e-18! See section B.1 for further analysis on this point. In summary, we find that many architectures for which the NLC correctly predicts high test error are nonetheless trainable. Of course, one might expect that a high sensitivity leads to poor generalization. As a sanity check, we conducted an experiment where we corrupted the test set with small random perturbations and measured how large these perturbations could be before the test error increased significantly. We plot this in FIG2. As expected, for the majority of high-NLC architectures, labels can be corrupted and the error increased with incredibly small perturbations. Summary We interpret the observed so far as follows. To generalize, the network must attain a critical NLC after training. This is only possible in practice if the initial NLC is already close. In that case, the networks often learns automatically to adopt a more ideal NLC. However, unless the initial NLC is itself in the critical range, we cannot attain optimal performance. Further predictors: bias and skip connections In figure 2, we mark in red all points corresponding to architectures that have a very biased output, i.e. 1000Q j (S x f (x, j)) < Q j,x f (x, j). We note that all of these architecture attain a high test error, even if their NLC is small. In FIG2, we plot the output bias before training against test error. We find that indeed, to achieve an optimal test error, a low initial bias is required. In section B.2, we further show that just as the NLC, the bias also tends to decline throughout training and that attaining a very low bias after training is even more essential. Table 1: Activation functions used in this study with important metrics. See main text for explanation and section E.4 for details. In FIG1, we plot in blue all points corresponding to architectures that have skip connections. argued that skip connections reduce the gradient growth of general architectures as well as make a further contribution to performance. Correspondingly, we find that skip connections lead to lower NLC values and to lower test errors for a given NLC level. To enable a more convenient visual comparison, we plot for architectures with and without skip connections separately in section B.3. In this section, we expose the connection between the NLC and low-level properties of the activation functions the network uses. Given a 1d activation function τ, we define DISPLAYFORM0 It is easy to check that if the input x is distributed according to a Gaussian with zero mean and covariance matrix σI, and f simply applies τ to each input component, then we have DISPLAYFORM1. Consider a randomly initialized network where each layer is made up of a fully-connected linear operation, batch normalization, and an activation function τ that is applied component-wise. It turns out that if the network is sufficiently wide, the pre-activations of τ are approximately unit Gaussian distributed. This follows from the central limit theorem . Hence, we expect each layer to contribute approximately N LC τ to the nonlinearity of the entire network. To verify this, we train a 2-layer network with batchnorm, which contains a single copy of τ at the single hidden layer. In table 1, we show N LC τ for all 8 activation functions we used (line A), as well as the median empirical NLC over 10 random initializations of the 2-layer network (line B). We indeed find a close match between the two values. We then measure the NLC of 49-layer batchnorm networks, which contain 48 activation functions. For 6 out of 8 activation functions, this NLC (line D) closely matches the exponentiated N LC τ 48 (line C). Hence, we find that nonlinearity compounds exponentially and that the NLC of a network is closely tied to which activation function is used. Note that the reason that the NLC value of the'square' and'odd square' activation functions diverge from N LC τdepth−1 at high depth is because those activation functions are unstable, which causes some inputs to growth with depth whereas the vast majority of inputs collapse to zero. We then verified that N LC τ is a meaningful measure of nonlinearity for an activation function. We computed the best linear fit for each τ given unit Gaussian input and then measured the ratio of the power of the signal filtered by this best linear fit over the power of the preserved signal. In table 1(line E), we find that for ReLU, SELU, tanh, sigmoid and Gaussian activation functions, there is a close correspondence in that this linear approximation error is around N LC τ − 1. While this relationship breaks down for the 3 most nonlinear activation functions, their linear approximation error still exceeds those of the other 5. We conclude that N LC τ is a meaningful measure of nonlinearity and that the NLC of an architecture can be calibrated by changing the linear approximability of the activation functions used. In this section, we discuss how the NLC compares against other metrics that are used for predicting test error in the deep learning field. We focus our analysis on five popular and representative examples: the gradient vector components, the size of gradient vector length, the Lipschitz constant, correlation information preservation, and depth. We find that each metric is susceptible to very simple confounders that render them unreliable in practice. The NLC, by design, is invariant to all these confounders. The first drawback is that this metric is confounded by simple multiplicative rescaling. For example, assume we are using a plain network that begins with a linear operation followed by batch normalization or layer normalization BID2. Then we can re-scale the input data with an arbitrary constant c and not only preserve the output of the network in the initialized state, but the entire trajectory of the parameter during training and therefore the final test error. Yet, multiplying the input by c causes the GVCS to shrink by c. Thus we can arbitrarily control GVCS while preserving test error, and therefore GVCS cannot be used as a direct predictor of test error. We observe a similar effect when the network output is re-scaled with a constant c. This causes GVCS to grow by c. As long as the loss function can handle large incoming values, which softmax+cross-entropy can do at least in some circumstances , we can again control GVCS arbitrarily without compromising performance. This drawback shows up in even more insidious ways. Consider a plain network with activation functions of form τ (s) = 1 + 1 k sin(ks). As k → ∞, τ eliminates all meaningful structure in the input data. The NLC converges to infinity to reflect this. Yet, GVCS is stable. Examining the NLC, we find that the increase in nonlinearity is captured by the Q i (S x f (x, i)) term, but not by the gradient. The crux is that the variability of the network output is down-scaled in proportion to the increase in nonlinearity, and this confounds the GVCS. The same effect occurs with He-initialized plain ReLU networks as depth increases. The second drawback is that the GVCS is also confounded by changing the width of the network. For example, consider a network that begins with a linear operation and has input dimension d in. Then we can increase the input dimension by an integer factor c by duplicating each input dimension c times. We can approximately maintain the learning dynamics by reducing the scale of initial weights of the first linear operator by √ c and the learning rate for that operator by c. Again, this transformation leaves the NLC unchanged but reduces GVCS by √ c, allowing us again to control GVCS without compromising performance. Gradient vector length / Lipschitz constant While less popular than GVCS, these two metrics are also used as a predictor for network performance (e.g. / respectively). Both metrics are susceptible to multiplicative scaling just as GVCS, and the same examples apply. However, in contrast to GVCS, they are not susceptible to a change in input width.;. They claim that preserving the correlation of two inputs as they pass through the network is essential for trainability, and hence also for a low test error. However, this metric also has an important confounder: additive bias. Assume we are using a network that employs batchnorm. Then biases in the features of the input do not significantly affect learning dynamics, as this bias will be removed by the first batchnorm operation. Yet, adding a constant c to the input can arbitrarily increase correlation between inputs without affecting the correlation of the outputs. So, again, the degree of correlation change through the network can be increased arbitrarily without altering network performance. ). Most of these works focus on finding specific functions which can be represented easily by deep networks, but require a prohibitively large number of neurons to represent for a shallow network. In FIG4, we plot the test error achieved by our architectures on CIFAR10 against depth. We find that there is actually a positive correlation between both quantities. We suspect this is mainly because deeper networks tend to have a larger NLC. Of course, depth cannot be used as a direct predictor of performance as it does not account for all the confounders discussed throughout this section. We introduced the nonlinearity coefficient, a measure of neural network nonlinearity that is closely tied to the relative diameter of linearly approximable regions in the input space of the network, to the sensitivity of the network output with respect to small input changes, as well as to the linear approximability of activation functions used in the network. Because of this conceptual grounding, because its value in the randomly initialized state is highly predictive of test error while also remaining somewhat stable throughout training, because it is robust to simple network changes that confound other metrics such as raw gradient size or correlation information, because it is cheap to compute and conceptually simple, we argue that the NLC is the best standalone metric for predicting test error in fully-connected feedforward networks. It has clear applications to neural architecture search and design as it allows sub-optimal architectures to be discarded before training. In addition to a right-sized NLC, we also found that avoiding excessive output bias and using skip connections play important independent roles in performance. This paper makes important contributions to several long-standing debates. We clearly show that neural networks are capable of overfitting when the output is too sensitive to small changes in the input. In fact, our random architecture sampling scheme shows that such architectures are not unlikely to arise. However, overfitting seems to be tied not to depth or the number of parameters, but rather to nonlinearity. In contrast to; , we find that a very high output sensitivity does not harm trainability, but only generalization. This difference is likely caused by our very extensive learning rate search and 64 bit precision training. While the popular guidance for architecture designers is to avoid exploding and vanishing gradients, we argue that achieving an ideal nonlinearity level is the more important criterion. While the raw gradient is susceptible to confounders and cannot be directly linked to meaningful network properties, the NLC captures what appears to be a deep and robust property. It turns out that architectures that were specifically designed to attain a stable gradient, such as He-initialized ReLU networks, in fact display a divergent NLC at great depth. It has been argued that the strength of deep networks lies in their exponential expressivity (e.g. ;). While we show that the NLC indeed exhibits exponential behavior, we find this property to be largely harmful, not helpful, as did e.g.. While very large datasets likely benefit from greater expressivity, in our study such expressivity only leads to lack of generalization rather than improved trainability. In fact, at least in fully-connected feedforward networks, we conjecture that great depth does not confer significant practical benefit. In future work, we plan to study whether the ideal range of NLC values we discovered for our three datasets (1 N LC 3) holds also for larger datasets and if not, how we might predict this ideal range a priori. We plan to investigate additional causes for why certain architectures perform badly despite a right-sized NLC, as well as extend our study to convolutional and densely-connected networks. We are interested in studying the connection of the NLC to e.g. adversarial robustness, quantizability, sample complexity, training time and training noise. Finally, unfortunately, we found the empirical measurement of the NLC to be too noisy to conclusively detect an underfitting regime. We plan to study this regime in future. The goal of this section is to provide an intuitive, graphical explanation of the NLC in addition to the mathematical derivation and analysis in section 3 for readers interested in developing a better intuition of this concept. In FIG5, we illustrate the meaning of the NLC in the case of an example function f with a single input and output dimension, and a bounded domain D and co-domain F. f is a simple sin curve, shown in blue. x 1 and x 2 are two sample inputs. We plot the location of (x 1, f (x 1)) in red and the location of (x 2, f (x 2)) in olive. The thick red and olive lines correspond to the local linear approximation of f at x 1 and x 2 respectively, which is simply the tangent line of the blue curve. The shaded olive and red regions correspond to the intervals in which the local linear approximations fall inside the co-domain F.It is easy to check that the proportion of the domain covered by the red interval and olive interval is DISPLAYFORM0 respectively. The insight behind the NLC is that both linear approximations can only be accurate while they remain inside their respective shaded area, or at least close to it. This is evidently true in both cases, as both tangent lines quickly move away from the co-domain outside the shaded region. In the case of x 2, this bound is also tight as the tangent tracks f closely everywhere in the olive region. However, in the case of x 1, the bound is loose, as the red line completely decoupled from f throughout a large part of the red region. The inverse value, DISPLAYFORM1, can be viewed as the number of shaded regions required to cover the entire domain. The NLC is simply the generalization of this concept to multiple dimensions, where the diameter is proxied with the average distance of two points, and the expectation is taken over the data distribution. It is worth noting that the NLC attempts to measure the ratio of diameter of domain and linearly approximable region, not the ratio of volumes. Informally speaking, the number of linearly approximable regions required to cover the domain behaves as N LC din. In this section, we illustrate the function computed by neural networks at varying levels of nonlinearity. Specifically, in TAB3, we depict the function computed by plain, fully-connected, Heinitialized batchnorm-ReLU networks at seven different depths in their randomly initialized state. We set d out = 3 and set the width of all other layers to 100. We then generated three 100-dimensional Gaussian random inputs x, x and x. We associated each point (a, b, c) that lies on the unit sphere in R 3, i.e. that has a 2 + b 2 + c 2 = 1, with the value ax + bx + cx. We call the sphere of points (a, b, c) associated with these inputs the "input sphere".. Note: Architectures which did not achieve a better-thanrandom test error were omitted in (A) and architectures that did not achieve a better-than-random training error were omitted in (B). We set those thresholds at 80% for CIFAR10 (10 different labels) and 50% for waveform-noise (3 different labels). We propagate each of those inputs forward through the network. We obtain a 3-dimensional output, which we divide by its length. Now the output lies on the unit sphere in R 3. Each point on that "output sphere" is associated with a color as shown in FIG7. Finally, we color each point on the input sphere according to its respective color on the output sphere. These colored input spheres are shown in table 2 as azimuthal projections. The RGB values of colors on the output sphere are chosen so that the R component is largest whenever the first output neuron is largest, the G component is largest whenever the second output neuron is largest and the B component is largest whenever the third output neuron is largest. If we imagine that the output is fed into a softmax operation for 3-class classification, then "purer" colors correspond to more confident predictions. For comparison, we show the NLC on CIFAR10 for batchnormReLU networks of the same depth (median of 10 random initializations). We find that as depth and the NLC of the network increases, the color, and thus the value of the output, change more and more quickly. Of course, this chaotic behavior of the output correspondingly implies smaller linearly approximable regions. In this section, we expand upon findings from our large-scale empirical study that were outlined in section 4. One of the hallmarks of our study was the fact that we conducted an exhaustive search over the starting learning rate for training with SGD. We trained our 750 architectures with 40 different. Note: Architectures which did not achieve a better-than-random test error were omitted in (A) and architectures that did not achieve a better-than-random training error were omitted in (B). We set those thresholds at 80% for CIFAR10 (10 different labels) and 50% for waveform-noise (3 different labels).starting learning rates each. Those learning rates formed a geometric sequence with spacing factor 3. The sequence was not the same for each architecture. In fact, the smallest of the 40 learning rates was chosen so that the weight update could still be meaningfully applied in 32 bit precision. See section E.2 for details. Of course, this was simply a heuristic, with the aim of providing a range of learning rates that would contain the ideal learning rate with very high probability. To verify that this goal was achieved, in figure 7A, we plot a histogram of the index of the training run that yielded the lowest validation error for CIFAR10. The training run with index 1 used the lowest starting learning rate, whereas the training run with index 40 used the largest starting learning rate. Note that we did not plot architectures that did not attain a test error of under 80%, i.e. a nonrandom test error, as for those architectures the learning rate was not chosen meaningfully. We find that while a wide range of training run indeces were chosen, there was a wide margin on each side of training runs that were never chosen. This is precisely what that confirms that, with high probability, we found the ideal learning rate for each architecture that has the potential to generalize. We also retrained 50 randomly chosen waveform-noise architectures without applying early stopping based on the validation error. Instead, we continued training to determine the lowest training classification error that could be achieved. The were plotted in FIG2. For this experiment, we used 60 training runs. Here, the smallest starting learning rate was chosen so that the weight updates could still be meaningfully applied in 64 bit precision. In FIG6, we find that indeed the range of training run indeces used is much wider. For 2 architectures, the chosen training run falls outside the range of the original 40 training runs. We hypothesized that architectures that have very high NLCs and cannot generalize are nonetheless trainable with very small learning rates in 64 bit precision. This is precisely what we find in practice. In FIG8, we plot the NLC in the randomly initialized state against the starting learning rate corresponding to the chosen training run. FIG8 depicts learning rates which minimized validation error on CIFAR10 and figure 8B depicts learning rates which minimized training error on waveform-noise. In other words, we show the same training runs as in FIG6, and again we removed architectures for which generalization / training failed completely, respectively. While the range of learning rates that lead to good generalization falls in a comparatively smaller range, some architectures can be trained successfully with a learning rate as small as 5e-18! In general, the reason for this trend is that a large NLC is associated with large gradients, and these gradients need to be down-scaled to keep weight updates bounded. Intriguingly, FIG8 suggests that as the NLC grows, the learning rate should decay as the square of the NLC. This observation mirrors that of , who observed that the magnitude of weight updates should scale inversely as the gradient increases, which would require the learning rate to scale with the inverse square. In FIG2, we show that high bias before training, defined as Qj,xf (x,j) Qj (Sxf (x,j)), leads to high test error. In FIG9, we investigate this quantity further. In FIG9, we find that just like the NLC, the bias decreases during training in many cases. In fact, it often reaches a near-zero value. In FIG9, we find that this is in fact necessary for the network to achieve a better-than-random test error at all. This is not entirely surprising for a dataset like CIFAR10, where each label occurs equally frequently. In FIG9, we show that many architectures (those near the bottom of the chart) attain a high bias but a low NLC. This confirms that a high makes an independent contribution to test error prediction. All bias values were computed on the training set. Finally, we note that at the time of writing, we are working on an "improved" version of SGD that can successfully train high-bias architectures and enable them to generalize. Discussing this algorithm, as well as the other signals that exist in figure 9 (many architectures cluster around 1D subspaces in all three graphs ...), unfortunately, goes beyond the scope of this paper. In figure 2, we show in blue all architectures that have skip connections, whereas we show in black architectures without skip connections. In that figure, we find that architectures with skip connections not only exhibit a lower NLC overall, but also tend to outperform architectures without skip connections that have similar NLCs. As it can be hard to distinguish colors in a scatter plot, in figure 10, we plot the for both types of architectures separately. Both the first row of graphs (A/B/C) and the second row of graphs (D/E/F) are identical to figure 2, except the top row shows only architectures without skip connections and the bottom row shows only architectures with skip connections. The difference in behavior is clear. In this section, we describe the randomly sampled architectures that we used for our large-scale study. Each network layer is composed out of a fully-connected linear operation with bias and an activation function. Some architectures have a normalization operation between the linear operation and the activation function. The last layer does not contain an activation function. Some architectures have skip connections, which always bypass two layers as in. They start after either the ). The multiplier is chosen to approximately preserve the scale of the incoming signal in the forward pass. This projection matrix is not trained and remains fixed throughout training. Each architecture was selected independently at random via the following procedure.• depth: Depth is chosen uniformly from the set of odd numbers between and including 3 and 49. We used odd numbers to avoid conflicts with our skip connections, each of which bypass two linear operations but do not bypass the first linear operation.• width: Width was chosen automatically as a function of depth so that the number of trainable parameters in the network is approximately 1 million. The width of all layers except the input and output layer, which are determined by the data, is identical.•. We used the multiplier of max(1, doutgoing dincoming) so that the scale of the signal is approximately preserved as it passes forward through the weight matrix, which is a well-accepted practice for avoiding exponential growth or decay in the forward pass used in e.g. He initialization and SELU initialization . With a probability of 50%, we initialize the all trainable bias vectors as zero vectors and with a probability of 50%, we initialize their components as independent zero mean Gaussians with a variance of 0.05. We took the 0.05 value from. If the biases are initialized as nonzero, we scale the weight matrices with a factor of √ 0.95 to approximately preserve the scale of the output of the entire linear operation. Finally, with a 25% probability, we then additionally multiply all weight matrices and biases jointly by 0.9 and with a 25% probability, we multiply them jointly by 1.1.• normalization: With a 50% probability, no normalization is used. With a 25% probability, batch normalization is used. With a 25% probability, layer normalization BID2 ) is used. Normalization operations do not use trainable bias and variance parameters.• activation function: We select one of the 8 activation functions shown in FIG0. We select ReLU, SeLU and Gaussian with probability 2 11 each and tanh, even tanh, sigmoid, square and odd square with probability 1 11 each. We downweighted the probabilities of tanh, even tanh and sigmoid as we considered them similar. The same holds for square and odd square. After choosing the initial activation function, we added additional modifications. If the initial activation function is τ (s), we replace it by cτ (ds + t) + b. First, d and t are chosen. d is 1 with a 50% probability, 1.2 with a 25% probability and 0.8 with a 25% probability. t is 0 with a 50% probability, 0.2 with a 25% probability and -0.2 with a 25% probability. Then, with a 50% probability, we set b so that if s follows a unit Gaussian distribution, τ (s) is unbiased, i.e. E s∼N τ (s) = 0. Debiasing follows the example of BID1. Finally, we always set c so that if s is a unit Gaussian, then Q s τ (s) = 1. Again, this follows the principle of avoiding exponential growth / decay in the forward pass as mentioned above. d, b, c and t are fixed throughout training.• skip connections: With a 50% probability, no skip connections are used. With a 25% probability, skip connections of strength 1 are used, as is usually done in practice. With a 25% chance, we choose a single value uniformly at random between 0 and 1 and set the strength of all skip connections to that value. With a 50% chance, all skip connections start after the linear operation. With a 50% chance, they start after the normalization operation. We introduced these variations to obtain a more diverse range of NLCs amongst networks with skip connections. Note that normalizing the signal between skip connections rather than only within a skip block reduces the gradient damping of the skip connections for reasons related to the k-dilution principle .After sampling, we apply one step of post-processing. All networks that have square or odd square activation functions, or skip connections, that also do not have normalization were assigned either batch normalization or layer normalization with 50% probability each. This is, again, to avoid exponential instability in the forward pass. This post-processing lead to the following changes in aggregate frequencies: no normalization -20.4%, batchnorm -39.8%, layer norm -39.8%.We sampled 250 architectures for each of three datasets. Results pertaining to those architectures are shown in figures 1, 2, 3, 4, 7, 8 and 9.We used softmax+cross-entropy as the loss function, as is done in the overwhelming number of practical cases. Crucially, after initializing each architecture, we measured the scale c of activations fed into the loss function, i.e. c = Q x,j f (x, j). We then had the loss function divide the incoming activations by c before applying softmax. This was done so that the loss functions, which yields very different training dynamics when presented with inputs of different sizes, did not confound the outcomes of our study. We believe that the preference of softmax+cross-entropy for outputs of a certain size has confounded the of studies in the past. c remained fixed throughout training. When designing our sampling scheme, we attempted to strike a balance between relevance and diversity. On the one hand, we did not want to include architectures that are pathological for known reasons. We initialized all architectures so that the signal could not grow or decay too quickly in the forward pass. Also, we always used orthogonal initialization. The advantages of orthogonal initialization over Gaussian initialization, at least for fully-connected layers has, in our opinion, been demonstrated to the point where we believe this should be the default going forward. On the other hand, we introduced many variations such as activation function dilation and shift, and skip connection strength that made our architectures more diverse. While those variations are not necessarily common in practice, we made sure that we never deviated from the "default case" by a large amount in any particular area. We wanted to conduct experiments on three different datasets. First, we chose MNIST and CIFAR10 as they are the two most popular datasets for evaluating deep neural networks, and are small enough so that we could conduct a very large number of training runs with the computational resources we had available. The MNIST dataset is composed of 28 by 28 black and white images of handwritten digits associated with a digit label that is between 0 and 9 (citation: MNIST-dataset). The CIFAR10 dataset is composed of 32 by 32 color images of objects from 10 categories associated with a category label (citation: CIFAR10-dataset).We decided to choose our third dataset from the UCI repository of machine learning datasets. recently validated the SELU nonlinearity, which has since become popular, on a large number of datasets from this repository. We wanted to choose a dataset that also used. To decide upon the specific dataset, we applied the following filters:• The most frequent class should not be more than 50% more frequent than the average class.• The dataset should contain between 1.000 and 100.000 datapoints.• Datapoints should contain at least 10 features.• The dataset should not be composed of images, as we already study 2 image datasets.• The dataset should not contain categorical or very sparse features.• We only considered datasets that we were actually able to find on the repository website. After applying all these filters, we were left with two datasets: waveform and waveform-noise. They are very similar. We chose the latter because of the greater number of input features. The inputs of the waveform-noise dataset are composed of wave attributes. Each input is associated with one of three category labels based on the wave type (citation: waveform-noise dataset). For waveform-noise, we normalized the mean and variance of the features. We processed CIFAR10 via the following procedure.1. We normalize the mean and variance of each datapoint.2. We normalize the mean of each feature.3. Via PCA, we determine the number of dimensions that hold 99% of the variance. That number is 810.4. We map each datapoint to an 810-dimensional vector via multiplication with a 3072 × 810 submatrix of a 3072 × 3072 uniformly random orthogonal matrix.5. Finally, we multiply the entire dataset with a single constant so that we obtain Q x,i x(i) = 1.We used the exact same procedure for MNIST, except that the number of dimensions of the final dataset was 334 instead of 810.During preliminary experiments, we found that this pre-processing scheme lead to faster training and lower error values than training on the raw data where only the features are normalized. The reason we designed this scheme in the first place was to reduce input dimensionality so that we could avoid an excessive amount of computation being allocated to the first layer, which would strain our computational budget. The MNIST dataset contains 60.000 training data points and 10.000 test data points. The training data was randomly split into a training set of size 50.000 and validation set of size 10.000. The CIFAR10 dataset contains 50.000 training data points and 10.000 test data points. The training data was randomly split into a training set of size 40.000 and a validation set of size 10.000. The waveform-noise dataset contains 5.000 data points, which were randomly split into a training set of size 3.000, a validation set of size 1.000 and a test set of size 1.000.As mentioned, for CIFAR10, our input dimensionality was 810. For MNIST, it was 334. For waveform-noise, it was 40. For CIFAR10 and MNIST, the output dimensionality / number of classes was 10. For waveform-noise, it was 3.E EXPERIMENTAL DETAILS E.1 LINEARLY APPROXIMABLE REGIONS STUDY (FIGURE 1) We begin by defining the relative diameter of a linearly approximable region for a given network f, input x from distribution D, input direction u ∈ R din and output direction v ∈ R dout. Starting from x, traversing the input space k times for some fractional value k in the direction of u yields x:= x + ku ||u||22Qi(Sxx(i)). As in the definition of the NLC, we use 2Q i (S x x(i)) as a proxy for the diameter of the input space. The radius of the linearly approximable region, relative to the size of the input space, is then the largest value of k such that the linear approximation induced by the Jacobian at x is still close to the true value of f at x. Therefore, we take the diameter of the linearly approximable region, relative to the size of the input space, as the largest value of k such that the linear approximation induced by the Jacobian at x is still close to the true value of f at x = x + ku ||u||2Qi(Sxx(i)). (Note that x is simply twice as far from x as x .) Specifically, we define "close" as DISPLAYFORM0 In plain words, the change in function value in the direction of v predicted by the local linear approximation must be at least half and at most 2 times the actual change in function value. We now generalize this quantity to minibatches, in order for it to be meaningfully defined for networks using batchnorm. Again, consider some network f. Now also consider a data batch X ∈ R din×b containing b randomly drawn datapoints from D. Also consider an input direction matrix U ∈ R din×b and output direction matrix V ∈ R dout×b. Now we define the relative diameter of the linearly approximable region as the largest k such that when setting X = X + kU ||U ||2Qi(S x a column of X x(i)), we have DISPLAYFORM1 Here f can be taken to be applied independently to each column of X if it does not use batchnorm and is taken to be applied jointly to all inputs in X if f does contain batchnorm. The "largest k" is computed by starting with k = 10 −9 and then checking the condition for increasing k until it fails. The values of k we checked formed a geometric series with spacing factor 10 1 10. We could not reliably check smaller values of k due to numerical underflow, which is why architectures with an NLC less than 10 −9 are not shown in figure 1.For each architecture, we considered a single random initialization. All values were computed in the randomly initialized state. No training was conducted. We use 10 minibatches of size 250 from the respective dataset and draw 10 random Gaussian U and 10 random Gaussian V. We obtain one relative region size value for each of 10 * 10 * 10 = 1000 configurations. Finally, in figure 1, we report the median across those 1000 values for each architecture. The NLC is computed as described in section G.E.2 PREDICTIVENESS STUDY (FIGURES 2, 3, 4, 7, 8, 9 AND 10) For each architecture, we considered a single random initialization. We trained them with SGD using minibatches of size 250. To ensure that there is no bias with regards to learning rate, we tuned the starting learning rate independently for each architecture by conducting a large number of training runs with various starting learning rates. A training run is conducted as follows. We train with the starting learning rate until the validation classification error (VCE) has not decreased for 10 epochs. Then we rewind the state of the network by 10 epochs (when the lowest VCE was achieved), divide the learning rate by 3 and continue training until the validation classification error has not improved for 5 epochs. We divide the learning rate by 3 again, rewind and continue training until the validation classification error has not improved for 5 epochs. This process continues until the step size has been divided by 3 ten times. When the VCE has again not improved for 5 epochs, we rewind one last time and terminate the training run. For each architecture we completed 40 total training runs with 40 different starting learning rates that form a geometric series with spacing factor 3. For each architecture, the smallest starting learning rate considered was computed as follows. We ran the SGD optimizer for 1 epoch with a learning rate of 1 without actually applying the updates computed. For the weight matrix in each layer, we thus obtained one update per minibatch. Let δW lb denote the update obtained for layer l and minibatch b and W l the initial value of the weight matrix in layer l. Finally, we used the value 10 DISPLAYFORM2 as our smallest starting learning rate. The rational behind this choice was that no individual weight matrix update obtained with the smallest starting learning rate would perturb any weight matrix during any iteration by more than approximately 10 −8. We chose 10 −8specifically so that our smallest starting learning rate would be less than the smallest learning rate that can be meaningfully used under 32 bit precision. Nonetheless, we trained all networks using 64 bit precision. Of course, this choice of smallest starting learning rate is merely a heuristic. We validated this heuristic by checking that no architecture that obtained a non-random test error attained its lowest validation error with either the smallest five or largest five starting learning rates. This condition was fulfilled for all architectures and datasets. Henceforth, we refer to the'trained network' as the network that was obtained during the training run that yielded the lowest validation classification error and the'initial network' as the network in the randomly initialized state. In FIG6, we show which training runs were used and in figure 8, we show which learning rates were used, plotted against the NLC of the initial network. The NLC was computed as in section G.In FIG1, we plot the test error of the trained network against the NLC of the initial network, again, computed as in section G. We mark in red all points corresponding to architectures for which 1000Q j (S x f (x, j)) < Q j,x f (x, j) for the initial network. We mark in blue all points corresponding to architectures that have skip connections. In FIG4, we plot depth versus test error of the trained network. In FIG2, we plot the bias value Qj,xf (x,j) Qj (Sxf (x,j)) of the initial network against the test error of the trained network. In FIG2, we plot the NLC of the initial network against the NLC of the trained network. If FIG2, we plot the NLC of the trained network against the test error of the trained network. In both 3B and 3C, the NLC was computed on the training set. However, the value of the NLC computed on the test set was very similar. We further compare the bias of the initial network against the bias of the trained network, against test error and against the NLC of the initial network in FIG9. The bias and NLC were always computed on the training set. In FIG0, we break down the of figure 2 into architectures with skip connections and architectures without skip connections. We then selected 50 random architectures from our 250 waveform-noise architectures. We then trained these architectures again, with two changes to the protocol: We reduced the learning rate by a factor of 3 only once the training classification error had not been reduced for 10 / 5 epochs respectively; and we considered 60 different step sizes which formed a geometric series with spacing factor 3 and start value 10 DISPLAYFORM3. Therefore, we considered even the smallest step size that was meaningful for 64 bit precision training. This change allowed us to successfully train even architectures with very high NLC. See section B.1 for an analysis on this point. The reason we only trained 50 architectures for this scenario is because training can take a very long time without using the validation set for early stopping, leading to considerable computational expense. The are presented in FIG2.Finally, for FIG2, we re-trained our 250 waveform-noise architectures with Adam instead of SGD. The protocol was the same (40 training runs), except before obtaining our measurements for δW lb, we first ran Adam for 4 epochs, again without applying updates, in order to warm-start the running averages. Only then did we run it for another epoch where we actually gathered values for δW lb. Again, we verified that the first and last 5 training runs were never used. We computed the maximal error-preserving perturbation shown in FIG2 similarly to the linearly approximable region relative diameter in section E.1. The difference is that instead of requiring that the local linear approximation be close to the true function, we required that the test error over the path from X to X be at most 5% higher than than the test error at X. The test error'over the path' is defined as the fraction of inputs in the batch that were correctly classified everywhere on the line from X to X. Again, we started with k = 10 −9 and increased it by 10 1 10 at each step, checking whether each input is correctly or incorrectly classified. We chose the 5% threshold so that architectures with a test error of around 90% on CIFAR10 / MNIST would yield finite outcomes. The values shown in FIG2 are the median over 10 * 10 = 100 values obtained from 10 random minibatches of size 250 and 10 Gaussian random direction matrices U. The random direction matrix V used in section E.1 does not come into play here. N LC τ was computed as defined in section 5. N LC τ 48 is simply the exponentiated value. The linear approximation error is computed as (DISPLAYFORM0) 2, whereτ is the best linear fit to τ for inputs drawn from N, i.e. arg minτ linear Q s∼N (τ (s) −τ (s)).NLC was computed as in section G. We show the median across 10 random initializations. The values for different initializations show little variation except for 49-layer networks with square or odd square activation functions. Let A be an m × n matrix and u a random vector of fixed length and uniformly random orientation. Then we have Further, we have DISPLAYFORM0 Here, both x and x are drawn independently from D. Both Q i (S x x(i)) and Q j (S x f (x, j)) can be computed exactly over the dataset in trivial fashion if f does not use batchnorm. If f does use batchnorm, however, Q j (S x f (x, j)) depends on the batch selection. In this case, we replace Q j (S x f (x, j)) in the definition with the NLC by Q j (S X,β f (X, j, β)). Here, X is a data batch matrix of dimensionality d in × b, where b is the minibatch size, each column of X is an independently drawn input from D and β is uniformly drawn from {1, .., b}. f (X, j, β) is the (j, β)'th entry of the output of f when X is jointly propagated through the network. In plain terms, we generalize the NLC to batchnorm networks by joining each input x with every possible minibatch X. We compute Q j (S X,β f (X, j, β)) in practice by simply dividing the dataset once into minibatches of size 250 and then taking the standard deviation of all output activation values observed during this single pass. Now we turn our attention to Q x ||J (x)|| F. Before we tackle this quantity, we show a property of the Frobenius norm similar to that shown in proposition 1. Let A be a m × n matrix and let u be an m-dimensional unit Gaussian vector. Then we have So, specifically, we have Q x ||J (x)|| F = Q x,u ||uJ (x)|| 2 for unit Gaussian u. Therefore, we can estimate Q x ||J (x)|| F stochastically by replacing the loss gradient at the output layer with Gaussian random vectors during backpropagation and taking the quadratic expectation over the input gradient. In practice, we compute Q x ||J (x)|| F by sampling 100 minibatches of size 250 and clamping independently drawn unit Gaussian random vectors at the output each time. Finally, let's look at Q x ||J (x)|| F for batchnorm networks. Again, this quantity is dependent on batch selection. In fact, in the definition of the NLC, we replace Q x ||J (x)|| F by 1 √ b Q X ||J (X)|| F, where J (X) is a bd out × bd in matrix that contains the gradient of every component of f (X) with respect to every component of X. It is easy to check that this value equals Q x ||J (x)|| F for networks without batchnorm. In that case, J (X) simply takes the form of a block-diagonal matrix with the individual J (x) from the batch forming the blocks. As before, we have Q X ||J (X)|| F = Q u,X ||uJ (X)|| 2, where u is a bd out -dimensional unit Gaussian random vector. Hence, just as before, we can stochastically compute Q X ||J (X)|| F by sampling random minibatches and backpropagating what is now effectively a d out × b-dimensional unit Gaussian matrix, and computing the quadratic expectation on the ing input gradient. As before, we sample 100 random minibatches of size 250.

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

It's important to consider optimization in function space, not just parameter space. We introduce a learning rule that reduces distance traveled in function space, just like SGD limits distance traveled in parameter space.


Question: Represent the article for finding a single-sentence TLDR

Question: This paper gives a rigorous analysis of trained Generalized Hamming Networks (GHN) proposed by and discloses an interesting finding about GHNs, i.e. stacked convolution layers in a GHN is equivalent to a single yet wide convolution layer. The revealed equivalence, on the theoretical side, can be regarded as a constructive manifestation of the universal approximation theorem;. In practice, it has profound and multi-fold implications. For network visualization, the constructed deep epitomes at each layer provide a visualization of network internal representation that does not rely on the input data. Moreover, deep epitomes allows the direct extraction of features in just one step, without resorting to regularized optimizations used in existing visualization tools. Despite the great success in recent years, neural networks have long been criticized for their blackbox natures and the lack of comprehensive understanding of underlying mechanisms e.g. in BID3; BID12; BID30; BID29. The earliest effort to interpret neural computing in terms of logic inferencing indeed dated back to the seminal paper of BID24, followed by recent attempts to provide explanations from a multitude of perspectives (reviewed in Section 2).As an alternative approach to deciphering the mysterious neural networks, various network visualization techniques have been actively developed in recent years (e.g. BID11 ; BID28 and references therein). Such visualizations not only provide general understanding about the learning process of networks, but also disclose operational instructions on how to adjust network architecture for performance improvements. Majority of visualization approaches probe the relations between input data and neuron activations, by showing either how neurons react to some sample inputs or, reversely, how desired activations are attained or maximized with regularized reconstruction of inputs BID7; BID20; BID36; BID31; BID23; BID33; BID0. Input data are invariably used in visualization to probe how the information flow is transformed through the different layers of neural networks. Although insightful, visualization approaches as such have to face a critical open question: to what extend the drawn from the analysis of sample inputs can be safely applied to new data?In order to furnish confirmatory answer to the above-mentioned question, ideally, one would have to employ a visualization tool that is independent of input data. This ambitious mission appears impossible at a first glance -the final neuron outputs cannot be readily decomposed as the product of inputs and neuron weights because the thresholding in ReLU activations is input data dependent. By following the principle of fuzzy logic, BID8 recently demonstrated that ReLUs are not essential and can be removed from the so called generalized hamming network (GHN). This simplified network architecture, as reviewed in section 3, facilitates the analysis of neuron interplay based on connection weights only. Consequently, stacked convolution layers can be merged into a single hidden layer without taking into account of inputs from previous layers. Equivalent weights of the merged GHN, which is called deep epitome, are computed analytically without resorting to any learning or optimization processes. Moreover, deep epitomes constructed at different layers can be readily applied to new data to extract hierarchical features in just one step (section 4). Despite the great success in recent years, neural networks have long been criticized for their blackbox natures e.g. in BID3: "they capture hidden relations between inputs and outputs with a highly accurate approximation, but no definitive answer is offered for the question of how they work". The spearhead BID24 attempted to interpret neural computing in terms of logic inferencing, followed by more "recent" interpretations e.g. in terms of the universal approximation framework BID6; BID15, restricted Boltzmann machine BID14, information bottleneck theory BID30, Nevertheless the mission is far from complete and the training of neural networks (especially deep ones) is still a trail-and-error based practice. The early 1990s witnessed the birth of fuzzy neural networks (FNN) BID21; BID13 which attempted to furnish neural networks with the interpretability of fuzzy logic BID34 BID37; BID2. On the other hand, neural networks have been used as a computational tool to come up with both membership functions and fuzzy inference rules BID9 BID32. This joint force endeavour remains active in the new millennium e.g. BID27; BID22; Nauck & Nürnberger; BID19; BID16. Nevertheless, FNNs have been largely overlooked nowadays by scholars and engineers in machine learning (ML) community, partially due to the lack of convincing demonstrations on ML problems with large datasets. The exception case is the recent BID8, which re-interpreted celebrated ReLU and batch normalization with a novel Generalized Hamming Network (GHN) and demonstrated the state-of-the-art performances on a variety of machine learning tasks. While GHNs adopted deep networks with multiple convolution layers, in this paper, we will show how to merge multiple stacked convolution layers into a single yet wide convolution layer. There are abundant empirical evidences backing the belief that deep network structures is preferred to shallow ones BID10, on the other hand, it was theoretically proved by the universal approximation theorem that, a single hidden layer network with non-linear activation can well approximate any arbitrary decision functions BID6 BID15. Also, empirically, it was shown that one may reduce depth and increase width of network architecture while still attaining or outperforming the accuracies of deep and residual network BID35. Nevertheless, it was unclear how to convert a trained deep network into a shallow equivalent network. To this end, the equivalence revealed in Section 3 can be treated as a constructive manifestation of the universal approximation theorem. Various network visualization techniques have been actively developed in recent years, with BID7 interpreting high level features via maximizing activation and sampling; BID20 BID36 learning hierarchical convolutional features via energy or cost minimization; BID31 computing class saliency maps for given images; BID23 reconstructing images from CNN features with an natural image prior applied; BID33 visualizing live activations as well as deep features via regularized optimization; BID0 monitoring prediction errors of individual linear classifiers at multiple iterations. Since all these visualization methods are based on the analysis of examples, the applicability of visualization methods to new data is questionable and no confirmatory answers are provided in a principled manner. The name "deep epitome" is reminiscent of BID17; BID4; BID18; BID5, in which miniature, condensed "epitomes" consisting of the most essential elements were extracted to model and reconstruct a set of given images. During the learning process, the self-similarity of image(s), either in terms of pixel-to-pixel comparison or spatial configuration, was exploited and a "smooth" mapping between epitome and input image pixels was estimated. We briefly review generalized hamming networks (GHN) introduced in BID8 and present in great detail a method to derive the deep epitome of a trained GHN. Note that we follow notations in BID8 with minor modifications for the sake of clarity and brevity. According to BID8, the cornerstone notion of generalized hamming distance (GHD) is defined as g(a, b):= a ⊕ b = a + b − 2 · a · b for any a, b ∈ R. Then the negative GHD is used to quantify the similarity between neuron inputs x and weights w: DISPLAYFORM0 in which L denotes the length of neuron weights e.g. in convolution kernels, and g(w, x) is the arithmetic mean of generalized hamming distance between elements of w and x. By dividing the constant 2 L, becomes the common representation of neuron computing (w · x + b) provided that: DISPLAYFORM1 It was proposed by BID8 that neuron bias terms should follow the condition analytically without resorting to an optimization approach. Any networks that fulfil this requirement are thus called generalized hamming networks (GHN). In the light of fuzzy logic, the negative of GHD quantifies the degree of equivalence between inputs x and weights w, i.e. the fuzzy truth value of the statement x ↔ w where ↔ denotes a fuzzy equivalence relation. Moreover, g(x, x) leads to a measurement of fuzziness in x, which reaches the maximal fuzziness when x = 0.5 and monotonically decreases when x deviates from 0.5. Also it can be shown that GHD followed by a non-linear activation induces a fuzzy XOR connective BID8.When viewed in this GHN framework, the ReLU activation function max(0, 0.5−g(x, w)) actually sets a minimal hamming distance threshold of 0.5 on neuron outputs. BID8 then argued that the use of ReLU activation is not essential because bias terms are analytically set in GHNs. BID8 reported only negligible influences when ReLU was completely skipped for the easy MNIST classification problem. For more challenging CIFAR10/100 classifications, removing ReLUs merely prolonged the learning process but the final classification accuracies remained almost the same. To this end, we restrict our investigation in this paper to those GHNs which have no ReLUs. As illustrated below, this simplification allows for strict derivation of deep epitome from individual convolution layers in GHNs.3.2 GENERALIZED HAMMING DISTANCE AND EPITOME BID8 postulated that one may analyse the entire GHN in terms of fuzzy logic inference rules, yet no elaboration on the analysis was given. Inspired by the universal approximation framework, we show below how to unravel a deep GHN by merging multiple convolution layers into a single hidden layer. We first reformulate the convolution operation in terms of generalized hamming distance (GHD) for each layer, then illustrate how to combine multiple convolution operations across different layers. As said, this combination is only made possible with GHNs in which bias terms strictly follow condition. Without loss of generality, we illustrate derivations and proofs for 1D neuron inputs and weights (with complete proofs elaborated in appendix A). Nevertheless, it is straightforward to extend the derivation to 2D or high dimensions. And appendices B to D illustrate deep epitomes of GHNs trained for 2D MNIST and CIFAR10/100 image classifications. Definition 1. For two given tuples DISPLAYFORM2 1... L, where ⊕ denotes the generalized hamming distance operator. Then the product has following properties, DISPLAYFORM3 K but they are permutation equivalent, in the sense that there exist permutation matrices P and Q such that x DISPLAYFORM4 2. non-linear: in contrast to the standard outer product which is bilinear in each of its entry, the hamming outer product is non-linear since in general x DISPLAYFORM5 where µ ∈ R is a scalar. Therefore, the hamming outer product defined as such is a pseudo outer product. DISPLAYFORM6 M because of the associativity of GHD. This property holds for arbitrary number of tuples. iterated operation: the definition can be trivially extended to multiple tuples DISPLAYFORM0 Definition 2. The convolution of hamming outer product or hamming convolution, denoted *, of two tuples is a binary operation that sums up corresponding hamming outer product entries: DISPLAYFORM1 where the subsets DISPLAYFORM2 DISPLAYFORM3 The hamming convolution has following properties, DISPLAYFORM4 K since the partition subsets S(n) remains the same.2. non-linear: this property is inherited from the non-linearity of the hamming outer product. DISPLAYFORM5 M since the summation of GHDs is non-associative. Note this is in contrast to the associativity of the hamming outer product. iterated operation: likewise, the definition can be extended to multiple tuples x DISPLAYFORM0 Figure 1 illustrates an example in which GHDs are accumulated through two consecutive convolutions. Note that the conversion from the hamming outer products to its convolution is non-invertible, Figure 2: The hamming convolution of two banks of epitomes. Remarks: a) for the inputs A, B the number of epitomes M a must be the same as the number of channels C b; and for the output bank DISPLAYFORM1. b) the notation * refers to the hamming convolution between two banks of epitomes (see Definition 5 for details). The convolution of two single-layered epitomes is treated as a special case with all M a, C a, M b, C b = 1. c) the notation refers to the summation of multiple epitomes of the same length, which is defined in Definition 7. d) multiple (coloured) epitomes in D correspond to different (coloured) epitomes in B; and different (shaded) channels in D correspond to different (shaded) channels of inputs in A.in the sense that, it is impossible to recover individual summands x k ⊕ y l from the summation DISPLAYFORM2 As proved in proposition 4, it is possible to compute the convolution of tuples in two (or more) stacked layers without explicitly recovering individual outer product entries of each layer. Due to the non-linearity of the hamming convolutions, computing the composite of two hamming convolutions is non-trivial as elaborated in Section 3.3. In order to illustrate how to carry out this operation, let us first introduce the epitome of a hamming convolution as follows. Definition 3. An epitome consists of a set of N pairs E = (g n, s n), n = 1,..., N where g n denotes the summation of GHD entries from some hamming convolutions, s n the number of summands or the cardinality of the subset S(n) defined above, and N is called the length of the epitome. A normalized epitome is an epitome with s n = 1 for all n = 1,... N. Any epitome can then be normalized by setting (g n /s n, 1) for all elements. A normalized epitome may also refer to input data x or neuron weights w that are not yet involved in any convolution operations. In the latter case, g n is simply the input data x or neuron weights w. Remark: the summation of GHD entries g n is defined abstractly, and depending on different scenarios, the underlying outer product may operate on arbitrary number of tuples DISPLAYFORM3 Fuzzy logic interpretation: in contrast to the traditional signal processing point of view, in which neuron weights w are treated as parameters of linear transformation and bias terms b are appropriate thresholds for non-linear activations, the generalized hamming distance approach treats w as fuzzy templates and sets bias terms analytically according to. In this view, the normalization g n /s n is nothing but the mean GHD of entries in the subset S(n), which indicates a grade of fitness (or a fuzzy set) between templates w and inputs x at location n. This kind of arithmetic mean operator has been used for aggregating evidences in fuzzy sets and empirically performed quite well in decision making environments (e.g. see BID37).Still in the light of signal processing, the generalized hamming distance naturally induces an information enhancement and suppression mechanism. Since the gradient of g(x, w) with respect to x is 1 − 2w, the information in x is then either enhanced or suppressed according to w: a) the output g(x, w) is always x for w = 0 (conversely 1 − x for w = 1) with no information loss in x; b) for w = 0.5, the output g(x, w) is always 0.5 regardless of x, thus input information in x is completely suppressed; c) for w < 0.0 or w > 1.0 information in x is proportionally enhanced. It was indeed observed, during the learning process in our experiments, a small faction of prominent feature pixels in weights w gradually attain large positive or negative values, so that corresponding input pixels play decisive roles in classification. On the other hand, large majority of obscure pixels remain in the fuzzy regime near 0.5, and correspondingly, input pixels have virtually no influence on the final decision (see experimental in Section 4). This observation is also in accordance with the information compression interpretation advocated by BID30, and the connection indicates an interesting research direction for future work. This subsection only illustrates main concerning how to merge multiple hamming convolution operations in stacked layers into a single-layer of epitomes i.e. deep epitome. Detailed proofs are given in appendix A. Theorem 10. A generalized hamming network consisting of multiple convolution layers, is equivalent to a bank of epitome, called deep epitome [D], which can be computed by iteratively applying the composite hamming convolution in equation to individual layer of epitomes: DISPLAYFORM0 in which = C a is the number of channels in the first bank A, = M z is the number of epitomes in the last bank Z, and DISPLAYFORM1 is the length of composite deep epitome. Note that for the hamming convolution to be a valid operation, the number of epitomes in the previous layer and the number channels in the current layer must be the same e.g. DISPLAYFORM2 Proof. For given inputs represented as a bank of normalized epitomes DISPLAYFORM3 Ly Cx ] is obtained by recursively applying equation to outputs from the previous layers, and factoring out the input due to the associativity proved in proposition 9: DISPLAYFORM4 Remark: due to the non-linearity of underlying hamming outer products, to prove the associativity of the convolution of epitomes is by no means trivial (see proposition 9). In essence, we have to use proposition 4 to compute the convolution of two epitomes even though individual entries of the underlying hamming outer product are not directly accessible. Consequently, the updating rule outlined in equations and play the crucial role in setting due bias terms analytically for generalized hamming networks (GHN), as opposed to the optimization approach often adopted by many non-GHN deep convolution networks. Fuzzy logic inferencing with deep epitomes: Eq. can be treated as a fuzzy logic inferencing rule, with which elements of input x are compared with respect to corresponding elements of deep epitomes d. More specifically, the negative of GHD quantifies the degree of equivalence between inputs x and epitome weights d, i.e. the fuzzy truth value of the assertion x ↔ d where ↔ denotes a fuzzy logical biconditional. Therefore, output scores in y indicate the grade of fuzzy equivalences truth values between x and the shifted d at different spatial locations. This inferencing rule, in the same vein of BID8, is applicable to either a single layer neuron weights or the composite deep epitomes as proved by. Constructive manifestation of the universal approximation theorem: it was proved that a single hidden layer network with non-linear activation can well approximate any arbitrary decision functions BID6; BID15, yet it was also argued by BID10 that such a single layer may be infeasibly large and may fail to learn and generalize correctly. Theorem 10 proves that such a simplified single hidden layer network can actually be constructed from a trained GNH. In this sense Theorem 10 illustrates a concrete solution which materializes the universal approximation theorem. We illustrate below deep epitomes extracted from three generalized hamming networks trained with MNIST, CIFAR10/100 classification respectively. Detailed descriptions about the network architectures (number of layers, channels etc.) are included in the appendix. Deep epitomes derived in the previous section allows one to build up and visualize hierarchical features in an on-line manner during the learning process. This approach is in contrast to many existing approaches, which often apply additional optimization or learning processes with various type of regularizations e.g. in BID7 BID33. Figures 5, 8 and 11, 12 in appendices illustrate deep epitomes learnt by three generalized hamming networks for the MNIST and CIFAR10/100 image classification tasks. It was observed that geometrical structures of hierarchical features were formed at different layers, rather early during the learning process (e.g. 1000 out of 10000 iterations). Substantial follow up efforts were invested on refining features for improved details. The scrutinization of normalized epitome histograms in FIG2 showed that a majority of pixel values remain relatively small during the learning process, while a small fraction of epitome weights gradually accumulate large values over thousands of iterations to form prominent features. The observation of sparse features has been reported and interpreted in terms of sparse coding e.g. BID26 or the information compression mechanism as advocated by BID30. we adopt the notion of fuzziness (also reviewed in Section 3.1) to provide a fuzzy logic interpretation: prominent features correspond to neuron weights with low fuzziness. It was indeed observed in FIG4 that fuzziness of deep epitomes in general decrease during the learning process despite of fluctuations at some layers. The inclination towards reduced fuzziness seems in accord with the minimization of classification errors, although the fuzziness is not explicitly minimized. Finally we re-iterate that the internal representation of deep epitomes is input data independent. For instance in MNIST handwritten images, it is certain constellations of strokes instead of digits that are learnt at layer 3 (see Figure 5). The matching of arbitrary input data with such "fuzzy templates" is then quantified by the generalized hamming distance, and can be treated as generic fuzzy logic It must be noted that the extraction of these hierarchical salient features is not entirely new and has been reported e.g. in BID7 BID20. Nevertheless, the equivalence of deep epitomes disclosed in Theorem 10 leads to an unique characteristic of GHNs -deep layer features do not necessarily rely on features extracted from previous layers, instead, they can be extracted in one step using deep epitomes at desired layers. For extremely deep convolution networks e.g. those with over 100 layers, this simplification may bring about substantial reduction of computational and algorithmic complexities. This potential advantage is worth follow up exploration in future research. We have proposed in this paper a novel network representation, called deep epitome, which is proved to be equivalent to stacked convolution layers in generalized hamming networks (GHN). Theoretically this representation provides a constructive manifestation for the universal approximation theorem BID6 BID15, which states that a single layered network, in principle, is able to approximate any arbitrary decision functions up to any desired accuracy. On the other hand, it is a dominant belief BID10, which is supported by abundant empirical evidences, that deep structures play an indispensable role in decomposing the combinatorial optimization problem into layer-wise manageable sub-problems. We concur with the view and supplement with our demonstration that, a trained deep GHN can be converted into a simplified networks for the sake of high interpretability, reduced algorithmic and computational complexities. The success of our endeavours lies in the rigorous derivation of convolving epitomes across different layers in eq. and, which set due bias terms analytically without resorting to optimizationbased approaches. Consequently, deep epitomes at all convolution layers can be computed without using any input data. Moreover, deep epitomes can be used to extract hierarchical features in just one step at any desired layers. In the light of fuzzy logic, the normalized epitome (definition 3) encodes a grade of fitness between the learnt templates and given inputs at certain spatial locations. This fuzzy logic interpretation furnishes a refreshing perspective that, in our view, will open the black box of deep learning eventually. APPENDIX A Definition 1. For two given tuples DISPLAYFORM0.., y L }, the hamming outer product, denoted, is a set of corresponding elements x DISPLAYFORM1.. L, where ⊕ denotes the generalized hamming distance operator. Then the product has following properties, DISPLAYFORM2 K but they are permutation equivalent, in the sense that there exist permutation matrices P and Q such that x DISPLAYFORM3 2. non-linear: in contrast to the standard outer product which is bilinear in each of its entry, the hamming outer product is non-linear since in general x DISPLAYFORM4 where µ ∈ R is a scalar. Therefore, the hamming outer product defined as such is a pseudo outer product. DISPLAYFORM5 M because of the associativity of GHD. This property holds for arbitrary number of tuples. iterated operation: the definition can be trivially extended to multiple tuples DISPLAYFORM0 Proof. associativity: by definition it suffices to prove element-wise (x k ⊕y l)⊕z m = x k ⊕(y l ⊕z m) because of the associativity of the generalized hamming distance. DISPLAYFORM1, then it suffices to prove non-linearity for each element i.e. DISPLAYFORM2 Definition 2. The convolution of hamming outer product or hamming convolution, denoted *, of two tuples is a binary operation that sums up corresponding hamming outer product entries: DISPLAYFORM3 where the subsets S(n):= {(k, l) k + (L − l) = n} for n = 1,..., K + L − 1, and the union of all subsets constitute a partition of all indices n=1,...,K+L−1 DISPLAYFORM4 The hamming convolution has following properties, DISPLAYFORM5 K since the partition subsets S(n) remains the same.2. non-linear: this property is inherited from the non-linearity of the hamming outer product.3. non-associative: DISPLAYFORM6 M since the summation of GHDs is non-associative. Note this is in contrast to the associativity of the hamming outer product. iterated operation: likewise, the definition can be extended to multiple tuples x DISPLAYFORM0 Proof. non-associativity: by definition it suffices to prove element-wise in general DISPLAYFORM1 Definition 3. An epitome consists of a set of N pairs E = (g n, s n), n = 1,..., N where g n denotes the summation of GHD entries from some hamming convolutions, s n the number of summands or the cardinality of the subset S(n) defined above, and N is called the length of the epitome. A normalized epitome is an epitome with s n = 1 for all n = 1,... N. Any epitome can then be normalized by setting (g n /s n, 1) for all elements. A normalized epitome may also refer to input data x or neuron weights w that are not yet involved in any convolution operations. In the latter case, g n is simply the input data x or neuron weights w. Proposition 4. Given two tuples x = {x k |k = 1 . . . K} and y = {y l |l = 1 . . . L}, then DISPLAYFORM2 Proof. DISPLAYFORM3 Remark: eq. allows one to compute summation of all hamming outer product elements on the right hand side, even though individual elements x k and y l are unable to recover from the given summands k x k and l y l. The definition below immediately follows and illustrates how to merge elements of two epitomes. DISPLAYFORM4 the convolution of two epitomes E c = E a * E b is given by: DISPLAYFORM5 where Remark: this operation is applicable to the case when two epitomes are merged via spatial convolution (see Figure 2 for an example). Note that this merging operation is associative due to the following theorem. Theorem 6. The convolution of multiple epitomes, as defined in 5, is associative: DISPLAYFORM6 DISPLAYFORM7 DISPLAYFORM8 Proof. By definition 5, elements of E a * Remark: this associative property is of paramount importance for the derivation of deep epitomes, which factor out the inputs x from subsequent convolutions with neuron weights w. Definition 7. Given two epitomes of the same size E a = {(g n, s n)|n = 1,... N }, E b = {(g n, s n)|n = 1,... N }, the summation of two epitomes E c = E a E b is trivially defined by element-wise summation:E c = {(g n, s n)|n = 1,..., N }; where g n = g n + g n, s n = s n + s n. Remark: the summation operation is applicable to the case when epitomes are (iteratively) merged cross different channels (see Figure 2 for an example). Note that the size of two input epitomes must be the same, and the size of output epitome remain unchanged. Moreover, the operation is trivially extended to multiple epitomes DISPLAYFORM0 The output of this operation, in turn, is a bank with Figure 2 for an example. DISPLAYFORM1 Proposition 9. The composite convolutions of multiple epitome banks, as given in definition 8, is associative: DISPLAYFORM2 Proof. The associativity immediately follows the associativity of Theorem 6 and definition 7.Remark: this associative property, which is inherited from theorem 6, can be trivially extended to multiple banks and lead to the main theorem of the paper as follows. Theorem 10. A generalized hamming network consisting of multiple convolution layers, is equivalent to a bank of epitome, called deep epitome [D], which can be computed by iteratively applying the composite hamming convolution in equation to individual layer of epitomes: DISPLAYFORM3 in which = C a is the number of channels in the first bank A, = M z is the number of epitomes in the last bank Z, and = L a +(L b −1)+...+(L z −1) is the length of composite deep epitome. Note that for the hamming convolution to be a valid operation, the number of epitomes in the previous layer and the number channels in the current layer must be the same e.g. DISPLAYFORM4 Proof..APPENDIX B: DEEP EPITOMES WITH MNIST HANDWRITTEN RECOGNITION Figure 5: Deep epitomes at layers 1,2 and 3 for a GHN trained with MNIST classification at iterations 100 and 10000 respectively.. Pseudo colour images correspond to three channels of features outputs for input RGB colour channels.

Represent the short summary for finding the corresponding scientific article (long)!

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.