Question: Represent the scientific article to find a short TLDR

Question: The recent success of neural networks for solving difficult decision tasks has incentivized incorporating smart decision making "at the edge." However, this work has traditionally focused on neural network inference, rather than training, due to memory and compute limitations, especially in emerging non-volatile memory systems, where writes are energetically costly and reduce lifespan. Yet, the ability to train at the edge is becoming increasingly important as it enables applications such as real-time adaptability to device drift and environmental variation, user customization, and federated learning across devices. In this work, we address four key challenges for training on edge devices with non-volatile memory: low weight update density, weight quantization, low auxiliary memory, and online learning. We present a low-rank training scheme that addresses these four challenges while maintaining computational efficiency. We then demonstrate the technique on a representative convolutional neural network across several adaptation problems, where it out-performs standard SGD both in accuracy and in number of weight updates. Deep neural networks have shown remarkable performance on a variety of challenging inference tasks. As the energy efficiency of deep-learning inference accelerators improves, some models are now being deployed directly to edge devices to take advantage of increased privacy, reduced network bandwidth, and lower inference latency. Despite edge deployment, training happens predominately in the cloud. This limits the privacy advantages of running models on-device and in static models that do not adapt to evolving data distributions in the field. Efforts aimed at on-device training address some of these challenges. Federated learning aims to keep data on-device by training models in a distributed fashion (Konecný et al., 2016). On-device model customization has been achieved by techniques such as weight-imprinting , or by retraining limited sets of layers. On-chip training has also been demonstrated for handling hardware imperfections . Despite this progress with small models, on-chip training of larger models is bottlenecked by the limited memory size and compute horsepower of edge processors. Emerging non-volatile (NVM) memories such as resistive random access memory (RRAM) have shown great promise for energy and area-efficient inference . However, on-chip training requires a large number of writes to the memory, and RRAM writes cost significantly more energy than reads (e.g., 10.9 pJ/bit versus 1.76 pJ/bit ). Additionally, RRAM endurance is on the order of 10 6 writes , shortening the lifetime of a device due to memory writes for on-chip training. In this paper, we present an online training scheme amenable to NVM memories to enable next generation edge devices. Our contributions are an algorithm called Streaming Kronecker Sum Approximation (SKS), and its analysis, which addresses the two key challenges of low write density and low auxiliary memory; two techniques "gradient max-norm" and "streaming batch norm" to help training specifically in the online setting; a suite of adaptation experiments to demonstrate the advantages of our approach. Efficient training for resistive arrays. Several works have aimed at improving the efficiency of training algorithms on resistive arrays. Of the three weight-computations required in training (forward, backprop, and weight update), weight updates are the hardest to parallelize using the array structure. Stochastic weight updates allow programming of all cells in a crossbar at once, as opposed to row/column-wise updating. Online Manhattan rule updating can also be used to update all the weights at once. Several works have proposed new memory structures to improve the efficiency of training . The number of writes has also been quantified in the context of chip-in-the-loop training . Distributed gradient descent. Distributed training in the data center is another problem that suffers from expensive weight updates. Here, the model is replicated onto many compute nodes and in each training iteration, the mini-batch is split across the nodes to compute gradients. The distributed gradients are then accumulated on a central node that computes the updated weights and broadcasts them. These systems can be limited by communication bandwidth, and compressed gradient techniques have therefore been developed. , the gradients are accumulated over multiple training iterations on each compute node and only gradients that exceed a threshold are communicated back to the central node. In the context of on-chip training with NVM, this method helps reduce the number of weight updates. However, the gradient accumulator requires as much memory as the weights themselves, which negates the density benefits of NVM. Low-Rank Training. Our work draws heavily from previous low-rank training schemes that have largely been developed for use in recurrent neural networks to uncouple the training memory requirements from the number of time steps inherent to the standard truncated backpropagation through time (TBPTT) training algorithm. Algorithms developed since then to address the memory problem include Real-Time Recurrent Learning (RTRL) , Unbiased Online Recurrent Optimization (UORO) , Kronecker Factored RTRL (KF-RTRL) , and Optimal Kronecker Sums (OK) . These latter few techniques rely on the weight gradients in a weight-vector product looking like a sum of outer products (i.e., Kronecker sums) of input vectors with backpropagated errors. Instead of storing a growing number of these sums, they can be approximated with a low-rank representation involving fewer sums. The meat of most deep learning systems are many weight matrix -activation vector products W · a. Fully-connected (dense) layers use them explicitly: layer, where σ is a non-linear activation function (more details are discussed in detail in Appendix C.1). Recurrent neural networks use one or many matrix-vector products per recurrent cell. Convolutional layers can also be interpreted in terms of matrix-vector products by unrolling the input feature map into strided convolution-kernel-size slices. Then, each matrix-vector product takes one such input slice and maps it to all channels of the corresponding output pixel (more details are discussed in Appendix C.2). The ubiquity of matrix-vector products allows us to adapt the techniques discussed in "Low-Rank Training" of Section 2 to other network architectures. Instead of reducing the memory across time steps, we can reduce the memory across training samples in the case of a traditional feedforward neural network. However, in traditional training (e.g., on a GPU), this technique does not confer advantages. Traditional training platforms often have ample memory to store a batch of activations and backpropagated gradients, and the weight updates ∆W can be applied directly to the weights W once they are computed, allowing temporary activation memory to be deleted. The benefits of low-rank training only become apparent when looking at the challenges of proposed NVM devices: Low write density (LWD). In NVM, writing to weights at every sample is costly in energy, time, and endurance. These concerns are exacerbated in multilevel cells, which require several steps of an iterative write-verify cycle to program the desired level. We therefore want to minimize the number of writes to NVM.. NVM is the densest form of memory. In 40nm technology, RRAM 1T-1R bitcells @ 0.085 um 2 are 2.8x smaller than 6T SRAM cells @ 0.242 um 2 . Therefore, NVM should be used to store the memory-intensive weights. By the same token, no other on-chip memory should come close to the size of the on-chip NVM. In particular, if our b−bit NVM stores a weight matrix of size n o × n i, we should use at most r(n i + n o)b auxiliary non-NVM memory, where r is a small constant. Despite these space limitations, the reason we might opt to use auxiliary (large, high endurance, low energy) memory is because there are places where writes are frequent, violating LWD if we were to use NVM. In the traditional minibatch SGD setting with batch size B, an upper limit on the write density per cell per sample is easily seen: 1/B. However, to store such a batch of updates without intermediate writes to NVM would require auxiliary memory proportional to B. Therefore, a trade-off becomes apparent. If B is reduced, LAM is satisfied at the cost of LWD. If B is raised, LWD is satisfied at the cost of LAM. Using low-rank training techniques, the auxiliary memory requirements are decoupled from the batch size, allowing us to increase B while satisfying both LWD and LAM 1. Additionally, because the low-rank representation uses so little memory, a larger bitwidth can be used, potentially allowing for gradient accumulation in a way that is not possible with low bitwidth NVM weights. In the next section, we elaborate on the low-rank training method. Let z (i) = W a (i) + b be the standard affine transformation building block of some larger network, e.g., y where. A minibatch SGD weight update accumulates this gradient over B samples: For a rank-r training scheme, approximate the sum by iteratively updating two rankr matricesL ∈ R no×r,R ∈ R ni×r with each new outer product:. Therefore, at each sample, we convert the rank-q = r + 1 systemLR + dz into the rank-rLR. In the next sections, we discuss how to compute rankReduce. One option for rankReduce(X) to convert from rank q = r + 1 X to rank r is a minimum error estimator, which is implemented by selecting the top r components of a singular value decomposition (SVD) of X. However, a naïve implementation is computationally infeasible and biased: solves these problems by proposing a minimum variance unbiased estimator for rankReduce, which they call the OK algorithm 2. The OK algorithm can be understood in two key steps: first, an efficient method of computing the SVD of a Kronecker sum; second, a method of splitting the singular value matrix Σ into two rank-r matrices whose outer product is a minimum-variance, unbiased estimate of Σ. Details can be found in their paper, however we include a high-level explanation in Sections 4.1.1 and 4.1.2 to aid our discussions. Note that our variable notation differs from. Recall that rankReduce should turn rank-q LR into an updated rank-rLR. q×q. Then we can find the SVD of & ), making it computationally feasible on small devices. Now we have: which gives the SVD of LR since Q L U C and Q R V C are orthogonal and Σ is diagonal. This SVD computation has a time complexity of O((n i +n o +q)q 2 ) and a space complexity of O((n i +n o +q)q). , it is shown that the problem of finding a rank-r minimum variance unbiased estimator of LR can be reduced to the problem of finding a rank-r minimum variance unbiased estimator of Σ and plugging it in to. Further, it is shown that such an optimal approximator for Σ = diag(σ 1, σ 2, . . ., σ q), where σ 1 ≥ σ 2 ≥ · · · ≥ σ q will involve keeping the m − 1 largest singular values and mixing the smaller singular values σ m,..., σ q within their (k + 1) × (k + 1) submatrix with m, k defined below. Let: Note that ||x 0 || 2 = 1. Let X ∈ R (k+1)×(k) be orthogonal such that its left nullspace is the span of x 0. Then XX = I − x 0 x 0. Now, let s ∈ {−1, 1} (k+1)×1 be uniform random signs and define: where is an element-wise product. ThenΣ LΣ R =Σ is a minimum variance, unbiased 3 rank-r approximation of Σ. PluggingΣ into, gives us a minimum variance, unbiased, rank-r approximationLR. Although the standalone OK algorithm presented by has good asymptotic computational complexity, our vector-vector outer product sum use case permits further optimizations. In this section we present these optimizations, and we refer readers to the explicit implementation called Streaming Kronecker Sum Approximation (SKS) in Algorithm 1 of Appendix A. The main optimization is a method of avoiding recomputing the QR factorization of L and R at every step. Instead, we keep track of orthogonal matrices Q L, Q R, and weightings c x such that Upon receiving a new sample, a single inner loop of the numerically-stable modified Gram-Schmidt (MGS) algorithm (Björck, 1967) can be used to update Q L and computed during MGS can be used to find the new value of C = c L c R + diag(c x). 3 The fact that it is unbiased: E[Σ] = Σ can be easily verified. After computingΣ L =Σ R in, we can orthogonalize these matrices intoΣ. With this formulation, we can maintain orthogonality in Q L, Q R by setting: These matrix multiplies require O((n i +n o)q 2 ) multiplications, so this optimization does not improve asymptotic complexity bounds. This optimization may nonetheless be practically significant since matrix multiplies are easy to parallelize and would typically not be the bottleneck of the computation compared to Gram-Schmidt. The next section discusses how to orthogonalizeΣ L efficiently and why Orthogonalization ofΣ L is relatively straightforward. From, the columns ofΣ L are orthogonal since Z is orthogonal. However, they do not have unit norm. We can therefore pull out the norm into a separate diagonal matrix R x with diagonal elements √ c x: We generated X by finding an orthonormal basis that was orthogonal to a vector x 0 so that we could have XX = I − x 0 x 0. An efficient method of producing this basis is through Householder matrices (x 0, X) = I − 2 vv /||v|| 2 where v = x 0 − e and (x 0, X) is a k + 1 × k + 1 matrix with first column x 0 and remaining columns X (; user1551, 2013 . The OK/SKS methods require O((n i + n o + q)q 2 ) operations per sample and O(n i n o q) operations after collecting B samples, giving an amortized cost of O((n i + n o + q)q 2 + n i n o q/B) operations per sample. Meanwhile, a standard approach expands the Kronecker sum at each sample, costing O(n i n o) operations per sample. If q B, n i, n o then the low rank method is superior to minibatch SGD in both memory and computational cost. SKS introduces variance into the gradient estimates, so here we analyze the implications for online convex convergence. We analyze the case of strongly convex loss landscapes f t (w t) for flattened weight vector w t and online sample t. In Appendix B, we show that with inverse squareroot learning rate, when the loss landscape Hessians satisfy 0 ≺ cI ∇ 2 f t (w t) and under constraint for the size of gradient errors ε t, where w * is the optimal offline weight vector, the online regret is sublinear in the number of online steps T. We can approximate ||ε|| and show that convex convergence is likely when is satisfied in the biased, zero-variance case (equivalent to raw SVD, i.e., not applying Section 4.1.2), or when is satisfied in the unbiased, minimum-variance case. Equations suggest conditions under which fast convergence may be more or less likely and also point to methods for improving convergence. We discuss these in more detail in Appendix B.3. We validate with several linear regression experiments on a static input batch X ∈ R and target Y t ∈ R 256×100. In Figure 1 (a), Gaussian noise at different strengths (represented by different colors) is added to the true batch gradients at each update step. Notice that convergence slows significantly to the right of the dashed lines, which is the region where no longer holds 4. In Figure 1 (b), we validate Equations by testing the SVD and SKS cases with rank r = 10. In these particular experiments, SKS adds too much variance, causing it to operate to the right of the dashed lines. However, both SVD and SKS can be seen to reduce their variance as training progresses. In the case of SVD, it is able to continue training as it tracks the right dashed line. Quantization. The NN is quantized in both the forward and backward directions with uniform power-of-2 quantization, where the clipping ranges are fixed at the start of training 5. Weights are quantized to 8 bits between -1 and 1, biases to 16 bits between -8 and 8, activations to 8 bits between 0 and 2, and gradients to 8 bits between -1 and 1. Both the weights W and weight updates ∆W are quantized to the same LSB so that weights cannot be used for accumulation beyond the fixed quantization dynamic range. This is in contrast to using high bitwidth or floating point accumulators. See Appendix D for more details on quantization. Gradient Max-Norming. State-of-the-art methods in training, such as Adam , use auxiliary memory per parameter to normalize the gradients. Unfortunately, we lack the memory budget to support these additional variables, especially if they must be updated every sample 6. Instead, we propose dividing each gradient tensor by the maximum absolute value of its elements. This stabilizes the range of gradients across samples. See Appendix E for more details on gradient max-norming. In the experiments, we refer to this method as "max-norm" (opposite "no-norm"). Streaming Batch Normalization. Batch normalization is a powerful technique for improving training performance which has been suggested to work by smoothing the loss landscape . We hypothesize that this may be especially helpful when parameters are quantized as in our case. However, in the online setting, we receive samples one-at-a-time rather than in batches. We therefore propose a streaming batch norm that uses moving average statistics rather than batch statistics as described in detail in Appendix F. To test the effectiveness of SKS, experiments are performed on a representative CNN with four 3 × 3 convolution layers and two fully-connected layers. We generate "offline" and "online" datasets based on MNIST (see Appendix G), including one in which the statistical distribution shifts every 10k images. We then optimize an online SGD and rank-4 SKS model for fair comparison (see Appendix H). To see the importance of different training techniques, we run several ablations in Appendix I. Finally, we compare these different training schemes in different environments, meant to model real life. In these hypothetical scenarios, a model is first trained on the offline training set, and is then deployed to a number of devices at the edge that make supervised predictions (they make a prediction, then are told what the correct prediction would have been). We present on four hypothetical scenarios. First, a control case where both external/environment and internal/NVM drift statistics are exactly the same as during offline training. Second, a case where the input image statistical distribution shifts every 10k samples, selecting from augmentations such as spatial transforms and gradients (see Section G). Third and fourth are cases where the NVM drifts from the programmed values, roughly modeling NVM memory degradation. In the third case, Gaussian noise is applied to the weights as if each weight was a single multi-level memory cell whose analog value drifted in a Brownian way. In the fourth case, random bit flips are applied as if each weight was represented by b memory cells (see Appendix G for details). For each hypothetical scenario, we plot five different training schemes: pure quantized inference (no training), bias-only training, standard SGD training, SKS training, and SKS training with max-normed gradients. In SGD training and for training biases, parameters are updated at every step in an online fashion. These are seen as different colored curves in Figure 2. Inference does best in the control case, but does poorly in adaptation experiments. SGD doesn't improve significantly on bias-only training, likely because SGD cannot accumulate gradients less than a weight LSB. SKS, on the other hand, shows significant improvement, especially after several thousand samples in the weight drift cases. Additionally, SKS shows about three orders of magnitude improvement compared to SGD in the worst case number of weight updates. Much of this reduction is due to the convolutions, where updates are applied at each pixel. However, reduction in fullyconnected writes is still important because of potential energy savings. SKS/max-norm performs best in terms of accuracy across all environments and has similar weight update cost to SKS/no-norm. To test the broader applicability of low rank training techniques, we run several experiments on ImageNet with ResNet-34 , a potentially realistic target for dense NVM inference on-chip. For ImageNet-size images, updating the low-rank approximation at each pixel quickly becomes infeasible, both because of the single-threaded nature of the algorithm, and because of the increased variance of the estimate at larger batch sizes. Instead, we focus on training the final layer weights (1000 × 512). ResNet-34 weights are initialized to those from and the convolution layers are used to generate feature vectors for 10k ImageNet training images 7, which are quantized and fed to a one-layer quantized 8 neural network. To speed up experiments, the layer weights are initialized to the pretrain weights, modulated by random noise that causes inference top-1 accuracy to fall to 52.7% ± 0.9%. In Table 1, we see that the unbiased SKS has the strongest recovery accuracies, although biased SVD also does quite well. The high-variance UORO and true SGD have weak or non-existent recoveries. -+0.3 ± 0.2 +0.3 ± 0.2 +0.3 ± 0.2 +0.9 ± 0.2 −3.9 ± 0.8 UORO 1 +0.4 ± 0.2 +0.3 ± 0.4 −1.8 ± 0.9 −7.6 ± 1.6 −31.7 ± 1.6 SVD 1 +1.9 ± 0.2 +5.8 ± 1.0 −3.4 ± 1.0 −19.4 ± 0.9 −40.7 ± 1.1 2 +1.4 ± 0.4 +6.5 ± 0.7 +6.3 ± 0.6 −5.2 ± 0.9 −36.3 ± 0.9 4 +1.3 ± 0.4 +6.5 ± 0.7 +5.2 ± 0.8 −3.3 ± 1.0 −33.8 ± 0.8 8 +1.4 ± 0.3 +5.6 ± 0.8 +4.3 ± 0.9 −2.4 ± 1.0 −32.8 ± 0.9 SKS 1 +0.3 ± 0.2 +0.3 ± 0.2 −0.7 ± 0.4 −2.7 ± 1.7 −26.5 ± 2.6 2 +0.3 ± 0.2 +0.4 ± 0.3 −0.1 ± 0.4 +1.3 ± 0.9 −12.9 ± 1.1 4 +0.4 ± 0.2 +0.6 ± 0.2 +1.9 ± 0.3 +8.0 ± 1.1 −5.1 ± 1.1 8 +0.4 ± 0.2 +1.1 ± 0.2 +3.3 ± 0.7 +4.8 ± 1.5 −15.8 ± 1.7 We demonstrated the potential for SKS to solve the major challenges facing online training on NVM-based edge devices: low write density and low auxiliary memory. SKS is a computationallyefficient, memory-light algorithm capable of decoupling batch size from auxiliary memory, allowing larger effective batch sizes, and consequently lower write densities. Additionally, we noted that SKS may allow for training under severe weight quantization constraints as rudimentary gradient accumulations are handled by the L, R matrices, which can have high bitwidths (as opposed to SGD, which may squash small gradients to 0). We found expressions for when SKS might have better convergence properties. Across a variety of online adaptation problems and a large-scale transfer learning demonstration, SKS was shown to match or exceed the performance of SGD while using a small fraction of the number of updates. Finally, we suspect that these techniques could be applied to a broader range of problems. Auxiliary memory minimization may be analogous to communication minimization in training strategies such as federated learning, where gradient compression is important. State: In this section we will attempt to bound the regret (defined below) of an SGD algorithm using noisy SKS estimatesg = g + ε in the convex setting, where g are the true gradients and ε are the errors introduced by the low rank SKS approximation. Here, g is a vector of size N and can be thought of as a flattened/concatenated version of the gradient tensors (e.g., N = n i · n o). Our proof follows the proof in. We define F as the convex feasible set (valid settings for our weight tensors) and assume that F is bounded with D = max w,v∈F ||w − v|| being the maximum distance between two elements of F. Further, assume a batch t of B samples out of T total batches corresponds to a loss landscape f t (w t) that is strongly convex in weight parameters w t, so there are positive constants C ≥ c > 0 such that cI ∇ 2 f t (w t) CI for all t (, Section 9. 3). We define regret as where w * = argmin w T t=1 f t (w) (i.e., it is an optimal offline minimizer of The gradients seen during SGD are g t = ∇f t (w t) and we assume they are bounded by G = max w∈F,t∈ [1,T] For the unbiased, minimum-variance case, Theorem A.4 from states that the minimum variance is s. Assuming errors between samples are uncorrelated, this leads to a total variance: For either case, ||ε|| 2 ≈ N σ 2 ε. For the t-th batch and i-th sample, we denote σ (t,i) q as the q-th singular value. For simplicity, we focus on the biased, zero-variance case (the unbiased case is similar). From, an approximately sufficient condition for sublinear-regret convergence is: B.3 DISCUSSION ON CONVERGENCE Equation suggests that as w t → w *, the constraints for achieving sublinear-regret convergence become more difficult to maintain. However, in practice this may be highly problem-dependent as the σ q will also tend to decrease near optimal solutions. To get a better sense of the behavior of the left-hand side of, suppose that: (no×ni) are the matrix weight W t gradients at batch t and || · || F is a Frobenius norm. We therefore expect both the left (proportional to ||G t || 2 F) and the right (proportional to ||w t − w * || 2) of to decrease during training as w t → w *. This behavior is in fact what is seen in Figure 1(b). If achieving convergence is found to be difficult, provides some insight for convergence improvement methods. One solution is to reduce batch size B to satisfy the inequality as necessary. This minimizes the weight updates during more repetitive parts of training while allowing dense weight updates (possibly approaching standard SGD with small batch sizes) during more challenging parts of training. Another solution is to reduce σ q. One way to do this is to increase the rank r so that the spectral energy of the updates are spread across more singular components. There may be alternate approaches based on conditioning the inputs to shape the distribution of singular values in a beneficial way. A third method is to focus on c, the lower bound on curvature of the convex loss functions. Perhaps a technique such as weight regularization can increase c by adding constant curvature in all Eigendirections of the loss function Hessian (although this may also increase the LHS of). Alternatively, perhaps low-curvature Eigen-directions are less important for loss minimization, allowing us to raise the c that we effectively care about. This latter approach requires no particular action on our part, except the recognition that fast convergence may only be guaranteed for high-curvature directions. This is exemplified in Figure 1(b), where we can see SVD track the curve for C more so than c. Finally, we note that this analysis focuses solely on the errors introduced by a floating-point version of SKS. Quantization noise can add additional error into the ε t term. We expect this to add a constant offset to the LHS of. For a weight LSB ∆, quantization noise has variance ∆ 2 /12, so we desire: C KRONECKER SUMS IN NEURAL NETWORK LAYERS A dense or fully-connected layer transforms an input a ∈ R ni×1 to an intermediate z = W · a + b to an output y = σ(z) ∈ R no×1 where σ is a non-linear activation function. Gradients of the loss function with respect to the weight parameters can be found as: which is exactly the per-sample Kronecker sum update we saw in linear regression. Thus, at every training sample, we can add (dz (i) ⊗ a (i) ) to our low rank estimate with SKS. A convolutional layer transforms an input feature map A ∈ R hin×win×cin to an intermediate feature map Z = W kern * A + b ∈ R hout×wout×cout through a 2D convolution * with weight kernel W kern ∈ R cout×k h ×kw×cin. Then it computes an output feature map y = σ(z) where σ is a non-linear activation function. Convolutions can be interpreted as matrix multiplications through the im2col operation which converts the input feature map A into a matrix A col ∈ R (houtwout)×(k h kwcin) where the i th row is a flattened version of the sub-tensor of a which is dotted with W kern to produce the i th pixel of the output feature map . We can multiply A col by a flattened version of the kernel, W ∈ R cout×(k h hwcin) to perform the W kern * A convolution operation with a matrix multiplication. Under the matrix multiplication interpretation, weight gradients can be represented as: which is the same as h out w out Kronecker sum updates. Thus, at every output pixel j of every training sample i, we can add (dZ col,j) to our low rank estimate with SKS. Note that while we already save an impressive factor of B/q in memory when computing gradients for the dense layer, we save a much larger factor of Bh out w out /q in memory when computing gradients for the convolution layers, making the low rank training technique even more crucial here. However, some care must be taken when considering activation memory for convolutions. For compute-constrained edge devices, image dimensions may be small and in minimal intermediate feature map memory requirements. However, if image dimensions grow substantially, activation memory could dominate compared to weight storage. Clever dataflow strategies may provide a way to reduce intermediate activation storage even when performing backpropagation 9. In a real device, operations are expected to be performed in fixed point arithmetic. Therefore, all of our training experiments are conducted with quantization in the loop. Our model for quantization is shown in Figure 3. The green arrows describe the forward computation. Ignoring quantization for a moment, we would have a = ReLU α W * a −1 + b, where * can represent either a convolution or a matrix multiply depending on the layer type and α is the closest power-of-2 to He initialization . For quantization, we rely on four basic quantizers: Qw, Qb, Qa, Qg, which describe weight quantization, bias and intermediate accumulator quantization, activation quantization, and gradient quantization, respectively. All quantizers use fixed clipping ranges as depicted and quantize uniformly within those ranges to the specified bitwidths. In the backward pass, follow the orange arrows from δ. Backpropagation follows standard backpropagation rules including using the straight-through estimator for quantizer gradients. However, because we want to perform training on edge devices, these gradients must themselves be quantized. The first place this happens is after passing backward through the ReLU derivitive. The other two places are before feeding back into the network parameters W, b, so that W, b cannot be used to accumulate values smaller than their LSB. Finally, instead of deriving ∆W from a backward pass through the * operator, the SKS method is used. SKS collects a −1, dz for many samples before computing the approximate ∆W. It accumulates information in two low rank matrices L, R which are themselves quantized to 16 bits with clipping ranges determined dynamically by the max absolute value of elements in each matrix. While SKS accumulates for B samples, leading to a factor of B reduction in the rate of updates to W, b is updated at every sample. This is feasible in hardware because b is small enough to be stored in more expensive forms of memory that have superior endurance and write power performance. Because of the coarse weight LSB size, weight gradients may be consistently quantized to 0, preventing them from accumulating. To combat this, we only apply an update if a minimum update density ρ min = 0.01 would be achieved, otherwise we continue accumulating samples in L and R, which have much higher bitwidths. When an update does finally happen, the "effective batch size" will be a multiple of B and we increase the learning rate correspondingly. In the literature, a linear scaling rule is suggested (see), however we empirically find square-root scaling works better (see Appendix H). E GRADIENT MAX-NORMING Figure 4: Maximum magnitude of weight gradients versus training step for standard SGD on a CNN trained on MNIST. Figure 4 plots the magnitude of gradients seen in a weight tensor over training steps. One apparent property of these gradients is that they have a large dynamic range, making them difficult to quantize. Even when looking at just the spikes, they assume a wide range of magnitudes. One potential method of dealing with this dynamic range is to scale tensors so that their max absolute element is 1 (similar to a per-tensor AdaMax or Range Batch-Norm applied to gradients). Optimizers such as Adam, which normalize by gradient variance, provide a justification for why this sort of scaling might work well, although they work at a per-element rather than per-tensor level. We choose max-norming rather than variance-based norming because the former is easier computational and potentially more ammenable to quantization. However, a problem with the approach of normalizing tensors independently at each sample is that noise might be magnified during regions of quiet as seen in the Figure. What we therefore propose is normalization by the maximum of both the current max element and a moving average of the max element. Explicitly, max-norm takes two parameters -a decay factor β = 0.999 and a gradient floor ε = 10 −4 and keeps two state variables -the number of evaluations k:= 0 and the current maximum moving average x mv:= ε. Then for a given input x, max-norm modifies its internal state and returns x norm: Standard batch normalization normalizes a tensor X along some axes, then applies a trainable affine transformation. For each slice X of X that is normalized independently: where µ b, σ b are mean and standard deviation statistics of a minibatch and γ, β are trainable affine transformation parameters. In our case, we do not have the memory to hold a batch of samples at a time and must compute µ b, σ b in an online fashion. To see how this works, suppose we knew the statistics of each sample µ i, σ i for i = 1... B in a batch of B samples. For simplicity, assume the i th sample is a vector X i,: ∈ R n containing elements X i,j. Then: In other words, the batch variance is not equal to the average of the sample variances. However, if we keep track of the sum-of-square values of samples σ for each sample i. After B samples, we divide both state variables by B and apply to get the desired batch statistics. Unfortunately, in an online setting, all samples prior to the last one in a given batch will only see statistics generated from a portion of the batch, ing in noisier estimates of µ b, σ b. In streaming batch norm, we alter the above formula slightly. Notice that in online training, only the most recently viewed sample is used for training, so there is no reason to weight different samples of a given batch equally. Therefore we can use an exponential moving average instead of a true average to track µ s, sq s. Specifically, let: If we set η = 1 − 1/B, a weighting of 1/B is seen on the current sample, just as in standard averages with a batch of size B, but now all samples receive similarly clean batch statistic estimates, not just the last few samples in a batch. For our experiments, we construct a dataset comprising an offline training, validation, and test set, as well as an online training set. Specifically, we start with the standard MNIST dataset of and split the 60k training images into partitions of size 9k, 1k, and 50k. Elastic transforms are used to augment each of these partitions to 50k offline training samples, 10k offline validation samples, and 100k online training samples, respectively. Elastic transforms are also applied to the 10k MNIST test images to generate the offline test samples. The source images for the 100k online training samples are randomly drawn with replacement, so there is a certain amount of data leakage in that an online algorithm may be graded on an image that has been generated from the same image a previous sample it has trained on has been generated from. This is intentional and is meant to mimic a real-life scenario where a deployed device is likely to see a restrictive and repetitive set of training samples. Our experiments include comparisons to standard SGD to show that SKS's improvement is not merely due to overfitting the source images. From the online training set, we also generate a "distribution shift" dataset by applying unique additional augmentations to every contiguous 10k samples of the 100k online training samples. Four types of augmentations are explored. Class distribution clustering biases training samples belonging to similar classes to have similar indices. For example, the first thousand images may be primarily "0"s and "3"s, whereas the next thousand might have many "5"s. Spatial transforms rotate, scale, and shift images by random amounts. Background gradients both scale the contrast of the images and apply black-white gradients across the image. Finally, white noise is random Gaussian noise added to each pixel. In addition to distribution shift for testing adaptation, we also look at internal statistical shift of weights in two ways -analog and digital. For analog weight drift, we apply independent additive Gaussian noise to each weight every d = 10 steps with σ = σ 0 / 1M/d where σ 0 = 10 and re-clip the weights between -1 and 1. This can be interpreted as each cell having a Gaussian cumulative error with σ = σ 0 after 1M steps. For digital weight drift, we apply independent binary random flips to the weight matrix bits every d steps with probability p = p 0 /(1M/d) where p 0 = 10. This can be interpreted as each cell flipping an average of p 0 times over 1M steps. Note that in real life, σ 0, p 0 depend on a host of issues such as the environmental conditions of the device (temperature, humidity, etc), as well as the rate of seeing training samples. In order to compare standard SGD with the SKS approach, we sweep the learning rates of both to optimize accuracy. In Figure 6, we compare accuracies across a range of learning rates for four different cases: SGD or SKS with or without max-norming gradients. Optimal accuracies are found when learning rate is around 0.01 for all cases. For most experiments, 8b weights, activations, and gradients, and 16b biases are used. Experiments similar to those in Section I are used to select some of the hyperparameters related to the SKS method in particular. In most experiments, rank-4 SKS with batch sizes of 10 (for convolution layers) or 100 (for fully-connected layers) are used. Additional details can be found in the supplemental code. Accuracy (Last 500 of 10k) Figure 6: The left two heat maps are used to select the base / standard SGD learning rate. The right two heat maps are used to select the SKS learning rate using the optimal SGD learning rate for bias training from the previous sweeps. For the SKS sweeps, the learning rate is scaled proportional to the square-root of the batch size B. This in an approximately constant optimal learning rate across batch size, especially for the max-norm case. Accuracy is reported averaged over the last 500 samples from a 10k portion of the online training set, trained from scratch. In Figure 7, rank and weight bitwidth is swept for SKS with gradient max-norming. As expected, training accuracy improves with both higher SKS rank and bitwidth. In dense NVM applications, higher bitwidths may be achievable, allowing for corresponding reductions in the SKS rank and therefore, reductions in the auxiliary memory requirements. In Table 2, biased (zero-variance) and unbiased (low-variance) versions of SKS are compared. Accuracy improvements are generally seen moving from biased to unbiased SKS although the pattern differs between the no-norm and max-norm cases. In the no-norm case, a significant improvement is seen favoring unbiased SKS for fully-connected layers. In the max-norm case, the choice of biased or unbiased SKS has only a minor impact on accuracy. It might be expected that as the number of accumulated samples for a given pseduobatch increases, lower variance would be increasingly important at the expense of bias. For our network, this implies convolutions, which receive updates at every pixel of an output feature map, would preferentially have biased SKS, while the fully-connected layer would preferentially be unbiased. This hypothesis is supported by the no-norm experiments, but not by the max-norm experiments. In Table 3, several ablations are performed on SKS with max-norm. Most notably, weight training is found to be extremely important for accuracy as bias-only training shows a ≈ 15 − 30% accuracy hit depending on whether max-norming is used. Streaming batch norm is also found to be quite helpful, especially in the no-norm case. Now, we explain the κ th ablation. In Section 4.1.1, we found the SVD of a small matrix C and its singular values σ 1,..., σ q. This allows us to easily find the condition number of C as κ(C) = σ 1 /σ q. We suspect high condition numbers provide relatively useless update information akin to noise, especially in the presence of L, R quantization. Therefore, we prefer not to update L, R on samples whose condition number exceeds threshold κ th. We can avoid performing an actual SVD (saving computation) by noting that C is often nearly diagonal, leading to the approximation κ(C) ≈ C 1,1 /C q,q. Empirically, this rough heuristic works well to reduce computation load while having minor impact on accuracy. In Table 3, κ th = 10 8 does not appear to ubiquitously improve on the default κ th = 100, despite being ≈ 2× slower to compute. Table 3: Miscellaneous selected ablations. Accuracy is calculated from the last 500 samples of 10k samples trained from scratch. Mean and unbiased standard deviation are calculated from five runs of different random seeds. Accuracy (no-norm) Accuracy (max-norm) baseline (no modifications) 80.2% ± 1.0% 83.0% ± 1.1% bias-only training 51.8% ± 3.2% 68.6% ± 1.4% no streaming batch norm 68.2% ± 1.9% 81.8% ± 1.3% no bias training 81.3% ± 1.0% 83.0% ± 1.4% κ th = 10 8 instead of 100 79.8% ± 1.4% 84.2% ± 1.4%

We use Kronecker sum approximations for low-rank training to address challenges in training neural networks on edge devices that utilize emerging memory technologies.

Represent the TLDR for finding the corresponding scientific article

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.


Question: Represent the scientific article to find a short TLDR

Question: Knowledge extraction techniques are used to convert neural networks into symbolic descriptions with the objective of producing more comprehensible learning models. The central challenge is to find an explanation which is more comprehensible than the original model while still representing that model faithfully. The distributed nature of deep networks has led many to believe that the hidden features of a neural network cannot be explained by logical descriptions simple enough to be understood by humans, and that decompositional knowledge extraction should be abandoned in favour of other methods. In this paper we examine this question systematically by proposing a knowledge extraction method using \textit{M-of-N} rules which allows us to map the complexity/accuracy landscape of rules describing hidden features in a Convolutional Neural Network (CNN). Experiments reported in this paper show that the shape of this landscape reveals an optimal trade off between comprehensibility and accuracy, showing that each latent variable has an optimal \textit{M-of-N} rule to describe its behaviour. We find that the rules with optimal tradeoff in the first and final layer have a high degree of explainability whereas the rules with the optimal tradeoff in the second and third layer are less explainable. The shed light on the feasibility of rule extraction from deep networks, and point to the value of decompositional knowledge extraction as a method of explainability. Recently there has been an increase in interest in explainable Artificial Intelligence (AI). Although in the past decade there have been major advances in the performance of neural network models, these models tend not to be explainable . In large part, this is due to the use of very large networks, specifically deep networks, which rely on distributed representations to model data accurately BID11. In contrast with symbolic AI, in which specific features are often hand picked for a problem, or symbolic Machine Learning (ML), which takes a localist approach BID15, the features used by a distributed representation do not necessarily correlate with obviously identifiable features of the data. A distributed representation may owe its strength to weak statistical correlations that a human would not be able to detect or describe in any comprehensible way. Knowledge extraction seeks to increase the explainability of neural networks by attempting to uncover the knowledge that a neural network has learned implicitly in its weights. One way of doing this is to translate trained neural networks into a set of symbolic rules or decision trees similar to the ones found in symbolic AI, ML and logic programming BID16 BID7. Rule extraction techniques have been around for decades BID20 ) with a number of rule extraction algorithms having been developed over the years BID12 BID4 BID22 ) (d'Avila BID5 . These techniques generally take one of two approaches: decompositional, in which the parameters of the network are used to generate rules, or pedagogical, in which the behaviour of the network is used to generate rules BID1 . In either case, the major issue with rule extraction is the complexity of the extracted rules. Even if it is possible to find a symbolic system which describes exactly a neural network (for example, feedforward, Boolean, deterministic networks can always be written as a logic program), a very large rule set derived from a very large CNN may be no more comprehensible than the original network. Perhaps the main reason knowledge extraction proves difficult (and in particular decompositional methods of extraction) is the distributed representations found in neural networks BID11. This means that important concepts which can be used for reasoning are not always represented by single neurons but by patterns of activity over many neurons. It has been argued that the distributed nature of neural networks plays an important part in many of their capabilities BID19. Distributed representations have been identified as one of the fundamental properties of connectionism BID18. This has led many to conclude that attempting to explain latent features using symbolic knowledge extraction is a dead end, and that methods akin to distillation should be adopted instead BID7. Distillation has also been proposed as a method for improving robustness but it's efficacy has been questioned BID13 BID3. Other approaches take a more practical view. Rather than attempting to open the black box, one may settle for some guarantees on the network's behaviour, or for visualizations seeking to explain individual classifications rather than the learned model BID9 BID17 BID10 In this paper, we develop a method for empirically examining the explainability of the latent variables in neural networks. We use rule extraction by searching through a space of M-of-N rules BID20 ) describing a latent variable, and measuring the error and complexity of each rule. By selecting various error/complexity trade-offs, we are able to map out a rule extraction landscape which shows the relationship between how complex the extracted rules are allowed to be and how accurately they capture the behaviour of a network. When applied to a standard 4-layer CNN trained on fashion MNIST, we find that some layers have very accurate rules whereas this is not the case for others even when using very complex rules. The discovery of a'critical point' on the rule extraction landscape shows that there is an ideal M-of-N rule to describe each latent variable. The accuracy of those rules depends highly on the variable that we are attempting to describe, with the overall explainability trends differing greatly between layers and architectures. All layers showed similarly shaped curves but in the convolutional layers the rules extracted with no penalty in complexity were much more complex relatively than the ones extracted from the fully connected layers with relative complexities over 0.4 in the convolutional layers and complexities of under 0.2 in the fully connected layers. Additionally, it was possible to find rules with near 0% error in the first and final layer whereas rules from the second and third layer could not do much better than 15% error. In Section 2 we give a brief overview of previous algorithms used for knowledge extraction. In Section 3 we give definitions of accuracy and complexity for M-of-N rules and outline the extraction process. In Section 4 we give the experimental of our rule extraction process for the mapping of the accuracy/complexity landscape before concluding in Section 5. One of the first attempts at knowledge extraction used a decompositional approach applied to feedforward networks, in particular the Knowledge-based Artificial Neural Networks (KBANN) BID21. This algorithm used the weights of a hidden variable to extract symbolic rules of the form IF M out of a set of N neurons (or concepts) are activated (or hold) THEN a given neuron (concept) is activated (holds), called M-of-N rules BID20. This was followed by more sophisticated algorithms which generate binary trees in which each node is an M-of-N rule BID12 BID4 (Notice that these binary trees can be reduced to IF-THEN propositional logic sentences as before). These more recent algorithms are pedagogical in that they select an M-of-N rule using the input units as the concepts (called literals in logic), based on the maximum information gain with respect to the output. By default these methods do not attempt to explain any of the latent variables of a network and simply treat the model as a black box which can be queried as an oracle to generate data for rule extraction. Many other extraction methods can be described as eclectic containing both aspects of pedagogical and decompositional methods (c.f. (d'Avila BID5) for examples. Other methods abandon the knowledge extraction paradigm and opt for alternative techniques, often more visually oriented BID9 BID7. A survey of various methods developed to solve the'black-box' problem of neural networks can be found in BID8.Most decompositional rule extraction techniques have been applied only to shallow networks with other techniques focusing on input/output relationships rather than attempting to explain the latent features of a deep network. The multiple hidden layers in a deep network mean that in order to explain an arbitrary hidden feature in terms of the input, a decompositional technique has to produce a hierarchy of rules (see BID22 for an example of hierarchical rule extraction). With many hidden layers, the extracted rules can quickly grow far too complex for a human to understand, unless each constituent of the rule hierarchy is exceedingly simple. Thus, the use of decompositional techniques to explain the features of a deep network end-to-end seems impractical, as argued in BID7. Nevertheless, experiments reported in this paper show that some layers of a deep network may be associated with highly explainable rules, and that within a layer some extracted rules may explain the network's behaviour remarkably well in terms of certain features. This opens the possibility of rule extraction being used as a tool for the modular explanation of network models, and it could provide insight into the similarity and disentanglement BID2 of latent features by comparing their optimal extracted rules. In what follows, we define the above ideas around rule explainability and optimality formally. In logic programming, a logical rule is an implication of the form A ← B, called A if B. The literal A is called the head of the rule and B stands for a conjunction of literals, B 1 ∧ B 2 ∧... ∧ B n called the body of the rule. Disjunctions in the body can be modelled simply as multiple rules having the same head. Most logic programs adopt a negation by failure paradigm so that A is true if and only if B is true BID6. When using rules to explain a neural network the literals will refer to the states of neurons. For example, if a neuron x takes binary values {0,1} then we define the literal X by X = T rue if x = 1, and X = F alse if x = 0. For neurons with continuous activation values, we can define a literal by including a threshold a such that X = T rue if x > a, and X = F alse otherwise. In other words, the literal X is shorthand for the statement x > a. In neural networks, a latent variable is usually poorly described by a single conjunctive rule since there are many different input configurations which will activate a neuron. Rather than simply adding a rule for each input pattern that activates a neuron (which essentially turns the network into a large lookup table), we look for M-of-N rules which have been commonly used in rule extraction starting with BID20. M-of-N rules soften the conjunctive constraint on the body of logical rules by requiring only M of the variables in the body to be true for some specific value of M < N (notice that when M = N we are left with a conjunction). For example, the rule DISPLAYFORM0 where ¬ stands for negation by failure. M-of-N rules are an attractive candidate for rule extraction for several reason. First, they offer a compact representation which is more general than a conjunction and which reflects naturally the input/output dependencies of the neurons in a neural network fan-in. Second, M-of-N rules are only a subset of all propositional formulas (it is easy to see that XOR cannot be represented as an M-of-N rule), meaning that our rule extraction process will not simply generate a lookup table to explain a neuron. Finally, M-of-N rules share a structural similarity with neural networks. This can be seen by viewing M-of-N rules as'weightless perceptrons'. Any M-of-N rule can be represented by a perceptron with its output neuron representing the head, and visible neurons representing the body of the rule. In order to encode an M-of-N rule in a neural network, one just needs to set the bias of the output neuron to M and the weights of each input neuron to 1 or −1 for neurons corresponding to positive or negative literals, respectively. M-of-N rules have been used in the early days of knowledge extraction but have since been forgotten. This paper brings M-of-N rules to the forefront of the debate on explainability again. When our network has continuous activation values, in order to define the literals to use for rule extraction we must choose a splitting value a for each neuron which will lead to a literal of the form x > a. In order to choose splitting values for continuous neurons we use information gain BID14. Given a target neuron we wish to explain, h, we generate a literal for the target neuron by selecting a split based on the information gain with respect to the output labels of the network. That is, given a set of test examples, choose the value of the target neuron h which splits the examples in such a way as to in the maximum decrease in entropy of the network outputs on the test examples. The input literals are then generated from the inputs to the target neuron by choosing splits for each input which maximize the information gain with respect to the target literal generated in the previous step. In practice this means that each target literal in a layer will have its own set of input literals, each corresponding to the same set of input neurons but with different splits. In the case that the layer is convolution, each feature map corresponds to a group of neurons each with a different input patch. Rather than test every single neuron in the feature map we only test the one whose optimal split has the maximum information gain with respect to the network output. This gives us a single rule for each feature map rather than a collection of them. The two metrics we are concerned with in rule extraction are comprehensibility and accuracy. For a given rule we can define accuracy in terms of a soundness measure. This is simply the expected difference between the predictions made by the rules and the network. More concretely given a neuron h in a neural network with input neurons x i, we can use the network to compute the state of h from the state of the input neurons which then determines the truth of literal H. Thus we can use the network to determine the truth of H, call this N (x). Furthermore, if we have some rule R relating variables H and X i, we can use the state of the input x to determine the value of the variables X i, and then use R to determine the value of H, call this R(x). Given a set of input configurations to test I (not necessarily from the test set of the network) we can measure the discrepancy between the output of the rules and the network as DISPLAYFORM0 In other words we measure the average error of the rules when trying to predict the output of the network over a test set. Comprehensibility is more difficult to define as there is a degree of subjectivity. The approach we take is to look at the complexity of a rule. Here, we think of complexity in an analogous way to the Kolmogorov complexity which is determined by a minimal description. Thus we determine the complexity of a rule by the length of its body when expressed by a (minimal) rule in disjunctive normal form (DNF). For an M-of-N rule, the complexity is simply M N M, where denotes the binomial coefficient. For our experiments we measure complexity in a relative manner by normalizing w.r.t. a maximum complexity. Given N possible input variables the maximum complexity is DISPLAYFORM1, where denotes the ceiling function (rounding to the next highest integer). Finally in order to control for growth we take the logarithm giving the following normalized complexity measure. DISPLAYFORM2 As an example, suppose we have a simple perceptron whose output unit has a bias of 1 and two binary visible units with weights w 1,1 = 1 and w 2,1 = −0.5. Then consider the rule h = 1 ⇐⇒ 1-of{x 1 = 1, ¬(x 2 = 1)}. Over the entire input space we see that R(x) = N (x) only when x 1 = 0 and x 2 = 1 giving us an error of 0.25. Furthermore, a 1 − of − 2 rule is the most complex rule possible for 2 variables as it has the longest DNF of any M-of-N rule giving us a complexity of 1. Using Eqs. and we define a loss function for a rule R as a weighted sum in which a parameter β ∈ R + determines the trade-off between soundness and complexity. DISPLAYFORM3 By using a brute force search procedure with various values of β we are able to explicitly determine the relationship between the allowed complexity of a rule and its maximum accuracy. For β = 0 the rule with the minimum loss will simply be the rule with minimum error regardless of complexity, and for β large enough the rule with the minimum loss will be a rule with 0 complexity, either a 1 − of − 1 rule or one of the trivial rules which either always predicts true or always predicts false (these can be represented as M-of-N rules by 0−of −N and N +1−of −N respectively). Given a neuron h j with n input neurons x i, we generate splits for each neuron using the technique just described to give us a set of literals H j and X i. Then, we negate the literals corresponding to neurons which have a negative weight to h j. Using these we search through O(n 2) M-of-N rules with variables X i in the body and H j in the head, which minimize L(R). To do this we reorder the variables according to the magnitude of the weight connecting x i to h j (such that we have |w 1,j | ≥ |w 2,j | ≥ ... ≥ |w n,j |). Then we consider the rule M − of − {X 1, ..., X N} for each 1 ≤ N ≤ n and each 0 ≤ M ≤ N + 1. The search procedure only relies on the ordering of the variables X i. Generate a split, s, for h by choosing the value which maximizes the information gain with respect to the network output. Use this to define the literal H for Each neuron x which is an input of h, do Generate a split for x by choosing the value which maximizes the information gain with respect to H. Use this value to define the literal X if the connection between x and h is positive, and use it to define ¬X otherwise end for A neuron with n input neurons has O(2 n) possible M-of-N rules which makes an exhaustive search intractable. However, here we rely on the assumption that the most accurate M-of-N rules with N literals use the literals corresponding to the N neurons with the strongest weights (i.e. the highest weight values). This assumption is easily justified by the conditional independence of hidden units in all layers except for the final one. Here, since we use a softmax function, the conditional independence of the output neurons is no longer valid. One way around this would be to order the literals by their information gains rather than their weights. However, the high accuracy found in the experimental of the rules extracted from softmax layer when ordering the literals by weight seem to suggest that this is not necessary. By defining an order on the literals we reduce an exponential search space to a polynomial one. However, this is still computationally difficult when a large number of test examples and input neurons exist. In order to complete the rule extraction in a reasonable time, the algorithm was implemented in Spark and run on IBM cloud services. The examples used to measure the accuracy of the extracted rules were taken from the training set rather than the test set as we evaluate the accuracy of the rules with respect to the output of the network rather than the actual output and using examples that the network was trained on was deemed to be a better representation of the behaviour the network learned. By running the search in parallel, we can map the accuracy/complexity graph for about 50 hidden neurons in the second and third layer in several hours. Increasing the number of examples used in the accuracy calculation greatly increases the time taken and for this reason we only use 1000 examples. To demonstrate the procedure we will examine the extraction process for the first hidden feature in the CNN trained on the fashion MNIST data set. First we select 1000 random input examples from the training set and use them to compute the activations of each neuron in the CNN as well as the predicted labels of the network. Since the CNN pads the input with zeros and the input images are 28 × 28 we have 28 × 28 = 784 neurons per feature in the first layer. Each of these neurons correspond to a different 5 × 5 patch of the input. To select a neuron to test we find the optimal splitting value of each neuron by computing the information gain of each neuron with respect to the networks predicted labels. For the first feature we find that the neuron with the maximum information gain is neuron 96 which has an information gain of 0.015 when split on the value 0.0004. This neuron corresponds to the image patch centered at (96/28, 96%28) =. With this split we define the variable H by H:= 1 iff h 96 ≥ 0.0004.Using this variable we define the input splits by choosing the values which in the maximum information gain with respect to H. Note our test input consists of the 1000 5 × 5 image patches centered at taken from the input examples. We then search through the M-of-N rules whose bodies consist of the input variables defined by the splits to determine an optimal M-of-N rule explaining H for various error/complexity tradeoffs. As we increase the complexity penalty we extract three different rules which are visualized in Figure 1. We can see from the figure that many of Figure 1: An example of the various rules extracted for the first feature. The first image represents the weights of the neruon and the following three are rules of decreasing complexity explaining the neuron. Here grey indicates that the input feature is not included in the M-of-N rule, white indicates a positive literal, and black indicates a negative literal the weights are filtered out by the rules. The most complex rule is a 5-of-13 rule which has a 0.025 error, or in other words the output of the rule agrees with the network 97.5% of the time. Adding a mild penalty to complexity changes the optimal rule to the much simpler 3-of-4 rule but raises the error to 0.043. Finally a heavy penalty to complexity produces the trivial 1-of-1 rule which has the significantly higher error of 0.13 In order to demonstrate the on a simple network known to be amenable to rule extraction we apply the technique to the classic example of the DNA promoter dataset. Training a feed forward network with a single hidden layer of 100 nodes We find that, like we will see with the Fashion MNIST dataset, the relationship between complexity and error in the first layer is exponential, suggesting an ideal complexity/error tradeoff (FIG1 . Furthermore, in the output layer we find that the rule 1 − of − {H 39, H 80} gives 100% fidelity to the network. Since the splits for the hidden layer are defined only by the information gain with respect to the output we can describe each of the literals in our 1-of-2 rule with an M-of-N rule extracted from the input layer. Extracting with no complexity penalty, the rules in question are of the form 64-of-119 for the variable H 3 9 which produces a single incorrect classification of H 3 9, and a rule of the form 32-of-61 for the variable H 8 0 which produces no incorrect classifications for H 8 0. Denoting L 119 as the set of literals in the rule explaining H 3 9 and L 61 as the set of literals in the rule explaining H 8 0 the output of the network can be predicted by the rule DISPLAYFORM0 In general the errors through a layer propagate when stacking rules that don't perfectly approximate each layer. This is exacerbated by the fact that in most networks, when we choose splits for the input neurons of a layer and then move down a layer we will end up choosing different splits for the same layer when they are treated as output. In order to replace a network with an end to set of heirarchical rules we must decide on a single set of splits for each layer. This can be done by simpy moving down one layer at a time and selecting the input splits based on the information gain against all configurations of the new output layer but it introduces more error. With this in mind we conduct our experiments layer by layer independently in order to provide an idealized complexity/error curve that can provide a baseline for the best achievable rules when doing rule extraction with M-of-N rules. By doing this we can examine under what circumstances M-of-N rule extraction might be useful as well as provide a method for evaluating other extraction algorithms. In order to examine the rule extraction landscape of a neural network trained on a practical example, we tested the layerwise rule extraction search on a basic CNN trained on fashion MNIST in tensorflow. The CNN had a standard architecture with a 32 feature convolutional layer with a 5 × 5 convolutional window followed by a 2×2 max pooling layer, then a 64 feature convolution with a 5×5 convolutional window and another 2 × 2 max pooling layer followed by a final hidden fully connected layer of 1024 units. All units used a rectified linear activation function; 1000 random inputs from the fashion MNIST training data were chosen to test extracted rules against the network. For the convolutional layers each feature was tested on the image patches corresponding to the maximum information gain as described in Section 3.2. For the third layer 50 features were chosen at random to test. In the third layer we limited the search to rules with 1000 literals or less to save time as it was very rare for even the most complex rules extracted to have over 900 literals. For the final layer the output was tested as 10 one-hot neurons, each of which underwent the rule searching procedure. For each layer, we repeated the search procedure for 5 different values of β, 0, 0.1, 0.2, 1, 5, 1000. This produced 5 different sets of extracted rules each with a different error/complexity trade-off. For a given value of β, we averaged the complexities and errors of rules extracted for each target neuron. This allowed us to produce a graph mapping out the error/complexity landscape for rules extracted from each layer (see Figure 1.) Here we can see that the Complexity/Error trade-off for the extracted rules from each layer are different. For the first and final layers we were able to extract quite accurate rules approaching a near 0 error whereas the second and third layer had a similar accuracy/complexity tradeoff with second layer showing a very slight improvement in accuracy with much more complex rules whereas the minimum error of the third layer is not able to be improved at all with increasingly complex rules. Additionally the trivial rules perform much worse on the second than the third layer. Importantly we see that the optimal accuracy/complexity tradeoff is not simply a function of the number of input nodes since the third layer performs about as well as the second layer despite having 3136 vs 800 input nodes. Similarly the final layer provides much more accurate rules that are relatively less complex than the first layer despite having 1024 vs 25 input nodes. Despite the actual values varying for each layer, the paint a similar picture for the shape of the rule extraction landscape in each case. Rather than a gradual decline in accuracy there is a critical point at which the error starts to increase rapidly as the penalty on complexity increases. This suggests that there is a'natural' set of rules for explaining the latent features. Although ultimately the trade off is subjective, the set of rules existing at the critical points cannot be made significantly more accurate if they are more complex, and will start to do exponentially worse if they are made any simpler. Current rule extraction algorithms do not explicitly take complexity into account for their optimization. Although they may include steps to attempt to simplify the extracted rules, it is not clear a priori where on the Error/Complexity graph these rules will land. To the best of our knowledge, this paper is the first to make rule complexity an integral part of the extraction algorithm, as exemplified in the analysis of Figure 1.Empirical evaluation of popular extraction algorithms should be an important step in their validation. The limitations, as well as potential, of rule extraction algorithms are also outlined in these . In certain cases, such as we see in the second and third layer of the CNN, there are simply no simple explanations for the features in terms of their input with even the most complex rules having an error rate of 15%. However, in other cases, such as in the final layer in the CNN, the behaviour of the output neurons can be accurately captured by relatively simple rules with extracted rules from the final layer obtaining near 0% error even for rules with complexity of under 0.05. This lends weight to the idea that decompositional rule extraction as a general method of explainability is not possible, but opens up the possibility of selective use of decompositional algorithms depending on which layer we wish to explain in terms of the previous layer. The black box problem of neural networks presents an obstacle to their deployment into society. The black box problem has been an issue for neural networks since their creation, but as neural networks have become more integrated into society, the need for explainability has attracted considerably more attention. The success of knowledge extraction in this endeavor has overall been mixed with most large neural networks today remaining difficult to interpret and explain. Traditionally knowledge extraction has been a commonly used paradigm and it has been applied to various tasks. Critics, however, point out that the distributed nature of neural networks makes the specific method of decomposition rule extraction unfeasible as individual latent features and unlikely to represent anything of significance. We test this claim by applying a novel search method for M-of-N rules to explain the latent features of a CNN, and find that generally latent features can be described by an'optimal' rule representing an ideal error/complexity trade-off for the explanation. We do this by including rule complexity as an explicit measure in the search for extracted rules. The large discrepancy in this trade-off between neurons in different layers, neurons in different layers with different architectures, and even different neurons in the same layer, suggests that rule extraction as a general technique is unlikely to provide adequate descriptions for all, or even most latent variables. However, the fact that in many cases the explanations can be made much simpler without reducing the accuracy of the rules suggests that rule extraction can be a useful tool when examining networks with features that are likely to be easily understandable. These indicate that decompositional rule extraction may still be an important tool for understanding the behaviour of networks. Further research would examine the effects on the accuracy/interpretability landscape of using different transfer functions, other data sets, different architectures, and various forms of regularization of the learning.

Represent the TLDR for finding the corresponding scientific article

Systematically examines how well we can explain the hidden features of a deep network in terms of logical rules.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.


Question:
represent the article for finding a single-sentence tldr

E.g. ' Large deep neural networks require huge memory to run and their running speed is sometimes too slow for real applications. Therefore network size reduction with keeping accuracy is crucial for practical applications. We present a novel neural network operator, chopout, with which neural networks are trained, even in a single training process, so as to truncated sub-networks perform as well as possible. Chopout is easy to implement and integrate into most type of existing neural networks. Furthermore it enables to reduce size of networks and latent representations even after training just by truncating layers. We show its effectiveness through several experiments. where where grad is a gradient and m is the number drawn in the forward pass. DISPLAYFORM0 At test time, chopout is defined to behave as a identity function, that is, just pass through the input 45 vector without any modification. This definition of chopout in prediction mode is contrastive to that 46 of dropout, which, in prediction time, dropout scale inputs to make it consistent with training time. Training a fully-connected neural network with applying chopout can be interpreted as simultaneous 48 training of randomly sampled sub-networks which are obtained by cuttinng out former parts of the 49 original fully-connected neural network with sharing parameters. In higher dimensional cases, chopout can be easily extended as a random truncation of channels 51 instead of dimensions. For example, when applied to a tensor x ∈ R c×h×w, the forward-propagation 52 of chopout is defined as DISPLAYFORM0 where P ({0, 1, · · ·, c}) is an arbitrary distribution. Back-propagation is defined in the same way. Throughout experiments, we use uniform distributions over {1, · · ·, d} for P m ({0, 1, · · ·, d}). We train autoencoders on MNIST (LeCun et al., Table 1, FIG4 ). We see that by applying 58 chopout on the hidden layer of the autoencoder, the reconstruction is kept well even after the hidden 59 layer is truncated. We apply chopout for embeddings trained through skip-gram models (Mikolov et al. [2013a,b] ). We 62 use text8 corpus 1. We set the window size to 5 and ignore infrequent words which appear less than 63 20 times in the corpus. The TAB2 shows the consistency of embeddings. ran ran news news good The distribution P m ({0, 1, · · ·, d}) should be explored. If we put chopouts in every layer of a 79 neural network, then, in training, there could be a layer where drawn m ∼ P m ({0, 1, · · ·, d}) is very 80 small and it could be a bottleneck of the prediction accuracy. Chopout can be used for network pruning.' == 'We present a novel simple operator, chopout, with which neural networks are trained, even in a single training process, so as to truncated sub-networks perform as well as possible.' != 'We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.'

Question: Recent findings show that deep generative models can judge out-of-distribution samples as more likely than those drawn from the same distribution as the training data. In this work, we focus on variational autoencoders (VAEs) and address the problem of misaligned likelihood estimates on image data. We develop a novel likelihood function that is based not only on the parameters returned by the VAE but also on the features of the data learned in a self-supervised fashion. In this way, the model additionally captures the semantic information that is disregarded by the usual VAE likelihood function. We demonstrate the improvements in reliability of the estimates with experiments on the FashionMNIST and MNIST datasets. Deep Generative Models (DGMs) have gained in popularity due to their ability to model the density of the observed training data from which one can draw novel samples. However, as pointed out in their recent paper, the inferences made by likelihood-based models, such as Variational Autoencoders (VAEs) and flow-based models (; van den), are not always reliable. They can judge out-of-distribution (OOD) samples to be more likely than in-distribution (ID) samples that are drawn from the same distribution as the training data. Concretely, a DGM trained on the FashionMNIST dataset will on average assign higher likelihoods to images from the MNIST dataset than to test images from the FashionMNIST dataset (see for example top left image in Figure 1(a) ). In this work we tackle the problem of misaligned likelihood estimates produced by VAEs on image data and propose a novel likelihood estimation during test time. Our method leverages findings reported in our earlier work Bütepage et al., which are summarised in Section 2, and is based on the idea to evaluate a given test image not only locally, using individual parameters returned by a VAE as it is usually done, but also globally using learned feature representations of the data. The main contribution of this paper is the introduction of a feature-based likelihood trained in a self-supervised fashion. This likelihood evaluates the model also based on the semantics of a given image and not solely on the values of each pixel. We elaborate on this idea in Section 3 and demonstrate the improvements with an empirical evaluation presented in Section 4. We emphasise that the aim of our work is exclusively to improve the reliability of the likelihood estimation produced by VAEs. We focus on image data in particular as we have not observed the misalignment in our earlier experiments on various non-image datasets from UCI Machine Learning Repository . We plan to investigate this further in the future work. Due to the lack of space we omit the experiments on non-image data as well as the specifics of VAEs for which we refer the reader to;. This section provides a on the evaluation of VAEs and summarizes our earlier work presented in (Bütepage et al., 2019). In VAEs, the observed random variable X is assumed to be generated from the joint distribution p(X, Z) = p(X|Z)p(Z) where Z denotes the latent variables. Using variational inference the intractable true posterior distribution p * (Z|X) is approximated with a simpler parametrised distribution q(Z|X). VAEs employ amortized inference where encoder and decoder neural networks, φ z (X) and φ x (Z), are jointly trained to represent the approximate posterior distribution q(Z|φ z (X)) and likelihood function p(X|φ x (Z)), respectively. From a Bayesian perspective, we can evaluate a successfully trained VAE using two different evaluation schemes where p P R V AE denotes the prior predictive (PR) and p AP O V AE the approximate posterior predictive (APO) distribution. Bütepage et al. argue that the likelihood estimates produced by a trained VAE are influenced by both 1) the choice of the above listed evaluation scheme and 2) the choice of the parametrisation of the likelihood function p(X|φ x (Z)). Here, two common choices are a Gaussian distribution in the case of colored images or a Bernoulli distribution in the case of black and white (or grey-scaled) images. The effect of both 1) and 2) is best demonstrated in Figure 1 (a) where we visualise the log likelihood estimates from a VAE V 1, parametrised by a Bernoulli likelihood (top row), and a VAE V 2, parametrised by a Gaussian likelihood (bottom row), using both PR (left column) and APO (right column) evaluation schemes from Equations and. Both VAEs were trained on the FashionMNIST dataset and tested on test images from both the FashionMNIST and MNIST datasets. In the case of V 1 the pixel values of the images were binarised with threshold 0.5, and in the case of V 2 scaled to the interval. The choice of the evaluation scheme influences the variance of the estimates of the training data as it directly affects the variability of the parameters φ x (z) returned by the VAE (see left vs right column in Figure 1(a) ). Namely, PR produces more diverse parameters corresponding to the latent representations of the whole training data while APO generates more homogeneous samples corresponding to the latent representation of a given test point x. On the other hand, the choice of the likelihood parametrisation (top vs bottom row in Figure 1(a) ) influences the actual values of the estimates since images are evaluated under distributions of different shapes. We refer the interested reader to (Bütepage et al., 2019) for a detailed discussion. Note that only the top-left combination in Figure 1(a) reproduces the reported in . (b) Using the improved p F EV AE (x|φ x (z)) likelihood from Equation. This section describes the self-supervised feature-based likelihood function which is the main contribution of this work. In addition to the influencing factors discussed in Section 2, we hypothesise that the likelihood estimates are also affected by the assumption that image pixels are independent and identically distributed (iid) around the likelihood function parameterised by the decoder. Let a test image x be represented as a concatenated vector of length D and let x d denote its d-th component. Using the assumption of iid pixels, the likelihood function becomes a product of individual pixel-wise likelihoods: Therefore, when computing the probability of x, the likelihood only captures pixel-wise errors that are evaluated locally under the parameters φ x (z) returned by the VAE and does not take into account the "global" information contained in the image (such as the semantics of the dataset). To mitigate the lack of the global evaluation, we propose to weight the likelihood term during test time with an additional term that relates the semantic information of both the test point x and the parameters φ x (z) to the semantics of the whole training dataset. We define the details below. We separately train a self-supervised classifier Γ and use its l-th layer to extract a low dimensional feature representation f x = l(x) of an image x. We train Γ on the same training datasetX = {x 1, . . ., x N} as we train the VAE. We then fit a Bayesian Gaussian Mixture (BGM) model with C components to the set F = {f x 1, . . ., f xn} of feature representations extracted from a randomly sampled subset ofX of size n < N (see also Section 4 for details). Let f x be the feature representation of a test image x. During the evaluation of the BGM on f x each mixture component is assigned a weight that indicates its contribution to the generation of f x. Let C x denote the mixture component with the highest weight. Given likelihood parameters φ x (z) returned by the VAE, we define the global likelihood of x as the product where p F E (f x |C φx(z) ) is the likelihood of the test point in feature space under the mixture component C φx(z) determined by the representation f φx(z) of the parameters φ x (z) and ) is the likelihood of f φx(z) under the same component C φx(z). The first term can be seen as a global likelihood of the test point under the decoded parameters and the second term represents a global likelihood of the parameters themselves. We then propose to evaluate the test image x under the combined likelihood function where p V AE as before captures local pixel-wise errors and p F E additionally captures the global (semantic) likelihood. We evaluate our method with experiments on FashionMNIST and MNIST datasets and present the below. Feature extraction We obtained low dimensional features of the training data by deploying a self-supervised Jigsaw classifier Γ presented by. The classifier receives a Jigsaw puzzle, which is a shuffled 3×3 grid of tiles extracted from a given image, and outputs (the class of) the permutation that was applied to the original unshuffled grid (see Appendix A for the implementation details). Note that any self-supervised learning strategy could be deployed as long as the obtained low dimensional features are of high quality and represent the training data well. After the completed training we randomly sampled n = 10000 training images {x 1, . . ., x n} and obtained their low dimensional representations {f 1, . . ., f n} from the first layer l 1 of the classifier Γ, to which we fitted a BGM model with C = 15 components. The parameters n and C were determined using a hyperparameter grid search. We used representations from the first layer because we hypothesise that the earlier layers of the classifier carry useful information about the training data while the later layers carry information about the task itself. We leave experiments with representations obtained from different layers for the future work. Experiment We trained two VAEs, V 1 and V 2, and two Jigsaw classifiers, Γ 1 and Γ 2 with specifications described in Appendix A on the FashionMNIST dataset. Here, the subscripts 1 and 2 denote that the model in consideration was trained on images binarised with threshold 0.5 and on images with pixel values scaled to the interval, respectively. As in the experiment producing the in Figure 1 (a), V 1 additionally assumes a Bernoulli likelihood and V 2 a Gaussian likelihood. For a given (binarised) test image x and parameters φ x (z) obtained from the trained VAE V i, we first calculate the VAE likelihood p V AE in the usual way using the assumption of iid pixels. We then obtain their low dimensional features f x = l 1 i (x) and f φx(z) = l 1 i (φ x (z)) from the first layer l 1 i of the trained Jigsaw classifier Γ i and calculate p F E under the fitted BGM following Equation. The product of the two likelihoods then equals the newly proposed likelihood p F EV AE from Equation. Given this pipeline, VAE i + Γ i for i = 1, 2 and our likelihood p F EV AE, we compared the log likelihood estimates using the PR and APO evaluation schemes from Equations and on the images from the test splits of FashionMNIST and MNIST datasets. The are visualised in Figure 1(b). We see that our method significantly improves the estimates when using Gaussian likelihood parametrisation (bottom row) as it clearly separates the OOD samples from the ID samples. Note that the VAE parameters φ x (z) in the PR evaluation always reflect the distribution of the entire training data. This means that the global likelihood of a test point evaluates the test point under all classes that were presented during training time. In practice this means that the PR evaluation of the global likelihood averages over all classes which in a less distinct separation of the OOD samples. When using Bernoulli likelihood (top row) our method increases the variance of the likelihood of OOD samples but fails to achieve the same separation as in the Gaussian case. This is because a significant amount of the semantic information is lost during the binarisation process of the FashionMNIST dataset. The ing binarised images are often unrecognisable with a sparse pixel distribution which makes the task of solving Jigsaw puzzles more difficult. Since digits in MNIST images are also sparse they become likely under p F E. We observe their estimates fusing with FashionMNIST estimates if we corrupt the using salt and pepper noise (see Figure 2 in Appendix B). We therefore hypothesise that in this particular case OOD samples simply become too similar to the ID samples, suggesting that the Bernoulli likelihood is not the most appropriate modelling choice. The inadequacy of the Bernoulli distribution in VAEs has also recently been discussed by who instead suggest to use their fully characterized continuous Bernoulli distribution. We have discussed how the problematic assumption that the image pixels are iid around the decoded parameters narrows the focus of the VAE likelihood function p V AE to a local area of the data density. Thus, the model likelihood function disregards the global data density, including the semantic information. Our proposed likelihood function mitigates this problem by leveraging self-supervised feature learning. In the future, we aim to evaluate our method on more complex datasets, such as CIFAR-10 and SVHN, and to design an end-to-end training procedure of VAEs using our proposed likelihood.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Improved likelihood estimates in variational autoencoders using self-supervised feature learning

represent the short summary for finding the corresponding scientific article (long)

To understand the information stored in the latent space, we train a GAN-style decoder constrained to produce images that the VAE encoder will map to the same region of latent space.