Question: Represent the scientific article to find a short TLDR

Question: We provide a novel perspective on the forward pass through a block of layers in a deep network. In particular, we show that a forward pass through a standard dropout layer followed by a linear layer and a non-linear activation is equivalent to optimizing a convex objective with a single iteration of a $\tau$-nice Proximal Stochastic Gradient method. We further show that replacing standard Bernoulli dropout with additive dropout is equivalent to optimizing the same convex objective with a variance-reduced proximal method. By expressing both fully-connected and convolutional layers as special cases of a high-order tensor product, we unify the underlying convex optimization problem in the tensor setting and derive a formula for the Lipschitz constant $L$ used to determine the optimal step size of the above proximal methods. We conduct experiments with standard convolutional networks applied to the CIFAR-10 and CIFAR-100 datasets and show that replacing a block of layers with multiple iterations of the corresponding solver, with step size set via $L$, consistently improves classification accuracy. Deep learning has revolutionized computer vision and natural language processing and is increasingly applied throughout science and engineering BID20. This has motivated the mathematical analysis of various aspects of deep networks, such as the capacity and uniqueness of their representations BID28 BID24 and their global training convergence properties BID10. However, a complete characterization of deep networks remains elusive. For example, Bernoulli dropout layers are known to improve generalization BID29, but a thorough theoretical understanding of their behavior remains an open problem. While basic dropout layers have proven to be effective, there are many other types of dropout with various desirable properties BID22. This raises many questions. Can the fundamental block of layers that consists of a dropout layer followed by a linear transformation and a non-linear activation be further improved for better generalization? Can the choice of dropout layer be made independently from the linear transformation and non-linear activation? Are there systematic ways to propose new types of dropout?We attempt to address some of these questions by establishing a strong connection between the forward pass through a block of layers in a deep network and the solution of convex optimization problems of the following form: DISPLAYFORM0 Note that when f i (a i x) = 1 2 (a i x − y i) 2 and g(x) = x 2 2, Eq. is standard ridge regression. When g(x) = x 1, Eq. has the form of LASSO regression. We show that a block of layers that consists of dropout followed by a linear transformation (fullyconnected or convolutional) and a non-linear activation has close connections to applying stochastic solvers to. Interestingly, the choice of the stochastic optimization algorithm gives rise to commonly used dropout layers, such as Bernoulli and additive dropout, and to a family of other types of dropout layers that have not been explored before. As a special case, when the block in question does not include dropout, the stochastic algorithm reduces to a deterministic one. Our contributions can be summarized as follows. (i) We show that a forward pass through a block that consists of Bernoulli dropout followed by a linear transformation and a non-linear activation is equivalent to a single iteration of τ -nice Proximal Stochastic Gradient, Prox-SG BID34 when it is applied to an instance of. We provide various conditions on g that recover (either exactly or approximately) common non-linearities used in practice. (ii) We show that the same block with an additive dropout instead of Bernoulli dropout is equivalent to a single iteration of mS2GD BID16 ) -a mini-batching form of variance-reduced SGD BID12 ) -applied to an instance of. (iii) By expressing both fully-connected and convolutional layers (referred to as linear throughout) as special cases of a high-order tensor product BID2, we derive a formula for the Lipschitz constant L of ∇F (x). As a consequence, we can compute the optimal step size for the stochastic solvers that correspond to blocks of layers. We note that concurrent work BID26 used a different analysis strategy to derive an equivalent for computing the singular values of convolutional layers. (iv) We validate our theoretical analysis experimentally by replacing blocks of layers in standard image classification networks with corresponding solvers and show that this improves the accuracy of the models. Optimization algorithms can provide insight and guidance in the design of deep network architectures BID31 BID14 BID35 BID36. For example, BID35 have proposed a deep network architecture for compressed sensing. Their network, dubbed ADMM-Net, is inspired by ADMM updates BID4 on the compressed sensing objective. Similarly, BID36 demonstrated that unrolling a proximal gradient descent solver BID1 ) on the same problem can further improve performance. The work of BID14 demonstrated a relation between incremental proximal methods and ResNet blocks; based on this observation, they proposed a new architecture (variational networks) for the task of image reconstruction. BID0 proposed to embed optimization problems, in particular linearly-constrained quadratic programs, as structured layers in deep networks. BID21 replaced proximal operators in optimization algorithms by neural networks. BID11 proposed a new matrix layer, dubbed ReEig, that applies a thresholding operation to the eigenvalues of intermediate feature representations that are stacked in matrix form. ReEig can be tightly connected to a proximal operator of the set of positive semi-definite matrices. proposed a new architecture based on a sparse representation construct, Multi-Layer Convolutional Sparse Coding (ML-CSC), initially introduced by BID23. Sparsity on the intermediate representations was enforced by a multi-layer form of basis pursuit. This body of work has demonstrated the merits of connecting the design of deep networks with optimization algorithms in the form of structured layers. Yet, with few exceptions BID0, previous works propose specialized architectures for specific tasks. Our work aims to contribute to a unified framework that relates optimization algorithms to deep layers. A line of work aims to provide rigorous interpretation for dropout layers. For example, BID32 showed that dropout is linked to an adaptively balanced 2 -regularized loss. BID33 showed that approximating the loss with a normal distribution leads to a faster form of dropout. BID7 BID38 developed a framework that connects dropout with approximate variational inference in Bayesian models. We provide a complementary perspective, in which dropout layers arise naturally in an optimization-driven framework for network design. This section is organized as follows. We introduce our notation and preliminaries in Section 3.1. In Section 3.2, we present a motivational example relating a single iteration of proximal gradient descent (Prox-GD) on to the forward pass through a fully-connected layer followed by a nonlinear activation. We will show that several commonly used non-linear activations can be exactly or approximately represented as proximal operators of g(x). In Section 3.3, we unify fully-connected and convolutional layers as special cases of a high-order tensor product. We propose a generic instance of in a tensor setting, where we provide a formula for the Lipschitz constant L of the finite sum structure of. In Section 3.4, we derive an intimate relation between stochastic solvers, namely τ -nice Prox-SG and mS2GD, and two types of dropout layers. FIG6 shows an overview of the connections that will be developed. FIG6: An overview of the tight relation between a single iteration of a stochastic solver and the forward pass through the l th layer in a network that consists of dropout followed by a linear transformation and a non-linear activation. We study an instance of problem with quadratic F (x), where x l−1 are the input activations and x l, the variables being optimized, correspond to the output activations. Varying the type of stochastic solver changes the nature of the dropout layer, while the prior g(x) on the output activations determines the non-linearity Prox 1 L g . As we will be working with tensors, we will follow the tensor notation of BID15. The order of a tensor is the number of its dimensions. In particular, scalars are tensors of order zero, vectors are tensors of order one, and matrices are tensors of order two. We denote scalars by lowercase letters a, vectors by bold lowercase letters a, and matrices by bold capital letters A. We use subscripts a i to refer to individual elements in a vector. Tensors of order three or more will be denoted by cursive capital letters A ∈ R J1×J2×···×Jn. Throughout the paper, we will handle tensors that are of at most order four. High-order tensors with a second dimension of size equal to one are traditionally called vector tensors and denoted A ∈ R J1×1×J3×J4. We use A(i, j, k, z) to refer to an element in a tensor and A(i, j, k, :) to refer to a slice of a tensor. The inner product between tensors of the same size is denoted A, B = i1,...,i N A (i 1, . . ., i N) B (i 1, . . ., i N). The squared Frobenius norm of a tensor A is defined as A 2 F = A, A. Lastly, the superscripts and H are used to denote the transpose and the Hermitian transpose, respectively. As a motivating example, we consider the l th linear layer in a deep network that is followed by a non-linear activation ρ, i.e. x l = ρ(Ax l−1 + b), where A ∈ R n2×n1 and b ∈ R n2 are the weights and biases of the layer and x l−1 and x l are the input and output activations, respectively. Now consider an instance of with a convex function g(x) and DISPLAYFORM0 where A (i, :) is the i th row of A. Such an objective can be optimized iteratively in x l using Prox-GD with the following update equation: DISPLAYFORM1 where the Lipschitz constant L = λ max AA and λ max denotes the maximum eigenvalue. By initializing the iterative optimization at x l = 0, it becomes clear that a single iteration of is equivalent to a fully-connected layer followed by a non-linearity that is implemented by the proximal operator BID6. The choice of g(x) determines the specific form of the non-linearity ρ. Several popular activation functions can be traced back to their corresponding g(x). The ReLU, which enforces non-negative output activations, corresponds to the indicator function g(x) = 1 x≥0; the corresponding instance of problem is a non-negative quadratic program. Similar observations for the ReLU have been made in other contexts BID0 BID23. We observe that many other activation functions fit this framework. For example, when g(x) is a squared hinge loss, i.e. DISPLAYFORM2 DISPLAYFORM3, a single update of FORMULA2 is equivalent to a linear layer followed by a Leaky ReLU. TAB0 lists some other choices of g(x) and their induced activations. is not required to exhibit a simple, coordinate-wise separable structure. More complex functions can be used, as long as the proximal operator is easy to evaluate. Interesting examples arise when the output activations have matrix structure. For instance, one can impose nuclear norm regularization g(X) = X * to encourage X to be low rank. Alternatively, one can enforce positive semi-definite structure on the matrix X by defining g(X) = 1 X 0. A similar activation has been used for higher-order pooling BID11.In what follows, we will show that this connection can be further extended to explain dropout layers. Interestingly, specific forms of dropout do not arise from particular forms of objective FORMULA0, but from different stochastic optimization algorithms that are applied to it. Before presenting our main on the equivalence between a forward pass through a block of layers and solving with stochastic algorithms, we provide some key lemmas. These lemmas will be necessary for a unified treatment of fully-connected and convolutional layers as generic linear layers. This generic treatment will enable efficient computation of the Lipschitz constant for both fully-connected and convolutional layers. Lemma 1. Consider the l th convolutional layer in a deep network with some non-linear activation, e.g. Prox g , where the weights A ∈ R n2×n1×W ×H, biases B ∈ R n2×1×W ×H, and input activations X l−1 ∈ R n1×1×W ×H are stacked into 4 th -order tensors. We can describe the layer as DISPLAYFORM0 where HO is the high-order tensor product. Here n 1 is the number of input features, n 2 is the number of output features (number of filters), and W and H are the spatial dimensions of the features. As a special case, a fully-connected layer follows naturally, since HO reduces to a matrix-vector multiplication when W = H = 1.The proof can be found in supplementary material. Note that the order of the dimensions is essential in this notation, as the first dimension in A corresponds to the number of independent filters while the second corresponds to the input features that will be aggregated after the 2D convolutions. Also note that according to the definition of HO in BID2, the spatial size of the filters in A, namely W and H, has to match the spatial dimensions of the input activations X l−1, since the operator HO performs 2D circular convolutions while convolutions in deep networks are 2D linear convolutions. This is not a restriction, since one can perform linear convolution through a zero-padded circular convolution. Lastly, we assume that the values in B are replicated along the spatial dimensions W and H in order to recover the behaviour of biases in deep networks. Given this notation, we will refer to either a fully-connected or a convolutional layer as a linear layer throughout the rest of the paper. Since we are interested in a generic linear layer followed by a non-linearity, we will consider the tensor quadratic version of F (x), denoted F (X): DISPLAYFORM1 Note that if A ∈ R n2×n1×W ×H, then A H ∈ R n1×n2×W ×H, where each of the frontal slices of A(:, :, i, j) is transposed and each filter, A(i, j, :, :), is rotated by 180•. This means that A H HO X aggregates the n 2 filters after performing 2D correlations. This is performed n 1 times independently. This operation is commonly referred to as a transposed convolution. Details can be found in supplementary material. Next, the following lemma provides a practical formula for the computation of the Lipschitz constant L of the finite sum part of FORMULA6: DISPLAYFORM2 whereÂ is the 2D discrete Fourier transform along the spatial dimensions W and H.The proof can be found in supplementary material. Lemma 2 states that the Lipschitz constant L is the maximum among the set of maximum eigenvalues of all the possible W × H combinations of the outer product of frontal slicesÂ (:, :, i, j)Â H (:, :, i, j). Note that if W = H = 1, thenÂ = A ∈ R n2×n1 since the 2D discrete Fourier transform of scalars (i.e. matrices of size 1 × 1) is an identity mapping. As a consequence, we can simplify to L = max i=j=1 {λ max A(:, :, i, j)A H (:, :, i, j) } = λ max AA, which recovers the Lipschitz constant for fully-connected layers. In this subsection, we present two propositions. The first shows the relation between standard Bernoulli dropout (p is the dropout rate), BerDropout p BID29, and τ -nice Prox-SG. The second proposition relates additive dropout, AddDropout, to mS2GD BID16. We will first introduce a generic notion of sampling from a set. This is essential as the stochastic algorithms sample unbiased function estimates from the set of n 1 functions in. Definition 3.1. BID9. A sampling is a random set-valued mapping with values being the subsets of [n 1] = {1, . . ., n 1}. A sampling S is τ -nice if it is uniform, i.e. Prob (i ∈ S) = Prob (j ∈ S) ∀ i, j, and assigns equal probabilities to all subsets of [n 1] of cardinality τ and zero probability to all others. Various other types of sampling can be found in BID9. We are now ready to present our first proposition. Proposition 1. A single iteration of Prox-SG with τ -nice sampling S on with τ = (1 − p)n 1, zero initialization, and unit step size can be shown to exhibit the update DISPLAYFORM0 which is equivalent to a forward pass through a BerDropout p layer that drops exactly n 1 p input activations followed by a linear layer and a non-linear activation. We provide a simplified sketch for fully-connected layers here. The detailed proof is in the supplement. To see how reduces to the functional form of BerDropout p followed by a fully-connected layer and a non-linear activation, consider W = H = 1. The argument of Prox 1 L g in (without the bias term) reduces to n 1 τ i∈S A(:, i, :, : DISPLAYFORM1 The first equality follows from the definition of HO, while the second equality follows from trivially reparameterizing the sum, with BerDropout p being equivalent to a mask that zeroes out exactly pn 1 input activations. Note that if τ -nice Prox-SG was replaced with Prox-GD, i.e. τ = n 1, then this corresponds to having a BerDropout p layer with dropout rate p = 0; thus, reduces to A BerDropout p (X l−1) = A X l−1, which recovers our motivating example that relates Prox-GD with the forward pass through a fully-connected layer followed by a non-linearity. Note that Proposition 1 directly suggests how to apply dropout to convolutional layers. Specifically, complete input features from n 1 should be dropped and the 2D convolutions should be performed only on the τ -sampled subset, where τ = (1 − p)n 1.Similarly, the following proposition shows that a form of additive dropout, AddDropout, can be recovered from a different choice of stochastic solver. Proposition 2. A single outer-loop iteration of mS2GD BID16 ) with unit step size and zero initialization is equivalent to a forward pass through an AddDropout layer followed by a linear layer and a non-linear activation. The proof is given in the supplement. It is similar to Proposition 1, with mS2GD replacing τ -nice Prox-SG. Note that any variance-reduced algorithm where one full gradient is computed at least once can be used here as a replacement for mS2GD. For instance, one can show that the serial sampling version of mS2GD, S2GD BID16, and SVRG BID12 can also be used. Other algorithms such as Stochastic Coordinate Descent with arbitrary sampling are discussed in the supplement. A natural question arises as a consequence of our framework: If common layers in deep networks can be understood as a single iteration of an optimization algorithm, what happens if the algorithm is applied for multiple iterations? We empirically answer this question in our experiments. In particular, we embed solvers as a replacement to their corresponding blocks of layers and show that this improves the accuracy of the models without an increase in the number of network parameters. Experimental setup. We perform experiments on CIFAR-10 and CIFAR-100 BID17 ). In all experiments, training was conducted on 90% of the training set while 10% was left for validation. The networks used in the experiments are variants of LeNet , AlexNet BID18, and VGG16 BID27. We used stochastic gradient descent with a momentum of 0.9 and a weight decay of 5 × 10 −4. The learning rate was set to (10 −2, 10 −3, 10 −4) for the first, second, and third 100 epochs, respectively. For finetuning, the learning rate was initially set to 10 −3 and reduced to 10 −4 after 100 epochs. Moreover, when a block of layers is replaced with a deterministic solver, i.e. Prox-GD, the step size is set to the optimal constant 1/L, where L is computed according to Lemma 2 and updated every epoch without any zero padding as a circular convolution operator approximates a linear convolution in large dimensions BID38. In Prox-SG, a decaying step size is necessary for convergence; therefore, the step size is exponentially decayed as suggested by BID3, where the initial step size is again set according to Lemma 2. Finally, to guarantee convergence of the stochastic solvers, we add the strongly convex function λ 2 X 2 F to the finite sum in, where we set λ = 10 −3 in all experiments. Note that for networks that include a stochastic solver, the network will be stochastic at test time. We thus report the average accuracy and standard deviation over 20 trials. Replacing fully-connected layers with solvers. In this experiment, we demonstrate that (i) training networks with solvers replacing one or more blocks of layers can improve accuracy when trained from scratch, and (ii) the improvement is consistently present when one or more blocks are replaced with solvers at different layers in the network. To do so, we train a variant of LeNet on the CIFAR-10 dataset with two BerDropout p layers. The last two layers are fully-connected layers with ReLU activation. We consider three variants of this network: Both fully-connected layers are augmented with BerDropout p (LeNet-D-D), only the last layer is augmented with BerDropout p (LeNet-ND-D), and finally only the penultimate layer is augmented with BerDropout p (LeNet-D-ND). In all cases, we set the dropout rate to p = 0.5. We replace the BerDropout p layers with their corresponding stochastic solvers and run them for 10 iterations with τ = n 1 /2 (the setting corresponding to a dropout rate of p = 0.5). We train these networks from scratch using the same procedure as the baseline networks. The are summarized in Table 2. It can be seen that replacing BerDropout p with the corresponding stochastic solver (τ -nice Prox-SG) improves performance significantly, for any choice of layer. The indicate that networks that incorporate stochastic solvers can be trained stably and achieve desirable generalization performance. LeNet-D-ND LeNet-ND-D Baseline 64.39% 71.72% 68.54% Prox-SG 72.86% ± 0.177 75.20% ± 0.205 76.23% ± 0.206 Table 2: Comparison in accuracy between variants of the LeNet architecture on the CIFAR-10 dataset. The variants differ in the location (D or ND) and number of BerDropout p layers for both the baseline networks and their stochastic solver counterpart Prox-SG. Accuracy consistently improves when Prox-SG is used. Accuracy is reported on the test set. Convolutional layers and larger networks. We now demonstrate that solvers can be used to improve larger networks. We conduct experiments with variants of AlexNet 1 and VGG16 on both CIFAR-10 and CIFAR-100. We start by training strong baselines for both AlexNet and VGG16, achieving 77.3% and 92.56% test accuracy on CIFAR-10, respectively. Note that performance on this dataset is nearly saturated. We then replace the first convolutional layer in AlexNet with the deterministic Prox-GD solver, since this layer is not preceded by a dropout layer. The are summarized in Table 3. We observe that finetuning the baseline network with the solver leads to an improvement of ≈ 1.2%, without any change in the network's capacity. A similar improvement is observed on the harder CIFAR-100 dataset. AlexNet AlexNet-Prox-GD CIFAR-10 77.30% 78.51% CIFAR-100 44.20%45.53% Table 3: Replacing the first convolutional layer of AlexNet by the deterministic Prox-GD solver yields consistent improvement in test accuracy on CIFAR-10 and CIFAR-100.Results on VGG16 are summarized in Table 4. Note that VGG16 has two fully-connected layers, which are preceded by a BerDropout p layer with dropout rate p = 0.5. We start by replacing only the last layer with Prox-SG with 30 iterations and τ = n 1 /2 (VGG16-Prox-SG-ND-D). We further replace both fully-connected layers that include BerDropout p with solvers (VGG16-Prox-SG-D-D). We observe comparable performance for both settings on CIFAR-10. We conjecture that this might be due to the dataset being close to saturation. On CIFAR-100, a more pronounced increase in accuracy is observed, where VGG-16-Prox-SG-ND-D outperforms the baseline by about 0.7%.We further replace the stochastic solver with a deterministic solver and leave the dropout layers unchanged. We denote this setting as VGG16-Prox-GD in Table 4. Interestingly, this setting performs the best on CIFAR-10 and comparably to VGG16-Prox-SG-ND-D on CIFAR-100. CIFAR-10 92.56% 92.44% ± 0.028 92.57% ± 0.029 92.80% CIFAR-100 70.27%70.95% ± 0.042 70.44% ± 0.077 71.10% Table 4: Experiments with the VGG16 architecture on CIFAR-10 and CIFAR-100. Accuracy is reported on the test set. Dropout rate vs. τ -nice sampling. In this experiment, we demonstrate that the improvement in performance is still consistently present across varying dropout rates. Since Proposition 1 has established a tight connection between the dropout rate p and the sampling rate τ in, we observe that for different choices of dropout rate the baseline performance improves upon replacing a block of layers with a stochastic solver with the corresponding sampling rate τ. We conduct experiments with 1 AlexNet BID18 was adapted to account for the difference in spatial size of the images in CIFAR-10 and ImageNet BID5 ). The first convolutional layer has a padding of 5, and all max-pooling layers have a kernel size of 2. A single fully-connected layer follows at the end. VGG16 on CIFAR-100. We train four different baseline models with varying choices of dropout rate p ∈ {0, 0.1, 0.9.0.95} for the last layer. We then replace this block with a stochastic solver with a sampling rate τ and finetune the network. Table 5 reports the accuracy of the baselines for varying dropout rates p and compares to the accuracy of the stochastic solver with corresponding τ (Prox-SG). With a high dropout rate, the performance of the baseline network drops drastically. When using the stochastic solver, we observe a much more graceful degradation. For example, with a sampling rate τ that corresponds to an extreme dropout rate of p = 0.95 (i.e. 95% of all input activations are masked out), the baseline network with BerDropout p suffers a 56% reduction in accuracy while the stochastic solver declines by only 5%. Prox-SG Table 5: Comparison of the VGG16 architecture trained on CIFAR-100 with varying dropout rates p in the last BerDropout p layer. We compare the baseline to its stochastic solver counterpart with corresponding sampling rate τ = (1 − p)n 1. Accuracy is reported on the test set. In summary, our experiments show that replacing common layers in deep networks with stochastic solvers can lead to better performance without increasing the number of parameters in the network. The ing networks are stable to train and exhibit high accuracy in cases where standard dropout is problematic, such as high dropout rates. We have presented equivalences between layers in deep networks and stochastic solvers, and have shown that this can be leveraged to improve accuracy. The presented relationships open many doors for future work. For instance, our framework shows an intimate relation between a dropout layer and the sampling S from the set [n 1] in a stochastic algorithm. As a consequence, one can borrow theory from the stochastic optimization literature to propose new types of dropout layers. For example, consider a serial importance sampling strategy with Prox-SG to solve BID37 BID34, where serial sampling is the sampling that satisfies Prob (i ∈ S, j ∈ S) = 0. A serial importance sampling S from the set of functions f i (X) is the sampling such that Prob DISPLAYFORM0 i.e. each function from the set [n 1] is sampled with a probability proportional to the norm of the gradient of the function. This sampling strategy is the optimal serial sampling S that maximizes the rate of convergence solving BID37. From a deep layer perspective, performing Prox-SG with importance sampling for a single iteration is equivalent to a forward pass through the same block of layers with a new dropout layer. Such a dropout layer will keep each input activation with a non-uniform probability proportional to the norm of the gradient. This is in contrast to BerDropout p where all input activations are kept with an equal probability 1 − p. Other types of dropout arise when considering non-serial importance sampling where |S| = τ > 1.In summary, we have presented equivalences between stochastic solvers on a particular class of convex optimization problems and a forward pass through a dropout layer followed by a linear layer and a non-linear activation. Inspired by these equivalences, we have demonstrated empirically on multiple datasets and network architectures that replacing such network blocks with their corresponding stochastic solvers improves the accuracy of the model. We hope that the presented framework will contribute to a principled understanding of the theory and practice of deep network architectures. A LEAKY RELU AS A PROXIMAL OPERATOR Proof. The proximal operator is defined as Prox g (a) = arg min DISPLAYFORM1 Note that the problem is both convex and smooth. The optimality conditions are given by: DISPLAYFORM2 Since the problem is separable in coordinates, we have: DISPLAYFORM3 The Leaky ReLU is defined as DISPLAYFORM4 which shows that Prox g is a generalized form of the Leaky ReLU with a shift of λ and a slope α = Proof. The proximal operator is defined as Prox g (a) = arg min DISPLAYFORM5 Note that the function g(x) is elementwise separable, convex, and smooth. By equating the gradient to zero and taking the positive solution of the ing quadratic polynomial, we arrive at the closedform solution: DISPLAYFORM6 where denotes elementwise multiplication. It is easy to see that this operator is close to zero for x i << 0, and close to x i for x i >> 0, with a smooth transition for small |x i |.Note that the function Prox g (a) approximates the activation SoftPlus = log(1 + exp (a)) very well. An illustrative example is shown in FIG2. Proposition 5. The proximal operator to g(x) = −γ i log(1 − x i x i) approximates the Tanh non-linearity. Proof. To simplify the exposition, we derive the proximal operator for the case γ = 1. The general case for γ > 0 follows analogously. The proximal operator is defined as Prox g (a) = arg min DISPLAYFORM0 Note that the logarithm is taken element wise, and the objective is convex and smooth. By equating the gradient to zero, it can be seen that the optimal solution is a root of a cubic equation: DISPLAYFORM1 which is defined in each coordinate i separately. DISPLAYFORM2 Since q 2 − p 3 < 0, ∀a i ∈ R, it is guaranteed that all roots are real and distinct. Consequently, the roots can be described as DISPLAYFORM3 Since g(x) is only defined on x ∈ [−1, 1] d, the root that minimizes has to satisfy DISPLAYFORM4 DISPLAYFORM5 It is straightforward to check that DISPLAYFORM6 By substituting f k into and checking inequality it becomes clear that the root corresponding to k = 2 minimizes. By using trigonometric identities the root corresponding to k = 2 can be further simplified to DISPLAYFORM7 which has the approximate shape of the Tanh activation. An example of this operator is shown in FIG2. The proximal operator corresponding to the Sigmoid activation can be derived in a similar fashion by setting g(x) = −γ log(x) − γ log(1 − x). The exposition presented in the paper requires some definitions related to tensors and operators on tensors. We summarize the material here. In all subsequent definitions, we assume D ∈ R n1×n2×n3×n4 and X ∈ R n2×1×n3×n4.Definition D.1. BID2 The t-product between high-order tensors is defined as DISPLAYFORM0 where circ HO (D) ∈ R n1n3n4×n2n3n4 and MatVec HO X ∈ R n2n3n4×1.The operator circ HO unfolds an input tensor into a structured matrix. On the other hand, MatVec HO unfolds an input tensor into a vector. The fold and unfold procedures are detailed in BID2. Definition D.2. BID2 The operator DISPLAYFORM1 DISPLAYFORM2 where bdiag : C n1×n2×n3×n4 → C n1n3n4×n2n3n4, maps a tensor to a block diagonal matrix of all the frontal faces D(:, :, i, j). Note that if n 3 = n 4 = 1, bdiag is an identity mapping. Moreover if n 1 = n 2 = 1, bdiag is a diagonal matrix. Due to the structure of the tensor unfold of circ HO , the ant matrix circ HO (D) exhibits the following blockwise diagonalization: DISPLAYFORM3 where F n is the n × n Normalized Discrete Fourier Matrix. Note thatD has the dimensions n 3 and n 4 replaced with the corresponding 2D Discrete Fourier Transforms. That isD(i, j, :, :) is the 2D Discrete Fourier Transform of D(i, j, :, :).For more details, the reader is advised to start with third order tensors in the work of BID13 and move to the work of BID2 is equivalent to (i) performing 2D convolutions spatially along the third and fourth dimensions.(ii) It aggregates the along the feature dimension n 1. (iii) It repeats the procedure for each of the n 2 filters independently. We will show the following using direct manipulation of the properties of HO BID2. DISPLAYFORM4 Note that Equation FORMULA1 shows that features are aggregated along the n 1 dimension. Now, by showing that A (:, i, :, :) HO X l−1 (i, :, :, :) performs n 2 independent 2D convolutions along on the i th channel, the Lemma 1 is proven. For ease of notation, consider two tensors U ∈ R n2×1×W ×H and Y ∈ R 1×1×W ×H then we have the following: DISPLAYFORM5 2D-Inverse Fourier Transform with stride of of n2 DISPLAYFORM6 Note that Ĝ is the elementwise product of the 2D Discrete Fourier Transform between a feature of an input activation Y and the 2D Discrete Fourier Transform of every filter of the n 2 in Y. Since DISPLAYFORM7 is the inverse 2D Fourier transform along each of the n 2 filters ing in Ĝ. Thus U HO Y performs 2D convolutions independently along each of the n 2 filters, combined with; thus, Lemma 1 is proven. DISPLAYFORM8 Lemma 3. For τ -nice Prox-SG, DISPLAYFORM9 is an unbiased estimator to F (X).Proof. DISPLAYFORM10 The first equality follows by introducing an indicator function where 1 i∈S = 1 if i ∈ S and zero otherwise. The last equality follows from the uniformity across elements of the τ -nice S.From Lemma 3, and with zero initialization it follows that DISPLAYFORM11 is an unbiased estimator of ∇F (X) X =0. The last iteration follows by noting that A = A H H. Therefore, the first iteration of τ -nice Prox-SGD with zero initialization and unit step size is: DISPLAYFORM12 Note that the previous stochastic sum in with τ = (1 − p)n 1 can be reparameterized as follows: DISPLAYFORM13 where M ∈ R n1×1×W ×H is a mask tensor. Note that since τ = (1 − p)n 1, M has exactly pn 1 slices M(i, :, :, :) that are completely zero. This equivalent to a dropout layer where the layer drops exactly pn 1 input activations. It follows that is equivalent to a forward pass through a BerDropout p layer followed by a linear layer and non-linear activation. H PROOF OF PROPOSITION 2 mS2GD BID16 with zero initialization at the first epoch defines the following update: DISPLAYFORM14 With zero initialization at the first epoch we have Y = 0, therefore DISPLAYFORM15 I RANDOMIZED COORDINATE DESCENT PERMUTES DROPOUT AND LINEAR LAYERWe present an additional insight to the role of stochastic solvers on in network design. In particular, we show that performing a randomized coordinate descent, RCD, on ignoring the finite sum structure, is equivalent to a linear transformation followed by BerDropout p and a non-linear activation. That is, performing RCD permutes the order of linear transformation and dropout. For ease of notation, we show this under the special case of fully connected layers. Proposition 6. A single iteration of Randomized Coordinate Descent, e.g. NSync, with τ -nice sampling of coordinates of with τ = (1 − p)n 2, unit step sizes along each partial derivative, and with zero initialization is equivalent to:Prox 1 L g i∈S A(i, :, :, :) HO X l−1 + B(i, :, :, :), which is equivalent to a forward pass through a linear layer followed by a BerDropout p layer (that drops exactly n 2 p output activations) followed by a non-linear activation. Proof. We provide a sketch of the proof on the simple quadratic F (x) = 1 2 A x − x l−1 2 − b x where the linear layer is a fully-connected layer. Considering a randomized coordinate descent, e.g. NSync, with τ -nice sampling of the coordinates we have the following: DISPLAYFORM16 Note that e i is a vector of all zeros except the i th coordinate which is equal to 1. Moreover, since the step sizes along each partial derivative is 1, v = 1. Equation is equivalent to a forward pass through a linear layer followed by a BerDropout p layer and a non-linear activation.

A framework that links deep network layers to stochastic optimization algorithms; can be used to improve model accuracy and inform network design.

Represent the short summary for finding the corresponding scientific article (long):

General analysis of sign-based methods (e.g. signSGD) for non-convex optimization, built on intuitive bounds on success probabilities.


Question: Represent the scientific article to find a short TLDR.

Question: Deep networks run with low precision operations at inference time offer power and space advantages over high precision alternatives, but need to overcome the challenge of maintaining high accuracy as precision decreases. Here, we present a method for training such networks, Learned Step Size Quantization, that achieves the highest accuracy to date on the ImageNet dataset when using models, from a variety of architectures, with weights and activations quantized to 2-, 3- or 4-bits of precision, and that can train 3-bit models that reach full precision baseline accuracy. Our approach builds upon existing methods for learning weights in quantized networks by improving how the quantizer itself is configured. Specifically, we introduce a novel means to estimate and scale the task loss gradient at each weight and activation layer's quantizer step size, such that it can be learned in conjunction with other network parameters. This approach works using different levels of precision as needed for a given system and requires only a simple modification of existing training code. Deep networks are emerging as components of a number of revolutionary technologies, including image recognition , speech recognition, and driving assistance . Unlocking the full promise of such applications requires a system perspective where task performance, throughput, energy-efficiency, and compactness are all critical considerations to be optimized through co-design of algorithms and deployment hardware. Current research seeks to develop methods for creating deep networks that maintain high accuracy while reducing the precision needed to represent their activations and weights, thereby reducing the computation and memory required for their implementation. The advantages of using such algorithms to create networks for low precision hardware has been demonstrated in several deployed systems (; ;). It has been shown that low precision networks can be trained with stochastic gradient descent by updating high precision weights that are quantized, along with activations, for the forward and backward pass . This quantization is defined by a mapping of real numbers to the set of discrete values supported by a given low precision representation (often integers with 8-bits or less). We would like a mapping for each quantized layer that maximizes task performance, but it remains an open question how to optimally achieve this. To date, most approaches for training low precision networks have employed uniform quantizers, which can be configured by a single step size parameter (the width of a quantization bin), though more complex nonuniform mappings have been considered . Early work with low precision deep networks used a simple fixed configuration for the quantizer , while starting with , later work focused on fitting the quantizer to the data, either based on statistics of the data distribution (; ;) or seeking to minimize quantization error during training (c;). Most recently, work has focused on using backpropagation with , FAQ , LQ-Nets , PACT (b), Regularization (c), and NICE (stochastic gradient descent to learn a quantizer that minimizes task loss (; ; b; a; ; ;). While attractive for their simplicity, fixed mapping schemes based on user settings place no guarantees on optimizing network performance, and quantization error minimization schemes might perfectly minimize quantization error and yet still be non optimal if a different quantization mapping actually minimizes task error. Learning the quantization mapping by seeking to minimize task loss is appealing to us as it directly seeks to improve on the metric of interest. However, as the quantizer itself is discontinuous, such an approach requires approximating its gradient, which existing methods have done in a relatively coarse manner that ignore the impact of transitions between quantized states (b; a;). Here, we introduce a new way to learn the quantization mapping for each layer in a deep network, Learned Step Size Quantization (LSQ), that improves on prior efforts with two key contributions. First, we provide a simple way to approximate the gradient to the quantizer step size that is sensitive to quantized state transitions, arguably providing for finer grained optimization when learning the step size as a model parameter. Second, we propose a simple heuristic to bring the magnitude of step size updates into better balance with weight updates, which we show improves convergence. The overall approach is usable for quantizing both activations and weights, and works with existing methods for backpropagation and stochastic gradient descent. Using LSQ to train several network architectures on the ImageNet dataset, we demonstrate significantly better accuracy than prior quantization approaches (Table 1) and, for the first time that we are aware of, demonstrate the milestone of 3-bit quantized networks reaching full precision network accuracy (Table 4). We consider deep networks that operate at inference time using low precision integer operations for computations in convolution and fully connected layers, requiring quantization of the weights and activations these layers operate on. Given data to quantize v, quantizer step size s, the number of positive and negative quantization levels Q P and Q N, respectively, we define a quantizer that computesv, a quantized and integer scaled representation of the data, andv, a quantized representation of the data at the same scale as v:v Here, clip(z, r 1, r 2) returns z with values below r 1 set to r 1 and values above r 2 set to r 2, and z rounds z to the nearest integer. Given an encoding with b bits, for unsigned data (activations) Q N = 0 and Q P = 2 b − 1 and for signed data (weights) Q N = 2 b−1 and Q P = 2 b−1 − 1. For inference,w andx values can be used as input to low precision integer matrix multiplication units underlying convolution or fully connected layers, and the output of such layers then rescaled by the step size using a relatively low cost high precision scalar-tensor multiplication, a step that can potentially be algebraically merged with other operations such as batch normalization (Figure 1). LSQ provides a means to learn s based on the training loss by introducing the following gradient through the quantizer to the step size parameter: This gradient is derived by using the straight through estimator to approximate the gradient through the round function as a pass through operation (though leaving the round itself in place for the purposes of differentiating down stream operations), and differentiating all other operations in Equations 1 and 2 normally. This gradient differs from related approximations (Figure 2), which instead either learn a transformation of the data that occurs completely prior to the discretization itself , or estimate the gradient by removing the round operation from the forward equation, algebraically canceling terms, and then differentiating such that ∂v /∂s = 0 where −Q N < v /s < Q P (b; a). In both such previous approaches, the relative proximity of v to the transition point between quantized states does not impact the gradient to the quantization parameters. However, one can reason that the Figure 2: Given s = 1, Q N = 0, Q P = 3, A) quantizer output and B) gradients of the quantizer output with respect to step size, s, for LSQ, or a related parameter controlling the width of the quantized domain (equal to s(Q P + Q N)) for QIL and PACT (b). The gradient employed by LSQ is sensitive to the distance between v and each transition point, whereas the gradient employed by QIL is sensitive only to the distance from quantizer clip points, and the gradient employed by PACT (b) is zero everywhere below the clip point. Here, we demonstrate that networks trained with the LSQ gradient reach higher accuracy than those trained with the QIL or PACT gradients in prior work. closer a given v is to a quantization transition point, the more likely it is to change its quantization bin (v) as a of a learned update to s (since a smaller change in s is required), thereby ing in a large jump inv. Thus, we would expect ∂v /∂s to increase as the distance from v to a transition point decreases, and indeed we observe this relationship in the LSQ gradient. It is appealing that this gradient naturally falls out of our simple quantizer formulation and use of the straight through estimator for the round function. For this work, each layer of weights and each layer of activations has a distinct step size, represented as an fp32 value, initialized to 2 |v| / √ Q P, computed on either the initial weights values or the first batch of activations, respectively. It has been shown that good convergence is achieved during training where the ratio of average update magnitude to average parameter magnitude is approximately the same for all weight layers in a network . Once learning rate has been properly set, this helps to ensure that all updates are neither so large as to lead to repeated overshooting of local minima, nor so small as to lead to unnecessarily long convergence time. Extending this reasoning, we consider that each step size should also have its update magnitude to parameter magnitude proportioned similarly to that of weights. Thus, for a network trained on some loss function L, the ratio should on average be near 1, where z denotes the l 2 -norm of z. However, we expect the step size parameter to be smaller as precision increases (because the data is quantized more finely), and step size updates to be larger as the number of quantized items increases (because more items are summed across when computing its gradient). To correct for this, we multiply the step size loss by a gradient scale, g, where for weight step size g = 1 / √ N W Q P and for activation step size g = 1 / √ N F Q P, where N W is the number of weights in a layer and N f is the number of features in a layer. In section 3.4 we demonstrate that this improves trained accuracy, and we provide reasoning behind the specific scales chosen in the Section A of the Appendix. Model quantizers are trained with LSQ by making their step sizes learnable parameters with loss gradient computed using the quantizer gradient described above, while other model parameters can be trained using existing techniques. Here, we employ a common means of training quantized networks , where full precision weights are stored and updated, quantized weights and activations are used for forward and backward passes, the gradient through the quantizer round function is computed using the straight through estimator such that and stochastic gradient descent is used to update parameters. For simplicity during training, we usev as input to matrix multiplication layers, which is algebraically equivalent to the previously described inference operations. We set input activations and weights to either 2-, 3-, 4-, or 8-bit for all matrix multiplication layers except the first and last, which always use 8-bit, as making the first and last layers high precision has become standard practice for quantized networks and demonstrated to provide a large benefit to performance. All other parameters are represented using fp32. All quantized networks are initialized using weights from a trained full precision model with equivalent architecture before fine-tuning in the quantized space, which is known to improve performance (; ;). Networks were trained with a momentum of 0.9, using a softmax cross entropy loss function, and cosine learning rate decay without restarts . Under the assumption that the optimal solution for 8-bit networks is close to the full precision solution , 8-bit networks were trained for 1 epoch while all other networks were trained for 90 epochs. The initial learning rate was set to 0.1 for full precision networks, 0.01 for 2-, 3-, and 4-bit networks and to 0.001 for 8-bit networks. All experiments were conducted on the ImageNet dataset , using pre-activation ResNet, VGG with batch norm, or SqueezeNext . All full precision networks were trained from scratch, except for VGG-16bn, for which we used the pretrained version available in the PyTorch model zoo. Images were resized to 256 × 256, then a 224 × 224 crop was selected for training, with horizontal mirroring applied half the time. At test time, a 224 × 224 centered crop was chosen. We implemented and tested LSQ in PyTorch. We expect that reducing model precision will reduce a model's tendency to overfit, and thus also reduce the regularization in the form of weight decay necessary to achieve good performance. To investigate this, we performed a hyperparameter sweep on weight decay for ResNet-18 (Table 2), and indeed found that lower precision networks reached higher accuracy with less weight decay. Performance was improved by reducing weight decay by half for the 3-bit network, and reducing it by a quarter for the 2-bit network. We used these weight decay values for all further experiments. We trained several networks using LSQ and compare accuracy with other quantized networks and full precision baselines (Table 1). To facilitate comparison, we only consider published models that quantize all convolution and fully connected layer weights and input activations to the specified precision, except for the first and last layers which may use higher precision (as for the LSQ models). In some cases, we report slightly higher accuracy on full precision networks than in their original publications, which we attribute to our use of cosine learning rate decay . We found that LSQ achieved a higher top-1 accuracy than all previous reported approaches for 2-, 3-and 4-bit networks with the architectures considered here. For nearly all cases, LSQ also achieved the best-to-date top-5 accuracy on these networks, and best-to-date accuracy on 8-bit versions of these networks. In most cases, we found no accuracy advantage to increasing precision from 4-bit to 8-bit. It is worth noting that the next best low precision method used progressive fine tuning (sequentially training a full precision to 5-bit model, then the 5-bit model to a 4-bit model, and so on), significantly increasing training time and complexity over our approach which fine tunes directly from a full precision model to the precision of interest. It is interesting to note that when comparing a full precision to a 2-bit precision model, top-1 accuracy drops only 2.9 for ResNet-18, but 14.0 for SqueezeNext-23-2x. One interpretation of this is that the SqueezeNext architecture was designed to maximize performance using as few parameters as possible, which may have placed it at a design point extremely sensitive to reductions in precision. For a model size limited application, it is important to choose the highest performing model that fits within available memory limitations. To facilitate this choice, we plot here network accuracy against corresponding model size (Figure 3). We can consider the frontier of best performance for a given model size of the architectures considered here. On this metric, we can see that 2-bit ResNet-34 and ResNet-50 networks offer an absolute advantage over using a smaller network, but with higher precision. We can also note that at all precisions, VGG-16bn exists below this frontier, which is not surprising as this network was developed prior to a number of recent innovations in achieving higher performance with fewer parameters. Figure 3: Accuracy vs. model size for the networks considered here show some 2-bit networks provide the highest accuracy at a given model size. Full precision model sizes are inset for reference. To demonstrate the impact of the step size gradient scale (Section 2.2), we measured R (see Equation 4) averaged across 500 iterations in the middle of the first training epoch for ResNet-18, using different step size gradient scales (the network itself was trained with the scaling as described in the methods to avoid convergence problems). With no scaling, we found that relative to parameter size, updates to step size were 2 to 3 orders of magnitude larger than updates to weights, and this imbalance increased with precision, with the 8-bit network showing almost an order of magnitude greater imbalance than the 2-bit network (Figure 4, left). Adjusting for the number of weights per layer (g = 1 / √ N W), the imbalance between step size and weights largely went away, through the imbalance across precision remained (Figure 4, center). Adjusting for the number of number of weights per layer and precision (g = 1 / √ N W Q P), this precision dependent imbalance was largely removed as well (Figure 4, right). We considered network accuracy after training a 2-bit ResNet-18 using different step size gradient scales (Table 3). Using the network with the full gradient scale (g Step size gradient scale for weight and activation step size respectively) as baseline, we found that adjusting only for weight and feature count led to a 0.3 decrease in top-1 accuracy, and when no gradient scale was applied the network did not converge unless we dropped the initial learning rate. Dropping the initial learning rate in multiples of ten, the best top-1 accuracy we achieved using no gradient scale was 3.4 below baseline, using an initial learning rate of 0.0001. Finally, we found that using the full gradient scaling with an additional ten-fold increase or decrease also reduced top-1 accuracy. Overall, this suggests a benefit to our chosen heuristic for scaling the step size loss gradient. We chose to use cosine learning rate decay in our experiments as it removes the need to select learning rate schedule hyperparameters, is available in most training frameworks, and does not increase training time. To facilitate comparison with in other publications that use step-based learning rate decay, we trained a 2-bit ResNet-18 model with LSQ for 90 epochs, using an initial learning rate of 0.01, which was multiplied by 0.1 every 20 epochs. This model reached a top-1 accuracy of 67.2, a reduction of 0.4 from the equivalent model trained with cosine learning rate decay, but still marking an improvement of 1.5 over the next best training method (see Table 1). We next sought to understand whether LSQ learns a solution that minimizes quantization error (the distance betweenv and v on some metric), despite such an objective not being explicitly encouraged. For this purpose, for a given layer we define the final step size learned by LSQ asŝ and let S be the set of discrete values {0.01ŝ, 0.02ŝ, ..., 20.00ŝ}. For each layer, on a single batch of test data we computed the value of s ∈ S that minimizes mean absolute error, ) where p and q are probability distributions. For purposes of relative comparison, we ignore the first term of Kullback-Leibler divergence, as it does not depend onv, and approximate the second term as −E[log(q(v(s)))], where the expectation is over the sample distribution. For a 2-bit ResNet-18 model we foundŝ = 0.949 ± 0.206 for activations andŝ = 0.025 ± 0.019 for weights (mean ± standard deviation). The percent absolute difference betweenŝ and the value of s that minimizes quantization error, averaged across activation layers was 50% for mean absolute error, 63% for mean square error, and 64% for Kullback-Leibler divergence, and averaged across weight layers, was 47% for mean absolute error, 28% for mean square error, and 46% for Kullback-Leibler divergence. This indicates that LSQ learns a solution that does not in fact minimize quantization error. As LSQ achieves better accuracy than approaches that directly seek to minimize quantization error, this suggests that simply fitting a quantizer to its corresponding data distribution may not be optimal for task performance. To better understand how well low precision networks can reproduce full precision accuracy, we combined LSQ with same-architecture knowledge distillation, which has been shown to improve low precision network training . Specifically, we used the distillation loss function of with temperature of 1 and equal weight given to the standard loss and the distillation loss (we found this gave comparable to weighting the the distillation loss two times more or less than the standard loss on 2-bit ResNet-18). The teacher network was a trained full precision model with frozen weights and of the same architecture as the low precision network trained. As shown in Table 4, this improved performance, with top-1 accuracy increasing by up to 1.1 (3-bit ResNet-50), and with 3-bit networks reaching the score of the full precision baseline (see Table 1 for comparison). As a control, we also used this approach to distill from the full precision teacher to a full precision (initially untrained) student with the same architecture, which did not lead to an improvement in the student network accuracy beyond training the student alone. These reinforce previous work showing that knowledge-distillation can help low precision networks catch up to full precision performance . Table 4: Accuracy for low precision networks trained with LSQ and knowledge distillation, which is improved over using LSQ alone, with 3-bit networks reaching the accuracy of full precision (32-bit) baselines (shown for comparison). The presented here demonstrate that on the ImageNet dataset across several network architectures, LSQ exceeds the performance of all prior approaches for creating quantized networks. We found best performance when rescaling the quantizer step size loss gradient based on layer size and precision. Interestingly, LSQ does not appear to minimize quantization error, whether measured using mean square error, mean absolute error, or Kullback-Leibler divergence. The approach itself is simple, requiring only a single additional parameter per weight or activation layer. Although our goal is to train low precision networks to achieve accuracy equal to their full precision counterparts, it is not yet clear whether this goal is achievable for 2-bit networks, which here reached accuracy several percent below their full precision counterparts. However, we found that such 2-bit solutions for state-of-the-art networks are useful in that they can give the best accuracy for the given model size, for example, with an 8MB model size limit, a 2-bit ResNet-50 was better than a 4-bit ResNet-34 (Figure 3). This work is a continuation of a trend towards steadily reducing the number of bits of precision necessary to achieve good performance across a range of network architectures on ImageNet. While it is unclear how far it can be taken, it is noteworthy that the trend towards higher performance at lower precision strengthens the analogy between artificial neural networks and biological neural networks, which themselves employ synapses represented by perhaps a few bits of information and single bit spikes that may be employed in small spatial and/or temporal ensembles to provide low bit width data representation. Analogies aside, reducing network precision while maintaining high accuracy is a promising means of reducing model size and increasing throughput to provide performance advantages in real world deployed deep networks. We compute our gradient scale value by first estimating R (Equation 4), starting with the simple heuristic that for a layer with N W weights To develop this approximation, we first note that the expected value of an l 2 -norm should grow with the square root of the number of elements normalized. Next, we assume that where Q P = 1, step size should be approximately equal to average weight magnitude so as to split the weight distribution into zero and non zero values in a roughly balanced fashion. Finally, we assume that for larger Q P, step size should be roughly proportional to 1 /Q P, so that as the number of available quantized states increases, data between the clip points will be quantized more precisely, and the clip points themselves (equal to sQ N and sQ P) will move further out to better encode outliers. We also note that, in the expectation, ∇ w L and ∇ s L are of approximately the same order. This can be shown by starting from the chain rule then assuming ∂ŵi /∂s is reasonably close to 1 (see for example Figure 2), and treating all ∂L /∂ŵi as uncorrelated zero-centered random variables, to compute the following expectation across weights: By assuming ∂ŵ /∂w = 1 for most weights, we similarly approximate Bringing all of this together, we can then estimate Knowing this expected imbalance, we compute our gradient scale factor for weights by simply taking the inverse of R, so that g is set to 1 / √ N W Q P. As most activation layers are preceded by batch normalization , and assuming updates to the learned batch normalization scaling parameter is the primary driver of changes to pre-quantization activations, we can use a similar approach to the above to show that there is an imbalance between step size updates and update driven changes to activations that grows with the number of features in a layer, N F as well as Q P. Thus, for activation step size we set g to 1 / √ N F Q P. In this section we provide pseudocode to facilitate the implementation of LSQ. We assume the use of automatic differentiation, as supported by a number of popular deep learning frameworks, where the desired operations for the training forward pass are coded, and the automatic differentiation engine computes the gradient through those operations in the backward pass. Our approach requires two functions with non standard gradients, gradscale (Function 1) and roundpass (Function 2). We implement the custom gradients by assuming a function called detach that returns its input (unmodified) during the forward pass, and whose gradient during the backward pass is zero (thus detaching itself from the backward graph). This function is used in the form: so that in the forward pass, y = x 1 (as the x 2 terms cancel out), while in the backward pass ∂L /∂x1 = 0 (as detach blocks gradient propagation to x 1) and ∂L /∂x2 = ∂L /∂y. We also assume a function nf eatures that given an activation tensor, returns the number of features in that tensor, and nweights that given a weight tensor, returns the number of weights in that tensor. Finally, the above are used to implement a function called quantize, which quantizes weights and activations prior to their use in each convolution or fully connected layer. The pseudocode provided here is chosen for simplicity of implementation and broad applicability to many training frameworks, though more compute and memory efficient approaches are possible. This example code assumes activations are unsigned, but could be modified to quantize signed activations.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A method for learning quantization configuration for low precision networks that achieves state of the art performance for quantized networks.

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the scientific article to find a short TLDR

Question: There have been several studies recently showing that strong natural language understanding (NLU) models are prone to relying on unwanted dataset biases without learning the underlying task, ing in models which fail to generalize to out-of-domain datasets, and are likely to perform poorly in real-world scenarios. We propose several learning strategies to train neural models which are more robust to such biases and transfer better to out-of-domain datasets. We introduce an additional lightweight bias-only model which learns dataset biases and uses its prediction to adjust the loss of the base model to reduce the biases. In other words, our methods down-weight the importance of the biased examples, and focus training on hard examples, i.e. examples that cannot be correctly classified by only relying on biases. Our approaches are model agnostic and simple to implement. We experiment on large-scale natural language inference and fact verification datasets and their out-of-domain datasets and show that our debiased models significantly improve the robustness in all settings, including gaining 9.76 points on the FEVER symmetric evaluation dataset, 5.45 on the HANS dataset and 4.78 points on the SNLI hard set. These datasets are specifically designed to assess the robustness of models in the out-of-domain setting where typical biases in the training data do not exist in the evaluation set. Recent neural models (; ;) have achieved high and even near human-performance on several large-scale natural language understanding benchmarks. However, it has been demonstrated that neural models tend to rely on existing idiosyncratic biases in the datasets, and leverage superficial correlations between the label and existing shortcuts in the training dataset to perform surprisingly well 1, without learning the underlying task (; ; ; ; ;). For instance, natural language inference (NLI) consists of determining whether a hypothesis sentence (There is no teacher in the room) can be inferred from a premise sentence (Kids work at computers with a teacher's help) 2 . However, recent work has demonstrated that large-scale NLI benchmarks contain annotation artifacts; certain words in the hypothesis are highly indicative of inference class that allow models with poor premise grounding to perform unexpectedly well . As an example, in some NLI benchmarks, negation words such as "nobody", "no", and "not" in the hypothesis are often highly correlated with the contradiction label. As a consequence, NLI models do not need to learn the true relationship between the premise and hypothesis and instead can rely on statistical cues, such as learning to link negation words with the contradiction label. As a of the existence of such biases, models exploiting statistical shortcuts during training often perform poorly on out-of-domain datasets, especially if they are carefully designed to limit the spurious cues. To allow proper evaluation, recent studies have tried to create new evaluation datasets that do not contain such biases . Unfortunately, it is hard to avoid spurious statistical cues in the construction of large-scale benchmarks, and collecting 1 We use biases, heuristic patterns or shortcuts interchangeably. 2 The given sentences are in the contradictory relation and the hypothesis cannot be inferred from the premise. new datasets is costly . It is therefore crucial to develop techniques to reduce the reliance on biases during the training of the neural models. In this paper, we propose several end-to-end debiasing techniques to adjust the cross-entropy loss to reduce the biases learned from datasets, which work by down-weighting the biased examples so that the model focuses on learning hard examples. Figure 1 illustrates an example of applying our strategy to prevent an NLI model from predicting the labels using existing biases in the hypothesis. Our strategy involves adding a bias-only branch f B on top of the base model f M during training (In case of NLI, the bias-only model only uses the hypothesis). We then compute the combination of the two models f C in a way to motivate the base model to learn different strategies than the ones used by the bias-only branch f B. At the end of the training, we remove the bias-only classifier and use the predictions of the base model. We propose three main debiasing strategies, detailed in Section 2.2. In our first two proposed methods, the combination is done with an ensemble method which combines the predictions of the base and the bias-only models. The training loss of the base model is then computed on the output of this combined model f C. This has the effect of reducing the loss going from the combined model to the base model for the examples which the bias-only model classifies correctly. For the third method, the bias-only predictions are used to directly weight the loss of the base model, explicitly modulating the loss depending on the accuracy of the bias-only model. All strategies work by allowing the base model to focus on learning the hard examples, by preventing it from learning the biased examples. Our approaches are simple and highly effective. They require training a simple classifier on top of the base model. Furthermore, our methods are model agnostic and general enough to be applicable for addressing common biases seen in several datasets in different domains. We evaluate our models on challenging benchmarks in textual entailment and fact verification. For entailment, we run extensive experiments on HANS (Heuristic Analysis for NLI Systems) , and hard NLI sets of Stanford Natural Language Inference (SNLI) and MultiNLI (MNLI) datasets (. We additionally construct hard MNLI datasets from MNLI development sets to facilitate the out-of-domain evaluation on this dataset 3 . Furthermore, we evaluate our fact verification models on FEVER Symmetric test set . The selected datasets are highly challenging and have been carefully designed to be unbiased to allow proper evaluation of the out-of-domain performance of the models. We show that including our strategies on training baseline models including BERT provide substantial gain on out-of-domain performance in all the experiments. In summary, we make the following contributions: 1) Proposing several debiasing strategies to train neural models that make them more robust to existing biases in the dataset. 2) An empirical evaluation of the proposed methods on two large-scale NLI benchmarks and obtaining substantial gain on their challenging out-of-domain data, including 5.45 points on HANS and 4.78 points on SNLI hard set. 3) Evaluating our models on fact verification, obtaining 9.76 points gain on FEVER symmetric test set, improving the of prior work by 4.65 points. To facilitate future work, we release our datasets and code. Problem formulation We consider a general multi-class classification problem. Given a dataset consisting of the input data x i ∈ X, and labels y i ∈ Y, the goal of the base model is to learn a mapping f M parameterized by θ M which computes the predictions over the label space given the input data, shown as f M: X → R |Y|. Our goal is to optimize θ M parameters such that we build a model which is more resistant to benchmark biases to improve its robustness to domain changes when the typical biases observed in the training data do not exist in the evaluation dataset. The key idea of our approach, depicted in Figure 1 is first to identify the dataset biases and heuristic patterns which the base model is susceptible to relying on. Then, we use a bias-only branch to capture these biases. We propose several strategies to incorporate the bias-only knowledge into the training of the base model to make a robust version of it. After training we remove the biasonly model and use the predictions of the base model. In this section, we explain each of these components. We assume that we do not have access to any data from the out-of-domain dataset, so we need to know a priori about the possible types of shortcut patterns we would like the base model to avoid relying on them. Once these shortcut patterns are identified, we train a bias-only model designed to capture the identified biases which only uses the biased features. For instance, it has been shown that a hypothesis-only model in the large-scale NLI datasets can correctly classify the majority of samples using the artifacts . Therefore, our bias-only model for NLI only uses hypothesis sentences. But note that the bias-only model can, in general, have any form, and is not limited to models which are using only a part of input data. Let x b i ∈ X b be biased features of x i which are predictive of y i. We then formalize this bias-only model as a mapping f B: X b → R |Y| parameterized by θ B trained using cross-entropy loss L B: where a i is the one-hot representation of the true label for the i th example. In the next section, we explain how we use the bias-only model to make a robust version of the base model. We propose several strategies to incorporate the bias-only f B knowledge into training of the base model f M and update its parameters θ M using the obtained loss L C of the combined classifier f C. All these strategies have the form illustrated in Figure 1, where the predictions of the bias-only model are combined with either the predictions of the base model or its error to down-weight the loss from the biased examples, thereby affecting the error backpropagated into the base model. As also illustrated in Figure 1, it is often convenient for the bias-only model to share parameters with the base model, such as sharing a sentence encoder. To prevent the base model from learning the biases, the bias-only loss L B is not back-propagated to these shared parameters of the base model. To accommodate this sharing, the bias-only and the base models are trained together. Next, we explain how the loss of the combined classifier, L C, is computed for each of our debiasing methods. Our first approach is based on the idea of the product of experts ensemble method : "It is possible to combine multiple probabilistic models of the same data by multiplying the probabilities together and then renormalizing.". Here, we use this notion to combine the bias-only and base model predictions by computing the element-wise product between their predictions as f B (x We compute this combination in the logarithmic space, which works better in practice: The key intuition behind this model is to combine the probability distributions of the bias-only and the base model to allow them to make predictions based on different characteristics of the input; the bias-only branch covers prediction based on biases, and the base model focuses learning the actual task. We then compute L C as the cross-entropy loss of the combined predictions f C . Then the base model parameters θ M are trained using the cross-entropy loss of the combined classifier f C : When this loss is backpropagated to base model parameters θ M, the predictions of the bias-only model decrease the updates for examples which it can accurately predict. propose a model called RUBI to alleviate unimodal biases learned by Visual Question Answering (VQA) models.'s study is limited to alleviating biases in VQA benchmarks. We, however, evaluate the effectiveness of their formulation together with our newly proposed variations in the natural language understanding context on several challenging NLU datasets. We first apply a sigmoid function to the bias-only model's predictions to obtain a mask containing an importance weight between 0 and 1 for each possible label. We then compute the element-wise product between the obtained mask and the base model's predictions: The main intuition is to dynamically adjust the predictions of the base model to prevent the base model from leveraging the shortcuts. We note two properties of this loss. When the bias-only model correctly classifies the example, the mask increases the value of the correct prediction while decreases the scores for other labels. As a , the loss of biased examples is down-weighted. For the hard examples that cannot be correctly classified using bias-only model, the obtained mask increases the score of the wrong answer. This, in turn, increases the contribution of hard examples and encourages the base model to learn the importance of correcting them. We additionally propose the following new variants of this model: 1. Computing the combination in logarithmic space, which we refer to it as RUBI + log space. 2. Normalizing the output of the bias-only model, followed by RUBI model, which we refer to it as RUBI + normalize: As with our first method, we then update the parameters of the base model θ M by backpropagating the cross-entropy loss L C of the combined classifier. Focal loss was originally proposed in to improve a single classifier by downweighting the well-classified points. We propose a novel variant of this loss, in which we leverage the bias-only branch's predictions to reduce the relative importance of the most biased examples and allow the model to focus on learning the hard examples. We define Debiased Focal Loss as: where γ is the focusing parameter, which impacts the down-weighting rate. When γ is set to 0, our Debiased Focal Loss is equivalent to the normal cross-entropy loss. For γ > 0, as the value of γ is increased, the effect of down-weighting is increased. We set γ = 2 through all experiments, which works well in practice and avoid fine-tuning it further. We note the properties of the Debiased Focal Loss: When the example x i is unbiased, and bias-only branch does not do well, is small, therefore the scaling factor is close to 1, and the loss remains unaffected. As the sample is more biased and softmax (f B (x b i)) is closer to 1, the modulating factor approaches 0 and the loss for the most biased examples is down-weighted. For this debiasing strategy, Debiased Focal Loss is then used to update the parameters of the base model θ M. Note that this loss has a different form from that used for the first two methods. We provide experiments on two large-scale NLI datasets, namely SNLI and MNLI, and FEVER dataset for our fact verification experiment and evaluate the models' performance on their challenging unbiased evaluation datasets proposed very recently. In most of our experiments, we consider BERT 4 as our baseline which is known to work well for these tasks, and additionally, we have included other baselines used in the prior work to compare against them. In all the experiments, we kept the hyperparameters of baselines as the default. We include low-level details in the appendix. Dataset: FEVER dataset contains claim-evidence pairs generated from Wikipedia. collect a new evaluation set for FEVER dataset to avoid the idiosyncrasies observed in the claims of this benchmark. They make the original claim-evidence pairs of FEVER evaluation dataset symmetric, by augmenting the dataset and making each claim and evidence appear with each label. Therefore, by balancing the artifacts, relying on cues from claim to classify samples is equivalent to a random guess. The collected dataset is challenging and the performance of the models evaluated on this dataset drop significantly. Base models: We consider BERT as the baseline, which works the best on this dataset , and predicts the relations based on the concatenation of the claim and the evidence with a delimiter token (see Appendix A). The bias-only model predicts the labels using only claims as input. Results: Table 1 shows the . The obtained improvement of our debiasing methods varies between 1.11-9.76 absolute points. The Product of experts and Debiased Focal loss are highly effective, boosting the performance of the baseline model by 9.76 and 7.53 absolute points respectively, significantly surpassing the prior work . Datasets: We evaluate on hard SNLI and MNLI datasets which are the split of these datasets where a hypothesis-only model cannot correctly predict the labels. show that the success of the recent textual entailment models is attributed to the biased examples, and the performance of these models are substantially lower on hard sets. Base models: We consider InferSent , and BERT as our base models. We choose InferSent to be able to compare against the prior work (b). The bias-only model only uses the hypothesis to predict the labels (see Appendix B). , to predict the labels using only the hypothesis and consider the subset of the samples on which our trained hypothesis-only classifier failed as hard examples. Table 3 shows the on the development sets and their corresponding hard sets. For BERT baseline, on MNLI matched hard dataset, the product of experts and RUBI+normalize improve the the most by 1.46 and 1.11 points. On MNLI mismatched hard, the Debiased Focal Loss and product of experts obtain 1.37, and 1.68 points gain respectively. For InferSent baseline, on MNLI matched hard, the product of experts and RUBI improve the by 2.34 and 0.94 points. On MNLI mismatched hard, the Product of experts and Debiased Focal Loss improve the by 2.61 and 2.52 points. To comply with limited access to the submission system of MNLI, we evaluate only the best of baseline and our models on the test sets. Table 4 shows the on the MNLI test and hard sets. Our product of expert model improves the performance on MNLI matched hard set by 0.93 points and 1.08 points on MNLI Mismatched hard set while maintaining the in-domain accuracy. show that NLI models can rely on superficial syntactic heuristics to perform the task. They introduce HANS dataset, which covers several examples on which the models employing the syntactic heuristics fail. Base model: We use BERT as our base model and train it on MNLI dataset. We consider several features for the bias-only model. The first three features are based on the syntactic heuristics proposed in: 1) Whether all the words in the hypothesis are included in the premise. 2) If the hypothesis is the contiguous subsequence of the premise. 3) If the hypothesis is a subtree in the premise's parse tree 4) The number of tokens shared between premise and hypothesis normalized by the number of tokens in the premise. We additionally include some similarity features: 5) The cosine similarity between premise and hypothesis tokens followed by mean and max-pooling. We consider the same weight for contradiction and neutral labels in the bias-only loss to allow the model to recognize entailment from not-entailment. During the evaluation, we map the neutral and contradiction labels to not-entailment. Results: As shown in Table 5, the Product of experts and Debiased Focal loss improve the the most by 5.45, 3.89 points. We provide the accuracy for each label on HANS dataset in Appendix C. To understand the impact of γ in Debiased Focal Loss, we train InferSent models with this loss for different values of γ on SNLI dataset and evaluate its performance on SNLI and SNLI hard sets. As illustrated in Figure 2, increasing γ focuses the loss on learning hard examples, and reduces the attention on learning biased examples. Consequently, the in-domain accuracy on SNLI is dropped but out-of-domain accuracy on SNLI hard set is increased. Results: Through extensive experiments on different datasets, our methods improve out-of-domain performance in all settings. Debiased Focal Loss and Product of experts models consistently obtain the highest gains. Within RUBI variations, RUBI+log space outperforms the other variations on SNLI with BERT baseline and HANS dataset. RUBI+normalize does better than the rest on FEVER experiment and MNLI matched hard set with BERT baseline. RUBI performs the best on SNLI and MNLI experiments with InferSent baseline, and MNLI mismatched hard with BERT baseline. As expected, improving the out-of-domain performance could come at the expense of the decreased in-domain performance, since the removed biases are useful for performing the in-domain task. This especially happens for Debiased Focal Loss, in which there is a trade-off between in-domain and out-of-domain performance as discussed depending on the parameter γ, and when the baseline model is not very powerful like InferSent. Our other models with BERT baseline consistently remain the in-domain performance. Biases in NLU benchmarks and other domains Recent studies have shown that large-scale NLU benchmarks contain biases.;; demonstrate that textual entailment models can rely on annotation artifacts and heuristic patterns to perform unexpectedly well. On ROC Stories corpus , show that considering only sample endings without story contexts performs exceedingly well. A similar phenomenon is observed in fact verification , argument reasoning comprehension , and reading comprehension . Finally, several studies confirm biases in VQA datasets, leading to accurate question-only models ignoring visual content. Existing techniques to alleviate biases The most common strategy to date to address biases is to augment the datasets by balancing the existing cues . In another line of work, to address the shortcoming in Stanford Question Answering dataset , propose to create an adversarial dataset in which they insert adversarial sentences to the input paragraphs. However, collecting new datasets especially in large-scale is costly and it remains an unsatisfactory solution. It is, therefore, crucial to develop strategies to allow training models on the existing biased datasets, while improving their out-of-domain performance. propose to first compute the n-grams existing in the claims which are the most associated with each label. They then solve an optimization problem to assign a balancing weight to each training sample to alleviate the biases. In contrast, we propose several end-to-end debiasing strategies. Additionally, Belinkov et al. (2019a) propose adversarial techniques to remove from the sentence encoder the features which allow a hypothesis-only model to succeed. However, we believe that in general the features used by the hypothesis-only model can include some information necessary to perform the NLI task, and removing such information from the sentence representation can hurt the performance of the full model. Their approach consequently degrades the performance on hard SNLI dataset which is expected to be less biased. In contrast to their method, we propose to train a bias-only model to use its predictions to dynamically adapt the classification loss to reduce the importance of the most biased examples during training. Concurrently to our own work,; have also proposed to use the product of experts models. However, we have evaluated on new domains and datasets, and have proposed several different ensemble-based debiasing techniques. We propose several novel techniques to reduce biases learned by neural models. We introduce a bias-only model that is designed to capture biases and leverages the existing shortcuts in the datasets to succeed. Our debiasing strategies then work by adjusting the cross-entropy loss based on the performance of this bias-only model to focus learning on the hard examples and down-weight the importance of the biased examples. Our proposed debiasing techniques are model agnostic, simple and highly effective. Extensive experiments show that our methods substantially improve the model robustness to domain-shift, including 9.76 points gain on FEVER symmetric test set, 5.45 on HANS dataset and 4.78 points on SNLI hard set. Base model: InferSent uses a separate BiLSTM encoder to learn sentence representations for premise and hypothesis, it then combines these embeddings following and feeds them to the default nonlinear classifier. For InferSent we train all models for 20 epochs as default without using early-stopping. We use the default hyper-parameters and following, we set BiLSTM dimension to 512. We use the default nonlinear classifier with 512 and 512 hidden neurons with Tanh nonlinearity. For Bert model, we finetune the models for 3 epochs. Bias-only model For BERT model, we use the same shallow nonlinear classifier explained in Appendix A, and for the InferSent model, we use a shallow linear classifier with 512, and 512 hidden units. Base model: We finetune all the models for 3 epochs. We use a nonlinear classifier with 6 and 6 hidden units with Tanh nonlinearity. Results: Table 6 shows the performance for each label (entailment and non entailment) on HANS dataset and its individual heuristics.

We propose several general debiasing strategies to address common biases seen in different datasets and obtain substantial improved out-of-domain performance in all settings.

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.


Question: Represent the scientific article to find a short TLDR

Question: Reconstruction of few-view x-ray Computed Tomography (CT) data is a highly ill-posed problem. It is often used in applications that require low radiation dose in clinical CT, rapid industrial scanning, or fixed-gantry CT. Existing analytic or iterative algorithms generally produce poorly reconstructed images, severely deteriorated by artifacts and noise, especially when the number of x-ray projections is considerably low. This paper presents a deep network-driven approach to address extreme few-view CT by incorporating convolutional neural network-based inference into state-of-the-art iterative reconstruction. The proposed method interprets few-view sinogram data using attention-based deep networks to infer the reconstructed image. The predicted image is then used as prior knowledge in the iterative algorithm for final reconstruction. We demonstrate effectiveness of the proposed approach by performing reconstruction experiments on a chest CT dataset. Computed Tomography (CT) reconstruction is an inverse problem where images are reconstructed from a collection of multiple x-ray projections known as sinogram. Conventional CT imaging systems use densely sampled x-ray projections (roughly equal to one projection per detector column) with a full angular range (180-360 degrees). Unlike the conventional CT setup, on the other hand, some CT systems use different imaging configurations that require rapid scanning or reduced radiation dose. In those cases, the CT imaging uses a small number of x-ray projections, referred to as few-view CT. Reconstructing images from a few x-ray projections becomes an extremely under-determined inverse problem, which in significant image degradation. The reconstructed images from extremely few-view sinogram measurement (10 views or less) are often characterized by severe artifacts and noise, even with state-of-the-art regularized iterative algorithms as well as with the widely used Filtered Backprojection (FBP). In recent years, deep learning-based approaches have been successfully applied to a number of image restoration, denoising, inpainting and other image processing applications. Methods in this category use perceptual information as well as contextual features to improve the image quality. In CT imaging applications, several deep convolutional neural network (CNN) approaches have been proposed to address different ill-conditioned CT reconstruction applications. Methods in proposed CNN-based approaches to improve the image quality for low-dose (sparse-view) imaging. These approaches aim to infer the noise distribution to generate a cleaner image from the noisy image. However, these methods do not employ the sinogram to ensure that the reconstructed image is consistent with the measurement. Gupta et al. proposed a method using a CNN-based projector for moderate sparse-view reconstruction (45 and 144 views). Anirudh et al. proposed a CNN-based sinogram completion approach to address limited-angle CT reconstruction. In this paper, we present a CNN inference-based reconstruction algorithm to address extremely few-view CT imaging scenarios. For the initial reconstruction, we employ a CNN-based inference model, based on CT-Net, that directly uses the input measurement (few-view sinogram data) to predict the reconstructed image. In the cases where the sinogram measurements are extremely undersampled, the images reconstructed by existing analytic and iterative methods may suffer from too much noise with little high frequency information, and the methods in may repair the missing or noisy part with perceptually created, but incorrect content. Thus, we pursue a method that directly uses the sinogram so that the reconstructed content is consistent with the input measurement, as an inverse problem. Furthermore, instead of performing the sinogram completion in optimized for limited-angle reconstruction, we propose to use the predicted image from the CNN inference model as an image prior in state-of-the-art iterative algorithms in order to improve the final reconstruction. Our experiments on a chest CT dataset show that the proposed model outperforms existing analytical and state-of-the-art iterative algorithms as well as the sinogram completion. The proposed approach is comprised of two stages: (a) CNN-driven inference, (b) iterative reconstruction with the image prior, as illustrated in Figure 1. The algorithm begins with CNN-based deep inferencing to predict the reconstructed images from few-view sinogram data. Then we perform an iterative algorithm with the predicted images as initial guess. For the CNN-based deep inferencing, we adopt CT-Net proposed in but modify it for few-view CT applications. Our network architecture is similar to the original CT-Net, consisting of 1D-2D CNNs to infer images directly from the sinogram data. Unlike limited-angle reconstruction, however, the sinogram in few-view CT is spatially less coherent and the number of projections is smaller compared to limited-angle reconstruction, and we use smaller filter sizes in the 1D CNN. For the training loss, we use SSIM loss in addition to L 2 and adversarial loss with a discriminator to generate more realistic CT images. We empirically choose the weight between L 2 and SSIM losses (0.7 in our experiment). The predicted reconstructed images are then used as image prior in the Regularized Least Squares (RLS)-based iterative algorithm. In RLS, images are reconstructed by solving the following optimization:X = arg min where y is measured sinogram data,X is the reconstructed attenuation coefficients of the object, A is the x-ray projection matrix subject to the projection geometry, and R(x) is a Total Variation regularization functional, and β serves as a weight to control the strength of the regularization. The β term is determined empirically. The optimization typically requires 50 − 100 iterations, starting with a randomly chosen initial guess. However, the optimization with a carefully selected initial guess enables high quality reconstruction as well as fast convergence. To this end, we propose to use our deep network-driven prediction as image prior (initial guess) in this iterative algorithm, which enables to recover edge-preserving high frequency regions in the reconstructed images. There are a set of parameters to be chosen empirically in the RLS algorithm. For our experiment on the chest CT dataset, we chose 100 iterations with β = 2e −2 in and the non-negative constraint. To demonstrate effectiveness of the proposed approach, we perform experiments on a chest CT dataset provided by the National Cancer Institute. The chest CT dataset is comprised of 1595 patients of chest scans for the study of lung nodule detection as an early stage cancer diagnosis. We split the entire scans into 1,000 patients of scans for training and the rest for testing. The reconstructed volumes are 512 × 512 × N where N is the number of slices ranging from 94 to 541, and each slice is treated independently to demonstrate the parallel-beam projection based reconstruction. We down-sampled the original 512 × 512 slices into 128 × 128 to fit the proposed model to a single GPU memory. Since the dataset does not provide the original sinogram, we performed forward projection using Livermore Tomography Tools to generate sinogram of 180 × 128 where 180 is the number of projection views. We then sampled sinogram for the training and test input to perform few-view experiments. In 9-view reconstruction, for example, we sampled views at 0, 20, 40, 60, 80, 100, 120, 140, 160 degrees from 180 views. For the training, we used Adam optimizer with learning rate of 1 × 10 −3 and the exponential decay of every 10,000 steps with a base of 0.97. The mini-batch size is 50 and the number of epochs is 50. We trained our model using Tensorflow on NVIDIA GeForce RTX 2080 Ti. To evaluate the reconstruction performance, we randomly sampled 50 images from the test dataset and performed our reconstruction together with FBP, RLS and Sinogram Completion (SC) as baseline algorithms. We report PSNR and SSIM of the reconstruction . Figure 2 shows qualitative from two different views, 6 and 12, respectively. We observe that the proposed method yields sharper edges and more high frequency regions with less artifact and noise, compared to the baseline algorithms. Figure 3 shows quantitative evaluation on 6 different few-view experiments (3, 6, 9, 12, 15, 18 views). PSNR and SSIM show that the proposed approach outperforms the baselines although both metrics are not sufficient to measure perceptual quality. Another observation is that RLS improves the image quality as the number of views increases while our method outperforms by a large margin especially when the number of views is extremely small. This document was prepared as an account of work sponsored by an agency of the United States government. Neither the United States government nor Lawrence Livermore National Security, LLC, nor any of their employees makes any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States government or Lawrence Livermore National Security, LLC. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States government or Lawrence Livermore National Security, LLC, and shall not be used for advertising or product endorsement purposes.

We present a CNN inference-based reconstruction algorithm to address extremely few-view CT.

Represent the TLDR for finding the corresponding scientific article!

In this paper we introduced an algorithm to learn the connectivity of deep multi-branch networks. The approach is evaluated on image categorization where it consistently yields accuracy gains over state-of-the-art models that use fixed connectivity.