Question: Represent the scientific article to find a short TLDR

Question: Nesterov SGD is widely used for training modern neural networks and other machine learning models. Yet, its advantages over SGD have not been theoretically clarified. Indeed, as we show in this paper, both theoretically and empirically, Nesterov SGD with any parameter selection does not in general provide acceleration over ordinary SGD. Furthermore, Nesterov SGD may diverge for step sizes that ensure convergence of ordinary SGD. This is in contrast to the classical in the deterministic setting, where the same step size ensures accelerated convergence of the Nesterov's method over optimal gradient descent. To address the non-acceleration issue, we introduce a compensation term to Nesterov SGD. The ing algorithm, which we call MaSS, converges for same step sizes as SGD. We prove that MaSS obtains an accelerated convergence rates over SGD for any mini-batch size in the linear setting. For full batch, the convergence rate of MaSS matches the well-known accelerated rate of the Nesterov's method. We also analyze the practically important question of the dependence of the convergence rate and optimal hyper-parameters on the mini-batch size, demonstrating three distinct regimes: linear scaling, diminishing returns and saturation. Experimental evaluation of MaSS for several standard architectures of deep networks, including ResNet and convolutional networks, shows improved performance over SGD, Nesterov SGD and Adam. Many modern neural networks and other machine learning models are over-parametrized. These models are typically trained to have near zero training loss, known as interpolation and often have strong generalization performance, as indicated by a range of empirical evidence including (23; 3). Due to a key property of interpolation -automatic variance reduction (discussed in Section 2.1), stochastic gradient descent (SGD) with constant step size is shown to converge to the optimum of a convex loss function for a wide range of step sizes. Moreover, the optimal choice of step size η * for SGD in that setting can be derived analytically. The goal of this paper is to take a step toward understanding momentum-based SGD in the interpolating setting. Among them, stochastic version of Nesterov's acceleration method (SGD+Nesterov) is arguably the most widely used to train modern machine learning models in practice. The popularity of SGD+Nesterov is tied to the well-known acceleration of the deterministic Nesterov's method over gradient descent. Yet, has not not theoretically clear whether Nesterov SGD accelerates over SGD. As we show in this work, both theoretically and empirically, Nesterov SGD with any parameter selection does not in general provide acceleration over ordinary SGD. Furthermore, Nesterov SGD may diverge, even in the linear setting, for step sizes that guarantee convergence of ordinary SGD. Intuitively, the lack of acceleration stems from the fact that, to ensure convergence, the step size of SGD+Nesterov has to be much smaller than the optimal step size for SGD. This is in contrast to the deterministic Nesterov method, which accelerates using the same step size as optimal gradient descent. As we prove rigorously in this paper, the slow-down of convergence caused by the small step size negates the benefit brought by the momentum term. We note that a similar lack of acceleration for the stochastic Heavy Ball method was analyzed in. To address the non-acceleration of SGD+Nesterov, we introduce an additional compensation term to allow convergence for the same range of step sizes as SGD. The ing algorithm, MaSS (Momentum-added Stochastic Solver) 1 updates the weights w and u using the following rules (with the compensation term underlined): Figure 1: Non-acceleration of Nesterov SGD and fast convergence of MaSS. w t+1 ← u t − η 1∇ f (u t), u t+1 ← (1 + γ)w t+1 − γw t + η 2∇ f (u t). (Here,∇ represents the stochastic gradient. The step size η 1, the momentum parameter γ ∈ and the compensation parameter η 2 are independent of t. We proceed to analyze theoretical convergence properties of MaSS in the interpolated regime. Specifically, we show that in the linear setting MaSS converges exponentially for the same range of step sizes as plain SGD, and the optimal choice of step size for MaSS is exactly η * which is optimal for SGD. Our key theoretical shows that MaSS has accelerated convergence rate over SGD. Furthermore, in the full batch (deterministic) scenario, our analysis selects η 2 = 0, thus reducing MaSS to the classical Nesterov's method. In this case our convergence rate also matches the well-known convergence rate for the Nesterov's method (15; 4). This acceleration is illustrated in Figure 1. Note that SGD+Nesterov (as well as Stochastic Heavy Ball) does not converge faster than SGD, in line with our theoretical analysis. We also prove exponential convergence of MaSS in more general convex setting under additional conditions. We further analyze the dependence of the convergence rate e −s(m)t and optimal hyper-parameters on the mini-batch size m. We identify three distinct regimes of dependence defined by two critical values m * 1 and m * 2: linear scaling, diminishing returns and saturation, as illustrated in Figure 2. The convergence speed per iteration s(m), as well as the optimal hyper-parameters, increase linearly as m in the linear scaling regime, sub-linearly in the diminishing returns regime, and can only increase by a small constant factor in the saturation regime. The critical values m * 1 and m * 2 are derived analytically. We note that the intermediate "diminishing terurns" regime is new and is not found in SGD. To the best of our knowledge, this is the first analysis of mini-batch dependence for accelerated stochastic gradient methods. We also experimentally evaluate MaSS on deep neural networks, which are non-convex. We show that MaSS outperforms SGD, SGD+Nesterov and Adam both in optimization and generalization, on different architectures of deep neural networks including convolutional networks and ResNet. The paper is organized as follows: In section 2, we introduce notations and preliminary . In section 3, we discuss the non-acceleration of SGD+Nesterov. In section 4 we introduce MaSS and analyze its convergence and optimal hyper-parameter selection. In section 5, we analyze the mini-batch MaSS. In Section 6, we show experimental . Over-parameterized models have drawn increasing attention in the literature as many modern machine learning models, especially neural networks, are over-parameterized and show strong generalization performance (16; 23; 2). Over-parameterized models usually in nearly perfect fit (or interpolation) of the training data (23; 18; 3). Exponential convergence of SGD with constant step size under interpolation and its dependence on the batch size is analyzed in. There are a few works that show or indicate the non-acceleration of existing stochastic momentum methods. First of all, the work theoretically proves non-acceleration of stochastic Heavy Ball method (SGD+HB) over SGD on certain synthetic data. Furthermore, these authors provide experimental evidence that SGD+Nesterov also converges at the same rate as SGD on the same data. The work theoretically shows that, for sufficiently small step-sizes, SGD+Nesterov and SGD+HB is equivalent to SGD with a larger step size. However, the in do not exclude the possibility that acceleration is possible when the step size is larger. The work concludes that "momentum hurts the convergence within the neighborhood of global optima", based on a theoretical analysis of SGD+HB. These are consistent with our analysis of the standard SGD+Nesterov. However, this does not apply to all momentum methods. Indeed, we will show that MaSS provably improves convergence over SGD. There is a large body of work, both practical and theoretical, on SGD with momentum, including (10; 8; 1). Adam, and its variant AMSGrad, are among the most practically used SGD methods with momentum. Unlike our method, Adam adaptively adjusts the step size according to a weight-decayed accumulation of gradient history. In the authors proposed an accelerated SGD algorithm, which can be written in the form shown on the right hand side in Eq.8, but with different hyper-parameter selection. Their ASGD algorithm also has a tail-averaging step at the final stage. In the interpolated setting (no additive noise) their analysis yields a convergence rate of O(P oly(κ,κ) exp(−)) for our algorithm with batch size 1. We provide some experimental comparisons between their ASGD algorithm and MaSS in Fig. 4. The work proposes and analyzes another first-order momentum algorithm and derives convergence rates under a different set of conditions -the strong growth condition for the loss function in addition to convexity. As shown in Appendix F.3, on the example of a Gaussian distributed data, the rates obtained in can be slower than those for SGD. In contrast, our algorithm is guaranteed to never have a slower convergence rate than SGD. Furthermore, in the same Gaussian setting MaSS matches the optimal accelerated full-gradient Nesterov rate. Additionally, in our work we consider the practically important dependence of the convergence rate and optimal parameter selection on the mini-batch size, which to the best of our knowledge, has not been analyzed for momentum methods., where f i only depends on a single data point (x i, y i). Let ∇f denote the exact gradient, and∇ m f denote the unbiased stochastic gradient evaluated based on a mini-batch of size m. For simplicity, we also denote∇f (w):=∇ 1 f (w). We use the concepts of strong convexity and smoothness of functions, see definitions in Appendix B.1. For loss function with µ-strong convexity and L-smoothness, the condition number κ is defined as κ = L/µ. In the case of the square loss,, and the Hessian matrix is H:. Let L and µ be the largest and the smallest non-zero eigenvalues of the Hessian respectively. Then the condition number is then κ = L/µ (note that zero eigenvalues can be ignored in our setting, see Section 4). Given a mini-batch size m, we define the m-stochastic condition number as κ m:= L m /µ. Following, we introduce the quantityκ (called statistical condition number in), which is the smallest positive real number such that E x 2 Hence, the quadratic loss function is also L m -smooth, for all m ≥ 1. By the definition of κ m, we also have Remark 2. It is important to note thatκ ≤ κ 1, since E x 2 We consider over-parametrized models that have zero training loss solutions on the training data (e.g.,). A solution f i (w) which fits the training data perfectly f i (w) = 0, ∀i = 1, 2, · · ·, n, is known as interpolating. In the linear setting, interpolation implies that the linear system {x has at least one solution. A key property of interpolation is Automatic Variance Reduction (AVR), where the variance of the stochastic gradient decreases to zero as the weight w approaches the optimal w *. For a detailed discussion of AVR see Appendix B.2. Thanks to AVR, plain SGD with constant step size can be shown to converge exponentially for strongly convex loss functions (13; 19; 14; 12). The set of acceptable step sizes is (0, 2/L m), where L m is defined in Eq.2 and m is the mini-batch size. Moreover, the optimal step size η * (m) of SGD that induces fastest convergence guarantee is proven to be 1/L m. In this section we prove that SGD+Nesterov, with any constant hyper-parameter setting, does not generally improve convergence over optimal SGD. Specifically, we demonstrate a setting where SGD+Nesterov can be proved to have convergence rate of (1 − O(1/κ)) t, which is same (up to a constant factor) as SGD. In contrast, the classical accelerated rate for the deterministic Nesterov's method is We will consider the following two-dimensional data-generating component decoupled model. Fix an arbitrary w * ∈ R 2 and randomly sample z from N. The data points (x, y) are constructed as follow: where e 1, e 2 ∈ R 2 are canonical basis vectors, The following theorem gives a lower bound for the convergence of SGD+Nesterov, regarding the linear regression problem on the component decoupled data model. See Appendix C for the proof. be a dataset generated according to the component decoupled model. Consider the optimization problem of minimizing quadratic function 2. For any step size η > 0 and momentum parameter γ ∈ of SGD+Nesterov with random initialization, with probability one, there exists a T ∈ N such that ∀t > T, where C > 0 is a constant. Compared with the convergence rate (1−1/κ) t of SGD, this theorem shows that SGD+Nesterov does not accelerate over SGD. This is very different from that in the deterministic gradient scenario, where the classical Nesterov's method has a strictly faster convergence guarantee than gradient descent. Intuitively, the key reason for the non-acceleration of SGD+Nesterov is a condition on the step size η required for non-divergence of the algorithm. Specifically, when momentum parameter γ is close to 1, η is required to be less than 2 ) (precise formulation is given in Lemma 1 in Appendix C). The slow-down ing from the small step size necessary to satisfy that condition cannot be compensated by the benefit of the momentum term. In particular, the condition on the step-size of SGD+Nesterov excludes η * that achieves fastest convergence for SGD. We show in the following corollary that, with the step size η *, SGD+Nesterov diverges. This is different from the deterministic scenario, where the Nesterov method accelerates using the same step size as gradient descent. Corollary 1. Consider the same optimization problem as in Theorem 1. Let step-size η = and acceleration parameter γ ∈ [0.6, 1], then SGD+Nesterov, with random initialization, diverges with probability 1. We empirically verify the non-acceleration of SGD+Nesterov as well as Corollary 1, in Section 6 and Appendix F.2. In this section, we propose MaSS, which introduces a compensation term (see Eq.1) onto SGD+Nesterov. We show that MaSS can converge exponentially for all the step sizes that in convergence of SGD, i.e., η ∈ (0, 2/L m). Importantly, we derive a convergence rate exp(−t/ √ κ 1κ), whereκ ≤ κ 1, for MaSS which is faster than the convergence rate for SGD exp(−t/κ 1). Moreover, we give an analytical expression for the optimal hyper-parameter setting. For ease of analysis, we rewrite update rules of MaSS in Eq.1 in the following equivalent form (introducing an additional variable v): There is a bijection between the hyper-parameters (η 1, η 2, γ) and (η, α, δ), which is given by: Remark 3 (SGD+Nesterov). In the literature, the Nesterov's method is sometimes written in a similar form as the R.H.S. of Eq.8. Since SGD+Nesterov has no compensation term, δ has to be fixed as η/α, which is consistent with the parameter setting in. Assumptions. We first assume square loss function, and later extend the analysis to general convex loss functions under additional conditions. For square loss function, the solution set W *:= {w ∈ R d |f (w) = 0} is an affine subspace in the parameter space R d. Given any w, we denote its closest solution as w *:= arg min v∈W * w − v, and define the error = w − w *. Be aware that different w may correspond to different w *, and that and (stochastic) gradients are always perpendicular to W * (see discussion in Appendix B.3). Hence, no actual update happens along W *. For this reason, we can ignore zero eigenvalues of H and restrict our analysis to the span of the eigenvectors of the Hessian with non-zero eigenvalues. Based on the equivalent form of MaSS in Eq.8, the following theorem shows that, for square loss function in the interpolation setting, MaSS is guaranteed to have exponential convergence when hyper-parameters satisfy certain conditions. Theorem 2 (Convergence of MaSS). Consider minimizing a quadratic loss function in the interpolation setting. Let µ be the smallest non-zero eigenvalue of the Hessian matrix H. Let L m be as defined in Eq.2. Denoteκ m:=κ/m + (m − 1)/m. In MaSS with mini batch of size m, if the positive hyper-parameters η, α, δ satisfy the following two conditions: then, after t iterations, Consequently, for some constant C > 0 which depends on the initialization. Remark 4. By condition Eq.10, the admissible step size η is (0, 2/L m), exactly the same as SGD for interpolated setting. Remark 5. One can easily check that the hyper-parameter setting of SGD+Nesterov does not satisfy the conditions in Eq.10. Proof sketch for Theorem 2. Denote F t:= E v t+1 − w * 2 we show that, under the update rules of MaSS in Eq.8, By the condition in Eq.10, c 1 ≤ 0, c 2 ≤ 0, then the last two terms are non-positive. Hence, Using that w t −w * 2 ≤ αF t /δ, we get the final . See detailed proof in Appendix D. Hyper-parameter Selection. From Theorem 2, we observe that the convergence rate is determined by (1−α) t. Therefore, larger α is preferred for faster convergence. Combining the conditions in Eq.10, we have By setting η * = 1/L m, which maximizes the right hand side of the inequality, we obtain the optimal selection α * = 1/ √ κ mκm. Note that this setting of η * and α * determines a unique δ * = α * /µ by the conditions in Eq.10. In summary, By Eq.9, the optimal selection of (η 1, η 2, γ) would be: κ m is usually larger than 1, which implies that the coefficient η * 2 of the compensation term is non-negative. The non-negative coefficient η 2 indicates that the weight u t is "over-descended" in SGD+Nesterov and needs to be compensated along the gradient direction. It is important to note that the optimal step size for MaSS as in Eq.13 is exactly the same as the optimal one for SGD. With such hyper-parameter selection given in Eq.14, we have the following theorem for optimal convergence: Theorem 3 (Acceleration of MaSS). Under the same assumptions as in Theorem 2, if we set hyperparameters in MaSS as in Eq.13, then after t iteration of MaSS with mini batch of size m, for some constant C > 0 which depends on the initialization. Remark 6. With the optimal hyper-parameters in Eq.13, the asymptotic convergence rate of MaSS is which is faster than the rate O(e −t/κm) of SGD (see ), since κ m ≥κ m. Remark 7 (MaSS Reduces to the Nesterov's method for full batch). In the limit of full batch m → ∞, we have κ m → κ,κ m → 1, the optimal parameter selection in Eq.14 reduces to It is interesting to observe that, in the full batch (deterministic) scenario, the compensation term vanishes and η * 1 and γ * are the same as those in Nesterov's method. Hence MaSS with the optimal hyper-parameter selection reduces to Nesterov's method in the limit of full batch. Moreover, the convergence rate in Theorem 3 reduces to O(e −t/ √ κ), which is exactly the well-known convergence rate of Nesterov's method (15; 4). Extension to Convex Case. First, we extend the definition of L 1 to convex functions,, for some > 0. In MaSS, if the hyper-parameters are set to be: then after t iterations, there exists a constant t. Based on our analysis, we discuss the effect of selection of mini-batch size m. We show that the domain of mini-batch size m can be partitioned into three intervals by two critical points: The three intervals/regimes are depicted in Figure 2, and the detailed analysis is in Appendix G. The optimal selection of hyper-parameters is approximated by: and the convergence rate in Eq. In the linear scaling regime, the hyper-parameter selections follow a Linear Scaling Rule (LSR): When the mini-batch size is multiplied by k, multiply all hyperparameters (η, α, δ) by k. This parallels the linear scaling rule for SGD which is an accepted practice for training neural networks. This three regimes partition is different from that for SGD, where only linear scaling and saturation regimes present. An empirical verification of the dependence of the convergence speed on m is shown in Figure 3. See the setup in Appendix G. Synthetic Data. We empirically verify the non-acceleration of SGD+Nesterov and the fast convergence of MaSS on synthetic data. Specifically, we optimize the quadratic function is generated by the component decoupled model described in Section 3. We compare the convergence behavior of SGD+Nesterov with SGD, as well as our proposed method, MaSS, and several other methods: SGD+HB, ASGD. We select the best hyper-parameters from dense grid search for SGD+Nesterov (step-size and momentum parameter), SGD+HB (step-size and momentum parameter) and SGD (step-size). For MaSS, we do not tune the hyper-parameters but use the hyper-parameter setting suggested by our theoretical analysis in Section 4; For ASGD, we use the setting provided by. Hyperparameters: we use optimal parameters for SGD+Nesterov, SGD+HB and SGD; the setting in for ASGD; Eq. 13 for MaSS. 9. We observe that the fastest convergence of SGD+Nesterov is almost identical to that of SGD, indicating the non-acceleration of SGD+Nesterov. We also observe that our proposed method, MaSS, clearly outperforms the others. In Appendix F.2, we provide additional experiments on more settings of the component decoupled data, and Gaussian distributed data. We also show the divergence of SGD+Nesterov with the same step size as SGD and MaSS in Appendix F.2. Real data: MNIST and CIFAR-10. We compare the optimization performance of SGD, SGD+Nesterov and MaSS on the following tasks: classification of MNIST with a fullyconnected network (FCN), classification of CIFAR-10 with a convolutional neural network (CNN) and Gaussian kernel regression on MNIST. See detailed description of the architectures in Appendix H.1. In all the tasks and for all the algorithms, we select the best hyper-parameter setting over dense grid search, except that we fix the momentum parameter γ = 0.9 for both SGD+Nesterov and MaSS, which is typically used in practice. All algorithms are implemented with mini batches of size 64 for neural network training. Test Performance. We show that the solutions found by MaSS have good generalization performance. We evaluate the classification accuracy of MaSS, and compare with SGD, SGD+Nesterov and Adam, on different modern neural networks: CNN and ResNet. See description of the architectures in Appendix H.1. In the training processes, we follow the standard protocol of data augmentation and reduction of learning rate, which are typically used to achieve state-of-the-art in neural networks. In each task, we use the same initial learning rate for MaSS, SGD and SGD+Nesterov, and run the same number of epochs (150 epochs for CNN and 300 epochs for ResNet-32). Detailed experimental settings are deferred to Appendix H.2. Average of 3 runs that converge. Some runs diverge. † Adam uses initial step size 0.001. Table 6 compares the classification accuracy of these algorithms on the test set of CIFAR-10 (average of 3 independent runs). We observe that MaSS produces the best test performance. We also note that increasing initial learning rate may improves performance of MaSS and SGD, but degrades that of SGD+Nesterov. Moreover, in our experiment, SGD+Nesterov with large step size η = 0.3 diverges in 5 out of 8 runs on CNN and 2 out of 5 runs on ResNet-32 (for random initialization), while MaSS and SGD converge on every run. Step-size η 1, secondary step-size η 2, acceleration parameter γ ∈.. end while Output: weight w t. Note that the proposed algorithm initializes the variables w 0 and u 0 with the same vector, which could be randomly generated. As discussed in section 4, MaSS can be equivalently implemented using the following update rules: In this case, variables u 0, v 0 and w 0 should be initialized with the same vector. There is a bijection between the hyper-parameters (η 1, η 2, γ) and (η, α, δ), which is given by: B ADDITIONAL PRELIMINARIES ). A differentiable function f: R d → R is µ-strongly convex (µ > 0), if f (x) ≥ f (z) + ∇f (z), x − z + µ 2 x − z 2, ∀x, z ∈ R d. Definition 2 (Smoothness). A differentiable function f: R d → R is L-smooth (L > 0), if f (x) ≤ f (z) + ∇f (z), x − z + L 2 x − z 2, ∀x, z ∈ R d. In the interpolation setting, one can write the square loss as A key property of interpolation is that the variance of the stochastic gradient of decreases to zero as the weight w approaches an optimal solution w *. Proposition 1 (Automatic Variance Reduction). For the square loss function f in the interpolation setting, the stochastic gradient at an arbitrary point w can be written as Moreover, the variance of the stochastic gradient Since E[(H m − H) 2 ] is independent of w, the above proposition unveils a linear dependence of variance of stochastic gradient on the norm square of error. This observation underlies exponential convergence of SGD in certain convex settings (20; 13; 19; 14; 12). Consider the square loss function, f (w) = 1 2n where the stochastic gradient is computed based on a randomly sampled batch of size m. Recall that the solution set W *:= {w ∈ R d |f (w) = 0} is an affine subspace in the parameter space, and that w * is the solution in W * that is closest to w. Hence, w − w * is perpendicular to W *, i.e., w − w Hence the (stochastic) gradient is perpendicular to W *. The key proof technique is to consider the asymptotic behavior of SGD+Nesterov in the decoupled model of data when the condition number becomes large. Notations and proof setup. Recall that the square loss function based on the component decoupled data D, define in Eq.6, is in the interpolation regime, then for SGD+Nesterov, we have the recurrence relation It is important to note that each component of w t evolves independently, due to the fact thatH is diagonal. With:= w − w *, we define for each component j = 1, 2 that where [j] is the j-th component of vector. The recurrence relation in Eq.26 can be rewritten as with For the ease of analysis, we define u:= 1 − γ ∈ and t j:= ησ 2 j, j = 1, 2. Without loss of generality, we assume σ 2 1 = 1 in this section. In this case, t 1 = η and t 2 = η/κ, where κ is the condition number. [j]:, where Proof idea. For the two-dimensional component decoupled data, we have By definition of Φ in Eq.27, we can see that the convergence rate is lower bounded by the convergence rates of the sequences {Φ [j] t } t. By the relation Eq.28, we have that the convergence rate of the sequence {Φ t} t is controlled by the magnitude of the top eigenvalue λ max of B, if Φ t has nonzero component along the eigenvector of B with eigenvalue λ max (B). Specifically, if |λ max | > 1, Φ [j] t grows at a rate of |λ max | t, indicating the divergence of SGD+Nesterov; if |λ max | < 1, then converges at a rate of |λ max | t. In the following, We use the eigen-systems of matrices B [j], especially the top eigenvalue, to analyze the convergence behavior of SGD+Nesterov with any hyper-parameter setting. We show that, for any choice of hyper-parameters (i.e., step-size and momentum parameter), at least one of the following statements must holds: • B has an eigenvalue larger than 1. • B has an eigenvalue of magnitude This is formalized in the following two lemmas. Lemma 1. For any u ∈, if step size then, B has an eigenvalue larger than 1. We will analyze the dependence of the eigenvalues on κ, when κ is large to obtain Lemma 2. For any u ∈, if step size then, B has an eigenvalue of magnitude 1 − O (1/κ). Finally, we show that Φ t has non-zero component along the eigenvector of B with eigenvalue λ max, hence the convergence of SGD+Nesterov is controlled by the eigenvalue of B [j] with the largest magnitude. Lemma 3. Assume SGD+Nesterov is initialized with w 0 such that both components w 0 − w *, e 1 and w 0 − w *, e 2 are non-zero. Then, for all t > 2, Φ [j] t has a non-zero component in the eigen direction of B [j] that corresponds to the eigenvalue with largest magnitude. Remark 8. When w is randomly initialized, the conditions w 0 −w *, e 1 = 0 and w 0 −w *, e 2 = 0 are satisfied with probability 1, since complementary cases form a lower dimensional manifold which has measure 0. By combining Lemma 1, 2 and 3, we have that SGD+Nesterov either diverges or converges at a rate of (1 − O(1/κ)) t, and hence, we conclude the non-acceleration of SGD+Nesterov. In addition, Corollary 1 is a special case of Theorem 1 and is proven by combining Lemma 1 and 3. In high level, the proof ideas of Lemma 1 and 3 is analogous to those of, which proves the non-acceleration of stochastic Heavy Ball method over SGD. But the proof idea of Lemma 2 is unique. Proof of Lemma 1. The characteristic polynomial of B j, j = 1, 2, are: First note that lim λ→∞ D j (λ) = +∞ > 0. In order to show B 1 has an eigenvalue larger than 1, it suffices to verity that Replacing γ by 1 − u and ησ 2 1 by t 1, we have Solving for the inequality D 1 < 0, we have, for positive step size η. Proof of Lemma 2. We will show that at least one of the eigenvalues of B is 1 − O(1/κ), under the condition in Eq.32. First, we note that t 2 = t 1 /κ = η/κ, which is O(1/κ). We consider the following cases separately: 2 ) and o; and 5) u is Θ, the last of which includes the case where momentum parameter is a constant. Note that, for cases 1-4, u is o. In such cases, the step size condition Eq.32 can be written as It is interesting to note that η must be o to not diverge, when u is o. This is very different to SGD where a constant step size can in convergence, see. 2 ). In this case, the terms u 6, u 4 t 2, u 2 t 2 2 and t 3 2 are of the same order. We find that 2 ). Hence, Write t 2 = cu 2 asymptotically for some constant c. If 4t 2 ≥ u 2, i.e., 4c − 1 ≥ 0, then If 4t 2 ≤ u 2, i.e., 4c − 1 ≤ 0, then In either case, the first-order term is of order u. Recall that t 2 = η/κ, then we have Case 2: u is o(t 0.5 2) and ω(t 2). In this case, ). Hence, 2 ). This implies that t 0.5 2 = o(1/κ) and u = o(1/κ). Therefore, all the eigenvalues λ i are of order 1 − O(1/κ). Case 3: u is O(t 2). This case if forbidden by the assumption of this lemma. This is because, κ is o ). This is contradictory to u is O(t 2). Case 4: u is Ω(t 0.5 2) and o. In this case, we first consider the terms independent of t 2, i.e., constant term and u-only terms. These terms can be obtained by putting t 2 = 0. In such a setting, the eigenvalues are simplified as: Note that the u-only terms cancel in λ 2, so the first order after the constant term must be of t 2 (could be t 2 /u 2, t 2 /u etc.). In the following we are going to analyze the t 2 terms. Since u is Ω(t 0.5 2), u 2 has lower order than t 2, and t 2 /u 2 is o. This allows us to do Taylor expansion: where f (u) and g(u) are u-terms only, which, by the above analysis in Eq.37, are shown to contribute nothing to λ 2. Hence, we use the first terms of T 1 and T 2 above to analyze the first order term of λ 2. Plugging in these term to the expression of λ 2, and keeping the lowest order of t 2, we find a zero coefficient of the lowest order t 2 -term: Hence, λ 2 can be written as: where c is the coefficient. On the other hand, by Eq.35 and t 2 = η/κ, we have Therefore, we can write Eq.38 as λ 2 = 1 − O(1/κ).. This is the case where the momentum parameter is κ-independent. Using the same argument as in case 4, we have zero u-only terms. Then, directly taking Taylor expansion with respect to t 2 in: Proof of Lemma 3. This proof follows the idea of the proof for stochastic Heavy Ball method in. The idea is to examine the subspace spanned by Φ t, t = 0, 1, 2, · · ·, j = 1, 2, and to prove that the eigenvector of B [j] corresponding to top eigenvalue (i.e., eigenvalue with largest magnitude) is not orthogonal to this spanned subspace. This in turn implies that there exists a non-zero component of Φ [j] t in the eigen direction of B [j] corresponding to top eigenvalue, and this decays/grows at a rate of λ t max (B [j] ). [j] t has non-zero component in the eigen direction of B [j] with top eigenvalue, then Φ [j] t should also have non-zero component in the same direction. Thus, it suffices to show that at least one of Φ 3 has non-zero component in the eigen direction with top eigenvalue. SinceH is diagonal for this two-dimensional decoupled data, w and w evolves independently, and we can analyze each component separately. In addition, it can be seen that each of the initial values w, which is non-zero by the assumption of this lemma, just acts as a scale factor during the training. Hence, without loss of generality, we can assume w 0 − (w *) [j] = 1, for each j. Then, according to the recurrence relation of SGD+Nesterov in Eq.26, where s [j] Denote the vectorized form of Φ t ), which is a 4 × 1 column vector. We stack the vectorized forms of Φ 3 to make a 4 × 4 matrix, denoted as M [j]: Note that Φ [j] t, t = 0, 1, 2, 3, are symmetric tensors, which implies that M [j] contains two identical rows. Specifically, the second and third row of M [j] are identical. Therefore, the vector v T is an eigenvector of M [j] with eigenvalue 0. In fact, v is also an eigenvector of B [j] with eigenvalue γ(1 − ησ Hence, v is not the eigenvector along top eigenvalue, and therefore, is orthogonal to the eigen space with top eigenvalue. In order to prove at least one of Φ t, t = 0, 1, 2, 3, has a non-zero component along the eigen direction of top eigenvalue, it suffices to verify that M [j] is rank 3, i.e., spans a three-dimensional space. Equivalently, we consider the following matrix where we omitted the superscript [j] for simplicity of the expression. If the determinant of M [j] is not zero, then it is full rank, and hence M [j] spans a three-dimensional space. Plug in the expressions in Eq.41, then we have where t j = ησ We note that, for all u ∈ both t j are not positive. This means that, for all u and positive t j, the determinant det(M [j] ) can never be zero. Therefore, for each j = 1, 2, M [j] is full rank, and M [j] spans a three-dimensional space, which includes the eigenvector with the top eigenvalue of B [j]. Hence, at least one of Φ t, t ∈ {0, 1, 2, 3} has non-zero component in the eigen direction with top eigenvalue. By Φ t also have non-zero component in the eigen direction with top eigenvalue of B [j]. Proof of Theorem 1. Lemma 1 and 2 show that, for any hyper-parameter setting (η, γ) with η > 0 and γ ∈, either top eigenvalue of B is larger than 1 or top eigenvalue of B is 1 − O(1/κ). Hence, |λ max | is either greater than 1 or is 1 − O(1/κ). Lemma 3 shows that Φ t has non-zero component along the eigenvector of B with eigenvalue λ max (B). By Eq.28 and Lemma 1 and 2, the sequence {Φ t} t either diverges or converges at a rate of t. By definition of Φ in Eq.27, we have that {t} t either diverges or converges at a rate of (1 − O(1/κ)) t. Note that, for the two-dimensional component decoupled data, we have Therefore, the convergence rate of SGD+Nesterov is lower bounded by (1−O(1/κ)) t. Note that the convergence rate of SGD on this data is (1 − O(1/κ)) t, hence SGD+Nesterov does not accelerate SGD on the two-dimensional component decoupled dataset. Proof of Corollary 1. When η = 1/L 1 and γ ∈ [0.6, 1], the condition in Eq.31 is satisfied. By Lemma 1, the top eigenvalue λ max of B is larger than 1. By Lemma 3, Φ has non-zero component along the eigenvector with this top eigenvalue λ max. Hence, | w t − w *, e 1 | grows at a rate of We first give a lemma that is useful for dealing with the mini-batch scenario: Lemma 4. If square loss f is in the interpolated setting, i.e., there exists w * such that f (w Proof. onto Eq.53 and add it to Eq.52, then If the hyper-parameters are selected as: then the last two terms in Eq.54 are non-positive. Hence, which implies Since f (w t) ≤ L/2 · w t − w * 2 (by smoothness), then we have the final with C being a constant. Fix an arbitrary w * ∈ R 2 and let z be randomly drawn from the zero-mean Gaussian distribution with variance E[z 2] = 2, i.e. z ∼ N. The data points (x, y) ∈ D are constructed as follow: where e 1, e 2 ∈ R 2 are canonical basis vectors, σ 1 > σ 2 > 0. Note that the corresponding square loss function on D is in the interpolation regime, since f (w *) = 0. The Hessian and stochastic Hessian matrices turn out to be Note thatH is diagonal, which implies that stochastic gradient based algorithms applied on this data evolve independently in each coordinate. This allows a simplified directional analysis of the algorithms applied. Here we list some useful for our analysis. The fourth-moment of Gaussian variable Gaussian Data. Suppose the data feature vectors {x i} are zero-mean Gaussian distributed, and y i = w *, x i, ∀i, where w * is fixed but unknown. Then, by the fact that for zero-mean Gaussian random variables z 1, z 2, z 3 and z 4, we have In this subsection, we show additional empirical verification for the fast convergence of MaSS, as well as the non-acceleration of SGD+Nesterov, on synthetic data. In addition, we show the divergence of SGD+Nesterov when using the same step size as SGD and MaSS, as indicated by Corollary 1. We consider two families of synthetic datasets: • Component decoupled: (as defined in Section 3). Fix an arbitrary w * ∈ R 2 with all components non-zero. x i is drawn from N (0, diag(2σ 2)) with probability 0.5 each. y i = w *, x i for all i. • 3-d Gaussian: Fix an arbitrary w * ∈ R 3 with all components non-zero. x i are independently drawn from N (0, diag(σ Step size: η * = 1/L 1 = 1/6, and momentum parameter: γ is 0.9 or 0.99. performed on either 3-d Gaussian or component decoupled data with fixed σ 1 and σ 2 . For each setting of (σ 1, σ 2), we randomly select w *, and generate 2000 samples for the dataset. Batch sizes for all algorithms are set to be 1. We report the performances of SGD, SGD+Nesterov and SGD+HB using their best hyper-parameter setting selected from dense grid search. On the other hand, we do not tune hyper-parameters of MaSS, but use the suggested setting by our theoretical analysis, Eq. 14. Specifically, we use Component decoupled: 3-d Gaussian: For ASGD, we use the setting suggested by. Figure 6 (in addition to Fig 4) and Figure 7 show the curves of the compared algorithms under various data settings. We observe that: 1) SGD+Nesterov with its best hyper-parameters is almost identical to the optimal SGD; 2) MaSS, with the suggested hyper-parameter selections, converges faster than all of the other algorithms, especially SGD. These observations are consistent with our theoretical about non-acceleration of SGD+Nesterov, Theorem 1, and accelerated convergence of MaSS, Theorem 3. Recall that MaSS differs from SGD+Nesterov by only a compensation term, this experiment illustrates the importance of this term. Note that the vertical axis is log scaled. Then the linear decrease of log losses in the plots implies an exponential loss decrease, and the slopes correspond to the coefficients in the exponents. Divergence of SGD+Nesterov with large step size. As discussed in Corollary 1, SGD+Nesterov diverges with step size η * = 1/L 1 (when γ ∈ [0.6, 1]), which is the optimal choice of step size for both SGD and MaSS. We run SGD+Nesterov, with step size η * = 1/L 1, to optimize the square loss function on component decoupled data mentioned above. Figure 8 shows the divergence of SGD+Nesterov with two common choices of momentum parameter γ: 0.9 and 0.99. with the parameter ρ on the loss function they prove convergence rate 1 − 1/ρ 2 κ t of their method (called SGD with Nesterov acceleration in their paper), where t is the iteration number. In the following, we show that, on a simple (zero-mean) Gaussian distributed data, this rate is slower than that of SGD, which has a rate of (1 − 1/κ) t. On the other hand, MaSS achieves the accelerated rate 1 − 1/ (2 + d)κ t. We empirically verify the three-regime partition observed in section 5 using zero-mean Gaussian data. In this evaluation, we set the covariant matrix of the (zero-mean) Gaussian to be: In the experiments, we run MaSS with a variaty of mini-batch size m, ranging from 1 to 160, on this Gaussian dataset. For each training process, we compute the convergence speed s(m), which is defined to be the inverse of the number of iterations needed to achieve a training error of ε. Fully-connected Network. The fully-connected neural network has 3 hidden layers, with 100 ReLU-activated neurons in each layer. After each hidden layer, there is a dropout layer with keep probability 0.5. This network takes 784-dimensional vectors as input, and has 10 softmax-activated output neurons. It has ≈99k trainable parameters in total. Convolutional Neural Network (CNN). The CNN we considered has three convolutional layers with kernel size of 5 × 5 and without padding. The first two convolutional layers have 64 channels each, while the last one has 128 channels. Each convolutional layer is followed by a 2 × 2 max pooling layer with stride of 2. On top of the last max pooling layer, there is a fully-connected ReLU-activated layer of size 128 followed by the output layer of size 10 with softmax non-linearity. A dropout layer with keep probability 0.5 is applied after the full-connected layer. The CNN has ≈576k trainable parameters in total. Residual Network (ResNet). We train a ResNet with 32 convolutional layers. The ResNet-32 has a sequence of 15 residual blocks: the first 5 blocks have an output of shape 32 × 32 × 16, the following 5 blocks have an output of shape 16×16×32 and the last 5 blocks have an output of shape 8×8×64. On top of these blocks, there is a 2×2 average pooling layer with stride of 2, followed by a output layer of size 10 with softmax non-linearity. The ResNet-32 has ≈467k trainable parameters in total. We use the fully-connected network to classify the MNIST dataset, and use CNN and ResNet to classify the CIFAR-10 dataset.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work proves the non-acceleration of Nesterov SGD with any hyper-parameters, and proposes new algorithm which provably accelerates SGD in the over-parameterized setting.

Represent the TLDR for finding the corresponding scientific article

We investigate the convergence of popular optimization algorithms like Adam , RMSProp and propose new variants of these methods which provably converge to optimal solution in convex  settings.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: We propose the Fixed Grouping Layer (FGL); a novel feedforward layer designed to incorporate the inductive bias of structured smoothness into a deep learning model. FGL achieves this goal by connecting nodes across layers based on spatial similarity. The use of structured smoothness, as implemented by FGL, is motivated by applications to structured spatial data, which is, in turn, motivated by domain knowledge. The proposed model architecture outperforms conventional neural network architectures across a variety of simulated and real datasets with structured smoothness. The effectiveness of predictive models often depends on the choice of inductive bias, and the extent to which this inductive bias captures real-world structure. For instance, one example of such bias encoding leading to improved performance is convolution. In principle, convolutional weights could be learned directly from data. However, in practice, imposing this structure leads to improved performance when compared to fully connected models, and as a , Convolutional neural networks (CNNs) have enjoyed wide use for computer vision tasks . Similarly, recurrent neural networks such as LSTMs are effective for text , and certain graphical models are ideal for sentence segmentation and labeling . Our work follows this philosophy. Specifically, we propose a feedforward layer for deep neural networks that is suitable for neuroimaging and potentially useful for other data where variables can be grouped due to underlying structure. Data with multiple input variables often exhibit some structure. For example, the El Nino dataset consists of measurements by weather buoys in the ocean, and one expects that nearby buoys can be grouped together. Similarly, socio-economic data can often be grouped together by geographic proximity. Financial market data of individual stocks can be grouped together based on the industrial sector to which a company belongs. Along similar lines, brain parcellations are a well studied paradigm for capturing the structure of brain activity, often via statistical parcellation based on ward clustering . The of ward clustering is a tree where leaf nodes represent voxels of the brain and interior nodes represent grouping of voxels into spatial clusters. Figure 1 visualizes the output of ward clustering at various granularities when applied to the human connectome project resting state brain data . Contribitions: Our primary technical contribution is the Fixed Grouping Layer (FGL). FGL is designed to extract features within each group, and additionally guarantees that each output vector is only affected by the input vectors related to it by the specified grouping. We demonstrate the benefit of using FGL on simulated experiments and real neuroimaging data. We compare FGL against fully connected networks, convolutional neural networks, CoordConv , and a closely related method proposed by. We extensively evaluate the performance of FGL on simulated and real brain imaging data showing improved performance. Functional Magnetic Resonance Imaging (fMRI) is a popular brain imaging technique which measures a physiological correlate of neuron activity . The brain imaging scans are generally of two kinds: resting state and task data. Resting state data (rfMRI) is collected while the subject is at rest, i.e., while the subject is not actively engaged in a task. Task data (tfMRI) is collected while the subject is engaged in a predefined task, for example, a motor task such as moving their fingers. fMRI data can be represented as 3-dimensional images, and have rich structure that has been studied extensively, including in the ML literature . Importantly, coarse correspondences have been discovered between brain regions and specific functions or behavior We particularly focus on brain decoding -a standard task in fMRI brain data analysis where the brain image is used to predict the associated task or stimulus. Broadly, there are two types of brain decoding models: end to end models and models which perform dimensionality reduction followed by a low-dimensional prediction. On one hand, dimension reduction directly captures the notion of grouping variables together. On the other hand, end to end models rarely employ brain spatial structure. This observation motivates our work. In recent years, decoding from fMRI studies has been attempted using a variety of methods: use factored logistic regression while use convolutional neural networks. use a factored model after performing dimensionality reduction. Inspired by similar motivations, construct a regularizer using feature groupings obtained from a fast clustering method. They demonstrate that such a regularizer outperforms dropout and L 2 regularization. However, they do not consider employing this structure in a deep model, opting instead for a wide shallow approach. Compared to , our illustrate the benefits of depth combined with spatial sparsity for brain image decoding. Next, we formally define our idea of groups. Given a set of input variables X = {x i : 0 ≤ i < n in, i ∈ Z}, a grouping of variables, denoted by G is a subset of the power-set of X such that each x i is in at least one set in G. That is, G ⊂ 2 X, such that, ∀x i ∈ X: x i ∈ G. Each set in G is a group of variables. For example, in the case of a colored image, each pixel can be considered a variable with an associated feature vector of length 3, i.e., each x i represents a pixel and A spatially smooth grouping of these variables corresponds to a segmentation of the image. Optionally, the groups can be mutually exclusive. An FGL layer takes as input n in vectors of c in length each, where the n in vectors are grouped into n out groups. Further, let c out be the length of the vector associated with each group. Note that these groups do not have to be mutually exclusive, but mutually exclusive groups offer benefits that we describe in the supplementary. The model architecture is allowed to use multiple channels (analogous to the # of filters of standard convolutional networks). c in and c out are the number of input and output channels. Mathematically, the Fixed Grouping Layer is given by: nout,cout is the matrix representing the output with each row represents one group. x ∈ R nin,cin is a matrix representing the input -one row for each input vector. A is a binary matrix that represents the grouping -A j,i = 1 if and only if x i is in group j. represents the Hadamard product (elementwise multiplication) . u, v, b are parameters of the model. v is used for a linear transformation from R cin to R cout, i.e., v ∈ R cin,cout. u is a matrix of size n in × c out. b the bias, is a matrix of size n out × c out We denote the i th input vector, i.e., the i th row of x, by x i. Observe that FGL is a fully connected layer when there is only one group which contains all variables. We construct a deep neural network for classification using repeated layers of FGL (and activation functions), followed by either a fully connected network or an FGL that groups all inputs into a single group. This is inspired from traditional CNN based classification models. We provide a visualization of a simplified model in Figure S2. Regularization: We use weight normalization , which is a reparameterization of the weights of a neural network that decouples the norm and the direction of the weights. That is, for a single dimension of a fully connected layer, weight normalization reparameterizes w as w = g(θ/||θ||), where θ is a vector of the same length as w and g is a scalar. The network now learns g, θ instead of w. For FGL we apply weight norm on both u and v. Weight norm is applied by treating u as c out different vectors and v as the weights of a fully connected layer. We use Pytorch to implement models, and nilearn to preprocess and visualize fMRI images. Training was done using Adam on 4 K80 GPUs. Code is available at https://www.github.com/anon/repo and a minimal version is provided in the supplementary. We consider brain decoding as a classification task, and use two common types of models as baselines: fully connected networks and convolution based models such as standard Convolutional Neural Networks (CNNs) and their CoordConv variant. To the best of our knowledge, our baselines include state of the art approaches on the considered datasets. Multinomial logistic regression is a standard model (a), given by: and bias b ∈ R k where k is the number of possible labels. Here,ŷ is the vector of predicted probabilities for each class or label. Clearly multinomial logistic regression by itself uses no spatial information. Feedforward Neural Networks (FNN): FNNs are ideal for tasks where the input has neither a grid-like structure nor a sequential structure. The architecture is an alternating sequence of linear transformations and activation functions . Convolutional Neural Networks (CNNs): , are a popular tool in deep learning. Many problems, where the inputs have a spatial representation, lend themselves to convolution. CNNs are popular not only for their flexibility but also because of the assumptions they make about the nature of the input -one of them being that dependencies between pixels are local and spatially invariant. These assumptions are usually appropriate for natural images. However, in the case of brain fMRI, since features are also dependent on position, i.e., features are not position invariant, CNNs might not work as well. demonstrate that CNNs are sometimes unable to capture locationdependent spatial representations. e.g. transforming a pair of coordinates to a one-hot representation. One reason for this failure could be that the coordinate transformation problem directly conflicts with the underlying assumptions of CNNs. They propose the CoordConv layer as a solution. CoordConv is essentially a convolutional layer except that it includes coordinates as additional input channels. Thus, CoordConv enables CNNs where local dependencies change across the image. First, we discuss grouping via Voronoi Diagrams. Given a set of points, called sites, a Voronoi diagram is the division of a plane into regions based on distance to the sites. Usually, positions whose closest site is the same are grouped together. Consider the grouping induced by a set of m sites P = {p i : 2, 0 ≤ i < m} for some s indicating the size of the plane: We use Voronoi diagrams because they create regions which are spatially connected. To increase the complexity of the task, we use groupings which are unions of arbitrarily chosen Voronoi regionsing in groups comprised of multiple spatially connected regions which may not be connected to each other. We provide an example in Figure 2b. Consider input data sampled from a Gaussian prior: x ∼ N (0, S), where 0 is a zero-vector and S is an arbitrary covariance matrix of appropriate size. We let x ∈ R s 2 for an integer s -the idea being that x is a flattened version of a grayscale image of size s × s. Next, suppose that datapoints are labelled based on a linear function. That is, z|x ∼ N (F x, Σ), for a fixed covariance matrix Σ, and a matrix F of size k × s 2 where k is the number of labels. The label, y, is assigned based on z: We briefly analyze this simple generative model in the context of FGL. Using conjugate priors (b), it is straightforward to show that, To explore the implications of equation, consider an F that is sparse such that the non-zero positions in each row of F correspond to a segmentation of the input image (a grouping of pixels). For example, circular patches on the image, or in our case, Voronoi diagrams. Additionally, if Σ is an identity matrix, then each dimension of z corresponds to the sum of values of a group of pixels. We create a dataset which samples x from the the Gaussian prior with S being an identity matrix. We use s = 128 so that each x can be interpreted as a square image. We create F by first creating a Voronoi diagram of 512 randomly selected points, and then merging these regions into k = 32 groups. We sample z from the the corresponding conditional distribution. We fix a random W and then assign the label y with highest likelihood to the datapoint x. We sample 50000 points to create the simulated dataset. A visualization of the process is provided in Figure 2a. To ensure that the dataset wasn't too noisy, we plot a histogram of probability of assigned label in Figure 2c. The Figure 3: (a) Test accuracy (with error bars) on held out 20% of simulated dataset vs. fraction of data used for training. The graph indicates that FGL has better empirical sample complexity. The small magnitude of error in estimation of performance indicates that models are well trained and the difference is due to the models themselves. (b) Minimum (across classes) F1 Score on held out test set vs. fraction of data used for training. The difference in performance is not due to performance on a single class/region, but rather across all labels. (c) Histogram of ground truth probability of labels for points where FGL is correct but CNN misclassifies. This demonstrates that the misclassification by CNN is not only on noisy datapoints but also for datapoints where the label should be clear. histogram shows that only a small number of datapoints are noisy -in most cases, the assigned label has a probability of at least 0.5. To demonstrate the benefit of using the Voronoi Diagram during classification, we train 4 modelsLogistic Regression (LR), a Convolutional Neural Network (Conv), a CoordConv variant (CC) of the same CNN, and a model using our proposed layer -FGL followed by a fully connected network. Our FGL model is provided the voronoi regions. The number of parameters in each model is roughly the same. Since the dataset uses labels that are linear in terms of x, we use no non-linear activations in any of our models. We found that using maxpooling in the CNN and CoordConv hurt performance. We don't report with an FNN because it performs similar to LR. We create a test set using 20% of the simulated dataset. The remaining points are used for training. For each model, we train using various quantities of available data, and test on the held out set. The are aggregated over 10 runs -with a randomly sampled test set for each run. A plot of the test accuracy vs. fraction of data used for training is given in Figure 3. We find that the standard deviation of accuracies of these models is small -indicating that the failures are not due to poor initialization or poor training but rather a difference in models. This experiment was designed to demonstrate a failure of convolution based models and also fully connected methods. Although this satisfies our intuition that using spatial structure should help drastically improve performance, we investigate the datapoints at which the CNN failed but FGL did not. The first thing to check was the probability of assigned labels for these points -a histogram of the same for a random subset of the testing set is provided in Figure 3c. The next sanity check is to ensure that the drop in performance isn't just for one set of regions or one class. To that end, we plot the lowest F1 score (lowest across classes) in Figure 3b. We see the same trend -FGL performs better than CNNs, CoordConv and Logistic Regression. These plots indicate the validity of the gain in performance -it is due to neither noisy labels, nor failure on any one label. Hence, using a grouping of variables seems to provide a significant benefit. We evaluate our models on 5 datasets which were used by: Archi , Brainomics , Cam-CAN , LA5c , an aggregation of fMRI datasets. Our only required preprocessing for the task contrast data was to upsample Cam-CAN and Brainomics to the same resolution as HCP. The assessment of fMRI decoding is an important topic in its own right due to unique spatial characteristics and typical sample sizes. show that leave-one-out strategies for cross validation can be unstable, and suggest that using reasonable defaults is a good strategy. Additionally, it is well known that having common subjects between train and test datasets can lead to misleading . This is because such a test set does not measure how well a model can generalize from one subject to another. Hence, we evaluate models via out-of-sample accuracy, i.e., we hold out some subjects (30%) for the testing dataset in each run. Further, we train all models with reasonable defaults that we found were not critical for model performance. showed that ward clustering provides good parcellations of the brain, we perform ward clustering on a fraction of HCP resting state data (total size of 4TB). We downsample the resting state time series data to about 1% of the original frequency, and use the brain activations at each position as the feature vectors for clustering. The downsampling is needed due to hardware constraints. We did not use task datasets for clustering since we would have to hold out more data for clustering -exacerbating data scarcity. Additionally, resting state data is more easily acquired , and there are strong correlations between tfMRI and rfMRI . Thus, using rfMRI should provide a good if not better parcellation of the brain. To make a deep network using FGL we require a hierarchical clustering and not just a parcellation. Hence, instead of using the segmentation produced by the parcellation algorithm provided by nilearn, we use the computed ward tree. We then slice into the ward clustering to produce parcellations with 32, 256 and 1024 regions. These have been visualized in Figure 1. Clearly, these groups are spatially connected. We need groupings of voxels into 1024 groups, then a grouping of these 1024 groups into 256 groups and finally a grouping of 256 groups into 32 groups. Since ward clustering is a hierarchical clustering scheme and outputs a tree structure, we extract these groupings by making appropriate cuts into the tree. Fully Connected Models: We experimented with Fully Connected Neural Networks (FNN) and Multinomial Logistic Regression (LR) with and without Dimension Reduction using our parcellation. We found that using Dimension Reduction reduced performance and hence do not report it. For FNNs, we tried 2 and 3 layer versions with intermediate sizes chosen from 64, 128, 256 and 512. The model with intermediate layer sizes of 512 and 128 worked best. The aforementioned models take a masked fMRI image as input and we used the MNI152 mask provided by nilearn. Emperically, we find that, for brain decoding, using a linear activation performs better than using non-linear activations. For this reason, we use a linear activation for our models. Unfortunately, we are unclear about why this occurs for this domain. We also evaluate Feature Grouping suggested by which is also designed to exploit structured sparsity but does this using a wide model (implemented using multiple randomized clusterings), unlike our deep FGL approach. We used code provided by the authors. We experimented with a variety of architectures and found no improvement by using residual connections or Batch-Norm. We also report using CoordConv. We found that using non-linear activations hurt the model's performance, similar to our finding with FNNs. Further, maxpooling also reduced performance. The architecture is 5 3-D convolution layers of stride 2 and kernel size 4. The input volumes have size 91 × 109 × 91, and convolution reduces the volume to 2 × 3 × 2 with 128 channels. We flatten this volume and pass it through a fully connected network to get the score for each label. The architecture for the CoordConv is identical to the CNN since CoordConv only concatenates a few input channels to the input image. We use Conv to refer to the Convolutional Neural Network and CC to refer to the CoordConv variant. We use 3 layers of FGL, each of which use the Parcellation described earlier. The input images have 212455 voxels after masking. We treat every voxel as a variable with a single feature. These voxels are then reduced to 1024 groups with feature vectors of length 8 each. Next, these groups are reduced to 256 variables with 64 features and finally to 32 variables with 128 features. The final prediction is made by flattening the output of the last FGL layer and passing it through a fully connected layer. The ing number of parameters is roughly 2 million, which is also roughly the same number of parameters used for CNN and CC. While this is a lot of parameters, we found that reducing the number of parameters by changing the number of features for each intermediate variable decreases performance for both convolution and FGL. We split each dataset multiple times into a train and test set. The split is done such that no subject in the test set appears in the training set. In each split, 30% of subjects are used for testing, and all or a part of the remaining subjects are used for training. Convolution based models were trained for 50 epochs, feedforward neural networks for 30 and FGL for 20. These hyperparameters were selected by monitoring for overfitting on the training set (using a further validation split). We perform experiments to study: the benefit of FGL given a lot of data, the benefits of FGL at small sample sizes, the effect of depth, and the effect of intermediate vector length. Large sample setting: The first experiment uses all of the training data (70% of total data) to train. We report out-of-sample accuracy in Table 1. We also report the p-values of one sided Wilcoxon rank sum tests between the performance of each model compared to 3 Layer FGL on the HCP dataset. Small sample setting: The second set of experiments varies the fraction of data used for training on the smaller datasets -namely, Archi, Brainomics and Cam-CAN. We explore small sample performance because limited sample sizes are typical for fMRI studies. Figure 4 plots test accuracy against fraction of data used for training. It demonstrates that FGL outperforms the baselines even when only a small amount of training data is used. Effect of depth, FGL depth vs. width: We train two additional models to study the effect of depth: one which uses only the first layer of FGL, and another that uses the first two layers. In Table 1, they're referred to as "FGL (1 Layer)" and "FGL (2 Layers)" respectively. The show that increasing FGL depth provides a statistically significant improvement in test accuracy on the HCP dataset. When compared to Feature grouping , the in Table 1 show that even after extensive tuning (we used the reported best performing settings and attempted additional tuning), this approach is not competitive -suggesting a significant representation benefit by exploiting FGL depth vs. width for this task. Unfortunately, the provided code did not scale to the largest datasets -hence the missing on HCP. To study the effect of intermediate vector length, we train five single layer FGL models with intermediate vector lengths (c out) of 1, 2, 4, 8 and 16. We plot the test accuracy against c out in Figures 5a and 5b. On 4 out of 5 datasets, increasing c out improves test accuracy. This is expected because the classification task is not binary and each region of the brain could contribute to multiple cognitive processes in different ways. However, the effect is not as pronounced on the Cam-CAN dataset. We suspect that this is because Cam-CAN has fewer labels than the other datasets. We note that c out could be interpreted as width of our model, similar to channel size of CNNs, which is known to be important. These experiments demonstrate the clear benefit of using FGL compared to other models with roughly the same number of parameters. When using 70% of data for training, FGL provides 2-6% of improvement in test accuracy on 4 of 5 datasets. A similar trend exists even when smaller amounts of data is used. As for the 5th dataset, Brainomics, FGL is on par with CNN based methods but better than Fully connected networks. While the effect of depth is not clear on smaller datasets, we note that on the HCP dataset, deeper models have a statistically significant improvement in performance. Further, an increase in c out also improves performance. Our main argument is that current methods discard important information about spatial smoothness as encoded by hierarchical spatial clustering. As pointed out, the main cost of our method is in constructing these hierarchies. For brain imaging, most datasets include readily available resting state data. From the larger view, we plan to encourage application communities to develop applicationspecific FGL architectures, which can be shared across several related tasks. In this work we propose a new layer architecture, the Fixed Grouping Layer (FGL), parameterized by a grouping of input variables. FGL explicitly extracts features within each input group. This is in contrast to convolution which extracts local features across the input, and fully connected networks which extract both global and local features. We demonstrate the benefit of using FGL on 5 real fMRI datasets of different sizes. Future work will involve the application of FGL to other tasks and application domains. Input Specification: In this work we deal with image-like data, either in 2D or 3D. Consider an input with s pixels or voxels in c channels -for example, a 64 × 64 image with RGB colors will have s = 4096 and c = 3. Such an input is treated as s variables with feature vectors of c length for each variable. Groupings: Since the output of the first FGL layer is feature vectors for each group, the grouping of the second FGL layer must group together the outputs of the first layer. Hence, we need a hierarchical structure with input variables at the leaf nodes. In this work, we use a Ward clustering of the brain, however, other clusterings may be more appropriate in other settings. We ran an experiment using the simulated dataset to estimate how robust FGL was to imperfect A. Apart from providing FGL the true voronoi diagrams, we also ran FGL using the clusters from K-Means clustering That is, A ji = 1 if pixel i was in cluster j according to K-Means. We did this using 16, 32 and 48 clusters obtained by clustering pixels in the training dataset. The are reported in S1. We see that although there is a drop when using imperfect A, FGL still outperforms logistic regression and convolution. This emphasises the benefit of capturing structure. We do see a larger drop when we use a clustering that is not representative enough (16 clusters), but don't see much gain from using a more representative one (48 clusters). While the FGL model is straightforward, multiple variants of it are possible. First, notice that FGL is essentially the following three operations (ignoring the bias): • Linear transformation: The multiplication, xv, transforms the data x from one basis to another using a linear transform v. • Rescaling: The hadamard product with u rescales each vector along each dimension independently • Aggregation: The multiplication by A aggregates the vectors (xv) u within each group using summation. Performing these operations in a different order creates some basic variants: for example, we could aggregate within groups, then rescale, and finally perform a linear transformation. These changes to operation order will require the parameters to be defined differently. For example, if the hadamard product with u is done after aggregation, then u will need to have n out rows. Another interesting variant is to replace the aggregation with a max operation within each group along each dimension. We think this is similar to doing a maxpool operation in convolution neural networks while the summation by A is similar to a weighted-sum-pool depending on the values of A ji. Early showed worse when using the max reduction variant and hence we did not investigate it further. However, it might prove effective in cases where a signal being present in one variable within a group is equivalent to the group showing that variable. Another possible benefit that we do not investigate is the use of multiple variable groupings -we can concatenate the A matrices that represent each grouping to make FGL extract features within each group from the union of both groups. That is, if A, A are the matrices that represent two groupings, we could use. This would allow one to make use of multiple types of groupings. For example, we could create parcellations at different points of the accuracyreproducibility tradeoff studied by , and make use of both. Similarly, one could create parcellations from different datasets and use them at once. However, using a single parcellation was sufficient to create a significant gain in performance, hence we don't go deeper in this direction. We mention a few other variants that did not perform as well in supplement S3. Our implementation of FGL using PyTorch is available at https://github.com/ anonymous/link. In this section, we discuss some challenges in implementation -If the number of input variables is large, performing A((xv) u) as a matrix multiplication is expensive. There are some ways to work around this: • Since A is a binary matrix, we can treat ((xv) u) as a matrix of embeddings, and lookup the indices at which A is non-zero, then performing necessary aggregation. • If the variable groups are mutually exclusive -that is, each input variable only belongs to one group, then A((xv) u) can be performed by scattering (xv) u according to the indices at which A is non zero. Prior literature (; ;) has shown that initialization of deep networks matters. Generally, a layer's weights are randomly initialized by sampling from a uniform distribution, denoted by U [−m, m] for some m based on the number of inputs, outputs and the activation function. Hence, after minor modifications for different activation functions, we use the following initialization:, v ij ∼ U − 1 1 + 5c in, 1 1 + 5c in Figure S2: Illustration of FGL. Given the numbered hierarchical segmentation, FGL extracts features for each segment. The inputs are 9 variables corresponding to segments of a square, which are first grouped as {{1, 2}, {3, 4, 5}, {6, 7}, {8, 9}}. The ing 4 groups are then grouped into 2 groups using the grouping {{1, 2}, {3, 4}}. The output is passed to a fully connected network to predict labels. Note that intermediate layers can use feature vectors of length greater than 1. One of the major benefits of using convolution is that it performs parameter sharing -which comes with its own benefits. Adapting FGL to perform parameter sharing is much harder. Typically, a fully connected from n in × c in numbers to n out × c out numbers would require n in × n out × c in × c out parameters. But this number is astronomical. To avoid using as many parameters, we decompose the operation into a multiplication by v followed by a Hadamard product with u. Doing so reduces the number of parameters to c in × c out + n in × c out. This is much more tractable, but more reduction might be possible: sharing parameters between groups seems lucrative, unfortunately, different groups can have different sizes and an arbitrary ordering -prevent us from parameter sharing further. If group sizes were constant and an ordering of variables was fixed, it would be possible to further reduce the number of parameters from O(n in) to O(groupsize). We use a straightforward architecture for convolution -repeated convolutional layers of stride 2 and a kernel size of 4 with appropriate padding, followed by a fully connected network. We found that using maxpool for downsampling reduced performance, and so did using non-linear activation functions. A visualization is provided in S1.

A feedforward layer to incorporate structured smoothness into a deep learning model

represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.