Question: Represent the article for finding a single-sentence TLDR

Question: First-order methods such as stochastic gradient descent (SGD) are currently the standard algorithm for training deep neural networks. Second-order methods, despite their better convergence rate, are rarely used in practice due to the pro- hibitive computational cost in calculating the second-order information. In this paper, we propose a novel Gram-Gauss-Newton (GGN) algorithm to train deep neural networks for regression problems with square loss. Our method draws inspiration from the connection between neural network optimization and kernel regression of neural tangent kernel (NTK). Different from typical second-order methods that have heavy computational cost in each iteration, GGN only has minor overhead compared to first-order methods such as SGD. We also give theoretical to show that for sufficiently wide neural networks, the convergence rate of GGN is quadratic. Furthermore, we provide convergence guarantee for mini-batch GGN algorithm, which is, to our knowledge, the first convergence for the mini-batch version of a second-order method on overparameterized neural net- works. Preliminary experiments on regression tasks demonstrate that for training standard networks, our GGN algorithm converges much faster and achieves better performance than SGD. First-order methods such as Stochastic Gradient Descent (SGD) are currently the standard choice for training deep neural networks. The merit of first-order methods is obvious: they only calculate the gradient and therefore are computationally efficient. In addition to better computational efficiency, SGD has even more advantages among the first-order methods. At each iteration, SGD computes the gradient only on a mini-batch instead of all training data. Such randomness introduced by sampling the mini-batch can lead to better generalization (; ; ; ;) and better convergence (; a; b), which is crucial when the function class is highly overparameterized deep neural networks. Recently there is a huge body of works trying to develop more efficient first-order methods beyond SGD (; ; ;). Second-order methods, despite their better convergence rate, are rarely used to train deep neural networks. At each iteration, the algorithm has to compute second order information, for example, the Hessian or its approximation, which is typically an m by m matrix where m is the number of parameters of the neural network. Moreover, the algorithm needs to compute the inverse of this matrix. The computational cost is prohibitive and usually it is not even possible to store such a matrix. Formula and require subtle implementation tricks to use backpropagation. In contrast, GGN has simpler update rule and better guarantee for neural networks. In a concurrent and independent work, Zhang et al. (2019a) showed that natural gradient method and K-FAC have a linear convergence rate for sufficiently wide networks in full-batch setting. In contrast, our method enjoys a higher-order (quadratic) convergence rate guarantee for overparameterized networks, and we focus on developing a practical and theoretically sound optimization method. We also reveal the relation between our method and NTK kernel regression, so using based on NTK (b), one can easily give generalization guarantee of our method. Another independent work proposed a preconditioned Q-learning algorithm which has similar form of our update rule. Unlike the methods considered in Zhang et al. (2019a); which contain the learning rate that needed to be tuned, our derivation of GGN does not introduce a learning rate term (or understood as suggesting that the learning rate can be fixed to be 1 to get good performance which is verified in Figure 2 (c)). Nonlinear least squares regression problem is a general machine learning problem. Given data pairs {x i, y i} n i=1 and a class of nonlinear functions f, e.g. neural networks, parameterized by w, the nonlinear least squares regression aims to solve the optimization problem min w∈R m L(w) = 1 2 In the seminal work , the authors consider the case when f is a neural network with infinite width. They showed that optimization on this problem using gradient flow involves a special kernel which is called neural tangent kernel (NTK). The follow-up works further extended the relation between optimization and NTK which can be concluded in the following lemma: Lemma 1 (Lemma 3.1 in Arora et al. (2019a), see also; ). Consider optimizing problem by gradient descent with infinitesimally small learning rate: dwt dt = −∇L(w t). where w t is the parameters at time t. Let f t = (f (w t, x i)) n i=1 ∈ R n be the network outputs on all x i's at time t, and y = (y i) n i=1 be the desired outputs. Then f t follows the following evolution: where G t is an n × n positive semidefinite matrix, i.e. the Gram matrix w.r.t. the NTK at time t, whose (i, j)-th entry is ∇ w f (w t, x i), ∇ w f (w t, x j). The key idea of and its extensions (b; a; ; a; ; ; ; b; a; ;) is that when the network is sufficiently wide, the Gram matrix at initialization G 0 is close to a fixed positive definite matrix defined by the infinite-width kernel and G t is close to G 0 during training for all t. Under this situation, G t remains invertible, and the above dynamics is then identical to the dynamics of solving kernel regression with gradient flow w.r.t. the current kernel at time t. In fact, Arora et al. (2019a) rigorously proves that a fully-trained sufficiently wide ReLU neural network is equivalent to the kernel regression predictor. As pointed out in , the idea of NTK can be summarized as a linear approximation using first order Taylor expansion. We give an example of this idea on the NTK at initialization: where ∇ w f (w 0, x) can then be viewed as an explicit expression of feature map at x, w − w 0 is the parameter in reproducing kernel Hilbert space (RKHS) induced by NTK and f (w, The idea of linear approximation is also used in the classic Gauss-Newton method to obtain an acceleration algorithm for solving nonlinear least squares problem. Concretely, at iteration t, Gauss-Newton method takes the following first-order approximation: where w t stands for the parameter at iteration t. We note that this is also the linear expansion for deriving NTK at time t. According to Eq. and, to update the parameter, one can instead solve the following problem. where f t, y have the same meaning as in Lemma 1, and J t = (∇ w f (w t, x 1), · · ·, ∇ w f (w t, x n)) ∈ R n×m is the Jacobian matrix. A necessary and sufficient condition for w to be the solution of Eq. is Below we will denote H t:= J t J t ∈ R m×m. For under-parameterized model (i.e., the number of parameters m is less than the number of data n), H t is invertible, and the update rule is This can also be viewed as an approximate Newton's method using H t = J t J t to approximate the Hessian matrix. In fact, the exact Hessian matrix is In the case when f is only mildly nonlinear w.r.t. w at data point x i's, ∇ 2 w f (w t, x i) ≈ 0, and H t is close to the real Hessian. In this situation, the behavior of the Gauss-Newton method is similar to that of Newton's method, and thus can achieve a superlinear convergence rate . The classic second-order methods using approximate Hessian such as Gauss-Newton method described in the previous section face obvious difficulties dealing with the intractable approximate Hessian matrix when the regression model is an overparameterized neural network. In Section 3.1, we develop a Gram-Gauss-Newton (GGN) method which is inspired by NTK kernel regression and does not require the computation of the approximate Hessian. In Section 3.2, we show that for sufficiently wide neural networks, GGN has quadratic convergence rate. In Section 3.3, we show that the additional computational cost (per iteration) of GGN compared to SGD is small. We now describe our GGN method to learn overparameterized neural networks for regression problems. As mentioned in the previous sections, for sufficiently wide networks, using gradient descent for solving the regression problem has similar dynamics as using gradient descent for solving NTK kernel regression (Lemma 1) w.r.t. NTK at each step. However, one can also solve the kernel regression problem w.r.t. the NTK at each step immediately using the explicit formula of kernel regression. By explicitly solving instead of using gradient descent to solve NTK kernel regression, one can expect the optimization to get accelerated. We propose our Gram-Gauss-Newton (GGN) method to directly solve the NTK kernel regression with Gram matrix G t at each time step t. Note that the feature map of NTK at time t, based on the approximation in Eq., can be expressed as x → ∇ w f (w t, x), and the linear parameters in RKHS are w − w t, also the target is f (w, x i) − f (w t, x i). Therefore, the kernel (ridgeless) regression solution where J t,S is the matrix of features at iteration t computed on the training data set S which is equal to the Jacobian, f t,S and y S are the vectorized outputs of neural network and the corresponding targets on S respectively, and is the Gram matrix of the NTK on S. One may wonder what is the relation between our derivation from NTK kernel regression and the Gauss-Newton method. We point out that for overparameterized models, there are infinitely many solutions of Eq. but our update rule essentially uses the minimum norm solution. In other words, the GGN update rule re-derives the Gauss-Newton method with the minimum norm solution. This somewhat surprising connection is due to the fact that in kernel learning, people usually choose a kernel with powerful expressivity, i.e. the dimension of feature space is large or even infinite. However, by the representer theorem , the solution of kernel (ridgeless) regression lies in the n-dimensional subspace of RKHS and minimizes the RKHS norm. We refer the readers to Chapter 11 of for details. As mentioned in Section 1, the design of learning algorithms should consider not only optimization but also generalization. It has been shown that using mini-batch instead of full batch to compute derivatives is crucial for the learned model to have good generalization ability (; ; ; ;). Therefore, we propose a mini-batch version of GGN. The update rule is the following: where B t is the mini-batch used at iteration t, J t,Bt and G t,Bt are the Jacobian and the Gram matrix computed using the data of B t respectively, and f t,Bt, y Bt are the vectorized outputs and the corresponding targets on B t respectively. G t,Bt = J t,Bt J t,Bt is a very small matrix when using a typical batch size. One difference between Eq. and Eq. is that our update rule only requires to compute the Gram matrix G t,Bt and its inverse. Note that the size of G t,Bt is equal to the size of the mini-batch and is typically very small. So this also greatly reduces the computational cost. Using the idea of kernel ridge regression (which can also be viewed as Levenberg-Marquardt extension of Gauss-Newton method), we introduce the following variant of GGN: where λ > 0 is another hyper-parameter controlling the learning process. Our algorithm is formally described in Algorithm 1. Fetch a mini-batch B t from the dataset. Calculate the Jacobian matrix J t,Bt. Calculate the Gram matrix G t,Bt = J t,Bt J t,Bt. Update the parameter by w t+1 = w t − J t,Bt (λG t,Bt + αI) −1 (f t,Bt − y Bt). 8: In this subsection, we show that for two-layer neural networks, if the width is sufficiently large, then: Full-batch GGN converges with quadratic convergence rate. Mini-batch GGN converges linearly. (For clarity, here we only present a proof for two-layer neural networks, but we believe that it is not hard for the to be extended to deep neural networks using the techniques developed in Du et al. (2018a); ). As we explained through the lens of NTK, the is a consequence of the fact that for wide enough neural networks, if the weights are initialized according to a suitable probability distribution, then with high probability the output of the network is close to a linear function w.r.t. the parameters (but nonlinear w.r.t. the input of the network) in a neighborhood containing the initialization point and a global optimum. Although the neural networks used in practice are far from that wide, this still motivates us to design the GGN algorithm. Neural network structure. We use the following two-layer network where x ∈ R d is the input, M is the network width, W = (w 1, · · ·, w M) and σ(·) is the activation function. Each entry of W is i.i.d. initialized with the standard Gaussian distribution w r ∼ N (0, I d) and each entry of a is initialized from the uniform distribution on {±1}. Similar to Du et al. (2018b), we only train the network on parameter W just for the clarity of the proof. We also assume the activation function σ(·) is -Lipschitz and β-smooth, and and β are regarded as O absolute constants. The key finding, as pointed out in; Du et al. (2018b; a), is that under such initialization, the Gram matrix G has an asymptotic limit, which is, under mild conditions (e.g. input data not being degenerate etc., see Lemma F.2 of Du et al. (2018a) ), a positive definite matrix Assumption 1 (Least Eigenvalue of the Limit of Gram Matrix). We assume the matrix K defined in above is positive definite, and denote its least eigenvalue as Now we are ready to state our theorem of full-batch GGN: Theorem 1 (Quadratic Convergence of Full-batch GGN on Overparameterized Neural Networks). Assume Assumption 1 holds. Assume the scale of the data is, then with probability 1 − δ over the random initialization, the full-batch version of GGN whose update rule is given in Eq. satisfies the following: 1) The Gram matrix G t,S at each iteration is invertible; 2) The loss converges to zero in a way that for some C that is independent of M, which is a second-order convergence. For the mini-batch version of GGN, by the analysis of its NTK limit, the algorithm is essentially doing serial subspace correction on subspaces induced by mini-batch. So mini-batch GGN is similar to the Gauss-Siedel method applied to solving systems of linear equations, as shown in the proof of the following theorem. Similar to the full batch situation, GGN takes the exact solution of the "kernel regression problem on the subspace" which is faster than just doing a gradient step to optimize on the subspace. Moreover, we note that existing of the convergence of SGD on overparameterized networks usually use the idea that when the step size is bounded by a quantity related to smoothness, the SGD can be reduced to GD. However, our analysis takes a different way from the analysis of GD, thus does not rely on small step size. In the following, we denote G 0 ∈ R n×n as the initial Gram matrix. Let n = bk, where b is the batch size and k is the number of batches, and let where represents the block-diagonal and block-lower-triangular parts of G 0. We will show that the convergence of mini-batch GGN is highly related to the spectral radius of A. To simplify the proof, we make the following mild assumption on A: Assumption 2 (Assumption on the Iteration Matrix). Assume the matrix A defined in above is diagonalizable. So we choose an arbitary diagonalization of A as A = P −1 QP and denote We note that Assumption 2 is only for the sake of simplicity. Even if it does not hold, an infinitesimally small perturbation can make any matrix diagonalizable, and it will not affect the proof. Now we are ready to state the theorem for mini-batch GGN. Theorem 2 (Convergence of Mini-batch GGN on Overparameterized Neural Networks). Assume Assumption 1 and 2 hold. Assume the scale of the data is We use the mini-batch version of GGN whose update rule is given in Eq., and the batch B t is chosen sequentially and cyclically with a fixed batch size b and k = n/b updates per epoch. If the network width then with probability 1 − δ over the random initialization, we have the following: 1) The Gram matrix G t,Bt at each iteration is invertible; 2) The loss converges to zero in a way that after T epochs, we have Proof sketch for Theorem 1 and 2. Denote J t = J(W t) and For the full-batch version, we have Then we control the first term in Eq. in a way similar to the following: can be upper bounded, we get our Eq.. For the mini-batch version, similarly we have where the subscript B t denotes the sub-matrix/vector corresponding to the batch, andG is a zero-padded version of G −1 t,Bt to make Eq. hold. Therefore, after one epoch (from the (tk + 1)-th to the ((t + 1)k)-th update), we have We will see that the matrix A t is close to the matrix A defined in, so it boils down to analyzing the spectral properties of A. For both theorems, we can compute that as M increases, the norm of the update W t − W t+1 F does not increase with M, so the update is small compared to the Gaussian initialization where. From this we can derive that the matrices J, G etc. remain close to their initialization, which makes bounding their norms possible. The full proof is in the appendix. In , the accelerated convergence is related to the local linearity and the stability of the Jacobian and Gram matrix. We emphasize that our theorems serve more as a motivation than a justification of our GGN algorithm, because we expect that GGN works in practice, even under milder situations when M is not as large as the theorem demands or for deep networks with different architectures, and that GGN would still perform much better than first-order methods. We have proved that for sufficiently overparametrized deep neural networks, full-batch GGN has quadratic convergence rate. In this subsection, we analyze the per-iteration computational cost of GGN, and compare it to that of SGD. For every mini-batch (i.e., iteration), there are two major steps of computation in GGN: • (A). Forward, and then backpropagate for computing the Jacobian matrix J. • (B). Use J to compute the update J (λG + αI) We show that the computational complexity of (A) is the same as that of SGD with the same batch size; and the computational complexity of (B) is small compared to (A) for typical networks and batch sizes. Thus, the per-iteration computation overhead of GGN is very small compared to SGD. Overall, in terms of training time, GGN can be much faster than SGD. For the computation in step (A), the forward part is just the same as that of SGD. For the backward part, for every input data, GGN keeps track of the output's derivative for the nodes in the middle of the computational graph. This part is just the same as backpropagation in SGD. What is different is that GGN also, for every input data, keeps track of the output's derivative for the parameters; while in SGD the derivatives for the parameters are averaged over a batch of data. However, it is not difficult to see the computational costs of GGN and SGD are the same. For the computation in step (B), observe that the size of the Jacobian is b × m where b is the batch size and m is the number of parameters. The Gram matrix G t,Bt = J t,Bt J t,Bt in our GramGauss-Newton method is of size b × b and it only requires O(b 2 m + b 3) for computing G t,Bt and a matrix inverse. Multiplying the two matrices to f − y requires even less computation. Overall, the computational cost in step (B) is small compared to that of step (A). Given the theoretical findings above, in this section, we compare our proposed GGN algorithm with several baseline algorithms in real applications. In particular, we mainly study two regression tasks, AFAD-LITE and RSNA Bone Age (rsn, 2017). AFAD-LITE task is to predict the age of human from the facial information. The training data of the AFAD-LITE task contains 60k facial images and the corresponding age for each image. We choose ResNet-32 as the base model architecture. During training, all input images are resized to 64 * 64. We study two variants of the ResNet-32 architecture: ResNet-32 with batch normalization layer (referred to as ResNetBN), and ResNet-32 with Fixup initialization (b) (referred to as ResNetFixup). In both settings, we use SGD as our baseline algorithm. In particular, we follow to use its momentum variant and set the hyper-parameters lr=0.003 and momentum=0.9 determined by selecting the best optimization performance using grid search. Since batch normalization is computed over all samples within a mini-batch, it is not consistent with our assumption in Section 2 that the regression function has the form of f (w, x), which only depends on w and a single input datum x. For this reason, the GGN algorithm does not directly apply to ResNetBN, and we test our proposed algorithm on ResNetFixup only. We set λ = 1 and α = 0.3 for GGN. We follow the common practice to set the batch size to 128 for our proposed method and all baseline algorithms. Mean square loss is used for training. RSNA Bone Age task is a part of the 2017 Pediatric Bone Age Challenge organized by the Radiological Society of North America (RSNA). It contains 12,611 labeled images. Each image in this dataset is a radiograph of a left hand labeled with the corresponding bone age. During training, all input images are resized to 64 * 64. We also choose ResNetBN and ResNetFixup for this experiment, and use ResNetBN and ResNetFixup trained in the first task as warm-start initialization. We use lr= 0.01 and momentum= 0.9 for SGD, and use λ = 1 and α = 0.1 for GGN. Batch size is set to 128 in these experiments, and mean square loss is used for training. Convergence. The training loss curves of different optimization algorithms for AFAD-LITE and RSNA Bone Age tasks are shown in Figure 1. On both tasks, our proposed method converges much faster than the baselines. We can see from Figure 1a and Figure 1b that, on the AFAD-LITE task, the loss using our GGN method quickly decreases to nearly zero in 30 epochs. On the contrary, for both baselines using SGD, the loss decays much slower than our method in terms of wall clock time and epochs. Similar advantage of GGN can also be observed on the RSNA bone age task. Generalization performance and different hyper-parameters. We can see that our proposed method trains much faster than other baselines. However, as a machine learning model, generalization performance also needs to be evaluated. Due to space limitation, we only provide the test curve for the RSNA Bone Age task in Figure 2a. From the figure, we can see that the test loss of our proposed method also decreases faster than the baseline methods. Furthermore, the loss of our GGN algorithm is lower than those of the baselines. These show that the GGN algorithm can not only accelerate the whole training process, but also learn better models. We then study the effect of hyper-parameters used in the GGN algorithm. We try different λ and α on the RSNA Bone Age task and report the training loss of all experiments at the 10 th epoch. All are plotted in Figure 2c. In the figure, the x-axis is the value of λ and the y-axis is the value of α. The gray value of each point corresponds to the loss, the lighter the color, the higher the loss. We can see that the model converges faster when λ is close to 1. In GGN, α can be considered as the inverse value of the learning rate in SGD. Empirically, we find that the convergence speed of training loss is not that sensitive to α given a proper λ, such as λ = 1. Some training loss curves of different hyper-parameter configurations are shown in Figure 2b. We propose a novel Gram-Gauss-Newton (GGN) method for solving regression problems with square loss using overparameterized neural networks. Despite being a second-order method, the computation overhead of the GGN algorithm at each iteration is small compared to SGD. We also prove that if the neural network is sufficiently wide, GGN algorithm enjoys a quadratic convergence rate. Experimental on two regression tasks demonstrate that GGN compares favorably to SGD on these data sets with standard network architectures. Our work illustrates that second-order methods have the potential to compete with first-order methods for learning deep neural networks with huge number of parameters. In this paper, we mainly focus on the regression task, but our method can be easily generalized to other tasks such as classification as well. Consider the k-category classification problem, the neural network outputs a vector with k entries. Although this will increase the computational complexity of getting the Jacobian whose size increases k times, i.e., J ∈ R (bk)×m, each row of J can be still computed in parallel, which means the extra cost only comes from parallel computation overhead when we calculate in a fully parallel setting. While most first-order methods for training neural networks can hardly make use of the computational resource in parallel or distributed settings to accelerate training, our GGN method can exploit this ability. For first-order methods, basically extra computational resource can only be used to calculate more gradients at a time by increasing batch size, which harms generalization a lot. But for GGN, more resource can be used to refine the gradients and achieve accelerated convergence speed with the help of second-order information. It is an important future work to study the application of GGN to classification problems. Notations. We use the following notations for the rest of the sections. • Let [n] = {1, · · ·, n}. • J W,x ∈ R M ×d denotes the gradient ∂W, which is of the same size of W. • The bold J W or J(W) denotes the Jacobian with regard to all n data, with each gradient for W vectorized, i.e. • w r denotes the r-th row of W, which is the incoming weights of the r-th neuron. • W 0 stands for the parameters at initialization. • d W,x:= σ (Wx) ∈ R M ×1 denotes the (entry-wise) derivative of the activation function. • We use ·, · to denote the inner product, · 2 to denote the Euclidean norm for vectors or the spectral norm for matrices, and · F to denote the Frobenius norm for matrices. where • is the point-wise product. So we can also easily solve G as Our analysis is based on the fact that G stays not too far from its infinite-width limit which is a positive definite matrix with least eigenvalue denoted λ 0, and we assume λ 0 > 0. λ 0 is a small data-dependent constant, and without loss of generality we assume λ 0 ≤ 1, or else we can just The first lemma is about the estimation of relevant norms at initialization. Lemma 2 (Bounds on Norms at Initialization). If M = Ω (d log(16n/δ)), then with probability at least 1 − δ/2 the following holds (c).). Notice that W 0 ∈ R M ×d is a Gaussian random matrix, the Corollary states that with probability 1 − 2e By choosing M = max(d, 2 log(8/δ)), we obtain W 0 2 ≤ 3 √ M with probability 1 − δ/4. (b). First, a r, r ∈ [M] are Rademacher variables, thereby 1-sub-Gaussian, so with probability 1 − 2e This means if we take M = Ω (log(16/δ)), Next, the vector is a standard Gaussian vector. Suppose the activation σ(·) is l-Lipschitz and l is O by our assumption, with the vector a fixed, the function According to the classic on the concentration of a Lipschitz function over Gaussian variables (see Theorem 2.26 of), we have holds jointly for all i ∈ [n] with probability 1 − δ/8. Note that Plugging in and into, we see that as long as M = Ω(log(16n/δ)), then with probability According to we can easily know that The next lemma is about the least eigenvalue of the Gram matrix G at initialization. It shows that when M is large, G W0 is close to K and has a lower bounded least eigenvalue. It is the same as Lemma 3.1 in Du et al. (2018b), but here we restate it and its proof for the reader's convenience., then with probability at least 1 − δ/2 over random initialization, we have Proof. Because σ is Lipschitz, σ (wx i)σ (wx j) is bounded by O. For every fixed (i, j) pair, at initialization G ij is an average of independent random variables, and by Hoeffding's inequality, applying union bound for all n 2 of (i, j) pairs, with probability 1 − δ/2 at initialization we have and then Next, in Lemma 4 and 5 we will bound the relevant norms and the least eigenvalue of G inside some scope of W that covers the whole optimization trajectory starting from W 0. Specifically, we consider the range where R is determined later to make sure that the optimization trajectory remains inside B(R). The idea of the whole convergence theorem is that when the width M is large, R is very small compared to its initialization scale:. This way, neither the Jacobian nor the Gram matrix changes much during optimization. Lemma 4 (Bounds on Norms in the Optimization Scope). Suppose the events in Lemma 2 hold. There exists a constant C > 0 such that if M ≥ CR 2, we have the following: (a) For any W ∈ B(R), we have Also, for any W ∈ B(R), we have Proof. (a). This is straightforward from Lemma 2(a), the definition of B(R), and M = Ω(R 2). (b). According to the O-smoothness of the activation, we have so we can bound And according to, we have Also, taking W 1 = W and W 2 = W 0, combining with Lemma 2(c), we see there exists C such that for M ≥ CR 2 we have J W F = O, and naturally, we have and thus combined with Lemma 3, we know that G W remains invertible when W ∈ B(R) and satisfies Proof. Based on the in Lemma 4(b), we have To make the above less than Proof idea. In this section, we use W t, t ∈ {0, 1, · · ·} to represent the parameter W after t iterations. For convenience, J t, G t, f t is short for J Wt, G Wt, f Wt respectively. We introduce For each iteration, if G t is invertible, we have Then we control the first term of the right hand side based on Lemma 4 in the following form and plugging into along with norm bounds on J and G we obtain a second-order convergence. Formal proof. Let R t = W t − W t+1 F for t ∈ {0, 1, · · ·}. We take R = Θ n λ0 in Lemma 4 and 5 (the constant is chosen to make the right hand side of hold). We prove that there exists an M = Ω max (with enough constant) that suffices. First we can easily verify that all the requirements for M in Lemma 2-5 can be satisfied. Hence, with probability at least 1 − δ all the events in Lemma 2-5 hold. Under this situation, we do induction on t to show the following: • (a). W t ∈ B(R). • As long as (b) is true for all t, then choosing M large enough to make sure the series {f t − y 2} ∞ t=0 converges to zero, we obtain the second-order convergence property. For t = 0, (a) and (b) hold by definition. Suppose the proposition holds for t = 0, · · ·, T. Then for t = 0, · · ·, T, G t is invertible. Recall that the update rule is vec((Lemma 4 and 5) According to Lemma 2(b) and the assumption that the target label y i = O, we have f 0 − y 2 2 = O(n). When T > 1, the decay ratio at the first step is bounded as ) with enough constant can make sure r is a constant less than 1, in which case the second-order convergence property (b) will ensure a faster ratio of decay at each subsequent step in f t − y T t=0. Combining, we have for variables r ∈ R n×1 (where vec(W) = vec(W 0) + J r), using the Gauss-Siedel method, which means solving for the i-th batch in every epoch. Therefore, it is natural that the matrix is introduced. We will show later that the real update follows In order to prove the theorem, we need some additional lemmas as follows. Lemma 6 (Formula for the Update). If the Gram matrix at each step is invertible, then: (b) The formula for f − y is (c) The update of f − y satisfies where Or, if we denote then we have Specifically, we have where Proof. (a) For, this is exactly the update formula for the i-th batch where the Jacobian and Gram matrix are J ti,i and G ti,ii respectively. Note that), we obtain. (c). Based on we know that where the index goes in decreasing order from left to right. So in order to prove we only need to prove that which we will prove by induction on i. For i = 0 it is trivial that D t − L t = U t0 by definition. Suppose holds for i, then... which proves. Note that by the definition, we have U ti = D ti − L ti, we can then obtain by By, we can see that the iteration matrix A t is close to the matrix A defined in. The convergence of the algorithm is much related to the eigenvalues of A. In the next two lemmas, we bound the spectral radius of A and provide an auxiliary on the convergence on perturbed matrices based on the spectral radius. Lemma 7. Suppose the least eigenvalue of the initial Gram matrix G 0 satisfies λ min (G 0) ≥ 3 4 λ 0, (which is true with probability 1 − δ if M = Ω n 2 log(2n/δ) λ 2 0, according to Lemma 3). Also assume J W0,x l F = O for any l ∈ [n] (which is true with probability 1 − δ, according to Lemma 2). Then the spectral radius A, or equivalently, maximum norm of eigenvalues of A, denoted ρ(A), satisfies Proof. For any eigenvalue λ ∈ C of A, it is an eigenvalue of A, so there exists a corresponding where. It is not hard to see that Also, since by our assumption each entry of L (or of G 0) is at most O, we have Now we use take an inner product of v with and get and therefore This concludes the proof for this lemma. Lemma 8. Denote ρ(A) = ρ 0 ≤ 1. Let A be diagonalized as A = P −1 QP and µ = P 2 P −1 2 (see Assumption 2). Suppose we have A t − A 0 2 ≤ δ for t ≤ T, then In addition to the bounds used in Theorem 1, we provide the next lemma with useful bounds of the norms and eigenvalues of relevant matrices in the optimization scope Lemma 9 (Relevant Bounds for the Matrices in the Optimization Scope). We assume the events in Lemma 2 and 3 hold, and let M = for some large enough constant C, then: For any (t, i)-pair, we assume W t i ∈ B(R) for all i ∈ [k] when t ≤ t and i ∈ [i] when t = t in the following propositions.. Suppose up to t, W t is in B(R) (which means for i ∈ [k + 1], and t ∈ [t − 1], W ti ∈ B(R)). (By the positive-definiteness of G 0 and Lemma 3) (b). By Lemma 4 we know that within B(R), the size of the Jacobian J x l F w.r.t. data x l is O,, this can be applied to each entry of D, L, U, etc., including those J t (i,i +1) terms by the convexity of B(R), and we can see that each entry of these matrices has size at most O and varies inside an O R √ M range. Therefore we get and suffices. With all the preparations, now we are ready to prove Theorem 2. The logic of the proof is just the same as what we did in the formal proof of Theorem 1, where we then used induction on t and now we use induction on the pair (t, i). The key, is still selecting some R so that in each step W ti remains in B(R). Combined with the previous Lemmas, in B(R) we have A t being close to A, and then convergence is guaranteed. Formal proof of Theorem 2. Let R ti = W t(i+1) − W ti F. We take in the range B(R) (where the constant is chosen to make the right hand side of hold). We prove that there exists an with a large enough constant that suffices. First we can easily verify that all the requirements for M in Lemma 2-9 (most importantly, Lemma 9) can be satisfied. Hence, with probability at least 1 − δ all the events in Lemma 2-9 hold. Under this situation, we do induction on (t, i) (t ∈ {1, 2, · · ·}, i ∈ [k], by dictionary order) to show that: For (t, i) =, it holds by definition. Suppose the proposition holds up to (t, i). Then since W ti ∈ B(R), by Lemma 4 we know that 2. This naturally gives us λ min (G ti,ii) ≥ λ0 2, which means G ti,ii is invertible. Similar to the argument in the proof of Lemma 9, we know that each entry of J ti, J t(i,i+1), D ti, L ti, U ti, etc., whose index of (t, i) only contains i with i ≤ i, is of scale Based on the update rule, we have Since this also hold for previous (t, i) pairs, we have which is the reason why we need to take R = Θ. This means that W ti ∈ B(R) holds. And by induction, we have proved that W remains in B(R) throughout the optimization process. The last thing to do is to bound f t − y. By the same logic from above, we have, which proves our theorem. In this section, we give test performance curve of AFAD-LITE dataset in Figure 3 under the same setting with Section 4. In addition, we provide more baseline , e.g. Adam and K-FAC on RSNA Bone Age dataset. Since we find that, as another alternation of BN, Group Normalization (GN) can largely improve the performance of Adam, we also implement the GN layer for our GGN method. We use grid search to obtain approximately best hyper-parameters for every experiment. All experiments are performed with batch size 128, input size 64*64 and weight decay 10 −4. We set the number of groups to 8 for ResNetGN. Other hyper-parameters are listed below. • SGD+ResNetBN: learning rate 0.01, momentum 0.9. • Adam+ResNetBN: learning rate 0.001. • SGD+ResNetGN: learning rate 0.002, momentum 0.9. • Adam+ResNetGN: learning rate 0.0005. • K-FAC+ResNet: learning rate 0.02, momentum 0.9, = 0.1, update frequency 100. • GGN+ResNetGN: λ = 1, α = 0.075. The convergence are summarized in Figure 4. Note to make comparison clearer, we use logarithmic scale for training curves.

Represent the TLDR for finding the corresponding scientific article.

A novel Gram-Gauss-Newton method to train neural networks, inspired by neural tangent kernel and Gauss-Newton method, with fast convergence speed both theoretically and experimentally.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Recent pretrained sentence encoders achieve state of the art on language understanding tasks, but does this mean they have implicit knowledge of syntactic structures? We introduce a grammatically annotated development set for the Corpus of Linguistic Acceptability (CoLA;), which we use to investigate the grammatical knowledge of three pretrained encoders, including the popular OpenAI Transformer and BERT . We fine-tune these encoders to do acceptability classification over CoLA and compare the models’ performance on the annotated analysis set. Some phenomena, e.g. modification by adjuncts, are easy to learn for all models, while others, e.g. long-distance movement, are learned effectively only by models with strong overall performance, and others still, e.g. morphological agreement, are hardly learned by any model. The effectiveness and ubiquity of pretrained sentence embeddings for natural language understanding has grown dramatically in recent years. Recent sentence encoders like OpenAI's Generative Pretrained Transformer (GPT;) and BERT achieve the state of the art on the GLUE benchmark . Among the GLUE tasks, these stateof-the-art systems make their greatest gains on the acceptability task with the Corpus of Linguistic Acceptability (CoLA;). CoLA contains example sentences from linguistics publications labeled by experts for grammatical acceptability, and written to show subtle grammatical features. Because minimal syntactic differences can separate acceptable sentences from unacceptable ones (What did Bo write a book about? / *What was a book about written by Bo?), and acceptability classifiers are more reliable when trained on GPT and BERT than on recurrent models, it stands to reason that GPT and BERT have better implicit knowledge of syntactic features relevant to acceptability. Our goal in this paper is to develop an evaluation dataset that can locate which syntactic features that a model successfully learns by identifying the syntactic domains of CoLA in which it performs the best. Using this evaluation set, we compare the syntactic knowledge of GPT and BERT in detail, and investigate the strengths of these models over the baseline BiLSTM model published by. The analysis set includes expert annotations labeling the entire CoLA development set for the presence of 63 fine-grained syntactic features. We identify many specific syntactic features that make sentences harder to classify, and many that have little effect. For instance, sentences involving unusual or marked argument structures are no harder than the average sentence, while sentences with long distance dependencies are hard to learn. We also find features of sentences that accentuate or minimize the differences between models. Specifically, the transformer models seem to learn long-distance dependencies much better than the recurrent model, yet have no advantage on sentences with morphological violations. Sentence Embeddings Robust pretrained word embeddings like word2vec and GloVe have been extemely successful and widely adopted in machine learning applications for language understanding. Recent research tries to reproduce this success at the sentence level, in the form of reusable sentence embeddings with pretrained weights. These rep- Table 1: A random sample of sentences from the CoLA development set, shown with their original acceptability labels (= acceptable, *=unacceptable) and with a subset of our new phenomenon-level annotations.resentations are useful for language understanding tasks that require a model to classify a single sentence, as in sentiment analysis and acceptability classification; or a pair of sentences, as in paraphrase detection and natural language inference (NLI); or that require a model to generate text based on an input text, as in question-answering. Early work in this area primarily uses recurrent models like Long Short-Term Memory (, LSTM) networks to reduce variable length sequences into fixed-length sentence embeddings. Current state of the art sentence encoders are pretrained on language modeling or related tasks with unlabeled-data. Among these, ELMo uses a BiLSTM architecture, while GPT and BERT use the Transformer architecture . Unlike most earlier approaches where the weights of the encoder are frozen after pretraining, the last two fine-tune the encoder on the downstream task. With additional fine-tuning on secondary tasks like NLI, these are the top performing models on the GLUE benchmark . The evaluation and analysis of sentence embeddings is an active area of research. One branch of this work uses probing tasks which can reveal how much syntactic information a sentence embedding encodes about, for instance, tense and voice , sentence length and word content , or syntactic depth and morphological number .Related work indirectly probes features of sentence embeddings using language understanding tasks with custom datasets manipulating specific grammatical features. uses several tasks including acceptability classification of sentences with manipulated verbal inflection to investigate whether LSTMs can identify violations in subject-verb agreement, and therefore a (potentially long distance) syntactic dependency. test whether sentence embeddings encode the scope of negation and semantic roles using semi-automatically generated sentences exhibiting carefully controlled syntactic variation. also semiautomatically generate data and use acceptability classification to test whether word and sentence embeddings encode information about verbs and their argument structures. CoLA & Acceptability Classification The Corpus of Linguistic Acceptability is a dataset of 10k example sentences including expert annotations for grammatical acceptability. The sentences are example sentences taken from 23 theoretical linguistics publications, mostly about syntax, including undergraduate textbooks, research articles, and dissertations. Such example sentences are usually labeled for acceptability by their authors or a small group of native English speakers. A small random sample of the CoLA development set (with our added annotations) can be seen in Table 1.Within computational linguistics, the acceptability classification task has been explored in var-ious settings. train RNNs to do acceptability classification over sequences of POS tags corresponding to example sentences from a syntax textbook. also train RNNs, but using naturally occurring sentences that have been automatically manipulated to be unacceptable. predict acceptability from language model probabilities, applying this technique to sentences from a syntax textbook, and sentences which were translated round-trip through various languages. Lau et al. attempt to model gradient crowdsourced acceptability judgments, rather than binary expert judgments. This reflects an ongoing debate about whether binary expert judgments like those in CoLA are reliable . We remain agnostic as to the role of binary judgments in linguistic theory, taking the expert judgments in CoLA at face value. measure human performance on a subset of CoLA (see TAB5), finding that new human annotators, while not in perfect agreement with the judgments in CoLA, still outperform the best neural network models by a wide margin. We introduce a grammatically annotated version of the entire CoLA development set to facilitate detailed error analysis of acceptability classifiers. These 1043 sentences are expert-labeled for the presence of 63 minor grammatical features organized into 15 major features. Each minor feature belongs to a single major feature. A sentence belongs to a major feature if it belongs to one or more of the relevant minor features. The Appendix includes descriptions of each feature along with examples and the criteria used for annotation. The 63 minor features and 15 major features are illustrated in TAB1. Considering minor features, an average of 4.31 features is present per sentence (SD=2.59). The average feature is present in 71.3 sentences (SD=54.7). Turning to major features, the average sentence belongs to 3.22 major features (SD=1.66), and the average major feature is present in 224 sentences (SD=112). Every sentence is labeled with at least one feature. The sentences were annotated manually by one of the authors, who is a PhD student with extensive training in formal linguistics. The features were developed in a trial stage, in which the annotator performed a similar annotation with different annotation schema for several hundred sentences from CoLA not belonging to the development set. Here we briefly summarize the feature set in order of the major features. Many of these constructions are well-studied in syntax, and further can be found in textbooks such as BID0 and.Simple This major feature contains only one minor feature, SIMPLE, including sentences with a syntactically simplex subject and predicate. Pred(icate) These three features correspond to predicative phrases, including copular constructions, small clauses (I saw Bo jump), and atives/depictives (Bo wiped the table clean).Adjunct These six features mark various kinds of optional modifiers. This includes modifiers of NPs (The boy with blue eyes gasped) or VPs (The cat meowed all morning), and temporal (Bo swam yesterday) or locative (Bo jumped on the bed).Argument types These five features identify syntactically selected arguments, differentiating, for example, obliques (I gave a book to Bo), PP arguments of NPs and VPs (Bo voted for Jones), and expletives (It seems that Bo left).Argument Alternations These four features mark VPs with unusual argument structures, including added arguments (I baked Bo a cake) or dropped arguments (Bo knows), and the passive (I was applauded).Imperative This contains only one feature for imperative clauses (Stop it!).Bind These are two minor features, one for bound reflexives (Bo loves himself), and one for other bound pronouns (Bo thinks he won).Question These five features apply to sentences with question-like properties. They mark whether the interrogative is an embedded clause (I know who you are), a matrix clause (Who are you?), or a relative clause (Bo saw the guy who left); whether it contains an island out of which extraction is unacceptable (*What was a picture of hanging on the wall?); or whether there is pied-piping or a multiword wh-expressions (With whom did you eat?). S-Syntax These seven features mark various unrelated syntactic constructions, including dislocated phrases (The boy left who was here earlier); movement related to focus or information structure (This I've gotta see); coordination, subordinate clauses, and ellipsis (I can't); or sentencelevel adjuncts (Apparently, it's raining).Determiner These four features mark various determiners, including quantifiers, partitives (two of the boys), negative polarity items (I *do/don't have any pie), and comparative constructions. Violations These three features apply only to unacceptable sentences, and only ones which are ungrammatical due to a semantic or morphological violation, or the presence or absence of a single salient word. We wish to emphasize that these features are overlapping and in many cases are correlated, thus not all from using this analysis set will be independent. We analyzed the pairwise Matthews Correlation Coefficient (MCC;) of the 63 minor features (giving 1953 pairs), and of the 15 major features (giving 105 pairs). MCC is a special case of Pearson's r for Boolean variables. 1 These are summarized in TAB3. Regarding the minor features, 60 pairs had a correlation of 0.2 or greater, 17 had a correlation of 0.4 or greater, and 6 had a correlation of 0.6 or greater. None had an anti-correlation of greater magnitude than -0.17. Turning to the major features, 6 pairs had a correlation of 0.2 or greater, and 2 had an anti-correlation of greater magnitude than -0.2.We can see at least three reasons for these observed correlations. First, some correlations can be attributed to overlapping feature definitions. For instance, EXPLETIVE arguments (e.g. There are birds singing) are, by definition, non-canonical arguments, and thus are a subset of ADD ARG. However, some added arguments, such as benefactives (Bo baked Mo a cake), are not expletives. Second, some correlations can be attributed to grammatical properties of the relevant constructions. For instance, QUESTION and AUX are correlated because main-clause questions in English require subject-aux inversion and in many cases the insertion of auxiliary do (Do lions meow?). Third, some correlations may be a consequence of the sources sampled in CoLA and the phenomena they focus on. For instance, the unusually high correlation of EMB-Q and ELLIPSIS/ANAPHOR can be attributed to , which is an article about the sluicing construction involving ellipsis of an embedded interrogative (e.g. I saw someone, but I don't know who).Finally, two strongest anti-correlations between major features are between SIMPLE and the two features related to argument structure, ARGU-MENT TYPES and ARG ALTERN. This follows from the definition of SIMPLE, which excludes any sentence containing a large number or unusual configuration of arguments. We train MLP acceptability classifiers for CoLA on top of three sentence encoders: the CoLA baseline encoder with ELMo-style embeddings, OpenAI GPT, and BERT. We use publicly available sentence encoders with pretrained weights. 2 LSTM encoder: CoLA baseline The CoLA baseline model is the sentence encoder with the highest performance on CoLA from Warstadt et al. The encoder uses a BiLSTM, which reads the sentence word-by-word in both directions, with maxpooling over the hidden states. Similar to ELMo , the inputs to the BiLSTM are the hidden states of a language model (only a forward language model is used in contrast with ELMo). The encoder is trained on a real/fake discrimination task which requires it to identify whether a sentence is naturally occurring or automatically generated. We train acceptability classifiers on CoLA using the CoLA baselines codebase with 20 random restarts, following the original authors' transfer-learning approach: The sentence encoder's weights are frozen, and the sentence embedding serves as input to an MLP with a single hidden layer. All hyperparameters are held constant across restarts. Transformer encoders: GPT and BERT In contrast with recurrent models, GPT and BERT use a self attention mechanism which combines representations for each (possibly non-adjacent) pair of words to give a sentence embedding. GPT is trained using a standard language modeling task, while BERT is trained with masked language modeling and next sentence prediction tasks. For each encoder, we use the jiant toolkit 3 to train 20 random restarts on CoLA feeding the pretrained models published by these authors into a single output layer. Following the methods of the original authors, we fine-tune the encoders during training on CoLA. All hyperparameters are held constant across restarts. The overall performance of the three sentence encoders is shown in TAB5. Performance on CoLA is measured using MCC . We present the best single restart for each encoder, the mean over restarts for an encoder, and the of ensembling the restarts for a given encoder, i.e. taking the majority classification for a given sentence, or the majority label of acceptable if tied. 4 For BERT , we exclude 5 out of the 20 restarts because they were degenerate (MCC=0). Across the board, BERT outperforms GPT, which outperforms the CoLA baseline. However, BERT and GPT are much closer in performance than they are to CoLA baseline. While ensemble performance exceeded the average for BERT and GPT, it did not outperform the best single model. The for the major features and minor features are shown in FIG1, respectively. For each feature, we measure the MCC of the sentences including that feature. We plot the mean of these across the different restarts for each model, and error bars mark the mean ±1 standard deviation. For the VIOLATIONS features, MCC is technically undefined because these features only contain unacceptable sentences. We report MCC in these cases by including for each feature a single acceptable example that is correctly classified by all models. Comparison across features reveals that the presence of certain features has a large effect on performance, and we comment on some overall patterns below. Within a given feature, the effect of model type is overwhelmingly stable, and resembles the overall difference in performance. However, we observe several interactions, i.e. specific features where the relative performance of models does not track their overall relative performance. Comparing Features Among the major features FIG1, performance is universally highest on the SIMPLE sentences, and is higher than each model's overall performance. Though these sentences are simple, we notice that the proportion of ungrammatical ones is on par with the entire dataset. Otherwise we find that a model's performance on sentences of a given feature is on par with or lower than its overall performance, reflecting the fact that features mark the presence of unusual or complex syntactic structure. Performance is also high (and close to overall performance) on sentences with marked argument structures (ARGUMENT TYPES and ARG(UMENT) ALT(ERNATION)). While these models are still worse than human (overall) per- formance on these sentences, this indicates that argument structure is relatively easy to learn. Comparing different kinds of embedded content, we observe higher performance on sentences with embedded clauses (major feature=COMP CLAUSE) embedded VPs (major feature=TO-VP) than on sentences with embedded interrogatives (minor features=EMB-Q, REL CLAUSE). An exception to this trend is the minor feature NO C-IZER, which labels complement clauses without a complementizer (e.g. I think that you're crazy). Low performance on these sentences compared to most other features in COMP CLAUSE might indicate that complementizers are an important syntactic cue for these models. As the major feature QUESTION shows, the difficulty of sentences with question-like syntax applies beyond just embedded questions. Excluding polar questions, sentences with question-like syntax almost always involve extraction of a wh-word, creating a long-distance dependency between the wh-word and its extraction site, which may be difficult for models to recognize. The most challenging features are all related to VIOLATIONS. Low performance on INFL/AGR VIOLATIONS, which marks morphological violations (He washed yourself, This is happy), is especially striking because a relatively high proportion (29%) of these sentences are SIMPLE. These models are likely to be deficient in encoding morphological features is that they are word level models, and do not have direct access sub-word information like inflectional endings, which indicates that these features are difficult to learn effectively purely from lexical distributions. Finally, unusual performance on some features is due to small samples, and have a high standard deviation, suggesting the is unreliable. This includes CP SUBJ, FRAG/PAREN, IMPERATIVE, NPI/FCI, and COMPARATIVE.Comparing Models Comparing within-feature performance of the three encoders to their overall performance, we find they have differing strengths and weaknesses. BERT stands out over other models in DEEP EMBED, which includes challenging sentences with doubly-embedded, as well as in several features involving extraction (i.e. longdistance dependencies) such as VP+EXTRACT and INFO-STRUC. The transformer models show evidence of learning long-distance dependencies better than the CoLA baseline. They outperform the CoLA baseline by an especially wide margin on BIND:REFL, which all involves establishing a dependency between a reflexive and its antecedent (Bo tries to love himself). They also have a large advantage in DISLOCATION, in which expressions are separated from their dependents (Bo practiced on the train an important presentation). The advantage of BERT and GPT may be due in part to their use of the transformer architecture. Unlike the BiLSTM used by the CoLA baseline, the transformer uses a self-attention mechanism that associates all pairs of words regardless of distance. In some cases models showed surprisingly good or bad performance, revealing possible idiosyncrasies of the sentence embeddings they output. For instance, the CoLA baseline performs on par with the others on the major feature ADJUNCT, especially considering the minor feature PARTICLE (Bo looked the word up).Furthermore, all models struggle equally with sentences in VIOLATION, indicating that the advantages of the transformer models over the CoLA baseline does not extend to the detection of morphological violations (INFL/AGR VIOLATION) or single word anomalies (EXTRA/MISSING EXPR). For comparison, we analyze the effect of sentence length on acceptability classifier performance. The are shown in FIG3. The for the CoLA baseline are inconsistent, but do drop off as sentence length increases. For BERT and GPT, performance decreases very steadily with length. Exceptions are extremely short sentences (length 1-3), which may be challenging due to insufficient information; and extremely long sentences, where we see a small (but somewhat unreliable) boost in BERT's performance. BERT and GPT are generally quite close in performance, except on the longest sentences, where BERT's performance is considerably better. Using a new grammatically annotated analysis set, we identify several syntactic phenomena that are predictive of good or bad performance of current state of the art sentence encoders on CoLA. We also use these to develop hypotheses about why BERT is successful, and why transformer models outperform sequence models. Our findings can guide future work on sentence embeddings. A current weakness of all sentence encoders we investigate, including BERT, is the identification of morphological violations. Future engineering work should investigate whether switching to a character-level model can mitigate this problem. Additionally, transformer models appear to have an advantage over sequence models with long-distance dependencies, but still struggle with these constructions relative to more local phenomena. It stands to reason that this performance gap might be widened by training larger or deeper transformer models, or training on longer or more complex sentences. This analysis set can be used by engineers interested in evaluating the syntactic knowledge of their encoders. Finally, these findings suggest possible controlled experiments that could confirm whether there is a causal relation between the presence of the syntactic features we single out as interesting and model performance. Our are purely correlational, and do not mark whether a particular construction is crucial for the acceptability of the sentence. Future experiments following and can semi-automatically generate datasets manipulating, for example, length of long-distance dependencies, inflectional violations, or the presence of interrogatives, while controlling for factors like sentence length and word choice, in order determine the extent to which these features impact the quality of sentence embeddings. Included a. John owns the book. b. Park Square has a festive air. c. *Herself likes Mary's mother. FORMULA0 Excluded a. Bill has eaten cake. b. I gave Joe a book. A. These are sentences including the verb be used predicatively. Also, sentences where the object of the verb is itself a predicate, which applies to the subject. Not included are auxiliary uses of be or other predicate phrases that are not linked to a subject by a verb. These sentences involve predication of a nonsubject argument by another non-subject argument, without the presence of a copula. Some of these cases may be analyzed as small clauses. (see , pp. 189-193) These are adjuncts modifying noun phrases. Adjuncts are (usually) optional, and they do not change the category of the expression they modify. Single-word prenominal adjectives are excluded, as are relative clauses (this has another category). These are adjuncts of VPs and NPs that specify a time or modify tense or aspect or frequency of an event. Adjuncts are (usually) optional, and they do not change the category of the expression they modify. These are adjuncts of VPs and NPs not described by some other category (with the exception of), i.e. not temporal, locative, or relative clauses. Adjuncts are (usually) optional, and they do not change the category of the expression they modify. Prepositional Phrase arguments of NPs or APs are individual-denoting arguments of a noun or adjective which are marked by a proposition. Arguments are selected for by the head, and they are (generally) not optional, though in some cases they may be omitted where they are understood or implicitly existentially quantified over. Prepositional arguments introduced with by. Usually, this is the (semantic) subject of a passive verb, but in rare cases it may be the subject of a nominalized verb. Arguments are usually selected for by the head, and they are generally not optional. In this case, the argument introduced with by is semantically selected for by the verb, but it is syntactically optional. See Adger (2003, p.190) and. (22 Expletives, or dummy arguments, are semantically inert arguments. The most common expletives in English are it and there, although not all occurrences of these items are expletives. Arguments are usually selected for by the head, and they are generally not optional. In this case, the expletive occupies a syntactic argument slot, but it is not semantically selected by the verb, and there is often a syntactic variation without the expletive. See Adger (2003, p.170-172) and Kim and Sells (2008, p.82-83 The passive voice is marked by the demotion of the subject (either complete omission or to a byphrase) and the verb appearing as a past participle. In the stereotypical construction there is an auxiliary be verb, though this may be absent. See Kim and Sells (2008, p.175-190) et al. (2013, p.163-186) and Sag et al. (2003, p.203-226). A.7.2 Binding:Other (Binding of Other Pronouns) These are cases in which a non-reflexive pronoun appears along with its antecedent. This includes donkey anaphora, quantificational binding, and bound possessives, among other bound pronouns. See Sportiche et al. (2013, p.163-186) and Sag et al. (2003, p.203-226). These are sentences in which the matrix clause is interrogative (either a wh-or polar question). See Adger (2003, pp.282-213), Kim and Sells (2008, pp.193-222), and Carnie (2013, p.315-350 Relative clauses are noun modifiers appearing with a relativizer (either that or a wh-word) and an associated gap. See Kim and Sells (2008, p.223-244

Represent the short summary for finding the corresponding scientific article (long)

We investigate the implicit syntactic knowledge of sentence embeddings using a new analysis set of grammatically annotated sentences with acceptability judgments.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR

Question: When considering simultaneously a finite number of tasks, multi-output learning enables one to account for the similarities of the tasks via appropriate regularizers. We propose a generalization of the classical setting to a continuum of tasks by using vector-valued RKHSs. Several fundamental problems in machine learning and statistics can be phrased as the minimization of a loss function described by a hyperparameter. The hyperparameter might capture numerous aspects of the problem: (i) the tolerance w. r. t. outliers as the -insensitivity in Support Vector Regression , (ii) importance of smoothness or sparsity such as the weight of the l 2 -norm in Tikhonov regularization , l 1 -norm in LASSO , or more general structured-sparsity inducing norms BID3, (iii) Density Level-Set Estimation (DLSE), see for example one-class support vector machines One-Class Support Vector Machine (OCSVM, Schölkopf et al., 2000), (iv) confidence as exemplified by Quantile Regression , or (v) importance of different decisions as implemented by Cost-Sensitive Classification . In various cases including QR, CSC or DLSE, one is interested in solving the parameterized task for several hyperparameter values. Multi-Task Learning provides a principled way of benefiting from the relationship between similar tasks while preserving local properties of the algorithms: ν-property in DLSE or quantile property in QR .A natural extension from the traditional multi-task setting is to provide a prediction tool being able to deal with any value of the hyperparameter. In their seminal work, extended multi-task learning by considering an infinite number of parametrized tasks in a framework called Parametric Task Learning (PTL). Assuming that the loss is piecewise affine in the hyperparameter, the authors are able to get the whole solution path through parametric programming, relying on techniques developed by.In this paper 1, we relax the affine model assumption on the tasks as well as the piecewise-linear assumption on the loss, and take a different angle. We propose Infinite Task Learning (ITL) within the framework of functionvalued function learning to handle a continuum number of parameterized tasks using Vector-Valued Reproducing Kernel Hilbert Space (vv-). After introducing a few notations, we gradually define our goal by moving from single parameterized tasks to ITL through multi-output learning. A supervised parametrized task is defined as follows. Let (X, Y) ∈ X × Y be a random variable with joint distribution P X,Y which is assumed to be fixed but unknown; we also assume that Y ⊂ R. We have access to n independent identically distributed (i. i. d.) observations called training samples: S:=((x i, y i)) n i=1 ∼ P ⊗n X,Y. Let Θ be the domain of hyperparameters, and v θ: Y × Y → R be a loss function associated to θ ∈ Θ. Let H ⊂ F (X ; Y) denote our hypothesis class; throughout the paper H is assumed to be a Hilbert space with inner product ·, · H. For a given θ, the goal is to estimate the minimizer of the expected risk DISPLAYFORM0 over H, using the training sample S. This task can be addressed by solving the regularized empirical risk minimization problem DISPLAYFORM1 where R θ S (h):= 1 n n i=1 v θ (y i, h(x i)) is the empirical risk and Ω: H → R is a regularizer. Below we give two examples. Quantile Regression: In this setting θ ∈. For a given hyperparameter θ, in Quantile Regression the goal is to predict the θ-quantile of the real-valued output conditional distribution P Y |X. The task can be tackled using the pinball loss defined in Eq.. DISPLAYFORM2 Density Level-Set Estimation: Examples of parameterized tasks can also be found in the unsupervised setting. For instance in outlier detection, the goal is to separate outliers from inliers. A classical technique to tackle this task is OCSVM (Schölkopf et al., 2000). OCSVM has a free parameter θ ∈, which can be proven to be an upper bound on the fraction of outliers. This unsupervised learning problem can be empirically described by the minimization of a regularized empirical risk R S θ (h, t) + Ω(h), solved jointly over h ∈ H and t ∈ R with DISPLAYFORM3 In the aforementioned problems, one is rarely interested in the choice of a single hyperparameter value (θ) and associated risk R S θ, but rather in the joint solution of multiple tasks. The naive approach of solving the different tasks independently can easily lead to inconsistencies. A principled way of solving many parameterized tasks has been cast as a MTL problem which takes into account the similarities between tasks and helps providing consistent solutions. For example it is possible to encode the similarities of the different tasks in MTL through an explicit constraint function . In the current work, the similarity between tasks is designed in an implicit way through the loss function and the use of a kernel on the hyperparameters. Moreover, in contrast to MTL, in our case the input space and the training samples are the same for each task; a task is specified by a value of the hyperparameter. This setting is sometimes refered to as multi-output learning (Álvarez et al., 2012).Formally, assume that we have p tasks described by parameters (θ j) p j=1. The idea of multi-task learning is to minimize the sum of the local loss functions R S θj, i. e.arg min DISPLAYFORM4 where the individual tasks are modelled by the real-valued h j functions, the overall R p -valued model is the vectorvalued function h: x → (h 1 (x),..., h p (x)), and Ω is a regularization term encoding similarities between tasks. Such approaches have been developed in for QR and in for DLSE.Learning a continuum of tasks: In the following, we propose a novel framework called Infinite Task Learning in which we learn a function-valued function h ∈ F (X ; F (Θ; Y)). Our goal is to be able to handle new tasks after the learning phase and thus, not to be limited to given predefined values of the hyperparameter. Regarding this goal, our framework generalizes the Parametric Task Learning approach introduced by , by allowing a wider class of models and relaxing the hypothesis of piece-wise linearity of the loss function. Moreover a nice byproduct of this vv-RKHS based approach is that one can benefit from the functional point of view, design new regularizers and impose various constraints on the whole continuum of tasks, e. g.,• The continuity of the θ → h(x)(θ) function is a natural desirable property: for a given input x, the predictions on similar tasks should also be similar.• Another example is to impose a shape constraint in QR: the conditional quantile should be increasing w. r. t. the hyperparameter θ. This requirement can be imposed through the functional view of the problem.• In DLSE, to get nested level sets, one would want that for all x ∈ X, the decision function θ → 1 R+ (h(x)(θ) − t(θ)) changes its sign only once. To keep the presentation simple, in the sequel we are going to focus on ITL in the supervised setting; unsupervised tasks can be handled similarly. Assume that h belongs to some space H ⊆ F (X ; F (Θ; Y)) and introduce an integrated loss function DISPLAYFORM5 where the local loss v: Θ × Y × Y → R denotes v θ seen as a function of three variables including the hyperparameter and µ is a probability measure on Θ which encodes the importance of the prediction at different hyperparameter values. Without prior information and for compact Θ, one may consider µ to be uniform. The true risk reads then DISPLAYFORM6 Intuitively, minimizing the expectation of the integral over θ in a rich enough space corresponds to searching for a pointwise minimizer x → h * (x)(θ) of the parametrized tasks introduced in Eq. with, for instance, the implicit space constraint that θ → h * (x)(θ) is a continuous function for each input x. We show in Proposition S.4.1 that this is precisely the case in QR.Interestingly, the empirical counterpart of the true risk minimization can now be considered with a much richer family of penalty terms: DISPLAYFORM7 Here, Ω(h) can be a weighted sum of various penalties as seen in Section 3. Many different models (H) could be applied to solve this problem. In our work we consider Reproducing Kernel Hilbert Spaces as they offer a simple and principled way to define regularizers by the appropriate choice of kernels and exhibit a significant flexibility. This section is dedicated to solving the ITL problem defined in Eq.. We first focus on the objective (V), then detail the applied vv-RKHS model family with various penalty examples, followed by representer theorems which give rise to computational tractability. In practice solving Eq. can be rather challenging due to the integral over θ. One might consider different numerical integration techniques to handle this issue. We focus here on Quasi Monte Carlo (QMC) methods as they allow (i) efficient optimization over vv-RKHSs which we will use for modelling H (Proposition 3.1), and (ii) enable us to derive generalization guarantees (Proposition 3.3). Indeed, let DISPLAYFORM0 be the QMC approximation of Eq.. Let w j = m −1 F −1 (θ j), and (θ j) m j=1 be a sequence with values in d such as the Sobol or Halton sequence where µ is assumed to be absolutely continuous w. r. t. the Lebesgue measure and F is the associated cdf. Using this notation and the training samples S = ((x i, y i)) n i=1, the empirical risk takes the form DISPLAYFORM1 and the problem to solve is DISPLAYFORM2 Hypothesis class (H): Recall that H ⊆ F (X ; F (Θ; Y)), in other words h(x) is a Θ → Y function for all x ∈ X. In this work we assume that the Θ → Y mapping can be described by an RKHS H kΘ associated to a k Θ: Θ × Θ → R scalar-valued kernel defined on the hyperparameters. Let k X: X × X → R be a scalar-valued kernel on the input space. The x → (hyperparameter → output) relation, i. e. h: X → H kΘ is then modelled by the Vector-Valued Reproducing Kernel Hilbert Spa- DISPLAYFORM3 where the operator-valued kernel K is defined as K(x, z) = k X (x, z)I, and I = I H k Θ is the identity operator on H kΘ.This so-called decomposable Operator-Valued Kernel has several benefits and gives rise to a function space with a well-known structure. One can consider elements h ∈ H K as mappings from X to H kΘ, and also as functions from (X × Θ) to R. It is indeed known that there is an isometry between H K and H k X ⊗ H kΘ, the RKHS associated to the product kernel k X ⊗ k Θ. The equivalence between these views allows a great flexibility and enables one to follow a functional point of view (to analyse statistical aspects) or to leverage the tensor product point of view (to design new kind of penalization schemes). Below we detail various regularizers before focusing on the representer theorems.• Ridge Penalty: For QR, a natural regularization is the squared vv-RKHS norm DISPLAYFORM4 This choice is amenable to excess risk analysis (see Proposition 3.3). It can be also seen as the counterpart of the classical (multi-task regularization term introduced by Sangnier et al. FORMULA0, compatible with an infinite number of tasks. · 2 H K acts by constraining the solution to a ball of a finite radius within the vv-RKHS, whose shape is controlled by both k X and k Θ .• L 2,1 -penalty: For DLSE, it is more adequate to apply an L 2,1 -RKHS mixed regularizer: DISPLAYFORM5 which is an example of a Θ-integrated penalty. This Ω choice allows the preservation of the θ-property (see Fig. S. 3), i. e. that the proportion of the outliers is θ.• Shape Constraints: Taking the example of QR it is advantageous to ensure the monotonicity of the estimated quantile function Let ∂ Θ h denotes the derivative of h(x)(θ) with respect to θ. Then one should solve arg min DISPLAYFORM6 However, the functional constraint prevents a tractable optimization scheme. To mitigate this bottleneck, we penalize if the derivative of h w. r. t. θ is negative: DISPLAYFORM7 When P:=P X this penalization can rely on the same anchors and weights as the ones used to approximate the integrated loss function: DISPLAYFORM8 Thus, one can modify the overall regularizer in QR to be DISPLAYFORM9 Representer Theorems: Apart from the flexibility of regularizer design, the other advantage of applying vv-RKHS as hypothesis class is that it gives rise to finite-dimensional representation of the ITL solution under mild conditions. Proposition 3.1 (Representer). Assume that for ∀θ ∈ Θ, v θ is a proper lower semicontinuous convex function with respect to its second argument. Then DISPLAYFORM10 with Ω(h) defined as in Eq. FORMULA0, has a unique solution h *, and DISPLAYFORM11 For DLSE, we similarly get a representer theorem with the following modelling choice. Let DISPLAYFORM12 2 Then, learning a continuum of level sets boils down to the minimization problem arg min DISPLAYFORM13 where DISPLAYFORM14 Remarks:• Relation to Joint Quantile Regression (JQR): In Infinite Quantile Regression (∞-QR), by choosing k Θ to be the (JQR) framework as a special case of our approach. In contrast to the JQR, however, in ∞-QR one can predict the quantile value at any θ ∈, even outside the (θ j) m j=1 used for learning. DISPLAYFORM15 • Relation to q-OCSVM: In DLSE, by choosing k Θ (θ, θ) = 1 (for all θ, θ ∈ Θ) to be the constant kernel, DISPLAYFORM16 δ θj, our approach specializes to q-OCSVM .• Relation to: Note that Operator-Valued Kernels for functional outputs have also been used in , under the form of integral operators acting on L 2 spaces. Both kernels give rise to the same space of functions; the benefit of our approach being to provide an exact finite representation of the solution (see Proposition 3.1).• Efficiency of the decomposable kernel: this kernel choice allows to rewrite the expansions in Propositions 3.1 and 3.2 as a Kronecker products and the complexity of the prediction of n points for m quantile becomes DISPLAYFORM17 Excess Risk Bounds: Below we provide a generalization error analysis to the solution of Eq. FORMULA10 for QR (with Ridge regularization and without shape constraints) by stability argument BID4, extending the work of BID2 to Infinite-Task Learning. The proposition (finite sample bounds are given in Corollary S.5.6) instantiates the guarantee for the QMC scheme. Proposition 3.3 (Generalization). Let h * ∈ H K be the solution of Eq. for the QR problem with QMC approximation. Under mild conditions on the kernels k X, k Θ and P X,Y, stated in the supplement, one has DISPLAYFORM18 (n, m) Trade-off: The proposition reveals the interplay between the two approximations, n (the number of training samples) and m (the number of locations taken in the integral approximation), and allows to identify the regime in λ = λ(n, m) driving the excess risk to zero. Indeed by choosing m = √ n and discarding logarithmic factors for simplicity, λ n −1 is sufficient. The mild assumptions imposed are: boundedness on both kernels and the random variable Y, as well as some smoothness of the kernels. Numerical Experiments: The efficiency of the ITL scheme for QR has been tested on several benchmarks; the are summarized in Table S.1 for 20 real datasets from the UCI repository. An additional experiment concerning the non-crossing property on a synthetic dataset can be found in Fig. S Let us recall the expression of the pinball loss: DISPLAYFORM19 Proposition S.4.1. Let X, Y be two random variables (r. v.s) respectively taking values in X and R, and q: X → F(, R) the associated conditional quantile function. Let µ be a positive measure on such that DISPLAYFORM20 where R is the risk defined in Eq..Proof. The proof is based on the one given in for a single quantile. Let f ∈ F (X ; F (; R)), θ ∈ and (x, y) ∈ X × R. Let also DISPLAYFORM21 Then, notice that DISPLAYFORM0 and since q is the true quantile function, DISPLAYFORM1 Moreover, (t − s) is negative when q(x)(θ) ≤ y ≤ h(x)(θ), positive when h(x)(θ) ≤ y ≤ q(x)(θ) and 0 otherwise, thus the quantity (t − s)(y − h(x)(θ)) is always positive. As a consequence, DISPLAYFORM2 There are several ways to solve the non-smooth optimization problems associated to the QR, DLSE and CSC tasks. One could proceed for example by duality-as it was done in -, or apply sub-gradient descent techniques (which often converge quite slowly). In order to allow unified treatment and efficient solution in our experiments we used the L-BFGS-B optimization scheme which is widely popular in large-scale learning, with non-smooth extensions (; Keskar & Wächter, 2017). The technique requires only evaluation of objective function along with its gradient, which can be computed automatically using reverse mode automatic differentiation (as in BID0). To benefit from from the available fast smooth implementations , we applied an infimal convolution on the non-differentiable terms of the objective. Under the assumption that m = O(√ n) (see Proposition 3.3), the complexity per L-BFGS-B iteration is O(n 2 √ n).The efficiency of the non-crossing penalty is illustrated in Fig. S.2 on a synthetic sine wave dataset where n = 40 and m = 20 points have been generated. Many crossings are visible on the right plot, while they are almost not noticible on the left plot, using the non-crossing penalty. Concerning our real-world examples (20 UCI datasets), to study the efficiency of the proposed scheme in quantile regression the following experimental protocol was applied. Each dataset was splitted randomly into a training set (70%) and a test set (30%). We optimized the hyperparameters by minimizing a 5-folds cross validation with a Bayesian optimizer 3. Once the hyperparameters were obtained, a new regressor was learned on the whole training set using the optimized hyperparameters. We report the value of the pinball loss and the crossing loss on the test set for three methods: our technique is called ∞-QR, we refer to's approach as JQR, and independent learning (abbreviated as IND-QR) represents a further baseline. We repeated 20 simulations (different random training-test splits); the are also compared using a Mann-WhitneyWilcoxon test. A summary is provided in Table S.1.Notice that while JQR is tailored to predict finite many quantiles, our ∞-QR method estimates the whole quantile function hence solves a more challenging task. Despite the more difficult problem solved, as Table S.1 suggest that the performance in terms of pinball loss of ∞-QR is comparable to that of the state-of-the-art JQR on all the twenty studied benchmarks, except for the'crabs' and'cpus' datasets (p.-val. < 0.25%). In addition, when considering the non-crossing penalty one can observe that ∞-QR outperforms the IND-QR baseline on eleven datasets (p.-val. < 0.25%) and JQR on two datasets. This illustrates the efficiency of the constraint based on the continuum scheme. The analysis of the generalization error will be performed using the notion of uniform stability introduced in BID4. For a derivation of generalization bounds in vv-RKHS, we refer to . In their framework, the goal is to minimize a risk which can be expressed as DISPLAYFORM0 where S = ((x 1, y 1),..., (x n, y n)) are i. i. d. inputs and λ > 0. We almost recover their setting by using losses defined as DISPLAYFORM1 where V is a loss associated to some local cost defined in Eq.. Then, they study the stability of the algorithm which, given a dataset S, returns DISPLAYFORM2 There is a slight difference between their setting and ours, since they use losses defined for some y in the output space of the vv-RKHS, but this difference has no impact on the validity of the proofs in our case. The use of their theorem requires some assumption that are listed below. We recall the shape of the OVK we use: DISPLAYFORM3, where k X and k Θ are both bounded scalar-valued kernels, in other words there exist (κ X, κ Θ) ∈ R 2 such that sup DISPLAYFORM4 Remark 1. Assumptions 1, 2 are satisfied for our choice of kernel. Assumption 3. The application (y, h, x) → (y, h, x) is σ-admissible, i. e. convex with respect to f and Lipschitz continuous with respect to f (x), with σ as its Lipschitz constant. Assumption 4. ∃ξ ≥ 0 such that ∀(x, y) ∈ X × Y and ∀S training set, (y, h * S, x) ≤ ξ. Definition S.5.1. Let S = ((x i, y i)) n i=1 be the training data. We call S i the training data S i = ((x 1, y 1 DISPLAYFORM5 Definition S.5.2. A learning algorithm mapping a dataset S to a function h * S is said to be β-uniformly stable with respect to the loss function if ∀n ≥ 1, ∀1 ≤ i ≤ n, ∀S training set, DISPLAYFORM6 Proposition S.5.1. BID4 Let S → h * S be a learning algorithm with uniform stability β with respect to a loss satisfying Assumption 4. Then ∀n ≥ 1, ∀δ ∈, with probability at least 1 − δ on the drawing of the samples, it holds that DISPLAYFORM7 Proposition S. Quantile Regression: We recall that in this setting, v(θ, y, h(x)(θ)) = max (θ(y − h(x)(θ)), (1 − θ)(y − h(x)(θ))) and the loss is DISPLAYFORM8 Moreover, we will assume that |Y | is bounded by B ∈ R as a r. v.. We will therefore verify the hypothesis for y ∈ [−B, B] and not y ∈ R.Lemma S.5.3. In the case of the QR, the loss is σ-admissible with σ = 2κ Θ.Proof. Let h 1, h 2 ∈ H K and θ ∈. ∀x, y ∈ X × R, it holds that DISPLAYFORM9 where s = 1 y≤h1(x)(θ) and t = 1 y≤h2(x)(θ). We consider all possible cases for t and s: DISPLAYFORM10 | because of the conditions on t, s. DISPLAYFORM11 By summing this expression over the (θ j) m j=1, we get that DISPLAYFORM12 and is σ-admissible with σ = 2κ Θ.Lemma S.5.4. Let S = ((x 1, y 1),..., (x n, y n)) be a training set and λ > 0. Then ∀x, θ ∈ X ×, it holds that DISPLAYFORM13 Proof. Since h * S is the output of our algorithm and 0 ∈ H K, it holds that DISPLAYFORM14 Lemma S.5.5. Assumption 4 is satisfied for ξ = 2 B + κ X κ Θ B λ.Proof. Let S = ((x 1, y 1),..., (x n, y n)) be a training set and h * S be the output of our algorithm. DISPLAYFORM15 Corollary S.5.6. The QR learning algorithm defined in Eq. is such that ∀n ≥ 1, ∀δ ∈, with probability at least 1 − δ on the drawing of the samples, it holds that DISPLAYFORM16 Proof. This is a direct consequence of Proposition S.5.2, Proposition S.5.1, Lemma S.5.3 and Lemma S.5.5.Definition S.5.3 (Hardy-Krause variation). Let Π be the set of subdivisions of the interval Θ =. A subdivision will be denoted σ = (θ 1, θ 2, . . ., θ p) and f: Θ → R be a function. We call Hardy-Krause variation of the function f the quantity sup DISPLAYFORM17 Remark 2. If f is continuous, V (f) is also the limit as the mesh of σ goes to zero of the above quantity. In the following, let f: DISPLAYFORM18 This function is of primary importance for our analysis, since in the Quasi Monte-Carlo setting, the bound of Proposition 3.3 makes sense only if the function f has finite Hardy-Krause variation, which is the focus of the following lemma. Lemma S.5.7. Assume the boundeness of both scalar kernels k X and k Θ. Assume moreover that k Θ is C 1 and that its partial derivatives are uniformly bounded by some constant C. Then DISPLAYFORM19 Proof. It holds that DISPLAYFORM20 The supremum of the integral is smaller than the integral of the supremum, as such DISPLAYFORM21 where f x,y: θ → v(θ, y, h * S (x)(θ)) is the counterpart of the function f at point (x, y). To bound this quantity, let us first bound locally V (f x,y). To that extent, we fix some (x, y) in the following. Since f x,y is continuous (because k Θ is C 1), then using Choquet (1969, Theorem 24.6), it holds that DISPLAYFORM22 Moreover since k ∈ C 1 and ∂k θ = (∂ 1 k)(·, θ) has a finite number of zeros for all θ ∈ ×, one can assume that in the subdivision considered afterhand all the zeros (in θ) of the residuals y − h * S (x)(θ) are present, so that y − h * S (x)(θ i+1) and y − h * S (x)(θ i) are always of the same sign. Indeed, if not, create a new, finer subdivision with this property and work with this one. Let us begin the proper calculation: let σ = (θ 1, θ 2, . . ., θ p) be a subdivision of Θ, it holds that ∀i ∈ {1, . . ., p − 1}: DISPLAYFORM23 We now study the two possible outcomes for the residuals: DISPLAYFORM24 Since k Θ is C 1, with partial derivatives uniformly bounded by C, |k Θ (θ i+1, θ i+1) − k Θ (θ i+1, θ i)| ≤ C(θ i+1 − θ i) and |k Θ (θ i, θ i) − k Θ (θ i+1, θ i)| ≤ C(θ i+1 − θ i) so that |h * S (x)(θ i) − h * S (x)(θ i+1)| ≤ κ X 2BC λ θ i+1 − θ i and overall DISPLAYFORM25 • If y − h(x)(θ i+1) ≤ 0 and y − h(x)(θ i) ≤ 0 then |f x,y (θ i+1) − f x,y (θ i)| = |(1 − θ i+1)(y − h * S (x)(θ i+1)) − (1 − θ i)(y − h * S (x)(θ i))| ≤ |h * S (x)(θ i) − h * S (x)(θ i+1)| + |(θ i+1 − θ i)y| + |(θ i − θ i+1)h * S (x)(θ i+1)| + |θ i (h * S (x)(θ i) − h * S (x)(θ i+1))| so that with similar arguments one gets DISPLAYFORM26 Therefore, regardless of the sign of the residuals y − h(x)(θ i+1) and y − h(x)(θ i), one gets Eq.. Since the square root function has Hardy-Kraus variation of 1 on the interval Θ =, it holds that DISPLAYFORM27 Combining this with Eq. finally gives DISPLAYFORM28 Lemma S.5.8. Let R be the risk defined in Eq. for the quantile regression problem. Assume that (θ) m j=1 have been generated via the Sobol sequence and that k Θ is C 1 and that its partial derivatives are uniformly bounded by some constant C. Then Proof of Proposition 3.3. Combine Lemma S.5.8 and Corollary S.5.6 to get an asymptotic behaviour as n, m → ∞. To assess the quality of the estimated model by ∞-OCSVM, we illustrate the θ-property (Schölkopf et al., 2000): the proportion of inliers has to be approximately 1 − θ (∀θ ∈). For the studied datasets (Wilt, Spambase) we used the raw inputs without applying any preprocessing. Our input kernel was the exponentiated χ 2 kernel k X (x, z):= exp −γ X d k=1 (x k − z k) 2 /(x k + z k) with bandwidth γ X = 0.25. A Gauss-Legendre quadrature rule provided the integral approximation in Eq. FORMULA8, with m = 100 samples. We chose the Gaussian kernel for k Θ; its bandwidth parameter γ Θ was the 0.2−quantile of the pairwise Euclidean distances between the θ j's obtained via the quadrature rule. The margin (bias) kernel was k b = k Θ. As it can be seen in FIG6, the θ-property holds for the estimate which illustrates the efficiency of the proposed continuum approach for density level-set estimation.

represent the short summary for finding the corresponding scientific article (long)

We propose an extension of multi-output learning to a continuum of tasks using operator-valued kernels.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.