Question: represent the next text

Question: We develop new approximation and statistical learning theories of convolutional neural networks (CNNs) via the ResNet-type structure where the channel size, filter size, and width are fixed. It is shown that a ResNet-type CNN is a universal approximator and its expression ability is no worse than fully-connected neural networks (FNNs) with a \textit{block-sparse} structure even if the size of each layer in the CNN is fixed. Our is general in the sense that we can automatically translate any approximation rate achieved by block-sparse FNNs into that by CNNs. Thanks to the general theory, it is shown that learning on CNNs satisfies optimality in approximation and estimation of several important function classes. As applications, we consider two types of function classes to be estimated: the Barron class and H\"older class. We prove the clipped empirical risk minimization (ERM) estimator can achieve the same rate as FNNs even the channel size, filter size, and width of CNNs are constant with respect to the sample size. This is minimax optimal (up to logarithmic factors) for the H\"older class. Our proof is based on sophisticated evaluations of the covering number of CNNs and the non-trivial parameter rescaling technique to control the Lipschitz constant of CNNs to be constructed. Convolutional Neural Network (CNN) is one of the most popular architectures in deep learning research, with various applications such as computer vision , natural language processing , and sequence analysis in bioinformatics (,). Despite practical popularity, theoretical justification for the power of CNNs is still scarce from the viewpoint of statistical learning theory. For fully-connected neural networks (FNNs), there is a lot of existing work, dating back to the 80's, for theoretical explanation regarding their approximation ability (, , , , and) and generalization power (, , and). See also and for surveys of earlier works. Although less common compared to FNNs, recently, statistical learning theory for CNNs has been studied, both about approximation ability (, ,) and about generalization power . One of the standard approaches is to relate the approximation ability of CNNs with that of FNNs, either deep or shallow. For example, proved that CNNs are a universal approximator of the Barron class (,), which is a historically important function class in the approximation theory. Their approach is to approximate the function using a 2-layered FNN (i.e., an FNN with a single hidden layer) with the ReLU activation function and transform the FNN into a CNN. Very recently independent of ours, showed any function realizable with an FNN can extend to an equivariant function realizable by a CNN that has the same order of parameters. However, to the best of our knowledge, no CNNs that achieves the minimax optimal rate (, Giné &) in important function classes, including the Hölder class, can keep the number of units in each layer constant with respect to the sample size. Architectures that have extremely large depth, while moderate channel size and width have become feasible, thanks to recent methods such as identity mappings (,), sophisticated initialization schemes (,), and normalization techniques (,). Therefore, we would argue that there are growing demands for theories which can accommodate such constant-size architectures. In this paper, we analyze the learning ability of ResNet-type ReLU CNNs which have identity mappings and constant-width residual blocks with fixed-size filters. There are mainly two reasons that motivate us to study this type of CNNs. First, although ResNet is the de facto architecture in various practical applications, the approximation theory for ResNet has not been explored extensively, especially from the viewpoint of the relationship between FNNs and CNNs. Second, constant-width CNNs are critical building blocks not only in ResNet but also in various modern CNNs such as Inception , DenseNet , and U-Net , to name a few. Our strategy is to replicate the learning ability of FNNs by constructing tailored ResNet-type CNNs. To do so, we pay attention to the block-sparse structure of an FNN, which roughly means that it consists of a linear combination of multiple (possibly dense) FNNs (we define it rigorously in the subsequent sections). Block-sparseness decreases the model complexity coming from the combinatorial sparsity patterns and promotes better bounds. Therefore, it is often utilized, both implicitly or explicitly, in the approximation and learning theory of FNNs (e.g., Bölcskei et al., ). We first prove that if an FNN is block-sparse with M blocks (M -way block-sparse FNN), we can realize the FNN with a ResNet-type CNN with O(M) additional parameters, which are often negligible since the original FNN already has Ω(M) parameters. Using this approximation, we give the upper bound of the estimation error of CNNs in terms of the approximation errors of block sparse FNNs and the model complexity of CNNs. Our is general in the sense that it is not restricted to a specific function class, as long as we can approximate it using block-sparse FNNs. To demonstrate the wide applicability of our methods, we derive the approximation and estimation errors for two types of function classes with the same strategy: the Barron class (of parameter s = 2) and Hölder class. We prove, as corollaries, that our CNNs can achieve the approximation error of orderÕ(M) for the β-Hölder class, where M is the number of parameters (we used M here, same as the number of blocks because it will turn out that CNNs have O(M) blocks for these cases), N is the sample size, and D is the input dimension. These rates are same as the ones for FNNs ever known in the existing literature. An important consequence of our theory is that the ResNet-type CNN can achieve the minimax optimal estimation error (up to logarithmic factors) for β-Hölder class even if its filter size, channel size and width are constant with respect to the sample size, as opposed to existing works such as and , where optimal FNNs or CNNs could have a width or a channel size goes to infinity as N → ∞.In summary, the contributions of our work are as follows:• We develop the approximation theory for CNNs via ResNet-type architectures with constant-width residual blocks. We prove any M -way block-sparse FNN is realizable such a CNN with O(M) additional parameters. That means if FNNs can approximate a function with O(M) parameters, we can approximate the function with CNNs at the same rate (Theorem 1).• We derive the upper bound of the estimation error in terms of the approximation error of FNNs and the model complexity of CNNs (Theorem 2). This gives the sufficient conditions to derive the same estimation error as that of FNNs (Corollary 1).• We apply our general theory to the Barron class and Hölder class and derive the approximation (Corollary 2 and 4) and estimation (Corollary 3 and 5) error rates, which are identical to those for FNNs, even if the CNNs have constant channel and filter size with respect to the sample size. In particular, this is minimax optimal for the Hölder case. We summarize in Table 1 the differences in the CNN architectures between our work and and , which established the approximation theory of CNNs via FNNs. First and foremost, only considered a specific function class -the Barron class -as a target function class, although their method is applicable to any function class that can be realized by a 2-layered ReLU FNN. Regarding the architecture, they considered CNNs with a single channel and whose width is "linearly increasing" layer by layer. For regression or classification problems, it is rare to use such an architecture. In addition, since they did not bound the norm of parameters in the approximating CNNs, we cannot derive the estimation error from this method. fully utilized the group invariance structure of underlying input spaces to construct CNNs. Such a structure makes theoretical analysis easier, especially for investigating the equivariance properties of CNNs since it enables us to incorporate mathematical tools such as group theory, Fourier analysis, and representation theory. Although their are quite strong in that it is applicable to any function that can be approximated by FNNs, their assumption on the group structure excludes the padding convolution layer, an important and popular type of convolution operations. Another point is that if we simply apply their construction method to derive the estimation error for (equivariant) Hölder functions, combined with the approximation of , the ing CNN that achieves the minimax optimal rate hasÕ(ε − D β) channels where ε is the approximation error threshold. It is partly because their construction is not aware of the internal sparse structure of approximating FNNs. Finally, the filter size of their CNN is as large as the input dimension. As opposed to these two works, we employ padding-and ResNettype CNNs which have multiple channels, fixed-size filters, and constant widths. , our is applicable to any function, as long as the FNNs to be approximated are block sparse, including the Barron and Hölder cases. If we apply our theorem to these classes, we can show that the optimal CNNs can achieve the same approximation and estimation rate as FNNs, while the number of channels is independent of the sample size. Further, this is minimax optimal up to the logarithmic factors for the Hölder class. Due to its practical success, theoretical analysis for ResNet has been explored recently (e.g., , , , and). From the viewpoint of statistical learning theory, and investigated the generalization power of ResNet from the perspective of the boosting interpretation. However, they did not discuss the function approximation ability of ResNet. To the best of our knowledge, our theory is the first work to provide the approximation ability of the CNN class that can accommodate the ResNet-type ones. We import the approximation theories for FNNs, especially ones for the Barron class and Hölder class. The approximation theory for the Barron class has been investigated in e.g., , , and. considered the parameter s = 1 (see Definition 3) and the activation function σ satisfying σ(z) → 1 as z → ∞ and σ(z) → 0 as z → −∞. Using this bound, proved that the estimation error of the ERM estimator is O(N − 2β 2β+D), which is minimax optimal up to logarithmic factors (see, e.g.,). We consider a regression task in this paper. Let X be a [−1, 1] D -valued random variable with unknown probability distribution P X and ξ be an independent random noise drawn from the Gaussian distribution with an unknown variance DISPLAYFORM0 • rigorously in the theorems later). We define a random variable Y by Y:= f• (X) + ξ. We denote the joint distribution of (X, Y) by P. Suppose we are given a dataset D = ((x 1, y 1),..., (x N, y N)) independently and identically sampled from the distribution P, we want to estimate the true function f• from the finite dataset D.We evaluate the performance of an estimator by the squared error. For a measurable function f: DISPLAYFORM1 Here, clip is the clipping operator defined DISPLAYFORM2 we define the L 2 -norm (weighted by P X) and the sup norm of f by DISPLAYFORM3 The task is to estimate the approximation error min f ∈F f − f • ∞ and the estimation error of the clipped ERM estimator: R(f) − R(f •). Note that the estimation error is a random variable with respect the choice of the training dataset D. By the definition of R and the independence of X and ξ, the estimation error equals to f − f DISPLAYFORM4 In this section, we define CNNs used in this paper. For this purpose, it is convenient to introduce 0, the set of real-valued sequences whose finitely many elements are non-zero: 0:= {w = (w n) n∈N>0 | ∃N ∈ N >0 s.t. w n = 0, ∀n ≥ N }. w = (w 1, . . ., w K) ∈ R K can be regarded as an element of 0 by setting w n = 0 for all n > K. Likewise, for C, C ∈ N >0, which will be the input and output channel sizes, respectively, we can think of (w k,j,i DISPLAYFORM0, we define the one-sided padding and stride-one convolution by w as an order-4 tensor DISPLAYFORM1 Here, i (resp. j) runs through 1 to C (resp. C) and α and β runs through 1 to D. Since we fix the input dimension D throughout the paper, we will omit the subscript D and write as L w if it is obvious from context. DISPLAYFORM2 Using this equality, we can expand a size-K filter to size-K.We can interpret L w as a linear mapping from DISPLAYFORM3 Next, we define the building block of CNNs: convolutional layers and fully-connected layers. Let C, C, K ∈ N >0 be the input channel size, output channel size, and filter size, respectively. For a weight tensor w ∈ R K×C ×C, a bias vector b ∈ R C, and an activation function σ: R → R, we define the convolutional layer Conv DISPLAYFORM4 where, ⊗ is the outer product of vectors and σ is applied in element-wise manner. Similarly, let W ∈ R D×C, b ∈ R, and σ: R → R, we define the fully-connected layer FC DISPLAYFORM5 is the vectorization operator that flattens a matrix into a vector. Finally, we define the ResNet-type CNN as a sequential concatenation of one convolution block, M residual blocks, and one fully-connected layer. FIG2 is the schematic view of the CNN we adopt in this paper. Definition 1 (Convolutional Neural Networks (CNNs)). Let M ∈ N >0 and L m ∈ N >0, which will be the number of residual blocks and the depth of m-th block, respectively. Let C DISPLAYFORM6 m ∈ R be the weight tensors and biases of l-th layer of the m-th block in the convolution part, respectively. Finally, let W ∈ R D×C (L 0) 0 and b ∈ R be the weight matrix and the bias for the fully-connected layer part, respectively. For θ:= ((w DISPLAYFORM7 is the identity function. Although CNN σ θ in this definition has a fully-connected layer, we refer to the stack of convolutional layers both with or without the final fully-connect layer as a CNN in this paper. We say a linear convolutional layer or a linear CNN when the activation function σ is the identity function and a ReLU convolution layer or a ReLU CNN when σ is ReLU defined by ReLU(x) = x ∨ 0. We borrow the term from ResNet and call Conv For architecture parameters C = (C DISPLAYFORM8, and norm parameters for convolution layers B (conv) > 0 and for fully-connected layers DISPLAYFORM9, the hypothesis class consisting of ReLU CNNs, as follows: DISPLAYFORM10 Here, the domain of CNNs is restricted to [−1, 1] D. Note that we impose norm constraints to the convolution part and fully-connected part separately. We emphasize that we do not impose any sparse constraints (e.g., restricting the number of non-zero parameters in a CNN to some fixed value) to F (CNN), as opposed to previous literature such as Yarotsky FORMULA24, , and. Since the notation is cluttered, we sometimes omit the subscripts as we do in the above. Remark 2. In this paper, we adopted one-sided padding, which is not often used practically, in order to make proofs simple. However, with slight modifications, all statements are true for equallypadded convolutions, the widely employed padding style which adds (approximately) same numbers of zeros to both ends of an input signal, with the exception that the filter size K is restricted to DISPLAYFORM11 We also discuss our design choice, especially the comparison with the original ResNet proposed in in Section G of the appendix. In this section, we mathematically define FNNs we consider in this paper, in parallel with the CNN case. Our FNN, which we coin a block-sparse FNN, consists of M possibly dense FNNs (blocks) concatenated in parallel, followed by a single fully-connected layer. We sketch the architecture of a block-sparse FNN in Figure 2. Let DISPLAYFORM0 m ∈ R be the weight matrix and the bias of the l-th layer of mth block (with the convention DISPLAYFORM1 be the weight (sub)vector of the final fully-connected layer corresponding to the m-th block and b ∈ R be the bias for the last layer. DISPLAYFORM2 We call a block-sparse FNN with M blocks a M -way block-sparse FNN. We say θ is compatible with (D [Lm] and norm parameters for the block part B (bs) > 0 and for the final layer DISPLAYFORM3 DISPLAYFORM4, the set of function realizable by FNNs: DISPLAYFORM5 Again, the domain is restricted to [−1, 1] D. Similar to the CNN case, we sometimes remove subscripts of the function class for simplicity. With the preparation in the previous sections, we state our main of this paper. We only describe statements of theorems and corollaries and key ideas in the main article. All complete proofs are deferred to the appendix. Our first main theorem claims that any M -way block-sparse FNN is realizable by a ResNet-type CNN with fixed-sized channels and filters by adding O(M) parameters, if we treat the widths D (l) m of the FNN as constants with respect to M. DISPLAYFORM0 m, and DISPLAYFORM1 that is, any FNN in DISPLAYFORM2 An immediate consequence of this theorem is that if we can approximate a function f • with a blocksparse FNN, we can also approximate f• with a CNN. Our second main theorem bounds the estimation error of the clipped ERM estimatorf. DISPLAYFORM0 (conv) and B (fc) as in Theorem 1. Suppose L m, C, K satisfies the equation FORMULA24 DISPLAYFORM1 Here, DISPLAYFORM2. M 1 and M 2 are defined by DISPLAYFORM3 The first term of FORMULA27 is the approximation error achieved by F (FNN). On the other hand, M 1 and M 2 are determined by the architectural parameters of F (CNN) -M 1 corresponds to the Lipschitz constant of a function realized by a CNN and M 2 is the number of parameters, including zeros, of a CNN. Therefore, the second term of represents the model complexity of F (CNN). There is a trade-off between the two terms. Using appropriately chosen M to balance them, we can evaluate the order of estimation error with respect to the sample size N. Corollary 1. Under the same assumptions as Theorem 2, suppose further log DISPLAYFORM4 The Barron class is an example of the function class that can be approximated by block-sparse FNNs. We employ the definition of Barron functions used in.Definition 3 (Barron class). We say a measurable function f DISPLAYFORM0 Here, F andF are the Fourier transformation and the inverse Fourier transformation, respectively. studied the approximation of the Barron function f• with the parameter s = 2 by a linear combination of M ridge functions (i.e., a 2-layered ReLU FNN). Specifically, they showed that there exists a function f M of the form DISPLAYFORM1 with |b m | ≤ 1, a m 1 = 1 and |t m | ≤ 1, such that f DISPLAYFORM2. Using this approximator f M, we can derive the same approximation order using CNNs by applying Theorem 1 with DISPLAYFORM3 Barron function with the parameter s = 2 such that f• = 0 and ∇f DISPLAYFORM4 with M residual blocks, each of which has depth O and at most 4 channels, and whose filter size is at DISPLAYFORM5 We have one design choice when we apply Corollary 1 to derive the estimation error: how to set B (bs) and B (fin). Looking at, the naive choice would be B (1 + ρ m) whose logarithm is O(M). We want its logarithm to beÕ. In order to do that, we change the relative scale between parameters in the block-sparse part and the fully-connected part using the homogeneous property of the ReLU function: ReLU(ax) = aReLU(x) for a > 0. The rescaling operation enables us to choose B, depth of each residual block L = O, channel size C = O, filter size K ∈ {2, . . ., D}, and norm bounds for the con- DISPLAYFORM6, and for the fully-connected part DISPLAYFORM7 such that for sufficiently large N, the clipped ERM estimatorf of F: DISPLAYFORM8. Here, DISPLAYFORM9 We next consider the approximation and error rates of CNNs when the true function is a β-Hölder function. Definition 4 (Hölder class). Let β > 0, f DISPLAYFORM0 Yarotsky FORMULA24 showed that FNNs with O(S) non-zero parameters can approximate any D variate β-Hölder function (β > 0) with the order ofÕ(S − β D). also proved a similar statement using a different construction method. They only specified their width (only), depth, and non-zero parameter counts of the approximating FNN and did not write in detail how non-zero parameters are distributed explicitly in the statements (see Theorem 1 of and Theorem 5 of Schmidt-Hieber FORMULA24). However, if we carefully look at their proofs, we find that we can transform the FNNs they constructed into the block-sparse ones. Therefore, we can utilize these FNNs and apply Theorem 1. To meet the assumption of Corollary 1, we again rescale the parameters of the FNNs, as we did in the Barron class case, so that log M 1 =Õ. We can derive the approximation and estimation errors by setting γ 1 = β D and γ 2 = 1. and O channels, and whose filter size is at most K, such that f DISPLAYFORM1 Corollary 5. There exist the number of residual blocks DISPLAYFORM2, depth of each residual block L =Õ, channel size C = O, filter size K ∈ {2, . . ., D}, norm bounds for the convolution part B (conv) = O, and for the fully-connected part B (fc) > 0 (log B (fc) = O(log N)) such that for sufficiently large N, the clipped ERM estimatorf of F: DISPLAYFORM3. Here, DISPLAYFORM4 Since the estimation error rate of the β-Hölder class is DISPLAYFORM5 (see, e.g.,), Corollary 5 implies that our CNN can achieve the minimax optimal rate up to logarithmic factors even the width D, the channel size C, and the filter size K are constant with respect to the sample size N. In this paper, we established new approximation and statistical learning theories for CNNs by utilizing the ResNet-type architecture of CNNs and the block-sparse structure of FNNs. We proved that any M -way block-sparse FNN is realizable using CNNs with O(M) additional parameters, when the width of the FNN is fixed. Using this , we derived the approximation and estimation errors for CNNs from those for block-sparse FNNs. Our theory is general because it does not depend on a specific function class, as long as we can approximate it with block-sparse FNNs. To demonstrate the wide applicability of our , we derived the approximation and error rates for the Barron class and Hölder class in almost same manner and showed that the estimation error of CNNs is same as that of FNNs, even if the CNNs have a constant channel size, filter size, and width with respect to the sample size. The key techniques were careful evaluations of the Lipschitz constant of CNNs and non-trivial weight parameter rescaling of FNNs. One of the interesting open questions is the role of the weight rescaling. We critically use the homogeneous property of the ReLU activation function to change the relative scale between the block-sparse part and the fully-connected part, if it were not for this property, the estimation error rate would be worse. The general theory for rescaling, not restricted to the Barron nor Hölder class would be beneficial for deeper understanding of the relationship between the approximation and estimation capabilities of FNNs and CNNs. Another question is when the approximation and estimation error rates of CNNs can exceed that of FNNs. We can derive the same rates as FNNs essentially because we can realize block-sparse FNNs using CNNs that have the same order of parameters (see Theorem 1). Therefore, if we dig into the internal structure of FNNs, like repetition, more carefully, the CNNs might need fewer parameters and can achieve better estimation error rate. Note that there is no hope to enhance this rate for the Hölder case (up to logarithmic factors) because the estimation rate using FNNs is already minimax optimal. It is left for future research which function classes and constraints of FNNs, like block-sparseness, we should choose. For tensor a, a +:= a∨0 where maximum operation is performed in element-wise manner. Similarly a −:= −(−a ∨ 0). Note that a = a + − a − holds for any tensor a. For normed spaces (V, · V), (W, · W) and linear operator T: V → W we denote the operator norm of T by T op:= sup v V =1 T v W. For a sequence w = (w,..., w (L) ) and l ≤ l, we denote its subsequence from the l-th to l -th elements by w[l : l]:= (w (l),..., w (l) ). 1 P equals to 1 if the statement P is true, equals to 0 otherwise. DISPLAYFORM0, we realize a CNN f (CNN) using M residual blocks by "serializing" blocks in the FNN and converting them into convolution layers. First, we double the channel size using the m = 0 part of CNN (i.e., D (L0) 0 = 2). We will use the first channel for storing the original input signal for feeding to downstream (i.e., m ≥ 1) blocks and the second one for accumulating the output of each blocks, that is, m size-1 filters made from the weight parameters of the corresponding layer of the FNN. Observing that the convolution operation with size-1 filter is equivalent to a dimension-wise affine transformation, the first coordinate of the output of l-th layer of the CNN is inductively same as that of the m-th block of the FNN. After computing the m-th block FNN using convolutions, we add its output to the accumulating channel in the identity mapping. Finally, we pick the first coordinate of the accumulating channel and subtract the bias term using the final affine transformation. We relate the approximation error of Theorem 2 with the estimation error using the covering number of the hypothesis class F (CNN). Although there are several theorems of this type, we employ the one in due to its convenient form (Lemma 5). We can prove that the logarithm of the covering number is upper bounded by M 2 log((B (conv) ∨ B (fc) )M 1 /ε) (Lemma 4) using the similar techniques to the one in. Theorem 2 is the immediate consequence of these two lemmas. To prove Cororraly 1, we set M = O(N α) for some α ≥ 0. Then, under the assumption of the corolarry, we have DISPLAYFORM0 from Theorem 2. The order of the right hand side with respect to N is minimized when α = DISPLAYFORM1 and b ∈ R such that 1. DISPLAYFORM2 Proof. First, observe that the convolutional layer constructed from u = [u 1 . . . u K] ∈ R K×1×1 takes the inner product with the first K elements of the input signal: DISPLAYFORM3 K×1×1 works as the "left-translation" by K − 1. Therefore, we should define w so that it takes the inner product with the K left-most elements in the first channel and shift the input signal by K − 1 with the second channel. Specifically, we define DISPLAYFORM4 We set b:= (0, . . ., 0 L0 − 1 times, t). Then w and b satisfy the condition of the lemma. The following lemma shows that we can convert any linear CNN to a ReLU CNN that has approximately 4 times larger parameters. This type of lemma is also found in (Lemma 2.3). DISPLAYFORM0 be filter sizes. Let (l). Consider the linear convolution layers constructed from w and b: DISPLAYFORM1 DISPLAYFORM2 ∞, and DISPLAYFORM3 Proof. We definew andb as follows: DISPLAYFORM4 By definition, a pair (w,b) satisfies the conditions FORMULA24 and FORMULA27. For any x ∈ R D, we set y DISPLAYFORM5 for any α, β ∈ [D]. Summing them up and using the definition ofb DISPLAYFORM6 (y DISPLAYFORM7 (y DISPLAYFORM8 (y DISPLAYFORM9 −(w (l+1) ) α,:,: (y DISPLAYFORM10 for any α, β ∈ [D]. Again, by taking the summation and using the definition ofb (l+1), we get DISPLAYFORM11.By applying ReLU, we get DISPLAYFORM12 By using the induction hypothesis, we get DISPLAYFORM13 Therefore, the claim holds for l + 1. By induction, the claim holds for L, which is what we want to prove. We can concatenate two CNNs with the same depths and filter sizes in parallel. Although it is almost trivial, we state it formally as a proposition. In the following proposition, C and C is not necessarily 1. DISPLAYFORM0 We define w and b in the same way, with the exception that C (l) is replaced with C (l). We definew = (w,...,w (L) ) and DISPLAYFORM1. Then, we have, DISPLAYFORM2 for any x, x ∈ R D×C and any σ: R → R.Note that by the definition of · 0 and · ∞, we have DISPLAYFORM3 and DISPLAYFORM4. We will construct the desired CNN consisting of M residual blocks, whose m-th residual block is made from the ingredients of the corresponding m-th block in f Lm], and w m ). DISPLAYFORM5 [The m = 0 Block]: We prepare a single convolutional layer with 2 output channels and 2 size-1 filters suth that the first filter works as the identity function and the second filter inserts zeros to the second channel. Weight parameters of this convolutional layer are all zeros except single one. We denote this block by Conv 0. DISPLAYFORM6 1×D is the d-th row of the matrix DISPLAYFORM7 m ×D. We apply Lemma 1 and Lemma 2 and obtain ReLU CNNs realizing the hinge functions. By combining them in parallel using Proposition 1, we have a learnable parameter θ DISPLAYFORM8 Since we double the channel size in the m = 0 part, the identity mapping has 2 channels. Therefore, we made Conv ReLU θ m so that it has 2 input channels and neglects the input signals coming from the second one. This is possible by adding filters consisting of zeros appropriately. Next, for l-th layer (l = 2, . . ., L m), we prepare size-1 filters w DISPLAYFORM9 where ⊗ is the Kronecker product of matrices. Intuitively, the l = 2 layer will pick all odd indices of the output of Conv DISPLAYFORM10. By the inductive calculation, we have DISPLAYFORM11 By definition, Conv m has the depth of L 0 + L m − 1, at most 4D DISPLAYFORM12 DISPLAYFORM13 Then, we have DISPLAYFORM14 (the subscript 1 represents the first coordinate).[Final Fully-connected Layer] Finally, we set w:= B We assume w, w ∈ R K×J×I, b, b ∈ R, and x ∈ R D×I unless specified. We have in mind that the activation function σ is either the ReLU function or the identity function id. But the following proposition holds for any 1-Lipschitz function such that σ = 0. Remember that we can treat L w as a linear operator from R D×I to R D×J. We endow R D×I and R D×J with the sup norm and denote the operator norm DISPLAYFORM0 is evaluated as follows: DISPLAYFORM1 Note that the first inequality holds because the ReLU function is 1-Lipschitz. DISPLAYFORM2 In the following propositions in this subsection, we assume W, W ∈ R D×C, b, b ∈ R, and x ∈ R D×C. Again, these propositions hold for any 1-Lipschitz function σ: R → R such that σ = 0. But σ = ReLU or id is enough for us. DISPLAYFORM0 The number of non-zero summand in the summation is at most W 0 and each summand is bounded by W ∞ x ∞ Therefore, we have |FC DISPLAYFORM1 In this section, we denote the architecture of CNNs by DISPLAYFORM2 and the norm constraint on the convolution part by DISPLAYFORM3 and (conv). Then, for any DISPLAYFORM4 DISPLAYFORM5 Proof. We write in shorthand as C DISPLAYFORM6 By Proposition 2 and assumptions w DISPLAYFORM7, it is further bounded by DISPLAYFORM8 by Proposition 2 and 4) DISPLAYFORM9 by Proposition 2 and 5) DISPLAYFORM10 We denote the l-th convolution layer of the m-th block by C (l) m and the m-th residual block of by C m: Proof. By using Proposition 9 inductively, we have DISPLAYFORM11 DISPLAYFORM12 Lemma 3. Let ε > 0. Suppose θ and θ are within distance ε, that is, max m,l w DISPLAYFORM13 We will bound each term of. By Proposition 8 and Proposition 11, DISPLAYFORM14 On the other hand, for m = 0,..., M, DISPLAYFORM15 DISPLAYFORM16 By applying and FORMULA37 to FORMULA37, we have DISPLAYFORM17 For a metric space (M 0, d) and ε > 0, we denote the (external) covering number of DISPLAYFORM0 Proof. The idea of the proof is same as that of Lemma 12 of Schmidt-Hieber (2017 (i.e., 2B DISPLAYFORM1 can be realized by parameters such that every pair of corresponding parameters are in a same bin, then, f − f ∞ ≤ ε by Lemma 3. We make a subset F 0 of F (CNN) by picking up every combination of bins for M 2 parameters. Then, for each f ∈ F (CNN), there exists f 0 ∈ F 0 such that f − f 0 ∞ ≤ ε. There are at most 2BM 1 ε −1 choices of bins for each parameter. Therefore, the cardinality of F 0 is at most DISPLAYFORM2 D.2 PROOF OF THEOREM 2 AND COROLLARY 1We use the lemma in to bound the estimation error of the clipped ERM estimatorf. Since our problem setting is slightly different from one in the paper, we restate the statement. Lemma 5 (cf. Lemma 10). Let F be a family of measurable functions from [−1, 1] D to R. Letf be the clipped ERM estimator of the regression problem described in Section 3.1. Suppose the covering number of F satisfies N (ε, F, · ∞) ≥ 3. Then, Proof. Basically, we convert our problem setting so that it fits to the assumptions of Lemma 10 of • (x n)). Then, the probability that D is drawn from P ⊗N is same as the probability that D is drawn from P ⊗N where P is the joint distribution of (X, Y). Also, we can show thatf is the ERM estimator of the regression problem Y = f DISPLAYFORM3 • +ξ using the dataset D:f 1 ∈ arg min f ∈F R D (f). We apply the Lemma 10 of with n ← N, d ← D, ε ← 1, δ ← 1 N, ∆ n ← 0, F ← F, F ← 2F,f ←f 1 and use the fact that the estimation error of the clipped ERM estimator is no worse than that of the ERM estimator, that is, Next, we prove the existence of a block-sparse FNN with constant-width blocks that optimally approximates a given β-Hölder function. It is almost same as the proof of Theorem 5 of. However, we need to construct the FNN so that it has a block-sparse structure. FORMULA24 ) layers. It is left for future research whether our can extend to the ResNet-type CNNs with pooling or Batch Normalization layers. Second, our CNN does not have ReLU activation after the junction points and the final layer of the 0-th block, while they have in the original ResNet. We choose this design to make proofs simpler. We can easily extend our to the architecture that adds the ReLU activations to those points with slight modifications using similar techniques appeared in Lemma 2 of the appendix. DISPLAYFORM4 DISPLAYFORM5

represent the short summary for finding the corresponding scientific article (long)

It is shown that ResNet-type CNNs are a universal approximator and its expression ability is not worse than fully connected neural networks (FNNs) with a \textit{block-sparse} structure even if the size of each layer in the CNN is fixed.

Represent the short summary for finding the corresponding scientific article (long)

We provide necessary and sufficient analytical forms for the critical points of the square loss functions for various neural networks, and exploit the analytical forms to characterize the landscape properties for the loss functions of these neural networks.


Question: Represent the scientific article to find a short TLDR

Question: Few shot image classification aims at learning a classifier from limited labeled data. Generating the classification weights has been applied in many meta-learning approaches for few shot image classification due to its simplicity and effectiveness. However, we argue that it is difficult to generate the exact and universal classification weights for all the diverse query samples from very few training samples. In this work, we introduce Attentive Weights Generation for few shot learning via Information Maximization (AWGIM), which addresses current issues by two novel contributions. i) AWGIM generates different classification weights for different query samples by letting each of query samples attends to the whole support set. ii) To guarantee the generated weights adaptive to different query sample, we re-formulate the problem to maximize the lower bound of mutual information between generated weights and query as well as support data. As far as we can see, this is the first attempt to unify information maximization into few shot learning. Both two contributions are proved to be effective in the extensive experiments and we show that AWGIM is able to achieve state-of-the-art performance on benchmark datasets. While deep learning methods achieve great success in domains such as computer vision , natural language processing , reinforcement learning , their hunger for large amount of labeled data limits the application scenarios where only a few data are available for training. Humans, in contrast, are able to learn from limited data, which is desirable for deep learning methods. Few shot learning is thus proposed to enable deep models to learn from very few samples . Meta learning is by far the most popular and promising approach for few shot problems (; ; ; ;). In meta learning approaches, the model extracts high level knowledge across different tasks so that it can adapt itself quickly to a new-coming task . There are several kinds of meta learning methods for few shot learning, such as gradient-based and metric-based . Weights generation, among these different methods, has shown effectiveness with simple formulation (; ; ; . In general, weights generation methods learn to generate the classification weights for different tasks conditioned on the limited labeled data. However, fixed classification weights for different query samples within one task might be sub-optimal, due to the few shot challenge. We introduce Attentive Weights Generation for few shot learning via Information Maximization (AWGIM) in this work to address these limitations. In AWGIM, the classification weights are generated for each query sample specifically. This is done by two encoding paths where the query sample attends to the task context. However, we show in experiments that simple cross attention between query samples and support set fails to guarantee classification weights fitted to diverse query data since the query-specific information is lost during weights generation. Therefore, we propose to maximize the lower bound of mutual information between generated weights and query, support data. As far as we know, AWGIM is the first work introducing Variational Information Maximization in few shot learning. The induced computational overhead is minimal due to the nature of few shot problems. Furthermore, by maximizing the lower bound of mutual information, AWGIM gets rid of inner update without compromising performance. AWGIM is evaluated on two benchmark datasets and shows state-of-the-art performance. We also conducted detailed analysis to validate the contribution of each component in AWGIM. 2 RELATED WORKS 2.1 FEW SHOT LEARNING Learning from few labeled training data has received growing attentions recently. Most successful existing methods apply meta learning to solve this problem and can be divided into several categories. In the gradient-based approaches, an optimal initialization for all tasks is learned . learned a meta-learner LSTM directly to optimize the given fewshot classification task. learned the transformation for activations of each layer by gradients to better suit the current task. In the metric-based methods, a similarity metric between query and support samples is learned. (; ; ; ; a). Spatial information or local image descriptors are also considered in some works to compute richer similarities (; b;). Generating the classification weights directly has been explored by some works. generated classification weights as linear combinations of weights for base and novel classes. and both generated the classification weights from activations of a trained feature extractor. Graph neural network denoising autoencoders are used in . proposed to generate "fast weights" from the loss gradient for each task. All these methods do not consider generating different weights for different query examples, nor maximizing the mutual information. There are some other methods for few-shot classification. Generative models are used to generate or hallucinate more data in; ). and used the closed-form solutions directly for few shot classification. integrated label propagation on a transductive graph to predict the query class label. Attention mechanism shows great success in computer vision and natural language processing . It is effective in modeling the interaction between queries and key-value pairs from certain context. Based on the fact that keys and queries point to the same entities or not, people refer to attention as self attention or cross attention. In this work, we use both types of attention to encode the task and query-task information. The work most similar to ours is Attentive Neural Processes, which also employs self and cross attention. However, we are using attention for few-shot image classification via maximizing the mutual information. In stark contrast, worked on regression from the perspective of a stochastic process and the variational objective is optimized. Given two random variables x and y, mutual information I(x; y) measures the decrease of uncertainty in one random variable when another is known. It is defined as the Kullback-Leibler divergence between joint distribution p(x, y) and product of marginal distributions p(x) ⊗ p(y), When x and y are independent, p(x, y) = p(x) ⊗ p(y) so that I(x, y) = 0, indicating that knowing x does not reveal any information about y. When y is a deterministic function of x, I(x, y) achieves its maximum value. Mutual information has been widely applied in applications such as Generative Adversarial Networks, self-supervised learning , visual question generation and so on. Similarly, the attentive path enables the query samplex to be equipped with task knowledge. Both paths are achieved by attention mechanism.x ap is repeated to concatenate with X cp. The weight generator g takes these concatenated representations as input to generate classification weights W specific forx, denoted by the colorful matrix with slash. It can be used to predict the class label forx and X. W is also used to reconstruct the inputs of the generator g by two networks r 1 and r 2. In this way, the lower bound of mutual information is maximized and g is forced to generate classification weights sensitive to different query samples. In this section, we provide the problem formulation first. Then the proposed model is described in Sec. 3.3. The objective function, which maximizes the mutual information between certain variables, and theoretical analysis are provided in Sec. 3.4. Following many popular meta-learning methods for few shot classification, we formulate the problem under episodic training paradigm . One N -way K-shot task sampled from an unknown task distribution P (T) includes support set and query set: where S = {(x cn;k, y cn;k)|k = 1,..., K; n = 1,..., N }, Q = {(x 1, ...,x |Q|)}. Support set S contains N K labeled samples. Query set Q includesx and we need to predict labelŷ forx based on S. During meta-training, the meta-loss is estimated on Q to optimize the model. During metatesting, the performance of meta-learning method is evaluated on Q, provided the labeled S. The classes used in meta-training and meta-testing are disjoint so that the meta-learned model needs to learn the knowledge transferable across tasks and adapt itself quickly to novel tasks. Our proposed approach follows the general framework to generate the classification weights (; ; ; ; . In this framework, there is a feature extractor to output image feature embeddings. The meta-learner needs to generate the classification weights for different tasks. Latent Embedding Optimization (LEO) is one of the weights generation methods that is most related to our work. In LEO, the latent code z is generated by h conditioned on support set S, described as z = h(S). h is instantiated as relation networks . Classification weights w can be decoded from z with l, w = l(z). In the inner loop, we use w to compute the loss (usually cross entropy) on the support set and then update z: where L S indicates that the loss is evaluated on S only. The updated latent code z is used to decode new classification weights w with generating function l. w is adopted in the outer loop for query set Q and the objective function of LEO then can be written as min Here θ stands for the parameters of h and l and we omit the regularization terms for clarity. LEO avoids updating high-dimensional w in the inner loop by learning a lower-dimensional latent space, from which sampled z can be used to generate w. The most significant difference between LEO and AWGIM is that we do not need inner updates to adapt the model. Instead, AWGIM is a feedforward network trained to maximize the mutual information so that it fits to different tasks well. On the other hand, AWGIM learns to generate optimal classification weights for each query sample while LEO generates fixed weights conditioned on the support set within one task. In Section 3.4 we will show LEO can be casted as a special case of AWGIM under certain conditions. The framework of our proposed method is shown in Figure 1. Assume that we have a feature extractor, which can be a simple 4-layer Convnet or a deeper Resnet. All the images included in the sampled task T are processed by this feature extractor and represented as d-dimensional vectors afterwards, i.e., x cn;k,x ∈ R d. There are two paths to encode the task context and the individual query sample respectively, which are called contextual path and attentive path. The outputs of both paths are concatenated together as input to the generator for classification weights. Generated classification weights are used to not only predict the label ofx, but also maximize the lower bound of mutual information between itself and other variables, which will be discussed in the following section 3.4. The encoding process includes two paths, namely the contextual path and attentive path. The contextual path aims at learning representations for only the support set with a multi-head self-attention network f cp sa . The outputs of contextual path X cp ∈ R N K×d h 1 thus contain richer information about the task and can be used later for weights generation. Existing weights generation methods generate the classification weights conditioned on the support set only, which is equivalent to using contextual path. However, the classification weights generated in this way might be sub-optimal. This is because estimating the exact and universal classification weights from very few labeled data in the support set is difficult and sometimes impossible. The generated weights are usually in lack of adaptation to different query samples. We address this issue by introducing attentive path, where the individual query example attends to the task context and then is used to generate the classification weights. Therefore, the classification weights are adaptive to different query samples and aware of the task context as well. In the attentive path, a new multi-head self-attention network f ap sa on the support set is employed to encode the global task information. f ap sa is different from f cp sa in contextual path because the selfattention network in contextual path emphasizes on generating the classification weights. On the contrary, outputs of self-attention here plays the role of providing the V alue context for different query samples to attend in the following cross attention. Sharing the same self-attention networks might limit the expressiveness of learned representations in both paths. The cross attention network f ap ca applied on each query sample and task-aware support set is followed to produceX ap ∈ R |Q|×d h. We use multi-head attention with h heads in both paths. In one attention block, we produce h different sets of queries, keys and values. Multi-head attention is claimed to be able to learn more comprehensive and expressive representations from h different subspaces . More details of these two paths can be found in A.2. We replicate X cp ∈ R N K×d h andX ap ∈ R |Q|×d h for |Q| and N K times respectively and reshape them afterwards. Then we have X cp ∈ R |Q|×N K×d h andX ap ∈ R |Q|×N K×d h. These two tensors are concatenated to become X cp⊕ap ∈ R |Q|×N K×2d h. X cp⊕ap can be interpreted that each query sample has its own latent representations for support set to generate specific classification weights, which are both aware of the task-context and adaptive to individual query sample. cp⊕ap is decoded by the weights generator g: R 2d h → R 2d. We assume that the classification weights follow Gaussian distribution with diagonal covariance. g outputs the distribution parameters and we sample the weights from learned distribution during meta-training. The sampled classification weights are represented as W ∈ R |Q|×N K×d. To reduce complexity, we compute the mean value on K classification weights for each class to have W f inal ∈ R |Q|×N ×d. Therefore, ith query sample has its specific classification weight matrix W f inal i,:,: ∈ R N ×d. The prediction for query data can be computed byXW f inalT. The support data X is replicated for |Q| times and reshaped as X s ∈ R |Q|×N K×d. So the prediction for support data can also be computed as X s W f inalT. Besides the weights generator g, we have another two decoders r 1: They both take the generated weights W as inputs and learn to reconstruct X cp andX ap respectively. The outputs of r 1 and r 2 are denoted as X cp re,X ap re ∈ R |Q|×N K×d h. The reason we are using reconstruction as auxiliary tasks will be discussed in following Sec. 3.4. In this section, we perform the analysis for one query sample without loss of generality. The subscripts for classification weights are omitted for clarity. In general, we use (x, y) and (x,ŷ) to represent support and query samples respectively. Since the classification weights w generated from g are encoded with attentive path and contextual path, it is expected that we can directly have the query-specific weights. However, we show in the experiments that simply doing this does not outperform a weight generator conditioned only on the S significantly, which implies that the generated classification weights from two paths are not sensitive to different query samples. In other words, the information from attentive path is not kept well during the weights generation. To address this limitation, we propose to maximize the mutual information between generated weights w and support as well as query data. The objective function can be described as maxI((x,ŷ); w) + (x,y)∈S I((x, y); w) According to the chain rule of mutual information, we have I((x,ŷ); w) = I(x; w) + I(ŷ; w|x). Equation 6 Directly computing the mutual information in Equation 7 is intractable since the true posteriori distributions like p(ŷ|x, w), p(x|w) are still unknown. Therefore, we use Variational Information Maximization (; to compute the lower bound of Equation 5. We use p θ (x|w) to approximate the true posteriori distribution, where θ represents the model parameters. As a , we have H(·) is the entropy of a random variable. H(x) is a constant value for given data. We can maximize this lower bound as the proxy for the true mutual information. Similar to I(x; w), p θ (x|w), p θ (x, y|w) are used to approximate the true posteriori distribution p(x|w) and p(x, y|w). Put the lower bounds back into Equation 7. Omit the constant entropy terms and the expectation subscripts for clarity, we have the new objective function as The first two terms are maximizing the log likelihood of label for both support and query data with respective to the network parameters, given the generated classification weights. This is equivalent to minimizing the cross entropy between prediction and ground-truth. We assume that p θ (x|w) and p θ (x|w) are Gaussian distributions. r 1 and r 2 are used to approximate the mean of these two Gaussian distributions. Therefore maximizing the log likelihood is equivalent to reconstruct x cp and x ap with L2 loss. Thus the loss function to train the network can be written as is not equal to real log likelihood and we have to decide the weightage for each one. λ 1, λ 2, λ 3 are thus hyper-parameters for trade-off of different terms. With the help of last three terms, the generated classification weights are forced to carry information about the support data and the specific query sample. In LEO , the inner update loss is computed as cross entropy on support data. If we merge the inner update into outer loop, then the loss becomes the summation of first two terms in Equation 15. However, the weight generation in LEO does not involve specific query samples, thus making reconstructingx ap impossible. In this sense, LEO can be regarded as a special case of our proposed method, where only contextual path exits and λ 2 = λ 3 = 0. The encoding process in contextual path in computational complexity O((N K) 2 ) due to self-attention. Similarly, the computational complexity of attentive path is O((N K) 2 + |Q|(N K)). In total, the complexity is O((N K) 2 + |Q|(N K)). However, because of the nature of few-shot learning problem, the value of (N K) 2 is usually negligible. The value of |Q| depends on the setting and the cross attention can be implemented parallelly via matrix multiplication. Therefore, the induced computational overhead will be negligible. AWGIM avoids the inner update without compromising the performance, which furthers reduces both training and inference time significantly. The empirical evaluation is presented in A.3.4. We conduct experiments on miniImageNet and tieredImageNet , two commonly used benchmark datasets, to compare with other methods and analyze our model. Both datasets are subsets of ILSVRC-12 dataset . miniImageNet contains 100 randomly sampled classes with 600 images per class. We follow the train/test split in , where 64 classes are used for meta-training, 16 for meta-validation and 20 for meta-testing. tieredImageNet is a larger dataset compared to miniImageNet. There are 608 classes and 779,165 images in total. They are selected from 34 higher level nodes in ImageNet hierarchy. 351 classes from 20 high level nodes are used for meta-training, 97 from 6 nodes for meta-validation and 160 from 8 nodes for meta-testing. We use the image features in LEO provided by the authors 2. They trained a 28-layer Wide Residual Network on the meta-training set. Each image then is represented by a 640 dimensional vector, which is used as the input to our model. For N -way K-shot experiments, we randomly sample N classes from meta-training set and each of them contains K samples as the support set and 15 as query set. Similar to other works, we train 5-way 1-shot and 5-shot models on two dataset. During meta-testing, 600 N -way K-shot tasks are sampled from meta-testing set and the average accuracy for query set is reported with 95% confidence interval, as done in recent works (; ;). We use TensorFlow to implement our method and the code will be made available. d = 640 is the dimension of feature embeddings. d h is set to be 128. The number of heads h in attention module is set to be 4. g, r 1 and r 2 are 2-layer MLPs with 256 hidden units. We decide λ 1 = 1, λ 2 = λ 3 = 0.001 by meta-validation performance. Conv-4 48.70 ± 1.84% 63.11 ± 0.92% Meta LSTM Conv-4 43.44 ± 0.77% 60.60 ± 0.71% Prototypical Nets Conv-4 49.42 ± 0.78% 68.20 ± 0.66% Relation Nets Conv-4 50.44 ± 0.82% 65.32 ± 0.70% SNAIL Resnets-12 55.71 ± 0.99% 68.88 ± 0.92% TPN Resnets-12 59.46 75.65 MTL Resnets-12 61.20 ± 1.80% 75.50 ± 0.80 Dynamic WRN-28-10 60.06 ± 0.14% 76.39 ± 0.11% Prediction WRN-28-10 59.60 ± 0.41% 73.74 ± 0.19% DAE-GNN WRN-28-10 62.96 ± 0.15% 78.85 ± 0.10% LEO WRN-28-10 61.76 ± 0.08% 77.59 ± 0.12% AWGIM (ours) WRN-28-10 63.12 ± 0.08% 78.40 ± 0.11% Table 2: Accuracy comparison with other approaches on tieredImageNet. The are averaged on 600 tasks from meta-testing set with 95% confidence interval. Best are highlighted. Model Feature Extractor 5-way 1-shot 5-way 5-shot MAML Conv-4 51.67 ± 1.81% 70.30 ± 1.75% Prototypical Nets Conv-4 53.31± 0.89% 72.69 ± 0.74% Relation Nets Conv-4 54.48 ± 0.93% 71.32 ± 0.78% TPN Conv-4 59.91 ± 0.96% 72.85 ± 0.74% is used to optimize the network with weight decay 1 × 10 −6. The initial learning rate is set to 0.0002 for 5-way 1-shot and 0.001 for 5-way 5-shot, which is decayed by 0.2 for every 15,000 iterations. We train the model for 50,000 iterations. Batch size is 64 for 5-way 1-shot and 32 for 5-way 5-shot. Similar to LEO , we first train the model on meta-training set and choose the optimal hyper-parameters by validation . Then we train the model on meta-training and meta-validation sets together using fixed hyper-parameters. We compare the performance of our approach AWGIM on two datasets with several state-of-theart methods proposed in recent years. The of MAML, Prototypical Nets, Relation Nets on tieredImageNet are evaluated by. The of Dynamic on miniImageNet with WRN-28-10 as the feature extractor is reported in . The other are reported in the corresponding original papers. We also include the backbone network structure of the used feature extractor for reference. The on miniImageNet and tieredImageNet are shown in Table 1 and 2 respectively. The top half parts of Table 1 and 2 display the methods belonging with different meta learning categories, such as metric-based(Matching Networks, Prototypical Nets), gradient-based (MAML, MTL), graph-based (TPN). The bottom part shows the classification weights generation approaches including Dynamic, Prediction, DAE-GNN, LEO and our proposed AWGIM. AWGIM can outperform all the methods in top parts of two table. Comparing with other classification weights generation methods in the bottom part, AWGIM still shows very competitive performance, namely the best on tieredImageNet and close to the state-of-the-art on miniImageNet. We note that all the classification weights generation methods are using WRN-28-10 as backbone network, which makes the comparison fair. In particular, AWGIM can outperform LEO in all settings. Table 3: Analysis of our proposed AWGIM. In the top half, the attentive path is removed to compare with LEO. In the bottom part, ablation analysis with respective to different components is provided. We also shuffle the generated classification weights randomly to show that they are indeed optimal for different query samples. We perform detailed analysis on AWGIM, shown in Table 3. We include the of for reference. "Generator in LEO" means that there is no inner update in LEO. In the upper part of the table, we first studied the effect of attentive path. We implemented two generators including only the contextual path during encoding. "Generator conditioned on S with IM" indicates that we add the cross entropy loss and reconstruction loss for support set. It can be observed that "Generator conditioned on S only" is trained with cross entropy on query set, which is similar to "Generator in LEO" without inner update. It is able to achieve similar or slightly better than "Generator in LEO", which implies that self-attention is no worse than relation networks used in LEO to model task-context. With information maximization, our generator is able to obtain slightly better performance than LEO. The effect of attention is investigated by replacing the attention modules with 2-layer MLPs, which is shown as "MLP encoding". More specifically, one MLP in contextual path is used for support set and another MLP in attentive path for query samples. We can see that even without attention to encode the task-contextual information, "MLP encoding" can achieve accuracy close to LEO, for the sake of information maximization. However, if we let λ 1 = λ 2 = λ 3 = 0 for MLP encoding, the performance drops significantly, which demonstrates the importance of maximizing the information. We conducted ablation analysis with respective to λ 1, λ 2 and λ 3 to investigate the effect of information maximization. First, λ 1, λ 2 and λ 3 are all set to be 0. In this case, the accuracy is similar to "generator conditioned on S only", showing that the generated classification weights are not fitted for different query samples, even with the attentive path. It can also be observed that maximizing the mutual information between weights and support is more crucial since λ 1 = λ 2 = 0 degrades accuracy significantly, comparing with λ 3 = 0. We further investigate the relative importance of the classification on support as well as reconstruction. λ 1 = 0 affects the performance noticeably. We conjecture that the support label prediction is more critical for information maximization. The classification weights are generated specifically for each query sample in AWGIM. To this point, we shuffle the classification weights between query samples within the same classes and between different classes as well to study whether the classification weights are adapted for different query samples. Assume there are T query samples per class in one task. W f inal ∈ R |Q|×N ×d can be reshaped into W f inal ∈ R N ×T ×N ×d. Then we shuffle this weight tensor along the first and second axis randomly. The are shown as "random shuffle between classes" and "random shuffle in class" in Table 3. For 5-way 1-shot experiments, the random shuffle between classes degrades the accuracy noticeably while the random shuffle in class dose not affect too much. This indicates that when the support data are very limited, the generated weights for query samples from the same class are very similar to each other while distinct for different classes. When there are more labeled data in support set, two kinds of random shuffle show very close or even the same in 5-way 5-shot experiments, which are both worse than the original ones. This implies that the generated classification weights are more diverse and specific for each query sample in 5-way 5-shot setting. The possible reason is that larger support set provides more knowledge to estimate the optimal classification weights for each query example. More analysis is provided in Appendix A.3. In this work, we introduce Attentive Weights Generation via Information Maximization (AWGIM) for few shot image classification. AWGIM learns to generate optimal classification weights for each query sample within the task by two encoding paths. To guarantee this, the lower bound of mutual information between generated weights and query, support data is maximized. As far as we know, AWGIM is the first work utilizing mutual information techniques for few shot learning. The effectiveness of AWGIM is demonstrated by state-of-the-art performance on two benchmark datasets and extensive analysis. The multi-head attention can be described as X h1 ∈ R N K×d h is the matrix where each row stands for one support sample x cn;k h1. For one N -way K-shot task, the outputs of f sa cp are represented by a matrix X cp ∈ R N K×d h. The attentive path is instantiated by attention, similar to contextual path. First, a MLP f 2: andx h2. Then we employ another H-head selfattention network f cn;k h2 to encode the global task information to each support sample, The cross attention between query and context-aware support samples are computed aŝ HereX ap ∈ R |Q|×d h is the matrix form ofx q, where each query sample is context-aware. ∈ R 2d h, where i, j stands for ith query sample and jth support sample. x cp⊕ap is decoded by the weights generator g: R 2d h → R 2d. We assume that the classification weights follow Gaussian distribution with diagonal covariance and we sample the weights from this distribution during meta-training, shown in Equation 23 and 24. A.3 EXPERIMENTAL ANALYSIS A.3.1 FEW SHOT REGRESSION AWGIM can be applied to few shot regression task by slight modification. During meta-training, we set the number of classes N equal to 1 and adapt the cross entropy loss to mean square error. We use the data points (x, y) as inputs to AWGIM and generate weight as well as bias parameters for a three layer MLP with hidden dimension 40. This is consistent with few shot regression experimental setting in LEO. The few shot regression tasks are constructed as either sinusoidal or linear regression tasks. For sinusoidal regression tasks, the amplitude range is We replace the multi-head attention in the two paths with single-head attention and conduct the 5-way 1-shot and 5-way 5-shot experiments on miniImageNet dataset. The are shown in Table 4. We can see clearly that multi-head attention improve the performance. In particular, for 5-way 1-shot experiment, single head attention gives close to MLP encoding, which indicates that single head attention struggles when data are extremely scarce. We compare AWGIM with LEO in terms of convergence speed. The batch size is set to be 16 for both methods. We use the hyper-parameters tuned by authors to train LEO. The accuracy of metavalidation set during meta-training on 5-way 1-shot miniImageNet is plotted, shown in Figure 3. we can see clearly that AWGIM converges faster than LEO and outperforms LEO except for the first few iterations. We measure the inference time of AWGIM to show that it induces minimal computational overhead. In comparison, we use "MLP encoding" in two paths, which has time complexity O(N K + |Q|). We use two set-ups on miniImageNet and the batch size is set to be 64. 100 batches are processed and we report the average consumed time for one batch. All these experiments on done with the same GPU and workstation. The are shown in Table 5. It can be observed that the usage of self-attention and cross attention in AWGIM occurs negligible overhead, compared with MLP encoding. This is because the values of N, K, |Q| are all relatively small and matrix multiplication further can be processed very fast by GPU. We visualize the generated classification weights by t-SNE . First we sample 400 tasks from meta-validation set of 5-way 1-shot miniImageNet experiment. Each task contains 5 query samples from 5 different classes. Thus in total there are 400 × 5 × 5 = 10, 000 weight vectors to visualize. As comparison, inputs to the generator g are also plotted. The visualization are shown in Figure 4. The inputs to g are displayed in (a, b) and the generated classification weights in (c, d). From the comparison between (a) and (c), we can see the decoded weights for each class in (c) are clustered closer than (a) in general. Red and blue dots in (b, d) denotes the classification weights for two query samples from two classes within one task. It can be observed that g can generate adapted weights for different query samples. This is consistent with Table 3, where the of "random shuffle between classes" suggest that query samples from different class have distinct classification weights. (a) (b) (c) (d) Figure 4: t-SNE visualization of the inputs to g in (a, b) and the generated classification weights in (c, d). Blue and red dots in (b) and (d) are the classification weights for two query samples in the same task.

Represent the TLDR for finding the corresponding scientific article.

A novel few shot learning method to generate query-specific classification weights via information maximization.

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Conversational question answering (CQA) is a novel QA task that requires the understanding of dialogue context. Different from traditional single-turn machine reading comprehension (MRC), CQA is a comprehensive task comprised of passage reading, coreference resolution, and contextual understanding. In this paper, we propose an innovative contextualized attention-based deep neural network, SDNet, to fuse context into traditional MRC models. Our model leverages both inter-attention and self-attention to comprehend the conversation and passage. Furthermore, we demonstrate a novel method to integrate the BERT contextual model as a sub-module in our network. Empirical show the effectiveness of SDNet. On the CoQA leaderboard, it outperforms the previous best model's F1 score by 1.6%. Our ensemble model further improves the F1 score by 2.7%. Machine reading comprehension (MRC) is a core NLP task in which a machine reads a passage and then answers related questions. It requires a deep understanding of both the article and the question, as well as the ability to reason about the passage and make inferences. These capabilities are essential in applications like search engines and conversational agents. In recent years, there have been numerous studies in this field (; ;), with various innovations in text encoding, attention mechanisms and answer verification. However, traditional MRC tasks often take the form of single-turn question answering. In other words, there is no connection between different questions and answers to the same passage. This oversimplifies the conversational manner humans naturally take when probing a passage, where question turns are assumed to be remembered as context to subsequent queries. Figure 1 demonstrates an example of conversational question answering in which one needs to correctly refer "she" in the last two rounds of questions to its antecedent in the first question, "Cotton." To accomplish this kind of task, the machine must comprehend both the current round's question and previous rounds of utterances in order to perform coreference resolution, pragmatic reasoning and semantic implication. To facilitate research in conversation question answering (CQA), several public datasets have been published that evaluate a model's efficacy in this field, such as CoQA , QuAC and QBLink . In these datasets, to generate correct responses, models need to fully understand the given passage as well as the dialogue context. Thus, traditional MRC models are not suitable to be directly applied to this scenario. Therefore, a number of models have been proposed to tackle the conversational QA task. DrQA+PGNet combines evidence finding and answer generation to produce answers. BiDAF++ achieves better by employing answer marking and contextualized word embeddings on the MRC model BiDAF . FlowQA leverages a recurrent neural network over previous rounds of questions and answers to absorb information from its history context. Once upon a time, in a barn near a farm house, there lived a little white kitten named Cotton. Cotton lived high up in a nice warm place above the barn where all of the farmer's horses slept. But Cotton wasn't alone in her little home above the barn, oh no. She shared her hay bed with her mommy and 5 other sisters... What color was Cotton? A1: white Q2: Where did she live? A2: in a barn Q3: Did she live alone? A3: no Figure 1: Example passage and first three rounds of question and answers from CoQA dataset . Pronouns requiring coreference resolution is marked in bold. In this paper, we propose SDNet, a contextual attention-based deep neural network for the conversational question answering task. Our network stems from machine reading comprehension models, but it has several unique characteristics to tackle context understanding. First, we apply both interattention and self-attention on the passage and question to obtain a more effective understanding of the passage and dialogue history. Second, we prepend previous rounds of questions and answers to the current question to incorporate contextual information. Third, SDNet leverages the latest breakthrough in NLP: BERT contextual embeddings . Different from the canonical way of employing BERT as a monolithic structure with a thin linear task-specific layer, we utilize BERT as a contextualized embedder and absorb its structure into our network. To accomplish this, we align the traditional tokenizer with the Byte Pair Encoding (BPE) tokenizer in BERT. Furthermore, instead of using only the last layer's output from BERT , we employ a weighted sum of BERT layer outputs to take advantage of all levels of semantic abstraction. Finally, we lock the internal parameters of BERT during training, which saves considerable computational cost. These techniques are also applicable to other NLP tasks. We evaluate SDNet on the CoQA dataset, and it improves on the previous state-of-the-art F 1 score by 1.6% (from 75.0% to 76.6%). The ensemble model further increases the F 1 score to 79.3%. In this section, we propose our neural model, SDNet, for the conversational question answering task. We first formulate the problem and then present an overview of the model before delving into the details of the model structure. Given a passage/context C, and question-answer pairs from previous rounds of conversation Q 1, A 1, Q 2, A 2,..., Q k−1, A k−1, the task is to generate response A k given the latest question Q k. The response is dependent on both the passage and historic questions and answers. To incorporate conversation history into response generation, we employ the idea from DrQA+PGNet to prepend the latest N rounds of QAs to the current question Q k. The problem is then converted into a single-turn machine reading comprehension task, where the reformulated question is Encoding layer encodes each token in passage and question into a fixed-length vector, which includes both word embeddings and contextualized embeddings. For contextualized embedding, we utilize the pretrained language understanding model BERT . Different from previous work, we fix the parameters in BERT model and use the linear combination of embeddings from different layers in BERT. Integration layer uses multi-layer recurrent neural networks (RNN) to capture contextual information within passage and question. To characterize the relationship between passage and question, we conduct word-level attention from question to passage both before and after the RNNs. We employ the idea of history-of-word from FusionNet to reduce the dimension of output hidden vectors. Furthermore, we conduct self-attention to extract relationship between words at different positions of context and question. Output layer computes the final answer span. It uses attention to condense the question into a fixedlength vector, which is then used in a bilinear projection to obtain the probability that the answer should start and end at each position. An illustration of our model SDNet is in Figure 2. We first use GloVe embedding for each word in the context and question. Additionally, we compute a feature vector f w for each context word, following the approach in DrQA. This feature vector contains a 12-dim POS embedding, an 8-dim NER embedding, a 3-dim exact matching vector em i indicating whether this word, its lower-case form or its stem appears in the question, and a 1-dim normalized term frequency. BERT as Contextual Embedder. We design a number of methods to leverage BERT as a contextualized embedder in our model. First, because BERT uses Byte Pair Encoding (BPE) as the tokenizer, the generated tokens are sub-words and may not align with traditional tokenizer . To incorporate BERT into our network, we first use a conventional tokenizer (e.g. spaCy) to get word sequences, and then apply the BPE tokenizer from BERT to partition each word w in the sequence into subwords w = (b 1, ..., b s). This alignment makes it possible to concurrently use BERT embeddings and other word-level features. The contextual embedding of w is defined to be the averaged BERT embedding of all sub-words b j, 1 ≤ j ≤ s. proposes the method to append thin task-specific linear layers to BERT, which takes the from the last transformer layer as input. However, as BERT contains multiple layers, we employ a weighted sum of these layer outputs to take advantage of information from all levels of semantic abstraction. This can help boost the performance compared with using only the last transformer's output. Third, as BERT contains hundreds of millions of parameters, it takes a lot of time and space to compute and store their gradients during optimization. To tackle this problem, we lock the internal weights of BERT during training, only updating the linear combination weights. This can significantly increase the efficiency during training, which can be especially useful when computing resource is limited. To summarize, suppose a word w is tokenized to s BPE tokens w = (b 1, b 2, ..., b s), and BERT has L layers that generate L embedding outputs for each BPE token, h The contextual embedding BERT w for word w is computed as: where α 1,..., α L are trainable parameters. Word-level Inter-Attention. We conduct attention from question to context (passage) based on GloVe word embeddings. Suppose the context word embeddings are {h where D ∈ R k×k is a diagonal matrix and U ∈ R d×k, k is the attention hidden size. To simplify notation, we denote the above attention function as Attn(A, B, C), which linearly combines the vector set C using attention scores computed from vector sets A and B. This resembles the definition of attention in transformer . It follows that the word-level interattention can be rewritten as Attn({h . Therefore, the input vector for each context word and question word is: RNN. In this component, we use two separate bidirectional LSTMs to form the contextualized understanding for C and Q: where h and K is the number of RNN layers. We use variational dropout for the input vector to each layer of RNN, i.e. the dropout mask is shared over different timesteps. Question Understanding. For each question word in Q, we employ one more RNN layer to generate a higher level of understanding of the question. Self-Attention on Question. As the question has integrated previous utterances, the model needs to directly relate the previously mentioned concept with the current question for context understanding. Therefore we employ self-attention on question: is the final representation of question words. Multilevel Inter-Attention. After multiple RNN layers extract different levels of semantic abstraction, we conduct inter-attention from question to context based on these representations. However, the cumulative output dimensions from all previous layers can be very large and computationally inefficient. Here we leverage the history-of-word idea from FusionNet : the attention uses all previous layers to compute scores, but only linearly combines one RNN layer output. In detail, we conduct K + 1 times of multilevel inter-attention from each RNN layer output of question to context {m where HoW is the history-of-word vector: An additional RNN layer is added to context C: Self Attention on the Context. Similar to questions, SDNet applies self-attention to the context. Again, it uses the history-of-word concept to reduce the output dimensionality: The self-attention is followed by an additional RNN layer to generate the final representation of context words: {u 2.5 OUTPUT LAYER Question Condensation. The question is condensed into a single representation vector: where w is a trainable vector. Generating answer span. As SDNet outputs answers of interval forms, the output layer generates the probability that the answer starts and ends at the i-th context word, 1 ≤ i ≤ m: where W S, W E are parameters. The use of GRU is to transfer information from start position to end position computation. Special answer types. SDNet can also output special types of answer, such as affirmation "yes", negation "no" or no answer "unknown". We separately generate the probabilities of these three answers: P Y, P N, P U . For instance, the probability that the answer is "yes", P Y, is computed as: where W Y and w Y are parametrized matrix and vector, respectively. During training, all rounds of questions and answers for the same passage form a batch. The goal is to maximize the probability of the ground-truth answer, including span start/end position, affirmation, negation and no-answer situations. Therefore, we minimize the cross-entropy loss function L: indicate whether the k-th ground-truth answer is a passage span, "yes", "no" and "unknown", respectively. During inference, we pick the largest span/yes/no/unknown probability. The span is constrained to have a maximum length of 15. We evaluated our model on CoQA , a large-scale conversational question answering dataset. In CoQA, many questions require understanding of both the passage and previous rounds of questions and answers, which poses challenge to conventional machine reading models. Table 1 summarizes the domain distribution in CoQA. As shown, CoQA contains passages from multiple domains, and the average number of question answering turns is more than 15 per passage. For each in-domain dataset, 100 passages are in the development set, and 100 passages are in the test set. The rest in-domain dataset are in the training set. The test set also includes all of the out-of-domain passages. We use spaCy for word tokenization and employ the uncased BERT-large model to generate contextual embedding. During training, we use a dropout rate of 0.4 for BERT layer outputs and 0.3 for other layers. We use Adamax as the optimizer, with a learning rate of α = 0.002, β = (0.9, 0.999) and = 10 −8. We train the model for 30 epochs. The gradient is clipped at 10. The word-level attention has a hidden size of 300. The self attention layer for question words has a hidden size of 300. The RNNs for question and context have K = 2 layers and each layer has a hidden size of 125. The multilevel attention from question to context has a hidden size of 250. The self attention layer for context has a hidden size of 250. The final RNN layer for context words has a hidden size of 125. We compare SDNet 2 with the following baseline models: DrQA+PGNet , BiDAF++ (and FlowQA . Aligned with the official leaderboard, we use F 1 as the evaluation metric, which is the harmonic mean of precision and recall at word level between the predicted answer and ground truth. Table 2 shows the performance of SDNet and baseline models. 4 As shown, SDNet achieves significantly better than baseline models. In detail, the single SDNet model improves overall F 1 by 1.6%, compared with previous state-of-art model on CoQA, FlowQA. We also trained an ensemble model consisting of 12 SDNet models with the same structure but different random seeds for initialization. The ensemble model uses the answer from the most number of models as its predicted answer. Ensemble SDNet model further improves overall F 1 score by 2.7%. Figure 3 shows the F 1 score of SDNet on development set during training. As seen, SDNet overpasses all but one baseline models after the second epoch, and achieves state-of-the-art after 8 epochs. Ablation Studies. We conduct ablation studies on SDNet to verify the effectiveness of different parts of the model. As Table 3 shows, our proposed weighted sum of per-layer output from BERT is crucial, boosting the performance by 1.75% compared with the canonical method of using only the last layer's output. This shows that the output from each layer in BERT is useful in downstream tasks. Using BERT-base instead of the BERT-large pretrained model hurts the F 1 score by 2.61%.Variational dropout and self attention can each improve the performance by 0.24% and 0.75% respectively. Contextual history. In SDNet, we utilize conversation history via prepending the current question with previous N rounds of questions and ground-truth answers. We experimented with the effect of N and present the in Table 4. As shown, excluding dialogue history (N = 0) can reduce the F 1 score by as much as 8.56%, manifesting the importance of contextual information in conversational QA task. The performance of our model peaks when N = 2, which was used in the final SDNet model. In this paper, we propose a novel contextual attention-based deep neural network, SDNet, to tackle the conversational question answering task. By leveraging inter-attention and self-attention on passage and conversation history, the model is able to comprehend dialogue flow and the passage. Furthermore, we leverage the latest breakthrough in NLP, BERT, as a contextual embedder. We design the alignment of tokenizers, linear combination and weight-locking techniques to adapt BERT into our model in a computation-efficient way. SDNet achieves superior over previous approaches. On the public dataset CoQA, SDNet outperforms previous state-of-the-art model by 1.6% in overall F 1 score and the ensemble model further improves the F 1 by 2.7%. Our future work is to apply this model to open-domain multiturn QA problem with large corpus or knowledge base, where the target passage may not be directly available. This will be a more realistic setting to human question answering.

A neural method for conversational question answering with attention mechanism and a novel usage of BERT as contextual embedder

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.