Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Min-max formulations have attracted great attention in the ML community due to the rise of deep generative models and adversarial methods, and understanding the dynamics of (stochastic) gradient algorithms for solving such formulations has been a grand challenge. As a first step, we restrict to bilinear zero-sum games and give a systematic analysis of popular gradient updates, for both simultaneous and alternating versions. We provide exact conditions for their convergence and find the optimal parameter setup and convergence rates. In particular, our offer formal evidence that alternating updates converge "better" than simultaneous ones. Min-max optimization has received significant attention recently due to the popularity of generative adversarial networks (GANs) and adversarial training , just to name some examples. Formally, given a bivariate function f (x, y), we aim to find a saddle point (x *, y *) such that f (x *, y) ≤ f (x *, y *) ≤ f (x, y *), ∀x ∈ R n, ∀y ∈ R n. (1.1) Since the beginning of game theory, various algorithms have been proposed for finding saddle points (; ; ; ; ; ; ; ;). Due to its recent resurgence in ML, new algorithms specifically designed for training GANs were proposed (; ; b;). However, due to the inherent non-convexity in deep learning formulations, our current understanding of the convergence behaviour of new and classic gradient algorithms is still quite limited, and existing analysis mostly focused on bilinear games or strongly-convex-strongly-concave games (; ; b; ; b). Nonzero-sum bilinear games, on the other hand, are known to be PPAD-complete (for finding approximate Nash equilibria, see e.g.). In this work, we study bilinear zero-sum games as a first step towards understanding general min-max optimization, although our apply to some simple GAN settings (a). It is well-known that certain gradient algorithms converge linearly on bilinear zero-sum games (; b; ;). These iterative algorithms usually come with two versions: Jacobi style updates or Gauss-Seidel (GS) style. In a Jacobi style, we update the two sets of parameters (i.e., x and y) simultaneously whereas in a GS style we update them alternatingly (i.e., one after the other). Thus, Jacobi style updates are naturally amenable to parallelization while GS style updates have to be sequential, although the latter is usually found to converge faster (and more stable). In numerical linear algebra, the celebrated Stein-Rosenberg theorem formally proves that in solving certain linear systems, GS updates converge strictly faster than their Jacobi counterparts, and often with a larger set of convergent instances. However, this does not readily apply to bilinear zero-sum games. Our main goal here is to answer the following questions about solving bilinear zero-sum games: • When exactly does a gradient-type algorithm converge? • What is the optimal convergence rate by tuning the step size or other parameters? • Can we prove something similar to the Stein-Rosenberg theorem for Jacobi and GS updates? Table 2: Optimal convergence rates. In the second column, β * denotes a specific parameter that depends on σ 1 and σ n (see equation 4.2). In the third column, the linear rates are for large κ. The optimal parameters for both Jacobi and Gauss-Seidel EG algorithms are the same. α denotes the step size (α 1 = α 2 = α), and β 1 and β 2 are hyper-parameters for EG and OGD, as given in §2. Algorithm α β 1 β 2 rate exponent Comment Jacobi and Gauss-Seidel Jacobi OGD 2β 1 β * β 1 ∼ 1 − 1/(6κ 2) β 1 = β 2 = α/2 GS OGD √ 2/σ 1 √ 2σ 1 /(σ Contributions We summarize our main from §3 and §4 in Table 1 and 2 respectively, with supporting experiments given in §5. We use σ 1 and σ n to denote the largest and the smallest singular values of matrix E (see equation 2.1), and κ:= σ 1 /σ n denotes the condition number. The algorithms will be introduced in §2. Note that we generalize gradient-type algorithms but retain the same names. Table 1 shows that in most cases that we study, whenever Jacobi updates converge, the corresponding GS updates converge as well (usually with a faster rate), but the converse is not true (§3). This extends the well-known Stein-Rosenberg theorem to bilinear games. Furthermore, Table 2 tells us that by generalizing existing gradient algorithms, we can obtain faster convergence rates. In the study of GAN training, bilinear games are often regarded as an important simple example for theoretically analyzing and understanding new algorithms and techniques (e.g. ; a; b;). It captures the difficulty in GAN training and can represent some simple GAN formulations (; ; a;). Mathematically, bilinear zero-sum games can be formulated as the following min-max problem: min x∈R n max y∈R n x Ey + b x + c y. The set of all saddle points (see definition in eq. (1.1)) is: Throughout, for simplicity we assume E to be invertible, whereas the seemingly general case with non-invertible E is treated in Appendix G. The linear terms are not essential in our analysis and we take b = c = 0 throughout the paper 1. In this case, the only saddle point is. For bilinear games, it is well-known that simultaneous gradient descent ascent does not converge and other gradient-based algorithms tailored for min-max optimization have been proposed (; ; a;). These iterative algorithms all belong to the class of general linear dynamical systems (LDS, a.k.a. matrix iterative processes). Using state augmentation z (t):= (x (t), y (t) ) we define a general k-step LDS as follows: where the matrices A i and vector d depend on the gradient algorithm (examples can be found in Appendix C.1). Define the characteristic polynomial, with A 0 = −I: The following well-known decides when such a k-step LDS converges for any initialization: Theorem 2.1 (e.g.). The LDS in eq. (2.3) converges for any initialization (z,..., z (k−1) ) iff the spectral radius r:= max{|λ| : p(λ) = 0} < 1, in which case {z (t) } converges linearly with an (asymptotic) exponent r. Therefore, understanding the bilinear game dynamics reduces to spectral analysis. The (sufficient and necessary) convergence condition reduces to that all roots of p(λ) lie in the (open) unit disk, which can be conveniently analyzed through the celebrated Schur's theorem ). The roots of a real polynomial p(λ) = a 0 λ n + a 1 λ n−1 + · · · + a n are within the (open) unit disk of the complex plane iff ∀k ∈ {1, 2, . . ., n}, det(In the theorem above, we denoted 1 S as the indicator function of the event S, i.e. 1 S = 1 if S holds and 1 S = 0 otherwise. For a nice summary of related stability tests, see . We therefore define Schur stable polynomials to be those polynomials whose roots all lie within the (open) unit disk of the complex plane. Schur's theorem has the following corollary (proof included in Appendix B.2 for the sake of completeness): Corollary 2.1 (e.g.). A real quadratic polynomial λ 2 + aλ + b is Schur stable iff b < 1, |a| < 1 + b; A real cubic polynomial λ 3 + aλ 2 + bλ + c is Schur stable iff |c| < 1, Let us formally define Jacobi and GS updates: Jacobi updates take the form while Gauss-Seidel updates replace x (t−i) with the more recent x (t−i+1) in operator T 2, where T 1, T 2: R nk × R nk → R n can be any update functions. For LDS updates in eq. (2.3) we find a nice relation between the characteristic polynomials of Jacobi and GS updates in Theorem 2.3 (proof in Appendix B.1), which turns out to greatly simplify our subsequent analyses: and L i is strictly lower block triangular. Then, the characteristic polynomial of Jacobi updates is p(λ, 1) while that of Gauss-Seidel updates is p(λ, λ). Compared to the Jacobi update, in some sense the Gauss-Seidel update amounts to shifting the strictly lower block triangular matrices L i one step to the left, as p(λ, λ) can be rewritten as det This observation will significantly simplify our comparison between Jacobi and Gauss-Seidel updates. Next, we define some popular gradient algorithms for finding saddle points in the min-max problem We present the algorithms for a general (bivariate) function f although our main will specialize f to the bilinear case in eq. (2.1). Note that we introduced more "step sizes" for our refined analysis, as we find that the enlarged parameter space often contains choices for faster linear convergence (see §4). We only define the Jacobi updates, while the GS counterparts can be easily inferred. We always use α 1 and α 2 to define step sizes (or learning rates) which are positive. The generalized GD update has the following form: When α 1 = α 2, the convergence of averaged iterates (a.k.a. Cesari convergence) for convex-concave games is analyzed in (; ; Nedić &). We study a generalized version of EG, defined as follows: EG was first proposed in with the restriction α 1 = α 2 = γ 1 = γ 2, under which linear convergence was proved for bilinear games. A slightly more generalized version was analyzed in where α 1 = α 2, γ 1 = γ 2, again with linear convergence proved. For later convenience we define β 1 = α 2 γ 1 and β 2 = α 1 γ 2. Optimistic gradient descent (OGD) We study a generalized version of OGD, defined as follows: 10) The original version of OGD was given in with α 1 = α 2 = 2β 1 = 2β 2, and its linear convergence for bilinear games was proved in. A slightly more generalized version with α 1 = α 2 and β 1 = β 2 was analyzed in Mokhtari et al. (2019b), again with linear convergence proved. Momentum method Generalized heavy ball method was analyzed in Gidel et al. (2019b): 11) (2.12) This is a modification of Polyak's heavy ball (HB) , which also motivated Nesterov's accelerated gradient algorithm (NAG) . Note that for both x-update and the y-update, we add a scale multiple of the successive difference (e.g. proxy of the momentum). For this algorithm our below improves those obtained in Gidel et al. (2019b), as will be discussed in §3. EG and OGD as approximations of proximal point algorithm It has been observed recently in Mokhtari et al. (2019b) that for convex-concave games, EG (α 1 = α 2 = γ 1 = γ 2 = η) and OGD (α 1 /2 = α 2 /2 = β 1 = β 2 = η) can be treated as approximations of the proximal point algorithm when η is small. With this , one can show that EG and OGD converge to saddle points sublinearly for smooth convex-concave games (a). We give a brief introduction of the proximal point algorithm in Appendix A (including a linear convergence for the slightly generalized version). The above algorithms, when specialized to a bilinear function f (see eq. (2.1)), can be rewritten as a 1-step or 2-step LDS (see. eq. (2.3)). See Appendix C.1 for details. With tools from §2, we formulate necessary and sufficient conditions under which a gradient-based algorithm converges for bilinear games. We sometimes use "J" as a shorthand for Jacobi style updates and "GS" for Gauss-Seidel style updates. For each algorithm, we first write down the characteristic polynomials (see derivation in Appendix C.1) for both Jacobi and GS updates, and present the exact conditions for convergence. Specifically, we show that in many cases the GS convergence regions strictly include the Jacobi convergence regions. The proofs for Theorem 3.1, 3.2, 3.3 and 3.4 can be found in Appendix C.2, C.3, C.4, and C.5, respectively. GD The characteristic equations can be computed as: Scaling symmetry From section 3 we obtain a scaling symmetry (α 1, α 2) → (tα 1, α 2 /t), with t > 0. With this symmetry we can always fix α 1 = α 2 = α. This symmetry also holds for EG and momentum. For OGD, the scaling symmetry is slightly different with (α 1, β 1, α 2, β 2) → (tα 1, tβ 1, α 2 /t, β 2 /t), but we can still use this symmetry to fix α 1 = α 2 = α. Theorem 3.1 (GD). Jacobi GD and Gauss-Seidel GD do not converge. However, Gauss-Seidel GD can have a limit cycle while Jacobi GD always diverges. When α 1 = α 2, this theorem was proved by Gidel et al. (2019a). EG The characteristic equations can be computed as: Theorem 3.2 (EG). For generalized EG with α 1 = α 2 = α and γ i = β i /α, Jacobi and Gauss-Seidel updates achieve linear convergence iff for any singular value σ of E, we have:, the convergence region of GS updates strictly include that of Jacobi updates. OGD The characteristic equations can be computed as: Theorem 3.3 (OGD). For generalized OGD with α 1 = α 2 = α, Jacobi and Gauss-Seidel updates achieve linear convergence iff for any singular value σ of E, we have: The convergence region of GS updates strictly include that of Jacobi updates. Momentum The characteristic equations can be computed as: Theorem 3.4 (momentum). For the generalized momentum method with α 1 = α 2 = α, the Jacobi updates never converge, while the GS updates converge iff for any singular value σ of E, we have: This condition implies that at least one of β 1, β 2 is negative. Prior to our work, only sufficient conditions for linear convergence were given for the usual EG and OGD; see §2 above. For the momentum method, our improves upon Gidel et al. (2019b) where they only considered specific cases of parameters. For example, they only considered β 1 = β 2 ≥ −1/16 for Jacobi momentum (but with explicit rate of divergence), and β 1 = −1/2, β 2 = 0 for GS momentum (with convergence rate). Our Theorem 3.4 gives a more complete picture and formally justifies the necessity of negative momentum. In the theorems above, we used the term "convergence region" to denote a subset of the parameter space (with parameters α, β or γ) where the algorithm converges. Our shares similarity with the celebrated Stein-Rosenberg theorem , which only applies to solving linear systems with non-negative matrices (if one were to apply it to our case, the matrix S in eq. (F.1) in Appendix F needs to have non-zero diagonal entries, which is not possible). In this sense, our extend the Stein-Rosenberg theorem to cover nontrivial bilinear games. In this section we study the optimal convergence rates of EG and OGD. We define the exponent of linear convergence as r = lim t→∞ ||z (t) ||/||z (t−1) || which is the same as the spectral radius. For ease of presentation we fix α 1 = α 2 = α > 0 (using scaling symmetry) and we use r * to denote the optimal exponent of linear convergence (achieved by tuning the parameters α, β, γ). Our show that by generalizing gradient algorithms one can obtain better convergence rates. Theorem 4.1 (EG optimal). Both Jacobi and GS EG achieve the optimal exponent of linear conver- Note that we defined β i = γ i α in Section 2. In other words, we are taking very large extra-gradient steps (γ i → ∞) and very small gradient steps (α → 0).. For Jacobi OGD with β 1 = β 2 = β, to achieve the optimal exponent of linear convergence, we must have α ≤ 2β. For the original OGD with α = 2β, the optimal exponent of linear convergence r * satisfies. For GS OGD with β 2 = 0, the optimal exponent of convergence is r * = (κ 2 − 1)/(κ 2 + 1), at α = √ 2/σ 1 and Remark The original OGD with α = 2β may not always be optimal. For example, take one-dimensional bilinear game and σ = 1, and denote the spectral radius given α, β as r(α, β). If we fix α = 1/2, by numerically solving section 3 we have i.e, α = 1/2, β = 1/3 is a better choice than α = 2β = 1/2. Numerical method We provide a numerical method for finding the optimal exponent of linear convergence, by realizing that the unit disk in Theorem 2.2 is not special. Let us call a polynomial to be r-Schur stable if all of its roots lie within an (open) disk of radius r in the complex plane. We can scale the polynomial with the following lemma: With the lemma above, one can rescale the Schur conditions and find the convergence region where the exponent of linear convergence is at most r (r < 1). A simple binary search would allow one to find a better and better convergence region. See details in Appendix D.3. Bilinear game We run experiments on a simple bilinear game and choose the optimal parameters as suggested in Theorem 4.1 and 4.2. The are shown in the left panel of Figure 1, which confirms the predicted linear rates. Figure 2: Heat maps of the spectral radii of different algorithms. We take σ = 1 for convenience. The horizontal axis is α and the vertical axis is β. Top row: Jacobi updates; Bottom row: Gauss-Seidel updates. Columns (left to right): EG; OGD; momentum. If the spectral radius is strictly less than one, it means that our algorithm converges. In each column, the Jacobi convergence region is contained in the GS convergence region (for EG we need an additional assumption, see Theorem 3.2). Density plots We show the density plots (heat maps) of the spectral radii in Figure 2. We make plots for EG, OGD and momentum with both Jacobi and GS updates. These plots are made when β 1 = β 2 = β and they agree with our theorems in §3. Wasserstein GAN As in , we consider a WGAN that learns the mean of a Gaussian: where s(x) is the sigmoid function. It can be shown that near the saddle point (θ *, φ *) = (0, v) the min-max optimization can be treated as a bilinear game (Appendix E.1). With GS updates, we find that Adam diverges, SGD goes around a limit cycle, and EG converges, as shown in the middle panel of Figure 1. We can see that Adam does not behave well even in this simple task of learning a single two-dimensional Gaussian with GAN. Our next experiment shows that generalized algorithms may have an advantage over traditional ones. Inspired by Theorem 4.1, we compare the convergence of two EGs with the same parameter β = αγ, and find that with scaling, EG has better convergence, as shown in the right panel of Figure 1. Finally, we compare Jacobi updates with GS updates. In Figure 3, we can see that GS updates converge even if the corresponding Jacobi updates do not. Mixtures of Gaussians (GMMs) Our last experiment is on learning GMMs with a vanilla GAN that does not directly fall into our analysis. We choose a 3-hidden layer ReLU network for both the generator and the discriminator, and each hidden layer has 256 units. We find that for GD and OGD, Jacobi style updates converge more slowly than GS updates, and whenever Jacobi updates converge, the corresponding GS updates converges as well. These comparisons can be found in Figure 4 and 5, which implies the possibility of extending our to non-bilinear games. Interestingly, we observe that even Jacobi GD converges on this example. We provide additional comparison between the Jacobi and GS updates of Adam in Appendix E.2. In this work we focus on the convergence behaviour of gradient-based algorithms for solving bilinear games. By drawing a connection to discrete linear dynamical systems (§2) and using Schur's theorem, we provide necessary and sufficient conditions for a variety of gradient algorithms, for both simultaneous (Jacobi) and alternating (Gauss-Seidel) updates. Our show that Gauss-Seidel updates converge more easily than Jacobi updates. Furthermore, we find the optimal exponents of linear convergence for EG and OGD, and provide a numerical method for searching that exponent. We performed a number of experiments to validate our theoretical findings and suggest further analysis. There are many future directions to explore. For example, our preliminary experiments on GANs suggest that similar (local) might be obtained for more general games. Indeed, the local convergence behaviour of min-max nonlinear optimization can be studied through analyzing the spectrum of the Jacobian matrix of the update operator (see, e.g., ; Gidel et al. (2019b) ). We believe our framework that draws the connection to linear discrete dynamic systems and Schur's theorem is a powerful machinery that can be applied in such problems and beyond. It would be interesting to generalize our to the constrained case (even for bilinear games), initiated in the recent work of. Extending our to account for stochastic noise (as empirically tested in our experiments) is another interesting direction, with some initial in Gidel et al. (2019a A PROXIMAL POINT (PP) ALGORITHM PP was originally proposed by with α 1 = α 2 and then carefully studied by. The linear convergence for bilinear games was also proved in the same reference. Note that we do not consider Gauss-Seidel PP since we do not get a meaningful solution after a shift of steps 2. where x (t+1) and y (t+1) are given implicitly by solving the equations above. For bilinear games, one can derive that: We can compute the exact form of the inverse matrix, but perhaps an easier way is just to compute the spectrum of the original matrix (the same as Jacobi GD except that we flip the signs of α i) and perform λ → 1/λ. Using the fact that the eigenvalues of a matrix are reciprocals of the eigenvalues of its inverse, the characteristic equation is: With the scaling symmetry (α 1, α 2) → (tα 1, α 2 /t), we can take α 1 = α 2 = α > 0. With the notations in Corollary 2.1, we have a = −2/(1 + α 2 σ 2) and b = 1/(1 + α 2 σ 2), and it is easy to check |a| < 1 + b and b < 1 are always satisfied, which means linear convergence is always guaranteed. Hence, we have the following theorem: Theorem A.1. For bilinear games, the proximal point algorithm always converges linearly. Although the proximal point algorithm behaves well, it is rarely used in practice since it is an implicit method, i.e., one needs to solve (x (t+1), y In this section we apply Theorem 2.1 to prove Theorem 2.3, an interesting connection between Jacobi and Gauss-Seidel updates: and L i is strictly lower block triangular. Then, the characteristic polynomial of Jacobi updates is p(λ, 1) while that of Gauss-Seidel updates is p(λ, λ). Let us first consider the block linear iterative process in the sense of Jacobi (i.e., all blocks are updated simultaneously):...... where A i,j is the j-th column block of A i. For each matrix A i, we decompose it into the sum where L i is the strictly lower block triangular part and U i is the upper (including diagonal) block triangular part. Theorem 2.1 indicates that the convergence behaviour of equation B.1 is governed by the largest modulus of the roots of the characteristic polynomial: Alternatively, we can also consider the updates in the sense of Gauss-Seidel (i.e., blocks are updated sequentially): We can rewrite the Gauss-Seidel update elegantly 3 as: i.e., where L k+1:= 0. Applying Theorem 2.1 again we know the convergence behaviour of the GaussSeidel update is governed by the largest modulus of roots of the characteristic polynomial: Note that A 0 = −I and the factor det(I − L 1) −1 can be discarded since multiplying a characteristic polynomial by a non-zero constant factor does not change its roots. B.2 PROOF OF COROLLARY 2.1 Corollary 2.1 (e.g.). A real quadratic polynomial λ 2 + aλ + b is Schur stable iff b < 1, |a| < 1 + b; A real cubic polynomial λ 3 + aλ 2 + bλ + c is Schur stable iff |c| < 1, Proof. It suffices to prove the for quartic polynomials. We write down the matrices: We require det(2 and thus |c − ad| < 1 − d 2 due to the first condition. δ 4 > 0 simplifies to: 14) which yields |a + c| < |b + d + 1|. Finally, δ 3 > 0 reduces to: Denote p(λ):= λ 4 + aλ 3 + bλ 2 + cλ + d, we must have p > 0 and p(−1) > 0, as otherwise there is a real root λ 0 with |λ 0 | ≥ 1. Hence we obtain b + d + 1 > |a + c| > 0. Also, from |c − ad| < 1 − d 2, we know that: So, the second factor in B.15 is negative and the positivity of the first factor reduces to: To obtain the Schur condition for cubic polynomials, we take d = 0, and the quartic Schur condition becomes: To obtain the Schur condition for quadratic polynomials, we take c = 0 in the above and write: The proof is now complete. Some of the following proofs in Appendix C.4 and C.5 rely on Mathematica code (mostly with the built-in function Reduce) but in principle the code can be verified manually using cylindrical algebraic decomposition. In this appendix, we derive the exact forms of LDSs (eq. (2.3)) and the characteristic polynomials for all gradient-based methods introduced in §2, with eq. (2.4). The following lemma is well-known and easy to verify using Schur's complement: Gradient descent From equation 2.6 the update equation of Jacobi GD can be derived as: and with Lemma C.1, we compute the characteristic polynomial as in eq. (2.4): With spectral decomposition we obtain equation 3.1. Taking α 2 → λα 2 and with Theorem 2.3 we obtain the corresponding GS updates. Therefore, the characteristic polynomials for GD are: Extra-gradient From eq. (2.7) and eq. (2.8), the update of Jacobi EG is: the characteristic polynomial is: Since we assumed α 2 > 0, we can left multiply the second row by β 2 E/α 2 and add it to the first row. Hence, we obtain: With Lemma C.1 the equation above becomes: which simplifies to equation 3.2 with spectral decomposition. Note that to obtain the GS polynomial, we simply take α 2 → λα 2 in the Jacobi polynomial as shown in Theorem 2.3. For the ease of reading we copy the characteristic equations for generalized EG: Optimistic gradient descent We can compute the LDS for OGD with eq. (2.9) and eq. (2.10): With eq. (2.4), the characteristic polynomial for Jacobi OGD is Taking the determinant and with Lemma C.1 we obtain equation 3.6. The characteristic polynomial for GS updates in equation 3.7 can be subsequently derived with Theorem 2.3, by taking (α 2, β 2) → (λα 2, λβ 2). For the ease of reading we copy the characteristic polynomials from the main text as: Momentum method With eq. (2.11) and eq. (2.12), the LDS for the momentum method is: From eq. (2.4), the characteristic polynomial for Jacobi momentum is Taking the determinant and with Lemma C.1 we obtain equation 3.10, while equation 3.11 can be derived with Theorem 2.3, by taking α 2 → λα 2. For the ease of reading we copy the characteristic polynomials from the main text as: C.2 PROOF OF THEOREM 3.1: SCHUR CONDITIONS OF GD Theorem 3.1 (GD). Jacobi GD and Gauss-Seidel GD do not converge. However, Gauss-Seidel GD can have a limit cycle while Jacobi GD always diverges. Proof. With the notations in Corollary 2.1, for Jacobi GD, b = 1 + α 2 σ 2 > 1. For Gauss-Seidel GD, b = 1. The Schur conditions are violated. For generalized EG with α 1 = α 2 = α and γ i = β i /α, Jacobi and Gauss-Seidel updates achieve linear convergence iff for any singular value σ of E, we have: If β 1 + β 2 + α 2 < 2/σ 2 1, the convergence region of GS updates strictly include that of Jacobi updates. Both characteristic polynomials can be written as a quadratic polynomial λ 2 + aλ + b, where: Compared to Jacobi EG, the only difference between Gauss-Seidel and Jacobi updates is that the α 2 σ 2 in b is now in a, which agrees with Theorem 2.3. Using Corollary 2.1, we can derive the Schur conditions equation 3.4 and equation 3.5. More can be said if β 1 + β 2 is small. For instance, if β 1 + β 2 + α 2 < 2/σ More precisely, to show that the GS convergence region strictly contains that of the Jacobi convergence region, simply take β 1 = β 2 = β. The Schur condition for Jacobi EG and Gauss-Seidel EG are separately: It can be shown that if β = α 2 /3 and α → 0, equation C.21 is always violated whereas equation C.22 is always satisfied. Conversely, we give an example when Jacobi EG converges while GS EG does not. Let β 1 σ 2 = β 2 σ 2 ≡ 3 2, then Jacobi EG converges iff α 2 σ 2 < 3 4 while GS EG converges iff α 2 σ 2 < 1 4. In this subsection, we fill in the details of the proof of Theorem 3.3, by first deriving the Schur conditions of OGD, and then studying the relation between Jacobi OGD and GS OGD. Theorem 3.3 (OGD). For generalized OGD with α 1 = α 2 = α, Jacobi and Gauss-Seidel updates achieve linear convergence iff for any singular value σ of E, we have: The convergence region of GS updates strictly include that of Jacobi updates. The Jacobi characteristic polynomial is now quartic in the form λ 4 + aλ 3 + bλ 2 + cλ + d, with Comparably, the GS polynomial equation 3.7 can be reduced to a cubic one λ 3 + aλ 2 + bλ + c with First we derive the Schur conditions equation 3.8 and equation 3.9. Note that other than Corollary 2.1, an equivalent Schur condition can be read from Cheng & Chiou (2007, Theorem 1) as: Theorem C.1 (With equation C.23 and Theorem C.1, it is straightforward to derive equation 3.8. With equation C.24 and Corollary 2.1, we can derive equation 3.9 without much effort. Now, let us study the relation between the convergence region of Jacobi OGD and GS OGD, as given in equation 3.8 and equation 3.9. Namely, we want to prove the last sentence of Theorem 3.3. The outline of our proof is as follows. We first show that each region of (α, β 1, β 2) described in equation 3.8 (the Jacobi region) is contained in the region described in equation 3.9 (the GS region). Since we are only studying one singular value, we slightly abuse the notations and rewrite β i σ as β i (i = 1, 2) and ασ as α. From equation 3.6 and equation 3.7, β 1 and β 2 can switch. WLOG, we assume β 1 ≥ β 2. There are four cases to consider: The third Jacobi condition in equation 3.8 now is redundant, and we have α > β 1 or α < β 2 for both methods. Solving the quadratic feasibility condition for α gives: where u = (β 1 β 2 + 1)(. On the other hand, assume α > β 1, the first and third GS conditions are automatic. Solving the second gives: 2 /2 and g(β 2):= (β 2 + 4 + 5β 2 2)/(2(1 + β 2 2)), and one can show that (C.28) Furthermore, it can also be shown that given 0 < β 2 < 1 and β 2 ≤ β 1 < g(β 2), we have • β 1 ≥ β 2 = 0. The Schur condition for Jacobi and Gauss-Seidel updates reduces to: One can show that given β 1 ∈, we have Reducing the first, second and fourth conditions of equation 3.8 yields: This region contains the Jacobi region. It can be similarly proved that even within this larger region, GS Schur condition equation 3.9 is always satisfied. • β 2 ≤ β 1 < 0. We have u < 0, tv < 0 and thus α < (u + √ u 2 + tv)/t < 0. This contradicts our assumption that α > 0. Combining the four cases above, we know that the Jacobi region is contained in the GS region. To show the strict inclusion, take β 1 = β 2 = α/5 and α → 0. One can show that as long as α is small enough, all the Jacobi regions do not contain this point, each of which is described with a singular value in equation 3.8. However, all the GS regions described in equation 3.9 contain this point. The proof above is still missing some details. We provide the proofs of equation C.26, equation C.28, equation C.29 and equation C.32 in the sub-sub-sections below, with the help of Mathematica, although one can also verify these claims manually. Moreover, a one line proof of the inclusion can be given with Mathematica code, as shown in Section C.4.5. The fourth condition of equation 3.8 can be rewritten as: where we used |β 1 β 2 | < 1 in both cases. So, equation C.33 becomes: Combining with α > β 1 or α < β 2 obtained from the second condition, we have: The first case is not possible, with the following code: u = (b1 b2 + 1) (b1 + b2); v = b1 b2 (b1 b2 + 1) (b1 b2 -3); t = (b1^2 + 1) (b2^2 + 1); Reduce[b2 t > u -Sqrt[u^2 + t v] && b1 >= b2 > 0 && Abs[b1 b2] < 1], and we have: Therefore, the only possible case is β 1 < α < (u + √ u 2 + tv)/t. Where the feasibility region can be solved with: What we get is: 1 + b2^2) ), {b2, b1}], we can remove the first constraint and get: 0 < b2 < 1 && b2 <= b1 < b2/(2 (1 + b2^2)) + 1/2 Sqrt[(4 + 5 b2^2)/(1 + b2^2)^2]. The second Jacobi condition simplifies to α > β 1 and the fourth simplifies to equation C.34. Combining with the first Jacobi condition: we have: This can be further simplified to achieve equation C.32. In fact, there is another very simple proof: Reduce[ForAll[{b1, b2, a}, (a -b1) (a -b2) > 0 && (a + b1) (a + b2) > -4 && Abs[b1 b2] < 1 && a^2 (b1^2 + 1) (b2^2 + 1) < (b1 b2 + 1) (2 a (b1 + b2) + b1 b2 (b1 b2 -3)), (a -b1) (a -b2) > 0 && (a + b1) (a + b2) < 4 && (a b1 + 1) (a b2 + 1) > (1 + b1 b2)^2], {b2, b1, a}] True. However, this proof does not tell us much information about the range of our variables. Theorem 3.4 (momentum). For the generalized momentum method with α 1 = α 2 = α, the Jacobi updates never converge, while the GS updates converge iff for any singular value σ of E, we have: This condition implies that at least one of β 1, β 2 is negative. Jacobi condition We first rename ασ as al and β 1, β 2 as b1, b2. With Theorem C.1: We obtain: {Abs[b1 b2] < 1, Abs[2 + b1 + b2] < 3 + b1 b2, al^2 > 0, al^2 + 4 (1 + b1) (1 + b2) > 0, al^2 (-1 + b1 b2)^2 < 0}. The last condition is never satisfied and thus Jacobi momentum never converges. Gauss-Seidel condition With Theorem C.1, we compute: The is: {Abs[b1 b2] < 1, Abs[2 -al^2 + b1 + b2] < 3 + b1 b2, al^2 > 0, 4 (1 + b1) (1 + b2) > al^2, al^2 (b1 + b2 + (-2 + al^2 -b1) b1 b2 + b1 (-1 + 2 b1) b2^2) < 0}, which can be further simplified to equation??. With Theorem 3.4, we can actually show that in general at least one of β 1 and β 2 must be negative. There are three cases to consider, and in each case we simplify equation??: 1. β 1 β 2 = 0. WLOG, let β 2 = 0, and we obtain −1 < β 1 < 0 and α 2 σ 2 < 4(1 + β 1). (C.36) 2. β 1 β 2 > 0. We have 3. β 1 β 2 < 0. WLOG, we assume β 1 ≥ β 2. We obtain: The constraints for α are α > 0 and: These conditions can be further simplified by analyzing all singular values. They only depend on σ 1 and σ n, the largest and the smallest singular values. Now, let us derive equation C.37, equation C.38 and equation C.39 more carefully. Note that we use a for ασ. Reduce[Abs[b1 b2] < 1 && Abs[-a^2 + b1 + b2 + 2] < b1 b2 + 3 && 4 (b1 + 1) (b2 + 1) > a^2 && a^2 b1 b2 < (1 -b1 b2) (2 b1 b2 -b1 -b2) && b1 b2 > 0 && a > 0, {b2, b1, a}] -1 < b2 < 0 && -1 < b1 < 0 && 0 < a < Sqrt[4 + 4 b1 + 4 b2 + 4 b1 b2] C.5.4 PROOF OF EQUATIONS C.38 AND C.39 Reduce[Abs[b1 b2] < 1 && Abs[-a^2 + b1 + b2 + 2] < b1 b2 + 3 && 4 (b1 + 1) (b2 + 1) > a^2 && a^2 b1 b2 < (1 -b1 b2) (2 b1 b2 -b1 -b2) && b1 b2 < 0 && b1 >= b2 && a > 0, {b2, b1, a}] (-1 < b2 <= -(1/3) && ((0 < b1 <= b2/(-1 + 2 b2) && 0 < a < Sqrt[4 + 4 b1 + 4 b2 + 4 b1 b2]) || (b2/(-1 + 2 b2) < b1 < -(1/(3 b2)) && Sqrt[(-b1 -b2 + 2 b1 b2 + b1^2 b2 + b1 b2^2 -2 b1^2 b2^2)/(b1 b2)] < a < Sqrt[4 + 4 b1 + 4 b2 + 4 b1 b2]))) || (-(1/3) < b2 < 0 && ((0 < b1 <= b2/(-1 + 2 b2) && 0 < a < Sqrt[4 + 4 b1 + 4 b2 + 4 b1 b2]) || (b2/(-1 + 2 b2) < b1 < -(b2/(1 + 2 b2)) && Sqrt[(-b1 -b2 + 2 b1 b2 + b1^2 b2 + b1 b2^2 -2 b1^2 b2^2)/(b1 b2)] < a < Sqrt[4 + 4 b1 + 4 b2 + 4 b1 b2]))) Some further simplication yields equation C.38 and equation C.39. For bilinear games and gradient-based methods, a Schur condition defines the region of convergence in the parameter space, as we have seen in Section 3. However, it is unknown which setting of parameters has the best convergence rate in a Schur stable region. We explore this problem now. Due to Theorem 3.1, we do not need to study GD. The remaining cases are EG, OGD and GS momentum (Jacobi momentum does not converge due to Theorem 3.4). Analytically (Section D.1 and D.2), we study the optimal linear rates for EG and special cases of generalized OGD (Jacobi OGD with β 1 = β 2 and Gauss-Seidel OGD with β 2 = 0). The special cases include the original form of OGD. We also provide details for the numerical method described at the end of Section 4. The optimal spectral radius is obtained by solving another min-max optimization problem: where θ denotes the collection of all hyper-parameters, and r(θ, σ) is defined as the spectral radius function that relies on the choice of parameters and the singular value σ. We also use Sv(E) to denote the set of singular values of E. In general, the function r(θ, σ) is non-convex and thus difficult to analyze. However, in the special case of quadratic characteristic polynomials, it is possible to solve equation D.1. This is how we will analyze EG and special cases of OGD, as r(θ, σ) can be expressed using root functions of quadratic polynomials. For cubic and quartic polynomials, it is in principle also doable as we have analytic formulas for the roots. However, these formulas are extremely complicated and difficult to optimize and we leave it for future work. For EG and OGD, we will show that the optimal linear rates depend only on the conditional number κ:= σ 1 /σ n. For simplicity, we always fix α 1 = α 2 = α > 0 using the scaling symmetry studied in Section 3. D.1 PROOF OF THEOREM 4.1: OPTIMAL CONVERGENCE RATE OF EG Theorem 4.1 (EG optimal). Both Jacobi and GS EG achieve the optimal exponent of linear convergence r * = (κ 2 − 1)/(κ 2 + 1) at α → 0 and β 1 = β 2 = 2/(σ 2 1 + σ 2 n). As κ → ∞, r * → 1 − 2/κ 2. For Jacobi updates, if β 1 = β 2 = β, by solving the roots of equation 3.2, the min-max problem is: If σ 1 = σ n = σ, we can simply take α → 0 and β = 1/σ 2 to obtain a super-linear convergence rate. Otherwise, let us assume σ 1 > σ n. We obtain a lower bound by taking α → 0 and equation D.2 reduces to: The optimal solution is given at 1 − βσ 2 n = βσ From general β 1, β 2, it can be verified that the optimal radius is achieved at β 1 = β 2 and the problem reduces to the previous case. The optimization problem is: In the first case, a lower bound is obtained at α 2 = (β 1 − β 2) 2 σ 2 /4 and thus the objective only depends on β 1 + β 2. In the second case, the lower bound is obtained at α → 0 and β 1 → β 2. Therefore, the function is optimized at β 1 = β 2 and α → 0. Our analysis above does not mean that α → 0 and β 1 = β 2 = 2/(σ 2 1 + σ 2 n) is the only optimal choice. For example, when σ 1 = σ n = 1, we can take β 1 = 1 + α and β 2 = 1 − α to obtain a super-linear convergence rate. For Gauss-Seidel updates and β 1 = β 2 = β, we do the following optimization: where by solving equation 3.3: r(σ, β, σ 2) is quasi-convex in σ 2, so we just need to minimize over α, β at both end points. Hence, equation D.5 reduces to: min α,β max{r(α, β, σ 1), r(α, β, σ n)}. By arguing over three cases: n, we find that the minimum (κ 2 − 1)/(κ 2 + 1) can be achieved at α → 0 and β = 2/(σ 2 1 + σ 2 n), the same as Jacobi EG. This is because α → 0 decouples x and y and it does not matter whether the update is Jacobi or GS. For general β 1, β 2, it can be verified that the optimal radius is achieved at β 1 = β 2. We do the following transformation: β i → ξ i − α 2 /2, so that the characteristic polynomial becomes: Denote ξ 1 + ξ 2 = φ, and (ξ 1 − α 2 /2)(ξ 2 − α 2 /2) = ν, we have: The discriminant is ∆:= σ 2 (σ 2 (φ 2 − 4ν) − 4α 2 ). We discuss two cases: 1. φ 2 − 4ν < 0. We are minimizing: with a ∨ b:= max{a, b} a shorthand. A minimizer is at α → 0 and ν → φ 2 /4 (since φ 2 < 4ν), where β 1 = β 2 = 2/(σ 2 1 + σ 2 n) and α → 0. 2. φ 2 − 4ν ≥ 0. A lower bound is:. This is only possible if α → 0 and φ 2 → 4ν, which yields β 1 = β 2 = 2/(σ From what has been discussed, the optimal radius is (κ 2 − 1)/(κ 2 + 1) which can be achieved at β 1 = β 2 = 2/(σ 2 1 + σ 2 n) and α → 0. Again, this might not be the only choice. For instance, take σ 1 = σ 2 n = 1, from equation 3.3, a super-linear convergence rate can be achieved at β 1 = 1 and D.2 PROOF OF THEOREM 4.2: OPTIMAL CONVERGENCE RATE OF OGD Theorem 4.2 (OGD optimal). For Jacobi OGD with β 1 = β 2 = β, to achieve the optimal linear rate, we must have α ≤ 2β. For the original OGD with α = 2β, the optimal linear rate r * satisfies. For Gauss-Seidel OGD with β 2 = 0, the optimal linear rate is r * = (κ 2 − 1)/(κ 2 + 1), at α = √ 2/σ 1 and For OGD, the characteristic polynomials equation 3.6 and equation 3.7 are quartic and cubic separately, and thus optimizing the spectral radii for generalized OGD is difficult. However, we can study two special cases: for Jacobi OGD, we take β 1 = β 2; for Gauss-Seidel OGD, we take β 2 = 0. In both cases, the spectral radius functions can be obtained by solving quadratic polynomials. We assume β 1 = β 2 = β in this subsection. The characteristic polynomial for Jacobi OGD equation 3.6 can be written as: (D.10) Factorizing it gives two equations which are conjugate to each other: 11) The roots of one equation are the conjugates of the other equation. WLOG, we solve λ(λ − 1) + i(λα − β)σ = 0 which gives (1/2)(u ± v), where (D.12), v can be expressed as: therefore, the spectral radius r(α, β, σ) satisfies: (D.14) and the minimum is achieved at α = 2β. From now on, we assume α ≤ 2β, and thus v = a + ib. We write: E SUPPLEMENTARY MATERIAL FOR SECTIONS 5 AND 6 We provide supplementary material for Sections 5 and 6. We first prove that when learning the mean of a Gaussian, WGAN is locally a bilinear game in Appendix E.1. For mixtures of Gaussians, we provide supplementary experiments about Adam in Appendix E.2. This implies that in some cases, Jacobi updates are better than GS updates. We further verify this claim in Appendix E.3 by showing an example of OGD on bilinear games. Optimizing the spectral radius given a certain singular value is possible numerically, as in Appendix E.4. Inspired by , we consider the following WGAN : with s(x):= 1/(1 + e −x) the sigmoid function. We study the local behavior near the saddle point (v, 0), which depends on the Hessian: with E v a shorthand for E x∼N (v,σ 2 I) and E φ for E z∼N (φ,σ 2 I). At the saddle point, the Hessian is simplified as: Therefore, this WGAN is locally a bilinear game. Given the same parameter settings as in Section 5, we train the vanilla GAN using Adam, with the step size α = 0.0002, and β 1 = 0.9, β 2 = 0.999. As shown in Figure 6, Jacobi updates converge faster than the corresponding GS updates.: Contour plot of spectral radius equal to 0.8. The red curve is for the Jacobi polynomial and the blue curve is for the GS polynomial. The GS region is larger but for some parameter settings, Jacobi OGD achieves a faster convergence rate. Take α = 0.9625, β 1 = β 2 = β = 0.5722, and σ = 1, the Jacobi and GS OGD radii are separately 0.790283 and 0.816572 (by solving equation 3.6 and equation 3.7), which means that Jacobi OGD has better performance for this setting of parameters. A more intuitive picture is given as Figure 7, where we take β 1 = β 2 = β. We minimize r(θ, σ) for a given singular value numerically. WLOG, we take σ = 1, since we can rescale parameters to obtain other values of σ. We implement grid search for all the parameters within the range [−2, 2] and step size 0.05. For the step size α, we take it to be positive. We use {a, b, s} as a shorthand for {a, a + s, a + 2s, . . ., b}. • We first numerically solve the characteristic polynomial for Jacobi OGD equation 3.6, fixing α 1 = α 2 = α with scaling symmetry. With α ∈ {0, 2, 0.05}, β i ∈ {−2, 2, 0.05}, the best parameter setting is α = 0.7, β 1 = 0.1 and β 2 = 0.6. β 1 and β 2 can be switched. The optimal radius is 0.6. • We also numerically solve the characteristic polynomial for Gauss-Seidel OGD equation 3.7, fixing α 1 = α 2 = α with scaling symmetry. With α ∈ {0, 2, 0.05}, β i ∈ {−2, 2, 0.05}, the best parameter setting is α = 1.4, β 1 = 0.7 and β 2 = 0. β 1 and β 2 can be switched. The optimal rate is 1/(5 √ 2). This rate can be further improved to be zero where α = √ 2, β 1 = 1/ √ 2 and β 2 = 0. • Finally, we numerically solve the polynomial for Gauss-Seidel momentum equation 3.11, with the same grid. The optimal parameter choice is α = 1.8, β 1 = −0.1 and β 2 = −0.05. β 1 and β 2 can be switched. The optimal rate is 0.5. In this appendix, we interpret the gradient-based algorithms (except PP) we have studied in this paper as splitting methods , for both Jacobi and Gauss-Seidel updates. By doing this, one can understand our algorithms better in the context of numerical linear algebra and compare our in Section 3 with the Stein-Rosenberg theorem. For EG, we need to compute an inverse: Given det(α 1 α 2 I + β 1 β 2 EE) = 0, the inverse always exists. The splitting method can also work for second-step methods, such as OGD and momentum. We split S = M − N − P and solve: For OGD, we have: For EG, we need to compute an inverse: The splitting method can also work for second-step methods, such as OGD and momentum. We split S = M − N − P and solve: z t+1 = M −1 N z t + M In this paper we considered the bilinear game when E is a non-singular square matrix for simplicity. Now let us study the general case where E ∈ R m×n. As stated in Section 2, saddle points exist iff b ∈ R(E), c ∈ R(E). The set of saddle points is: {(x, y)|y ∈ N (E), x ∈ N (E)}. (G.3)

We systematically analyze the convergence behaviour of popular gradient algorithms for solving bilinear games, with both simultaneous and alternating updates.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Most approaches in generalized zero-shot learning rely on cross-modal mapping between an image feature space and a class embedding space or on generating artificial image features. However, learning a shared cross-modal embedding by aligning the latent spaces of modality-specific autoencoders is shown to be promising in (generalized) zero-shot learning. While following the same direction, we also take artificial feature generation one step further and propose a model where a shared latent space of image features and class embeddings is learned by aligned variational autoencoders, for the purpose of generating latent features to train a softmax classifier. We evaluate our learned latent features on conventional benchmark datasets and establish a new state of the art on generalized zero-shot as well as on few-shot learning. Moreover, our on ImageNet with various zero-shot splits show that our latent features generalize well in large-scale settings. Generalized zero-shot learning (GZSL) is a classification task where no labeled training examples are available from some of the classes. Many approaches learn a mapping between images and their class embeddings BID11 BID19 BID0. For instance, ALE maps CNN features of images to a perclass attribute space. An orthogonal approach to GZSL is to augment data by generating artificial image features, such as BID21 who proposed to generate image features via a conditional WGAN. As a third approach, BID16 proposed to learn a latent space embedding by transforming both modalities to the latent spaces of autoencoders and match the corresponding distributions by minimizing the Maximum Mean Discrepancy (MMD). Learning such cross-modal embeddings can be beneficial for potential downstream tasks that require multimodal fusion. In this regard, BID13 recently used a cross-modal autoencoder to extend visual question answering to previously unseen objects. Although recent cross-modal autoencoder architectures represent class prototypes in a latent space BID10 BID16, better generalization can be achieved if the shared representation space is more amenable to interpolation between different classes. Variational Autoencoders (VAEs) are known for their capability in accurate interpolation between representations in their latent space, i.e. as demonstrated for sentence interpolation BID2 and image interpolation BID6. Hence, in this work, we train VAEs to encode and decode features from different modalities, and align their latent spaces by matching the parametrized latent distributions and by enforcing a cross-modal reconstruction criterion. Since we learn representations that are oblivious to their origin, a zero-shot visual classifier can be trained using latent space features from semantic data. Our contributions in this work are as follows. FORMULA1 Generalized Zero-shot Learning Let S = {(x, y, c(y))| x ∈ X, y ∈ Y S, c(y) ∈ C} be a set of training examples, consisting of image-features x, e.g. extracted by a CNN, class labels y available during training and class-embeddings c(y). Typical class-embeddings are vectors of continuous attributes or Word2Vec features. In addition, an set U = {(u, c(u))| u ∈ Y u, c(u) ∈ C} is used, where u denote unseen classes from a set Y u, which is disjoint from Y S. Here, C(U) = {c(u 1),..., c(u L)} is the set of class-embeddings of unseen classes. In the legacy challenge of ZSL, the task is to learn a classifier f ZSL: X → Y U. However, in this work, we focus on the more realistic and challenging setup of generalized zero-shot learning (GZSL) where the aim is to learn a classifier DISPLAYFORM0 The Objective Function CADA-VAE is trained with pairs of image features and attribute vectors of seen classes. The data of each pair has to belong to the same class. In this process, an image feature encoder and an attribute encoder learn to transform the training data into the shared latent space. The encoders belong to two VAEs with a common latent space. Once the VAEs are trained, a softmax classifier is trained on both seen image data and unseen attributes, after they are transformed into the latent representation. As the VAE encoding is non-deterministic, many latent features are sampled for each datapoint. Since we only have one attribute vector per class, we oversample latent-space encoded features of unseen classes. To test the classifier, the visual test data is first transformed into the latent space, using only the predicted means µ of the latent representation. The Objective function for training the VAEs is derived as follows. For every modality i (image features, attributes), a VAE is trained. The basic VAE loss for a feature x of modality i ∈ 1, 2,..M is: DISPLAYFORM1 where D KL represents the Kullback-Leibler Divergence, β is a weight, q(z|x (i) ) = N (µ, Σ) is the VAE encoder consisting of a multilayer perceptron, and p(z) is a Gaussian prior. Additionally, each encoded datapoint is decoded into every available modality, e.g. encoded image features are decoded into attributes and vice versa. Consequently, we minimize the L1 cross-reconstruction loss: DISPLAYFORM2 where γ is a weight. The L1 loss empirically proved to provide slighthly better than L2. Furthermore, the 2-Wasserstein W distance between the multivariate Gaussian latent distribution of image features and attributes is minimized: DISPLAYFORM3 The VAE is trained using the final objective L = L basic +L CA +L DA. We refer to the Cross-Aligned and Distribution-Aligned VAE as CADA-VAE. In addition, we test the variant L = L basic + L CA, termed CA-VAE, and the variant L = L basic + L DA, referred to as DA-VAE. A latent size of 64 is used for all experiments, except 128 for ImageNet. We evaluate our framework on zero-shot learning benchmark datasets CUB-200-2011 BID18, SUN attribute BID12 BID7 BID20 for the GZSL setting. All image features used for training the VAEs are extracted from the 2048-dimensional final pooling layer of a ResNet-101. To avoid violating the zero-shot assumption, i.e. test classes need to be disjoint from the classes that ResNet-101 was trained with, we use the proposed training splits in BID20. As class embeddings, attribute vectors were utilized if available. For ImageNet we used Word2Vec embeddings provided by BID3. All hyperparameters were chosen on a validation set provided by BID20. We report the harmonic mean (H) between seen (S) and unseen (U) average per-class accuracy, i.e. the Top-1 accuracy is averaged on a per-class basis., AwA1 and 2 (Generalized Zero-Shot Learning We compare our model with 11 state-of-the-art models. Among those, CVAE, SE , and f-CLSWGAN BID21 learn to generate artificial visual data and thereby treat the zero-shot problem as a data-augmentation problem. On the other hand, the classic ZSL methods DeViSE, SJE BID0, ALE, EZSL BID14 and LATEM BID19 use a linear compatibility function or other similarity metrics to compare embedded visual and semantic features; CMT BID15 and LATEM BID19 utilize multiple neural networks to learn a non-linear embedding; and SYNC BID3 learns by aligning a class embedding space and a weighted bipartite graph. ReViSE BID16 proposes a shared latent manifold learning using an autoencoder between the image features and class attributes. The in TAB3 show that our CADA-VAE outperforms all other methods on all datasets. Moreover, our model achieves significant improvements over feature generating models most notably on CUB. Compared to the classic ZSL methods, our method leads to at least 100% improvement in harmonic mean accuracies. In the legacy challenge of ZSL setting, which is hardly realistic, our CADA-VAE provides competitive performance, i.e. 60.4 on CUB, 61.8 on SUN, 62.3 on AWA1, 64.0 on AWA2. However, in this work, we focus on the more practical and challenging GZSL setting. We believe the obtained increase in performance by our model can be explained as follows. CADA-VAE learns a shared representation in a weakly supervised fashion, through a crossreconstruction objective. Since the latent features have to be decoded into every involved modality, and since every modality encodes complementary information, the model is encouraged to learn an encoding that retains the information contained in all used modalities. In doing so, our method is less biased towards learning the distribution of the seen class image features, which is known as the projection domain shift problem BID5. As we generate a certain number of latent features per class using non-deterministic encoders, our method is also akin to data-generating approaches. However, the learned representations lie in a lower dimensional space, i.e. only 64, and therefore, are less prone to bias towards the training set of image features. In effect, our training is more stable than the adversarial training schemes used for data generation BID21. BID20 several evaluation splits were proposed with increasing granularity and size both in terms of the number of classes and the number of images. Note that since all the images of 1K classes are used to train ResNet-101, measuring seen class accuracies would be biased. However, we can still evaluate the accuracy of unseen class images in the GZSL search space that contains both seen and unseen classes. Hence, at test time the 1K seen classes BID15 49.8 7.2 12.6 21.8 8.1 11.8 87.6 0.9 1.8 90.0 0.5 1.0 SJE BID0 59 BID14 63.8 12.6 21.0 27.9 11.0 15.8 75.6 6.6 12.1 77.8 5.9 11.0 SYNC BID3 70.9 11.5 19.8 43.3 7.9 13.4 87.3 8.9 16.2 90.5 10.0 18.0 DeViSE 53.0 23.8 32.8 27.4 16.9 20.9 68.7 13.4 22.4 74.7 17.1 27.8 f- CLSWGAN Xian et al. (2018b) 57 act as distractors. For ImageNet, as attributes are not available, we use Word2Vec features as class embeddings provided by BID3. We compare our model with f-CLSWGAN BID21, i.e. an image feature generating framework which currently achieves the state of the art on ImageNet. We use the same evaluation protocol on all the splits. Among the splits, 2H and 3H are the classes 2 or 3 hops away from the 1K seen training classes of ImageNet according to the ImageNet hierarchy. M 500, M 1K and M 5K are the 500, 1000 and 5000 most populated classes, while L500, L1K and L5K are the 500, 1000 and 5000 least populated classes that come from the rest of the 21K classes. Finally,'All' denotes the remaining 20K classes of ImageNet. As shown in FIG1, our model significantly improves the state of the art in all the available splits. Note that the test time search space in the'All' split is 22K dimensional. Hence even a small improvement in accuracy on this split is considered to be compelling. The achieved substantial increase in performance by CADA-VAE shows that our 128-dim latent feature space constitutes a robust generalizable representation, surpassing the current state-of-the-art image feature generating framework f-CLSWGAN. In this work, we propose CADA-VAE, a cross-modal embedding framework for generalized zeroshot learning in which the modality-specific latent distributions are aligned by minimizing their Wasserstein distance and by using cross-reconstruction. This procedure leaves us with encoders that can encode features from different modalities into one cross-modal embedding space, in which a linear softmax classifier can be trained. We present different variants of cross-aligned and distribution aligned VAEs and establish new state-of-the-art in generalized zero-shot learning for four medium-scale benchmark datasets as well as the large-scale ImageNet. We further show that a cross-modal embedding model for generalized zero-shot learning achieves better performance than data-generating methods, establishing the new state of the art.

represent the tldr for finding the corresponding scientific article

We use VAEs to learn a shared latent space embedding between image features and attributes and thereby achieve state-of-the-art results in generalized zero-shot learning.

Represent the short summary for finding the corresponding scientific article (long):

We propose a novel autoencoding model with augmented adversarial reconstruction loss. We intoduce new metric for content-based assessment of reconstructions.


Question: Represent the article for finding a single-sentence TLDR:

Question: Intuitively, image classification should profit from using spatial information. Recent work, however, suggests that this might be overrated in standard CNNs. In this paper, we are pushing the envelope and aim to further investigate the reliance on and necessity of spatial information. We propose and analyze three methods, namely Shuffle Conv, GAP+FC and 1x1 Conv, that destroy spatial information during both training and testing phases. We extensively evaluate these methods on several object recognition datasets (CIFAR100, Small-ImageNet, ImageNet) with a wide range of CNN architectures (VGG16, ResNet50, ResNet152, MobileNet, SqueezeNet). Interestingly, we consistently observe that spatial information can be completely deleted from a significant number of layers with no or only small performance drops. Despite the fantastic performances of convolutional neural networks (CNNs) on computer vision tasks, their inner workings remain mostly obfuscated to us and analyzing them often in surprising . Generally, the majority of modern CNNs for image classification learn spatial information across all the convolutional layers: every layer in AlexNet, VGG, Inception, and ResNet applies 3×3 or larger filters. Such design choice is based on the assumption that spatial information remains important at every convolutional layer to consecutively increase the access to a larger spatial context. This is based on the observations that single local features can be ambiguous and should be related to other features in the same scene to make accurate predictions;. Recent work on restricting the receptive field of CNN architectures, scrambling the inputs or using wavelet feature networks ing in networks with shallow depth have all found it to be possible to acquire competitive performances on the respective classification tasks. This raises doubts on whether common CNNs learn representations of global context as small local features appear to be sufficient for classification. We add to the list of surprising findings surrounding the inner workings of CNNs and present a rigorous investigation on the necessity of spatial information in standard CNNs by avoiding learning spatial information at multiple layers. To this end, we propose three methods i.e., shuffle conv, GAP+FC and 1x1Conv, to eliminate the spatial information. Surprisingly, we find that the modified CNNs i.e., without the ability to access any spatial information at last layers, can still achieve competitive on several object recognition datasets. This indicates that the spatial information is overrated for standard CNNs and not necessary to reach competitive performances. In our experiments, the last layers of standard CNNs can be simplified by substituting them with our proposed GAP+FC or 1x1Conv layers which ignore spatial information, leading to a smaller model with less parameters. Moreover, our novel simplifications can be adapted to a wide range of CNN architectures and maintain state-of-the-art performance on various image classification datasets. The detail of the shuffle conv. Each feature map from the input tensor will be randomly and independently shuffled before being fed into an ordinary convolution. Training models for the task of object recognition, our intuitive understanding would be that global image context is beneficial for making accurate predictions. For that reason extensive efforts have been made to enhance the aggregation of spatial information in the decision-making progress of CNNs.; have made attempts to generalize the strict spatial sampling of convolutional kernels to allow for globally spread out sampling and have spurred a range of follow-up work on embedding global context layers with the help of spatial down-sampling. While all of these works have improved on a related classification metric in some way, it is not entirely evident whether the architectural changes alone can be credited, as there is an increasing number of work on questioning the importance of the extent of spatial information for common CNNs. One of the most recent observations by for example indicate that the VGG-16 architecture trained on ImageNet is invariant to scrambled images to a large extent, e.g. they reported only a drop of slightly over 10% points top-5 accuracy for a pre-trained VGG-16. Furthermore, they were also able to construct a modified ResNet architecture with a limited receptive field as small as 33 × 33 and were able to reach competitive on ImageNet, similar to the style of the traditional Bag-of-Visual-Words. The latter was also explicitly incorporated into the training of CNNs in the works by;; , the effect of neglecting global spatial information by design had surprisingly little effect on performance values. On a related note, has indicated with constructing object-texture mismatched images that models trained solely on ImageNet do not learn shape sensitive representations, which would be expected to require global spatial information, but instead are mostly sensitive to local texture features. Our work is motivated to push the envelope further in order to investigate the necessity of spatial information in the process pipeline of CNNs. While the related work has put the attention mainly on altering the input, we are interested in taking measures that remove the spatial information in intermediate layers to shed light on how CNNs process spatial information, thus evaluating its importance and make suggestions for architectural design choices. In order to test how spatial information is processed in the CNN processing pipeline, we propose three approaches: shuffle convolution, GAP+FC and 1x1Conv that neglect spatial information in different ways in intermediate layers and apply these to well established architectures. The evaluation is primarily done with comparing the classification accuracy for models that have been increasingly constrained with respect to how much spatial information can be propagated throughout the network. Section 3.1 elaborates details on our approaches and the experimental setup is discussed in section 3.2. Shuffle Convolution extends the ordinary convolution operation by prepending a random spatial shuffle operation, so that the input to the convolution is permuted. As illustrated in Fig. 1 right: Assume an input tensor of size c×h×w with c being the number of feature maps for a convolutional layer. We first take one feature map from the input tensor and flatten it into a 1-d vector with h × w elements, whose ordering is then permuted randomly. The ing vector is finally reshaped back into h × w and substitute the original feature map. This procedure is independently repeated c times for each feature map so that activations from the same location in the previous layer are misaligned, thereby preventing the information from being encoded by the spatial arrangement of the activations. The shuffled output becomes the input of an ordinary convolutional layer in the end. Even though shuffling itself is not differentiable, gradients can still be propagated through in the same way as Max Pooling. Therefore it can be embedded into the model directly for end-to-end training. As the indices are recomputed within each forward pass, the shuffled output is also independent across training and testing steps. Images within the same batch are shuffled in the same way for the sake of simplicity since we find empirically that it doesn't make a difference whether the images inside the same batch are shuffled in different ways. Instead of shuffling a single layer, we shuffle all layers from the last to the specific depth (last 2 convolutional layers are shuffled in Fig. 1) in order to prevent the model from remembering encountered permutations. Memorization of random patterns is something that deep networks have been shown to be powerful at. Global Average Pooling and Fully Connected Layers: Shuffle convolution is an intuitive way of destroying spatial information but it also makes it impossible to learn correlations across channels for a particular spatial location. Furthermore, shuffling introduces undesirable randomness into the model so that during evaluation multiple forward passes are needed to acquire an estimate of the mean of the output. A simple deterministic alternative achieving a similar goal is what we call GAP+FC. The deployment of Global Average Pooling (GAP) after an intermediate layer, and substitute all the subsequent ones by fully connected layers. Compared to shuffle conv, it is a much more efficient way to avoid learning spatial information at intermediate layers because it shrinks the spatial size of feature maps to one. Fig. 1 demonstrates a toy example of a CNN with the last two convolutional layers modified by GAP+FC. 1x1 Convolution: GAP+FC collapses the spatial information to a size of 1. However, reducing the spatial size potentially influences the expressive ability of the model. For example, the point-wise difference of two consecutive 7 × 7 feature maps lies in the 49 dimension space while the difference of two 1 × 1 feature maps is just a single value, so if the information would be conveyed by the order of the feature maps, larger feature map size tends to be more expressive. In order to retain the information kept in the spatial dimensions but restrict the model to be invariant to the relationships between spatial locations, we propose as an alternative the use of 1x1 convolutions, which replaces the 3x3 convolutions at last layers in a network. It differs from shuffle conv in that the activation at the same spatial location is aligned. Fig. 1 gives a small demonstration where the last 2 layers from a toy CNN are modified. It is worth noting that ResNets use stride-two convolution to downsample the feature maps at the end of bottleneck. Such down-sampling strategy is not ideal for 1x1 convolution because it ignores more than 3/4 of the input. So we use max or average pooling with 2x2 windows as our down-sampling method instead. We test different architectures with shuffle conv, GAP+FC and 1x1Conv on 3 datasets: CIFAR100, Small-ImageNet-32x32 and ImageNet. We measure in each experiment the top-1 accuracy and the number of model parameters. We will take an existing model and apply the modification to layers from the last layer on. The rest of the setup and hyper-parameters remain the same as the baseline model. By shuffle conv or GAP+FC or 1x1Conv, our modification on the baseline model always starts from the last layer and is consecutively extended to the first layer. We denote as K the number of modified convolutional layers or sub-modules counting from the last layer on. The rest of the operations, like skip connections, and modules remain the same. 2 × 2 average pooling with stride 2 is used for down-sampling in all experiments due to the ablation of down-sampling methods in section 4.4. For the VGG-16 architecture, the modification is only performed on the convolutional layers as illustrated in Fig. 1. K varies from 0 (representing the baseline) to 13 since 13 out of the 16 layers are convolutional. For the ResNet-50 architecture with 16 bottleneck sub-modules, one bottleneck is considered as one layer and the modification is only applied onto the 3 × 3 convolutions inside since they are the only operation with spatial extent, the rest of the configuration remains the same as in the baseline model (see Appendix foran example of modified ResNet-50 architecture). For CIFAR100 and Small-ImageNet-32x32 experiments, the first convolution in ResNet is set to 3 × 3 with stride 1 and the first max pooling layer is removed so that the final feature map size is 4 × 4. For each architecture, we first reproduce the original on the benchmark as our baseline, and then the same training scheme is directly used to train our models. All models in the same set of experiments are trained with the same setup from scratch and they are initialized by the same random seed. During testing, we make sure to use a different random seed than during training. We first present an in-depth study of our main observations on CIFAR100 for VGG-16 and ResNet-50 in section 4.1 and then verify them on other datasets and architectures in section 4.3. Finally, the influence of the depth and receptive field size is discussed in section 4.4. In this section, we first investigate the invariance of pre-trained models to the absence of the spatial information at test time, then we impose this invariance at training time with methods in section 3.1. Contradicting to the common sense, recent works suggest a less important role of spatial information in image classification task. Here we take a further step to study the robustness of the model against the absence of the spatial information at test time by applying Shuffle Conv. More specifically, we substitute the last 3 convolutional layers (see Appendix A.4 for more on other layers) of a pre-trained VGG-16 with shuffle conv at test time on CIFAR100 such that the spatial information is neglected in those layers. Because random shuffle is independent at each forward pass, the final test accuracy will be the average of 200 evaluations and the standard deviation is also present. The left table in 2 clean → shuffle shows the model from the clean training scheme gives around 1% test accuracy, which is the same as random guess on CIFAR100, when evaluated with random shuffle. However, if the shuffle conv is infused into the model at training time, then the baseline performance can be achieved no matter whether random shuffle appears at test time as shown in the left table of 2 (73.67% for shuffle → shuffle and 73.57% for shuffle → clean). We thus design the following experiment: we modify the last K convolutional or bottleneck layers of VGG-16 or ResNet-50 on CIFAR100 by Shuffle Conv (both at training and test time), GAP+FC, and 1x1Conv such that the spatial information is removed in different ways. Our modification on the baseline model always starts from the last layer and is consecutively extended to the first layer. The modified networks with different K are then trained on the training set with the same setup and evaluated on the hold-out validation set of CIFAR100. Table 1: Table summarizes the top-1 accuracy and the number of parameters of different K on CIFAR100 for VGG-16 and ResNet-50 with GAP+FC and 1x1Conv. K is defined as the number of modified layers counting from the last layer. The first column for each modification method shows the most compressed model within 1% accuracy difference to the corresponding baseline model and the second column presents the best performed model for each modification method. We can see that 1x1Conv gives even a slightly higher test accuracy while having fewer parameters. The on CIFAR100 for VGG-16 and ResNet-50 are shown in Fig. 2. The x-axis is the number of modified layers K, ranging from 0 to the total number of convolutional or bottleneck layers. K = 0 is the baseline model performance without modifying any layer. As we can see in the right of Fig. 2, with the increasing number of modified layers, the performance of ResNet-50 drops surprisingly slowly for our three methods consistently. For example, Shuffle conv can modify up to the last 5 layers of ResNet-50 while maintaining similar baseline performance i.e., Shuffle conv(K=5) achieves 77.89% accuracy v.s. 78.06% accuracy of the baseline (K=0). 1x1Conv and GAP+FC can preserve the baseline performance until K = 5 and K = 9, where the feature map size is 8 and 16, respectively. For VGG-16, as shown in the left of Fig. 2, a similar trend can be observed. Shuffle conv, GAP+FC, and 1x1Conv are able to tolerate modification of the last 5 layers without losing any accuracy. This is in strong contrast to the common belief that the spatial information is essential for object recognition tasks. One obvious advantage of our methods is that 1x1Conv and GAP+FC can reduce the number of model parameters without affecting the performance. As a side effect, we find that GAP+FC and 1x1Conv have a regularization effect that can lead to improved generalization performance when data augmentation is not applied. Fig. 2 shows the test accuracy of modified ResNet-50 via GAP+FC and 1x1Conv trained with and without data augmentation. While the models trained with data augmentation show similar test accuracy, we observe a significant performance improvement over the baseline on ResNet-50 trained without data augmentation, e.g 1x1Conv outperforms the baseline by 8.01% on CIFAR100 when several last layers are modified. Unfortunately, this effect doesn't hold across other architectures and datasets. Table 2: Left: Top-1 accuracy of VGG-16 with random shuffle enabled at either training and test time for the last 3 layers. Shuffled model is robust to the standard test scheme while the test accuracy of a standard VGG-16 drops to the random guess level if evaluated with shuffling. Right: Effect of data augmentation on classification for ResNet-50 on CIFAR100. The data augmentation here is the random flipping and the random cropping. We present here the best performed model for each method. We can see that modified models reach higher test accuracy when data augmentation is not applied. ResNet-50 with 1x1Conv trained without data augmentation shows a significant performance improvement over the baseline from 65.64% to 73.65% on CIFAR100. Table 3: Image classification on Small-ImageNet for VGG16 and ResNet50 with GAP+FC and 1x1Conv. K is defined as the number of modified layers counting from the last layer. Our experiments in Table 2 left clearly show that ordinary models by default don't possess the invariance to the absence of the spatial information. In contrast to the common wisdom, we find that spatial information can be neglected from a significant number of last layers without any performance drop if the invariance is imposed at training, which suggests that spatial information at last layers is not necessary for a good performance. We should however notice that it doesn't indicate that models whose prediction is based on the spatial information can't generalize well. Besides, unlike the common design manner that layers at different depth inside the network are normally treated equally, e.g. the same module is always used throughout the architecture, our observation implies it is beneficial to have different designs for different layers since there is no necessity to encode spatial information in the last layers (see Appendix A.3 for discussion on first layers), therefore reducing the model complexity. Comparing our three methods, we observe that 1x1Conv is more robust to the absence of the spatial information while Shuffle Conv and GAP+FC perform similarly for both VGG-16 and ResNet-50. This implies that CNNs can still benefit from the larger size of activation maps even though its spatial information is not presented. Since CIFAR100 is a relatively easy dataset with centered objects belonging to only 100 classes, we conduct in the following experiments on more complex inputs: small-ImageNet and ImageNet, whereas small-ImageNet is a down-sampled version of the latter (from 256 × 256 to 32 × 32). The on Small-ImageNet are summarized in the Table 3 (see more details in the Appendix). GAP+FC and 1x1Conv present a similar behavior as on CIFAR100 dataset. And the gap between the performance of GAP+FC and 1x1Conv increases, the maximal number of layers that can be modified on ResNet50 for GAP+FC and 1x1Conv are 3 and 6. This implies that spatial information at last layers of CNNs are not necessary for good performance on the datasets with enough complexity. Furthermore, we conduct experiments for different architectures on full ImageNet with an input image size of 224 × 224. We first reproduce baselines as in the original papers and then apply the same training scheme directly to train our models. Here we only evaluate 1x1Conv due to its superiority over GAP+FC and due to its excessive computational overhead training on the full ImageNet dataset. In Table 4, we observe that spatial information can be ignored at last layers So far, we have evaluated our methods with large models that have been shown to have incredible capacity to learn even from random labels. A hypothesis could be that the models we test are very complex to begin with such that it is of no surprise that they learn the relevant representations in earlier layers and can encode the information necessary to classify in very few dimensions. To approach this question, we deploy our experiments on architectures that have been specifically designed to be of minimal complexity in order to save memory and reduce the number of floating point operations. Hence, we evaluate MobileNetV2 with 3.5M parameters and with 1.25M parameters, both of which are able to reach competitive performance on ImageNet. MobileNetV2 uses the inverted residual bottleneck as their building block where the input tensor is first expanded along the channel dimension and then a 3 × 3 depth-wise convolution is performed before the number of channels is reduced to the output dimension by the final 1 × 1 convolution. In our modification we simply remove the 3 × 3 depthwise convolution together with its ReLU and batch normalization. SqueezeNet is composed of fire modules, which leverage the strategies from to reduce the model parameters. It first squeezes the number of channels by a 1 × 1 convolution and then expands by a mixture of 1 × 1 convolutions and 3 × 3 convolutions. In our modification, we replace all 3 × 3 convolutions in the expand phase by 1 × 1 convolutions. The in Table 4 show that the last two conv layers of both MobileNetV2 and SqueezeNet are also spatial invariant i.e., neglecting the spatial information at those 2 last layers does not affect the performance at all, despite the minimal model complexity. The experiments on Small-ImageNet and ImageNet confirm again the claim in section 4.2 that the spatial information at last layers is not necessary for a good performance and its generalizability across architectures can lead to a further reduction of the number of model parameters even on models that are already very efficient, e.g. MobileNetV2 and SqueezeNet. In the previous section, we observed that 1x1Conv gives the best performance in the sense that spatial information of more layers can be neglected without affect the test accuracy. Here we investigate whether these modified layers are of importance at all or whether they can be stripped of the architecture entirely. The relationship between the receptive field size of a layer and whether it can be modified without performance impact is evaluated subsequently. Importance of the Depth. We saw previously that 1x1Conv gives the best performance in the sense that spatial information at more layers can be neglected without affect the overall test accuracy. Here we ask whether those modified layers can be neglected altogether, effectively reducing the depth. We first pick the most compressed ResNet-50 with the same test accuracy as the baseline on Small-ImageNet, last 6 sub-modules are modified by 1x1Conv. We then strip off one modified layer at a time from the last layer on, ing in 6 different models which are trained with the same configuration. The is shown in Fig. 3 left. With the increase of the number of 1x1 convolutional layers, the test accuracy also increases. So even though the spatial information at last layers is not necessary, those last layers are still essential for good performance. The relation between the receptive field size and the test accuracy difference to the baseline for different image size on VGG-16 over CIFAR100 shows that the test accuracy saturates with the increase of the receptive field size for a given image size. The minimal required receptive field tends to be larger for larger image size and this minimum is normally larger than the actual image size. The exact relation is however unclear. that the spatial information is marginalized out at some particular depth and the ing non-linear transformations are solely used to disentangle the depth wise information. Relationship to the Receptive Field. A reason for that marginalization of spatial information could be hypothesized to be related to the receptive field size of a particular layer. If the receptive field size of a layer is greater or equal to the size of the image, does that tell us whether all following layers can be manipulated? We choose to ablate VGG-16 because the receptive field for a multibranch network is not properly defined as it can only state a theoretical upper bound and do so on CIFAR100 as each object normally occupies the entire image. We replace the 3 × 3 convolutional layers in VGG-16 by 1×1 convolutional layers from the last layer on and until the first layer, thereby varying the receptive field size of the last convolutional layer in our model. Results are shown in Fig. 3 right. Y-axis is the test accuracy difference between the modified model and the baseline model. We can see that the test accuracy saturates with the increase of the receptive field size for a given image size. In order to reach the saturation, it seems that the minimal required receptive field size has to exceeds the actual image size by a relatively large margin and this margin increases for larger image size. For example, the model reaches approximately the same test accuracy as a vanilla VGG-16 with receptive field being 50 for 32 × 32 input image, and the same number becomes around 120 for 64 × 64 input image. This is maybe because the effective receptive field is normally smaller than the theoretical receptive field. However, it is still not really possible to tell a quantitative relation between the required receptive field size and the image size since there are too few data points and it is hard to confirm if an architecture with a specific final receptive field size is sufficient to obtain the baseline performance. To conclude, we empirically show that last layers of CNNs are robust to the absence of the spatial information, which is commonly assumed to be important for object recognition tasks. Our proposed methods, without accessing any spatial information at last layers of modern CNNs, are able to achieve competitive on several object recognition datasets incuding CIFAR100, SmallImageNet and ImageNet. We suggest a good rule of thumb for CNN architectures: using 1x1 convolution or fully connected layers at last layers reduces the number of parameters without affecting the performance. An interesting future direction is to study whether our methods can generalize to other computer vision tasks, e.g., object detection and pose estimation where the spatial relationships are vital for localizing objects. A APPENDIX Convolution with stride 2 was suggested by for 3 × 3 filters as a replacement for pooling layers as the down-sampling method. For example, ResNets use 1 × 1 convolution with stride 2 to reduce the feature map size. However, a direct adaptation leads to a failure for our 1x1Conv. In figure 4, we observe a more rapid decrease of the test accuracy for stride 2 downsampling than average pooling and max pooling on ResNet50 over Small-ImageNet. With the same test accuracy as the baseline, the number of modifiable layers is 3 for convolution with stride 2 and 6 for average pooling. The reason for the failure of the stride 2 case may lie in the fact that 1x1 convolution doesn't have the spatial extent, so a down-sampling will ignore 75% of the activations even they may convey the majority of the information. In an ordinary bottleneck that performs down-sampling, the lost information in the main branch can be replenished from the skip connection where 3 × 3 convolution is deployed to ensure the information at each location is processed. In our modification, however, the skip connection branch will suffer from the loss of the information as well due to 1x1 convolution. Average pooling or max pooling on the other hand doesn't have this problem and their performance according to the plot doesn't have significant difference to each other. We test the necessity of spatial information by GAP+FC and 1x1Conv for VGG-16 and ResNet-50 on Small-ImageNet. Experimental setup is the same as the CIFAR100 experiment. Results are shown in Fig. 5. Within 1% test accuracy difference, GAP+FC manages to replace the last 4 layers in VGG-16 and 1x1Conv can replace the last 7 layers (46.05% and 45.44% compared to the baseline performance 46.59%, respectively). Similarly, the test accuracy can be preserved until K = 3 and K = 6 for GAP+FC and 1x1Conv, which confirms the better performance of 1x1Conv over GAP+FC. This indicates spatial information at last layers is not necessary for a good performance. Previous experiments always apply shuffle conv from one specific layer until the last layer in a network. We test here the impact of random shuffle at different depth by applying shuffle conv at one single layer at a time. The of VGG-16 on CIFAR100 is summarized in Fig. 6 where the x-axis is the layer index (VGG-16 has 13 convolutional layers). We plot the baseline performance with an horizontal line alongside the modified models in order to show a clearer comparison. We can see an overall similar trend as multiple layer shuffle in Fig. 2, the test accuracy drops slowly shuffle conv vanilla Figure 6: The orange curve is the test accuracy of the vanilla VGG-16 i.e. the baseline. The blue curve shows the test accuracy of the VGG-16 with a single convolutional layer modified by shuffle conv at different depth. The x-axis is the layer index with 13 being the last convolutional layer in VGG-16. Random shuffle is applied both at training and test time. The implies random shuffle has a larger impact at first layers than the last layers. with the decrease of the layer index. The baseline performance is maintained for the last 4 layers, which implies random shuffle has a larger impact at first layers than the last layers. In Table. 2 left, we presented the test accuracy of a specific model whose last 3 layers are replaced by shuffle conv under mismatched training and test schemes. We show here the complete of models with different K in Fig. 7. The green curve which is obtained by evaluating the baseline with different K at test time falls to random guess on CIFAR100, compared to the red curve which represents the baseline with clean training and test schemes. And the shuffled models which maintain the baseline accuracy have very similar behavior (the overlapped part of orange curve and blue curve) no matter whether random shuffle appears during evaluation. However, there is a gradually increasing gap between these 2 curve when the shuffled model can't preserve the baseline performance, that consistent schemes gives significant higher accuracy than the inconsistent one. Unfortunately, the reason is not fully understood. A.5 ON IMAGENET Fig. 8 and 9 show the complete of test accuracy of ResNet-152 and ResNet-50 being modified by 1x1Conv on ImageNet. All models are trained with the same scheme as in from scratch. The claim that spatial information is not necessary at last layers generalizes well on ImageNet. We test here another type of random shuffle along the depth of feature maps. It randomly swaps the order of the feature maps along the channel dimension in each forward pass and is denoted as channel shuffle. The experiments are run for VGG-16 on CIFAR100. Fig. 10 shows the change of the test accuracy with the number of layers K that is modified by channel shuffle increasing. Besides an overall decreasing trend, the test accuracy drops much faster than that from random spatial shuffle (74.10% to 71.49% with only the last layer being shuffled), which implies a much weaker robustness of the model against channel shuffle. We therefore assume a more important role of the order of the feature maps in encoding the information at last layers. A.7 RESNET50 ARCHITECTURE Fig. 11 shows an example of ResNet-50 with the last 3 bottlenecks being modified by shuffle conv, GAP+FC and 1x1Conv. Our modification is applied only the 3 × 3 convolution in side each bottleneck since it is the only operation that has the spatial extent.

Spatial information at last layers is not necessary for a good classification accuracy.

Represent the short summary for finding the corresponding scientific article (long).

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.