---
abstract: 'In this paper, we improve Generative Adversarial Networks by incorporating a manifold learning step into the discriminator. We consider locality-constrained linear and subspace-based manifolds[^1] , and locality-constrained non-linear manifolds. In our design, the manifold learning and coding steps are intertwined with layers of the discriminator, with the goal of attracting intermediate feature representations onto manifolds. We adaptively balance the discrepancy between feature representations and their manifold view, which represents a trade-off between denoising on the manifold and refining the manifold. We conclude that locality-constrained non-linear manifolds have the upper hand over linear manifolds due to their non-uniform density and smoothness. We show substantial improvements over different recent state-of-the-art baselines.'
author:
- |
    Yao Ni^^$^{\!, \dagger}$, Piotr Koniusz[^2]$\;^{,\S,\dagger}$, Richard Hartley$^{\dagger,\vardiamond}$, Richard Nock$^{\vardiamond\!,\clubsuit,\dagger}$\
    $^{\dagger}$The Australian National University $^\S$Data61/CSIRO $^{\vardiamond}$Google Research\
    mvaldez@example.com\
- |
    Yao Ni^^$^{\!, \dagger}$, Piotr Koniusz^^$^{, \S,\dagger}$, Richard Hartley$^{\dagger,\vardiamond}$, Richard Nock$^{\vardiamond\!,\clubsuit,\dagger}$\
    $^{\dagger}$The Australian National University $^\S$Data61/CSIRO $^{\vardiamond}$Google Research\
    mvaldez@example.com\
bibliography:
- 'lcsa-gan.bib'
title:
- Manifold Learning Benefits GANs
- 'Manifold Learning Benefits GANs (Supplementary Material)'
---

Introduction {#sec:intro}
============

Generative Adversarial Networks (GANs) @GAN are a family of powerful models for image generation @PGAN [@BigGAN; @StyleGANV2], sound generation @WaveGAN, image stylization @li2016precomputed and destylization @fatima_ijcv, super-resolution @yu2018face, feature generation @xian2018feature [@MetaGAN], . The original GAN learns to generate images @wang2018high [@PGAN; @BigGAN; @StyleGANV2; @StyleGAN] by performing the following min-max game:

$$\min\limits_{{\boldsymbol{\theta}}_{G}} \max\limits_{{\boldsymbol{\theta}}_{D}}
\mathcal{J}(D_{|{\boldsymbol{\theta}}_{D}}, G_{|{\boldsymbol{\theta}}_{G}}),
    \label{eq:gan1}
\vspace{-0.2cm}$$

where $\mathcal{J}(\cdot)\!=\!\mathds{E}_{{\boldsymbol{x}}\sim
p_x({\boldsymbol{x}})}\!\log(D({\boldsymbol{x}};{\boldsymbol{\theta}}_{D}))+\mathds{E}_{{\boldsymbol{z}}\sim
p_z({\boldsymbol{z}})}\!\log(1\!-\!D(G({\boldsymbol{z}})))$. <span>Eq</span> updates parameters ${\boldsymbol{\theta}}_{D}$ of discriminator $D({\boldsymbol{x}};{\boldsymbol{\theta}}_{D})$ to discriminate between samples from the data distribution $p_x({\boldsymbol{x}})$ and generative distributions $p_g(G)$. Simultaneously, parameters ${\boldsymbol{\theta}}_{G}$ of generator $G({\boldsymbol{z}};{\boldsymbol{\theta}}_{G})$ are updated to fool the discriminator $D$. As a result, the noise distribution $p_z({\boldsymbol{z}})$ can be mapped to $p_x({\boldsymbol{x}})$ via generator $G$.

![Our GAN pipeline. We equip the discriminator with residual blocks $B_1,\cdots,B_L$, each containing standard CNN operations, convolutions, ReLU, downsampling, residual link , and the manifold learner $\mathcal{M}_l$. Metaparameter $\beta$ controls the degree of mixing block conv. features with their view recovered from the manifold. The 'overfit?' detector increases $\beta$ when overfitting to ${\boldsymbol{x}}_\text{real}$ is suspected, which boosts the impact of manifold learners.<span data-label="fig:gan_pipe"></span>](images/gan_pipe_new.pdf){width="1\linewidth"}

However, GANs typically suffer from three problems: **1)** the training instability @kodali2017convergence, **2)** the so-called mode collapse @roth2017stabilizing, and **3)** overfitting of the discriminator @Webster_2019_CVPR. The training instability is an imbalanced competition between the generator and the discriminator due to non-overlapping support between the model distribution and the data distribution @kodali2017convergence [@PGAN; @Chu2020Smoothness], leading to poorer quality of generated data. Mode collapse has to do with sharply rising gradient around undesirable local equilibria, resulting in generation of the same image. Finally, discriminator overfitting results in excessive memorization and poor generalization.

Indeed, with an excessive number of parameters, the discriminator may memorize the training data instead of learning a meaningful distribution of generated and real data. Thus, the performance of GANs may result in a high real/fake classification accuracy on the training dataset and a low accuracy on the validation split @BigGAN [@ADA; @DiffAug]. Webster @Webster_2019_CVPR argue such a phenomenon mainly affects the discriminator and is undetectable in the generator, with the exception of hybrid adversarial and non-adversarial methods @pmlr-v80-bojanowski18a which impose a consistency loss on a generator to promote its output to be close to some groundtruth.

We also observed discriminator overfitting on baselines used in our experiments. Doubling the number of parameters of a baseline discriminator resulted in training and validation FID scores of baseline GANs diverging at some intermediate training stage.

Thus, to reduce overfitting of the discriminator, we propose intertwining a data-driven feature manifold-learning step with layers of the discriminator. In this way, the discriminator learns the feature manifold at different levels of object abstraction, from fine to coarse, which limits the complexity of parameter space and separates the signal from noise as both generated and real data are expressed on a common manifold. As a result, the generator diversifies the generated patterns according to their view on the manifold, on which the discriminator operates. The min-max game operates on a gradually learnt manifold (see Fig. \[fig:gan\_pipe\]).

Our contributions are threefold:

1.  We intertwine locality-constrained and subspace-based feature encoding and dictionary learning @me_ATN with blocks of the GAN discriminator to exploit manifold learning in an end-to-end scenario.

2.  We employ a balancing term to help blocks of the discriminator learn the data-driven manifold from the encoder intertwined with them, while permitting some degree of freedom in the vicinity of that manifold (§\[sec:pro\]).$\!\!\!\!\!\!$

3.  We show that locality-constrained soft assignment coding (the best coder in our experiments) acts as a denoiser @denoising_JMLR due to its Lipschitz continuity which we can control to vary its operating mode between the ordinary k-means quantization and locality-constrained linear coding, which promotes quantization of some parts of the feature space while approximately preserving linearity in other parts of the feature space (§\[sec:lcsa\]).

For contribution (i), we investigate Sparse Coding (SC) @lee_sparse [@yang_sparse], Non-negative Sparse Coding (SC~+~) @nnsc, Orthogonal Matching Pursuit (OMP) @pati_omp [@dmallat_omp], Locality-constrained Linear Coding (LLC) @wang_llc, Soft Assignment (SA) @bilmes_gmm [@gemert_kernel], and Locality-constrained Soft Assignment (LCSA) @liu_sadefense [@me_SAO], and Hard Assignment (HA) @steinhaus_ha [@csurka04_bovw]. We defer the discussion on properties of each of these coding methods, and share intuitions about them and necessary equations in §\[sec:pre\].

Problem Formulation {#sec:pro}
===================

Figure \[fig:gan\_pipe\] shows our pipeline (we do not indicate conditional cues for brevity). We build on BigGAN @BigGAN, OmniGAN @zhou2020omni, MSG-StyleGAN @karnewar2020msg, StyleGAN2@StyleGANV2 but we equip the discriminator with the manifold learner which is metacontrolled to reduce overfitting that typically happens in the discriminator rather than the generator @Webster_2019_CVPR. In §\[sup:limited\_data\] of the supplementary material, we also investigate the combination of our method with DA @DiffAug, ADA @ADA and LeCamGAN @lecamgan in limited data scenario.

![Blocks $B_1,\cdots,B_L$ of our discriminator contain a standard block denoted by ${\boldsymbol{f}}$ intertwined with the manifold learner. Metaparameter $\beta$ controls the mixing balance between ${\boldsymbol{f}}$ and ${\boldsymbol{h}}$.<span data-label="fig:res_blcoks"></span>](images/enc_blocks_new.pdf){width="1\linewidth"}

The discriminator $D({\boldsymbol{x}};{\boldsymbol{\theta}}_{D})$ of the GAN in <span>Eq</span> classifies input images as real or fake. Many architectures exist in the literature , GAN @GAN uses the discriminator based on a convolutional network, whereas recent architectures , BigGAN @BigGAN, OmniGAN @zhou2020omni, MSG-StyleGAN@karnewar2020msg and StyleGAN2@StyleGANV2, use residual discriminators with $L$ residual blocks , see BigGAN @BigGAN (their Fig. 16).

Let[^3] ${\boldsymbol{f}}:{\mathbb{R}^{d\times N}}\!\times{\mathbb{R}^{|{\boldsymbol{\theta}}_{B}|}}\rightarrow{\mathbb{R}^{d'\times N'}}$ (where $|{\boldsymbol{\theta}}_B|$ is the size of the set of parameters ${\boldsymbol{\theta}}_B$) be a function realized by a single discriminator block with parameters ${\boldsymbol{\theta}}_{B}$, where $d$ and $d'$ are the number of input/output channels, $N\!=\!WH$ and $N'\!=\!W'H'$ are the number of input/output spatial locations in feature maps of the block. Often, $N'$ may equal $N$.

We introduce an encoding function ${\boldsymbol{h}}\!:\!{\mathbb{R}^{d'\times
N'}}\!\rightarrow\!{\mathbb{R}^{d'\times {N'}}}$ that maps ${\mathbb{R}^{d'\times
N'}}$ into a subset, usually nowhere dense or of small volume in ${\mathbb{R}^{d'\times N'}}$, which we sometimes refer to as the <span>*feature space*</span>. In the cases we consider this mapping derives from a mapping ${\mathbb{R}^{d'}}\rightarrow {\mathbb{R}^{d'}}$ applied independently and equally over the second dimension ${\mathbb{R}^{N'}}$. The encoding introduces an error, measured by $
\|{\boldsymbol{h}}({\boldsymbol{X}}) - {\boldsymbol{X}}\|_F \le \epsilon
$ where $\epsilon$ is called the <span>reconstruction error</span>.

In dictionary-based encoding, the function ${\boldsymbol{h}}$ relies on a dictionary ${\boldsymbol{M}}\!=\![{\boldsymbol{m}}_1,\cdots,{\boldsymbol{m}}_k]\!\in\!{\mathbb{R}^{d'\times k}}$ containing $k$ column vectors, the so-called dictionary atoms (sometimes called anchors), defining the underlying manifold ${{\mathcal{M}}}$, and $k\!\gg\!d'$ ensures the dictionary is overcomplete. Then, after solving the optimization problem,

$$({\boldsymbol{\alpha}},{\boldsymbol{M}})=\operatorname*{arg\,min}_{{\boldsymbol{\alpha}}',{\boldsymbol{M}}'}  \|{\boldsymbol{X}}\!-\!{\boldsymbol{M}}'{\boldsymbol{\alpha}}'\|_F^2+\kappa\,
\Omega({\boldsymbol{\alpha}}', {\boldsymbol{M}}', {\boldsymbol{X}}),\label{eq:enc1}
\vspace{-0.3cm}$$

where ${\boldsymbol{\alpha}}\!\equiv\![{\boldsymbol{\alpha}}_1,\cdots,{\boldsymbol{\alpha}}_{N'}]\!\in\!{\mathbb{R}^{k\times
N'}}\!$, the function ${\boldsymbol{h}}$ is defined by ${\boldsymbol{h}}({\boldsymbol{X}}) = {\boldsymbol{M}}{\boldsymbol{\alpha}}$. Since ${\boldsymbol{\alpha}}$ depends on ${\boldsymbol{X}}$, we shall commonly write it as ${\boldsymbol{\alpha}}({\boldsymbol{X}})$. The mapping ${\boldsymbol{h}}$ maps ${\mathbb{R}^{d'}}$ into a subset ${{\mathcal{M}}}$ of ${\mathbb{R}^{d'}}$, that we will call the feature manifold (or simply manifold).

The choice of $\Omega({\boldsymbol{\alpha}}', {\boldsymbol{M}}', {\boldsymbol{X}})$ realizes some desired constraints via regularization (with $\kappa\!>\!0$) for example, $\Omega({\boldsymbol{\alpha}}',
{\boldsymbol{M}}', {\boldsymbol{X}})\!=\!\|{\boldsymbol{\alpha}}'\|_1$ encourages sparsity of ${\boldsymbol{\alpha}}$, while $\Omega({\boldsymbol{\alpha}}', {\boldsymbol{M}}', {\boldsymbol{X}})\!=\!
\sum_{n}[\|{\boldsymbol{x}}_n-{\boldsymbol{m}}_1\|_2^2,\cdots,\|{\boldsymbol{x}}_n-{\boldsymbol{m}}_k\|_2^2]^T]{\boldsymbol{\alpha}}_{n}|$ encourages locality to express each ${\boldsymbol{\alpha}}_n$ $\operatorname*{Span}({\boldsymbol{m}}_1,\cdots,{\boldsymbol{m}}_{k'})$, where ${\boldsymbol{m}}_1,\cdots,{\boldsymbol{m}}_{k'}$ are the $k'$ nearest neighbours of ${\boldsymbol{x}}_n$.

We intertwine the encoding step with blocks of the discriminator as follows:

$${\boldsymbol{X}}_{l+1}\!=\!
(1\!-\!\beta){\boldsymbol{\tilde{X}}}_{l} +\beta\, {\boldsymbol{h}}_{l}({\boldsymbol{\tilde{X}}}_{l})
\label{eq:enc2}
\vspace{-0.2cm}$$

where ${\boldsymbol{\tilde{X}}}_{l} = {\boldsymbol{f}}({\boldsymbol{X}}_{l};{\boldsymbol{\theta}}_{B_l})$ and ${\boldsymbol{h}}_{l}$ is the encoding function introduced just above, expressed in terms of a dictionary ${\boldsymbol{M}}_{l}$, whereas $\{{\boldsymbol{\theta}}_{B_l}\}_{l=1}^L$ and $\{{\boldsymbol{M}}_l\}_{l=1}^L$ are parameters of blocks of the discriminator and dictionaries for layers $1, \ldots, L$ respectively. Figure \[fig:res\_blcoks\] illustrates <span>Eq</span> applied to blocks $B_1,\cdots,B_L$ of the discriminator.

At the same time, we prevent <span>Eq</span> from becoming a residual link by adding a reconstruction loss to the GAN objective:

$$\begin{aligned}
\begin{split}
\mathcal{J}_\text{prox}\!&=\!\frac{\gamma}{L}
\sum_{l=1}^L{\boldsymbol{\epsilon}}({\boldsymbol{\tilde{X}}}_{l}; {\boldsymbol{M}}_{l}),\quad\text{ where }\\
\;{\boldsymbol{\epsilon}}({\boldsymbol{\tilde{X}}}_{l}; {\boldsymbol{M}}_{l})\!&=
\| {\boldsymbol{\tilde{X}}}_{l} - {\boldsymbol{h}}_{l}({\boldsymbol{\tilde{X}}}_{l}) \|_F^2 ~.
\label{eq:prox}
\end{split}\end{aligned}$$

Metaparameters $(\beta,\gamma)$ control the mixing balance and the proximity between ${\boldsymbol{f}}$ and ${\boldsymbol{h}}$.

**Meta-adaptation of $\boldsymbol{(\beta,\gamma)}$.** Discriminator overfitting can be detected with a hypothesis test on the expectation over decisions $r({\boldsymbol{x}}_\text{real})\!=\!\mathds{E}[\operatorname*{Sign}(D({\boldsymbol{x}}_
\text{real}))]$ samples ${\boldsymbol{x}}_\text{real}$, defined as $\operatorname*{Sign}(r({\boldsymbol{x}}_\text{real})\!>\!\eta)\!\in\!\{-1,0,1\}$, where $\eta\!=\!0.5$ is the threshold whose violation indicates potential overfitting, as the discriminator becomes increasingly good at distinguishing real datapoints @ADA. Thus, to update $(\beta,\gamma)$, we apply: <span> $$\begin{aligned}
& \beta_{t+1}\!=\!\beta_t\!+\!\Delta_\beta\!\cdot\!\operatorname*{Sign}\left(r({\boldsymbol{x}}_\text{real})
\!>\!\eta\right),\label{eq:beta}\\
& \gamma_{t+1}\!=\!\gamma_0\!+\!\Delta_\gamma\!\cdot\!\beta_{t+1},
\label{eq:gamma}\end{aligned}$$ where $\beta_0\!=\!0.1$. $\Delta_\beta\!=\!0.001$ ensures a gradual change of $\beta$ by increasing contributions from ${\boldsymbol{h}}({\boldsymbol{\tilde{X}}}_{l})$ in <span>Eq</span> when overfitting is detected, and increasing contributions from ${\boldsymbol{\tilde{X}}}_{l}$ in <span>Eq</span> when overfitting vanishes. By setting $\gamma_0\!=\!0.1$, we ensure that the proximity loss in <span>Eq</span> is always enabled, and $1\!\leq\!\Delta_\gamma\!\leq\!3$ controls the strength of proximity.</span>

![Understanding our manifold setting. Atoms ${\boldsymbol{m}}_1,\cdots,{\boldsymbol{m}}_k$ (crosses $\times$) define the geometry of the manifold. Samples ${\boldsymbol{x}}_1,\cdots,{\boldsymbol{x}}_{N'}$ (blue <span>$\circ$</span>) are projected onto the manifold ${{\mathcal{M}}}$ via function $\boldsymbol{\alpha}({\boldsymbol{x}})$ and then recovered via ${\boldsymbol{h}}({\boldsymbol{x}})$, which produces recovered samples ${\boldsymbol{\tilde{x}}}_1,\cdots,{\boldsymbol{\tilde{x}}}_{N'}$ (black $\circ$). The grey circles indicate $L_2$ balls imposed by the proximity operator in <span>Eq</span> while green triangles <span>$\triangle$</span> are trade-off samples between ${\boldsymbol{f}}$ and ${\boldsymbol{h}}$, controlled by $\beta$ within $L_2$ balls. We note that radii of $L_2$ balls are controlled by $\gamma$ but each ball can be larger or smaller depending whether this benefits the discrimination loss. Thus, where desired, trade-off samples are refined by ${\boldsymbol{f}}$ ${\boldsymbol{h}}$ within $L_2$ balls. Curly lines (sigmoid-like <span>$\backsim$</span>) indicate we use locally the sigmoid non-linearity.<span data-label="fig:res_man1"></span>](images/man1.pdf){width="0.95\linewidth"}

**Discussion.** Figure \[fig:res\_man1\] shows our manifold setting, which exploits the interplay between <span>Eq</span> and <span>Eq</span>. We alternately learn encoding of samples ${\boldsymbol{X}}_l$ on manifold ${{\mathcal{M}}}_l$ and refine dictionary ${\boldsymbol{M}}_l$. The proximity operator in <span>Eq</span> encourages samples ${\boldsymbol{\tilde{X}}}_l$ to stay in the proximity of their recovered view ${\boldsymbol{h}}({\boldsymbol{\tilde{X}}}_l)$ by controlling $L_2$ balls around ${\boldsymbol{h}}({\boldsymbol{\tilde{X}}}_l)$. <span>Eq</span> interpolates between ${\boldsymbol{h}}({\boldsymbol{\tilde{X}}}_l)$ and ${\boldsymbol{\tilde{X}}}_l$ within $L_2$ balls. We opt for such a design as (i) ${\boldsymbol{f}}$ may refine ${\boldsymbol{h}}$ if the discriminator loss spotting real/fake inputs deems it useful, (ii) ${\boldsymbol{h}}$ is preferred when overfitting is detected, whereas ${\boldsymbol{f}}$ introduces refined patterns otherwise, (iii) ${\boldsymbol{f}}$ is encouraged to learn from the piecewise-smooth ${\boldsymbol{h}}$ (see §\[sec:lcsa\]).

Related Works
=============

Below, we discuss state-of-the-art GAN pipelines.

**Modern GANs.** New variants of GANs @GAN use a modified generator or discriminator. After the original GAN @GAN (and DCGAN @radford2015unsupervised), a residual architecture was introduced into GANs @gulrajani2017improved, and improved by adding a self-attention block @SAGAN. Progressive GAN @PGAN uses several levels of layers for increasingly finer image resolution. StyleGAN @StyleGAN maps the input to an intermediate latent space and controls the generator through adaptive instance normalization (so-called AdaIN). MSG-GAN passes multi-scale gradients from the discriminator to the generator @karnewar2020msg.

**Improving GANs.** To address training instability, mode collapse and overfitting, researchers study (i) loss and distance formulations, (ii) regularization mechanisms and penalties, and (iii) architectural modifications.

Arjovsky propose Wasserstein GAN @arjovsky2017wasserstein which uses the Kantorovich-Rubinstein duality to realize the Wasserstein distance that improves the stability. Gulrajani @gulrajani2017improved improve the Wasserstein GAN by penalizing the norm of gradient of the critic. Mean & Covariance GAN @mroueh2017mcgan performs matching between the generated and real data distributions with first- and second-order statistics. A Maximum Mean Discrepancy GAN @NIPS2017_dfd7468a matches distributions in the Reproducing Kernel Hilbert Space (RKHS).

Spectral Normalization GAN @SNGAN applies spectral normalization on weights to stabilize training of the discriminator, whereas Spectral Regularization GAN @liu2019spectral performs detection of so-called spectral collapse. The Disconnected Manifold GAN @khayatkhoei2018disconnected assumes that natural images lie on a union of disjoint manifolds, realized with several generators. The Feature Quantization GAN (FQGAN) @FQGAN quantizes features of discriminator into a k-means style dictionary which improves the discriminator. The Denoising Feature Matching GAN @warde2016improving encourages proximity between the output of the generator and a denoising auto-encoder. Our work differs from the Disconnected Manifold GAN which models an entire image distribution as a union of non-explicit manifolds (a collection of generators). FQGAN imposes quantization on features of the discriminator and Denoising Feature Matching GAN learns a denoising auto-encoder on real images to apply it on generated images.

In contrast, we model coarse-to-fine features extracted from multiple blocks of the discriminator, which capture different semantic levels of abstraction. We encourage these features to lie on explicit locality-constrained non-linear manifolds (each block of our discriminator has its own learner). We adaptively control the mixing levels of conv. features and their views recovered from manifolds, and the smoothness of manifolds to prevent overfitting.

**Limiting Overfitting.** Overfiting in GANs can be limited with augmentations (rotations, clipping) @CRGAN [@DiffAug; @jeong2021training; @tran2021data]. However, augmentation artifacts leak into the generated images @ICRGAN [@ADA]. Injecting noises to the discriminator @CAGAN [@CTGAN] has been realized by dropout layers @srivastava2014dropout (implicit network ensemble), whereas we equip the discriminator with a data-manifold learner whose smoothness we can control.

Preliminaries {#sec:pre}
=============

Below, we explain GAN pipelines on which we build, and feature encoding and dictionary learning, our key tool.

Baseline GANs {#sec:base_gans}
-------------

**BigGAN** @BigGAN combines a projection-based loss @miyato2018cgans, spectral normalization @SNGAN, and self-attention @SAGAN. The projection-based score is a trade-off between a class-wise cosine similarity and a class independent term:

$$s({\boldsymbol{x}}, {\boldsymbol{y}}) = {\boldsymbol{y}}^T {\boldsymbol{V}}D({\boldsymbol{x}};{\boldsymbol{\theta}}_D) + {\boldsymbol{f}}'(D({\boldsymbol{x}};{\boldsymbol{\theta}}_D);
{\boldsymbol{\theta}}_{D'}),
\vspace{-0.2cm}$$

where ${\boldsymbol{y}}\!\in\!\{0,1\}^C$ and $\|{\boldsymbol{y}}\|_1\!=\!1$, ${\boldsymbol{V}}\!\in\!{\mathbb{R}^{C\times d'}}$ is a bilinear compatibility matrix that associates output features ${\boldsymbol{X}}_\text{out}$ from $D$ with the class label. As ${\boldsymbol{y}}$ is a one-hot vector, ${\boldsymbol{V}}\!\equiv\![{\boldsymbol{v}}_1,\cdots,{\boldsymbol{v}}_C]^T$ contains linear projectors ${\boldsymbol{v}}_c$, one per class $c\!\in\{0,\cdots,C\!-\!1\}$. Function ${\boldsymbol{f}}':{\mathbb{R}^{d'}}\!\rightarrow\!1$ is realized by an FC layer with parameters ${\boldsymbol{\theta}}_{D'}$. Scores $s({\boldsymbol{x}}, {\boldsymbol{y}})$ are passed to a hinge-based loss with two components: <span> $$\begin{aligned}
&\mathcal{J}_\text{discr}\!=\!\mathds{E}_{({\boldsymbol{x}},{\boldsymbol{y}})\sim p_{x\times
y}^\text{real}({\boldsymbol{x}},{\boldsymbol{y}})}\max(0, 1\!-\!s({\boldsymbol{x}}, {\boldsymbol{y}}))+\nonumber\\
&\quad\mathds{E}_{({\boldsymbol{z}},{\boldsymbol{y}}_z)\sim p_{z\times y_z}({\boldsymbol{z}},{\boldsymbol{y}}_z)}\max(0,
1\!+\!s(G({\boldsymbol{z}}, {\boldsymbol{y}}_z), {\boldsymbol{y}}_z)). $$ Our manifold-based pipeline combines loss $\mathcal{J}_\text{prox}$ from <span>Eq</span> with $\mathcal{J}_\text{discr}$ and the original $\mathcal{J}_\text{gen}$ from @BigGAN.</span>

**OmniGAN** @zhou2020omni uses a multi-label softmax loss (where a label vector ${\boldsymbol{y}}\!\in\!\{0,1\}^{C+2}$, $\|{\boldsymbol{y}}\|_1\!=\!2$ is a concatenation of the one-hot class label vector and one-hot real/fake vector): <span> $$\begin{aligned}
&\mathcal{J}_\text{discr}\!=\!\mathds{E}_{({\boldsymbol{x}},{\boldsymbol{y}})\sim p_{x\times
y}^\text{real}({\boldsymbol{x}},{\boldsymbol{y}})}s({\boldsymbol{x}},{\boldsymbol{y}})+\nonumber\\
&\qquad\quad\mathds{E}_{({\boldsymbol{z}},{\boldsymbol{y}}_z)\sim p_{z\times y_z}({\boldsymbol{z}},{\boldsymbol{y}}_z)}s(G({\boldsymbol{z}},
{\boldsymbol{y}}_z),{\boldsymbol{y}}_z)\quad\text{where}\\
&\qquad\quad s({\boldsymbol{x}},{\boldsymbol{y}})\!=\!\sum\limits_{c=0}^{C+1}\log\left(1\!+\!
e^{-\operatorname*{Sign}(y_c\!-\!0.5)\phi_c({\boldsymbol{x}})}\right). $$ We note that ${\boldsymbol{X}}_\text{out}$ represents output features from $D$, ${\boldsymbol{\phi}}({\boldsymbol{x}})\!=\!{\boldsymbol{f}}'(D({\boldsymbol{x}}; {\boldsymbol{\theta}}_{D}); {\boldsymbol{\theta}}_{D'})$, where function ${\boldsymbol{f}}':{\mathbb{R}^{d'}}\!\rightarrow\!C\!+\!2$ is realized by an FC layer with parameters ${\boldsymbol{\theta}}_{D'}$.</span>

Feature Coding and Dictionary Learning on Data-driven Manifolds {#sec:code}
---------------------------------------------------------------

Below, we formalize feature encoding and dictionary learning approaches listed in §\[sec:intro\]. In experiments, we substitute a chosen coding step into function ${\boldsymbol{h}}({\boldsymbol{x}})$ from §\[sec:pro\]. Moreover, ${\boldsymbol{M}}\!\equiv\![{\boldsymbol{m}}_1,\cdots,{\boldsymbol{m}}_k]\!\in\!
{\mathbb{R}^{d'\times k}}$ is a dictionary whose learning step is detailed at the bottom of §\[sec:code\]. We write ${\boldsymbol{\alpha}}({\boldsymbol{x}})$ to represent the encoding mapping on the simplex.

**Hard Assignment (HA).** This encoder assigns each ${\boldsymbol{x}}$ to its nearest ${\boldsymbol{m}}$ by solving the following optimisation problem: $$\begin{array}{l}
{\boldsymbol{\alpha}}({\boldsymbol{x}})=\operatorname*{arg\,min}\limits_{{\boldsymbol{\alpha}}'\in\left\{0,1\right\}^k}\;\rVert{
{\boldsymbol{x}}-{\boldsymbol{M}}{\boldsymbol{\alpha}}'}\lVert_2^2,\\
\qquad\quad\,\text{s.t.}\;\;\|{\boldsymbol{\alpha}}'\|_{1}=1. \end{array}\label{eq:ha}
$$ If ${\boldsymbol{M}}$ is formed by k-means clustering, HA becomes an equivalent of the quantizer from FQGAN @FQGAN.

**Sparse Coding (SC) & Non-negative Sparse Coding (SC~+~).** SC encodes ${\boldsymbol{x}}$ as a sparse linear combination of atoms ${\boldsymbol{M}}$ by optimising the following objective:

$$\label{eq:sp1}
{\boldsymbol{\alpha}}({\boldsymbol{x}})=\operatorname*{arg\,min}\limits_{{\boldsymbol{\alpha}}'}\;\rVert{ {\boldsymbol{x}}-{\boldsymbol{M}}{\boldsymbol{\alpha}}'}\lVert_2^2
\;+\;\kappa \rVert{{\boldsymbol{\alpha}}'}\rVert_1,
\vspace{-0.3cm}$$

whereas SC~+~ additionally imposes a constraint that ${\boldsymbol{\alpha}}'\!\geq\!0$. As already noted in §\[sec:pro\], both SC and SC~+~ encode ${\boldsymbol{x}}$ on a subspace of ${\boldsymbol{M}}$ of size controlled by the sparsity term. Thus, both coders learn subspace-based manifolds.

**Orthogonal Matching Pursuit (OMP).** This encoder expresses ${\boldsymbol{x}}$ as a sparse linear combination of atoms ${\boldsymbol{M}}$ by optimising the following objective:

$$\label{eq:omp}
\begin{array}{l}
{\boldsymbol{\alpha}}({\boldsymbol{x}})=\operatorname*{arg\,min}\limits_{{\boldsymbol{\alpha}}'}\;\rVert{
{\boldsymbol{x}}-{\boldsymbol{M}}{\boldsymbol{\alpha}}'}\lVert_2^2,\\\qquad\quad\,\text{s.t.}\;\;\rVert{{\boldsymbol{\alpha}}'}\rVert_0\!\leq\!\tau, 
\end{array}
\vspace{-0.3cm}$$

where the pseudo-norm $\rVert{{\boldsymbol{\alpha}}'}\rVert_0$ ensures the count of non-zero coefficients of ${\boldsymbol{\alpha}}'$ is at most $\tau$. Unlike SC and SC~+~, $\rVert{{\boldsymbol{\alpha}}'}\rVert_0$, the penalty enforces a strict limit on the number of non-zero elements in ${\boldsymbol{\alpha}}'$, but the problem itself is NP-hard.

**Approximate Locality-constrained Linear Coding (LLC).** LLC expresses ${\boldsymbol{x}}$ as a linear combination of $k'$ nearest neighbour atoms of ${\boldsymbol{x}}$ selected from ${\boldsymbol{M}}$, forming subspaces of size $k'$ on a piecewise-linear manifold:

$$\begin{array}{l}
{} {\boldsymbol{\alpha}}({\boldsymbol{x}})=\operatorname*{arg\,min}\limits_{{\boldsymbol{\alpha}}'}\;\rVert{ {\boldsymbol{x}}-{\boldsymbol{M}}{\boldsymbol{\alpha}}'}\lVert_2^2,\\
\qquad\quad\,\text{s.t.}\;\;\mathbf{1}^T{\boldsymbol{\alpha}}'=1,
\end{array}\label{eq:llc}
\vspace{-0.2cm}$$

and ${\boldsymbol{\alpha}}'$ is further constrained by $\alpha'_i = 0$ unless ${\boldsymbol{m}}_i$ is one of the $k'$ closest neighbours of ${\boldsymbol{x}}$.

**Soft Assignment (SA) & Locality-constrained Soft Assignment (LCSA).** SA expresses ${\boldsymbol{x}}$ as the membership probability of ${\boldsymbol{x}}$ belonging to each ${\boldsymbol{m}}_i$ in ${\boldsymbol{M}}$, a concept known from Gaussian Mixture Models (GMM). SA is derived assuming equal mixing probability and equal variance $\sigma$ of GMM components. SA has the closed form:

$$\begin{aligned}
\label{eq:gmm4}
{\boldsymbol{\alpha}}({\boldsymbol{x}};{\boldsymbol{M}},\sigma) &= S_\sigma(
 \|{\boldsymbol{x}}\!-\!{\boldsymbol{m}}_1\|_2,\cdots, \|{\boldsymbol{x}}\!-\!{\boldsymbol{m}}_k\|_2),
 \vspace{0.3cm}\end{aligned}$$

where $S_\sigma$ is the softmax function $S_\sigma: {\mathbb{R}^{k}} \rightarrow \Delta^{k-1}$, where $\Delta^{k-1}$ is the probability simplex and:

$$\label{eq:softmax}
S_\sigma(d_1, \ldots, d_k)_j = \frac{\exp(-d_j^2/2\sigma^2)}{\sum_i \exp(-d_i^2/2\sigma^2)} ~.
\vspace{0.2cm}$$

This model yields largest values of $\alpha'_i$ for atoms ${\boldsymbol{m}}_i$ in ${\boldsymbol{M}}$ that are close Euclidean neighbours of ${\boldsymbol{x}}$. However, $\alpha_i({\boldsymbol{x}})>0$ even for ${\boldsymbol{m}}_i$ that are far from ${\boldsymbol{x}}$. For this reason, SA is not strictly locality-constrained.

LCSA differs from SA by setting $\alpha_i({\boldsymbol{x}}) = 0$ unless ${\boldsymbol{m}}_i$ is among the $k'$ nearest-neighbour atoms for ${\boldsymbol{x}}$. The normalization in <span>Eq</span> is done such that in the denominator in <span>Eq</span>, the summation is only over the $k'$ nearest neighbours. Thus, LCSA maps ${\mathbb{R}^{d}}$ onto a set of probability simplices $\Delta^{k-1}$

As LCSA performed best in our experiments, we analyze it in §\[sec:lcsa\].

**Dictionary Learning (DL).** For all methods listed above, we employ a simple dictionary learning objective which follows <span>Eq</span>. That is, we assume some evaluated or fixed ${\boldsymbol{\alpha}}({\boldsymbol{X}})\!\equiv\![{\boldsymbol{\alpha}}_1,
\cdots,{\boldsymbol{\alpha}}_{N'}]$, and compute:

$$\label{eq:dl}
\begin{array}{l}
{\boldsymbol{M}}=\operatorname*{arg\,min}\limits_{{\boldsymbol{M}}'}\;\rVert{ {\boldsymbol{X}}-{\boldsymbol{M}}'{\boldsymbol{\alpha}}}\lVert_F^2,\\\end{array}
\vspace{-0.2cm}$$

where ${\boldsymbol{M}}'$ can be constrained to contain atoms $\rVert{{\boldsymbol{m}}'_i}\rVert_2\!\leq\!1$ if codes ${\boldsymbol{\alpha}}$ have non-restricted $L_2$ norm , for OMP.

**Inverse of ${\boldsymbol{\alpha}}$.** To reproject ${\boldsymbol{\alpha}}({\boldsymbol{X}})$ from the manifold ${{\mathcal{M}}}$ into the Euclidean space, we simply compute ${\boldsymbol{\tilde{X}}}\!=\!{\boldsymbol{M}}{\boldsymbol{\alpha}}({\boldsymbol{X}})$.

**Implementation Remarks.** Coding methods, dictionary learning, and their implementations are detailed in §\[sup:code\] of the supplementary material. For the dictionary learning step, we detach ${\boldsymbol{X}}$ and ${\boldsymbol{\alpha}}$, and run 1 iteration of gradient descent per mini-batch each ${\boldsymbol{M}}$ (no big gain for $\geq\!2$ iterations). For SC and SC~+~, we detach ${\boldsymbol{X}}$ and all ${\boldsymbol{M}}$, and let 5 iterations of gradient descent (no gain for $\geq\!6$ iterations). LLC has a closed-form solver @wang_llc. Our efficient OMP solves the system of linear equations (no matrix inversion). SA/LCSA enjoy a fast closed-form recipe. LLC and LCSA use the partial sort algorithm for selecting $k'$ nearest neighbours. We detach ${\boldsymbol{M}}{\boldsymbol{\alpha}}$ to compute the proximity loss in <span>Eq</span>.

Theoretical Analysis of LCSA {#sec:lcsa}
============================

As LCSA is consistently the best encoder in our experiments, we focus its theoretical properties below. All proofs for theories listed below are in §\[sup:proofs\] of the supplementary material.

In this section, we use the following notation. Suppose a dictionary of atoms ${\boldsymbol{M}}\!=\![{\boldsymbol{m}}_i]$ in ${\mathbb{R}^{d}}$ is given, and ${{2^{{\boldsymbol{M}}}_{k'}}}$ denotes the set of all subsets of size $k'$ of ${\boldsymbol{M}}$. We define by ${{\text{\rm NN}_{k'}}}: {\mathbb{R}^{d}} \rightarrow {{2^{{\boldsymbol{M}}}_{k'}}}$ to be the set-valued function that takes a point ${\boldsymbol{x}}$ to its set of $k'$ nearest neighbour atoms. A maximal subset of ${\mathbb{R}^{d}}$ on which ${{\text{\rm NN}_{k'}}}({\boldsymbol{x}})$ is constant is called a Voronoi cell. For a given set ${{U}}$ in ${{2^{{\boldsymbol{M}}}_{k'}}}$, then, a Voronoi cell $({{\text{\rm NN}_{k'}}})^{-1}({{U}})$ is a subset of ${\mathbb{R}^{d}}$ consisting of the all the points for which ${{U}}$ is the set of $k'$ nearest neighbours. The collection of all Voronoi cells constitutes a decomposition of ${\mathbb{R}^{d}}$ into disjoint polyhedral regions.

In the case where the set of $k'$ nearest elements of ${\boldsymbol{M}}$ is not unique, we leave the set ${{\text{\rm NN}_{k'}}}(x)$ undefined. Thus, the Voronoi cells are disjoint open polyhedral regions such that ${{\text{\rm NN}_{k'}}}({\boldsymbol{x}})$ is constant on each cell. The complement in ${\mathbb{R}^{d}}$ of the set of Voronoi cells is a subset of a finite set of hyperplanes in ${\mathbb{R}^{d}}$.

#### SA and LCSA encoding. 

Given a dictionary ${\boldsymbol{M}}\!=\! [{\boldsymbol{m}}_i]$ in ${\mathbb{R}^{d}}$ consisting of $k'$ elements, with $k'\!\le\!d\!+\!1$, we consider the function ${\boldsymbol{M}}{\boldsymbol{\alpha}}({\boldsymbol{x}})$ where ${\boldsymbol{\alpha}}({\boldsymbol{x}})\!=\!S_\sigma({\boldsymbol{x}})$ is the softmax mapping <span>Eq</span>. (The dictionary may be the dictionary of $k'$ nearest neighbors of some point ${\boldsymbol{x}}$.)

If $\sigma\!>\!0$, the mapping ${\boldsymbol{x}}\mapsto {\boldsymbol{M}}{\boldsymbol{\alpha}}({\boldsymbol{x}})$ is a smooth fibration from ${\mathbb{R}^{d}}$ <span>**onto**</span> the interior of the simplex $\Delta$ with vertices ${\boldsymbol{m}}_i$. The fibre of this mapping is equal to the linear subspace of ${\mathbb{R}^{d}}$ normal to the affine space spanned by the ${\boldsymbol{m}}_i$.

A fibre is the set of points that map to the same point in $\Delta^{k'-1}$ under this mapping.

\[prop:p1\] The following properties of LCSA hold true:

1.  If $\sigma\!\rightarrow\!0$ or $k'\!=\!1$, the ${\boldsymbol{\alpha}}$ codes converge to the HA solution (quantization).

2.  ${\boldsymbol{\alpha}}({\boldsymbol{x}})$ is an approximately linear coding of ${\boldsymbol{x}}$ in the proximity of ${\boldsymbol{\mu}}({\boldsymbol{x}})\!=\!
    \frac{1}{k'} \sum_{{\boldsymbol{m}}'\in{{\text{\rm NN}_{k'}}}({\boldsymbol{x}})} {\boldsymbol{m}}' ~.
    $

3.  For ${\boldsymbol{x}}$ with nearest neighbour atoms $\{{\boldsymbol{m}}_i\}={{\text{\rm NN}_{k'}}}({\boldsymbol{x}})$ with mean ${\boldsymbol{\mu}}({\boldsymbol{x}})$, and nearest neighbor atom $\mathbf{n}({\boldsymbol{x}})\!=\!\text{\rm NN}_1({\boldsymbol{x}})$, the reconstruction error satisfies

    $$\|{\boldsymbol{M}}{\boldsymbol{\alpha}}({\boldsymbol{x}})\!-\!{\boldsymbol{x}}\|_2 \le\\\max\Big(\|{\boldsymbol{x}}\!-\!\mathbf{n}({\boldsymbol{x}})\|_2, \|{\boldsymbol{x}}\!-\!{\boldsymbol{\mu}}({\boldsymbol{x}})\|_2\Big).
    \vspace{-0.3cm}$$

4.  The reconstruction error varies smoothly on each Voronoi cell. For ${\boldsymbol{x}}$ and ${\boldsymbol{x}}'$ in the same Voronoi cell and $\Delta$ the simplex with vertices ${{\text{\rm NN}_{k'}}}({\boldsymbol{x}})$, we have

    $\;$Local Lipschitz continuity: if $\sigma> 0$, then $$\|{\boldsymbol{M}}{\boldsymbol{\alpha}}({\boldsymbol{x}})\!-\!{\boldsymbol{M}}{\boldsymbol{\alpha}}({\boldsymbol{x}}')\|\!\leq\!K\|{\boldsymbol{x}}\!-\!{\boldsymbol{x}}'\|$$ where $K\!=\!{D^2/\sigma^2}$ and $D$ is diameter of simplex $\Delta$ (the maximum distance between its vertices). The Lipschitz condition holds for $\|\cdot\|_1$ and $\|\cdot\|_2$ norms.

    $\;$The biggest change of the reconstruction error on a Voronoi cell for HA ($\sigma\!=\!0)$ is less than or equal to $D$.

5.  The LCSA encoding ${\boldsymbol{M}}{\boldsymbol{\alpha}}({\boldsymbol{x}})$ is non-continuous at the boundaries of the Voronoi regions.

\[prop:den\] Our design fulfils the principles of GAN with Denoising Auto-Encoder (DAE) @denoising_JMLR with loss:

$$\mathcal{L}_\text{dae}=\frac{1}{N}\sum\limits_{n=1}^N\Big(\,\Big\lVert{\boldsymbol{r}}(
{\boldsymbol{x}}_n)-{\boldsymbol{x}}_n\Big\rVert_2^2 + \sigma'^2\Big\lVert
\frac{\partial{\boldsymbol{r}}({\boldsymbol{x}})}{\partial{\boldsymbol{x}}}\Big|_{{\boldsymbol{x}}={\boldsymbol{x}}_n}\Big\rVert_2^2\,\Big),
\label{eq:dae}
\vspace{-0.1cm}$$

where ${\boldsymbol{r}}({\boldsymbol{x}}_n)$ is the reconstruction of ${\boldsymbol{x}}_n$ akin to our ${\boldsymbol{h}}({\boldsymbol{x}}_n)$ in <span>Eq</span>. More importantly, $\sigma'^2$ specifies the noise variance. Specifically, we note the following:

1.  Not surprisingly, the proximity loss in <span>Eq</span> fulfills somewhat similar role to $\lVert{\boldsymbol{r}}({\boldsymbol{x}}_n)-{\boldsymbol{x}}_n\rVert_2^2$ in <span>Eq</span>.

2.  LCSA implicitly fulfils the denoising role (apart from locality-constrained non-linear coding) of $\sigma'^2\Big\lVert
    \frac{\partial{\boldsymbol{r}}({\boldsymbol{x}})}{\partial{\boldsymbol{x}}}\Big\rVert_F^2$, with $\sigma$ of LCSA and $\sigma'$ of DAE related , $\sigma'^2$ is proportional to our $\sigma^2$. To this end, we notice that DAE penalises the Frobenius norm of the Jacobian matrix $\Big\lVert
    \frac{\partial{\boldsymbol{r}}({\boldsymbol{x}})}{\partial{\boldsymbol{x}}}\Big\rVert_F$ by increasing $\sigma'^2$. We penalise the spectral norm of Jacobian matrix $\Big\lVert\frac{\partial{\boldsymbol{M}}\boldsymbol{\alpha}({\boldsymbol{x}})}{\partial{\boldsymbol{x}}}\Big\rVert_2\!=\!K$ via de facto controlling the Lipschitz constant $K\!=\!{D^2}/{\sigma^2}$.

Experiments
===========

We evaluate our method on CIFAR-10 & CIFAR-100 @CIFAR & ImageNet @deng2009imagenet (conditional GAN) and Oxford-102 Flowers @nilsback_flower102 and FFHQ @StyleGAN (unconditional setting). We show our LCSA harmonizes with BigGAN @BigGAN, OmniGAN @zhou2020omni, MSG-StyleGAN @karnewar2020msg and StyleGAN2 @StyleGANV2.

**Datasets.** CIFAR-10 has $50K$ and $10K$ training and testing images ($32\!\times\!32$) from 10 classes, whereas CIFAR-100 has 100 categories. ImageNet has $1.2M$ and $50K$ training and validation images with $1K$ classes. We center-crop and downscale its images to $64\!\times\!64$ and $128\!\times\!128$ pixels. Oxford-102 Flowers contains $8K$ images of 102 fine-grained flower species. We center-crop its images and resize to $256\!\times\!256$. FFHQ dataset provides $70K$ human face images at multiple resolutions (we opt for $256\!\times\!256$). Following @ADA, we augmented the $70K$ dataset to $140K$ with $x$-flips.

**Evaluation Metrics.** We generate $50K$ images per dataset to compute the commonly used Inception Score @Improved_GAN and Fréchet Inception Distance (FID) @FID. Mean/standard dev. are computed over 5 runs, where both reported. We report tFID, computed between $50K$ generated images and all training images. For CIFAR-10/CIFAR-100/ImageNet, we also compute vFID between $10K$/$10K$/$50K$ generated images and $10K$/$10K$/$50K$ real testing (val. on ImageNet) images. For Oxford-102 Flowers/FFHQ, we calculate FID between $10K$/$50K$ fake images and the entire training set.

Network Architecture and Hyper-parameters
-----------------------------------------

We build on OmniGAN/BigGAN/StyleGAN2 for CIFAR-10. For CIFAR-100/ImageNet ($64\!\times\!64$), we experiment with OmniGAN/BigGAN. For ImageNet ($128\!\times\!128$), we build upon OmniGAN given OmniGAN consistently outperforms BigGAN. We employ MSG-StyleGAN as our baseline for Oxford-102 Flowers. For FFHQ, we build upon StyleGAN2. Details can be found in §\[sup:hyper\] of the supplementary material.

Results of Image Generation
---------------------------

The generated images for each dataset are given in §\[sup:exim\] of the supplementary material.

**CIFAR-10.** Table \[table:comparison\_C10\] shows results on OminGAN+LCSA, which outperforms the baseline OmniGAN by 0.25 and 1.43 on the IS and tFID metrics ($d'\!=\!256$). With $d'\!=\!512$, we outperform OmniGAN by $0.32$ and 3.52. We obtain further improvements with $d'\!=\!1024$ while baselines struggle to converge. Comparisons of different models with $d'\!=\!1024$ are in the §\[sup:cifar\] of the supplementary material.

  Model                 $d'$     IS $\uparrow$      tFID$\downarrow$    vFID $\downarrow$
  -------------------- ------ -------------------- ------------------- -------------------
  BigGAN$^\dagger$                    9.14                7.05                 $-$
  FQGAN$^\dagger$                     9.16                6.16                 $-$
  OmniGAN$^\dagger$                   9.63                5.52                 $-$
  CR-GAN                              $-$                  $-$                11.48
  ContraGAN                           $-$                  $-$                10.32
  ICR-GAN                             $-$                  $-$                9.21
  OmniGAN+LCSA                 **9.88**$\pm$0.02    **4.09**$\pm$0.10   **8.16**$\pm$0.07
  BigGAN                              9.36                8.16                12.16
  FQGAN                               9.38                7.65                11.72
  OmniGAN                             9.70                6.88                10.65
  OmniGAN+LCSA                   10.02$\pm$0.05       3.36$\pm$0.06       7.40$\pm$0.06
  StyleGAN2+ADA                  10.14$\pm$0.09       2.42$\pm$0.04       6.54$\pm$0.06
  StyleGAN2+ADA+LCSA           **10.18**$\pm$0.06   **2.32**$\pm$0.05   **6.36**$\pm$0.10
  OmniGAN+LCSA          1024   **10.21**$\pm$0.03   **2.94**$\pm$0.02   **6.98**$\pm$0.04

  : Results on CIFAR-10. We combine OmniGAN and StyleGAN2+ADA with LCSA. $^\dagger$ are results collected from @zhou2020omni.<span data-label="table:comparison_C10"></span>

**CIFAR-100.** Table \[table:comparison\_C100\] shows results on OmniGAN+LCSA against the state of the art. For $d'\!=\!256$, our method gains $0.09$ and 1.9 on the IS and tFID metrics over the baseline OmniGAN. For $d'\!=\!512$, we outperform OmniGAN by 0.93 and 3.91 (IS and tFID). Further gains are achieved for $d'\!=\!1024$ while other methods struggle to converge. We include TAC-GAN @TAC-GAN for comparison.

  Model                $d'$     IS $\uparrow$      tFID $\downarrow$   vFID $\downarrow$
  ------------------- ------ -------------------- ------------------- --------------------
  BigGAN$^\dagger$                  10.89                10.18                $-$
  FQGAN$^\dagger$                   10.62                8.23                 $-$
  OmniGAN$^\dagger$                 13.51                8.14                 $-$
  TAC-GAN                            9.34                7.22                 $-$
  OmniGAN+LCSA                **13.60**$\pm$0.11   **6.24**$\pm$0.09   **11.02**$\pm$0.13
  BigGAN                            11.44                10.16               15.24
  FQGAN                             11.05                7.76                12.70
  OmniGAN                           12.78                9.13                13.82
  OmniGAN+LCSA                **13.71**$\pm$0.03   **5.22**$\pm$0.10   **9.98**$\pm$0.08
  OmniGAN+LCSA         1024   **13.88**$\pm$0.12   **4.97**$\pm$0.09   **9.72**$\pm$0.08

  : Comparison of OmniGAN+LCSA with others on CIFAR-100. $^\dagger$ are results collected from @zhou2020omni.<span data-label="table:comparison_C100"></span>

**ImageNet ($\mathbf {64\times64}$).** Table \[table:comparison\_I64\] shows that BiGAN+LCSA outperforms BigGAN by 2.97 and 3.01 (tFID & vFID). OmniGAN+LCSA improves OmniGAN by 6.86 and 2.72 (IS & tFID), which is the state of the art in GANs. **ImageNet ($\mathbf {128\times128}$).** Table \[table:comparison\_I128\] shows OmniGAN+LCSA improves OmniGAN by 23.32 and 2.28 (IS & tFID).

**Oxford-102 Flowers.** Figure \[fig:comparison\_oxford\](a) shows that MSG-StyleGAN+LCSA improves MSG-StyleGAN by 5.46 on FID. Moreover, Figure \[fig:comparison\_oxford\](b) shows that at around iteration no. $75K$, MSG-StyleGAN starts diverging while MSG-StyleGAN+LCSA (Ours) continues to decrease FID.

**FFHQ ($\mathbf {256\times256}$).** Table \[table:comparison\_FFHQ\] shows that StyleGAN2 with LCSA improves the FID of StyleGAN2 by 1.55/0.39 on $70K/140K$ dataset. Moreover, our LCSA can also be combined with bCR @ICRGAN to improve the performance.

**Data-limited Generation on CIFAR-10/100.** We provide comprehensive experiments (10% and 20% data) in §\[sup:limited\_data\] of the supplementary material. We summarize our findings as: 1) the augmentation-based ADA @ADA and DA @DiffAug leak augmentation artifacts to the generator, while ADA+LCSA and DA+LCSA alleviate this issue, 2) LCSA harmonizes with ADA, DA and LeCam loss @lecamgan. 3) we achieve the state of the art on this limited data setting.

  Model                                  IS $\uparrow$   tFID $\downarrow$   vFID $\downarrow$
  ------------------------------------- --------------- ------------------- -------------------
  Inst. Sel. GAN @devries2020instance      **43.30**           9.07                 $-$
  BigGAN                                     34.50             8.96                8.80
  FQGAN                                      33.14             8.27                8.15
  BigGAN+LCSA                                33.29           **5.99**            **5.79**
  OmniGAN                                    70.59             7.09                7.66
  OmniGAN+LCSA                             **77.45**         **4.26**            **4.94**

  : Results on ImageNet ($64\times64$). We combine OmniGAN and BigGAN with LCSA. We set $d'\!=\!384$.<span data-label="table:comparison_I64"></span>

  Model                   IS $\uparrow$   tFID $\downarrow$   vFID $\downarrow$
  ---------------------- --------------- ------------------- -------------------
  BigGAN$^\dagger$           104.57             9.19                9.18
  OmniGAN$^\dagger$          190.94             8.30                8.93
  OmniGAN$^{\ddagger}$       169.13             7.11                7.30
  OmniGAN+LCSA             **192.45**         **4.83**            **5.24**

  : Results on ImageNet ($128\!\times\!128$) with $d'\!=\!384$. $^\dagger$ stands for quoting from @zhou2020omni, $^{\ddagger}$ are results reproduced by us.

\[table:comparison\_I128\]

<span>0.45</span>

\[b\]

|@

l@

|@

c@

|@

c@

|

Model & Iters & FID $\downarrow$\
MSG-ProGAN$^\dagger$ & 53$K$ & 28.27\
MSG-StyleGAN$^\dagger$ & 50$K$ & 19.60\
MSG-StyleGAN & 125$K$ & 18.59\
Ours & 125$K$ & **13.13**\
\

<span>0.5</span> ![Results on Oxford-102 Flowers. (a) FID of different models ($^\dagger$ are results collected from @karnewar2020msg). (b) FID the iteration number for MSG-StyleGAN and MSG-StyleGAN+LCSA on Oxford-102 Flowers. Black dots indicate the minimum FID.<span data-label="fig:comparison_oxford"></span>](images/Flowers_FIDs_CVPR.pdf "fig:"){width="\linewidth"}

  Data    StyleGAN2   +ADA   +LeCam   +bCR   +LCSA   +LCSA+bCR
  ------ ----------- ------ -------- ------ ------- -----------
  70k       5.28      4.30     -      3.79   3.83    **3.47**
  140k      3.71      3.81    3.66    3.53   3.32    **3.20**

  : The FID $\downarrow$ results on FFHQ (256$\times$256) dataset.<span data-label="table:comparison_FFHQ"></span>

Impact of Hyperparameters
-------------------------

\[h\]

<span>0.24</span> ![image](images/different_knns.pdf){width="\linewidth"}

<span>0.24</span> ![image](images/different_sigmas.pdf){width="\linewidth"}

<span>0.24</span> ![image](images/different_dict_size.pdf){width="\linewidth"}

<span>0.24</span> ![image](images/different_layers.pdf){width="\linewidth"}

Below, we conduct ablations on CIFAR-100 for $d'\!=\!512$.

**Nearest neighbours $\boldsymbol{k'}$, $\boldsymbol{\sigma}$ and $\boldsymbol{k}$.** Figure \[different\_hyper\_parameters\](a) shows results on OmniGAN+LCSA $k'$ on CIFAR-100, which verifies that the locality-constrained manifold can be constructed with $8\!\leq\!k'\!\leq\!32\!\ll\!k$ with a sufficiently small reconstruction error $\epsilon^2$ below the worst case (Point 3, Prop. \[prop:p1\]). For ImageNet, $k'\!=\!8$ also led to best results. Fig. \[different\_hyper\_parameters\](b) verifies that $1\!\leq\!\sigma\!\leq\!1.5$ provides the best trade-off in terms of smoothness (quantization linearization) (Points 1, 2 and 4 of Prop. \[prop:p1\]). Figure \[different\_hyper\_parameters\](c) verifies the benefit of dictionary overcompleteness $d'\!\ll\!k$ as $k\!=\!1024$.

**Blocks $\boldsymbol{l\!\in\!\{1,\cdots,L\}}$ and LCSA.** Figure \[different\_hyper\_parameters\](d) shows the results for various combinations of injection of LCSA into blocks of discriminator. It appears that injecting LCSA into all blocks yields the lowest tFID/vFID, which verifies that constructing multiple manifolds at coarse-to-fine semantic level controls the complexity of the discriminator.

**Metaparameter $\boldsymbol{\beta}$ and impact of $\boldsymbol{\eta}$.** $\beta$ in <span>Eq</span> controls the mixing of conv. features obtained by ${\boldsymbol{f}}$ and their view ${\boldsymbol{h}}$ recovered from the manifold, as in <span>Eq</span>. Fig. \[figure:eta\_beta\](a) shows that in early iterations, blocks of discriminator learn conv. features. Around $25K$ iterations, the model starts oscillating between prevention of overfitting and refining conv. features, as indicated by the green curve that continues to gradually grow for $\eta\!=\!0.5$ (flat curve for $\eta\!=\!0.3$). Figure \[figure:eta\_beta\](b) also shows that $\eta\!=\!0.5$ is a universally good threshold.

Analysis and Ablation studies
-----------------------------

We analyse our method on CIFAR-10/100 ($d'\!=\!512$).

**Preventing Discriminator Overfitting.** Fig. \[figure:Dout\_acc\_C10\] verifies that the discriminator of standard methods yields high real/fake accuracy on training images but low accuracy on testing images. Thus, they overfit to the training set (see the large discrepancy between real and fake images). Thus, standard methods diverge early (see FID) but our method continues learning, as shown in Fig.\[figure:fid\_C10\] and Fig. \[fig:comparison\_oxford\](b).

<span>0.493</span> ![Evolution of metaparameter $\beta$ and the impact of $\eta$ which controls the behavior of the detector of overfitting.<span data-label="figure:eta_beta"></span>](images/beta.pdf "fig:"){width="\linewidth"}

<span>0.493</span> ![Evolution of metaparameter $\beta$ and the impact of $\eta$ which controls the behavior of the detector of overfitting.<span data-label="figure:eta_beta"></span>](images/different_etas.pdf "fig:"){width="\linewidth"}

![Discriminator predictions on CIFAR-10. We plot the output of discriminator on training images and generated fake images. We test Acc (real/fake accuracy predicted by the discriminator) on the training images and testing images. We use different colors to represent different models. Solid/dash/dot lines indicate training/fake/testing images.<span data-label="figure:Dout_acc_C10"></span>](images/D_out.pdf){width="1\linewidth"}

![Evolution of tFID and vFID for different models on CIFAR-10. Black dots indicate the minimum FID. <span data-label="figure:fid_C10"></span>](images/C10_FIDs.pdf){width="1\linewidth"}

**Impact of Different Components.** In Table \[table:coding\_methods\], we ablate (i) dictionary learning (Eq(\[eq:dl\])), (ii) the adaptive mixing input Eq(\[eq:enc2\]) and (iii) proximity loss Eq(\[eq:prox\]). We conduct experiments with settings: (1) ACM: removing our mainfold learner and the proximity loss, and changing Eq(\[eq:enc2\]) to be ${\boldsymbol{X}}_{l+1}\!=\!(1\!-\!\beta){\boldsymbol{\tilde{X}}}_l$ to adaptively control the magnitude of ${\boldsymbol{\tilde{X}}}_l$; (2) LCSA($\gamma\!=\!0$): removing the proximity loss but keeping the adaptive mixing input; (3) LCSA($\beta\!=\!0$): removing the adaptive mixing input but keeping the adaptive proximity loss; (4) LCSA(EMA): replacing our dictionary learning with the exponential moving average; (5) LCSA(fixed ($\beta$, $\gamma$)): fixed meta-controller ($\beta$, $\gamma$) (different combination of ($\beta$ and $\gamma$) is given in §\[sup:fixx\] of the supplementary material). Table \[table:coding\_methods\] shows that (i) LCSA($\gamma\!=\!0$) is better than ACM and OmniGAN which verifies the benefit of adaptive mixing input. (ii) LCSA($\beta\!=\!0$) is better than OmniGAN which verifies the importance of controlling the complexity of learning. (iii) EMA performs worse than our dictionary learning. (iv) As fixed ($\beta$, $\gamma$) scores lower thus the adaptive meta-controller is useful. In §\[sup:residual\] (supplementary material), LCSA yields larger errors at foregrounds and smaller at backgrounds, thus intertwining step (mixing input) helps refine/denoise features before input to next layer.

**Different Coders.** Table \[table:coding\_methods\] compares coding methods, each applied to all $l\!=\!1,\cdots,4$ blocks of discriminator. LCSA is the best performer followed by SA and LLC or SC, which validates that using locality-constrained soft assignment whose continuity is controlled via the Lipschitz constant (Prop. \[prop:p1\]) inverse-proportional to $\sigma^2$ is more robust than locally-linear coding, due to the quantization linear reconstruction trade-off and the role of $\sigma^2$ in denoising (Prop. \[prop:den\]). We also tried replacing the LCSA coder with Denoising Auto-Encoder (DAE) @denoising_JMLR (the setup is in §\[sup:dae\] of the supplementary material). LCSA achieves better tFID and vFID than locality-constrained linear coding, subspace learning, and DAE. Figure \[fig:var1\] of the supplementary material shows the variance computed over LCSA codes for indivisual images. While visually mundane regions have low variance (, single code of $\boldsymbol{\alpha}$ encodes it), the visually diverse regions have variance that is somewhat higher for LCSA than other coders, preventing overfitting where it matters.

  --------------------------------- --------------- ------------------- ------------------- --------------- ------------------- -------------------
                                                                                                                                
                                     IS $\uparrow$   tFID $\downarrow$   vFID $\downarrow$   IS $\uparrow$   tFID $\downarrow$   vFID $\downarrow$
  OmniGAN                                9.70              6.88                10.65             12.78             9.13                13.82
  +ACM                                   9.79              6.03                9.99              12.82             10.61               15.51
  +LCSA($\gamma=0$)                      9.80              5.15                8.35              13.42             8.12                12.78
  +LCSA($\beta=0$)                       9.73              4.77                8.77              13.64             5.54                10.37
  +LCSA(EMA)                             9.91              3.83                7.89              13.71             5.63                10.40
  +LCSA(fixed($\beta$, $\gamma$))        10.01             4.06                8.03              13.64             5.46                10.29
  +LCSA                                  10.09           **3.29**            **7.31**            13.73           **5.12**            **9.91**
  +HA                                    9.88              4.52                8.49              13.68             5.52                10.33
  +SC$_+$                                10.09             3.53                7.65              13.82             5.49                10.31
  +SC                                    10.14             3.48                7.46              13.68             5.40                10.20
  +OMP                                   10.02             3.83                7.83              13.58             5.52                10.40
  +LLC                                   10.04             3.77                7.75              13.61             5.35                10.14
  +SA                                    10.00             3.46                7.45              13.76             5.31                10.19
  +DAE                                   10.08             3.89                7.91              13.65             5.41                10.22
  --------------------------------- --------------- ------------------- ------------------- --------------- ------------------- -------------------

  : Results for ablation studies on CIFAR-10 & CIFAR-100.<span data-label="table:coding_methods"></span>

Conclusions
===========

We have applied data-manifold learning (LCSA) in the coarse-to-fine manner to conv. features of discriminator to prevent its overfitting by adaptively balancing the trade-off between denoising on the mainfold and refining the mainfold (intertwining and dictionary learning steps), controlling the complexity of learning (the proximity loss). Locality-constrained soft assignment is controlled via the Lipschitz constant, resulting in a trade-off between quantization and linear coding, yielding state-of-the-art results.

Our code will be released in due course.

-   §\[sup:notations\] explains our mathematical notation.

-   §\[sup:cifar\] provides further evaluations on CIFAR-10 and CIFAR-100 (32$\times$32) datasets @CIFAR given the large feature size of $d'\!=\!1024$. We believe that it is valuable to demonstrate that our method can benefit from such a large $d'$ while other methods cannot benefit much when applying $d'\!=\!1024$ instead of the usual $d'\!=\!512$.

-   §\[sup:limited\_data\] provides results for the limited data setting on CIFAR-10 and CIFAR-100.

-   §\[sup:imagenet32\] provides results on the ImageNet (32$\times$32) dataset @deng2009imagenet in order to demonstrate that our LCSA-based pipeline benefits experiments on GAN for lower resolution images, which are favoured over high resolution images for instance when enriching training datasets of few-shot learning approaches. Note that results on ImageNet (64$\times$64) and ImageNet (128$\times$128) are given in our main submission.

-   §\[sup:note\_fqgan\] explains why FQGAN suffers from discriminator overfitting.

-   Section \[sup:dae\] investigates the use of Denoising Auto-Encoder (DAE) @denoising_JMLR in place of LCSA, and concludes that LCSA is more beneficial.

-   §\[sup:fixx\] investigates the use of fixed $\beta$ and $\gamma$ rather than metaoptimized $\beta$ and $\gamma$. The conclusion is that with mixing and proximity operators controlled by these parameters, we can reduce overfititng in the discriminator while training the manifold dictionary.

-   §\[sup:residual\] illustrates the residual error between ${\boldsymbol{\tilde{X}}}$ and $h({\boldsymbol{\tilde{X}}})$. The results support our belief that conv. encoder refines LCSA on difficult foreground objects while generating coarse-to-fine cov. features close to the manifold.

-   §\[sup:hyper\] details hyperparameters used in our experiments. We point out that the majority of parameters remain unchanged throughout many experiments.

-   §\[sup:code\] provides details of the feature encoding methods and the dictionary learning step.

-   §\[sup:exim\] illustrates examples of images generated with/without the LCSA coder.

-   §\[sup:proofs\] offers proofs for claims of §\[sec:lcsa\].

Notations {#sup:notations}
=========

Capitalized boldface symbols are matrices (, ${\boldsymbol{X}}$), lowercase boldface symbols are vectors (, ${\boldsymbol{x}}$), and regular fonts denote scalars , $n$, $N$, $l$, $L$, $X_{i,j}$ is a $(i,j)$-th coefficient of ${\boldsymbol{X}}$, $x_{i}$ is an $i$-th coefficient of ${\boldsymbol{x}}$. We define a vector of all-ones as $\mathbf{1}\!=\!\left[1,\cdots,1\right]^T$ and concatenation of $\alpha_i$ as $[\alpha_i]_{i=1,\cdots,I}$.

Experiments on CIFAR-10/100 ($\boldsymbol{d'\!=\!1024}$) {#sup:cifar}
========================================================

Below we demonstrate that utilizing LCSA in BigGAN and OmniGAN further enables them to utilize large feature sizes ($d'\!=\!1024$), improving results further. Original BigGAN and OmniGAN struggle to attain good results for such a large feature size. Tables \[table:CIFAR\_1024\] verifies our pipelines on CIFAR-10 and CIFAR-100. Figures \[figure:fid\_1024\_C10\] and \[figure:fid\_1024\_C100\] show that our BigGAN+LCSA and OmniGAN+LCSA continue to learn while other models struggle to converge.

  -------------- --------------- ------------------- ------------------- --------------- ------------------- -------------------
                                                                                                             
                  IS $\uparrow$   tFID $\downarrow$   vFID $\downarrow$   IS $\uparrow$   tFID $\downarrow$   vFID $\downarrow$
  BigGAN              9.64              10.71               14.86             11.54             15.11               19.97
  FQGAN               9.44              12.59               16.69           **11.74**           8.49                13.51
  BigGAN+LCSA       **9.81**          **3.51**            **7.55**            11.60           **5.49**            **10.37**
  OmniGAN             9.92              12.11               16.29             12.42             10.11               14.85
  OmniGAN+LCSA      **10.21**         **2.94**            **6.98**          **13.88**         **4.97**            **9.72**
  -------------- --------------- ------------------- ------------------- --------------- ------------------- -------------------

  : Results for different models on CIFAR-10 and CIFAR-100 with $d'\!=\!1024$.<span data-label="table:CIFAR_1024"></span>

![Evolution of tFID and vFID for different models on CIFAR-10 with $d'\!=\!1024$. Black dots indicate the minimum FID. <span data-label="figure:fid_1024_C10"></span>](images/C10_1024_FIDs.pdf){width="1\linewidth"}

![Evolution of tFID and vFID for different models on CIFAR-100 with $d'\!\!=\!\!1024$. Black dots indicate the minimum FID. <span data-label="figure:fid_1024_C100"></span>](images/C100_1024_FIDs.pdf){width="1\linewidth"}

\[ht!\]

|@

l@

|c@

|c@

|c@

|c@

|@

c@

|@

c@

|

& &\
& 100% & 20% & 10%& 100% & 20% & 10%\
& IS$\uparrow$ / tFID$\downarrow$ / vFID$\downarrow$ & IS$\uparrow$ / tFID$\downarrow$ / vFID$\downarrow$& IS$\uparrow$ / tFID$\downarrow$ / vFID$\downarrow$& IS$\uparrow$ / tFID$\downarrow$ / vFID$\downarrow$& IS$\uparrow$ / tFID$\downarrow$ / vFID$\downarrow$& IS$\uparrow$ / tFID$\downarrow$ / vFID$\downarrow$\
BigGAN($d'\!=\!256$) & 9.26 /   5.46 /   9.38 & 8.70 / 16.25 / 20.37 & 8.20 / 31.42 / 35.58 & 10.99 / 7.89 / 12.74 &   9.94 / 25.96 / 30.89 &   7.54 / 50.89 / 55.15\
+DA & 9.39 /   4.47 /   8.58 & 8.95 /   9.38 / 13.26 & 8.65 / 18.35 / 22.04 & 10.91 / 7.30 / 11.99 &   9.73 / 16.32 / 20.88 &   9.33 / 27.01 / 31.32\
+LeCam & 9.48 /   4.25 /   8.27 & 8.96 / 11.32 / 15.25 & 8.50 / 26.30 / 30.55 & **11.44** / 6.53 / 11.23 & 10.00 / 20.82 / 25.77 &   8.10 / 39.33 / 44.23\
+LCSA & **9.51** /   **4.12** /   **8.20** & **8.99** /   **8.72** / **12.73** & **8.74** / **12.36** / **16.46** & 11.02 / **6.38** / **11.13** & **10.38** / **13.22** / **18.06** & **10.08** / **21.13** / **25.87**\

+LeCam+DA & 9.47 /   4.29 /   8.29 & 9.05 /   7.53 / 11.46 & 8.84 / 12.15 / 15.92 & 11.15 / 6.55 / 11.34 & 10.42 / 13.01 / 17.65 &   9.92 / 21.73 / 26.19\
+LCSA+DA & **9.50** /   **3.75** /   **7.83** & **9.08** /   7.29 / 11.31 & 8.86 / 10.86 / 14.78 & **11.21** / **5.76** / **10.52** & 10.55 / 12.03 / 16.92 & **10.68** / 19.38 / 24.21\
+LCSA+LeCam+DA & 9.47 /   3.80 /   7.89 & 9.04 /   **6.95** / **10.95** & **8.96** / **10.05** / **13.88** & 11.17 / 5.85 / 10.64 & **10.67** / **10.16** / **15.00** & 10.28 / **18.24** / **23.12**\

OmniGAN($d'\!=\!1024$) & 9.98 /   6.89 / 10.76 & 8.62 / 37.86 / 42.18 & 6.59 / 54.04 / 58.71 & 12.61 / 8.43 / 13.21 & 10.11 / 40.57 / 44.86 &   6.87 / 63.41 / 67.46\
+DA & 10.10 /   4.26 /   8.15 & 9.47 / 13.56 / 17.34 & 8.96 / 19.59 / 23.60 & 12.96 / 7.48 / 12.11 & 11.42 / 17.72 / 22.49 & 10.21 / 32.61 / 36.86\
+ADA & **10.27** /   5.03 /   9.15 & 9.44 / 27.20 / 31.05 & 7.72 / 41.82 / 45.36 & 785-943-2312 & 12.18 / 13.66 / 18.34 &   8.81 / 46.74 / 51.03\
+LCSA & 10.20 /   **2.58** /    **6.65** & **9.97** /   **5.15** /   **9.01** & **9.74** /   **7.66** / **11.47** & **13.74** / **4.94** /   **9.67** & **12.86** / **11.15** / **15.90** & **11.85** / **13.82** / **18.56**\
+LCSA+DA & 10.22 /   2.52 /   6.60 & 9.93 /   5.01 /   8.89 & 9.75 /   7.46 / 11.35 & 13.79 / 4.58 /   9.46 & 12.92 / 11.06 / 15.73 & 11.88 / 13.68 / 18.44\
+LCSA+ADA & **10.38** /    **2.35** /   **6.38** & **10.12** /   **4.48** /   **8.41** & **10.04** /   **6.45** / **10.39** & **13.80** / **4.07** /   **8.90** & **13.78** /   **7.45** / **12.11** & **12.67** / **10.18** / **14.87**\

<span>0.485</span> ![tFID different $d'$ on CIFAR-10 and CIFAR-100 under the limited data setting.<span data-label="fig:limited_feature_size"></span>](images/C10lim10_20.pdf "fig:"){width="\linewidth"}

<span>0.485</span> ![tFID different $d'$ on CIFAR-10 and CIFAR-100 under the limited data setting.<span data-label="fig:limited_feature_size"></span>](images/C100lim10_20.pdf "fig:"){width="\linewidth"}

<span>0.485</span> ![Augmentation strength ($p$) of ADA ADA+LCSA during training on 10%/20% CIFAR-10 data (OmniGAN $d'\!\!=\!\!256$).<span data-label="fig:aug_p"></span>](images/aug_p_C10lim10.pdf "fig:"){width="\linewidth"}

<span>0.485</span> ![Augmentation strength ($p$) of ADA ADA+LCSA during training on 10%/20% CIFAR-10 data (OmniGAN $d'\!\!=\!\!256$).<span data-label="fig:aug_p"></span>](images/aug_p_C10lim20.pdf "fig:"){width="\linewidth"}

Data-limited Generation on CIFAR-10/100 {#sup:limited_data}
=======================================

Both DA @DiffAug and ADA @ADA limit discriminator overfitting by augmenting the real and generated images that are passed to the discriminator. However, in case of DA and ADA, augmentation artifacts leak into the generated images as shown in Figure \[fig:C10\_lim10\]. The LeCamGAN @lecamgan restricts discriminator overfitting by reducing the real/fake loss discrepancy at the output of the discriminator. However, it is not enough to limit overfitting by just operating at the output of the discriminator (otherwise a well-designed loss would deal with all overfitting issues which is not the case in the literature). Therefore, we prevent the discriminator overfitting by modifying its blocks by encoding both the features of real and fake images into a common manifold and recovering their view from it, which has the ability to control the complexity of our feature space.

Below, we compare our LCSA with DA @DiffAug, ADA @ADA on OmniGAN (and/or BigGAN) to variants without LCSA. We conduct experiments based on BigGAN to compare our LCSA with LeCamGAN @lecamgan due to the issue of combining LeCam loss @lecamgan with the multi-output loss of OmniGAN. Following @DiffAug, we augmented the real images with $x$-flip, and trained the OmniGAN and BigGAN for $1K$ epochs on the full data and $5K$ epochs on 10%/20% data setting. The DiffAug used translation and cutout augmentations. The ADA used 18 augmentations, including rotations and scaling. Details can be founded in @ADA.

Table \[tab:comparison\_aug\] shows that our LCSA outperforms DA, ADA and LeCamGAN. LCSA can be always combined with them to improve their performance. Specifically, we achieve state-of-the-art results on CIFAR-10 (100%, 20% and 10%) and CIFAR-100 (20% and 10%) based on OmniGAN ($d'\!=\!1024$). In addition, in Figure \[fig:limited\_feature\_size\], OmniGAN, DA and ADA become worse when increasing the $d'$, while combining them with our LCSA improves the tFID given the larger model size, on which the discriminator can memorize the training data easier, demonstrating that our LCSA has stronger performance while preventing overfitting better than ADA and DA alone. Moreover, we find that the DA and ADA leak the augmentation cues to the generated images, as shown in Figure \[fig:C10\_lim10\], indicating that their discriminators overfit to the augmentation cues instead of learning meaningful discrimination boundary. ADA strives to control the strength of augmentations by a controller which observes the discriminator output and adjusts the desired augmentation strength. Thus, in Figure \[fig:aug\_p\] we plot the augmentation strength of ADA and LCSA+ADA. We notice that **LCSA+ADA is able to limit the augmentation strength of ADA, thus reducing the leakage of augmentation artifacts.** This means LCSA can deal with the discriminator overfitting well. Concluding, LCSA enjoys better performance than ADA, DA, and LeCam for data-limited generation. LCSA gains further improvements with larger model size $d'$, and LCSA lowers the risk of leaking augmentation artifacts. Thus, LCSA appears to have the superior capability of preventing discriminator overfitting than other strategies.

Experiments on ImageNet (32$\times$32) {#sup:imagenet32}
======================================

We preprocess images by center-cropping and downscaling them to $32\times32$ pixels. We train the network for 100 epochs. We set mini-batch size to 64, $\beta_0\!=\!0.$, $\gamma_0\!=\!0.1$, $\eta\!=\!0.5$, $\sigma\!=\!1.2$, whereas the dictionary learning rate is set to 1e-3. We equip each residual block of the discriminator with our LCSA module. We set $\Delta_{\gamma}\!=\!1.2$ for $d'\!=\!512$. For OmniGAN and OmniGAN+LCSA, we set weight decay of discriminator to be 1e-4 given $d'\!=\!512$. Table \[tab:my\_label\] presents our results in the above setting.

  Model           IS $\uparrow$   tFID $\downarrow$   vFID $\downarrow$
  -------------- --------------- ------------------- -------------------
  BigGAN              13.25             5.44                5.60
  BigGAN+LCSA       **13.61**         **4.82**            **5.01**
  OmniGAN             29.55             3.74                4.56
  OmniGAN+LCSA      **30.53**         **3.53**            **4.37**

  : Results for ImageNet ($32\!\times\!32$).<span data-label="tab:my_label"></span>

A difference between FQGAN and Hard Assignment {#sup:note_fqgan}
==============================================

As we note in §\[sec:code\], if ${\boldsymbol{M}}$ is formed by k-means clustering, Hard Assignment becomes an equivalent of the quantizer from FQGAN. However, the FQGAN suffers from discriminator overfitting, as we shown in Figures \[figure:fid\_C10\], \[figure:fid\_1024\_C10\] and \[figure:fid\_1024\_C100\]. The FQGAN uses the proximity loss Eq(\[eq:prox\]), however, it has no intertwining step between the manifold learning step with blocks of discriminator. That is, for FQGAN, the feature input to the next layer from our pipeline Eq(\[eq:enc2\]) is changed to:

$${\boldsymbol{X}}_{l+1}\!={\boldsymbol{\tilde{X}}}_l
\vspace{-0.2cm}$$

Without the interwining step, without the adaptive mechanism, and without LCSA, the FQGAN cannot prevent blocks in discriminator from over-expressing the learned features, which may lie outside of the manifold, resulting in discriminator overfitting.

Replacing LCSA with Denoising Auto-Encoder (DAE) {#sup:dae}
================================================

![Architecture of DAE. We plug DAE in place of LCSA in Figure \[fig:res\_blcoks\]: we replace ${\boldsymbol{h}}(\cdot)$ based on LCSA with ${\boldsymbol{h}}(\cdot)$ based on DAE.<span data-label="fig:dae"></span>](images/dae_new.pdf){width="8.0cm"}

We conduct experiments in which we replace the LCSA coder with Denoising Auto-Encoder(DAE) @denoising_JMLR. As illustrated in Figure \[fig:dae\], two FC layers, each intertwined with LeakyReLU, form our DAE module. Moreover, we apply the mixing mechanism from <span>Eq</span> and the proximity operator from <span>Eq</span> which is an equivalent of the reconstruction loss of DAE given in <span>Eq</span>. As Figure \[fig:dae\] indicates, DAE is equipped with a noise injector $\boldsymbol{\varepsilon}\!\sim\!\mathcal{N}(
\boldsymbol{0},\boldsymbol{\sigma'}^2)$ with variance $\boldsymbol{\sigma'}^2$ (equiv. in <span>Eq</span>). We use the multivariate normal distribution which produces noise samples $\boldsymbol{\varepsilon}$ that are added to ${\boldsymbol{X}}'_l$, while $\boldsymbol{\sigma'}^2$ is a vector of on-diagonals equal $\sigma'^2$ (off-diagonal coefficients equal 0).

Instead of $k$ (dictionary size), we vary the size of FC layers $(d'\!\times\!k_*)$ and $(k_*\!\times\!d')$, by varying the size of hidden representation $k_*$. We investigate $k_*\!\in\!\{8, 16, 32, 64, 128, 256\}$ and $\sigma'^2\!\in\!\{0.01, 0.04, 0.25, 0.64, 1.0\}$. After searching for the best set of parameters, we obtain $k_*\!=\!64$, $\sigma'^2\!=\!0.04$ for CIFAR-10 and $k_*\!=\!32$, $\sigma'^2\!=\!0.04$ for CIFAR-100. The DAE module was updated once at each mini-batch.

Table \[table:dae\] shows that OmniGAN+LCSA outperforms OmniGAN+DAE, which validates our assumptions on locality-constrained non-linear manifold learning and its ability to deal with the noise and regularization of discriminator.

  -------------- --------------- ------------------- ------------------- --------------- ------------------- -------------------
                                                                                                             
                  IS $\uparrow$   tFID $\downarrow$   vFID $\downarrow$   IS $\uparrow$   tFID $\downarrow$   vFID $\downarrow$
  OmniGAN             9.70              6.88                10.65             12.78             9.13                13.82
  OmniGAN+DAE         10.08             3.89                7.91              13.65             5.52                10.33
  OmniGAN+LCSA      **10.09**         **3.29**            **7.31**          **13.73**         **5.12**            **9.91**
  -------------- --------------- ------------------- ------------------- --------------- ------------------- -------------------

  : OmniGAN+DAE OmniGAN+LCSA ($d'\!=\!512$).<span data-label="table:dae"></span>

Fixed $\beta$ and $\gamma$ {#sup:fixx}
==========================

To verify that metaparameter learning of $\beta$ and $\gamma$ works better than setting fixed $\beta$ and $\gamma$, Table \[table:fixed\_betaa\] shows for the best possible fixed $\beta$. For each experiment, $\gamma\!=\!\Delta_{\gamma}\beta$ and $\Delta_{\gamma}\!=\!1.2$ (same as for learnable experiment). For fixed $\beta$ and $\gamma$, the best results are obtained with ($\beta\!=\!0.4$, $\gamma\!=\!0.48$) for CIFAR-10 and ($\beta\!=\!0.2$, $\gamma\!=\!0.24$) for CIFAR-100. Figure \[fig:fixed\_beta\_gamma\] provides further ablations.

  ----------- --------------- ------------------- ------------------- --------------- ------------------- -------------------
                                                                                                          
               IS $\uparrow$   tFID $\downarrow$   vFID $\downarrow$   IS $\uparrow$   tFID $\downarrow$   vFID $\downarrow$
  Fixed            10.01             4.06                8.03              13.64             5.46                10.29
  Learnable      **10.09**         **3.29**            **7.31**          **13.73**         **5.12**            **9.91**
  ----------- --------------- ------------------- ------------------- --------------- ------------------- -------------------

  : Best fixed $\beta$ and $\gamma$ best metalearnable $\beta$ and $\gamma$.<span data-label="table:fixed_betaa"></span>

<span>0.485</span> ![Fixed $\beta$ and $\gamma$, $\gamma=\Delta_{\gamma}\beta$ where $\Delta_{\gamma}=1.2$ (as in the adaptive setting). We show results for several fixed values of $\beta$.<span data-label="fig:fixed_beta_gamma"></span>](images/fixed_beta_gamma_C10.pdf "fig:"){width="\linewidth"}

<span>0.485</span> ![Fixed $\beta$ and $\gamma$, $\gamma=\Delta_{\gamma}\beta$ where $\Delta_{\gamma}=1.2$ (as in the adaptive setting). We show results for several fixed values of $\beta$.<span data-label="fig:fixed_beta_gamma"></span>](images/fixed_beta_gamma.pdf "fig:"){width="\linewidth"}

Illustration of the residual error  {#sup:residual}
===================================

\[!t\]

<span>0.24</span> ![image](images/car.jpg){width="\linewidth"}

<span>0.24</span> ![image](images/car_layer1.jpg){width="\linewidth"}

<span>0.24</span> ![image](images/bird.jpg){width="\linewidth"}

<span>0.24</span> ![image](images/bird_layer1.jpg){width="\linewidth"}

<span>0.24</span> ![image](images/dog.jpg){width="\linewidth"}

<span>0.24</span> ![image](images/dog_layer1.jpg){width="\linewidth"}

<span>0.24</span> ![image](images/horse.jpg){width="\linewidth"}

<span>0.24</span> ![image](images/horse_layer1.jpg){width="\linewidth"}

We randomly sample images from real dataset on CIFAR-10, and input them into the discriminator. We obtain $\|{\boldsymbol{X}}_1\!-\!{\boldsymbol{\tilde{X}}}_1\|_F^2$ at the first residual block and normalize for visualization. Figure \[fig:visualization\] shows LCSA coder perpetrates larger errors at locations of difficult objects and smaller ones at locations of backgrounds. As <span>Eq</span>.

Network architecture and Hyperparameters {#sup:hyper}
========================================

Unless specified otherwise, we set $k\!=\!1024$, $\sigma=1.2$, $\gamma_0\!=\!0$, $\beta_0\!=0.1$, and $\eta=0.5$.

**CIFAR-10.** Below, we experiment with OmniGAN @zhou2020omni which adopts the architecture of BigGAN @BigGAN. We follow OmniGAN mini-batch size of 32 and train our networks for 500 epochs. To this end, we equip each residual block of the discriminator with the LCSA module. For our experiments on CIFAR-10, we apply weight decays of 1e-4, 1e-5 and 0 on generator given $d'\!=\!256, 512, 1024$, weight decay 0 for the discriminator. To update the dictionary, we use the Adam optimizer @kingma2014adam with learning rate of 0.002. We set We use $\Delta_{\gamma}$ equal 1.0, 1.2 and 2.5 given $d'\!=\!256, 512, 1024$. For StyleGAN2+ADA+LCSA, we follow @ADA and set $\eta=0.6$. We set $\Delta_\gamma=0.1$ due to the architecture differences between StyleGAN2 and BigGAN.

**CIFAR-100.** We use CIFAR-10 settings but we set the learning rate of DL to 0.0015, $k'\!=\!32$, and weight decays of 5e-4, 2e-4, 5e-5 for discriminators with $d'\!=\!256, 512, 1024$.

**ImageNet ($\mathbf {64\!\times\!64}$).** We equip BigGAN and OmniGAN with LCSA (the last 3 blocks $l\!\in\!\{3,4,5\}$, $d'\!=\!384$). We use the architecture of BigGAN @BigGAN. For OmniGAN and OmniGAN+LCSA, we set mini-batch size to 256, learning rates of generator/discriminator to 4e-4, and take 4 discriminator steps per generator step to prevent OmniGAN from diverging. Weight decays are 0 and 2e-5 for generator and discriminator. For BigGAN, FQGAN and BigGAN+LCSA, we apply settings of FQGAN @FQGAN to set mini-batch size to 512, learning rates of generator and discriminator to 1e-4 and 4e-4. We set dictionary learning rate to 1e-3, $\Delta_{\gamma}\!=\!1.5$, $k'\!=\!8$. We set $\sigma\!=\!0.8$, $\beta_0\!=\!0$, $\gamma_0\!=\!0.1$ given the discriminator does not overfit on this diverse dataset at the early training stage. All models are trained for 200 epochs.

**ImageNet ($\mathbf {128\!\times\!128}$)**. We only equip OmniGAN with LCSA as OmniGAN consistently outperforms BigGAN. We use setting from ImageNet ($64\!\times\!64$) but set dict. learning rate to 2e-4 and weight decay of discriminator to 5e-6. The two models are trained for 300 epochs.

**Oxford-102 Flowers ($\mathbf {256\!\times\!256}$)**. We employ the state-of-the-art model for this dataset, MSG-StyleGAN @karnewar2020msg, as our baseline. Following @karnewar2020msg, we set the size of mini-batch to 32 while $d'\!=\!512$. We set the number of iteration to $125K$, $k'\!=\!32$, $\Delta_{\gamma}\!=\!1.$, $\eta=0.7$, dictionary learning rate is 5e-4, and LCSA is applied to blocks 4 and 5 of discriminator.

**FFHQ ($\mathbf {256\!\times\!256}$)**. Following @ADA, we set the mini-batch size to 64, $d'\!\!=\!\!512$ and train the networks until the discriminator had seen $25M$ real images. We equip StyleGAN2 with LCSA (the last 4 blocks $l\!\in\!\{4,5,6,7\}$). We set $k'\!\!=\!\!32$, $\Delta_{\gamma}\!\!=\!\!0.1$, $\eta\!\!=\!\!0.9$, dictionary learning rate is 5e-4. Table \[table:hyperparams\] summarizes hyperparameters used in our experiments. We used a very limited range of these parameters (most par. do not change between experiments) , dict. learning rate equals 1e-3, 1.5e-3, 2e-3, 5e-4 or 2e-4, $\beta_0$ is set to 0 or 0.1, $\gamma_0$ is set to 0 or 0.1, $\Delta_\gamma$ is set to 0.1, 1, 1.2, 1.5 or 2.5, $\sigma$ is set to 0.8 or 1.2, $\eta$ is set to 0.5, 0.6, 0.7 or 0.9, and $k'$ to 8, 32, or 128. Table \[table:time\_cost\] provides the training time for models without or with LCSA coder.

\[h\]

      dataset       $d'$         Model          dict LR   $\beta_0$   $\gamma_0$   $\Delta_{\gamma}$   $k'$   $\sigma$   $\eta$      $L$      GWD    DWD
  ---------------- ------ -------------------- --------- ----------- ------------ ------------------- ------ ---------- -------- ----------- ------ ------
                    256       OmniGAN+LCSA       2e-3        0.1          0               1.0          128      1.2       0.5     {1,2,3,4}   1e-4    0
                                 BigGAN           $-$        $-$         $-$              $-$          $-$      $-$       $-$        $-$       0      0
                              BigGAN+LCSA        2e-3        0.1          0               1.2          128      1.2       0.5     {1,2,3,4}    0      0
                                OmniGAN           $-$        $-$         $-$              $-$          $-$      $-$       $-$        $-$      1e-5    0
                              OmniGAN+LCSA       2e-3        0.1          0               1.2          128      1.2       0.5     {1,2,3,4}   1e-5    0
                           StyleGAN2+ADA+LCSA    2e-3        0.1          0               0.1          128      1.2       0.6     {1,2,3,4}    0      0
                                 BigGAN           $-$        $-$         $-$              $-$          $-$      $-$       $-$        $-$       0      0
                              BigGAN+LCSA        2e-3        0.1          0               2.5          128      1.2       0.5     {1,2,3,4}    0      0
                                OmniGAN           $-$        $-$         $-$              $-$          $-$      $-$       $-$        $-$       0      0
                              OmniGAN+LCSA       2e-3        0.1          0               2.5          128      1.2       0.5     {1,2,3,4}    0      0
                    256       OmniGAN+LCSA      1.5e-3       0.1          0               1.0           32      1.2       0.5     {1,2,3,4}   1e-4   5e-4
                                 BigGAN           $-$        $-$         $-$              $-$          $-$      $-$       $-$        $-$       0      0
                              BigGAN+LCSA       1.5e-3       0.1          0               1.2           32      1.2       0.5     {1,2,3,4}    0      0
                                OmniGAN           $-$        $-$         $-$              $-$          $-$      $-$       $-$        $-$      1e-5   2e-4
                              OmniGAN+LCSA      1.5e-3       0.1          0               1.2           32      1.2       0.5     {1,2,3,4}   1e-5   2e-4
                                 BigGAN           $-$        $-$         $-$              $-$          $-$      $-$       $-$        $-$       0      0
                              BigGAN+LCSA       1.5e-3       0.1          0               2.5           32      1.2       0.5     {1,2,3,4}    0      0
                                OmniGAN           $-$        $-$         $-$              $-$          $-$      $-$       $-$        $-$       0     5e-5
                              OmniGAN+LCSA      1.5e-3       0.1          0               2.5           32      1.2       0.5     {1,2,3,4}    0     5e-5
                                 BigGAN           $-$        $-$         $-$              $-$          $-$      $-$       $-$        $-$       0      0
      ImageNet                BigGAN+LCSA        1e-3         0          0.1              1.5           8       0.8       0.5      {3,4,5}     0      0
    $64\times64$                OmniGAN           $-$        $-$         $-$              $-$          $-$      $-$       $-$        $-$       0     2e-5
                              OmniGAN+LCSA       1e-3         0          0.1              1.5           8       0.8       0.5      {3,4,5}     0     2e-5
      ImageNet                  OmniGAN           $-$        $-$         $-$              $-$          $-$      $-$       $-$        $-$       0     5e-6
   $128\times128$             OmniGAN+LCSA       2e-4         0          0.1              1.5           8       0.8       0.5      {4,5,6}     0     5e-6
     Oxford-102               MSG-StyleGAN        $-$        $-$         $-$              $-$          $-$      $-$       $-$        $-$       0      0
      Flowers              MSG-StyleGAN+LCSA     5e-4        0.1          0                1            32      1.2       0.7       {4,5}      0      0
        FFHQ        512     StyleGANV2+LCSA      5e-4        0.1          0               0.1           32      1.2       0.9     {4,5,6,7}    0      0

  Dataset / Baseline / GPUs                     Baseline   +LCSA
  -------------------------------------------- ---------- --------
  CIFAR-10 / OmniGAN($d'=512$) / 1               27h11m    30h54m
  CIFAR-100 / OmniGAN($d'=512$) / 1              27h29m    30h32m
  ImageNet($64\!\times\!64$) / OmniGAN / 4       3d15h     4d01h
  ImageNet($128\!\times\!128$) / OmniGAN / 4     20d03h    22d10h
  Oxford-102 Flowers / MSG-StyleGAN / 4          22h46m    23h09m
  FFHQ / StyleGAN2 / 4                           58h42m    59h51m

  : <span>Training time for models with without the LCSA encoder</span>. Experiments were performed on NVIDA Tesla V100 GPUs.<span data-label="table:time_cost"></span>

Coding Methods {#sup:code}
==============

Algorithms \[code:nnsc\] and \[code:sc\] detail our implementations of Non-Negative Sparse Coding (SC$_+$) and Sparse Coding (SC).

\[1\]<span>**except**\#1</span> \[1\]<span>**else**\#1</span>

\[1\]<span>**if**\#1</span>

\[!h\]

<span>**Input:**</span> ${\boldsymbol{X}}, {\boldsymbol{M}}$ for a given forward pass, $\kappa$: the $L_1$ norm penalty, $\iota$: the number of iterations, $\omega$: the learning rate.

\[1\] $\{\alpha_{in}\!\sim\!\mathcal{U}(1e\!-\!6,1)\}_{i\!=\!1,\cdots,k\text{
and } n\!=\!1,\cdots,N'}$ ${\boldsymbol{\alpha}}\!\gets\!\left[\frac{{\boldsymbol{\alpha}}_n}{\|{\boldsymbol{\alpha}}_n\|_1+1e\!-\!6}
\right]_{n\!=\!1,\cdots,N'}$(concat. of ${\boldsymbol{\alpha}}_n$ into matrix ${\boldsymbol{\alpha}}$) $\Delta{\boldsymbol{\alpha}}\!=\!2{\boldsymbol{M}}^T\!({\boldsymbol{X}}\!-\!{\boldsymbol{M}}{\boldsymbol{\alpha}})+\kappa$ (gradient computation) ${\boldsymbol{\alpha}}\!\gets\!{\boldsymbol{\alpha}}\!-\!\frac{\omega}{(1+i)^{0.3}}\Delta{\boldsymbol{\alpha}}$ (stochastic gradient descent) ${\boldsymbol{\alpha}}\!\gets\!\text{ReLU}({\boldsymbol{\alpha}})$ (reprojection into the feasible set)

<span>**Output:**</span> ${\boldsymbol{\alpha}}$: sparse codes

\[!h\]

<span>**Input:**</span> ${\boldsymbol{X}}, {\boldsymbol{M}}$ for a given forward pass, $\kappa$: the $L_1$ norm penalty, $\iota$: the number of iterations, $\omega$: the learning rate.

\[1\] $\{\alpha_{in}\!\sim\!\mathcal{U}(-1,1)\}_{i\!=\!1,\cdots,k\text{ and }
n\!=\!1,\cdots,N'}$ ${\boldsymbol{\alpha}}_+\!=\!\text{ReLU}({\boldsymbol{\alpha}})\!+\!1e\!-\!6$ and ${\boldsymbol{\alpha}}_-\!=\!\text{ReLU}(-{\boldsymbol{\alpha}})\!+\!1e\!-\!6$ ${\boldsymbol{\alpha}}_+\!\gets\!\left[\frac{{\boldsymbol{\alpha}}_{+,n}}{\|{\boldsymbol{\alpha}}_{+,n}\|_1+1e\!-
\!6}\right]_{n\!=\!1,\cdots,N'}$, ${\boldsymbol{\alpha}}_-\!\gets\!\left[\frac{
{\boldsymbol{\alpha}}_{-,n}}{\|{\boldsymbol{\alpha}}_{-,n}\|_1+1e\!-\!6}\right]_{n\!=\!1,\cdots,N'}\!\!\!\!
\!\!\!\!$ $\Delta{\boldsymbol{\alpha}}\!=\!2{\boldsymbol{M}}^T\!({\boldsymbol{X}}\!-\!{\boldsymbol{M}}({\boldsymbol{\alpha}}_+\!-\!{\boldsymbol{\alpha}}_-))$ (gradient computation) $\Delta{\boldsymbol{\alpha}}_+\!=\!\Delta{\boldsymbol{\alpha}}+\kappa$ and $\Delta{\boldsymbol{\alpha}}_-\!=\!-\Delta{\boldsymbol{\alpha}}+\kappa$ ${\boldsymbol{\alpha}}_+\!\gets\!{\boldsymbol{\alpha}}_+\!-\!\frac{\omega}{(1+i)^{0.3}}\Delta
{\boldsymbol{\alpha}}_+$ and ${\boldsymbol{\alpha}}_-\!\gets\!{\boldsymbol{\alpha}}_-\!-\!\frac{\omega}{(1+i)^{0.3}}\Delta
{\boldsymbol{\alpha}}_-$ ${\boldsymbol{\alpha}}_+\!\gets\!\text{ReLU}({\boldsymbol{\alpha}}_+)$ and ${\boldsymbol{\alpha}}_-\!\gets\!\text{ReLU}({\boldsymbol{\alpha}}_-)$ ${\boldsymbol{\alpha}}\!\gets\!{\boldsymbol{\alpha}}_+\!-\!{\boldsymbol{\alpha}}_-$

<span>**Output:**</span> ${\boldsymbol{\alpha}}$: sparse codes

\[!h\]

<span>**Input:**</span> ${\boldsymbol{X}}, {\boldsymbol{M}}$ for a given forward pass, $\tau$: the number of non-zero elements (iterations).

\[1\] $\boldsymbol{E}\!=\!{\boldsymbol{X}}$ (initialize matrix of residuals) ${\boldsymbol{M}}^{*}\!=\!\emptyset$ (active dictionary tensor ${\boldsymbol{M}}^{*}\!\in\!{\mathbb{R}^{d'\times 0 \times N'}}$) $\boldsymbol{P}\!=\!{\boldsymbol{M}}^T\!\boldsymbol{E}$ (projection of ${\boldsymbol{X}}$ onto residual $\boldsymbol{E}$) $\boldsymbol{\varphi^i}\!=\!\text{idx\_max}(|\boldsymbol{P}|)$ (index of maximum coefficient $\qquad\qquad\qquad\quad\;\;\;$per column) ${\boldsymbol{M}}^*\!=\text{cat}({\boldsymbol{M}}^*, [{\boldsymbol{m}}_{\varphi_1^i},\cdots,
{\boldsymbol{m}}_{\varphi_{N'}^i}]; 1)$ (active atoms $\qquad$are added to tensor ${\boldsymbol{M}}^{*}$ so that ${\boldsymbol{M}}^{*}\!\in\!{\mathbb{R}^{d'\times i \times N'}})$ ${\boldsymbol{\bar{M}}}\!=\!\text{squeeze}({\boldsymbol{M}}_{:,:,n})$ (dict. ${\boldsymbol{\bar{M}}}\!\in\!{\mathbb{R}^{d'\times i}}$ for ${\boldsymbol{x}}_n$) ${\boldsymbol{\alpha}}_n\!=\!\text{solve}({\boldsymbol{\bar{M}}}^T\!{\boldsymbol{x}}_n, {\boldsymbol{\bar{M}}}^T\!{\boldsymbol{\bar{M}}})$ (system of linear $\quad$equations solved by batch solver torch.solve($\cdot$) $\boldsymbol{e}_n\!=\!{\boldsymbol{x}}_n\!-\!{\boldsymbol{\bar{M}}}{\boldsymbol{\alpha}}_n$ (residual solved by batch opera- $\qquad\;\;\;$tions at once, that is, $\boldsymbol{E}\!\equiv\![\boldsymbol{e}_1,\cdots,\boldsymbol{e}_{N'}]$) ${\boldsymbol{\alpha}}\!=\!\text{one\_hot}([\boldsymbol{\varphi^1,\cdots,\boldsymbol{
\varphi^\tau}}])\!\odot\![{\boldsymbol{\alpha}}_1,\cdots,{\boldsymbol{\alpha}}_{N'}]$

<span>**Output:**</span> ${\boldsymbol{\alpha}}$: sparse codes

Algorithm \[code:omp\] is our efficient implementation of the Orthogonal Matching Pursuit (OMP) which is based on the batch support of PyTorch. Operations <span>*cat*</span>, <span>*squeeze*</span>, <span>*solve*</span> and <span>*one\_hot*</span> are equivalent to PyTorch methods with the same names while $\odot$ is the element-wise multiplication.

\[!h\]

<span>**Input:**</span> ${\boldsymbol{X}}, {\boldsymbol{M}}$ for a given forward pass, $k'$: the $k'$ nearest neighbours, $\rho$: a small reg. constant equal $1e\!-\!6$.

\[1\] $\boldsymbol{C}_n\!=\!({\boldsymbol{M}}\!-\!\boldsymbol{1}{\boldsymbol{x}}_n^T)({\boldsymbol{M}}\!-\!
\boldsymbol{1}{\boldsymbol{x}}_n^T)^T$ (so-called data cov.) ${\boldsymbol{\alpha}}_n\!=\!\text{solve}(\boldsymbol{1},
\boldsymbol{C}_n+\text{diag}(\boldsymbol{1})\!\cdot\!\rho)$ (matrix left div. by $\boldsymbol{1}$) ${\boldsymbol{\alpha}}_n\!\gets\!{\boldsymbol{\alpha}}_n/\sum_j\alpha_{jn}$ (normalization) ${\boldsymbol{\alpha}}\!=\![{\boldsymbol{\alpha}}_1,\cdots,{\boldsymbol{\alpha}}_{N'}]$ (concatenation of vectors ${\boldsymbol{\alpha}}_n$ into $\qquad\qquad\qquad\qquad\!$matrix ${\boldsymbol{\alpha}}$)

<span>**Output:**</span> ${\boldsymbol{\alpha}}$: locality-constrained linear codes

\[!h\]

<span>**Input:**</span> ${\boldsymbol{X}}, {\boldsymbol{M}}$ for a given forward pass, $k'$: the $k'$ nearest neighbours, $\sigma$: the bandwidth controlling the slope rate.

\[1\] $\boldsymbol{\varphi}\!=\!\text{idx\_nn}({\boldsymbol{x}}_n; {\boldsymbol{M}}, k')$ ($k'$ indexes of $k'$ nearest ${\boldsymbol{m}}_j$ $\qquad\qquad\qquad\qquad\qquad\!\!\!$of ${\boldsymbol{x}}_n$) ${\boldsymbol{\alpha}}'_n\!=\!{\boldsymbol{\alpha}}'({\boldsymbol{x}}_n;[{\boldsymbol{m}}_{\varphi_1},\cdots,{\boldsymbol{m}}_{
\varphi_{k'}}])$ (SA computed by $\qquad\qquad\qquad\qquad$<span>Eq</span> on local dictionary) ${\boldsymbol{\alpha}}_n\!=\!\pi({\boldsymbol{\alpha}}'_n; \boldsymbol{\varphi},k)$ (deploy coeff. of ${\boldsymbol{\alpha}}'_n$ to locations $\qquad\qquad\qquad\qquad\quad\;$$\boldsymbol{\varphi}$ of $k$ dim. zeroed vector) ${\boldsymbol{\alpha}}\!=\![{\boldsymbol{\alpha}}_1,\cdots,{\boldsymbol{\alpha}}_{N'}]$ (concat. vectors ${\boldsymbol{\alpha}}_n$ into matrix ${\boldsymbol{\alpha}}$)

<span>**Output:**</span> ${\boldsymbol{\alpha}}$: locality-constrained soft-assigned codes

Algorithm \[code:llc\] illustrates that Locality-constrained Linear Coding (LLC) is computed with operations which enjoy the batch support of PyTorch. Once more, operation <span>*solve*</span> is equivalent to the batch-wise linear solver of PyTorch.

Algorithm \[code:lcsa\] is a sketch implementation of Locality-constrained Soft Assignment (LCSA). Function <span>*idx\_nn*</span> firstly computes distances $\boldsymbol{D}$ of ${\boldsymbol{X}}$ to ${\boldsymbol{M}}$ as: $$\begin{aligned}
&\boldsymbol{D}\!=\![\|{\boldsymbol{x}}_1\|_2^2,\cdots,\|{\boldsymbol{x}}_{N'}\|_2^2]\!\cdot\!
\boldsymbol{1}^T+\boldsymbol{1}\!\cdot\![\|{\boldsymbol{m}}_1\|_2^2,\cdots,\|{\boldsymbol{m}}_{k}\|_2^2]^T
\nonumber\\
&\qquad-2{\boldsymbol{M}}^T\!{\boldsymbol{X}}.\end{aligned}$$ Subsequently, function <span>*ktop*</span> implemented in PyTorch selects $k'$ indexes of $k'$ smallest distances from $\boldsymbol{D}$ for each ${\boldsymbol{x}}_n$.

\[!h\]

<span>**Input:**</span> ${\boldsymbol{X}}, {\boldsymbol{M}}, {\boldsymbol{\alpha}}$ for a given forward pass, $\omega$: the learning rate.

\[1\] $\{m_{ij}\!\sim\!\mathcal{U}(-1,1)\}_{i\!=\!1,\cdots,d'\text{ and }
j\!=\!1,\cdots,k}$ ${\boldsymbol{M}}\!\gets\!\left[\frac{{\boldsymbol{m}}_j}{\|{\boldsymbol{m}}_j\|_1+1e\!-\!6}\right]_{j\!=\!1,
\cdots,k}$(concat. of ${\boldsymbol{m}}_j$ into matrix ${\boldsymbol{M}}$)$\!\!\!\!\!\!\!\!$ $\Delta{\boldsymbol{M}}\!=\!2({\boldsymbol{X}}\!-\!{\boldsymbol{M}}{\boldsymbol{\alpha}}){\boldsymbol{\alpha}}^T$ (gradient computation) ${\boldsymbol{M}}\!\gets\!{\boldsymbol{M}}\!-\!\frac{\omega}{(1+i)^{0.3}}\Delta{\boldsymbol{M}}$ (stochastic gradient descent)

<span>**Output:**</span> ${\boldsymbol{M}}$: dictionary atoms

Algorithm \[code:dl\] illustrates how we perform the Dictionary Learning (DL) step. While we could solve the Least Squares Problem in the closed-form, we noted that a simple one step of SGD per mini-batch is sufficient to obtain good results. Of course, where needed and/or requested by reviewers, we will refine our claims and proofs further.

\[t\]

<span>0.24</span> ![image](images/C10_omni256.jpg){width="\linewidth"}

<span>0.24</span> ![image](images/C10_omnilcsa256.jpg){width="\linewidth"}

<span>0.24</span> ![image](images/C10_omnilcsa512.jpg){width="\linewidth"}

<span>0.24</span> ![image](images/C10_omnilcsa1024.jpg){width="\linewidth"}

Generated Images {#sup:exim}
================

Figures \[fig:gen\_C10\], \[fig:gen\_C100\], \[fig:imagenet64\_proj\], \[fig:imagenet64\_omni\], and \[fig:imagenet128\] show examples of images generated with OminGAN OmniGAN+LCSA after training on CIFAR-10 ($32\!\times\!32$), CIFAR-100 ($32\!\times\!32$), ImageNet ($64\!\times\!64$) (we also include BigGAN & BigGAN+LCSA) and ImageNet ($128\!\times\!128$). Figure \[fig:flowers\_interpolation\] shows examples of images generated with MSG-StyleGAN MSG-StyleGAN+LCSA after training on Oxford-102 Flowers ($256\!\times\!256$). Figure \[fig:ffhq\] shows examples of images generated with StyleGAN2 StyleGAN2+LCSA for training on FFHQ ($256\!\times\!256$).

\[h\]

<span>0.24</span> ![image](images/C100_omni256.jpg){width="\linewidth"}

<span>0.24</span> ![image](images/C100_omnilcsa256.jpg){width="\linewidth"}

<span>0.24</span> ![image](images/C100_omnilcsa512.jpg){width="\linewidth"}

<span>0.24</span> ![image](images/C100_omnilcsa1024.jpg){width="\linewidth"}

\[h\]

<span>0.485</span> ![image](images/I64_proj_novq_cut.jpg){width="\linewidth"}

<span>0.485</span> ![image](images/I64_proj_lcsa_cut.jpg){width="\linewidth"}

\[h\]

<span>0.485</span> ![image](images/I64_omni_novq_cut.jpg){width="\linewidth"}

<span>0.485</span> ![image](images/I64_omni_lcsa_cut.jpg){width="\linewidth"}

\[h\]

<span>0.485</span> ![image](images/I128_omni_novq.jpg){width="\linewidth"}

<span>0.485</span> ![image](images/I128_omni_lcsa.jpg){width="\linewidth"}

\[h\]

<span>0.485</span> ![image](images/mean_flowers_msg.jpg){width="\linewidth"}

<span>0.485</span> ![image](images/mean_flowers_lcsa.jpg){width="\linewidth"}

\[h\]

<span>0.485</span> ![image](images/FFHQ_StyleGANV2.jpg){width="\linewidth"}

<span>0.485</span> ![image](images/FFHQ_LCSA.jpg){width="\linewidth"}

\[!ht\]

<span>0.24</span> ![image](images/C10_lim10_diffaug.jpg){width="\linewidth"}

<span>0.24</span> ![image](images/C10_lim10_lcsa_diffaug.jpg){width="\linewidth"}

<span>0.24</span> ![image](images/C10_lim10_ada.jpg){width="\linewidth"}

<span>0.24</span> ![image](images/C10_lim10_lcsa_ada.jpg){width="\linewidth"}

![image](images/variance/C10_real.jpg){width="\colwidth"}

![image](images/variance/LCSA_var0.jpg){width="\colwidth"}

![image](images/variance/LLC_var0.jpg){width="\colwidth"}

![image](images/variance/NNSC_var0.jpg){width="\colwidth"}

![image](images/variance/OMP_var0.jpg){width="\colwidth"}

![image](images/variance/LCSA_var1.jpg){width="\colwidth"}

![image](images/variance/LLC_var1.jpg){width="\colwidth"}

![image](images/variance/NNSC_var1.jpg){width="\colwidth"}

![image](images/variance/OMP_var1.jpg){width="\colwidth"}

![image](images/variance/LCSA_var2.jpg){width="\colwidth"}

![image](images/variance/LLC_var2.jpg){width="\colwidth"}

![image](images/variance/NNSC_var2.jpg){width="\colwidth"}

![image](images/variance/OMP_var2.jpg){width="\colwidth"}

![image](images/variance/LCSA_var3.jpg){width="\colwidth"}

![image](images/variance/LLC_var3.jpg){width="\colwidth"}

![image](images/variance/NNSC_var3.jpg){width="\colwidth"}

![image](images/variance/OMP_var3.jpg){width="\colwidth"}\

<span><span>0.093</span></span> ![image](images/variance/LCSA_cls_var0.jpg){width="\linewidth"}

<span><span>0.093</span></span> ![image](images/variance/LLC_cls_var0.jpg){width="\linewidth"}

<span><span>0.093</span></span> ![image](images/variance/NNSC_cls_var0.jpg){width="\linewidth"}

<span><span>0.093</span></span> ![image](images/variance/OMP_cls_var0.jpg){width="\linewidth"}

[^1]: The coding spaces considered in this paper are loosely termed manifolds. In most cases they are not manifolds in the strict mathematical sense, but rather topological spaces such as varieties, or simplicial complexes. The word will be used only in an informal sense.

[^2]: Equal contribution.$\qquad^\clubsuit$ Brain team (mvaldez@example.com).

[^3]: Our notations are explained in §\[sup:notations\] of the supplementary material.

{'source': 'arXiv_out/2112'}