Question: Represent the scientific article to find a short TLDR

Question: Despite recent progress in generative image modeling, successfully generating high-resolution, diverse samples from complex datasets such as ImageNet remains an elusive goal. To this end, we train Generative Adversarial Networks at the largest scale yet attempted, and study the instabilities specific to such scale. We find that applying orthogonal regularization to the generator renders it amenable to a simple "truncation trick", allowing fine control over the trade-off between sample fidelity and variety by reducing the variance of the Generator's input. Our modifications lead to models which set the new state of the art in class-conditional image synthesis. When trained on ImageNet at 128x128 resolution, our models (BigGANs) achieve an Inception Score (IS) of 166.3 and Frechet Inception Distance (FID) of 9.6, improving over the previous best IS of 52.52 and FID of 18.65. Figure 1: Class-conditional samples generated by our model. The state of generative image modeling has advanced dramatically in recent years, with Generative Adversarial Networks at the forefront of efforts to generate highfidelity, diverse images with models learned directly from data. GAN training is dynamic, and sensitive to nearly every aspect of its setup (from optimization parameters to model architecture), but a torrent of research has yielded empirical and theoretical insights enabling stable training in a variety of settings. Despite this progress, the current state of the art in conditional ImageNet modeling achieves an Inception Score of 52.5, compared to 233 for real data. In this work, we set out to close the gap in fidelity and variety between images generated by GANs and real-world images from the ImageNet dataset. We make the following three contributions towards this goal:• We demonstrate that GANs benefit dramatically from scaling, and train models with two to four times as many parameters and eight times the batch size compared to prior art. We introduce two simple, general architectural changes that improve scalability, and modify a regularization scheme to improve conditioning, demonstrably boosting performance.• As a side effect of our modifications, our models become amenable to the "truncation trick," a simple sampling technique that allows explicit, fine-grained control of the tradeoff between sample variety and fidelity.• We discover instabilities specific to large scale GANs, and characterize them empirically. Leveraging insights from this analysis, we demonstrate that a combination of novel and existing techniques can reduce these instabilities, but complete training stability can only be achieved at a dramatic cost to performance. Our modifications substantially improve class-conditional GANs. When trained on ImageNet at 128×128 resolution, our models (BigGANs) improve the state-of-the-art Inception Score (IS) and Fréchet Inception Distance (FID) from 52.52 and 18.65 to 166.5 and 7.4 respectively. We also successfully train BigGANs on ImageNet at 256×256 and 512×512 resolution, and achieve IS and FID of 232.5 and 8.1 at 256×256 and IS and FID of 241.5 and 11.5 at 512×512. Finally, we train our models on an even larger dataset -JFT-300M -and demonstrate that our design choices transfer well from ImageNet. Code and weights for our pretrained generators are publicly available 1. A Generative Adversarial Network (GAN) involves Generator (G) and Discriminator (D) networks whose purpose, respectively, is to map random noise to samples and discriminate real and generated samples. Formally, the GAN objective, in its original form involves finding a Nash equilibrium to the following two player min-max problem: DISPLAYFORM0 where z ∈ R dz is a latent variable drawn from distribution p(z) such as N (0, I) or U[−1, 1]. When applied to images, G and D are usually convolutional neural networks . Without auxiliary stabilization techniques, this training procedure is notoriously brittle, requiring finely-tuned hyperparameters and architectural choices to work at all. Much recent research has accordingly focused on modifications to the vanilla GAN procedure to impart stability, drawing on a growing body of empirical and theoretical insights (; Sønderby et al., 2017;). One line of work is focused on changing the objective function BID1;; BID3 ) to encourage convergence. Another line is focused on constraining D through gradient penalties (; ;) or normalization , both to counteract the use of unbounded loss functions and ensure D provides gradients everywhere to G.Of particular relevance to our work is Spectral Normalization , which enforces Lipschitz continuity on D by normalizing its parameters with running estimates of their first singular values, inducing backwards dynamics that adaptively regularize the top singular direction. analyze the condition number of the Jacobian of G and find that performance is dependent on G's conditioning. find that employing Spectral Normalization in G improves stability, allowing for fewer D steps per iteration. We extend on these analyses to gain further insight into the pathology of GAN training. Other works focus on the choice of architecture, such as SA- GAN which adds the self-attention block from to improve the ability of both G and D to model global structure. ProGAN trains high-resolution GANs in the single-class setting by training a single model across a sequence of increasing resolutions. In conditional GANs class information can be fed into the model in various ways. In it is provided to G by concatenating a 1-hot class vector to the noise vector, and the objective is modified to encourage conditional samples to maximize the corresponding class probability predicted by an auxiliary classifier. de for ablations of our proposed modifications. Batch is batch size, Param is total number of parameters, Ch. is the channel multiplier representing the number of units in each layer, Shared is using shared embeddings, Skip-z is using skip connections from the latent to multiple layers, Ortho. is Orthogonal Regularization, and Itr indicates if the setting is stable to 10 6 iterations, or it collapses at the given iteration. Other than rows 1-4, are computed across 8 random initializations. modify the way class conditioning is passed to G by supplying it with classconditional gains and biases in BatchNorm layers. , D is conditioned by using the cosine similarity between its features and a set of learned class embeddings as additional evidence for distinguishing real and generated samples, effectively encouraging generation of samples whose features match a learned class prototype. Objectively evaluating implicit generative models is difficult . A variety of works have proposed heuristics for measuring the sample quality of models without tractable likelihoods (; ; BID4). Of these, the Inception Score and Fréchet Inception Distance have become popular despite their notable flaws BID2. We employ them as approximate measures of sample quality, and to enable comparison against previous work. In this section, we explore methods for scaling up GAN training to reap the performance benefits of larger models and larger batches. As a baseline, we employ the SA-GAN architecture of , which uses the hinge loss GAN objective. We provide class information to G with class-conditional BatchNorm (; de) and to D with projection . The optimization settings follow (notably employing Spectral Norm in G) with the modification that we halve the learning rates and take two D steps per G step. For evaluation, we employ moving averages of G's weights following;; , with a decay of 0.9999. We use Orthogonal Initialization , whereas previous works used N (0, 0.02I) or Xavier initialization . Each model is trained on 128 to 512 cores of a Google TPUv3 Pod , and computes BatchNorm statistics in G across all devices, rather than per-device as is typical. We find progressive growing unnecessary even for our 512×512 models. Additional details are in Appendix C.We begin by increasing the batch size for the baseline model, and immediately find tremendous benefits in doing so. Rows 1-4 of TAB1 show that simply increasing the batch size by a factor of 8 improves the state-of-the-art IS by 46%. We conjecture that this is a of each batch covering more modes, providing better gradients for both networks. One notable side effect of this scaling is that our models reach better final performance in fewer iterations, but become unstable and undergo complete training collapse. We discuss the causes and ramifications of this in Section 4. For these experiments, we report scores from checkpoints saved just before collapse. We then increase the width (number of channels) in each layer by 50%, approximately doubling the number of parameters in both models. This leads to a further IS improvement of 21%, which we posit is due to the increased capacity of the model relative to the complexity of the dataset. Doubling the depth did not initially lead to improvement -we addressed this later in the BigGAN-deep model, which uses a different residual block structure. We note that class embeddings c used for the conditional BatchNorm layers in G contain a large number of weights. Instead of having a separate layer for each embedding , we opt to use a shared embedding, which is linearly projected to each layer's gains and biases . This reduces computation and memory costs, and improves training speed (in number of iterations required to reach a given performance) by 37%. Next, we add direct skip connections (skip-z) from the noise vector z to multiple layers of G rather than just the initial layer. The intuition behind this design is to allow G to use the latent space to directly influence features at different resolutions and levels of hierarchy. In BigGAN, this is accomplished by splitting z into one chunk per resolution, and concatenating each chunk to the conditional vector c which gets projected to the BatchNorm gains and biases. In BigGAN-deep, we use an even simpler design, concatenating the entire z with the conditional vector without splitting it into chunks. Previous works have considered variants of this concept; our implementation is a minor modification of this design. Skip-z provides a modest performance improvement of around 4%, and improves training speed by a further 18%. Unlike models which need to backpropagate through their latents, GANs can employ an arbitrary prior p(z), yet the vast majority of previous works have chosen to draw z from either N (0, I) or U[−1, 1]. We question the optimality of this choice and explore alternatives in Appendix E.Remarkably, our best come from using a different latent distribution for sampling than was used in training. Taking a model trained with z ∼ N (0, I) and sampling z from a truncated normal (where values which fall outside a range are resampled to fall inside that range) immediately provides a boost to IS and FID. We call this the Truncation Trick: truncating a z vector by resampling the values with magnitude above a chosen threshold leads to improvement in individual sample quality at the cost of reduction in overall sample variety. FIG0 (a) demonstrates this: as the threshold is reduced, and elements of z are truncated towards zero (the mode of the latent distribution), individual samples approach the mode of G's output distribution. Related observations about this trade-off were made in .This technique allows fine-grained, post-hoc selection of the trade-off between sample quality and variety for a given G. Notably, we can compute FID and IS for a range of thresholds, obtaining the variety-fidelity curve reminiscent of the precision-recall curve (FIG17). As IS does not penalize lack of variety in class-conditional models, reducing the truncation threshold leads to a direct increase in IS (analogous to precision). FID penalizes lack of variety (analogous to recall) but also rewards precision, so we initially see a moderate improvement in FID, but as truncation approaches zero and variety diminishes, the FID sharply drops. The distribution shift caused by sampling with different latents than those seen in training is problematic for many models. Some of our larger models are not amenable to truncation, producing saturation artifacts FIG0 ) when fed truncated noise. To counteract this, we seek to enforce amenability to truncation by conditioning G to be smooth, so that the full space of z will map to good output samples. For this, we turn to Orthogonal Regularization BID5, which directly enforces the orthogonality condition: DISPLAYFORM0 where W is a weight matrix and β a hyperparameter. This regularization is known to often be too limiting , so we explore several variants designed to relax the constraint while still imparting the desired smoothness to our models. The version we find to work best removes the diagonal terms from the regularization, and aims to minimize the pairwise cosine similarity between filters but does not constrain their norm: DISPLAYFORM1 where 1 denotes a matrix with all elements set to 1. We sweep β values and select 10 −4, finding this small added penalty sufficient to improve the likelihood that our models will be amenable to truncation. Across runs in TAB1, we observe that without Orthogonal Regularization, only 16% of models are amenable to truncation, compared to 60% when trained with Orthogonal Regularization. We find that current GAN techniques are sufficient to enable scaling to large models and distributed, large-batch training. We find that we can dramatically improve the state of the art and train models up to 512×512 resolution without need for explicit multiscale methods like. Despite these improvements, our models undergo training collapse, necessitating early stopping in practice. In the next two sections we investigate why settings which were stable in previous works become unstable when applied at scale. Much previous work has investigated GAN stability from a variety of analytical angles and on toy problems, but the instabilities we observe occur for settings which are stable at small scale, necessitating direct analysis at large scale. We monitor a range of weight, gradient, and loss statistics during training, in search of a metric which might presage the onset of training collapse, similar to . We found the top three singular values σ 0, σ 1, σ 2 of each weight matrix to be the most informative. They can be efficiently computed using the Alrnoldi iteration method (Golub & der), which extends the power iteration method, used in , to estimation of additional singular vectors and values. A clear pattern emerges, as can be seen in FIG1 (a) and Appendix F: most G layers have well-behaved spectral norms, but some layers (typically the first layer in G, which is over-complete and not convolutional) are ill-behaved, with spectral norms that grow throughout training and explode at collapse. To ascertain if this pathology is a cause of collapse or merely a symptom, we study the effects of imposing additional conditioning on G to explicitly counteract spectral explosion. First, we directly regularize the top singular values σ 0 of each weight, either towards a fixed value σ reg or towards some ratio r of the second singular value, r · sg(σ 1) (with sg the stop-gradient operation to prevent the regularization from increasing σ 1). Alternatively, we employ a partial singular value decomposition to instead clamp σ 0. Given a weight W, its first singular vectors u 0 and v 0, and σ clamp the value to which the σ 0 will be clamped, our weights become: DISPLAYFORM0 where σ clamp is set to either σ reg or r · sg(σ 1). We observe that both with and without Spectral Normalization these techniques have the effect of preventing the gradual increase and explosion of either σ 0 or σ0 σ1, but even though in some cases they mildly improve performance, no combination prevents training collapse. This evidence suggests that while conditioning G might improve stability, it is insufficient to ensure stability. We accordingly turn our attention to D. As with G, we analyze the spectra of D's weights to gain insight into its behavior, then seek to stabilize training by imposing additional constraints. FIG1 (b) displays a typical plot of σ 0 for D (with further plots in Appendix F). Unlike G, we see that the spectra are noisy, σ0 σ1 is well-behaved, and the singular values grow throughout training but only jump at collapse, instead of exploding. The spikes in D's spectra might suggest that it periodically receives very large gradients, but we observe that the Frobenius norms are smooth (Appendix F), suggesting that this effect is primarily concentrated on the top few singular directions. We posit that this noise is a of optimization through the adversarial training process, where G periodically produces batches which strongly perturb D. If this spectral noise is causally related to instability, a natural counter is to employ gradient penalties, which explicitly regularize changes in D's Jacobian. We explore the R 1 zero-centered gradient penalty from Mescheder et al. FORMULA0: DISPLAYFORM0 With the default suggested γ strength of 10, training becomes stable and improves the smoothness and boundedness of spectra in both G and D, but performance severely degrades, ing in a 45% reduction in IS. Reducing the penalty partially alleviates this degradation, but in increasingly ill-behaved spectra; even with the penalty strength reduced to 1 (the lowest strength for which sudden collapse does not occur) the IS is reduced by 20%. Repeating this experiment with various strengths of Orthogonal Regularization, DropOut , and L2 (See Appendix I for details), reveals similar behaviors for these regularization strategies: with high enough penalties on D, training stability can be achieved, but at a substantial cost to performance. We also observe that D's loss approaches zero during training, but undergoes a sharp upward jump at collapse (Appendix F). One possible explanation for this behavior is that D is overfitting to the training set, memorizing training examples rather than learning some meaningful boundary between real and generated images. As a simple test for D's memorization (related to Gulrajani et al. FORMULA0), we evaluate uncollapsed discriminators on the ImageNet training and validation sets, and measure what percentage of samples are classified as real or generated. While the training accuracy is consistently above 98%, the validation accuracy falls in the range of 50-55%, no better than random guessing (regardless of regularization strategy). This confirms that D is indeed memorizing the training set; we deem this in line with D's role, which is not explicitly to generalize, but to distill the training data and provide a useful learning signal for G. Additional experiments and discussion are provided in Appendix G. We find that stability does not come solely from G or D, but from their interaction through the adversarial training process. While the symptoms of their poor conditioning can be used to track and.0/275 BigGAN-deep 128 5.7 ±.3/124.5 ± 2 6.3 ±.3/148.1 ± 4 7.4 ±.6/166.5 ± 1 25 ± 2/253 ± 11 BigGAN-deep 256 6.9 ±.2/171.4 ± 2 7.0 ±.1/202.6 ± 2 8.1 ±.1/232.5 ± 2 27 ± 8/317 ± 6 BigGAN-deep 512 7.5/152.8 7.7/181.4 11.5/241.5 39.7/298Table 2: Evaluation of models at different resolutions. We report scores without truncation (Column 3), scores at the best FID (Column 4), scores at the IS of validation data (Column 5), and scores at the max IS (Column 6). Standard deviations are computed over at least three random initializations.identify instability, ensuring reasonable conditioning proves necessary for training but insufficient to prevent eventual training collapse. It is possible to enforce stability by strongly constraining D, but doing so incurs a dramatic cost in performance. With current techniques, better final performance can be achieved by relaxing this conditioning and allowing collapse to occur at the later stages of training, by which time a model is sufficiently trained to achieve good .5 EXPERIMENTS We evaluate our models on ImageNet ILSVRC 2012 (at 128×128, 256×256, and 512×512 resolutions, employing the settings from TAB1, row 8. The samples generated by our models are presented in FIG2, with additional samples in Appendix A, and online 2 . We report IS and FID in TAB3 . As our models are able to trade sample variety for quality, it is unclear how best to compare against prior art; we accordingly report values at three settings, with complete curves in Appendix D. First, we report the FID/IS values at the truncation setting which attains the best FID. Second, we report the FID at the truncation setting for which our model's IS is the same as that attained by the real validation data, reasoning that this is a passable measure of maximum sample variety achieved while still achieving a good level of "objectness." Third, we report FID at the maximum IS achieved by each model, to demonstrate how much variety must be traded off to maximize quality. In all three cases, our models outperform the previous state-of-the-art IS and FID scores achieved by and .In addition to the BigGAN model introduced in the first version of the paper and used in the majority of experiments (unless otherwise stated), we also present a 4x deeper model (BigGAN-deep) which uses a different configuration of residual blocks. As can be seen from Table 3: BigGAN on JFT-300M at 256×256 resolution. The FID and IS columns report these scores given by the JFT-300M-trained Inception v2 classifier with noise distributed as z ∼ N (0, I) (non-truncated). The (min FID) / IS and FID / (max IS) columns report scores at the best FID and IS from a sweep across truncated noise distributions ranging from σ = 0 to σ = 2. Images from the JFT-300M validation set have an IS of 50.88 and FID of 1.94.extend to other architectures, and that increased depth leads to improvement in sample quality. Both BigGAN and BigGAN-deep architectures are described in Appendix B.Our observation that D overfits to the training set, coupled with our model's sample quality, raises the obvious question of whether or not G simply memorizes training points. To test this, we perform class-wise nearest neighbors analysis in pixel space and the feature space of pre-trained classifier networks (Appendix A). In addition, we present both interpolations between samples and class-wise interpolations (where z is held constant) in Figures 8 and 9. Our model convincingly interpolates between disparate samples, and the nearest neighbors for its samples are visually distinct, suggesting that our model does not simply memorize training data. We note that some failure modes of our partially-trained models are distinct from those previously observed. Most previous failures involve local artifacts , images consisting of texture blobs instead of objects , or the canonical mode collapse. We observe class leakage, where images from one class contain properties of another, as exemplified by FIG2. We also find that many classes on ImageNet are more difficult than others for our model; our model is more successful at generating dogs (which make up a large portion of the dataset, and are mostly distinguished by their texture) than crowds (which comprise a small portion of the dataset and have more large-scale structure). Further discussion is available in Appendix A. To confirm that our design choices are effective for even larger and more complex and diverse datasets, we also present of our system on a subset of JFT-300M . The full JFT-300M dataset contains 300M real-world images labeled with 18K categories. Since the category distribution is heavily long-tailed, we subsample the dataset to keep only images with the 8.5K most common labels. The ing dataset contains 292M images -two orders of magnitude larger than ImageNet. For images with multiple labels, we sample a single label randomly and independently whenever an image is sampled. To compute IS and FID for the GANs trained on this dataset, we use an Inception v2 classifier trained on this dataset. Quantitative are presented in Table 3. All models are trained with batch size 2048. We compare an ablated version of our model -comparable to SA-GAN but with the larger batch size -against a "full" BigGAN model that makes uses of all of the techniques applied to obtain the best on ImageNet (shared embedding, skip-z, and orthogonal regularization). Our show that these techniques substantially improve performance even in the setting of this much larger dataset at the same model capacity (64 base channels). We further show that for a dataset of this scale, we see significant additional improvements from expanding the capacity of our models to 128 base channels, while for ImageNet GANs that additional capacity was not beneficial. In FIG6 (Appendix D), we present truncation plots for models trained on this dataset. Unlike for ImageNet, where truncation limits of σ ≈ 0 tend to produce the highest fidelity scores, IS is typically maximized for our JFT-300M models when the truncation value σ ranges from 0.5 to 1. We suspect that this is at least partially due to the intra-class variability of JFT-300M labels, as well as the relative complexity of the image distribution, which includes images with multiple objects at a variety of scales. Interestingly, unlike models trained on ImageNet, where training tends to collapse without heavy regularization (Section 4), the models trained on JFT-300M remain stable over many hundreds of thousands of iterations. This suggests that moving beyond ImageNet to larger datasets may partially alleviate GAN stability issues. The improvement over the baseline GAN model that we achieve on this dataset without changes to the underlying models or training and regularization techniques (beyond expanded capacity) demonstrates that our findings extend from ImageNet to datasets with scale and complexity thus far unprecedented for generative models of images. We have demonstrated that Generative Adversarial Networks trained to model natural images of multiple categories highly benefit from scaling up, both in terms of fidelity and variety of the generated samples. As a , our models set a new level of performance among ImageNet GAN models, improving on the state of the art by a large margin. We have also presented an analysis of the training behavior of large scale GANs, characterized their stability in terms of the singular values of their weights, and discussed the interplay between stability and performance. In the BigGAN model FIG3 ), we use the ResNet GAN architecture of , which is identical to that used by , but with the channel pattern in D modified so that the number of filters in the first convolutional layer of each block is equal to the number of output filters (rather than the number of input filters, as in Miyato et al. FORMULA0 ; Gulrajani et al. FORMULA0). We use a single shared class embedding in G, and skip connections for the latent vector z (skip-z). In particular, we employ hierarchical latent spaces, so that the latent vector z is split along its channel dimension into chunks of equal size (20-D in our case), and each chunk is concatenated to the shared class embedding and passed to a corresponding residual block as a conditioning vector. The conditioning of each block is linearly projected to produce per-sample gains and biases for the BatchNorm layers of the block. The bias projections are zero-centered, while the gain projections are centered at 1. Since the number of residual blocks depends on the image resolution, the full dimensionality of z is 120 for 128 × 128, 140 for 256 × 256, and 160 for 512 × 512 images. The BigGAN-deep model (Figure 16) differs from BigGAN in several aspects. It uses a simpler variant of skip-z conditioning: instead of first splitting z into chunks, we concatenate the entire z with the class embedding, and pass the ing vector to each residual block through skip connections. BigGAN-deep is based on residual blocks with bottlenecks , which incorporate two additional 1 × 1 convolutions: the first reduces the number of channels by a factor of 4 before the more expensive 3 × 3 convolutions; the second produces the required number of output channels. While BigGAN relies on 1 × 1 convolutions in the skip connections whenever the number of channels needs to change, in BigGAN-deep we use a different strategy aimed at preserving identity throughout the skip connections. In G, where the number of channels needs to be reduced, we simply retain the first group of channels and drop the rest to produce the required number of channels. In D, where the number of channels should be increased, we pass the input channels unperturbed, and concatenate them with the remaining channels produced by a 1 × 1 convolution. As far as the network configuration is concerned, the discriminator is an exact reflection of the generator. There are two blocks at each resolution (BigGAN uses one), and as a BigGAN-deep is four times deeper than BigGAN. Despite their increased depth, the BigGAN-deep models have significantly fewer parameters mainly due to the bottleneck structure of their residual blocks. For example, the 128 × 128 BigGAN-deep G and D have 50.4M and 34.6M parameters respectively, while the corresponding original BigGAN models have 70.4M and 88.0M parameters. All BigGAN-deep models use attention at 64 × 64 resolution, channel width multiplier ch = 128, and z ∈ R 128. ReLU, Global sum pooling DISPLAYFORM0 ReLU, Global sum pooling in G. For BigGAN-deep, we use the learning rate of 2 · 10 −4 in D and 5 · 10 −5 in G for 128 × 128 models, and 2.5 · 10 −5 in both D and G for 256 × 256 and 512 × 512 models. We experimented with the number of D steps per G step (varying it from 1 to 6) and found that two D steps per G step gave the best . We use an exponential moving average of the weights of G at sampling time, with a decay rate set to 0.9999. We employ cross-replica BatchNorm in G, where batch statistics are aggregated across all devices, rather than a single device as in standard implementations. Spectral Normalization is used in both G and D, following SA-GAN . We train on a Google TPU v3 Pod, with the number of cores proportional to the resolution: 128 for 128×128, 256 for 256×256, and 512 for 512×512. Training takes between 24 and 48 hours for most models. We increase from the default 10 −8 to 10 −4 in BatchNorm and Spectral Norm to mollify low-precision numerical issues. We preprocess data by cropping along the long edge and rescaling to a given resolution with area resampling. The default behavior with batch normalized classifier networks is to use a running average of the activation moments at test time. Previous works have instead used batch statistics when sampling images. While this is not technically an invalid way to sample, it means that are dependent on the test batch size (and how many devices it is split across), and further complicates reproducibility. We find that this detail is extremely important, with changes in test batch size producing drastic changes in performance. This is further exacerbated when one uses exponential moving averages of G's weights for sampling, as the BatchNorm running averages are computed with non-averaged weights and are poor estimates of the activation statistics for the averaged weights. To counteract both these issues, we employ "standing statistics," where we compute activation statistics at sampling time by running the G through multiple forward passes (typically 100) each with different batches of random noise, and storing means and variances aggregated across all forward passes. Analogous to using running statistics, this in G's outputs becoming invariant to batch size and the number of devices, even when producing a single sample. We run our networks on CIFAR-10 using the settings from TAB1, row 8, and achieve an IS of 9.22 and an FID of 14.73 without truncation. We compute the IS for both the training and validation sets of ImageNet. At 128×128 the training data has an IS of 233, and the validation data has an IS of 166. At 256×256 the training data has an IS of 377, and the validation data has an IS of 234. At 512×512 the training data has an IS of 348, and the validation data has an IS of 241. The discrepancy between training and validation scores is due to the Inception classifier having been trained on the training data, ing in high-confidence outputs that are preferred by the Inception Score. FIG6: JFT-300M IS vs. FID at 256×256. We show truncation values from σ = 0 to σ = 2 (top) and from σ = 0.5 to σ = 1.5 (bottom). Each curve corresponds to a row in Table 3. The curve labeled with baseline corresponds to the first row (with orthogonal regularization and other techniques disabled), while the rest correspond to rows 2-4 -the same architecture at different capacities (Ch). While most previous work has employed N (0, I) or U[−1, 1] as the prior for z (the noise input to G), we are free to choose any latent distribution from which we can sample. We explore the choice of latents by considering an array of possible designs, described below. For each latent, we provide the intuition behind its design and briefly describe how it performs when used as a drop-in replacement for z ∼ N (0, I) in an SA-GAN baseline. As the Truncation Trick proved more beneficial than switching to any of these latents, we do not perform a full ablation study, and employ z ∼ N (0, I) for our main to take full advantage of truncation. The two latents which we find to work best without truncation are Bernoulli {0, 1} and Censored Normal max (N (0, I), 0), both of which improve speed of training and lightly improve final performance, but are less amenable to truncation. We also ablate the choice of latent space dimensonality (which by default is z ∈ R 128), finding that we are able to successfully train with latent dimensions as low as z ∈ R 8, and that with z ∈ R 32 we see a minimal drop in performance. While this is substantially smaller than many previous works, direct comparison to single-class networks (such as those in , which employ a z ∈ R 512 latent space on a highly constrained dataset with 30,000 images) is improper, as our networks have additional class information provided as input. • N (0, I). A standard choice of the latent space which we use in the main experiments.• U[−1, 1]. Another standard choice; we find that it performs similarly to N (0, I).• Bernoulli {0, 1}. A discrete latent might reflect our prior that underlying factors of variation in natural images are not continuous, but discrete (one feature is present, another is not). This latent outperforms N (0, I) (in terms of IS) by 8% and requires 60% fewer iterations.• max (N (0, I), 0), also called Censored Normal. This latent is designed to introduce sparsity in the latent space (reflecting our prior that certain latent features are sometimes present and sometimes not), but also allow those latents to vary continuously, expressing different degrees of intensity for latents which are active. This latent outperforms N (0, I) (in terms of IS) by 15-20% and tends to require fewer iterations.• Bernoulli {−1, 1}. This latent is designed to be discrete, but not sparse (as the network can learn to activate in response to negative inputs). This latent performs near-identically to N (0, I).• Independent Categorical in {−1, 0, 1}, with equal probability. This distribution is chosen to be discrete and have sparsity, but also to allow latents to take on both positive and negative values. This latent performs near-identically to N (0, I).• N (0, I) multiplied by Bernoulli {0, 1}. This distribution is chosen to have continuous latent factors which are also sparse (with a peak at zero), similar to Censored Normal but not constrained to be positive. This latent performs near-identically to N (0, I).• Concatenating N (0, I) and Bernoulli {0, 1}, each taking half of the latent dimensions. This is inspired by, and is chosen to allow some factors of variation to be discrete, while others are continuous. This latent outperforms N (0, I) by around 5%.• Variance annealing: we sample from N (0, σI), where σ is allowed to vary over training. We compared a variety of piecewise schedules and found that starting with σ = 2 and annealing towards σ = 1 over the course of training mildly improved performance. The space of possible variance schedules is large, and we did not explore it in depth -we suspect that a more principled or better-tuned schedule could more strongly impact performance.• Per-sample variable variance: N (0, σ i I), where σ i ∼ U[σ l, σ h] independently for each sample i in a batch, and (σ l, σ h) are hyperparameters. This distribution was chosen to try and improve amenability to the Truncation Trick by feeding the network noise samples with non-constant variance. This did not appear to affect performance, but we did not explore it in depth. One might also consider scheduling (σ l, σ h), similar to variance annealing. Figure 21: G training statistics with σ 0 in G regularized towards 1. Collapse occurs after 125000 iterations.(a) σ0 (b) DISPLAYFORM0 Figure 22: D training statistics with σ 0 in G regularized towards 1. Collapse occurs after 125000 iterations. DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 DISPLAYFORM4 In this section, we present and discuss additional investigations into the stability of our models, expanding upon the discussion in Section 4. The symptoms of collapse are sharp and sudden, with sample quality dropping from its peak to its lowest value over the course of a few hundred iterations. We can detect this collapse when the singular values in G explode, but while the (unnormalized) singular values grow throughout training, there is no consistent threshold at which collapse occurs. This raises the question of whether it is possible to prevent or delay collapse by taking a model checkpoint several thousand iterations before collapse, and continuing training with some hyperparameters modified (e.g., the learning rate).We conducted a range of intervention experiments wherein we took checkpoints of a collapsed model ten or twenty thousand iterations before collapse, changed some aspect of the training setup, then observed whether collapse occurred, when it occurred relative to the original collapse, and the final performance attained at collapse. We found that increasing the learning rates (relative to their initial values) in either G or D, or both G and D, led to immediate collapse. This occurred even when doubling the learning rates from 2 · 10 −4in D and 5 · 10 −5 in G, to 4 · 10 −4 in D and 1 · 10 −4 in G, a setting which is not normally unstable when used as the initial learning rates. We also tried changing the momentum terms (Adam's β 1 and β 2), or resetting the momentum vectors to zero, but this tended to either make no difference or, when increasing the momentum, cause immediate collapse. FIG0 ), the noise spikes resemble an impulse response: at each spike, the spectra jump upwards, then slowly decrease, with some oscillation. One possible explanation is that this behavior is a consequence of D memorizing the training data, as suggested by experiments in Section 4.2. As it approaches perfect memorization, it receives less and less signal from real data, as both the original GAN loss and the hinge loss provide zero gradients when D outputs a confident and correct prediction for a given example. If the gradient signal from real data attenuates to zero, this can in D eventually becoming biased due to exclusively received gradients that encourage its outputs to be negative. If this bias passes a certain threshold, D will eventually misclassify a large number of real examples and receive a large gradient encouraging positive outputs, ing in the observed impulse responses. This argument suggests several fixes. First, one might consider an unbounded loss (such as the Wasserstein loss BID1) which would not suffer this gradient attentuation. We found that even with gradient penalties and brief re-tuning of optimizer hyperparameters, our models did not stably train for more than a few thousand iterations with this loss. We instead explored changing the margin of the hinge loss as a partial compromise: for a given model and minibatch of data, increasing the margin will in more examples falling within the margin, and thus contributing to the loss. We explored a range of novel and existing techniques which ended up degrading or otherwise not affecting performance in our setting. We report them here; our evaluations for this section are not as thorough as those for the main architectural choices. Our intention in reporting these is to save time for future work, and to give a more complete picture of our attempts to improve performance or stability. We note, however, that these must be understood to be specific to the particular setup we used. A pitfall of reporting negative is that one might report that a particular technique doesn't work, when the reality is that this technique did not have the desired effect when applied in a particular way to a particular problem. Drawing overly general might close off potentially fruitful avenues of research.• We found that doubling the depth (by inserting an additional Residual block after every upor down-sampling block) hampered performance.• We experimented with sharing class embeddings between both G and D (as opposed to just within G). This is accomplished by replacing D's class embedding with a projection from G's embeddings, as is done in G's BatchNorm layers. In our initial experiments this seemed to help and accelerate training, but we found this trick scaled poorly and was sensitive to optimization hyperparameters, particularly the choice of number of D steps per G step.• We tried replacing BatchNorm in G with WeightNorm , but this crippled training. We also tried removing BatchNorm and only having Spectral Normalization, but this also crippled training.• We tried adding BatchNorm to D (both class-conditional and unconditional) in addition to Spectral Normalization, but this crippled training.• We tried varying the choice of location of the attention block in G and D (and inserting multiple attention blocks at different resolutions) but found that at 128×128 there was no noticeable benefit to doing so, and compute and memory costs increased substantially. We found a benefit to moving the attention block up one stage when moving to 256×256, which is in line with our expectations given the increased resolution.• We tried using filter sizes of 5 or 7 instead of 3 in either G or D or both. We found that having a filter size of 5 in G only provided a small improvement over the baseline but came at an unjustifiable compute cost. All other settings degraded performance.• We tried varying the dilation for convolutional filters in both G and D at 128×128, but found that even a small amount of dilation in either network degraded performance.• We tried bilinear upsampling in G in place of nearest-neighbors upsampling, but this degraded performance.• In some of our models, we observed class-conditional mode collapse, where the model would only output one or two samples for a subset of classes but was still able to generate samples for all other classes. We noticed that the collapsed classes had embedings which had become very large relative to the other embeddings, and attempted to ameliorate this issue by applying weight decay to the shared embedding only. We found that small amounts of weight decay (10 −6) instead degraded performance, and that only even smaller values (10 −8) did not degrade performance, but these values were also too small to prevent the class vectors from exploding. Higher-resolution models appear to be more resilient to this problem, and none of our final models appear to suffer from this type of collapse.• We experimented with using MLPs instead of linear projections from G's class embeddings to its BatchNorm gains and biases, but did not find any benefit to doing so. We also experimented with Spectrally Normalizing these MLPs, and with providing these (and the linear projections) with a bias at their output, but did not notice any benefit.• We tried gradient norm clipping (both the global variant typically used in recurrent networks, and a local version where the clipping value is determined on a per-parameter basis) but found this did not alleviate instability.

Represent the TLDR for finding the corresponding scientific article

represent the tldr for finding the corresponding scientific article

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.


Question: Represent the article for finding a single-sentence TLDR

Question: In this work, we present a novel upper bound of target error to address the problem for unsupervised domain adaptation. Recent studies reveal that a deep neural network can learn transferable features which generalize well to novel tasks. Furthermore, provide an upper bound for target error when transferring the knowledge, which can be summarized as minimizing the source error and distance between marginal distributions simultaneously. However, common methods based on the theory usually ignore the joint error such that samples from different classes might be mixed together when matching marginal distribution. And in such case, no matter how we minimize the marginal discrepancy, the target error is not bounded due to an increasing joint error. To address this problem, we propose a general upper bound taking joint error into account, such that the undesirable case can be properly penalized. In addition, we utilize constrained hypothesis space to further formalize a tighter bound as well as a novel cross margin discrepancy to measure the dissimilarity between hypotheses which alleviates instability during adversarial learning. Extensive empirical evidence shows that our proposal outperforms related approaches in image classification error rates on standard domain adaptation benchmarks. The advent of deep convolutional neural networks brings visual learning into a new era. However, the performance heavily relies on the abundance of data annotated with ground-truth labels. Since traditional machine learning assumes a model is trained and verified in a fixed distribution (single domain), where generalization performance is guaranteed by VC theory (N.), thus it cannot always be applied to real-world problem directly. Take image classification task as an example, a number of factors, such as the change of light, noise, angle in which the image is pictured, and different types of sensors, can lead to a domain-shift thus harm the performance when predicting on test data. Therefore, in many practical cases, we wish that a model trained in one or more source domains is also applicable to another domain. As a solution, domain adaptation (DA) aims to transfer the knowledge learned from a source distribution, which is typically fully labeled into a different (but related) target distribution. This work focus on the most challenging case, i.e, unsupervised domain adaptation (UDA), where no target label is available. suggests that target error can be minimized by bounding the error of a model on the source data, the discrepancy between distributions of the two domains, and a small optimal joint error. Owing to the strong representation power of deep neural nets, many researchers focus on learning domain-invariant features such that the discrepancy of two feature spaces can be minimized. For aligning feature distributions across domains, mainly two strategies have been substantially explored. The first one is bridging the distributions by matching all their statistics . The second strategy is using adversarial learning to build a minimax game between domain discriminator and feature extractor, where a domain discriminator is trained to distinguish the source from the target while the feature extractor is learned to confuse it simultaneously (; ;). In spite of the remarkable empirical accomplished by feature distribution matching schemes, they still suffer from a major limitation: the joint distributions of feature spaces and categories are not well aligned across data domains. As is reported in , such methods fail to generalize in certain closely related source/target pairs, e.g., digit classification adaptation from MNIST to SVHN. One potential reason is when matching marginal distributions of source and target domains, samples from different classes can be mixed together, where the joint error becomes nonnegligible since no hypothesis can classify source and target at the same time. This work aims to address the above problem by incorporating joint error to formalize an optimizable upper bound such that the undesired overlap due to a wrong match can be properly penalized. We evaluate our proposal on several different classification tasks. In some experimental settings, our method outperforms other methods by a large margin. The contributions of this work can be summarized as follows: · We propose a novel upper bound taking joint error into account and theoretically prove that our proposal can reduce to several other methods under certain simplifications. · We construct a constrained hypothesis space such that a much tighter bound can be obtained during optimization. · We adopt a novel measurement, namely cross margin discrepancy, for the dissimilarity of two hypotheses on certain domain to alleviate the instability during adversarial learning and provide reliable performance. The upper bound proposed by invokes numerous approaches focusing on reducing the gap between source and target domains by learning domain-invariant features, which can be achieved through statistical moment matching. Long et al. (2015; use maximum mean discrepancy (MMD) to match the hidden representations of certain layers in a deep neural network. Transfer Component Analysis (TCA) tries to learn a subspace across domains in a Reproducing Kernel Hilbert Space (RKHS) using MMD that dramatically minimize the distance between domain distributions. Adaptive batch normalization (AdaBN) modulates the statistics from source to target on batch normalization layers across the network in a parameterfree way. Another way to learn domain-invariant features is by leveraging generative adversarial network to produce target features that exactly match the source. relax divergence measurement in the upper bound by a worst case which is equivalent to the maximum accuracy that a discriminator can possibly achieve when distinguishing source from target. follow this idea but separate the training procedure into classification stage and adversarial learning stage where an independent feature extractor is used for target. Saito et al. (2017b) explore a tighter bound by explicitly utilizing task-specific classifiers as discriminators such that features nearby the support of source samples will be favored by extractor. introduce margin disparity discrepancy, a novel measurement with rigorous generalization bounds, tailored to the distribution comparison with the asymmetric margin loss to bridge the gap between theory and algorithm. Methods perform distribution alignment on pixel-level in raw input, which is known as image-to-image translation, are also proposed (; ; ; ; ;). Distribution matching may not only bring the source and target domains closer, but also mix samples with different class labels together. Therefore, Saito et al. (2017a);; aim to use pseudo-labels to learn target discriminative representations encouraging a lowdensity separation between classes in the target domain . However, this usually requires auxiliary data-dependent hyper-parameter to set a threshold for a reliable prediction. present conditional adversarial domain adaptation, a principled framework that conditions the adversarial adaptation models on discriminative information conveyed in the classifier predictions, where the back-propagation of training objective is highly dependent on pseudo-labels. We consider the unsupervised domain adaptation as a binary classification task (our proposal holds for multi-class case) where the learning algorithm has access to a set of n labeled points {( . from the source domain S and a set of m unlabeled points {(x i t) ∈ X} m i=1 sampled i.i.d. from the target domain T. Let f S: X → {0, 1} and f T: X → {0, 1} be the optimal labeling functions on the source and target domains, respectively. Let (usually 0-1 loss) denotes a distance metric between two functions over a distribution that satisfies symmetry and triangle inequality. As a commonly used notation, the source risk of hypothesis h: X → {0, 1} is the error w.r.t. the true labeling function f S under domain S, i.e., S (h):= S (h, f S). Similarly, we use T (h) to represent the risk of the target domain. With these notations, the following bound holds: For simplicity, we use The above upper bound is minimized when h = f S, and it is equivalent to T (f S, f T) owing to the triangle inequality: Furthermore, we demonstrate in such case, our proposal is equivalent to an upper bound of optimal joint error λ because: Fig. 1b illustrates a case where common methods fail to penalize the undesirable situation when samples from different classes are mixed together during distribution matching, while our proposal is capable to do so (for simplicity we assume f S takes a specific form, then T (f S, f T) measures the overlapping area 2 and 5, which is equivalent to the optimal joint error λ). Since optimal labeling functions f S, f T are not available during training, we shall further relax the upper bound by taking supreme w.r.t f S, f T within a hypothesis space H: Then minimizing target risk T (h) becomes optimizing a minimax game and since the max-player taking two parameters f 1, f 2 is too strong, we introduce a feature extractor g to make the min-player stronger. Applying g to the source and target distributions, the overall optimization problem can be written as: min However, if we leave H unconstrained, the supreme term can be arbitrary large. In order to obtain a tight bound, we need to restrict the size of hypothesis space as well as maintain the upper bound. For f S ∈ H 1 ≤ H and f T ∈ H 2 ≤ H, the following holds: The constrained subspace for H 1 is trivial as according to its definition, f S must belong to the space consisting of all classifiers for source domain, namely H sc. However, the constrained subspace for H 2 is a little problematic since we have no access to the true labels of the target domain, thus it is hard to locate f T. Therefore, the only thing we can do is to construct a hypothesis space for H 2 that most likely contains f T. As is illustrated in Fig. 1c, when matching distributions of source and target domain, if the ideal case is achieved where the conditional distributions of source and target are perfectly aligned, then it is fare to assume f T ∈ H sc. However, if the worst case is reached where samples from different class are mixed together, then we tend to believe f T / ∈ H sc. Considering this, we present two proposals in the following sections based on different constraints. We assume H 2 is a space where the hypothesis can classify the samples from the source domain with an accuracy of γ ∈, namely H γ sc, such that we can avoid the worst case by choosing a small value for the hyper-parameter γ when a huge domain shift exists. In practice, it is difficult to actually build such a space and sample from it due to a huge computational cost. Instead, we use a weighted source risk to constrain the behavior of f 2 as an approximation to the sample from H γ sc, which leads to the final training objective: Firstly, we build a space consisting of all classifiers for approximate target domain {( based on pseudo labels which can be obtained by the prediction of h during training procedure, namely Ht c . Here, we assume H 2 is an intersection between two hypothesis spaces, i.e. Given enough reliable pseudo labels, we can be confident about f T ∈ H 2 . Analogously, the training objective is given by: The reason we make such an assumption for H 2 can be intuitively explained by Fig. 2 . If H 2 = H sc, then f 2 must perfectly classify the source samples, and it is possible that f 2 does not pass through some target samples (shadow are in 2a), especially when two domains differ a lot. In such case, the feature extractor can move those samples into either side of the decision boundary to reduce the training objective (shadow area) which is not a desired behavior. With an appropriate constraint (2b), as for the extractor, the only way to reduce the objective (shadow area) is to move those samples (orange) inside of f 2. Following the above notations, we consider a score function s(x, y) for multi-class classification where the output indicates the confidence of the prediction on class y. Thus an induced labeling function named l s from X → Y is given by: As a well-established theory, the margin between data points and the classification surface plays a significant role in achieving strong generalization performance. In order to quantify into differentiable measurement as a surrogate of 0-1 loss, we introduce the margin theory developed by , where a typical form of margin loss can be interpreted as: We aim to utilize this concept to further improve the reliability of our proposed method by leveraging this margin loss to define a novel measurement of the discrepancy between two hypotheses f 1, f 2 (e.g. softmax) over a distribution D, namely cross margin discrepancy: Before further discussion, we firstly construct two distributions D f1, D f2 induced by f 1, f 2 respectively, where Then we consider the case where two hypotheses f 1 and f 2 disagree, i.e. y 1 = l f1 (x) = l f2 (x) = y 2, and the primitive loss is defined as: Then the cross margin discrepancy can be viewed as: which is a sum of the margin loss for f 1 on D f2 and the margin loss for f 2 on D f1, if we use the logarithm of softmax as the score function. Thanks to the trick introduced by to mitigate the burden of exploding or vanishing gradients when performing adversarial learning, we further define a dual form as: This dual loss resembles the objective of the generative adversarial network, where two hypotheses try to increase the probability of their own prediction and simultaneously decrease the probability of their opponents; whereas the feature extractor is trained to increase the probability of their opponents, such that the discrepancy can be minimized without unnecessary oscillation. However, a big difference here is when training extractor, GANs usually maximize an alternative term log f 1 (x, y 2) + log f 2 (x, y 1) instead of directly minimizing log(1 − f 1 (x, y 2)) + log(1 − f 2 (x, y 1)) since the original term is close to zero if the discriminator achieves optimum. In our case, the hypothesis can hardly beat the extractor thus the original form can be more smoothly optimized. During the training procedure, the two hypotheses will eventually agree on some points (l f1 (x) = l f2 (x) = y) such that we need to define a new form of discrepancy measurement. Analogously, the primitive loss and its dual form are given by: Another reason why we propose such a discrepancy measurement is that it helps alleviate instability for adversarial learning. As is illustrated in Fig. 3b, during optimization of a minimax game, when two hypotheses try to maximize the discrepancy (shadow area), if one moves too fast around the decision boundary such that the discrepancy is actually maximized w.r.t some samples, then these samples can be aligned on either side to decrease the discrepancy by tuning the feature extractor, which is not a desired behavior. From Fig. 3a, we can see that our proposed cross margin discrepancy is flat for the points around original, i.e. the gradient w.r.t those points nearby the decision boundary will be relatively small, which helps to prevent such failure. propose a novel margin-aware generalization bound based on scoring functions and a new divergence MDD. The training objective used in MDD can be alternatively interpreted as (here (h, f) denotes the margin disparity): min Recall Eq.7, if we set f 1 = f 2 = f and free the constraint of f to any f ∈ H, our proposal degrades exactly to MDD. As is discussed above, when matching distribution, if and only if the ideal case is achieved, where the conditional distributions of induced feature spaces for source and target perfectly match (which is not always possible), can we assume two optimal labeling functions f S, f T to be identical. Besides, an unconstrained hypothesis space for f is definitely not helpful to construct a tight bound. Saito et al. (2017b) propose two task-specific classifiers f 1, f 2 that are used to separate the decision boundary on source domain, such that the extractor is encouraged to produce features nearby the support of the source samples. The objective used in MCD can be alternatively interpreted as (here (f 1, f 2) is quantified by L 1 ): Again, recall Eq.7, if we set γ = 1 and h = f 1, MCD is equivalent to our proposal. As is proved in section 3.1, the upper bound is optimized when h = f S. However, it no longer holds since the upper bound is relaxed by taking supreme to form an optimizabel objective, i.e. setting h = f 1 does not necessarily minimize the objective. Besides, as we discuss above, a fixed γ = 1, i.e H 2 = H sc lacks generality since we have no idea about where f T might be, such that it is not likely to be applicable to those cases where a huge domain shift exists. In this experiment, our proposal is assessed in four types of adaptation scenarios by adopting commonly used digits datasets (Fig. 6 in Appendix),i.e. MNIST , Street View House Numbers (SVHN) , and USPS such that the could be easily compared with other popular methods. All experiments are performed in an unsupervised fashion without any kinds of data augmentation. Details are omitted due to the limit of space (see A.1). We report the accuracy of different methods in Tab. 1. Our proposal outperforms the competitors in almost all settings except a single compared with GPDA . However, their solution requires sampling that increases data size and is equivalent to adding Gaussian noise to the last layer of a classifier, which is considered as a type of augmentation. Our success partially owes to combining the upper bound with the joint error, especially when optimal label functions differ from each other (e.g. MNIST →SVHN). Moreover, as most scenarios are relatively easy for adaptation thus we can be more confident about the hypothesis space constraint owing to reliable pseudo-labels, which leads to a tighter bond during optimization. The demonstrate our proposal can improve generalization performance by adopting both of these advantages. Fig. 4a shows that our original proposal is quite sensitive to the hyper-parameter γ. In short, setting γ = 1 here yields the best performance in most situations, since f S, f T can be quite close after aligning distributions, especially in these easily adapted scenarios. However, in MNIST → SVHN, setting γ = 0.1 gives the optimum which means that f S, f T are so far away due to a huge domain shift that no extractor is capable of introducing an identical conditional distribution in feature space. The improvement is not that much, but at least we outperform the directly comparable MCD and show the importance of hypothesis space constraint. Furthermore, Fig. 4d empirically proves simply minimizing the discrepancy between the marginal distribution does not necessarily lead to a reliable adaptation, which demonstrates the importance of joint error. In addition, Fig. 4b,Fig. 4c show the superiority of the cross margin discrepancy which accelerates the convergence and provides a slightly better . We further evaluate our method on object classification. The VisDA dataset is used here, which is designed for 12-class adaptation task from synthetic object to real object images. Source domain contains 152,397 synthetic images (Fig. 7a in Appendix), which are generated by rendering 3D CAD models. Data of the target domain is collected from MSCOCO consisting of 55,388 real images (Fig. 7b in Appendix). Since the 3D models are generated without the and color diversity, the synthetic domain is quite different from the real domain, which makes it a much more difficult problem than digits adaptation. Again, this experiment is performed in unsupervised fashion and no data augmentation technique excluding horizontal flipping is allowed. Details are omitted due to the limit of space (see A.2). We report the accuracy of different methods in Tab. 2, and find that our proposal outperforms the competitors in all settings. The image structure of this dataset is more complex than that of digits, yet our method provides reliable performance even under such a challenging condition. Another key observation is that some competing methods (e.g., DANN, MCD), which can be categorized as distribution matching based on adversarial learning, perform worse than MDD which simply matches statistics, in classes such as plane and horse, while our methods perform better across all classes, which clearly demonstrates the importance of taking the joint error into account. As for the original proposal (Fig. 5c), performance drops when relaxing the constraint which actually confuses us. Because we expect an improvement here since it is unbelievable that f S, f T eventually lie in a similar space judging from the relatively low prediction accuracy. As for the alternative proposal (Fig. 5d), we test the adaptation performance for different η and the prediction accuracy drastically drops when η goes beyond 0.2. One possible cause is that f 2 and h might almost agree on target domain, such that the prediction of h could not provide more accurate information for the target domain without introducing noisy pseudo labels. Fig. 5a, Fig. 5b again demonstrate the superiority of cross margin discrepancy and the importance of joint error. In this work, we propose a general upper bound that takes the joint error into account. Then we further pursuit a tighter bound with reasonable constraint on the hypothesis space. Additionally, we adopt a novel cross domain discrepancy for dissimilarity measurement which alleviates the instability during adversarial learning. Extensive empirical evidence shows that learning an invariant representation is not enough to guarantee a good generalization in the target domain, as the joint error matters especially when the domain shift is huge. We believe our take an important step towards understanding unsupervised domain adaptation, and also stimulate future work on the design of stronger adaptation algorithms that manage to align conditional distributions without using pseudo-labels from the target domain. layer and a 0.5 rate of dropout is conducted. Nesterov accelerated gradient is used for optimization with a mini-batch size of 32 and an initial learning rate of 10 −3 which decays exponentially. As for the hyper-parameter, we test for γ = {0.1, 0.5, 0.9, 1} and η = {0, 0.5, 0.8, 0.9}. For a direct comparison, we report the accuracy after 10 epochs. Office-Home is a complex dataset (Fig. 8) containing 15,500 images from four significantly different domains: Art (paintings, sketches and/or artistic depictions), Clipart (clip art images), Product (images without ), and Real-world (regular images captured with a camera). In this experiment, following the protocol from , we evaluate our method by fine-tuning a ResNet-50 model pretrained on ImageNet . The model except the last layer combined with a single-layer bottleneck is used as feature extractor and a randomly initialized 2-layer fully-connected network with width 1024 is used as a classifier, where batch normalization is applied to each layer and a 0.5 rate of dropout is conducted. For optimization, we use the SGD with the Nesterov momentum term fixed to 0.9, where the batch size is 32 and learning rate is adjusted according to. From Tab. 3, we can see the adaptation accuracy of the source-only method is rather low, which means a huge domain shift is quite likely to exist. In such case, simply minimizing the discrepancy between source and target might not work as the joint error can be increased when aligning distributions, thus the assumption of the basic theory does not hold anymore. On the other hand, our proposal incorporates the joint error into the target error upper bound which can boost the performance especially when there is a large domain shift. Figure 8: Sample images from the Office-Home dataset . Office-31 (Fig. 9) is a popular dataset to verify the effectiveness of a domain adaptation algorithm, which contains three diverse domains, Amazon from Amazon website, Webcam by web camera and DSLR by digital SLR camera with 4,652 images in 31 unbalanced classes. In this experiment, following the protocol from , we evaluate our method by fine-tuning a ResNet-50 model pretrained on ImageNet . The model used here is almost identical to the one in Office-Home experiment except a different width 2048 for classifiers. For optimization, we use the SGD with the Nesterov momentum term fixed to 0.9, where the batch size is 32 and learning rate is adjusted according to. The on Office-31 are reported in Tab. 4. As for the tasks D→A and W→A, judging from the adaptation accuracy of those previous methods that do not consider the joint error, it is quite likely that samples from different classes are mixed together when matching distributions. Our method shows an advantage in such case which demonstrates that the proposal manage to penalize the undesired matching between source and target. As for the tasks A→W and A→D, our proposal shows relatively high variance and poor performance especially in A→W. One possible reason is that our method depends on building reliable classifiers for the source domain to satisfy the constraint. However, the Amazon dataset contains a lot of noise (Fig. 10) such that the decision boundary of the source classifiers varies drastically in each iteration during training procedure, which can definitely harm the convergence.

joint error matters for unsupervised domain adaptation especially when the domain shift is huge

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We propose a method based on the adversarial training strategy to learn discriminative features unbiased and invariant to the confounder(s) by incorporating a loss function that encourages a vanished correlation between the bias and learned features.