Question: Represent the scientific article to find a short TLDR

Question: Generative adversarial networks (GANs) are a class of deep generative models which aim to learn a target distribution in an unsupervised fashion. While they were successfully applied to many problems, training a GAN is a notoriously challenging task and requires a significant amount of hyperparameter tuning, neural architecture engineering, and a non-trivial amount of ``tricks". The success in many practical applications coupled with the lack of a measure to quantify the failure modes of GANs ed in a plethora of proposed losses, regularization and normalization schemes, and neural architectures. In this work we take a sober view of the current state of GANs from a practical perspective. We reproduce the current state of the art and go beyond fairly exploring the GAN landscape. We discuss common pitfalls and reproducibility issues, open-source our code on Github, and provide pre-trained models on TensorFlow Hub. Deep generative models are a powerful class of unsupervised machine learning models. The power of these models was recently harnessed in a variety of applications, including image generation, learned compression, and domain transfer BID13; BID0 BID0. Generative adversarial networks BID8 are one of the main approaches to learning such models in a fully unsupervised fashion. The GAN framework can be viewed as a two-player game where the first player, the generator, is learning to transform some simple input distribution (usually a standard multivariate Normal or uniform) to a distribution on the space of images, such that the second player, the discriminator, cannot tell whether the samples belong to the true distribution or were synthesized. Both players aim to minimize their own loss and the solution to the game is the Nash equilibrium where neither player can improve their loss unilaterally. This powerful framework can also be derived by minimizing a divergence between the model distribution and the true distribution BID20.Training GANs involves solving a minimax problem over the parameters of the generator and the discriminator which are usually parameterized as deep convolutional neural networks. Consequently, this minimax problem is notoriously hard to solve in practice. As a , a plethora of loss functions, regularization and normalization schemes, coupled with neural architecture choices, have been proposed BID8; BID19 BID9 BID18.Our contributions. In this work we provide a thorough empirical analysis of these competing approaches, and help the researchers and practitioners navigate this space. We first define the GAN landscape -the set of loss functions, normalization and regularization schemes, and the most commonly used architectures. We explore this search space on several modern large-scale data sets by means of hyperparameter optimization, considering both "good" sets of hyperparameters reported in the literature, as well as ones obtained by Gaussian Process regression. By analyzing the impact of the loss function, we conclude that the non-saturating loss is sufficiently stable across data sets, architectures and hyperparameters. We then proceed to decompose the effect of various normalization and regularization schemes, as well as varying architectures. We show that both gradient penalty BID9 as well as spectral normalization BID19 are useful in the context of high-capacity architectures. Finally, we discuss some common pitfalls, reproducibility issues, and practical considerations. We provide reference implementations, including training and evaluation code on Github 1 and provide pre-trained models on TensorFlow Hub. 2 2 THE GAN LANDSCAPE Let P denote the target (true) distribution and Q the model distribution. BID8 suggest two loss functions: the minimax GAN and the non-saturating (NS) GAN. In the former the discriminator minimizes the negative log-likelihood for the binary classification task. In the latter the generator maximizes the probability of generated samples being real. In this work we consider the non-saturating loss as it is known to outperform the minimax variant. The corresponding loss functions are DISPLAYFORM0 In Wasserstein GAN (WGAN) ) the authors propose to consider the Wasserstein divergence instead of the original Jensen-Shannon (JS). In particular, under the optimal discriminator, minimizing the proposed value function with respect to the generator minimizes the Wasserstein distance between P and Q. The drawback is that one has to ensure a 1-Lipschitz discriminator due to exploited Kantorovich-Rubenstein duality. The corresponding loss functions are DISPLAYFORM1 Finally, we consider the least-squares loss (LS) which corresponds to minimizing the Pearson χ 2 divergence between P and Q BID18. The intuition is that this loss function is smooth and saturates slower than the sigmoid cross-entropy loss of the JS formulation. The corresponding loss functions are DISPLAYFORM2 Gradient norm penalty. In the context of Wasserstein GANs this penalty can be interpreted as a soft penalty for the violation of 1-Lipschitzness (WGAN GP) BID9. Hereby, the gradient is evaluated on a linear interpolation between training points and generated samples as a proxy to the optimal coupling. The gradient penalty can also be evaluated around the data manifold which encourages the discriminator to be piece-wise linear in that region (Dragan) BID16. However, the gradient norm penalty can be considered purely as a regularizer for the discriminator and it was shown that it can improve the performance for other losses BID7. Furthermore, the penalty can be scaled by the "confidence" of the discriminator in the context of f-divergences . A drawback of gradient penalty (GP) regularization scheme is that it can depend on the model distribution Q which changes during training. One drawback of Dragan is that it is unclear to which extent the Gaussian assumption for the manifold holds. Finally, computing the gradient norms implies a non-trivial running time penalty -essentially doubling the running time. We also investigate the impact of a regularizer ubiquitous in supervised learning -the L 2 penalty on all the weights of the network. Discriminator normalization. Normalizing the discriminator can be useful from both the optimization perspective (more efficient gradient flow, a more stable optimization), as well as from the representation perspective -the representation richness of the layers in a neural network depends on the spectral structure of the corresponding weight matrices BID19.From the optimization point of view, several techniques have found their way into the GAN literature, namely batch normalization (BN) BID12 and layer normalization (LN) BID2. Batch normalization in the context of GANs was suggested by BID6 and further popularized by. It normalizes the pre-activations of nodes in a layer to mean β and standard deviation γ, where both β and γ are parameters learned for each node in the layer. The normalization is done on the batch level and for each node separately. In contrast, with Layer normalization, all the hidden units in a layer share the same normalization terms β and γ, but different samples are normalized differently BID2. Layer normalization was first applied in the context of GANs in BID9.From the representation point of view, one has to consider the neural network as a composition of (possibly non-linear) mappings and analyze their spectral properties. In particular, for the discriminator to be a bounded linear operator it suffices to control the maximum singular value. This approach is followed in BID19 where the authors suggest dividing each weight matrix, including the matrices representing convolutional kernels, by their spectral norm. Furthermore, the authors argue that a key advantage of spectral normalization over competing approaches is that it in discriminators of higher rank. We explore two classes of architectures in this study: deep convolutional generative adversarial networks (DCGAN) and residual networks (ResNet) BID10, both of which are ubiquitous in GAN research. Recently, BID19 defined a variation of DCGAN, so called SNDCGAN. Apart from minor updates (cf. Section 4) the main difference to DCGAN is the use of an eight-layer discriminator network. The details of both networks are summarized in TAB3. The other architecture, ResNet19, is an architecture with five ResNet blocks in the generator and six ResNet blocks in the discriminator, that can operate on 128 × 128 images. We follow the ResNet setup from BID19, with the small difference that we simplified the design of the discriminator. The detailed parameters of discriminator and generator are summarized in TAB4 and TAB4. With this setup we were able to reproduce the current state of the art . An ablation study on various ResNet modifications is available in the Appendix. We focus on several recently proposed metrics well suited to the image domain. For an in-depth overview of quantitative metrics we refer the reader to BID4.Inception Score (IS). Proposed by , IS offers a way to quantitatively evaluate the quality of generated samples. Intuitively, the conditional label distribution of samples containing meaningful objects should have low entropy, and the variability of the samples should be high. which can be expressed as DISPLAYFORM0 The authors found that this score is well-correlated with scores from human annotators. Drawbacks include insensitivity to the prior distribution over labels and not being a proper distance. As an alternative BID11 proposed the Frechet Inception Distance (FID). Samples from P and Q are first embedded into a feature space (a specific layer of InceptionNet). Then, assuming that the embedded data follows a multivariate Gaussian distribution, the mean and covariance are estimated. Finally, the Fréchet distance between these two Gaussians is computed, i.e. DISPLAYFORM1 where (µ x, Σ x), and (µ y, Σ y) are the mean and covariance of the embedded samples from P and Q, respectively. The authors argue that FID is consistent with human judgment and more robust to noise than IS. Furthermore, the score is sensitive to the visual quality of generated samplesintroducing noise or artifacts in the generated samples will reduce the FID. In contrast to IS, FID can detect intra-class mode dropping, i.e. a model that generates only one image per class can score a perfect IS, but will suffer from have a high FID BID17. BID3 argued that FID has no unbiased estimator and suggest Kernel Inception distance (KID) instead. In Appendix B we empirically compare KID to FID and observe that both metrics are very strongly correlated (Spearman rank-order correlation coefficient of 0.994 for LSUN-BEDROOM and 0.995 for CELEBA-HQ-128 datasets). As a we focus on FID as it is likely to in the same ranking. Multi-scale Structural Similarity for Image Quality (MS-SSIM) and Diversity. A critical issue in GANs are mode collapse and mode-dropping -failing to capture a mode, or low-diversity of generated samples from a given mode. The MS-SSIM score is used for measuring the similarity of two images where higher MS-SSIM score indicates more similar images. Several recent works suggest using the average pairwise MS-SSIM score within a given class as a proxy for the diversity of generated samples BID21 BID7. The drawback of this [10 DISPLAYFORM2 approach is that we do not know the class corresponding to the generated sample, so it is usually applied on one-class data sets, such as CELEBA-HQ-128. In this work we use the same setup as in BID7 . In particular, given a batch size b, we compute the average pairwise MS-SSIM score on 5 batches, of 5 × b × (b − 1)/2 image pairs in total. We stress that the diversity should only be taken into account together with the FID and IS metrics. We consider three data sets, namely CIFAR10, CELEBA-HQ-128, and LSUN-BEDROOM. The LSUN-BEDROOM data set contains slightly more than 3 million images 3. We randomly partition the images into a train and test set whereby we use 30588 images as the test set. Secondly, we use the CELEBA-HQ data set of 30k images BID14. We use the 128 × 128 × 3 version obtained by running the code provided by the authors. 4 We use 3000 examples as the test set and the remaining examples as the training set. Finally, we also include the CIFAR10 data set which contains 70K images (32x32x3), partitioned into 60000 training instances and 10000 testing instances. The baseline FID scores are 12.6 for CELEBA-HQ-128, 3.8 for LSUN-BEDROOM, and 5.19 for CIFAR10. Details on FID computation are presented in Section 4. The search space for GANs is prohibitively expensive: exploring all combinations of all losses, normalization and regularization schemes, and architectures is outside of the practical realm. Instead, in this study we analyze several slices of this tensor for each data set. In particular, to ensure that we can reproduce existing , we perform a study over the subset of this tensor on CIFAR10. We then proceed to analyze the performance of these models across CELEBA-HQ-128 and LSUN-BEDROOM. In Section 3.1 we fix everything but the loss. In Section 3.2 we fix everything but the regularization and normalization scheme. Finally, in Section 3.3 we fix everything but the architecture. This allows us to decouple some of these design choices and provide some insight on what matters most. As noted in BID17, one major issue preventing further progress is the hyperparameter tuning -currently, the community has converged to a small set of parameter values which work on some data sets, and may completely fail on others. In this study we combine the best hyperparameter settings found in the literature BID19, and perform Gaussian Process regression in the bandit setting to possibly uncover better hyperparameter settings. We then consider the top performing models and discuss the impact of the computational budget. We summarize the fixed hyperparameter settings in TAB0 which contains the "good" parameters reported in recent publications BID7 BID19 BID9. In particular, we consider the cross product of these parameters to obtain 24 hyperparameter settings to reduce the bias. Finally, to provide a fair comparison, we perform Gaussian Process optimization in the bandit setting on the parameter ranges provided in TAB0. We run 12 rounds (i.e. we communicate with the oracle 12 times) of the optimization, each with a batch of 10 hyperparameter sets selected based on the FID scores from the of the previous iterations. Figure 1: Impact of the loss function: FID distribution for top 5% models. The non-saturating (NS) loss is stable over both data sets. Gradient penalty and spectral normalization improve the sample quality. From the computational budget perspective (i.e. how many models one needs to train to reach a certain FID), both spectral normalization and gradient penalty perform better than the baseline, but the former is more efficient. As we explore the number of discriminator updates per generator update (1 or 5), this leads to an additional 240 hyperparameter settings which in some cases outperform the previously known hyperparameter settings. Batch size is set to 64 for all the experiments. We use a fixed the number of discriminator update steps of 100K for LSUN-BEDROOM data set and CELEBA-HQ-128 data set, and 200K for CIFAR10 data set. We apply the Adam optimizer BID15. Given that there are 4 major components (loss, architecture, regularization, normalization) to analyze for each data set, it is infeasible to explore the whole landscape. Hence, we opt for a more pragmatic solution -we keep some dimensions fixed, and vary the others. For each experiment we highlight three aspects: FID distribution of the top 5% of the trained models, the corresponding sample diversity score, and the tradeoff between the computational budget (i.e. number of models to train) and model quality in terms of FID. Each model was retrained 5 times with a different random seed and we report the median score. The variance for models obtained by Gaussian Process regression is handled implicitly so we train each model once. Here the loss is either the non-saturating loss (NS) BID8, the least-squares loss (LS) BID18, or the Wasserstein loss (WGAN). We use the ResNet19 with generator and discriminator architectures detailed in TAB4. We consider the most prominent normalization and regularization approaches: gradient penalty BID9, and spectral normalization BID19. Both studies were performed on CELEBA-HQ-128 and LSUN-BEDROOM with hyperparameter settings shown in TAB0.The are presented in Figure 1. We observe that the non-saturating loss is stable over both data sets. Spectral normalization improves the quality of the model on both data sets. Similarly, the gradient penalty can help improve the quality of the model, but finding a good regularization tradeoff is non-trivial and requires a high computational budget. Models using the GP penalty benefit from 5:1 ratio of discriminator to generator updates as suggested by BID9. We also performed a study on hinge loss BID19 and present it in the Appendix. Figure 2: Impact of regularization and normalization: FID distribution for top 5% models. Both gradient penalty (GP) and spectral normalization (SN) outperform the baseline and should be considered, while former being more computationally expensive. Unfortunately none fully address the stability issues. DISPLAYFORM0 The goal of this study is to compare the relative performance of various regularization and normalization methods presented in the literature. To this end, and based on the loss study, we fix the loss to non-saturating loss BID8. We use the ResNet19 with generator and discriminator architectures described in TAB4. Finally, we consider batch normalization (BN) BID12, layer normalization (LN) BID2, spectral normalization (SN), gradient penalty (GP) BID9, dragan penalty (DR) BID16, or L 2 regularization. We consider both CELEBA-HQ-128 and LSUN-BEDROOM with the hyperparameter settings shown in TAB0.The are presented in Figure 2. We observe that adding batch norm to the discriminator hurts the performance. Secondly, gradient penalty can help, but it doesn't stabilize the training. In fact, it is non-trivial to strike a balance of the loss and regularization strength. Spectral normalization helps improve the model quality and is more computationally efficient than gradient penalty. This is consistent with recent in. Similarly to the loss study, models using GP penalty benefit from 5:1 ratio of discriminator to generator updates. Furthermore, in a separate ablation study we observed that running the optimization procedure for an additional 100K steps is likely to increase the performance of the models with GP penalty. Impact of Simultaneous Regularization and Normalization. Given the folklore that the Lipschitz constant of the discriminator is critical for the performance, one may expect simultaneous Test set Noisy test set Figure 3: Impact of simultaneous normalization and regularization: FID distribution for top 5% models. Gradient penalty coupled with spectral normalization (SN) or layer normalization (LN) strongly improves the performance over the baseline. DISPLAYFORM0 regularization and normalization could improve model quality. To quantify this effect, we fix the loss to non-saturating loss BID8, use the Resnet19 architecture (as above), and combine several normalization and regularization schemes, with hyperparameter settings shown in TAB0 coupled with 24 randomly selected parameters. The are presented in Figure 3. We observe that one may benefit from additional regularization and normalization. However, a lot of computational effort has to be invested for somewhat marginal gains in FID. Nevertheless, given enough computational budget we advocate simultaneous regularization and normalization -spectral normalization and layer normalization seem to perform well in practice. An interesting practical question is whether our findings also hold for a different model capacity. To this end, we also perform a study on SNDCGAN from BID19. We consider the non-saturating GAN loss, gradient penalty and spectral normalization. While for smaller architectures regularization is not essential BID17, the regularization and normalization effects might become more relevant due to deeper architectures and optimization considerations. The are presented in FIG4. We observe that both architectures achieve comparable and benefit from regularization and normalization. Spectral normalization strongly outperforms the baseline for both architectures. In this section we focus on several pitfalls we encountered while trying to reproduce existing and provide a fairly and accurate comparison. Metrics. There already seems to be a divergence in how the FID score is computed: Some authors report the score on training data, yielding a FID between 50k training and 50k generated samples . Some opt to report the FID based on 10k test samples and 5k generated samples and use a custom implementation BID19. Finally, BID17 report the score with respect to the test data, in particular FID between 10k test samples, and 10k generated samples. The subtle differences will in a mismatch between the reported FIDs, in some cases of more than 10%. We argue that FID should be computed with respect to the test data set as and use 10k test samples and 10k generated samples on CIFAR10 and LSUN-BEDROOM, and 3k vs 3k on CELEBA-HQ-128 as in in BID17. Similarly, there are several ways to compute a diversity score using MS-SSIM and we follow the approach from BID7. We provide the implementation details in Section G of the Appendix. Details of neural architectures. Even in popular architectures, like ResNet, there is still a number of design decision one needs to make, that are often omitted from the reported . Those include the exact design of the ResNet cell (order of layers, when is ReLu applied, when to upsample and downsample, how many filters to use). Some of these differences might lead to potentially unfair comparison. As a , we suggest to use the architectures presented within this work as a solid baseline. An ablation study on various ResNet modifications is available in the Appendix. Data sets. A common issue is related to data set processing -does LSUN-BEDROOM always correspond to the same data set? In most cases the precise algorithm for upscaling or cropping is not clear which introduces inconsistencies between on the "same" data set. Implementation details and non-determinism. One major issue is the mismatch between the algorithm presented in a paper and the code provided online. We are aware that there is an embarrassingly large gap between a good implementation and a bad implementation of a given model. Hence, when no code is available, one is forced to guess which modifications were done. Another particularly tricky issue is removing randomness from the training process. After one fixes the data ordering and the initial weights, obtaining the same score by training the same model twice is non-trivial due to randomness present in certain GPU operations BID5. Disabling the optimizations causing the non-determinism often in an order of magnitude running time penalty. While each of these issues taken in isolation seems minor, they compound to create a mist which introduces friction in practical applications and the research process . A recent large-scale study on GANs and Variational Autoencoders was presented in BID17. The authors consider several loss functions and regularizers, and study the effect of the loss function on the FID score, with low-to-medium complexity data sets (MNIST, CIFAR10, CELEBA), and a single (InfoGAN style) architecture. In this limited setting, the authors found that there is no statistically significant difference between recently introduced models and the original non-saturating GAN. A study of the effects of gradient-norm regularization in GANs was recently presented in BID7. The authors posit that the gradient penalty can also be applied to the non-saturating GAN, and that, to a limited extent, it reduces the sensitivity to hyperparameter selection. In a recent work on spectral normalization, the authors perform a small study of the competing regularization and normalization approaches BID19. We are happy to report that we could reproduce these and we present them in the Appendix. Inspired by these works and building on the available open-source code from BID17, we take one additional step in all dimensions considered therein: more complex neural architectures, more complex data sets, and more involved regularization and normalization schemes. In this work we study the GAN landscape: losses, regularization and normalization schemes, and neural architectures, and their impact on the on the quality of generated samples which we assess by recently introduced quantitative metrics. Our fair and thorough empirical evaluation suggests that one should consider non-saturating GAN loss and spectral normalization as default choices when applying GANs to a new data set. Given additional computational budget, we suggest adding the gradient penalty from BID9 and train the model until convergence. Furthermore, additional marginal gains can be obtained by combining normalization and regularization empirically confirming the importance of the Lipschitz constant of the discriminator. Furthermore, both types of architectures proposed up-to this point perform reasonably well. A separate ablation study uncovered that most of the tricks applied in the ResNet style architectures lead to marginal changes in the quality and should be avoided due to the high computational cost. As a of this large-scale study we identify the common pitfalls standing in the way of accurate and fair comparison and propose concrete actions to demystify the future -issues with metrics, data set preprocessing, non-determinism, and missing implementation details are particularly striking. We hope that this work, together with the open-sourced reference implementations and trained models, will serve as a solid baseline for future GAN research. We present an empirical study with SNDCGAN and ResNet CIFAR architectures on CIFAR10 in figure 5 and figure 6. In addition to Section 3.1, we evaluate one more kind of loss on CIFAR10. Here HG, NS and WGAN stand for hinge loss, non saturating loss and Wasserstein loss respectively. We observe that hinge loss performs very similar to non-saturating loss. DISPLAYFORM0 The KID metric introduced by BID3 is an alternative to FID. We use models from our Regularization and Normalization study (see Section 3.2) to compare both metrics. Here, by model we denote everything that needs to be specified for the training -including all hyper-parameters, like learning rate, λ, Adam's β, etc. The Spearman rank-order correlation coefficient between KID and FID scores is approximately 0.994 for LSUN-BEDROOM and 0.995 for CELEBA-HQ-128 datasets. To evaluate a practical setting of selecting several best models, we compare the intersection between the set of "best K models by FID" and the set of "best K models by KID" for K ∈ 5, 10, 20, 50, 100. The are summarized in TAB2.This experiment suggests that FID and KID metrics are very strongly correlated, and for the practical applications one can choose either of them. Also, the from our studies based on FID should transfer to studies based on KID. We used the same architecture as BID19, with the parameters copied from the GitHub page 5. In TAB3, we describe the operations in layer column with order. Kernel size is described in format [f ilter h, f ilter w, stride], input shape is h × w and output shape is h × w × channels. The slopes of all lReLU functions are set to 0.1. The input shape h × w is 128 × 128 for CELEBA-HQ-128 and LSUN-BEDROOM, 32 × 32 for CIFAR10. The ResNet19 architecture is described in TAB4. RS column stands for the resample of the residual block, with downscale(D)/upscale(U)/none(-) setting. MP stands for mean pooling and BN for batch normalization. ResBlock is defined in TAB5. The addition layer merges two paths by adding them. The first path is a shortcut layer with exactly one convolution operation, while the second path consists of two convolution operations. The downscale layer and upscale layer are marked in TAB5. We used average pool with kernel for downscale, after the convolution operation. We used unpool from https://github.com/tensorflow/ tensorflow/issues/2169 for upscale, before convolution operation. h and w are the input shape to the ResNet block, output shape depends on the RS parameter. ci and co are the input channels and output channels for a ResNet block. TAB6 described the ResNet CIFAR architecture we used in Figure 5 for reproducing the existing . Note that RS is set to none for third ResBlock and fourth ResBlock in discriminator. In this case, we used the same ResNet block defined in TAB5 without resampling. DISPLAYFORM0 ReLU, MP --512 (a) ResBlock discriminator DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 ReLU, MP --128 DISPLAYFORM4 We have noticed six minor differences on Resnet architecture comparing to implementation from https: //github.com/pfnet-research/chainer-gan-lib/blob/master/common/net.py BID19. We did ablation study to verify the impact of these differences. FIG7 shows the impact of the ablation study, with details described as following.• DEFAULT: ResNet CIFAR architecture with spectral normalization and non-saturating GAN loss.• SKIP: Use input as output for the shortcut connection in the discriminator ResBlock. By default it was a conv layer with 3x3 kernel.• CIN: Use ci for the discriminator ResBlock hidden layer output channels. By default it was co in our setup, while BID19 used co for first ResBlock and ci for the rest.• OPT: Use an optimized setup for the first discriminator ResBlock, which includes: no ReLU, a conv layer for the shortcut connections, use co instead of ci in ResBlock.• CIN OPT: Use CIN and OPT together. It means the first ResBlock is optimized while the remaining ResBlocks use ci for the hidden output channels.• SUM: Use reduce sum for the discriminator output. By default it was reduce mean.• TAN: Use tanh for the generator output, as well as range [-1, 1] for discriminator input. By default it was sigmoid and discriminator input range.• EPS: Use a bigger epsilon 2e − 5 for generator batch normalization. By default it was 1e − 5 in TensorFlow.• ALL: Apply all the above differences together. In the ablation study, the CIN experiment obtained the worst FID score. Combining with OPT, the CIN were improved to the same level as the others which is reasonable because the first block has three input channels, which becomes a bottleneck for the optimization. Hence, using OPT and CIN together performs as well as the others. Overall, the impact of these differences are minor according to the study on CIFAR10. DISPLAYFORM5 To make the future GAN training simpler, we propose a set of best parameters for three setups: Best parameters without any regularizer. Best parameters with only one regularizer. Best parameters with at most two regularizers. TAB7 summarize the top 2 parameters for SNDCGAN architecture, ResNet19 architecture and ResNet CIFAR architecture, respectively. Models are ranked according to the median FID score of five different random seeds with fixed hyper-parameters in TAB0. Note that ranking models according to the best FID score of different seeds will achieve better but unstable . Gaussian Process optimization hyper-parameters are not included in this table. For ResNet19 architecture with at most two regularizers, we have run it only once due to computational overhead. To show the model stability, we listed the best FID score out of five seeds from the same parameters in column best. Spectral normalization is clearly outperforms the other normalizers on SNDCGAN and ResNet CIFAR architectures, while on ResNet19 both layer normalization and spectral normalization work well. To visualize the FID score on each data set, Figure 8, Figure 9 and Figure 10 show the generated examples by GANs. We select the examples from the best FID run, and then increase the FID score for two more plots. For each architecture and hyper-parameter we estimate its impact on the final FID. Figure 11 presents heatmaps for hyperparameters, namely the learning rate, β1, β2, n disc, and λ for each combination of neural architecture and data set. We used the MS-SSIM scorer from TensorFlow with default power factors . Note that the default filter size for each scale layer is 11, the minimum image edge is 11 × 2 4 = 176. To adapt it to CELEBA-HQ-128 data set with size 128 × 128, we used the minimum of filter size 11 and image size in last scale layer to allow the computation followed the previous work BID7.

Represent the short summary for finding the corresponding scientific article (long)

A sober view on the current state of GANs from a practical perspective

Represent the TLDR for finding the corresponding scientific article

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the scientific article to find a short TLDR

Question: As deep reinforcement learning (RL) is applied to more tasks, there is a need to visualize and understand the behavior of learned agents. Saliency maps explain agent behavior by highlighting the features of the input state that are most relevant for the agent in taking an action. Existing perturbation-based approaches to compute saliency often highlight regions of the input that are not relevant to the action taken by the agent. Our approach generates more focused saliency maps by balancing two aspects (specificity and relevance) that capture different desiderata of saliency. The first captures the impact of perturbation on the relative expected reward of the action to be explained. The second downweights irrelevant features that alter the relative expected rewards of actions other than the action to be explained. We compare our approach with existing approaches on agents trained to play board games (Chess and Go) and Atari games (Breakout, Pong and Space Invaders). We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that our approach generates saliency maps that are more interpretable for humans than existing approaches. and deep sequential models . However, interpretability for RL-based agents has received significantly less attention. Interpreting the strategies learned by RL agents can help users better understand the problem that the agent is trained to solve. For instance, interpreting the actions of a chess-playing agent in a position could provide useful information about aspects of the position. Interpretation of RL agents is also an important step before deploying such models to solve real-world problems. Inspired by the popularity and use of saliency maps to interpret in computer vision, a number of existing approaches have proposed similar methods for reinforcement learning-based agents. derive saliency maps that explain RL agent behavior by applying a Gaussian blur to different parts of the input image. They generate saliency maps using differences in the value function and policy vector between the original and perturbed state. They achieve promising on agents trained to play Atari games. compute saliency maps using a difference in the action-value (Q(s, a)) between the original and perturbed state. There are two primary limitations to these approaches. The first is that they highlight features whose perturbation affects actions apart from the one we are explaining. This is illustrated in Figure 1, which shows a chess position (it is white's turn). Stockfish 1 plays the move Bb6 in this position, which traps the black rook (a5) and queen (c7) 2. The knight protects the white bishop on a4, and hence the move works. In this position, if we consider the saliency of the white queen (square d1), then it is apparent that the queen is not involved in the tactic and hence the saliency should be low. However, perturbing the state (by removing the queen) leads to a state with substantially different values for Q(s, a) and V (s). Therefore, existing approaches ) mark the queen as salient. The second limitation is that they highlight features that are not relevant to the action to be explained. In Figure 1c, perturbing the state by removing the black pawn on c6 alters the expected reward for actions other than the one to be explained. Therefore, it alters the policy vector and is marked salient. However, the pawn is not relevant to explain the move played in the position (Bb6). In this work, we propose a perturbation based approach for generating saliency maps for black-box agents that builds on two desired properties of action-focused saliency. The first, specificity, captures the impact of perturbation only on the Q-value of the action to be explained. In the above example, this term downweights features such as the white queen that impact the expected reward of all actions equally. The second, relevance, downweights irrelevant features that alter the expected rewards of actions other than the action to be explained. It removes features such as the black pawn on c6 that increase the expected reward of other actions (in this case, Bb4). By combining these aspects, we generate a saliency map that highlights features of the input state that are relevant for the action to be explained. Figure 1 illustrates how the saliency map generated by our approach only highlights pieces relevant to the move, unlike existing approaches. We use our approach to explain the actions taken by agents for board games (Chess and Go), and for Atari games (Breakout, Pong and Space Invaders). Using a number of illustrative examples, we show that our proposed approach obtains more focused and accurate interpretations for all of these setups when compared to and. We also demonstrate that our approach is more effective in identifying important pieces in chess puzzles, and further, in aiding skilled chess players to solve chess puzzles (improves accuracy of solving them by nearly 25% and reduces the time taken by 31% over existing approaches). We are given an agent M, operating on a state space S, with the set of actions A s for s ∈ S, and a Q-value function denoted as Q(s, a) for s ∈ S, a ∈ A s. Following a greedy policy, let the action that was selected by the agent at state s beâ, i.e.â = arg max a Q(s, a). The states are parameterized in terms of state-features F. For instance, in a board game such as chess, the features are the 64 squares. For Atari games, the features are pixels. We are interested in identifying which features of the state s are important for the agent in taking actionâ. We assume that the agent is in the exploitation phase 1 https://stockfishchess.org/ 2 We follow the coordinate naming convention where columns are'a-h' (left-right), rows'8-1' (top-bottom), and pieces are labeled using the first letter of its name in upper case (e.g. 'B' denotes the bishop). A move consists of the piece and the position it moves to, e.g.'Bb6' indicates that the bishop moves to position'b6'. and therefore plays the action with the highest expected reward. This feature importance is described by an importance-score or saliency for each feature f, denoted by S, where S[f] ∈ denotes the saliency of the f th feature of s for the agent taking actionâ. A higher value indicates that the f th feature of s is more important for the agent when taking actionâ. The general outline of perturbation based saliency approaches is as follows. For each feature f, first perturb s to get s. For instance, in chess, we can perturb the board position by removing the piece in the f th square. In Atari, perturb the input image by adding a Gaussian blur centered on the f th pixel. Second, query M to get Q(s, a) ∀a ∈ A s ∩ A s'. We take the intersection of A s and A s to represent the case where some actions may be legal in s but not in s and vice versa. For instance, when we remove a piece in chess, actions that were legal earlier may not be legal anymore. In the rest of this section, when we use "all actions" we mean all actions that are legal in both the states s and s. Finally, compute S[f] based on how different Q(s, a) and Q(s, a)) are, i.e. intuitively, S[f] should be higher if Q(s, a) is significantly different from Q(s, a). compute the saliency map using 2, while s,â). In this work, we will propose an alternative approach to compute S[f]. Properties We define two desired properties of an accurate saliency map for policy-based agents: 1. Specificity: Saliency S[f] should focus on the effect of the perturbation specifically on the action being explained,â, i.e. it should be high if perturbing the f th feature of the state reduces the relative expected reward of the selected action. Stated another way, S[f] should be high if Q(s,â) − Q(s,â) is substantially higher than Q(s, a) − Q(s, a), a =â. For instance, in figure 1, removing pieces such as the white queen impact all actions uniformly (Q(s, a) − Q(s, a) is roughly equal for all actions). Therefore, such pieces should not be salient for explainingâ. On the other hand, removing pieces such as the white knight on a4 specifically impacts the move (â =Bb6) we are trying to explain (Q(s, Bb6) − Q(s, Bb6) Q(s, a) − Q(s, a) for other actions a). Therefore, such pieces should be salient forâ. Figure 1, removing the black pawn on c6 increases the expected reward of other actions (in this case, Bb4). However, it does not effect the expected reward of the action to be explained (Bb6). Therefore, the pawn is not salient for explaining the move. In general, such features that are irrelevant toâ should not be salient. Existing approaches to saliency maps do not capture these properties in how they compute the saliency. Both the saliency approaches used in are not focusing on the action-specific effects since they aggregate the change over all actions. Although the saliency computation in is somewhat more specific to the action, i.e. S[f] = Q(s,â) − Q(s,â), it is ignoring whether the effects on Q are relevant only tô a, or effect all the other actions as well. This is illustrated in Figure 1. To focus on the effect of the change on the action, we are interested in whether the relative returns ofâ change with the perturbation. Using Q(s,â) directly, as in , does not capture the relative changes to Q(s, a) for other actions. To support specificity, we use the softmax over Q-values to normalize the values (as is also used in softmax action selection): and compute ∆p = P (s,â) − P (s,â), the difference in the relative expected reward of the action to be explained between the original and the perturbed state. A high value of ∆p thus implies that the feature is important for the specific choice of actionâ by the agent, while a low value indicates that the effect is not specific to the action. Identifying Relevant Changes Apart from focusing on the change in Q(s,â), we also want to ensure that the perturbation leads to minimal effect on the relative expected returns for other actions. To capture this intuition, we will compute the relative returns of all other actions, and compute saliency in proportion to their similarity. Specifically, we normalize the Q-values using a softmax apart from the selected actionâ. We use the KL-Divergence D KL = P rem (s, a)||P rem (s, a) to measure the difference between P rem (s, a) and P rem (s, a). A high D KL indicates that the relative expected reward of taking some actions (other than the original action) changes significantly between s and s. In other words, a high D KL indicates that the effect of the feature is spread over other actions, i.e. the feature may not be relevant for the selected actionâ. Computing the Saliency To compute salience S[f], we need to combine ∆p and should be low, regardless of whether ∆p is high; the perturbation is affecting many other actions. Conversely, when D KL is low, S[f] should depend on ∆p. To be able to compare these properties on a similar scale, we define a normalized measure of distribution similarity K using D KL: As D KL goes from 0 to ∞, K goes from 1 to 0. Thus, S[f] should be low if either ∆p is low or K is low. Harmonic mean provides this desired effect in a robust, smooth manner, and therefore we define S[f] to be the harmonic mean of ∆p and K: Equation 4 captures our desired properties of saliency maps. If perturbing the f th feature affects the expected rewards of all actions uniformly, then ∆p is low and subsequently S[f] is low. This low value of ∆p captures the property of specificity defined above. If perturbing the f th feature of the state affects the rewards of some actions other than the action to be explained, then D KL is high, K is low, and S[f] is low. This low value of K captures the property of relevance defined above. To show that our approach produces more meaningful saliency maps than existing approaches, we use sample positions from Chess, Atari (Breakout, Pong and Space Invaders) and Go (Section 3.1). To show that our approach generates saliency maps that provide useful information to humans, we conduct human studies on problem-solving for chess puzzles (Section 3.2). To automatically compare the saliency maps generated by different perturbation-based approaches, we introduce a Chess saliency dataset (Section 3.3). We use the dataset to show how our approach is better than existing approaches in identifying chess pieces that humans deem relevant in several positions. In Section 3.4, we show how our approach can be used to understand common tactical ideas in chess by interpreting the action of a trained agent. To show that our approach works for black-box agents, regardless of whether they are trained using reinforcement learning, we use a variety of agents. We only assume access to the agent's Q(s, a) function for all experiments. For experiments on chess, we use the Stockfish agent 3. For experiments on Go, we use the pre-trained MiniGo RL agent 4. For experiments on Atari agents and for generating saliency maps for, we use their code and pre-trained RL agents 5. For generating saliency maps using , we use our own implementation 6. All of our code and more detailed are available in our Github repository 6. In this section, we provide examples of generated saliency maps to highlight the qualitative differences between our approach that is action-focused and existing approaches that are not. Chess Figure 1 shows sample positions where our approach produces more meaningful saliency maps than existing approaches for a chess-playing agent (Stockfish). and generate saliency maps that highlight pieces that are not relevant to the move played by the agent. This is because they use differences in Q(s, a), V (s) or the the L 2 norm of the policy vector between the original and perturbed state to calculate the saliency maps. Therefore, pieces such as the white queen that affect the value estimate of the state are marked salient. In contrast, the saliency map generated by our approach only highlights pieces relevant to the move. Atari To show that our approach generates saliency maps that are more focused than those generated by, we compare the approaches on three Atari games: Breakout, Pong, and Space Invaders. Figures 2, 3, and 4 shows the . Our approach highlights regions of the input image more precisely, while the approach highlights several regions of the input image that are not relevant to explain the action taken by the agent. Go Figure 5 shows a board position in Go. It is black's turn. The four white stones threaten the three black stones that are in one row at the top left corner of the board. To save those three black stones, black looks at the three white stones that are directly below the three black ones. Due to another white stone below the three white stones, the continuous row of three white stones cannot be captured easily. Therefore black moves to place a black stone below that single white stone in an attempt to start capturing the four white stones. It takes the next few turns to surround the structure of four white stones with black ones, thereby saving its pieces. The method described in generates a saliency map that highlights almost all the pieces on the board. Therefore, it reveals little about the pieces that the agent thinks are important. On the other hand, the map produced by highlights only a few pieces. The saliency map generated by our approach correctly highlights the structure of four white stones and the black stones already present around them that may be involved in capturing them. Figure 6: ROC curves comparing approaches on the chess saliency dataset To show that our approach generates saliency maps that provide useful information to humans, we conduct human studies on problem-solving for chess puzzles. We show fifteen chess players (ELO 1600-2000) ten chess puzzles from https://www.chess.com (average difficulty ELO 1800). For each puzzle, we show either the puzzle without a saliency map, or the puzzle with a saliency map generated by our approach,, or. The player is then asked to solve the puzzle. We measure the accuracy (number of puzzles correctly solved) and the average time taken to solve the puzzle, shown in Table 1. The saliency maps generated by our approach are more helpful for humans when solving puzzles than those generated by other approaches. We observed that the saliency maps generated by often confuse humans, because they highlight several pieces unrelated to the tactic. The maps generated by highlight few pieces and therefore are marginally better than showing no saliency maps for solving puzzles. To automatically compare the saliency maps generated by different perturbation-based approaches, we introduce a Chess saliency dataset. The dataset consists of 100 chess puzzles 7. Each puzzle has a single correct move. For each puzzle, we ask three human experts (ELO > 2200) to mark the pieces that are important for playing the correct move. We take a majority vote of the three experts to obtain a list of pieces that are important for the move played in the position. The complete dataset is available in our Github repository 7. We use this dataset to compare our approach to existing approaches ). Each approach generates a list of squares and a score that indicates how salient the piece on the square is for a particular move. We scale the scores between 0 and 1 to generate ROC curves. Figure 6a shows the . Our approach generates saliency maps that are better than existing approaches at identifying chess pieces that humans deem relevant in certain positions. To evaluate the relative importance of the two components in our saliency computation (S[f]; Equation 4), we compute saliency maps and ROC curves using each component individually, i.e. S[f] = ∆p or S[f] = K, and compare harmonic mean to other ways to combine them, i.e. using the average, geometric mean, and minimum of ∆p and K. Figure 6b shows the . Combination Figure 7: Saliency maps generated by our approach that demonstrate common tactical motifs in chess of the two properties via harmonic mean leads to more accurate saliency maps than alternative approaches. In this section, we show how our approach can be used to understand common tactical ideas in chess by interpreting the action of a trained agent. Figure 7 illustrates common tactical positions in chess. The corresponding saliency maps are generated by interpreting the moves played by the Stockfish agent in these positions. In Figure 7a, it is white to move. The surprising Rook x d6 is the move played by Stockfish. Figure 7d shows the saliency map generated by our approach. The map illustrates the key idea in the position. Once black's rook recaptures white's rook, white's bishop pins it to the black king. Therefore, white can increase the number of attackers on the rook. The additional attacker is the pawn on e4 highlighted by the saliency map. In Figure 7b, it is white to move. Stockfish plays Queen x h7. A queen sacrifice! Figure 7e shows the saliency map. The map highlights the white rook and bishop, along with the queen. The key idea is that once black captures the queen with his king (a forced move), then the white rook moves to h5 with checkmate. This checkmate is possible because the white bishop guards the important escape square on g6. The saliency map highlights both pieces. In Figure 7c, it is black to move. Stockfish plays the sacrifice rook x d4. The saliency map in Figure 7f illustrates several key aspects of the position. The black queen and light-colored bishop are threatening mate on g2. The white queen protects g2. The white rook on a5 is unguarded. Therefore, once white recaptures the sacrificed rook with the pawn on c3, black can attack both the white rook and queen with the move bishop to b4. The idea is that the white queen is "overworked" or "overloaded" on d2, having to guard both the g2-pawn and the a5-Rook against black's double attack. We are also interested in evaluating the robustness of the generated saliency maps: is the saliency different if non-salient changes are made to the state? To evaluate the robustness of our approach, we perform two irrelevant perturbations to the positions in the chess saliency dataset. First, we pick a random piece amongst the ones labeled non-salient by human experts in a particular position, and remove it from the board. We repeat this for each puzzle in the dataset to generate a new perturbed saliency dataset. Second, we remove a random piece amongst ones labeled non-salient by our approach for each puzzle, creating another perturbed saliency dataset. In order to evaluate the effect of non-salient perturbations on our generated saliency maps, we compute the AUC values for the generated saliency maps, as above, for these perturbed datasets. Since we remove non-salient pieces, we expect the saliency maps and subsequently AUC value to be similar to the value on the original dataset. For both these perturbations, we get an AUC value of 0.92, same as the value on the non-perturbed dataset, confirming the robustness of our saliency maps to these non-relevant perturbations. Since understanding RL agents is important both for deploying RL agents to the real world and for gaining insights about the tasks, a number of different kinds of interpretations have been introduced. A number of approaches generate natural language explanations to explain RL agents (; ;). They assume access to an exact MDP model and that the policies map from interpretable, high-level state features to actions. More recently, analyze execution traces of an agent to extract explanations. A shortcoming of this approach is that it explains policies in terms of hand-crafted state representations that are semantically meaningful to humans. This is often not practical for board games or Atari games where the agents learn from raw board/visual input. apply t-SNE on the last layer of a deep Q-network (DQN) to cluster states of behavior of the agent. They use Semi-Aggregated Markov Decision Processes (SAMDPs) to approximate the black box RL policies. They use the more interpretable SAMDPs to gain insight into the agent's policy. An issue with the explanations is that they emphasize t-SNE clusters that are difficult to understand for non-experts. To build user trust and increase adoption, it is important that the insight into agent behavior should be in a form that is interpretable to the untrained eye and obtained from the original policy instead of a distilled one. Most relevant to our approach are the visual interpretable explanations of deep networks using saliency maps. Methods for computing saliency can be classified broadly into two categories. Gradient-based methods identify input features that are most salient to the trained DNN by using the gradient to estimate their influence on the output. use gradient magnitude heatmaps, which was expanded upon by more sophisticated methods to address their shortcoming, such as guided backpropagation , excitation backpropagation , DeepLIFT , GradCAM (, and GradCAM++ . Integrate gradients provide two axioms to further define the shortcomings of these approaches: sensitivity (relative to a baseline) and implementation invariance, and use them to derive an approach. Nonetheless, all gradient-based approaches still depend on the shape in the immediate neighborhood of a few points, and conceptually, use perturbations that lack physical meaning, making them difficult to use and vulnerable to adversarial attacks in form of imperceivable noise . Further, they are not applicable to scenarios with black-box access to the agent, and even with white-box access to model internals, they are not applicable when agents are not fully differentiable, such as Stockfish for chess. We are more interested in perturbation-based methods for black-box agents: methods that compute the importance of an input feature by removing, altering, or masking the feature in a domain-aware manner and observing the change in output. It is important to choose a perturbation that removes information without introducing any new information. As a simple example, consider a classifier that predicts'True' if a certain input image contains a bird and'False' otherwise. Removing information from the part of the image which contains the bird should change the classifier's prediction, whereas removing information from other areas should not. Several kinds of perturbations have been explored, e.g.; remove information by replacing a part of the input with a gray square. Note that these approaches are implementation invariant by definition, and are sensitive with respect to the perturbation function. Existing perturbation-based approaches for ), however, by focusing on the complete Q (or V), tend to produce saliency maps that are not specific to the action of interest. Our proposed approach addresses this by measuring the impact only on the action being selected, ing in more focused and useful saliency maps, as we show in our experiments. Saliency maps focus on visualizing the dependence between the input and output to the model, essentially identifying the situation-specific explanation for the decision. Although such local explanations have applications in understanding, debugging, and developing trust with machine learning systems, they do not provide any direct insights regarding the general behavior of the model, or guarantee that the explanation is applicable to a different scenario. Attempts to provide a more general understanding of the model include carefully selecting prototype explanations to show to the user (van der) and crafting explanations that are precise and actionable . We will explore such ideas for the RL setting in future, to provide explanations that accurately characterize the behavior of the policy function, in a precise, testable, and intuive manner. There are a number of limitations of our approach to generating saliency maps in our current implementation. First, we perturb the state by removing information (removing pieces in Chess/Go, blurring pixels in Atari). Therefore, our approach cannot highlight the importance of absence of certain attributes, i.e. saliency of certain positions being empty. In games such as Chess and Go, an empty square or file (collection of empty squares) can often be important for a particular move. Future work will explore perturbation functions that add information to the state (perhaps in the form of adding pieces in Chess/Go). Such functions, along with our approach, can be used to calculate the importance of empty squares. Second, it is possible that perturbations may explore states that lie outside the manifold, i.e. they lead to invalid states. For example, unless explicitly prohibited like we do, our approach will compute the saliency of the king pieces by removing them, which is not allowed in the game, or remove the paddle from Pong. In future, we will explore strategies that take the valid state space into account when computing the saliency. Last we are estimating the saliency of each feature independently, ignoring feature dependencies and correlations, which may lead to incorrect saliency maps. We will investigate approaches that perturb multiple features to quantify the importance of each feature , and combine them with our approach to explaining black-box policy-based agents. We presented a perturbation-based approach that generates more focused saliency maps than existing approaches by balancing two aspects (specificity and relevance) that capture different desired characteristics of saliency. We showed through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that our approach generates saliency maps that are more interpretable for humans than existing approaches. The of our technique show that saliency can provide meaningful insights into a black-box RL agent's behavior. For experiments on Go, we use the pre-trained MiniGo RL agent: https://github.com/ tensorflow/minigo. This agent was trained using the AlphaGo Algorithm. It also adds features and architecture changes from the. For experiments on Atari agents and for generating saliency maps for, we use their code and pre-trained RL agents available at https://github.com/greydanus/ visualize_atari. These agents are trained using the Asynchronous Advantage Actor-Critic Algorithm (A3C) . For generating saliency maps using , we use our implementation. All of our code and more detailed are available in our Github repository: https://github.com/ rl-interpretation/understandingRL. For chess and Go, we perturb the board position by removing one piece at a time. We do not remove a piece if the ing position is illegal. For instance, in chess, we do not remove the king. For Atari, we use the perturbation technique described in. The technique perturbs the input image by adding a Gaussian blur localized around a pixel. The blur is constructed using the Hadamard product to interpolate between the original input image and a Gaussian blur. The saliency maps for Atari agents have been computed on the frames provided by in their code repository. The puzzles for conducting the Chess human studies, creating the Chess Saliency Dataset, and providing illustrative examples have been taken from Lichess: https://database.lichess. org/. The puzzles for illustrative examples on Go have been taken from OnlineGo: https: //online-go.com/puzzles. Figure 8 shows the saliency maps generated by our approach for the top 3 moves in a chess position. The maps highlight the different pieces that are salient for each move. For instance, Figure 8a shows that for the move Qd4, the pawn on g7 is important. This is because the queen move protects the pawn. For the saliency maps in Figures 8b and 8c, the pawn on g7 is not highlighted. To show that our approach generates meaningful saliency maps in Chess for RL agents, we interpret the LeelaZero Deep RL agent https://github.com/leela-zero/leela-zero. Figure 9 shows the . As discussed in Section 1, the saliency maps generated by

Represent the TLDR for finding the corresponding scientific article!

We propose a model-agnostic approach to explain the behaviour of black-box deep RL agents, trained to play Atari and board games, by highlighting relevant features of an input state.

represent the tldr for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.