Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Learning domain-invariant representation is a dominant approach for domain generalization. However, previous methods based on domain invariance overlooked the underlying dependency of classes on domains, which is responsible for the trade-off between classification accuracy and the invariance. This study proposes a novel method {\em adversarial feature learning under accuracy constraint (AFLAC)}, which maximizes domain invariance within a range that does not interfere with accuracy. Empirical validations show that the performance of AFLAC is superior to that of baseline methods, supporting the importance of considering the dependency and the efficacy of the proposed method to overcome the problem. In supervised learning we typically assume that samples are obtained from the same distribution in training and testing; however, because this assumption does not hold in many practical situations it reduces the classification accuracy for the test data BID20. One typical situation is domain generalization (DG) BID1 BID18 BID19 BID2: we have labeled data from several source domains and collectively exploit them such that the trained system generalizes to other unseen, but somewhat similar, target domain(s). This paper considers DG under the situation where domain d and class y labels are statistically dependent owing to some common latent factor z FIG0 -(c)), which we referred to as domainclass dependency. For example, the WISDM Activity Prediction dataset (WISDM, BID10), where y and d correspond to activities and wearable device users, exhibits this dependency because some activities (e.g., jogging) are strenuous to the extent that some unathletic subjects avoided them (data characteristics), or other activities were added only after the study began and the initial subjects could not perform them (data-collection errors). The dependency is common in real-world datasets BID23 and a similar setting has been investigated in domain adaptation (DA) studies, but most prior DG studies overlooked the dependency. Most prior DG methods utilize invariant feature learning (IFL) (e.g.,). IFL attempts to learn feature representation h from input data x which is invariant to d. When source and target domains have some common structure (see,), IFL prevents the classifier from overfitting to source domains FIG0 ). However, under the dependency, merely imposing the domain invariance can adversely affect the classification accuracy as pointed out by BID21 and illustrated in FIG0. Although that trade-off occurs in source domains (because DG uses only source data during optimization), it can also negatively affect the classification performance for target domain(s). For example, if the target domain has characteristics similar (or same as an extreme case) to those of a certain source domain, giving priority to domain invariance obviously interferes with the DG performance (FIG0).In this paper, considering that prioritizing domain invariance under the trade-off can negatively affect the DG performance, we propose a novel method adversarial feature learning under accuracy constraint (AFLAC), which maximizes domain invariance within a range that does not interfere with the classification accuracy FIG0 -(e)) on adversarial training. Specifically, AFLAC is intended to achieve accuracy-constrained domain invariance, which we define as the maximum H(d|h) (H denotes entropy) value under the condition H(y|x) = H(y|h) (h has as much y information as x). Empirical validations show that the performance of AFLAC is superior to that of baseline methods, supporting the importance of considering domain-class dependency and the efficacy of the proposed approach for overcoming the issue. DG has been attracting considerable attention in recent years, and most prior DG methods utilize IFL BID2 BID4. In particular, our proposed method is based on Domain Adversarial Nets (DAN), which was originally invented for DA BID3 and BID21 demonstrated its efficacy in DG. In addition, BID21 intuitively explained the trade-off between classification accuracy and domain invariance, but they did not suggest any solution to the problem except for carefully tuning a weighting parameter. Several studies that address DG without utilizing IFL have been conducted. For example, CCSA BID16, CIDG BID13, and CIDDG BID14 proposed to make use of semantic alignment, which attempts to make latent representation given class label (p(h|y)) identical within source domains. This approach was originally proposed by BID6 in the DA context, but its efficacy to overcome the trade-off problem is not obvious. CrossGrad BID18, which is one of the recent state-of-the-art DG methods, utilizes data augmentation with adversarial examples. However, because the method relies on the assumption that y and d are independent, it might not be directly applicable to our setting. In DA, BID23; BID6 address the situation where p(y) changes across the source and target domains by correcting the change of p(y) using unlabeled target data, which is often accomplished at the cost of classification accuracy for the source domain. However, this approach is not applicable (or necessary) to DG because we are agnostic on target domain(s), and this paper is concerned with the change of p(y) within source domains. Instead, we propose to maximize the classification accuracy for source domains while improving the domain invariance. Here we provide the notion of accuracy-constrained domain invariance, which is the maximum domain invariance within a range that does not interfere with the classification accuracy. The reason for the constraint is that the primary purpose of DG is the classification for unseen domains rather than domain itself, and the improvement of the invariance could detrimentally affect the performance. Theorem 1 Let h = f (x), i.e., h is a deterministic mapping of x with a function f. We define accuracy-constrained domain invariance as the maximum H(d|h) value under the constraint that DISPLAYFORM0 Proof 1 Using the properties of entropy, the following inequation holds: DISPLAYFORM0 By assumption, H(y|x) = H(y|h) = 0, and thus the following inequation holds: DISPLAYFORM1 Thus, the maximum H(d|h) value under the constraints is H(d|y). We propose a novel method named AFLAC, which is designed to achieve accuracy-constraind domain invariance. Formally, we denote f E (x), q M (y|h), and q D (d|h) (E, M, and D are the parameters) as the deterministic encoder, probabilistic model of the label classifier, and that of domain discriminator, respectively. Then, the objective function of AFLAC is described as follows: DISPLAYFORM0 Here γ denotes a weighting parameter. Note that, although we cannot obtain true distribution p(d|y), we can use the maximum likelihood estimator of it when y and d are discrete, as is usual with DG.Here we formally show that AFLAC is intended to achieve H(d|h) = H(d|y) (accuracy-constrained domain invariance) by a Nash equilibrium analysis similar to BID7; BID21. We define D * and M * as the solutions to Eq. 3 and Eq. 4 with fixed E. They obviously satisfy q * D = p(d|h), q * M = p(y|h), respectively. Thus, V in Eq. 3 can be written as follows: DISPLAYFORM1 E *, which we define as the solution to Eq. 5 and in Nash equilibrium, satisfies not only H(y|h) = H(y|x) (optimal classification accuracy) but also E h,y∼p (h,y) [D KL [p(d|y) BMNISTR We created the Biased Rotated MNIST dataset (BMNISTR) by modifying the sample size of the popular benchmark dataset MNISTR BID5, such that the class distribution differed among the domains. In MNISTR, each domain was created by rotating images by 15 degree increments: 0, 15,..., 75 (referred to as M0, ..., M75). We created four variants of MNISTR that have different types of domain-class dependency, referred to as BMNISTR-1 through BMNISTR-3. As shown in TAB0 -left, BMNISTR-1, -2 have similar trends but different degrees of dependency, whereas BMNISTR-1 and BMNISTR-3 differ in terms of their trends. In training, we employed a leave-one-domain-out setting BID5: we trained the models on five of the six domains and tested them using the remaining one. WISDM WISDM contains sensor data of accelerometers of six human activities (walking, jogging, upstairs, downstairs, sitting, and standing) performed by 36 users (domains). WISDM has the dependency for the reason noted in Sec. 1. we randomly selected <10 / 26> and <26 / 10> users as <source / target> domains, and split the source data into training and validation data. We compared AFLAC with the following methods. CNN is a vanilla convolutional networks trained on the aggregation of data from all source domains. DAN BID21 ) is expected to generalize across domains utilizing domain-invariant representation, but it can be affected by the trade-off as pointed out by BID21. CIDDG is our re-implementation of BID14, which is designed to achieve semantic alignment on adversarial training. Additionally, we used AFLAC-Abl, which is a version of AFLAC modified for ablation studies. AFLAC- DISPLAYFORM0 We first investigated the extent to which domain-class dependency affects the performance of domain-invariance-based methods. In TAB0 -right, we compared the mean F-measures for the classes 0 through 4 and classes 5 through 9 in BMNISTR with the target domain M0. Recall that the sample sizes for the classes 0∼4 are variable across domains, whereas the classes 5∼9 have identical sample sizes across domains. The F-measures show that AFLAC outperformed baselines in most dataset-class pairs, which supports that the dependency reduces the performance of IFL methods and that AFLAC can mitigate the problem. Further, the relative improvement of AFLAC to AFLAC-Abl is more significant for the classes 0∼4 than for 5∼9 in BMNISTR-1 and BMNISTR-3, suggesting that AFLAC tends to increase performance more significantly for classes in which the dependency occurs. Moreover, the improvement is more significant in BMNISTR-1 than in BMNISTR-2, suggesting that the stronger the dependency is, the lower the performance of domain-invariance-based methods becomes. Finally, although the dependencies of BMNISTR-1 and BMNISTR-3 have different trends, AFLAC improved the F-measures in both datasets. Next we investigated the relationship between the strength of regularization and performance. c, d) show that the accuracy gaps of AFLAC-Abl and AFLAC increase with strong regularization (such as when γ = 10), suggesting that AFLAC, as it was designed, does not tend to reduce accuracy with strong regularizer, and thus AFLAC is robust toward hyperparameter choice. In this paper, we proposed a novel method AFLAC, which maximizes domain invariance within a range that does not interfere with classification accuracy on adversarial training. Empirical validations show the superior DG performance of AFLAC to the baseline methods, supporting the importance of the domain-class dependency in domain generalization tasks and the efficacy of the proposed method for overcoming the issue.

Address the trade-off caused by the dependency of classes on domains by improving domain adversarial nets

represent this text to find a much much longer article about it with say hundreds-thousands of words

Example:
Provided: "Elastic-InfoGAN is a modification of InfoGAN that learns, without any supervision, disentangled representations in class imbalanced data" Match: "We propose a novel unsupervised generative model, Elastic-InfoGAN, that learns to disentangle object identity from other low-level aspects in class-imbalanced datasets. We first investigate the issues surrounding the assumptions about uniformity made by InfoGAN, and demonstrate its ineffectiveness to properly disentangle object identity in imbalanced data. Our key idea is to make the discovery of the discrete latent factor of variation invariant to identity-preserving transformations in real images, and use that as the signal to learn the latent distribution's parameters. Experiments on both artificial (MNIST) and real-world (YouTube-Faces) datasets demonstrate the effectiveness of our approach in imbalanced data by: (i) better disentanglement of object identity as a latent factor of variation; and (ii) better approximation of class imbalance in the data, as reflected in the learned parameters of the latent distribution. Generative models aim to model the true data distribution, so that fake samples that seemingly belong to the modeled distribution can be generated (; ;). Recent deep neural network based models such as Generative Adversarial Networks (; ;) and Variational Autoencoders have led to promising in generating realistic samples for high-dimensional and complex data such as images. More advanced models show how to discover disentangled representations;;;; ), in which different latent dimensions can be made to represent independent factors of variation (e.g., pose, identity) in the data (e.g., human faces). InfoGAN ) in particular, tries to learn an unsupervised disentangled representation by maximizing the mutual information between the discrete or continuous latent variables and the corresponding generated samples. For discrete latent factors (e.g., digit identities), it assumes that they are uniformly distributed in the data, and approximates them accordingly using a fixed uniform categorical distribution. Although this assumption holds true for many existing benchmark datasets (e.g.,), real-word data often follows a long-tailed distribution and rarely exhibits perfect balance between the categories. Indeed, applying InfoGAN on imbalanced data can in incoherent groupings, since it is forced to discover potentially non-existent factors that are uniformly distributed in the data; see Fig. 1. In this work, we augment InfoGAN to discover disentangled categorical representations from imbalanced data. Our model, Elastic-InfoGAN, makes two modifications to InfoGAN which are simple and intuitive. First, we remodel the way the latent distribution is used to fetch the latent variables; we lift the assumption of any knowledge about class imbalance, where instead of deciding and fixing them beforehand, we treat the class probabilities as learnable parameters of the optimization process. To enable the flow of gradients back to the class probabilities, we employ the Gumbel-Softmax distribution , which acts as a proxy for the categorical distribution, generating differentiable samples having properties similar to that of categorical samples. Second, we enforce our network to assign the same latent category for an image I and its transformed image I, which induces the discovered latent factors to be invariant to identity-preserving transformations like illumination, translation, rotation, and scale changes. Although there are multiple meaningful ways to partition unlabeled data-e.g., with digits, one partitioning could be based Samples generated with an InfoGAN model learned with a fixed uniform categorical distribution Cat(K = 10, p = 0.1) on balanced and imbalanced data, respectively. Each row corresponds to a different learned latent category. (Right): Samples generated with Elastic-InfoGAN using its automatically learned latent categorical distribution. Although InfoGAN discovers digit identities in the balanced data, it produces redundant/incoherent groupings in the imbalanced data. In contrast, our model is able to discover digit identities in the imbalanced data. on identity, whereas another could be based on stroke width-we aim to discover the partitioning that groups objects according to a high-level factor like identity while being invariant to low-level "nuisance" factors like lighting, pose, and scale changes. Such partitionings focusing on object identity are more likely to be useful for downstream visual recognition applications (e.g., semi-supervised object recognition). In sum, our modifications to InfoGAN lead to better disentanglement and categorical grouping of the data (Fig. 1), while at the same time enabling the discovery of the original imbalance through the learned probability parameters of the Gumbel softmax distribution. Importantly, these modifications do not impede InfoGAN's ability to jointly model both continuous and discrete factors in either balanced or imbalanced data scenarios. Our contributions can be summarized as follows: To our knowledge, our work is the first to tackle the problem of unsupervised generative modeling of categorical disentangled representations in imbalanced data. We show qualitatively and quantitatively our superiority in comparison to Info-GAN and other relevant baselines. Our work takes a step forward in the direction of modeling real data distributions, by not only explaining what modes of a factor of variation are present in the data, but also discovering their respective proportions. Disentangled representation learning Learning disentangled representations of the data has a vast literature . InfoGAN ) is one of the most popular unsupervised GAN based disentanglement methods, which learns disentanglement by maximizing the mutual information between the latent codes and generated images. It has shown promising for discovering meaningful latent factors in balanced datasets like MNIST , CelebA , and SVHN . The recent method of JointVAE extends beta-VAE by jointly modeling both continuous and discrete factors, using Gumbel-Softmax sampling. However, both InfoGAN and JointVAE assume uniformly distributed data, and hence fail to be equally effective in imbalanced data, evident by Fig. 1 and our experiments. Our work proposes modifications to InfoGAN to enable it to discover meaningful latent factors in imbalanced data. Learning from imbalanced data Real world data have a long-tailed distribution , which can impede learning, since the model can get biased towards the dominant categories. To alleviate this issue, researchers have proposed re-sampling (; ; ; ;) and class reweighting techniques (; ; ;) to oversample rare classes and down-weight dominant classes. These methods have shown to be effective for the supervised setting, in which the class distributions are known a priori. There are Figure 2: Elastic-InfoGAN takes a sampled categorical code from a Gumbel-Softmax distribution and a noise vector to generate fake samples. Apart from the original InfoGAN ) loss functions, we have two additional constraints: We take real images x and create a transformed version x using identity-preserving operations (e.g., small rotation), and force their inferred latent code distributions to be close; We also constrain their entropy to be low. The use of differentiable latent variables from the Gumbel-Softmax enables gradients to flow back to the class probabilities to update them. also unsupervised clustering methods that deal with imbalanced data in unknown class distributions (e.g., ;). Our model works in the same unsupervised setting; however, unlike these methods, we propose an unsupervised generative model method that learns to disentangle latent categorical factors in imbalanced data. Leveraging data augmentation for unsupervised image grouping Some works (; ; ;) use data augmentation for image transformation invariant unsupervised clustering or representation learning. The main idea is to maximize the mutual information or similarity between the features of an image and its corresponding transformed image. However, unlike our approach, these methods do not target imbalanced data and do not perform generative modeling. Let X = {x 1, x 2, . . ., x N} be a dataset of N unlabeled images from k different classes. No knowledge about the nature of class imbalance is known beforehand. Our goal is twofold: (i) learn a generative model G which can learn to disentangle object category from other aspects (e.g., digits in MNIST , face identity in YouTube-Faces ); (ii) recover the unknown true class imbalance distribution via the generative modeling process. In the following, we first briefly discuss InfoGAN ), which addressed this problem for the balanced setting. We then explain how InfoGAN can be extended to the scenario of imbalanced data. Learning disentangled representations using the GAN framework was introduced in InfoGAN ). The intuition is for generated samples to retain the information about latent variables, and consequently for latent variables to gain control over certain aspects of the generated image. In this way, different types of latent variables (e.g., discrete categorical vs. continuous) can control properties like discrete (e.g., digit identity) or continuous (e.g., digit rotation) variations in the generated images. Formally, InfoGAN does this by maximizing the mutual information between the latent code c and the generated samples G(z, c), where z ∼ P noise (z) and G is the generator network. The mutual information I(c, G(c, z)) can then be used as a regularizer in the standard GAN training objective. Computing I(c, G(c, z)) however, requires P (c|x), which is intractable and hard to compute. The authors circumvent this by using a lower bound of I(c, G(c, z)), which can approximate P (c|x) via a neural network based auxiliary distribution Q(c|x). The training objective hence becomes: Figure 3: Different ways for unsupervised learning based methods to group unlabeled data; based on rotation (left) vs. digit identity (right). Here, we show two different groups for each grouping. where D is the discriminator network, and H(c) is the entropy of the latent code distribution. Training with this objective in latent codes c having control over the different factors of variation in the generated images G(z, c). To model discrete variations in the data, InfoGAN employs nondifferentiable samples from a uniform categorical distribution with fixed class probabilities; i.e., c ∼ Cat(K = k, p = 1/k) where k is the number of discrete categories to be discovered. As shown in Fig. 1, applying InfoGAN to an imbalanced dataset in suboptimal disentanglement, since the uniform prior assumption does not match the actual ground-truth data distribution of the discrete factor (e.g., digit identity). To address this, we propose two augmentations to InfoGAN. The first is to enable learning of the latent distribution's parameters (class probabilities), which requires gradients to be backpropagated through latent code samples c, and the second is to enforce identity-preserving transformation invariance in the learned latent variables so that the ing disentanglement favors groups that coincide with object identities. Learning the prior distribution To learn the prior distribution, we replace the fixed categorical distribution in InfoGAN with the Gumbel-Softmax distribution , which enables sampling of differentiable samples. The continuous Gumbel-Softmax distribution can be smoothly annealed into a categorical distribution. Specifically, if p 1, p 2..., p k are the class probabilities, then sampling of a k-dimensional vector c can be done in a differentiable way: Here g i, g j are samples drawn from Gumbel, and τ (softmax temperature) controls the degree to which samples from Gumbel-Softmax resemble the categorical distribution. Low values of τ make the samples possess properties close to that of a one-hot sample. In theory, InfoGAN's behavior in the class balanced setting (Fig. 1 left) can be replicated in the imbalanced case (where grouping becomes incoherent, Fig. 1 center), by simply replacing the fixed uniform categorical distribution with Gumbel-Softmax with learnable class probabilities p i's; i.e. gradients can flow back to update the class probabilities (which are uniformly initialized) to match the true class imbalance. And once the true imbalance gets reflected in the class probabilities, the possibility of proper categorical disentanglement (Fig. 1 right) becomes feasible. Empirically, however, this ideal behavior is not observed in a consistent manner. As shown in Fig. 3 (left), unsupervised grouping can focus on non-categorical attributes such as rotation of the digit. Although this is one valid way to group unlabeled data, our goal in this work is to prefer groupings that correspond to class identity as in Fig. 3 (right). Learning object identities To capture object identity as the factor of variation, we make another modification to InfoGAN. Specifically, to make the model focus on high level object identity and be invariant to low level factors like rotation, thickness, illumination, etc., we explicitly create these identity-preserving transformations on real images, and enforce the latent prediction Q(c|x) to be invariant to these transformations. Note that such transformations (aka data augmentations) are standard for learning invariant representations for visual recognition tasks. Formally, for any real image x ∼ P data (x), we apply a set of transformations δ to obtain a transformed image x = δ(x). It is important to point out that these transformations are not learned over the optimization process. Instead we use fixed simple transformations which guarantee that the human defined object identity label for the original image x and the transformed image x image remain the same. For example, the digit identity of a'one' from MNIST will remain the same if a transformation of rotation (±10 degree) is applied. Similarly, a face identity will remain the same upon horizontal flipping. We hence formulate our transformation constraint loss function: where d(·) is a distance metric (e.g., cosine distance), and Q(c x |x), Q(c x |x), are the latent code predictions for real image x and transformed image x, respectively. Note that ideally Q(c|x), for either x ∼ P data (x) or x ∼ P g (G), should have low entropy (peaky class distribution) for proper inference about the latent object category. Eq. 2 automatically enforces a peaky class distribution for Q(c|x) for x ∼ P g (G), because the sampled input latent code c from Gumbel-Softmax is peaky. For x ∼ P data (x) though, Eq. 4 alone isn't sufficient as it can be optimized in a sub-optimal manner (e.g., if c x ≈ c x, but both have high entropy). We hence add an additional entropy loss which forces c x and c x to have low entropy (s) class distributions: The losses L trans and L ent, along with Gumble-Softmax, constitute our overall training objective: V Inf oGAN plays the role of generating realistic images and associating the latent variables to correspond to some factor of variation in the data, while the addition of L trans will push the discovered factor of variation to be close to object identity. Finally, L ent's objective is to ensure Q behaves similarly for real and fake image distributions. The latent codes sampled from Gumbel-softmax, generated fake images, and losses operating on fake images are all functions of class probabilities p i's too. Thus, during the minimization phase of Eqn. 6, the gradients are used to optimize the class probabilities along with G and Q in the backward pass. In this section, we perform quantitative and qualitative analyses to demonstrate the advantage of Elastic-InfoGAN in discovering categorical disentanglement for imbalanced datasets. We use: MNIST and YouTube-Faces . MNIST is by default a balanced dataset with 70k images, with a similar number of training samples for each of 10 classes. We artificially introduce imbalance over 50 random splits (max imbalance ratio 10:1 between the largest and smallest class). YouTube-Faces is a real world imbalanced video dataset with varying number of training samples (frames) for the 40 face identity classes (as used in). The smallest/largest class has 53/695 images, with a total of 10,066 tightly-cropped face images. All are reported over the average of: (i) 50 runs (over 50 random imbalances) for MNIST, (ii) 5 runs over the same imbalanced dataset for YouTube-Faces. We use MNIST to provide a proof-of-concept of our approach. For example, one of the ways in which different'ones' in MNIST vary is rotation, which can be used as a factor (as opposed to object identity) to group data in imbalanced cases (recall Fig. 3 left). Thus, using rotation as a transformation in L trans should alleviate this problem. We ultimately care most about the YouTube-Faces since it is more representative of real world data, both in terms of challenging visual variations (e.g., facial pose, scale, expression, and lighting changes) a well as inherent class imbalance. For this reason, the effect of augmentations in L trans will be more reflective of how well our model can work in real world data. We design different baselines to show the importance of having learnable priors for different latent variables and applying our transformation constraints. • Uniform InfoGAN ): This is the original InfoGAN with fixed and uniform categorical distribution. • Ground-truth InfoGAN: This is InfoGAN with a fixed, but imbalanced categorical distribution where the class probabilities reflect the ground-truth class imbalance. • Ground-truth InfoGAN + Transformation constraint: Similar to the previous baseline but with our data transformation constraint (L trans). • Gumbel-softmax: In this case, InfoGAN does not have a fixed prior for the latent variables. Instead, the priors are learned using the Gumbel-softmax technique . • Gumbel-softmax + Transformation constraint: Apart from having a learnable prior we also apply our transformation constraint (L trans). This is a variant of our final approach that does not have the entropy loss (L ent). • Gumbel-softmax + Transformation constraint + Entropy Loss (Elastic-InfoGAN): This is our final model with all the losses, L trans and L ent, in addition to V Inf oGAN (D, G, Q). • JointVAE : We also include this VAE based baseline, which performs joint modeling of disentangled discrete and continuous factors. Our evaluation should capture: how well we learn class-specific disentanglement for the imbalanced dataset, and recover the ground-truth class distribution of the imbalanced dataset. To capture these aspects, we apply three evaluation metrics: • Average Entropy (ENT): Evaluates two properties: (i) whether the images generated for a given categorical code belong to the same ground-truth class i.e., whether the ground-truth class histogram for images generated for each categorical code has a low entropy; (ii) whether each ground-truth class is associated with a single unique categorical code. We generate 1000 images for each of the k latent categorical codes, compute class histograms using a pre-trained classifier 2 to get a k × k matrix (where rows index latent categories and columns index ground-truth categories). We report the average entropy across the rows (tests (i)) and columns (tests (ii)). • Normalized Mutual Information (NMI) : We treat our latent category assignments of the fake images (we generate 1000 fake images for each categorical code) as one clustering, and the category assignments of the fake images by the pre-trained classifier as another clustering. NMI measures the correlation between the two clusterings. The value of NMI will vary between 0 to 1; higher the NMI, stronger the correlation. • Root Mean Square Error (RMSE) between predicted and actual class distributions: measures the accuracy of approximating the true class distribution of the imbalanced dataset. Since the learned latent distribution may not be aligned to the ground-truth distribution (e.g., the first dimension for the learned distribution might capture 9's in MNIST whereas the first dimension for the groundtruth distribution may be for 0's), we need a way to align the two. For this, we use the pre-trained classifier to classify the generated images for a latent variable and assign the variable to the most frequent class. If more than one latent variable is assigned to the same class, then their priors are added before computing its distance with the known prior of the ground-truth class. We first evaluate disentanglement quality as measured by NMI and average entropy (ENT); see Figure 4: Representative image generations on a random imbalanced MNIST split. Each row corresponds to a learned latent variable. Our approach generates inconsistent images in only row 2 whereas Uniform InfoGAN does so in rows 1,2,6,8 and JointVAE does so in rows 3,5,6,7,9,10. particular, our full model obtains significant boosts of 0.101 and 0.104 in NMI, and -0.222 and -0.305 in ENT compared to the Uniform InfoGAN baseline for MNIST and YouTube-Faces, respectively. The boost is even more significant when compared to JointVAE: 0.1977, 0.3380 in NMI, and -0.4658, -0.9963 in ENT for MNIST and YouTube-Faces, respectively. This again is a of the assumption of a uniform categorical prior by JointVAE, along with poorer quality generations. We see that our transformation constraint generally improves the performance for both when the ground-truth prior is known (Ground-truth InfoGAN vs. Ground-truth InfoGAN + Transformation constraint) as well as when the prior is learned (Gumbel-softmax vs. Gumbel-softmax + Transformation constraint). This shows that enforcing the network to learn groupings that are invariant to identity-preserving transformations helps it to learn a disentangled representation in which the latent dimensions correspond more closely to identity-based classes. Also, learning the prior using the Gumbel-softmax leads to better categorical disentanglement than fixed uniform priors, which demonstrates the importance of learning the prior distribution in imbalanced data. Overall, our approach using Gumbel-softmax to learn the latent prior distribution together with our transformation constraint works better than applying them individually, which demonstrates their complementarity. Interestingly, using a fixed ground-truth prior (Ground-truth InfoGAN) does not in better disentanglement than learning the prior (Gumbel-softmax). This requires further investigation, but we hypothesis that having a rigid prior makes optimization more difficult compared to allowing the network to converge to a distribution on its own, as there are multiple losses that need to be simultaneously optimized. Finally, in Table 2, we evaluate how well the Gumbel-softmax can recover the ground-truth prior distribution. For this, we compute the RMSE between the learned prior distribution and ground- truth prior distribution. Our full model (transformation constraint + entropy loss) produces the best estimate of the true class imbalance for both datasets, as evident through lowest RMSE. Our improvement over the Gumbel-Softmax baseline indicates the importance of our tranformation L trans and entropy L ent losses in approximating the class imbalance. We next qualitatively evaluate the disentanglement achieved by our approach. Figs. 4, 5, and 7 show for MNIST and YouTube-Faces. Overall, Elastic-InfoGAN generates more consistent images for each latent code compared to Uniform InfoGAN and JointVAE. For example, in Fig. 4, ElasticInfoGAN only generates inconsistent images in the second row whereas the baseline approaches generate inconsistent images in several rows. Similarly, in Fig. 7, Elastic-InfoGAN generates faces of the same person corresponding to a latent variable more consistently than the baselines. Both Uniform InfoGAN and JointVAE on the other hand tend to mix up identities within the same categorical code because they incorrectly assume a prior uniform distribution. Finally, we demonstrate that Elastic-InfoGAN does not impede modeling of continuous factors in the imbalanced setting. Specifically, one can augment the input with continuous latent codes (e.g. r1, r2 ∼ Unif(-1, 1)) along with the existing categorical and noise vectors. In Fig. 6, we show the of continuous code interpolation; we can see that each of the two continuous codes largely captures a particular continuous factor (stroke width on left, and digit rotation on the right). In this work, we proposed a new unsupervised generative model that learns categorical disentanglement in imbalanced data. Our model learns the class distribution of the imbalanced data and enforces invariance to be learned in the discrete latent variables. Our demonstrate superior performance over alternative baselines. We hope this work will motivate other researchers to pursue this interesting research direction in generative modeling of imbalanced data. For MNIST, we operate on the original 28x28 image size, with 10-dimensional categorical code to represent 10 digit categories. For YouTube-Faces, we crop the faces using bounding box annotations provided, and then resize them to 64x64 resolution, and use a 40-dimensional categorical code to represent 40 face identities (first 40 categories sorted in alphabetical manner), as done in. Pre-trained classification architecture used for evaluation for MNIST: 2 Conv + 2 FC layers, with max pool and ReLU after every convolutional layer. For YouTube-Faces classification, we fine-tune a ResNet-50 network pretrained on VGGFace2, for face recognition. We set λ 1 = 1 (for L 1), λ 2 = 10 (for L trans), and λ 3 = 1 (for L ent). These hyperparameters were chosen to balance the magnitude of the different loss terms. Finally, one behavior we observe is that if the random initialization of class probabilities is too skewed (only few classes have high probability values), then it becomes very difficult for them to get optimized to the ideal state. We hence initialize them with the uniform distribution, which makes training much more stable. We follow the exact architecture as described in InfoGAN ): The generator network G takes as input a 64 dimensional noise vector z ∼ N and 10 dimensional samples from Gumbel-Softmax distribution. The discriminator D and the latent code prediction network Q share most of the layers except the final fully connected layers. Elastic-InfoGAN architecture for YouTube Faces We operate on cropped face images resized to 64x64 resolution. Our architecture is based on the one proposed in StackGANv2 , where we use its 2-stage version for generating 64x64 resolution images. The input is a 100 dimensional noise vector z ∼ N and 40 dimensional samples (c) from the Gumbel-Softmax distribution. There is an initial fully connected layer which maps the input (concatenation of z and c) to an intermediate feature representation. A series of a combination of upsampling + convolutional (interleaved with batch normalization and Gated Linear Units) increase the spatial resolution of the feature representation, starting from 1024 (feature size: 4 x 4 x 1024) channels to 64 (feature size: 64 x 64 x 64) channels. For the first stage, a convolutional network transforms the feature representation into a 3 channel output, while maintaining the spatial resolution; this serves as the fake image from the first stage. The next stage uses the 64 x 64 x 64 resolution features, forwards it through a network containing residual blocks and convolutional layers, while again maintaining the spatial resolution of 64 x 64. For the second stage, again a convolutional layer maps the ing feature into a 64 x 64 resolution fake image, which is the one used by the model for evaluation purposes. The discriminator networks are identical at both stages. It consists of 4 convolutional layers interleaved with batch normalization and leaky ReLU layers, which serve as the common layers for both the D and Q networks. After that, D has one non-shared convolutional layer which maps the feature representation into a scalar value reflecting the real/fake score. For Q, we have a pair of non-shared convolutional layers which map the feature representation into a 40 dimensional latent code prediction. We employ a similar way of training the generative and discriminative modules as described in. We first update the discriminator based on the real/fake adversarial loss. In the next step, after computing the remaining losses (mutual information + L trans + L ent), we update the generator (G) + latent code predictor (Q) + latent distribution parameters at once. Our optimization process alternates between these two phases. For MNIST, we train all baselines for 200 epochs, with a batch size of 64. For YouTube-Faces, we train until convergence, as measured via qualitative realism of the generated images. We use a batch size of 50. τ = 0.1 when used for sampling from Gumbel-Softmax, which in samples having very low entropy (very close to one hot vectors from a categorical distribution). Here we describe the exact class imbalance used in our experiments. For MNIST, we include below the 50 random imbalances created. For YouTube-Faces, we include the true ground truth class imbalance in the first 40 categories. The imbalances reflect the class frequency. A.2.1 MNIST 147, 0.037, 0.033, 0.143, 0.136, 0.114, 0.057, 0.112, 0.143, 0.078 • 0.061, 0.152, 0.025, 0.19, 0.12, 0.036, 0.092, 0.185, 0.075, 0.064 • 0.173, 0.09, 0.109, 0.145, 0.056, 0.114, 0.075, 0.03, 0.093, 0.116 • 0.079, 0.061, 0.033, 0.139, 0.145, 0.135, 0.057, 0.062, 0.169, 0.121 • 0.053, 0.028, 0.111, 0.142, 0.13, 0.121, 0.107, 0.066, 0.125, 0.118 • 0.072, 0.148, 0.092, 0.081, 0.119, 0.172, 0.05, 0.109, 0.085, 0.073 • 0.084, 0.143, 0.07, 0.082, 0.059, 0.163, 0.156, 0.063, 0.074, 0.105 • 0.062, 0.073, 0.065, 0.183, 0.099, 0.08, 0.05, 0.16, 0.052, 0.177 • 0.139, 0.113, 0.074, 0.06, 0.068, 0.133, 0.142, 0.13, 0.112, 0.03 • 0.046, 0.128, 0.059, 0.112, 0.135, 0.164, 0.142, 0.125, 0.051, 0.037 • 0.107, 0.057, 0.154, 0.122, 0.05, 0.111, 0.032, 0.044, 0.136, 0.187 • 0.129, 0.1, 0.039, 0.112, 0.119, 0.095, 0.047, 0.14, 0.156, 0.064 • 0.146, 0.08, 0.06, 0.072, 0.051, 0.119, 0.176, 0.11, 0.158, 0.028 A.3 DISCUSSION ABOUT EVALUATING PREDICTED CLASS IMBALANCE IN SEC. 4.2 To measure the ability of a generative model to approximate the class imbalance present in the data, we derive a metric in Section 4.2 of the main paper, the of which are presented in Table 2. Even though we do get better as measured by RMSE between the approximated and the original imbalance distribution, we would like to discuss certain flaws associated with this metric. In its current form, we compute the class histogram (using the pre-trained classifier, which classifies each fake image into one of the ground-truth categories) for a latent code and associate the latent code to the most frequent class. If multiple latent codes get associated to the same ground-truth class, there will be ground-truth classes for which the predicted class probability will be zero. This is rarely an issue for MNIST, as it only has 10 ground-truth classes, and thus in most cases both our method and the baselines assign each latent code to a unique ground-truth class. However, for YouTube-Faces, after associating latent codes to the ground truth categories in this manner, roughly 10-13 ground-truth classes (out of 40) get associated with 0 probability for both our approach and the baselines (due to multiple latent codes being associated to the same majority ground-truth class). Our metric therefore may be too strict, especially for difficult settings with many confusing groundtruth categories. The tricky part about evaluating how well the model is approximating the class imbalance is that there are two key aspects that need to be simultaneously measured. Specifically, not only should (i) the raw probability values discovered match the ground-truth class imbalance distribution, but (ii) the class probabilities approximated by the latent codes must correspond to the correct ground-truth classes. For example, if the original data had 80% samples from class A and 20% from class B, the generative model should not only estimate the imbalance as 80%-20%, but the model must associate 80% to class A and 20% to class B (instead of 80% to class B and 20% to class A). Another way to evaluate whether a model is capturing the ground-truth class imbalance could be the FID score, but it's worth noting that a method can still have a good FID score without disentangling the different factors of variations. Given the limitation with our metric on YouTube-Faces, we have also measured the min/max of predicted prior values. For YouTube-Faces, the min/max of predicted and ground-truth priors are: Gumbel-Softmax: Min 2.76748415e-05, Max: 0.0819286481; Ours without L ent: Min 0.00211485, Max: 0.06152404; Ours complete: Min 0.00336615, Max: 0.06798439; and GroundTruth: Min 0.005265, Max: 0.069044. Our full method's min/max more closely matches that of the ground-truth, and the overall ordering of the methods follows that of Table 2 using our RMSE based metric. In sum, we have made an effort to evaluate accurate class imbalance prediction in multiple ways, but it is important to note that this is an area which calls for better metrics to evaluate the model's ability to approximate the class imbalance distribution."

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Recent advances in deep generative models have lead to remarkable progress in synthesizing high quality images. Following their successful application in image processing and representation learning, an important next step is to consider videos. Learning generative models of video is a much harder task, requiring a model to capture the temporal dynamics of a scene, in addition to the visual presentation of objects. While recent generative models of video have had some success, current progress is hampered by the lack of qualitative metrics that consider visual quality, temporal coherence, and diversity of samples. To this extent we propose Fréchet Video Distance (FVD), a new metric for generative models of video based on FID. We contribute a large-scale human study, which confirms that FVD correlates well with qualitative human judgment of generated videos. Recent advances in deep generative models have lead to remarkable success in synthesizing highquality images . A natural next challenge is to consider video generation. This is a much harder task, requiring a model to capture the temporal dynamics of a scene, in addition to the visual presentation of objects. Generative models of video will enable many applications, including missing-frame prediction , improved instance segmentation , or complex (relational) reasoning tasks by conducting inference .While great progress has been made in recent years, video generation models are still in their infancy, and generally unable to synthesize more than a few seconds of video . Learning a good dynamics model remains a major challenge in generating real world videos. However, in order to qualitatively measure progress in synthesizing videos, we require metrics that consider visual quality, temporal coherence, and diversity of generated samples. We contribute Fréchet Video Distance (FVD), a new metric for generative models of video. FVD builds on the principles underlying Fréchet Inception Distance (FID;), which has been successfully applied to images. We introduce a feature representation that captures the temporal coherence of the content of a video, in addition to the quality of each frame. Unlike popular * Both authors contributed equally to this work while interning at Google Brain. Figure 1: Generated videos by various models ranked according to FVD (lower is better). metrics such as Peak Signal to Noise Ratio (PSNR) or the Structural Similarity (SSIM;) index, FVD considers a distribution over videos, thereby avoiding the drawbacks of framelevel metrics . We contribute extensive experiments to evaluate FVD, including a large-scale human study which confirms that FVD coincides well with qualitative human judgment of generated videos. A generative model of videos must capture the underlying data distribution with which the observed data was generated. The distance between the real world data distribution P R and the distribution defined by the generative model P G is a natural evaluation metric. An analytic expression of either distribution is usually unavailable, which prohibits straightforward application of many common distance functions. For example, the popular Fréchet Distance (or 2-Wasserstein distance) between P R and P G is defined by: DISPLAYFORM0 2 where the minimization is over all random variables X and Y with distributions P R and P G respectively. This expression is difficult to solve for the general case, but has a closed form solution when when P R and P G are multivariate Gaussians : DISPLAYFORM1 where µ R and µ G are the means and Σ R and Σ G are the co-variance matrices of P R and P G. A multivariate Gaussian is seldom an accurate representation of the underlying data distribution, but when using a suitable feature space, this is a reasonable approximation. For distributions over real world images, used a learned feature embedding to calculate the distance between P R and P G as follows: First samples from P R and P G are fed through an Inception network network that was trained on ImageNet and their feature representations (activations) in one of the hidden layers are recorded. Then the Fréchet Inception Distance (FID;) is computed via Eq. 1 using the means and covariances obtained from the recorded responses of the real, and generated samples. The feature representation learned by the pre-trained neural network greatly affects the quality of the metric. When training on ImageNet, the learned features expose information useful in reasoning about objects in images, whereas other information content may be suppressed. Likewise, different layers of the network encode features at different abstraction levels. In order to obtain a suitable feature representation for videos we require a pre-trained network that considers the temporal coherence of the visual content across a sequence of frames, in addition to its visual presentation at any given point in time. In this work we investigate several variations of a pre-trained Inflated 3D Convnet (I3D;), and name the ing metric the Fréchet Video Distance (FVD) 1 2.The I3D network generalizes the Inception architecture to sequential data, and is trained to perform action-recognition on the Kinetics data set consisting of human-centered YouTube videos. Action recognition requires visual context and temporal evolution to be considered simultaneously, and I3D has been shown to excel at this task. We explore two different feature representations (logits, avg. pool) learned by I3D networks pre-trained on Kinetics-400, and Kinetics-600.A potential downside in using Eq. 1 is the potentially large error in estimating Gaussian distributions over the learned feature space. Bińkowski et al. proposed to use the Maximum Mean Discrepancy (MMD;) as an alternative in the case of images, and we will explore this variation in the context of videos as well. MMD is a kernel-based approach, which provides a means to calculate the distance between two empirical distributions without assuming a particular form. Bińkowski et al. proposed to use a polynomial kernel k(a, b):= a T b + 1 3, which we will apply to the learned features of the I3D network to obtain the Kernel Video Distance (KVD).1 A similar adaptation of FID was used by to evaluate their vid2vid model. Here we introduce FVD as a general metric for videos and an focus on an extensive empirical study.2 Code to compute FVD is available at https://git.io/fpuEH. In the following we present of a noisy study, and human study of FVD. Additional experiments that analyze its sensitivity to sample size, and resolution are available in Appendix B & D. We test how sensitive FVD is to basic distortions by adding noise to the real videos. We consider static noise added to individual frames, and temporal noise, which distorts the entire sequence of frames. We applied these distortions (details in Appendix A) at up to six different intensities, and computed the FVD and KVD between videos from the BAIR , Kinetics-400 and HMDB51 data sets and their noisy counterparts. As potential embeddings, we considered the top-most pooling layer, and the logits layer of the I3D model pre-trained on the Kinetics-400 data set, as well as the same layers in a variant of the I3D model pre-trained on the extended Kinetics-600 data set. As a baseline, we compared to a naive extension of FID for videos in which the Inception network (pre-trained on ImageNet) is evaluated for each frame individually, and the ing embeddings (or their pair-wise differences) are averaged to obtain a single embedding for each video. This "FID" score is then computed according to Eq. 1.All variants were able to detect the various injected distortions to some degree, with the pre-trained Inception network generally being inferior at detecting temporal distortions as was expected. FIG0 shows that the logits layer of the I3D model pre-trained on Kinetics-400 has the best average rank correlation with the sequence of noise intensities. An overview of its scores on the noise experiments can be seen in FIG2. Table 1: Agreement of metrics with human judgment when considering models with a fixed value for a given metric (eq.), or with spread values over a wide range (spr.). One important criterion for the performance of generative models is the visual fidelity of the samples as judged by human observers , as a metric for generative models must ultimately correlate well with human judgment. Thus we trained several conditional video generation models, and asked human raters to compare the quality of the generated videos in different scenarios. Results The of the human evaluation studies can be seen in Table 1, and additional for KVD and Avg. FID in Appendix C. We find that FVD is the superior choice compared to all other metrics tested. The obtained for eq. FVD and spr. FVD are of key importance as they determine how users will experience FVD in practice. From the spr. FVD column we can conclude that no other metric can improve upon the ranking induced by FVD, and the eq. FVD column tells us that no other metric can reliably distinguish between good models that are equal in terms of FVD.On the other hand, FVD is able to distinguish models when other metrics can not (eq. SSIM, eq. PSNR), agreeing well with human judgment (74.9 %, and 81.0 % agreement respectively). Likewise FVD consistently improves on the ranking induced by other metrics (spr. SSIM, spr. PSNR), even though these scenarios are clearly favorable for the metric under consideration. We introduced the Fréchet Video Distance (FVD), a new evaluation metric for generative models of video, and an important step towards better evaluation of models for video generation. Our experiments confirm that FVD is accurate in evaluating videos that were modified to include static noise, and temporal noise. More importantly, a large scale human study among generated videos from several recent generative models reveals that FVD consistently outperforms SSIM and PSNR in agreeing with human judgment. To test whether FVD can detect static noise we added one of the following distortions to each frame in a sequence of video frames: a black rectangle drawn at a random location in the frame, Gaussian blur, which applies a Gaussian smoothing kernel to the frame, Gaussian noise, which interpolates between the observed frame and standard Gaussian noise, and Salt & Pepper noise, which sets each pixel in the frame to either black or white with a fixed probability. Temporal noise was injected by locally swapping a number of randomly chosen frames with its neighbor in the sequence globally swapping a number of randomly chosen pairs of frames selected across the whole sequence, interleaving the sequence of frames corresponding to multiple different videos to obtain new videos, and by switching from one video to another video after a number of frames to obtain new videos. We applied these distortions at up to six different intensities that are unique to each type, e.g. related to the size of the black rectangle, the number of swaps to perform, or the number of videos to interleave. We conduct the noise study on HMDB (Correlation of Noise Intensity and Metric FVD kinetics400 logits FVD kinetics600 logits FVD kinetics400 pool FVD kinetics600 pool KVD kinetics400 logits KVD kinetics600 logits KVD kinetics400 pool KVD kinetics600 pool FID average over frames FID average over frame-diffs We consider the accuracy with which FVD is able to calculate the true underlying distance between a distribution of generated videos and a target distribution. To calculate the FVD according to Eq. 1 we need to estimate µ R, µ G and Σ R, Σ G from the available samples. The larger the sample size, the better these estimates will be, and the better FVD will reflect the true underlying distance between the distributions. For an accurate generative model these distributions will typically be fairly close, and the noise from the estimation process will primarily affect our . This effect has been wellstudied for FID (; Bińkowski et al., 2018), and is depicted for FVD in FIG3. It can be seen that even when the underlying distributions are identical, FVD will typically be larger than zero because our estimates of the parameters µ R, µ G, Σ R and Σ G are noisy. It can also be seen that for a fixed number of samples the standard errors (measured over 50 tries) are small, and an accurate comparison can be made. Hence, it is critical that in comparing FVD values across models, one uses the same sample size 3.2 5 2 6 2 7 2 8 2 9 2 10 2 11 2 12Number of ) on the BAIR data set. Using a wide range of possible hyperparameter settings, and by including model parameters at various stages of training we obtain over 3 000 different models. Generated videos are obtained by combining 2 frames of context with the proceeding 14 output frames. Following prior work we obtain the PSNR and SSIM scores by generating 100 videos for each input context (conditioning frames) and returning the best frame-averaged value among these videos. We consider 256 video sequences (unseen by the model) to estimate the target distribution when computing FVD.We conduct several human studies based on different subsets of the trained models. In particular, we select models according to two different scenarios:One Metric Equal We consider models that are indistinguishable according to a single metric, and evaluate to what degree human raters and other competing metrics are able to distinguish these models in terms of the quality of the generated videos. We choose 10 models having roughly equal values for a given metric that are close to the best quartile of the overall distribution of that metric, i.e., the models were chosen such that they are worse than 25 % of the remaining models and better than 75 % of the remaining models as determined by the metric under consideration. We were able to choose models whose values where identical up to the first 4-5 significant digits for each metric. One Metric Spread In a second setting we consider to what degree models having very different scores for a given metric, coincide with the subjective quality of their generated videos as judged by humans. We choose 10 models which were equidistant between the 10 % and 90 % percentile of the overall distribution of that metric. In this case, there should be clear differences in terms of the quality of the generated videos among the models under consideration (provided that the metric is accurate), suggesting high agreement with human judgment for the metric under consideration in comparison to competing metrics. For the human evaluation, we used 3 generated videos from each selected model. Human raters would be shown a video from two models, and then asked to identify which of the two looked better, or alternatively report that their quality was indistinguishable. Each pair of compared videos was shown to up to 3 independent raters, where the third rater was only asked if the first two raters disagreed. The raters were given no prior indication about which video was thought to be better. We calculated the correspondence between these human ratings and the ratings determined by the various metrics under consideration. Table 3: Agreement of metrics with human judgment when considering models with a fixed value for a given metric (eq.), or with spread values over a wide range (spr.). FVD is superior at judging generated videos based on subjective quality. Table 3 contains the of the human evaluation for the KVD and Avg. FID metrics in addition to the for all other metrics from the main paper. Avg. FID is computed by averaging the Inception embedding for each frame, before computing the Fréchet distance.• Avg. FID We find that Avg. FID performs markedly worse compared to FVD in most scenarios, except on spr. SSIM, where it achieves slightly better performance. It suggests that it is preferential to judge the wide range of videos (sampled from each decile as determined by SSIM) based primarily on frame-level quality. On the other hand, when considering videos of similar quality in eq. SSIM, we find that judging based on temporal coherence (in addition to frame-level quality) is beneficial and Avg. FID performs worse.• KVD We find that KVD is highly correlated with FVD (spearman: 0.9), although in most scenarios it performs slightly worse than FVD in terms of agreement with human judgment. In general we may conclude from Table 3 that FVD is the superior choice compared to all other metrics tested. The obtained for eq. FVD and spr. FVD are of key importance as they determine how users will experience FVD in practice. From the spr. FVD column we can conclude that no other metric can improve upon the ranking induced by FVD, and the eq. FVD column tells us that no other metric can reliably distinguish between good models that are equal in terms of FVD. Table 3 also reports the agreement among raters. These are computed as the fraction of the comparisons in which the first two raters agreed for a given video pair, averaged across all comparisons to obtain the final percentage. It can be seen that in most cases the raters are confident in comparing generated videos. While the in Appendix B demonstrates that FVD are highly reproducible for a fixed sample size, it does not consider to what degree small differences in FVD can be considered meaningful. To answer this question, human raters were asked to compare videos generated by a randomly chosen model having an FVD of 200 / 400 (base200 / base400) and generated videos by models that were 10, 20, 50, 100, 200, 300, 400, and 500 FVD points worse. In each case we selected 5 models from the models available at these FVD scores and generated 3 videos for each model, ing in a total of 1 800 comparisons. For a given video comparison, raters were asked to decide which of the two videos looked better, or if they were of similar quality. For each of these pairs, we asked up to 3 human raters for their opinion. In Figure 5 it can be seen that when the difference in FVD is smaller than 50, the agreement with human raters is close to random (but never worse), and increases rapidly once two models are more than 50 FVD points apart. Hence, it can be concluded that differences of 50 FVD or more typically correspond to differences in the quality of the generated videos that can be perceived by humans. Figure 5: Fraction of human raters that agree with FVD on which of two models is better, as a function of the difference in FVD between the models. Error bars are standard errors, and raters deciding that video pairs are of similar quality are counted as not agreeing with FVD.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose FVD: a new metric for generative models of video based on FID. A large-scale human study confirms that FVD correlates well with qualitative human judgment of generated videos.

represent the short summary for finding the corresponding scientific article (long):

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.