Question: Represent the article for finding a single-sentence TLDR

Question: Hyperparameter optimization can be formulated as a bilevel optimization problem, where the optimal parameters on the training set depend on the hyperparameters. We aim to adapt regularization hyperparameters for neural networks by fitting compact approximations to the best-response function, which maps hyperparameters to optimal weights and biases. We show how to construct scalable best-response approximations for neural networks by modeling the best-response as a single network whose hidden units are gated conditionally on the regularizer. We justify this approximation by showing the exact best-response for a shallow linear network with L2-regularized Jacobian can be represented by a similar gating mechanism. We fit this model using a gradient-based hyperparameter optimization algorithm which alternates between approximating the best-response around the current hyperparameters and optimizing the hyperparameters using the approximate best-response function. Unlike other gradient-based approaches, we do not require differentiating the training loss with respect to the hyperparameters, allowing us to tune discrete hyperparameters, data augmentation hyperparameters, and dropout probabilities. Because the hyperparameters are adapted online, our approach discovers hyperparameter schedules that can outperform fixed hyperparameter values. Empirically, our approach outperforms competing hyperparameter optimization methods on large-scale deep learning problems. We call our networks, which update their own hyperparameters online during training, Self-Tuning Networks (STNs). Regularization hyperparameters such as weight decay, data augmentation, and dropout are crucial to the generalization of neural networks, but are difficult to tune. Popular approaches to hyperparameter optimization include grid search, random search BID3, and Bayesian optimization . These approaches work well with low-dimensional hyperparameter spaces and ample computational resources; however, they pose hyperparameter optimization as a black-box optimization problem, ignoring structure which can be exploited for faster convergence, and require many training runs. We can formulate hyperparameter optimization as a bilevel optimization problem. Let w denote parameters (e.g. weights and biases) and λ denote hyperparameters (e.g. dropout probability). Let L T and L V be functions mapping parameters and hyperparameters to training and validation losses, respectively. We aim to solve 1: DISPLAYFORM0 Substituting the best-response function w * (λ) = arg min w L T (λ, w) gives a single-level problem: DISPLAYFORM1 If the best-response w * is known, the validation loss can be minimized directly by gradient descent using Equation 2, offering dramatic speed-ups over black-box methods. However, as the solution to a high-dimensional optimization problem, it is difficult to compute w * even approximately. , we propose to approximate the best-response w * directly with a parametric functionŵ φ. We jointly optimize φ and λ, first updating φ so thatŵ φ ≈ w * in a neighborhood around the current hyperparameters, then updating λ by usingŵ φ as a proxy for w * in Eq. 2: DISPLAYFORM2 Finding a scalable approximationŵ φ when w represents the weights of a neural network is a significant challenge, as even simple implementations entail significant memory overhead. We show how to construct a compact approximation by modelling the best-response of each row in a layer's weight matrix/bias as a rank-one affine transformation of the hyperparameters. We show that this can be interpreted as computing the activations of a base network in the usual fashion, plus a correction term dependent on the hyperparameters. We justify this approximation by showing the exact best-response for a shallow linear network with L 2 -regularized Jacobian follows a similar structure. We call our proposed networks Self-Tuning Networks (STNs) since they update their own hyperparameters online during training. STNs enjoy many advantages over other hyperparameter optimization methods. First, they are easy to implement by replacing existing modules in deep learning libraries with "hyper" counterparts which accept an additional vector of hyperparameters as input 2. Second, because the hyperparameters are adapted online, we ensure that computational effort expended to fit φ around previous hyperparameters is not wasted. In addition, this online adaption yields hyperparameter schedules which we find empirically to outperform fixed hyperparameter settings. Finally, the STN training algorithm does not require differentiating the training loss with respect to the hyperparameters, unlike other gradient-based approaches , allowing us to tune discrete hyperparameters, such as the number of holes to cut out of an image BID12, data-augmentation hyperparameters, and discrete-noise dropout parameters. Empirically, we evaluate the performance of STNs on large-scale deep-learning problems with the Penn Treebank and CIFAR-10 datasets , and find that they substantially outperform baseline methods. A bilevel optimization problem consists of two sub-problems called the upper-level and lower-level problems, where the upper-level problem must be solved subject to optimality of the lower-level problem. Minimax problems are an example of bilevel programs where the upper-level objective equals the negative lower-level objective. Bilevel programs were first studied in economics to model leader/follower firm dynamics and have since found uses in various fields (see BID10 for an overview). In machine learning, many problems can be formulated as bilevel programs, including hyperparameter optimization, GAN training , meta-learning, and neural architecture search BID18.Even if all objectives and constraints are linear, bilevel problems are strongly NP-hard . Due to the difficulty of obtaining exact solutions, most work has focused on restricted settings, considering linear, quadratic, and convex functions. In contrast, we focus on obtaining local solutions in the nonconvex, differentiable, and unconstrained setting. Let F, f: R n × R m → R denote the upper-and lower-level objectives (e.g., L V and L T) and λ ∈ R n, w ∈ R m denote the upper-and lower-level parameters. We aim to solve: DISPLAYFORM0 subject to w ∈ arg min DISPLAYFORM1 It is desirable to design a gradient-based algorithm for solving Problem 4, since using gradient information provides drastic speed-ups over black-box optimization methods . The simplest method is simultaneous gradient descent, which updates λ using ∂F /∂λ and w using ∂f /∂w. However, simultaneous gradient descent often gives incorrect solutions as it fails to account for the dependence of w on λ. Consider the relatively common situation where F doesn't depend directly on λ, so that ∂F /∂λ ≡ 0 and hence λ is never updated. A more principled approach to solving Problem 4 is to use the best-response function . Assume the lower-level Problem 4b has a unique optimum w * (λ) for each λ. Substituting the best-response function w * converts Problem 4 into a single-level problem: DISPLAYFORM0 If w * is differentiable, we can minimize Eq. 5 using gradient descent on F * with respect to λ. This method requires a unique optimum w * (λ) for Problem 4b for each λ and differentiability of w *. In general, these conditions are difficult to verify. We give sufficient conditions for them to hold in a neighborhood of a point (λ 0, w 0) where w 0 solves Problem 4b given λ 0. Lemma 1. Let w 0 solve Problem 4b for λ 0. Suppose f is C 2 in a neighborhood of (λ 0, w 0) and the Hessian ∂ 2 f /∂w 2 (λ 0, w 0) is positive definite. Then for some neighborhood U of λ 0, there exists a continuously differentiable function w *: U → R m such that w * (λ) is the unique solution to Problem 4b for each λ ∈ U and w * (λ 0) = w 0.Proof. See Appendix B.1.The gradient of F * decomposes into two terms, which we term the direct gradient and the response gradient. The direct gradient captures the direct reliance of the upper-level objective on λ, while the response gradient captures how the lower-level parameter responds to changes in the upper-level parameter: DISPLAYFORM1 Even if ∂F /∂λ ≡ 0 and simultaneous gradient descent is possible, including the response gradient can stabilize optimization by converting the bilevel problem into a single-level one, as noted by for GAN optimization. Conversion to a single-level problem ensures that the gradient vector field is conservative, avoiding pathological issues described by. In general, the solution to Problem 4b is a set, but assuming uniqueness of a solution and differentiability of w * can yield fruitful algorithms in practice. In fact, gradient-based hyperparameter optimization methods can often be interpreted as approximating either the best-response w * or its Jacobian ∂w * /∂λ, as detailed in Section 5. However, these approaches can be computationally expensive and often struggle with discrete hyperparameters and stochastic hyperparameters like dropout probabilities, since they require differentiating the training loss with respect to the hyperparameters. Promising approaches to approximate w * directly were proposed by , and are detailed below.1. Global Approximation. The first algorithm proposed by approximates w * as a differentiable functionŵ φ with parameters φ. If w represents neural net weights, then the mappingŵ φ is a hypernetwork . If the distribution p(λ) is fixed, then gradient descent with respect to φ minimizes: DISPLAYFORM0 If support(p) is broad andŵ φ is sufficiently flexible, thenŵ φ can be used as a proxy for w * in Problem 5, ing in the following objective: min DISPLAYFORM1 2. Local Approximation. In practice,ŵ φ is usually insufficiently flexible to model w * on support(p). The second algorithm of locally approximates w * in a neighborhood around the current upper-level parameter λ. They set p(|σ) to a factorized Gaussian noise distribution with a fixed scale parameter σ ∈ R n +, and found φ by minimizing the objective: DISPLAYFORM2 Intuitively, the upper-level parameter λ is perturbed by a small amount, so the lower-level parameter learns how to respond. An alternating gradient descent scheme is used, where φ is updated to minimize equation 9 and λ is updated to minimize equation 8. This approach worked for problems using L 2 regularization on MNIST . However, it is unclear if the approach works with different regularizers or scales to larger problems. It requiresŵ φ, which is a priori unwieldy for high dimensional w. It is also unclear how to set σ, which defines the size of the neighborhood on which φ is trained, or if the approach can be adapted to discrete and stochastic hyperparameters. In this section, we first construct a best-response approximationŵ φ that is memory efficient and scales to large neural networks. We justify this approximation through analysis of simpler situations. Then, we describe a method to automatically adjust the scale of the neighborhood φ is trained on. Finally, we formally describe our algorithm and discuss how it easily handles discrete and stochastic hyperparameters. We call the ing networks, which update their own hyperparameters online during training, Self-Tuning Networks (STNs). We propose to approximate the best-response for a given layer's weight matrix W ∈ R Dout×Din and bias b ∈ R Dout as an affine transformation of the hyperparameters λ 3: DISPLAYFORM0 Here, indicates elementwise multiplication and row indicates row-wise rescaling. This architecture computes the usual elementary weight/bias, plus an additional weight/bias which has been scaled by a linear transformation of the hyperparameters. Alternatively, it can be interpreted as directly operating on the pre-activations of the layer, adding a correction to the usual pre-activation to account for the hyperparameters: DISPLAYFORM1 This best-response architecture is tractable to compute and memory-efficient: it requires D out (2D in + n) parameters to representŴ φ and D out (2 + n) parameters to representb φ, where n is the number of hyperparameters. Furthermore, it enables parallelism: since the predictions can be computed by transforming the pre-activations (Equation 11), the hyperparameters for different examples in a batch can be perturbed independently, improving sample efficiency. In practice, the approximation can be implemented by simply replacing existing modules in deep learning libraries with "hyper" counterparts which accept an additional vector of hyperparameters as input 4. Given that the best-response function is a mapping from R n to the high-dimensional weight space R m, why should we expect to be able to represent it compactly? And why in particular would equation 10 be a reasonable approximation? In this section, we exhibit a model whose best-response function can be represented exactly using a minor variant of equation 10: a linear network with Jacobian norm regularization. In particular, the best-response takes the form of a network whose hidden units are modulated conditionally on the hyperparameters. Consider using a 2-layer linear network with weights w = (Q, s) ∈ R D×D × R D to predict targets t ∈ R from inputs x ∈ R D: a(x; w) = Qx, y(x; w) = s a(x; w)Suppose we use a squared-error loss regularized with an L 2 penalty on the Jacobian ∂y /∂x, where the penalty weight λ lies in R and is mapped using exp to lie R +: DISPLAYFORM0 Theorem 2. Let w 0 = (Q 0, s 0), where Q 0 is the change-of-basis matrix to the principal components of the data matrix and s 0 solves the unregularized version of Problem 13 given DISPLAYFORM1 where σ is the sigmoid function. Proof. See Appendix B.2.Observe that y(x; w * (λ)) can be implemented as a regular network with weights w 0 = (Q 0, s 0) with an additional sigmoidal gating of its hidden units a(x; w * (λ)): DISPLAYFORM2 This architecture is shown in FIG0. Inspired by this example, we use a similar gating of the hidden units to approximate the best-response for deep, nonlinear networks. The sigmoidal gating architecture of the preceding section can be further simplified if one only needs to approximate the best-response function for a small range of hyperparameter values. In particular, for a narrow enough hyperparameter distribution, a smooth best-response function can be approximated by an affine function (i.e. its first-order Taylor approximation). Hence, we replace the sigmoidal gating with linear gating, in order that the weights be affine in the hyperparameters. The following theorem shows that, for quadratic lower-level objectives, using an affine approximation to the best-response function and minimizing E ∼p(|σ) [f (λ +,ŵ φ (λ +))] yields the correct best-response Jacobian, thus ensuring gradient descent on the approximate objective F (λ,ŵ φ (λ)) converges to a local optimum: DISPLAYFORM0 is Gaussian with mean 0 and variance DISPLAYFORM1 Proof. See Appendix B.3. The effect of the sampled neighborhood. Left: If the sampled neighborhood is too small (e.g., a point mass) the approximation learned will only match the exact best-response at the current hyperparameter, with no guarantee that its gradient matches that of the best-response. Middle: If the sampled neighborhood is not too small or too wide, the gradient of the approximation will match that of the best-response. Right: If the sampled neighborhood is too wide, the approximation will be insufficiently flexible to model the best-response, and again the gradients will not match. The entries of σ control the scale of the hyperparameter distribution on which φ is trained. If the entries are too large, thenŵ φ will not be flexible enough to capture the best-response over the samples. However, the entries must remain large enough to forceŵ φ to capture the shape locally around the current hyperparameter values. We illustrate this in FIG1. As the smoothness of the loss landscape changes during training, it may be beneficial to vary σ. To address these issues, we propose adjusting σ during training based on the sensitivity of the upperlevel objective to the sampled hyperparameters. We include an entropy term weighted by τ ∈ R + which acts to enlarge the entries of σ. The ing objective is: DISPLAYFORM0 This is similar to a variational inference objective, where the first term is analogous to the negative log-likelihood, but τ = 1. As τ ranges from 0 to 1, our objective interpolates between variational optimization and variational inference, as noted by. Similar objectives have been used in the variational inference literature for better training BID5 and representation learning .Minimizing the first term on its own eventually moves all probability mass towards an optimum λ *, ing in σ = 0 if λ * is an isolated local minimum. This compels σ to balance between shrinking to decrease the first term while remaining sufficiently large to avoid a heavy entropy penalty. When benchmarking our algorithm's performance, we evaluate F (λ,ŵ φ (λ)) at the deterministic current hyperparameter λ 0. (This is a common practice when using stochastic operations during training, such as batch normalization or dropout.) We now describe the complete STN training algorithm and discuss how it can tune hyperparameters that other gradient-based algorithms cannot, such as discrete or stochastic hyperparameters. We use an unconstrained parametrization λ ∈ R n of the hyperparameters. Let r denote the element-wise function which maps λ to the appropriate constrained space, which will involve a non-differentiable discretization for discrete hyperparameters. Let L T and L V denote training and validation losses which are (possibly stochastic, e.g., if using dropout) functions of the hyperparameters and parameters. Define functions f, F by f (λ, w) = L T (r(λ), w) and F (λ, w) = L V (r(λ), w). STNs are trained by a gradient descent scheme which alternates between updating φ for T train steps to minimize E ∼p(|σ) [f (λ +,ŵ φ (λ +))] (Eq. 9) and updating λ and σ for T valid steps to minimize DISPLAYFORM0 (Eq. 15). We give our complete algorithm as Algorithm 1 and show how it can be implemented in code in Appendix G. The possible non-differentiability of r due to discrete hyperparameters poses no problem. To estimate the derivative of E ∼p(|σ) [f (λ +,ŵ φ (λ +))] with respect to φ, we can use the reparametrization trick and compute ∂f /∂w and ∂ŵ φ/∂φ, neither of whose computation paths involve the discretization r. DISPLAYFORM1 with respect to a discrete hyperparameter λ i, there are two cases we must consider: Initialize: Best-response approximation parameters φ, hyperparameters λ, learning rates DISPLAYFORM0 while not converged do DISPLAYFORM1 Case 1: For most regularization schemes, L V and hence F does not depend on λ i directly and thus the only gradient is throughŵ φ. Thus, the reparametrization gradient can be used. Case 2: If L V relies explicitly on λ i, then we can use the REINFORCE gradient estimator BID15 to estimate the derivative of the expectation with respect to λ i. The number of hidden units in a layer is an example of a hyperparameter that requires this approach since it directly affects the validation loss. We do not show this in Algorithm 1, since we do not tune any hyperparameters which fall into this case. We applied our method to convolutional networks and LSTMs , yielding self-tuning CNNs (ST-CNNs) and self-tuning LSTMs (ST-LSTMs). We first investigated the behavior of STNs in a simple setting where we tuned a single hyperparameter, and found that STNs discovered hyperparameter schedules that outperformed fixed hyperparameter values. Next, we compared the performance of STNs to commonly-used hyperparameter optimization methods on the CIFAR-10 and PTB datasets. Due to the joint optimization of the hypernetwork weights and hyperparameters, STNs do not use a single, fixed hyperparameter during training. Instead, STNs discover schedules for adapting the hyperparameters online, which can outperform any fixed hyperparameter. We examined this behavior in detail on the PTB corpus using an ST-LSTM to tune the output dropout rate applied to the hidden units. The schedule discovered by an ST-LSTM for output dropout, shown in Figure 3, outperforms the best, fixed output dropout rate (0.68) found by a fine-grained grid search, achieving 82.58 vs 85.83 validation perplexity. We claim that this is a consequence of the schedule, and not of regularizing effects from sampling hyperparameters or the limited capacity ofŵ φ.To rule out the possibility that the improved performance is due to stochasticity introduced by sampling hyperparameters during STN training, we trained a standard LSTM while perturbing its dropout rate around the best value found by grid search. We used random Gaussian perturbations, and sinusoid perturbations for a cyclic regularization schedule. STNs outperformed both perturbation methods (Table 2 : Final validation and test performance of each method on the PTB word-level language modeling task, and the CIFAR-10 image-classification task. To determine whether the limited capacity ofŵ φ acts as a regularizer, we trained a standard LSTM from scratch using the schedule for output dropout discovered by the ST-LSTM. Using this schedule, the standard LSTM performed nearly as well as the STN, providing evidence that the schedule itself (rather than some other aspect of the STN) was responsible for the improvement over a fixed dropout rate. To further demonstrate the importance of the hyperparameter schedule, we also trained a standard LSTM from scratch using the final dropout value found by the STN (0.78), and found that it did not perform as well as when following the schedule. The final validation and test perplexities of each variant are shown in TAB0.Next, we show in Figure 3 that the STN discovers the same schedule regardless of the initial hyperparameter values. Because hyperparameters adapt over a shorter timescale than the weights, we find that at any given point in training, the hyperparameter adaptation has already equilibrated. As shown empirically in Appendix F, low regularization is best early in training, while higher regularization is better later on. We found that the STN schedule implements a curriculum by using a low dropout rate early in training, aiding optimization, and then gradually increasing the dropout rate, leading to better generalization. We evaluated an ST-LSTM on the PTB corpus , which is widely used as a benchmark for RNN regularization due to its small size (; ; BID14 . We used a 2-layer LSTM with 650 hidden units per layer and 650-dimensional word embeddings. We tuned 7 hyperparameters: variational dropout rates for the input, hidden state, and output; embedding dropout (that sets rows of the embedding matrix to 0); DropConnect on the hidden-to-hidden weight matrix; and coefficients α and β that control the strength of activation regularization and temporal activation regularization, respectively. For LSTM tuning, we obtained the best when using a fixed perturbation scale of 1 for the hyperparameters. Additional details about the experimental setup and the role of these hyperparameters can be found in Appendix D. We compared STNs to grid search, random search, and Bayesian optimization. 6 FIG2 shows the best validation perplexity achieved by each method over time. STNs outperform other meth- ods, achieving lower validation perplexity more quickly. The final validation and test perplexities achieved by each method are shown in Table 2. We show the schedules the STN finds for each hyperparameter in FIG2; we observe that they are nontrivial, with some forms of dropout used to a greater extent at the start of training (including input and hidden dropout), some used throughout training (output dropout), and some that are increased over the course of training (embedding and weight dropout). We evaluated ST-CNNs on the CIFAR-10 dataset, where it is easy to overfit with high-capacity networks. We used the AlexNet architecture , and tuned: continuous hyperparameters controlling per-layer activation dropout, input dropout, and scaling noise applied to the input, discrete data augmentation hyperparameters controlling the length and number of cut-out holes , and continuous data augmentation hyperparameters controlling the amount of noise to apply to the hue, saturation, brightness, and contrast of an image. In total, we considered 15 hyperparameters. We compared STNs to grid search, random search, and Bayesian optimization. FIG4 shows the lowest validation loss achieved by each method over time, and Table 2 shows the final validation and test losses for each method. Details of the experimental setup are provided in Appendix E. Again, STNs find better hyperparameter configurations in less time than other methods. The hyperparameter schedules found by the STN are shown in FIG3. Bilevel Optimization. BID10 provide an overview of bilevel problems, and a comprehensive textbook was written by BID1. When the objectives/constraints are restricted to be linear, quadratic, or convex, a common approach replaces the lower-level problem with its KKT conditions added as constraints for the upper-level problem . In the unrestricted setting, our work loosely resembles trust-region methods BID9, which repeatedly approximate the problem locally using a simpler bilevel program. In closely related work, used evolutionary techniques to estimate the best-response function iteratively. Hypernetworks. First considered by Schmidhuber (1993; BID15, hypernetworks are functions mapping to the weights of a neural net. Predicting weights in CNNs has been developed in various forms BID11 BID16 . used hypernetworks to generate weights for modern CNNs and RNNs. BID6 used hypernetworks to globally approximate a bestresponse for architecture search. Because the architecture is not optimized during training, they require a large hypernetwork, unlike ours which locally approximates the best-response. Gradient-Based Hyperparameter Optimization. There are two main approaches. The first approach approximates w * (λ 0) using w T (λ 0, w 0), the value of w after T steps of gradient descent on f with respect to w starting at (λ 0, w 0). The descent steps are differentiated through to approximate ∂w * /∂λ(λ 0) ≈ ∂w T /∂λ(λ 0, w 0). This approach was proposed by BID13 and used by , and. The second approach uses the Implicit Function Theorem to derive ∂w * /∂λ(λ 0) under certain conditions. This was first developed for hyperparameter optimization in neural networks and developed further by. Similar approaches have been used for hyperparameter optimization in log-linear models , kernel selection BID8 ), and image reconstruction (; BID7 . Both approaches struggle with certain hyperparameters, since they differentiate gradient descent or the training loss with respect to the hyperparameters. In addition, differentiating gradient descent becomes prohibitively expensive as the number of descent steps increases, while implicitly deriving ∂w * /∂λ requires using Hessian-vector products with conjugate gradient solvers to avoid directly computing the Hessian. Model-Based Hyperparameter Optimization. A common model-based approach is Bayesian optimization, which models p(r|λ, D), the conditional probability of the performance on some metric r given hyperparameters λ and a dataset D = {(λ i, r i)}. We can model p(r|λ, D) with various methods (; ; ; . D is constructed iteratively, where the next λ to train on is chosen by maximizing an acquisition function C(λ; p(r|λ, D)) which balances exploration and exploitation. Training each model to completion can be avoided if assumptions are made on learning curve behavior . These approaches require building inductive biases into p(r|λ, D) which may not hold in practice, do not take advantage of the network structure when used for hyperparameter optimization, and do not scale well with the number of hyperparameters. However, these approaches have consistency guarantees in the limit, unlike ours. Model-Free Hyperparameter Optimization. Model-free approaches include grid search and random search. BID3 advocated using random search over grid search. Successive Halving and Hyperband extend random search by adaptively allocating resources to promising configurations using multi-armed bandit techniques. These methods ignore structure in the problem, unlike ours which uses rich gradient information. However, it is trivial to parallelize model-free methods over computing resources and they tend to perform well in practice. Hyperparameter Scheduling. Population Based Training (PBT) considers schedules for hyperparameters. In PBT, a population of networks is trained in parallel. The performance of each network is evaluated periodically, and the weights of under-performing networks are replaced by the weights of better-performing ones; the hyperparameters of the better network are also copied and randomly perturbed for training the new network clone. In this way, a single model can experience different hyperparameter settings over the course of training, implementing a schedule. STNs replace the population of networks by a single best-response approximation and use gradients to tune hyperparameters during a single training run. We introduced Self-Tuning Networks (STNs), which efficiently approximate the best-response of parameters to hyperparameters by scaling and shifting their hidden units. This allowed us to use gradient-based optimization to tune various regularization hyperparameters, including discrete hyperparameters. We showed that STNs discover hyperparameter schedules that can outperform fixed hyperparameters. We validated the approach on large-scale problems and showed that STNs achieve better generalization performance than competing approaches, in less time. We believe STNs offer a compelling path towards large-scale, automated hyperparameter tuning for neural networks. We thank Matt Johnson for helpful discussions and advice. MM is supported by an NSERC CGS-M award, and PV is supported by an NSERC PGS-D award. RG acknowledges support from the CIFAR Canadian AI Chairs program. Best-response of the parameters to the hyperparameters The (validation loss) direct (hyperparameter) gradient DISPLAYFORM0 The (elementary parameter) response gradient DISPLAYFORM1 The (validation loss) response gradient DISPLAYFORM2 The hyperparameter gradient: a sum of the validation losses direct and response gradients B PROOFS B.1 LEMMA 1Because w 0 solves Problem 4b given λ 0, by the first-order optimality condition we must have: DISPLAYFORM3 The Jacobian of ∂f /∂w decomposes as a block matrix with sub-blocks given by: DISPLAYFORM4 We know that f is C 2 in some neighborhood of (λ 0, w 0), so ∂f /∂w is continuously differentiable in this neighborhood. By assumption, the Hessian ∂ 2 f /∂w 2 is positive definite and hence invertible at (λ 0, w 0). By the Implicit Function Theorem, there exists a neighborhood V of λ 0 and a unique continuously differentiable function w *: V → R m such that ∂f /∂w(λ, w * (λ)) = 0 for λ ∈ V and w * (λ 0) = w 0.Furthermore, by continuity we know that there is a neighborhood DISPLAYFORM5 Combining this with ∂f /∂w(λ, w * (λ)) = 0 and using second-order sufficient optimality conditions, we conclude that w * (λ) is the unique solution to Problem 4b for all λ ∈ U. This discussion mostly follows from. We let X ∈ R N ×D denote the data matrix where N is the number of training examples and D is the dimensionality of the data. We let t ∈ R N denote the associated targets. We can write the SVD decomposition of X as: DISPLAYFORM0 where U and V are N × D and D × D orthogonal matrices and D is a diagonal matrix with entries DISPLAYFORM1 We next simplify the function y(x; w) by setting u = s Q, so that y(x; w) = s Qx = u x. We see that the Jacobian ∂y /∂x ≡ u is constant, and Problem 13 simplifies to standard L 2 -regularized least-squares linear regression with the following loss function: DISPLAYFORM2 It is well-known (see , Chapter 3) that the optimal solution u * (λ) minimizing Equation 19 is given by: DISPLAYFORM3 Furthermore, the optimal solution u * to the unregularized version of Problem 19 is given by: DISPLAYFORM4 Recall that we defined Q 0 = V, i.e., the change-of-basis matrix from the standard basis to the principal components of the data matrix, and we defined s 0 to solve the unregularized regression problem given Q 0. Thus, we require that Q 0 s 0 = u * which implies s 0 = D −1 U t. There are not unique solutions to Problem 13, so we take any functions Q(λ), s(λ) which satisfy Q(λ) s(λ) = v * (λ) as "best-response functions". We will show that our chosen functions Q * (λ) = σ(λv + c) row Q 0 and s * (λ) = s 0, where v = −1 and c i = 2 log(d i) for i = 1,..., D, meet this criteria. We start by noticing that for any d ∈ R +, we have: DISPLAYFORM5 It follows that: DISPLAYFORM6... DISPLAYFORM7... DISPLAYFORM8... DISPLAYFORM9 B.3 THEOREM 3By assumption f is quadratic, so there exist A ∈ R n×n, B ∈ R n×m, C ∈ R m×m and d ∈ R n, e ∈ R m such that: DISPLAYFORM10 One can easily compute that: DISPLAYFORM11 Since we assume ∂ 2 f /∂w 2 0, we must have C 0. Setting the derivative equal to 0 and using second-order sufficient conditions, we have: DISPLAYFORM12 Hence, we find: DISPLAYFORM13 We letŵ φ (λ) = U λ + b, and definef to be the function given by: DISPLAYFORM14 Substituting and simplifying: DISPLAYFORM15 Expanding, we find that equation 36 is equal to: DISPLAYFORM16 where we have: DISPLAYFORM17 We can simplify these expressions considerably by using linearity of expectation and that ∼ p(|σ) has mean 0: DISPLAYFORM18 We can use the cyclic property of the Trace operator, E ∼p(|σ) [] = σ 2 I, and commutability of expectation and a linear operator to simplify the expectations of 2 and 3: DISPLAYFORM19 We can then differentiatef by making use of various matrix-derivative equalities to find: DISPLAYFORM20 Setting the derivative ∂f /∂b(λ 0, U, b, σ) equal to 0, we have: DISPLAYFORM21 Setting the derivative for ∂f /∂U(λ 0, U, b, σ) equal to 0, we have: DISPLAYFORM22 DISPLAYFORM23 DISPLAYFORM24 This is exactly the best-response Jacobian ∂w * /∂λ(λ) as given by Equation 34. Substituting U = C −1 B into the equation 50 gives: DISPLAYFORM25 This is w * (λ 0) − ∂w * /∂λ(λ 0), thus the approximate best-response is exactly the first-order Taylor series of w * about λ 0. updated the model parameters, but did not update hyperparameters. We terminated training when the learning rate dropped below 0.0003.We tuned variational dropout (re-using the same dropout mask for each step in a sequence) on the input to the LSTM, the hidden state between the LSTM layers, and the output of the LSTM. We also tuned embedding dropout, which sets entire rows of the word embedding matrix to 0, effectively removing certain words from all sequences. We regularized the hidden-to-hidden weight matrix using DropConnect (zeroing out weights rather than activations) . Because DropConnect operates directly on the weights and not individually on the mini-batch elements, we cannot use independent perturbations per example; instead, we sample a single DropConnect rate per mini-batch. Finally, we used activation regularization (AR) and temporal activation regularization (TAR). AR penalizes large activations, and is defined as: DISPLAYFORM26 where m is a dropout mask and h t is the output of the LSTM at time t. TAR is a slowness regularizer, defined as: DISPLAYFORM27 For AR and TAR, we tuned the scaling coefficients α and β. For the baselines, the hyperparameter ranges were: [0, 0.95] for the dropout rates, and for α and β. For the ST-LSTM, all the dropout rates and the coefficients α and β were initialized to 0.05 (except in Figure 3, where we varied the output dropout rate). Here, we present additional details on the CNN experiments. For all , we held out 20% of the training data for validation. We trained the baseline CNN using SGD with initial learning rate 0.01 and momentum 0.9, on mini-batches of size 128. We decay the learning rate by 10 each time the validation loss fails to decrease for 60 epochs, and end training if the learning rate falls below 10 −5 or validation loss has not decreased for 75 epochs. For the baselines-grid search, random search, and Bayesian optimization-the search spaces for the hyperparameters were as follows: dropout rates were in the range We trained the ST-CNN's elementary parameters using SGD with initial learning rate 0.01 and momentum of 0.9, on mini-batches of size 128 (identical to the baselines). We use the same decay schedule as the baseline model. The hyperparameters are optimized using Adam with learning rate 0.003. We alternate between training the best-response approximation and hyperparameters with the same schedule as the ST-LSTM, i.e. T train = 2 steps on the training step and T valid = 1 steps on the validation set. Similarly to the LSTM experiments, we used five epochs of warm-up for the model parameters, during which the hyperparameters are fixed. We used an entropy weight of τ = 0.001 in the entropy regularized objective (Eq. 15). The cutout length was restricted to lie in {0, . . ., 24} while the number of cutout holes was restricted to lie in {0, . . ., 4}. All dropout rates, as well as the continuous data augmentation noise parameters, are initialized to 0.05. The cutout length is initialized to 4, and the number of cutout holes is initialized to 1. Overall, we found the ST-CNN to be relatively robust to the initialization of hyperparameters, but starting with low regularization aided optimization in the first few epochs. Here, we draw connections between hyperparameter schedules and curriculum learning. Curriculum learning BID2 ) is an instance of a family of continuation methods BID0, which optimize non-convex functions by solving a sequence of functions that are ordered by increasing difficulty. In a continuation method, one considers a family of training criteria C λ (w) with a parameter λ, where C 1 (w) is the final objective we wish to minimize, and C 0 (w) represents the training criterion for a simpler version of the problem. One starts by optimizing C 0 (w) and then gradually increases λ from 0 to 1, while keeping w at a local minimum of C λ (w) BID2 ). This has been hypothesized to both aid optimization and improve generalization. In this section, we explore how hyperparameter schedules implement a form of curriculum learning; for example, a schedule that increases dropout over time increases stochasticity, making the learning problem more difficult. We use the of grid searches to understand the effects of different hyperparameter settings throughout training, and show that greedy hyperparameter schedules can outperform fixed hyperparameter values. First, we performed a grid search over 20 values each of input and output dropout, and measured the validation perplexity in each epoch. FIG8 shows the validation perplexity achieved by different combinations of input and output dropout, at various epochs during training. We see that at the start of training, the best validation loss is achieved with small values of both input and output dropout. As we train for more epochs, the best validation performance is achieved with larger dropout rates. Next, we present a simple example to show the potential benefits of greedy hyperparameter schedules. For a single hyperparameter-output dropout-we performed a fine-grained grid search and constructed a dropout schedule by using the hyperparameter values that achieve the best validation perplexity at each epoch in training. As shown in FIG9, the schedule formed by taking the best output dropout value in each epoch yields better generalization than any of the fixed hyperparameter values from the initial grid search. In particular, by using small dropout values at the start of training, the schedule achieves a fast decrease in validation perplexity, and by using larger dropout later in training, it achieves better overall validation perplexity. FIG10 shows the perturbed values for output dropout we used to investigate whether the improved performance yielded by STNs is due to the regularization effect, and not the schedule, in Section 4.1. In this section, we provide PyTorch code listings for the approximate best-response layers used to construct ST-LSTMs and ST-CNNs: the HyperLinear and HyperConv2D classes. We also provide a simplified version of the optimization steps used on the training set and validation set.

We use a hypernetwork to predict optimal weights given hyperparameters, and jointly train everything together.

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent text

Question: Conditional Generative Adversarial Networks (cGANs) are finding increasingly widespread use in many application domains. Despite outstanding progress, quantitative evaluation of such models often involves multiple distinct metrics to assess different desirable properties, such as image quality, conditional consistency, and intra-conditioning diversity. In this setting, model benchmarking becomes a challenge, as each metric may indicate a different "best" model. In this paper, we propose the Frechet Joint Distance (FJD), which is defined as the Frechet distance between joint distributions of images and conditioning, allowing it to implicitly capture the aforementioned properties in a single metric. We conduct proof-of-concept experiments on a controllable synthetic dataset, which consistently highlight the benefits of FJD when compared to currently established metrics. Moreover, we use the newly introduced metric to compare existing cGAN-based models for a variety of conditioning modalities (e.g. class labels, object masks, bounding boxes, images, and text captions). We show that FJD can be used as a promising single metric for model benchmarking. The use of generative models is growing across many domains (van den c; ; ; ;). Among the most promising approaches, Variational Auto-Encoders (VAEs) , auto-regressive models (van den a; b), and Generative Adversarial Networks (GANs) have been driving significant progress, with the latter at the forefront of a wide-range of applications (; ; a; ; ; ;). In particular, significant research has emerged from practical applications, which require generation to be based on existing context. For example, tasks such as image inpainting, super-resolution, or text-to-image synthesis have been successfully addressed within the framework of conditional generation, with conditional GANs (cGANs) among the most competitive approaches. Despite these outstanding advances, quantitative evaluation of GANs remains a challenge . In the last few years, a significant number of evaluation metrics for GANs have been introduced in the literature (; ; Bińkowski et al., 2018; ; ; Kynkäänniemi et al., 2019;). Although there is no clear consensus on which quantitative metric is most appropriate to benchmark GAN-based models, Inception Score (IS) and Fréchet Inception Distance (FID) have been extensively used. However, both IS and FID were introduced in the context of unconditional image generation and, hence, focus on capturing certain desirable properties such as visual quality and sample diversity, which do not fully encapsulate all the different phenomena that arise during conditional image generation. In conditional generation, we care about visual quality, conditional consistency -i.e., verifying that the generation respects its conditioning, and intra-conditioning diversity -i.e., sample diversity per conditioning. Although visual quality is captured by both metrics, IS is agnostic to intra-conditioning diversity and FID only captures it indirectly. 1 Moreover, neither of them can capture conditional con-sistency. In order to overcome these shortcomings, researchers have resorted to reporting conditional consistency and diversity metrics in conjunction with ). Consistency metrics often use some form of concept detector to ensure that the requested conditioning appears in the generated image as expected. Although intuitive to use, these metrics require pretrained models that cover the same target concepts in the same format as the conditioning (i.e., classifiers for image-level class conditioning, semantic segmentation for mask conditioning, etc.), which may or may not be available off-the-shelf. Moreover, using different metrics to evaluate different desirable properties may hinder the process of model selection, as there may not be a single model that surpasses the rest in all measures. In fact, it has recently been demonstrated that there is a natural trade-off between image quality and sample diversity , which calls into question how we might select the correct balance of these properties. In this paper we introduce a new metric called Fréchet Joint Distance (FJD), which is able to implicitly assess image quality, conditional consistency, and intra-conditioning diversity. FJD computes the Fréchet distance on an embedding of the joint image-conditioning distribution, and introduces only small computational overhead over FID compared to alternative methods. We evaluate the properties of FJD on a variant of the synthetic dSprite dataset and verify that it successfully captures the desired properties. We provide an analysis on the behavior of both FID and FJD under different types of conditioning such as class labels, bounding boxes, and object masks, and evaluate a variety of existing cGAN models for real-world datasets with the newly introduced metric. Our experiments show that FJD captures the three highlighted properties of conditional generation; it can be applied to any kind of conditioning (e.g., class, bounding box, mask, image, text, etc.); and when applied to existing cGAN-based models, FJD demonstrates its potential to be used as a promising unified metric for hyper-parameter selection and cGAN benchmarking. To our knowledge, there are no existing metrics for conditional generation that capture all of these key properties. Conditional GANs have witnessed outstanding progress in recent years. Training stability has been improved through the introduction of techniques such as progressive growing, , spectral normalization and the two time-scale update rule . Architecturally, conditional generation has been improved through the use of auxiliary classifiers and the introduction of projection-based conditioning for the discriminator . Image quality has also benefited from the incorporation of self-attention (a), as well as increases in model capacity and batch size . All of this progress has led to impressive , paving the road towards the challenging task of generating more complex scenes. To this end, a flurry of works have tackled different forms of conditional image generation, including class-based (; ; ; ; ;), image-based (a; b; ; ;), mask-and bounding box-based (; ; ;, as well as text- (; ; 2018a; ;) and dialogue-based conditionings . This intensified research has lead to the development of a variety of metrics to assess the three factors of conditional image generation process quality, namely: visual quality, conditional consistency, and intra-conditioning diversity. Visual quality. A number of GAN evaluation metrics have emerged in the literature to assess visual quality of generated images in the case of unconditional image generation. Most of these metrics either focus on the separability between generated images and real images (; ; ;), compute the distance between distributions (; ;), assess sample quality and diversity from conditional or marginal distributions (; ;), measure the similarity between generated and real images (; ; ;) or are log-likelihood based 2. Among these, the most accepted automated visual quality metrics are Inception Score (IS) and Fréchet Inception Distance (FID) . Conditional consistency. To assess the consistency of the generated images with respect to model conditioning, researchers have reverted to available, pre-trained feed-forward models. The structure of these models depends on the modality of the conditioning (e.g. segmentation models are used for mask conditioning or image captioning models are applied to evaluate text conditioning). Moreover, the metric used to evaluate the forward model on the generated distribution depends on the conditioning modality and includes: accuracy in the case of class-conditioned generation, Intersection over Union when using bounding box-and mask-conditionings, BLEU , METEOR or CIDEr in the case of text-based conditionings, and Structural Similarity (SSIM) or peak signal-to-noise ratio (PSNR) for image-conditioning. Intra-conditioning diversity. The most common metric for evaluating sample diversity is Learned Perceptual Image Patch Similarity (LPIPS) (b), which measures the distance between samples in a learned feature space. Alternatively, proposed Intra-FID, which calculates a FID score separately for each conditioning and reports the average score over all conditionings. This method should in principle capture the desirable properties of image quality, conditional consistency, and intra-class diversity. However, it scales poorly with the number of unique conditions, as the computationally intensive FID calculation must be repeated for each case, and because FID behaves poorly when the sample size is small (Bińkowski et al., 2018). Furthermore, in cases where the conditioning cannot be broken down into a set of discrete classes (e.g., pixel-based conditioning), Intra-FID is intractable. As a , it has not been applied beyond class-conditioning. FID aims to compare the statistics of generated samples to samples from a real dataset. Given two multivariate Gaussian distributions N (µ, Σ) and N (μ,Σ), Fréchet Distance (FD) is defined as: When evaluating a generative model, N (µ, Σ) represents the data (reference) distribution, obtained by fitting a Gaussian to samples from a reference dataset, and N (μ,Σ) represents the learned (generated) distribution, a of fitting to samples from a generative model. In FID, both the real images and model samples are embedded in a learned feature space using a pre-trained Inception v3 model . Thus, the Gaussian distributions are defined in the embedded space. More precisely, given a dataset of images {x, a set of model samples, and an Inception embedding function f, we estimate the Gaussian parameters µ, Σ,μ andΣ as: In conditional image generation, a dataset is composed of image-condition pairs {(, where the conditioning can take variable forms, such as image-level classes, segmentation masks, or text. The goal of conditional image generation is to produce realistic looking, diverse imagesx that are consistent with the conditioningŷ. Thus, a set of model samples with corresponding conditioning can be defined as: . As discussed in Section 3, the Fréchet distance (FD) compares any two Gaussians defined over arbitrary spaces. In FJD, we propose to compute the FD between two Gaussians defined over the joint image-conditioning embedding space. More precisely, given an image embedding function f, a conditioning embedding function h, a conditioning embedding scaling factor α, and a merging function g that combines the image embedding with the conditioning embedding into a joint one, we can estimate the respective Gaussian parameters µ, Σ,μ andΣ as: Note that by computing the FD over the joint image-conditioning distribution, we are able to simultaneously assess image quality, conditional consistency, and intra-conditioning diversity, all of which are important factors in evaluating the quality of conditional image generation models. To ensure reproducibility, when reporting FJD scores it is important to include details such as which conditioning embedding function was used, which dataset is used for the reference distribution, and the α value. We report these values for all of our experiments in Appendix B. Sentence-BERT The purpose of the embedding function h is to reduce the dimensionality and extract a useful feature representation of the conditioning. As such, the choice of h will vary depending on the modality of conditioning. In most cases, an off-the-shelf, pretrained embedding can be used for the purposes of extracting a useful representation. In the absence of preexisting, pretrained conditioning embedding functions, a new one should be learned. For example, for bounding box and mask conditionings the embedding function could be learned with an autoencoder. 3 For suggested assignments of conditioning modalities to embedding functions please refer to Table 1. In order to control the relative contribution of the image component and the conditioning component to the final FJD value, we scale the conditioning embedding by a constant α. In essence, α indicates how much we care about the conditioning component compared to the image component. When α = 0, the conditioning component is ignored and FJD is equivalent to FID. As the value of α increases, the perceived importance of the conditioning component is also increased and reflected accordingly in the ing measure. To equally weight the image component and the conditioning component, we recommend setting α to be equal to the ratio between the average L 2 norm of the image embedding and the conditioning embedding. This weighting ensures that FJD retains consistent behaviour across conditioning embeddings, even with varying dimensionality or magnitude. We note that α should be calculated on data from the reference distribution (real data distribution), and then applied to all conditioning embeddings thereafter. See Appendix F for an example of the effect of the α hyperparameter. 3 In the initial stages of this project, we also explored methods to bypass this additional training step by projecting a visual representation of bounding box or mask conditioning into an Inceptionv3 embedding space. However, the Inceptionv3 embedding may not properly capture object positions as it is trained to classify, discarding precise spatial information. Therefore, we consider autoencoders (AE) to be better suited to our setup since they are trained to recover both object appearance and spatial information from the embedded representation. The purpose of the merging function g is to combine the image embedding and conditioning embedding into a single joint embedding. We compared several candidate merging functions and found concatenation of the image embedding and conditioning embedding vectors to be most effective, both in terms of simplicity and performance. As such, concatenation is used as the merging function in all following experiments. In this section, we demonstrate that FJD captures the three desiderata of conditional image generation, namely image quality, conditional consistency and intra-conditioning diversity. dSprite-textures. The dSprite dataset is a synthetic dataset where each image depicts a simple 2D shape on a black . Each image can be fully described by a set of factors, including shape, scale, rotation, x position, and y position. We augment the dSprite dataset to create dSprite-textures by adding three texture patterns for each sample. Additionally, we include class labels indicating shape, as well as bounding boxes and mask labels for each sample (see Figure 1). In total, the dataset contains 2,211,840 unique images. This synthetic dataset allows us to exactly control our sample distribution and, thereby, simulate a generator with image-conditioning inconsistencies or reduced sample diversity. To embed the conditioning for calculating FJD in the following experiments, we use one-hot encoding for the class labels, and autoencoder representations for the bounding box and mask labels. 4 We are releasing the code to generate dSprite-textures. In this subsection, we aim to test the sensitivity of FJD to image quality perturbations. To do so, we draw 10k random samples from the dSprite-textures dataset to form a reference dataset. The generated dataset is simulated by duplicating the reference dataset and adding Gaussian noise drawn from N (0, σ) to the images, where σ ∈ [0, 0.25] and pixel values are normalized (and clipped after noise addition) to the range. The addition of noise mimics a generative model that produces low quality images. We repeat this experiment for all three conditioning types in dSprite-textures: class, bounding box, and mask. Results are shown in Figure 2, where we plot both FID and FJD as a function of the added Gaussian noise (σ is indicated on the x-axis as Noise Magnitude). We find that, in all cases, FJD has a very similar trend to FID, indicating that it successfully captures image quality. Additional image quality experiments on the large scale COCO-Stuff dataset can be found in Appendix C. In this subsection, we aim to highlight the sensitivity of FJD to conditional consistency. In particular, we target specific types of inconsistencies, such as incorrect scale, orientation, or position. We draw a set of 10k samples from the dSprite-textures dataset and duplicate it to represent the reference dataset and the generated dataset, each with identical image and conditioning marginal distributions. For 30% of the generated dataset samples we swap conditionings of pairs of samples that are identical in all but one of the attributes (scale, orientation, x position or y position). For example, if one generated sample has attribute x position 4 and a second generated sample has attribute x position 7, swapping their conditionings leads to generated samples that are offset by 3 pixels w.r.t. their ground truth x position. Swapping conditionings in this manner allows us to control for specific attributes' conditional consistency, while keeping the image and conditioning marginal distributions unchanged. As a , all changes in FJD can be attributed solely to conditional inconsistencies. Figure 3 depicts the of this experiment for four different types of alterations: scale, orientation, and x and y positions. We observe that the FID between image distributions (solid blue line) remains constant even as the degree of conditional inconsistency increases. For class conditioning (dotted orange line), FJD also remains constant, as changes to scale, orientation, and position are independent of the object class. Bounding box and mask conditionings, as they contain spatial information, produce variations in FJD that are proportional to the offset. Interestingly, for the orientation offsets, FJD with mask conditioning fluctuates rather than increasing monotonically. This behaviour is due to the orientation masks partially re-aligning with the ground truth around 90 • and 180 •. Each of these cases emphasize the effective sensitivity of FJD with respect to conditional consistency. Additional conditional consistency experiments with text conditioning can be found in Appendix D. In this subsection, we aim to test the sensitivity of FJD to intra-conditioning diversity 5, by alternating the per-conditioning image texture variability. More precisely, we vary the texture based on four different image attributes: shape that is captured in all tested conditionings, as well as scale, orientation and position that are captured by bounding box and mask conditionings only. To create attribute-texture assignments, we stratify attributes based on their values. For example, one possible shape-based stratification of a dataset with three shapes might be: [squares, ellipses, hearts]. To quantify the dataset intra-conditioning diversity, we introduce a diversity score. A diversity score of 1 means that the per-attribute texture distribution is uniform across stratas, while a diversity score of 0 means that each strata is assigned to a single texture. Middling diversity scores indicate that the textural distribution is skewed towards one texture type in each strata. We create our reference dataset by randomly drawing 10k samples. The generated distribution is created by duplicating the reference distribution and adjusting the per-attribute texture variability to achieve the desired diversity score. The of these experiments are shown in Figure 4, which plots the increase in FID and FJD, for different types of conditioning, as the diversity of textures within each subset decreases. For all tested scenarios, we observe that FJD is sensitive to intra-conditioning diversity changes. Moreover, not surprisingly, since a change in the joint distribution of attributes and textures also implies a change to the image marginal distribution, we observe that FID increases with reduced diversity. This experiment suggests that FID is able to capture intra-conditioning diversity changes when the image conditional distribution is also affected. However, if the image marginal distribution were to stay constant, FID would be blind to intra-conditioning diversity changes (as is shown in Section 5.3). In this section, we seek to demonstrate the application of FJD to evaluate models with several different conditioning modalities, in contrast to FID and standard conditional consistency and diversity metrics. We focus on testing class-conditioned, image-conditioned, and text-conditioned image generation tasks, which have been the focus of numerous works 6. Multi-label, bounding box, and mask conditioning are also explored in Appendix I. We note that FJD and FID yield similar rankings of models in this setting, which is to be expected since most models use similar conditioning mechanisms. Rankings are therefore dominated by image quality, rather than conditional consistency. We refer the reader to Appendix F and H for examples of cases where FJD ranks models differently than FID. Class-conditioned cGANs. . Accuracy is used to evaluate conditional consistency, and is computed as the Inception v3 accuracy of each model's generated samples, using their conditioning as classification ground truth. Class labels from the validation set are used as conditioning to generate 50k samples for each model, and the training set is used as the reference distribution. One-hot encoding is used to embed the class conditioning for the purposes of calculating FJD. We find that FJD follows the same trend as FID for class-conditioned models, preserving their ranking and highlighting the FJD's ability to capture image quality. Additionally, we note that the difference between FJD and FID correlates with each model's classification accuracy, with smaller gaps appearing to indicate better conditional consistency. Diversity scores, however, rank models in the opposite order compared to all other metrics. This behaviour evokes the trade-off between realism and diversity highlighted by. Ideally, we would like a model that produces diverse outputs, but this property is not as attractive if it also in a decrease in image quality. At what point should diversity be prioritized over image quality, and vice versa? FJD is a suitable metric for answering this question if the goal is to find a model that best matches the target conditional data generating distribution. Image-conditioned cGANs. In this setting we encounter some ambiguity with regards to model selection, as for all datasets, each metric ranks the models differently. BicycleGAN appears to have the best image quality, Pix2pix produces images that are most visually similar to the ground truth, and MSGAN and MUNIT achieve the best sample diversity scores. This scenario demonstrates the benefits of using a single unified metric for model selection, for which there is only a single best model. Text-conditioned cGANs. Table 4 shows FJD and FID scores for three state-of-the-art text-conditioned models trained on the Caltech-UCSD Birds 200 dataset (CUB-200) at 256 × 256 resolution: HDGan (c), StackGAN++ (a), and AttnGAN . Conditional consistency is evaluated using visual-semantic similarity, as proposed by Zhang et al. (2018c). Conditioning from the test set captions is used to generate 30k images, and the same test set is also used as the reference distribution. We use pre-computed Char-CNN-RNN sentence embeddings as the conditioning embedding for FJD, since they are commonly used with CUB-200 and are readily available. In this case we find that AttnGAN dominates in terms of conditional consistency compared to HDGan and StackGAN++, while all models are comparable in terms of diversity. AttnGAN is ranked best overall by FJD. In cases where the biggest differentiator between the models is image quality, FID and FJD will provide a consistent ranking as we see here. In cases where the trade-off is more subtle we believe practitioners will opt for a metric that measurably captures intra-conditioning diversity. In this paper we introduce Fréchet Joint Distance (FJD), which is able to assess image quality, conditional consistency, and intra-conditioning diversity within a single metric. We compare FJD to FID on the synthetic dSprite-textures dataset, validating its ability to capture the three properties of interest across different types of conditioning, and highlighting its potential to be adopted as a unified cGAN benchmarking metric. We also demonstrate how FJD can be used to address the potentially ambiguous trade-off between image quality and sample diversity when performing model selection. Looking forward, FJD could serve as valuable metric to ground future research, as it has the potential to help elucidate the most promising contributions within the scope of conditional generation. In this section, we illustrate the claim made in Section 1 that FID cannot capture intra-conditioning diversity when the joint distribution of two variables changes but the marginal distribution of one of them is not altered. Consider two multivariate Gaussian distributions, (X 1, Y 1) ∼ N (0, Σ 1) and (X 2, Y 2) ∼ N (0, Σ 2), where If we let X i take the role of the embedding of the conditioning variables (e.g., position) and Y i take the role of the embedding of the generated variables (i.e., images), then computing FID in this example would correspond to computing the FD between f Y1 and f Y2, which is zero. On the other hand, computing FJD would correspond to the FD between f X1,Y1 and f X2,Y2, which equals 0.678. But note that Dist1 and Dist2 have different degrees of intra-conditioning diversity, as illustrated by Figure 5 (right), where two histograms of f Yi|Xi∈(0.9,1.1) are displayed, showing marked differences to each other (similar plots can be constructed for other values of X i). Therefore, this example illustrates a situation in which FID is unable to capture changes in intra-conditioning diversity, while FJD is able to do so. Important details pertaining to the computation of the FID and FJD metrics for different experiments included in this paper are reported in Table 5. For each dataset we report which conditioning modality was used, as well as the conditioning embedding function. Information about which split and image resolution are used for the reference and generated distributions is also included, as well as how many samples were generated per conditioning. Values for α reported here are calculated according to the balancing mechanism recommended in Section 4.2. Datasets splits marked by "-" indicate that the distribution is a randomly sampled subset of the full dataset. We repeat the experiment initially conducted in Section 5.2 on a real world dataset to see how well FJD tracks image quality. Specifically, we use the COCO-Stuff dataset , which provides class labels, bounding box annotations, and segmentation masks. We follow the same experimental procedure as outlined in Section 5.2: Gaussian noise is drawn from N (0, σ) and add to the images, where σ ∈ [0, 0.25] and pixel values are normalized (and clipped after noise addition) to the range. The original dataset of clean images is used as the reference distribution, while noisy images are used to simulate a generated distribution with poor image quality. For the purposes of calculating FJD, we use N-hot encoding to embed the labels of the classes present in each image, and autoencoder representations for the bounding box and mask labels. As shown in Figure 6, FID and FJD both track image quality well, increasing as more noise is added to the generated image distribution. In order to test the effectiveness of FJD at detecting conditional inconsistencies in the text domain, we use the Caltech-UCSD Birds 200 dataset . This dataset is a common benchmark for text conditioned image generation models, containing 200 fine-grained bird categories, 11,788 images, and 10 descriptive captions per images. Also included in the dataset are vectors of detailed binary annotations describing the attributes of the bird in each image. Each annotation indicates the presence or absence of specific features, such as has\_bill\_shape::curved or has\_wing\_color::blue. Our goal in this experiment is to swap captions between images, and in this fashion introduce inconsistencies between images and their paired captions, while preserving the marginal distributions of images and labels. We compare attribute vectors belonging to each image using the Hamming distance to get an indication for how well the captions belonging to one image might describe another. Small Hamming distances indicate a good match between image and caption, while at larger values the captions appear to describe a very different bird than what is pictured (as demonstrated in Figure 7). This bird is fully covered in red except for some parts of wing and it has brown feet. A red bird with a short bill with a black cheek patch. This small bird is bright red with black wings, black eyeing, and short black beak. The small brown bird has a yellow beak and black round eyes. 39 The body of the bird is ivory and the crown is bright red while the wing is black and ivory speckled. 51 Figure 7: A summer tanager, as described by a variety of captions (ground truth caption highlighted in blue). The Hamming distance between attribute vectors associated with each caption and the ground truth caption provides an indication of how well each caption describes the image. To test FJD we create two datsets: one which contains the original image-captions pairs from CUB-200 to act as the reference distribution, and another in which captions have been swapped to act as a generated distribution that has poor conditional consistency. Char-CNN-RNN embeddings are used to encode the captions for the purposes of calculating FJD. In Figure 8 we observe that as the average Hamming distance across captions increases (i.e., the captions become worse at describing their associated images), FJD also increases. FID, which is unable to detect these inconsistencies, remains constant throughout. E LIST OF SOURCES OF PRE-TRAINED MODEL Table 6 includes the hyperlinks to all of the pretrained conditional generation models used in our experiments in Section 6. The α parameter in the FJD equation acts as a weighting factor indicating the importance of the image component versus the conditional component. When α = 0, then FJD is equal to FID, since we only care about the image component. As the value of α increases, the magnitude of the conditional component's contribution to the value of FJD increases as well. In our experiments, we attempt to find a neutral value for α that will balance the contribution from the conditional component and the image component. This balancing is done by finding the value of α that would in equal magnitude between the image and conditioning embeddings (as measured by the average L2 norm of the embedding vectors). Instead of reporting FJD at a single α, an alternative approach is to calculate and plot FJD for a range of α values, as shown in Figure 9. Plotting α versus FJD allows us to observe any change in rank of models as the importance weighting on the conditional component is increased. Here we use the truncation trick to evaluate BigGAN at several different truncation values σ. The truncation trick is a technique wherein the noise vector used to condition a GAN is scaled by σ in order to trade sample diversity for image quality and conditional consistency, without needing to retrain the model (as shown in Table 7). Table 7: Comparison of BigGAN model evaluated with different truncation values σ. FJD is calculated at α = 17.8. As σ increases, classification accuracy decreases and diversity increases. Note that FID and FJD are consistent in their choice of the preferred model at σ = 1.0, however, the relative ranking of σ = 0.25 and σ = 2.0 is different between the two metrics. We find that in several cases, the ranking of models changes when comparing them at α = 0 (equivalent to FID), versus comparing them using FJD at higher α values. Models with low truncation values σ initially achieve good performance when α is also low. However, as α increases, these models rapidly drop in rank due to lack of sample diversity, and instead models with higher σ values are favoured. This is most obvious when comparing σ = 0.25 and σ = 1.75 (blue and yellow lines in Figure 9) respectively. To create embeddings for the bounding box and mask conditionings evaluated in this paper we utilize a variant of the Regularized AutoEncoder with Spectral Normalization (RAE-SN) introduced by and enhance it with residual connections (Tables 8 and 9). For better reconstruction quality, we substitute the strided convolution and transposed convolution for average pooling and nearest neighbour upsampling, respectively. Spectral normalization is applied to all linear and convolution layers in the decoder, and an L 2 penalty is applied to the latent representation z during training. Hyperparameters such as the weighting factor on the L 2 penalty and the number of dimensions in the latent space are selected based on which combination produces the best reconstructions on a held-out validation set. In Tables 10 and 11 we depict the architecture for an autoencoder with 64 × 64 input resolution, but this can be scaled up or down by adding or removing residual blocks as required. ch represents a channel multiplier which is used to control the capacity of the model. M represents the number of latent dimensions in the latent representation. C indicates the number of classes in the bounding box or mask representation. ResBlock down 2ch → 4ch ResBlock down 4ch → 8ch ResBlock up 4ch → 2ch ResBlock up 2ch → ch Conv ch → C In order to demonstrate the utility of FJD for the purposes of model selection and hyperparameter tuning, we consider the loss function of the generator from an auxiliary classifier GAN (ACGAN) , as shown in Equation 7 to 9. Here S indicates the data source, and C indicates the class label. The generator loss L G is maximized during training, and consists of two components: an adversarial component L S, which encourages generated samples to look like real samples, and a classification component L C, which encourages samples to look more like their target class. In this experiment we add a weighting parameter λ, which weights the importance of the conditional component of the generator loss. The original formulation of ACGAN is equivalent to always setting λ = 1, but it is unknown whether this is the most suitable setting as it is never formally tested. To this end, we train models on the MNIST dataset and perform a sweep over the λ parameter in the range, training a single model for each λ value tested. Each model is evaluated using FID, FJD, and classification accuracy to indicate conditional consistency. For FID and FJD we use the training set as the reference distribution, and generate 50, 000 samples for the generated distribution. Classification accuracy is measured using a pretrained LeNet classifier , where the conditioning label is used as the groundtruth. Scores from best performing models as indicated by FID, FJD, and classification accuracy are shown in Table 12. Sample sheets are provided in Figure 10, where each column is conditioned on a different digit from 0 to 9. We find that FID is optimized when λ = 0.25 (Figure 10a). This produces a model with good image quality, but almost no conditional consistency. Accuracy is optimized when λ = 5.0 (Figure 10c), yielding a model with good conditional consistency, but limited image quality. Finally, FJD is optimized when λ = 1.0 (Figure 10b), producing a model that demonstrates a balance between image quality and conditional consistency. These demonstrate the importance of considering both image quality and conditional consistency simultaneously when performing hyperparameter tuning. To demonstrate FJD applied to multi-label, bounding box, and mask conditioning on a real world dataset, we train a GAN on the COCO-Stuff dataset . To this end, we train three generative models, one for each conditioning type. Following , we select only images containing between 3 and 8 objects, and also ignore any objects that occupy less than 2% of the total image area. Two image resolutions are considered: 64 × 64 and 128 × 128. We adopt a BigGAN-style model , but modify the design such that a single fixed architecture can be trained with any of the three conditioning types. See Section I.1 for architectural details. We train each model 5 times, with different random seeds, and report mean and standard deviation of both FID and FJD in Table 13. N-hot encoding is used as the embedding function for the multi-label conditioning, while autoencoder representations are used to calculate FJD for bounding box and mask conditioning. In most cases we find that FID values are very close between conditioning types. A similar trend is observed in FJD at the 128 × 128 resolution. For models trained at 64 × 64 resolution however, we notice a more drastic change in FJD between conditioning types. Mask conditioning achieves the lowest FJD score, followed by multi-label conditioning and bounding box conditioning. This could indicate that the mask conditioning models are more conditionally consistent (or diverse) compared to other conditioning types. In order to modify to work with multiple types of conditioning we make two major changes. The first change occurs in the generator, where we replace the conditional batch normalization layers with SPADE . This substitution allows the generator to receive spatial conditioning such as bounding boxes or masks. In the case of class conditioning with a spatially tiled class vector, SPADE behaves similarly to conditional batch normalization. The second change we make is to the discriminator. The original BigGAN implementation utilizes a single projection layer in order to provide class-conditional information to the discriminator. To extend this functionality to bounding box and mask conditioning, we add additional projection layers after each ResBlock in the discriminator. The input to each projection layer is a downsampled version of the conditioning that has been resized using nearest neighbour interpolation to match the spatial resolution of each layer. In this way we provide conditioning information at a range of resolutions, allowing the discriminator to use whichever is most useful for the type of conditioning it has received. Aside from these specified changes, and using smaller batch sizes, models are trained with the same hyperparameters and training scheme as specified in . In this section, we present some random 128 × 128 samples of conditional generation for the models covered in Section I. In particular, Figures 11-13 show class, bounding box, and mask conditioning samples, respectively. Each row displays a depiction of conditioning, followed by 4 different samples, and finally the real image corresponding to the conditioning. As shown in Figure 11, conditioning on classes leads to variable samples w.r.t. object positions, scales and textures. As we increase the conditioning strength, we reduce the freedom of the generation and hence, in Figure 12, we observe how the variability starts appearing in more subtle regions. Similarly, in Figure 13, taking different samples per conditioning only changes the textures. Although the degrees of variability decrease as the conditioning strength increases, we obtain sharper, better looking images.

Represent the TLDR for finding the corresponding scientific article

We propose a new metric for evaluating conditional GANs that captures image quality, conditional consistency, and intra-conditioning diversity in a single measure.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Parametric adversarial divergences implicitly define more meaningful task losses for generative modeling, we make parallels with structured prediction to study the properties of these divergences and their ability to encode the task of interest.