Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Generative adversarial networks (GANs) are one of the most popular approaches when it comes to training generative models, among which variants of Wasserstein GANs are considered superior to the standard GAN formulation in terms of learning stability and sample quality. However, Wasserstein GANs require the critic to be 1-Lipschitz, which is often enforced implicitly by penalizing the norm of its gradient, or by globally restricting its Lipschitz constant via weight normalization techniques. Training with a regularization term penalizing the violation of the Lipschitz constraint explicitly, instead of through the norm of the gradient, was found to be practically infeasible in most situations. Inspired by Virtual Adversarial Training, we propose a method called Adversarial Lipschitz Regularization, and show that using an explicit Lipschitz penalty is indeed viable and leads to competitive performance when applied to Wasserstein GANs, highlighting an important connection between Lipschitz regularization and adversarial training. In recent years, Generative adversarial networks (GANs) have been becoming the state-of-the-art in several generative modeling tasks, ranging from image generation to imitation learning . They are based on an idea of a two-player game, in which a discriminator tries to distinguish between real and generated data samples, while a generator tries to fool the discriminator, learning to produce realistic samples on the long run. Wasserstein GAN (WGAN) was proposed as a solution to the issues present in the original GAN formulation. Replacing the discriminator, WGAN trains a critic to approximate the Wasserstein distance between the real and generated distributions. This introduced a new challenge, since Wasserstein distance estimation requires the function space of the critic to only consist of 1-Lipschitz functions. To enforce the Lipschitz constraint on the WGAN critic, originally used weight clipping, which was soon replaced by the much more effective method of Gradient Penalty (GP) , which consists of penalizing the deviation of the critic's gradient norm from 1 at certain input points. Since then, several variants of gradient norm penalization have been introduced (; ; ; b). Virtual Adversarial Training (VAT) is a semi-supervised learning method for improving robustness against local perturbations of the input. Using an iterative method based on power iteration, it approximates the adversarial direction corresponding to certain input points. Perturbing an input towards its adversarial direction changes the network's output the most. Inspired by VAT, we propose a method called Adversarial Lipschitz Regularization (ALR), enabling the training of neural networks with regularization terms penalizing the violation of the Lipschitz constraint explicitly, instead of through the norm of the gradient. It provides means to generate a pair for each input point, for which the Lipschitz constraint is likely to be violated with high probability. In general, enforcing Lipschitz continuity of complex models can be useful for a lot of applications. In this work, we focus on applying ALR to Wasserstein GANs, as regularizing or constraining Lipschitz continuity has proven to have a high impact on training stability and reducing mode collapse. Source code to reproduce the presented experiments is available at https://github.com/dterjek/adversarial_lipschitz_regularization. • We propose Adversarial Lipschitz Regularization (ALR) and apply it to penalize the violation of the Lipschitz constraint directly, ing in Adversarial Lipschitz Penalty (ALP). • Applying ALP on the critic in WGAN (WGAN-ALP), we show state-of-the-art performance in terms of Inception Score and Fréchet Inception Distance among non-progressive growing methods trained on CIFAR-10, and competitive performance in the high-dimensional setting when applied to the critic in Progressive Growing GAN trained on CelebA-HQ. Generative adversarial networks (GANs) provide generative modeling by a generator network g that transforms samples of a low-dimensional latent space Z into samples from the data space X, transporting mass from a fixed noise distribution P Z to the generated distribution P g. The generator is trained simultaneously with another network f called the discriminator, which is trained to distinguish between fake samples drawn from P g and real samples drawn from the real distribution P r, which is often represented by a fixed dataset. This network provides the learning signal to the generator, which is trained to generate samples that the discriminator considers real. This iterative process implements the minimax game played by the networks f and g. This training procedure minimizes the approximate Jensen-Shannon divergence (JSD) between P r and P g . However, during training these two distributions might differ strongly or even have non-overlapping supports, which might in gradients received by the generator that are unstable or zero. Wasserstein GAN (WGAN) was proposed as a solution to this instability. Originating from Optimal Transport theory , the Wasserstein metric provides a distance between probability distributions with much better theoretical and practical properties than the JSD. It provides a smooth optimizable distance even if the two distributions have non-overlapping supports, which is not the case for JSD. It raises a metric d X from the space X of the supports of the probability distributions P 1 and P 2 to the space of the probability distributions itself. For these purposes, the Wasserstein-p distance requires the probability distributions to be defined on a metric space and is defined as where Π(P 1, P 2) is the set of distributions on the product space X × X whose marginals are P 1 and P 2, respectively. The optimal π achieving the infimum in is called the optimal coupling of P 1 and P 2, and is denoted by π *. The case of p = 1 has an equivalent formulation called the Kantorovich-Rubinstein formula , where f: X → R is called the potential function, f L ≤ 1 is the set of all functions that are 1-Lipschitz with respect to the ground metric d X, and the Wasserstein-1 distance corresponds to the supremum over all 1-Lipschitz potential functions. The smallest Lipschitz constant for a real-valued function f with the metric space (X, d X) as its domain is given by Based on, the critic in WGAN implements an approximation of the Wasserstein-1 distance between P g and P r. The minimax game played by the critic f and the generator g becomes min a formulation that proved to be superior to the standard GAN in practice, with substantially more stable training behaviour and improved sample quality, although recent GAN variants do not always use this objective . With WGAN, the challenge became effectively restricting the smallest Lipschitz constant of the critic f, sparking the birth of a plethora of Lipschitz regularization techniques for neural networks. A general definition of the smallest Lipschitz constant of a function where the metric spaces (X, d X) and (Y, d Y) are the domain and codomain of the function f, respectively. The function f is called Lipschitz continuous if there exists a real constant for any x, y ∈ X. Then, the function f is also called K-Lipschitz. Theoretical properties of neural networks with low Lipschitz constants were explored in , and , showing that it induces better generalization. Learning mappings with Lipschitz constraints became prevalent in the field of deep learning with the introduction of WGAN. Enforcing the Lipschitz property on the critic was first done by clipping the weights of the network. This approach achieved superior compared to the standard GAN formulation, but still sometimes yielded poor quality samples or even failed to converge. While clipping the weights enforces a global Lipschitz constant, it also reduces the function space, which might not include the optimal critic any more. Soon this method has been replaced by a softened one called Gradient Penalty (GP) . Motivated by the fact that the optimal critic should have unit gradient norm on lines connecting the coupled points (x 1, x 2) ∼ π * according to, they proposed a regularizer that enforces unit gradient norm along these lines, which not only enforces the Lipschitz constraint, but other properties of the optimal solution as well. However, π * is not known in practice, which is why proposed to apply GP on samples of the induced distribution P i, by interpolating samples from the marginals P 1 and P 2. The critic in the WGAN-GP formulation is regularized with the loss where P i denotes the distribution of samples obtained by interpolating pairs of samples drawn from P r and P g, and λ is a hyperparameter acting as a Lagrange multiplier. Theoretical arguments against GP were pointed out by and , arguing that unit gradient norm on samples of the distribution P i is not valid, as the pairs of samples being interpolated are generally not from the optimal coupling π *, and thus do not necessarily need to match gradient norm 1. Furthermore, they point out that differentiability assumptions of the optimal critic are not met. Therefore, the regularizing effect of GP might be too strong. As a solution, suggested using a loss penalizing the violation of the Lipschitz constraint either explicitly with or implicitly with where in both cases (a) + denotes max(0, a). The first method has only proved viable when used on toy datasets, and led to considerably worse on relatively more complex datasets like CIFAR-10, which is why used the second one, which they termed Lipschitz Penalty (LP). Compared to GP, this term only penalizes the gradient norm when it exceeds 1. As P τ they evaluated the interpolation method described above, and also sampling random local perturbations of real and generated samples, but found no significant improvement compared to P i. proposed dropout in the critic as a way for creating perturbed input pairs to evaluate the explicit Lipschitz penalty, which led to improvements, but still relied on using GP simultaneously. A second family of Lipschitz regularization methods is based on weight normalization, restricting the Lipschitz constant of a network globally instead of only at points of the input space. One such technique is called spectral normalization (SN) proposed in , which is a very efficient and simple method for enforcing a Lipschitz constraint with respect to the 2-norm on a per-layer basis, applicable to neural networks consisting of affine layers and K-Lipschitz activation functions. proposed a similar approach, which can be used to enforce a Lipschitz constraint with respect to the 1-norm and ∞-norm in addition to the 2-norm, while also being compatible with batch normalization and dropout. argued that any Lipschitzconstrained neural network must preserve the norm of the gradient during backpropagation, and to this end proposed another weight normalization technique (showing that it compares favorably to SN, which is not gradient norm preserving), and an activation function based on sorting. VAT is a semi-supervised learning method that is able to regularize networks to be robust to local adversarial perturbation. Virtual adversarial perturbation means perturbing input sample points in such a way that the change in the output of the network induced by the perturbation is maximal in terms of a distance between distributions. This defines a direction for each sample point called the virtual adversarial direction, in which the perturbation is performed. It is called virtual to make the distinction with the adversarial direction introduced in clear, as VAT uses unlabeled data with virtual labels, assigned to the sample points by the network being trained. The regularization term of VAT is called Local Distributional Smoothness (LDS). It is defined as where p is a conditional distribution implemented by a neural network, D(p, p) is a divergence between two distributions p and p, for which chose the Kullback-Leibler divergence (KLD), and is the virtual adversarial perturbation, where is a hyperparameter. VAT is defined as a training method with the regularizer applied to labeled and unlabeled examples. An important detail is that is minimized by keeping p(y|x) fixed and optimizing p(y|x + r vadv) to be close to it. The adversarial perturbation is approximated by the power iteration r vadv ≈ r k, where r 0 is a randomly sampled unit vector and ξ is another hyperparameter. This iterative scheme is an approximation of the direction at x that induces the greatest change in the output of p in terms of the divergence D. found that k = 1 iteration is sufficient in practical situations. 3 argued that penalizing the norm of the gradient as in is more effective than penalizing the Lipschitz quotient directly as in, as the former penalizes the slope of f in all spatial directions around x, unlike the latter, which does so only along (x − y). We hypothesize that using the explicit Lipschitz penalty in itself is insufficient because if one takes pairs of samples x, y randomly from P r, P g or P i (or just one sample and generates a pair for it with random perturbation), the violation of the Lipschitz penalty evaluated at these sample pairs will be far from its maximum, hence a more sophisticated strategy for sampling pairs is required. As we will show, a carefully chosen sampling strategy can in fact make the explicit penalty favorable over the implicit one. Consider the network f as a mapping from the metric space (X, d X) to the metric space (Y, d Y). Let us rewrite with y = x + r to get A given mapping f is K-Lipschitz if and only if for any given x ∈ X, taking the supremum over r in in a value K or smaller. Assuming that this supremum is always achieved for some r, we can define a notion of adversarial perturbation with respect to the Lipschitz continuity for a given x ∈ X as r adv = arg max and the corresponding maximal violation of the K-Lipschitz constraint as We define Adversarial Lipschitz Regularization (ALR) as the method of adding as a regularization term to the training objective that penalizes the violation of the Lipschitz constraint evaluated at sample pairs obtained by adversarial perturbation. We call this term Adversarial Lipschitz Penalty (ALP). To put it in words, ALP measures the deviation of f from being K-Lipschitz evaluated at pairs of sample points where one is the adversarial perturbation of the other. If added to the training objective, it makes the learned mapping approximately K-Lipschitz around the sample points it is applied at. We found that in the case of the WGAN critic it is best to minimize without keeping f (x) fixed. See Appendix A.1 for the semi-supervised case and Appendix A.2 for how VAT can be seen as a special case of Lipschitz regularization. In general, computing the adversarial perturbation is a nonlinear optimization problem. A crude and cheap approximation is r adv ≈ r k, where is the approximated adversarial direction with r 0 being a randomly sampled unit vector. The derivation of this formula is essentially the same as the one described in , but is included in Appendix A.3 for completeness. Unlike in VAT, we do not fix, but draw it randomly from a predefined distribution P over R + to apply the penalty at different scales. Theoretically, ALR can be used with all kinds of metrics d X and d Y, and any kind of model f, but the approximation of r adv imposes a practical restriction. It approximates the adversarial perturbation of x as a translation with length with respect to the 2-norm in the adversarial direction, which is only a perfect approximation if the ratio in is constant for any > 0. This idealized setting is hardly ever the case, which is why we see the search for other approximation schemes as an important future direction. There is a large number of methods for generating adversarial examples besides the one proposed in VAT (; ;), which could possibly be combined with ALR either to improve the approximation performance or to make it possible with new kinds of metrics. The latter is important since one of the strengths of the Wasserstein distance is that it can be defined with any metric d X, a fact that and built on by extending GP to work with metrics other than the Euclidean distance. emphasized the fact that through explicit Lipschitz penalties one could extend WGANs to more general metric spaces as well. In practice, one adds the Monte Carlo approximation of the expectation (averaged over a minibatch of samples) of either or the square of (or both) to the training objective, multiplied by a Lagrange multiplier λ. While VAT adds the expectation of to the training objective, for WGAN we have added the square of the expectation of. To train the Progressive GAN, we have added both the expectation and its square. In the semi-supervised setting, we added only the expectation similarly to VAT. We have found these choices to work best in these scenarios, but a principled answer to this question is beyond the scope of this paper. The target Lipschitz constant K can be tuned by hand, or in the presence of labeled data it is possible to calculate the Lipschitz constant of the dataset . The hyperparameters of the approximation scheme are k, ξ and those of P. Choosing the right hyperparameters can be done by monitoring the number of adversarial perturbations found by the algorithm for which the Lipschitz constraint is violated (and hence contribute a nonzero value to the expectation of), and tuning the hyperparameters in order to keep this number balanced between its maximum (which is the minibatch size) and its minimum (which is 0). If it is too high, it means that either K is too small and should be increased, or the regularization effect is too weak, so one should increase λ. If it is too low, then either the regularization effect is too strong, or ALR is parametrized in a way that it cannot find Lipschitz constraint violations efficiently. In the former case, one should decrease λ. In the latter, one should either decrease K, tune the parameters of P, or increase the number of power iterations k for the price of increased runtime. We have not observed any significant effect when changing the value of ξ in any of the tasks considered. In terms of efficiency when applied to WGANs, ALR compares favorably to the implicit methods penalizing the gradient norm, and to weight normalization techniques as well, as demonstrated in the experiments section. See Appendix A.4 for a showcase of the differences between weight normalization methods, implicit penalty methods and explicit penalty methods, represented by SN, LP and ALR, respectively. The key takeaways are that • penalty methods in a softer regularization effect than SN, • ALR is preferable when the regularized network contains batch normalization (BN) layers, and • ALR gives more control over the regularization effect, which also means there are more hyperparameters to tune. The performance of ALR mostly depends on the speed of the approximation of r adv. The current method requires 1 step of backpropagation for each power iteration step, which means that running time will be similar to that of LP and GP with k = 1. SN is much cheaper computationally than each penalty method, although we believe ALR has the potential to become relatively cheap as well by adopting new techniques for obtaining adversarial examples . We specialize the ALP formula with f being the critic, d X (x, y) = x − y 2, d Y (x, y) = |x − y| and K = 1, and apply it to the WGAN objective to arrive at a version with the explicit penalty, which uses adversarial perturbations as a sampling strategy. It is formulated as where P r,g is a combination of the real and generated distributions (meaning that a sample x can come from both), λ is the Lagrange multiplier, and the adversarial perturbation is defined as This formulation of WGAN in a stable explicit Lipschitz penalty, overcoming the difficulties experienced when one tries to apply it to random sample pairs as shown in. To evaluate the performance of WGAN-ALP, we trained one on CIFAR-10, consisting of 32 × 32 RGB images, using the residual architecture from , implemented in TensorFlow. Closely following , we used the Adam optimizer with parameters β 1 = 0, β 2 = 0.9 and an initial learning rate of 2 × 10 −4 decaying linearly to 0 over 100000 iterations, training the critic for 5 steps and the generator for 1 per iteration with minibatches of size 64 (doubled for the generator). We used as a loss function to optimize the critic. K = 1 was an obvious choice, and we found λ = 100 to be optimal (the training diverged for λ = 0.1, and was stable but performed worse for λ = 10 and 1000). The hyperparameters of the approximation of r adv were set to ξ = 10, P being the uniform distribution over [0.1, 10], and k = 1 power iteration. Both batches from P r and P g were used for regularization. We used Inception Score and FID as our evaluation metrics. The former correlates well with human judgment of image quality and is the most widely used among GAN models, and the latter has been shown to capture model issues such as mode collapse, mode dropping and overfitting, while being a robust and efficient metric . We monitored the Inception Score and FID during training using 10000 samples every 1000 iteration, and evaluated them at the end of training using 50000 samples. We ran the training setting described above 10 times with different random seeds, and calculated the mean and standard deviation of the final Inception Scores and FIDs, while also recording the maximal Inception Score observed during training. We report these values for WGAN-ALP and other relevant GANs (; ; a; ; ; ;) in Table 1. We did not run experiments to evaluate competing models, but included the values reported in the corresponding papers (with the exception of the FID for WGAN-GP, which was taken from Zhou et al. (2019a) ). They used different methods to arrive at the cited , from which that of is the one closest to ours. We show some generated samples in Figure 1a. We also trained WGAN-LP in our implementation. During training, the best observed Inception Score and FID were 8.13 and 18.49, while at the end of training the best final Inception Score and FID were 8.01 and 15.42. To see that ALR indeed restricts the Lipschitz constant of the critic, we monitored the gradient norms during training, which converged to ≈ 5 with λ = 100. This was also the case using LP with λ = 0.1, but the number of Lipschitz constraint violations found by the algorithm were much higher in this case than with ALR. Our toy example in Appendix A.4 showed that when the regularized network contains BN layers, ALR seems to work better than competing methods. In order to see if this still applies in more complex settings, we have trained a variant of WGAN in which the critic contains BN layers (WGAN-BN). did not use BN in the critic as they argued that GP is not valid in that setting, and indeed when we trained WGAN-BN with GP, the best Inception Score observed during training was only 6.29. When we applied ALP to WGAN-BN, the were nearly on par with the original setting without BN, producing an even better maximal Inception Score of 8.71. We leave the question of how BN affects Lipschitz continuity for future work. Generated samples are shown in Figure 1b. made the distinction between one-sided and two-sided penalties, represented by and. The latter is based on the fact that in WGAN, the optimal critic has unit gradient norm on lines connecting points from the optimal coupling π *. showed that since π * is not known in practice, one should use the one-sided penalty, while proposed a method to approximate π * with an auto-encoding scheme. In the limit r 2 → 0 the expression inside the arg max operator in is equivalent to the directional derivative of f along r, and the vector r adv corresponding to the maximum value of the directional derivative at x is equivalent to ∇ x f (x). Since the critic f corresponds to the potential function in the dual formulation of the optimal transport problem, at optimality its gradient at x points towards its coupling y, where (x, y) ∼ π *. From this perspective, sampling pairs (x, x + r adv) using can be seen as an approximation of the optimal coupling π *. To test how reasonable this approximation is, we have trained a WGAN variant with the two-sided explicit penalty formulated as which performed similarly to the one-sided case with λ = 10, but was less stable for other values of λ. The findings of were similar for the case of the implicit penalty. Improving the approximation scheme of r adv might render the formulation using the two-sided penalty preferable in the future. To show that ALR works in a high-dimensional setting as well, we trained a Progressive GAN on the CelebA-HQ dataset , consisting of 1024 × 1024 RGB images. We took the official TensorFlow implementation and replaced the loss function of the critic, which originally used GP, with a version of ALP. Using as the training objective was stable until the last stage of progressive growing, but to make it work on the highest resolution, we had to replace it with meaning that we used the sum of the absolute and squared values of the Lipschitz constraint violation as the penalty. The optimal hyperparameters were λ = 0.1, P being the uniform distribution over [0.1, 100], ξ = 10 and k = 1 step of power iteration. The best FID seen during training with the original GP version was 8.69, while for the modified ALP version it was 14.65. The example shows that while ALP did not beat GP in this case (possibly because the implementation was fine-tuned using GP), it does work in the high-dimensional setting as well. For samples generated by the best performing ALR and GP variants see Appendix A.5. Inspired by VAT, we proposed ALR and shown that it is an efficient and powerful method for learning Lipschitz constrained mappings implemented by neural networks. Resulting in competitive performance when applied to the training of WGANs, ALR is a generally applicable regularization method. It draws an important parallel between Lipschitz regularization and adversarial training, which we believe can prove to be a fruitful line of future research. Since VAT is a semi-supervised learning method, it is important to see how ALR fares in that regime. To show this, we have replicated one of the experiments from. We trained the ConvLarge architecture to classify images from CIFAR-10 with the same setting as described in , except that we did not decay the learning rate, but kept it fixed at 3 × 10 −4. We split the 50000 training examples into 4000 samples for the classification loss, 45000 samples for regularization and 1000 for validation, with equally distributed classes. Test performance was evaluated on the 10000 test examples. We have found that unlike in the unsupervised setting, here it was important to assume f (x) fixed when minimizing the regularization loss, and also to complement the smoothing effect with entropy minimization . The baseline VAT method was ALR specialized with K = 0, d X being the Euclidean metric, d Y being the KL divergence, fixed = 8 and λ = 1. This setting achieved maximal validation performance of 84.2% and test performance 82.46%. After some experimentation, the best performing choice was K = 0, d X being the l 2 metric, d Y the mean squared difference over the logit space (which parametrize the categorical output distribution over which the KL divergence is computed in the case of VAT), P being the uniform distribution over and λ = 1. This way the maximal validation performance was 85.3% and test performance 83.54%. Although this ≈ 1% is improvement is not very significant, it shows that ALR can be a competitive choice as a semi-supervised learning method as well. VAT was defined by considering neural networks implementing conditional distributions p(y|x), where the distribution over discrete labels y was conditioned on the input image x. To see why LDS, the regularization term of VAT, can be seen as special kind of Lipschitz continuity, we will use a different perspective. Consider a mapping f: X → Y with domain X and codomain Y, where X is the space of images and Y is the probability simplex (the space of distributions over the finite set of labels). Since a divergence is in general a premetric (prametric, quasi-distance) on the space of probability measures , and Lipschitz continuity is defined for mappings between metric spaces, let us restrict the divergence D from the VAT formulation to be a metric d Y. used KLD in their experiments, which is not a metric, but one can use e.g. the square root of JSD or the Hellinger distance, which are metrics. Let us metrize the space of images X with d X being the Euclidean metric. From this perspective, the network f is a mapping from the metric space (X, d X) to the metric space (Y, d Y). Let us also assume that we aim to learn a mapping f with the smallest possible f L by setting K to 0. To enforce the condition x + r ∈ X in, we bound the Euclidean norm of r from above by some predefined > 0. If we make the additional assumption that the supremum is always achieved with an r of maximal norm, the denominator in will be constant, hence the formulas with and without it will be equivalent up to a scaling factor. With these simplifications, and reduce to and which are equivalent to and, respectively. Let us consider the question of keeping f (x) fixed when minimizing an implementation detail. With this discrepancy aside, we have recovered VAT as a special case of Lipschitz regularization. We assume that f and d Y are both twice differentiable with respect to their arguments almost everywhere, the latter specifically at x = y. Note that one can easily find a d Y for which the last assumption does not hold, for example the l1 distance. If d Y is translation invariant, meaning that for each u ∈ Y, then its subderivatives at x = y will be independent of x, hence the method described below will still work. Otherwise, one can resort to using a proxy metric in place of d Y for the approximation, for example the l2 distance. so that the second-order Taylor approximation of d(r, x) is d(r, x) ≈ 1 2 r T H(x)r, where H(x) = ∇∇ r d(r, x) r=0 is the Hessian matrix. The eigenvector u of H(x) corresponding to its eigenvalue with the greatest absolute value is the direction of greatest curvature, which is approximately the adversarial direction that we are looking for. The power iteration defined by where r 0 is a randomly sampled unit vector, converges to u if u and r 0 are not perpendicular. Calculating H(x) is computationally heavy, which is why H(x)r i is approximated using the finite differences method as where the equality follows from. The hyperparameter ξ = 0 is introduced here. In summary, the adversarial direction is approximated by the iterative scheme of which one iteration is found to be sufficient and necessary in practice. To showcase the differences between weight normalization methods, implicit penalty methods and explicit penalty methods, represented by SN, LP and ALR, respectively, we devised the following toy example. Suppose that we want to approximate the following real-valued mapping on the 2-dimensional interval [−4, 4] 2: for −4 ≤ x, y ≤ 4. In addition, we want the approximation to be 1-Lipschitz. It is easy to see that the optimal approximation with respect to the mean squared error iŝ This example has connections to WGAN, as the optimal critic is 1-Lipschitz, and its approximation will provide the learning signal to the generator in the form of gradients. Therefore, it is important to closely approximate the gradient of the optimal critic, which is achieved indirectly by Lipschitz regularization. In this example, we will see how closely the different Lipschitz regularization methods can match the gradient of the optimal approximationf opt. We implemented the example in PyTorch. For the approximationf, we use an MLP with 3 hidden layers containing 20, 40 and 20 neurons, respectively, with ReLU activations after the hidden layers, and a variant which also has batch normalization (BN) before the activations, since it has been found that BN hurts adversarial robustness , and hence it should also hurt Lipschitz continuity. We trained the networks for 2 14 iterations, with batches consisting of an input, a corresponding output, and an additional input for regularization. The inputs are drawn uniformly at random from [−4, 4] 2 and the output is defined by. The minibatch size was 64 for input-output pairs, and 1024 for regularization inputs. We used heatmaps to visualize the gradient norm surfaces of the optimal and learned mappings, with the color gradient going from black at 0 to white at 1, see Figure 2. This example is not intended to rank the competing Lipschitz regularization methods, as it always depends on the particular application which one is the best suited, but to show that they are fundamentally different and competent in their own way. (h) ∇xf ALP,λ=0.1,k=0 2 (i) ∇xf ALP,λ=1,k=0 2 (j) ∇xf ALP,λ=10,k=0 2 (k) ∇xf ALP,λ=0.1,k=1 2 (l) ∇xf ALP,λ=1,k=1 2 (m) ∇xf ALP,λ=10,k=1 2 (n) ∇xf ALP,λ=0.1,k=5 2 (o) ∇xf ALP,λ=1,k=5 2 (p) ∇xf ALP,λ=10,k=5 2 Figure 2: Gradient norm surfaces of optimal and learned approximations of f Without any kind of regularization, the network learned to approximate the target function very well, but its gradients look nothing like that off opt, although somehow it is a better match with BN. When we apply SN to the MLP layers, the without BN will be a very smooth mapping with maximum gradient norm far below 1. SN is not compatible with BN, the being only slightly better than the unregularized case. A detail not visible here is that because SN considers weight matrices as linear maps from R n to R m and normalizes them layer-wise, it regularizes globally instead of around actual data samples. In this case, on the whole of R 2 instead of just [−4, 4] 2. For WGANs trained on CIFAR-10, the input space consists of 32 × 32 RGB images with pixel values in [−1, 1], but the trained mapping is regularized on R 32×32×3 instead of just [−1, 1] 32×32×3 (which contains the supports of the real and fake distributions). This can hurt performance if the optimal mapping implemented by a particular network architecture is K-Lipschitz inside these supports, but not in some other parts of R 32×32×3. When the network is regularized using LP, the regularization strength can be controlled by tuning the value of λ. We trained with λ = 0.1, 1 and 10. Without BN, the highest of these values seems to work the best. With BN, the ing mapping is visibly highly irregular. With ALR, in addition to λ, we have additional control over the regularization by the hyperparameters of the approximation scheme of r adv. After some experimentation, we have found the best P for this case was the uniform distribution over [10 −6, 10 −5]. We trained with λ = 0.1, 1 and 10, and k = 0, 1 and 5 power iterations. Arguably, both with and without BN the λ = 1 and k = 5 case seems like the best choice. Without BN, the are quite similar to the LP case, but when BN is introduced, the ing mappings are much smoother than the ones obtained with LP. Figure 4: Images generated using Progressive GAN trained with GP

Represent the short summary for finding the corresponding scientific article (long)

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Multi-task learning promises to use less data, parameters, and time than training separate single-task models. But realizing these benefits in practice is challenging. In particular, it is difficult to define a suitable architecture that has enough capacity to support many tasks while not requiring excessive compute for each individual task. There are difficult trade-offs when deciding how to allocate parameters and layers across a large set of tasks. To address this, we propose a method for automatically searching over multi-task architectures that accounts for resource constraints. We define a parameterization of feature sharing strategies for effective coverage and sampling of architectures. We also present a method for quick evaluation of such architectures with feature distillation. Together these contributions allow us to quickly optimize for parameter-efficient multi-task models. We benchmark on Visual Decathlon, demonstrating that we can automatically search for and identify architectures that effectively make trade-offs between task resource requirements while maintaining a high level of final performance. Multi-task learning allows models to leverage similarities across tasks and avoid overfitting to the particular features of any one task . This can in better generalization and more robust feature representations. While this makes multi-task learning appealing for its potential performance improvements, there are also benefits in terms of resource efficiency. Training a multi-task model should require less data, fewer training iterations, and fewer total parameters than training an equivalent set of task-specific models. In this work we investigate how to automatically search over high performing multi-task architectures while taking such resource constraints into account. Finding architectures that offer the best accuracy possible given particular resource constraints is nontrivial. There are subtle trade-offs in performance when increasing or reducing use of parameters and operations. Furthermore, with multiple tasks, one must take into account the impact of shared operations. There is a large space of options for tweaking such architectures, in fact so large that it is difficult to tune an optimal configuration manually. Neural architecture search (NAS) allows researchers to automatically search for models that offer the best performance trade-offs relative to some metric of efficiency. Here we define a multi-task architecture as a single network that supports separate outputs for multiple tasks. These outputs are produced by unique execution paths through the model. In a neural network, such a path is made up of a subset of the total nodes and operations in the model. This subset may or may not overlap with those of other tasks. During inference, unused parts of the network can be ignored by either pruning out nodes or zeroing out their activations (Figure 1). Such architectures mean improved parameter efficiency because redundant operations and features can be consolidated and shared across a set of tasks. We seek to optimize for the computational efficiency of multi-task architectures by finding models that perform as well as possible while reducing average node use per task. Different tasks will require different capacities to do well, so reducing average use requires effectively identifying which tasks will ask more of the model and which tasks can perform well with less. In addition, performance is affected by how nodes are shared across tasks. It is unclear when allocating resources whether sets of tasks would benefit from sharing parameters or would instead interfere. Figure 1: Feature partitioning can be used to control how much network capacity is used by tasks, and how much sharing is done across tasks. In this work we identify effective partitioning strategies to maximize performance while reducing average computation per task. When searching over architectures, differences in resource use can be compared at different levels of granularity. Most existing work in NAS and multi-task learning searches over the allocation and use of entire layers (; ;), we instead partition out individual feature channels within a layer. This offers a greater degree of control over both the computation required by each task and the sharing that takes place between tasks. The main obstacle to address in searching for effective multi-task architectures is the vast number of possibilities for performing feature partitioning as well as the significant amount of computation required to evaluate and compare arrangements. A naive brute search over different partitioning strategies is prohibitively expensive. We leverage our knowledge of the search space to explore it more effectively. We propose a parameterization of partitioning strategies to reduce the size of the search space by eliminating unnecessary redundancies and more compactly expressing the key features that distinguish different architectures. In addition, the main source of overhead in NAS is evaluation of sampled architectures. It is common to define a surrogate operation that can be used in place of training a full model to convergence. Often a smaller model will be trained for a much shorter number of iterations with the hope that the differences in accuracy that emerge early on correlate with the final performance of the full model. We propose a strategy for evaluating multi-task architectures using feature distillation which provides much faster feedback on the effectiveness of a proposed partitioning strategy while correlating well with final validation accuracy. In this work we provide: • a parameterization that aids automatic architecture search by providing a direct and compact representation of the space of sharing strategies in multi-task architectures. • an efficient method for evaluating proposed parameterizations using feature distillation to further accelerate the search process. • on Visual Decathlon to demonstrate that our search strategy allows us to effectively identify trade-offs between parameter use and performance on diverse and challenging image classification datasets. Multi-Task Learning: There is a wide body of work on multi-task learning spanning vision, language, and reinforcement learning. The following discussion will center around designing multi-task architectures for deep learning in vision (; ;). There are many obstacles to overcome in multi-task architecture design, but the most pressing concerns depend largely on how the problem setting has been defined. Two distinguishing factors include: • Task Ordering: Are all tasks available at all times or are they presented one after the other? • Fixed vs Learned Strategies: Is a uniform strategy applied across tasks or is a task-specific solution learned? The former is important as work in which tasks are presented sequentially must address catastrophic forgetting . This is less of a concern in our work as we train on all tasks at once. As for the latter, finding a solution tuned to a specific set of tasks requires the same sort of outer-loop optimization seen in neural architecture search which is time-consuming and expensive. The contributions presented in this work seek to make this process more manageable. A strong baseline for multi-task architectures is the use of a single shared network . Deep networks are overparameterized in such a way that the same layers can be applied across different domains while producing features that are useful to different ends. Using a shared architecture is common in reinforcement learning to train a single agent to perform many tasks with a uniform observation and action space . A common technique to train single shared models well in both reinforcement learning and vision is distillation of multiple models into one (; ; ; ;). In work where tasks are presented sequentially, the focus is often to build on top of an existing network while not disrupting its ability to solve its original task (; ;). Currently, many methods freeze the network's weights so they are not changed while learning a new task. Examples include masking out specific filter weights or introducing auxiliary layers . Another approach is to dynamically expand a network with additional capacity for each new task . All of these methods build on top of a fixed model, meaning that new tasks must perform the computation required for the original task as well as take additional steps to be task-specific. It is also common to build multi-task architectures from sets of layers that are run in parallel (; ; ;). Cross-stitch networks compute activations for each task as a learned weighted sum across these layers . This sort of soft attention over features can be seen in other multi-task architecture work as well (; . There are approaches to search over paths through these layers such that each task has a unique, optimal execution path . Similar to work in single-task NAS, the best path is found by either reinforcement learning or evolutionary algorithms . The optimal trade-offs in parameter sharing may occur at a more fine-grained level than entire layers, so instead of working with parallel blocks of layers we divide up individual feature channels. Neural Architecture Search: There are three main areas in which contributions are made for more effective architecture search: search space, optimization, and sample evaluation. Search space: With a well-designed search space, it is possible to randomly sample and arrive at high performing solutions (; b). In general, NAS operates in a discrete space where entire layers are included or not. We instead propose a continuous search space where slight changes can be made in how resources are allocated across tasks. This allows alternatives for optimization that would not apply in other NAS work. Optimization: Leading approaches either use reinforcement learning or genetic algorithms for NAS (; ;). This search is difficult and the tradeoffs between approaches are unclear . We test the effectiveness of random sampling and evolutionary strategies optimization . Evaluating Samples: Training a model to convergence is time-consuming and resource intensive. It is not realistic to sample thousands of architectures and train them all. Instead one must use a cheaper form of evaluation. Some options include preserving weights across samples for faster training , successive halving , progressive steps to increase complexity (a), as well as techniques to model the expected performance of sampled architectures (; ;). It is unclear how well surrogate functions correlate with final model performance . We investigate the use of distillation for performing this evaluation. Concretely, given a feature tensor F ∈ R c×h×w we define a binary mask m f ∈ {0, 1} c for each task, and during a forward pass of the network multiply F by m f to zero out all channels not associated with a particular task. We further define a mask for the backward pass, m b ∈ {0, 1} c, whose non-zero elements are a subset of the non-zero elements in m f. Gradients are calculated as usual through standard backpropagation, but any weights we wish to leave unchanged will have their gradients zeroed out according to m b. Together, these masks can capture the training dynamics seen in many existing multi-task architecture designs . For example, one can devote an outsized proportion of features to a task like ImageNet classification, then make these features available during forward inference on a new smaller dataset. A backward mask, m b, can then be defined to ensure that the ImageNet weights remain untouched when finetuning on the new task. There are a number of advantages to allocating resources at the channel level. There is enough flexibility to allow fine-grained control allotting specific weights to particular subsets of tasks. And after training, it is straightforward to prune the network according to any mask configuration. Meaning for simple tasks that only require a small subset of channels we can use a fraction of compute at test time while still leveraging the advantages of joint training with other tasks. An important implementation detail is that masks are applied every other layer. Consider, for example, making a task use half the model. You might think to set half the values of m f to one and apply it to each layer. But that would mean c 2 inputs and c 2 outputs at each layer which only uses one quarter of the original model. Instead, applying a mask at every other layer produces the desired behavior of allocating half the model to the task. Now that we have decided to partition up feature channels, how do we go about finding the best masks for each task? Consider defining a binary matrix that specifies all partitioning masks: c×n, where c is the number of feature channels and n is the total number of tasks. A direct search over this matrix is problematic. It is not straightforward to optimize over a space of many discrete values, and one must account for significant redundancy given that all permutations of channels are equivalent. Moreover, naive random sampling would never cover the full space of partitioning strategies (consider the probability of randomly sampling two masks m that were mutually exclusive). In order to see diverse degrees of feature sharing, the overlap of channels between masks must be explicitly accounted for. Thus instead of searching over M, we propose searching directly over the features of M that determine performance: 1) the number of feature channels used by each task, and 2) the amount of sharing between each pair of tasks. The former decides the overall capacity available for the task while the latter shapes how tasks help or interfere with each other. We explicitly parameterize these factors in a matrix P. We can compute P = 1 c M T M, where the diagonal elements of P provide the percentage of feature channels used by each task and the off-diagonal elements define the percentage of overlapping features between task pairs. When sampling new partitioning strategies, we sample directly from P and identify a corresponding mask M to match it. To remove some ill posed parts of the space we take additional steps to adjust P. More details of this process as well as how we derive M from P can be found in the appendix. This representation has a number of distinct advantages. It is not tied to the number of channels in a given layer, so a single parameterization can be used for layers of different sizes. It is low dimensional, particularly since n is typically much smaller than c. And it is interpretable, providing a clear impression of which tasks require more or less network capacity and which tasks train well together. Moreover we get an immediate and direct measurement of average node usage per task, this is simply the mean of the diagonal of P. We will use this metric to compare the resource efficiency of different proposed partitioning strategies. In order to optimize over different parameterizations P, there are two key ideas to cover: how we choose samples from the space and how we evaluate and compare different samples. We treat our search setting as a black box optimization problem where given a particular parameterization we have a function which returns a score assessing its quality. Based on this score we can then choose how to further sample new parameterizations. We investigate two strategies for finding good constraint matrices. Random sampling: The first is to simply randomly sample values. This has already been demonstrated to serve as a strong baseline in some architecture search work (; a). With the low dimensionality of the matrix as well the additional steps taken to preprocess constraints, it is not unreasonable that much of the space can be covered with random samples. Random samples serve well to map out large swaths of the search space and identify the principle choices that affect final performance. Concretely, a random matrix P can be sampled with values taken uniformly from 0 to 1. If a particular resource target is desired, it is trivial to bias or restrict samples to a specific range of parameter usage. Evolutionary strategies: Because P is continuous, it is possible to search over parameterizations with gradient-based optimization. We run experiments using evolutionary strategies 1. More specifically, we use a simple implementation with the modifications as described by. A gradient direction is approximated by sampling several random directions in the parameter space and computing finite differences to see which directions seem most promising. A weighted average is then computed across all directions to calculate the gradient to update the current parameters. A key feature of our approach is that we modify the algorithm to prioritize parameterizations that use as few channels as necessary per task. An additional L2 weight regularization term is added to the parameters on the diagonal of P. This serves to reduce the number of channels used by each task, in particular those that can be pulled down without affecting the overall accuracy and performance of the model. By controlling the strength of this regularization we can tune the importance of resource efficiency in the search process. Using this optimization strategy is only possible because of the parameterization defined in 3.1. Approximating gradients make sense in the context of the continuous constraints defined in P, and we can more effectively explore the space of multi-task architectures using this signal. This is different from existing architecture search work where search decisions correspond to the coarse selection of entire computational blocks and their connections to each other. Finally, we must evaluate different partitioning schemes. But as discussed, determining the relative effectiveness of one partitioning over another by training models to convergence is expensive. One possible strategy is to train models for a short period of time assuming that the relative differences in performance that appear early in training should correlate well with differences in performance when trained for longer. We instead propose to use feature distillation to observe the representational capacity of a partitioned layer. We test how well shared multi-task layers can reproduce the activations of corresponding single-task layers. By only focusing on a few layers, we reduce total computation and the number of weights that need to be tuned. In addition, directly distilling to intermediate layer activations provides a more direct training signal than a final classification loss. Given a proposed partitioning mask, we initialize new layers to be distilled and load reference models for each target task. Input to the layer is generated by passing through the task-specific pretrained model up to a target depth. The ing features are then passed through the subsequent layers of the pretrained model as well as the new shared layers. In the new layers, intermediate features are masked according to the proposed partitioning. This procedure is illustrated in Figure 2. We use a mean-squared error loss to supervise the shared layers such that their output features match those produced by the reference teacher models. We can measure the effectiveness of this distillation by replacing the original pretrained layers with the new shared layers and measuring the updated model accuracy. Figure 2: Multi-task distillation: At a given step, a teacher network pretrained on a single task is chosen from the available tasks. Features at a target depth are produced then passed through the next residual block of both the teacher and student (where a mask is applied). Features are then compared using a standard MSE loss. This process can be applied to multiple layers at once. It is important to emphasize that we are not using distillation to get a final multi-task model as we do not want to be limited by the performance of individual pre-trained models. Instead, distillation serves as a proxy task for quickly evaluating partitioning strategies. We do not run the distillation process to convergence, only for a brief interval, and it serves as a sufficient signal to provide feedback on different parameterizations. This leads to a dramatic reduction in the time required to evaluate a particular masking strategy. We run a number of experiments to investigate the role our proposed parameterization and distillation play in finding multi-task architectures that minimize task computation and parameter use while achieving high accuracy. All experiments are performed using the Visual Decathlon dataset . Visual Decathlon is composed of many well-established computer vision classification datasets of various sizes and respective difficulties. There is sufficient diversity that it is difficult to determine which datasets would benefit from more or less network capacity and parameter sharing. We investigate how a model performs when trained on nine Decathlon tasks at once (all datasets except for ImageNet). We initialize a shared ResNet model with a separate fully connected layer output for each task. To simplify experiments, we freeze the first two-thirds of the model and only apply feature partioning to the last third. For training, we alternate mini-batches sampled from each dataset and apply a standard cross-entropy classification loss at the appropriate task specific output. More thorough implementation and experiment details can be found in the appendix. It is unclear that performance will necessarily be better with any feature restriction as opposed to using the full model for all tasks. One question is whether partitioning well leads to a reduction in interference across tasks and perhaps improved performance. In addition, we wish to see the overall relationship between performance and feature restriction in this multi-task setting. What's the best performance possible as average feature use is reduced further and further? Before performing our search we need to know that given a sampled set of feature masks M, distillation performance correlates well with the final accuracy of a model trained to convergence. This will determine whether our proposed distillation is a reasonable surrogate in lieu of full training. The higher the correlation between the two, the more confidence we can place in our search process. When performing distillation we initialize the child layers with pretrained layers from an ImageNet model, since the parent single-task networks have also been initialized from the same model. This accelerates the distillation process. The whole process takes just one minute on a P100 GPU. Further details are available in the appendix. We sample many random partitioning masks and run both the distillation procedure and full training to convergence. As a baseline, we also see how well final validation accuracy compares to accuracies seen earlier in training. We compare to the accuracies after 5k and 10k iterations (corresponding to 5 and 10 minutes of training). As seen in Table 1, the distillation procedure (which takes a fraction This allows us to sample and compare many more parameterizations during our search process and have more confidence that the top performing parameterizations will do well when training a full model. Randomly sampling parameterizations: To map out the performance of different partitioning strategies we sample random parameterizations and plot distillation performance against the average percentage of allocated features (the mean of the diagonal of P) in Figure 4 (left). From the distribution of random samples we get an impression of the best performance possible at different degrees of resource use. The combination of fast feedback with distillation plus effective search space coverage with our proposed parameterization produces this information with less samples and in less time. At high levels of average feature use, choice of partitioning can only make so much of a difference. We are interested in the opposite -how well we can do when restricting task computation as much as possible. Here, partitioning well is necessary to achieve high performance. Also it is important to note that explicitly specifying the degree of sharing between tasks is critical. This is shown in the middle plot of Figure 4. We evaluate different partitioning strategies with three fixed sets of values for the diagonal of P, only adjusting the amount of sharing that takes place between tasks. There can be a significant difference in performance in each of these cases, and as expected, sharing affects performance more and more as average parameter use goes down as there is more flexibility when choosing how to overlap features. It is important that feature sharing is parameterized when doing any sort of optimization. Finally, we look at per-task (Figure 5). In this particular setting, every task benefits from using as many features as possible. This may have to do with using a model pretrained on ImageNet, but it makes sense that tasks benefit from using as many features as they can. An important facet to this is how much of those features are shared. As average parameter usage increases across tasks (indicated by a lighter color), individual tasks suffer as they now share more of their features and must deal with the interference of other tasks. Table 2: Validation on Visual Decathlon. Evolutionary strategies: As mentioned above, the distribution of random samples gives an immediate impression of the best level of performance possible as a function of average parameter use. Evolutionary strategies provides a means to more directly push this edge of performance even further. We visualize the search process by plotting samples over the course of optimization overlaid over the distribution of samples found by random sampling (Figure 4 (right) ). ES optimization quickly identifies samples that provide the best accuracy given their current level of parameter use and densely samples in this space, making slight changes for any last available improvements to performance. Furthermore, adjusting the weight decay penalty used during optimization controls the resource use of the final partitioning strategy. This allows us to easily tune the optimization to reach the best architecture that meets specific resource needs. The best parameterization found with evolutionary strategies outperforms a number of baselines for performing partitioning as seen in Table 2. We compare across several strategies with different degrees of feature use and sharing. We measure validation accuracy of models trained to convergence (averaged over five trials). These baselines include: independent partitions that split features evenly across tasks, sharing half of available feature channels and splitting the rest, and finally, sharing all feature channels. In line with our random sampling, the more channels given across tasks, the better performance. Sharing everything does the best amongst these baselines. However, using the parameterization found from our optimization both reduces average channel use and achieves better performance overall. We see that there exist partitioning strategies that cut down average feature dramatically while still maintaining the same overall performance. This is in large part due to simple tasks that only need a small fraction of channels (DPed for example in Fig 5). By taking away the interference caused by these simpler tasks, harder tasks stand to gain more and that can be seen in Table 2 with tasks like CIFAR100, Flowers, and Omniglot seeing the largest gains from an effective partitioning strategy. In this work we investigate efficient multi-task architecture search to quickly find models that achieve high performance under a limited per-task budget. We propose a novel strategy for searching over feature partitioning that automatically determines how much network capacity should be used by each task and how many parameters should be shared between tasks. We design a compact representation to serve as a search space, and show that we can quickly estimate the performance of different partitioning schemes by using feature distillation. Refining P: We define a simple set of operations that convert from the raw search space P to a constraint matrix P that is more likely to correspond to feasible masks. Given the knowledge that pairwise values of P are conditioned on its diagonal terms, it is not possible for there to be more overlap between two tasks than the channels used by any one task. That is, no off-diagonal element M ij should be greater than the corresponding diagonal elements M ii and M jj. We remap all off-diagonal elements to appropriate values determined by the diagonal of the matrix. This means that for any off-diagonal element in P, 0 now maps to the minimum possible overlap and 1 to the maximum possible overlap of the two tasks. The procedure is defined as follows: Figure 6: Measuring correlation of feature activations across three blocks of task-specific ResNet models given a shared input image. Even though no restriction is made when finetuning individual models, the features produced are highly correlated through the first two-thirds of the model, and greater task differentiation is not seen until the end. Differentiation in early blocks would normally occur due to different batch normalization statistics across datasets, but that is controlled for here. Multi-task training: For full model training, we use a batchsize of 64 with SGD and momentum at a learning rate of 0.05 for 100k iterations, dropping to a learning rate of 0.005 at iteration 75k. All training was done on a single Nvidia P100 GPU. We followed the exact training, validation, and test splits provided by Visual Decathlon. Several steps are taken to ensure that a model trained simultaneously on multiple tasks converges well: • Batch normalization: We maintain separate batch normalization statistics per task as done in . This adds minimal parameter overhead and accelerates training. • Momentum: We maintain separate gradient statistics when using momentum with SGD. This is important given our use of feature partitioning. At any given training step, we do not want the unused weights associated with other tasks to be updated. • Training curriculum: Rather than uniformly sampling across tasks we apply a simple strategy to choose mini-batches from tasks inversely proportional to their current training accuracy. Tasks that lag behind get sampled more often. Evidence for this approach has been demonstrated in a multi-task reinforcement learning setting . We find a curriculum over tasks more effective than a curriculum over individual samples . • Pretrained ImageNet model: All experiments are performed with a pretrained ImageNet model. The model is trained using the PyTorch implementation made available by. Because ImageNet is orders of magnitude larger than any of the other datasets in Decathlon and takes much longer to train we exclude it in our partitioning experiments to focus on the interactions of other datasets. The main hyperparameters that determine performance were learning rate and a temperature term that controlled the task sampling curriculum. This temperature term determines whether minibatches are sampled uniformly across tasks or whether tasks with low training accuracy are weighted more heavily. For both hyperparameters we arrive at the final value by a simple grid search. Final validation accuracy reported in Table 2 (in the main paper) is averaged across 5 trials. Frozen layers: To further simplify experiments, we freeze and share the first two-thirds of the network. Partitioning is thus only performed on the last third of the model. By only updating the weights of the last block, we focus attention on the layers where task-specific features are most important without restricting the model's representational capacity to fit each task. In all of our experiments we use an ImageNet-pretrained ResNet model made up of three computational blocks with four layers each. We freeze the first two computational blocks and only perform partitioning on the last set of layers. The justification for this stems from analysis performed with individual single task models. We compare feature differences across finetuned task-specific models. These models were trained with no restrictions initialized from an ImageNet model until converging to high accuracy on some target task. Because we start with a pretrained model we can make mean-ingful comparisons of each model's channel activations to see how task feature use diverges after finetuning. We compare intermediate task features after passing in a shared image into every model. An important detail here is that we control for the batch normalization statistics associated with the dataset that the image is sampled from. The subsequent features produced by each model are almost identical all the way up through the first two-thirds of the model. Aside from subtle differences, task-specific differentiation did not occur until the final third of the model where features were still somewhat correlated but differed dramatically model to model. This is visualized in Figure 6. Because of this we decided the task-specific differentiation afforded by feature partitioning would not be as important in earlier stages of the model, and experiments would be more informative and also faster to run while focusing only on the last set of layers. Distillation details: We do not use the accuracy-based curriculum used in normal training during distillation and instead alternate mini-batches uniformly across each task. Distillation training is done for a brief 3000 iterations with a batch size of 4 and a learning rate of 1 which is dropped by a factor of 10 at iteration 2000. Distillation is done on the last four ResNet layers at once to match the final training setting as closely as possible. All scores reported when performing distillation are averaged across three trials. The optimization curves shown in the paper are from runs that have each taken 1000 samples, these were performed on machines with 4 P100 GPUs. Given that sample evaluation takes roughly a minute, the whole procedure takes just over four hours. At each step, 16 random parameter directions are sampled and these are both added and subtracted from the current parameterization P to produce 32 new samples to evaluate. A gradient is calculated based on the of these samples, and a gradient descent step is applied to the current parameters with a learning rate of 0.1. Both clipping and a sigmoid operation were tested to ensure that values remain between 0 and 1 with no discernible difference in optimization performance.

automatic search for multi-task architectures that reduce per-task feature use

Represent the short summary for finding the corresponding scientific article (long)

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.