Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to the original performance, given a fixed network initialization. We evaluate our method in various initialization settings. Experiments on multiple datasets, MNIST, CIFAR10, PASCAL-VOC, and CUB-200, demonstrate the ad-vantage of our approach compared to alternative methods. Finally, we include a real-world application of dataset distillation to the continual learning setting: we show that storing distilled images as episodic memory of previous tasks can alleviate forgetting more effectively than real images. proposed network distillation as a way to transfer the knowledge from an ensemble of many separately-trained networks into a single, typically compact network, performing a type of model compression. In this paper, we are considering a related but orthogonal task: rather than distilling the model, we propose to distill the dataset. Unlike network distillation, we keep the model fixed but encapsulate the knowledge of the entire training dataset, which typically contains thousands to millions of images, into a small number of synthetic training images. We show that we can go as low as one synthetic image per category, training the same model to reach surprisingly good performance on these synthetic images. For example, in Figure 1a, we compress 60, 000 training images of MNIST digit dataset into only 10 synthetic images (one per category), given a fixed network initialization. Training the standard LENET on these 10 images yields test-time MNIST recognition performance of 94%, compared to 99% for the original dataset. For networks with unknown random weights, 100 synthetic images train to 89%. We name our method Dataset Distillation and these images distilled images. But why is dataset distillation interesting? First, there is the purely scientific question of how much data is encoded in a given training set and how compressible it is? Second, we wish to know whether it is possible to "load up" a given network with an entire dataset-worth of knowledge by a handful of images. This is in contrast to traditional training that often requires tens of thousands of data samples. Finally, on the practical side, dataset distillation enables applications that require compressing data with its task. We demonstrate that under the continual learning setting, storing distilled images as memory of past task and data can alleviate catastrophic forgetting . A key question is whether it is even possible to compress a dataset into a small set of synthetic data samples. For example, is it possible to train an image classification model on synthetic images that are not on the manifold of natural images? Conventional wisdom would suggest that the answer is no, as the synthetic training data may not follow the same distribution of the real test data. Yet, in this work, we show that this is indeed possible. We present an optimization algorithm for synthesizing a small number of synthetic data samples not only capturing much of the original training data but also tailored explicitly for fast model training with only a few data point. To achieve our goal, we first derive the network weights as a We distill the knowledge of tens of thousands of images into a few synthetic training images called distilled images. On MNIST, 100 distilled images can train a standard LENET with a random initialization to 89% test accuracy, compared to 99% when fully trained. On CIFAR10, 100 distilled images can train a network with a random initialization to 41% test accuracy, compared to 80% when fully trained. In Section 3.6, we show that these distilled images can efficiently store knowledge of previous tasks for continual learning. differentiable function of our synthetic training data. Given this connection, instead of optimizing the network weights for a particular training objective, we optimize the pixel values of our distilled images. However, this formulation requires access to the initial weights of the network. To relax this assumption, we develop a method for generating distilled images for randomly initialized networks. To further boost performance, we propose an iterative version, where the same distilled images are reused over multiple gradient descent steps so that the knowledge can be fully transferred into the model. Finally, we study a simple linear model, deriving a lower bound on the size of distilled data required to achieve the same performance as training on the full dataset. We demonstrate that a handful of distilled images can be used to train a model with a fixed initialization to achieve surprisingly high performance. For networks pre-trained on other tasks, our method can find distilled images for fast model fine-tuning. We test our method on several initialization settings: fixed initialization, random initialization, fixed pre-trained weights, and random pre-trained weights. Extensive experiments on four publicly available datasets, MNIST, CIFAR10, PASCAL-VOC, and CUB-200, show that our approach often outperforms existing methods. Finally, we demonstrate that for continual learning methods that store limited-size past data samples as episodic memory , storing our distilled data instead is much more effective. Our distilled images contain richer information about the past data and tasks, and we show experimental evidence on standard continual learning benchmarks. Our code, data, and models will be available upon publication. Knowledge distillation. The main inspiration for this paper is network distillation , a widely used technique in ensemble learning and model compression (; ;). While network distillation aims to distill the knowledge of multiple networks into a single model, our goal is to compress the knowledge of an entire dataset into a few synthetic data. Our method is also related to the theoretical concept of teaching dimension, which specifies the minimal size of data needed to teach a target model to a learner . However, methods (; 2015) inspired by this concept require the existence of target models, which our method does not. Dataset pruning, core-set construction, and instance selection. Another way to distill knowledge is to summarize the entire dataset by a small subset, either by only using the "valuable" data for model training (; ;) or by only labeling the "valuable" data via active learning . Similarly, core-set construction (; ; ;) and instance selection (Olvera-López et al., 2010) methods aim to select a subset of the entire training data, such that models trained on the subset will perform as well as the model trained on the full dataset. For example, solutions to many classical linear learning algorithms, e.g., Perceptron and SVMs , are weighted sums of subsets of training examples, which can be viewed as core-sets. However, algorithms constructing these subsets require many more training examples per category than we do, in part because their "valuable" images have to be real, whereas our distilled images are exempt from this constraint. Gradient-based hyperparameter optimization. Our work bears similarity with gradient-based hyperparameter optimization techniques, which compute the gradient of hyperparameter w.r.t. the final validation loss by reversing the entire training procedure (; ; ;). We also backpropagate errors through optimization steps. However, we use only training set data and focus more heavily on learning synthetic training data rather than tuning hyperparameters. To our knowledge, this direction has only been slightly touched on previously . We explore it in greater depth and demonstrate the idea of dataset distillation in various settings. More crucially, our distilled images work well across random initialization weights, not possible by prior work. Understanding datasets. Researchers have presented various approaches for understanding and visualizing learned models (; ; ; ;). Unlike these approaches, we are interested in understanding the intrinsic properties of the training data rather than a specific trained model. Analyzing training datasets has, in the past, been mainly focused on the investigation of bias in datasets . For example, proposed to quantify the "value" of dataset samples using cross-dataset generalization. Our method offers a different perspective for understanding datasets by distilling full datasets into a few synthetic samples. Given a model and a dataset, we aim to obtain a new, much-reduced synthetic dataset which performs almost as well as the original dataset. We first present our main optimization algorithm for training a network with a fixed initialization with one gradient descent (GD) step (Section 3.1). In Section 3.2, we derive the resolution to a more challenging case, where initial weights are random rather than fixed. In Section 3.3, we further study a linear network case to help readers understand both the properties and limitations of our method. We also discuss the distribution of initial weights with which our method can work well. In Section 3.4, we extend our approach to reuse the same distilled images over 2, 000 gradient descent steps and largely improve the performance. Finally, Section 3.5 discusses dataset distillation for different initialization distributions. Finally, in Section 3.6, we show that our distilled images can be used as effective episodic memory for continual learning tasks., we parameterize our neural network as θ and denote (x i, θ) as the loss function that represents the loss of this network on a data point x i. Our task is to find the minimizer of the empirical error over entire training data: where for notation simplicity we overload the (·) notation so that (x, θ) represents the average error of θ over the entire dataset. We make the mild assumption that is twice-differentiable, which holds true for the majority of modern machine learning models and tasks. Standard training usually applies minibatch stochastic gradient descent or its variants. At each step t, a minibatch of training data x t = {x t,j} n j=1 is sampled to update the current parameters as where η is the learning rate. Such a training process often takes tens of thousands or even millions of update steps to converge. Instead, we learn a tiny set of synthetic distilled training datax with M N and a corresponding learning rateη so that a single GD step such as Input: p(θ0): distribution of initial weights; M: the number of distilled data Input: α: step size; n: batch size; T: the number of optimization iterations;η0: initial value forη either from N (0, I) or from real training images. Initializeη ←η0 2: for each training step t = 1 to T do 3: Get a minibatch of real training data xt = {xt,j} n j=1 4: Sample a batch of initial weights θ for each sampled θ Compute updated parameter with GD: θ Evaluate the objective function on real training data: 10: end for Output: distilled datax and optimized learning rateη using these learned synthetic datax can greatly boost the performance on the real test set. Given an initial θ 0, we obtain these synthetic datax and learning rateη by minimizing the objective below L: where we derive the new weights θ 1 as a function of distilled datax and learning rateη using Equation 2 and then evaluate the new weights over all the real training data x. The loss L(x,η; θ 0) is differentiable w.r.t.x andη, and can thus be optimized using standard gradient-based methods. In many classification tasks, the data x may contain discrete parts, e.g., class labels in data-label pairs. For such cases, we fix the discrete parts rather than learn them. Unfortunately, the above distilled data is optimized for a given initialization, and does not generalize well to other initializations, as it encodes the information of both the training dataset x and a particular network initialization θ 0. To address this issue, we turn to calculate a small number of distilled data that can work for networks with random initializations from a specific distribution. We formulate the optimization problem as follows: where the network initialization θ 0 is randomly sampled from a distribution p(θ 0). During our optimization, the distilled data are optimized to work well for randomly initialized networks. In practice, we observe that the final distilled data generalize well to unseen initializations. In addition, these distilled images often look quite informative, encoding the discriminative features of each category (e.g., in Figure 2). Algorithm 1 illustrates our main method. As the optimization (Equation 4) is highly non-linear and complex, the initialization ofx plays a critical role in the final performance. We experiment with different initialization strategies and observe that using random real images as initialization often produces better distilled images compared to random initialization, e.g., N (0, I). For a compact set distilled data to be properly learned, it turns out having only one GD step is far from sufficient. Next, we derive a lower bound on the size of distilled data needed for a simple model with arbitrary initial θ 0 in one GD step, and discuss its implications on our algorithm. This section studies our formulation in a simple linear regression problem with quadratic loss. We derive a lower bound of the size of distilled data needed to achieve the same performance as training on the full dataset for arbitrary initialization with one GD step. Consider a dataset x containing N data-target pairs, where d i ∈ R D and t i ∈ R, which we represent as two matrices: an N × D data matrix d and an N × 1 target matrix t. Given the mean squared error metric and a D × 1 weight matrix θ, we have We aim to learn M synthetic data-target pairsx = (d,t), whered is an M × D matrix,t an M × 1 matrix (M N), andη the learning rate, to minimize (x, θ 0 −η∇ θ0 (x, θ 0)). The updated weight matrix after one GD step with these distilled data is For the quadratic loss, there always exists distilled datax that can achieve the same performance as training on the full dataset x (i.e., attaining the global minimum) for any initialization θ 0. For example, given any global minimum solution θ *, we can choosed = N · I andt = N · θ *. But how small can the size of the distilled data be? For such models, the global minimum is attained at any θ * in the condition above, we have Here we make the mild assumption that the feature columns of the data matrix d are independent (i.e., d T d has full rank). For ax = (d,t) to satisfy the above equation for any θ 0, we must have which implies thatd Td has full rank and M ≥ D. Discussion. The analysis above only considers a simple case but suggests that any small number of distilled data fail to generalize to arbitrary initial θ 0. This is intuitively expected as the optimization target (x, θ 1) = (x, θ 0 −η∇ θ0 (x, θ 0)) depends on the local behavior of (x, ·) around θ 0 (e.g., gradient magnitude), which can be drastically different across various initializations θ 0. The lower bound M ≥ D is a quite restricting one, considering that real datasets often have thousands to even hundreds of thousands of dimensions (e.g., images). This analysis motivates us to avoid the limitation of using one GD step by extending to multiple steps in the next section. We extend Algorithm 1 to more than one gradient descent steps by changing Line 6 to multiple sequential GD steps on the same batch of distilled data, i.e., each step i performs and changing Line 9 to backpropagate through all steps. We do not share the same learning rates across steps as later steps often require lower learning rates. Naively computing gradients is memory and computationally intensive. Therefore, we exploit a recent technique called back-gradient optimization, which allows for significantly faster gradient calculation in reverse-mode differentiation . Specifically, back-gradient optimization formulates the necessary second-order terms into efficient Hessian-vector products , which can be easily calculated with modern automatic differentiation systems such as PyTorch . There is freedom in choosing the distribution of initial weights p(θ 0). In this work, we explore the following four practical choices in the experiments: • Random initialization: Distribution over random initial weights, e.g., He Initialization and Xavier Initialization for neural networks. • Fixed initialization: A particular fixed network initialized by the method above. • Random pre-trained weights: Distribution over models pre-trained on other tasks or datasets, e.g., ALEXNET networks trained on ImageNet . • Fixed pre-trained weights: A particular fixed network pre-trained on other tasks and datasets. Distillation with pre-trained weights. Such learned distilled data essentially fine-tune weights pre-trained on one dataset to perform well for a new dataset, thus bridging the gap between the two domains. Domain mismatch and dataset bias represent a challenging problem in machine learning (; ;). In this work, we characterize the domain mismatch via distilled data. In Section 4.1.2, we show that a small number of distilled images are sufficient to quickly adapt convolutional neural network (CNN) models to new datasets and tasks. To guard against domain shift, several continual learning methods store a subset of training samples in a small memory buffer, and restrict future updates to maintain reasonable performance on these stored samples (; ; ;). As our distilled images contain rich information about the past training data and task, they could naturally serve as a compressed memory of the past. To test this, we modify a recent continual learning method called Gradient Episodic Memory (GEM) . GEM enforces inequality constraints such that the new model, after being trained on the new data and task, should perform at least as well as the old model on the previously stored data and tasks. Here, we store our distilled data for each task instead of randomly drawn training samples as used in GEM. We use the distilled data to construct inequality constraints, and solve the optimization using quadratic programming, same as in GEM. As shown in Section 4.2, our method compares favorably against several baselines that rely on real images. In this section, we report experiments of regular image classifications on MNIST and CIFAR10 , adaptation from ImageNet to PASCAL-VOC and CUB-200 , and continual learning on permuted MNIST and CIFAR100. Baselines. For each experiment, in addition to baselines specific to the setting, we generally compare our method against baselines trained with data derived or selected from real training images: • Random real images: We randomly sample the same number of real images per category. • Optimized real images: We sample different sets of random real images as above, and choose the top 20% best performing sets. • k-means++: We apply k-means++ clustering to each category, and extract the cluster centroids. Table 1: Comparison between our method and various baselines. All methods use ten images per category (100 in total), except for the average real images baseline, which reuses the same images in different GD steps. For MNIST, our method uses 2000 GD steps, and baselines use the best among #steps ∈ {1, 100, 500, 1000, 2000}. For CIFAR10, our method uses 50 GD steps, and baselines use the best among #steps ∈ {1, 5, 10, 20, 500}. In addition, we include a K-nearest neighbors (KNN) baseline, and report best among all combinations of distance metric ∈ {l1, l2} and one or three neighbors. • Average real images: We compute the average image for each category. Please see the appendix for more details about training and baselines, and additional . We first present experimental on training classifiers either from scratch or adapting from pre-trained weights. For MNIST, the distilled images are trained with LENET , which achieves about 99% test accuracy if conventionally trained. For CIFAR10, we use a network architecture that achieves around 80% test accuracy if conventionally trained. For ImageNet adaptations, we use an ALEXNET. We use 2000 GD steps for MNIST and 50 GD steps for CIFAR10. For random initializations and random pre-trained weights, we report means and standard deviations over 200 held-out models, unless otherwise stated. For baselines, we perform each evaluation on 200 held-out models using all possible combinations of learning rate ∈ {distilled learning ratesη *, 1e-3, 3e-3, 1e-2, 3e-2, 1e-1, 3e-1} and several choices of numbers of training GD steps (see table captions for details), and report with the best performing combination. Fixed initialization. With access to initial network weights, distilled images can directly train a fixed network to reach high performance. Experiment show that just 10 distilled images (one per class) can boost the performance of a LENET with an initial accuracy 8.25% to a final accuracy of 93.82% on MNIST in 2000 GD steps. Using 100 distilled images (ten per class) can raise the final accuracy can be raised to 94.41%, as shown in the first column of Table 1. Similarly, 100 distilled images can train a network with an initial accuracy 10.75% to test accuracy of 45.15% on CIFAR10 in 50 GD steps. Figure 2 distilled images trained with randomly sampled initializations using Xavier Initialization . While the ing average test accuracy from these images are not as high as those for fixed initialization, these distilled images crucially do not require a specific initial point, and thus could potentially generalize to a much wider range of starting points. In Section 4.2 below, we present preliminary of achieving nontrivial gains from applying such distilled images to classifier networks during a continual learning training process. Table 2: Adapting models among MNIST (M), USPS (U), and SVHN (S) using 100 distilled images. Our method outperforms few-shot domain adaptation and other baselines in most settings. Due to computation limitations, the 100 distilled images are split into 10 minibatches applied in 10 sequential GD steps, and the entire set of 100 distilled images is iterated through 3 times (30 GD steps in total). For baselines, we train the model using the same number of images with {1, 3, 5} times and report the best . Table 3: Adapting an ALEXNET pre-trained on ImageNet to PASCAL-VOC and CUB-200. We use one distilled image per category, repeatedly applied via three GD steps. Our method significantly outperforms the baselines. For baselines, we train the model with {1, 3, 5} GD steps and report the best. Results are over 10 runs. Multiple gradient descent steps. Section 3.3 has shown theoretical limitations of using only one step in a simple linear case. In Figure 3, we empirically verify for deep networks that using multiple steps drastically outperforms the single step method, given the same number of distilled images. Table 1 summarizes the of our method and all baselines. Our method with both fixed and random initializations outperforms all the baselines on CIFAR10 and most of the baselines on MNIST. Next, we show the extended setting of our algorithm discussed in Section 3.5, where the weights are not randomly initialized but pre-trained on a particular dataset. In this section, for random initial weights, we train the distilled images on 2000 pre-trained models and evaluate them on 200 unseen models. Fixed and random pre-trained weights on digits. As shown in Section 3.5, we can optimize distilled images to quickly fine-tune pre-trained models on a new dataset. Table 2 shows that our method is more effective than various baselines on adaptation between three digits datasets: MNIST, USPS , and SVHN . We also compare our method against a stateof-the-art few-shot domain adaptation method . Although our method uses the entire training set to compute the distilled images, both methods use the same number of images to distill the knowledge of target dataset. Prior work is outperformed by our method with fixed pre-trained weights on all the tasks, and by our method with random pre-trained weights on two of the three tasks. This shows that our distilled images effectively compress the information of target datasets. Fixed pre-trained ALEXNET to PASCAL-VOC and CUB-200. In Table 3, we adapt a widely used ALEXNET model pre-trained on ImageNet to image classification on PASCAL-VOC and CUB-200 datasets. Given only one distilled image per category, our method outperforms various baselines significantly. Our method is on par with fine-tuning on the full datasets with thousands of images. We modify Gradient Episodic Memory (GEM) to store distilled data for each task rather than real training images. Experiments in use large memory buffers, up to 25% of the training set. Instead, we focus on a more realistic scenario where the buffer is rather small (≤ 1% of the training set). Following the experiment settings and architecture choices from , we consider two continual learning tasks: Permuted MNIST CIFAR100 Memory size per task = 10 iCaRL -42.4 GEM 67 No memory buffer EWC 63.5 45.6 Table 4: Continual learning . Distilled images are trained with random Xavier Initialization distribution. For permuted MNIST, they are trained with 2000 GD steps. For CIFAR100, they are trained for 200 GD steps. • Permuted MNIST: 20 classification tasks each formed by using a different permutation to arrange pixels from MNIST images. Each task contains 1, 000 training images. The classifier used has 2 hidden layers each with 100 neurons. • CIFAR100: 20 classification tasks formed by splitting the 100 classes into 20 equal subsets of 5 classes. Each task contains 2, 500 training images. The classifier used is RESNET18 . Table 4 shows that using distilled data drastically improves final overall accuracy on all tasks, and reduces buffer size by up to 5× compared to the original GEM that uses real images. We only report the basic iCaRL setting on CIFAR100 because it requires similar input distributions across all tasks, and it is unclear how to properly inject distilled images into its specialized examplar selection procedure. The appendix details the hyper-parameters tested for each continual learning algorithm. In this paper, we have presented dataset distillation for compressing the knowledge of entire training data into a few synthetic training images. We demonstrate how to train a network to reach surprisingly good performance with only a small number of distilled images. Finally, the distilled images can efficiently store the memory of previous tasks in the continual learning setting. Many challenges remain for knowledge distillation of data. Although our method generalizes well to random initializations, it is still limited to a particular network architecture. Since loss surfaces for different architectures might be drastically different, a more flexible method of applying the distilled data may overcome this difficulty. Another limitation is the increasing computation and memory requirements for finding the distilled data as the number of images and steps increases. To compress large-scale datasets such as ImageNet, we may need first-order gradient approximations to make the optimization computationally feasible. Nonetheless, we are encouraged by the findings in this paper on the possibilities of training large models with a few distilled data, leading to potential applications such as accelerating network evaluation in neural architecture search . We believe that the ideas developed in this work might give new insights into the quantity and type of data that deep networks are able to process, and hopefully inspire others to think along this direction. In our experiments, we disable dropout layers in the networks due to the randomness and computational cost they introduce in distillation. Moreover, we initialize the distilled learning rates with a constant between 0.001 and 0.02 depending on the task, and use the Adam solver with a learning rate of 0.001. For random initialization and random pre-trained weights, we sample 4 to 16 initial weights in each optimization step. We run all the experiments on NVIDIA 1080 Ti, Titan Xp, and V100 GPUs. We use one GPU for fixed initial weights and up to four GPUs for random initial weights. Each training typically takes 1 to 6 hours. Below we describe the details of our baselines using real training images. • Random real images: We randomly sample the same number of real images per category. We evaluate the performance over 10 randomly sampled sets. • Optimized real images: We sample 50 sets of real images using the procedure above, pick 10 sets that achieve the best performance on 20 held-out models and 1024 randomly chosen training images, and evaluate the performance of these 10 sets. • k-means++: For each category, we use k-means++ clustering to extract the same number of cluster centroids as the number of distilled images in our method. We evaluate the method over 10 runs. • Average real images: We compute the average image of all the images in each category, which is repeated to match the same total number of images. We evaluate the model only once because average images are deterministic. To enforce our optimized learning rate to be positive, we apply softplus to a scalar trained parameter. For continual learning experiment on CIFAR10 dataset, to compare with GEM , we replace the Batch normalization with Group normalization in RESNET18 , as it is difficult to run back-gradient optimization through batch norm running statistics. For a fair comparison, we use the same architecture for our method and other baselines. For dataset distillation experiments with pre-trained initial weights, distilled images are initialized with N at the beginning of training. For other experiments, distilled images are initialized with random real samples, unless otherwise stated. For the compared continual learning methods, we report the best report from the following combinations of hyper-parameters: • GEM: -γ ∈ {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1}. -learning rate = 0.1. • iCARL: -regularization ∈ {0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1.0}. -learning rate = 0.1. • Figures 4 and 5 show distilled images trained for random initializations on MNIST and CI-FAR10. • Figures 6, 7, and 8 show distilled images trained for adapting random pre-trained models on digits datasets including MNIST, USPS, and SVHN. Step: 0 LRs: 0.0073, 0.0072, 0.0153 0 1 2 3 4 5 6 7 8 9 Step: 1 LRs: 0.0156, 0.0144, 0.0246 Step: 2 LRs: 0.0114, 0.0543, 0.0371 Step: 3 LRs: 0.0151, 0.0564, 0.0631 Step: 4 LRs: 0.0161, 0.0437, 0.0441 Step: 5 LRs: 0.0538, 0.1200, 0.0960 Step: 6 LRs: 0.0324, 0.0490, 0.0362 Step: 7 LRs: 0.1045, 0.0609, 0.0532 Step: 8 LRs: 0.0375, 0.0465, 0.0437 Step: 9 LRs: 0.1236, 0.1507, 0.0439 Figure 6: Dataset distillation for adapting random pretrained models from USPS to MNIST. 100 distilled images are split into 10 GD steps, shown as 10 rows here. Top row is the earliest GD step, and bottom row is the last. The 10 steps are iterated over three times to finish adaptation, leading to a total of 30 GD steps. These images train average test accuracy on 200 held out models from 67.54% ± 3.91% to 92.74% ± 1.38%. Step: 0 LRs: 0.0038, 0.0027, 0.0063 0 1 2 3 4 5 6 7 8 9 Step: 1 LRs: 0.0035, 0.0044, 0.0030 Step: 2 LRs: 0.0039, 0.0040, 0.0047 Step: 3 LRs: 0.0035, 0.0034, 0.0026 Step: 4 LRs: 0.0029, 0.0040, 0.0050 Step: 5 LRs: 0.0022, 0.0032, 0.0027 Step: 6 LRs: 0.0031, 0.0039, 0.0019 Step: 7 LRs: 0.0005, 0.0005, 0.0024 Step: 8 LRs: 0.0008, 0.0005, 0.0018 Step: 9 LRs: 0.0018, 0.0010, 0.0009 Figure 7: Dataset distillation for adapting random pretrained models from MNIST to USPS. 100 distilled images are split into 10 GD steps, shown as 10 rows here. Top row is the earliest GD step, and bottom row is the last. The 10 steps are iterated over three times to finish adaptation, leading to a total of 30 GD steps. These images train average test accuracy on 200 held out models from 90.43% ± 2.97% to 95.38% ± 1.81%. Step: 0 LRs: 0.0050, 0.0105, 0.0119 0 1 2 3 4 5 6 7 8 9 Step: 1 LRs: 0.0099, 0.0269, 0.0162 Step: 2 LRs: 0.0049, 0.0232, 0.0160 Step: 3 LRs: 0.0143, 0.0532, 0.0438 Step: 4 LRs: 0.0072, 0.0195, 0.0389 Step: 5 LRs: 0.0228, 0.0540, 0.0382 Step: 6 LRs: 0.0392, 0.0347, 0.0489 Step: 7 LRs: 0.0277, 0.0373, 0.0308 Step: 8 LRs: 0.0525, 0.0225, 0.0192 Step: 9 LRs: 0.0321, 0.0707, 0.0250 Figure 8: Dataset distillation for adapting random pretrained models from SVHN to MNIST. 100 distilled images are split into 10 GD steps, shown as 10 rows here. Top row is the earliest GD step, and bottom row is the last. The 10 steps are iterated over three times to finish adaptation, leading to a total of 30 GD steps. These images train average test accuracy on 200 held out models from 51.64% ± 2.77% to 85.21% ± 4.73%.

Represent the TLDR for finding the corresponding scientific article

We propose to distill a large dataset into a small set of synthetic data that can train networks close to original performance.

represent the tldr for finding the corresponding scientific article
Fewshot example: "We introduce an underparameterized, nonconvolutional, and simple deep neural network that can, without training, effectively represent natural images and solve image processing tasks like compression and denoising competitively." == "Deep neural networks, in particular convolutional neural networks, have become highly effective tools for compressing images and solving inverse problems including denoising, inpainting, and reconstruction from few and noisy measurements. This success can be attributed in part to their ability to represent and generate natural images well. Contrary to classical tools such as wavelets, image-generating deep neural networks have a large number of parameters---typically a multiple of their output dimension---and need to be trained on large datasets. In this paper, we propose an untrained simple image model, called the deep decoder, which is a deep neural network that can generate natural images from very few weight parameters. The deep decoder has a simple architecture with no convolutions and fewer weight parameters than the output dimensionality. This underparameterization enables the deep decoder to compress images into a concise set of network weights, which we show is on par with wavelet-based thresholding. Further, underparameterization provides a barrier to overfitting, allowing the deep decoder to have state-of-the-art performance for denoising. The deep decoder is simple in the sense that each layer has an identical structure that consists of only one upsampling unit, pixel-wise linear combination of channels, ReLU activation, and channelwise normalization. This simplicity makes the network amenable to theoretical analysis, and it sheds light on the aspects of neural networks that enable them to form effective signal representations. Data models are central for signal and image processing and play a key role in compression and inverse problems such as denoising, super-resolution, and compressive sensing. These data models impose structural assumptions on the signal or image, which are traditionally based on expert knowledge. For example, imposing the assumption that an image can be represented with few non-zero wavelet coefficients enables modern (lossy) image compression BID1 and efficient denoising BID6.In recent years, it has been demonstrated that for a wide range of imaging problems, from compression to denoising, deep neural networks trained on large datasets can often outperform methods based on traditional image models BID19 BID0 BID18 BID4 BID22. This success can largely be attributed to the ability of deep networks to represent realistic images when trained on large datasets. Examples include learned representations via autoencoders BID12 and generative adversarial models BID8. Almost exclusively, three common features of the recent success stories of using deep neural network for imaging related tasks are i) that the corresponding networks are over-parameterized (i.e., they have much more parameters than the dimension of the image that they represent or generate), ii) that the networks have a convolutional structure, and perhaps most importantly, iii) that the networks are trained on large datasets. An important exception that breaks with the latter feature is a recent work by Ulyanov et al. BID20, which provides an algorithm, called the deep image prior (DIP), based on deep neural networks, that can solve inverse problems well without any training. Specifically, Ulyanov et al. demonstrated that fitting the weights of an over-parameterized deep convolutional network to a single image, together with strong regularization by early stopping of the optimization, performs competitively on a variety of image restoration problems. This is surprising because it does not involve a training dataset, which means that the notion of what makes an image'natural' is contained in a combination of the network structure and the regularization. However, without regularization the proposed network has sufficient capacity to overfit to noise, preventing meaningful image denoising. These prior works demonstrating the effectiveness of deep neural networks for image generation beg the question whether there may be a deep neural network model of natural images that is underparameterized and whose architecture alone, without algorithmic assistance, forms an efficient model for natural images. In this paper, we propose a simple image model in the form of a deep neural network that can represent natural images well while using very few parameters. This model thus enables image compression, denoising, and solving a variety of inverse problems with close to or state of the art performance. We call the network the deep decoder, due to its resemblance to the decoder part of an autoencoder. The network does not require training, and contrary to previous approaches, the network itself incorporates all assumptions on the data, is under-parameterized, does not involve convolutions, and has a simplicity that makes it amenable to theoretical analysis. The key contributions of this paper are as follows:• The network is under-parameterized. Thus, the network maps a lower-dimensional space to a higher-dimensional space, similar to classical image representations such as sparse wavelet representations. This feature enables image compression by storing the coefficients of the network after its weights are optimized to fit a single image. In Section 2, we demonstrate that the compression is on-par with wavelet thresholding BID1, a strong baseline that underlies JPEG-2000. An additional benefit of underparameterization is that it provides a barrier to overfitting, which enables regularization of inverse problems.• The network itself acts as a natural data model. Not only does the network require no training (just as the DIP BID20); it also does not critically rely on regularization, for example by early stopping (in contrast to the DIP). The property of not involving learning has at least two benefits: The same network and code is usable for a number of applications, and the method is not sensitive to a potential misfit of training and test data.• The network does not use convolutions. Instead, the network does have pixelwise linear combinations of channels, and, just like in a convolutional neural network, the weights are shared among spatial positions. Nonetheless, these are not convolutions because they provide no spatial coupling between pixels, despite how pixelwise linear combinations are sometimes called'1x1 convolutions.' In contrast, the majority of the networks for image compression, restoration, and recovery have convolutional layers with filters of nontrivial spatial extent BID19; BID0; BID18; BID4 BID22. This work shows that relationships characteristic of nearby pixels of natural images can be imposed directly by upsampling layers.• The network only consists of a simple combination of few building blocks, which makes it amenable to analysis and theory. For example, we prove that the deep decoder can only fit a small proportion of noise, which, combined with the empirical observation that it can represent natural images well, explains its denoising performance. The remainder of the paper is organized as follows. In Section 2, we first demonstrate that the deep decoder enables concise image representations. We formally introduce the deep decoder in Section 3. In Section 4, we show the performance of the deep decoder on a number of inverse problems such as denoising. In Section 5 we discuss related work, and finally, in Section 6 we provide theory and explanations on what makes the deep decoder work. Intuitively, a model describes a class of signals well if it is able to represent or approximate a member of the class with few parameters. In this section, we demonstrate that the deep decoder, an untrained, non-convolutional neural network, defined in the next section, enables concise representation of an image-on par with state of the art wavelet thresholding. The deep decoder is a deep image model G: R N → R n, where N is the number of parameters of the model, and n is the output dimension, which is (much) larger than the number of parameters (n N). The parameters of the model, which we denote by C, are the weights of the network, and not the input of the network, which we will keep fixed. To demonstrate that the deep decoder enables concise image representations, we choose the number of parameters of the deep decoder, N, such that it is a small fraction of the output dimension of the deep decoder, i.e., the dimension of the images 1.We draw 100 images from the ImageNet validation set uniformly at random and crop the center to obtain a 512x512 pixel color image. For each image x *, we fit a deep decoder model G(C) by minimizing the loss DISPLAYFORM0 with respect to the network parameters C using the Adam optimizer. We then compute for each image the corresponding peak-signal-to-noise ratio, defined as 10 log 10 (1/MSE), where DISPLAYFORM1 is the image generated by the network, and x * is the original image. We compare the compression performance to wavelet compression BID1 by representing each image with the N -largest wavelet coefficients. Wavelets-which underly JPEG 2000, a standard for image compression-are one of the best methods to approximate images with few coefficients. In Fig. 1 we depict the . It can be seen that for large compression factors (3 · 512 2 /N = 32.3), the representation by the deep decoder is slightly better for most images (i.e., is above the red line), while for smalle compression factors (3 · 512 2 /N = 8), the wavelet representation is slightly better. This experiment shows that deep neural networks can represent natural images well with very few parameters and without any learning. The observation that, for small compression factors, wavelets enable more concise representations than the deep decoder is intuitive because any image can be represented exactly with sufficiently many wavelet coefficients. In contrast, there is no reason to believe a priori that the deep decoder has zero representation error because it is underparameterized. The main point of this experiment is to demonstrate that the deep decoder is a good image model, which enables applications like solving inverse problems, as in Section 4. However, it also suggest that the deep decoder can be used for lossy image compression, by quantizing the coefficients C and saving the quantized coefficients. In the appendix, we show that image representations of the deep decoder are not sensitive to perturbations of its coefficients, thus quantization does not have a detrimental effect on the image quality. Deep networks were used successfully before for the compression of images BID19 BID0 BID18. In contrast to our work, which is capable of compressing images without any learning, the aforementioned works learn an encoder and decoder using convolutional recurrent neural networks BID19 and convolutional autoencoders BID18 based on training data. We consider a decoder architecture that transforms a randomly chosen and fixed tensor B 0 ∈ R n0×k0 consisting of k 0 many n 0 -dimensional channels to an n d × k out dimensional image, where k out = 1 for a grayscale image, and k out = 3 for an RGB image with three color channels. Throughout, n i has two dimensions; for example our default configuration has n 0 = 16 × 16 and n d = 512 × 512. The network transforms the tensor B 0 to an image by pixel-wise linearly combining the channels, upsampling operations, applying rectified linear units (ReLUs), and normalizing DISPLAYFORM0... lin. comb., upsampling, ReLU, CN lin. comb., sigmoid Figure 1: The deep decoder (depicted on the right) enables concise image representations, onpar with state-of-the-art wavelet based compression. The crosses on the left depict the PSNRs for 100 randomly chosen ImageNet-images represented with few wavelet coefficients and with a deep decoder with an equal number of parameters. A cross above the red line means the corresponding image has a smaller representation error when represented with the deep decoder. The deep decoder is particularly simple, as each layer has the same structure, consisting of a pixel-wise linear combination of channels, upsampling, ReLU nonlinearities, and channelwise normalization (CN). the channels. Specifically, the channels in the (i + 1)-th layer are given by DISPLAYFORM1 Here, the coefficient matrices C i ∈ R ki×ki+1 contain the weights of the network. Each column of the tensor B i C i ∈ R ni×ki+1 is formed by taking linear combinations of the channels of the tensor B i in a way that is consistent across all pixels. Then, cn(·) performs a channel normalization operation which is equivalent to normalizing each channel individually, and can be viewed as a special case of the popular batch normalization proposed in BID13. Specifically, let Z i = relu(U i B i C i) be the channels in the i-th layer, and let z ij be the j-th channel in the i-th layer. Then channel normalization performs the following transformation: DISPLAYFORM2, where mean and var compute the empirical mean and variance, and γ ij and β ij are parameters, learned independently for each channel, and is a fixed small constant. Learning the parameter γ and β helps the optimization but is not critical. This is a special case of batch normalization with batch size one proposed in BID13, and significantly improves the fitting of the model, just like how batch norm alleviates problems encountered when training deep neural networks. The operator U i ∈ R ni+1×ni is an upsampling tensor, which we choose throughout so that it performs bi-linear upsampling. For example, if the channels in the input have dimensions n 0 = 16×16, then the upsampling operator U 0 upsamples each channel to dimensions 32 × 32. In the last layer, we do not upsample, which is to say that we choose the corresponding upsampling operator as the identity. Finally, the output of the d-layer network is formed as DISPLAYFORM3 where Fig. 1 for an illustration. Throughout, our default architecture is a d = 6 layer network with k i = k for all i, and we focus on output images of dimensions n d = 512 × 512 and number of channels k out = 3. Recall that the parameters of the network are given by C = {C 0, C 1, . . ., C d}, and the output of the network is only a function of C, since we choose the tensor B 0 at random and fix it. Therefore, we write x = G(C). Note that the number of parameters is given by DISPLAYFORM4 DISPLAYFORM5 where the term 2k i corresponds to the two free parameters associated with the channel normalization. Thus, the number of parameters is N = dk 2 + 2dk + 3k. In the default architectures with d = 6 and k = 64 or k = 128, we have that N = 25,536 (for k = 64) and N =100,224 (k = 128) out of an RGB image space of dimensionality 512 × 512 × 3 = 786,432 parameters. We finally note that naturally variations of the deep decoder are possible; for example in a previous version of this manuscript, we applied upsampling after applying the relu-nonlinearity, but found that applying it before yields slightly better . While the deep decoder does not use convolutions, its structure is closely related to that of a convolutional neural network. Specifically, the network does have pixelwise linear combinations of channels, and just like in a convolutional neural network, the weights are shared among spatial positions. Nonetheless, pixelwise linear combinations are not proper convolutions because they provide no spatial coupling of pixels, despite how they are sometimes called 1 × 1 convolutions. In the deep decoder, the source of spatial coupling is only from upsampling operations. In contrast, a large number of networks for image compression, restoration, and recovery have convolutional layers with filters of nontrivial spatial extent BID19; BID0; BID18; BID4; BID22. Thus, it is natural to ask whether using linear combinations as we do, instead of actual convolutions yields better . Our simulations indicate that, indeed, linear combinations yield more concise representations of natural images than p × p convolutions, albeit not by a huge factor. Recall that the number of parameters of the deep decoder with d layers, k channels at each layer, and 1 × 1 convolutions is N (d, k; 1) = dk 2 + 3k + 2dk. If we consider a deep decoder with convolutional layers with filters of size p × p, then the number of parameters is: DISPLAYFORM0 If we fix the number of channels, k, but increase p to 3, the representation error only decreases since we increase the number of parameters (by a factor of approximately 32). We consider image reconstruction as described in Section 2. For a meaningful comparison, we keep the number of parameters fixed, and compare the representation error of a deep decoder with p = 1 and k = 64 (the default architecture in our paper) to a variant of the deep decoder with p = 3 and k = 22, so that the number of parameters is essentially the same in both configurations. We find that the representation of the deep decoder with p = 1 is better (by about 1dB, depending on the image), and thus for concise image representations, linear combinations (1 × 1 convolutions) appear to be more effective than convolutions of larger spatial extent. In this section, we use the deep decoder as a structure-enforcing model or regularizers for solving standard inverse problems: denoising, super-resolution, and inpainting. In all of those inverse problems, the goal is to recover an image x from a noisy observation y = f (x) + η. Here, f is a known forward operator (possibly equal to identity), and η is structured or unstructured noise. We recover the image x with the deep decoder as follows. Motivated by the finding from the previous section that a natural image x can (approximately) be represented with the deep decoder as G(C), we estimate the unknown image from the noisy observation y by minimizing the loss DISPLAYFORM0 with respect to the model parameters C. LetĈ be the of the optimization procedure. We estimate the image asx = G(Ĉ).We use the Adam optimizer for minimizing the loss, but have obtained comparable with gradient descent. Note that this optimization problem is non-convex and we might not reach a global minimum. Throughout, we consider the least-squares loss (i.e., we take · 2 to be the 2 norm), but the loss function can be adapted to account for structure of the noise. We remark that fitting an image model to observations in order to solve an inverse problem is a standard approach and is not specific to the deep decoder or deep-network-based models in general. Specifically, a number of classical signal recovery approaches fit into this framework; for example solving a compressive sensing problem with 1 -norm minimization amounts to choosing the forward operator as f (x) = Ax and minimizing over x in a 1 -norm ball. We start with the perhaps most basic inverse problem, denoising. The motivation to study denoising is at least threefold: First, denoising is an important problem in practice, second, many inverse problem can be solved as a chain of denoising steps BID15 Figure 2: An application of the deep decoder for denoising the astronaut test image. The deep decoder has performance on-par with state of the art untrained denoising methods, such as the DIP method BID20 and the BM3D algorithm BID5.problem is simple to model mathematically, and thus a common entry point for gaining intuition on a new method. Given a noisy observation y = x + η, where η is additive noise, we estimate an image with the deep decoder by minimizing the least squares loss G(C) − y 2 2, as described above. The in Fig. 2 and Table 1 demonstrate that the deep decoder has denoising performance onpar with state of the art untrained denoising methods, such as the related Deep Image Prior (DIP) method BID20 (discussed in more detail later) and the BM3D algorithm BID5. Since the deep decoder is an untrained method, we only compared to other state-of-the-art untrained methods (as opposed to learned methods such as BID22).Why does the deep decoder denoise well? In a nutshell, from Section 2 we know that the deep decoder can represent natural images well even when highly underparametrized. In addition, as a consequence of being under-parameterized, the deep decoder can only represent a small proportion of the noise, as we show analytically in Section 6, and as demonstrated experimentally in FIG2. Thus, the deep decoder "filters out" a significant proportion of the noise, and retains most of the signal. How to choose the parameters of the deep decoder? The larger k, the larger the number of latent parameters and thus the smaller the representation error, i.e., the error that the deep decoder makes when representing a noise-free image. On the other hand, the smaller k, the fewer parameters, and the smaller the range space of the deep decoder G(C), and thus the more noise the method will remove. The optimal k trades off those two errors; larger noise levels require smaller values of k (or some other form of regularization). If the noise is significantly larger, then the method requires either choosing k smaller, or it requires another means of regularization, for example early stopping of the optimization. For example k = 64 or 128 performs best out of {32, 64, 128}, for a PSNR of around 20dB, while for a PSNR of about 14dB, k = 32 performs best. We next super-resolve images with the deep denoiser. We define a forward model f that performs downsampling with the Lanczos filter by a factor of four. We then downsample a given image by a factor of four, and then reconstruct it with the deep decoder (with k = 128, as before). We compare performance to bi-cubic interpolation and to the deep image prior, and find that the deep decoder outperforms bicubic interpolation, and is on-par with the deep image prior (see Table 1 in the appendix). Finally, we use the deep decoder for inpainting, where we are given an inpainted image y, and a forward model f mapping a clean image to an inpainted image. The forward model f is defined by a mask that describes the inpainted region, and simply maps that part of the image to zero. FIG1 and Table 1 demonstrate that the deep decoder performs well on the inpainting problems; however, the deep image prior performs slightly better on average over the examples considered. For the impainting problem we choose a significantly more expressive prior, specifically k = 320. Image compression, restoration, and recovery algorithms are either trained or untrained. Conceptually, the deep decoder image model is most related to untrained methods, such as sparse representations in overcomplete dictionaries (for example wavelets BID6 and curvelets BID17). A number of highly successful image restoration and recovery schemes are not directly based on generative image models, but rely on structural assumptions about the image, such as exploiting self-similarity in images for denoising BID5 and super-resolution BID7 ).Since the deep decoder is an image-generating deep network, it is also related to methods that rely on trained deep image models. Deep learning based methods are either trained end-to-end for tasks ranging from compression BID19 BID0 BID18 BID4 BID22 to denoising BID4 BID22, or are based on learning a generative image model (by training an autoencoder or GAN BID12 BID8) and then using the ing model to solve inverse problems such as compressed sensing BID3, denoising BID11, phase retrieval, and blind deconvolution BID2, by minimizing an associated loss. In contrast to the deep decoder, where the optimization is over the weights of the network, in all the aforementioned methods, the weights are adjusted only during training and then are fixed upon solving the inverse problem. Most related to our work is the Deep Image Prior (DIP), recently proposed by Ulyanov et al. BID20. The deep image prior is an untrained method that uses a network with an hourglass or encoder-decoder architecture, similar to the U-net and related architectures that work well as autoencoders. The key differences to the deep decoder are threefold: i) the DIP is over-parameterized, whereas the deep decoder is under-parameterized. ii) Since the DIP is highly over-parameterized, it critically relies on regularization through early stopping and adding noise to its input, whereas the deep decoder does not need to be regularized (however, regularization can enhance performance). iii) The DIP is a convolutional neural network, whereas the deep decoder is not. We further illustrate point ii) comparing the DIP and deep decoder by denoising the astronaut image from Fig. 2. In FIG2 we plot the Mean Squared Error (MSE) over the number of iterations of the optimizer for fitting the noisy astronaut image x + η. Note that to fit the model, we minimize the error G(C) − (x + η) 2 2, because we are only given the noisy image, but we plot the MSE between the representation and the actual, true image G(C t) − x 2 2 at iteration t. Here, C t are the parameters of the deep decoder after t iterations of the optimizer. In FIG2 and (c), we plot the loss or MSE associated with fitting the noiseless astronaut image, x (G(C t) − x 2 2 ) and the noise itself, η, (G(C t) − η 2 2 ). Models are fitted independently for the noisy image, the noiseless image, and the noise. The plots in FIG2 show that with sufficiently many iterations, both the DIP and the DD can fit the image well. However, even with a large number of iterations, the deep decoder can not fit the noise well, whereas the DIP can. This is not surprising, given that the DIP is over-parameterized and the deep decoder is under-parameterized. In fact, in Section 6 we formally show that due to the The third panel shows the MSE of the output of DD or DIP for an image consisting purely of noise, as computed relative to that noise. Due to under-parameterization, the deep decoder can only fit a small proportion of the noise, and thus enables image denoising. Early stopping can mildly enhance the performance of DD; to see this note that in panel (a), the minimum is obtained at around 5000 iterations and not at 50,000. The deep image prior can fit noise very well, but fits an image faster than noise, thus early stopping is critical for denoising performance.underparameterization, the deep decoder can only fit a small proportion of the noise, no matter how and how long we optimize. As a consequence, it filters out much of the noise when applied to a natural image. In contrast, the DIP relies on the empirical observation that the DIP fits a structured image faster than it fits noise, and thus critically relies on early stopping. In the previous sections we empirically showed that the deep decoder can represent images well and at the same time cannot fit noise well. In this section, we formally show that the deep decoder can only fit a small proportion of the noise, relative to the degree of underparameterization. In addition, we provide insights into how the components of the deep decoder contribute to representing natural images well, and we provide empirical observations on the sensitivity of the parameters and their distribution. We start by showing that an under-parameterized deep decoder can only fit a proportion of the noise relative to the degree of underparameterization. At the heart of our argument is the intuition that a method mapping from a low-to a high-dimensional space can only fit a proportion of the noise relative to the number of free parameters. For simplicity, we consider a one-layer network, and ignore the batch normalization operation. Then, the networks output is given by DISPLAYFORM0 Here, we take C = (C 0, c 1), where C 0 is a k × k matrix and c 1 is a k-dimensional vector, assuming that the number of output channels is 1. While for the performance of the deep decoder the choice of upsampling matrix is important, it is not relevant for showing that the deep decoder cannot represent noise well. Therefore, the following statement makes no assumptions about the upsampling matrix U 0. Proposition 1. Consider a deep decoder with one layer and arbitrary upsampling and input matrices. That is, let B 0 ∈ R n0×k and U 0 ∈ R n×n0. Let η ∈ R n be zero-mean Gaussian noise with covariance matrix σI, σ > 0. Assume that k 2 log(n 0)/n ≤ 1/32. Then, with probability at least DISPLAYFORM1 The proposition asserts that the deep decoder can only fit a small portion of the noise energy, precisely a proportion determined by its number of parameters relative to the output dimension, n. Our The blue curves show a one-dimensional piecewise smooth signal, and the red crosses show estimates of this signal by a one-dimensional deep decoder with either linear or convex upsampling. We see that linear upsampling acts as an indirect signal prior that promotes piecewise smoothness.simulations and preliminary analytic suggest that this statement extends to multiple layers in that the lower bound becomes 1 − c DISPLAYFORM2, where c is a numerical constant. Note that the lower bound does not directly depend on the noise variance σ since both sides of the inequality scale with σ 2. Upsampling is a vital part of the deep decoder because it is the only way that the notion of locality explicitly enters the signal model. In contrast, most convolutional neural networks have spatial coupling between pixels both by unlearned upsampling, but also by learned convolutional filters of nontrivial spatial extent. The choice of the upsampling method in the deep decoder strongly affects the'character' of the ing signal estimates. We now discuss the impacts of a few choices of upsampling matrices U i, and their impact on the images the model can fit. No upsampling: If there is no upsampling, or, equivalently, if U i = I, then there is no notion of locality in the ing image. All pixels become decoupled, and there is then no notion of which pixels are near to each other. Specifically, a permutation of the input pixels (the rows of B 0) simply induces the identical permutation of the output pixels. Thus, if a deep decoder without upsampling could fit a given image, it would also be able to fit random permutations of the image equally well, which is practically equivalent to fitting random noise. Nearest neighbor upsampling: If the upsampling operations perform nearest neighbor upsampling, then the output of the deep decoder consists of piecewise constant patches. If the upsampling doubles the image dimensions at each layer, this would in patches of 2 d × 2 d pixels that are constant. While this upsampling method does induce a notion of locality, it does so too strongly in the sense that squares of nearby pixels become identical and incapable of fitting local variation within natural images. Linear and convex, non-linear upsampling: The specific choice of upsampling matrix affects the multiscale'character' of the signal estimates. To illustrate this, FIG3 shows the signal estimate from a 1-dimensional deep decoder with upsampling operations given by linear upsampling (x 0, x 1, x 2, . . .) → (x 0, 0.5x 0 + 0.5x 1, x 1, 0.5x 1 + 0.5x 2, x 2, . . .) and convex nonlinear upsampling given by (x 0, x 1, x 2, . . .) → (x 0, 0.75x 0 + 0.25x 1, x 1, 0.75x 1 + 0.25x 2, x 2, . . .). Note that while both models are able to capture the coarse signal structure, the convex upsampling in a multiscale fractal-like structure that impedes signal representation. In contrast, linear upsampling is better able to represent smoothly varying portions of the signal. Linear upsampling in a deep decoder indirectly encodes the prior that natural signals are piecewise smooth and in some sense have approximately linear behavior at multiple scales 6.3 NETWORK INPUT Throughout, the network input is fixed. We choose the network input B 1 by choosing its entries uniformly at random. The particular choice of the input is not very important; it is however desirable that the rows are incoherent. To see this, as an extreme case, if any two rows of B 1 are equal and if the upsampling operation preserves the values of those pixels exactly (for example, as with the linear upsampling from the previous section), then the corresponding pixels of the output image is Figure 6: The left panel shows an image reconstruction after training a deep decoder on the MRI phantom image (PSNR is 51dB). The right panel shows how the deep decoder builds up an image starting from a random input. From top to bottom are the input to the network and the activation maps (i.e., relu(B i C i)) for eight out of the 64 channels in layers one to six. Table 1: Performance comparison of the deep decoder for denoising (DN), superresolution (SR), and inpainting (IP), in peak signal to noise ratio (PSNR). Note that identity corresponds to the PSNR of the noise and corruption in the DN and IP experiments, respectively. also exactly the same, which restricts the range space of the deep decoder unrealistically, since for any pair of pixels, the majority of natural images does not have exactly the same value at this pair of pixels. The deep decoder is tasked with coverting multiple noise channels into a structured signal primarily using pixelwise linear combinations, ReLU activation funcions, and upsampling. Using these tools, the deep decoder builds up an image through a series of successive approximations that gradually morph between random noise and signal. To illustrate that, we plot the activation maps (i.e., relu(B i C i)) of a deep decoder fitted to the phantom MRI test image (see Fig. 6). We choose a deep decoder with d = 5 layers and k = 64 channels. This image reconstruction approach is in contrast to being a semantically meaningful hierarchical representation (i.e., where edges get combined into corners, that get combined into simple sample, and then into more complicated shapes), similar to what is common in discriminative networks. RH is partially supported by NSF award IIS-1816986, an NVIDIA Academic GPU Grant, and would like to thank Ludwig Schmidt for helpful discussions on the deep decoder in general, and in particular for suggestions on the experiments in Section 2.Code to reproduce the is available at https://github.com/reinhardh/ supplement_deep_decoder APPENDIX A PROOF OF PROPOSITION 1Suppose that the network has one layer, i.e., G(C) = relu(U 0 B 0 C 0)c 1. We start by re-writing B 1 = relu(B 0 C 0) in a convenient form. For a given vector x ∈ R n, denote by diag(x > 0) the matrix that contains one on its diagonal if the respective entry of x is positive and zero otherwise. Let c jci denote the i-th column of C j, and denote by W ji ∈ {0, 1} k×k the corresponding diagonal matrix W ji = diag(U j B j c jci > 0). With this notation, we can write DISPLAYFORM0 where [c 1] i denotes the i-th entry of c 1. Thus, G(C) lies in the union of at-most-k 2 -dimensional subspaces of R n, where each subspace is determined by the matrices {W 0j} k j=1. The number of those subspaces is bounded by n k 2. This follows from the fact that for the matrix A:= U 0 B 0, by Lemma 1 below, the number of different matrices W 0j is bounded by n k. Since there are k matrices, the number of different sets of matrices is bounded by n k 2. Lemma 1. For any A ∈ R n×k and k ≥ 5, |{diag(Av > 0)A|v ∈ R k }| ≤ n k.Next, fix the matrixes {W 0j} j. As G(C) lies in an at-most-k 2 -dimensional subspace, let S be a k 2 -dimensional subspace that contains the range of G for these fixed {W 0j} j. It follows that Now, we make use of the following bound on the projection of the noise η onto a subspace. Lemma 2. Let S ⊂ R n be a subspace with dimension. Let η ∼ N (0, I n) and β ≥ 1. Then, P P S c η n, then P X − n ≥ 2 √ nx + 2x ≤ e −x, DISPLAYFORM1 With these, we obtain P [X ≥ 5βn] ≤ e −βn if β ≥ 1,P [X ≤ n/2] ≤ e −n/16.We have noiseless C 1 noisy C 2 noisy C 3 noisy C 4 noisy C 5 noisy C 6 noisy c 7 noisy Figure 7: Sensitivity to parameter perturbations of the weights in each layer, and images generated by perturbing the weights in different layers, and keeping the weights in the other layers constant. A.1 PROOF OF LEMMA 1Our goal is to count the number of sign patterns (Av > 0) ∈ {0, 1}. Note that this number is equal to the maximum number of partitions one can get when cutting a k-dimensional space with n many hyperplanes that all pass through the origin, and are perpendicular to the rows of A. This number if well known (see for example BID21) and is upper bounded by DISPLAYFORM2 Thus, DISPLAYFORM3 where the last inequality holds for k ≥ 5. The deep decoder is not overly sensitive to perturbations of its coefficients. To demonstrate this, fit the standard test image Barbara with a deep decoder with 6 layers and k = 128, as before. We then perturb the weights in a given layer i (i.e., the matrix C i) with Gaussian noise of a certain signal-tonoise ratio relative to C i and leave the other weights and the input untouched. We then measure the peak signal-to-noise ratio in the image domain, and plot the corresponding curve for each layer (see Fig. 7). It can be seen that the representation provided by the deep decoder is relatively stable with respect to perturbations of its coefficients, and that it is more sensitive to perturbations in higher levels. Finally, in FIG7 we depict the distribution of the weights of the network after fitted to the Barbara test image, and note that the weights are approximately Gaussian distributed. The distribution of the weighs is approximately Gaussian."

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We relate the minimax game of generative adversarial networks (GANs) to finding the saddle points of the Lagrangian function for a convex optimization problem, where the discriminator outputs and the distribution of generator outputs play the roles of primal variables and dual variables, respectively. This formulation shows the connection between the standard GAN training process and the primal-dual subgradient methods for convex optimization. The inherent connection does not only provide a theoretical convergence proof for training GANs in the function space, but also inspires a novel objective function for training. The modified objective function forces the distribution of generator outputs to be updated along the direction according to the primal-dual subgradient methods. A toy example shows that the proposed method is able to resolve mode collapse, which in this case cannot be avoided by the standard GAN or Wasserstein GAN. Experiments on both Gaussian mixture synthetic data and real-world image datasets demonstrate the performance of the proposed method on generating diverse samples. Generative adversarial networks (GANs) are a class of game theoretical methods for learning data distributions. It trains the generative model by maintaining two deep neural networks, namely the discriminator network D and the generator network G. The generator aims to produce samples resembling real data samples, while the discriminator aims to distinguish the generated samples and real data samples. The standard GAN training procedure is formulated as the following minimax game: DISPLAYFORM0 where p d (x) is the data distribution and p z (z) is the noise distribution. The generated samples G(z) induces a generated distribution p g (x). Theoretically, the optimal solution to is p * g = p d and D * (x) = 1/2 for all x in the support of data distribution. In practice, the discriminator network and the generator network are parameterized by θ θ θ d and θ θ θ g, respectively. The neural network parameters are updated iteratively according to gradient descent. In particular, the discriminator is first updated either with multiple gradient descent steps until convergence or with a single gradient descent step, then the generator is updated with a single descent step. However, the analysis of the convergence properties on the training approaches is challenging, as noted by Ian Goodfellow in BID10, "For GANs, there is no theoretical prediction as to whether simultaneous gradient descent should converge or not. Settling this theoretical question, and developing algorithms guaranteed to converge, remain important open research problems.". There have been some recent studies on the convergence behaviours of GAN training (; BID18 BID14 BID24 BID22 .The simultaneous gradient descent method is proved to converge assuming the objective function is convex-concave in the network parameters . The local stability property is established in BID14 BID24.One notable inconvergence issue with GAN training is referred to as mode collapse, where the generator characterizes only a few modes of the true data distribution BID11 BID18. Various methods have been proposed to alleviate the mode collapse problem. Feature matching for intermediate layers of the discriminator has been proposed in . In BID23, the generator is updated based on a sequence of previous unrolled discriminators. A mixture of neural networks are used to generate diverse samples (; BID15 BID2 . In, it was proposed that adding noise perturbation on the inputs to the discriminator can alleviate the mode collapse problem. It is shown that this training-with-noise technique is equivalent to adding a regularizer on the gradient norm of the discriminator . The Wasserstein divergence is proposed to resolve the problem of incontinuous divergence when the generated distribution and the data distribution have disjoint supports BID12. Mode regularization is used in the loss function to penalize the missing modes BID6 ). The regularization is usually based on heuristics, which tries to minimize the distance between the data samples and the generated samples, but lacks theoretical convergence guarantee. In this paper, we formulate the minimax optimization for GAN training as finding the saddle points of the Lagrangian function for a convex optimization problem. In the convex optimization problem, the discriminator function D(·) and the probabilities of generator outputs p g (·) play the roles of the primal variables and dual variables, respectively. This connection not only provides important insights in understanding the convergence of GAN training, but also enables us to leverage the primal-dual subgradient methods to design a novel objective function that helps to alleviate mode collapse. A toy example reveals that for some cases when standard GAN or WGAN inevitably leads to mode collapse, our proposed method can effectively avoid mode collapse and converge to the optimal point. In this paper, we do not aim at achieving superior performance over other GANs, but rather provide a new perspective of understanding GANs, and propose an improved training technique that can be applied on top of existing GANs. The contributions of the paper are as follows:• The standard training of GANs in the function space is formulated as primal-dual subgradient methods for solving convex optimizations.• This formulation enables us to show that with a proper gradient descent step size, updating the discriminator and generator probabilities according to the primal-dual algorithms will provably converge to the optimal point.• This formulation in a novel training objective for the generator. With the proposed objective function, the generator is updated such that the probabilities of generator outputs are pushed to the optimal update direction derived by the primal-dual algorithms. Experiments have shown that this simple objective function can effectively alleviate mode collapse in GAN training.• The convex optimization framework incorporates different variants of GANs including the family of f -GAN and an approximate variant of WGAN. For all these variants, the training objective can be improved by including the optimal update direction of the generated probabilities. In this section, we first describe the primal-dual subgradient methods for convex optimization. Later, we explicitly construct a convex optimization and relate the subgradient methods to standard GAN training. Consider the following convex optimization problem: DISPLAYFORM0 where x ∈ R k is a length-k vector, X is a convex set, and f i (x), i = 0 · · ·,, are concave functions mapping from R k to R. The Lagrangian function is calculated as DISPLAYFORM1 In the optimization problem, the variables x ∈ R k and λ λ λ ∈ R + are referred to as primal variables and dual variables, respectively. The primal-dual pair (x *, λ λ λ *) is a saddle-point of the Lagrangian fuction, if it satisfies: DISPLAYFORM2 Primal-dual subgradient methods have been widely used to solve the convex optimization problems, where the primal and dual variables are updated iteratively, and converge to a saddle point (Nedić & ; BID16 .There are two forms of algorithms, namely dual-driven algorithm and primal-dual-driven algorithm. For both approaches, the dual variables are updated according to the subgradient of L(x(t), λ λ λ(t)) with respect to λ λ λ(t) at each iteration t. For the dual-driven algorithm, the primal variables are updated to achieve maximum of L(x, λ λ λ(t)) over x. For the primal-dual-driven algorithm, the primal variables are updated according to the subgradient of L(x(t), λ λ λ(t)) with respect to x(t). The iterative update process is summarized as follows: DISPLAYFORM3 where P X (·) denotes the projection on set X and (x) + = max(x, 0).The following theorem proves that the primal-dual subgradient methods will make the primal and dual variables converge to the optimal solution of the convex optimization problem. Theorem 1 Consider the convex optimization. Assume the set of saddle points is compact. Suppose f 0 (x) is a strictly concave function over x ∈ X and the subgradient at each step is bounded. There exists some step size α (t) such that both the dual-driven algorithm and the primal-dual-driven algorithm yield x (t) → x * and λ λ λ (t) → λ λ λ *, where x * is the solution to, and λ λ λ * satisfies DISPLAYFORM4 Proof: See Appendix 7.1. We explicitly construct a convex optimization problem and relate it to the minimax game of GANs. We assume that the source data and generated samples belong to a finite set {x 1, · · ·, x n} of arbitrary size n. The extension to uncountable sets can be derived in a similar manner BID20. The finite case is of particular interest, because any real-world data has a finite size, albeit the size could be arbitrarily large. We construct the following convex optimization problem: DISPLAYFORM0 where D is some convex set. The primal variables are DISPLAYFORM1 ) is the Lagrangian dual associated with the i-th constraint. The Lagrangian function is thus DISPLAYFORM2 When D = {D : 0 ≤ D i ≤ 1, ∀i}, finding the saddle points for the Lagrangian function is exactly equivalent to solving the GAN minimax problem. This inherent connection enables us to utilize the primal-dual subgradient methods to design update rules for D(x) and p g (x) such that they converge to the saddle points. The following theorem provides a theoretical guideline for the training of GANs. Theorem 2 Consider the Lagrangian function given by with D = {D : ≤ D i ≤ 1 −, ∀i}, where 0 < < 1/2. If the discriminator and generator have enough capacity, and the discriminator output and the generated distribution are updated according to the primal-dual update rules and FORMULA4 with DISPLAYFORM3 Proof: The optimization problem is a particularized form of, where DISPLAYFORM4 The objective function is strictly concave over D. Moreover, since D is projected onto the compact set [, 1 −] at each iteration t, the subgradients ∂f i (D (t) ) are bounded. The assumptions of Theorem 1 are satisfied. Since the constraint (8b) gives an upper bound of D i ≤ 1/2, the solution to the above convex optimization is obviously DISPLAYFORM5 Since the problem is convex, the optimal primal solution is the primal saddle point of the Lagrangian function (, Chapter 5). DISPLAYFORM6, and the saddle point is unique. By Theorem 1, the primal-dual update rules will guarantee convergence of DISPLAYFORM7 It can be seen that the standard training of GAN corresponds to either dual-driven algorithm or primal-dual-driven algorithm BID11. A natural question arises: Why does the standard training fail to converge and lead to mode collapse? As will be shown later, the underlying reason is that standard training of GANs in some cases do not update the generated distribution according to. Theorem 2 inspires us to propose a training algorithm to tackle this issue. First, we present our training algorithm. Later, we will use a toy example to give intuitions of why our algorithm is effective to avoid mode collapse. The algorithm is described in Algorithm 1. The maximum step of discriminator update is k 0. In the context of primal-dual-driven algorithms, k 0 = 1. In the context of dual-driven algorithms, k 0 is some large constant, such that the discriminator is updated till convergence at each training epoch. The update of the discriminator is the same as standard GAN training. The main difference is the modified loss function for the generator update. The intuition is that when the generated samples have disjoint support from the data, the generated distribution at the data support may not be updated using standard training. This is exactly one source of mode collapse. Ideally, the modified loss function will always update the generated probabilities at the data support along the optimal direction. The generated probability mass at DISPLAYFORM0 where 1{·} is the indicator function. The indicator function is not differentiable, so we use a continuous kernel to approximate it. Define DISPLAYFORM1 where σ is some positive constant. The constant σ is also called bandwidth for kernel density estimation. The empirical generated distribution is thus approximately calculated as. There Initialization: Choose the objective function f 0 (·) and constraint function f 1 (·) according to the GAN realization. For the original GAN based on Jensen-Shannon divergence, f 0 (D) = log (D) and f 1 (D) = log(2(1 − D)). while the stopping criterion is not met do Sample minibatch m 1 data samples DISPLAYFORM0 Update the discriminator parameters with gradient ascent: DISPLAYFORM1 end for Update the target generated distribution as: DISPLAYFORM2 where α is some step size and DISPLAYFORM3 Withp g (x i) fixed, update the generator parameters with gradient descent: DISPLAYFORM4 end while are different bandwidth selection methods BID5 BID13. It can be seen that as σ → 0, k σ (x − y) tends to the indicator function, but it will not give large enough gradients to far areas that experience mode collapse. A larger σ implies a coarser quantization of the space in approximating the distribution. In practical training, the kernel bandwidth can be set larger at first and gradually decreases as the iteration continues. By the dual update rule, the generated probability of every x i should be updated as DISPLAYFORM5 This motivates us to add the second term of in the loss function, such that the generated distribution is pushed towards the target distribution.Although having good convergence guarantee in theory, the non-parametric kernel density estimation of the generated distribution may suffer from the curse of dimension. Previous works combining kernel learning and the GAN framework have proposed methods to scale the algorithms to deal with high-dimensional data, and the performances are promising BID19 BID17 ). One common method is to project the data onto a low dimensional space using an autoencoder or a bottleneck layer of a pretrained neurual network, and then apply the kernel-based estimates on the feature space. Using this approach, the estimated probability of x i becomes DISPLAYFORM6 where f φ is the projection of the data to a low dimensional space. We will leave the work of generating high-resolution images using this approach as future work. Mode collapse occurs when the generated samples have a very small probability to overlap with some families of the data samples, and the discriminator D(·) is locally constant around the region of the generated samples. We use a toy example to show that the standard training of GAN and Wasserstein may fail to avoid mode collapse, while our proposed method can succeed. Claim 1 Suppose the data distribution is p d (x) = 1{x = 1}, and the initial generated distribution is p g (x) = 1{x = 0}. The discriminator output D(x) is some function that is equal to zero for |x − 0| ≤ δ and is equal to one for |x − 1| ≤ δ, where 0 < δ < 1/2. Standard training of GAN and WGAN leads to mode collapse. Proof: We first show that the discriminator is not updated, and then show that the generator is not updated during the standard training process. In standard training of GAN and WGAN, the discriminator is updated according to the gradient of. For GAN, since 0 ≤ D(x) ≤ 1, the objective funtion for the discriminator is at most zero, i.e., DISPLAYFORM0 which is achieved by the current D(x) by assumption. For WGAN, the optimal discrminator output D(x) is some 1-Lipschitz function such that DISPLAYFORM1 where is due to the Lipschitz condition |D − D| ≤ 1. The current D(x) is obviously optimal. Thus, for both GAN and WGAN, the gradient of the loss function with respect to θ θ θ d is zero and the discriminator parameters are not updated. On the other hand, in standard training, the generator parameters θ θ θ g are updated with only the first term of. By the chain rule, DISPLAYFORM2 where is due to the assumption that D(x) is locally constant for x = 0. Therefore, the generator and the discriminator reach a local optimum point. The generated samples are all zeros. In our proposed training method, when x = 1, the optimal update direction is given by, wherẽ p g is a large value because D = 1. Therefore, by, the second term in the loss function is very large, which forces the generator to generate samples at G(z) = 1. As the iteration continues, the generated distribution gradually converges to data distribution, and D(x) gradually converges to 1/2, which makes ∂ pg(x) L(D(x), p g (x)) = log(2(1 − D(x))) become zero. The experiment in Section 5 demonstrates this training dynamic. In this paper, the standard training of GANs in function space has been formulated as primal-dual updates for convex optimization. However, the training is optimized over the network parameters in practice, which typically yields a non-convex non-concave problem. Theorem 2 tells us that as long as the discriminator output and the generated distribution are updated according to the primal-dual update rule, mode collapse should not occur. This insight leads to the addition of the second term in the modified loss function for the generator. In Section 5, experiments on the above-mentioned toy example and real-world datasets show that the proposed training technique can greatly improve the baseline performance. Consider the following optimization problem: DISPLAYFORM0 DISPLAYFORM1 where f 0 (·) and f 1 (·) are concave functions. Compared with the generic convex optimization problem, the number of constraint functions is set to be the variable alphabet size, and the constraint functions are DISPLAYFORM2 The objective and constraint functions in can be tailored to produce different GAN variants. For example, TAB0 shows the large family of f -GAN . The last row of TAB0 gives a new realization of GAN with a unique saddle point of D * (x) = 2 and DISPLAYFORM3 We also derive a GAN variant similar to WGAN, which is named "Approximate WGAN". As shown in TAB0, the objective and constraint functions yield the following minimax problem: DISPLAYFORM4 where is an arbitrary positive constant. The augmented term D 2 (x) is to make the objective function strictly concave, without changing the original solution. It can be seen that this problem has a unique saddle point p * g (x) = p d (x). As tends to 0, the training objective function becomes identical to WGAN. The optimal D(x) for WGAN is some Lipschitz function that maximizes E x∼p d (x) {D(x)} − E x∼pg(x) {D(x)}, while for our problem is D * (x) = 0. Weight clipping can still be applied, but serves as a regularizer to make the training more robust BID21.The training algorithms for these variants of GANs follow by simply changing the objective function f 0 (·) and constraint function f 1 (·) accordingly in Algorithm 1. 5.1 SYNTHETIC DATA FIG0 shows the training performance for a toy example. The data distribution is p g (x) = 1{x = 1}. The inital generated samples are concentrated around x = −3.0. The details of the neural network parameters can be seen in Appendix 7.3. FIG0 shows the generated samples in the 90 quantile as the training iterates. After 8000 iterations, the generated samples from standard training of GAN and WGAN are still concentrated around x = −3.0. As shown in FIG0, the discrminators hardly have any updates throughout the training process. Using the proposed training approach, the generated samples gradually converge to the data distribution and the discriminator output converges to the optimal solution with D = 1/2. Fig. 2 shows the performance of the proposed method for a mixture of 8 Gaussain data on a circle. While the original GANs experience mode collapse BID15 BID23, our proposed method is able to generate samples over all 8 modes. In the training process, the bandwidth of the Gaussian kernel is inialized to be σ 2 = 0.1 and decreases at a rate of 0.8 DISPLAYFORM0, where t is the iteration number. The generated samples are dispersed initially, and then gradually converge to the Gaussian data samples. Note that our proposed method involves a low complexity with a simple regularization term added in the loss function for the generator update. Figure 2: Performance of the proposed algorithm on 2D mixture of Gaussian data. The data samples are marked in blue and the generated samples are marked in orange. We also evaluate the performance of the proposed method on two real-world datasets: MNIST and CIFAR-10. Please refer to the appendix for detailed architectures. Inception score is employed to evaluate the proposed method. It applies a pretrained inception model to every generated image to get the conditional label distribution p(y|x). The Inception score is calculated as exp (E x {KL(p(y|x) p(y)}). It measures the quality and diversity of the generated images. The MNIST dataset contains 60000 labeled images of 28 × 28 grayscale digits. We train a simple LeNet-5 convolutional neural network classifier on MNIST dataset that achieves 98.9% test accuracy, and use it to compute the inception score. The proposed method achieves an inception score of 9.8, while the baseline method achieves an inception score of 8.8. The examples of generated images are shown in Fig. 3. The generated images are almost indistinguishable from real images. We further evaluated our algorithm on an augmented 1000-class MNIST dataset to further demonstrate the robustness of the proposed algorithm against mode collapse problem. More details of the experimental can be found in the Appendix. CIFAR is a natural scene dataset of 32 × 32. We use this dataset to evaluate the visual quality of the generated samples. Table 2 shows the inception scores of different GAN models on CIFAR-10 dataset. The inception score of the proposed model is much better than the baseline method WGAN MNIST CIFAR Figure 3: Examples of generated images using MNIST and CIFAR dataset. Method ScoreReal data 11.24 ± 0.16 WGAN 3.82 ± 0.06 MIX + WGAN BID2 4.04 ± 0.07 Improved-GAN 4.36 ± 0.04 ALI BID7 5.34 ± 0.05 DCGAN 6.40 ± 0.05Proposed method 4.53 ± 0.04 Table 2: Inception scores on CIFAR-10 dataset.that uses similar network architecture and training method. Note that although DCGGAN achieves a better score, it uses a more complex network architecture. Examples of the generated images are shown in Fig. 3. In this paper, we propose a primal-dual formulation for generative adversarial learning. This formulation interprets GANs from the perspective of convex optimization, and gives the optimal update of the discriminator and the generated distribution with convergence guarantee. By framing different variants of GANs under the convex optimization framework, the corresponding training algorithms can all be improved by pushing the generated distribution along the optimal direction. Experiments on two synthetic datasets demonstrate that the proposed formulation can effectively avoid mode collapse. It also achieves competitive quantitative evaluation scores on two benchmark real-world image datasets. The proof of convergence for dual-driven algorithms can be found in BID4, Chapter 3).The primal-dual-driven algorithm for continuous time update has been studied in BID8. Here, we show the convergence for the discrete-time case. We choose a step size α(t) that satisfies DISPLAYFORM0 Let z(t) = [x(t), λ λ λ(t)] T be a vector consisting of the primal and dual variables at the t-th iteration. The primal-dual-driven update can be expressed as: DISPLAYFORM1 where DISPLAYFORM2 and DISPLAYFORM3 Since the subgradient is bounded by assumption, there exists M > 0 such that ||T (·)|| 2 2 < M, where ||.|| 2 stands for the L 2 norm. Modes generated Inception Score BID23, and the networks are trained with Root Mean Square Propagation (RMSProp) with a learning rate of 1e-4. For GAN, the networks are trained with Adam with a learning rate of 1e-4. The minibatch size is 32. The bandwidth parameter for the Gaussian kernel is initialized to be σ = 0.5 and then is changed to 0.1 after 2000 iterations. We use the network structure in BID23 to evaluate the performance of our proposed method. The data is sampled from a mixture of 8 Gaussians of standard deviation of 0.02 uniformly located on a circle of radius 2. The noise samples are a vector of 256 independent and identically distributed (i.i.d.) Gaussian variables with mean zero and standard deviation of 1.The generator has two hidden layers of size 128 with ReLU activation. The last layer is a linear projection to two dimensions. The discriminator has one hidden layer of size 128 with ReLU activation followed by a fully connected network to a sigmoid activation. All the biases are initialized to be zeros and the weights are initalilzed via the "Xavier" initialization BID9. The training follows the primal-dual-driven algorithm, where both the generator and the discriminator are updated once at each iteration. The Adam optimizer is used to train the discriminator with 8e-4 learning rate and the generator with 4e-4 learning rate. The minibatch sample number is 64. For MNIST dataset, the generator network is a deconvolutional neural network. It has two fully connected layer with hidden size 1024 and 7 × ×7 × 128, two deconvolutional layers with number of units 64, 32, stride 2 and deconvolutional kernel size 4 × 4 for each layer, respectively, and a final convolutional layer with number of hidden unit 1 and convolutional kernel 4 × 4.. The discriminator network is a two layer convolutional neural network with number of units 64, 32 followed by two fully connected layer of hidden size 1024 and 1. The input noise dimension is 64.We employ ADAM optimization algorithm with initial learning rate 0.01 and β = 0.5. For CIFAR dataset, the generator is a 4 layer deconvolutional neural network, and the discriminator is a 4 layer convolutional neural network. The number of units for discriminator is, and the number of units for generator is. The stride for each deconvolutional and convolutional layer is two. We employ RMSProp optimization algorithm with initial learning rate of 0.0001, decay rate 0.95, and momentum 0.1.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a primal-dual subgradient method for training GANs and this method effectively alleviates mode collapse.

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO