Question: Represent the article for finding a single-sentence TLDR

Question: Generative neural networks map a standard, possibly distribution to a complex high-dimensional distribution, which represents the real world data set. However, a determinate input distribution as well as a specific architecture of neural networks may impose limitations on capturing the diversity in the high dimensional target space. To resolve this difficulty, we propose a training framework that greedily produce a series of generative adversarial networks that incrementally capture the diversity of the target space. We show theoretically and empirically that our training algorithm converges to the theoretically optimal distribution, the projection of the real distribution onto the convex hull of the network's distribution space. Generative Adversarial Nets (GAN) BID5 is a framework of estimating generative models. The main idea BID4 is to train two target network models simultaneously, in which one, called the generator, aims to generate samples that resemble those from the data distribution, while the other, called the discriminator, aims to distinguish the samples by the generator from the real data. Naturally, this type of training framework admits a nice interpretation as a twoperson zero-sum game and interesting game theoretical properties, such as uniqueness of the optimal solution, have been derived BID5. It is further proved that such adversarial process minimizes certain divergences, such as Shannon divergence, between the generated distribution and the data distribution. Simply put, the goal of training a GAN is to search for a distribution in the range of the generator that best approximates the data distribution. The range is often defined by the input latent variable z and its specific architecture, i.e., Π = {G(z, θ), θ ∈ Θ}. When the range is general enough, one could possibly find the real data distribution. However, in practice, the range is usually insufficient to perfectly describe the real data, which is typically of high dimension. As a , what we search for is in fact the I-projection BID2 of the real data distribution on Π. 1. The range of the generator Π is convex (see figure 1(a)), or it is not convex but the projection of the real data distribution on Π's convex hull (CONV Π) is in Π (see FIG0). 2. The range of the generator is non-convex and the projection of the real data distribution in CONV Π is not in the range Π (see figure 1(c)). In case 1, one can find the optimal distribution in Π to approximate real data set in CONV Π. But in case 2, using standard GANs with a single generator, one can only find the distribution in Π that is nearest to the projection. It then makes sense to train multiple generators and use a convex combination of them to better approximate the data distribution (than using a single generator in the non-convex case (see figure 1(c))).The above argument is based on the assumption that one could achieve global optimality by training, while this is not the case in general. When reaching a local optimal distribution, in order to improve performance, do we need to add more generators and restart training? In this paper, we put forward a sequential training procedure that adds generators one by one to improve the performance, without retraining the previously added generators. Our contributions can be summarized as follows.• We derive an objective function tailored for such a incremental training process. The objective function takes both the real data distribution and the pre-learned distribution into consideration. We show that with this new objective, we actually maximize marginal contribution when adding a new generator. We also put forward an incremental training algorithm based on the new objective function.• We prove that our algorithm always converges to the projection of real data distribution to the convex hull of the ranges of generators, which is the optimal solution with multiple generators. This property continues to hold in online settings where target distribution changes dynamically.• Our experiments show that our algorithm can overcome the local optimal issue mentioned above. We perform experiments on a synthetic dataset as well as two real world datasets, e.g., CelebA and MNIST, and conclude that our algorithm could improve the mixture distribution even in the case where the range is not sufficient enough.• Experiments also show that, compared with previous methods, our algorithm is fast and stable in reducing the divergence between mixture distribution and the real data. Recently, there have been intensive researches on improving the performance of generative adversarial neural networks. Two lines of works are closely related to our paper. They focus mainly on improving the discriminator and the generator respectively. The Unrolled GAN introduced by BID11 improves the discriminator by unrolling optimizing the objective during training, which stabilizes training and effectively reduces the mode collapse. D2GAN proposed by utilizes two discriminators to minimize the KL-divergence and the reverse KL-divergence respectively. It treats different modes more fairly, and thus avoids mode collapse. DFM introduced by brings a Denoising AutoEncoder (DAE) into the generator's objective to minimize the reconstruction error in order to get more information from the target manifold. BID12 proposed McGan based on mean and covariance feature matching to stabilize the training of GANs. Finally, WGAN introduced by employs the Wasserstein distance, which is a more appropriate measure of performance, and achieves more stable performance. These works are different from ours since they focus on the discriminator by measuring the divergence between the generated data and the real data more precisely. However, our work fixes the discriminator and tries to enrich the expressiveness of the generator by combining multiple generators.1.1.2 proposes two methods to improve the training process. The first is selfensembling GANs, which assembles the generators from different epochs to stabilize training. The other is Cascade GAN, where the authors train new generator using the data points with highest values from the discriminator. These two methods are heuristic ways to improve training, but with no theoretical guarantee. BID7 and BID3 proposed the methods called MGAN and multi-agent GANs respectively. The former introduces a classifier into the discriminator to catch different modes, while the later employ a new component into the generators' objective to promote diversity. BID1 introduces a new metric on distributions and proposes a MIX+GAN to search for an equilibrium. But all these methods need to train the multiple generators simultaneously, and none of them can deal with the case when the training process reaches a local optima. Also, these models lack flexibility, in the sense that when one tries to change the number of generators, all the generators need to be retrained. Another closely related work is BID15, in which the authors propose a method called AdaGAN, which is based on a robust reweighting scheme on the data set inspired from boosting. The idea is that the new generators should focus more on the previous bad training data. But AdaGAN and other boosting-like algorithms are based on the assumption that one generator could catch some modes precisely, which may not be reasonable since the generator always learns to generate the average samples among the real data set in order to obtain low divergence, especially when the generator's range is under condition of FIG0. In Section 5, we compare our algorithm with AdaGAN with different dataset. A GAN BID5 takes samples (a.k.a. latent variables z) from a simple and standard distribution as its input and generates samples in a high dimensional space to approximate the target distribution. This is done by training a generative neural network and an auxiliary discriminative neural network alternatively. An f- BID14 generalizes the adversarial training as minimizing the f-divergence between the real data distribution and the generated distribution, DISPLAYFORM0 dx. A GAN is a special f-GAN that minimizes the JensenShannon divergence. The general objective function of an f-GAN can be defined as follows: min DISPLAYFORM1 Here f * is the conjugate function of f in f-divergence; T represents a neural network regarded as the corresponding discriminator; finally, θ and ξ denote the parameters of the generator and the discriminator, respectively. The adversarial training method proposed by BID5 is playing a minimax game between the generator and the discriminator. Such a method can be caught in local optima and thus is undesirable (e.g., mode collapse).In this paper we propose a novel framework to train multiple generators sequentially: We maintain a group of generators (empty at the beginning) as well as their corresponding weights, then add new generators into the group one by one and rebalance the weights. In particular, only the newly added generator at each step is trained. The purpose here is to augment the capacity of the group of generators and mitigate the local optima issue. Define the distribution range of a generator as Π = {p | p = G(z, θ), θ ∈ Θ}, i.e., the set of distributions that the generator can produce with different parameter θ. The distribution range is determined by the distribution of input z and the architecture of the generative network. Define a generator group as G = {G 1, G 2,, . . ., G n}, where G i is the generator added in step i. We associate each generator with a weight ω i > 0. Then the mixed distribution of the group is: DISPLAYFORM0 ω i is the sum of weights. When a new generator G n+1 joins in the group G, the group becomes G = G ∪ {G n+1} and the mixed distribution becomes DISPLAYFORM1 In this section, we describe how we use a generator group to improve the performance and tackle the local optima issue mentioned previously. To train such a generator group, we propose an incremental training algorithm (algorithm 1) adding generators to the group sequentially. In algorithm 1, we use DISPLAYFORM0 repeat Build and initialize generator G i using the same network structure. Set target distribution for G i to be p target = DISPLAYFORM1 until Convergence D(·, ·) to denote the "distance" between two distributions, which can be any divergence (e.g., fdivergence or Wasserstein distance) or a general norm. The key step in algorithm 1 is the choice of the target distribution for training DISPLAYFORM2 j=1 ω j p j and after adding G i, the generator group G can perfectly produce the desired distribution p real. However, in general, we have D(p target, p i) = 0 and our algorithm proceeds in a greedy fashion, i.e., it always maximizes the marginal contribution of G i to the generator group G. We devote the rest of this section to proving the above statement. In algorithm 1, we use different loss functions for each generators. The marginal contribution of the (N + 1)-th generator is as follows when we adopt f-divergence as the distance measure: DISPLAYFORM0 To get a better approximation to the real distribution, we fix the existing generators in the group and tune the parameters of the new generator to minimize the distance between the new group and the real distribution. In fact, this is equivalent to maximizing the marginal contribution of the new generator DISPLAYFORM1 DISPLAYFORM2.To show this, we first introduce the χ 2 -divergence. DISPLAYFORM3 dx. Note that χ 2 -divergence is a special case of the f-divergence: DISPLAYFORM4 In fact, with some mild assumptions on f, the f -divergence is well-approximated by χ 2 -divergence when p and q are close. The following lemma can be obtained via Taylor expansion BID2. Lemma 1. For any f-divergence with f (u), if f (u) is twice differentiable at u = 1 and f > 0, then for any q and p close to q we have: DISPLAYFORM5 Proof of proposition 1. We rewrite the objective function equation 1 for χ 2 -divergence: DISPLAYFORM6 Based on the former definition, we obtain DISPLAYFORM7 +, which concludes the proof. According to algorithm 1, in each round, a new generator G N +1 is added and the loss function is set to be D(p target, p N +1). Therefore, when training each generator G i, the target distribution only depends on the real distribution and the previous generators in G. In particular, both of them are already known (figure 2).To minimize D(p target, p G N +1), we conduct adversarial training by using an auxiliary discriminator T: DISPLAYFORM0 where by the linearity of expectation: DISPLAYFORM1.Based on these, we propose an incremental training algorithm for G N +1 as algorithm 2. In this section, we show that although our framework which trains each generator in a greedy way, the output distribution of the generator group will always converge. Furthermore, the converged distribution is the closest one to the target distribution among the set of all possible distributions that a group of generators can produce (i.e., the optimal one within the distribution range of the group of generators).Recall our notation that the distribution range of a generator is Π. By taking a convex combination of multiple generators (with the same architecture), the set of all possible output distributions becomes the convex hull of Π: DISPLAYFORM0 Our algorithm greedily optimizes each G N +1 to minimize D(p target, p G N +1). By the Pinsker's inequality, the total variation distance between p target and p G N +1 is upper bounded by D(p target, p G N +1)/2 and we can easily extend it to χ 2 -divergence by D KL (p||q) ≤ D χ 2 (p||q) + 0.42. In other words, while greedily optimizing each G N +1, the distance between p target and p G N +1 is also approximately minimized. Hence it is reasonable to assume that for each G N +1, its distance to p target is approximately minimized with some tolerance ≥ 0, i.e., p G N +1 −p target ≤ inf p − p target +. Under such an assumption, our algorithm approximately converges to the the optimal distribution in CONV Π: Proposition 2. For any Π that is connected and bounded, algorithm 2 approximately converges to the optimal distribution within the closure of the convex hull CONV Π of Π.To simplify the argument, we fix each ω i to be 1 and embed the discrete probability distributions into a Hilbert space. In this case, each G N +1 approximately minimizes the distance to p target = (N + 1)p real − N i=1 p Gi can be formalized as: DISPLAYFORM0 and our algorithm approximately converges to the optimal distribution in CONV Π if as N → ∞, DISPLAYFORM1 Then proposition 2 is implied by the following lemma. Lemma 2. Consider a connected and bounded subset Π of a Hilbert space H and any target ρ ∈ H. Let {p * n} ∞ n=1 be a sequence of points in Π such that for ρ target = (n + 1)ρ − nT n, DISPLAYFORM2 Corollary 1. With the finite change of target distribution, algorithm 2 can converge to the new optimal distribution within CONV Π.Due to the space limit, we send the proof to the appendix. Based on corollary 1, regardless the change of target distribution, as long as it is an finite variation, algorithm 2 can converge to the projection of new target distribution. Due to the sequential nature and the above theoretical guarantee, our algorithm naturally generalizes the dynamic online settings. We test our algorithm on a synthesized Gaussian distribution dataset and two well-known real world datasets: CelebA and MNIST, which are the complex high dimensional real world distributions. We design the experiment to test our sequential training algorithm. The main purpose is not to demonstrate high quality , e.g., high definition pictures, but to show that our algorithm can search for an appropriate distribution that significantly improved the performance of mixture distributions as the number of generators increase, especially when the generator's range is rather limited. In all experiments, we use the Adam optimizer with learning rate of 5 × 10 −5, and β 1 = 0.5, β 2 = 0.9. Finally, we set weights ω i = 1 for convenience. Metric. As the method mentioned in BID14, when we fix the generator, we can train an auxiliary neural network to maximize the derived lower bound to measure the divergence between the generated data and the real data, i.e., D f (P ||Q). Based on these theories, we import an auxiliary neural network to measure the performance of different methods. The architecture of the auxiliary neural network is the same as the discriminator used in each experiment. We train it for 50 epoches, which is enough to measure the differences. Then we take the mean value of the last 100 iterations as the final output. Synthesized data. In this part, we design some experiments in R 2 space. The dataset is sampled from 8 independent two-dimensional Gaussian distributions (i.e., the blue points in FIG7 . The model is previously proposed by BID11 .Firstly, following the experiment designed in BID11 and BID7, we choose the latent variable z in a high dimensional space as z ∼ N (0, I 256), i.e., the distribution projection is likely to be in the generator's range, which meets the condition of FIG0. In FIG7, the blue points are the real data while the corresponding colored number represents the data points generated by each generator respectively. As FIG7 shows, we train up to 4 generators to approximate the data distribution and the first generator tends to catch the data with high probability around the centre of each Gaussian. As the number of generators increasing, generated data tends to cover the data away from the centre in order to be complementary to previous mixture distributions and thus gains a considerable marginal profit. These demonstrate our marginal maximization algorithm can promote the mixture distributions to cover the data with low probabilities. Secondly, we reduce the dimension of z to 1, i.e., z ∼ N and simplify the corresponding network architecture, so that the condition of figure 1(c) is likely met. In this part, we compare our algorithm with the state of the art incremental training method AdaGAN BID15 and the baseline method Orignal GAN. 1 We train up to 20 generators in each experiment with the same starting generator (i.e., identical first generator for each method), then measure the D χ 2 (p||q) between real distribution and the generated mixed distribution. We repeat the experiment for 30 times to reduce the effect of random noises. Figure 5 and figure 6 illustrate the average and the best performance with different numbers of generators, respectively. According to the , our algorithm approaches to p real faster than the other two methods and achieves the best performance among all three methods. In summary, our algorithm outperforms the other two both in terms of the speed of converging to the real distribution and the quality of the final under the case of figure 1(c).MNIST. In this experiment, we run our algorithm on the MNIST dataset BID9. We design this experiment to measure the performance of our algorithm for a more complex data distribution. We choose the latent variable as z ∼ N to limit the corresponding generator range and Then we train up to 22 generators to approximate the real distribution and the is showed in figure 4. Our algorithm outperforms the Original GAN but is inferior to the AdaGAN with the first 8 generators. As the number of generators increases, AdaGAN seems to run into a bottleneck while both our algorithm and the Original GAN gradually approximate to the real data distribution. In order to analysis the convergence, we further train up to 100 generators with both our algorithm and the original GAN. In FIG4, the horizontal dash lines represent the minimum value of the Wasserstein distance for the two method respectively. As showed in figure, the distance gradually decrease with the number of generators increasing, and our algorithm is much faster to reduce the distance and can even obtain a better performance. More over, as we tends to investigate the property of each generator in the generators' group, we measure the Wasserstein distance between distribution G(z, θ) and the real data, the experiment is showed in FIG5 and the dash lines in FIG5 represent the mean value of the 100 generators. Interestingly, the shows that, in each generator, original GAN tends to search a distribution in the distribution range that is closer to the real data distribution, while our algorithm is searching for a distribution that is complementary to the generators' group (i.e., a huge decrease in the mixture condition in FIG4) even if its own performance is poor (i.e., a high distance in the in FIG5).CelebA. We also conduct our experiment on the CelebA dataset BID10. As shown in figure 1, we start with an identical generator and train up to 6 generators using different methods. The measured Wasserstein distance between mixed distribution of Group G and the real-data distribution is showed in FIG0. In this experiment, we use the training method WGAN-GP proposed by BID6. The experiment indicates that our algorithm outperforms the other two methods after the second generator. It demonstrates the potential of our algorithm applying to real world datasets. Proof of lemma 2. Without loss of generality, we can assume ρ = 0, since otherwise we can add an offset −ρ to the Hilbert space H. DISPLAYFORM0 Letp ∈ Π be the point that minimizes the distance between −nT n and its projectionp ⊥ on line −nT n, i.e.,p = arg min p∈Π p ⊥ + nT n, where p ⊥ = p,nTn H nTn 2 · nT n. Then we can further bound d n+1 by p + nT n, DISPLAYFORM1 On one hand, since T n ∈ CONV Π,p ⊥ can be seen as the projection ofp on T n as well, hence p −p ⊥ ≤ p − T n. Note that Π is bounded, therefore p − T n is bounded by the diameter of Π, denoted as d ≥ 0.On the other hand, suppose that p * is the point closest to ρ = 0 within CONV Π, i.e., p * = arg min p∈CONV Π p. Since Π is connected, therefore the projection of Π on −nT n is the same with the projection of CONV Π. Hence, DISPLAYFORM2 In other words, DISPLAYFORM3 If p * = inf p∈CONV Π p > 0, then we have d n = n T n ≥ n p *. Hence DISPLAYFORM4 2 /2(n + 1) p * ≤ (p * +)(n + 1) + d 2 /2 p * · ln(n + 1).Then for any δ > 0, let N be sufficiantly large such that ln N N ≤ 2δ p * /d 2, we have DISPLAYFORM5 Otherwise, p * = 0 and d n+1 ≤ + d 2 + d 2 n. Note that the upper bound is increasing in d n, hence ∀n > 0, d n ≤ d * n for d * n defined as follows: DISPLAYFORM6 For which, we can easily prove by induction that d * n ≤ n + √ nd. Therefore T n = d n ≤ + d/ √ n, which immediately completes the proof. Proof of corollary 1. Without loss of generality, we assume the optimal projection of target distribution is changed from ρ to ρ after n 0 iterations, where n 0 ∈ R + is a constant value. Then we can derive T n+n0 − ρ ≤ n· Tn−ρ n+n0 DISPLAYFORM0, where n ∈ R + is the training iteration after change. Then based on lemma 2, we obtain lim n→+∞ T n − ρ ≤ inf p∈Π p − ρ +. On the other side, for a specific n 0, T n0 − ρ ≤ T n0 − ρ + ρ − ρ is a bounded value if the variation of target distribution is limited. Finally, we can obtain lim n→+∞ T n+n0 − ρ ≤ inf p∈Π p − ρ +, which concludes the proof. dx. For KL-divergence and χ 2 -divergence, the corresponding f (t) are f KL (t) = t log(t) and f χ 2 (t) = (t−1) 2 respectively. Import an auxiliary function as: DISPLAYFORM1 Then based on the monotonicity of F(t), we have F (t) min ≥ −0.42. DISPLAYFORM2

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a new method to incrementally train a mixture generative model to approximate the information projection of the real data distribution.

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.


Question: Represent the article for finding a single-sentence TLDR

Question: Generative priors have become highly effective in solving inverse problems including denoising, inpainting, and reconstruction from few and noisy measurements. With a generative model we can represent an image with a much lower dimensional latent codes. In the context of compressive sensing, if the unknown image belongs to the range of a pretrained generative network, then we can recover the image by estimating the underlying compact latent code from the available measurements. However, recent studies revealed that even untrained deep neural networks can work as a prior for recovering natural images. These approaches update the network weights keeping latent codes fixed to reconstruct the target image from the given measurements. In this paper, we optimize over network weights and latent codes to use untrained generative network as prior for video compressive sensing problem. We show that by optimizing over latent code, we can additionally get concise representation of the frames which retain the structural similarity of the video frames. We also apply low-rank constraint on the latent codes to represent the video sequences in even lower dimensional latent space. We empirically show that our proposed methods provide better or comparable accuracy and low computational complexity compared to the existing methods. Compressive sensing refers to a broad class of problems in which we aim to recover a signal from a small number of measurements -. Suppose we are given a sequence of measurements for t = 1,..., T as y t = A t x t + e t, where x t denotes the t th frame in the unknown video sequence, y t denotes its observed measurements, A t denotes the respective measurement operator, and e t denotes noise or error in the measurements. Our goal is to recover the video sequence (x t) from the available measurements (y t). The recovery problem becomes especially challenging as the number of measurements (in y t) becomes very small compared to the number of unknowns (in x t). Classical signal priors exploit sparse and low-rank structures in images and videos for their reconstruction -. However, the natural images exhibits far richer nonlinear structures than sparsity alone. We focus on a newly emerging generative priors that learn a function that maps vectors drawn from a certain distribution in a low-dimensional space into images in a highdimensional space. The generative model and optimization problems we use are inspired by recent work on using generative models for compressive sensing in -. Compressive sensing using generative models was introduced in, which used a trained deep generative network as a prior for image reconstruction from compressive measurements. Afterwards deep image prior (DIP) used an untrained convolutional generative model as a prior for solving inverse problems such as inpainting and denoising because of their tendency to generate natural images; the reconstruction problem involves optimization of generator network parameters. Inspired by these observations, a number of methods have been proposed for solving compressive sensing problem by optimizing generator network weights while keeping the latent code fixed at a random value,. Both DIP and deep decoder update the model weights to generate a given image; therefore, the generator can reconstruct wide range of images. One key difference between the two approaches is that the network used in DIP is highly overparameterized, while the one used in deep decoder is underparameterized. We observed two main limitations in the DIP and deep decoder-based video recovery that we seek to address in this paper. The latent codes in DIP and deep decoder methods are initialized at random and stay fixed throughout the recovery process. Therefore, we cannot infer the structural similarities in the images from the structural similarities in the latent codes. Both of these methods train one network per image. A naive approach to train one network per frame in a video will be computationally prohibitive, and if we train a single network to generate the entire video sequence, then their performance degrades. Therefore, we propose joint optimization over network weights γ and the latent codes z t to reconstruct video sequence. Thus we learn a single generator and a set of latent codes to represent a video sequence. We observe that when we optimize over latent code alongside network weights, the temporal similarity in the video frames is reflected in the latent code representation. To exploit similarities among the frames in a video sequence, we also include low-rank constraints on the latent codes. An illustration of different types of representations we use in this paper are shown in Figure 1. In this paper, we reconstruct a video sequence from the compressive measurements in by jointly optimizing over the latent codes z t and the network parameters γ. Since the frames in a video sequence exhibit rich redundancies in their representation, we impose a low-rank constraint on the latent codes to represent the video sequence with a more compact representation of the latent codes. The key contributions of this paper are as follows. • We demonstrate that joint optimization allows us to learn a single generator network for an entire video sequence and corresponding latent codes simultaneously. We demonstrate that this approach has lower computational complexity and requires less number of parameters to reliably generate the entire video sequence. Furthermore, joint optimization retains the similarity structure of the video frames in their latent representation which leaves further scope for different tasks which involves latent space manipulation. • Consecutive frames in a video sequence share lot of similarities. To encode similarities among the reconstructed frames, we introduce low-rank constraints on the generator latent codes. This enables us to represent a video sequence with even smaller number of parameters in the latent space. We show that, in some cases, the low-rank structure on the latent codes also provides a nice low-dimensional manifold. For a single image reconstruction, deep image prior solve the following optimization to obtain optimalγ, arg min In this optimization, z is initialized randomly and kept unaltered. To jointly optimize the latent codes and generator parameters for a video sequence, we use the similar formulation as in but optimize it over the z t and γ. The ing optimization problem can be written as The reconstructed video sequence can be generated using the estimated latent codes (ẑ 1, . . .,ẑ T) and generator weights (γ) asx t = Gγ(ẑ t). We initialize latent codes with samples drawn from a Gaussian distribution and normalize them to have unit norm. We initialize γ with random weights using the initialization scheme in. Initilizing the generator with a pretrained set of weights can potentially serve as a good initialization and lead to good and faster convergence. We tested both variants, but observed little difference in performance; therefore, we use random initialization of parameters in this paper. Each iteration of joint optimization consists of two steps: 1) latent code optimization and 2) network parameter optimization. After every gradient descent update of the latent codes, z t, we update the model parameters with stochastic gradient descent. In all of our experiments with joint optimization, we learned a single set of network weights for the entire sequence. We note that it is possible to divide a longer video sequences into small segments and learn different sets of network weights for each of them. At the end of our reconstruction process, we have a single set of trained weightsγ, reconstructed framesx t and their corresponding optimal latent codesẑ t. As we optimize over the latent codes and the network weights in joint optimization, the latent codes capture the temporal similarity of the video frames. To further exploit the redundancies in a video sequence, we assume that the variation in the sequence of images are localized and the latent codes sequence can be represented in a low-dimensional space compared to their ambient dimension. Let us define a matrix Z with all the latent codes as where z t is the latent code corresponding to t th image of the sequence. To impose a low-rank constraint, we solve the following constrained optimization: We solve using a projected gradient descent method in which we project the latent code estimates after every iteration to a manifold of rank-r matrices. To do that, we compute Z matrix and its rank-r approximation using principal component analysis (PCA) or singular value decomposition (SVD). In this manner, we can express each of the latent codes in terms of r orthogonal basis vectors vectors u 1,..., u r as where α ij is the weight of the corresponding basis vector. We can represent a video sequence with T frames with r orthogonal codes, and the lowrank representation of latent codes requires r × k + r × T parameters compared to T × k. This offers r(k) times compression to our latent code representation. As we observe later, we use r = 4 for k = 256 and T = 32 which gives us compression of 0.14 in latent code representation. In this paper we report the for one synthetic sequence which we refer to as'Rotating MNIST'. In this sequence, we resize one MNIST digit to 64 × 64 and rotate by 2 • per frame for a total of 32 frames. We experiment on different real video and UCF101 dataset. In Table I, we report our for'Handclapping','Handwaving' and'Walking' video sequences from KTH dataset;'Archery','Apply Eye Makeup' and'Band Marching' video sequences from UCF101 dataset. We centered and resized every frame in KTH videos to 64 × 64 and UCF101 videos to 256 × 256 pixels. We used the well-known DCGAN architecture for our generators, except that we do not use any batch-normalization layer. The latent code dimensions for grayscale 64 × 64, RGB 64 × 64 and RGB 256 × 256 video sequences are 64, 256 and 512 respectively. We use Adam optimizer for generator weights optimization and SGD for latent code optimization. Unless otherwise mentioned, we use rank=4 constraint as low rank constraint because we empirically found that we need a least rank=4 for a video sequence with 32 frames to get comparable performance. We show comparison with classical total variation minimization based TVAL3D (3D extension of TVAL3 ) algorithm and state-of-the-art untrained generative prior based deep decoder on denoising, inpainting, and compressive sensing tasks. We use two different deep decoder settings: underparameterized deep decoder (UP deep decoder) and overparameterized deep decoder (OP deepdecoder). Although the authors suggested deep decoder to be UP, we report the for OP deep decoder as well because it shows better performance and its hyperparameters are tuned by the authors of deep decoder. Other then denoising and inpainting, we performed compressive random projection experiments where we used separable measurements, Y = P T XP, where X, Y are reshaped versions of x, y as 2D matrices, P is a random projection matrix. We report the for denoising experiment at 20 dB SNR noise, inpainting experiment for 80% missing pixels and compressive sensing experiments for 20% available measurements in Table I. From the , we can observe that joint optimization with/without low-rank constraint outperform TVAL3D algorithm and UP deep decoder. It performs at par with OP deep decoder. In Figure 2, we show reconstruction performance for denoising, inpainting and compressive sensing at different measurement rate or noise level for'Handwaving' video sequence. We can get the similar observation from these curves as well. We report some reconstruction for'Handwaving' sequence in Figure 2. From the reconstructions, we can say that joint optimization performs at par with the comparing algorithms. It especially performs well in reconstructing details from masked frames. ) and deep decoder. The are average over five experiments with different random measurement matrices (or noise in the case of denoising). Rotating The computational complexity of our proposed methods vary with the choice of the generator structure. We have chosen DCGAN generator structure for our experiments. We calculate memory requirement for gradient descent using torchsummary package. For a single 64 × 64 RGB image, memory requirement for UP deep decoder, OP deep decoder and joint optimization is 2.75 MB, 66.48 MB and 2.06 MB respectively. For a single 256 × 256 RGB image, memory requirement for UP deep decoder, OP deep decoder and joint optimization is 44.03 MB, 1239.75 MB and 10.88 MB respectively. For a RGB video seqence with 32 frames, UP deep decoder will require 11, 304 × 32 parameters while OP deep decoder will have 397, 056 × 32 (12.7M) parameters. On the other hand we need 4,852,736 and 6,988,544 network parameters to represent RGB 64 × 64 and 256 × 256 video sequences, respectively in joint optimization method with DCGAN generator. Because of the huge memory requirement, OP deep decoder is not suitable for optimization over entire video sequence whereas the low capacity hinders UP deep decoder from generating entire video sequence. To investigate the similarity structure in the latent codes obtained by joint optimization, we performed another experiment in which we concatenated 16 frames from each of the The cosine similarity matrices for the video frames, compressive measurements, latent codes for with fixed latent codes (random), latent codes for joint optimization, and latent codes for joint optimization with low-rank are presented in Figure 3 (a)-(e). We can distinguish the video sequences from the pairwise similarity matrices of the latent codes we estimate with joint optimization. We also observe that the low-rank constraint improves the similarity matrix. We mentioned that using low rank constraint we can represent the video sequences in much lower dimensional space. If the generator function is continuous it makes sense that the latent space representation of a video sequence will retain their sequential structure in some low dimensional representation. We demonstrate one such example using rank=2 constraint on the latent codes while reconstructing'Rotating MNIST' sequence from its masked version with 80% pixels missing. As we are enforcing, rank=2 constraint taking mean and first principal component, the latent codes should fall on line. In Figure 4, we represent the latent codes in a 2D plane using 2 orthogonal basis vectors. t th point in Figure 4, represent latent code of t th frame. We can observe that latent codes are maintaining sequence in their 2D dimensional representation. For complex motions, it might take higher dimensional representation to observe such sequential pattern.

Represent the short summary for finding the corresponding scientific article (long)

Recover videos from compressive measurements by learning a low-dimensional (low-rank) representation directly from measurements while training a deep generator.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We develop VAEs where the encoder takes a model parameter vector as input, so we can do rapid inference for many models


Question: represent the article for finding a single-sentence tldr

Question: Magnitude-based pruning is one of the simplest methods for pruning neural networks. Despite its simplicity, magnitude-based pruning and its variants demonstrated remarkable performances for pruning modern architectures. Based on the observation that the magnitude-based pruning indeed minimizes the Frobenius distortion of a linear operator corresponding to a single layer, we develop a simple pruning method, coined lookahead pruning, by extending the single layer optimization to a multi-layer optimization. Our experimental demonstrate that the proposed method consistently outperforms the magnitude pruning on various networks including VGG and ResNet, particularly in the high-sparsity regime. The "magnitude-equals-saliency" approach has been long underlooked as an overly simplistic baseline among all imaginable techniques to eliminate unnecessary weights from over-parametrized neural networks. Since the early works of; which provided more theoretically grounded alternative of magnitude-based pruning (MP) based on second derivatives of loss function, a wide range of methods including Bayesian / information-theoretic approaches (; ; ;), pregularization (; ;), sharing redundant channels , and reinforcement learning approaches (; ;) has been proposed as more sophisticated alternatives. On the other hand, the capabilities of MP heuristics are gaining attention once more. Combined with minimalistic techniques including iterative pruning and dynamic reestablishment of connections , a recent large-scale study by claims that MP can achieve a state-of-the-art trade-off of sparsity and accuracy on ResNet-50. The unreasonable effectiveness of magnitude scores often extends beyond the strict domain of network pruning; a recent experiment by suggests an existence of an automatic subnetwork discovery mechanism underlying the standard gradient-based optimization procedures of deep, overparametrized neural networks by showing that the MP algorithm finds an efficient trainable subnetwork. These observations constitute a call to revisit the "magnitude-equals-saliency" approach for a better understanding of deep neural network itself. As an attempt to better understand the nature of MP methods, we study a generalization of magnitude scores under a functional approximation framework; by viewing MP as a relaxed minimization of distortion in layerwise operators introduced by zeroing out parameters, we consider a multi-layer extension of the distortion minimization problem. Minimization of the newly suggested distortion measure which'looks ahead' the impact of pruning on neighboring layers gives birth to a novel pruning strategy, coined lookahead pruning (LAP). In this paper, we focus on comparison of the proposed LAP scheme to its MP counterpart. We empirically demonstrate that LAP consistently outperforms the MP under various setups including linear networks, fully-connected networks, and deep convolutional and residual networks. In particular, the LAP consistently enables more than ×2 gain in the compression rate of the considered models, with increasing benefits under the high-sparsity regime. Apart from its performance, the lookahead pruning method enjoys additional attractive properties: • Easy-to-use: Like magnitude-based pruning, the proposed LAP is a simple score-based approach agnostic to model and data, which can be implemented by computationally light elementary tensor operations. Unlike most Hessian-based methods, LAP does not rely on an availability of training data except for the retraining phase. It also has no hyper-parameter to tune, in contrast to other sophisticated training-based and optimization-based schemes. • Versatility: As our method simply replaces the "magnitude-as-saliency" criterion with a lookahead alternative, it can be deployed jointly with algorithmic tweaks developed for magnitudebased pruning, such as iterative pruning and retraining or joint pruning and training with dynamic reconnections . The remainder of this manuscript is structured as follows: In Section 2, we introduce a functional approximation perspective toward MP and motivate LAP and its variants as a generalization of MP for multiple layer setups; in Section 3 we explore the capabilities of LAP and its variants with simple models, then move on to apply LAP to larger-scale models. We begin by a more formal description of the magnitude-based pruning (MP) algorithm . Given an L-layer neural network associated with weight tensors W 1,..., W L, the MP algorithm removes connections with smallest absolute weights from each weight tensor, until the desired level of sparsity has been achieved. This layerwise procedure is equivalent to finding a mask M whose entries are either 0 or 1, incurring a smallest Frobenius distortion, measured by min where denotes the Hadamard product, · 0 denotes the entry-wise 0 -norm, and s is a sparsity constraint imposed by some operational criteria. Aiming to minimize the Frobenius distortion (Eq.), the MP algorithm naturally admits a functional approximation interpretation. For the case of a fully-connected layer, the maximal difference between the output from a pruned and an unpruned layer can be bounded as Namely, the product of the layerwise Frobenius distortion upper bounds the output distortion of the network incurred by pruning weights. Note that this perspective on MP as a worst-case distortion minimization was already made in , which inspired an advent of the layerwise optimal brain surgery (L-OBS) procedure. A similar idea holds for convolutional layers. For the case of a two-dimensional convolution with a single input and a single output channel, the corresponding linear operator takes a form of a doubly block circulant matrix constructed from the associated kernel tensor (see, e.g.,). Here, the Frobenius distortion of doubly block circulant matrices can be controlled by the Frobenius distortion of the weight tensor of the convolutional layer. The case of multiple input/output channel or non-circular convolution can be dealt with similarly using channel-wise circulant matrices as a block. We refer the interested readers to Sedghi et al. (2019 The myopic optimization (Eq.) based on the per-layer Frobenius distortion falls short even in the simplest case of the two-layer linear neural network with one-dimensional output, where we consider predictors of a form Y = u W x and try to minimize the Frobenius distortion of u W (equivalent to 2 distortion in this case). Here, if u i is extremely large, pruning any nonzero element in the i-th row of W may incur a significant Frobenius distortion. Motivated by this observation, we consider a block approximation analogue of the magnitude-based pruning objective Eq.. Consider an L-layer neural network with associated weight tensors W 1,..., W L, and assume linear activation for simplicity (will be extended to nonlinear cases later in this section). Let J (W i) denote the Jacobian matrix corresponding to the linear operator characterized by W i. For pruning the i-th layer, we take into account the weight tensors of neighboring layers W i−1, W i+1 in addition to the original weight tensor W i. In particular, we propose to minimize the Frobenius distortion of the operator block An explicit minimization of the block distortion (Eq.), however, is computationally intractable in general (see Appendix D for a more detailed discussion). To avoid an excessive computational overhead, we propose to use the following score-based pruning algorithm, coined lookahead pruning (LAP), for approximating Eq.: For each entry w of W i, we prune away the weights with the smallest value of lookahead distortion (in a single step), defined as where W i | w=0 denotes the tensor whose entries are equal to the entries of W i except for having zeroed out w. We let both W 0 and W L+1 to be tensors consisting of ones. In other words, lookahead distortion (Eq.) measures the distortion (in Frobenius norm) induced by pruning w while all other weights remain intact. For three-layer blocks consisting only of fully-connected layers and convolutional layers, Eq. reduces to the following compact formula: for an edge w connected to the j-th input neuron/channel and the k-th output neuron/channel of the i-th layer, where its formal derivation is presented in Appendix E. where |w| denotes the weight of w, W [j, :] denotes the slice of W composed of weights connected to the j-th output neuron/channel, and W [:, k] denotes the same for the k-th input neuron/channel. In LAP, we compute the lookahead distortion for all weights, and then remove weights with smallest distortions in a single step (as done in MP). A formal description of LAP is presented in Algorithm 1. We also note the running time of LAP is comparable with that of MP (see Appendix G). LAP on linear networks. To illustrate the benefit of lookahead, we evaluate the performance of MP and LAP on a linear fully-connected network with a single hidden layer of 1,000 nodes, trained with MNIST image classification dataset. Fig. 2a and Fig. 2b depict the test accuracy of models pruned with each methods, before and after retraining steps. As can be expected from the discrepancy between the minimization objectives (Eqs. and), networks pruned with LAP outperform networks pruned with MP at every sparsity level, in terms of its performance before a retraining phase. Remarkably, we observe that test accuracy of models pruned with LAP monotonically increases from 91.2% to 92.3% as the sparsity level increases, until the fraction of surviving weights reaches 1.28%. At the same sparsity level, models pruned with MP achieves only 71.9% test accuracy. We also observe that LAP leads MP at every sparsity level even after a retraining phase, with an increasing margin as we consider a higher level of sparsity. Understanding LAP with nonlinear activations. Most neural network models in practice deploy nonlinear activation functions, e.g., rectified linear units (ReLU). Although the lookahead distortion was originally derived using linear activation functions, LAP can also be used for nonlinear networks, as the quantity L i (w) remains relevant to the original block approximation point of view. This is especially true when the network is severely over-parametrized. To see this, consider a case where one aims to prune a connection in the first layer of a two-layer fully-connected network with ReLU, i.e., where σ(x) = max{0, x} is applied entrywise. Under the over-parametrized scenario, zeroing out a single weight may alter the activation pattern of connected neurons with only negligible probability, which allows one to decouple the probability of activation of each neuron from the act of pruning each connection. This enables us to approximate the root mean square distortion of the network output introduced by pruning w of W 1 by √ p k L 1 (w), where k is the index of the output neuron that w is connected to, and p k denotes the probability of activation for the k-th neuron. In this sense, LAP (Algorithm 1) can be understood as assuming i.i.d. activations of neurons, due to a lack of an additional access to training data. In other word, LAP admits a natural extension to the regime where we assume an additional access to training data during the pruning phase. This variant, coined LAP-act, will be formally described in Appendix F, with experimental comparisons to another datadependent baseline of optimal brain damage (OBD) . Another theoretical justification of using the lookahead distortion (Eq.) for neural networks with nonlinear activation functions comes from recent discoveries regarding the implicit bias imposed by training via stochastic gradient descent . See Appendix M for a detailed discussion. As will be empiricically shown in Section 3.1, LAP is an effective pruning strategy for sigmoids and tanh activations, that are not piece-wise linear as ReLU. Batch normalization (BN), introduced by , aims to normalize the output of a layer per batch by scaling and shifting the outputs with trainable parameters. Based on our functional approximation perspective, having batch normalization layers in a neural network is not an issue for MP, which relies on the magnitudes of weights; batch normalization only affects the distribution of the input for each layer, not the layer itself. On the other hand, as the lookahead distortion (Eq.) characterizes the distortion of the multi-layer block, one must take into account batch normalization when assessing the abstract importance of each connection. The revision of lookahead pruning under the presence of batch normalization can be done fairly simply. Note that such a normalization process can be expressed as for some a, b ∈ R dim(x). Hence, we revise the lookahead pruning to prune the connections with a minimum value of where a i [k] denotes the k-th index scaling factor for the BN layer placed at the output of the i-th fully-connected or convolutional layer (if BN layer does not exist, let a i [k] = 1). This modification of LAP makes it an efficient pruning strategy, as will be empirically verified in Section 3.3. As the LAP algorithm (Algorithm 1) takes into account current states of the neighboring layers, LAP admits several variants in terms of lookahead direction, order of pruning, and sequential pruning methods; these methods are extensively studied in Section 3.2 Along with "vanilla" LAP, we consider in total, six variants, which we now describe below: Mono-directional LAPs. To prune a layer, LAP considers both preceding and succeeding layers. Looking forward, i.e., only considering the succeeding layer, can be viewed as an educated modification of the internal representation the present layer produces. Looking backward, on the other hand, can be interpreted as only taking into account the expected structure of input coming into the present layer. The corresponding variants, coined LFP and LBP, are tested. Order of pruning. Instead of using the unpruned tensors of preceding/succeeding layers, we also consider performing LAP on the basis of already-pruned layers. This observation brings up a question of the order of pruning; an option is to prune in a forward direction, i.e., prune the preceding layer first and use the pruned weight to prune the succeeding, and the other is to prune backward. Both methods are tested, which are referred to as LAP-forward and LAP-backward, respectively. Sequential pruning. We also consider a sequential version of LAP-forward/backward methods. More specifically, if we aim to prune total p% of weights from each layer, we divide the pruning budget into five pruning steps and gradually prune (p/5)% of the weights per step in forward/backward direction. Sequential variants will be marked with a suffix "-seq". In this section, we compare the empirical performance of LAP with that of MP. More specifically, we validate the applicability of LAP to nonlinear activation functions in Section 3.1. In Section 3.2, we test LAP variants from Section 2.3. In Section 3.3, we test LAP on VGG , ResNet , and Wide ResNet . Experiment setup. We consider five neural network architectures: The fully-connected network (FCN) under consideration is consist of four hidden layers, each with 500 neurons. The convolutional network (Conv-6) consists of six convolutional layers, followed by a fully-connected classifier with two hidden layers with 256 neurons each; this model is identical to that appearing in the work of suggested as a scaled-down variant of VGG. 2 VGG-19 is used, with an addition of batch normalization layers after each convolutional layers, and a reduced number of fully-connected layers from three to one. 3 ResNets of depths {18, 50} are used. WRN of 16 convolutional layers and widening factor 8 (WRN-16-8) is used. All networks used ReLU activation function, except for the experiments in Section 3.1. We mainly consider image classification tasks. In particular, FCN is trained on MNIST dataset , Conv-6, VGG, and ResNet are trained on CIFAR-10 dataset , and VGG, ResNet, and WRN are trained on Tiny-ImageNet. 4 We focus on the one-shot pruning of MP and LAP, i.e., models are trained with a single training-pruning-retraining cycle. All in this section are averaged over five independent trials. We provide more details on setups in Appendix A. We first compare the performance of LAP with that of MP on FCN using three different types of activation functions: sigmoid, and tanh, and ReLU. Figs. 3a to 3c depict the performance of models pruned with LAP (Green) and MP (Red) under various levels of sparsity. Although LAP was motivated primarily from linear networks and partially justified for positivehomogenous activation functions such as ReLU, the experimental show that LAP consistently outperforms MP even on networks using sigmoidal activation functions. We remark that LAP outperforms MP by a larger margin as fewer weights survive (less than 1%). Such a pattern will be observed repeatedly in the remaining experiments of this paper. In addition, we also check whether LAP still exhibits better test accuracy before retraining under the usage of nonlinear activation functions, as in the linear network case (Fig. 2b). Fig. 3d illustrates the test accuracy of pruned FCN using ReLU on MNIST dataset before retraining. We observe that the network pruned by LAP continues to perform better than MP in this case; the network pruned by LAP retains the original test accuracy until only 38% of the weights survive, and shows less than 1% performance drop with only 20% of the weights remaining. On the other hand, MP requires 54% and 30% to achieve the same level of performance, respectively. In other words, the models pruned with MP requires about 50% more survived parameters than the models pruned with LAP to achieve a similar level of performance before being retrained using additional training batches. Now we evaluate LAP and its variants introduced in Section 2.3 on FCN and Conv-6, each trained on MNIST and CIFAR-10, respectively. Table 1 summarizes the experimental on FCN and Table 2 summarizes the on Conv-6. In addition to the baseline comparison with MP, we also compare with random pruning (RP), where the connection to be pruned was decided completely independently. We observe that LAP performs consistently better than MP and RP with similar or smaller variance in any case. In the case of an extreme sparsity, LAP enjoys a significant performance gain; over 75% gain on FCN and 14% on Conv-6. This performance gain comes from a better training accuracy, instead of a better generalization; see Appendix L for more information. Comparing mono-directional lookahead variants, we observe that LFP performs better than LBP in the low-sparsity regime, while LBP performs better in the high-sparsity regime; in any case, LAP performed better than both methods. Intriguingly, the same pattern appeared in the case of the ordered pruning. Here, LAP-forward can be considered an analogue of LBP in the sense that they both consider layers closer to the input to be more important. Likewise, LAP-backward can be considered an analogue of LFP. We observe that LAP-forward performs better than LAP-backward in the high-sparsity regime, and vice versa in the low-sparsity regime. Our interpretation is as follows: Whenever the sparsity level is low, the importance of a carefully curating the input signal is not significant due to high redundancies in natural image signal. This causes a relatively low margin of increment by looking backward in comparison to looking forward. When the sparsity level is high, the input signal is scarce, and the relative importance of preserving the input signal is higher. Finally, we observe that employing forward/backward ordering and sequential methods leads to a better performance, especially in the high-sparsity regime. There is no clear benefit of adopting directional methods in the low-sparsity regime. The relative gain in performance with respect to LAP is either marginal, or unreliable. (Tables 3 and 4), and VGG-19, ResNet-50, and WRN-16-8 on TinyImageNet (Tables 5 to 7). For models trained on CIFAR-10, we also test LAP-forward to verify the observation that it outperforms LAP in the high-sparsity regime on such deeper models. We also report additional experimental on VGG-{11, 16} trained on CIFAR-10 in Appendix B. For models trained on Tiny-ImageNet, top-1 error rates are reported in Appendix C. From Tables 3 to 7, we make the following two observations: First, as in Section 3.2, the models pruned with LAP consistently achieve a higher or similar level of accuracy compared to models pruned with MP, at all sparsity levels. In particular, test accuracies tend to decay at a much slower rate with LAP. In Table 3, for instance, we observe that the models pruned by LAP retain test accuracies of 70∼80% even with less than 2% of weights remaining. In contrast, the performance of models pruned with MP falls drastically, to below 30% accuracy. This observation is consistent on both CIFAR-10 and Tiny-ImageNet datasets. Second, the advantages of considering an ordered pruning method (LAP-forward) over LAP is limited. While we observe from Table 3 that LAP-forward outperforms both MP and LAP in the highsparsity regime, the gain is marginal considering standard deviations. LAP-forward is consistently worse than LAP (by at most 1% in absolute scale) in the low-sparsity regime. In this work, we interpret magnitude-based pruning as a solution to the minimization of the Frobenius distortion of a single layer operation incurred by pruning. Based on this framework, we consider the minimization of the Frobenius distortion of multi-layer operation, and propose a novel lookahead pruning (LAP) scheme as a computationally efficient algorithm to solve the optimization. Although LAP was motivated from linear networks, it extends to nonlinear networks which indeed minimizes the root mean square lookahead distortion assuming i. τ fraction in all fully-connected layers, except for the last layer where we use (1 + q)/2 instead. For FCN, we use (p, q) = (0, 0.5). For Conv-6, VGGs ResNets, and WRN, we use (0.85, 0.8). For ResNet-{18, 50}, we do not prune the first convolutional layer. The range of sparsity for reported figures in all tables is decided as follows: we start from τ where test error rate starts falling below that of an unpruned model and report the at τ, τ + 1, τ + 2,... for FCN and Conv-6, τ, τ + 2, τ + 4,... for VGGs, ResNet-50, and WRN, and τ, τ + 3, τ + 6,... for ResNet-18. In this section, we show that the optimization in Eq. is NP-hard by showing the reduction from the following binary quadratic programming which is NP-hard : for some symmetric matrix A ∈ R n×n. Without loss of generality, we assume that the minimum eigenvalue of A (denoted with λ) is negative; if not, Eq. admits a trivial solution x = (0, . . ., 0). Assuming λ < 0, Eq. can be reformulated as: where H = A − λI. Here, one can easily observe that the above optimization can be solved by solving the below optimization for s = 1,..., n min x∈{0,1} n: i xi=s Finally, we introduce the below equality where 1 denotes a vector of ones, U is a matrix consisting of the eigenvectors of H as its column vectors, and Λ is a diagonal matrix with corresponding (positive) eigenvalues of H as its diagonal elements. The above equality shows that Eq. is a special case of Eq. by choosing W 1 = √ ΛU, W 2 = 1, W 3 = 1 and M = 1 − x. This completes the reduction from Eq. to Eq.. In this section, we provide a derivation of Eq. for the fully-connected layers. The convolutional layers can be handled similarly by substituting the multiplications in Eqs. and by the convolutions. The Jacobian matrix of the linear operator correponding to a fully-connected layer is the weight matrix itself, i.e. J (W i) = W i. From this, lookahead distortion can be reformulated as Now, we decompose the matrix product W i+1 W i W i−1 in terms of entries of W i as below: where, and j-th row of W i−1, respectively. The contribution of a single entry w: ]. Therefore, in terms of the Frobenius distortion, we conclude that which completes the derivation of Eq. for fully-connected layers. F LAP-ACT: IMPROVING LAP USING TRAINING DATA Recall two observations made from the example of two-layer fully connected network with ReLU activation appearing in Section 2.1: LAP is designed to reflect the lack of knowledge about the training data at the pruning phase; once the activation probability of each neuron can be estimated, it is possible to refine LAP to account for this information. In this section, we continue our discussion on the second observation. In particular, we study an extension of LAP called lookahead pruning with activation (LAP-act) which prunes the weight with smallest value of Here, W i is a scaled version of W i and w is the corresponding scaled value of w, defined by where I ij denotes the set of output indices in the j-th output neuron/channel of i-th layer (for fully connected layers, this is a singleton). Also, p k denotes the neuron's probability of activation, which can be estimated by passing the training data. We derive LAP-act (Eq.) in Appendix F.1 and perform preliminary empirical validations in Appendix F.2 with using optimal brain damage (OBD) as a baseline. We also evaluate a variant of LAP using Hessian scores of OBD instead of magnitude scores. It turns out that in the small networks (FCN, Conv-6), LAP-act outperforms OBD. Consider a case where one aims to prune a connection of a network with ReLU, i.e., where σ(x) = max{0, x} is applied entrywise. Under the over-parametrized scenario, zeroing out a single weight may alter the activation pattern of connected neurons with only negligible probability, which allows one to decouple the probability of activation of each neuron from the act of pruning each connection. From this observation, we first construct the below random distortion, following the philosophy of the linear lookahead distortion Eq. where J (W i) denotes a random matrix where ] and g i [k] is a 0-1 random variable corresponding to the activation, i.e., g i [k] = 1 if and only if the k-th output of the i-th layer is activated. However, directly computing the expected distortion with respect to the real activation distribution might be computationally expensive. To resolve this issue, we approximate the root mean-squared lookahead distortion by applying the mean-field approximation to the activation probability of neurons, i.e., all activations are assumed to be independent, as denotes the mean-field approximation of p(g). Indeed, the lookahead distortion with ReLU nonlinearity (Eq.) or three-layer blocks consisting only of the fully-connected layers and the convolutional layers can be easily computed by using the rescaled weight matrix W i: where I i,j denotes a set of output indices in the j-th output neuron/channel of the i-th layer. Finally, for an edge w connected to the j-th input neuron/channel and the k-th output neuron/channel of the i-th layer, Eq. reduces to where w denotes the rescaled value of w. This completes the derivation of Eq.. We compare the performance of three algorithms utilizing training data at the pruning phase: optimal brain damage (OBD) which approximates the loss via second order Taylor seris approximation with the Hessian diagonal , LAP using OBD instead of weight magnitudes (OBD+LAP), and LAP-act as described in this section. We compare the performances of three algorithms under the same experimental setup as in Section 3.2. To compute the Hessian diagonal for OBD and OBD+LAP, we use a recently introduced software package called "BackPACK," , which is the only open-source package supporting an efficient of Hessians, up to our knowledge. Note that the algorithms evaluated in this section are also evaluated for global pruning experiments in Appendix I. The experimental for FCN and Conv-6 are presented in Tables 13 and 14. Comparing to algorithms relying solely on the model parameters for pruning (MP/LAP in Tables 1 and 2), we observe that OBD performs better in general, especially in the high sparsity regime. This observation is coherent to the findings of. Intriguingly, however, we observe that applying lookahead critertion to OBD (OBD+LAP) significantly enhances to OBD significantly enhances the performance in the high sparsity regime. We hypothesize that LAP helps capturing a correlation among scores (magnitude or Hessian-based) of adjacent layers. Also, we observe that LAP-act consistently exhibits a better performance compared to OBD. This is somewhat surprising, in the sense that LAP-act only utilizes (easier-to-estimate) information about activation probabilities of each neuron to correct lookahead distortion. The average running time of OBD, OBD+LAP, and LAP-act is summarized in Table 15. We use Xeon E5-2630v4 2.20GHz for pruning edges, and additionally used a single NVidia GeForce GTX-1080 for the computation of Hessian diagonals (used for OBD, OBD+LAP) and activation probabiility (for LAP-act). We observe that LAP-act runs in a significantly less running time than OBD/OBD+LAP, and the gap widens as the number of parameters and the dimensionality of the dataset increases (from MNIST to CIFAR-10). MP comprises of three steps: computing the absolute value of the tensor, sorting the absolute values, and selecting the cut-off threshold and zero-ing out the weights under the threshold. Steps and remain the same in LAP, and typically takes O(n log n) steps (n denotes the number of parameters in a layer). On the other hand, Step is replaced by computing the lookahead distortion for each parameter w. Fortunately, this need not be computed separately for each parameter. Indeed, one can perform tensor operations to compute the squared lookahead distortion, which has the same ordering with lookahead distortion. For fully-connected layers with 2-dimensional Jacobians, the squared lookahead distortion for where 1 i denotes all-one matrix of size d i−2 × d i; multiplying 1 i denotes summing operation along an axis and duplicating summed into the axis, and 2 denotes the element-wise square operation. The case of convolutional layers can be handled similarly. We note that an implementation of Eq. Table 16, where we fixed the layerwise pruning rate to be uniformly 90%. The codes are implemented with PyTorch, and the computations have taken place on 40 CPUs of Intel Xeon E5-2630v4 @ 2.20GHz. All figures are averaged over 100 trials. We make two observations from Table 16. First, the time required for LAP did not exceed 150% of the time required for MP, confirming our claim on the computational benefits of LAP. Second, most of the added computation comes from considering the factors from batch normalization, without which the added computation load is ≈5%. In the main text, LAP is compared to the MP in the context of unstructured pruning, where we do not impose any structural constraints on the set of connections to be pruned together. On the other hand, the magnitude-based pruning methods are also being used popularly as a baseline for channel pruning , which falls under the category of structured pruning. MP in channel pruning is typically done by removing channels with smallest aggregated weight magnitudes; this aggregation can be done by either taking 1 -norm or 2 -norm of magnitudes. Similarly, we can consider channel pruning scheme based on an 1 or 2 aggregation of LAP distortions, which we will call LAP-1 and LAP-2 (as opposed to MP-1 and MP-2). We compare the performances of LAP-based channel pruning methods to MP-based channel pruning methods, along with another baseline of random channel pruning (denoted with RP). We test with Conv-6 (Table 17) and VGG-19 (Table 18) networks on CIFAR-10 dataset. All reported figures are averaged over five trials, experimental settings are identical to the unstructure pruning experiments unless noted otherwise. Similar to the case of unstructured pruning, we observe that LAP-based methods consistently outperform MP-based methods. Comparing 1 with 2 aggregation, we note that LAP-2 performs better than LAP-1 in both experiments, by a small margin. Among MP-based methods, we do not observe any similar dominance. Table 19 and Table 20. In this methods, we prune a fraction of weights with smallest scores (e.g. weight magnitude, lookahead distortion, Hessian-based scores) among all weights in the whole network. The suffix "-normalize" in the tables denotes that the score is normalized by the Frobenius norm of the corresponding layer's score. For MP, LAP, OBD+LAP and LAP-act, we only report the for global pruning with normalization, as the normalized versions outperform the unnormalized ones. In the case of OBD, whose score is already globally designed, we report the for both unnormalized and normalized versions. As demonstrated in Section 3.2 for fixed layerwise pruning rates, we observe that LAP and its variants perform better than their global pruning baselines, i.e. MP-normalize and OBD. We also note that LAP-normalize performs better than MP with pre-specified layerwise pruning rates (appeared in Section 3.2), with a larger gap for higher levels of sparsity. We test LAP-all on FCN under the same setup as in Section 3.2, and report the in Table 21. All figures are averaged over five trials. We observe that LAP-all achieves a similar level of performance to LAP, while LAP-all underperforms under a high-sparsity regime. We suspect that such shortfall originates from the accumulation of error terms incurred by ignoring the effect of activation functions, by which the benefits of looking further fades. An in-depth theoretical analysis for the determination of an optimal "sight range" of LAP would be an interesting future direction. As a sanity check, we compare the performance of large neural networks pruned via MP and LAP to the performance of a small network. In particular, we prune VGG-16, VGG-19, and ResNet-18 trained on CIFAR-10 dataset, to have a similar number of parameters to MobileNetV2 . For training and pruning VGGs and ResNet, we follows the prior setup in Appendix A while we use the same setup for training MobileNetV2 (Adam optimizer with learning rate of 3 · 10 −4 with batch size 60, and trained 60k steps). We observe that models pruned via LAP (and MP) exhibit better performance compared to MobileNetV2, even when pruned to have a smaller number of parameters. In this section, we briefly discuss where the benefits of the sub-network discovered by LAP comes from; does LAP subnetwork have a better generalizability or expressibility? For this purpose, we look into the generalization gap, i.e., the gap between the training and test accuracies, of the hypothesis learned via LAP procedure. Below we present a plot of test accuracies (Fig. 4a) and a plot of generalization gap (Fig. 4b) for FCN trained with MNIST dataset. The plot hints us that the network structure learned by LAP may not necessarily have a smaller generalizability. Remarkably, the generalization gap of the MP-pruned models and the LAP-pruned models are very similar to each other; the benefits of LAP subnetwork compared to MP would be that it can express a better-performing architecture with a network of similar sparsity and generalizability. remains constant for any hidden neuron j over training via gradient flow. In other words, the total outward flow of weights is tied to the inward flow of weights for each neuron. This observation hints at the possibility of a relative undergrowth of weight magnitude of an'important' connection, in the case where the connection shares the same input/output neuron with other'important' connections. From this viewpoint, the multiplicative factors in Eq. take into account the abstract notion of neuronal importance score, assigning significance to connections to the neuron through which more gradient signals have flowed through. Without considering such factors, LAP reduces to the ordinary magnitude-based pruning.

We study a multi-layer generalization of the magnitude-based pruning.

Represent the TLDR for finding the corresponding scientific article

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.