Question: Represent the scientific article to find a short TLDR

Question: Batch Normalization (BN) has become a cornerstone of deep learning across diverse architectures, appearing to help optimization as well as generalization. While the idea makes intuitive sense, theoretical analysis of its effectiveness has been lacking. Here theoretical support is provided for one of its conjectured properties, namely, the ability to allow gradient descent to succeed with less tuning of learning rates. It is shown that even if we fix the learning rate of scale-invariant parameters (e.g., weights of each layer with BN) to a constant (say, 0.3), gradient descent still approaches a stationary point (i.e., a solution where gradient is zero) in the rate of T^{−1/2} in T iterations, asymptotically matching the best bound for gradient descent with well-tuned learning rates. A similar with convergence rate T^{−1/4} is also shown for stochastic gradient descent. Batch Normalization (abbreviated as BatchNorm or BN) is one of the most important innovation in deep learning, widely used in modern neural network architectures such as ResNet BID8, Inception , and DenseNet . It also inspired a series of other normalization methods (; BID0 ;).BatchNorm consists of standardizing the output of each layer to have zero mean and unit variance. For a single neuron, if x 1,..., x B is the original outputs in a mini-batch, then it adds a BatchNorm layer which modifies the outputs to DISPLAYFORM0 where µ = i=1 (x i − µ) 2 are the mean and variance within the minibatch, and γ, β are two learnable parameters. BN appears to stabilize and speed up training, and improve generalization. The inventors suggested that these benefits derive from the following:1. By stabilizing layer outputs it reduces a phenomenon called Internal Covariate Shift, whereby the training of a higher layer is continuously undermined or undone by changes in the distribution of its inputs due to parameter changes in previous layers., 2. Making the weights invariant to scaling, appears to reduce the dependence of training on the scale of parameters and enables us to use a higher learning rate;3. By implictly regularizing the model it improves generalization. But these three benefits are not fully understood in theory. Understanding generalization for deep models remains an open problem (with or without BN). Furthermore, in demonstration that intuition can sometimes mislead, recent experimental suggest that BN does not reduce internal covariate shift either , and the authors of that study suggest that the true explanation for BN's effectiveness may lie in a smoothening effect (i.e., lowering of the Hessian norm) on the objective. Another recent paper tries to quantify the benefits of BN for simple machine learning problems such as regression but does not analyze deep models. Provable quantification of Effect 2 (learning rates). Our study consists of quantifying the effect of BN on learning rates. observed that without BatchNorm, a large learning rate leads to a rapid growth of the parameter scale. Introducing BatchNorm usually stabilizes the growth of weights and appears to implicitly tune the learning rate so that the effective learning rate adapts during the course of the algorithm. They explained this intuitively as follows. After BN the output of a neuron z = BN(w x) is unaffected when the weight w is scaled, i.e., for any scalar c > 0, BN(w x) = BN((cw) x).Taking derivatives one finds that the gradient at cw equals to the gradient at w multiplied by a factor 1/c. Thus, even though the scale of weight parameters of a linear layer proceeding a BatchNorm no longer means anything to the function represented by the neural network, their growth has an effect of reducing the learning rate. Our paper considers the following question: Can we rigorously capture the above intuitive behavior? Theoretical analyses of speed of gradient descent algorithms in nonconvex settings study the number of iterations required for convergence to a stationary point (i.e., where gradient vanishes). But they need to assume that the learning rate has been set (magically) to a small enough number determined by the smoothness constant of the loss function -which in practice are of course unknown. With this tuned learning rate, the norm of the gradient reduces asymptotically as T −1/2 in T iterations. In case of stochastic gradient descent, the reduction is like T −1/4. Thus a potential way to quantify the rate-tuning behavior of BN would be to show that even when the learning rate is fixed to a suitable constant, say 0.1, from the start, after introducing BN the convergence to stationary point is asymptotically just as fast (essentially) as it would be with a hand-tuned learning rate required by earlier analyses. The current paper rigorously establishes such auto-tuning behavior of BN (See below for an important clarification about scale-invariance).We note that a recent paper introduced a new algorithm WNgrad that is motivated by BN and provably has the above auto-tuning behavior as well. That paper did not establish such behavior for BN itself, but it was a clear inspiration for our analysis of BN.Scale-invariant and scale-variant parameters. The intuition of applies for all scale-invariant parameters, but the actual algorithm also involves other parameters such as γ and β whose scale does matter. Our analysis partitions the parameters in the neural networks into two groups W (scale-invariant) and g (scale-variant). The first group, W = {w,..., w (m) }, consists of all the parameters whose scales does not affect the loss, i.e., scaling w (i) to cw (i) for any c > 0 does not change the loss (see Definition 2.1 for a formal definition); the second group, g, consists of all other parameters that are not scale-invariant. In a feedforward neural network with BN added at each layer, the layer weights are all scale-invariant. This is also true for BN with p normalization strategies and other normalization layers, such as Weight Normalization , Layer Normalization BID0 ), Group Normalization (see Table 1 in BID0 for a summary). In this paper, we show that the scale-invariant parameters do not require rate tuning for lowering the training loss. To illustrate this, we consider the case in which we set learning rates separately for scale-invariant parameters W and scale-variant parameters g. Under some assumptions on the smoothness of the loss and the boundedness of the noise, we show that 1. In full-batch gradient descent, if the learning rate for g is set optimally, then no matter how the learning rates for W is set, (W ; g) converges to a first-order stationary point in the rate DISPLAYFORM0, which asymptotically matches with the convergence rate of gradient descent with optimal choice of learning rates for all parameters (Theorem 3.1); 2. In stochastic gradient descent, if the learning rate for g is set optimally, then no matter how the learning rate for W is set, (W ; g) converges to a first-order stationary point in the rate O(T −1/4 polylog(T)), which asymptotically matches with the convergence rate of gradient descent with optimal choice of learning rates for all parameters (up to a polylog(T) factor) (Theorem 4.2).In the usual case where we set a unified learning rate for all parameters, our imply that we only need to set a learning rate that is suitable for g. This means introducing scale-invariance into neural networks potentially reduces the efforts to tune learning rates, since there are less number of parameters we need to concern in order to guarantee an asymptotically fastest convergence. In our study, the loss function is assumed to be smooth. However, BN introduces non-smoothness in extreme cases due to division by zero when the input variance is zero (see equation 1). Note that the suggested implementation of BN by uses a smoothening constant in the whitening step, but it does not preserve scale-invariance. In order to avoid this issue, we describe a simple modification of the smoothening that maintains scale-invariance. Also, our cannot be applied to neural networks with ReLU, but it is applicable for its smooth approximation softplus BID5 ).We include some experiments in Appendix D, showing that it is indeed the auto-tuning behavior we analysed in this paper empowers BN to have such convergence with arbitrary learning rate for scale-invariant parameters. In the generalization aspect, a tuned learning rate is still needed for the best test accuracy, and we showed in the experiments that the auto-tuning behavior of BN also leads to a wider range of suitable learning rate for good generalization. Previous work for understanding Batch Normalization. Only a few recent works tried to theoretically understand BatchNorm. was described earlier. aims to find theoretical setting such that training neural networks with BatchNorm is faster than without BatchNorm. In particular, the authors analyzed three types of shallow neural networks, but rather than consider gradient descent, the authors designed task-specific training methods when discussing neural networks with BatchNorm. BID1 observes that the higher learning rates enabled by BatchNorm improves generalization. Convergence of adaptive algorithms. Our analysis is inspired by the proof for WNGrad , where the author analyzed an adaptive algorithm, WNGrad, motivated by Weight Normalization . Other works analyzing the convergence of adaptive methods are (; ; ;).Invariance by Batch Normalization. BID3 proposed to run riemmanian gradient descent on Grassmann manifold G(1, n) since the weight matrix is scaling invariant to the loss function. observed that the effective stepsize is proportional to ηw wt 2. In this section, we introduce our general framework in order to study the benefits of scale-invariance. Scale-invariance is common in neural networks with BatchNorm. We formally state the definition of scale-invariance below:Definition 2.1. (Scale-invariance) Let F(w, θ) be a loss function. We say that w is a scale-invariant parameter of F if for all c > 0, F(w, θ) = F(cw, θ); if w is not scale-invariant, then we say w is a scale-variant parameter of F.We consider the following L-layer "fully-batch-normalized" feedforward network Φ for illustration: DISPLAYFORM0 } is a mini-batch of B pairs of input data and ground-truth label from a data set D. f y is an objective function depending on the label, e.g., f y could be a cross-entropy loss in classification tasks. W,..., W (L) are weight matrices of each layer. σ: R → R is a nonlinear activation function which processes its input elementwise (such as ReLU, sigmoid). Given a batch of inputs z 1,..., z B ∈ R m, BN(z b) outputs a vectorz b defined as DISPLAYFORM1 where DISPLAYFORM2 are the mean and variance of z b, γ k and β k are two learnable parameters which rescale and offset the normalized outputs to retain the representation power. The neural network Φ is thus parameterized by weight matrices W (i) in each layer and learnable parameters γ k, β k in each BN.BN has the property that the output is unchanged when the batch inputs z 1,k,..., z B,k are scaled or shifted simultaneously. For z b,k = w kx b being the output of a linear layer, it is easy to see that w k is scale-invariant, and thus each row vector of weight matrices DISPLAYFORM3 In convolutional neural networks with BatchNorm, a similar argument can be done. In particular, each filter of convolutional layer normalized by BN is scale-invariant. With a general nonlinear activation, other parameters in Φ, the scale and shift parameters γ k and β k in each BN, are scale-variant. When ReLU or Leaky ReLU are used as the activation σ, the vector (γ 1, . . ., γ m, β 1, . . ., β m) of each BN at layer 1 ≤ i < L (except the last one) is indeed scale-invariant. This can be deduced by using the the (positive) homogeneity of these two types of activations and noticing that the output of internal activations is processed by a BN in the next layer. Nevertheless, we are not able to analyse either ReLU or Leaky ReLU activations because we need the loss to be smooth in our analysis. We can instead analyse smooth activations, such as sigmoid, tanh, softplus BID5, etc. Now we introduce our general framework. Let Φ be a neural network parameterized by θ. Let D be a dataset, where each data point z ∼ D is associated with a loss function F z (θ) (D can be the set of all possible mini-batches). We partition the parameters θ into (W ; g), where W = {w,..., w (m) } consisting of parameters that are scale-invariant to all F z, and g contains the remaining parameters. The goal of training the neural network is to minimize the expected loss over the dataset: DISPLAYFORM0 In order to illustrate the optimization benefits of scaleinvariance, we consider the process of training this neural network by stochastic gradient descent with separate learning rates for W and g: DISPLAYFORM1 Thanks to the scale-invariant properties, the scale of each weight w (i) does not affect loss values. However, the scale does affect the gradients. Let V = {v,..., v (m) } be the set of normalized weights, where DISPLAYFORM0 2. The following simple lemma can be easily shown: Lemma 2.2 (Implied by). For any W and g, DISPLAYFORM1 To make ∇ w (i) F z (W ; g) 2 to be small, one can just scale the weights by a large factor. Thus there are ways to reduce the norm of the gradient that do not reduce the loss. For this reason, we define the intrinsic optimization problem for training the neural network. Instead of optimizing W and g over all possible solutions, we focus on parameters θ in which w DISPLAYFORM2 This does not change our objective, since the scale of W does not affect the loss. 2 = 1 for all i} be the intrinsic domain. The intrinsic optimization problem is defined as optimizing the original problem in U: DISPLAYFORM0 For {θ t} being a sequence of points for optimizing the original optimization problem, we can define {θ t}, whereθ t = (V t ; g t), as a sequence of points optimizing the intrinsic optimization problem. In this paper, we aim to show that training neural network for the original optimization problem by gradient descent can be seen as training by adaptive methods for the intrinsic optimization problem, and it converges to a first-order stationary point in the intrinsic optimization problem with no need for tuning learning rates for W. We assume F z (W ; g) is defined and twice continuously differentiable at any θ satisfying none of DISPLAYFORM0 2, we assume that the following bounds on the smoothness: DISPLAYFORM1 In addition, we assume that the noise on the gradient of g in SGD is upper bounded by G g: DISPLAYFORM2 Smoothed version of motivating neural networks. Note that the neural network Φ illustrated in Section 2.1 does not meet the conditions of the smooothness at all since the loss function could be non-smooth. We can make some mild modifications to the motivating example to smoothen it 1:. The activation could be non-smooth. A possible solution is to use smooth nonlinearities, e.g., sigmoid, tanh, softplus BID5, etc. Note that softplus can be seen as a smooth approximation of the most commonly used activation ReLU.. The formula of BN shown in equation 3 may suffer from the problem of division by zero. To avoid this, the inventors of , add a small smoothening parameter > 0 to the denominator, i.e.,z DISPLAYFORM3 However, when z b,k = w kx b, adding a constant directly breaks the scale-invariance of w k. We can preserve the scale-invariance by making the smoothening term propositional to w k 2, i.e., replacing with w k 2. By simple linear algebra and letting DISPLAYFORM4, this smoothed version of BN can also be written as DISPLAYFORM5 Since the variance of inputs is usually large in practice, for small, the effect of the smoothening term is negligible except in extreme cases. Using the above two modifications, the loss function is already smooth. However, the scale of scale-variant parameters may be unbounded during training, which could cause the smoothness unbounded. To avoid this issue, we can either project scale-variant parameters to a bounded set, or use weight decay for those parameters (see Appendix C for a proof for the latter solution). The following lemma is our key observation. It establishes a connection between the scale-invariant property and the growth of weight scale, which further implies an automatic decay of learning rates:Lemma 2.4. For any scale-invariant weight w (i) in the network Ψ, we have: DISPLAYFORM0. w DISPLAYFORM1 Proof. Let θ t be all the parameters in θ t other than w (i)t. Taking derivatives with respect to c for the both sides of F zt (w DISPLAYFORM2 t, so the first proposition follows by taking c = 1. Applying Pythagorean theorem and Lemma 2.2, the second proposition directly follows. Using Lemma 2.4, we can show that performing gradient descent for the original problem is equivalent to performing an adaptive gradient method for the intrinsic optimization problem: DISPLAYFORM3 where Π is a projection operator which maps any vector w to w/ w 2 .Remark 2.6. Wu et al. FORMULA0 noticed that Theorem 2.5 is true for Weight Normalization by direct calculation of gradients. Inspiring by this, they proposed a new adaptive method called WNGrad. Our theorem is more general since it holds for any normalization methods as long as it induces scale-invariant properties to the network. The adaptive update rule derived in our theorem can be seen as WNGrad with projection to unit sphere after each step. Proof for Theorem 2.5. Using Lemma 2.2, we have DISPLAYFORM4 which implies the first equation. The second equation is by Lemma 2.4.While popular adaptive gradient methods such as AdaGrad BID4, RMSprop (Tieleman Assumptions on learning rates. We consider the case that we use fixed learning rates for both W and g, i.e., η w,0 = · · · = η w,T −1 = η w and η g,0 = · · · = η g,T −1 = η g . We assume that η g is tuned carefully to η g = (1 − c g)/L gg for some constant c g ∈. For η w, we do not make any assumption, i.e., η w can be set to any positive value. Theorem 3.1. Consider the process of training Φ by gradient descent with η g = 2(1 − c g)/L gg and arbitrary η w > 0. Then Φ converges to a stationary point in the rate of DISPLAYFORM0 where DISPLAYFORM1 This matches the asymptotic convergence rate of GD by BID2. The high level idea is to use the decrement of loss function to upper bound the sum of the squared norm of the gradients. Note that ∇L(DISPLAYFORM0 Thus the core of the proof is to show that the monotone increasing w DISPLAYFORM1 T 2 has an upper bound for all T . It is shown that for every w (i), the whole training process can be divided into at most two phases. In the first phase, the effective learning rate η w / w DISPLAYFORM2 2 is larger than some threshold 1 Ci (defined in Lemma 3.2) and in the second phase it is smaller. 2 is large enough and that the process enters the second phase, then by Lemma 3.2 in each step the loss function L will decrease by DISPLAYFORM0 by Lemma 2.4). Since L is lower-bounded, we can conclude w DISPLAYFORM1 2 is also bounded. For the second part, we can also show that by Lemma 3.2 DISPLAYFORM2 Thus we can concludeÕ(DISPLAYFORM3) convergence rate of ∇L(θ t) 2 as follows. DISPLAYFORM4 The full proof is postponed to Appendix A. In this section, we analyze the effect related to the scale-invariant properties when training a neural network by stochastic gradient descent. We use the framework introduced in Section 2.2 and assumptions from Section 2.4. Assumptions on learning rates. As usual, we assume that the learning rate for g is chosen carefully and the learning rate for W is chosen rather arbitrarily. More specifically, we consider the case that the learning rates are chosen as DISPLAYFORM0 We assume that the initial learning rate η g of g is tuned carefully to η g = (1 − c g)/L gg for some constant c g ∈. Note that this learning rate schedule matches the best known convergence rate O(T −1/4) of SGD in the case of smooth non-convex loss functions BID6.For the learning rates of W, we only assume that 0 ≤ α ≤ 1/2, i.e., the learning rate decays equally as or slower than the optimal SGD learning rate schedule. η w can be set to any positive value. Note that this includes the case that we set a fixed learning rate η w,0 = · · · = η w,T −1 = η w for W by taking α = 0. Remark 4.1. Note that the auto-tuning behavior induced by scale-invariances always decreases the learning rates. Thus, if we set α > 1/2, there is no hope to adjust the learning rate to the optimal strategy Θ(t −1/2). Indeed, in this case, the learning rate 1/G t in the intrinsic optimization process decays exactly in the rate ofΘ(t −α), which is the best possible learning rate can be achieved without increasing the original learning rate. Theorem 4.2. Consider the process of training Φ by gradient descent with η w,t = η w · (t + 1) DISPLAYFORM1 gg and η w > 0 is arbitrary. Then Φ converges to a stationary point in the rate of DISPLAYFORM2 where DISPLAYFORM3 and we see L gg = Ω.Note that this matches the asymptotic convergence rate of SGD, within a polylog(T) factor. We delay the full proof into Appendix B and give a proof sketch in a simplified setting where there is no g and α ∈ [0, 1 2). We also assume there's only one w i, that is, m = 1 and omit the index i. By Taylor expansion, we have DISPLAYFORM0 We can lower bound the effective learning rate η w,T w T 2 and upper bound the second order term respectively in the following way:. For all 0 ≤ α < 1 2, the effective learning rate DISPLAYFORM1. DISPLAYFORM2 Taking expectation over equation 14 and summing it up, we have DISPLAYFORM3 Plug the above bounds into the above inequality, we complete the proof. DISPLAYFORM4 In this paper, we studied how scale-invariance in neural networks with BN helps optimization, and showed that (stochastic) gradient descent can achieve the asymptotic best convergence rate without tuning learning rates for scale-invariant parameters. Our analysis suggests that scale-invariance in nerual networks introduced by BN reduces the efforts for tuning learning rate to fit the training data. However, our analysis only applies to smooth loss functions. In modern neural networks, ReLU or Leaky ReLU are often used, which makes the loss non-smooth. It would have more implications by showing similar in non-smooth settings. Also, we only considered gradient descent in this paper. It can be shown that if we perform (stochastic) gradient descent with momentum, the norm of scale-invariant parameters will also be monotone increasing. It would be interesting to use it to show similar convergence for more gradient methods. By the scale-invariant property of w (i), we know that L(W ; g) = L(V ; g). Also, the following identities about derivatives can be easily obtained: DISPLAYFORM0 Thus, the assumptions on the smoothness imply DISPLAYFORM1 Proof for Lemma 3.2. Using Taylor expansion, we have ∃γ ∈, such that for w DISPLAYFORM2 Note that w DISPLAYFORM3 t, we have DISPLAYFORM4 Thus, DISPLAYFORM5 By the inequality of arithmetic and geometric means, we have DISPLAYFORM6 Taking ∆w DISPLAYFORM7 We can complete the proof by replacing DISPLAYFORM8 Using the assumption on the smoothness, we can show that the gradient with respect to w (i) is essentially bounded: Lemma A.1. For any W and g, we have DISPLAYFORM9 Proof. A.1 Fix all the parameters except w (i). Then L(W ; g) can be written as a function f (w DISPLAYFORM10 Since f is continuous and S is compact, there must exist v DISPLAYFORM11 min is also a minimum in the entire domain and ∇f (w DISPLAYFORM12 For an arbitrary DISPLAYFORM13, and h goes along the geodesic from v (i) min to v (i) on the unit sphere S with constant speed. Let H(τ) = ∇f (h(τ)). By Taylor expansion, we have DISPLAYFORM14 where we use the fact that w DISPLAYFORM15 at the third line. We can bound S DISPLAYFORM16 Combining them together, we have DISPLAYFORM17 Taking sum over all i = 1,..., m and also subtracting G T on the both sides, we have DISPLAYFORM18 where DISPLAYFORM19 T + G T is used at the second line. Combining the lemmas above together, we can obtain our . Proof for Theorem 3.1. By Lemma A.2, we have DISPLAYFORM20 Thus min 0≤t<T ∇L(V t, g t) 2 converges in the rate of DISPLAYFORM21 Let F t = σ{z 0, . . ., z t−1} be the filtration, where σ{·} denotes the sigma field. We use L t:= L zt (θ t), F t:= F zt (θ t) for simplicity. As usual, we define v DISPLAYFORM0 i. Let k be the maximum i such that t i exists. Let t k+1 = T + 1. Then we know that DISPLAYFORM1 Thus, DISPLAYFORM2 Proof. Conditioned on F t, by Taylor expansion, we have DISPLAYFORM3 where Q t is DISPLAYFORM4 By the inequality DISPLAYFORM5 Taking this into equation 21 and summing up for all t, we have DISPLAYFORM6 and the right hand side can be expressed as DISPLAYFORM7 Proof for Theorem 4.2. Combining Lemma B.2 and Lemma B.4, for 0 ≤ α < 1/2, we have DISPLAYFORM8 2 +Õ(log T).Thus, DISPLAYFORM9 Similarly, for α = 1/2, we have DISPLAYFORM10 2 +Õ(log T).Thus, DISPLAYFORM11 In this section we prove that the modified version of the motivating neural network does meet the assumptions in Section 2.4. More specifically, we assume:• We use the network structure Φ in Section 2.1 with the smoothed variant of BN as described in Section 2.4;• The objective f y (·) is twice continuously differentiable, lower bounded by f min and Lipschitz (|f y (ŷ)| ≤ α f );• The activation σ(·) is twice continuously differentiable and Lipschitz (|f y (ŷ)| ≤ α σ );• We add an extra weight decay (L2 regularization) term First, we show that g t (containing all scale and shift parameters in BN) is bounded during the training process. Then the smoothness follows compactness using Extreme Value Theorem. We use the following lemma to calculate back propagation: DISPLAYFORM0 Proof. Letx ∈ R B be the vector wherex b:= (w (x b −u))/ w S+ I. It is easy to see x 2 2 ≤ B. Then DISPLAYFORM1 For x b, we have DISPLAYFORM2 Thus, DISPLAYFORM3 Lemma C.2. If g 0 2 is bounded by a constant, there exists some constant K such that g t 2 ≤ K.Proof. Fix a time t in the training process. Consider the process of back propagation. Define DISPLAYFORM4 b,k is the output of the k-th neuron in the i-th layer in the b-th data sample in the batch. By the Lipschitzness of the objective, R L can be bounded by a constant. If R i can be bounded by a constant, then by the Lipschitzness of σ and Lemma C.1, the gradient of γ and β in layer i can also be bounded by a constant. Note that DISPLAYFORM5 Thus γ and β in layer i can be bounded by a constant since DISPLAYFORM6 Also Lemma C.1 and the Lipschitzness of σ imply that R i−1 can be bounded if R i and γ in the layer i can be bounded by a constant. Using a simple induction, we can prove the existence of K for bounding the norm of g t 2 for all time t. Theorem C.3. If g 0 2 is bounded by a constant, then Φ satisfies the assumptions in Section 2.4.Proof. Let C be the set of parameters θ satisfying g ≤ K and w (i) 2 = 1 for all 1 ≤ i ≤ m. By Lemma C.2, C contains the set ofθ associated with the points lying between each pair of θ t and θ t+1 (including the endpoints).It is easy to show that F z (θ) is twice continously differentiable. Since C is compact, by the Extreme Value Theorem, there must exist such constants L In this section, we provide experimental evidence showing that the auto rate-tuning behavior does empower BN in the optimization aspect. We trained a modified version of VGGNet on Tensorflow. This network has 2 × conv64, pooling, 3 × conv128, pooling, 3 × conv256, pooling, 3 × conv512, pooling, 3 × conv512, pooling, fc512, fc10 layers in order. Each convolutional layer has kernel size 3 × 3 and stride 1. ReLU is used as the activation function after each convolutional or fullyconnected layer. We add a BN layer right before each ReLU. We set = 0 in each BN, since we observed that the network works equally well for being 0 or an small number (such as 10 −3, the default value in Tensorflow). We initialize the parameters according to the default configuration in Tensorflow: all the weights are initialized by Glorot uniform initializer BID7; β and γ in BN are initialized by 0 and 1, respectively. In this network, every kernel is scale-invariant, and for every BN layer except the last one, the concatenation of all β and γ parameters in this BN is also scale-invariant. Only β and γ parameters in the last BN are scale-variant (See Section 2.1). We consider the training in following two settings:1. Train the network using the standard SGD (No momentum, learning rate decay, weight decay and dropout);2. Train the network using Projected SGD (PSGD): at each iteration, one first takes a step proportional to the negative of the gradient calculated in a random batch, and then projects each scale-invariant parameter to the sphere with radius equal to its 2-norm before this iteration, i.e., rescales each scale-invariant parameter so that each maintains its length during training. Note that the projection in Setting 2 removes the adaptivity of the learning rates in the corresponding intrinsic optimization problem, i.e., Gt in equation 9 remains constant during the training. Thus, by comparing Setting 1 and Setting 2, we can know whether or not the auto-tuning behavior of BN shown in theory is effective in practice. The relationship between the training loss and the learning rate. For learning rate larger than 10, the training loss of PSGD or SGD with BN removed is always either very large or NaN, and thus not invisible in the figure. Left: The average training loss of the last 5 epochs (averaged across 10 experiments). In rare cases, the training loss becomes NaN in the experiments for the green curve (SGD, BN removed) with learning rate larger than 10 −0.7. We removed such data when taking the average. Right: The average training loss of each epoch (each curve stands for a single experiment). The relationship between the test accuracy and the learning rate. Left: The average test accuracy of the last 5 epochs (averaged across 10 experiments). Right: The test accuracy after each epoch (each curve stands for a single experiment). Due to the implementation of Tensorflow, outputing NaN leads to a test accuracy of 10%. Note that the magenta dotted curve (PSGD, lr=100), red dashed curve (SGD, BN removed, lr=1) and cyan dashed curve (SGD, BN removed, lr=10) are covered by the magenta dashed curve (SGD, BN removed, lr=100). They all have 10% test accuracy. As in our theoretical analysis, we consider what will happen if we set two learning rates separately for scale-invariant and scale-variant parameters. We train the network in either setting with different learning rates ranging from 10 −2 to 10 2 for 100 epochs. First, we fix the learning rate for scale-variant ones to 0.1, and try different learning rates for scaleinvariant ones. As shown in FIG3, for small learning rates (such as 0.1), the training processes of networks in Setting 1 and 2 are very similar. But for larger learning rates, networks in Setting 1 can still converge to 0 for all the learning rates we tried, while networks in Setting 2 got stuck with relatively large training loss. This suggests that the auto-tuning behavior of BN does takes effect when the learning rate is large, and it matches with the claimed effect of BN in that BN enables us to use a higher learning rate. Though our theoretical analysis cannot be directly applied to the network we trained due to the non-smoothness of the loss function, the experiment match with what we expect in our analysis. Next, we consider the case in which we train the network with a unified learning rate for both scaleinvariant and scale-variant parameters. We also compare Setting 1 and 2 with the setting in which we train the network with all the BN layers removed using SGD (we call it Setting 3).As shown in FIG4, the training loss of networks in Setting 1 converges to 0. On the contrast, the training loss of networks in Setting 2 and 3 fails to converge to 0 when a large learning rate is used, and in some cases the loss diverges to infinity or NaN. This suggests that the auto-tuning behavior of BN has an effective role in the case that a unified learning rate is set for all parameters. For a fair comparison, we also trained neural networks in Setting 3 with initialization essentially equivalent to the ones with BN. This is done in the same way as (Krähenbühl et al., 2015;) and Section 3 of : we first randomly initialize the parameters, then feed the first batch into the network and adjust the scaling and bias of each neuron to make its outputs have zero mean and unit variance. In this way, the loss of the networks converges to 0 when the learning rate is smaller than 10 −2.0, but for a slightly larger learning rate such as 10 −1.8, the loss fails to converge to 0, and sometimes even diverges to infinity or NaN. Compared with experimental in Setting 1, this suggests that the robustness of training brought by BN is independent of the fact that BN changes the effective initialization of parameters. Despite in Setting 1 the convergence of training loss for different learning rates, the convergence points can be different, which lead to different performances on test data. In FIG5, we plot the test accuracy of networks trained in Setting 1 and 2 using different unified learning rates, or separate learning rates with the learning rate for scale-variant parameters fixed to 0.1. As shown in the FIG5, the test accuracy of networks in Setting 2 decreases as the learning rate increases over 0.1, while the test accuracy of networks in Setting 1 remains higher than 75%. The main reason that the network in Setting 2 doesn't perform well is underfitting, i.e. the network in Setting 2 fails to fit the training data well when learning rate is large. This suggests that the autotuning behavior of BN also benefits generalization since such behavior allows the algorithm to pick learning rates from a wider range while still converging to small test error.

Represent the TLDR for finding the corresponding scientific article

We give a theoretical analysis of the ability of batch normalization to automatically tune learning rates, in the context of finding stationary points for a deep learning objective.

Represent the short summary for finding the corresponding scientific article (long)

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.


Question: Represent the text to find a very short summary (1-2 sentences max):

Question: Generative models of natural images have progressed towards high fidelity samples by the strong leveraging of scale. We attempt to carry this success to the field of video modeling by showing that large Generative Adversarial Networks trained on the complex Kinetics-600 dataset are able to produce video samples of substantially higher complexity and fidelity than previous work. Our proposed model, Dual Video Discriminator GAN (DVD-GAN), scales to longer and higher resolution videos by leveraging a computationally efficient decomposition of its discriminator. We evaluate on the related tasks of video synthesis and video prediction, and achieve new state-of-the-art Fréchet Inception Distance for prediction for Kinetics-600, as well as state-of-the-art Inception Score for synthesis on the UCF-101 dataset, alongside establishing a strong baseline for synthesis on Kinetics-600. Figure 1: Selected frames from videos generated by a DVD-GAN trained on Kinetics-600 at 256×256, 128 × 128, and 64 × 64 resolutions (top to bottom). Modern deep generative models can produce realistic natural images when trained on high-resolution and diverse datasets (; ; ; ;). Generation of natural video is an obvious further challenge for generative modeling, but one that is plagued by increased data complexity and computational requirements. For this reason, much prior work on video generation has revolved around relatively simple datasets, or tasks where strong temporal conditioning information is available. We focus on the tasks of video synthesis and video prediction (defined in Section 2.1), and aim to extend the strong of generative image models to the video domain. Building upon the state-of-the-art BigGAN architecture , we introduce an efficient spatio-temporal decomposition of the discriminator which allows us to train on Kinetics-600 -a complex dataset of Figure 2: Generated video samples with interesting behavior. In raster-scan order: a) On-screen generated text with further lines appearing.. b) Zooming in on an object. c) Colored detail from a pen being left on paper. d) A generated camera change and return. natural videos an order of magnitude larger than other commonly used datasets. The ing model, Dual Video Discriminator GAN (DVD-GAN), is able to generate temporally coherent, high-resolution videos of relatively high fidelity (Figure 1). Our contributions are as follows: • We propose DVD-GAN -a scalable generative model of natural video which produces high-quality samples at resolutions up to 256 × 256 and lengths up to 48 frames. • We achieve state of the art for video synthesis on UCF-101 and prediction on Kinetics-600. • We establish class-conditional video synthesis on Kinetics-600 as a new benchmark for generative video modeling, and report DVD-GAN as a strong baseline. The exact formulation of the video generation task can differ in the type of conditioning signal provided. At one extreme lies unconditional video synthesis where the task is to generate any video following the training distribution. Another extreme is occupied by strongly-conditioned models, including generation conditioned on another video for content transfer , per-frame segmentation masks (a), or pose information (; b;). In the middle ground there are tasks which are more structured than unconditional generation, and yet are more challenging from a modeling perspective than strongly-conditional generation (which gets a lot of information about the generated video through its input). The objective of class-conditional video synthesis is to generate a video of a given category (e.g., "riding a bike") while future video prediction is concerned with generation of continuing video given initial frames. These problems differ in several aspects, but share a common requirement of needing to generate realistic temporal dynamics, and in this work we focus on these two problems. Generative Adversarial Networks (GANs) are a class of generative models defined by a minimax game between a Discriminator D and a Generator G. The original objective was proposed by , and many improvements have since been suggested, mostly targeting improved training stability (; ; ; ;). We use the hinge formulation of the objective which is optimized by gradient descent (ρ is the elementwise ReLU function): GANs have well-known limitations including a tendency towards limited diversity in generated samples (a phenomenon known as mode collapse) and the difficulty of quantitative evaluation due to the lack of an explicit likelihood measure over the data. Despite these downsides, GANs have produced some of the highest fidelity samples across many visual domains . Kinetics is a large dataset of 10-second high-resolution YouTube clips originally created for the task of human action recognition. We use the second iteration of the dataset, Kinetics-600 , which consists of 600 classes with at least 600 videos per class for a total of around 500,000 videos. 1 Kinetics videos are diverse and unconstrained, which allows us to train large models without being concerned with the overfitting that occurs on small datasets with fixed objects interacting in specified ways . Among prior work, the closest dataset (in terms of subject and complexity) which is consistently used is UCF-101 . We focus on Kinetics-600 because of its larger size (almost 50x more videos than UCF-101) and its increased diversity (600 instead of 101 classes -not to mention increased intra-class diversity). Nevertheless for comparison with prior art we train on UCF-101 and achieve a state-of-the-art Inception Score there. Kinetics contains many artifacts expected from YouTube, including cuts (as in Figure 2d), title screens and visual effects. Except when specifically described, we choose frames with stride 2 (meaning we skip every other frame). This allows us to generate videos with more complexity without incurring higher computational cost. To the best of our knowledge we are the first to consider generative modelling of the entirety of the Kinetics video dataset 2, although a small subset of Kinetics consisting of 4,000 selected and stabilized videos (via a SIFT + RANSAC procedure) has been used in at least two prior papers . Due to the heavy pre-processing and stabilization present, as well as the sizable reduction in dataset size (two orders of magnitude) we do not consider these datasets comparable to the full Kinetics-600 dataset. Designing metrics for measuring the quality of generative models (GANs in particular) is an active area of research . In this work we report the two most commonly used metrics, Inception Score (IS) and Fréchet Inception Distance (FID) . The standard instantiation of these metrics is intended for generative image models, and uses an Inception model for image classification or feature extraction. For videos, we use the publicly available Inflated 3D Convnet (I3D) network trained on Kinetics-600 . Our Fréchet Inception Distance is therefore very similar to the Fréchet Video Distance (FVD) , although our implementation is different and more aligned with the original FID metric. 3 More details are in Appendix A.4. Our primary contribution is Dual Video Discriminator GAN (DVD-GAN), a generative video model of complex human actions built upon the state-of-the-art BigGAN architecture while introducing scalable, video-specific generator and discriminator architectures. An overview of the DVD-GAN architecture is given in Figure 3 and a detailed description is in Appendix A.2. Unlike some of the prior work, our generator contains no explicit priors for foreground, or motion (optical flow); instead, we rely on a high-capacity neural network to learn this in a data-driven manner. While DVD-GAN contains sequential components (RNNs), it is not autoregressive in time or in space. In other words, the pixels of each frame do not directly depend on other pixels in the video, as would be the case for auto-regressive models or models generating one frame at a time. Generating long and high resolution videos is a heavy computational challenge: individual samples from Kinetics-600 (just 10 seconds long) contain upwards of 16 million pixels which need to be generated in a consistent fashion. This is a particular challenge to the discriminator. For example, a generated video might contain an object which leaves the field of view and incorrectly returns with a different color. Here, the ability to determine this video is generated is only possible by comparing two different spatial locations across two (potentially distant) frames. Given a video with length T, height H, and width W, discriminators that process the entire video would have to process all H × W × T pixels -limiting the size of the model and the size of the videos being generated. DVD-GAN tackles this scale problem by using two discriminators: a Spatial Discriminator D S and a Temporal Discriminator D T. D S critiques single frame content and structure by randomly sampling k full-resolution frames and judging them individually. We use k = 8 and discuss this choice in Section 4.3. D S's final score is the sum of the per-frame scores. The temporal discriminator D T must provide G with the learning signal to generate movement (something not evaluated by D S). To make the model scalable, we apply a spatial downsampling function φ(·) to the whole video and feed its output to D T. We choose φ to be 2 × 2 average pooling, and discuss alternatives in Section 4.3. This in an architecture where the discriminators do not process the entire video's worth of pixels, since D S processes only k × H × W pixels and For a 48 frame video at 128 × 128 resolution, this reduces the number of pixels to process per video from 786432 to 327680: a 58% reduction. Despite this decomposition, the discriminator objective is still able to penalize almost all inconsistencies which would be penalized by a discriminator judging the entire video. D T judges any temporal discrepancies across the entire length of the video, and D S can judge any high resolution details. The only detail the DVD-GAN discriminator objective is unable to reflect is the temporal evolution of pixels within a 2 × 2 window. We have however not noticed this affecting the generated samples in practice. DVD-GAN's D S is similar to the per-frame discriminator D I in MoCoGAN . However MoCoGAN's analog of D T looks at full resolution videos, whereas D S is the only source of learning signal for high-resolution details in DVD-GAN. For this reason, D S is essential when φ is not the identity, unlike in MoCoGAN where the additional per-frame discriminator is less crucial. Generative video modeling is a widely explored problem which includes work on VAEs (; ; ;) and recurrent models (b; ; c;), auto-regressive models (; ; ;), normalizing flows , and GANs (; ; ;). Much prior work considers decompositions which model the texture and spatial consistency of objects separately from their temporal dynamics. One approach is to split G into foreground and models , while another considers explicit or implicit optical flow or motion in either G or D . Other methods decompose the generator (or encoder) to treat concepts like pose, content and motion separately from one another (; a). Similar to DVD-GAN, MoCoGAN discriminates individual frames in addition to a discriminator which operates on fixed-length K-frame slices of the whole video (where K < T). Though this potentially reduces the number of pixels to discriminate to describes discriminating sliding windows, which increases the total number of pixels. Other models follow this approach by discriminating groups of frames (; ;). TGANv2 proposes "adaptive batch reduction" for efficient training, an operation which randomly samples subsets of videos within a batch and temporal subwindows within each video. This operation is applied throughout TGANv2's G, with heads projecting intermediate feature maps directly to pixel space before applying batch reduction, and corresponding discriminators evaluating these lower resolution intermediate outputs. An effect of this choice is that TGANv2 discriminators only evaluate full-length videos at very low resolution. We show in Figure 6 that a similar reduction in DVD-GAN's resolution when judging full videos leads to a loss in performance. We expect further reduction (towards the resolution at which TGANv2 evaluates the entire length of video) to lead to further degradation of DVD-GAN's quality. Furthermore, this method is not easily adapted towards models with large batch sizes divided across a number of accelerators, with only a small batch size per replica. A detailed description of our training setup is in Appendix A.3. Each DVD-GAN was trained on TPU pods using between 32 and 512 replicas with an Adam optimizer. Video Synthesis models are trained for around 300,000 learning steps, whilst Video Prediction models are trained for up to 1,000,000 steps. Most models took between 12 and 96 hours to train. Our primary concern the problem of Video Synthesis. We provide our for the UCF-101 and Kinetics-600 datasets. With Kinetics-600 emerging as a new benchmark for generative video modelling, our establish a strong baseline for future work. In Table 1 we show the main of this paper: benchmarks for Class-Conditional Video Synthesis on Kinetics-600. In this regime, we train a single DVD-GAN on all classes of Kinetics-600, supplying per-sample class information to both G and D. We consider a range of resolutions and video lengths, and measure Inception Score and Fréchet Inception Distance (FID) for each (as described in Section 2.4). We further measure each model along a truncation curve, which we carry out by calculating FID and IS statistics while varying the standard deviation of the latent vectors between 0 and 1. There is no prior work with which to quantitatively compare these (for comparative experiments see Section 4.1.2 and Section 4.2.1), but we believe these samples to show a level of fidelity not yet achieved in datasets as complex as Kinetics-600 (see samples from each row in Appendix D.1). Because all videos are resized for the I3D network (to 224 × 224), it is meaningful to compare metrics across equal length videos at different resolutions. Neither IS nor FID are comparable across videos of different lengths, and should be treated as separate metrics. Generating longer and larger videos is a more challenging modeling problem, which is conveyed by the metrics (in particular, comparing 12-frame videos across 64 × 64, 128 × 128 and 256 × 256 resolutions). Nevertheless, DVD-GAN is able to generate plausible videos at all resolutions and with length spanning up to 4 seconds (48 frames). As can be seen in Appendix D.1, smaller videos display high quality textures, object composition and movement. At higher resolutions, generating coherent objects becomes more difficult (movement consists of a much larger number of pixels), but high-level details of the generated scenes are still extremely coherent, and textures (even complicated ones like a forest backdrop in Figure 1a) are generated well. It is further worth noting that the 48-frame models do not see more high resolution frames than the 12-frame model (due to the fixed choice of k = 8 described in Section 3.1), yet nevertheless learn to generate high resolution images. We further verify our by testing the same model on UCF-101 , a smaller dataset of 13,320 videos of human actions across 101 classes that has previously been used for video synthesis and prediction (; ;). Our model produces samples with an IS of 27.38, significantly outperforming the state of the art (see Table 2 for quantitative comparison and Appendix B.1 for more details). 11.85 ±.07 MoCoGAN 12.42 ±.03 ProgressiveVGAN 14.56 ±.05 TGANv2 24.34 ±.35 DVD- GAN (ours) 27.38 ± 0.53 Future Video Prediction is the problem of generating a sequence of frames which directly follow from one (or a number) of initial conditioning frames. Both this and video synthesis require G to learn to produce realistic scenes and temporal dynamics, however video prediction further requires G to analyze the conditioning frames and discover elements in the scene which will evolve over time. In this section, we use the Fréchet Video Distance exactly as: using the logits of an I3D network trained on Kinetics-400 as features. This allows for direct comparison to prior work. Our model, DVD-GAN-FP (Frame Prediction), is slightly modified to facilitate the changed problem, and details of these changes are given in Appendix A.5. For direct comparison with concurrent work on autoregressive video models we consider the generation of 11 frames of Kinetics-600 at 64 × 64 resolution conditioned on 5 frames, where the videos for training are not taken with any frame skipping. We show for all these cases in Table 4. Our frame-conditional model DVD-GAN-FP outperforms the prior work on frame-conditional prediction for Kinetics. The final row labeled DVD-GAN corresponds to 16-frame class-conditional Video Synthesis samples, generated without frame conditioning and without frame skipping. The FVD of this video synthesis model is notably better. On the one hand, we hypothesize that the synthesis model has an easier generative task: it can choose to generate (relatively) simple samples for each class, rather than be forced to continue frames taken from videos which are class outliers, or contain more complicated details. On the other hand, a certain portion of the FID/FVD metric undoubtedly comes from the distribution of objects and s present in the dataset, and so it seems that the prediction model should have a handicap in the metric by being given the ground truth distribution of s and objects with which to continue videos. The synthesis model's improved performance on this task seems to indicate that the advantage of being able to select videos to generate is greater than the advantage of having a ground truth distribution of starting frames. This is un-intuitive, as the frame conditional model has access to strictly more information about the data distribution it is trying to recover compared to the synthesis model (despite the fact that the two models are being trained by an identical objective). This experiment favors the synthesis model for FVD, but we highlight that other models or other metrics might produce the opposite ordering. We analyze several choices for k (the number of frames per sample in the input to D S) and φ (the downsampling function for D T). We expect setting φ to the identity or k = T to in the best model, but we are interested in the maximally compressive k and φ that reduce discriminator input size (and the amount of computation), while still producing a high quality generator. For φ, we consider: 2 × 2 and 4 × 4 average pooling, the identity (no downsampling), as well as a φ which takes a random half-sized crop of the input video (as in). Results can be seen in Figure 6. For each ablation, we train three identical DVD-GANs with different random initializations on 12-frame clips of Kinetics-600 at 64 × 64 resolution for 100,000 steps. We report mean and standard deviation (via the error bars) across each group for the whole training period. For k, we consider 1, 2, 8 and 10 frames. We see diminishing effect as k increases, so settle on k = 8. We note the substantially reduced IS of 4 × 4 downsampling as opposed to 2 × 2, and further note that taking half-sized crops (which in the same number of pixels input to D T as 2 × 2 pooling) is also notably worse. We approached the challenging problem of modeling natural video by introducing a GAN capable of capturing the complexity of a large video dataset. We showed that on UCF-101 and frame-conditional Kinetics-600 it quantitatively achieves the new state of the art, alongside qualitatively producing video synthesis samples with high complexity and diversity. We further wish to emphasize the benefit of training generative models on large and complex video datasets, such as Kinetics-600, and envisage the strong baselines we established on this dataset with DVD-GAN will be used as a reference point by the generative modeling community moving forward. While much remains to be done before realistic videos can be consistently generated in an unconstrained setting, we believe DVD-GAN is a step in that direction. A EXPERIMENT METHODOLOGY For all datasets we randomly shuffle the training set for each model replica independently. Experiments on the BAIR Robot Pushing dataset are conducted in the native resolution of 64 × 64, where for UCF-101 we operate at a (downsampled) 128 × 128 resolution. This is done by a bilinear resize such that the video's smallest dimension is mapped to 128 pixels while maintaining aspect ratio (144 for UCF-101). From this we take a random 128-pixel crop along the other dimension. We use the same procedure to construct datasets of different resolutions for Kinetics-600. All three datasets contain videos with more frames than we generate, so we take a random sequence of consecutive frames from the resized output. For UCF-101, we augmented the dataset by randomly performing left-right flips with probability 0.5. Our model adopts many architectural choices from including our nomenclature for describing network width, which is determined by the product of a channel multiplier ch with a constant for each layer in the network. The layer-wise constants for G are for 64 × 64 videos and for 128 × 128. The width of the i-th layer is given by the product of ch and the i-th constant and all layers prior to the residual network in G use the initial layer's multiplier and we refer to the product of that and ch as ch 0. ch in DVD-GAN is 128 for videos with 64 × 64 resolution and 96 otherwise. The corresponding ch lists for both D T and D S are for 64 × 64 resolution and for 128 × 128. The input to G consists of a Gaussian latent noise z ∼ N (0, I) and a learned linear embedding e(y) of the desired class y. Both inputs are 120-dimensional vectors. G starts by computing an affine transformation of [z; e(y)] to a [4, 4, ch 0]-shaped tensor (in Figure 3 this is represented as a 1 × 1 convolution). [z; e(y)] is used as the input to all class-conditional Batch Normalization layers throughout G (the gray line in Figure 7). This is then treated as the input (at each frame we would like to generate) to a Convolutional Gated Recurrent Unit whose update rule for input x t and previous output h t−1 is given by the following: In these equations σ and ρ are the elementwise sigmoid and ReLU functions respectively, the n operator represents a convolution with a kernel of size n × n, and the operator is an elementwise multiplication. Brackets are used to represent a feature concatenation. This RNN is unrolled once per frame. The output of this RNN is processed by two residual blocks (whose architecture is given by Figure 7). The time dimension is combined with the batch dimension here, so each frame proceeds through the blocks independently. The output of these blocks has width and height dimensions which are doubled (we skip upsampling in the first block). This is repeated a number of times, with the output of one RNN + residual group fed as the input to the next group, until the output tensors have the desired spatial dimensions. We do not reduce over the time dimension when calculating Batch Normalization statistics. This prevents the network from utilizing the Batch Normalization layers to pass information between timesteps. The spatial discriminator D S functions almost identically to BigGAN's discriminator, though an overview of the residual blocks is given in Figure 7 for completeness. A score is calculated for each of the uniformly sampled k frames (we default to k = 8) and the D S output is the sum over per-frame scores. The temporal discriminator D T has a similar architecture, but pre-processes the real or generated video with a 2 × 2 average-pooling downsampling function φ. Furthermore, the first two residual blocks of D T are 3-D, where every convolution is replaced with a 3-D convolution with a kernel size of 3 × 3 × 3. The rest of the architecture follows BigGAN . Sampling from DVD-GAN is very efficient, as the core of the generator architecture is a feed-forward convolutional network: two 64 × 64 48-frame videos can be sampled in less than 150ms on a single TPU core. The dual discriminator D is updated twice for every update of G and we use Spectral Normalization for all weight layers (approximated by the first singular value) and orthogonal initialization of weights . Sampling is carried out using the exponential moving average of G's weights, which is accumulated with decay γ = 0.9999 starting after 20,000 training steps. The model is optimized using Adam with batch size 512 and a learning rate of 1 · 10 −4 and 5 · 10 −4 for G and D respectively. Class conditioning in D is projection-based whereas G relies on class-conditional Batch Normalization (; ;): equivalent to standard Batch Normalization without a learned scale and offset, followed by an elementwise affine transformation where each parameter is a function of the noise vector and class conditioning. The FID we use for Synthesis on Kinetics-600 is calculated exactly as Fréchet Video Distance except that we use a different feature network: an I3D trained on Kinetics-600 (as opposed to the network trained on Kinetics-400 in FVD) and features from the final hidden layer instead of the logits. This metric can be implemented as a small change from the publically available FVD code by changing the name of the TF-Hub module to'https://tfhub.dev/deepmind/i3d-kinetics-600/1' and loading the tensor named'RGB/inception_i3d/Logits/AvgPool3D' from the ing graph. In order to provide on future video prediction problems we describe a simple modification to DVD-GAN to facilitate the added conditioning. A diagram of the extended model is in Figure 8. Each row is a separate video, the leftmost column is a (true) conditioning frame. Given C conditioning frames, our modified DVD-GAN-FP passes each frame separately through a deep residual network identical to D S. The (near) symmetric design of G and D S's residual blocks mean that each output from a D-style residual block has a corresponding intermediate tensor in G of the same spatial resolution. After each block the ing features for each conditioning frame are stacked in the channel dimension and passed through a 3 × 3 convolution and ReLU activation. The ing tensor is used as the initial state for the Convolutional GRU in the corresponding block in G. Note that the frame conditioning stack reduces spatial resolution while G increases resolution. Therefore the smallest features of the conditioning frames (which have been through the most layers) are input earliest in G and the larger features (which have been through less processing) are input to G towards the end. D T operates on the concatenation of the conditioning frames and the output of G, meaning that it does not receive any extra information detailing that the first C frames are special. However to reduce wasted computation we do not sample the first C frames for D S on real or generated data. This technically means that D S will never see the first few frames from real videos at full resolution, but this was not an issue in our experiments. Finally, our video prediction variant does not condition on any class information, allowing us to directly compare with prior art. This is achieved by settling the class id of all samples to 0. 4 This evaluation is performed by re-scaling the video to 128 × 128, normalizing the input features based on mean statistics of the ground truth dataset, then taking a 112 × 112 center crop and applying C3D. Our model produces samples with an IS of 27.38, significantly outperforming the state of the art (see Table 2). The DVD-GAN architecture on UCF-101 is identical to the model used for Kinetics, and is trained on 16-frame 128 × 128 clips from UCF-101. The lack of class information does hurt the performance of DVD-GAN, and training on UCF-101 with class labels leads to an improved model with an Inception Score of 32.97. This is directly comparable to which achieved an IS of 15.83 and is close to the IS reported for the ground truth data (34.49). However we note than many more recent video generation papers do not test in this regime. It is worth mentioning that our improved score is, at least partially, due to memorization of the training data. In Figure 10 we show interpolation samples from our best UCF-101 model. Like interpolations in Appendix D.2, we sample 2 latents (left and rightmost columns) and show samples from the linear interpolation in latent space along each row. Here we show 4 such interpolations (the first frame from each video). Unlike Kinetics-600 interpolations, which smoothly transition from one sample to the other, we see abrupt jumps in the latent space between highly distinct samples, and little intra-video diversity between samples in each group. It can be further seen that some generated samples highly correlate with samples from the training set. We show this both as a failure of the Inception Score metric, the commonly reported value for classconditional video synthesis on UCF-101, but also as strong signal that UCF-101 is not a complex or diverse enough dataset to facilitate interesting video generation. Each class is relatively small, and reuse of clips from shared underlying videos means that the intra-class diversity can be restricted to just a handful of videos per class. This suggests the need for larger, more diverse and challenging datasets for generative video modelling, and we believe that Kinetics-600 provides a better benchmark for this task. Here we detail a number of modifications or miscellaneous we experimented with which did not produce a conclusive . • We experimented with several variations of normalization which do not require calculating statistics over a batch of data. Group Normalization performed best, almost on a par with (but worse than) Batch Normalization. We further tried Layer Normalization , Instance Normalization (, and no normalization, but found that these significantly underperformed Batch Normalization. • We found that removing the final Batch Normalization in G, which occurs after the ResNet and before the final convolution, caused a catastrophic failure in learning. Interestingly, just removing the Batch Normalization layers within G's residual blocks still led to good (though slightly worse) generative models. In particular, variants without Batch Normalization in the residual blocks often achieve significantly higher IS (up to 110.05 for 64 × 64 12 frame samples -twice normal). But these models had substantially worse FID scores (1.22 for the aforementioned model) -and produced qualitatively worse video samples. • Early variants of DVD-GAN contained Batch Normalization which normalized over all frames of all batch elements. This gave G an extra channel to convey information across time. It took advantage of this, with the being a model which required batch statistics in order to produce good samples. We found that the version which normalizes over timesteps independently worked just as well and without the dependence on statistics. • Models based on the residual blocks of BigGAN-deep trained faster (in wall clock time) but slower with regards to metrics, and struggled to reach the accuracy of models based on BigGAN's residual blocks. It is difficult to accurately convey complicated generated video through still frames. Where provided, we recommend readers view the generated videos themselves via the provided links. We refer to videos within these batches by row/column number where the video in the 0th row and column is in the top left corner. We expect G to produce samples of higher quality from latents near the mean of the distribution (zero). This is the idea behind the Truncation Trick . Like BigGAN, we find that DVD-GAN is amenable to truncation. We also experiment with interpolations in the latent space and in the class embedding. In both cases, interpolations are evidence that G has learned a relatively smooth mapping from the latent space to real videos: this would be impossible for a network that has only memorized the training data, or which is only capable of generating a few exemplars per class. Note that while all latent vectors along an interpolation are valid (and therefore G should produce a reasonable sample), at no point during training is G asked to generate a sample halfway between two classes. Nevertheless G is able to interpolate between even very distinct classes. Figure 17: An example intra-class interpolation. Each column is a separate video (the vertical axis is the time dimension). The left and rightmost columns are randomly sampled latent vectors and are generated under a shared class. Columns in between represent videos generated under the same class across the linear interpolation between the two random samples. Note the smooth transition between videos at all six timesteps displayed here. Figure 18: An example of class interpolation. As before, each column is a sequence of timesteps of a single video. Here, we sample a single latent vector, and the left and rightmost columns represent generating a video of that latent under two different classes. Columns in between represent videos of that same latent generated across an interpolation of the class embedding. Even though at no point has DVD-GAN been trained on data under an interpolated class, it nevertheless produces reasonable samples.

Represent the TLDR for finding the corresponding scientific article.

We propose DVD-GAN, a large video generative model that is state of the art on several tasks and produces highly complex videos when trained on large real world datasets.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

In this paper we introduced an algorithm to learn the connectivity of deep multi-branch networks. The approach is evaluated on image categorization where it consistently yields accuracy gains over state-of-the-art models that use fixed connectivity.