Question: Represent the input

Question: Training activation quantized neural networks involves minimizing a piecewise constant training loss whose gradient vanishes almost everywhere, which is undesirable for the standard back-propagation or chain rule. An empirical way around this issue is to use a straight-through estimator (STE) in the backward pass only, so that the "gradient" through the modified chain rule becomes non-trivial. Since this unusual "gradient" is certainly not the gradient of loss function, the following question arises: why searching in its negative direction minimizes the training loss? In this paper, we provide the theoretical justification of the concept of STE by answering this question. We consider the problem of learning a two-linear-layer network with binarized ReLU activation and Gaussian input data. We shall refer to the unusual "gradient" given by the STE-modifed chain rule as coarse gradient. The choice of STE is not unique. We prove that if the STE is properly chosen, the expected coarse gradient correlates positively with the population gradient (not available for the training), and its negation is a descent direction for minimizing the population loss. We further show the associated coarse gradient descent algorithm converges to a critical point of the population loss minimization problem. Moreover, we show that a poor choice of STE leads to instability of the training algorithm near certain local minima, which is verified with CIFAR-10 experiments. Deep neural networks (DNN) have achieved the remarkable success in many machine learning applications such as computer vision , natural language processing and reinforcement learning . However, the deployment of DNN typically require hundreds of megabytes of memory storage for the trainable full-precision floating-point parameters, and billions of floating-point operations to make a single inference. To achieve substantial memory savings and energy efficiency at inference time, many recent efforts have been made to the training of coarsely quantized DNN, meanwhile maintaining the performance of their float counterparts (; ; ; ; b).Training fully quantized DNN amounts to solving a very challenging optimization problem. It calls for minimizing a piecewise constant and highly nonconvex empirical risk function f (w) subject to a discrete set-constraint w ∈ Q that characterizes the quantized weights. In particular, weight quantization of DNN have been extensively studied in the literature; see for examples (; ; ; ; 2018a; ; ;). On the other hand, the gradient ∇f (w) in training activation quantized DNN is almost everywhere (a.e.) zero, which makes the standard back-propagation inapplicable. The arguably most effective way around this issue is nothing but to construct a non-trivial search direction by properly modifying the chain rule. Specifically, one can replace the a.e. zero derivative of quantized activation function composited in the chain rule with a related surrogate. This proxy derivative used in the backward pass only is referred as the straight-through estimator (STE) . In the same paper, proposed an alternative approach based on stochastic neurons. In addition, proposed the feasible target propagation algorithm for learning hard-threshold (or binary activated) networks via convex combinatorial optimization. The idea of STE originates to the celebrated perceptron algorithm (; 1962) in 1950s for learning single-layer perceptrons. The perceptron algorithm essentially does not calculate the "gradient" through the standard chain rule, but instead through a modified chain rule in which the derivative of identity function serves as the proxy of the original derivative of binary output function 1 {x>0}. Its convergence has been extensive discussed in the literature; see for examples, and the references therein. extended this idea to train multi-layer networks with binary activations (a.k.a. binary neuron), namely, to backpropagate as if the activation had been the identity function. proposed a STE variant which uses the derivative of the sigmoid function instead. In the training of DNN with weights and activations constrained to ±1, substituted the derivative of the signum activation function with 1 {|x|≤1} in the backward pass, known as the saturated STE. Later the idea of STE was readily employed to the training of DNN with general quantized ReLU activations (; ; ; ; b), where some other proxies took place including the derivatives of vanilla ReLU and clipped ReLU. Despite all the empirical success of STE, there is very limited theoretical understanding of it in training DNN with stair-case activations. considers leaky ReLU activation of a one-hidden-layer network. They showed the convergence of the so-called Convertron algorithm, which uses the identity STE in the backward pass through the leaky ReLU layer. Other similar scenarios, where certain layers are not desirable for back-propagation, have been brought up recently by and . The former proposed an implicit weighted nonlocal Laplacian layer as the classifier to improve the generalization accuracy of DNN. In the backward pass, the derivative of a pre-trained fullyconnected layer was used as a surrogate. To circumvent adversarial defense , introduced the backward pass differentiable approximation, which shares the same spirit as STE, and successfully broke defenses at ICLR 2018 that rely on obfuscated gradients. Throughout this paper, we shall refer to the "gradient" of loss function w.r.t. the weight variables through the STE-modified chain rule as coarse gradient. Since the backward and forward passes do not match, the coarse gradient is certainly not the gradient of loss function, and it is generally not the gradient of any function. Why searching in its negative direction minimizes the training loss, as this is not the standard gradient descent algorithm? Apparently, the choice of STE is non-unique, then what makes a good STE? From the optimization perspective, we take a step towards understanding STE in training quantized ReLU nets by attempting these questions. On the theoretical side, we consider three representative STEs for learning a two-linear-layer network with binary activation and Gaussian data: the derivatives of the identity function (; ;), vanilla ReLU and the clipped ReLUs . We adopt the model of population loss minimization (; ; ;). For the first time, we prove that proper choices of STE give rise to training algorithms that are descent. Specifically, the negative expected coarse gradients based on STEs of the vanilla and clipped ReLUs are provably descent directions for the minimizing the population loss, which yield monotonically decreasing energy in the training. In contrast, this is not true for the identity STE. We further prove that the corresponding training algorithm can be unstable near certain local minima, because the coarse gradient may simply not vanish there. Complementary to the analysis, we examine the empirical performances of the three STEs on MNIST and CIFAR-10 classifications with general quantized ReLU. While both vanilla and clipped ReLUs work very well on the relatively shallow LeNet-5, clipped ReLU STE is arguably the best for the deeper VGG-11 and ResNet-20. In our CIFAR experiments in section 4.2, we observe that the training using identity or ReLU STE can be unstable at good minima and repelled to an inferior one with substantially higher training loss and decreased generalization accuracy. This is an implication that poor STEs generate coarse gradients incompatible with the energy landscape, which is consistent with our theoretical finding about the identity STE.To our knowledge, convergence guarantees of perceptron algorithm (; 1962) and Convertron algorithm were proved for the identity STE. It is worth noting that Convertron makes weaker assumptions than in this paper. These , however, do not generalize to the network with two trainable layers studied here. As aforementioned, the identity STE is actually a poor choice in our case. Moreover, it is not clear if their analyses can be extended to other STEs. Similar to Convertron with leaky ReLU, the monotonicity of quantized activation function plays a role in coarse gradient descent. Indeed, all three STEs considered here exploit this property. But this is not the whole story. A great STE like the clipped ReLU matches quantized ReLU at the extrema, otherwise the instability/incompatibility issue may arise. Organization. In section 2, we study the energy landscape of a two-linear-layer network with binary activation and Gaussian data. We present the main and sketch the mathematical analysis for STE in section 3. In section 4, we compare the empirical performances of different STEs in 2-bit and 4-bit activation quantization, and report the instability phenomena of the training algorithms associated with poor STEs observed in CIFAR experiments. Due to space limitation, all the technical proofs as well as some figures are deferred to the appendix. Notations. · denotes the Euclidean norm of a vector or the spectral norm of a matrix. 0 n ∈ R n represents the vector of all zeros, whereas 1 n ∈ R n the vector of all ones. I n is the identity matrix of order n. For any w, z ∈ R n, w z = w, z = i w i z i is their inner product. w z denotes the Hadamard product whose i th entry is given by (w z) i = w i z i. We consider a model similar to that outputs the prediction DISPLAYFORM0 for some input Z ∈ R m×n. Here w ∈ R n and v ∈ R m are the trainable weights in the first and second linear layer, respectively; Z i denotes the ith row vector of Z; the activation function σ acts component-wise on the vector Zw, i.e., σ(Zw) i = σ((Zw) i ) = σ(Z i w). The first layer serves as a convolutional layer, where each row Z i can be viewed as a patch sampled from Z and the weight filter w is shared among all patches, and the second linear layer is the classifier. The label is generated according to y * (Z) = (v *) σ(Zw *) for some true (non-zero) parameters v * and w *. Moreover, we use the following squared sample loss DISPLAYFORM1 Unlike in , the activation function σ here is not ReLU, but the binary function σ(x) = 1 {x>0}.We assume that the entries of Z ∈ R m×n are i.i.d. sampled from the Gaussian distribution N . Since (v, w; Z) = (v, w/c; Z) for any scalar c > 0, without loss of generality, we take w * = 1 and cast the learning task as the following population loss minimization problem: DISPLAYFORM2 where the sample loss (v, w; Z) is given by. With the Gaussian assumption on Z, as will be shown in section 2.2, it is possible to find the analytic expressions of f (v, w) and its gradient DISPLAYFORM0.The gradient of objective function, however, is not available for the network training. In fact, we can only access the expected sample gradient, namely, DISPLAYFORM1. By the standard back-propagation or chain rule, we readily check that DISPLAYFORM2 and DISPLAYFORM3 Note that σ is zero a.e., which makes inapplicable to the training. The idea of STE is to simply replace the a.e. zero component σ in with a related non-trivial function µ (; ; ;), which is the derivative of some (sub)differentiable function µ. More precisely, back-propagation using the STE µ gives the following non-trivial surrogate of ∂ ∂w (v, w; Z), to which we refer as the coarse (partial) gradient DISPLAYFORM4 Using the STE µ to train the two-linear-layer convolutional neural network (CNN) with binary activation gives rise to the (full-batch) coarse gradient descent described in Algorithm 1.Algorithm 1 Coarse gradient descent for learning two-linear-layer CNN with STE µ. DISPLAYFORM5 Let us present some preliminaries about the landscape of the population loss function f (v, w).To this end, we define the angle between w and w * as θ(w, w *):= arccos w w * w w * for any w = 0 n. Recall that the label is given by y * (Z) = (v *) Zw * from, we elaborate on the analytic expressions of f (v, w) and ∇f (v, w). Lemma 1. If w = 0 n, the population loss f (v, w) is given by DISPLAYFORM0 Lemma 2. If w = 0 n and θ(w, w *) ∈ (0, π), the partial gradients of f (v, w) w.r.t. v and w are DISPLAYFORM1 respectively. For any v ∈ R m, (v, 0 m) is impossible to be a local minimizer. The only possible (local) minimizers of the model are located at 1. Stationary points where the gradients given by and vanish simultaneously (which may not be possible), i.e., DISPLAYFORM2 2. Non-differentiable points where θ(w, w *) = 0 and v = v *, or θ(w, w *) = π and v = DISPLAYFORM3 are obviously the global minimizers of. We show that the stationary points, if exist, can only be saddle points, and DISPLAYFORM4 are the only potential spurious local minimizers. DISPLAYFORM5 give the saddle points obeying, and DISPLAYFORM6 are the spurious local minimizers. Otherwise, the model has no saddle points or spurious local minimizers. We further prove that the population gradient ∇f (v, w) given by and, is Lipschitz continuous when restricted to bounded domains. Lemma 3. For any differentiable points (v, w) and (ṽ,w) with min{w, w} = c w > 0 and max{v, ṽ} = C v, there exists a Lipschitz constant L > 0 depending on C v and c w, such that DISPLAYFORM7 We are most interested in the complex case where both the saddle points and spurious local minimizers are present. Our main are concerned with the behaviors of the coarse gradient descent summarized in Algorithm 1 when the derivatives of the vanilla and clipped ReLUs as well as the identity function serve as the STE, respectively. We shall prove that Algorithm 1 using the derivative of vanilla or clipped ReLU converges to a critical point, whereas that with the identity STE does not. Theorem 1 (Convergence). Let {(v t, w t)} be the sequence generated by Algorithm 1 with ReLU µ(x) = max{x, 0} or clipped ReLU µ(x) = min {max{x, 0}, 1}. Suppose w t ≥ c w for all t with some c w > 0. Then if the learning rate η > 0 is sufficiently small, for any initialization (v 0, w 0), the objective sequence {f (v t, w t)} is monotonically decreasing, and {(v t, w t)} converges to a saddle point or a (local) minimizer of the population loss minimization. In addition, if 1 m v * = 0 and m > 1, the descent and convergence properties do not hold for Algorithm 1 with the identity function µ(x) = x near the local minimizers satisfying θ(w, w *) = π and DISPLAYFORM0 The convergence guarantee for the coarse gradient descent is established under the assumption that there are infinite training samples. When there are only a few data, in a coarse scale, the empirical loss roughly descends along the direction of negative coarse gradient, as illustrated by Figure 1. As the sample size increases, the empirical loss gains monotonicity and smoothness. This explains why (proper) STE works so well with massive amounts of data as in deep learning. Remark 2. The same hold, if the Gaussian assumption on the input data is weakened to that their rows i.i.d. follow some rotation-invariant distribution. The proof will be substantially similar. In the rest of this section, we sketch the mathematical analysis for the main .sample size = 10 sample size = 50 sample size = 1000Figure 1: The plots of the empirical loss moving by one step in the direction of negative coarse gradient v.s. the learning rate (step size) η for different sample sizes. If we choose the derivative of ReLU µ(x) = max{x, 0} as the STE in FORMULA7, it is easy to see µ (x) = σ(x), and we have the following expressions of DISPLAYFORM0 Let µ(x) = max{x, 0} in. The expected coarse gradient w.r.t. w is DISPLAYFORM1 where DISPLAYFORM2 As stated in Lemma 5 below, the key observation is that the coarse partial gradient E Z g relu (v, w; Z) has non-negative correlation with the population partial gradient ∂f ∂w (v, w), and −E Z g relu (v, w; Z) together with −E Z ∂ ∂v (v, w; Z) form a descent direction for minimizing the population loss. Lemma 5. If w = 0 n and θ(w, w *) ∈ (0, π), then the inner product between the expected coarse and population gradients w.r.t. w is DISPLAYFORM3 Moreover, if further v ≤ C v and w ≥ c w, there exists a constant A relu > 0 depending on C v and c w, such that DISPLAYFORM4 Clearly, when DISPLAYFORM5 We redefine the second term as 0n in the case θ(w, w *) = π, or equivalently, DISPLAYFORM6 that the coarse gradient descent behaves like the gradient descent directly on f (v, w). Here we would like to highlight the significance of the estimate in guaranteeing the descent property of Algorithm 1. By the Lipschitz continuity of ∇f specified in Lemma 3, it holds that DISPLAYFORM7 where a) is due to. Therefore, if η is small enough, we have monotonically decreasing energy until convergence. Lemma 6. When Algorithm 1 converges, E Z ∂ ∂v (v, w; Z) and E Z g relu (v, w; Z) vanish simultaneously, which only occurs at the 1. Saddle points where is satisfied according to Proposition 1. Lemma 6 states that when Algorithm 1 using ReLU STE converges, it can only converge to a critical point of the population loss function. For the STE using clipped ReLU, µ(x) = min {max{x, 0}, 1} and µ (x) = 1 {0<x<1} (x). We have similar to Lemmas 5 and 6. That is, the coarse partial gradient using clipped ReLU STE E Z g crelu (v, w; Z) generally has positive correlation with the true partial gradient of the population loss ∂f ∂w (v, w) (Lemma 7)). Moreover, the coarse gradient vanishes and only vanishes at the critical points (Lemma 8). Lemma 7. If w = 0 n and θ(w, w *) ∈ (0, π), then DISPLAYFORM0 * same as in Lemma 5, and DISPLAYFORM1 2 )dr. The inner product between the expected coarse and true gradients w.r.t. w DISPLAYFORM2 Moreover, if further v ≤ C v and w ≥ c w, there exists a constant A crelu > 0 depending on C v and c w, such that DISPLAYFORM3 Lemma 8. When Algorithm 1 converges, E Z ∂ ∂v (v, w; Z) and E Z g crelu (v, w; Z) vanish simultaneously, which only occurs at the 1. Saddle points where is satisfied according to Proposition 1. Now we consider the derivative of identity function. Similar to Lemmas 5 and 6 are not valid anymore. It happens that the coarse gradient derived from the identity STE does not vanish at local minima, and Algorithm 1 may never converge there. Lemma 9. Let µ(x) = x in. Then the expected coarse partial gradient w.r.t. w is DISPLAYFORM0 If θ(w, w *) = π and DISPLAYFORM1 i.e., E Z g id (v, w; Z) does not vanish at the local minimizers if 1 m v * = 0 and m > 1.Lemma 10. If w = 0 n and θ(w, w *) ∈ (0, π), then the inner product between the expected coarse and true gradients w.r.t. w is DISPLAYFORM2 Lemma 9 suggests that if 1 m v * = 0, the coarse gradient descent will never converge near the spurious minimizers with θ(w, w *) = π and DISPLAYFORM3 does not vanish there. By the positive correlation implied by of Lemma 10, for some proper (v 0, w 0), the iterates {(v t, w t)} may move towards a local minimizer in the beginning. But when {(v t, w t)} approaches it, the descent property does not hold for E Z [g id (v, w; Z)] because of, hence the training loss begins to increase and instability arises. While our theory implies that both vanilla and clipped ReLUs learn a two-linear-layer CNN, their empirical performances on deeper nets are different. In this section, we compare the performances of the identity, ReLU and clipped ReLU STEs on MNIST and CIFAR-10 benchmarks for 2-bit or 4-bit quantized activations. As an illustration, we plot the 2-bit quantized ReLU and its associated clipped ReLU in Figure 3 in the appendix. Intuitively, the clipped ReLU should be the best performer, as it best approximates the original quantized ReLU. We also report the instability issue of the training algorithm when using an improper STE in section 4.2. In all experiments, the weights are kept float. The resolution α for the quantized ReLU needs to be carefully chosen to maintain the full-precision level accuracy. To this end, we follow and resort to a modified batch normalization layer without the scale and shift, whose output components approximately follow a unit Gaussian distribution. Then the α that fits the input of activation layer the best can be pre-computed by a variant of Lloyd's algorithm (; a) applied to a set of simulated 1-D half-Gaussian data. After determining the α, it will be fixed during the whole training process. Since the original LeNet-5 does not have batch normalization, we add one prior to each activation layer. We emphasize that we are not claiming the superiority of the quantization approach used here, as it is nothing but the HWGQ , except we consider the uniform quantization. The optimizer we use is the stochastic (coarse) gradient descent with momentum = 0.9 for all experiments. We train 50 epochs for LeNet-5 on MNIST, and 200 epochs for VGG-11 and ResNet-20 on CIFAR-10. The parameters/weights are initialized with those from their pre-trained full-precision counterparts. The schedule of the learning rate is specified in TAB2 in the appendix. The experimental are summarized in Table 1, where we record both the training losses and validation accuracies. Among the three STEs, the derivative of clipped ReLU gives the best overall performance, followed by vanilla ReLU and then by the identity function. For deeper networks, clipped ReLU is the best performer. But on the relatively shallow LeNet-5 network, vanilla ReLU exhibits comparable performance to the clipped ReLU, which is somewhat in line with our theoretical finding that ReLU is a great STE for learning the two-linear-layer (shallow) CNN. We report the phenomenon of being repelled from a good minimum on ResNet-20 with 4-bit activations when using the identity STE, to demonstrate the instability issue as predicted in Theorem 1. By Table 1, the coarse gradient descent algorithms using the vanilla and clipped ReLUs converge to the neighborhoods of the minima with validation accuracies (training losses) of 86.59% (0.25) and 91.24% (0.04), respectively, whereas that using the identity STE gives 54.16% (1.38). Note that the landscape of the empirical loss function does not depend on which STE is used in the training. Then we initialize training with the two improved minima and use the identity STE. To see if the algorithm is stable there, we start the training with a tiny learning rate of 10 −5. For both initializations, the training loss and validation error significantly increase within the first 20 epochs; see Figure 4.2. To speedup training, at epoch 20, we switch to the normal schedule of learning rate specified in TAB2 and run 200 additional epochs. The training using the identity STE ends up with a much worse minimum. This is because the coarse gradient with identity STE does not vanish at the good minima in this case (Lemma 9). Similarly, the poor performance of ReLU STE on 2-bit activated ResNet-20 is also due to the instability of the corresponding training algorithm at good minima, as illustrated by Figure 4 in Appendix C, although it diverges much slower. Figure 2: When initialized with weights (good minima) produced by the vanilla (orange) and clipped (blue) ReLUs on ResNet-20 with 4-bit activations, the coarse gradient descent using the identity STE ends up being repelled from there. The learning rate is set to 10 −5 until epoch 20. We provided the first theoretical justification for the concept of STE that it gives rise to descent training algorithm. We considered three STEs: the derivatives of the identity function, vanilla ReLU and clipped ReLU, for learning a two-linear-layer CNN with binary activation. We derived the explicit formulas of the expected coarse gradients corresponding to the STEs, and showed that the negative expected coarse gradients based on vanilla and clipped ReLUs are descent directions for minimizing the population loss, whereas the identity STE is not since it generates a coarse gradient incompatible with the energy landscape. The instability/incompatibility issue was confirmed in CIFAR experiments for improper choices of STE. In the future work, we aim further understanding of coarse gradient descent for large-scale optimization problems with intractable gradients. Figure 4: When initialized with the weights produced by the clipped ReLU STE on ResNet-20 with 2-bit activations (88.38% validation accuracy), the coarse gradient descent using the ReLU STE with 10 −5 learning rate is not stable there, and both classification and training errors begin to increase. Lemma 11. Let z ∈ R n be a Gaussian random vector with entries i.i.d. sampled from N. Given nonzero vectors w,w ∈ R n with the angle θ, we have DISPLAYFORM0 Proof of Lemma 11. The third identity was proved in Lemma A.1 of . To show the first one, without loss of generality we assume w = [w 1, 0 n−1] with w 1 > 0, then E 1 {z w>0} = P(z 1 > 0) = 1 2. We further assumew = [w 1,w 2, 0 n−2]. It is easy to see that DISPLAYFORM1 To prove the last identity, we use polar representation of two-dimensional Gaussian random variables, where r is the radius and φ is the angle with dP r = r exp(−r 2 /2)dr and dP φ = 1 2π dφ. Then E z i 1 {z w>0, z w>0} = 0 for i ≥ 3. Moreover, DISPLAYFORM2 Therefore, Lemma 12. Let z ∈ R n be a Gaussian random vector with entries i.i.d. sampled from N. Given nonzero vectors w,w ∈ R n with the angle θ, we have Moreover, E z i 1 {0<z w<1} = 0 for i ≥ 3. So the first identity holds. For the second one, we have DISPLAYFORM3 DISPLAYFORM4 DISPLAYFORM5 and similarly, E z 2 1 {0<z w<1,z w>0} = q(θ, w). Therefore, DISPLAYFORM6 DISPLAYFORM7 Proof of Lemma 13. DISPLAYFORM8 where the last inequality is due to the rearrangement inequality since both sin(φ) and ξ DISPLAYFORM9 2. Since cos(φ)ξ DISPLAYFORM10 is even, we have DISPLAYFORM11 The first inequality is due to part 1 which gives p(π/2, w) ≤ q(π/2, w), whereas the second one holds because sin(φ)ξ Proof of Lemma 14. 1. Since by Cauchy-Schwarz inequality, DISPLAYFORM12 we have DISPLAYFORM13 Therefore, DISPLAYFORM14 where we used the fact sin(x) ≥ 2. Since I n − ww w 2 w * is the projection of w * onto the complement space of w, and likewise for I n −ww w 2 w *, the angle between I n − ww w 2 w * and I n −ww w 2 w * is equal to the angle between w andw. Therefore, DISPLAYFORM15 Lemma 1. If w = 0 n, the population loss f (v, w) is given by DISPLAYFORM0 Proof of Lemma 1. We notice that DISPLAYFORM1 Let Z i be the i th row vector of Z. Since w = 0 n, using Lemma 11, we have DISPLAYFORM2 and for i = j, DISPLAYFORM3, DISPLAYFORM4 Then it is easy to validate the first claim. Moreover, if w = 0 n, then DISPLAYFORM5 Lemma 2. If w = 0 n and θ(w, w *) ∈ (0, π), the partial gradients of f (v, w) w.r.t. v and w are DISPLAYFORM6 Proof of Lemma 2. The first claim is trivial, and we only show the second one. Since θ(w, w DISPLAYFORM7 give the saddle points obeying FORMULA11 DISPLAYFORM8 From FORMULA1 it follows that DISPLAYFORM9 On the other hand, from it also follows that DISPLAYFORM10 where we used (I m + 1 m 1 m)1 m = (m + 1)1 m. Taking the difference of the two equalities above gives DISPLAYFORM11 By, we have θ(w, w DISPLAYFORM12 Furthermore, since ∂f ∂v (v, w) = 0, we have DISPLAYFORM13 Next, we check the local optimality of the stationary points. By ignoring the scaling and constant terms, we rewrite the objective function as DISPLAYFORM14 It is easy to check that its Hessian matrix DISPLAYFORM15 is indefinite. Therefore, the stationary points are saddle points. DISPLAYFORM16 where we used in the last identity. We consider an arbitrary point (v + ∆v, π + ∆θ) in the neighborhood of (v, π) with ∆θ ≤ 0. The perturbed objective value is DISPLAYFORM17 On the right hand side, since v = (I m + 1 m 1 m) −1 (1 m 1 m − I m)v * is the unique minimizer to the quadratic functionf (v, π), we have if ∆v = 0 m, DISPLAYFORM18 Moreover, for sufficiently small ∆v, it holds that ∆θ · (v + ∆v) v * > 0 for ∆θ < 0 because of. Therefore,f (v + ∆v, π + ∆θ) >f (v, π) whenever (∆v, ∆θ) is small and non-zero, and DISPLAYFORM19 To prove the second claim, suppose Lemma 3. For any differentiable points (v, w) and (ṽ,w) with min{w, w} = c w > 0 and max{v, ṽ} = C v, there exists a Lipschitz constant L > 0 depending on C v and c w, such that DISPLAYFORM20 DISPLAYFORM21 DISPLAYFORM22 where the last inequality is due to Lemma 14.1. DISPLAYFORM0 where the second last inequality is to due to Lemma 14.2. Combining the two inequalities above validates the claim. Lemma 4. The expected partial gradient of (v, w; Z) w.r.t. v is DISPLAYFORM1 Let µ(x) = max{x, 0} in. The expected coarse gradient w.r.t. w is DISPLAYFORM2 Proof of Lemma 4. The first claim is true because DISPLAYFORM3 Using the fact that µ = σ = 1 {x>0}, we have DISPLAYFORM4 Invoking Lemma 11, we have DISPLAYFORM5 and DISPLAYFORM6 3 We redefine the second term as 0n in the case θ(w, w *) = π, or equivalently, DISPLAYFORM7 Therefore, DISPLAYFORM8 and the follows. Lemma 5. If w = 0 n and θ(w, w *) ∈ (0, π), then the inner product between the expected coarse and true gradients w.r.t. w is DISPLAYFORM9 Moreover, if further v ≤ C v and w ≥ c w, there exists a constant A relu > 0 depending on C v and c w, such that DISPLAYFORM10 Proof of Lemma 5. By Lemmas 2 and 4, we have DISPLAYFORM11 Notice that I n − ww w 2 w = 0 n and w * = 1, if θ(w, w *) = 0, π, then we have DISPLAYFORM12 To show the second claim, without loss of generality, we assume w = 1. Denote θ:= θ(w, w *). By Lemma 1, we have DISPLAYFORM13 By Lemma 4, DISPLAYFORM14 where DISPLAYFORM15 and by the first claim, DISPLAYFORM16 Hence, for some A relu depending only on C v and c w, we have DISPLAYFORM17 Saddle points where is satisfied according to Proposition 1. Proof of Lemma 6. By Lemma 4, suppose we have DISPLAYFORM0 and DISPLAYFORM1 where DISPLAYFORM2 If θ(w, w *) = 0, then by, v = v *, and FORMULA2 If v v * = 0, then by, we have the expressions for v and θ(w, w *) from Proposition 1, and is satisfied. Lemma 7. If w = 0 n and θ(w, w *) ∈ (0, π), then DISPLAYFORM3 where DISPLAYFORM4 * same as in Lemma 5, and DISPLAYFORM5 2 )dr. The inner product between the expected coarse and true gradients w.r.t. w DISPLAYFORM6 Moreover, if further v ≤ C v and w ≥ c w, there exists a constant A crelu > 0 depending on C v and c w, such that DISPLAYFORM7 Proof of Lemma 7. Denote θ:= θ(w, w *). We first compute E Z g crelu (v, w; Z). By, DISPLAYFORM8 Since µ = 1 {0<x<1} and σ = 1 {x>0}, we have In the last equality above, we called Lemma 12. DISPLAYFORM9 Notice that I n − ww w 2 w = 0 n and w * = 1. If θ(w, w *) = 0, π, then the inner product between E Z g crelu (v, w; Z) and Combining the above estimate together with FORMULA2, FORMULA2 and FORMULA2, and using Cauchy-Schwarz inequality, we have DISPLAYFORM10 where p(0, w) and q(θ, w) are uniformly bounded. This completes the proof. Proof of Lemma 8. The proof of Lemma 8 is similar to that of Lemma 6, and we omit it here. The core part is that q(θ, w) defined in Lemma 12 is non-negative and equals 0 only at θ = 0, π, as well as p(0, w) ≥ p(θ, w) ≥ p(π, w) = 0.Lemma 9. Let µ(x) = x in. Then the expected coarse partial gradient w.r.t. w is Proof of Lemma 9. By, DISPLAYFORM11 Using the facts that µ = 1 and σ = 1 {x>0}, we have DISPLAYFORM12 In the last equality above, we called the third identity In the third equality, we used the identity (I m + 1 m 1 m)1 m = (m + 1)1 m twice. Lemma 10. If w = 0 n and θ(w, w *) ∈ (0, π), then the inner product between the expected coarse and true gradients w.r.t. w is

Represent the short summary for finding the corresponding scientific article (long)!

We make theoretical justification for the concept of straight-through estimator.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the article for finding a single-sentence TLDR

Question: This paper presents GumbelClip, a set of modifications to the actor-critic algorithm, for off-policy reinforcement learning. GumbelClip uses the concepts of truncated importance sampling along with additive noise to produce a loss function enabling the use of off-policy samples. The modified algorithm achieves an increase in convergence speed and sample efficiency compared to on-policy algorithms and is competitive with existing off-policy policy gradient methods while being significantly simpler to implement. The effectiveness of GumbelClip is demonstrated against existing on-policy and off-policy actor-critic algorithms on a subset of the Atari domain. Recent advances in reinforcement learning (RL) have enabled the extension of long-standing methods to complex and large-scale tasks such as Atari , Go, and DOTA . The key driver has been the use of deep neural networks, a non-linear function approximator, with the combination usually referred to as Deep Reinforcement Learning (DRL) . However, deep learning-based methods are usually data-hungry, requiring millions of samples before the network converges to a stable solution. As such, DRL methods are usually trained in a simulated environment where an arbitrary amount of data can be generated. RL algorithms can be classified as either learning in an off-policy or on-policy setting. In the onpolicy setting, an agent learns directly from experience generated by its current policy. In contrast, the off-policy setting enables the agent to learn from experience generated by its current policy or/and other separate policies. An algorithm that learns in the off-policy setting has much greater sample efficiency as old experience from the current policy can be reused; it also enables off-policy algorithms to learn an optimal policy while executing an exploration-focused policy . The most famous off-policy method is Q-Learning which learns an actionvalue function, Q(s, a), that maps the value to a state s and action a pair. Deep Q-Learning (DQN), the marriage of Q-Learning with deep neural networks, was popularised by and used various modifications, such as experience replay, for stable convergence. Within DQN, experience replay is often motivated as a technique for reducing sample correlation. Unfortunately, all action-value methods, including Q-Learning, have two significant disadvantages. First, they learn deterministic policies, which cannot handle problems that require stochastic policies. Second, finding the greedy action with respect to the Q function is costly for large action spaces. To overcome these limitations, one could use policy gradient algorithms , such as actor-critic methods, which learn in an on-policy setting at the cost of sample efficiency. The ideal solution would be to combine the sample efficiency of off-policy algorithms with the desirable attributes of on-policy algorithms. Work along this line has been done by using importance sampling or by combining several techniques together, as in ACER . However, the ing methods are quite complex and require many modifications to existing algorithms. This paper, proposes a set of adjustments to A2C, a parallel on-policy actor-critic algorithm, enabling off-policy learning from stored trajectories. Therefore, our contributions are as follows: • GumbelClip, a fully off-policy actor-critic algorithm, the of a small set of simple adjustments, in under 10 lines of code (LOC) 1, to the A2C algorithm. • GumbelClip has increased sample efficiency and overall performance over on-policy actorcritic algorithms, such as A2C. • GumbelClip performs similarily to other off-policy actor-critic algorithms, such as ACER, while being significantly simpler to implement. The paper is organized as follows: Section 2 covers information, Section 3 describes the GumbelClip algorithm, Section 4 details the experiments along with , discussion, and ablations of our methodology. Section 5 discusses possible future work, and finally Section 6 provides concluding remarks. Consider an agent interacting with a Markov Decision Process (MDP) consisting of a set of states S, a set of actions A, a transition function P:, and a reward function r: S × A → R. Within this work, discrete actions and time steps are assumed. A policy is a probability distribution over actions conditioned on states, π: At each time step t, the agent observes the environment state s t ∈ S, chooses an action a t ∈ A from its policy π(a t |s t), and receives a reward r t from the environment. The goal of the agent is to maximize the discounted future return The discount factor γ ∈ trades off the importance of immediate and future rewards. Following from this, the value function of a policy π, in a discounted problem, is defined as the expected return To optimize the parameters θ of the stochastic policy π the policy gradient theorem is used, which provides an expression for the gradient of the discounted reward objectives with respect to parameter θ. Therefore, the parameters θ of the differentiable stochastic policy π θ (a t |s t) are updated as: where Ψ π (s t, a t), as shown by Schulman et al. (2015b), can be replaced with quantities such as: the total reward of the trajectory, the TD residual, or the state-action value function Q π (s t, a t). The choice of Ψ π affects the variance of the estimated gradient. This work uses the advantage function, which provides a relative measure of value for each action. The advantage function helps to reduce the variance of the gradient estimator while keeping the bias unchanged. The Gumbel-Softmax distribution (GSD) is a continuous distribution used to approximate samples from a categorical distribution. The GSD uses standard Gumbel noise to sample directly from the softmax distribution. On its own, the Gumbel distribution is typically used to model the maximum of a set of independent samples. Given categorical class probabilities π 1, π 2,..., π k the Gumbel-Softmax distribution is defined as: 1 Assuming code for replay memory is available. In practice, the policy gradient is estimated from a trajectory of samples generated by the on-policy stationary distribution π(a|s). This limits the efficiency of typical policy gradient methods, such as actor-critic, compared to methods like Deep Q-Learning which can learn off-policy. A common approach to using off-policy samples is a technique known as importance sampling (; ; ;). Given a trajectory of samples generated by some behaviour policy B(a|s), the policy gradient from Equation 1 is modified to be: where ρ is the known as the importance weight and is defined as a ratio between the current policy π(a|s) and the behaviour policy B(a|s): Unfortunately, the importance weighted gradient in Equation 3 suffers from high variance. To reduce variance, Wawrzyński proposed truncating each importance weight to the interval [0, c] where c is some constant. GumbelClip builds off the on-policy actor-critic algorithm A2C. To enable off-policy learning GumbelClip uses clipped importance sampling, policy forcing through Gumbel noise, and large batches sampled from a replay memory. Psuedo-code for GumbelClip is provided in Algorithm 1. We begin by defining a few quantities. In importance weighting, two policy classes exist: the current policy π(a|s; θ) and the behaviour policy B(a|s). Because replay memory is being used, the behaviour policy is simply the distribution over actions with an old parameter setting θ *: GumbelClip introduces a third policy, the forced policy F(a|s) using the Gumbel-Softmax distribution, which from adding noise (i) sampled from a standard Gumbel distribution to the normalized logits of the current policy: As the name implies, adding Gumbel noise has the effect of "forcing" the sampled policy distribution to be more categorical such that one action contains most of the probability mass. As Equation 6 is identical to the Gumbel-Softmax distribution, with temperature τ = 1, we can refer to Figure 1b) to understand the characteristics of the ing sampled distribution. Initialize parameters θ and θv. Initialize replay memory D with capacity N. repeat for i ∈ {0, · · ·, k} do Perform ai according to π(·|si; θ). Receive reward ri and new state si+1., 0, c end for end for Perform update of θ using dθ and θv using dθv. until Max iteration or time reached. Similarly to previous work, this study uses importance sampling ρ to weight the updates of the loss function. However, instead of using the current policy, π(·|s t), in the numerator, it is replaced with the forced policy F(a t |s t): The range of ρ t is clipped to [0, c] and this clipped importance weight is referred to asρ t. Clipping the upper bound prevents the product of many importance weights from exploding and reduces the variance (Wawrzyński, 2009). Putting this all together the update equation for GumbelClip is as follows: The gradient is estimated by uniformly sampling b trajectories of length k from a replay memory with size N. In addition, the network parameters are updated using the forced policy F(a|s). The advantage function A(s t, a t) = R is the bootstrapped k-step return for time t. As the forced policy in the numerator tends towards a categorical distribution, it became evident that the importance weights had the habit of clumping near the boundaries of the interval and often near 1. Following from Figure 2, we can see the distribution of the ratio between GumbelClip and a typical application of truncated importance sampling to A2C as might be suggested by Wawrzyński. The effect of clipping and Gumbel noise has an interesting effect on the way the policy is updated. From Figure 2 (b), we see that three modes exist, which roughly correspond to the cases of agreement and disagreement. The case of agreement corresponds to ratios and the mode near 1 while disagreements correspond to ratios and modes at 0 and c. More exactly, when the forced policy disagrees with the behaviour policy, say F(·|s) ≈ 1 and B(·|s) ≈ 0, the update the policy receives is at most clipped by the upper bound of our interval: c. On the other hand, when the situation is reversed, but still in disagreement, with F(·|s) ≈ 0 and B(·|s) ≈ 1, the policy has an importance weight of~0. Our experiments focus on the Atari domain as there exists a large amount of variety between environments and the states are represented as raw high-dimensional pixels. The gym software package by was used to conduct all the experiments. All experiments used the same algorithm, network architecture, and hyper-parameters to learn to play Atari games using only raw pixel observations. This study used the same input pre-processing and network architecture as. The network architecture consists of three convolutional layers as follows: 32 8 × 8 filters with stride 4, 64 4 × 4 filters with stride 2, and 32 3 × 3 filters with stride 1. The final convolutional layer feeds into a fully-connected layer with 512 units. All layers are followed by rectified non-linearity. Finally, the network outputs a softmax policy over actions and a state-value. The experimental set-up used 16 threads running on a GPU equipped machine. All experiments trained for 40 million frames. A replay memory of size N = 250000 was kept, an update was performed every k = 5 steps in the environment, and a clamping coefficient of c = 4 was used. The optimization procedure used RMSProp with a learning rate of 5e − 4, entropy regularization of 0.01, and a discount factor of γ = 0.99. We sample 64 trajectories of length 5 for each update. Learning begins after we have collected 10, 000 samples in the replay memory. We tuned the hyperparameters and developed GumbelClip on the FishingDerby environment only; the other environments can be considered "out of sample". All experiments used the same hyperparameter settings and network architecture. We use the best 3 of 4 seeds and report the mean value with 1 standard deviation as shaded areas on all graphs. As GumbelClip uses larger batchsizes we see an improvement in total run time per seed of 6-7 hours over A2C which takes 9-10 hours per seed. Due to limited computational resources, we were only able to evaluate GumbelClip on a subset of the environments and with a smaller replay memory size. Therefore, in this study, an effort was made to select environments that best showcase the performance of off-policy (ACER) and on-policy (A2C) Figure 3: Training performance across 8 Atari games. We see the performance of GumbelClip (shown in blue) against ACER (shown in green), an off-policy actor-critic algorithm, and the onpolicy algorithm A2C (shown in red). The graphs show the average performance over 3 seeds with 1 standard deviation shown as the shaded region. GumbelClip matches or exceeds the performance of the on-policy algorithm on all environments shown; while in all cases achieving improved convergence speed. It also shows a respectable performance when compared to the ACER algorithm on many of the environments. actor-critic methods. We note that the performance can be expected to improve with a larger replay memory, as seen with DQN and other methods using replay memory. Additionally, we focused the examination of our ablations on the Alien environment to reduce computational requirements. GumbelClip is based on a modified version of the open-source A2C implementation by. The present work was compared with an off-policy actor-critic algorithm, ACER , and an on-policy actor-critic algorithm, A2C, the synchronous version of A3C. Both baseline models, A2C and ACER, used the baselines package provided by OpenAI . ACER used a replay ratio of 4, trust region updates (a), a discount factor of γ = 0.99, entropy regularization of 0.01, updated every 20 steps, and a 50000 capacity replay memory per thread. To test the proposed methodology, the performance of GumbelClip on a subset of Atari environments was examined. In particular, the following environments were investigated: Alien, BeamRider, Boxing, FishingDerby, MsPacman, Qbert, Seaquest, and SpaceInvaders. We report the average reward every 1000 episodes over 40 million frames. As mentioned previously, environments were chosen where either the on-policy algorithms perform well or where there is a clear difference in performance between an off-policy and on-policy method. From Figure 3, we see the performance of GumbelClip, shown in blue, in comparison to the onpolicy ACER algorithm, shown in green, and the off-policy A2C algorithm. We see that the use of replay memory and learning with off-policy samples significantly improves the sample efficiency of GumbelClip over A2C. Across the board, we see that GumbelClip converges significantly faster than A2C while also exceeding A2C's performance. We achieve similar sample efficiency between the off-policy actor-critics, GumbelClip and ACER, across each environment. The additive noise and aggressive clamping seem to have both positive and negative effects. We see that GumbelClip sees a faster initial increase in performance across almost all environments, even outperforming ACER in some cases. However, clamping has been noted by to have the possibility of introducing bias. We hypothesize this might be the cause of poor performance on the Qbert environment, which while better than A2C, is much lower when compared to ACER. The benefit of GumbelClip comes from its simplicity, requiring a small number of easy changes to the A2C algorithm, while still providing better performance than other on-policy actor-critic algorithms. Figure 4: Variations in distributions used to sample additive noise. We compare the impact of noise drawn from the standard Gumbel distribution to the standard Normal distribution and Uniform distribution between. Here we examine the distribution used to draw the additive noise (i) for use in the forced policy F(a|s). We compare the performance of noise sampled from the standard Gumbel distribution to that of the standard Normal distribution and the Uniform distribution. The experiments only adjust the sampling distribution with no other parameter changes. We investigate the performance on the Alien Atari game over 40 million frames. From the in Figure 4, we see that the Gumbel and Normal distributions have the same initial rate of improvement but diverge roughly midway through training. The Normal distribution appears to degrade in performance before becoming stable at~1500. The uniform distribution, sampled between, has the same convergence characteristic as the other two distributions but instead of diverging away, similar to the Normal distribution, it continues upward before converging at~2000. From Figure 4, we see that the Gumbel distribution is the most performant. Figure 5: Stability of GumbelClip with extended training time. GumbelClip is trained for 150M frames,~4x longer, on the Boxing and FishingDerby environments. We see that GumbelClip experiences little to no oscillations in performance even with continued weight updates. It is natural to inquire on the stability of this method, as we rely on additive noise which could cause instability after an optimal policy has been reached. To this end, we evaluate the stability of GumbelClip by increasing the number of training iterations such that 150 million frames are seen. The Boxing and FishingDerby environments are used for this evaluation. The environments were chosen as the policy had achieved the highest score during training, a known ceiling, and any instability would cause a divergence to a sub-optimal policy. From the rather uninteresting graphs, shown in Figure 5, we see that GumbelClip can converge to and maintain a stable policy even with continued parameter updates. To overcome the noise added by the Gumbel distribution requires the network to output a near one-hot-encoded categorical distribution. In our final set of experiments, we performed ablations over the various components of GumbelClip to tease apart the cause of improvements in performance. The of the ablation are shown in Figure 6 (a) with the complete GumbelClip algorithm in blue and each stripped version of GumbelClip as a red curve. We start with a base version, which is the application of truncated importance sampling and replay memory to the off-policy algorithm A2C; this base version is shown in the leftmost panel in Figure 6 (a). From Figure 6: Ablations of GumbelClip. We gradually introduce each component of GumbelClip to a base version of an off-policy actor-critic A2C algorithm. a) The full GumbelClip algorithm is shown as the blue curve while the stripped versions are shown in red. From left to right we gradually add the components onto the base version until we arrive at the full GumbelClip algorithm, shown in the last pane. The lines in the last pane are identical with one graph is stylized. b) The table provides the percent deltas between either the stripped version to GumbelClip or the current model to the last. We measure the change between the last 100 episodes. lower than GumbelClip. The simple addition of a larger batchsize, from 16 sampled trajectories, to 64 as shown in the second panel in Figure 6 (a) causes an increase of +21.30% over the base version and narrows the difference to GumbelClip to −26.49%. Using an aggressive clamp of 4 instead of 10 on the importance sampling ratioρ improves performance by an additional +12.33%. Finally, the addition of noise sampled from the Gumbel distribution closes the gap with a final increase of +21.09%. The addition of noise changes Equation 8 from being in terms of π(a|s), in both the policy being optimized and the numerator inρ, to F(a|s). It is clearly shown that the large batchsize and additive noise contribute the most to the performance increase between stripped versions. Additionally, while difficult to quantify, we can see from the plots that the aggressive clamp and additive noise improve sample efficiency the most. Figure 7: Change in distributions overρ as components are added, shown as histograms. The xaxis corresponds to the magnitude ofρ and the y-axis is the number of occurrences. From left to right we see the distribution overρ when using the base version. As components are added we see three modes, at approximately 0, 1, and c, become more pronounced. The addition of Gumbel noise increases the smoothness between 0 → 1, creates a fatter tail from the mode 1 → c, and increases the density of all modes. From Figure 7, we see the effect that each addition to the base version has on the ratioρ and therefore the updates to the network. In the last two panes of Figure 7, we see three clear modes at 0, 1, and c. Addition of Gumbel noise increases the density across all modes and reduces the "noise" seen from 0 → 1. The modes, as discussed in Section 3, correspond to "agreement", at 1, and "disagreement" at {0, c}. As GumbelClip relies heavily on the Gumbel distribution to sample noise from, it is best suited to environments with discrete actions. Therefore, an interesting avenue for future work would be to find a suitable distribution to sample noise from that works with continuous actions. Additionally, further investigation into possible annealing schedules for a temperature hyperparameter, in Equation 6, or around the clamping constant c could yield interesting . In this paper we have presented GumbelClip, a set of adjustments to the on-policy A2C algorithm, enabling full off-policy learning from stored trajectories in a replay memory. Our approach relies on aggressive clipping of the importance weight, large batchsize, and additive noise sampled from the Gumbel distribution. We have empirically validated the use of each component in GumbelClip through ablations and shown the stability of the algorithm. Furthermore, we have shown that GumbelClip achieves superior performance and higher sample efficiency than A2C. GumbelClip nears the performance and sample efficiency of ACER on many of the tested environments. Our methodology requires minimal changes to the A2C algorithm, which in contrast to ACER, makes the implementation of GumbelClip straightforward.

Represent the TLDR for finding the corresponding scientific article

With a set of modifications, under 10 LOC, to A2C you get an off-policy actor-critic that outperforms A2C and performs similarly to ACER. The modifications are large batchsizes, aggressive clamping, and policy "forcing" with gumbel noise.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We augment the Q-value estimates with a count-based bonus that ensures optimism during action selection and bootstrapping, even if the Q-value estimates are pessimistic.


Question: In the past few years, various advancements have been made in generative models owing to the formulation of Generative Adversarial Networks (GANs). GANs have been shown to perform exceedingly well on a wide variety of tasks pertaining to image generation and style transfer. In the field of Natural Language Processing, word embeddings such as word2vec and GLoVe are state-of-the-art methods for applying neural network models on textual data. Attempts have been made for utilizing GANs with word embeddings for text generation. This work presents an approach to text generation using Skip-Thought sentence embeddings in conjunction with GANs based on gradient penalty functions and f-measures. The of using sentence embeddings with GANs for generating text conditioned on input information are comparable to the approaches where word embeddings are used. Numerous efforts have been made in the field of natural language text generation for tasks such as sentiment analysis BID35 and machine translation BID7 BID24. Early techniques for generating text conditioned on some input information were template or rule-based engines, or probabilistic models such as n-gram. In recent times, state-of-the-art on these tasks have been achieved by recurrent BID23 BID20 and convolutional neural network models trained for likelihood maximization. This work proposes an Code available at: https://github.com/enigmaeth/skip-thought-gan approach for text generation using Generative Adversarial Networks with Skip-Thought vectors. GANs BID9 are a class of neural networks that explicitly train a generator to produce high-quality samples by pitting against an adversarial discriminative model. GANs output differentiable values and hence the task of discrete text generation has to use vectors as differentiable inputs. This is achieved by training the GAN with sentence embedding vectors produced by Skip-Thought, a neural network model for learning fixed length representations of sentences. Deep neural network architectures have demonstrated strong on natural language generation tasks BID32. Recurrent neural networks using combinations of shared parameter matrices across time-steps BID30 BID20 BID3 with different gating mechanisms for easing optimization BID13 BID3 have found some success in modeling natural language. Another approach is to use convolutional neural networks that reuse kernels across time-steps with attention mechanism to perform language generation tasks BID15.Supervised learning with deep neural networks in the framework of encoder-decoder models has become the state-of-the-art methods for approaching NLP problems (Young et al.). Stacked denoising autoencoder have been used for domain adaptation in classifying sentiments BID8 and combinatory categorical autoencoders demonstrate learning the compositionality of sentences BID12. Recent text generation models use a wide variety of GANs such as gradient policy based sequence generation framework BID34 BID6 for performing natural language generation tasks. Other architectures such as those proposed in with RNN and variational auto-encoder generator with CNN discriminator and in BID11 with leaky discriminator to guide generator through high-level extracted features have also shown great . This section introduces Skip-Thought Generative Adversarial Network with a on models that it is based on. The Skip-Thought model induces embedding vectors for sentences present in training corpus. These vectors constitute the real distribution for the discriminator network. The generator network produces sentence vectors similar to those from the encoded real distribution. The generated vectors are sampled over training and decoded to produce sentences using a Skip-Thought decoder conditioned on the same text corpus. Skip-Thought is an encoder-decoder framework with an unsupervised approach to train a generic, distributed sentence encoder. The encoder maps sentences sharing semantic and syntactic properties to similar vector representations and the decoder reconstructs the surrounding sentences of an encoded passage. The sentence encoding approach draws inspiration from the skip-gram model in producing vector representations using previous and next sentences. The Skip-Thought model uses an RNN encoder with GRU activations BID4 and an RNN decoder with conditional GRU, the combination being identical to the RNN encoder-decoder of used in neural machine translation. For a given sentence tuple (s i−1, s i, s i+1), let w t i denote the t-th word for sentence s i, and let x t i denote its word embedding. The model has three components: Encoder. Encoded vectors for a sentence s i with N words w i, w i+1,...,w n are computed by iterating over the following sequence of equations: DISPLAYFORM0 Decoder. A neural language model conditioned on the encoder output h i serves as the decoder. Bias matrices C z, C r, C are introduced for the update gate, reset gate and hidden state computation by the encoder. Two decoders are used in parallel, one each for sentences s i + 1 and s i − 1. The following equations are iterated over for decoding: DISPLAYFORM1 Objective. For the same tuple of sentences, objective function is the sum of log-probabilities for the forward and backward sentences conditioned on the encoder representation: DISPLAYFORM2 Generative Adversarial Networks BID9 are deep neural net architectures comprised of two networks, contesting with each other in a zero-sum game framework. For a given data, GANs can mimic learning the underlying distribution and generate artificial data samples similar to those from the real distribution. Generative Adversarial Networks consists of two players -a Generator and a Discriminator. The generator G tries to produce data close to the real distribution P (x) from some stochastic distribution P (z) termed as noise. The discriminator D's objective is to differentiate between real and generated data G(z).The two networks -generator and discriminator compete against each other in a zero-sum game. The minimax strategy dictates that each network plays optimally with the assumption that the other network is optimal. This leads to Nash equilibrium which is the point of convergence for GAN model. Objective. BID9 have formulated the minimax game for a generator G, discriminator D adversarial network with value function V (G, D) as: DISPLAYFORM0 The STGAN model uses a deep convolutional generative adversarial network, similar to the one used in (Radford et al.). The generator network is updated twice for each discriminator network update to prevent fast convergence of the discriminator network. The Skip-Thought encoder for the model encodes sentences with length less than 30 words using 2400 GRU units BID4 with word vector dimensionality of 620 to produce 4800-dimensional combineskip vectors.. The combine-skip vectors, with the first 2400 dimensions being uni-skip model and the last 2400 bi-skip model, are used as they have been found to be the best performing in the experiments 1. The decoder uses greedy decoding taking argmax over softmax output distribution for given time-step which acts as input for next time-step. It reconstructs sentences conditioned on a sentence vector by randomly sampling from the predicted distributions with or without a preset beam width. Unknown tokens are not included in the vocabulary. A 620 dimensional RNN word embeddings is used with 1600 hidden GRU decoding units. Gradient clipping with Adam optimizer BID16 ) is used, with a batch size of 16 and maximum sentence length of 100 words for decoder. The training process of a GAN is notably difficult BID28 and several improvement techniques such as batch normalization, feature matching, historical averaging BID28 and unrolling GAN (Metz et al.) have been suggested for making the training more stable. Training the Skip-Thought GAN often in mode dropping (Arjovsky & Bottou; Srivastava et al.) with a parameter setting where it outputs a very narrow distribution of points. To overcome this, it uses minibatch discrimination by looking at an entire batch of samples and modeling the distance between a given sample and all the other samples present in that batch. The minimax formulation for an optimal discriminator in a vanilla GAN is Jensen-Shannon Distance between the generated distribution and the real distribution. used Wasserstein distance or earth mover's distance to demonstrate how replacing distance measures can improve training loss for GAN. BID10 have incorporated a gradient penalty regularizer in WGAN objective for discriminator's loss function. The experiments in this work use the above f-measures to improve performance of Skip-Thought GAN on text generation. GANs can be conditioned on data attributes to generate samples BID21 BID25. In this experiment, both the generator and discriminator are conditioned on Skip-Thought encoded vectors. The encoder converts 70000 sentences from the BookCorpus dataset with a training/test/validation split of 5/1/1 into vectors used as real samples for discriminator. The decoded sentences are used to evaluate model performance under corpus level BLEU-2, BLEU-3 and BLEU-4 metrics (Papineni et al.), once using only test set as reference and then entire corpus as reference. i can n't see some shopping happened. i had a police take watch out of my wallet.get him my camera found a person's my watch. here i collect my telephone card and telephone number delta airlines flight six zero two from six p.m. to miami, please? Table 2. Sample sentences generated from training on CMU-SE Dataset; mode collapse is overcome by using minibatch discrimination. Formation of sentences further improved by changing f-measure to Wasserstein distance along with gradient penalty regularizer. Language generation is done on a dataset comprising simple English sentences referred to as CMU-SE 2 in BID26. The CMU-SE dataset consists of 44,016 sentences with a vocabulary of 3,122 words. For encoding, the vectors are extracted in batches of sentences having the same length. The samples represent how mode collapse is manifested when using least-squares distance BID18 f-measure without minibatch discrimination. Table 2(a) contains sentences generated from STGAN using least-squares distance BID18 in which there was no mode collapse observed, while 2(b) contains examples wherein it is observed. Table 2(c) shows generated sentences using gradient penalty regularizer(GAN-GP). Table 2 (d) has samples generated from STGAN when using Wasserstein distance f-measure as WGAN ) and 2(e) contains samples when using a gradient penalty regularizer term as WGAN-GP BID10. Another performance metric that can be computed for this setup has been described in BID26 which is a parallel work to this. Simple CFG 3 and more complex ones like Penn Treebank CFG generate samples BID5 which are used as input to GAN and the model is evaluated by computing the diversity and accuracy of generated samples conforming to the given CFG.Skip-Thought sentence embeddings can be used to generate images with GANs conditioned on text vectors for text-to-image conversion tasks like those achieved in BID27 BID2. These embeddings have also been used to Models like neuralstoryteller 4 which use these sentence embeddings can be experimented with generative adversarial networks to generate unique samples.

Represent the short summary for finding the corresponding scientific article (long):

Generating text using sentence embeddings from Skip-Thought Vectors with the help of Generative Adversarial Networks.

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.


Question: Represent the article for finding a single-sentence TLDR

Question: Autoregressive recurrent neural decoders that generate sequences of tokens one-by-one and left-to-right are the workhorse of modern machine translation. In this work, we propose a new decoder architecture that can generate natural language sequences in an arbitrary order. Along with generating tokens from a given vocabulary, our model additionally learns to select the optimal position for each produced token. The proposed decoder architecture is fully compatible with the seq2seq framework and can be used as a drop-in replacement of any classical decoder. We demonstrate the performance of our new decoder on the IWSLT machine translation task as well as inspect and interpret the learned decoding patterns by analyzing how the model selects new positions for each subsequent token.

new out-of-order decoder for neural machine translation

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.