Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Direct policy gradient methods for reinforcement learning and continuous control problems are a popular approach for a variety of reasons: 1) they are easy to implement without explicit knowledge of the underlying model; 2) they are an "end-to-end" approach, directly optimizing the performance metric of interest; 3) they inherently allow for richly parameterized policies. A notable drawback is that even in the most basic continuous control problem (that of linear quadratic regulators), these methods must solve a non-convex optimization problem, where little is understood about their efficiency from both computational and statistical perspectives. In contrast, system identification and model based planning in optimal control theory have a much more solid theoretical footing, where much is known with regards to their computational and statistical properties. This work bridges this gap showing that (model free) policy gradient methods globally converge to the optimal solution and are efficient (polynomially so in relevant problem dependent quantities) with regards to their sample and computational complexities. Recent years have seen major advances in the control of uncertain dynamical systems using reinforcement learning and data-driven approaches; examples range from allowing robots to perform more sophisticated controls tasks such as robotic hand manipulation (; BID1 ; ; ; a), to sequential decision making in game domains, e.g. AlphaGo and Atari game playing . Deep reinforcement learning (DeepRL) are becoming increasingly popular for tackling such challenging sequential decision making problems. Many of these successes have relied on sampling based reinforcement learning algorithms such as policy gradient methods, including the DeepRL approaches; here, there is little theoretical understanding of their efficiency, either from a statistical or a computational perspective. In contrast, control theory (optimal and adaptive control) has a rich body of tools, with provable guarantees, for related sequential decision making problems, particularly those that involve continuous control. These latter techniques are often model-based -they estimate an explicit dynamical model first (e.g. system identification) and then design optimal controllers. This work builds bridges between these two lines of work, namely, between optimal control theory and sample based reinforcement learning methods, using ideas from mathematical optimization. In the standard optimal control problem, the dynamics model f t, where f t is specified as x t+1 = f t (x t, u t, w t), maps a state x t ∈ R d, a control (the action) u t ∈ R k, and a disturbance w t, to the next state x t+1 ∈ R d. The objective is to find the control input u t which minimizes the long term cost, minimize T t=1 c t (x t, u t)such that x t+1 = f t (x t, u t, w t).Here the u t are allowed to depend on the history of observed states. In practice, this is often solved by considering the linearized control (sub-)problem where the dynamics are approximated by x t+1 = A t x t + B t u t + w t, and the costs are approximated by a quadratic function in x t and u t, e.g. . This work considers an important special case: the time homogenous, infinite horizon problem referred to as the linear quadratic regulator (LQR) problem. The herein can also be extended to the finite horizon, time in-homogenous setting, discussed in Section 5.In the LQR problem, the objective is minimize E ∞ t=0 (x t Qx t + u t Ru t)such that x t+1 = Ax t + Bu t, x 0 ∼ D.where initial state x 0 ∼ D is assumed to be randomly distributed according to distribution D; the matrices A ∈ R d×d and B ∈ R d×k are referred to as system (or transition) matrices; Q ∈ R d×d and R ∈ R k×k are both positive definite matrices that parameterize the quadratic costs. For clarity, this work does not consider a noise disturbance but only a random initial state. The importance of (some) randomization for analyzing direct methods is discussed in Section 3.Throughout, assume that A and B are such that the optimal cost is finite (for example, the controllability of the pair (A, B) would ensure this). Optimal control theory BID2 BID13 BID5 BID6 shows that the optimal control input can be written as a linear function in the state, DISPLAYFORM0 Planning with a known model. Planning can be achieved by solving the algebraic Riccati equation, DISPLAYFORM1 for a positive definite matrix P which parameterizes the "cost-to-go" (the optimal cost from a state going forward). The optimal control gain is then given as: DISPLAYFORM2 There are both algebraic solution methods to find P and (convex) SDP formulations to solve for P. More broadly, even though there are convex formulations for planning, these formulations: 1) do not directly parameterize the policy 2) they are not "end-to-end" approaches in that they are not directly optimizing the cost function of interest and 3) it is not immediately clear how to utilize these approaches in the model-free setting, where the agent only has simulation access. These formulations are discussed in Section A, where there is a discussion of how the standard SDP formulation is not a direct method that minizes the cost over the set of feasible policies. Even in the most basic case of the standard linear quadratic regulator model, little is understood as to how direct (model-free) policy gradient methods fare. This work provides rigorous guarantees, showing that, while in fact the approach is a non-convex one, directly using (model free) local search methods leads to finding the globally optimal policy. The main contributions are as follows:• (Exact case) Even with access to exact gradient evaluation, little is understood about whether or not convergence to the optimal policy occurs, even in the limit, due to the non-convexity in the problem. This work shows that global convergence does indeed occur (and does so efficiently) for local search based methods.• (Model free case) Without a model, this work shows how one can use simulated trajectories (as opposed to having knowledge of the model) in a stochastic policy gradient method where provable convergence to a globally optimal policy is guaranteed, with (polynomially) efficient computational and sample complexities.• (The natural policy gradient) Natural policy gradient methods BID18 ) -and related algorithms such as Trust Region Policy Optimization and the natural actor critic -are some of the most widely used and effective policy gradient methods (see BID12). While many argue in favor of this method based on either information geometry BID18 BID3 or based on connections to actor-critic methods BID11, these do not provably show an improved convergence rate. This work is the first to provide a guarantee that the natural gradient method enjoys a considerably improved convergence rate over its naive gradient counterpart. More broadly, the techniques in this work merge ideas from optimal control theory, mathematical (and zeroth order) optimization, and sample based reinforcement learning methods. These techniques may ultimately help in improving upon the existing set of algorithms, addressing issues such as variance reduction or improving upon the natural policy gradient method (with, say, a GaussNewton method). The Discussion touches upon some of these issues. In the reinforcement learning setting, the model is unknown, and the agent must learn to act through its interactions with the environment. Here, solution concepts are typically divided into: modelbased approaches, where the agent attempts to learn a model of the world, and model-free approaches, where the agent directly learns to act and does not explicitly learn a model of the world. The related work on provably learning LQRs is reviewed from this perspective. Model-based learning approaches. In the context of LQRs, the agent attempts to learn the dynamics of "the plant" (i.e. the model) and then plans, using this model, for control synthesis. Here, the classical approach is to learn the model with subspace identification . BID14 provides a provable learning (and non-asymptotic) , where the quality of the policy obtained is shown to be near optimal (efficiency is in terms of the persistence of the training data and the controllability Gramian). BID0 also provides provable, nonasymptotic learning (in a regret context), using a bandit algorithm that achieves lower sample complexity (by balancing exploration-exploitation more effectively); the computational efficiency of this approach is less clear. More recently, BID10 expands on an explicit system identification process, where a robust control synthesis procedure is adopted that relies on a coarse model of the plant matrices (A and B are estimated up to some accuracy level, naturally leading to a "robust control" setup). Arguably, this is the most general (and non-asymptotic) , that is efficient from both a statistical perspective (computationally, the method works with a finite horizon to approximate the infinite horizon). This only needs the plant to be controllable; the work herein needs the stronger assumption that the initial policy in the local search procedure is a stable controller (an assumption which may be inherent to local search procedures, discussed in Section 5).Model-free learning approaches. Model-free approaches that do not rely on an explicit system identification step typically either: 1) estimate value functions (or state-action values) through Monte Carlo simulation which are then used in some approximate dynamic programming variant or 2) directly optimize a (parameterized) policy, also through Monte Carlo simulation. Model-free approaches for learning optimal controllers is not well understood, from a theoretical perspective. Here, BID7 provides an asymptotic learnability using a value function approach, namely Q-learning. This work seeks to characterize the behavior of (direct) policy gradient methods, where the policy is linearly parameterized, as specified by a matrix K ∈ R k×d which generates the controls: DISPLAYFORM0 The cost of this K is denoted as: DISPLAYFORM1 where {x t, u t} is the trajectory induced by following K, starting with x 0 ∼ D. The importance of (some) randomization, either in x 0 or noise through having a disturbance, for analyzing gradient methods is discussed in Section 3. Here, K * is a minimizer of C(·).Gradient descent on C(K), with a fixed stepsize η, follows the update rule: DISPLAYFORM2 It is helpful to explicitly write out the functional form of the gradient. Define P K as the solution to: DISPLAYFORM3 and, under this definition, it follows that C(K) can be written as: DISPLAYFORM4 Also, define Σ K as the (un-normalized) state correlation matrix, i.e. DISPLAYFORM5 Lemma 1. (Policy Gradient Expression) The policy gradient is: DISPLAYFORM6 Observe: DISPLAYFORM7 This implies: DISPLAYFORM8 x t x t using recursion and that x 1 = (A − BK)x 0. Taking expectations completes the proof. Sample based policy gradient methods introduce some randomization for estimating the gradient. REINFORCE. Let π θ (u|x) be a parametric stochastic policy, where u ∼ π θ (·|x). The policy gradient of the cost, C(θ), is: DISPLAYFORM0 where the expectation is with respect to the trajectory {x t, u t} induced under the policy π θ and where Q π θ (x, u) is referred to as the state-action value. The REINFORCE algorithm uses Monte Carlo estimates of the gradient obtained by simulating π θ.The natural policy gradient. The natural policy gradient BID18 ) follows the update: DISPLAYFORM1 where G θ is the Fisher information matrix. There are numerous succesful related approaches (; ; BID12). An important special case is using a linear policy with additive Gaussian noise (b) DISPLAYFORM2 where K ∈ R k×d and σ 2 is the noise variance. Here, the natural policy gradient of K (when σ is considered fixed) takes the form: DISPLAYFORM3 To see this, one can verify that the Fisher matrix of size kd × kd, which is indexed as DISPLAYFORM4 where i, i ∈ {1, . . . k} and j, j ∈ {1, . . . d}, has a block diagonal form where the only non-zeros blocks are [G K] (i,·),(i,·) = Σ K (this is the block corresponding to the i-th coordinate of the action, as i ranges from 1 to k). This form holds more generally, for any diagonal noise. Zeroth order optimization. Zeroth order optimization is a generic procedure BID9 ) for optimizing a function f (x), using only query access to the function values of f (·) at input points x (and without explicit query access to the gradients of f). This is also the approach in using "evolutionary strategies" . The generic approach can be described as follows: define the perturbed function as DISPLAYFORM5 For small σ, the smooth function is a good approximation to the original function. Due to the Gaussian smoothing, the gradient has the particularly simple functional form (see BID9): DISPLAYFORM6. This expression implies a straightforward method to obtain an unbiased estimate of the ∇f σ 2 (x), through obtaining only the function values f (x + ε) for random ε. This section provides a brief characterization of the optimization landscape, in order to help provide intuition as to why global convergence is possible and as to where the analysis difficulties lie. Lemma 2. (Non-convexity) If d ≥ 3, there exists an LQR optimization problem, min K C(K), which is not convex, quasi-convex, and star-convex. Section B provides a specific example. In general, for a non-convex optimization problem, gradient descent may not even converge to the global optima in the limit. For the case of LQRs, the following corollary (of Lemma 8) provides a characterization of the stationary points. Corollary 3. (Stationary point characterization) If ∇C(K) = 0, then either K is an optimal policy or Σ K is rank deficient. This lemma is the motivation for using a distribution over x 0 (as opposed to a deterministic starting point): E x0∼D x 0 x 0 being full rank guarantees that Σ K is full rank, which implies all stationary points are a global optima. An additive disturbance in the dynamics model also suffices. The concept of gradient domination is important in the non-convex optimization literature (; ;). A function f: R d → R is said to be gradient dominated if there exists some constant λ, such that for all x, DISPLAYFORM0 If a function is gradient dominated, this implies that if the magnitude of the gradient is small at some x, then the function value at x will be close to that of the optimal function value. The following corollary of Lemma 8 shows that C(K) is gradient dominated. DISPLAYFORM1 where λ is a problem dependent constant (and ·, · denotes the trace inner product).With gradient domination and no (spurious) local optima, one may hope that recent on escaping saddle points (; BID16 BID17 immediately imply that gradient descent converges quickly. This is not the case due to that it is not straightforward to characterize the (local) smoothness properties of C(K); this is a difficulty well studied in the optimal control theory literature, related to robustness and stability. In fact, if it were the case that C(K) is a smooth function 1 (in addition to being gradient dominated), then classical mathematical optimization would not only immediately imply global convergence, these would also imply convergence at a linear rate. First, on exact gradient methods are provided. From an analysis perspective, this is the natural starting point; once global convergence is established for exact methods, the question of using simulation-based, model-free methods can be approached with zeroth-order optimization methods. Notation. Z denotes the spectral norm of a matrix Z; Tr(Z) denotes the trace of a square matrix; σ min (Z) denotes the minimal singular value of a square matrix Z. Also, it is helpful to define DISPLAYFORM0 The following three exact update rules are considered: DISPLAYFORM0 Natural policy gradient descent: DISPLAYFORM1 Gauss-Newton: DISPLAYFORM2 Kn. The natural policy gradient descent direction is defined so that it is consistent with the stochastic case, as per Equation 4. It is straightforward to verify that the policy iteration algorithm is a special case of the Gauss-Newton method when η = 1 (for the case of policy iteration, convergence in the limit is provided in Todorov & Li ( The Gauss-Newton method requires the most complex oracle to implement: it requires access to ∇C(K), Σ K, and R+B P K B; it also enjoys the strongest convergence rate guarantee. At the other extreme, gradient descent requires oracle access to only ∇C(K) and has the slowest convergence rate. The natural policy gradient sits in between, requiring oracle access to ∇C(K) and Σ K, and having a convergence rate between the other two methods. Theorem 5. (Global Convergence of Gradient Methods) Suppose C(K 0) is finite and and µ > 0.• Gauss-Newton case: For a stepsize η = 1 and for DISPLAYFORM3 the Gauss-Newton algorithm (Equation 7) enjoys the following performance bound: DISPLAYFORM4 • Natural policy gradient case: For a stepsize DISPLAYFORM5 and for DISPLAYFORM6 natural policy gradient descent (Equation 6) enjoys the following performance bound: DISPLAYFORM7 Algorithm 1 Model-Free Policy Gradient (and Natural Policy Gradient) Estimation 1: Input: K, number of trajectories m, roll out length, smoothing parameter r, dimension d 2: DISPLAYFORM8 Sample a policy K i = K + U i, where U i is drawn uniformly at random over matrices whose (Frobenius) norm is r. Simulate K i for steps starting from x 0 ∼ D. Let C i and Σ i be the empirical estimates: DISPLAYFORM0 where c t and x t are the costs and states on this trajectory. 5: end for 6: Return the (biased) estimates: DISPLAYFORM1 • Gradient descent case: For an appropriate (constant) setting of the stepsize η, DISPLAYFORM2 and for DISPLAYFORM3, gradient descent (Equation 5) enjoys the following performance bound: DISPLAYFORM4 In comparison to model-based approaches, these require the (possibly) stronger assumption that the initial policy is a stable controller, i.e. C(K 0) is finite (an assumption which may be inherent to local search procedures). The Discussion mentions this as direction of future work. In the model free setting, the controller has only simulation access to the model; the model parameters, A, B, Q and R, are unknown. The standard optimal control theory approach is to use system identification to learn the model, and then plan with this learned model This section proves that model-free, policy gradient methods also lead to globally optimal policies, with both polynomial computational and sample complexities (in the relevant quantities).Using a zeroth-order optimization approach (see Section 2.2), Algorithm 1 provides a procedure to find (controllably biased) estimates, ∇C(K) and Σ K, of both ∇C(K) and Σ K. These can then be used in the policy gradient and natural policy gradient updates as follows: DISPLAYFORM0 Natural policy gradient descent: DISPLAYFORM1 where Algorithm 1 is called at every iteration to provide the estimates of ∇C(K n) and Σ Kn.The choice of using zeroth order optimization vs using REINFORCE (with Gaussian additive noise, as in Equation 3) is primarily for technical reasons 2. It is plausible that the REINFORCE estimation procedure has lower variance. One additional minor difference, again for technical reasons, is that Algorithm 1 uses a perturbation from the surface of a sphere (as opposed to a Gaussian perturbation).Theorem 6. (Global Convergence in the Model Free Setting) Suppose C(K 0) is finite, µ > 0, and that x 0 ∼ D has norm bounded by L almost surely. Also, for both the policy gradient method and the natural policy gradient method, suppose Algorithm 1 is called with parameters: DISPLAYFORM2 • Natural policy gradient case: For a stepsize DISPLAYFORM3 and for DISPLAYFORM4 then, with high probability, i.e. with probability greater than 1 − exp(−d), the natural policy gradient descent update (Equation 9) enjoys the following performance bound: DISPLAYFORM5 • Gradient descent case: For an appropriate (constant) setting of the stepsize η, DISPLAYFORM6 and if N satisfies DISPLAYFORM7, then, with high probability, gradient descent (Equation 8) enjoys the following performance bound: DISPLAYFORM8 This work has provided provable guarantees that model-based gradient methods and model-free (sample based) policy gradient methods convergence to the globally optimal solution, with finite polynomial computational and sample complexities. Taken together, the herein place these popular and practical policy gradient approaches on a firm theoretical footing, making them comparable to other principled approaches (e.g. subspace ID methods and algebraic iterative approaches).Finite C(K 0) assumption, noisy case, and finite horizon case. These methods allow for extensions to the noisy case and the finite horizon case. This work also made the assumption that C(K 0) is finite, which may not be easy to achieve in some infinite horizon problems. The simplest way to address this is to model the infinite horizon problem with a finite horizon one; the techniques developed in Section D.1 shows this is possible. This is an important direction for future work. Open Problems.• Variance reduction: This work only proved efficiency from a polynomial sample size perspective. An interesting future direction would be in how to rigorously combine variance reduction methods and model-based methods to further decrease the sample size.• A sample based Gauss-Newton approach: This work showed how the Gauss-Newton algorithm improves over even the natural policy gradient method, in the exact case. A practically relevant question for the Gauss-Newton method would be how to both: a) construct a sample based estimator b) extend this scheme to deal with (non-linear) parametric policies.• Robust control: In model based approaches, optimal control theory provides efficient procedures to deal with (bounded) model mis-specification. An important question is how to provably understand robustness in a model free setting. This section briefly reviews some parameterizations and solution methods for the classic LQR and related problems from control theory. Finite horizon LQR. First, consider the finite horizon case. The basic approach is to view it as a dynamic program with the value function x T t P t x t, where DISPLAYFORM0 which in turn gives optimal control DISPLAYFORM1 Another approach is to view the LQR problem as a linearly-constrained Quadratic Program in all x t and u t (where the constraints are given by the dynamics, and the problem size equals the horizon).The QP is clearly a convex problem, but this observation is not useful by itself as the problem size grows with the horizon, and naive use of quadratic programming scales badly. However, the special structure due to linear dynamics allows for simplifications and control-theoretic interpretation as follows: the Lagrange multipliers can be interpreted as "co-state" variables, and they follow a recursion that runs backwards in time known as the "adjoint system". Using Lagrange duality, one can show that this approach is equivalent to solving the Riccati recursion mentioned above. Popular use of the LQR in control practice is often in the receding horizon LQR, BID8;: at time t, an input sequence is found that minimizes the T -step ahead LQR cost starting at the current time, then only the first input in the sequence is used. The ing static feedback gain converges to the infinite horizon optimal solution as horizon T becomes longer. Infinite horizon LQR. Here, the constrained optimization view (QP) is not informative as the problem is infinite dimensional; the dynamic programming viewpoint readily extends. Suppose the system A, B is controllable (which guarantees optimal cost is finite). It turns out that the value function and the optimal controller are static (do not depend on t) and can be found by solving the Algebraic Riccati Equation (ARE) given in. The optimal K can then be found from equation.The main computational step is solving the ARE, which is extensively studied (e.g. ). One approach due to is to simply run the recursion DISPLAYFORM2 T P k A with P 1 = Q, which converges to the unique positive semidefinite solution of the ARE (since the fixed-point iteration is contractive). Other approaches are direct and based on linear algebra, which carry out an eigenvalue decomposition on a certain block matrix followed by a matrix inversion .Direct computation of the control input has also been considered in the optimal control literature, e.g., gradient updates in function spaces . For the linear quadratic setup, direct iterative computation of the feedback gain has been examined in (Mårtensson &), and explored further in (Mårtensson, 2012) with a view towards distributed implementations. There methods are presented as local search heuristics without provable guarantees of reaching the optimal policy. SDP formulation. The LQR problem can also be expressed as a semidefinite program (SDP) with variable P, as given in (section 5, equation, this is for a continuous-time system but there are similar discrete-time versions). This SDP can be derived by relaxing the equality in the Riccati equation to an inequality, then using the Schur complement formula to rewrite the ing Riccati inequality as linear matrix inequality; the objective in the case of LQR is the trace of the positive definite matrix variable. This formulation and its dual has been explored in BID4.It is important to note that while the optimal solution of this SDP is the unique positive semidefinite solution to the Riccati equation, which in turn gives the optimal policy K *, other feasible P (not equal to P *) do not necessarily correspond to a feasible, stabilizing policy K. This means that the feasible set of this SDP is not a convex characterization of all P that correspond to stabilizing K. Thus it also implies that if one uses any optimization algorithm that maintains iterates in the feasible set (e.g. interior point methods), no useful policy can be extracted from the iterates before convergence to P *. For this reason, this convex formulation is not helpful for parametrizing the space of policies K in manner that supports the use of local search methods (those that directly lower the cost function of interest), which is the focus of this work. Let K(A, B) denote the set of state feedback gains K such that A − BK is stable, i.e., its eigenvalues are inside the unit circle in the complex plane. This set is generally nonconvex. A concise counterexample to convexity is provided here. Let A and B be 3 × 3 identity matrices and This section provides the analysis of the convergence rates of the (exact) gradient based methods. First, some helpful lemmas for the analysis are provided. Throughout, it is convenient to use the following definition: DISPLAYFORM0 The policy gradient can then be written as: DISPLAYFORM1 Define the value V K (x), the state-action value Q K (x, u), and the advantage A K (x, u). V K (x, t) is the cost of the policy starting with x 0 = x and proceeding with K onwards: DISPLAYFORM0 is the cost of the policy starting with x 0 = x, taking action u 0 = u and then proceeding with K onwards: DISPLAYFORM1 The advantage can be viewed as the change in cost starting at state x and taking a one step deviation from the policy K.The next lemma is identical to that in BID19 BID20 for Markov decision processes. Lemma 7. (Cost difference lemma) Suppose K and K have finite costs. Let {x t} and {u t} be state and action sequences generated by K, i.e. starting with x 0 = x and using u t = −K x t. It holds that: DISPLAYFORM2 Also, for any x, the advantage is: DISPLAYFORM3 Proof. Let c t be the cost sequence generated by K. Telescoping the sum appropriately: DISPLAYFORM4 which completes the first claim. For the second claim, observe that: DISPLAYFORM5 And, for u = K x, DISPLAYFORM6 which completes the proof. This lemma is helpful in proving that C(K) is gradient dominated. Lemma 8. (Gradient domination) Let K * be an optimal policy. Suppose K has finite cost and µ > 0. It holds that: DISPLAYFORM7 For a lower bound, it holds that: DISPLAYFORM8 Proof. From Equation 10 and by completing the square, DISPLAYFORM9 with equality when DISPLAYFORM10 Let x * t and u * t be the sequence generated under K *. Using this and Lemma 7, DISPLAYFORM11 which completes the proof of the upper bound. For the lower bound, consider DISPLAYFORM12 −1 E K where equality holds in Equation 11. Let x t and u t be the sequence generated under K. Using that DISPLAYFORM13 which completes the proof. Recall that a function f is said to be smooth (or C 1 -smooth) if it satisfies for some finite β, it satisfies: DISPLAYFORM14 for all x, y (equivalently, it is smooth if the gradients of f are continuous). Lemma 9. ("Almost" smoothness) C(K) satisfies: DISPLAYFORM15 To see why this is related to smoothness (e.g. compare to Equation 12), suppose K is sufficiently close to K so that: DISPLAYFORM16 and the leading order term 2Tr(Σ K (K − K) E K ) would then behave as Tr((K − K) ∇C(K)). The challenge in the proof (for gradient descent) is quantifying the lower order terms in this argument. Proof. The claim immediately from Lemma 7, by using Equation 10 and taking an expectation. The next lemma spectral norm bounds on P K and Σ K are helpful: Lemma 10. It holds that: DISPLAYFORM17 Proof. For the first claim, C(K) is lower bounded as: DISPLAYFORM18 Alternatively, C(K) can be lower bounded as: DISPLAYFORM19 which proves the second claim. The next lemma bounds the one step progress of Gauss-Newton. Lemma 11. Suppose that: DISPLAYFORM0 −1 E K. Using Lemma 9 and the condition on η, DISPLAYFORM1 where the last step uses Lemma 8.With this lemma, the proof of the convergence rate of the Gauss Newton algorithm is immediate. Proof. (of Theorem 5, Gauss-Newton case) The theorem is due to that η = 1 leads to a contraction of 1 − ηµ Σ K * at every step. The next lemma bounds the one step progress of the natural policy gradient. Lemma 12. Suppose: DISPLAYFORM0. It holds that: DISPLAYFORM1 Proof. Since K = K − ηE K, Lemma 9 implies: DISPLAYFORM2 The last term can be bounded as: DISPLAYFORM3 Continuing and using the condition on η, DISPLAYFORM4 using Lemma 8.With this lemma, the proof of the natural policy gradient convergence rate can be completed. Proof. (of Theorem 5, natural policy gradient case) Using Lemma 10, DISPLAYFORM5 The proof is completed by induction: DISPLAYFORM6, since Lemma 12 can be applied. The proof proceeds by arguing that Lemma 12 can be applied at every step. If it were the case that DISPLAYFORM7 and by Lemma 12: DISPLAYFORM8 which completes the proof. As informally argued by Equation 13, the proof seeks to quantify how Σ K changes with η. Then the proof bounds the one step progress of gradient descent. DISPLAYFORM0 This subsections aims to prove the following: Lemma 13. (Σ K perturbation) Suppose K is such that: DISPLAYFORM1 It holds that: DISPLAYFORM2 The proof proceeds by starting with a few technical lemmas. First, define a linear operator on symmetric matrices, T K (·), which can be viewed as a matrix on d+1 2dimensions. Define this operator on a symmetric matrix X as follows: DISPLAYFORM3 Also define the induced norm of T as follows: DISPLAYFORM4 where the supremum is over all symmetric matrices X (whose spectral norm is non-zero).Also, define DISPLAYFORM5 Proof. For a unit norm vector v ∈ R d and unit spectral norm matrix X, DISPLAYFORM6 The proof is completed using the upper bound on Σ K in Lemma 10.Also, with respect to K, define another linear operator on symmetric matrices: DISPLAYFORM7 Let I to denote the identity operator on the same space. Define the induced norm · of these operators as in Equation 14. Note these operators are related to the operator T K as follows:Lemma 15. When (A − BK) has spectral radius smaller than 1, DISPLAYFORM8 Proof. When (A − BK) has spectral radius smaller than 1, T K is well defined and is the solution of DISPLAYFORM9 Since, DISPLAYFORM10 The proof of Lemma 13 seeks to bound: DISPLAYFORM11 The following two perturbation bounds are helpful in this. Lemma 16. It holds that: DISPLAYFORM12 Proof. Let ∆ = K − K. For every matrix X, DISPLAYFORM13 The operator norm of F K − F K is the maximum possible ratio in spectral norm of (F K − F K)(X) and X. Then the claim follows because AX ≤ A X. DISPLAYFORM14 Proof. Define A = I − F K, and DISPLAYFORM15 Observe: DISPLAYFORM16. DISPLAYFORM17 and so DISPLAYFORM18 This proves the main inequality. The last step of the inequality is just applying definition of the norm of DISPLAYFORM19 With these Lemmas, the proof is completed as follows: Proof. (of Lemma 13) First, the proof shows T K F K − F K ≤ 1/2, which is the desired condition in Lemma 17. First, observe that under the assumed condition on K − K, implies that DISPLAYFORM20 using that DISPLAYFORM21 ≤ 1 due to Lemma 10. Using Lemma 16, DISPLAYFORM22 Using this and Lemma 14, DISPLAYFORM23 where the last step uses the condition on K − K.Thus, DISPLAYFORM24 using Lemmas 10 and 16. Equipped with these lemmas, the one step progress of gradient descent can be bounded. Lemma 18. Suppose that DISPLAYFORM0 It holds that: DISPLAYFORM1 Proof. By Lemma 9, DISPLAYFORM2 where the last step uses Lemma 8.By Lemma 13, DISPLAYFORM3 using the assumed condition on η. Using this last claim and Lemma 10, DISPLAYFORM4 using the condition on η. In order to prove a gradient descent convergence rate, the following bounds are helpful: Lemma 19. It holds that DISPLAYFORM5 and that: DISPLAYFORM6 Proof. Using Lemma 10, DISPLAYFORM7 By Lemma 8, DISPLAYFORM8 µ which proves the first claim. Again using Lemma 8, DISPLAYFORM9 which proves the second claim. With these lemmas, the proof of the gradient descent convergence rate follows:Proof. (of Theorem 5, gradient descent case) First, the following argues that progress is made at t = 1. Based on Lemma 10 and Lemma 19, by choosing η to be an appropriate polynomial in DISPLAYFORM10, σ min (Q) and µ, the stepsize condition in Equation 16 is satisfied. Hence, by Lemma 18, DISPLAYFORM11 which implies that the cost decreases at t = 1. Proceeding inductively, now suppose that C(K t) ≤ C(K 0), then the stepsize condition in Equation 16 is still satisfied (due to the use of C(K 0) in bounding the quantities in Lemma 19). Thus, Lemma 18 can again be applied for the update at time t + 1 to obtain: DISPLAYFORM12 ≤ ε, and the follows. This section shows how techniques from zeroth order optimization allow the algorithm to run in the model-free setting with only black-box access to a simulator. The dependencies on various parameters are not optimized, and the notation h is used to represent different polynomial factors in the relevant factors (DISPLAYFORM0 µσmin(Q), A, B, R, 1/σ min (R)). When the polynomial also depend on dimension d or accuracy 1/, this is specified as parameters (h FIG2).The section starts by showing how the infinite horizon can be approximated with a finite horizon. This section shows that as long as there is an upper bound on C(K), it is possible to approximate both C(K) and Σ(K) with any desired accuracy. Lemma 20. For any K with finite C(K), let Σ DISPLAYFORM0 Proof. First, the bound on Σ K is proved. Define the operators T K and F K as in Section C.4, observe DISPLAYFORM1, this follows immediately from the form of F K (X) = (A + BK)X(A + BK). If X is PSD then W XW is also PSD for any W. Now, since tr(DISPLAYFORM2 (Here the last step is by Lemma 10), and all traces are nonnegative, then there must exists j < such that tr(DISPLAYFORM3 Therefore as long as DISPLAYFORM4, it follows that: DISPLAYFORM5 Here the first step is again because of all the terms are PSD, so using more terms is always better. The last step follows because F j (Σ K) is also a PSD matrix so the spectral norm is bounded by trace. In fact, it holds that tr(DISPLAYFORM6 Therefore if DISPLAYFORM7 The next lemma show that the function value and its gradient are approximate preserved if a small perturbation to the policy K is applied. Lemma 21. (C K perturbation) Suppose K is such that: DISPLAYFORM0, K then: DISPLAYFORM1 As in the proof of Lemma 16, the assumption implies that T K F K − F K ≤ 1/2, and, from Equation 15, that DISPLAYFORM2 Hence, DISPLAYFORM3 For the first term, DISPLAYFORM4 Combining the two terms completes the proof. The next lemma shows the gradient is also stable after perturbation. Lemma 22. (∇C K perturbation) Suppose K is such that: DISPLAYFORM5 Let's first look at the second term. By Lemma 8, DISPLAYFORM6 then by Lemma 13 DISPLAYFORM7 Therefore the second term is bounded by DISPLAYFORM8 By the previous lemma, DISPLAYFORM9 Since K ≤ 2 K, and K can be bounded by C(K) (Lemma 19), all the terms can be bounded by polynomials of related parameters multiplied by K − K. This section analyzes the smoothing procedure and completes the proof of gradient descent. Although Gaussian smoothing is more standard, the objective C(K) is not finite for every K, therefore technically E u∼N (0,σ 2 I) [C(K + u)] is not well defined. This is avoidable by smoothing in a ball. Let S r represent the uniform distribution over the points with norm r (boundary of a sphere), and B r represent the uniform distribution over all points with norm at most r (the entire sphere). When applying these sets to matrix a U, the Frobenius norm ball is used. The algorithm performs gradient descent on the following function DISPLAYFORM0 The next lemma uses the standard technique (e.g. in BID15) to show that the gradient of C r (K) can be estimated just with an oracle for function value. DISPLAYFORM1 This is the same as Lemma 2.1 in BID15, for completeness the proof is provided below. Proof. By Stokes formula, DISPLAYFORM2 By definition, DISPLAYFORM3 Under review as a conference paper at ICLR 2018Also, DISPLAYFORM4 The Lemma follows from combining these equations, and use the fact that DISPLAYFORM5 From the lemma above and standard concentration inequalities, it is immediate that it suffices to use a polynomial number of samples to approximate the gradient. Lemma 24. Given an, there are fixed polynomials h r (1/), h sample (d, 1/) such that when r ≤ 1/h r (1/), with m ≥ h sample (d, 1/) samples of U 1,..., U n ∼ S r, with high probability (at least DISPLAYFORM6 is also close to ∇C(K) with high probability. Proof. For the first part, the difference is broken into two terms: DISPLAYFORM7 For the first term, choose h r (1/) = min{1/r 0, 2h grad /} (r 0 is chosen later). By Lemma 22 when r is smaller than 1/h r (1/) = /2h grad, every point u on the sphere have ∇C(K + U) − ∇C(K) ≤ /4. Since ∇C r (K) is the expectation of ∇C(K + U), by triangle inequality ∇C r (K) − ∇C(K) ≤ /2.The proof also makes sure that r ≤ r 0 such that for any U ∼ S r, it holds that C(K + U) ≤ 2C(K). By Lemma 21, 1/r 0 is a polynomial in the relevant factors. Adding these two terms and apply triangle inequality gives the . For the second part, the proof breaks it into more terms. Let ∇ be equal to The third term is just what was bounded earlier, using h r,trunc (1/) = h r (2/) and making sure h sample,trunc (d, 1/) ≥ h sample (d, 2/), this guarantees that it is smaller than /2.For the second term, choose ≥ Therefore,∇ − ∇ is again a sum of independent vectors with bounded norm, so by Vector Bernstein's inequality, when h sample,trunc (d, 1/, L 2 /µ) is a large enough polynomial, ∇ − ∇ ≤ /4 with high probability. Adding all the terms finishes the proof. Theorem 25. There are fixed polynomials h GD,r (1/), h GD,sample (d, 1/, L 2 /µ), h GD, (d, 1/) such that if every step the gradient is computed as Lemma 24 (truncated at step), pick step size η and T the same as the gradient descent case of Theorem 5, it holds that C(K T) − C(K) ≤ with high probability (at least 1 − exp(−d)). Before the Theorem for natural gradient is proven, the following lemma shows the variance can be estimated accurately. satisfies Σ − Σ K ≤. Further, when ≤ µ/2, it holds that σ min (Σ K) ≥ µ/2.Proof. This is broken into three terms: let Σ Therefore, standard concentration bounds show that when h varsample,trunc is a large enough polynomial, Σ −Σ ≤ /2 holds with high probability. Adding these three terms gives the . Finally, the bound on σ min (Σ K) follows simply from Weyl's Theorem. Theorem 27. Suppose C(K 0) is finite and and µ > 0. The natural gradient follows the update rule: DISPLAYFORM0 Suppose the stepsize is set to be: DISPLAYFORM1 If the gradient and variance are estimated as in Lemma 24, Lemma 26 with r = 1/h N GD,r (1/), with m ≥ h N GD,sample (d, 1/, L 2 /µ) samples, both are truncated to h N GD, (d, 1/) iterations, then with high probability (at least 1 − exp(−d)) in T iterations where DISPLAYFORM2 µσ min (R) log 2(C(K 0) − C(K *)) ε then the natural gradient satisfies the following performance bound: DISPLAYFORM3

This paper shows that model-free policy gradient methods can converge to the global optimal solution for non-convex linearized control problems.

This paper provides a rigorous study of the variance reduced TD learning and characterizes its advantage over vanilla TD learning


Question: Represent the article for finding a single-sentence TLDR

Question: Low-dimensional vector embeddings, computed using LSTMs or simpler techniques, are a popular approach for capturing the “meaning” of text and a form of unsupervised learning useful for downstream tasks. However, their power is not theoretically understood. The current paper derives formal understanding by looking at the subcase of linear embedding schemes. Using the theory of compressed sensing we show that representations combining the constituent word vectors are essentially information-preserving linear measurements of Bag-of-n-Grams (BonG) representations of text. This leads to a new theoretical about LSTMs: low-dimensional embeddings derived from a low-memory LSTM are provably at least as powerful on classification tasks, up to small error, as a linear classifier over BonG vectors, a that extensive empirical work has thus far been unable to show. Our experiments support these theoretical findings and establish strong, simple, and unsupervised baselines on standard benchmarks that in some cases are state of the art among word-level methods. We also show a surprising new property of embeddings such as GloVe and word2vec: they form a good sensing matrix for text that is more efficient than random matrices, the standard sparse recovery tool, which may explain why they lead to better representations in practice. Much attention has been paid to using LSTMs BID15 and similar models to compute text embeddings BID3 BID7. Once trained, the LSTM can sweep once or twice through a given piece of text, process it using only limited memory, and output a vector with moderate dimensionality (a few hundred to a few thousand), which can be used to measure text similarity via cosine similarity or as a featurization for downstream tasks. The powers and limitations of this method have not been formally established. For example, can such neural embeddings compete with and replace traditional linear classifiers trained on trivial Bag-of-n-Grams (BonG) representations? Tweaked versions of BonG classifiers are known to be a surprisingly powerful baseline and have fast implementations BID17. They continue to give better performance on many downstream supervised tasks such as IMDB sentiment classification BID21 than purely unsupervised LSTM representations BID19 BID13 BID25. Even a very successful character-level (and thus computation-intensive, taking a month of training) approach does not reach BonG performance on datasets larger than IMDB BID31. Meanwhile there is evidence suggesting that simpler linear schemes give compact representations that provide most of the benefits of word-level LSTM embeddings (; BID1 . These linear schemes consist of simply adding up, with a few modifications, standard pretrained word embeddings such as GloVe or word2vec BID24 BID29 .The current paper ties these disparate threads together by giving an information-theoretic account of linear text embeddings. We describe linear schemes that preserve n-gram information as lowdimensional embeddings with provable guarantees for any text classification task. The previous linear schemes, which used unigram information, are subcases of our approach, but our best schemes can also capture n-gram information with low additional overhead. Furthermore, we show that the original unigram information can be (approximately) extracted from the low-dimensional embedding using sparse recovery/compressed sensing BID6. Our approach also fits in the tradition of the older work on distributed representations of structured objects, especially the works of BID30 and BID18. The following are the main achieved by this new world-view:1. Using random vectors as word embeddings in our linear scheme (instead of pretrained vectors) already allows us to rigorously show that low-memory LSTMs are provably at least as good as every linear classifier operating on the full BonG vector. This is a novel theoretical in deep learning, obtained relatively easily. By contrast, extensive empirical study of this issue has been inconclusive (apart from character-level models, and even then only on smaller datasets BID31). Note also that empirical work by its nature can only establish performance on some available datasets, not on all possible classification tasks. We prove this theorem in Section 4 by providing a nontrivial generalization of a combining compressed sensing and learning BID5 ). In fact, before our work we do not know of any provable quantification of the power of any text embedding.2. We study theoretically and experimentally how our linear embedding scheme improves when it uses pretrained embeddings (GloVe etc.) instead of random vectors. Empirically we find that this improves the ability to preserve Bag-of-Words (BoW) information, which has the following restatement in the language of sparse recovery: word embeddings are better than random matrices for "sensing" BoW signals (see Section 5). We give some theoretical justification for this surprising finding using a new sparse recovery property characterizing when nonnegative signals can be reconstructed by 1 -minimization.3. Section 6 provides empirical supporting the above theoretical work, reporting accuracy of our linear schemes on multiple standard classification tasks. Our embeddings are consistently competitive with recent and perform much better than all previous linear methods. Among unsupervised word-level representations they achieve state of the art performance on both the binary and fine-grained SST sentiment classification tasks BID33. Since our document representations are fast, compositional, and simple to implement given standard word embeddings, they provide strong baselines for future work. Neural text embeddings are instances of distributed representations, long studied in connectionist approaches because they decay gracefully with noise and allow distributed processing. BID14 provided an early problem formulation, and BID30 provided an elementary solution, the holographic distributed representation, which represents structured objects using circular vector convolution and has an easy and more compact implementation using the fast Fourier transform (FFT). Plate suggested applying such ideas to text, where "structure" can be quantified using parse trees and other graph structures. Our method is also closely related in form and composition to the sparse distributed memory system of BID18. In the unigram case our embedding reduces to the familiar sum of word embeddings, which is known to be surprisingly powerful , and with a few modifications even more so BID1.Representations of BonG vectors have been studied through the lens of compression by BID28, who computed representations based on classical lossless compression algorithms using a linear program (LP). Their embeddings are still high-dimensional (d > 100K) and quite complicated to implement. In contrast, linear projection schemes are simpler, more compact, and can leverage readily available word embeddings. BID25 also used a linear scheme, representing documents as an average of learned word and bigram embeddings. However, the motivation and benefits of encoding BonGs in low-dimensions are not made explicit. The novelty in the current paper is the connection to compressed sensing, which is concerned with recovering high-dimensional sparse signals x ∈ R N from low-dimensional linear measurements Ax, specifically by studying conditions on matrix A ∈ R d×N when this is possible (see Appendix A for some on compressed sensing and the previous work of BID5 that we build upon). In this section we define the two types of representations that our analysis will relate:1. high-dimensional BonG vectors counting the occurrences of each k-gram for k ≤ n 2. low-dimensional embeddings, from simple vector sums to novel n-gram-based embeddings Although some of these representations have been previously studied and used, we define them so as to make clear their connection via compressed sensing, i.e. that representations of the second type are simply linear measurements of the first. We now define some notation. Let V be the number of words in the vocabulary and V n be the number of n-grams (independent of word order), so that V = V 1. Furthermore set V sum n = k≤n V k and V max n = max k≤n V k. We will use words/n-grams and indices interchangeably, e.g. if (a, b) is the ith of V 2 bigrams then the one-hot vector e (a,b) will be 1 at index i. Where necessary we will use {,} to denote a multi-set and (,) to denote a tuple. For any m vectors v i ∈ R d for i = 1,..., m we define [v 1, . . ., v m] to be their concatenation, which is thus an element of R md. Finally, for any subset X ⊂ R N we denote by ∆X the set {x − x : x, x ∈ X}. Assigning to each word a unique index i ∈ [V] we define the Bag-of-Words (BoW) representation x BoW of a document to be the V -dimensional vector whose ith entry is the number of times word i occurs in the document. The n-gram extension of BoW is the Bag-of-n-Grams (BonG) representation, which counts the number of times any k-gram for k ≤ n appears in a document. Linear classification over such vectors has been found to be a strong baseline .For ease of analysis we simplify the BonG approach by merging all n-grams in the vocabulary that contain the same words but in a different order. We call these features n-cooccurrences and find that the modification does not affect performance significantly (see Table 3 in Appendix F.1). Formally for a document w 1,..., w T we define the Bag-of-n-Cooccurrences (BonC) vector as the concatenation DISPLAYFORM0 e wt,..., DISPLAYFORM1 which is thus a V sum n -dimensional vector. Note that for unigrams this is equivalent to the BoW vector. Now suppose each word w has a vector v w ∈ R d for some d V. Then given a document w 1,..., w T we define its unigram embedding as z u = T t=1 v wt. While this is a simple and widely used featurization, we focus on the following straightforward relation with BoW: if A ∈ R d×V is a matrix whose columns are word vectors v w then Ax BoW = T t=1 Ae wt = T t=1 v wt = z u. Thus in terms of compressed sensing the unigram embedding of a document is a d-dimensional linear measurement of its Bag-of-Words vector. We could extend this unigram embedding to n-grams by first defining a representation for each ngram as the tensor product of the vectors of its constituent words. Thus for each bigram b = (w 1, w 2) we would have v b = v w1 v T w2 and more generally v g = n t=1 v wt for each n-gram g = (w 1, . . ., w n). The document embedding would then be the sum of the tensor representations of all n-grams. The major drawback of this approach is of course the blowup in dimension, which in practice prevents its use beyond n = 2. To combat this a low-dimensional sketch or projection of the tensor product can be used, such as the circular convolution operator of BID30. Since we are interested in representations that can also be constructed by an LSTM, we instead sketch this tensor product using the element-wise multiplication operation, which we find also usually works better than circular convolution in practice (see Table 4 in Appendix F.1). Thus for the n-cooccurrence g = {w 1, . . ., w n}, we define the distributed cooccurrence (DisC) embeddingṽ g = d n−1 2 n t=1 v wt. The coefficient is required when the vectors v w are random and unit norm to ensure that the product also has close to unit norm (see Lemma B.1). In addition to their convenient form, DisC embeddings have nice theoretical and practical properties: they preserve the original embedding dimension, they reduce to unigram (word) embeddings for n = 1, and under mild assumptions they satisfy useful compressed sensing properties with overwhelming probability (Lemma 4.1).We then define the DisC document embedding to be the nd-dimensional weighted concatenation, over k ≤ n, of the sum of the DisC vectors of all k-grams in a document: DISPLAYFORM0 Here scaling factors C k are set so that all spans of d coordinates have roughly equal norm (for random embeddings C k = 1; for word embeddings C k = 1/k works well). Note that sinceṽ wt = v wt we have z = z u in the unigram case. Furthermore, as with unigram embeddings by comparing and one can easily construct a DISPLAYFORM1 As discussed previously, LSTMs have become a common way to apply the expressive power of RNNs, with success on a variety of classification, representation, and sequence-to-sequence tasks. For document representation, starting with h 0 = 0 m an m-memory LSTM initialized with word vectors v w ∈ R d takes in words w 1,..., w T one-by-one and computes the document representation DISPLAYFORM0 where h t ∈ R m is the hidden representation at time t, the forget gate f, input gate i, and input function g are a.e. differentiable nondecreasing elementwise "activation" functions R m → R m, and affine transformations T * (x, y) = W * x + U * y + b * have weight matrices W * ∈ R m×d, U * ∈ R m×m and bias vectors b * ∈ R m. The LSTM representation of a document is then the state at the last time step, i.e. z LSTM = h T. Note that we will follow the convention of using LSTM memory to refer to the dimensionality of the hidden states. Since the LSTM is initialized with an embedding for each word it requires O(m 2 + md + V d) computer memory, but the last term is just a lookup table so the vocabulary size does not factor into iteration or representation complexity. From our description of LSTMs it is intuitive to see that one can initialize the gates and input functions so as to construct the DisC embeddings defined in the previous section. We state this formally and give the proof in the unigram case (the full proof appears in Appendix B.3): Proposition 3.1. Given word vectors v w ∈ R d, one can initialize an O(nd)-memory LSTM that takes in words w 1,..., w T (padded by an end-of-document token assigned vector 0 d) and constructs the DisC embedding (up to zero padding), i.e. such that for all documents z LSTM = z (n). DISPLAYFORM1 By Proposition 3.1 we can construct a fixed LSTM that can compute compressed BonC representations on the fly and be further trained by stochastic gradient descent using the same memory. Our main contribution is to provide the first rigorous analysis of the performance of the text embeddings that we are aware of, showing that the embeddings of Section 3.2 can provide performance on downstream classification tasks at least as well any linear classifier over BonCs. Before stating the theorem we make two mild simplifying assumptions on the BonC vectors:1. The vectors are scaled by DISPLAYFORM0, where T is the maximum document length. This assumption is made without loss of generality.2. No n-cooccurrence contains a word more than once. While this is (infrequently) violated in practice, the problem can be circumvented by merging words as a preprocessing step. DISPLAYFORM1 (1 − γ)(1 − 2δ) the classifierŵ minimizing the 2 -regularized logistic loss over its representations satisfies DISPLAYFORM2 The above theoretical bound shows that LSTMs match BonC performance as ε → 0, which can be realized by increasing the embedding dimension d (c.f. FIG7). Compressed sensing is concerned with recovering a high-dimensional k-sparse signal x ∈ R N from a few linear measurements; given a design matrix A ∈ R d×N this is formulated as DISPLAYFORM0 where z = Ax is the measurement vector. As l 0 -minimization is NP-hard, research has focused on sufficient conditions for tractable recovery. One such condition is the Restricted Isometry Property (RIP), for which BID6 proved that can be solved by convex relaxation: DISPLAYFORM1 We will abuse notation and say (k, ε)-RIP when X is the set of k-sparse vectors. This is the more common definition, but ours allows a more general Theorem 4.2 and a tighter bound in Theorem 4.1. DISPLAYFORM2 If is a λ-Lipschitz convex loss function and w 0 ∈ R N is its minimizer over D then w.p. 1 − 2δ the linear classifierŵ A ∈ R d minimizing the 2 -regularized empirical loss function DISPLAYFORM3 for appropriate choice of C. Recall that ∆X = {x − x : x, x ∈ X} for any X ⊂ R N.While a detailed proof of this theorem is spelled out in Appendix C, the main idea is to compare the distributional loss incurred by a classifierŵ in the original space to the loss incurred by Aŵ in the compressed space. We show that the minimizer of the regularized empirical loss in the original space (ŵ) is a bounded-coefficient linear combination of samples in S, so its loss depends only on inner products between points in X. Thus using RIP and a generalization error by BID34 we can bound the loss ofŵ A, the regularized classifier in the compressed domain. Note that to get back from Theorem 4.2 the O(√ ε) bound for k-sparse inputs of BID5 we can set X to the be the set of k-sparse vectors and assume A is (2k, ε)-RIP. To apply Theorem 4.2 we need the design matrix A (n) transforming BonCs into the DisC embeddings of Section 3.2 to satisfy the following RIP condition (Lemma 4.1), which we prove using a restricted isometry for structured random sampling matrices in Appendix D: Lemma 4.1. Assume the setting of Theorem 4.1 and let A (n) be the nd × V sum n matrix relating DisC and BonC representations of any document by DISPLAYFORM0 is the set of BonCs of documents of length at most T.Proof of Theorem 4. DISPLAYFORM1 is the set of BonC vectors of documents of length at most T. By BonC assumption all BonCs lie within the unit ball, so we can apply Theorem 4.2 with the logistic loss, λ = 1, and R = 1 to get that a classifierŵ trained using 2 -regularized logistic loss overŜ will satisfy the required bound. Since by Proposition 3.1 one can DISPLAYFORM2 T, this completes the proof. Theorem 4.1 is proved using random vectors as the word embeddings in the scheme of Section 3. However, in practice LSTMs are often initialized with standard word vectors such as GloVe. Such embeddings cannot satisfy traditional compressed sensing properties such as RIP or incoherence. This follows essentially from the definition: word embeddings seek to capture word similarity, so similar words (e.g. synonyms) have embeddings with high inner product, which violates both properties. Thus the efficacy of real-life LSTMs must have some other explanation. But in this section we present the surprising empirical finding that pretrained word embeddings are more efficient than random vectors at encoding and recovering BoW information via compressed sensing. We further sketch a potential explanation for this , though a rigorous explanation is left for subsequent work. In recent years word embeddings have been discovered to have many remarkable properties, most famously the ability to solve analogies BID24. Our connection to compressed sensing indicates that they should have another: preservation of sparse signals as low-dimensional linear measurements. To examine this we subsample documents from the SST BID33 and IMDB BID21 classification datasets, embed them as d-dimensional unigram embeddings z = Ax for d = 50, 100, 200,..., 1600 (where A ∈ R d×V is the matrix of word embeddings and x is a document's BoW vector), solve the following LP, known as Basis Pursuit (BP), which is the standard 1 -minimization problem for sparse recovery in the noiseless case (see Appendix A): DISPLAYFORM0 Success is measured as the F 1 score of retrieved words. We use Squared Norm (SN) vectors BID0 trained on a corpus of Amazon reviews BID23 and normalized i.i.d. Rademacher vectors as a baseline. SN is used due to similarity to GloVe and its formulation via an easy-to-analyze generative model that may provide a framework to understand the (see Appendix F.2), while the Amazon corpus is used for its semantic closeness to the sentiment datasets. compared to dimension. Pretrained word embeddings (SN trained on Amazon reviews) need half the dimensionality of normalized Rademacher vectors to achieve near-perfect recovery. Note that IMDB documents are on average more than ten times longer than SST documents. Figures 1 and 2 show that pretrained embeddings require a lower dimension d than random vectors to recover natural language BoW. This is surprising as the training objective goes against standard conditions such as approximate isometry and incoherence; indeed as shown in FIG3 recovery is poor for randomly generated word collections. The latter outcome indicates that the fact that a document is a set of mutually meaningful words is important for sparse recovery using embeddings trained on co-occurrences. We achieve similar with other objectives (e.g. GloVe/word2vec) and other corpora (see Appendix F.1), although there is some sensitivity to the sparse recovery method, as other 1 -minimization methods work well but greedy methods, such as Orthogonal Matching Pursuit (OMP), work poorly, likely due to their dependence on incoherence BID36.For the n-gram case (i.e. BonC recovery for n > 1), although we know by Lemma 4.1 that DisC embeddings composed from random vectors satisfy RIP, for pretrained vectors it is unclear how to reason about suitable n-gram embeddings without a rigorous understanding of the unigram case, and experiments do not show the same recovery benefits. One could perhaps do well by training on cooccurrences of word tuples, but such embeddings could not be used by a word-level LSTM. As shown in FIG3, the success of pretrained embeddings for linear sensing is a local phenomenon; recovery is only efficient for naturally occurring collections of words. However, applying statistical RIP/incoherence ideas BID2 to explain this is ruled out since they require collections to be incoherent with high probability, whereas word embeddings are trained to give high inner product to words appearing together. Thus an explanation must come from some other, weaker condition. The usual necessary and sufficient requirement for recovering all signals with support S ⊂ [N] is the local nullspace property (NSP), which stipulates that vectors in the kernel of A not have too much mass on S (see Definition A.2). While NSP and related properties such as restricted eigenvalue (see Definition A.3) are hard to check, we can impose some additional structure to formulate an intuitive, verifiable perfect recovery condition for our setting. Specifically, since our signals (BoW vectors) are nonnegative, we can improve upon solving BP by instead solving nonnegative BP (BP+): minimize w 1 subject to Aw = z, w ≥ 0 d The following geometric then characterizes when solutions of BP+ recover the correct signal: Theorem 5.1 BID9. Consider a matrix A ∈ R d×N and an index subset S ⊂ [N] of size k. Then any nonnegative vector x ∈ R N + with support supp(x) = S is recovered from Ax by BP+ iff the set A S of columns of A indexed by S comprise the vertices of a k-dimensional face of the convex hull conv(A) of the columns of A together with the origin. This theorem equates perfect recovery of a BoW vector via BP+ with the vectors of its words being the vertices of some face of the polytope conv(A). The property holds for incoherent columns since the vectors are far enough that no one vector is inside the simplex formed by any k others. On the other hand, pretrained embeddings satisfy it by having commonly co-occurring words close together and other words far away, making it easier to form a face from columns indexed by the support of a BoW. We formalize this intuition as the Supporting Hyperplane Property (SHP): SHP is a very weak property implied by NSP (Corollary E.1). However, it can be checked by using convex optimization to see if the hyperplane exists (Appendix E.2). Furthermore, we show (full proof in Appendix E.1) that this hyperplane is the supporting hyperplane of the face of conv(A) with vertices A S, from which it follows by Theorem 5.1 that SHP characterizes recovery using BP+: Corollary 5.1. BP+ recovers any x ∈ R N + with supp(x) = S from Ax iff A satisfies S-SHP.Proof Sketch. By Theorem 5.1 it suffices to show equivalence of S-SHP with the column set A S comprising the vertices of a k-dimensional face of conv(A). A face F of polytope P is defined as its intersection with some hyperplane such that all points in P \F lie on one side of the hyperplane. (=⇒) Let F be the face of conv(A) formed by the columns A S. Then there must be a supporting hyperplane H containing F. Since the columns of A are in general position, all columns A S = A\A S lie in conv(A)\F and hence must all be on one side of H, so H is the desired hyperplane. (⇐=) Let H be the hyperplane supporting A S, with all other columns on one side of H. By convexity, H contains the simplex F of A S. Any point in conv(A)\F can be written as a convex combination of points in F and columns A S, with a positive coefficient on at least one of the columns, and so must lie on the same side of H as A S. Thus A S comprises the vertices of a face F.Thus perfect recovery of a BoW via BP+ is equivalent to the existence of a hyperplane separating embeddings of words in the document from those of the rest of the vocabulary. Intuitively, words in the same document are trained to have similar embeddings and so will be easier to separate out, providing some justification for why pretrained vectors are better for sensing. We verify that SHP is indeed more likely to be satisfied by such designs in FIG4, which also serves as an empirical check of Corollary 5.1 since SHP satisfaction implies BP recovery as the latter can do no better than BP+. We further compare to recovery using OMP/OMP+ (the latter removes negative values and recomputes the set of atoms at each iteration); interestingly, while OMP+ recovers the correct signal from SN almost as often as BP/BP+, it performs quite poorly for GloVe, indicating that these embeddings may have quite different sensing properties despite similar training objectives. As similarity properties that may explain these also relate to downstream task performance, we conjecture a relationship between embeddings, recovery, and classification that may be understood under a generative model (see Appendix F.2). However, the Section 4 bounds depend on RIP, not recovery, so these experiments by themselves do not apply. They do show that the compressed sensing framework remains relevant even in the case of non-random, pretrained word embeddings. BID1 Reported performance of best hyperparameter using Amazon GloVe embeddings. BID31 shown for comparison. The top three for each dataset are bolded, the best is italicized, and the best word-level performance is underlined. Our theoretical show that simple tensor product sketch-based n-gram embeddings can approach BonG performance and be computed by a low-memory LSTM. In this section we compare these text representations and others on several standard tasks, verifying that DisC performance approaches that of BonCs as dimensionality increases and establishing several baselines for text classification. Code to reproduce is provided at https://github.com/NLPrinceton/text_embedding. We test classification on MR movie reviews BID27, CR customer reviews BID16, SUBJ subjectivity dataset BID26, MPQA opinion polarity subtask , TREC question classification BID20, SST sentiment classification (binary and fine-grained) BID33, and IMDB movie reviews BID21. The first four are evaluated using 10-fold cross-validation, while the others have train-test splits. In all cases we use logistic regression with 2 -regularization determined by cross-validation. We further test DisC on the SICK relatedness and entailment tasks BID22 and the MRPC paraphrase detection task BID8. The inputs here are sentences pairs (a, b) and the standard featurization for document embeddings x a and x b of a and b is et al., 2015). We use logistic regression for SICK entailment and MRPC and use ridge regression to predict similarity scores for SICK relatedness, with 2 -regularization determined by cross-validation. Since BonGs are not used for pairwise tasks our theory says nothing about performance here; we include these evaluations to show that our representations are also useful for other tasks. DISPLAYFORM0 Embeddings: In the main evaluation TAB1 we use normalized 1600-dimensional GloVe embeddings BID29 ) trained on the Amazon Product Corpus BID23, which are released at http://nlp.cs.princeton.edu/DisC. We also compare the SN vectors of Section 5 trained on the same corpus with random vectors when varying the dimension FIG7 ). Results: We find that DisC representation performs consistently well relative to recent unsupervised methods; among word-level approaches it is the top performer on the SST tasks and competes on many others with skip-thoughts and CNN-LSTM, both concatenations of two LSTM representations. While success may be explained by training on a large and in-domain corpus, being able to use so much text without extravagant computing resources is one of the advantages of a simple approach. Overall our method is useful as a strong baseline, often beating BonCs and many more complicated approaches while taking much less time to represent and train on documents than neural representations FIG6 ).Finally, we analyze empirically how well our model approximates BonC performance. As predicted by Theorem 4.1, the performance of random embeddings on IMDB approaches that of BonC as dimension increases and the isometry distortion ε decreases (FIG7). Using pretrained (SN) vectors, DisC embeddings approach BonC performance much earlier, surpassing it in the unigram case. In this paper we explored the connection between compressed sensing, learning, and natural language representation. We first related LSTM and BonG methods via word embeddings, coming up with simple new document embeddings based on tensor product sketches. Then we studied their classification performance, proving a generalization of the compressed learning of BID5 to convex Lipschitz losses and a bound on the loss of a low-dimensional LSTM classifier in terms of its (modified) BonG counterpart, an issue which neither experiments nor theory have been able to resolve. Finally, we showed how pretrained embeddings fit into this sparse recovery framework, demonstrating and explaining their ability to efficiently preserve natural language information. A COMPRESSED SENSING The field of compressed sensing is concerned with recovering a high-dimensional k-sparse signal x ∈ R N from few linear measurements. In the noiseless case this is formulated as minimize w 0 subject to Aw = zwhere A ∈ R d×N is the design matrix and z = Ax is the measurement vector. Since 0 -minimization is NP-hard, a foundational approach is to use its convex surrogate, the 1 -norm, and characterize when the solution to is equivalent to that of the following LP, known as basis pursuit (BP): DISPLAYFORM0 Related approaches such as Basis Pursuit Denoising (LASSO) and the Dantzig Selector generalize BP to handle signal or measurement noise BID11; however, the word embeddings case is noiseless so these methods reduce to BP. Note that throughout Section 5 and the Appendix we say that an 1 -minimization method recovers x from Ax if its optimal solution is unique and equivalent to the optimal solution of.An alternative way to approximately solve FORMULA1 is to use a greedy algorithm such as matching pursuit (MP) or orthogonal matching pursuit (OMP), which pick basis vectors one at a time by multiplying the measurement vector by A T and choosing the column with the largest inner product BID36. One condition through which recovery can be guaranteed is the Restricted Isometry Property (RIP): DISPLAYFORM0 A line of work started by BID6 used the RIP property to characterize matrices A such that FORMULA1 and FORMULA1 have the same minimizer for any k-sparse signal x; this occurs with overwhelming probability when d = Ω k log N k and DISPLAYFORM1 Since the ability to recover a signal x from a representation Ax implies information preservation, a natural next step is to consider learning after compression. BID5 DISPLAYFORM2 and a (2k, ε)-RIP matrix A, the hinge loss of a classifier trained on {( DISPLAYFORM3 is bounded by that of the best linear classifier over the original samples. Theorem 4.2 provides a generalization of this to any convex Lipschitz loss function. RIP is a strong requirement, both because it is not necessary for perfect, stable recovery of k-sparse vectors usingÕ(k) measurements and because in certain settings we are interested in using the above ideas to recover specific signals -those statistically likely to occur-rather than all k-sparse signals. The usual necessary and sufficient condition to recover any vector x ∈ R N with index support set S ⊂ [N] is the local nullspace property (NSP), which is implied by RIP: Definition A.2 BID11. A matrix A ∈ R d×N satisfies NSP for a set S ⊂ [N] if w S 1 < w S 1 for all nonzero w ∈ ker(A) = {v : Av = 0 d}.Theorem A.1 BID11. BP recovers any x ∈ R N + with supp(x) = S from Ax iff A satisfies NSP for S.A related condition that implies NSP is the local restricted eigenvalue property (REP): Definition A.3 BID32 DISPLAYFORM4 Lastly, a simple condition that can sometimes provide recovery guarantees is mutual incoherence: Definition A.4. A ∈ R d×N is µ-incoherent if max a,a |a T a | ≤ µ, where the maximum is taken over any two distinct columns a, a of A.While incoherence is easy to verify (unlike the previous recovery properties), word embeddings tend to have high coherence due to the training objective pushing together vectors of co-occurring words. Apart from incoherence, the properties above are hard show empirically. However, we are compressing BoW vectors, so our signals are nonnegative and we can impose an additional constraint on FORMULA1 The polytope condition is equivalent to nonnegative NSP (NSP+), a weaker form of NSP: Definition A.5 BID10. DISPLAYFORM0 w i > 0 for all nonzero w ∈ ker(A).Lemma A.1. If A ∈ R d×N satisfies NSP for some S ⊂ [N] then it also satisfies NSP+ for S.Proof (Adapted from BID10). Since A satisfies NSP, we have w S 1 < w S 1. Then for a nonzero w ∈ ker(A) such that w S ≥ 0 we will have DISPLAYFORM1 Lemma A.2. BP+ recovers any x ∈ R N + with supp(x) = S from Ax iff A satisfies NSP+ for S. For any nonzero w ∈ ker(A) such that w S ≥ 0, ∃ λ > 0 such that x + λw ≥ 0 N and A(x + λw) = Ax. Since BP+ uniquely recovers x, we have x + λw 1 > x 1, so NSP+ follows from the following inequality and the fact that λ is positive: DISPLAYFORM0 For any x ≥ 0 such that Ax = Ax we have that w = x − x ∈ ker(A) and w S = x S ≥ 0 since the support of x is S. Thus by NSP+ we have that DISPLAYFORM1 w i > 0, which yields DISPLAYFORM2 Thus BP+ will recover x uniquely. Lemma A.2 shows that NSP+ is equivalent to the polytope condition in Theorem A.2, as they are both necessary and sufficient conditions for BP+ recovery. Table 3: The performance of an l 2 -regularized logit classifier over Bag-of-n-Grams (BonG) vectors is generally similar to that of Bag-of-n-Cooccurrences (BonC) vectors for n = 2, 3 (largest differences bolded). Evaluation settings are the same as in Section 6. Note that for unigrams the two representations are equivalent. Table 4: Performance comparison of element-wise product (DisC) and circular convolution for encoding local cooccurrences (best for each task is bolded). Evaluation settings are the same as in Section 6. Note that for unigrams the two representations are equivalent. In this section we compare the performance of several alternative representations with the ones presented in the main evaluation TAB1. Table 3 provides a numerical justification for our use of unordered n-grams (cooccurrences) instead of n-grams, as the performance of the two featurizations are closely comparable. In Table 4 we examine the use of circular convolution instead of elementwise multiplication as linear measurements of BonC vectors BID30. To construct the former from a document w 1,..., w T we compute DISPLAYFORM0 where F is the discrete Fourier transform and F −1 its inverse. Note that for n = 1 this is equivalent to the simple unigram embedding (and thus also to the DisC embedding in). d} then for any n-gram g = (w 1, . . ., w n) we have E ṽ g 2 2 = 1. The same holds true with the additional assumption that all words in g are distinct if the word vectors are i.i.d. d-dimensional spherical Gaussians. Proof. DISPLAYFORM0 Substituting these parameters into the LSTM update and using h 0 = 0 we have ∀ t > 0 that DISPLAYFORM1... DISPLAYFORM2, by padding the end of the document with an end-of-document token whose word vector is 0 d the entries in those dimensions will be set to zero by the update at the last step. Thus up to zero padding we will have z LSTM = h T =z (n).C PROOF OF THEOREM 4.2Throughout this section we assume the setting described in Theorem 4.2. Furthermore for some positive constant C define the 2 -regularization of the loss function as DISPLAYFORM3, where (·, ·) is a convex λ-Lipschitz function in the first cordinate. Then DISPLAYFORM4 where |α i | ≤ λC m ∀ i. This holds in the compressed domain as well. Proof. If is an λ-Lipschitz function, its sub-gradient at every point is bounded by λ. So by convexity, the unique optimizer is given by taking first-order conditions: DISPLAYFORM5 Since is Lipschitz, |∂ w T xi (w T x i, y i)| ≤ λ. Therefore the first-order optimal solution ofŵ can be expressed as FORMULA1 for some α 1,..., α m satisfying |α i | ≤ λC m ∀ i, which is the desired . DISPLAYFORM6 Also since 0 N ∈ X, A is also (X, ε)-RIP and the then follows by the same argument as in (, Lemma 4.2-3). DISPLAYFORM7 Proof. The first bound follows by expanding ŵ 2 2 and using x 2 ≤ R; the second follows by expanding ŵ A 2 2, applying Lemma C.2 to bound inner product distortion, and using x 2 ≤ R. Lemma C.3. Letŵ be the linear classifier minimizing L S. Then DISPLAYFORM8 Proof. By Lemma C.1 we can re-expressŵ using Equation 14 and then apply the inequality from Lemma C.2 to get DISPLAYFORM9 for any x ∈ R N. Since is λ-Lipschitz taking expectations over D implies DISPLAYFORM10 Substituting Equation 14 applying Lemma C.2 also yields DISPLAYFORM11 Together the inequalities bounding the loss term and the regularization term FORMULA1 imply the . Lemma C.4. Letŵ be the linear classifier minimizing L S and let w * be the linear classifier minimizing L D. Then with probability 1 − γ DISPLAYFORM12 This holds in the compressed domain as well. Proof. By Corollary C.1 we have thatŵ is contained in a closed convex subset independent of S. Therefore since is λ-Lipschitz, L is 1 C -strongly convex, and x 2 ≤ O(R), we have by BID34, Theorem 1) that with probability 1 − γ DISPLAYFORM13, which substituted into the previous equation completes the proof. Proof of Theorem 4.2. Applying Lemma C.4 in the compressed domain yields DISPLAYFORM14, so together with Lemma C.3 and the previous inequality we have DISPLAYFORM15 We now apply Lemma C.4 in the sparse domain to get DISPLAYFORM16 2, so by the previous inequality we have DISPLAYFORM17 Substituting the C that minimizes the r.h.s. of this inequality completes the proof. D PROOF OF LEMMA 4.1We assume the setting described in Lemma 4.1, where we are concerned with the RIP condition of the matrix A (n) when multiplying vectors x ∈ X (n)T, the set of BonC vectors for documents of length at most T. This matrix can be written as DISPLAYFORM18 where A p is the d × V p matrix whose columns are the DisC embeddings of all p-grams in the vocabulary (and thus A = A 1 = A, the matrix of the original word embeddings). Note that from any x ∈ X (n) T can be written as x = [x 1, . . ., x n], where x p is a T -sparse vector whose entries correspond to p-grams. Thus we also have DISPLAYFORM19 Proof. By union bound we have that A p is (2k, ε)-RIP ∀ p ∈ [n] with probability at least 1 − nγ. Thus by Definition 4.1 we have w. DISPLAYFORM20 Similarly, DISPLAYFORM21. From Definition 4.1, taking the square root of both sides of both inequalities completes the proof. Definition D.1 BID11. Let D be a distribution over a subset S ⊂ R n. Then the set Φ = {φ 1, . . ., φ N} of functions φ i: S → R is a bounded orthonormal system (BOS) with constant B if we have E D (φ i φ j) = 1 i=j ∀ i, j and sup s∈S |φ i (s)| ≤ B ∀ i. Note that by definition B ≥ 1. DISPLAYFORM22 Proof. Note that by Theorem D.1 it suffices to show that √ dA p is a random sampling matrix associated with a BOS with constant B = 1. Let D = U V {±1} be the uniform distribution over V i.i.d. Rademacher random variables indexed by words in the vocabulary. Then by definition the matrix A p ∈ R d×Vp can be constructed by drawing random variables DISPLAYFORM23 and assigning to the ijth entry of DISPLAYFORM24 gt, where each function φ j: {±1} V → R is uniquely associated to its p-gram. It remains to be shown that this set of functions is a BOS with constant B = 1.For any two p-grams g, g and their functions φ i, φ j we have E D (φ i φ j) = E x∼D p t=1 x gt x g t, which will be 1 iff each word in g ∪ g occurs an even number of times in the product and 0 otherwise. Because all p-grams are uniquely defined under any permutation of its words (i.e. we are in fact using p-cooccurrences) and we have assumed that no p-gram contains a word more than once, each word occurs an even number of times in the product iff g = g ⇐⇒ i = j. Furthermore we have that DISPLAYFORM25 V ∀ i by construction. Thus according to Definition D.1 the set of functions {φ 1, . . ., φ Vp} associated to the p-grams in the vocabulary is a BOS with constant B = 1.Proof of Lemma 4.1. DISPLAYFORM26. Applying Lemma D.1 yields the . In Section 5.2, Definition 5.1 we introduced the Supporting Hyperplane Property (SHP), which by Corollary 5.1 characterizes when BP+ perfectly recovers a nonnegative signal. Together with Lemmas A.1 and A.2 this fact also shows that SHP is a weaker condition than the well-known nullspace property (NSP): Corollary E.1. If a matrix A ∈ R d×N with columns in general position satisfies NSP for some S ⊂ [N] then it also satisfies S-SHP.In this section we give the entire proof of Corollary 5.1 and describe how to verify SHP given a design matrix and a set of support indices. E.1 PROOF OF COROLLARY 5.1Recall that it suffices to show equivalence of A being S-SHP with the columns A S forming the vertices of a k-dimensional face of conv(A), where we can abuse notation to set A ∈ R d×(N +1), with the extra column being the origin 0 d, so long as we constrain N + 1 ∈ S.(=⇒): The proof of the forward direction appeared in full in the proof sketch (see Section 5.2). DISPLAYFORM0 We also know that F = {i∈S λ i A i : λ ∈ ∆ |S|} ⊆ H by convexity of H. Since any point y ∈ conv(A)\F can be written as y = DISPLAYFORM1 ∈ S such that λ j = 0, we have that DISPLAYFORM2 This implies that conv(A)\F ⊆ H − and F = conv(A) ∩ H, so since the columns of A are in general position F is a k-dimensional face of conv(A) whose vertices are the columns A S. Recall that a matrix R d×N satisfies S-SHP for S ⊂ [N] if there is a hyperplane containing the set of all columns of A indexed by S and the set of all other columns together with the origin are on one side of it. Due to Corollary 5.1, checking S-SHP allows us to know whether all nonnegative signals with index support S will be recovered by BP+ without actually running the optimization on any one of them. The property can be checked by solving a convex problem of the form The constraint enforces the property that the hyperplane contains all support embeddings, while the optimal objective value is zero iff SHP is satisfied (this follows from the fact that scaling h does not affect the constraint so if the minimal objective is zero for any single ε > 0 it is zero for all ε > 0). The problem can be solved via using standard first or second-order equality-constrained convex optimization algorithms. We set ε = 1 and p = 3 (to get a C 2 objective) and adapt the second-order method from Boyd & Vandenberghe (2004, Chapter 10). Our implementation can be found at https://github.com/NLPrinceton/sparse_recovery. Figure 7: Efficiency of pretrained embeddings as sensing vectors at d = 300 dimensions, measured via the F 1 -score of the original BoW. 200 documents from each dataset were compressed and recovered in this experiment. For fairness, the number of words V is the same for all embeddings so all documents are required to be subsets of the vocabulary of all corpora. word2vec embeddings trained on Google News and GloVe vectors trained on Common Crawl were obtained from public repositories BID24 BID29 while Amazon and Wikipedia embeddings were trained for 100 iterations using a symmetric window of size 10, a min count of 100, for SN/GloVe a cooccurrence cutoff of 1000, and for word2vec a down-sampling frequency cutoff of 10 −5 and a negative example setting of 3. 300-dimensional normalized random vectors are used as a baseline. We show in Figure 7 that the surprising effectiveness of word embeddings as linear measurement vectors for BoW signals holds for other embedding objectives and corpora as well. Specifically, we see that widely used embeddings, when normalized, match the efficiency of random vectors for retrieving SST BoW and are more efficient when retrieving IMDB BoW. Interestingly, SN vectors are most efficient and are also the only embeddings for normalizing is not needed for good performance. In Section 5.2 we gave some intuition for why pretrained word embeddings are efficient sensing vectors for natural language BoW by examining a geometric characterization of local equivalence due to BID9 in light of the usual similarity properties of word embeddings. However, this analysis does not provide a rigorous theory for our empirical . In this section we briefly discuss a model-based justification that may lead to a stronger understanding. We need a model relating BoW generation to the word embeddings trained over words co-occurring in the same BoW. As a starting point consider the model of BID0, in which a corpus is generated by a random walk c t over the surface of a ball in R d; at each t a word w is emitted w.p. DISPLAYFORM0 Minimizing the SN objective approximately maximizes corpus likelihood under this model. Thus in an approximate sense a document of length T is generated by setting a context vector c and emitting T words via with c t = c. This model is a convenient one for analysis due its simplicity and invariance to word order as well as the fact that the approximate maximum likelihood document vector is the sum of the embeddings of words in the document. Building upon the intuition established following Corollary 5.1 one can argue that, if we have the true latent SN vectors, then embeddings of words in the same document (i.e. emitted by the same context vector) will be close to each other and thus easy to separate from the embeddings of other words in the vocabulary. However, we find empirically that not all of the T words closest to the sum of the word embeddings (i.e. the context vector) are the ones emitted; indeed individual word vectors in a document may have small, even negative inner product with the context vector and still be recovered via BP. Thus any further theoretical argument must also be able to handle the recovery of lower probability words whose vectors are further away from the context vector than those of words that do not appear in the document. We thus leave to future work the challenge of explaining why embeddings ing from this (or another) model provide such efficient sensing matrices for natural language BoW.

Represent the short summary for finding the corresponding scientific article (long)

We use the theory of compressed sensing to prove that LSTMs can do at least as well on linear text classification as Bag-of-n-Grams.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.