Question: represent the text to find a very short summary (1-2 sentences max) E.g. Value-based methods constitute a fundamental methodology in planning and deep reinforcement learning (RL). In this paper, we propose to exploit the underlying structures of the state-action value function, i.e., Q function, for both planning and deep RL. In particular, if the underlying system dynamics lead to some global structures of the Q function, one should be capable of inferring the function better by leveraging such structures. Specifically, we investigate the low-rank structure, which widely exists for big data matrices. We verify empirically the existence of low-rank Q functions in the context of control and deep RL tasks (Atari games). As our key contribution, by leveraging Matrix Estimation (ME) techniques, we propose a general framework to exploit the underlying low-rank structure in Q functions, leading to a more efficient planning procedure for classical control, and additionally, a simple scheme that can be applied to any value-based RL techniques to consistently achieve better performance on''low-rank'' tasks. Extensive experiments on control tasks and Atari games confirm the efficacy of our approach. Value-based methods are widely used in control, planning and reinforcement learning (; ;). To solve a Markov Decision Process (MDP), one common method is value iteration, which finds the optimal value function. This process can be done by iteratively computing and updating the state-action value function, represented by Q(s, a) (i.e., the Q-value function). In simple cases with small state and action spaces, value iteration can be ideal for efficient and accurate planning. However, for modern MDPs, the data that encodes the value function usually lies in thousands or millions of dimensions (; 2019), including images in deep reinforcement learning . These practical constraints significantly hamper the efficiency and applicability of the vanilla value iteration. Yet, the Q-value function is intrinsically induced by the underlying system dynamics. These dynamics are likely to possess some structured forms in various settings, such as being governed by partial differential equations. In addition, states and actions may also contain latent features (e.g., similar states could have similar optimal actions). Thus, it is reasonable to expect the structured dynamic to impose a structure on the Q-value. Since the Q function can be treated as a giant matrix, with rows as states and columns as actions, a structured Q function naturally translates to a structured Q matrix. In this work, we explore the low-rank structures. To check whether low-rank Q matrices are common, we examine the benchmark Atari games, as well as 4 classical stochastic control tasks. As we demonstrate in Sections 3 and 4, more than 40 out of 57 Atari games and all 4 control tasks exhibit low-rank Q matrices. This leads us to a natural question: How do we leverage the low-rank structure in Q matrices to allow value-based techniques to achieve better performance on "low-rank" tasks? We propose a generic framework that allows for exploiting the low-rank structure in both classical planning and modern deep RL. Our scheme leverages Matrix Estimation (ME), a theoretically guaranteed framework for recovering low-rank matrices from noisy or incomplete measurements . In particular, for classical control tasks, we propose Structured Value-based Planning (SVP). For the Q matrix of dimension |S| × |A|, at each value iteration, SVP randomly updates a small portion of the Q(s, a) and employs ME to reconstruct the remaining elements. We show that planning problems can greatly benefit from such a scheme, where much fewer samples (only sample around 20% of (s, a) pairs at each iteration) can achieve almost the same policy as the optimal one. For more advanced deep RL tasks, we extend our intuition and propose Structured Value-based Deep RL (SV-RL), applicable for any value-based methods such as DQN . Here, instead of the full Q matrix, SV-RL naturally focuses on the "sub-matrix", corresponding to the sampled batch of states at the current iteration. For each sampled Q matrix, we again apply ME to represent the deep Q learning target in a structured way, which poses a low rank regularization on this "sub-matrix" throughout the training process, and hence eventually the Q-network's predictions. Intuitively, as learning a deep RL policy is often noisy with high variance, if the task possesses a low-rank property, this scheme will give a clear guidance on the learning space during training, after which a better policy can be anticipated. We confirm that SV-RL indeed can improve the performance of various value-based methods on "low-rank" Atari games: SV-RL consistently achieves higher scores on those games. Interestingly, for complex, "high-rank" games, SV-RL performs comparably. ME naturally seeks solutions that balance low rank and a small reconstruction error (cf. Section 3.1). Such a balance on reconstruction error helps to maintain or only slightly degrade the performance for "high-rank" situation. We summarize our contributions as follows: • We are the first to propose a framework that leverages matrix estimation as a general scheme to exploit the low-rank structures, from planning to deep reinforcement learning. • We demonstrate the effectiveness of our approach on classical stochastic control tasks, where the low-rank structure allows for efficient planning with less computation. • We extend our scheme to deep RL, which is naturally applicable for any value-based techniques. Across a variety of methods, such as DQN, double DQN, and dueling DQN, experimental on all Atari games show that SV-RL can consistently improve the performance of value-based methods, achieving higher scores for tasks when low-rank structures are confirmed to exist. To motivate our method, let us first investigate a toy example which helps to understand the structures within the Q-value function. We consider a simple deterministic MDP, with 1000 states, 100 actions and a deterministic state transition for each action. The reward r(s, a) is randomly generated first for each (s, a) pair, and then fixed throughout. A discount factor γ = 0.95 is used. The deterministic nature imposes a strong relationship among connected states. In this case, our goal is to explore: what kind of structures the Q function may contain; and how to effectively exploit such structures. The Low-rank Structure Under this setting, Q-value could be viewed as a 1000 × 100 matrix. To probe the structure of the Q-value function, we perform the standard Q-value iteration as follows: where s denotes the next state after taking action a at state s. We randomly initialize Q. In Fig. 1, we show the approximate rank of Q (t) and the mean-square error (MSE) between Q (t) and the optimal Q *, during each value iteration. Here, the approximate rank is defined as the first k singular values (denoted by σ) that capture more than 99% variance of all singular values, i.e., k i=1 σ 2 i / j σ 2 j ≥ 0.99. As illustrated in Fig. 1 (a) and 1(b), the standard theory guarantees the convergence to Q *; more interestingly, the converged Q * is of low rank, and the approximate rank of Q (t) drops fast. These observations give a strong evidence for the intrinsic low dimensionality of this toy MDP. Naturally, an algorithm that leverages such structures would be much desired. The previous motivate us to exploit the structures for efficient planning. The idea is simple: If the eventual matrix is low-rank, why not enforcing such a structure throughout the iterations? In other words, with the existence of a global structure, we should be capable of exploiting it during intermediate updates and possibly also regularizing the to be in the same low-rank space. In particular, at each iteration, instead of every (s, a) pair (i.e., Eq.), we would like to only calculate Q (t+1) for some (s, a) pairs and then exploit the low-rank structure to recover the whole Q (t+1) matrix. We choose matrix estimation (ME) as our reconstruction oracle. The reconstructed matrix is often with low rank, and hence regularizing the Q matrix to be low-rank as well. We validate this framework in Fig. 1 (c) and 1(d), where for each iteration, we only randomly sample 50% of the (s, a) pairs, calculate their corresponding Q (t+1) and reconstruct the whole Q (t+1) matrix with ME. Clearly, around 40 iterations, we obtain comparable to the vanilla value iteration. Importantly, this comparable performance only needs to directly compute 50% of the whole Q matrix at each iteration. It is not hard to see that in general, each vanilla value iteration incurs a computation cost of O(|S| 2 |A| 2). The complexity of our method however only scales as O(p|S| 2 |A| 2)+O M E, where p is the percentage of pairs we sample and O M E is the complexity of ME. In general, many ME methods employ SVD as a subroutine, whose complexity is bounded by O(min{|S| 2 |A|, |S||A| 2}) . For low-rank matrices, faster methods can have a complexity of order linear in the dimensions . In other words, our approach improves computational efficiency, especially for modern high-dimensional applications. This overall framework thus appears to be a successful technique: it exploits the low-rank behavior effectively and efficiently when the underlying task indeed possesses such a structure. Having developed the intuition underlying our methodology, we next provide a formal description in Sections 3.1 and 3.2. One natural question is whether such structures and our method are general in more realistic control tasks. Towards this end, we provide further empirical support in Section 3.3. ME considers about recovering a full data matrix, based on potentially incomplete and noisy observations of its elements. Formally, consider an unknown data matrix X ∈ R n×m and a set of observed entries Ω. If the observations are incomplete, it is often assumed that each entry of X is observed independently with probability p ∈. In addition, the observation could be noisy, where the noise is assumed to be mean zero. Given such an observed set Ω, the goal of ME is to produce an estimatorM so that ||M − X|| ≈ 0, under an appropriate matrix norm such as the Frobenius norm. The algorithms in this field are rich. Theoretically, the essential message is: exact or approximate recovery of the data matrix X is guaranteed if X contains some global structure (Candès & ; ;). In the literature, most attention has been focusing on the low-rank structure of a matrix. Correspondingly, there are many provable, practical algorithms to achieve the desired recovery. Early convex optimization methods (Candès &) seek to minimize the nuclear norm, ||M || *, of the estimator. For example, fast algorithms, such as the Soft-Impute algorithm solves the following minimization problem: Since the nuclear norm || · || * is a convex relaxation of the rank, the convex optimization approaches favor solutions that are with small reconstruction errors and in the meantime being relatively low-rank, which are desirable for our applications. Apart from convex optimization, there are also spectral methods and even non-convex optimization approaches (; ;). In this paper, we view ME as a principled reconstruction oracle to effectively exploit the low-rank structure. For faster computation, we mainly employ the Soft-Impute algorithm. We now formally describe our approach, which we refer as structured value-based planning (SVP). Fig. 2 illustrates our overall approach for solving MDP with a known model. The approach is based on the Q-value iteration. At the t-th iteration, instead of a full pass over all state-action pairs: State-action value function Incomplete observation Reconstructed value function Figure 2: An illustration of the proposed SVP algorithm for leveraging low-rank structures. 1. SVP first randomly selects a subset Ω of the state-action pairs. In particular, each state-action pair in S × A is observed (i.e., included in Ω) independently with probability p. 2. For each selected (s, a), the intermediateQ(s, a) is computed based on the Q-value iteration: 3. The current iteration then ends by reconstructing the full Q matrix with matrix estimation, from the set of observations in Ω. That is, Q (t+1) = ME {Q(s, a)} (s,a)∈Ω. Overall, each iteration reduces the computation cost by roughly 1 − p (cf. discussions in Section 2). In Appendix A, we provide the pseudo-code and additionally, a short discussion on the technical difficulty for theoretical analysis. Nevertheless, we believe that the consistent empirical benefits, as will be demonstrated, offer a sounding foundation for future analysis. We empirically evaluate our approach on several classical stochastic control tasks, including the Inverted Pendulum, the Mountain Car, the Double Integrator, and the Cart-Pole. Our objective is to demonstrate, as in the toy example, that if the optimal Q * has a low-rank structure, then the proposed SVP algorithm should be able to exploit the structure for efficient planning. We present the evaluation on Inverted Pendulum, and leave additional on other planning tasks in Appendix B and C. Inverted Pendulum In this classical continuous task, our goal is to balance an inverted pendulum to its upright position, and the performance metric is the average angular deviation. The dynamics is described by the angle and the angular speed, i.e., s = (θ,θ), and the action a is the torque applied. We discretize the task to have 2500 states and 1000 actions, leading to a 2500 × 1000 Q-value matrix. The Low-rank Structure We first verify that the optimal Q * indeed contains the desired low-rank structure. We run the vanilla value iteration until it converges. The converged Q matrix is found to have an approximate rank of 7. For further evidence, in Appendix C, we construct "low-rank" policies directly from the converged Q matrix, and show that the policies maintain the desired performance. The SVP Policy Having verified the structure, we would expect our approach to be effective. To this end, we apply SVP with different observation probability p and fix the overall number of iterations to be the same as the vanilla Q-value iteration. Fig. 3 confirms the success of our approach. Fig. 3(a), 3(b) and 3(c) show the comparison between optimal policy and the final policy based on SVP. We further illustrate the performance metric, the average angular deviation, in Fig. 3(d). Overall, much fewer samples are needed for SVP to achieve a comparable performance to the optimal one. So far, our focus has been on tabular MDPs where value iteration can be applied straightforwardly. However, the idea of exploiting structure is much more powerful: we propose a natural extension of our approach to deep RL. Our scheme again intends to exploit and regularize structures in the Q-value function with ME. As such, it can be seamlessly incorporated into value-based RL techniques that include a Q-network. We demonstrate this on Atari games, across various value-based RL techniques. Before diving into deep RL, let us step back and review the process we took to develop our intuition. Previously, we start by treating the Q-value as a matrix. To exploit the structure, we first verify that certain MDPs have essentially a low-rank Q *. We argue that if this is indeed the case, then enforcing the low-rank structures throughout the iterations, by leveraging ME, should lead to better algorithms. A naive extension of the above reasoning to deep RL immediately fails. In particular, with images as states, the state space is effectively infinitely large, leading to a tall Q matrix with numerous number of rows (states). Verifying the low-rank structure for deep RL hence seems intractable. However, by definition, if a large matrix is low-rank, then almost any row is a linear combination of some other rows. That is, if we sample a small batch of the rows, the ing matrix is most likely low-rank as well. To probe the structure of the deep Q function, it is, therefore, natural to understand the rank of a randomly sampled batch of states. In deep RL, our target for exploring structures is no longer the optimal Q *, which is never available. In fact, like SVP, the natural objective should be the converged values of the underlying algorithm, which in deep scenarios, are the eventually learned Q function. With the above discussions, we now provide evidence for the low-rank structure of learned Q function on some Atari games. We train standard DQN on 4 games, with a batch size of 32. To be consistent, the 4 games all have 18 actions. After the training process, we randomly sample a batch of 32 states, evaluate with the learned Q network and finally synthesize them into a matrix. That is, a 32 × 18 data matrix with rows the batch of states, the columns the actions, and the entries the values from the learned Q network. Note that the rank of such a matrix is at most 18. The above process is repeated for 10,000 times, and the histogram and empirical CDF of the approximate rank is plotted in Fig. 4. Apparently, there is a strong evidence supporting a highly structured low-rank Q function for those games -the approximate ranks are uniformly small; in most cases, they are around or smaller than 3. Having demonstrated the low-rank structure within some deep RL tasks, we naturally seek approaches that exploit the structure during the training process. We extend the same intuitions here: if eventually, the learned Q function is of low rank, then enforcing/regularizing the low rank structure for each iteration of the learning process should similarly lead to efficient learning and potentially better performance. In deep RL, each iteration of the learning process is naturally the SGD step where one would update the Q network. Correspondingly, this suggests us to harness the structure within the batch of states. Following our previous success, we leverage ME to achieve this task. We now formally describe our approach, referred as structured value-based RL (SV-RL). It exploits the structure for the sampled batch at each SGD step, and can be easily incorporated into any Q-value based RL methods that update the Q network via a similar step as in Q-learning. In particular, Q-value based methods have a common model update step via SGD, and we only exploit structure of the sampled batch at this step -the other details pertained to each specific method are left intact. Precisely, when updating the model via SGD, Q-value based methods first sample a batch of B transitions {(s and form the following updating targets: For example, in DQN,Q is the target network. The Q network is then updated by taking a gradient step for the loss function 2, with respect to the parameter θ. To exploit the structure, we then consider reconstructing a matrix Q † fromQ, via ME. The reconstructed Q † will replace the role ofQ in Eq. to form the targets y (i) for the gradient step. In particular, the matrix Q † has a dimension of B × |A|, where the rows represent the "next states" in the batch, the columns represent actions, and the entries are reconstructed values. Let S B {s . The SV-RL alters the SGD update step as illustrated in Algorithm 1 and sample a set Ω of state-action pairs from S B × A. In particular, each state-action pair in S B × A is observed (i.e., included in Ω) with probability p, independently. evaluate every state-action pair in Ω viaQ, whereQ is the network that would be used to form the targets {y in the original value-based methods (cf. Eq.). 6: based on the evaluated values, reconstruct a matrix Q † with ME, i.e., / * new targets with reconstructed Q † for the gradient step * / 8: replaceQ in Eq. with Q † to evaluate the targets {y SV-RL Targets: 9: update the Q network with the original targets replaced by the SV-RL targets. Note the resemblance of the above procedure to that of SVP in Section 3.2. When the full Q matrix is available, in Section 3.2, we sub-sample the Q matrix and then reconstruct the entire matrix. When only a subset of the states (i.e., the batch) is available, naturally, we look at the corresponding sub-matrix of the entire Q matrix, and seek to exploit its structure. We conduct extensive experiments on Atari 2600. We apply SV-RL on three representative value-based RL techniques, i.e., DQN, double DQN and dueling DQN. We fix the total number of training iterations and set all the hyper-parameters to be the same. For each experiment, averaged across multiple runs are reported. Additional details are provided in Appendix D. We present representative of SV-RL applied to the three value-based deep RL techniques in Fig. 6. These games are verified by method mentioned in Section 4.1 to be low-rank. Additional on all Atari games are provided in Appendix E. The figure reveals the following . First, games that possess structures indeed benefit from our approach, earning mean rewards that are strictly higher than the vanilla algorithms across time. More importantly, we observe consistent improvements across different value-based RL techniques. This highlights the important role of the intrinsic structures, which are independent of the specific techniques, and justifies the effectiveness of our approach in consistently exploiting such structures. Further Observations Interestingly however, the performance gains vary from games to games. Specifically, the majority of the games can have benefits, with few games performing similarly or slightly worse. Such observation motivates us to further diagnose SV-RL in the next section. So far, we have demonstrated that games which possess structured Q-value functions can consistently benefit from SV-RL. Obviously however, not all tasks in deep RL would possess such structures. As such, we seek to diagnose and further interpret our approach at scale. Diagnosis We select 4 representative examples (with 18 actions) from all tested games, in which SV-RL performs better on two tasks (i.e., FROSTBITE and KRULL), slightly better on one task (i.e., ALIEN), and slightly worse on the other (i.e., SEAQUEST). The intuitions we developed in Section 4 incentivize us to further check the approximate rank of each game. As shown in Fig. 7, in the two better cases, both games are verified to be approximately low-rank (∼ 2), while the approximate rank in ALIEN is moderately high (∼ 5), and even higher in SEAQUEST (∼ 10). Consistent Interpretations As our approach is designed to exploit structures, we would expect to attribute the differences in performance across games to the "strength" of their structured properties. Games with strong low-rank structures tend to have larger improvements with SV-RL (Fig. 7 (a) and 7(b)), while moderate approximate rank tends to induce small improvements (Fig. 7(c) ), and high approximate rank may induce similar or slightly worse performances (Fig. 7(d) ). The empirical are well aligned with our arguments: if the Q-function for the task contains low-rank structure, SV-RL can exploit such structure for better efficiency and performance; if not, SV-RL may introduce slight or no improvements over the vanilla algorithms. As mentioned, the ME solutions balance being low rank and having small reconstruction error, which helps to ensure a reasonable or only slightly degraded performance, even for "high rank" games. We further observe consistent on ranks vs. improvement across different games and RL techniques in Appendix E, verifying our arguments. We plot games where the SV-based method performs differently. More structured games (with lower rank) can achieve better performance with SV-RL. Structures in Value Function Recent work in the control community starts to explore the structure of value function in control/planning tasks (; ; ; 2018). These work focuses on decomposing the value function and subsequently operating on the reduced-order space. In spirit, we also explore the low-rank structure in value function. The central difference is that instead of decomposition, we focus on "completion". We seek to efficiently operate on the original space by looking at few elements and then leveraging the structure to infer the rest, which allows us to extend our approach to modern deep RL. In addition, while there are few attempts for basis function learning in high dimensional RL , functions are hard to generate in many cases and approaches based on basis functions typically do not get the same performance as DQN, and do not generalize well. In contrast, we provide a principled and systematic method, which can be applied to any framework that employs value-based methods or sub-modules. Value-based Deep RL Value-based methods are fundamental in deep RL, exemplified by DQN (; 2015). There has been a large body of literature on its variants, such as double DQN , dueling DQN , IQN and other techniques that improve exploration . Our approach focuses on general value-based RL methods. As long as the method has a similar model update step as in Q-learning, our approach can leverage the structure to help with the task. We empirically show that deep RL tasks that have structured value functions indeed benefit from our scheme. Matrix Estimation ME is the primary tool we leverage to exploit the low-rank structure in value functions. The techniques have been widely studied and applied to different domains (; ;), and recently even in robust deep learning . The field is relatively mature, with extensive algorithms and provable recovery guarantees for structured matrix . Because of the strong promise, we view ME as a principled reconstruction oracle to exploit the low-rank structures within matrices. We investigated the structures in value function, and proposed a complete framework to understand, validate, and leverage such structures in various tasks, from planning to deep reinforcement learning. The proposed SVP and SV-RL algorithms harness the strong low-rank structures in the Q function, showing consistent benefits for both planning tasks and value-based deep reinforcement learning techniques. Extensive experiments validated the significance of the proposed schemes, which can be easily embedded into existing planning and RL frameworks for further improvements. randomly sample a set Ω of observed entries from S × A, each with probability p 4: / * update the randomly selected state-action pairs * / 5: end for 8: / * reconstruct the Q matrix via matrix estimation * / 9: apply matrix completion to the observed values {Q(s, a)} (s,a)∈Ω to reconstruct Q (t+1): While based on classical value iteration, we remark that a theoretical analysis, even in the tabular case, is quite complex. Although the field of ME is somewhat mature, the analysis has been largely focused on the "one-shot" problem: recover a static data matrix given one incomplete observation. Under the iterative scenario considered here, standard assumptions are easily broken and the analysis warrants potentially new machinery. Furthermore, much of the effort in the ME community has been devoted to the Frobenius norm guarantees rather than the infinite norm as in value iteration. Non-trivial infinite norm bound has received less attention and often requires special techniques . Resolving the above burdens would be important future avenues in its own right for the ME community. Henceforth, this paper focuses on empirical analysis and more importantly, generalizing the framework successfully to modern deep RL contexts. As we will demonstrate, the consistent empirical benefits offer a sounding foundation for future analysis. Inverted Pendulum As stated earlier in Sec. 3.3, the goal is to balance the inverted pendulum on the upright equilibrium position. The physical dynamics of the system is described by the angle and the angular speed, i.e., (θ,θ). Denote τ as the time interval between decisions, u as the torque input on the pendulum, the dynamics can be written as : A reward function that penalizes control effort while favoring an upright pendulum is used: In the simulation, the state space is (−π, π] for θ and [−10, 10] forθ. We limit the input torque in [−1, 1] and set τ = 0.3. We discretize each dimension of the state space into 50 values, and action space into 1000 values, which forms an Q-value function a matrix of dimension 2500 × 1000. We follow to handle the policy of continuous states by modelling their transitions using multi-linear interpolation. Mountain Car We select another classical control problem, i.e., the Mountain Car , for further evaluations. In this problem, an under-powered car aims to drive up a steep hill . The physical dynamics of the system is described by the position and the velocity, i.e., (x,ẋ). Denote u as the acceleration input on the car, the dynamics can be written as The reward function is defined to encourage the car to get onto the top of the mountain at x 0 = 0.5: We follow standard settings to restrict the state space as [−0.07, 0.07] for x and [−1.2, 0.6] forẋ, and limit the input u ∈ [−1, 1]. Similarly, the whole state space is discretized into 2500 values, and the action space is discretized into 1000 values. The evaluation metric we are concerned about is the total time it takes to reach the top of the mountain, given a randomly and uniformly generated initial state. We consider the Double Integrator system , as another classical control problem for evaluation. In this problem, a unit mass brick moving along the x-axis on a frictionless surface, with a control input which provides a horizontal force, u . The task is to design a control system to regulate this brick to x = T. The physical dynamics of the system is described by the position and the velocity (i.e., (x,ẋ)), and can be derived as , we use the quadratic cost formulation to define the reward function, which regulates the brick to We follow standard settings to restrict the state space as [−3, 3] for both x andẋ, limit the input u ∈ [−1, 1] and set τ = 0.1. The whole state space is discretized into 2500 values, and the action space is discretized into 1000 values. Similarly, we define the evaluation metric as the total time it takes to reach to x = T, given a randomly and uniformly generated initial state. Cart-Pole Finally, we choose the Cart-Pole problem , a harder control problem with 4-dimensional state space. The problem consists a pole attached to a cart moving on a frictionless track. The cart can be controlled by means of a limited force within 10N that is possible to apply both to the left or to the right of the cart. The goal is to keep the pole on the upright equilibrium position. The physical dynamics of the system is described by the angle and the angular speed of the pole, and the position and the speed of the cart, i.e., (θ,θ, x,ẋ). Denote τ as the time interval between decisions, u as the force input on the cart, the dynamics can be written as x:= u + ml θ2 sin θ −θ cos θ where g = 9.8m/s 2 corresponds to the gravity acceleration, m c = 1kg denotes the mass of the cart, m = 0.1kg denotes the mass of the pole, l = 0.5m is half of the pole length, and u corresponds to the force applied to the cart, which is limited by u ∈ [−10, 10]. A reward function that favors the pole in an upright position, i.e., characterized by keeping the pole in vertical position between |θ| ≤ 12π 180, is expressed as r(θ) = cos 4 (15θ). In the simulation, the state space is [− .5] foṙ x. We limit the input force in [−10, 10] and set τ = 0.1. We discretize each dimension of the state space into 10 values, and action space into 1000 values, which forms an Q-value function a matrix of dimension 10000 × 1000. We further verify that the optimal Q * indeed contains the desired low-rank structure. To this end, we construct "low-rank" policies directly from the converged Q matrix. In particular, for the converged Q matrix, we sub-sample a certain percentage of its entries, reconstruct the whole matrix via ME, and finally construct a corresponding policy. Fig. 8 illustrates the , where the policy heatmap as well as the performance (i.e., the angular error) of the "low-rank" policy is essentially identical to the optimal one. The reveal the intrinsic strong low-rank structures lie in the Q-value function. We provide additional for the inverted pendulum problem. We show the policy trajectory (i.e., how the angle of the pendulum changes with time) and the input changes (i.e., how the input torque changes with time), for each policy. In Fig. 9, we first show the comparison between the optimal policy and a "low-rank" policy. Recall that the low-rank policies are directly reconstructed from the converged Q matrix, with limited observation of a certain percentage of the entries in the converged Q matrix. As shown, the "low-rank" policy performs nearly identical to the optimal one, in terms of both the policy trajectory and the input torque changes. This again verifies the strong low-rank structure lies in the Q function. Further, we show the policy trajectory and the input torque changes of the SVP policy. We vary the percentage of observed data for SVP, and present the policies with 20% and 60% for demonstration. As reported in Fig. 10, the SVP policies are essentially identical to the optimal one. Interestingly, when we further decrease the observing percentage to 20%, the policy trajectory vibrates a little bit, but can still stabilize in the upright position with a small average angular deviation ≤ 5 •. Similarly, we first verify the optimal Q * contains the desired low-rank structure. We use the same approach to generate a "low-rank" policy based on the converged optimal value function. Fig. 11 (a) and 11(b) show the policy heatmaps, where the reconstructed "low-rank" policy maintains visually identical to the optimal one. In Fig. 11(c) and 12, we quantitatively show the average time-to-goal, the policy trajectory and the input changes between the two schemes. Compared to the optimal one, even with limited sampled data, the reconstructed policy can maintain almost identical performance. We further show the of the SVP policy with different amount of observed data (i.e., 20% and 60%) in Fig. 13 and 14. Again, the SVP policies show consistently comparable to the optimal policy, over various evaluation metrics. Interestingly, the converged Q matrix of vanilla value iteration is found to have an approximate rank of 4 (the whole matrix is 2500 × 1000), thus the SVP can harness such strong low-rank structure for perfect recovery even with only 20% observed data. For the Double Integrator, We first use the same approach to generate a "low-rank" policy. Fig. 15(a) and 15(b) show that the reconstructed "low-rank" policy is visually identical to the optimal one. In Further, we show the of the SVP policy with different amount of observed data (i.e., 20% and 60%) in Fig. 17 and 18. As shown, the SVP policies show consistently decent , which demonstrates that SVP can harness such strong low-rank structure even with only 20% observed data. Finally, we evaluate SVP on the Cart-Pole system. Note that since the state space has a dimension of 4, the policy heatmap should contain 4 dims, but is hard to visualize. Since the metric we care is the angle deviation, we here only plot the first two dims (i.e., the (θ,θ) tuple) with fixed x andẋ, to visualize the policy heatmaps. We first use the same approach to generate a "low-rank" policy. Fig. 19(a) and 19(b) show the policy heatmaps, where the reconstructed "low-rank" policy is visually identical to the optimal one. In Fig. 19(c) and 20, we quantitatively show the average time-to-goal, the policy trajectory and the input changes between the two schemes. As demonstrated, the reconstructed policy can maintain almost identical performance with only small amount of sampled data. We finally show the of the SVP policy with different amount of observed data (i.e., 20% and 60%) in Fig. 21 and 22. Even for harder control tasks with higher dimensional state space, the SVP policies are still essentially identical to the optimal one. Across various stochastic control tasks, we demonstrate that SVP can consistently leverage strong low-rank structures for efficient planning. Training Details and Hyper-parameters The network architectures of DQN and dueling DQN used in our experiment are exactly the same as in the original papers (; ;). We train the network using the Adam optimizer . In all experiments, we set the hyper-parameters as follows: learning rate α = 1e −5, discount coefficient γ = 0.99, and a minibatch size of 32. The number of steps between target network updates is set to 10, 000. We use a simple exploration policy as the -greedy policy with the decreasing linearly from 1 to 0.01 over 3e 5 steps. For each experiment, we perform at least 3 independent runs and report the averaged . † formed by the current batch of states, we mainly employ the Soft-Impute algorithm throughout our experiments. We set the sub-sample rate to p = 0.9 of the Q matrix, and use a linear scheduler to increase the sampling rate every 2e 6 steps. Experiments across Various Value-based RL We show more across DQN, double DQN and dueling DQN in Fig. 23, 24, 25, 26, 27 and 28, respectively. For DQN, we complete all 57 Atari games using the proposed SV-RL, and verify that the majority of tasks contain low-rank structures (43 out of 57), where we can obtain consistent benefits from SV-RL. For each experiment, we associate the performance on the Atari game with its approximate rank. As mentioned in the main text, majority of the games benefits consistently from SV-RL. We note that roughly only 4 games, which have a significantly large rank, perform slightly worse than the vanilla DQN. Consistency and Interpretation Across all the experiments we have done, we observe that when the game possesses structures (i.e., being approximately low-rank), SV-RL can consistently improve the performance of various value-based RL techniques. The superior performance is maintained through most of the experiments, verifying the ability of the proposed SV-RL to harness the structures for better efficiency and performance in value-based deep RL tasks. In the meantime, when the approximate rank is relatively higher (e.g., SEAQUEST), the performance of SV-RL can be similar or worse than the vanilla scheme, which also aligns well with our intuitions. Note that the majority of the games have an action space of size 18 (i.e., rank is at most 18), while some (e.g., PONG) only have 6 or less (i.e., rank is at most 6). We provide an additional study on the Inverted Pendulum problem with respect to the discretization scale. As described in Sec. 3, the dynamics is described by the angle and the angular speed as s = (θ,θ), and the action a is the torque applied. To solve the task with value iteration, the state and action spaces need to be discretized into fine-grids. Previously, the two-dimensional state space was discretized into 50 equally spaced points for each dimension and the one-dimensional action space was evenly discretized into 1000 actions, leading to a 2500 × 1000 Q-value matrix. Here we choose three different discretization values for state-action pairs: 400 × 100, 2500 × 1000, and 10000 × 4000, to provide different orders of discretization for both state and action values. As Table 1 reports, the approximate rank is consistently low when discretization varies, demonstrating the intrinsic low-rank property of the task. Fig. 29 and Table 1 further demonstrates the effectiveness of SVP: it can achieve almost the same policy as the optimal one even with only 20% observations. The reveal that as long as the discretization is fine enough to represent the optimal policy for the task, we would expect the final Q matrix after value iteration to have similar rank. Figure 29: Additional study on discretization scale. We choose three different discretization value on the Inverted Pendulum task, i.e. 400 (states, 20 each dimension) × 100 (actions), 2500 (states, 50 each dimension) × 1000 (actions), and 10000 (states, 100 each dimension) × 4000 (actions). First row reports the optimal policy, second row reports the SVP policy with 20% observation probability. Table 1: Additional study on discretization scale (cont.). We report the approximate rank as well as the performance metric (i.e., the average angular deviation) on different discretization scales. To further understand our approach, we provide another study on batch size for games of different rank properties. Two games from Fig. 7 are investigated; one with a small rank (Frostbite) and one with a high rank (Seaquest). Different batch sizes, 32, 64, and 128, are explored and we show the in Fig. 30. Intuitively, for a learning task, the more complex the learning task is, the more data it would need to fully learn the characteristics. For a complex game with higher rank, a small batch size may not be sufficient to capture the game, leading the recovered matrix via ME to impose a structure that deviates from the original, more complex structure of the game. In contrast, with more data, i.e., a larger batch size, the ME oracle attempts to find the best rank structure that would effectively describe the rich observations and at the same time, balance the reconstruction error. Such a structure is more likely to be aligned with the underlying complex task. Indeed, this is what we observe in Fig. 30. As expected, for Seaquest (high rank), the performance is worse than the vanilla DQN when the batch size is small. However, as the batch size increases, the performance gap becomes smaller, and eventually, the performance of SV-RL is the same when the batch size becomes 128. On the other hand, for games with low rank, one would expect that a small batch size would be enough to explore the underlying structure. Of course, a large batch size would not hurt since the game is intrinsically low-rank. In other words, our intuition would suggest SV-RL to perform better across different batch sizes. Again, we observe this phenomenon as expected in Fig. 30. For Frostbite (low rank), under different batch sizes, vanilla DQN with SV-RL consistently outperforms vanilla DQN by a certain margin. Figure 30: Additional study on batch size. We select two games for illustration, one with a small rank (Frostbite) and one with a high rank (Seaquest). We vary the batch size with 32, 64, and 128, and report the performance with and without SV-RL. == We propose a generic framework that allows for exploiting the low-rank structure in both planning and deep reinforcement learning. != To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.

Question: Cloze test is widely adopted in language exams to evaluate students' language proficiency. In this paper, we propose the first large-scale human-designed cloze test dataset CLOTH in which the questions were used in middle-school and high-school language exams. With the missing blanks carefully created by teachers and candidate choices purposely designed to be confusing, CLOTH requires a deeper language understanding and a wider attention span than previous automatically generated cloze datasets. We show humans outperform dedicated designed baseline models by a significant margin, even when the model is trained on sufficiently large external data. We investigate the source of the performance gap, trace model deficiencies to some distinct properties of CLOTH, and identify the limited ability of comprehending a long-term context to be the key bottleneck. In addition, we find that human-designed data leads to a larger gap between the model's performance and human performance when compared to automatically generated data. Being a classic language exercise, the cloze test BID26 is an accurate assessment of language proficiency BID7 BID11 BID27 and has been widely employed in language examinations. Under standard setting, a cloze test requires examinees to fill in the missing word (or sentence) that best fits the surrounding context. To facilitate natural language understanding, automatically generated cloze datasets were introduced to measure the ability of machines in reading comprehension BID8 BID9 BID17. In these datasets, each cloze question typically consists of a context paragraph and a question sentence. By randomly replacing a particular word in the question sentence with a blank symbol, a single test case is created. For instance, the CNN/Daily Mail BID8 take news articles as the context and the summary bullet points as the question sentence. Only named entities are considered when creating the blanks. Similarly, in Children's Books test (CBT) BID9, the cloze question is obtained by removing a word in the last sentence of every consecutive 21 sentences, with the first 20 sentences being the context. Different from the CNN/Daily Mail datasets, CBT also provides each question with a candidate answer set, consisting of randomly sampled words with the same part-of-speech tag from the context as that of the ground truth. Thanks to the automatic generation process, these datasets can be very large in size, leading to significant research progress. However, compared to how humans would create cloze questions, the automatic generation process bears some inevitable issues. Firstly, the blanks are chosen uniformly without considering which aspect of the language phenomenon the question will test. Hence, quite a portion of automatically generated questions can be purposeless or even trivial to answer. Another issue involves the ambiguity of the answer. Given a context and a blanked sentence, there can be multiple words that fit almost equally well into the blank. A possible solution is to include a candidate option set, as done by CBT, to get rid of the ambiguity. However, automatically generating the candidate option set can be problematic since it cannot guarantee the ambiguity is removed. More importantly, automatically generated candidates can be totally irrelevant or simply grammatically unsuitable for the blank, ing in again trivial questions. Probably due to these unsatisfactory issues, it has been shown neural models have achieved comparable performance with human within very short time BID3 BID6 BID23. While there has been work trying to incorporate human design into cloze question generation BID30, the MSR Sentence Completion Challenge created by this effort is quite small in size, limiting the possibility of developing powerful neural models on it. Motivated by the aforementioned drawbacks, we propose CLOTH, a large-scale cloze test dataset collected from English exams. Questions in the dataset are designed by middle-school and highschool teachers to prepare Chinese students for entrance exams. To design a cloze test, teachers firstly determine the words that can test students' knowledge of vocabulary, reasoning or grammar; then replace those words with blanks and provide three candidate options for each blank. If a question does not specifically test grammar usage, all of the candidate options would complete the sentence with correct grammar, leading to highly confusing questions. As a , human-designed questions are usually harder and are a better assessment of language proficiency. Note that, different from the reading comprehension task, a general cloze test does not focus on testing reasoning abilities but evaluates several aspects of language proficiency including vocabulary, reasoning and grammar. To verify if human-designed cloze questions are difficult for current models, we train dedicated models as well as the state-of-the-art language model and evaluate their performance on this dataset. We find that the state-of-the-art model lags behind human performance even if the model is trained on a large external corpus. We analyze where the model fails compared to human. After conducting error analysis, we assume the performance gap from the model's inability to use long-term context. To verify this assumption, we evaluate humans' performance when they are only allowed to see one sentence as the context. Our assumption is confirmed by the matched performances of the model and human when given only one sentence. In addition, we demonstrate that human-designed data is more informative and more difficult than automatically generated data. Specifically, when the same amount of training data is given, human-designed training data leads to better performance. Additionally, it is much easier for the same model to perform well on automatically generated data. In this section, we introduce the CLOTH dataset that is collected from English examinations, and study the assessed abilities of this dataset. We collected the raw data from three free websites 2 in China that gather exams designed by English teachers. These exams are used to prepare students for college/high school entrance exams. Before cleaning, there are 20, 605 passages and 332, 755 questions. We perform the following processes to ensure the validity of the data: 1. We remove questions with an inconsistent format such as questions with more than four options; 2. We filter all questions whose validity relies on external information such as pictures or tables; 3. Further, we delete duplicated passages; 4. On one of the websites, the answers are stored as images. We use two OCR software, tesseract 3 and ABBYY FineReader 4, to extract the answers from images. We discard the question when from the two software are different. After the cleaning process, we obtain a dataset of 7, 131 passages and 99, 433 questions. Since high school questions are more difficult than middle school questions, we divided the datasets into CLOTH-M and CLOTH-H, which stand for the middle school part and the high school part. We split 11% of the data for both the test set and the dev set. The detailed statistics of the whole dataset and two subsets are presented in TAB1. In order to evaluate students' mastery of a language, teachers usually design tests so that questions cover different aspects of a language. Specifically, they first identity words in the passage that can Table 2.Passage: Nancy had just got a job as a secretary in a company. Monday was the first day she went to work, so she was very 1 and arrived early. She 2 the door open and found nobody there. " I am the 3 to arrive." She thought and came to her desk. She was surprised to find a bunch of 4 on it. They were fresh. She 5 them and they were sweet. She looked around for a 6 to put them in. " Somebody has sent me flowers the very first day!" she thought 7. " But who could it be?" she began to 8. The day passed quickly and Nancy did everything with 9 interest. For the following days of the 10, the first thing Nancy did was to change water for the followers and then set about her work. Then came another Monday. 11 she came near her desk she was overjoyed to see a(n) 12 bunch of flowers there. She quickly put them in the vase, 13 the old ones. The same thing happened again the next Monday. Nancy began to think of ways to find out the 14. On Tuesday afternoon, she was sent to hand in a plan to the 15. She waited for his directives at his secretary's 16. She happened to see on the desk a half-opened notebook, which 17: "In order to keep the secretaries in high spirits, the company has decided that every Monday morning a bunch of fresh flowers should be put on each secretarys desk." Later, she was told that their general manager was a business management psychologist. Questions: DISPLAYFORM0 A Table 2: A Sample passage from our dataset. The correct answers are highlighted. To understand the assessed abilities on this dataset, we divide questions into several types and label the proportion of each type of questions. We find that the questions can be divided into the following types:• Grammar: The question is about grammar usage, involving tense, preposition usage, active/passive voices, subjunctive mood and so on.• Short-term-reasoning: The question is about content words and can be answered based on the information within the same sentence.• Matching/paraphrasing: The question is answered by copying/paraphrasing a word.• Long-term-reasoning: The answer must be inferred from synthesizing information distributed across multiple sentences. We sample 100 passages in the high school category and the middle school category respectively. Each passage in the high school category has 20 questions and each passage in the middle school category has 10 questions. The types of the 3000 question are labeled on Amazon Turk. We pay $1 and $0.5 for high school passage and middle school passage respectively. The proportion of different questions is shown in TAB4. We find that the majority of questions are short-term-reasoning questions, in which the examinee needs to utilize grammar knowledge, vocabulary knowledge and simple reasoning to answer the questions. Note that questions in middle school are easier since they have more grammar questions. Finally, only approximately 22.4% of data needs long-term information, in which the long-term-reasoning questions constitute a large proportion. In this section, we study if human-designed cloze test is a challenging problem for state-of-the-art models. We find that the language model trained on large enough external corpus could not solve the cloze test. After conducting error analysis, we hypothesize that the model is not able to deal with long-term dependencies. We verify the hypothesis by evaluating human's performance when human only see one sentence as the context. LSTM To test the performance of RNN based supervised models, we train a bidirectional LSTM BID10 to predict the missing word given the context, with only labeled data. The implementation details are in Appendix A.1.Attention Readers To enable the model to gather information from a longer context, we augment the supervised LSTM model with the attention mechanism BID1, so that the representation at the blank is used as a query to find the relevant context in the document and a blank-specific representation of the document is used to score each candidate answer. Specifically, we adapt the Stanford Attention Reader BID3 and the position-aware attention model BID29 to the cloze test problem. With the position-aware attention model, the attention scores are based on both the context match and the distances of two words. Both attention models are trained only with the human-designed blanks just as the LSTM model. Language model Language modeling and cloze test are similar since, in both tasks, a word is predicted based on the context. In cloze test, the context on both sides may determine the correct answer. Suppose x i is the missing word and x 1, · · ·, x i−1, x i+1, · · ·, x n are the context. Although language model is trained to predict the next word only using the left context, to utilize the surrounding context, we could choose x i that maximizes the joint probability p(x 1, · · ·, x n), which essentially maximizes the conditional likelihood p(DISPLAYFORM0 . Therefore, language model can be naturally adapted to cloze test. In essence, language model treats each word as a possible blank and learns to predict it. As a , it receives more supervision than the supervised model trained on human-labeled questions. Additionally, it can be trained on a very large unlabeled corpus. Interested in whether the state-ofthe-art language model can solve cloze test, we first train a neural language model on the training set of our corpus, then we test the language model trained on One Billion Word Benchmark BID2) (referred as 1-billion-language-model) that achieves a perplexity of 30.0 BID13 5. To make the evaluation time tractable, we limit the context length to one sentence or three sentences. Human performance We measure the performance of Amazon Turkers on 3, 000 sampled questions when the whole passage is given. The comparison is shown in Table 4. Both attention models achieve a similar accuracy to the LSTM. We hypothesize the attention model's unsatisfactory performance is due to the difficulty to learn to comprehend longer context when the majority of the training data only requires understanding short-term information. The language model trained on our dataset achieves an accuracy of 0.548 while the supervised model's accuracy is 0.484, indicating that more training data in better generalization. When only one sentence is given as context, the accuracy of 1-billion-languagemodel is 0.695, which shows that the amount of data is an essential factor affecting the model's performance. It also indicates that the language model can learn sophisticated language regularities when given enough data. The same can also be drawn from state-of-the-art on six language tasks ed from applying language model representations as word vectors BID0. However, if we increase the context length to three sentences, the accuracy of 1-billionlanguage-model only improves to 0.707. In contrast, human outperforms 1-billion-language-model by a significant margin, which demonstrates that deliberately designed questions in CLOTH are not completely solved even for state-of-the-art models. Table 4: Model and human's performance on CLOTH. Attention model does not leads to performance improvement compared to vanilla LSTM. Language model outperforms LSTM since it receives more supervisions in learning to predict each word. Training on large external corpus further significantly enhances the accuracy. In this section, we would like to understand why the state-of-the-art model lags behind human performance. We find that most of the errors made by the large language model involve long-term reasoning. Additionally, in a lot of cases, the dependency is within the context of three sentences. Several errors made by the large language model are shown in Table 5. In the first example, the model does not know that Nancy found nobody in the company means that Nancy was the first one to arrive at the company. In the second and third example, the model fails probably because of the coreference from "they" to "flowers". The dependency in the last case is longer. It depends on the fact that "Nancy" was alone in the company. Based on the case study, we hypothesize that the language model is not able to take long-term information into account, although it achieves a surprisingly good overall performance. Moreover, the 1-billion-language-model is trained on the sentence level, which might also in paying more attention to short-term information. However, we do not have enough computational resources to train a large model on 1 Billion Word Benchmark to investigate the differences of training on sentence level or on paragraph level. She smelled them and they were sweet. She looked around for a to put them in. A. vase B. room C. glass D. bottle "Somebody has sent me flowers the very first day!" " But who could it be?" she began to. The day passed quickly and Nancy did A. seek B. wonder C. work D. ask everything with great interest. Table 5: Error analysis of 1-billion-language-model with three sentences as the context. The questions are sampled from the sample passage shown in Table 2. The correct answer is in bold text. The incorrectly selected options are in italics. An available comparison is to test the model's performance on different types of questions. We find that the model's accuracy is 0.591 on long-term-reasoning questions of CLOTH-H while achieving 0.693 on short-term-reasoning, which partially confirms that long-term-reasoning is harder. However, we could not completely rely on the performance on specific questions types, partly due to the small sample size. A more fundamental reason is that the question type labels are subjective and their reliability depends on whether turkers are careful enough. For example, in the error analysis shown in Table 5, a careless turker would label the second example as short-term-reasoning without noticing that the meaning of "they" relies on a long context span. To objectively verify if the language model's strengths are in dealing with short-term information, we obtain the ceiling performance of only utilizing short-term information. Showing only one sentence as the context, we ask the turkers to label all possible options that they deem to be correct given the insufficient information. We also ask them to select a single option based on their best guesses. By limiting the context span manually, the ceiling performance with only the access to short context is estimated accurately. The performances of turkers and 1-billion-language-model are shown in TAB8. The performance of 1-billion-language-model using one sentence as the context can almost match the ceiling performance of only using short-term information. Hence we conclude that the language model can almost perfectly solve all short-term cloze questions. However, the performance of language model is not improved significantly when the needed long-term context is given, indicating that the performance gap is due to the inability of long-term reasoning. Assuming the majority of question type labels is reliable, we verify the strengths and weaknesses of models and human by studying the performance of models and human on different question categories. The comparison is shown in Figure 1.The human study on short-term ceiling performance also reveals that the options are carefully picked. Specifically, when a Turker thinks that a question has multiple answers, 3.41 out of 4 options are deemed to be possibly correct, which means that teachers design the options so that three or four options all make sense if we only look at the local context.4 COMPARING HUMAN-DESIGNED DATA AND AUTOMATICALLY GENERATED DATAIn this section, we demonstrate that human-designed data is a better test bed than automatically generated data for general cloze test since it in a larger gap between the model's performance and human performance. However, the distributional mismatch between two types of data makes the human-designed data an unsuitable training source for solving automatically generated questions. In addition, we improve the model's performance by finding generated data that resembles humandesigned data. At a casual observation, a cloze test can be created by randomly deleting words and randomly sampling candidate options. In fact, to generate large-scale data, similar generation processes have been introduced and widely used in machine comprehension BID8 BID9 BID17. However, research on cloze test design BID22 shows that tests created by deliberately deleting words are more reliable than tests created by randomly or periodically deleting words. To design accurate language proficiency assessment, teachers usually select words in order to examine students' proficiency in grammar, vocabulary and reasoning. Moreover, in order to make the question non-trivial, the three incorrect options provided by teachers are usually grammatically correct and relevant to the context. For instance, in the fourth problem of the sample passage shown in Table 2, "grapes", "flowers" and "bananas" all fit the description of being fresh. We know "flowers" is the correct answer after seeing the sentence "Somebody has sent me flowers the very first day!".Naturally, we hypothesize that the distribution of human-generated data is different from automatically generated data. To verify this assumption, we compare the LSTM model's performance when given different proportion of the two types of data. Specifically, to train a model with α percent of automatically generated data, we randomly replace a percent blanks with blanks at random positions, while keeping the remaining 100 − α percent questions the same. The candidate options for the generated blanks are random words sampled from the unigram distribution. We test the trained model on human-designed data and automatically generated data respectively. Table 7: We train a model on α percent of automatically generated data and 100 − α percent of human-designed data and test it on human-designed data and automatically generated data respectively. The performance is shown in Table 7. We have the following observations: human-designed data leads to a larger gap between the model's performance and the human performance, when given the same model. The model's performance and human's performance on the human-designed data are 0.484 and 0.860 respectively, leading to a gap of 0.376. In comparison, the performance gap on the automatically generated data is at most 0.185 since the model's performance reaches 0.815 when trained on generated data. It shows that the distributions of human-designed data and automatically generated data are quite different. the distributional mismatch between two types of data makes it difficult to transfer a model trained on human-designed data to automatically generated data. Specifically, the model's performance on automatically generated data monotonously increases when given a higher ratio of automatically generated training data. To conclude, human-designed data is a good test base because of the larger gap between performances of the model and the human, although the distributional mismatch problem makes it difficult to be the best training source for out-of-domain cloze test such as automatically generated cloze test.4.2 COMBINING HUMAN-DESIGNED DATA WITH AUTOMATICALLY GENERATED DATA In Section 3.1, we show that language model is able to take advantage of more supervisions since it predicts each word based on the context. In essence, each word can provide an automatically generated question. At the same time, we also show that human-designed data and the automatically generated data are quite different in Section 4.1. In this section, we propose to combine humandesigned data with automatically generated data to achieve better performance. Note that discriminative models can also treat all words in a passage as automatically generated questions, just like a language model (Please see the Appendix A.3 for details). We study two methods of leveraging automatically generated data and human-designed data:Equally averaging Let J h be the average loss for all human-designed questions and J u be the average loss for all automatically generated questions in the passage. A straightforward method is to optimize J h + λJ u so that the model learns to predict words deleted by human and all other words in the passage. We set λ to 1 in our experiments. This model treats each automatically generated questions as equally important. Representativeness-based weighted averaging A possible avenue towards having large-scale indomain data is to automatically pick out questions which are representative of in-domain data among a large number of out-of-domain samples. Hence, we mimick the design behavior of language teachers by training a network to predict the representativeness of each automatically generated question. Note that the candidate option set for a automatically generated question is the whole vocabulary. We leave the candidate set prediction for future work. The performance of the representativeness prediction network and an example are shown in Appendix A.4.Let J i denotes the negative log likelihood loss for the i−th question and let l i be the outputted representativeness of the i-th question (The definition of l i is in Appendix A.2). We define the representativeness weighted loss function as DISPLAYFORM0 where H is the set of all human-generated questions and α is the temperature of the Softmax function. When the temperature is +∞, the model degenerate into equally averaging objective function without using the representativeness. When the temperature is 0, only the most representative question is used. We set α to 2 based on the performance on the dev set. We present the in Table 8. When all other words are treated as equally important, the accuracy is 0.543, similar to the performance of language model. Representativeness-based weighted averaging leads to an accuracy of 0.565. When combined with human-designed data, the performance can be improved to 0.583 6. Large-scale automatically generated cloze test BID8 BID9 BID17 leaded to significant research advancement. However, the generated questions do not consider Table 8: Overall on CLOTH. The "representativeness" means weighted averaging the loss of each question using the predicted representativeness. "equal-average" means to equally average losses of questions.the language phenomenon to be tested and are relatively easy to solve. Recently proposed reading comprehension datasets are all labeled by human to ensure their qualities BID20 BID12 BID28 BID16. Aiming to evaluate machines under the same conditions human is evaluated, there are a growing interests in obtaining data from examinations. NTCIR QA Lab BID24 contains a set of real-world university entrance exam questions. The Entrance Exams task at CLEF QA Track BID18 BID21 evaluates machine's reading comprehension ability. The AI2 Elementary School Science Questions dataset 7 provides 5, 060 scientific questions used in elementary and middle schools. BID15 proposes the first large-scale machine comprehension dataset obtained from exams. They show that questions designed by teachers have a significant larger proportion of reasoning questions. Our dataset focuses on evaluating language proficiency while the focus of reading comprehension is reasoning. In Section 4.2, we employ a simple supervised approach that predicts how likely a word is selected by teachers as a cloze question. It has been shown that features such as morphology information and readability are beneficial in cloze test prediction BID25 BID5. We leave investigating the advanced approaches of automatically designing cloze test to future work. In this paper, we propose a large-scale cloze test dataset CLOTH that is designed by teachers. With the missing blanks and candidate options carefully created by teachers to test different aspects of language phenomenon, CLOTH requires a deep language understanding and better captures the complexity of human language. We find that human outperforms state-of-the-art models by a significant margin, even if the model is trained on a large corpus. After detailed analysis, we find that the performance gap is due to model's inability to understanding a long context. We also show that, compared to automatically-generated questions, human-designed questions are more difficult and leads to a larger margin between human performance and the model's performance. A predicted sample is shown in FIG2. Clearly, words that are too obvious have low scores, such as punctuation marks, simple words "a" and "the". In contrast, content words whose semantics are directly related to the context have a higher score, e.g., "same", "similar", "difference" have a high score when the difference between two objects is discussed and "secrets" has a high score since it is related to the subsequent sentence "does not want to share with others".Our prediction model achieves an F1 score of 36.5 on the test set, which is understandable since there are many plausible questions within a passage.

Represent the short summary for finding the corresponding scientific article (long)

A cloze test dataset designed by teachers to assess language proficiency

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

Humans in the loop revise documents to accord with counterfactual labels, resulting resource helps to reduce reliance on spurious associations.