Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Value-based methods constitute a fundamental methodology in planning and deep reinforcement learning (RL). In this paper, we propose to exploit the underlying structures of the state-action value function, i.e., Q function, for both planning and deep RL. In particular, if the underlying system dynamics lead to some global structures of the Q function, one should be capable of inferring the function better by leveraging such structures. Specifically, we investigate the low-rank structure, which widely exists for big data matrices. We verify empirically the existence of low-rank Q functions in the context of control and deep RL tasks (Atari games). As our key contribution, by leveraging Matrix Estimation (ME) techniques, we propose a general framework to exploit the underlying low-rank structure in Q functions, leading to a more efficient planning procedure for classical control, and additionally, a simple scheme that can be applied to any value-based RL techniques to consistently achieve better performance on''low-rank'' tasks. Extensive experiments on control tasks and Atari games confirm the efficacy of our approach. Value-based methods are widely used in control, planning and reinforcement learning (; ;). To solve a Markov Decision Process (MDP), one common method is value iteration, which finds the optimal value function. This process can be done by iteratively computing and updating the state-action value function, represented by Q(s, a) (i.e., the Q-value function). In simple cases with small state and action spaces, value iteration can be ideal for efficient and accurate planning. However, for modern MDPs, the data that encodes the value function usually lies in thousands or millions of dimensions (; 2019), including images in deep reinforcement learning . These practical constraints significantly hamper the efficiency and applicability of the vanilla value iteration. Yet, the Q-value function is intrinsically induced by the underlying system dynamics. These dynamics are likely to possess some structured forms in various settings, such as being governed by partial differential equations. In addition, states and actions may also contain latent features (e.g., similar states could have similar optimal actions). Thus, it is reasonable to expect the structured dynamic to impose a structure on the Q-value. Since the Q function can be treated as a giant matrix, with rows as states and columns as actions, a structured Q function naturally translates to a structured Q matrix. In this work, we explore the low-rank structures. To check whether low-rank Q matrices are common, we examine the benchmark Atari games, as well as 4 classical stochastic control tasks. As we demonstrate in Sections 3 and 4, more than 40 out of 57 Atari games and all 4 control tasks exhibit low-rank Q matrices. This leads us to a natural question: How do we leverage the low-rank structure in Q matrices to allow value-based techniques to achieve better performance on "low-rank" tasks? We propose a generic framework that allows for exploiting the low-rank structure in both classical planning and modern deep RL. Our scheme leverages Matrix Estimation (ME), a theoretically guaranteed framework for recovering low-rank matrices from noisy or incomplete measurements . In particular, for classical control tasks, we propose Structured Value-based Planning (SVP). For the Q matrix of dimension |S| × |A|, at each value iteration, SVP randomly updates a small portion of the Q(s, a) and employs ME to reconstruct the remaining elements. We show that planning problems can greatly benefit from such a scheme, where much fewer samples (only sample around 20% of (s, a) pairs at each iteration) can achieve almost the same policy as the optimal one. For more advanced deep RL tasks, we extend our intuition and propose Structured Value-based Deep RL (SV-RL), applicable for any value-based methods such as DQN . Here, instead of the full Q matrix, SV-RL naturally focuses on the "sub-matrix", corresponding to the sampled batch of states at the current iteration. For each sampled Q matrix, we again apply ME to represent the deep Q learning target in a structured way, which poses a low rank regularization on this "sub-matrix" throughout the training process, and hence eventually the Q-network's predictions. Intuitively, as learning a deep RL policy is often noisy with high variance, if the task possesses a low-rank property, this scheme will give a clear guidance on the learning space during training, after which a better policy can be anticipated. We confirm that SV-RL indeed can improve the performance of various value-based methods on "low-rank" Atari games: SV-RL consistently achieves higher scores on those games. Interestingly, for complex, "high-rank" games, SV-RL performs comparably. ME naturally seeks solutions that balance low rank and a small reconstruction error (cf. Section 3.1). Such a balance on reconstruction error helps to maintain or only slightly degrade the performance for "high-rank" situation. We summarize our contributions as follows: • We are the first to propose a framework that leverages matrix estimation as a general scheme to exploit the low-rank structures, from planning to deep reinforcement learning. • We demonstrate the effectiveness of our approach on classical stochastic control tasks, where the low-rank structure allows for efficient planning with less computation. • We extend our scheme to deep RL, which is naturally applicable for any value-based techniques. Across a variety of methods, such as DQN, double DQN, and dueling DQN, experimental on all Atari games show that SV-RL can consistently improve the performance of value-based methods, achieving higher scores for tasks when low-rank structures are confirmed to exist. To motivate our method, let us first investigate a toy example which helps to understand the structures within the Q-value function. We consider a simple deterministic MDP, with 1000 states, 100 actions and a deterministic state transition for each action. The reward r(s, a) is randomly generated first for each (s, a) pair, and then fixed throughout. A discount factor γ = 0.95 is used. The deterministic nature imposes a strong relationship among connected states. In this case, our goal is to explore: what kind of structures the Q function may contain; and how to effectively exploit such structures. The Low-rank Structure Under this setting, Q-value could be viewed as a 1000 × 100 matrix. To probe the structure of the Q-value function, we perform the standard Q-value iteration as follows: where s denotes the next state after taking action a at state s. We randomly initialize Q. In Fig. 1, we show the approximate rank of Q (t) and the mean-square error (MSE) between Q (t) and the optimal Q *, during each value iteration. Here, the approximate rank is defined as the first k singular values (denoted by σ) that capture more than 99% variance of all singular values, i.e., k i=1 σ 2 i / j σ 2 j ≥ 0.99. As illustrated in Fig. 1 (a) and 1(b), the standard theory guarantees the convergence to Q *; more interestingly, the converged Q * is of low rank, and the approximate rank of Q (t) drops fast. These observations give a strong evidence for the intrinsic low dimensionality of this toy MDP. Naturally, an algorithm that leverages such structures would be much desired. The previous motivate us to exploit the structures for efficient planning. The idea is simple: If the eventual matrix is low-rank, why not enforcing such a structure throughout the iterations? In other words, with the existence of a global structure, we should be capable of exploiting it during intermediate updates and possibly also regularizing the to be in the same low-rank space. In particular, at each iteration, instead of every (s, a) pair (i.e., Eq.), we would like to only calculate Q (t+1) for some (s, a) pairs and then exploit the low-rank structure to recover the whole Q (t+1) matrix. We choose matrix estimation (ME) as our reconstruction oracle. The reconstructed matrix is often with low rank, and hence regularizing the Q matrix to be low-rank as well. We validate this framework in Fig. 1 (c) and 1(d), where for each iteration, we only randomly sample 50% of the (s, a) pairs, calculate their corresponding Q (t+1) and reconstruct the whole Q (t+1) matrix with ME. Clearly, around 40 iterations, we obtain comparable to the vanilla value iteration. Importantly, this comparable performance only needs to directly compute 50% of the whole Q matrix at each iteration. It is not hard to see that in general, each vanilla value iteration incurs a computation cost of O(|S| 2 |A| 2). The complexity of our method however only scales as O(p|S| 2 |A| 2)+O M E, where p is the percentage of pairs we sample and O M E is the complexity of ME. In general, many ME methods employ SVD as a subroutine, whose complexity is bounded by O(min{|S| 2 |A|, |S||A| 2}) . For low-rank matrices, faster methods can have a complexity of order linear in the dimensions . In other words, our approach improves computational efficiency, especially for modern high-dimensional applications. This overall framework thus appears to be a successful technique: it exploits the low-rank behavior effectively and efficiently when the underlying task indeed possesses such a structure. Having developed the intuition underlying our methodology, we next provide a formal description in Sections 3.1 and 3.2. One natural question is whether such structures and our method are general in more realistic control tasks. Towards this end, we provide further empirical support in Section 3.3. ME considers about recovering a full data matrix, based on potentially incomplete and noisy observations of its elements. Formally, consider an unknown data matrix X ∈ R n×m and a set of observed entries Ω. If the observations are incomplete, it is often assumed that each entry of X is observed independently with probability p ∈. In addition, the observation could be noisy, where the noise is assumed to be mean zero. Given such an observed set Ω, the goal of ME is to produce an estimatorM so that ||M − X|| ≈ 0, under an appropriate matrix norm such as the Frobenius norm. The algorithms in this field are rich. Theoretically, the essential message is: exact or approximate recovery of the data matrix X is guaranteed if X contains some global structure (Candès & ; ;). In the literature, most attention has been focusing on the low-rank structure of a matrix. Correspondingly, there are many provable, practical algorithms to achieve the desired recovery. Early convex optimization methods (Candès &) seek to minimize the nuclear norm, ||M || *, of the estimator. For example, fast algorithms, such as the Soft-Impute algorithm solves the following minimization problem: Since the nuclear norm || · || * is a convex relaxation of the rank, the convex optimization approaches favor solutions that are with small reconstruction errors and in the meantime being relatively low-rank, which are desirable for our applications. Apart from convex optimization, there are also spectral methods and even non-convex optimization approaches (; ;). In this paper, we view ME as a principled reconstruction oracle to effectively exploit the low-rank structure. For faster computation, we mainly employ the Soft-Impute algorithm. We now formally describe our approach, which we refer as structured value-based planning (SVP). Fig. 2 illustrates our overall approach for solving MDP with a known model. The approach is based on the Q-value iteration. At the t-th iteration, instead of a full pass over all state-action pairs: State-action value function Incomplete observation Reconstructed value function Figure 2: An illustration of the proposed SVP algorithm for leveraging low-rank structures. 1. SVP first randomly selects a subset Ω of the state-action pairs. In particular, each state-action pair in S × A is observed (i.e., included in Ω) independently with probability p. 2. For each selected (s, a), the intermediateQ(s, a) is computed based on the Q-value iteration: 3. The current iteration then ends by reconstructing the full Q matrix with matrix estimation, from the set of observations in Ω. That is, Q (t+1) = ME {Q(s, a)} (s,a)∈Ω. Overall, each iteration reduces the computation cost by roughly 1 − p (cf. discussions in Section 2). In Appendix A, we provide the pseudo-code and additionally, a short discussion on the technical difficulty for theoretical analysis. Nevertheless, we believe that the consistent empirical benefits, as will be demonstrated, offer a sounding foundation for future analysis. We empirically evaluate our approach on several classical stochastic control tasks, including the Inverted Pendulum, the Mountain Car, the Double Integrator, and the Cart-Pole. Our objective is to demonstrate, as in the toy example, that if the optimal Q * has a low-rank structure, then the proposed SVP algorithm should be able to exploit the structure for efficient planning. We present the evaluation on Inverted Pendulum, and leave additional on other planning tasks in Appendix B and C. Inverted Pendulum In this classical continuous task, our goal is to balance an inverted pendulum to its upright position, and the performance metric is the average angular deviation. The dynamics is described by the angle and the angular speed, i.e., s = (θ,θ), and the action a is the torque applied. We discretize the task to have 2500 states and 1000 actions, leading to a 2500 × 1000 Q-value matrix. The Low-rank Structure We first verify that the optimal Q * indeed contains the desired low-rank structure. We run the vanilla value iteration until it converges. The converged Q matrix is found to have an approximate rank of 7. For further evidence, in Appendix C, we construct "low-rank" policies directly from the converged Q matrix, and show that the policies maintain the desired performance. The SVP Policy Having verified the structure, we would expect our approach to be effective. To this end, we apply SVP with different observation probability p and fix the overall number of iterations to be the same as the vanilla Q-value iteration. Fig. 3 confirms the success of our approach. Fig. 3(a), 3(b) and 3(c) show the comparison between optimal policy and the final policy based on SVP. We further illustrate the performance metric, the average angular deviation, in Fig. 3(d). Overall, much fewer samples are needed for SVP to achieve a comparable performance to the optimal one. So far, our focus has been on tabular MDPs where value iteration can be applied straightforwardly. However, the idea of exploiting structure is much more powerful: we propose a natural extension of our approach to deep RL. Our scheme again intends to exploit and regularize structures in the Q-value function with ME. As such, it can be seamlessly incorporated into value-based RL techniques that include a Q-network. We demonstrate this on Atari games, across various value-based RL techniques. Before diving into deep RL, let us step back and review the process we took to develop our intuition. Previously, we start by treating the Q-value as a matrix. To exploit the structure, we first verify that certain MDPs have essentially a low-rank Q *. We argue that if this is indeed the case, then enforcing the low-rank structures throughout the iterations, by leveraging ME, should lead to better algorithms. A naive extension of the above reasoning to deep RL immediately fails. In particular, with images as states, the state space is effectively infinitely large, leading to a tall Q matrix with numerous number of rows (states). Verifying the low-rank structure for deep RL hence seems intractable. However, by definition, if a large matrix is low-rank, then almost any row is a linear combination of some other rows. That is, if we sample a small batch of the rows, the ing matrix is most likely low-rank as well. To probe the structure of the deep Q function, it is, therefore, natural to understand the rank of a randomly sampled batch of states. In deep RL, our target for exploring structures is no longer the optimal Q *, which is never available. In fact, like SVP, the natural objective should be the converged values of the underlying algorithm, which in deep scenarios, are the eventually learned Q function. With the above discussions, we now provide evidence for the low-rank structure of learned Q function on some Atari games. We train standard DQN on 4 games, with a batch size of 32. To be consistent, the 4 games all have 18 actions. After the training process, we randomly sample a batch of 32 states, evaluate with the learned Q network and finally synthesize them into a matrix. That is, a 32 × 18 data matrix with rows the batch of states, the columns the actions, and the entries the values from the learned Q network. Note that the rank of such a matrix is at most 18. The above process is repeated for 10,000 times, and the histogram and empirical CDF of the approximate rank is plotted in Fig. 4. Apparently, there is a strong evidence supporting a highly structured low-rank Q function for those games -the approximate ranks are uniformly small; in most cases, they are around or smaller than 3. Having demonstrated the low-rank structure within some deep RL tasks, we naturally seek approaches that exploit the structure during the training process. We extend the same intuitions here: if eventually, the learned Q function is of low rank, then enforcing/regularizing the low rank structure for each iteration of the learning process should similarly lead to efficient learning and potentially better performance. In deep RL, each iteration of the learning process is naturally the SGD step where one would update the Q network. Correspondingly, this suggests us to harness the structure within the batch of states. Following our previous success, we leverage ME to achieve this task. We now formally describe our approach, referred as structured value-based RL (SV-RL). It exploits the structure for the sampled batch at each SGD step, and can be easily incorporated into any Q-value based RL methods that update the Q network via a similar step as in Q-learning. In particular, Q-value based methods have a common model update step via SGD, and we only exploit structure of the sampled batch at this step -the other details pertained to each specific method are left intact. Precisely, when updating the model via SGD, Q-value based methods first sample a batch of B transitions {(s and form the following updating targets: For example, in DQN,Q is the target network. The Q network is then updated by taking a gradient step for the loss function 2, with respect to the parameter θ. To exploit the structure, we then consider reconstructing a matrix Q † fromQ, via ME. The reconstructed Q † will replace the role ofQ in Eq. to form the targets y (i) for the gradient step. In particular, the matrix Q † has a dimension of B × |A|, where the rows represent the "next states" in the batch, the columns represent actions, and the entries are reconstructed values. Let S B {s . The SV-RL alters the SGD update step as illustrated in Algorithm 1 and sample a set Ω of state-action pairs from S B × A. In particular, each state-action pair in S B × A is observed (i.e., included in Ω) with probability p, independently. evaluate every state-action pair in Ω viaQ, whereQ is the network that would be used to form the targets {y in the original value-based methods (cf. Eq.). 6: based on the evaluated values, reconstruct a matrix Q † with ME, i.e., / * new targets with reconstructed Q † for the gradient step * / 8: replaceQ in Eq. with Q † to evaluate the targets {y SV-RL Targets: 9: update the Q network with the original targets replaced by the SV-RL targets. Note the resemblance of the above procedure to that of SVP in Section 3.2. When the full Q matrix is available, in Section 3.2, we sub-sample the Q matrix and then reconstruct the entire matrix. When only a subset of the states (i.e., the batch) is available, naturally, we look at the corresponding sub-matrix of the entire Q matrix, and seek to exploit its structure. We conduct extensive experiments on Atari 2600. We apply SV-RL on three representative value-based RL techniques, i.e., DQN, double DQN and dueling DQN. We fix the total number of training iterations and set all the hyper-parameters to be the same. For each experiment, averaged across multiple runs are reported. Additional details are provided in Appendix D. We present representative of SV-RL applied to the three value-based deep RL techniques in Fig. 6. These games are verified by method mentioned in Section 4.1 to be low-rank. Additional on all Atari games are provided in Appendix E. The figure reveals the following . First, games that possess structures indeed benefit from our approach, earning mean rewards that are strictly higher than the vanilla algorithms across time. More importantly, we observe consistent improvements across different value-based RL techniques. This highlights the important role of the intrinsic structures, which are independent of the specific techniques, and justifies the effectiveness of our approach in consistently exploiting such structures. Further Observations Interestingly however, the performance gains vary from games to games. Specifically, the majority of the games can have benefits, with few games performing similarly or slightly worse. Such observation motivates us to further diagnose SV-RL in the next section. So far, we have demonstrated that games which possess structured Q-value functions can consistently benefit from SV-RL. Obviously however, not all tasks in deep RL would possess such structures. As such, we seek to diagnose and further interpret our approach at scale. Diagnosis We select 4 representative examples (with 18 actions) from all tested games, in which SV-RL performs better on two tasks (i.e., FROSTBITE and KRULL), slightly better on one task (i.e., ALIEN), and slightly worse on the other (i.e., SEAQUEST). The intuitions we developed in Section 4 incentivize us to further check the approximate rank of each game. As shown in Fig. 7, in the two better cases, both games are verified to be approximately low-rank (∼ 2), while the approximate rank in ALIEN is moderately high (∼ 5), and even higher in SEAQUEST (∼ 10). Consistent Interpretations As our approach is designed to exploit structures, we would expect to attribute the differences in performance across games to the "strength" of their structured properties. Games with strong low-rank structures tend to have larger improvements with SV-RL (Fig. 7 (a) and 7(b)), while moderate approximate rank tends to induce small improvements (Fig. 7(c) ), and high approximate rank may induce similar or slightly worse performances (Fig. 7(d) ). The empirical are well aligned with our arguments: if the Q-function for the task contains low-rank structure, SV-RL can exploit such structure for better efficiency and performance; if not, SV-RL may introduce slight or no improvements over the vanilla algorithms. As mentioned, the ME solutions balance being low rank and having small reconstruction error, which helps to ensure a reasonable or only slightly degraded performance, even for "high rank" games. We further observe consistent on ranks vs. improvement across different games and RL techniques in Appendix E, verifying our arguments. We plot games where the SV-based method performs differently. More structured games (with lower rank) can achieve better performance with SV-RL. Structures in Value Function Recent work in the control community starts to explore the structure of value function in control/planning tasks (; ; ; 2018). These work focuses on decomposing the value function and subsequently operating on the reduced-order space. In spirit, we also explore the low-rank structure in value function. The central difference is that instead of decomposition, we focus on "completion". We seek to efficiently operate on the original space by looking at few elements and then leveraging the structure to infer the rest, which allows us to extend our approach to modern deep RL. In addition, while there are few attempts for basis function learning in high dimensional RL , functions are hard to generate in many cases and approaches based on basis functions typically do not get the same performance as DQN, and do not generalize well. In contrast, we provide a principled and systematic method, which can be applied to any framework that employs value-based methods or sub-modules. Value-based Deep RL Value-based methods are fundamental in deep RL, exemplified by DQN (; 2015). There has been a large body of literature on its variants, such as double DQN , dueling DQN , IQN and other techniques that improve exploration . Our approach focuses on general value-based RL methods. As long as the method has a similar model update step as in Q-learning, our approach can leverage the structure to help with the task. We empirically show that deep RL tasks that have structured value functions indeed benefit from our scheme. Matrix Estimation ME is the primary tool we leverage to exploit the low-rank structure in value functions. The techniques have been widely studied and applied to different domains (; ;), and recently even in robust deep learning . The field is relatively mature, with extensive algorithms and provable recovery guarantees for structured matrix . Because of the strong promise, we view ME as a principled reconstruction oracle to exploit the low-rank structures within matrices. We investigated the structures in value function, and proposed a complete framework to understand, validate, and leverage such structures in various tasks, from planning to deep reinforcement learning. The proposed SVP and SV-RL algorithms harness the strong low-rank structures in the Q function, showing consistent benefits for both planning tasks and value-based deep reinforcement learning techniques. Extensive experiments validated the significance of the proposed schemes, which can be easily embedded into existing planning and RL frameworks for further improvements. randomly sample a set Ω of observed entries from S × A, each with probability p 4: / * update the randomly selected state-action pairs * / 5: end for 8: / * reconstruct the Q matrix via matrix estimation * / 9: apply matrix completion to the observed values {Q(s, a)} (s,a)∈Ω to reconstruct Q (t+1): While based on classical value iteration, we remark that a theoretical analysis, even in the tabular case, is quite complex. Although the field of ME is somewhat mature, the analysis has been largely focused on the "one-shot" problem: recover a static data matrix given one incomplete observation. Under the iterative scenario considered here, standard assumptions are easily broken and the analysis warrants potentially new machinery. Furthermore, much of the effort in the ME community has been devoted to the Frobenius norm guarantees rather than the infinite norm as in value iteration. Non-trivial infinite norm bound has received less attention and often requires special techniques . Resolving the above burdens would be important future avenues in its own right for the ME community. Henceforth, this paper focuses on empirical analysis and more importantly, generalizing the framework successfully to modern deep RL contexts. As we will demonstrate, the consistent empirical benefits offer a sounding foundation for future analysis. Inverted Pendulum As stated earlier in Sec. 3.3, the goal is to balance the inverted pendulum on the upright equilibrium position. The physical dynamics of the system is described by the angle and the angular speed, i.e., (θ,θ). Denote τ as the time interval between decisions, u as the torque input on the pendulum, the dynamics can be written as : A reward function that penalizes control effort while favoring an upright pendulum is used: In the simulation, the state space is (−π, π] for θ and [−10, 10] forθ. We limit the input torque in [−1, 1] and set τ = 0.3. We discretize each dimension of the state space into 50 values, and action space into 1000 values, which forms an Q-value function a matrix of dimension 2500 × 1000. We follow to handle the policy of continuous states by modelling their transitions using multi-linear interpolation. Mountain Car We select another classical control problem, i.e., the Mountain Car , for further evaluations. In this problem, an under-powered car aims to drive up a steep hill . The physical dynamics of the system is described by the position and the velocity, i.e., (x,ẋ). Denote u as the acceleration input on the car, the dynamics can be written as The reward function is defined to encourage the car to get onto the top of the mountain at x 0 = 0.5: We follow standard settings to restrict the state space as [−0.07, 0.07] for x and [−1.2, 0.6] forẋ, and limit the input u ∈ [−1, 1]. Similarly, the whole state space is discretized into 2500 values, and the action space is discretized into 1000 values. The evaluation metric we are concerned about is the total time it takes to reach the top of the mountain, given a randomly and uniformly generated initial state. We consider the Double Integrator system , as another classical control problem for evaluation. In this problem, a unit mass brick moving along the x-axis on a frictionless surface, with a control input which provides a horizontal force, u . The task is to design a control system to regulate this brick to x = T. The physical dynamics of the system is described by the position and the velocity (i.e., (x,ẋ)), and can be derived as , we use the quadratic cost formulation to define the reward function, which regulates the brick to We follow standard settings to restrict the state space as [−3, 3] for both x andẋ, limit the input u ∈ [−1, 1] and set τ = 0.1. The whole state space is discretized into 2500 values, and the action space is discretized into 1000 values. Similarly, we define the evaluation metric as the total time it takes to reach to x = T, given a randomly and uniformly generated initial state. Cart-Pole Finally, we choose the Cart-Pole problem , a harder control problem with 4-dimensional state space. The problem consists a pole attached to a cart moving on a frictionless track. The cart can be controlled by means of a limited force within 10N that is possible to apply both to the left or to the right of the cart. The goal is to keep the pole on the upright equilibrium position. The physical dynamics of the system is described by the angle and the angular speed of the pole, and the position and the speed of the cart, i.e., (θ,θ, x,ẋ). Denote τ as the time interval between decisions, u as the force input on the cart, the dynamics can be written as x:= u + ml θ2 sin θ −θ cos θ where g = 9.8m/s 2 corresponds to the gravity acceleration, m c = 1kg denotes the mass of the cart, m = 0.1kg denotes the mass of the pole, l = 0.5m is half of the pole length, and u corresponds to the force applied to the cart, which is limited by u ∈ [−10, 10]. A reward function that favors the pole in an upright position, i.e., characterized by keeping the pole in vertical position between |θ| ≤ 12π 180, is expressed as r(θ) = cos 4 (15θ). In the simulation, the state space is [− .5] foṙ x. We limit the input force in [−10, 10] and set τ = 0.1. We discretize each dimension of the state space into 10 values, and action space into 1000 values, which forms an Q-value function a matrix of dimension 10000 × 1000. We further verify that the optimal Q * indeed contains the desired low-rank structure. To this end, we construct "low-rank" policies directly from the converged Q matrix. In particular, for the converged Q matrix, we sub-sample a certain percentage of its entries, reconstruct the whole matrix via ME, and finally construct a corresponding policy. Fig. 8 illustrates the , where the policy heatmap as well as the performance (i.e., the angular error) of the "low-rank" policy is essentially identical to the optimal one. The reveal the intrinsic strong low-rank structures lie in the Q-value function. We provide additional for the inverted pendulum problem. We show the policy trajectory (i.e., how the angle of the pendulum changes with time) and the input changes (i.e., how the input torque changes with time), for each policy. In Fig. 9, we first show the comparison between the optimal policy and a "low-rank" policy. Recall that the low-rank policies are directly reconstructed from the converged Q matrix, with limited observation of a certain percentage of the entries in the converged Q matrix. As shown, the "low-rank" policy performs nearly identical to the optimal one, in terms of both the policy trajectory and the input torque changes. This again verifies the strong low-rank structure lies in the Q function. Further, we show the policy trajectory and the input torque changes of the SVP policy. We vary the percentage of observed data for SVP, and present the policies with 20% and 60% for demonstration. As reported in Fig. 10, the SVP policies are essentially identical to the optimal one. Interestingly, when we further decrease the observing percentage to 20%, the policy trajectory vibrates a little bit, but can still stabilize in the upright position with a small average angular deviation ≤ 5 •. Similarly, we first verify the optimal Q * contains the desired low-rank structure. We use the same approach to generate a "low-rank" policy based on the converged optimal value function. Fig. 11 (a) and 11(b) show the policy heatmaps, where the reconstructed "low-rank" policy maintains visually identical to the optimal one. In Fig. 11(c) and 12, we quantitatively show the average time-to-goal, the policy trajectory and the input changes between the two schemes. Compared to the optimal one, even with limited sampled data, the reconstructed policy can maintain almost identical performance. We further show the of the SVP policy with different amount of observed data (i.e., 20% and 60%) in Fig. 13 and 14. Again, the SVP policies show consistently comparable to the optimal policy, over various evaluation metrics. Interestingly, the converged Q matrix of vanilla value iteration is found to have an approximate rank of 4 (the whole matrix is 2500 × 1000), thus the SVP can harness such strong low-rank structure for perfect recovery even with only 20% observed data. For the Double Integrator, We first use the same approach to generate a "low-rank" policy. Fig. 15(a) and 15(b) show that the reconstructed "low-rank" policy is visually identical to the optimal one. In Further, we show the of the SVP policy with different amount of observed data (i.e., 20% and 60%) in Fig. 17 and 18. As shown, the SVP policies show consistently decent , which demonstrates that SVP can harness such strong low-rank structure even with only 20% observed data. Finally, we evaluate SVP on the Cart-Pole system. Note that since the state space has a dimension of 4, the policy heatmap should contain 4 dims, but is hard to visualize. Since the metric we care is the angle deviation, we here only plot the first two dims (i.e., the (θ,θ) tuple) with fixed x andẋ, to visualize the policy heatmaps. We first use the same approach to generate a "low-rank" policy. Fig. 19(a) and 19(b) show the policy heatmaps, where the reconstructed "low-rank" policy is visually identical to the optimal one. In Fig. 19(c) and 20, we quantitatively show the average time-to-goal, the policy trajectory and the input changes between the two schemes. As demonstrated, the reconstructed policy can maintain almost identical performance with only small amount of sampled data. We finally show the of the SVP policy with different amount of observed data (i.e., 20% and 60%) in Fig. 21 and 22. Even for harder control tasks with higher dimensional state space, the SVP policies are still essentially identical to the optimal one. Across various stochastic control tasks, we demonstrate that SVP can consistently leverage strong low-rank structures for efficient planning. Training Details and Hyper-parameters The network architectures of DQN and dueling DQN used in our experiment are exactly the same as in the original papers (; ;). We train the network using the Adam optimizer . In all experiments, we set the hyper-parameters as follows: learning rate α = 1e −5, discount coefficient γ = 0.99, and a minibatch size of 32. The number of steps between target network updates is set to 10, 000. We use a simple exploration policy as the -greedy policy with the decreasing linearly from 1 to 0.01 over 3e 5 steps. For each experiment, we perform at least 3 independent runs and report the averaged . † formed by the current batch of states, we mainly employ the Soft-Impute algorithm throughout our experiments. We set the sub-sample rate to p = 0.9 of the Q matrix, and use a linear scheduler to increase the sampling rate every 2e 6 steps. Experiments across Various Value-based RL We show more across DQN, double DQN and dueling DQN in Fig. 23, 24, 25, 26, 27 and 28, respectively. For DQN, we complete all 57 Atari games using the proposed SV-RL, and verify that the majority of tasks contain low-rank structures (43 out of 57), where we can obtain consistent benefits from SV-RL. For each experiment, we associate the performance on the Atari game with its approximate rank. As mentioned in the main text, majority of the games benefits consistently from SV-RL. We note that roughly only 4 games, which have a significantly large rank, perform slightly worse than the vanilla DQN. Consistency and Interpretation Across all the experiments we have done, we observe that when the game possesses structures (i.e., being approximately low-rank), SV-RL can consistently improve the performance of various value-based RL techniques. The superior performance is maintained through most of the experiments, verifying the ability of the proposed SV-RL to harness the structures for better efficiency and performance in value-based deep RL tasks. In the meantime, when the approximate rank is relatively higher (e.g., SEAQUEST), the performance of SV-RL can be similar or worse than the vanilla scheme, which also aligns well with our intuitions. Note that the majority of the games have an action space of size 18 (i.e., rank is at most 18), while some (e.g., PONG) only have 6 or less (i.e., rank is at most 6). We provide an additional study on the Inverted Pendulum problem with respect to the discretization scale. As described in Sec. 3, the dynamics is described by the angle and the angular speed as s = (θ,θ), and the action a is the torque applied. To solve the task with value iteration, the state and action spaces need to be discretized into fine-grids. Previously, the two-dimensional state space was discretized into 50 equally spaced points for each dimension and the one-dimensional action space was evenly discretized into 1000 actions, leading to a 2500 × 1000 Q-value matrix. Here we choose three different discretization values for state-action pairs: 400 × 100, 2500 × 1000, and 10000 × 4000, to provide different orders of discretization for both state and action values. As Table 1 reports, the approximate rank is consistently low when discretization varies, demonstrating the intrinsic low-rank property of the task. Fig. 29 and Table 1 further demonstrates the effectiveness of SVP: it can achieve almost the same policy as the optimal one even with only 20% observations. The reveal that as long as the discretization is fine enough to represent the optimal policy for the task, we would expect the final Q matrix after value iteration to have similar rank. Figure 29: Additional study on discretization scale. We choose three different discretization value on the Inverted Pendulum task, i.e. 400 (states, 20 each dimension) × 100 (actions), 2500 (states, 50 each dimension) × 1000 (actions), and 10000 (states, 100 each dimension) × 4000 (actions). First row reports the optimal policy, second row reports the SVP policy with 20% observation probability. Table 1: Additional study on discretization scale (cont.). We report the approximate rank as well as the performance metric (i.e., the average angular deviation) on different discretization scales. To further understand our approach, we provide another study on batch size for games of different rank properties. Two games from Fig. 7 are investigated; one with a small rank (Frostbite) and one with a high rank (Seaquest). Different batch sizes, 32, 64, and 128, are explored and we show the in Fig. 30. Intuitively, for a learning task, the more complex the learning task is, the more data it would need to fully learn the characteristics. For a complex game with higher rank, a small batch size may not be sufficient to capture the game, leading the recovered matrix via ME to impose a structure that deviates from the original, more complex structure of the game. In contrast, with more data, i.e., a larger batch size, the ME oracle attempts to find the best rank structure that would effectively describe the rich observations and at the same time, balance the reconstruction error. Such a structure is more likely to be aligned with the underlying complex task. Indeed, this is what we observe in Fig. 30. As expected, for Seaquest (high rank), the performance is worse than the vanilla DQN when the batch size is small. However, as the batch size increases, the performance gap becomes smaller, and eventually, the performance of SV-RL is the same when the batch size becomes 128. On the other hand, for games with low rank, one would expect that a small batch size would be enough to explore the underlying structure. Of course, a large batch size would not hurt since the game is intrinsically low-rank. In other words, our intuition would suggest SV-RL to perform better across different batch sizes. Again, we observe this phenomenon as expected in Fig. 30. For Frostbite (low rank), under different batch sizes, vanilla DQN with SV-RL consistently outperforms vanilla DQN by a certain margin. Figure 30: Additional study on batch size. We select two games for illustration, one with a small rank (Frostbite) and one with a high rank (Seaquest). We vary the batch size with 32, 64, and 128, and report the performance with and without SV-RL.

represent the tldr for finding the corresponding scientific article

Examples:
Provided: "It is shown that ResNet-type CNNs are a universal approximator and its expression ability is not worse than fully connected neural networks (FNNs) with a \textit{block-sparse} structure even if the size of each layer in the CNN is fixed." Match: "We develop new approximation and statistical learning theories of convolutional neural networks (CNNs) via the ResNet-type structure where the channel size, filter size, and width are fixed. It is shown that a ResNet-type CNN is a universal approximator and its expression ability is no worse than fully-connected neural networks (FNNs) with a \textit{block-sparse} structure even if the size of each layer in the CNN is fixed. Our is general in the sense that we can automatically translate any approximation rate achieved by block-sparse FNNs into that by CNNs. Thanks to the general theory, it is shown that learning on CNNs satisfies optimality in approximation and estimation of several important function classes. As applications, we consider two types of function classes to be estimated: the Barron class and H\"older class. We prove the clipped empirical risk minimization (ERM) estimator can achieve the same rate as FNNs even the channel size, filter size, and width of CNNs are constant with respect to the sample size. This is minimax optimal (up to logarithmic factors) for the H\"older class. Our proof is based on sophisticated evaluations of the covering number of CNNs and the non-trivial parameter rescaling technique to control the Lipschitz constant of CNNs to be constructed. Convolutional Neural Network (CNN) is one of the most popular architectures in deep learning research, with various applications such as computer vision , natural language processing , and sequence analysis in bioinformatics (,). Despite practical popularity, theoretical justification for the power of CNNs is still scarce from the viewpoint of statistical learning theory. For fully-connected neural networks (FNNs), there is a lot of existing work, dating back to the 80's, for theoretical explanation regarding their approximation ability (, , , , and) and generalization power (, , and). See also and for surveys of earlier works. Although less common compared to FNNs, recently, statistical learning theory for CNNs has been studied, both about approximation ability (, ,) and about generalization power . One of the standard approaches is to relate the approximation ability of CNNs with that of FNNs, either deep or shallow. For example, proved that CNNs are a universal approximator of the Barron class (,), which is a historically important function class in the approximation theory. Their approach is to approximate the function using a 2-layered FNN (i.e., an FNN with a single hidden layer) with the ReLU activation function and transform the FNN into a CNN. Very recently independent of ours, showed any function realizable with an FNN can extend to an equivariant function realizable by a CNN that has the same order of parameters. However, to the best of our knowledge, no CNNs that achieves the minimax optimal rate (, Giné &) in important function classes, including the Hölder class, can keep the number of units in each layer constant with respect to the sample size. Architectures that have extremely large depth, while moderate channel size and width have become feasible, thanks to recent methods such as identity mappings (,), sophisticated initialization schemes (,), and normalization techniques (,). Therefore, we would argue that there are growing demands for theories which can accommodate such constant-size architectures. In this paper, we analyze the learning ability of ResNet-type ReLU CNNs which have identity mappings and constant-width residual blocks with fixed-size filters. There are mainly two reasons that motivate us to study this type of CNNs. First, although ResNet is the de facto architecture in various practical applications, the approximation theory for ResNet has not been explored extensively, especially from the viewpoint of the relationship between FNNs and CNNs. Second, constant-width CNNs are critical building blocks not only in ResNet but also in various modern CNNs such as Inception , DenseNet , and U-Net , to name a few. Our strategy is to replicate the learning ability of FNNs by constructing tailored ResNet-type CNNs. To do so, we pay attention to the block-sparse structure of an FNN, which roughly means that it consists of a linear combination of multiple (possibly dense) FNNs (we define it rigorously in the subsequent sections). Block-sparseness decreases the model complexity coming from the combinatorial sparsity patterns and promotes better bounds. Therefore, it is often utilized, both implicitly or explicitly, in the approximation and learning theory of FNNs (e.g., Bölcskei et al., ). We first prove that if an FNN is block-sparse with M blocks (M -way block-sparse FNN), we can realize the FNN with a ResNet-type CNN with O(M) additional parameters, which are often negligible since the original FNN already has Ω(M) parameters. Using this approximation, we give the upper bound of the estimation error of CNNs in terms of the approximation errors of block sparse FNNs and the model complexity of CNNs. Our is general in the sense that it is not restricted to a specific function class, as long as we can approximate it using block-sparse FNNs. To demonstrate the wide applicability of our methods, we derive the approximation and estimation errors for two types of function classes with the same strategy: the Barron class (of parameter s = 2) and Hölder class. We prove, as corollaries, that our CNNs can achieve the approximation error of orderÕ(M) for the β-Hölder class, where M is the number of parameters (we used M here, same as the number of blocks because it will turn out that CNNs have O(M) blocks for these cases), N is the sample size, and D is the input dimension. These rates are same as the ones for FNNs ever known in the existing literature. An important consequence of our theory is that the ResNet-type CNN can achieve the minimax optimal estimation error (up to logarithmic factors) for β-Hölder class even if its filter size, channel size and width are constant with respect to the sample size, as opposed to existing works such as and , where optimal FNNs or CNNs could have a width or a channel size goes to infinity as N → ∞.In summary, the contributions of our work are as follows:• We develop the approximation theory for CNNs via ResNet-type architectures with constant-width residual blocks. We prove any M -way block-sparse FNN is realizable such a CNN with O(M) additional parameters. That means if FNNs can approximate a function with O(M) parameters, we can approximate the function with CNNs at the same rate (Theorem 1).• We derive the upper bound of the estimation error in terms of the approximation error of FNNs and the model complexity of CNNs (Theorem 2). This gives the sufficient conditions to derive the same estimation error as that of FNNs (Corollary 1).• We apply our general theory to the Barron class and Hölder class and derive the approximation (Corollary 2 and 4) and estimation (Corollary 3 and 5) error rates, which are identical to those for FNNs, even if the CNNs have constant channel and filter size with respect to the sample size. In particular, this is minimax optimal for the Hölder case. We summarize in Table 1 the differences in the CNN architectures between our work and and , which established the approximation theory of CNNs via FNNs. First and foremost, only considered a specific function class -the Barron class -as a target function class, although their method is applicable to any function class that can be realized by a 2-layered ReLU FNN. Regarding the architecture, they considered CNNs with a single channel and whose width is "linearly increasing" layer by layer. For regression or classification problems, it is rare to use such an architecture. In addition, since they did not bound the norm of parameters in the approximating CNNs, we cannot derive the estimation error from this method. fully utilized the group invariance structure of underlying input spaces to construct CNNs. Such a structure makes theoretical analysis easier, especially for investigating the equivariance properties of CNNs since it enables us to incorporate mathematical tools such as group theory, Fourier analysis, and representation theory. Although their are quite strong in that it is applicable to any function that can be approximated by FNNs, their assumption on the group structure excludes the padding convolution layer, an important and popular type of convolution operations. Another point is that if we simply apply their construction method to derive the estimation error for (equivariant) Hölder functions, combined with the approximation of , the ing CNN that achieves the minimax optimal rate hasÕ(ε − D β) channels where ε is the approximation error threshold. It is partly because their construction is not aware of the internal sparse structure of approximating FNNs. Finally, the filter size of their CNN is as large as the input dimension. As opposed to these two works, we employ padding-and ResNettype CNNs which have multiple channels, fixed-size filters, and constant widths. , our is applicable to any function, as long as the FNNs to be approximated are block sparse, including the Barron and Hölder cases. If we apply our theorem to these classes, we can show that the optimal CNNs can achieve the same approximation and estimation rate as FNNs, while the number of channels is independent of the sample size. Further, this is minimax optimal up to the logarithmic factors for the Hölder class. Due to its practical success, theoretical analysis for ResNet has been explored recently (e.g., , , , and). From the viewpoint of statistical learning theory, and investigated the generalization power of ResNet from the perspective of the boosting interpretation. However, they did not discuss the function approximation ability of ResNet. To the best of our knowledge, our theory is the first work to provide the approximation ability of the CNN class that can accommodate the ResNet-type ones. We import the approximation theories for FNNs, especially ones for the Barron class and Hölder class. The approximation theory for the Barron class has been investigated in e.g., , , and. considered the parameter s = 1 (see Definition 3) and the activation function σ satisfying σ(z) → 1 as z → ∞ and σ(z) → 0 as z → −∞. Using this bound, proved that the estimation error of the ERM estimator is O(N − 2β 2β+D), which is minimax optimal up to logarithmic factors (see, e.g.,). We consider a regression task in this paper. Let X be a [−1, 1] D -valued random variable with unknown probability distribution P X and ξ be an independent random noise drawn from the Gaussian distribution with an unknown variance DISPLAYFORM0 • rigorously in the theorems later). We define a random variable Y by Y:= f• (X) + ξ. We denote the joint distribution of (X, Y) by P. Suppose we are given a dataset D = ((x 1, y 1),..., (x N, y N)) independently and identically sampled from the distribution P, we want to estimate the true function f• from the finite dataset D.We evaluate the performance of an estimator by the squared error. For a measurable function f: DISPLAYFORM1 Here, clip is the clipping operator defined DISPLAYFORM2 we define the L 2 -norm (weighted by P X) and the sup norm of f by DISPLAYFORM3 The task is to estimate the approximation error min f ∈F f − f • ∞ and the estimation error of the clipped ERM estimator: R(f) − R(f •). Note that the estimation error is a random variable with respect the choice of the training dataset D. By the definition of R and the independence of X and ξ, the estimation error equals to f − f DISPLAYFORM4 In this section, we define CNNs used in this paper. For this purpose, it is convenient to introduce 0, the set of real-valued sequences whose finitely many elements are non-zero: 0:= {w = (w n) n∈N>0 | ∃N ∈ N >0 s.t. w n = 0, ∀n ≥ N }. w = (w 1, . . ., w K) ∈ R K can be regarded as an element of 0 by setting w n = 0 for all n > K. Likewise, for C, C ∈ N >0, which will be the input and output channel sizes, respectively, we can think of (w k,j,i DISPLAYFORM0, we define the one-sided padding and stride-one convolution by w as an order-4 tensor DISPLAYFORM1 Here, i (resp. j) runs through 1 to C (resp. C) and α and β runs through 1 to D. Since we fix the input dimension D throughout the paper, we will omit the subscript D and write as L w if it is obvious from context. DISPLAYFORM2 Using this equality, we can expand a size-K filter to size-K.We can interpret L w as a linear mapping from DISPLAYFORM3 Next, we define the building block of CNNs: convolutional layers and fully-connected layers. Let C, C, K ∈ N >0 be the input channel size, output channel size, and filter size, respectively. For a weight tensor w ∈ R K×C ×C, a bias vector b ∈ R C, and an activation function σ: R → R, we define the convolutional layer Conv DISPLAYFORM4 where, ⊗ is the outer product of vectors and σ is applied in element-wise manner. Similarly, let W ∈ R D×C, b ∈ R, and σ: R → R, we define the fully-connected layer FC DISPLAYFORM5 is the vectorization operator that flattens a matrix into a vector. Finally, we define the ResNet-type CNN as a sequential concatenation of one convolution block, M residual blocks, and one fully-connected layer. FIG2 is the schematic view of the CNN we adopt in this paper. Definition 1 (Convolutional Neural Networks (CNNs)). Let M ∈ N >0 and L m ∈ N >0, which will be the number of residual blocks and the depth of m-th block, respectively. Let C DISPLAYFORM6 m ∈ R be the weight tensors and biases of l-th layer of the m-th block in the convolution part, respectively. Finally, let W ∈ R D×C (L 0) 0 and b ∈ R be the weight matrix and the bias for the fully-connected layer part, respectively. For θ:= ((w DISPLAYFORM7 is the identity function. Although CNN σ θ in this definition has a fully-connected layer, we refer to the stack of convolutional layers both with or without the final fully-connect layer as a CNN in this paper. We say a linear convolutional layer or a linear CNN when the activation function σ is the identity function and a ReLU convolution layer or a ReLU CNN when σ is ReLU defined by ReLU(x) = x ∨ 0. We borrow the term from ResNet and call Conv For architecture parameters C = (C DISPLAYFORM8, and norm parameters for convolution layers B (conv) > 0 and for fully-connected layers DISPLAYFORM9, the hypothesis class consisting of ReLU CNNs, as follows: DISPLAYFORM10 Here, the domain of CNNs is restricted to [−1, 1] D. Note that we impose norm constraints to the convolution part and fully-connected part separately. We emphasize that we do not impose any sparse constraints (e.g., restricting the number of non-zero parameters in a CNN to some fixed value) to F (CNN), as opposed to previous literature such as Yarotsky FORMULA24, , and. Since the notation is cluttered, we sometimes omit the subscripts as we do in the above. Remark 2. In this paper, we adopted one-sided padding, which is not often used practically, in order to make proofs simple. However, with slight modifications, all statements are true for equallypadded convolutions, the widely employed padding style which adds (approximately) same numbers of zeros to both ends of an input signal, with the exception that the filter size K is restricted to DISPLAYFORM11 We also discuss our design choice, especially the comparison with the original ResNet proposed in in Section G of the appendix. In this section, we mathematically define FNNs we consider in this paper, in parallel with the CNN case. Our FNN, which we coin a block-sparse FNN, consists of M possibly dense FNNs (blocks) concatenated in parallel, followed by a single fully-connected layer. We sketch the architecture of a block-sparse FNN in Figure 2. Let DISPLAYFORM0 m ∈ R be the weight matrix and the bias of the l-th layer of mth block (with the convention DISPLAYFORM1 be the weight (sub)vector of the final fully-connected layer corresponding to the m-th block and b ∈ R be the bias for the last layer. DISPLAYFORM2 We call a block-sparse FNN with M blocks a M -way block-sparse FNN. We say θ is compatible with (D [Lm] and norm parameters for the block part B (bs) > 0 and for the final layer DISPLAYFORM3 DISPLAYFORM4, the set of function realizable by FNNs: DISPLAYFORM5 Again, the domain is restricted to [−1, 1] D. Similar to the CNN case, we sometimes remove subscripts of the function class for simplicity. With the preparation in the previous sections, we state our main of this paper. We only describe statements of theorems and corollaries and key ideas in the main article. All complete proofs are deferred to the appendix. Our first main theorem claims that any M -way block-sparse FNN is realizable by a ResNet-type CNN with fixed-sized channels and filters by adding O(M) parameters, if we treat the widths D (l) m of the FNN as constants with respect to M. DISPLAYFORM0 m, and DISPLAYFORM1 that is, any FNN in DISPLAYFORM2 An immediate consequence of this theorem is that if we can approximate a function f • with a blocksparse FNN, we can also approximate f• with a CNN. Our second main theorem bounds the estimation error of the clipped ERM estimatorf. DISPLAYFORM0 (conv) and B (fc) as in Theorem 1. Suppose L m, C, K satisfies the equation FORMULA24 DISPLAYFORM1 Here, DISPLAYFORM2. M 1 and M 2 are defined by DISPLAYFORM3 The first term of FORMULA27 is the approximation error achieved by F (FNN). On the other hand, M 1 and M 2 are determined by the architectural parameters of F (CNN) -M 1 corresponds to the Lipschitz constant of a function realized by a CNN and M 2 is the number of parameters, including zeros, of a CNN. Therefore, the second term of represents the model complexity of F (CNN). There is a trade-off between the two terms. Using appropriately chosen M to balance them, we can evaluate the order of estimation error with respect to the sample size N. Corollary 1. Under the same assumptions as Theorem 2, suppose further log DISPLAYFORM4 The Barron class is an example of the function class that can be approximated by block-sparse FNNs. We employ the definition of Barron functions used in.Definition 3 (Barron class). We say a measurable function f DISPLAYFORM0 Here, F andF are the Fourier transformation and the inverse Fourier transformation, respectively. studied the approximation of the Barron function f• with the parameter s = 2 by a linear combination of M ridge functions (i.e., a 2-layered ReLU FNN). Specifically, they showed that there exists a function f M of the form DISPLAYFORM1 with |b m | ≤ 1, a m 1 = 1 and |t m | ≤ 1, such that f DISPLAYFORM2. Using this approximator f M, we can derive the same approximation order using CNNs by applying Theorem 1 with DISPLAYFORM3 Barron function with the parameter s = 2 such that f• = 0 and ∇f DISPLAYFORM4 with M residual blocks, each of which has depth O and at most 4 channels, and whose filter size is at DISPLAYFORM5 We have one design choice when we apply Corollary 1 to derive the estimation error: how to set B (bs) and B (fin). Looking at, the naive choice would be B (1 + ρ m) whose logarithm is O(M). We want its logarithm to beÕ. In order to do that, we change the relative scale between parameters in the block-sparse part and the fully-connected part using the homogeneous property of the ReLU function: ReLU(ax) = aReLU(x) for a > 0. The rescaling operation enables us to choose B, depth of each residual block L = O, channel size C = O, filter size K ∈ {2, . . ., D}, and norm bounds for the con- DISPLAYFORM6, and for the fully-connected part DISPLAYFORM7 such that for sufficiently large N, the clipped ERM estimatorf of F: DISPLAYFORM8. Here, DISPLAYFORM9 We next consider the approximation and error rates of CNNs when the true function is a β-Hölder function. Definition 4 (Hölder class). Let β > 0, f DISPLAYFORM0 Yarotsky FORMULA24 showed that FNNs with O(S) non-zero parameters can approximate any D variate β-Hölder function (β > 0) with the order ofÕ(S − β D). also proved a similar statement using a different construction method. They only specified their width (only), depth, and non-zero parameter counts of the approximating FNN and did not write in detail how non-zero parameters are distributed explicitly in the statements (see Theorem 1 of and Theorem 5 of Schmidt-Hieber FORMULA24). However, if we carefully look at their proofs, we find that we can transform the FNNs they constructed into the block-sparse ones. Therefore, we can utilize these FNNs and apply Theorem 1. To meet the assumption of Corollary 1, we again rescale the parameters of the FNNs, as we did in the Barron class case, so that log M 1 =Õ. We can derive the approximation and estimation errors by setting γ 1 = β D and γ 2 = 1. and O channels, and whose filter size is at most K, such that f DISPLAYFORM1 Corollary 5. There exist the number of residual blocks DISPLAYFORM2, depth of each residual block L =Õ, channel size C = O, filter size K ∈ {2, . . ., D}, norm bounds for the convolution part B (conv) = O, and for the fully-connected part B (fc) > 0 (log B (fc) = O(log N)) such that for sufficiently large N, the clipped ERM estimatorf of F: DISPLAYFORM3. Here, DISPLAYFORM4 Since the estimation error rate of the β-Hölder class is DISPLAYFORM5 (see, e.g.,), Corollary 5 implies that our CNN can achieve the minimax optimal rate up to logarithmic factors even the width D, the channel size C, and the filter size K are constant with respect to the sample size N. In this paper, we established new approximation and statistical learning theories for CNNs by utilizing the ResNet-type architecture of CNNs and the block-sparse structure of FNNs. We proved that any M -way block-sparse FNN is realizable using CNNs with O(M) additional parameters, when the width of the FNN is fixed. Using this , we derived the approximation and estimation errors for CNNs from those for block-sparse FNNs. Our theory is general because it does not depend on a specific function class, as long as we can approximate it with block-sparse FNNs. To demonstrate the wide applicability of our , we derived the approximation and error rates for the Barron class and Hölder class in almost same manner and showed that the estimation error of CNNs is same as that of FNNs, even if the CNNs have a constant channel size, filter size, and width with respect to the sample size. The key techniques were careful evaluations of the Lipschitz constant of CNNs and non-trivial weight parameter rescaling of FNNs. One of the interesting open questions is the role of the weight rescaling. We critically use the homogeneous property of the ReLU activation function to change the relative scale between the block-sparse part and the fully-connected part, if it were not for this property, the estimation error rate would be worse. The general theory for rescaling, not restricted to the Barron nor Hölder class would be beneficial for deeper understanding of the relationship between the approximation and estimation capabilities of FNNs and CNNs. Another question is when the approximation and estimation error rates of CNNs can exceed that of FNNs. We can derive the same rates as FNNs essentially because we can realize block-sparse FNNs using CNNs that have the same order of parameters (see Theorem 1). Therefore, if we dig into the internal structure of FNNs, like repetition, more carefully, the CNNs might need fewer parameters and can achieve better estimation error rate. Note that there is no hope to enhance this rate for the Hölder case (up to logarithmic factors) because the estimation rate using FNNs is already minimax optimal. It is left for future research which function classes and constraints of FNNs, like block-sparseness, we should choose. For tensor a, a +:= a∨0 where maximum operation is performed in element-wise manner. Similarly a −:= −(−a ∨ 0). Note that a = a + − a − holds for any tensor a. For normed spaces (V, · V), (W, · W) and linear operator T: V → W we denote the operator norm of T by T op:= sup v V =1 T v W. For a sequence w = (w,..., w (L) ) and l ≤ l, we denote its subsequence from the l-th to l -th elements by w[l : l]:= (w (l),..., w (l) ). 1 P equals to 1 if the statement P is true, equals to 0 otherwise. DISPLAYFORM0, we realize a CNN f (CNN) using M residual blocks by "serializing" blocks in the FNN and converting them into convolution layers. First, we double the channel size using the m = 0 part of CNN (i.e., D (L0) 0 = 2). We will use the first channel for storing the original input signal for feeding to downstream (i.e., m ≥ 1) blocks and the second one for accumulating the output of each blocks, that is, m size-1 filters made from the weight parameters of the corresponding layer of the FNN. Observing that the convolution operation with size-1 filter is equivalent to a dimension-wise affine transformation, the first coordinate of the output of l-th layer of the CNN is inductively same as that of the m-th block of the FNN. After computing the m-th block FNN using convolutions, we add its output to the accumulating channel in the identity mapping. Finally, we pick the first coordinate of the accumulating channel and subtract the bias term using the final affine transformation. We relate the approximation error of Theorem 2 with the estimation error using the covering number of the hypothesis class F (CNN). Although there are several theorems of this type, we employ the one in due to its convenient form (Lemma 5). We can prove that the logarithm of the covering number is upper bounded by M 2 log((B (conv) ∨ B (fc) )M 1 /ε) (Lemma 4) using the similar techniques to the one in. Theorem 2 is the immediate consequence of these two lemmas. To prove Cororraly 1, we set M = O(N α) for some α ≥ 0. Then, under the assumption of the corolarry, we have DISPLAYFORM0 from Theorem 2. The order of the right hand side with respect to N is minimized when α = DISPLAYFORM1 and b ∈ R such that 1. DISPLAYFORM2 Proof. First, observe that the convolutional layer constructed from u = [u 1 . . . u K] ∈ R K×1×1 takes the inner product with the first K elements of the input signal: DISPLAYFORM3 K×1×1 works as the "left-translation" by K − 1. Therefore, we should define w so that it takes the inner product with the K left-most elements in the first channel and shift the input signal by K − 1 with the second channel. Specifically, we define DISPLAYFORM4 We set b:= (0, . . ., 0 L0 − 1 times, t). Then w and b satisfy the condition of the lemma. The following lemma shows that we can convert any linear CNN to a ReLU CNN that has approximately 4 times larger parameters. This type of lemma is also found in (Lemma 2.3). DISPLAYFORM0 be filter sizes. Let (l). Consider the linear convolution layers constructed from w and b: DISPLAYFORM1 DISPLAYFORM2 ∞, and DISPLAYFORM3 Proof. We definew andb as follows: DISPLAYFORM4 By definition, a pair (w,b) satisfies the conditions FORMULA24 and FORMULA27. For any x ∈ R D, we set y DISPLAYFORM5 for any α, β ∈ [D]. Summing them up and using the definition ofb DISPLAYFORM6 (y DISPLAYFORM7 (y DISPLAYFORM8 (y DISPLAYFORM9 −(w (l+1) ) α,:,: (y DISPLAYFORM10 for any α, β ∈ [D]. Again, by taking the summation and using the definition ofb (l+1), we get DISPLAYFORM11.By applying ReLU, we get DISPLAYFORM12 By using the induction hypothesis, we get DISPLAYFORM13 Therefore, the claim holds for l + 1. By induction, the claim holds for L, which is what we want to prove. We can concatenate two CNNs with the same depths and filter sizes in parallel. Although it is almost trivial, we state it formally as a proposition. In the following proposition, C and C is not necessarily 1. DISPLAYFORM0 We define w and b in the same way, with the exception that C (l) is replaced with C (l). We definew = (w,...,w (L) ) and DISPLAYFORM1. Then, we have, DISPLAYFORM2 for any x, x ∈ R D×C and any σ: R → R.Note that by the definition of · 0 and · ∞, we have DISPLAYFORM3 and DISPLAYFORM4. We will construct the desired CNN consisting of M residual blocks, whose m-th residual block is made from the ingredients of the corresponding m-th block in f Lm], and w m ). DISPLAYFORM5 [The m = 0 Block]: We prepare a single convolutional layer with 2 output channels and 2 size-1 filters suth that the first filter works as the identity function and the second filter inserts zeros to the second channel. Weight parameters of this convolutional layer are all zeros except single one. We denote this block by Conv 0. DISPLAYFORM6 1×D is the d-th row of the matrix DISPLAYFORM7 m ×D. We apply Lemma 1 and Lemma 2 and obtain ReLU CNNs realizing the hinge functions. By combining them in parallel using Proposition 1, we have a learnable parameter θ DISPLAYFORM8 Since we double the channel size in the m = 0 part, the identity mapping has 2 channels. Therefore, we made Conv ReLU θ m so that it has 2 input channels and neglects the input signals coming from the second one. This is possible by adding filters consisting of zeros appropriately. Next, for l-th layer (l = 2, . . ., L m), we prepare size-1 filters w DISPLAYFORM9 where ⊗ is the Kronecker product of matrices. Intuitively, the l = 2 layer will pick all odd indices of the output of Conv DISPLAYFORM10. By the inductive calculation, we have DISPLAYFORM11 By definition, Conv m has the depth of L 0 + L m − 1, at most 4D DISPLAYFORM12 DISPLAYFORM13 Then, we have DISPLAYFORM14 (the subscript 1 represents the first coordinate).[Final Fully-connected Layer] Finally, we set w:= B We assume w, w ∈ R K×J×I, b, b ∈ R, and x ∈ R D×I unless specified. We have in mind that the activation function σ is either the ReLU function or the identity function id. But the following proposition holds for any 1-Lipschitz function such that σ = 0. Remember that we can treat L w as a linear operator from R D×I to R D×J. We endow R D×I and R D×J with the sup norm and denote the operator norm DISPLAYFORM0 is evaluated as follows: DISPLAYFORM1 Note that the first inequality holds because the ReLU function is 1-Lipschitz. DISPLAYFORM2 In the following propositions in this subsection, we assume W, W ∈ R D×C, b, b ∈ R, and x ∈ R D×C. Again, these propositions hold for any 1-Lipschitz function σ: R → R such that σ = 0. But σ = ReLU or id is enough for us. DISPLAYFORM0 The number of non-zero summand in the summation is at most W 0 and each summand is bounded by W ∞ x ∞ Therefore, we have |FC DISPLAYFORM1 In this section, we denote the architecture of CNNs by DISPLAYFORM2 and the norm constraint on the convolution part by DISPLAYFORM3 and (conv). Then, for any DISPLAYFORM4 DISPLAYFORM5 Proof. We write in shorthand as C DISPLAYFORM6 By Proposition 2 and assumptions w DISPLAYFORM7, it is further bounded by DISPLAYFORM8 by Proposition 2 and 4) DISPLAYFORM9 by Proposition 2 and 5) DISPLAYFORM10 We denote the l-th convolution layer of the m-th block by C (l) m and the m-th residual block of by C m: Proof. By using Proposition 9 inductively, we have DISPLAYFORM11 DISPLAYFORM12 Lemma 3. Let ε > 0. Suppose θ and θ are within distance ε, that is, max m,l w DISPLAYFORM13 We will bound each term of. By Proposition 8 and Proposition 11, DISPLAYFORM14 On the other hand, for m = 0,..., M, DISPLAYFORM15 DISPLAYFORM16 By applying and FORMULA37 to FORMULA37, we have DISPLAYFORM17 For a metric space (M 0, d) and ε > 0, we denote the (external) covering number of DISPLAYFORM0 Proof. The idea of the proof is same as that of Lemma 12 of Schmidt-Hieber (2017 (i.e., 2B DISPLAYFORM1 can be realized by parameters such that every pair of corresponding parameters are in a same bin, then, f − f ∞ ≤ ε by Lemma 3. We make a subset F 0 of F (CNN) by picking up every combination of bins for M 2 parameters. Then, for each f ∈ F (CNN), there exists f 0 ∈ F 0 such that f − f 0 ∞ ≤ ε. There are at most 2BM 1 ε −1 choices of bins for each parameter. Therefore, the cardinality of F 0 is at most DISPLAYFORM2 D.2 PROOF OF THEOREM 2 AND COROLLARY 1We use the lemma in to bound the estimation error of the clipped ERM estimatorf. Since our problem setting is slightly different from one in the paper, we restate the statement. Lemma 5 (cf. Lemma 10). Let F be a family of measurable functions from [−1, 1] D to R. Letf be the clipped ERM estimator of the regression problem described in Section 3.1. Suppose the covering number of F satisfies N (ε, F, · ∞) ≥ 3. Then, Proof. Basically, we convert our problem setting so that it fits to the assumptions of Lemma 10 of • (x n)). Then, the probability that D is drawn from P ⊗N is same as the probability that D is drawn from P ⊗N where P is the joint distribution of (X, Y). Also, we can show thatf is the ERM estimator of the regression problem Y = f DISPLAYFORM3 • +ξ using the dataset D:f 1 ∈ arg min f ∈F R D (f). We apply the Lemma 10 of with n ← N, d ← D, ε ← 1, δ ← 1 N, ∆ n ← 0, F ← F, F ← 2F,f ←f 1 and use the fact that the estimation error of the clipped ERM estimator is no worse than that of the ERM estimator, that is, Next, we prove the existence of a block-sparse FNN with constant-width blocks that optimally approximates a given β-Hölder function. It is almost same as the proof of Theorem 5 of. However, we need to construct the FNN so that it has a block-sparse structure. FORMULA24 ) layers. It is left for future research whether our can extend to the ResNet-type CNNs with pooling or Batch Normalization layers. Second, our CNN does not have ReLU activation after the junction points and the final layer of the 0-th block, while they have in the original ResNet. We choose this design to make proofs simpler. We can easily extend our to the architecture that adds the ReLU activations to those points with slight modifications using similar techniques appeared in Lemma 2 of the appendix. DISPLAYFORM4 DISPLAYFORM5"

We propose a generic framework that allows for exploiting the low-rank structure in both planning and deep reinforcement learning.

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.