Question: Represent the scientific article to find a short TLDR

Question: Prefrontal cortex (PFC) is a part of the brain which is responsible for behavior repertoire. Inspired by PFC functionality and connectivity, as well as human behavior formation process, we propose a novel modular architecture of neural networks with a Behavioral Module (BM) and corresponding end-to-end training strategy. This approach allows the efficient learning of behaviors and preferences representation. This property is particularly useful for user modeling (as for dialog agents) and recommendation tasks, as allows learning personalized representations of different user states. In the experiment with video games playing, the show that the proposed method allows separation of main task’s objectives andbehaviors between different BMs. The experiments also show network extendability through independent learning of new behavior patterns. Moreover, we demonstrate a strategy for an efficient transfer of newly learned BMs to unseen tasks. Humans are highly intelligent species and are capable of solving a large variety of compound and open-ended tasks. The performance on those tasks often varies depending on a number of factors. In this work, we group them into two main categories: Strategy and Behaviour. The first group contains all the factors leading to the achievement of a defined set of goals. On the other hand, Behaviour is responsible for all the factors not directly linked to the goals and having no significant effect on them. Examples of such factors can be current sentiment status or the unique personality and preferences that affect the way an individual makes decisions. Existing Deep Networks have been focused on learning of a Strategy component. This was achieved by optimization of a model for defined sets of goals, also the goal might be decomposed into sub-goals first, as in FeUdal Networks BID29 or Policy Sketches approach BID1. Behavior component, in turn, obtained much less attention from the DL community. Although some works have been conducted on the identification of Behavior component in the input, such as works in emotion recognition BID15 BID11 BID17. To the best of our knowledge, there was no previous research on incorporation on Behavior Component or Behavior Representation in Deep Networks before. Modeling Behaviour along with Strategy component is an important step to mimicking a real human behavior and creation of robust Human-Computer Interaction systems, such as a dialog agent, social robot or recommendation system. The early work of artificial neural networks was inspired by brain structure BID9 BID16, and the convolution operation and hierarchical layer design found in the network designed for visual analytic are inspired by visual cortex BID9 BID16. In this work, we again seek inspiration from the human brain architecture. In the neuroscience studies, the prefrontal cortex (PFC) is the region of the brain responsible for the behavioral repertoire of animals BID18 ). Similar to the connectivity of the brain cortex (as shown in Figure 1), we hypothesize that a behavior can be modeled as a standalone module within the deep network architecture. Thus, in this work, we introduce a general purpose modular architecture of deep networks with a Behavioural Module (BM) focusing on impersonating the functionality of PFC.Apart from mimicking the PFC connectivity in our model, we also borrow the model training strategy from human behavior formation process. As we are trying to mimic the functionality of a human brain we approached the problem from the perspective of Reinforcement Learning. This approach also aligns with the process of unique personality development. According to BID6 and BID5 unique personality can be explained by different dopamine functions caused by genetic influence. These differences are also a reason for different Positive Emotionality (PE) Sensory Cortex (Conv Layers) PFC (Behavior Module) Motor Cortex (FC layers) Figure 1: Abstract illustration of the prefrontal cortex (PFC) connections of the brain BID18 and corresponding parts of the proposed model. patterns (sensitivity to reward stimuli), which are in turn a significant factor in behavior formation process BID5. Inspired by named biological processes we introduce extra positive rewards (referring to positive-stimuli or dopamine release, higher the reward referring to higher sensitivity) to encourage specific actions and provoke the development of specific behavioral patterns in the trained agent. To validate our method, we selected the challenging domain of classic Atari 2600 games BID2, where the simulated environment allows an AI algorithm to learn game playing by repeatedly seek to understand the input space, objectives and solution. Based on this environment and an established agent (i.e. Deep Q-Network (DQN) BID20 ), the behavior of the agent can be represented by preferences over different sets of actions. In other words, in the given setting, each behaviour is represented by a probability distribution over given action space. In real-world tasks, the extra-reward can be represented by the human satisfaction by taken action along with the correctness of the output (main reward).Importantly, the effect of human behavior is not restricted to a single task and can be observed in various similar situations. Although it is difficult to correlate the effect of human behavior on completely different tasks, it is often easier to observe akin patterns in similar domains and problems. To verify this, we study two BM transfer strategies to transfer a set of newly learned BMs across different tasks. As a human PFC is responsible for behavior patterns in a variety of tasks, we also aim to achieve a zero-shot transfer of learned modules across different tasks. The contributions of our work are as follow:• We propose a novel modular architecture with behavior module and a learning method for the separation of behavior from a strategy component.• We provide a 0-shot transfer strategy for newly learned behaviors to previously unseen tasks. The proposed approach ensures easy extendability of the model to new behaviors and transferability of learned BMs.• We demonstrate the effectiveness of our approach on video games domain. The experimental show good separation of behavior with different BMs, as well as promising when transfer learned BMs to new tasks. Along with that, we study the effects of different hyper-parameters on the behavior separation process. Task separation is an important yet relatively unexplored topic in deep learning. BID22 BID22 explored this idea by simulating a simplified primate visual cortex by separation a network into two parts, responsible for shape classification task and shape localization on a binary image, respectively The topic was further studied in BID12 BID13 b), however, due to the limitations in computational resources at that time, it has not gotten much advancement. Recently, number researchers have revisited the idea of task separation and modular networks with evolutionary algorithms. So in BID28 BID28 and BID21 applied neuroevolution algorithms to evolve predefined modules responsible for the problem subtasks, where improved performance was reported when compared against monolithic architectures. BID23 BID24 proposed a neuroevolution approach to develop a multi-modular network capable of learning different agent behaviors. The module structure and the number of modules in the network were evolved in the training process. Although the multi-module architecture achieved better performance, it has not achieved separation of the tasks among the modules. A number of evolved modules appeared redundant and not used in the test phase, while others have used shared neurons. Moreover, the architecture was fixed once learned and did not assume changes in the structure. The same approach with modifications in mutation strategy' BID25, genome encoding BID27 and task complexity BID25, but has not achieved significant performance. In 2016, BID4 proposed to use a coevolutionary algorithm for domain transfer problem to avoid training from the scratch. It first independently learns a pool of networks on different Atari2600 games, During the transfer phase, the networks were frozen and used as a'building blocks' in a new network while combined with newly evolved neurons. In 2017, BID8 introduced PathNet to address the task-transfer module on the example of Atari2600 games. PathNet has a fixed size architecture (L layers by N modules), where each module was represented by either convolutional or fully-connected block. During the training phase, authors applied the tournament genetic algorithm to learn active paths between modules along with the weights. Once the task was learned, active modules and paths were frozen and the new task could start learning a new path. Recently proposed FeUdal Networks architecture BID29, also proposed a Modular design for Reinforcement Learning problems with sub-goals. In this work authors use Manager and Worker modules for learning abstract goals and primitive actions respectively. FeUdal networks are designed to tackle environments with long-term credit assignment and sparse reward signals. The modules in the named architecture are not transferable and designed to learn different time-span goal embeddings. BID0 proposed the Neural Module Network for Visual Question Answering (VQA) task. It consists of separate modules responsible for different tasks (e.g. Find, Transform, Combine, Describe and Measure modules), which could be combined in different ways depending on the network input. A similar dynamic architecture was proposed and applied to robot manipulator task BID7. The model was end-to-end trained and consisted of two modules (i.e. robotspecific and task-specific) and achieved good performance on a zero-shot learning task. The Modular Neural Network was also applied in Reinforcement Learning task in a robotics environment BID1. In this work, each module was responsible for a separate sub-task of the main task. However, the modules could be combined only in a sequential manner. Most of the previous works focused on multi-task problems or problems with sub-goals where the modules were responsible for learning explicit sub-functionality directly affecting the model performance. Our approach is different in a sense, we learn a behavior module responsible for representation of user sentiment states or preferences not affecting the main goals. This approach leads to high adaptability of the network performance to new preferences or states of an enduser without retraining of the whole network, expandability of the network to future variations, removability of BMs in case of unknown preferences, as well as high-potential to transfer of the learned representations to unseen tasks. To the best of our knowledge, there are no similar approaches. The goal of our modular network is to introduce Behavior component into Deep Networks, ensure separation of the behavior and main task functionalities into different components of the network and provide a strategy for an efficient transfer of learned behaviors. Our model has three main parts The Main Network is responsible for the main task (strategy component) functionality, a replaceable/removable Behavior Module (BM) encodes the agent behavior and separate it from the main network, and the Discriminator is used during the transfer stage and helps to learn similar feature representations among different tasks. An overview of the proposed network architecture is shown in FIG0. In the given architecture Convolutional layers correspond to (Visual) Sensory cortex, Fully-Connected layers of the Main Network to the Motor Cortex and Behaviour Module to PFC from Figure 1. In this work, we adopt the deep Q-Network (DQN) with target network and memory replay BID20 to solve the main task (denoted as main network). DQN has reported good performance The DQN has a fairly simple network structure, which consists of 3 consecutive convolutional layers followed by two fully-connected (fc) layers (see FIG0 . All the layers, except the last one, use ReLU activation functions. The network output is represented by the set of expected future discounted rewards for each of the available actions. The output obeys Bellman-equation and Root-Mean-Square Error is applied as a loss function (L m) during the training phase. In this work, we are interested in the separation of the behavioral component from the main task functionality. Specifically, we design a network where the behavior is modeled with a replaceable and removable module, which is denoted as Behavioral Module (BM). The BM is supposed to have no significant effect on the performance on the main task. The BM is modeled as two fc layers with the first layer having ReLU activation function and the second layer having linear activation. The proposed BM input is the output from the last convolutional layer of the main network, while its output is directly fed to the first fc layer of the main network. This architecture follows the PFC connectivity pathways described in Figure 1. The forward pass of the network is represented by the following equations: DISPLAYFORM0 where I is the network input, j is the index of the current behaviour, l i is the output of the i-th layer, l bi is the output of i-th layer of a BM, f i is the activation function at the i-th layer, and denotes 2d convolution operation. The Main Network contains layers from l 1 to l 5. Note that l b becomes zero vector if no behavior is required or enforced. The summation operator at the layer 4 ensures the influence of BM can be easily removed from the main network (l b is zeros in this case). It also minimizes the effects of BM on the gradients flow during the backpropagation stage. The training is conducted in an end-to-end manner as presented in Algorithm 1.In our approach, the introduction of BM does not require additional loss function and the loss is directly incorporated into the main network loss (L m). To do this we introduce additional rewards for desired behaviors of the agent, similar to PE effect on human behavior formation process. In our i ← random(1,size(B)) setting behavior is defined by agent's preference to play specific actions. Thus, each preferred action played was rewarded with an extra action reward. The action reward is subject to the game of interest and its designing process will be described in the Experiment section. One of the advantages of network modularization is to allow the learned BMs to be transferred to a different main network with minimal or no network re-training. This property is useful for knowledge sharing among a group of models in problems with a variety of possible implementations, changing environments and open-ended tasks. Once task objectives have changed or new behaviors were developed in another model, the target model can just apply a new module without any updates or training of the main network. This property allows easy extension of a learned model and knowledge share across different models without any additional training. The learned BMs from the previous section is used during the transfer phase. In this work, we consider two approaches, namely fine-tuning and adversarial transfer. The first approach uses a source task model and fine-tunes it for a new target task, where BMs are kept unchanged. In the adversarial transfer approach, we introduce a discriminator network (as shown in FIG0, which enforces the convolutional layers to learn features similar to features of the source task. To do so, we adopt the domain-adversarial training BID10 . In this case, the discriminator network has 2 fully-connected layers with Relu and Softmax non-linearity functions, which tries to classify output of the last convolutional layers as being from the source or target task. Different from the original paper, we minimize the softmax loss at the discriminator output and flip the gradient sign at the convolutional layers. The weights update can be formulated as follows: DISPLAYFORM0 where θ dj t are the parameters of the discriminator at timestep t, θ aj t are the parameters of the convolutional layers at timestep t, β is the parameter as described by BID10, L a is the classification loss of the descriminator. In this section, we delineate the experiments that focus on two main aspects of this work: the separation of agent's behavior from the main task, and cross-task transfer of learned behaviors. In order to demonstrate the flexibility and extendability of the proposed network, we also considered zero-shot setting so that an end-user will not require additional training for the case of behavior module transfer. We evaluate the proposed novel modular architecture on the classic Atari 2600 games BID2. The main reason is that video games provide a controlled environment, where it is easier to control agent behavior by representing it with distribution over available actions. In addition, Atari 2600 emulator does not require data collection and labeling, yet it provides a wide range of tasks in terms of different games. The loss function used to encourage the learning of a specific behavior is described in the next section. In this work, we evaluate our architecture on four games, namely Pong, Space Invaders, Demon Attack and Breakout, which consist of four available actions, namely No action, Fire, Up/Right, and Down/Left. Data pre-processing: The input of the network is a stack of 4 game-frames. Each frame was converted into a gray-scale image and resized to 84 × 84 pixels. As the consecutive game frame contain similar information, we sample one game frame on every fourth frame. and an additional action. Additionally, we tested the effect of zero-behavior (i.e. BM0) presence during the training stage. In other words, no actions were stimulated and BM is not applied. Training: To train the proposed network (i.e. main network and BMs), we used the standard Qlearning algorithm and the corresponding loss function using Bellman equation BID19. The training used a combined reward represented by the sum of game score and individual action rewards. The magnitude of the additional reward was represented by an average reward per frame of the game. All the rewards obtained during the game were clipped at 1 and -1 BID20.Evaluation Metrics: We evaluate the proposed models using two metrics. The first metric focus on the game play performance. As each game has different reward scales, we compute the mean of game scores ratios achieved by the proposed modular network and the Vanilla DQN model TAB0. We refer to this metric as Average Main Score Ratio (AMSR). If AMSR is equal to 1, it means the trained model with BM performs equally well as the Vanilla DQN model. Similarly, AMSR higher than 1 indicate our proposed model perform better than the Vanilla DQN, or worst if it is lower than 1. Thus, AMSR that is close to or more than 1 would indicate our modular network is comparable to baseline. The second metric reflects the capability of the proposed modular network in term of modeling the desired behavior. To do this, we define the Behavior Distance (BD) by calculating the Bhattacharyya distance BID3 between the BMs' action distribution to an ideal target distribution. The target distribution is computed by divide 1 with the rewarded actions (i.e. BM5's target distribution is [0.0, 0.0, 0.5, 0.5]). In the ideal case, the BD of the learned network should be close to 1 as our training only encourages over certain actions set. This experiment aims to show that the behavior functionality can be separated from the main task and learned independently. To demonstrate that we conducted the training in two stages. During the Stage 1, we first trained the main network with five behaviors (i.e., BM0 -BM2, BM4 -BM5 and BM8) using Algorithm 1. Given the trained network from Stage 1, Stage 2 focused on training of the remaining BMs (i.e. BM3, BM6, and BM7) while the main network was frozen. This includes behaviors stimulating 1, 2 and 3 actions, respectively. Effect of key parameters: First of all, we studied the effect of the action reward magnitude on the training process. We started with estimation of average reward per frame value (r) for each game (without additional action rewards) and observed performance of our model with various action rewards (i.e. 0.25r, 0.5r, r, 2r, and 5r). TAB2 show that action reward magnitude directly affects the quality of learned behavior in both stages, where increasing the action reward above r value leads to degradation of the main task's performance. Although additional reward magnitude selection depends on a desired main score and BD, we recommend the value equal to r as it leads to the highest BD score during Stage 2, and as a better functionality separation. Next, to see the effect of other parameters we set the value of the action reward to 0.5r, so that we can observe an effect of the changes on the main reward, as well as the behavior pattern. As the next step, we have studied the effect of the complexity of the BMs on the quality of the learned behaviors by trying a different number of layers. Also, we looked for a better separation of the Behavior component by studying the effects of dropout, BM0 and different learning rate for the Behavior module. According to the TAB3 use of 2 fully-connected layers ed in a significant improvement on the main task score compared to 1-layer version. However, adding more layers did not in a better performance. Similar effect demonstrated a higher BM learning rate compared to the main network TAB4, while lower value leads to lower main scores. Finally implementing a Dropout layer for the BM and using BM0 ed in a higher BD score during Stage 2 and main score during Stage 1.Results: Taking into account hyper-parameter effect we have trained a final model with 2 layer BMs and 0.5 dropout layers, applying 2 times higher BM learning rate, BM0 and action reward equal to r. The trained model showed high main task scores compared to the vanilla DQN model, as well as high similarity of learned behaviors to ideal target distributions at Stage 1, as well as after separate training of BMs at Stage 2 TAB0. Experiments also showed that removing the BMs does not lead to a performance degradation of the model on the main task. Importantly, the effect of the action reward magnitude directly correlated with agents preferences to play rewarded actions, which aligns with the PE effect in human behavior formation process. Thus, the development process of exact behavior pattern can be controlled through variations in action reward magnitude. Therefore, we conclude that the proposed model and training method allows a successful separation of the strategy (main task) and behavior functionalities and further network expansion. Implementation details: To achieve a zero-shot performance of the transferred modules, we aimed to achieve a similar feature representation of the target model to the source model. To achieve that we tested two approaches: fine-tuning and adversarial transfer. In the first case, we have fine-tuned the main network obtained in Stage 1 of Section 4.1 on a new game with frozen Stage 1 BMs, applied to every pair of games and following Algorithm 1. After that, we tested the performance on previously unseen Stage 2 BMs. In adversarial setting we follow the same procedure, but with the use of the Discriminator part FIG0 ). The performance was compared to the of transferring Stage 2 BMs to the best model configuration after Stage 1 from Section 4.1.Results: As it can be seen from the Table 5 even a simple zero-shot transfer of learned BMs based on fine-tuning in a good performance of the model on unseen BMs. BM0 and Stage 1 behaviors achieved close performance to an original network. Although the BD score of zero-shot adversarial transfer is approximately 9% lower, the main task performance of transferred modules on an unseen task is close to a separately trained network. This fact shows that zero-shot transfer of separately learned BMs to unseen tasks in slightly worse performance compared to the separately trained model. This leads to a that target performance of transferred BMs can be achieved through much less training compared to complete network retraining. In this work, we have proposed a novel Modular Network architecture with Behavior Module, inspired by human brain Pre-Frontal Cortex connectivity. This approach demonstrated the successful separation of the Strategy and Behavior functionalities among different network components. This is particularly useful for network expandability through independent learning of new Behavior Modules. Adversarial 0-shot transfer approach showed high potential of the learned BMs to be transferred to unseen tasks. Experiments showed that learned behaviors are removable and do not degrade the performance of the network on the main task. This property allows the model to work in a general setting, when user preferences are unknown. The also align with human behavior formation process. We also conducted an exhaustive study on the effect of hyper-parameters on behavior learning process. As a future work, we are planning to extend the work to other domains, such as style transfer, chat bots, and recommendation systems. Also, we will work on improving module transfer quality. In this appendix, we show the details of our preliminary study on various key parameters. The experiments were conducted on the Behavior Separation task.

represent the tldr for finding the corresponding scientific article

Extendable Modular Architecture is proposed for developing of variety of Agent Behaviors in DQN.

Our paper analyses the tremendous representational power of networks especially with 'skip connections', which may be used as a method  for better generalization.


Question: represent the scientific article to find a short tldr!

Question: A major component of overfitting in model-free reinforcement learning (RL) involves the case where the agent may mistakenly correlate reward with certain spurious features from the observations generated by the Markov Decision Process (MDP). We provide a general framework for analyzing this scenario, which we use to design multiple synthetic benchmarks from only modifying the observation space of an MDP. When an agent overfits to different observation spaces even if the underlying MDP dynamics is fixed, we term this observational overfitting. Our experiments expose intriguing properties especially with regards to implicit regularization, and also corroborate from previous works in RL generalization and supervised learning (SL). Generalization for RL has recently grown to be an important topic for agents to perform well in unseen environments. Complication arises when the dynamics of the environments entangle with the observation, which is often a high-dimensional projection of the true latent state. One particular framework, which we denote by zero-shot supervised framework (a; ;) and is used to study RL generalization, is to treat it analogous to a classical supervised learning (SL) problem -i.e. assume there exists a distribution of MDP's, train jointly on a finite "training set" sampled from this distribution, and check expected performance on the entire distribution, with the fixed trained policy. In this framework, there is a spectrum of analysis, ranging from almost purely theoretical analysis to full empirical on diverse environments ). However, there is a lack of analysis in the middle of this spectrum. On the theoretical side, previous work do not provide analysis for the case when the underlying MDP is relatively complex and requires the policy to be a non-linear function approximator such as a neural network. On the empirical side, there is no common ground between recently proposed empirical benchmarks. This is partially caused by multiple confounding factors for RL generalization that can be hard to identify and separate. For instance, an agent can overfit to the MDP dynamics of the training set, such as for control in Mujoco . In other cases, an RNN-based policy can overfit to maze-like tasks in exploration, or even exploit determinism and avoid using observations . Furthermore, various hyperparameters such as the batch-size in SGD , choice of optimizer , discount factor γ and regularizations such as entropy and weight norms can also affect generalization. Due to these confounding factors, it can be unclear what parts of the MDP or policy are actually contributing to overfitting or generalization in a principled manner, especially in empirical studies with newly proposed benchmarks. In order to isolate these factors, we study one broad factor affecting generalization that is most correlated with themes in SL, specifically observational overfitting, where an agent overfits due to properties of the observation which are irrelevant to the latent dynamics of the MDP family. To study this factor, we fix a single underlying MDP's dynamics and generate a distribution of MDP's by only modifying the observational outputs. Our contributions in this paper are the following: 1. We discuss realistic instances where observational overfitting may occur and its difference from other confounding factors, and design a parametric theoretical framework to induce observational overfitting that can be applied to any underlying MDP. 2. We study observational overfitting with linear quadratic regulators (LQR) in a synthetic environment and neural networks such as multi-layer perceptrons (MLPs) and convolutions in classic Gym environments. A primary novel we demonstrate for all cases is that implicit regularization occurs in this setting in RL. We further test the implicit regularization hypothesis on the benchmark CoinRun from using MLPs, even when the underlying MDP dynamics are changing per level. 3. In the Appendix, we expand upon previous experiments by including full training curve and hyperparamters. We also provide an extensive analysis of the convex one-step LQR case under the observational overfitting regime, showing that under Gaussian initialization of the policy and using gradient descent on the training cost, a generalization gap must necessarily exist. The structure of this paper is outlined as follows: Section 2 discusses the motivation behind this work and the synthetic construction to abstract certain observation effects. Section 3 demonstrates numerous experiments using this synthetic construction that suggest implicit regularization is at work. Finally, Section 3.4 tests the implicit regularization hypothesis on CoinRun, as well as ablates various ImageNet architectures and margin metrics in the Appendix. We start by showing an example of observational overfitting in Figure 1. This example highlights the issues surrounding MDP's with rich, textured observations -specifically, the agent can use any features that are correlated with progress, even those which may not generalize across levels. This is an important issue for vision-based policies, as many times it is not obvious what part of the observation causes an agent to act or generalize. Figure 1: Example of observational overfitting in Sonic from Gym Retro . Saliency maps highlight (in red) the top-left timer and objects such as clouds and textures because they are correlated with progress, as they move backwards while agent is moving forwards. The agent could memorize optimal actions for training levels even if its observation was only from the timer, and "blacking-out" the timer consistently improved generalization performance (see Appendix A.2.3). Currently most architectures used in model-free RL are simple (with fewer than one million parameters) compared to the much larger and more complex ImageNet architectures used for classification. This is due to the fact that most RL environments studied either have relatively simple and highly structured images (e.g. Atari) compared to real world images, or conveniently do not directly force the agent to observe highly detailed images. For instance in large scale RL such as DOTA2 or Starcraft 2 , the agent observations are internal minimaps pertaining to object xy-locations, rather than human-rendered observations. Several artificial benchmarks (b;) have been proposed before to portray this notion of overfitting, where an agent must deal with a changing however, a key difference in our work is that we explicitly require the "" to be correlated with the progress rather than loosely correlated (e.g. through determinism between the and the game avatar) or not at all. This makes a more explicit connection to causal inference (; ;) where spurious correlations between ungeneralizable features and progress may make training easy, but are detrimental to test performance because they induce false attributions. Previously, many works interpret the decision-making of an agent through saliency and other network visualizations (; on common benchmarks such as Atari. Other recent works such as analyze the interactions between noise-injecting explicit regularizations and the information bottleneck. However, our work is motivated by learning theoretic frameworks to capture this phenomena, as there is vast literature on understanding the generalization properties of SL classifiers (; ;) and in particular neural networks (b; ; ; c). For an RL policy with high-dimensional observations, we hypothesize its overfitting can come from more theoretically principled reasons, as opposed to purely good inductive biases on game images. As an example of what may happen in high dimensional observation space, consider linear least squares regression task where given the set X ∈ R m×d and Y ∈ R m, we want to find w ∈ R d that minimizes X,Y (w) = Y − Xw 2 where m is the number of samples and d is the input dimension. We know that if X X is full rank (hence d ≤ m), X,Y has a unique global minimum w * = (X X) −1 X Y. On the other hand if X X is not full rank (eg. when m < d), then there are many global minima w * such that Y = Xw * 1. Luckily, if we use any gradient based optimization to minimize the loss and initialize with w = 0, the solution will only span column spaces of X and converges to minimum 2 norm solution among all global minima due to implicit regularization . Thus a high dimensional observation space with a low dimensional state space can induce multiple solutions, some of which are not generalizable to other functions or MDP's but one could hope that implicit regularization would help avoiding this issue. We analyze this case in further detail for the convex one-step LQR case in Section 3.1 and Appendix A.4.3. In the zero-shot framework for RL generalization, we assume there exists a distribution D over MDP's M for which there exists a fixed policy π opt that can achieve maximal return on expectation over MDP's generated from the distribution. An appropriate finite training set M train = {M 1, . . ., M n} can then be created by repeatedly randomly sampling M ∼ D. Thus for a MDP M and any policy π, expected episodic reward is defined as R M (π). In many empirical cases, the support of the distribution D is made by parametrized MDP's where some process, given a parameter θ, creates a mapping θ → M θ (e.g. through procedural generation), and thus we may simplify notation and instead define a distribution Θ that induces D, which implies a set of samples Θ train = {θ 1, . . ., θ n} also induces a M train = {M 1, . . ., M n}, and we may redefine reward as R M θ (π) = R θ (π). 1 Given any X with full rank X X, it is possible to create many global minima by projecting the data onto As a simplified model of the observational problem from Sonic, we can construct a mapping θ → M θ by first fixing a base MDP M = (S, A, r, T), which corresponds to state space, action space, reward, and transition. The only effect of θ is to introduce an additional observation function φ θ: S → O, where the agent receives input from the high dimensional observation space O rather than from the state space S. Thus, for our setting, θ actually parameterizes a POMDP family which can be thought of as simply a combination of a base MDP M and an observational function φ θ, hence Let Θ train = {θ 1, . . ., θ n} be a set of n i.i.d. samples from Θ, and suppose we train π to optimize reward against is the average reward over this empirical sample. We want to generalize to the distribution Θ, which can be expressed as the average episode reward R over the full distribution, i.e. Thus we define the generalization gap as J Θ (π) − J Θ (π). We can model the effects of Figure 1 more generally, not specific to sidescroller games. We assume that there is an underlying state s (e.g. xy-locations of objects in a game), whose features may be very well structured, but that this state has been projected to a high dimensional observation space by φ θ. To abstract the notion of generalizable and non-generalizable features, we construct a simple and natural candidate class of functions, where In this setup, f (·) is a function invariant for the entire MDP population Θ, while g θ (·) is a function dependent on the sampled parameter θ. h is a "combination" function which combines the two outputs of f and g to produce a final observation. While f projects this latent data into salient and important, invariant features such as the avatar, monsters, and items, g θ projects the latent data to unimportant features that do not contribute to extra generalizable information, and can cause overfitting, such as the changing or textures. A visual representation is shown in Figure 2. This is a simplified but still insightful model relevant in more realistic settings. For instance, in settings where g θ does matter, learning this separation and task-identification could potentially help fast adaptation in meta-learning . From now on, we denote this setup as the (f, g)-scheme. This setting also leads to more interpretable generalization bounds -Lemma 2 of provides a high probability (1 − δ) bound for the "intrinsic" generalization gap when m levels are, where is the Rademacher Complexity under the MDP, where θ i are the ζ i parameters used in the original work, and the transition T and initialization I are fixed, therefore omitted, to accommodate our setting. The Rademacher Complexity term captures how invariant policies in the set Π with respect to θ. For most RL benchmarks, this is not interpretable due to multiple confounding factors such as the varying level dynamics. For instance, it is difficult to imagine what behaviors or network weights a policy would possess in order to produce the same total rewards, regardless of changing dynamics. However, in our case, because the environment parameters θ are only from g θ, the Rademacher Complexity is directly based on how much the policy "looks at" g θ. More formally, let Π * be the set of policies π * which are not be affected by changes in g θ; i.e. ∇ θ π * (φ θ (s)) = 0 ∀s and thus R θ (π *) = R const ∀θ, which implies that the environment parameter θ has no effect on the reward; Normally in a MDP such as a game, the concatenation operation may be dependent on time (e.g. textures move around in the frame). In the scope of this work, we simplify the concatenation effect and assume h(·) is a static concatenation, but still are able to demonstrate insightful properties. We note that this inductive bias on h allows explicit regularization to trivially solve this problem, by penalizing a policy's first layer that is used to "view" g θ (s) (Appendix A.1.1), hence we only focus on implicit regularizations. This setting is naturally attractive to analyzing architectural differences, as it is more closely related in spirit to image classifiers and SL. One particular line of work to explain the effects of certain architectural modifications in SL such as overparametrization and residual connections is implicit regularization (a; ;), as overparametrization through more layer depth and wider layers has proven to have no p -regularization equivalent , but rather precondition the dynamics during training. Thus, in order to fairly experimentally measure this effect, we always use fixed hyperparameters and only vary based on architecture. In this work, we only refer to architectural implicit regularization techniques, which do not have a explicit regularization equivalent. Some techniques e.g. coordinate descent are equivalent to explicit 1 -regularization. We first analyze the case of the LQR as a surrogate for what may occur in deep RL, which has been done before for various topics such as sample complexity and model-based RL. This is analogous to analyzing linear/logistic regression as a surrogate to understanding extensions to deep SL techniques (a;). In particular, this has numerous benefits -the cost (negative of reward) function is deterministic, and allows exact gradient descent (i.e. the policy can differentiate through the cost function) as opposed to necessarily using stochastic gradients in normal RL, and thus can cleanly provide evidence of implicit regularization. Furthermore, in terms of gradient dynamics and optimization, LQR readily possesses nontrivial qualities compared to linear regression, as the LQR cost is a non-convex function but all of its minima are global minima . To show that overparametrization alone is an important implicit regularizer in RL, LQR allows the use of linear policies (and consequently also allows stacking linear layers) without requiring a stochastic output such as discrete Gumbel-softmax or for the continuous case, a parametrized Gaussian. This is setting able to show that overparametrization alone can affect gradient dynamics, and is not a consequence of extra representation power due to additional non-linearities in the policy. There have been multiple recent works on this linear-layer stacking in SL and other theoretical problems such as matrix factorization and matrix completion (b; a;), but to our knowledge, we are the first to analyze this case in the context of RL generalization. We explicitly describe setup as follows: for a given θ, we let f (s) = W c · s, while g θ (s) = W θ · s where W c, W θ are semi-orthogonal matrices, to prevent information loss relevant to outputting the optimal action, as the state is transformed into the observation. Hence, if s t is the underlying state at time t, then the observation is o t = W c W θ s t and thus the action is a t = Ko t, where K is the policy matrix. While W c remains a constant matrix, we sample W θ randomly, using the "level ID" integer θ as the seed for random generation. In terms of dimensions, if s is of shape d state, then f also projects to a shape of d state, while g θ projects to a much larger shape d noise, implying that the observation to the agent is of dimension d signal + d noise. In our experiments, we set as default (d signal, d noise) =. If P is the unique minimizer of the original cost function, then the unique minimizer of the population. However, if we have a single level, then there exist multiple solutions, ∀α. This extra bottom component W θ P T causes overfitting. In Appendix A.4.3, we show that in the 1-step LQR case (which can be extended to convex losses whose gradients are linear in the input), gradient descent cannot remove this component, and thus overfitting necessarily occurs. Furthermore, we find that increasing d noise increases the generalization gap in the LQR setting. This is empirically verified in Figure 3 using an actual non-convex LQR loss, and the suggest that the gap scales by O(√ d noise). In terms of overparametrization, we experimentally added more (100 × 100) linear layers K = K 0 K 1,..., K j and increased widths for a 2-layer case (Figure 3), and observe that both settings reduce the generalization gap, and also reduce the norms (spectral, nuclear, Frobenius) of the final end-to-end policy K, without changing its expressiveness. This suggests that gradient descent under overparametrization implicitly biases the policy towards a "simpler" model in the LQR case. A ≥ 1 ∀A and A * = A F, A 1. We see that the naive spectral bound diverges at 2 layers, and the weight-counting sums are too loose. As a surrogate model for deep RL, one may ask if the generalization gap of the final end-to-end policy K can be predicted by functions of the layers K 0,..., K j. This is an important question as it is a required base case for predicting generalization when using stochastic policy gradient with nonlinear activations such as ReLU or Tanh. From examining the distribution of singular values on K (Appendix A.1.1), we find that more layers does not bias the policy towards a low rank solution in the nonconvex LQR case, unlike (b) which shows this does occur for matrix completion, and in general, convex losses. Ultimately, we answer in the negative: intriguingly, SL bounds have very little predictive power in the RL domain case. To understand why SL bounds may be candidates for the LQR case, we note that as a basic smoothness bound C(K) − C(K) ≤ O(K − K) (Appendix A.4) can lead to very similar reasoning with SL bounds. Since our setup is similar to SL in that "LQR levels" which may be interpreted as a dataset, we use bounds of the form ∆·Φ, where ∆ is a "macro" product term ∆ = ). However, the Φ terms increase too rapidly as shown in Figure 3. Terms such as Frobenius product and Fischer-Rao are effective for the SL depth case, but are both ineffective in the LQR depth case. For width, the only product which is effective is the nuclear norm product. In Section 3.1, we find that observational overfitting exists and overparametrization potentially helps in the linear setting. In order to analyze the case when the underlying dynamics are nonlinear, we let M be a classic Gym environment and we generate a M θ = (M, w θ) by performing the exact same (f, g)-scheme as the LQR case, i.e. sampling θ to produce an observation function We again can produce training/test sets of MDPs by repeatedly sampling θ, and for policy optimization, we use Proximal Policy Gradient . Although bounds on the smoothness term R θ (π) − R θ (π) affects upper bounds on Rademacher Complexity (and thus generalization bounds), we have no such theoretical guarantees in the Mujoco case as it is difficult to analyze the smoothness term for complicated transitions such as Mujoco's physics simulator. However, in Figure 4, we can observe empirically that the underlying state dynamics has a significant effect on generalization performance as the policy nontrivially increased test performance such as in CartPole-v1 and Swimmer-v2, while it could not for others. This suggests that the Rademacher complexity and smoothness on the reward function vary highly for different environments. Figure 4: Each Mujoco task is given 10 training levels (randomly sampling g θ parameters). We used a 2-layer ReLU policy, with 128 hidden units each. Dimensions of outputs of (f, g) were respectively. Even though it is common practice to use basic (2-layer) MLPs in these classic benchmarks, there are highly nontrivial generalization effects from modifying on this class of architectures. Our in Figures 5 and 6 show that increasing width and depth for basic MLPs can increase generalization and is significantly dependent on the choice of activation, and other implicit regularizations such as using residual layers can also improve generalization. Specifically, switching between ReLU and Tanh activations produces different during overparametrization. For instance, increasing Tanh layers improves generalization on CartPole-v1, and width increase with ReLU helps on Swimmer-v2. Tanh is noted to consistently improve generalization performance. However, stacking Tanh layers comes at a cost of also producing vanishing gradients which can produce subpar training performance, for e.g. HalfCheetah. To allow larger depths, we use ReLU residual layers, which also improves generalization and stabilizes training. Previous work did not find such an architectural pattern for GridWorld environments, suggesting that this effect may exist primarily for observational overfitting cases. While there have been numerous works which avoid overparametrization on simplifying policies or compactifying networks , we instead find that there are generalization benefits to overparametrization even in the nonlinear control case. From the above with MLPs, one may wonder if similar may carry to convolutional networks, as they are widely used for vision-based RL tasks. As a ground truth reference for our experiment, we the canonical networks proven to generalize well in the dataset CoinRun, which are from worst to best, NatureCNN Mnih et al., IMPALA Espeholt et al. (2018, and IMPALA-LARGE (IMPALA with more residual blocks and higher convolution depths), which have respective parameter numbers (600K, 622K, 823K). We setup a similar (f, g)-scheme appropriate for the inductive bias of convolutions, by passing the vanilla Gym 1D state corresponding to joint locations and velocities, through multiple deconvolutions. We do so rather than using the RGB image from env.render to enforce that the actual state is indeed low dimensional and minimize complications in experimentation, as e.g. inference of velocity information would require frame-stacking. Specifically in our setup, we project the actual state to a fixed length, reshaping it into a square, and replacing f and g θ both with the same orthogonally-initialized deconvolution architecture to each produce a 84 × 84 image (but g θ 's network weights are still generated by θ 1, ..., θ m similar to before). We combine the two outputs by using one half of the "image" from f, and one half from g θ, as shown back in Figure 2. shows that the same ranking between the three architectures exists as well on the GymDeconv dataset. We show that generalization ranking among NatureCNN/IMPALA/IMPALA-LARGE remains the same regardless of whether we use our synthetic constructions or CoinRun. This suggests that the RL generalization quality of a convolutional architecture is not limited to real world data, as our test purely uses numeric observations, which are not based on a human-prior. From these findings, one may conjecture that these RL generalization performances are highly correlated and may be due to common factors. One of these factors we suggest is due to implicit regularization. In order to support this claim, we perform a memorization test by only showing g θ's output to the policy. This makes the dataset impossible to generalize to, as the policy network cannot invert every single observation function {g θ1 (·), g θ2 (·),..., g θn (·)} simultaneously. also constructs a memorization test for mazes and grid-worlds, and showed that more parameters increased the memorization ability of the policy. While it is intuitive that more parameters would incur more memorization, we show in Figure 8 that this is perhaps not a complete picture when implicit regularization is involved. Using the underlying MDP as a Swimmer-v2 environment, we see that NatureCNN, IMPALA, IMPALA-LARGE have reduced memorization performances. IMPALA-LARGE, which has more depth parameters and more residual layers (and thus technically has more capacity), memorizes less than IMPALA due its inherent inductive bias. While memorization performance is dampened in 8, we perform another deconvolution memorization test using an LQR as the underlying MDP in Appendix A.1.1 that shows that there can exist specific hard limits to memorization, which also follows the same ranking above. We further test our overparametrization hypothesis from Sections 3.1, 3.2 to the CoinRun benchmark, using unlimited levels for training. For MLP networks, we downsized CoinRun from native 64 × 64 to 32 × 32, and flattened the 32 × 32 × 3 image for input to an MLP. Two significant differences from the synthetic cases are that 1. Inherent dynamics are changing per level in CoinRun, and 2. The relevant and irrelevant CoinRun features change locations across the 1-D input vector. Regardless, in Figure 9, we show that overparametrization can still improve generalization in this more realistic RL benchmark, much akin to (b) which showed that overparametrization for MLP's improved generalization on 32 × 32 × 3 CIFAR-10. Figure 9: Overparametrization improves generalization for CoinRun. While we also extend the case of large-parameter convolutional networks using ImageNet networks in Appendix A.2.1, an important question is how to predict the generalization gap only from the training phase. A particular set of metrics, popular in the SL community are margin distributions , as they deal with the case for softmax outputs which do not explicitly penalize the weight norm of a network, by normalizing the "confidence" margin of the logit outputs. While using margins on state-action pairs (from an on-policy replay buffer) is not technically rigorous, one may be curious to see if they have predictive power, especially as MLPs are relatively simple to norm-bound. We plotted these margin distributions in Appendix A.2.2, but found that the weight norm bounds used in SL are simply too dominant for this RL case. This, with the bound found earlier for the LQR case, suggests that current norm bounds are simply too loose for the RL case even though we have shown overparametrization helps generalization in RL, and hopefully this motivates more of the study of such theory. We have identified and isolated a key component of overfitting in RL as the particular case of "observational overfitting", which is particularly attractive for studying architectural implicit regularizations. We have analyzed this setting extensively, by examining 3 main components: 1. The analytical case of LQR and linear policies under exact gradient descent, which lays the foundation for understanding theoretical properties of networks in RL generalization. 2. The empirical but principled Projected-Gym case for both MLP and convolutional networks which demonstrates the effects of neural network policies under nonlinear environments. 3. The large scale case for CoinRun, which can be interpreted as a case where relevant features are moving across the input, where empirically, MLP overparametrization also improves generalization. We noted that current network policy bounds using ideas from SL are unable to explain overparametrization effects in RL, which is an important further direction. In some sense, this area of RL generalization is an extension of static SL classification from adding extra RL components. For instance, adding a nontrivial "combination function" between f and g θ that is dependent on time (to simulate how object pixels move in a real game) is both an RL generalization issue and potentially video classification issue, and extending to the memory-based RNN case will also be highly beneficial. Furthermore, it is unclear whether such overparametrization effects would occur in off-policy methods such as Q-learning and also ES-based methods. In terms of architectural design, recent works (; ;) have shed light on the properties of asymptotically overparametrized neural networks in the infinite width and depth cases and their performance in SL. Potentially such architectures (and a corresponding training algorithm) may be used in the RL setting which can possibly provide benefits, one of which is generalization as shown in this paper. We believe that this work provides an important initial step towards solving these future problems. We further verify that explicit regularization (norm based penalties) also reduces generalization gaps. However, explicit regularization may be explained due to the bias of the synthetic tasks, since the first layer's matrix may be regularized to only "view" the output of f, especially as regularizing the first layer's weights substantially improves generalization. Figure A2: Explicit Regularization on layer norms. We provide another deconvolution memorization test, using an LQR as the underlying MDP. While fg-Gym-Deconv shows that memorization performance is dampened, this test shows that there can exist specific hard limits to memorization. Specifically, NatureCNN can memorize 30 levels, but not 50; IMPALA can memorize 2 levels but not 5; IMPALA-LARGE cannot memorize 2 levels at all. Training, Test Rewards (f = NULL) IMPALA_2_levels IMPALA_5_levels IMPALA_30_levels IMPALA_LARGE_2_levels NatureCNN_30_levels NatureCNN_50_levels Figure A3: Deconvolution memorization test using LQR as underlying MDP. For reference, we also extend the case of large-parameter convolutional networks using ImageNet networks. We experimentally verify in Table 1 that large ImageNet models perform very differently in RL than SL. We note that default network with the highest test reward was IMPALA-LARGE-BN (IMPALA-LARGE, with Batchnorm) at ≈ 5.5 test score. In order to verify that this is inherently a feature learning problem rather than a combinatorial problem involving objects, such as in , we show that state-of-the-art attention mechanisms for RL such as Relational Memory Core (RMC) using pure attention on raw 32 × 32 pixels does not perform well here, showing that a large portion of generalization and transfer must be based on correct convolutional setups. We provide the training/testing curves for the ImageNet/large convolutional models used. Note the following: 1. RMC32x32 projects the native image from CoinRun from 64 × 64 to 32 × 32, and uses all pixels as components for attention, after adding the coordinate embedding found in . Optimal parameters were (mem slots = 4, head size = 32, num heads = 4, num blocks = 2, gate style = 'memory'). 2. Auxiliary Loss in ShakeShake was not used during training, only the pure network. 3. VGG-A is a similar but slightly smaller version of VGG-16. A key question is how to predict the generalization gap only from the training phase. A particular set of metrics, popular in the SL community are margin distributions , as they deal with the case for softmax categorical outputs which do not explicitly penalize the weight norm of a network, by normalizing the "confidence" margin of the logit outputs. While using margins on state-action pairs (from an on-policy replay buffer) is not technically rigorous, one may be curious to see if they have predictive power, especially as MLP's are relatively simple to norm-bound, and as seen from the LQR experiments, the norm of the policy may be correlated with the generalization performance. For a policy, the the margin distribution will be defined as (x, y) →, where F π (x) y is the logit value (before applying softmax) of output y given input x, and S is the matrix of states in the replay buffer, and R π is a norm-based Lipschitz measure on the policy network logits. In general, R π is a bound on the Lipschitz constant of the network but can also be simply expressions which allow the margin distribution to have high correlation with the generalization gap. Thus, we use measures inspired by recent literature in SL in which we designate Spectral-L1, Distance, and Spectral-Frobenius measures for R π, and we replace the classical supervised learning pair (x, y) = (s, a) with the state-action pairs found on-policy. The expressions for R π (after removing irrelevant constants) are as follows, with their analogous papers: 1. Spectral-L1 measure: ) 2. Distance measure: ) 3. Spectral-Fro measure: We verify in Figure A5, that indeed, simply measuring the raw norms of the policy network is a poor way to predict generalization, as it generally increases even as training begins to plateau. This is inherently because the softmax on the logit output does not penalize arbitrarily high logit values, and hence proper normalization is needed. The margin distribution converges to a fixed distribution even long after training has plateaued. However, unlike SL, the margin distribution is conceptually not fully correlated with RL generalization on the total reward, as a policy overconfident in some state-action pairs does not imply bad testing performance. This correlation is stronger if there are Lipschitz assumptions on state-action transitions, as noted in . For empirical datasets such as CoinRun, a metric-distance between transitioned states is ill-defined however. Nevertheless, the distribution over the on-policy replay buffer at each policy gradient iteration is a rough measure of overall confidence. We note that there are two forms of modifications, network dependent (explicit modifications to the policy -norm regularization, dropout, etc.) and data dependent (modifications only to the data in the replay buffer -action stochasticity, data augmentation, etc.). Ultimately however, we find that current norm measures R π become too dominant in the fraction, leading to the monotonic decreases in the means of the distributions as we increase parametrization. Figure A6: Margin Distributions at the end of training. In the Gym-Retro benchmark using Sonic , the agent is given 47 training levels with rewards corresponding to increases in horizontal location. The policy is trained until 5k reward. At test time, 11 unseen levels are partitioned into starting positions, and the rewards are measured and averaged. We briefly mention that the agent strongly overfits to the scoreboard (i.e. an artifact correlated with progress in the level), which may be interpreted as part of the output of g θ (·). In fact, the agent is still able to train to 5k reward from purely observing the timer as the observation. By blacking out this scoreboard with a black rectangle, we see an increase in test performance. Settings IMPALA NatureCNN Blackout 1250 ± 40 1141 ± 40 NoBlackout 1130 ± 40 1052 ± 40 for the the full solution and notations. Using the same notation (A, B, Q, R), denote C(K) = x0∼D x T 0 P K x 0 as the cost and u t = −Kx t as the policy, where P K satisfies the infinite case for the Lyapunov equation: We may calculate the precise LQR cost by vectorizing (i.e. flattening) both sides' matrices and using the Kroncker product ⊗, which leads to a linear regression problem on P K, which has a precise solution, implementable in TensorFlow: Parameter Generation A Orthogonal initialization, scaled 0.99 Orthogonal Initialization, scaled 0.5 The basis for producing f, g θ outputs is due to using batch matrix multiplication operations, or "BMV", where the same network architecture uses different network weights for each batch dimension, and thus each entry in a batchsize of B will be processed by the same architecture, but with different network weights. This is to simulate the effect of g θi. The numeric ID i of the environment is used as an index to collect a specific set of network weights θ i from a global memory of network weights (e.g. using tensorflow.gather). We did not use nonlinear activations for the BMV architectures, as they did not change the outcome of the . Architecture Setup BMV-Deconv (filtersize = 2, stride = 1, outchannel = 8, padding = "VALID") (filtersize = 4, stride = 2, outchannel = 4, padding = "VALID") (filtersize = 8, stride = 2, outchannel = 4, padding = "VALID") (filtersize = 8, stride = 3, outchannel = 3, padding = "VALID") BMV-Dense f: Dense 30, g: Dense 100 A.3.3 IMAGENET MODELS For the networks used in the supervised learning tasks, we direct the reader to the following repository: https://github.com/tensorflow/models/blob/master/research/ slim/nets/nets_factory.py. We also used the RMC: deepmind/sonnet/blob/ master/sonnet/python/modules/relational_memory.py See for the default parameters used for CoinRun. We only varied nminibatches in order to fit memory onto GPU. We also did not use RNN additions, in order to measure performance only from the feedforward network -the framestacking/temporal aspect is replaced by the option to present the agent velocity in the image. In this section, we use notation consistent with for our base proofs. However, in order to avoid confusion with a high dimensional policy K we described in 3.1, we denote our low dimensional base policy as P and state as s t rather than x t. Let · be the spectral norm of a matrix (i.e. largest singular value). Suppose C(P) was the infinite horizon cost for an (A, B, Q, R)-LQR where action a t = −P · s t, s t is the state at time t, state transition is s t+1 = A · s t + B · a t, and timestep cost is s T t Qs t + a T t Ra t. C(P) for an infinite horizon LQR, while known to be non-convex, still possess the property that when ∇C(P *) = 0, P * is a global minimizer, or the problem statement is rank deficient. To ensure that our cost C(P) always remains finite, we restrict our analysis when P ∈ P, where P = {P : P ≤ α and A − BP ≤ 1} for some constant α, by choosing A, B and the initialization of P appropriately, using the hyperparameters found in A.3.1. We further define the observation modified cost as C(K; W θ) = C K W c W θ T. As described in Lemma 16 of , we define and T P = sup X T P (X) X over all non-zero symmetric matrices X. Lemma 27 of provides a bound on the difference C(P) − C(P) for two different policies P, P when LQR parameters A, B, Q, R are fixed. During the derivation, it states that when P − P ≤ min σmin(Q)µ 4C(P) B (A−BP +1), P, then: C(P) − C(P) ≤ 2 T P (2 P R P − P + R P − P 2)+ 2 T P 2 2 B (A − BP + 1) P − P P 2 R Lemma 17 also states that: where Assuming that in our problem setup, x 0, Q, R, A, B were fixed, this means many of the parameters in the bounds are constant, and thus we conclude: C(P) − C(P) ≤ O C(P) 2 P 2 P − P (A − BP + B + 1) + P P − P 2 Since we assumed A − BP ≤ 1 or else T P (X) is infinite, we thus collect the terms: Since α is a bound on P for P ∈ P, note that P 2 P − P + P P − P 2 = P − P (P 2 + P + P − P) ≤ P − P (P 2 + P ( P + P) ≤ (3α 2) P − P From, this leads to the bound: Note that this directly implies a similar bound in the high dimensional observation case -in particular, We first start with a convex cost 1-step LQR toy example under this regime, which shows that linear components such as β 0 W θ T cannot be removed from the policy by gradient descent dynamics to improve generalization. To shorten notation, let W c ∈ R n×n and W θ ∈ R p×n, where n p. This is equivalent to setting d signal = d state = n and d noise = p, and thus the policy K ∈ R n×(n+p). In the 1-step LQR, we allow s 0 ∼ N (0, I), a 0 = K W c W θ s 0 and s 1 = s 0 + a 0 with cost 1 2 s 1 2, then C(K; W θ) = E s0 1 2 and Define the population cost as C(K):= E W θ [C(K; W θ)]. Let the notation O(p, n) denote the following set of orthogonal matrices: O(p, n) = {W ∈ R p×n : W T W = I}. We use the shorthand O(n) = O(n, n). Proposition 1. Suppose that W θ ∼ Unif(O(p, n)) and W c ∼ Unif(O(n)). Then Here, the expectation is over the randomness of the samples {W i} m i=1 and the initalization K 0. The contribution from E 1 is due to the generalization error of the minimum-norm stationary point of C m (·; {W i}). The contribution from E 2 is due to the full-rank initialization of K 0. We remark that our proof specifically relies on the rank of the Hessian as m increases, rather than a more common concentration inequality used in empirical risk minimization arguments, which leads to a 1 √ m scaling. Furthermore, the above expression for E[C(K ∞)] does not scale increasingly with poly(p) for the convex 1-Step LQR case, while empirically, the non-convex infinite LQR case does indeed increase from increasing the noise dimension p (as shown in Section 3.1). Interestingly, this suggests that there is an extra contribution from the non-convexity of the cost, where the observation-modified gradient dynamics tends to reach worse optima. A.4.3.2 PROOF OF THEOREM 1 Fix integers n, p with p ≥ n and suppose that n divides p. Draw a random W ∈ O(p) uniformly from the Haar measure on O(p) and divide W column-wise into W 1, W 2,..., W p/n (that is W i ∈ R p×n, W

represent this text to find a much much longer article about it with say hundreds-thousands of words

We isolate one factor of RL generalization by analyzing the case when the agent only overfits to the observations. We show that architectural implicit regularizations occur in this regime.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.