Question: Represent the scientific article to find a short TLDR

Question: Model-free deep reinforcement learning approaches have shown superhuman performance in simulated environments (e.g., Atari games, Go, etc). During training, these approaches often implicitly construct a latent space that contains key information for decision making. In this paper, we learn a forward model on this latent space and apply it to model-based planning in miniature Real-time Strategy game with incomplete information (MiniRTS). We first show that the latent space constructed from existing actor-critic models contains relevant information of the game, and design training procedure to learn forward models. We also show that our learned forward model can predict meaningful future state and is usable for latent space Monte-Carlo Tree Search (MCTS), in terms of win rates against rule-based agents. Model-free deep reinforcement learning (DRL) approaches (e.g., deep Q-learning BID14], DDPG BID12 ], A3C BID16 ], etc) have been applied extensively in many simulated environments with complete information and relatively simple game dynamics (e.g., Atari games, Go], Doom, etc). The learned agent, which acts reactively based on the current game situation, can even achieve superhuman performance. However, for complicated environments, planning ahead (or "predicting the future") before making an actual decision is important. Such a planning procedure requires a forward model that estimates the next state s t+1 given the current state s t and action a t, which is in general non-trivial to construct and estimate from the high-dimensional raw input. For partially observable environments (e.g., Real-time Strategy Games like StarCraft), constructing a forward model is more difficult even with a perfect domain knowledge of the game, due to the deliberate concealing of information and the additional requirement to capture the belief of the unknown for the agent. A natural question now arises. Could we borrow the success of model-free approach to learn a forward model? Note that in model-free approaches, a single shared network (called "trunk") is often used to extract features from the input game situation to obtain a latent representation. From the latent space, multiple reinforcement learning quantities (Q-function, value function V, advantage function A, etc) are predicted via simple linear transformations and used for decision making. Strong performance of these approaches indicates that the learned latent space must have captured key ingredients of the input situation and remains low-dimensional. Therefore, it is an excellent candidate for the state representation of a forward model. In this paper, we study whether it is possible to use the latent space learned by model-free approaches to construct forward models. We use MiniRTS ], an efficient and simple twoplayer Real-time Strategy (RTS) game. MiniRTS captures the basic dynamics of its kind: the agent builds units (workers and troops) that consume resources, gathers resources, explores regions out of sights ("fog of war"), defends enemy's attack, and invades enemy's base. This game is incomplete information, because the agent can only see within its sight, and does not know the action of its opponent by default. Rather than unit based control as in;; ], the agent uses 9 discrete actions to control the overall strategy (e.g., build a particular kind of troops, attack or defend).Our contributions are three-fold: First, we propose to study the relationship between the latent space learned by model-free approaches and the state representation of forward models. Very few works (e.g, DARLA BID10], DQN BID15 ]) in model-free RL study these properties in depth, let alone using the latent state in model-based approaches for incomplete information game. To our knowledge, we are one of the first works to explore such directions. Second, we improve the performance of model-based agent in MiniRTS by input feature design and show that the latent space learned from actor-critic models BID16 ] can reconstruct critical information of the game, e.g., Hit Point of the base and available resources. Finally, we propose novel algorithms that learn a forward model that maps a latent state h t to its future counterpart h t (t > t) with reduced drifting. Such a forward model enables us to use model-based planning such as Monte-Carlo Tree Search (MCTS) in incomplete information games. We show positive performance (8% higher than random planning) in terms of win rates against rule-based agents. Forward modeling Model-based approaches are one of the standard ways to model complicated yet physically calibrated robots BID8 ]. In these cases, it is assumed, or known by design, that a forward model can be obtained easily. Furthermore, the state representation of the forward model is usually the internal state of the system (e.g., position and velocity of each robot joint), and is thus low-dimensional. On the other hand, learning a forward model from a complicated and high-dimensional observation is in general difficult, even with the current deep learning architecture. Computer vision researchers try to predict the next video frame given the last few frames BID13; BID26 ]. To reduce the complexity, one natural idea is to project the high-dimensional input to low-dimensional state that captures the important aspect of the situation, on which a forward model might be easier to learn. BID0 ] learns a forward model directly from visual input and uses that for manipulation. To make their learned latent state nontrivial, a regularization term is applied to ensure the learned representation can reconstruct the input as much as possible. In comparison, we do not need a regularization since the latent space is pre-trained by model-free approaches. One issue in forward modeling is that its accumulative prediction over many steps might drift and is not usable for planning. BID29 ] uses forward model prediction as another feature for a model-free approach, while BID2 uses forward model as a way to guide exploration for model-free procedure. In this paper, we use multi-step long-term prediction to stabilize the forward model. Real-time strategy games. Most works on Real-time Strategy (RTS) games assume that the agents have access to complete information, in which the behaviors of both the player and the opponents are observed. Micro-management is often viewed as a standard RL task, addressed by model-free approaches; BID17 ]. On the other hand, Monte-Carlo Tree Search (MCTS) has been applied (e.g, ABCD BID4], MCTSCD BID21 ]), which focus on concurrent planning with perfect information and perfect forward model. BID24 ] learns a short-term shallow forward model for local combat, but again assuming full access to the game. Finally, BID5 ] deals with partial observability, but focuses particularly on scouting, rather than global strategies as we do. Grouping unit-based actions into macro (or strategic) actions is an open and challenging topic in RTS games. Following ], we use 9 pre-defined discrete strategic actions to facilitate training. Other recent works also model macro strategies from replays BID11 ] or from action scripts BID3 ], both with deep models. In model-free reinforcement learning, an agent learns to maximize its long-term reward, either from previous experience in the same environment (e.g. replay buffer), or from its own interaction with the environment with its behavior. In both cases, the agent does not need to have knowledge of the environment, nor know how the world changes after an action is taken, hence called model-free approach. On the other hand, if an agent learns to build a model to predict how the state (its own state and/or environment state) changes after its action, then the agent is considered to take a model-based approach. In this paper, we first use an off-policy version of Batch A3C; BID1 ] to train a model-free agent. During training, the parameters of the agent follows the gradient directions of the two objectives: DISPLAYFORM0 where π(a|s; θ π) and V (s; θ V) are the policy and value functions of the agent. DISPLAYFORM1 is an importance factor that captures the discrepancy between the policy distribution π(a t |s t ; θ old π) evaluated when we sample the action a t at state s t, and the policy we use now with the current parameter θ π. We apply the importance factor η t since Batch A3C does not start updating model parameters until it collects a full batch. During that period, the data that are first collected is considered to be off-policy. In many works that use shared feature extraction and multi-head predictions BID28;; ], the two set of the parameters θ V and θ π are mostly shared with a common trunk h with parameter θ U, and the policy/value function can be written as: DISPLAYFORM2 h compactly encodes the information used for both decision (policy π) and evaluation (value V) of the model. Compared to the original high-dimensional input that might contain a lot of irrelevant information, h typically contains a rich but low-dimensional representation of the current situation that is critical for its assigned task. That might include world states directly related to short-term/longterm reward, and belief of the unobserved part of the environment. From a model-based point of view, such a latent space learned from a successful model-free training is a good candidate for the state representation of a forward model. Following the reasoning, in order to build a strong forward model, a natural approach is to start with a strong pre-trained model-free agent that performs well in the environment. In this paper, we use MiniRTS ] as the environment, which is a fast incomplete information Real-time Strategy Game with two players. An agent which performs well in MiniRTS must build its own troops, gather resources efficiently, explore unseen territory, defend the attack from the enemy, and find the right timing to attack back. The game ends when one player's base is destroyed by the opponent. MiniRTS runs at 40K FPS on a laptop with quite complicated dynamics (e.g., collision detection, unit path planning, etc) and is suitable for our task. To strengthen the model-free agent, we improve upon ] by appending more input feature channels. We define three models Vanilla, BuildHistory and PrevSeen, each with a different set of features (Tbl. 1).• Vanilla Similar to ], we use basic features that include Unit Type, Hit Point ratio, and available resources to build units.• BuildHistory In an incomplete information game, the decision a player made two minutes ago is not the same as the decision the player is making now, even if the perceived situation is identical. This is because during the period, the opponent (which the player cannot see) might have done a lot of work. To make the agent aware of the situation, we record the build since tick of each unit, and attach related information in the feature plane.• PrevSeen Ideally the agent should remember all information since the game starts (e.g., the opponent's base location it saw at tick 100, even if the current tick is 5000). However, as shown in our experiment, Recurrent Network (RNN) is not able to carry such a longterm memory. Therefore, we modify the game engine so that the most recent information that the agent used to have access to, is also sent into the input. much faster convergence, and encourages fast map exploration and changes in strategies. As a , the bot learns more aggressive strategies like rush by itself. In Sec. 4.1, we show that PrevSeen has the strongest performance and its latent representation is a good candidate for forward models. To check whether the learned latent space is useful for forward modeling and understand what kind of game statistics it has captured, we further learn an interpretation network to predict the input from the hidden state. The interpretation network is designed to mirror the network that maps input to latent space, only replacing pooling layers with upsampling layers. Again, the latent space of PrevSeen can predict highly relevant input features, as shown in Sec. 4.2. Once we have a rich latent space that contain important information for decision making, we thus train a forward model that carries h t to the near future, which can be used for prediction and modelbased planning. One naive approach is to learn a forward model f to predict the next state h t+1 given h t and a t. However, a simple prediction might suffer from drifting, in which if we apply f multiple times to predict far future, then the predict error will quickly accumulate and makes the long-term prediction unusable. This leads to detrimental performance for planning methods (e.g. MCTS) which require accurate long-term predictions. To solve this issue, during training, we not only require f (h t, a t) to be close to h t+1, but also require f (t) (h t, a t, a t+1, . . ., a t+t −1) to be close to h t+t, where f (n) is to apply the same forward model n times. While a naive implementation requires O(T 3) gradient update, such a training procedure can be implemented efficiently with the current autograd approach with only O(T 2) gradient update. The trained forward model enables us to apply existing planning method to complicated incomplete information games and give a sensible performance boost over baselines. We first learn a model-free agent against rule-based AI (AI SIMPLE in MiniRTS). To maximize the performance of trained AI, we use T = 20 for all the experiments, as mentioned by BID9; BID7 ] that multi-step training with large T is useful. We also set the frame skip (how frequent the action is made) as 50 for AI, and as 20 for AI SIMPLE. This gives the trained agent a slight disadvantage. For network structure, we use a convolutional networks with 2 conv + 1 pooling + 2 conv + 1 pooling, each convolutional layer has 64 channels, except for the first and the last layer. The input features is 20-by-20 images with 35 channels (Tbl. 1). For all approaches, the overall number of parameters is kept constant as in BID29 ]. The ing latent representation is 35 × 5 × 5 = 875 dimensional, which is much lower than the original input space. After convolution, the latent representation is then used to predict discrete action distribution (policy π) and value function V. For RNN, the next latent state h t+1 is computed by first concatenating h t with a learnable embedding of a t, and then compressing them into a vector in the latent state by MLP. We also use frame-stacking (of size 4), which is an empirical trick to mimic short-term memory during training. Both RNN and frame-stacking use the feature sets of Vanilla. As in ], we use 9 discrete actions to drive the environments. All these actions are globally strategic actions (e.g., attack, defend, which unit to build) and agents do not need to worry about their details (e.g., where to build the unit).History Information. Adding historic data can help with the performance of RL approach. However, similar to previous works ], the performance of frame-stacking is already comparable to RNN approach, showing that the latent space really captures the short-term memory. On the other hand, PrevSeen outperforms both RNN and BuildHistory. In summary, while the short-term information can be encoded into the hidden space (either by RNN or frame-stacking), long-term information cannot be efficiently learned in the latent space and has to be encoded manually (as in PrevSeen).Complete/Incomplete information. This leads to difference in training curves FIG1 and final performance (Tbl. 2) whether the agent can see the opponent's behavior and whether the agent has immediate access to the information that encodes what it saw long-time before (e.g., sense the base of the opponent long before). Value function. In incomplete information setting, we see many sudden drops in the value function when game progresses. This is because the agent does not estimate well about the opponent's actions and thus thinks it is in good shape. To quantify it, we define the surprise metric as the difference between nearby estimated values: Fig. 3(b) shows that AI trained with complete information indeed knows more about the situations than those trained with incomplete information, in particular at the beginning of the game. As shown in Fig. 3(c), the distribution of surprise is different between complete and incomplete information bot. DISPLAYFORM0 However, even for the complete information bot, there are quite a few sudden drops. This is due to the fact of complicated dynamics of the game. Lots of small factors, e.g., path planning, collision detection, could lead to substantial different consequences of the game. For example, the consequence of an even battle is often determined by the micro-management of each player, for which our model does not have control. Different learned strategy. Although the final performance of PrevSeen and BuildHistory are similar, their behavior against rule-based AI is quite different. As shown in Fig. 4, AI trained from PrevSeen learned to explore the map and/or rush the enemy with a few tanks at the beginning of the game. Quantitatively, the number of nonzero seen-then-hidden events per game from PrevSeen is twice than that from BuildHistory (Fig. 5). We haven't seen similar behaviors in prior works. With the latent state, we can reconstruct the original 20x20 channels. We choose to predict one channel at a time. A joint model that predicts all input channels at once works poorly since dense channels (with many non-zero entries) dominate the prediction. We define the normalized reconstruction accuracy (NRA) to be 1 − c − c * / c *, where c is some channel and c * is its ground truth value. Fig. 6 shows the . The model correctly finds the most relevant features from the input (e.g., base HP ratio, the amount of available resource). We discover that the agent can partially recover the exact location of the workers during early stage of training, but ignores them at convergence, showing that the model learns to discard less relevant features. Interestingly, the prev-seen feature channels are not predicted well for the full-fledged agent, showing they are not important in the decision. However, models equipped with these channels train much faster. We also try predicting the 5x5 down-sampled version of the original 20x20 channels, by removing all upsampling layers. Overall the normalized reconstruction accuracy is much higher, as shown in the dark blue bars. This is because the latent state is also a 5 × 5 down-sampled state of the original game state, and the model only needs to reconstruct the features in a rough region instead of in each grid. In particular, we see the emergence of multiple relevant features such as the location of own RANGE ATTACKER and BASE, affiliation and unit HP ratio, etc. We also see that the agent learns to pay attention to the opponent's WORKER (which gathers resources) and MELEE ATTACKER (which attacks us). The model pays attention to the opponent's MELEE ATTACKER but not so much to RANGE ATTACKER because the rule based AI mostly builds MELEE ATTACKER. We train multiple forward models following Sec. 3.2, each with different training paradigm defined as follows. Pred1: basic forward model (FIG0); MatchPi: enforce the predicted future state to predict the future policy well FIG0; MatchA: enforce the predicted future state to predict the future action well FIG0; PredN: predict long-term future states (FIG0). Win rate from the decisions made by latent state h that is predicted by forward models. Top row: 2-hop prediction (ĥ t = f (h t−2)) on 5 models trained on 5 independent A3C models. Bottom row: 5-hop prediction. In both cases, the grey dotted line is the baselineĥ t = h t−2 (and h t = h t−5). MatchA(red) lines do not continue due to numerical instability. For evaluation, at a time step t, we recursively apply the learned forward model to estimate the current latent stateĥ t from previous state h t−t that is computed from observation s t−t. Then we useĥ t to make decision for time t, and check its win rate against the rule-based system. As a baseline, we use identity forward functionĥ t = h t−t.We evaluate these training paradigms in 5 independently trained models using model-free approaches. As shown in FIG4, MatchPi is the most stable method with highest win rate, consistently higher than the baseline approach that usesĥ t = h t−t. MatchA also learns quite fast, but runs into numerical instabilities after a few hundreds of iterations. PredN is also quite stable but its learning speed is not as fast compared to MatchA. When the delay is large, we clearly see the decay of the win rate performance. The performance metric used in Sec. 4.3 is only an intermediate metric. To finally test the performance of forward models, we plug them into a planning algorithm, e.g., Monte-Carlo Tree Search (MCTS). Note that traditional MCTS can only be applied when both the complete game state and the dynamics of the game environments are known. However, neither conditions are satisfied for RTS games, thus learned forward models and value function are necessary. In MCTS, We use shared tree (tree parallelization) among multiple threads. This is because compared to other approaches (e.g., root parallelization Chaslot et al. FORMULA0), expanding a tree node, as a costly operation that requires feed-forwarding a neural network, only happens once. To increase diversity of initial exploration, we use virtual loss of size 5. During the execution of MCTS, the network performs a projection step that maps the input state to its latent representation, a forwarding step that predicts the latent state h t+1 given h t and a t, and a prediction step that gives the value of the predicted state for back-propagation in MCTS.Interestingly, the models with strong performance in delayed prediction task (e.g., MatchPi) does not perform well in MCTS. On the contrary, PredN gives a consistently strong performance (25%) over 5 independently trained models, compared to the random baseline (17% win rate). Note that without a good forward model and a value function, we cannot even run reduced space MCTS. The proposed forward model makes it possible. However, compared to the performance of model-free MatchA MatchPi Pred1Figure 8: Win rate against AI SIMPLE with frame skip 20 on 1000 games using latent space MonteCarlo Tree Search. All uses 100 rollouts (5 threads, each thread with 20 rollouts). The pink line is the baseline of a random agent that picks the 9 actions uniformly at each step.approach (> 80%), there is still a long way to go. We also tried removing short-term predictions (FIG0), the performance is similar. We have also tried combining forward model with action probability given by model-free approach using PUCT []. However, there is no gain due to the fact that the model-free part has a much stronger performance, overshadowing the contribution of forward models. Latent space learned by model-free reinforcement learning encodes important information for an agent to make sensible decisions to maximize the reward in a complicated simulated environment. In this paper, we verify the power of latent space of successfully trained model-free agent, and propose several methods to learn forward models on this space, in a real-time strategy game with incomplete information. Despite an extremely hard problem, we learn forward models that make it possible to use planning approaches such as Monte Carlo Tree Search, and show consistently positive gains over baselines. A lot of future works follow. As a first step, although we show that it is possible to learn a forward model for incomplete information Real-time Strategy games to enable model-based planning in the latent space, it remains an open problem how to improve its performance. It is possible that despite a good forward model is learned, the value function is not good enough, e.g., putting too much focus on the on-policy trajectory, for Monte-Carlo Tree Search. Also, in this paper we use predefined 9 global actions for the game. How to automatically learn global actions from unit-based commands that are exponentially large, is still an challenging issue to solve.

The paper analyzes the latent space learned by model-free approaches in a miniature incomplete information game, trains a forward model in the latent space and apply it to Monte-Carlo Tree Search, yielding positive performance.

Represent the short summary for finding the corresponding scientific article (long):

Adaptation of an RL agent in a target environment with unknown dynamics is fast and safe when we transfer prior experience in a variety of environments and then select risk-averse actions during adaptation.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Several state of the art convolutional networks rely on inter-connecting different layers to ease the flow of information and gradient between their input and output layers. These techniques have enabled practitioners to successfully train deep convolutional networks with hundreds of layers. Particularly, a novel way of interconnecting layers was introduced as the Dense Convolutional Network (DenseNet) and has achieved state of the art performance on relevant image recognition tasks. Despite their notable empirical success, their theoretical understanding is still limited. In this work, we address this problem by analyzing the effect of layer interconnection on the overall expressive power of a convolutional network. In particular, the connections used in DenseNet are compared with other types of inter-layer connectivity. We carry out a tensor analysis on the expressive power inter-connections on convolutional arithmetic circuits (ConvACs) and relate our to standard convolutional networks. The analysis leads to performance bounds and practical guidelines for design of ConvACs. The generalization of these are discussed for other kinds of convolutional networks via generalized tensor decompositions. Recently, densely connected networks such as FractalNet BID8, ResNet BID6, and DenseNet BID7, have obtained state of the art performance on large problems where highly deep network configurations are used. Adding dense connections between different layers of a network virtually shortens its depth, thus allowing a better flow of information and gradient through the network. This makes possible the training of highly deep models. Models with these types of connections have been successfully trained with hundreds of layers. More specifically, DenseNets have achieved state of the art performance on the CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets, using models of up to 1 thousand layers in depth. Nevertheless, whether these connections provide a fundamental enhancement on the expressive power of a network, or just improve the training of the model, is still an open question. In BID7, DenseNet models with 3 times less parameters than its counterpart (ResNets) were able to achieve the same performance on the ImageNet challenge. Moreover, a theoretical understanding of why the connections used by DenseNets lead to better performance compared with FractalNets or ResNets is still pending. Despite the popularity of these models, there are few theoretical frameworks explaining the power of these models and providing insights to their performance. In, the authors considered convolutional networks with linear activations and product pooling layers, called convolutional arithmetic circuits (ConvACs), and argued for the expressiveness of deep networks using a tensor based analysis. This analysis has been extended to rectifier based convolutional networks via generalization of the tensor product. In, it was shown that ConvACs enjoy a greater expressive power than rectifier based models despite the popularity of rectifier based networks in practice. Indeed the empirical relevance of ConvAC was demonstrated through an architecture called SimNets. In addition, the generative ConvAC of BID11 achieved state of the art performance in classification of images with missing pixels. These served as motivation for the works of;; BID9; BID10, where different aspects of ConvACs were studied from a theoretical perspective. In the inductive bias introduced by pooling geometries was studied. Later, BID9 makes use of the quantum entanglement measure to analyze the inductive bias introduced by the correlations among the channels of ConvACs. Moreover, BID10 generalizes the convolutional layer of ConvACs by allowing overlapping receptive fields, in other words permitting stride values lower than the convolution patch size. These locally overlapping connections led to an enhancement on the expressive capacity of ConvACs. The notion of inter-layer connectivity for ConvACs was addressed by in the context of sequential data processing, such as audio and text related tasks. In that work, the expressive capabilities of interconnecting processing blocks from a sequence was studied. Nevertheless, these types of interconnections are related to the sequential nature of the problem and different from the ones used in ResNet, FractalNet and DenseNet. In this work, we extend the tensor analysis framework of to obtain insightful knowledge about the effect of dense connections, from the kind used in DenseNets, FractalNet and ResNet, on the expressiveness of deep ConvACs. We study the expressive capabilities provided by different types of dense connections. Moreover, from these we derive performance bounds and practical guidelines for selection of the hyperparameters of a deep ConvAC, such as layer widths and the topology of dense connections. These serve as the first step into understanding dense connectivity in rectifier networks as well, since they can be further extended to include rectifier linear units, in the same spirit as the generalization of the tensor products done by.The remainder of this paper is organized as follows. In Section 2, we introduce the notation and basic concepts from tensor algebra. In Section 3, we present the tensor representation of ConvACs as introduced by, and later in Section 4, we obtain tensor representations for densely connected ConvACs. In Section 5, performance bounds and design guidelines are derived for densely connected ConvACs. The term tensor refers to a multi-dimensional array, where the order of the tensor corresponds to the number of indexes required to access one of its entries. For instance, a vector is a tensor of order 1 while a matrix is a tensor of order 2. In general a tensor A of order N requires N indexes (d 1, . . ., d N) to access one of its elements. For the sake of notation, given I ∈ N, we use the expression [I] to denote the set {1, 2, . . ., I}. In addition, the (d 1, . . ., d N)-th entry of a given tensor of order N and size DISPLAYFORM0. Moreover, for the particular case of tensors of order N with symmetric sizes DISPLAYFORM1 ⊗N as shorthand notation for R M ×···×M. A crucial operator in tensor analysis is the tensor product ⊗, since it is necessary for defining the rank of a tensor. For two tensors B ∈ R M1×···×Mp and C ∈ R Mp+1×···×Mp+q, the tensor product is defined such that B ⊗ C ∈ R M1×···×Mp+q and (B ⊗ C) d1,...,dp+q = B d1,...,dp C dp+1,...,dp+q for all (d 1, . . ., d p+q). In tensor algebra, a tensor A ∈ R M1×M2×···×M N is said to have rank 1 if it can be expressed as DISPLAYFORM2 Moreover, any tensor A ∈ R M1×M2×···×M N can be expressed as a sum of rank-1 tensors, that is DISPLAYFORM3 where Z ∈ N is sufficiently large and v DISPLAYFORM4. Note that this statement is DISPLAYFORM5 On the other hand, when Z is the minimum number such that is satisfied, the rank of the tensor is defined to be rank(A) = Z and becomes equivalent to the well known CANDECOMP/PARAFAC (CP) decomposition of A. Another operator, that is on the core of the former works of;; BID9, is the matricization operator. The operator [A] denotes the matricization of a tensor A ∈ R M1×···×M N of order N. This matricization of the tensor A re-orders its elements into a matrix A ConvAC is a convolutional neural network that utilizes linear activation functions with product pooling, unlike most popular convolutional networks which make use of rectifier activations with max or average pooling. Moreover, the input of the network is modeled by X = (x 1, . . ., x N) ∈ (R s) N, where x i ∈ R s denotes the vectorization of the i-th patch of the input image. For this analysis, it is assumed that a set of M features is obtained from every patch, that is DISPLAYFORM6. These features are selected from a given parametric family F = {f θ : R s → R : θ ∈ Θ}, such as Gaussian kernels, wavelet functions, or learned features. Then, to determine whether an input X belongs to a class belonging to the set Y, the network evaluates the some score functions h y (X) ∈ R and decides for the class y ∈ Y such that DISPLAYFORM0 Using this formulation, in FIG0 (a) we observe an example of a single hidden layer ConvAC, while in FIG0 we observe the general case of a deep arithmetic circuit of L layers. As shown by, any score function of a ConvAC can be expressed as an homogeneous polynomial with degree N on the input features of the form DISPLAYFORM1 where A y d1,...,d N ∈ R are the polynomial coefficients stored in the grid-tensor DISPLAYFORM2, degree N, and M N polynomial coefficients stored in the grid-tensor A y.For the special case of a shallow ConvAC with 1 × 1 convolutions and Z hidden units 1, shown in FIG0 (a), the score functions are computed from the weight vectors a DISPLAYFORM3. This leads to the score function DISPLAYFORM4 The first step of the tensor analysis framework is to obtain an expression (in terms of the network parameters a y z and a z,i d) of the grid-tensor A y that represents this concrete network architecture. In other words, obtaining the expression for A y that transforms into. This expression was already obtained in as DISPLAYFORM5 where ⊗ denotes the tensor product. Note that FORMULA14 is in the form of a standard CP decomposition of the grid tensor A y. This implies that the rank of A y is bounded by rank(A y) ≤ Z. Moreover, the obtained where generalized in for the case of a deep ConvAC with size-2 pooling windows 2, thus L = log 2 N hidden layers as shown in FIG0 (b), leading to a grid-tensor given by the hierarchical tensor decomposition DISPLAYFORM6 where r 0,..., r L−1 ∈ N are the number of channels in the hidden layers, {a γ∈[r0] are the weights in the first hidden convolutions, DISPLAYFORM7 DISPLAYFORM8 are the weights of the hidden layers, and a L,1,y ∈ R r L−1 stores the weights corresponding to the output y in the output layer. The recent empirical success of densely connected networks (DenseNets), presented by BID7, has served as motivation for our theoretical analysis on dense connectivity. Dense connectivity in a convolutional neural network refers to the case when a number k ∈ N (known as growth rate) of previous layers serve as input of the forthcoming layer. More precisely, in BID7, a DenseNet performs this via concatenation along the feature dimension of the current layer inputs with the preceding layer features. Note that these feature must have compatible sizes along the spatial dimension for the concatenation to be possible. To address this issue, BID7 proposed to group blocks of the same spatial dimensions into a dense block, as shown in FIG1. These dense blocks do not contain operations such as pooling, that alter the spatial dimensions of the input features. Moreover, in the DenseNet architecture the layers that perform the pooling operation are called transition layers, since they serve as transition between dense blocks. For example, in FIG1 we depict a dense block of 4 layers with growth rate k = 2, followed by a transition layer. 1 We must mention that the generalization to w × w convolutions is straightforward and was already covered by.2 Note that the generalization to different pooling sizes is straight forward and was done by. In the original DenseNet these transition layers included one convolution layer before the pooling operation. Nevertheless, for this work we consider transition layers composed of only pooling operations. Note that this does not affect the generality of the model, since avoiding dense connections on the convolutional layer preceding the transition layer is equivalent to including a convolution in that transition layer 3.In the case of ConvACs, any dense block of size greater than 1 can be represented as a dense block of size 1, since the activation function is the linear function (the non-linearity comes from the product pooling operator in the transition layer). Therefore, for ConvACs, it is only reasonable to analyze dense blocks of size 1. Note that, if we only allow dense connections between hidden layers within a dense block, a ConvAC is limited to a maximum growth rate of k = 1. In order to analyze the effect of broader connectivity we extend the concept of growth rate by allowing dense connections between dense blocks. With proper pooling, outputs of hidden layers belonging to different dense blocks can also be concatenated along the feature dimension. In the reminder of this paper we refer to the dense connections between hidden layers of the same block as intra-block connections, while the connections between hidden layers of different blocks as inter-block connections. In this section we analyze the effect of intra-block connections. We first start by constructing a densely connected version of a single hidden layer ConvAC. The ing network with growth rate k = 1 is shown in FIG2 (a). In the same manner as in, this architecture leads to the score function DISPLAYFORM0 Then, we present the following proposition regarding shallow ConvACs with dense connections of growth rate k = 1. The network's function of a densely connected shallow ConvAC shown in corresponds to the grid tensor DISPLAYFORM0 where DISPLAYFORM1 Proof See appendix B.1.Note that the rank of this tensor is now bounded by rank(A y) ≤ Z + M instead of Z, but adding these dense connections increases the number of parameters of the network from M N Z + Z to We now generalize the obtained for the case of a L-layered dense arithmetic circuit, with growth rare k = 1, as the one in FIG2 (b). Similarly to, the obtained grid tensor has the hierarchical decomposition given by DISPLAYFORM2 DISPLAYFORM3... DISPLAYFORM4 From this we observe that inter block connections account for virtually increasing the width of the network's hidden layers from r l tor l r l + r l−1 for all l = 0, 1,..., L − 1, where r −1 M. Note that this increased width comes at the expense of increasing the network's parameters. Moreover, in Section 5 we discuss whether increasing the network's width via intra block dense connections leads to an enhancement in its overall expressive power. In this section we study broader connectivity via dense inter-block connections. As discussed in Section 4, proper pooling of the preceding features must take place before the concatenating them into the current layer. Since this type of connections have not been considered in the former DenseNets, we propose 3 possible ways of realizing such connections (via product, average, or max pooling). For a ConvAC with pooling window size w pool, an inter block connection that connects block DISPLAYFORM0 An example of an inter block connection of jump length L jump = 1 can be seen in FIG2 (c). To perform this inter block connections, the sizes along the spatial dimensions of preceding features must be reduced by L jump w pool, before concatenating them along the feature dimension of layer l. This spatial size reduction may be realized via pooling of the preceding features with window size L jump w pool. When using a pooling layer the size along the feature dimension remains unchanged. Moreover, the type of pooling employed (product, average, or maximum) affects the expressive potential of the ing ConvAC. Furthermore, the following proposition addresses the effect that adding dense inter block connections, via average pooling, has on the network function of a ConvAC.Proposition 2 Adding inter block connections via average pooling of jump length L jump ≥ 1 to a standard ConvAC with grid-tensor A y ∈ (R M) ⊗N leads to a network function of the form DISPLAYFORM1 where g(X) contains polynomial terms on DISPLAYFORM2 Remark 1 This is also valid when the connections are done by addition instead of concatenation, as it is done in ResNet and FractalNet. Proof See appendix B.2.From this proposition we conclude that adding inter block connections average pooling does not alter the grid tensor A y, instead these connections account for extra polynomial terms of degree strictly less than N. Note that, for the special case where the input features belong to an exponential kernel family, such as F = {f θ (x) = e θ T x: R s → R: θ ∈ Θ} or F = {f θ (x) = e θ−x p: R s → R: θ ∈ Θ} where · p denotes the p norm with p ∈ N, the number of polynomial terms is equivalent to the number of exponential basis that the network function can realize. Therefore, the another valid measure of expressiveness is the number of polynomial terms a ConvAC is able to realize. Given a certain ConvAC topology, the number of polynomial terms can be computed inductively by expanding the polynomial products of every layer via generalized binomial expansions. Such an analysis is left for future contributions. Moreover, if we perform this connections via product poling, the features to be concatenated correspond to polynomial terms of the same order. This leads to a generalization of the intra-block connections from 4.1, leading to virtually increased widths r l r l + Ljump q=1 r l−1−q. Finally, we leave the analysis of inter-block connections via maximum pooling for future work and consider only product pooling inter-block connections in the remainder of this paper. For the sake of comparison, let us assume networks with hidden layer widths r l decaying (or increasing) at an exponential rate of λ ∈ R. Formally, this is r l = λr l−1 ∈ N, thus r l = (λ) l r for all l = 0, 1,..., L − 1, where r r 0. To shorten the notation, we denote as (L, r, λ, k) to a ConvAC with of exponential width decay λ ∈ R, length L ∈ N, initial with r ∈ N and growth-rate k ∈ N. A growth-rate of k = 0 refers to a standard ConvAC with no dense connections. Definition 1 Suppose that the weights of a (L, r, λ, k) ConvAC, with L, k ∈ N and r, λ ∈ R, are randomly drawn according to some continuous non-vanishing distribution. Then, this (L, r, λ, k) ConvAC is said to have weak dense gain G w ∈ R if, with probability p > 0, we obtain score functions that cannot be realized by a (L, r, λ, 0) ConvAC with r < G w r. When p = 1, this (L, r, λ, k) ConvAC is said to have a strong dense gain G s = G w ∈ R.Using this definition we present a bound for the weak dense gain G w in the following theorem. DISPLAYFORM0 Proof See appendix B.3.This general bound may serve as guideline for tayloring M and the widths r 0,..., r L−1 such that we exploit the expressiveness added by dense connections. Proof See appendix B.3.Using this , we able able to quantify the expressive gain provided by dense inter block connections. If a ConvAC has a dense gain G w = (1 + 1 λ) that is already close to the general bound from Theorem 5.1 it is less encouraging to include broader dense connections, since it would increase the number of parameters of the model while there is no room for a significant expressive gain increase. In this scenario, connections as the ones in ResNet and FractalNet may more beneficial since they do not increase the size of the model, while at the same time enhancing its trainability. This last theorem shows that there exist a regime where this bounds can be achieved with strong dense gain. Whether this is true outside this regime is still an open question, since further knowledge about the rank of random tensors is limited. Moreover, these theorems does not consider the additional amount of parameters added by dense connections. We complete our analysis by addressing this issue in the following proposition. Proposition 3 Let ∆P dense ∈ N be the additional number of parameters that are added to a (L, r, λ, 0) ConvAC when we introduce dense connections of growth-rate k > 0. In the same manner, let ∆P stand ∈ N be the number of parameters that are added to a (L, r, λ, 0) ConvAC when we increase its initial width r by a factor G ∈ R. Then the ratio between ∆P dense and ∆P stand is greater than DISPLAYFORM1 Proof See appendix B.4.The factor G from this proposition directly relates to the dense gain of a ConvAC, thus this ratio may be used to decide whether is interesting to add dense connections to a model (we want this ratio to be as large as possible). Finally Theorems 5.1 and 5.2 directly bound this ratio, which give the practitioner a guideline to decide which connections (if any) should be added to a given model. Lemma 1 Given Z ∈ N, let A ∈ (R M) ⊗ P be a random tensor of even order P ≥ 2 such that DISPLAYFORM0 where a (k) z ∈ R M are randomly drawn from a non-vanishing continuous distribution for all k ∈ [P] and z ∈ [Z]. Then, if Z ≤ M P/2 we have that rank(A) = rank([A]) = Z with probability 1. This lemma also holds when for a subset Z ⊆ [Z] we have that a (k) z = a z e z ∈ R M for all z ∈ Z, where a z ∈ R are randomly drawn from a non-vanishing continuous distribution. Proof Using the definition of the matricization operator, we get that the matricization A is DISPLAYFORM1. FORMULA3 Note that, from this expression, it is straight forward to see that the rank of [A] is always less or equal than Z. DISPLAYFORM2 be a permuted version of [A] such that the first Z rows of U correspond to the rowsz ∈Z of [A], and the first Z columns of U correspond to the columnsz ∈Z of [A]. Since permuting the rows and the columns of a matrix does not alter its rank, we have that U has the same rank as [A]. Now, let us partition U into blocks as DISPLAYFORM3 where P is of size Z-by-Z, and Q, W, Z have matching dimensions. Note that, if rank(P) = Z then rank(U) ≥ Z, which leads to DISPLAYFORM4 Therefore, it is sufficient to show that rank(P) = Z with probability 1 to conclude this proof. To that end, let us define the mapping from x ∈ R M P Z to P = P(x) as DISPLAYFORM5 Note that this definition of x implies that a DISPLAYFORM6 is computed as in FORMULA3, we have that [A] = [A](x), thus Q = Q(x) and P = P(x). Now, det P(x) is a polynomial on x, then it either vanishes in a set of measure zero or its the zero-polynomial (see BID0).If we set x to be equal to some x 0 ∈ R M P Z such that a DISPLAYFORM7 is now a matrix with 1 on the entry (z,z) and zero elsewhere, we have that [A](x 0) is a diagonal matrix with ones on the diagonal elementsz ∈Z and zero elsewhere. This leads to P(x 0) = I Z which has a determinant det P(x 0) = 0. Finally, since there exist x 0 such that the polynomial det P(x 0) is not zero, we conclude that det P(x) is not the zero-polynomial, which means that det P(x) = 0 with probability 1, thus proving this lemma. DISPLAYFORM8 ⊗ P be random tensors of even order P ≥ 2 and Z ∈ N be tensors such that DISPLAYFORM9 z ∈ R M are randomly drawn from a non-vanishing continuous distribution. Then, if Z 1 ≤ M P/2 and Z 2 ≤ M P/2, we have that rank (A ⊗ B) = Z 1 Z 2 with probability 1.Proof Let C ∈ (R M) ⊗2P be a random tensor defined as C = A ⊗ B. Therefore, we may express C as DISPLAYFORM10 Then, we define rank-1 tensors C (q,z) to be C (q,z) = a DISPLAYFORM11 Since C is now expressed as a sum of Z 1 Z 2 rank-1 tensors, we have that rank(C) ≤ Z 1 Z 2.Since Z 1 ≤ M P/2 and Z 2 ≤ M P/2 we may use Lemma 1, this leads to rank([A]) = Z 1 and rank([B]) = Z 2 with probability 1. Finally we, use the properties of the Kronecker product to obtain the rank of the matricization C as rank(DISPLAYFORM12 with probability 1, thus proving the Lemma. DISPLAYFORM13 ⊗ P be tensors of order P > 2 and Z ∈ N be tensors such that Proof We reformulate this to have the same form as. To that end we define a DISPLAYFORM0 which has the same form as. Therefore, as done in FORMULA14, we obtain the grid tensor for this architecture as DISPLAYFORM1, thus proving this proposition. Proof DISPLAYFORM0 T ∈ R M N, the output of the l-th layer of a (L, r, λ, 0) ConvAC can be stored into the vectors of mappings DISPLAYFORM1. Moreover, since the entries of these vectors are the of l − 1 convolution-pooling layers with product pooling of window size 2, all the mappings δ l,j 1 (x) can be expressed as a sum of polynomial terms on x of degree 2 l. Now, let the coefficient vectors a DISPLAYFORM2 ], be the weight vectors for the convolution of the l-th layer. To shorten the notation we use a l,j,γ, DISPLAYFORM3 as shorthand for the convolution between these vectors. Then, the outputs the the layer l of this ConvAC are given by δ l+1,j ∈ R r l+1 with δ DISPLAYFORM4 If we recursively calculate these out vectors up to the L-th layer we obtain the score functions h DISPLAYFORM5 1 (x) ∈ R. We now consider the effecct of adding dense connections via average pooling from some k ∈ N preceding layers l − 1,..., l − k. To this end, letr l = k q=1 r l−q be the total size along the feature dimesnion of the vectors to be concatenated. In addition, let DISPLAYFORM6 Rr l be the vectors of mappings of the corresponding preceeding features at the layer l for j ∈ [N/r l]. In order to compute the convolutions of this layer, an additional vector of coefficients is required as b DISPLAYFORM7 Then, the outputs of the l-th layer of this (L, r, λ, k) ConvAC are the denoted as the vectorsδ DISPLAYFORM8 where DISPLAYFORM9 Note that the entries of ω l,j (x) are assumed to come from preceding layers with an appropiate average pooling. Since performing avergae pooling does not increase the degree of the polynomial terms involved (only product pooling does) and the jump length Ljump is at least 1, the entries of ω l,j (x) have at most polynomial degree 2 l−1, which is strictly less than the degree of the entries of δ l,j (x) (i.e., 2 l). Therefore, from the obtained expression of ω l+1,j we observe that it has polynomials withb degree no greater than 2 l + 2 l−1, while the entries of δ l+1,j have a strictly higher degree of DISPLAYFORM10 Moreover, since a l,j,γ, δ l,j + ω l,j can be expressed as DISPLAYFORM11 we can make use of the obtained in an unductive manner up to the L-th layer, thus leading to DISPLAYFORM12, where g(x) contains polynomial terms of x of order strictly less than N, thus proving this theorem. Note that this also applies to additive and resudial connections, as de ones used in ResNet and FractalNet, since they can be expressed as in.B.3 PROOF OF THEOREMS 5.1 TO 5.3 DISPLAYFORM13 For the forthcoming analysis let us assume r 0 ≤ M. This assumption is done, so that we can write min{r 0, M} = r 0, merely for notation purposes since we show that this does not affect the generality of the . Using this assumption, we upper bound the rank of the grid tensor as DISPLAYFORM14 It was shown in that, when the weights are independently generated from some continuous distribution, we have that rank φ 1,j,γ = min{r 0, M} with probability 1. Note that, the bounds obtained for r 0 values greater than M is the same as for r 0 = M, thus implying that the assumption of r 0 ≤ M does not affect the generality of the . Finally, by induction up to the L-th layer, we obtain a bound for the grid tensor rank as DISPLAYFORM15 Since we assumed networks with hidden layer widths r l decaying (or increasing) at an exponential rate of λ ∈ R. Formally, this is r l = λr l−1 ∈ N, thus r l = (λ) l r for all l = 0, 1,..., L − 1, where r r 0. Therefore, we may simplify the obtained bound to DISPLAYFORM16 We this analysis by proving Theorem 5.1. To that end let A y dense be the grid tensor of a dense (L, r, λ, k) ConvAC with k > 0, while A y stand is the grid tensor of a (L, r, λ, 0) ConvAC with r ∈ R. As discussed in Section 4, this dense version of the former (L, r, λ, 0) ConvAC is equivalent to virtually increasing the widths of the ConvAC, which translates extra additive terms in the expressions from 12. Moreover, using corollary 1 we observe that, if the ranks of the tensors φ l,j,γ are additive and multiplicative up to rank(A y dense) > rank(A y stand), so they are up to rank(A y stand). A weak dense gain value G w ∈ R is achieved when there is a set of functions realized by the (L, r, λ, k) ConvAC that cannot be realized by (L, r, λ, 0) ConvAC unless r = G w r. To bound this gain, let us assume the best case scenario where rank(A DISPLAYFORM17 which proves Theorem 5.1.For Theorem 5.2 we use consider the particular case of k = 1, which yields a core tensor given by the hierarchical tensor decomposition from. We use the same assumption of r 0 ≤ M and define the virtually increased widthsr l r l + r l−1 ∈ N for l = 1,..., L − 1 andr 0 M. This leads to rank φ 1,j,γ = rank Note that for r l = λr l−1 ∈ N (λ ∈ R), we get virtually increased widthsr l = (1 + λ) l r = λ 1 + As in for the proof of Theorem 5.1, the maximum dense gain G w is obtained when rank(A y dense) reaches the maximum possible rank. In this particular case, this corresponds to rank(A By definition, we have that ∆P std = P (L, Gr, λ, 0) − P (L, Gr, λ, 0) and ∆P dense = P (L, Gr, λ, k) − P (L, r, λ, 0), thus yielding Finnaly, we use these expressions to compute the ratio of interest as DISPLAYFORM18 2 ) l + (G 2 − 1) k q=1 λ −q which proves this proposition.

Represent the TLDR for finding the corresponding scientific article

We analyze the expressive power of the connections used in DenseNets via tensor decompositions.

We compare graph RNNs and graph ConvNets, and we consider the most generic class of graph ConvNets with residuality.


Question: Represent the scientific article to find a short TLDR

Question: We consider the following central question in the field of Deep Reinforcement Learning (DRL): How can we use implicit human feedback to accelerate and optimize the training of a DRL algorithm? State-of-the-art methods rely on any human feedback to be provided explicitly, requiring the active participation of humans (e.g., expert labeling, demonstrations, etc.). In this work, we investigate an alternative paradigm, where non-expert humans are silently observing (and assessing) the agent interacting with the environment. The human's intrinsic reactions to the agent's behavior is sensed as implicit feedback by placing electrodes on the human scalp and monitoring what are known as event-related electric potentials. The implicit feedback is then used to augment the agent's learning in the RL tasks. We develop a system to obtain and accurately decode the implicit human feedback (specifically error-related event potentials) for state-action pairs in an Atari-type environment. As a baseline contribution, we demonstrate the feasibility of capturing error-potentials of a human observer watching an agent learning to play several different Atari-games using an electroencephalogram (EEG) cap, and then decoding the signals appropriately and using them as an auxiliary reward function to a DRL algorithm with the intent of accelerating its learning of the game. Building atop the baseline, we then make the following novel contributions in our work: (i) We argue that the definition of error-potentials is generalizable across different environments; specifically we show that error-potentials of an observer can be learned for a specific game, and the definition used as-is for another game without requiring re-learning of the error-potentials. (ii) We propose two different frameworks to combine recent advances in DRL into the error-potential based feedback system in a sample-efficient manner, allowing humans to provide implicit feedback while training in the loop, or prior to the training of the RL agent. (iii) Finally, we scale the implicit human feedback (via ErrP) based RL to reasonably complex environments (games) and demonstrate the significance of our approach through synthetic and real user experiments. Deep Reinforcement Learning (DRL) algorithms have now beaten human experts in Go , taught robots to become parkour masters, and enabled truly autonomous vehicles . However, current state-of-the-art RL agents equipped with deep neural networks are inherently complex, difficult and time-intensive to train. Particularly in complex environments with sparse reward functions (e.g., maze navigation), the DRL agents need an inordinate amount of interaction with the environment to learn the optimal policy. Human participation can potentially help DRL algorithms by accelerating their training and reducing the learning costs without compromising final performance. This potential has inspired a several research efforts where either an alternative (or supplementary) feedback is obtained from the human participant . Such approaches despite being highly effective, severely burden the human-in-the-loop demanding either expert demonstrations or explicit feedback . In this paper, we investigate an alternative paradigm that substantially increases the richness of the reward functions, while not severely burdening the human-in-the-loop. We study the use of electroencephalogram (EEG) based brain waves of the human-in-the-loop to generate the reward functions that can be used by the DRL algorithms. Such a model will benefit from the natural rich activity of a powerful sensor (the human brain), but at the same time not burden the human if the activity being relied upon is intrinsic. This paradigm is inspired by a high-level error-processing system in humans that generates error-related potential/negativity (ErrP or ERN) .When a human recognizes an error made by an agent, the elicited ErrP can be captured through EEG to inform agent about the sub-optimality of the taken action in the particular state. As a baseline contribution, we demonstrate the feasibility of capturing error-potentials of a human observer watching an agent learning to play several different Atari-games, and then decoding the signals appropriately and using them as an auxiliary reward function to a DRL algorithm. We show that a full access approach to obtain feedback on every state-action pair while RL agent is learning, can significantly speedup the training convergence of RL agent. We contend that while obtaining such implicit human feedback through EEG is less burdensome, it is still a time-intensive task for the subject and the experimenter alike. This, combined with the noisy EEG signals and stochasticity in inferring error-potentials, raises significant challenges in terms of the practicality of the solution. In this context, we first argue that the definition of ErrPs is generalizable across different environments. We show that ErrPs of an observer can be learned for a specific game, and the definition used as-is for another game without requiring re-learning of the ErrP. This is notably different from previous approaches (Chavarriaga & Millán, 2010;), where the labeled ErrPs are obtained in the same environment (where the RL task is performed). For any new and unseen environment, it does not require the human to go through the training phase again, and assumes no prior knowledge about the optimal state-action pairs of the environment. We present two different frameworks to combine recent advances in DRL into the implicit human feedback mechanism (via ErrP) in a practical, sample-efficient manner. This reduces the cost of human supervision sufficiently allowing the DRL systems to train. Relying on Active Learning (AL) methods, our first framework allows humans to provide implicit feedback in the loop, while an RL agent is being trained. An uncertainty based acquisition function is modeled to select the samples state-action pairs for querying the implicit human feedback. However, as a human is always required to be in the loop, our second framework allows humans to provide their feedback implicitly before the agent starts training. Based on the human feedback obtained during pre-training, a quality (Q) function is learned over these imperfect demonstrations to provide the supplementary reward to the RL agent. We present from real ErrP experiments to evaluate the acceleration in learning, and sample efficiency, in both frameworks. In summary, the novel contributions of our work are, 1. We demonstrate the generalizability of error-potentials over various Atari-like environments (discrete grid-based navigation games, studied in this work), enabling the estimation of implicit human feedback in new and unseen environments. 2. We propose two different frameworks to combine recent advances in DRL into ErrP based feedback system in a practical, sample-efficient manner. The first framework allows humans to provide implicit feedback while training in the loop. Taking advantage of recent approaches in learning from imperfect demonstrations, in the second framework, the implicit human feedback is obtained prior to the training of the RL agent. 3. We scale the implicit human feedback (via ErrP) based RL to reasonably complex environments and demonstrate the significance of our approach through synthetic and real user experiments.;; studied RL from human rankings or ratings, however rely on explicit human feedback, and assume that the feedback is noiseless. Demonstrations have been commonly used to improve the efficiency of RL (; ;), and a common paradigm is to initialize RL algorithms with good policy or Q function (; ;). In this work, we use rely on implicit feedback from non-expert humans (via ErrPs) which is inherently noisy. (Chavarriaga & Millán, 2010; ;) demonstrate the benefit of ErrPs in a very simple setting (i.e., very small state-space), and use ErrP-based feedback as the only reward. Moreover, in all of these works, the ErrP decoder is trained on a similar game (or robotic task), essentially using the knowledge that is supposed to be unknown in the RL task. In our work, we use labeled ErrPs examples of very simple and known environments to train the ErrP decoder, and combine with the recent advances in DRL in a sample-efficient manner for reasonably complex environments. Consider a Markov Decision Process (MDP) problem M, as a tuple < X, A, P, P 0, R, γ >, with state-space X, action-space A, transition kernel P, initial state distribution P 0, accompanied with reward function R, and discounting factor 0 ≤ γ ≤ 1. Here the random variable Z(s, a) denotes the accumulated discounted future rewards starting from state s and action a. In this work, we only consider MDP with discrete actions and states. In model-free RL method, the central idea of most prominent approaches is to learn the Q-function by minimizing the Bellman residual, i.e., L(Q) = E π Q(x, a) − r − γQ(x,â) 2, and temporal difference (TD) update where the transition tuple (x, a, r, x) consists of a consecutive experience under behavior policy π. Modern techniques in DRL such as DQN and the target network are also adpoted throughout the paper. The humans intrinsic reactions to the agents behavior is sensed as implicit feedback by placing electrodes on the human scalp and monitoring what are known as event-related electric potentials. We rely on the Riemannian Geometry framework for the classification of error-related potentials (; presented in Appendix 7.1. We consider the classification of error-related potentials as a binary variable indicating the presence (i.e., action taken by the agent is incorrect) and absence of error (i.e., action taken by the agent is correct). With the availability of implicit human feedback, we explore how the training of state-of-the-art DRL algorithms can be accelerated. A trivial approach is to obtain feedback on every state-action pair while RL agent is learning (also known as full access). It is to add a negative penalty to the reward when ErrP is detected, and keep using the original reward from the environment without ErrP detected. The evaluation of this method based on real ErrP data is shown in section 5.1. The validate that this method can speed up the training convergence of RL agent significantly. We contend that while obtaining such implicit human feedback through EEG is less burdensome, it is still a time-intensive task for the subject and the experimenter alike. This, combined with the noisy EEG signals and stochasticity in inferring ErrPs, raises significant challenges in terms of the practicality of the solution. In this section, we discuss three approaches towards integrating the ErrP with recent advances in DRL in a practical manner. Firstly, we show that ErrPs of an observer can be learned for a specific game, and the definition used as-is for another game without requiring re-learning of the ErrP. Further, we discuss two frameworks to combine the recent advances in DRL into the implicit human feedback mechanism (via ErrP) to accelerate the RL agent learning in a sample-efficient manner. The first framework allows humans to provide implicit feedback while training in the loop, without any prior knowledge on the game. In the second framework, the implicit human feedback is obtained prior to the training of the RL agent. It exploits the initially given trajectories with ErrP labels to learn a reward function for augmenting the RL agent, where human with some prior knowledge is needed to specify some non-expert trajectories. Recently, Q function can be shown to have better generalization in state-space if trained with non-expert demonstrations . Error-potentials in the EEG signals is studied under two major paradigms in human-machine interaction tasks, (i) feedback and response ErrPs: error made by human (; ; ; ;), (ii) interaction ErrPs: error made by machine in interpreting human intent (Ferrez & Millán, 2005). Another interesting paradigm is when human is watching (and silently assessing) the machine performing a specific task (Chavarriaga & Millán, 2010). The manifestation of these potentials across these paradigms were found quite similar in terms of their general shape, timings of negative and positive peaks, frequency characteristics etc., (Ferrez & Millán, 2005; Chavarriaga & Millán, 2010). This prompts us to explore the consistency of the error-potentials across different environments (i.e., games, in our case). We restrict the score of our work to the paradigm of human acting as a silent observer of the machine actions. In Fig.5, we plot the grand average waveforms across three environments (Maze, Catch and Wobble), to visually validate the consistency of potentials. We can see that the shape of negativity, and the timings of the peaks is quite consistent across the three game environments studied in this work. Further, we perform experimental evaluation in section 5.2.1, to show that error-potentials are indeed generalizable across environments, and can further be used to inform deep reinforcement learning algorithm in a new and unseen environments. Active Learning (AL) frameworks have been proved quite successful in optimizing the learning task while minimizing the required number of labeled examples . In AL, an acquisition function is used to efficiently select the data points requested for labeling from an external oracle. We introduce a framework of training RL agents with implicit non-expert human feedback in the loop, leveraging recent advances in active learning methods. We present our active learning based framework in Fig. 2 (a). We use an uncertainty-based acquisition function to select the state-action pairs required for non-expert human labeling (via ErrP). Since it is critical to keep the coherence between consecutive state-action pairs shown to the human subject, a full trajectory from start to end of the game can be shown. The calculation of the acquisition function is based on the state-action pair uncertainty along the trajectory, as explained in Appendix 7.3. Specifically, we model the Deep-Q-Network (DQN) by Bayesian learning methods, which have strong capacity of uncertainty estimation . The DQN is trained with experience collected in the reply buffer, a structure commonly used in deep RL algorithms. In contrast to the full access method, the presented framework queries for ErrP based state-action pair labeling only at the end of every N E episodes. We further store the decoded ErrP labels into buckets, to be used for future training augmentation. In every step, the RL agent inquire the negativity of the current state-action pair from buckets, instead of ErrP labeling, which reduces the number of ErrP inquiries significantly. This negativity can add a negative penalty to the environmental reward as auxiliary. Trajectory Generation and Selection: ErrP labeling informs the RL agent about negativity of selected actions, ideally preventing the agent from deviating from the optimal paths in the game. However, these optimal paths are unknown a priori. For generating trajectories for ErrP labeling, we empirically found that following greedily the action with largest Q value in every state based on the most updated DQN performs very well. Then the trajectory with the largest acquisition function output is selected for querying ErrP labels. Three acquisition functions evaluated in experiments are all formulated based on the uncertainty estimation of Q values, and their formulations and approximations are introduced in Appendix 7.3. The framework are presented in Algorithm 1. RL algorithms deployed in the environment with sparse rewards demand heavy explorations (require a large number of trial-and-errors) during the initial stages of training. Imitation learning from a small number of demonstrations followed by RL fine-tuning is a promising paradigm to improve the sample efficiency in such cases (Večerík et al., 2017; ;). Inspired by the paradigm of imitation learning, we develop a novel framework that can robustly Starting from random initial state, the RL agent plays the game until the end of the episode; 4 Update the DQN Q by experiences randomly selected from the replay buffer; The flowchart of the second framework is in Fig. 2(b). In this framework, the trajectories in the demonstration are first criticized by ErrP labeling in experiments, and a quality (Q) function is learned from the labeled trajectories in the reward learning step. An alternative reward is derived from the learned quality function, augmenting the following RL algorithm. This approach is considerably different from our first framework (section 4.2), as we only make queries for ErrP labeling on trajectories initially given in the demonstration (rather than making queries continuously during every training step). These queries are made before the RL agent starts training, improving the efficiency of the total number of labeling (implicit, ErrP based) queries made to the external oracle (human). Similar to the first framework, the demonstrations for ErrP labeling can only consist of complete trajectories. We assumed that the trajectories in the demonstration are initially specified by human or other external sources, without any reward information. This is a reasonable assumption since the rewards may be unknown to humans in general cases. The human subject in the experiment provides feedback in an implicit manner (via ErrP) on state-action pairs along the trajectories, labeling every state-action pair as a positive or negative sample. Based on the decoded ErrP labels and initially given trajectories, the proposed framework learns the reward function based on maximum entropy RL methods , as explained in details in Appendix 7.4. Different from conventional imitation learning, these trajectories are not given by expert policies, allowing the non-experts to demonstrate. Moreover, the Q function learned from imperfect demonstrations can have better estimations on states unseen in the demonstration, and provide better generalization in the state-space . We have developed three discrete-grid based navigation games in OpenAI Gym emulating Atari framework , namely (i) Wobble, (ii) Catch, and (iii) Maze, shown in Fig. 3(a). We use the default Atari dimensions (i.e., 210x160 pixels). The source codes of the games can be found in the public repository 1, and can be used with the OpenAI Gym module. 1 source code is attached with the submission for anonymity purposes Wobble: Wobble is a simple 1-D cursor-target game, where the middle horizontal plane is divided into 20 discrete blocks. At the beginning of the game, the cursor appears at the center of the screen, and the target appears no more than three blocks away from the cursor position. The action space for the agent is moving one step either left or right. The game is finished when the cursor reaches the target. Once the game is finished, a new game is started with the cursor in place. Catch: Catch is a simplistic version of Eggomania 2 (Atari 2600 benchmark), where we display a single egg on the screen at a time. The screen dimensions are divided into 10x10 grid space, where the egg and the cart, both occupies one block. The action space of the agent consists of "NOOP" (no operation), "moving left" and "moving right". At the start of the game, the horizontal position of the egg is chosen randomly. At each time step, the egg falls one block in the vertical direction. Maze: Maze is a 2-D navigational game, where the agent has to reach to a fixed target. The Atari screen is centered and divided into 10x10 equal-sized blocks. The agent and target occupy one block. The action space consists of four directional movements. The maze architecture is kept fixed for the purpose of this work. If an agent moves, but hits a wall, a quick blinking of the agent is displayed, to show the action taken by the agent. We designed and developed an experimental protocol, where a machine agent plays a computer game, while a human silently observes (and assesses) the actions taken by the machine agent. These implicit human reactions are captured by placing raw electrodes on the scalp of the human brain in the form of EEG. The electrode cap was attached with the OpenBCI 3 platform, which was further connected to a desktop machine over the wireless channel. In the game design (developed on OpenAI Gym), we open a TCP port, and continuously transmit the current state-action pair using the TCP/IP protocol. We used OpenViBE software to record the human EEG data. OpenViBE continuously listens to the TCP port (for state-action pairs), and timestamps the EEG data in a synchronized manner. A total of five human subjects were recruited using standard procedures. We recruited five human subjects (mean age 26.8 ± 1.92, 1 female) for collecting the EEG data. For each subject, we conducted three separate sessions over multiple days. For each subject-game pair, the experimental duration was less than 15 minutes. The agent took action every 1.5 seconds. All the research protocols for the user data collection were reviewed and approved by the Institutional Review Board 4. The full access method as discussed in section 3 is the most preliminary approach to make ErrP labels augment the RL algorithm. It has the fastest training convergence rate (provides upper bound) but makes the maximum possible queries to the external oracle (human) for the implicit feedback. We use this method as a benchmark for comparing the data-efficiency of other RL augmentation methods. The with real ErrP data of 5 subjects are shown in Figure 4. Here the training data of ErrP decoder is from Catch game while the testing data is from Maze. We can see there is a significant improvement in the training convergence. It further validates the generalization capability of ErrP decoding from 1-D to 2-D navigation games. In this paper, "No ErrP" method refers to regular RL algorithms without the help of any human feedback. The success rate is defined as the ratio of success plays in the previous 32 episodes. The training completes when the success rate reaches to 1. In all plots of this paper, solid lines are average values over 10 random seeds, and shaded regions correspond to one standard deviation. In the evaluations of this paper, the Q network is modeled by Bayesian deep learning methods, such as Bayesian DQN or bootstrapped DQN, introduced in Appendix 7.2. In this subsection, we evaluate the performance of three approaches to practially integrate the DRL with implicit human feedback (via ErrPs). We first validate the feasibility of decoding ErrP signals using a 10-fold cross-validation scheme for each game. In this scheme, we train and test on the ErrP samples of the same game environment. In Fig. 5(a), we show the performance of three games in terms of AUC score, sensitivity and specificity, averaged over 5 subjects. The Maze game has the highest AUC score (0.89 ± 0.05) followed by Catch (0.83 ± 0.08) and Wobble (0.77 ± 0.09). To evaluate the generalization capability of errorpotential signals and the decoding algorithm, we train on the samples collected from Catch and test on Maze game. In Fig. 5(b), we provide the AUC score performance compared with the 10-fold CV AUC score of Maze. We can see that the Catch game is able to capture more than 80% of the variability in the ErrPs for Maze game. To provide deeper insights into the generalizability extent, we present the AUC score of generalizability performance over all combinations in fig. 5(c). In the later subsections, we experimentally show that these performance numbers are sufficient to achieve 2.25x improvement in training time (in terms of the number of episodes required). We performed preliminary experiments to gain fundamental insights into the extent of generalizability. All the three games considered in this work, differ in terms of their action space. Wobble can move either left or right (two actions), Catch has an additional "NOOP" (3 actions), and the agent in the Maze can move in either direction (4 actions). To understand the generalizability of ErrP in terms of the actions taken by the agent, we train on the Wobble, and test on the Catch game for two groups -(i) when the agent moves in either direction, and (ii) when the agent stays in the place. We obtain an average AUC score of 0.7359 (± 0.1294) and 0.6423 (± 0.1451) for both groups, respectively. Through a paired t-test, we found the difference in mean statistically significant. Similarly, for the Catch game, we test two groups -(i) when egg is close to the paddle, and (ii) when egg is far from the paddle. We found the mean AUC scores of 0.71 (± 0.1) and 0.84 (± 0.12) for each group, respectively. The difference of the mean of both groups was found statistically significant. In evaluating active RL framework, we explore three forms of acquisition functions, i.e., entropy, mutual information, and confidence interval. Their expressions and approximation techniques are illustrated with details in Appendix 7.3. The benchmark performance of full access method is shown in section 5.1. We first evaluate the performance of first framework with synthesized human feedback, which is presented in Appendix 7.5.1 on box world environment . In this section, we evaluate the first framework on Maze game with real ErrP experimental data. We use Bayesian DQN for the Q network. Three acquisition functions are compared in Figure 6 with detailed statistics on Table 7.5.1, which has similar as the synthetic case. Based on real ErrP data, we can show that compared with full access method, the first framework can reach similar performance with much less feedback inquiries. In the evaluation of this framework, the trajectories given initially are generated based on optimal paths randomly corrupted by wrong actions, which appear with the probability of 0.2. We evaluate the performance with 10 and 20 trajectories given initially. Prior to training the RL agent, each subject is asked to provide feedback via ErrP on the state-action pairs along these trajectories. We conducted experiments on 5 subjects, based on Maze game. Here the Q network is modeled by Bayesian DQN. The performance of augmented RL algorithms is shown in Figure 7. The reward function is shown to speed up the training convergence of the RL agent significantly. Since trajectories are randomly generated initially, the number of ErrP inquiries of the second framework is equal to 372.1(±58.2), based on the statistics in our simulations. The second framework even outperforms the full access method, with ErrP inquiries on only 20 trajectories, proving its data efficiency. However, this framework needs a human or external source, who has some prior knowledge of the game, to specify the initial trajectories. We first demonstrate the feasibility of capturing error-potentials of a human observer watching an agent learning to play several different Atari-games, and then decoding the signals appropriately and using them as an auxiliary reward function to a DRL algorithm. Then we argue that the definition of ErrPs is generalizable across different environment. In the ideal approach, we validate the augmentation effect of ErrP labels on RL algorithms by the full access method. Then, in the practical approach, we propose two augmentation frameworks for RL agent, applicable to different situations. The first is to integrate human into the training loop of RL agent based on active learning, while the second is to learn a reward function from imperfect demonstrations labeled by ErrP. The demonstration of the generalizability of error-potentials is limited across the environments presented in the paper. We have considered discrete grid-based reasonably complex navigation games. The validation of the generalization to a variety of Atari and Robotic environments is the subject of the future work. We also plan to test our framework of integrating implicit human feedback (via ErrPs) over robotic environments, and text the generalization capability of error-potentials between virtual and physical worlds. As future work, we plan to investigate as to how machines can be assisted in RL by using intrinsic EEG-based cooperations among humans and machines. are bandpass filtered in [0.5, 40] Hz. Epochs of 800ms were extracted relative to pre-stimulus 200ms baseline, and were subjected to spatial filtering. In spatial filtering, prototype responses of each class, i.e., "correct" and "erroneous", are computed by averaging all training trials in the corresponding classes("xDAWN Spatial Filter" (; ;). "xDAWN filtering" projects the EEG signals from sensor space (i.e., electrode space) to the source space (i.e., a low-dimensional space constituted by the actual neuronal ensembles in brain firing coherently). The covariance matrix of each epoch is computed, and concatenated with the prototype responses of the class. Further, dimensionality reduction is achieved by selecting relevant channels through backward elimination. The filtered signals are projected to the tangent space for feature extraction. The obtained feature vector is first normalized (using L1 norm) and fed to a regularized regression model. A threshold value is selected for the final decision by maximizing accuracy offline on the training set. We present the algorithm to decode the ErrP signals in Algorithm 2. Algorithm 2: Riemannian Geometry based ErrP classification algorithm Input: raw EEG signals EEG 1 Pre-process raw EEG signals; 2 Spatial Filtering: xDAWN Spatial Filter (nf ilter); 3 Electrode Selection: ElectrodeSelect (nelec, metric='riemann'); 4 Tangent Space Projection: TangentSpace(metric = "logeuclid") Normalize using L1 norm; 5 Regression: ElasticNet; 6 Select decision threshold by maximizing accuracy Here we introduce two DQN models adopted in this paper. Bayesian DQN The first model we use is a DQN architecture where the Q-function is approximated as a linear function, with weights ω a, of the feature representation of states φ θ (x) ∈ R d, parameterized by neural network with weights θ . Here by utilizing the DQN architecture and imposing Gaussian distributions on ω a, based on Bayesian linear regression (BLR) , the posterior of ω a can be calculated by where we construct disjoint replay buffer D a corresponding to experience with action a, and a matrix Φ θ a ∈ R d×|Da|, vector y a, i.e., the concatenation of state features and target values in set D a. Therefore the posterior of Q value can be the following the Gaussian distribution, Bootstrapped DQN Another Bayesian DQN model we use is bootstrapped DQN . It explores in a similar manner as the Bayesian DQN introduced above, but uses a bootstrapped neural network to approximate a posterior sample for the value. Bootstrapped DQN is also provably efficient, but adopts neural network instead of linear value function and bootstraps instead of Gaussian sampling. It is implemented by K ∈ N bootstrapped estimates of the Q value in parallel, i.e., Q k (s, a; θ), s = 1,..., K. In this work, three acquisition functions are explored in selecting trajectory for ErrP labeling. The trajectory is defined as a sequence of state-action pairs τ:= {(s 0, a 0),..., (s T, a T)}. We denote the trajectory set as D, the learned Q network as Q, and acquisition function as a(τ, Q). The selected trajectory maximizes the acquisition function given Q. In this work, we explore three acquisition functions: • Entropy: Select the trajectory with the maximum entropy, which measures the uncertainty of state-action pairs along the trajectory. 1.96 where N s t (N f t) are number of True (False) ErrP decoding for state-action pair at step t on the trajectory. Since implicit human feedback via ErrP is noisy (hence imperfect demonstrations), we model the reward learning as a probabilistic maximum entropy RL problem. Following the principle of maximum entropy, given Q function Q(·, ·), the policy distribution and value function in terms of Q function can be expressed as follows, where α is a free parameter, tuned empirically. The likelihood of positive and negative state-action pair are denoted as π Q (a|s) and 1 − π Q (a|s). When demonstrations and corresponding implicit human feedback are ready, we train the Q function by maximizing the likelihood of both positive and negative state-action pairs in the demonstrations. In order to refine the reward shape and attenuate the variance of learning updates, we introduce another baseline function t(s) in the Q function. Hence, the Q function becomes Q B (s, a):= Q(s, a) − t(s). It can be proved that Q B (·, ·) and Q(·, ·) induce the same optimal policy . The baseline function t * (·) can be learned by optimizing t * = arg min t J(t), and the objective is defined as where the loss function l(·) is chosen to be l 1 -norm through empirical evaluations. In addition to the demonstration D, we incorporate another set of demonstrations D R, containing transitions randomly sampled from environment without reward information. The set D R is to help the function t(·) to efficiently learn the state dynamics, and does not require human labeling, essentially keeping the number of queries same. After reward learning, consisting of learning Q function and baseline function, for any transition tuple (s, a, s), the learned reward function can be represented as Q B (s, a) − γ max a ∈A Q B (s, a). We then use this reward function to augment the following RL agent. 7.5 BOX WORLD GAME ENVIRONMENT This environment consists of an 8 × 8 pixels room with keys and boxes randomly scattered. The room also contains an agent, represented by a single black pixel, which can move in four directions: up, down, left, and right. Keys are represented by a single colored pixel, and boxes are represented by two adjacent colored pixels, where the pixel on the right represents the box's lock. A key can open a lock if its color matches the lock. Its screen shot is shown in Figure 8 (a). Here, we evaluate the first framework on the Box World game with synthetic human feedback. This environment is introduced with details in Appendix 7.5. The synthetic feedback gives noisy label on each state-action pair, where the correct (optimal) one is labeled as wrong (sub-optimal) with the probability of 1, and the wrong (sub-optimal) one is labeled as correct with the probability of 2. Here, the Q network is modeled by bootstrapped DQN. This game has a combinatorially complex environment which cannot be quickly solved by a regular RL algorithm. The simulation are shown in Figure 8, where "No Feedback" refers to the RL algorithms without the help of human feedback. The detailed statistics of evaluations are illustrated in Table 7.5.1. Three acquisition functions are evaluated for comparison. The mean complete episode for mutual information, entropy and confidence level are 167.0, 177.5 and 231.2 for Human 1, 166.3, 184.2 and 195.2 for Human 2. We can see the first framework can achieve similar performance as full access method in terms of convergence speed, with much smaller number of inquiries. The acquisition function of confidence interval performs worst, because it does not consider the properties of the trained model. The mutual information performs better than entropy, but needs a larger number of human feedback inquiries.

We use implicit human feedback (via error-potentials, EEG) to accelerate and optimize the training of a DRL algorithm, in a practical manner.

represent the tldr for finding the corresponding scientific article

The overall goal of this work is to enable sample-efficient imitation from expert demonstrations, both with and without the provision of expert action labels, through the use of f-divergences.