Question: Represent the scientific article to find a short TLDR

Question: Recurrent neural networks (RNNs) are an effective representation of control policies for a wide range of reinforcement and imitation learning problems. RNN policies, however, are particularly difficult to explain, understand, and analyze due to their use of continuous-valued memory vectors and observation features. In this paper, we introduce a new technique, Quantized Bottleneck Insertion, to learn finite representations of these vectors and features. The is a quantized representation of the RNN that can be analyzed to improve our understanding of memory use and general behavior. We present of this approach on synthetic environments and six Atari games. The ing finite representations are surprisingly small in some cases, using as few as 3 discrete memory states and 10 observations for a perfect Pong policy. We also show that these finite policy representations lead to improved interpretability. Deep reinforcement learning (RL) and imitation learning (IL) have demonstrated impressive performance across a wide range of applications. Unfortunately, the learned policies are difficult to understand and explain, which limits the degree that they can be trusted and used in high-stakes applications. Such explanations are particularly problematic for policies represented as recurrent neural networks (RNNs) BID16 BID14, which are increasingly used to achieve state-of-the-art performance BID15 BID21. This is because RNN policies use internal memory to encode features of the observation history, which are critical to their decision making, but extremely difficult to interpret. In this paper, we take a step towards comprehending and explaining RNN policies by learning more compact memory representations. Explaining RNN memory is challenging due to the typical use of high-dimensional continuous memory vectors that are updated through complex gating networks (e.g. LSTMs, GRUs BID10 BID5). We hypothesize that, in many cases, the continuous memory is capturing and updating one or more discrete concepts. If exposed, such concepts could significantly aid explainability. This motivates attempting to quantize the memory and observation representation used by an RNN to more directly capture those concepts. In this case, understanding the memory use can be approached by manipulating and analyzing the quantized system. Of course, not all RNN policies will have compact quantized representations, but many powerful forms of memory usage can be captured in this way. Our main contribution is to introduce an approach for transforming an RNN policy with continuous memory and continuous observations to a finite-state representation known as a Moore Machine. To accomplish this we introduce the idea of Quantized Bottleneck Network (QBN) insertion. QBNs are simply auto-encoders, where the latent representation is quantized. Given a trained RNN, we train QBNs to encode the memory states and observation vectors that are encountered during the RNN operation. We then insert the QBNs into the trained RNN policy in place of the "wires" that propagated the memory and observation vectors. The combination of the RNN and QBN in a policy represented as a Moore Machine Network (MMN) with quantized memory and observations that is nearly equivalent to the original RNN. The MMN can be used directly or fine-tuned to improve on inaccuracies introduced by QBN insertion. While training quantized networks is often considered to be quite challenging, we show that a simple approach works well in the case of QBNs. In particular, we demonstrate that "straight through" gradient estimators as in BID1 BID6 are quite effective. We present experiments in synthetic domains designed to exercise different types of memory use as well as benchmark grammar learning problems. Our approach is able to accurately extract the ground-truth MMNs, providing insight into the RNN memory use. We also did experiments on 6 Atari games using RNNs that achieve state-of-the-art performance. We show that in most cases it is possible to extract near-equivalent MMNs and that the MMNs can be surprisingly small. Further, the extracted MMNs give insights into the memory usage that are not obvious based on just observing the RNN policy in action. For example, we identify games where the RNNs do not use memory in a meaningful way, indicating the RNN is implementing purely reactive control. In contrast, in other games, the RNN does not use observations in a meaningful way, which indicates that the RNN is implementing an open-loop controller. There have been efforts made in the past to understand the internals of Recurrent Networks BID12 BID0 BID22 BID17 BID11 BID18. However, to the best of our knowledge there is no prior work on learning finite-memory representations of continuous RNN policies. Our work, however, is related to a large body of work on learning finite-state representations of recurrent neural networks. Below we summarize the branches of that work and the relationship to our own. There has been a significant history of work on extracting Finite State Machines (FSMs) from recurrent networks trained to recognize languages BID28 BID23 BID3. Typical approaches include discretizing the continuous memory space via gridding or clustering followed by minimization. A more recent approach is to use classic query-based learning algorithms to extract FSMs by asking membership and equivalence queries BID26. However, none of these approaches directly apply to learning policies, which require extending to Moore Machines. In addition, all of these approaches produce an FSM approximation that is separated from the RNN and thus serve as only a proxy of the RNN behavior. Rather, our approach directly inserts discrete elements into the RNN that preserves its behavior, but allows for a finite state characterization. This insertion approach has the advantage of allowing fine-tuning and visualization using standard learning frameworks. The work most similar to ours also focused on learning FSMs BID28. However, the approach is based on directly learning recurrent networks with finite memory, which are qualitatively similar to the memory representation of our MMNs. That work, however, focused on learning from scratch rather than aiming to describe the behavior of a continuous RNN. Our work extends that approach to learn MMNs and more importantly introduces the method of QBN insertion as a way of learning via guidance from a continuous RNN.This transforms any pre-trained recurrent policy into a finite representation. We note that there has been prior work on learning fully binary networks, where the activation functions and/or weights are binary (e.g. BID1 BID6 BID8). The goal of that line of work is typically to learn more time and space efficient networks. Rather, we focus on learning only discrete representations of memory and observations, while allowing the rest of the network to use arbitrary activations and weights. This is due to our alternative goal of supporting interpretability rather than efficiency. Recurrent neural networks (RNNs) are commonly used in reinforcement learning to represent policies that require or can benefit from internal memory. At each time step, an RNN is given an observation o t (e.g. image) and must output an action a t to be taken in the environment. During execution an RNN maintains a continuous-valued hidden state h t, which is updated on each transition and influences the action choice. In particular, given the current observation o t and current state h t, an RNN performs the following operations: 1) Extract a set of observation features f t from o t, for example, using a CNN when observations are images, 2) Outputting an action a t = π(h t) according to policy π, which is often a linear softmax function of h t, 3) transition to a new state h t+1 = δ(f t, h t) where δ is the transition function, which is often implemented via different types of gating networks such as LSTMs or GRUs. The continuous and high dimensional nature of h t and f t can make interpreting the role of memory difficult. This motivates our goal of extracting compact quantized representations of h t and f t. Such representations have the potential to allow investigating a finite system that captures the key features of the memory and observations. For this purpose we introduce Moore Machines and their deep network counterparts. Moore Machines. A classical Moore Machine (MM) is a standard finite state machine where all states are labeled by output values, which in our case will correspond to actions. In particular, a Moore Machine is described by a finite set of (hidden) statesĤ, an initial hidden stateĥ 0, a finite set of observationsÔ, a finite set of actions A, a transition functionδ, and a policyπ that maps hidden states to actions. The transition functionδ:Ĥ ×Ô →Ĥ returns the next hidden statê h t+1 =δ(ĥ t,ô t) given the current stateĥ t and observationô t. By convention we will use h t andĥ t to denote continuous and discrete states respectively and similarly for other quantities and functions. Moore Machine Networks. A Moore Machine Network (MMN) is a Moore Machine where the transition functionδ and policyπ are represented via deep networks. In addition, since the raw observations given to an MMN are often continuous, or from an effectively unbounded set (e.g. images), an MMN will also provide a mappingĝ from the continuous observations to a finite discrete observation spaceÔ. Hereĝ will also be represented via a deep network. In this work, we consider quantized state and observation representations where eachĥ ∈Ĥ is a discrete vector and each discrete observation inÔ is a discrete vector that describes the raw observation. We will denote the quantization level as k and the dimensions ofĥ andf by B h and B f respectively. Based on the above discussion, an MMN can be viewed as a traditional RNN, where: 1) The memory is restricted to be composed of k-level activation units, and 2) The environmental observations are intermediately transformed to a k-level representationf before being fed to the recurrent module. Given an approach for incorporating quantized units into the backpropagation process, it is straightforward, in concept, to learn MMNs from scratch via standard RNN learning algorithms. However, we have found that learning MMNs from scratch can be quite difficult for non-trivial problems, even when an RNN can be learned with relative ease. For example, we have not been able to train highperforming MMNs from scratch for Atari games. Below we introduce a new approach for learning MMNs that is able to leverage the ability to learn RNNs. Given a trained RNN, our key idea is to first learn quantized bottleneck networks (QBNs) for embedding the continuous observation features and hidden state into a k-level quantized representation. We will then insert the QBNs into the original recurrent net in such a way that its behavior is minimally changed with the option of fine-tuning after insertion. The ing network can be viewed as consuming quantized features and maintaining quantized state, which is effectively an MMN. Below we describe the steps in further detail, which are illustrated in FIG0. A QBN is simply an autoencoder where the latent representation between the encoder and decoder (i.e. the bottleneck) is constrained to be composed of k-level activation units. While, traditional autoencoders are generally used for the purpose of dimensionality reduction in continuous space BID9, QBNs are motivated by the goal of discretizing a continuous space. Conceptually, this can be done by quantizing the activations of units in the encoding layer. We represent a QBN via a continuous multilayer encoder E, which maps inputs x to a latent encoding E(x), and a corresponding multilayer decoder D. To quantize the encoding, the QBN output is given by DISPLAYFORM0 In our case, we use 3-level quantization in the form of +1, 0 and −1 using the quantize function, which assumes the outputs of E(x) are in the range [−1, 1]. 1 One choice for the output nodes of E(x) would be the tanh activation. However, since the gradient of tanh is close to 1 near 0, it can be difficult to produce quantization level 0 during learning. Thus, as suggested in , to support 3-valued quantization we use the following activation function, which is flatter in the region around zero input.φ(x) = 1.5 tanh(x) + 0.5 tanh(−3x)Of course introducing the quantize function in the QBN in b(x) being non-differentiable, making it apparently incompatible with backpropagation, since the gradients between the decoder and encoder will almost always be zero. While there are a variety of ways to deal with this issue, we have found that the straight-through estimator, as suggested and used in prior work BID8 BID1 BID6 ) is quite effective. In particular, the standard straightthrough estimator of the gradient simply treats the quantize function as the identity function during back-propagation. Overall, the inclusion of the quantize function in the QBN effectively allows us to view the last layer of E as producing a k-level encoding. We train a QBN as an autoencoder using the standard L 2 reconstruction error x − b(x) 2 for a given input x. Given a recurrent policy we can run the policy in the target environment in order to produce an arbitrarily large set of training sequences of triples (o t, f t, h t), giving the observation, corresponding observation feature, and hidden state at time t. Let F and H be the sets of all observed features and states respectively. The first step of our approach is to train two QBNs, b f and b h, on F and H respectively. If the QBNs are able to achieve low reconstruction error then we can view latent "bottlenecks" of the QBNs as a high-quality k-level encodings of the original hidden states and features. We now view b f and b h as "wires" that propagate the input to the output, with some noise due to imperfect reconstruction. We insert these wires into the original RNN in the natural way (stage-3 in FIG0). The b f QBN is inserted between the RNN units that compute the features f and the nodes those units are connected to. The b h QBN is inserted between the output of the recurrent network block and the input to the recurrent block. If b f and b h always produced perfect reconstructions, then the of inserting them as described would not change the behavior of the RNN. Yet, the RNN can now be viewed as an MMN since the bottlenecks of b f and b h provide a quantized representation of the features f t and states h t.Fine Tuning. In practice, the QBNs will not achieve perfect reconstruction and thus, the ing MMN may not behave identically to the original RNN. Empirically, we have found that the performance of the ing MMN is often very close to the RNN directly after insertion. However, when there is non-trivial performance degradation, we can fine-tune the MMN by training on the original rollout data of the RNN. Importantly, since our primary goal is to learn a representation of the original RNN, during fine-tuning our objective is to have the MMN match the softmax distribution over actions produced by the RNN. We found that training in this way was significantly more stable than training the MMN to simply output the same action as the RNN. After obtaining the MMN, one could use visualization and other analysis tools to investigate the memory and it's feature bits in order to gain a semantic understanding of their roles. Solving the full interpretation problem in a primitive way is beyond the scope of this work. Extraction. Another way to gain insight is to use the MMN to produce an equivalent Moore Machine over atomic state and observation spaces, where each state and observation is a discrete symbol. This machine can be analyzed to understand the role of different machine states and how they are related. In order to create the Moore Machine we run the learned MMN to produce a dataset of <ĥ t−1,f t,ĥ t, a t >, giving the consecutive pairs of quantized states, the quantized features that led to the transition, and the action selected after the transition. The state-space of the Moore Machine will correspond to the p distinct quantized states in the data and the observation-space of the machine will be the q unique quantized feature vectors in the data. The transition function of the machineδ is constructed from the data by producing a p × q transaction table that captures the transitions seen in the data. Minimization. In general, the number of states p in the ing Moore Machine will be larger than necessary in the sense that there is a much smaller, but equivalent, minimal machine. Thus, we apply standard Moore Machine minimization techniques to arrive at the minimal 2 equivalent Moore Machine BID19. This often dramatically reduces the number of distinct states and observations. Our experiments address the following questions: 1) Is it possible to extract MMNs from RNNs without significant loss in Performance? 2) What is the general magnitude of the number of states and observations in the minimal machines, especially for complex domains such as Atari? 3) Do the learned MMNs help with interpretability of the recurrent policies? In this section, we begin addressing these questions by considering two domains where ground truth Moore Machines are known. The first is a parameterized synthetic environment, Mode Counter, which can capture multiple types of memory use. Second, we consider benchmark grammar learning problems. The class of Mode Counter Environments (MCEs) allows us to vary the amount of memory required by a policy (including no memory) and the required type of memory usage. In particular, MCEs can require varying amounts of memory for remembering past events and implementing internal counters. An MCE is a restricted type of Partially Observable Markov Decision Process, which transitions between one of M modes over time according to a transition distribution, which can depend on the current mode and amount of time spent in the current mode. There are M actions, one for each mode, and the agent receives a reward of +1 at the end of the episode if it takes the correct action associated with the active mode at each time step. The agent does not observe the mode directly, but rather must infer the mode via a combination of observations and memory use. Different parameterizations place different requirements on how (and if) memory needs to be used to infer the mode and achieve optimal performance. Below we give an intuitive description of the MCEs 3 in our experiments. We conduct experiments in three MCE instances, which use memory and observations in fundamentally different ways. This tests our ability to use our approach for determining the type of memory use. 1) Amnesia. This MCE is designed so that the optimal policy does not need memory to track past information and can select optimal actions based on just the current observation. 2) Blind. Here we consider the opposite extreme, where the MCE observations provide no information about optimal actions. Rather, memory must be used to implement counters that keep track of a deterministic mode sequence for determining the optimal actions. 3) Tracker. This MCE is designed so that the optimal policy must both use observations and memory in order to select optimal actions. Intuitively the memory must implement counters that keep track of key time steps where the observations provide information about the mode. In all above instances, we used M = 4 modes. For each MCE instance we use the following recurrent architecture: the input feeds into 1 feed-forward layer with 4 Relu6 nodes BID13 ) (f t), followed by a 1-layer GRU with 8 hidden units (h t), followed by a fully connected softmax layer giving a distribution over the M actions (one per mode). Since we know the optimal policy in the MCEs we use imitation learning for training. For all of the MCEs in our experiments, the trained RNNs achieve 100% accuracy on the imitation dataset and appeared to produce optimal policies. MMN Training. The observation QBN b f and hidden-state QBN b h have the same architecture, except that the number of quantized bottleneck units B f and B h are varied in our experiments. The encoders consist of 1 feed-forward layer of tanh nodes, where the number of nodes is 4 times the size of the bottleneck. This layer feeds into 1 feedforward layer of quantized bottleneck nodes (see Section 4). The decoder for both b f and b h has a symmetric architecture to the encoder. Training of b f and b h in the MCE environments was extremely fast compared to the RNN training, since QBNs do not need to learn temporal dependencies. We trained QBNs with bottleneck sizes of B f ∈ {4, 8} and B h ∈ {4, 8}. For each combination of B f and B h we embedded the QBNs into the RNN to give a discrete MMN and then measured performance of the MMN before and after fine tuning. TAB0 gives the average test score over 50 test episodes. Score of 1 indicates the agent performed optimally for all episodes. In most of the cases no fine tuning was required (marked as '-') since the agent achieved perfect performance immediately after bottleneck insertion due to low reconstruction error. In all other cases, except for Tracker (B h = 4, B f = 4) fine-tuning ed in perfect MMN performance. The exception yielded 98% accuracy. Interestingly in that case, we see that if we only insert one of the bottlenecks at a time, we yield perfect performance, which indicates that the combined error accumulation of the two bottlenecks is responsible for the reduced performance. Moore Machine Extraction. TAB0 also gives the number of states and observations of the MMs extracted from the MMNs both before and after minimization. Recall that the number of states and obsevations before minimization is the number of distinct combinations of values observed for the bottleneck nodes during long executions of the MMN. We see that there are typically significantly more states and observations before minimization than after. This indicates that the MMN learning does not necessarily learn minimal discrete state and observation representations, though the representations accurately describe the RNN. After minimization (Section 4.3), however, in all but one case we get exact minimal machines for each MCE domain. The ground truth minimal machines that are found are shown in the Appendix (Figure 3). This shows that the MMNs learned via QBN insertions were equivalent to the true minimal machines and hence indeed optimal in most cases. The exception matches the case where the MMN did not achieve perfect accuracy. Examining these machines allows one to understand the memory use. For example, the machine for Blind has just a single observation symbol and hence its transitions cannot depend on the input observations. In contrast, the machine for Amnesia shows that each distinct observation symbol leads to the same state (and hence action choice) regardless of the source state. Thus, the policies action is completely determined by the current observation., 2017) ). Here we evaluate our approach over the 7 Tomita Grammars 4, where each grammar defines the set of binary strings that should be accepted or rejected. Since, our focus is on policy learning problems, we treat the grammars as environments with two actions'accept' and'reject'. Each episode corresponds to a random string that is either part of the particular grammar or not. The agent receives a reward of 1 if the correct action accept/reject is chosen on the last symbol of a string. RNN Training. The RNN for each grammar is comprised of a one-layer GRU with 10 hidden units, followed by a fully connected softmax layer with 2 nodes (accept/reject). Since we know the optimal policy, we again use imitation learning to train each RNN using the Adam optimizer and learning rate of 0.001. The training dataset is comprised of an equal number of accept/reject strings with lengths uniformly sampled in the range. TAB1 presents the test for the trained RNNs giving the accuracy over a test set of 100 strings drawn from the same distribution as used for training. Other than grammar #6 5, the RNNs were 100% accurate. MMN Training. Since the raw observations for this problem are from a finite alphabet, we don't need to employ a bottleneck encoder to discretize the observation features. Thus the only bottleneck learned here is b h for the hidden memory state. We use the same architecture for b h as used for the MCE experiments and conduct experiments with B h ∈ {8, 16}. These bottlenecks were then inserted in the RNNs to give MMNs. The performance of the MMNs before and after fine-tuning are shared in TAB1. In almost all cases, the MMN is able to maintain the performance of the RNN without fine-tuning. Fine tuning provides only minor improvements in other cases, which already are achieving high accuracy. Moore Machine Extraction. Our for MM extraction and minimization are in TAB1. In each case, we see a considerable reduction in the MM's state-space after minimization while accurately maintaining the MMN's performance. Again, this shows that the MMN learning does not directly in minimal machines, yet are equivalent to the minimal machines and hence are exact 4 The Tomita grammars are the following 7 languages over the alphabet {0, 1}: solutions. In all cases, except for grammar 6 the minimized machines are identical to the minimal machines that are known for these grammars BID24. DISPLAYFORM0 In this section, we consider applying our technique to RNNs learned for six Atari 6 games using the OpenAI gym BID2. Unlike the above experiments, where we knew the ground truth MMs, for Atari we did not have any preconception of what the MMs might look and how large they might be. The fact that the input observations for Atari (i.e. images) are much more complex than the previous experiments inputs makes it completely unclear if we can expect similar types of . There have been other recent efforts towards understanding Atari agents BID27 BID7. However, we are not aware of any other work which aims to extract finite state representations for Atari policies. RNN Training. All the Atari agents have the same recurrent architecture. The input observation is an image frame, preprocessed by gray-scaling, 2x down-sampling, cropping to an 80 × 80 square and normalizing the values to. The network has 4 convolutional layers (kernel size 3, strides 2, padding 1, and 32,32,16,4 filters respectively). We used Relu as the intermediate activation and Relu6 over the last convolutional layer. This is followed by a GRU layer with 32 hidden units and a fully connected layer with n+1 units, where n is the dimension of the Atari action space. We applied a softmax to first n neurons to obtain the policy and used the last neuron to predict the value function. We used the A3C RL algorithm BID16 ) (learning rate 10 −4, discount factor 0.99) and computed loss on the policy using Generalized Advantage Estimation (λ = 1.0) BID20. We report the trained RNN performance on our six games in the second column of TAB2.MMN Training. We used the same general architecture for the QBN b f as used for the MCE experiments, but adjusted the encoder input and decoder output sizes to match the dimension of the continuous observation features f t. For b h, the encoder has 3 feed-forward layers with (8 × B h), (4 × B h) and B h nodes. The decoder is symmetric to the encoder. For the Atari domains, the training data for b f and b h was generated using noisy rollouts. In particular, each training episode was generated by executing the learned RNN for a random number of steps and then executing an -greedy (with = 0.3) version of the RNN policy. This is intended to increase the diversity of the training data and we found that it helps to more robustly learn the QBNs. We trained bottlenecks for B h ∈ {64, 128} and B f ∈ {100, 400} noting that these values are significantly larger than for our earlier experiments due to the complexity of Atari. Note that while there are an enormous number of potential discrete states for these values of B h the actual number of states observed and hence the number of MMN states can be substantially smaller. Each bottleneck was trained to the point of saturation of training loss and then inserted into the RNN to give an MMN for each Atari game. MMN Performance. TAB2 gives the performance of the trained MMNs before and after finetuning for different combinations of B h and B f. We see that for 4 games, Pong, Freeway, Bowling, and Boxing, the MMNs after fine tuning either achieve identical scores to the RNN or very close (in the case of boxing). This demonstrates the ability to learn a discrete representation of the input and memory for these complex games with no impact on performance. We see that for Boxing and Pong fine tuning was required to match the RNN performance. In the case of Freeway and Bowling, fine-tuning was not required. In the remaining two games, Breakout and Space Invaders, we see that the MMNs learned after fine tuning achieve lower scores than the original RNNs, though the scores are still relatively good. On further investigation, we found that this drop in performance was due to poor reconstruction on some rare parts of the game. For example, in Breakout, after the first board is cleared, the policy needs to press the fire-button to continue, but the learned MMN does not do this and instead times out, which in less score. This motivates the investigation into more intelligent approaches for training QBNs to capture critical information in such rare, but critically important, states. MM Minimization. We see from TAB2 that before minimization, the MMs often have relatively large numbers of discrete states and observations. This is unsurprising given that we are using relatively large values of B h and B f. However, we see that after minimizing the MMNs the number of states and observations reduces by orders of magnitude, sometimes to just a single state and/or single observation. The number of states and observations in many cases are small enough to write out and analyze by hand, making them amenable to careful analysis. However, this analysis is likely to be non-trivial for moderately complex policies due to the need to understand the "meaning" of the observations and in turn of the states. Understanding Memory Use. We were surprised to observe in Atari the same three types of memory use considered for the MCE domains above. First, we see that the MM for Pong has just three states (one per action) and 10 discrete observation symbols (see Figure 2a). Most importantly we see that each observation transitions to the same state (and hence action) regardless of the current state. So we can view this MM as defining a set of rule that maps individual observations to actions with no memory necessary. In this sense, the Pong policy is analogous to the Amnesia MCE [5.1].In contrast, we see that in both Bowling and Freeway there is only one observation symbol in the minimal MM. This means that the MM actually ignores the input image when selecting actions. Rather the policies are open-loop controllers whose action just depends on the time-step rather than the observations. Thus, these policies are analogous to the Blind MCE [5.1]. Freeway has a particularly trivial policy that always takes the Up action at each time step. While this policy behavior could have been determined by looking at the action sequence of the rollouts, it is encouraging that our MM extraction approach also discovered this. As shown in Figure 2b, Bowling has a more interesting open-loop policy structure where it has an initial sequence of actions and then a loop is entered where the action sequence is repeated. It is not immediately obvious that this policy has such an open-loop structure by just watching the policy. Thus, we can see that the MM extraction approach we use here can provide significant additional insight. Breakout, Space Invaders and Boxing use both memory and observations based on our analysis of the MM transition structures. We have not yet attempted a full semantic analysis of the discrete observations and states for any of the Atari policies. This will require additional visualization and interaction tools and is an important direction of future work that is enabled by our approach. Motivated by the goal of better understanding memory use in RNN polices, we introduced an approach for extracting finite state Moore Machines from those policies. The key idea, bottleneck insertion, is to train Quantized Bottleneck Networks to produce binary encodings of continuous RNN memory and input features, and then insert those bottlenecks into the RNN. This yields a BID15, A3C BID16 From the MMN we then extract a discrete Moore machine that can then be transformed into an equivalent minimal machine for analysis and usage. Our on two environments where the ground truth machines are known show that our approach is able to accurately extract the ground truth. We also show experiments in six Atari games, where we have no prior insight into the ground truth machines. We show that, in most cases, the learned MMNs maintain similar performance to the original RNN policies. Further, the extracted machines provide insight into the memory usage of the policies. First, we see that the number of required memory states and observations is surprisingly small. Second, we can identify cases where the policy did not use memory in a significant way (e.g. Pong) and policies that relied only on memory and ignored the observations (e.g. Bowling and Freeway). To our knowledge, this is the first work where this type of insight was reported for policies in such complex domains. A key direction for future work is to develop tools and visualizations for attaching meaning to the discrete observations and in turn states, which will allow for an additional level of insight into the policies. It is also worth considering the use of tools for analyzing finite-state machine structure to gain further insight and analyze formal properties of the policies. An MCE is parameterized by the mode number M, a mode transition function P, a mode life span mapping ∆(m) that assigns a positive integer to each mode, and a count set C containing zero or more natural numbers. At time t the MCE hidden state is a tuple (m t, c t), where m t ∈ {1, 2, . . ., M} is the current mode and c t is the count of time-steps that the system has been consecutively in mode m t. The mode only changes when the lifespan is reached, i.e. c t = ∆(m t) − 1, upon which the next mode m t+1 is generated according to the transition distribution P (m t+1 | m t). The transition distribution also specifies the distribution over initial modes. The agent does not directly observe the state, but rather, the agent only receives a continuous-valued observations o t ∈ at each step, based on the current state (m t, c t). If c t ∈ C then o t is drawn uniformly at random from [m t /M, (m t + 1)/M )] and otherwise o t is drawn uniformly at random from. Thus, observations determine the mode when the mode count is in C and otherwise the observations are uninformative. Note that the agent does not observe the counter. This means that to keep track of the mode for optimal performance the agent must remember the current mode and use memory to keep track of how long the mode has been active, in order to determine when it needs to "pay attention" to the current observation. We conduct experiments with the following three MCE instances 7: 1) Amnesia. This MCE uses ∆(m) = 1 for all modes, C = {0}, and uniformly samples random initial mode and transition distributions. Thus, an optimal policy will not use memory to track information from the past, since the current observation alone determines the current mode. This tests our ability to use MMN extraction to determine that a policy is purely reactive, i.e. not using memory. 2) Blind. Here we use deterministic initial mode and transition distributions, mode life spans that can be larger than 1, and C = {}. Thus, the observations provide no information about the mode and optimal performance can only be achieved by using memory to keep track of the deterministic mode sequence. This allows us to test whether the extraction of an MMN could infer that the recurrent policy is ignoring observations and only using memory. 3) Tracker. This MCE is identical to Amnesia, except that the ∆(m) values can be larger than 1. This requires an optimal policy to pay attention to observations when c t = 0 and use memory to keep track of the current mode and mode count. This is the most general instance of the environment and can in difficult problems when the number of modes and their life-spans grow. In all above instances, we used M = 4. We use'A' and'R' to denote accept and reject states, respectively. Other than Grammar 6, all machines are 100% accurate.

Extracting a finite state machine from a recurrent neural network via quantization for the purpose of interpretability with experiments on Atari.

Querying a black-box neural network reveals a lot of information about it; we propose novel "metamodels" for effectively extracting information from a black box.


Question: Represent the scientific article to find a short TLDR

Question: Building upon the recent success of deep reinforcement learning methods, we investigate the possibility of on-policy reinforcement learning improvement by reusing the data from several consecutive policies. On-policy methods bring many benefits, such as ability to evaluate each ing policy. However, they usually discard all the information about the policies which existed before. In this work, we propose adaptation of the replay buffer concept, borrowed from the off-policy learning setting, to the on-policy algorithms. To achieve this, the proposed algorithm generalises the Q-, value and advantage functions for data from multiple policies. The method uses trust region optimisation, while avoiding some of the common problems of the algorithms such as TRPO or ACKTR: it uses hyperparameters to replace the trust region selection heuristics, as well as the trainable covariance matrix instead of the fixed one. In many cases, the method not only improves the comparing to the state-of-the-art trust region on-policy learning algorithms such as ACKTR and TRPO, but also with respect to their off-policy counterpart DDPG. The past few years have been marked by active development of reinforcement learning methods. Although the mathematical foundations of reinforcement learning have been known long before BID23, starting from 2013, the novel deep learning techniques allowed to solve vision based discrete control tasks such as Atari 2600 games BID15 as well as continuous control problems BID12. Many of the leading state-of-the-art reinforcement learning methods share the actor-critic architecture BID5. Actorcritic methods separate the actor, providing a policy, and the critic, providing an approximation for the expected discounted cumulative reward or some derived quantities such as advantage functions BID2. However, despite improvements, state-of-the-art reinforcement learning still suffers from poor sample efficiency and extensive parameterisation. For most real-world applications, in contrast to simulations, there is a need to learn in real time and over a limited training period, while minimising any risk that would cause damage to the actor or the environment. Reinforcement learning algorithms can be divided into two groups: on-policy and off-policy learning. On-policy approaches (e. g., SARSA BID18, ACKTR BID28) evaluate the target policy by assuming that future actions will be chosen according to it, hence the exploration strategy must be incorporated as a part of the policy. Off-policy methods (e. g., Qlearning BID27, DDPG BID12) separate the exploration strategy, which modifies the policy to explore different states, from the target policy. The off-policy methods commonly use the concept of replay buffers to memorise the outcomes of the previous policies and therefore exploit the information accumulated through the previous iterations BID13. BID15 combined this experience replay mechanism with Deep Q-Networks (DQN), demonstrating end-to-end learning on Atari 2600 games. One limitation of DQN is that it can only operate on discrete action spaces. BID12 proposed an extension of DQN to handle continuous action spaces based on the Deep Deterministic Policy Gradient (DDPG). There, exponential smoothing of the target actor and critic weights has been introduced to ensure stability of the rewards and critic predictions over the subsequent iterations. In order to improve the variance of policy gradients, BID20 proposed a Generalised Advantage Function. combined this advantage function learning with a parallelisation of exploration using differently trained actors in their Asynchronous Advantage Actor Critic model (A3C); however, BID26 demonstrated that such parallelisation may also have negative impact on sample efficiency. Although some work has been performed on improvement of exploratory strategies for reinforcement learning BID8, but it still does not solve the fundamental restriction of inability to evaluate the actual policy, neither it removes the necessity to provide a separate exploratory strategy as a separate part of the method. In contrast to those, state-of-the-art on-policy methods have many attractive properties: they are able to evaluate exactly the ing policy with no need to provide a separate exploration strategy. However, they suffer from poor sample efficiency, to a larger extent than off-policy reinforcement learning. TRPO method BID19 has introduced trust region policy optimisation to explicitly control the speed of policy evolution of Gaussian policies over time, expressed in a form of Kullback-Leibler divergence, during the training process. Nevertheless, the original TRPO method suffered from poor sample efficiency in comparison to off-policy methods such as DDPG. One way to solve this issue is by replacing the first order gradient descent methods, standard for deep learning, with second order natural gradient . BID28 used a Kroneckerfactored Approximate Curvature (K-FAC) optimiser BID14 in their ACKTR method. PPO method proposes a number of modifications to the TRPO scheme, including changing the objective function formulation and clipping the gradients. BID26 proposed another approach in their ACER algorithm: in this method, the target network is still maintained in the off-policy way, similar to DDPG BID12, while the trust region constraint is built upon the difference between the current and the target network. Related to our approach, recently a group of methods has appeared in an attempt to get the benefits of both groups of methods. BID7 propose interpolated policy gradient, which uses the weighted sum of both stochastic BID24 and deterministic policy gradient BID22. BID17 propose an off-policy trust region method, Trust-PCL, which exploits off-policy data within the trust regions optimisation framework, while maintaining stability of optimisation by using relative entropy regularisation. While it is a common practice to use replay buffers for the off-policy reinforcement learning, their existing concept is not used in combination with the existing on-policy scenarios, which in discarding all policies but the last. Furthermore, many on-policy methods, such as TRPO BID19, rely on stochastic policy gradient BID24, which is restricted by stationarity assumptions, in a contrast to those based on deterministic policy gradient BID22, like DDPG BID12. In this article, we describe a novel reinforcement learning algorithm, allowing the joint use of replay buffers with trust region optimisation and leading to sample efficiency improvement. The contributions of the paper are given as follows:1. a reinforcement learning method, enabling replay buffer concept along with on-policy data; 2. theoretical insights into the replay buffer usage within the on-policy setting are discussed; 3. we show that, unlike the state-of-the-art methods as ACKTR BID28, PPO and TRPO BID19, a single non-adaptive set of hyperparameters such as the trust region radius is sufficient for achieving better performance on a number of reinforcement learning tasks. As we are committed to make sure the experiments in our paper are repeatable and to further ensure their acceptance by the community, we will release our source code shortly after the publication. Consider an agent, interacting with the environment by responding to the states s t, t ≥ 0, from the state space S, which are assumed to be also the observations, with actions a t from the action space A chosen by the policy distribution π θ (·|s t), where θ are the parameters of the policy. The initial state distribution is ρ 0: S → R. Every time the agent produces an action, the environment gives back a reward r(s t, a t) ∈ R, which serves as a feedback on how good the action choice was and switches to the next state s t+1 according to the transitional probability P (s t+1 |s t, a t). Altogether, it can be formalised as an infinite horizon γ-discounted Markov Decision Process (S, A, P, r, ρ 0, γ), γ ∈ BID28 BID19. The expected discounted return BID3 ) is defined as per BID19: DISPLAYFORM0 The advantage function A π BID2, the value function V π and the Q-function Q π are defined as per; BID19: DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 In all above definitions s 0 ∼ ρ 0 (s 0), a t ∼ π(a t |s t), s t+1 ∼ P (s t+1 |s t, a t), and the policy π = π θ is defined by its parameters θ. A straightforward approach for learning a policy is to perform unconstrained maximisation ρ(π θ) with respect to the policy parameters θ. However, for the state-of-the-art iterative gradient-based optimisation methods, this approach would lead to unpredictable and uncontrolled changes in the policy, which would impede efficient exploration. Furthermore, in practice the exact values of ρ(π θ) are unknown, and the quality of its estimates depends on approximators which tend to be correct only in the vicinity of parameters of observed policies. (2015a) mention that in practice the algorithm's convergence rate and the complexity of maximum KL divergence computations makes it impractical to apply this method directly. Therefore, they proposed to replace the unconstrained optimisation with a similar constrained optimisation problem, the Trust Region Policy Optimisation (TRPO) problem: DISPLAYFORM0 where D KL is the KL divergence between the old and the new policy π θ old and π θ respectively, and δ is the trust region radius. Despite this improvement, it needs some further enhancements to solve this problem efficiently, as we will elaborate in the next section. Many of the state-of-the-art trust region based methods, including TRPO BID19 and ACKTR BID28, use second order natural gradient based actor-critic optimisation BID1 BID10. The motivation behind it is to eliminate the issue that gradient descent loss, calculated as the Euclidean norm, is dependent on parametrisation. For this purpose, the Fisher information matrix is used, which is, as it follows from and BID10, normalises per-parameter changes in the objective function. In the context of actor-critic optimisation it can be written as BID28 BID10, where p(τ) is the trajectory distribution p(s 0) T t=0 π(a t |s t)p(s t+1 |s t, a t): DISPLAYFORM0 However, the computation of the Fisher matrix is intractable in practice due to the large number of parameters involved; therefore, there is a need to resort to approximations, such as the Kroneckerfactored approximate curvature (K-FAC) method BID14, which has been first proposed for ACKTR in BID28. In the proposed method, as it is detailed in Algorithm 1, this optimisation method is used for optimising the policy. While the original trust regions optimisation method can only use the samples from the very last policy, discarding the potentially useful information from the previous ones, we make use of samples over several consecutive policies. The rest of the section contains definition of the proposed replay buffer concept adaptation, and then formulation and discussion of the proposed algorithm.3.1 USAGE OF REPLAY BUFFERS BID15 suggested to use replay buffers for DQN to improve stability of learning, which then has been extended to other off-policy methods such as DDPG BID12. The concept has not been applied to on-policy methods like TRPO BID19 or ACKTR BID28, which do not use of previous data generated by other policies. Although based on trust regions optimisation, ACER BID26 uses replay buffers for its off-policy part. In this paper, we propose a different concept of the replay buffers, which combines the on-policy data with data from several previous policies, to avoid the restrictions of policy distribution stationarity for stochastic policy gradient BID24. Such replay buffers are used for storing simulations from several policies at the same time, which are then utilised in the method, built upon generalised value and advantage functions, accommodating data from these policies. The following definitions are necessary for the formalisation of the proposed algorithm and theorems. We define a generalised Q-function for multiple policies {π 1, . . ., π n, . . ., π N} as DISPLAYFORM0 DISPLAYFORM1 |s n t, a n t ), a n t ∼ π n (a n t |s n t). We also define the generalised value function and the generalised advantage function as DISPLAYFORM2 DISPLAYFORM3 DISPLAYFORM4 P (s → x, k, π), as in BID24, is the probability of transition from the state s to the state x in k steps using policy π. Theorem 1. For the set of policies {π 1, . . ., π N} the following equality will be true for the gradient: DISPLAYFORM5 where θ are the joint parameters of all policies {π n} and b πn (s) is a bias function for the policy. The proof of Theorem 1 is given in Appendix B. Applying a particular case of the bias function b πn (s) = −V π (s) and using the likelihood ratio transformation, one can get DISPLAYFORM6 The proposed approach is summarised in Algorithm 1. The replay buffer R p contains data collected from several subsequent policies. The size of this buffer is RBP CAPACITY., increase i by the total number of timesteps in all new paths. {Stage 2} Put recorded paths into the policy paths replay buffer R p ← P. {Stage 3} For every path in R p compute the targets for the value function regression using equation FORMULA0. Ψ = Update the value function estimator parameters {Stage 4} For every path in R p, estimate the advantage function using Equation.{Stage 5} Update parameters of the policy θ for N ITER PL UPDATE iterations using the gradient from Equation FORMULA1 and a barrier function defined in Equation. end while During Stage 1, the data are collected for every path until the termination state is received, but at least TIMESTEPS PER BATCH steps in total for all paths. The policy actions are assumed to be sampled from the Gaussian distribution, with the mean values predicted by the policy estimator along with the covariance matrix diagonal. The covariance matrix output was inspired, although the idea is different, by the EPG paper BID4.At Stage 2, the obtained data for every policy are saved in the policy replay buffer R p.At Stage 3, the regression of the value function is trained using Adam optimiser BID11 with step size VF STEP SIZE for N ITER VF UPDATE iterations. For this regression, the sum-of-squares loss function is used. The value function target values are computed for every state s t for every policy in the replay buffer using the actual sampled policy values, where t max is the maximum policy step index:V DISPLAYFORM0 During Stage 4, we perform the advantage function estimation. BID20 proposed the Generalised Advantage Estimator for the advantage function A π (s t, a t) as follows: DISPLAYFORM1 DISPLAYFORM2 Here k > 0 is a cut-off value, defined by the length of the sequence of occured states and actions within the MDP, λ ∈ is an estimator parameter, andṼ π (s t) is the approximation for the value function V π (s t), with the approximation targets defined in Equation. As proved in BID20, after rearrangement this would in the generalised advantage function estimator DISPLAYFORM3 For the proposed advantage function (see Equation 11), the estimator could be defined similarly to BID20 as DISPLAYFORM4 DISPLAYFORM5 However, it would mean the estimation of multiple value functions, which diminishes the replay buffer idea. To avoid it, we modify this estimator for the proposed advantage function as DISPLAYFORM6 Theorem 2. The difference between the estimators FORMULA1 and FORMULA1 is DISPLAYFORM7 The proof of Theorem 2 is given in Appendix C. It shows that the difference between two estimators is dependent of the difference in the conventional and the generalised value functions; given the continuous value function approximator it reveals that the closer are the policies, within a few trust regions radii, the smaller will be the bias. During Stage 5, the policy function is approximated, using the K-FAC optimiser BID14 with the constant step size PL STEP SIZE. As one can see from the description, and differently from ACKTR, we do not use any adaptation of the trust region radius and/or optimisation algorithm parameters. Also, the output parameters include the diagonal of the (diagonal) policy covariance matrix. The elements of the covariance matrix, for the purpose of efficient optimisation, are restricted to universal minimum and maximum values MIN COV EL and MAX COV EL.As an extention from BID20 and following Theorem 1 with the substitution of likelihood ratio, the policy gradient estimation is defined as DISPLAYFORM8 To practically implement this gradient, we substitute the parameters θ π, derived from the latest policy for the replay buffer, instead of joint θ parameters assuming that the parameters would not deviate far from each other due to the trust region restrictions; it is still possible to calculate the estimation ofÃ πn (s n t, a n t) for each policy using Equation as these policies are observed. For the constrained optimisation we add the linear barrier function to the function ρ(θ): DISPLAYFORM9 where α > 0 is a barrier function parameter and θ old are the parameters of the policy on the previous iteration. Besides of removing the necessity of heuristical estimation of the optimisation parameters, it also conforms with the theoretical prepositions shown in and, while our approach is proposed independently, pursues the similar ideas of using actual constrained optimisation method instead of changing the gradient step size parameters as per BID19.The networks' architectures correspond to OpenAI Baselines ACKTR implementation,which has been implemented by the ACKTR authors BID28. The only departure from the proposed architecture is the diagonal covariance matrix outputs, which are present, in addition to the mean output, in the policy network. In order to provide the experimental evidence for the method, we have compared it with the on-policy ACKTR BID28, PPO and TRPO BID19 ) methods, as well as with the off-policy DDPG BID12 method on the MuJoCo BID25 robotic simulations. The technical implementation is described in Appendix A. BID25: comparison with TRPO BID19, ACKTR BID28 and PPO BID25: comparison between the proposed algorithm and DDPG BID12 In contrast to those methods, the method shows that the adaptive values for trust region radius can be advantageously replaced by a fixed value in a combination with the trainable policy distribution covariance matrix, thus reducing the number of necessary hyperparameters. The for ACKTR for the tasks HumanoidStandup, Striker and Thrower are not included as the baseline ACKTR implementation diverged at the first iterations with the predefined parameterisation. PPO are obtained from baselines implementation PPO1. Figure 2 compares for different replay buffer sizes; the size of the replay buffers reflects the number of policies in it and not actions (i.e. buffer size 3 means data from three successive policies in the replay buffer). We see that in most of the cases, the use of replay buffers show performance improvement against those with replay buffer size 1 (i.e., no replay buffer with only the current policy used for policy gradient); substantial improvements can be seen for HumanoidStandup task. Figure 3 shows the performance comparison with the DDPG method BID12. In all the tasks except HalfCheetah and Humanoid, the proposed method outperforms DDPG. For HalfCheetah, the versions with a replay buffer marginally overcomes the one without. It is also remarkable that the method demonstrates stable performance on the tasks HumanoidStandup, Pusher, Striker and Thrower, on which DDPG failed (and these tasks were not included into the DDPG article). The paper combines replay buffers and on-policy data for reinforcement learning. Experimental on various tasks from the MuJoCo suite BID25 show significant improvements compared to the state of the art. Moreover, we proposed a replacement of the heuristically calculated trust region parameters, to a single fixed hyperparameter, which also reduces the computational expences, and a trainable diagonal covariance matrix. The proposed approach opens the door to using a combination of replay buffers and trust regions for reinforcement learning problems. While it is formulated for continuous tasks, it is possible to reuse the same ideas for discrete reinforcement learning tasks, such as ATARI games. The parameters of Algorithm 1, used in the experiment, are given in Table 1; the parameters were initially set, where possible, to the ones taken from the state-of-the-art trust region approach implementation BID28, and then some of them have been changed based on the experimental evidence. As the underlying numerical optimisation algorithms are out of the scope of the paper, the parameters of K-FAC optimiser from have been used for the experiments; for the Adam algorithm BID11, the default parameters from Tensorflow BID0 implementation (β 1 = 0.9, β 2 = 0.999, = 1 · 10 −8) have been used. The method has been implemented in Python 3 using Tensorflow BID0 as an extension of the OpenAI baselines package. The neural network for the control experiments consists of two fully connected layers, containing 64 neurons each, following the OpenAI ACKTR network implementation Proof. Extending the derivation from Sutton et al. FORMULA1, one can see that: DISPLAYFORM0 Then, Proof. The difference between the two k-th estimators is given as DISPLAYFORM1 By substituting this into the GAE estimator difference one can obtain ∆Ã πn (s t, a t) = (1 − λ)(γ∆V 1 + λγ 2 ∆V 2 + λ 2 γ 3 ∆V 3 +... DISPLAYFORM2

Represent the short summary for finding the corresponding scientific article (long)

We investigate the theoretical and practical evidence of on-policy reinforcement learning improvement by reusing the data from several consecutive policies.

Represent the TLDR for finding the corresponding scientific article

Parameter space noise allows reinforcement learning algorithms to explore by perturbing parameters instead of actions, often leading to significantly improved exploration performance.


Question: represent

Question: Convolutional Neural Networks (CNN) have been successful in processing data signals that are uniformly sampled in the spatial domain (e.g., images). However, most data signals do not natively exist on a grid, and in the process of being sampled onto a uniform physical grid suffer significant aliasing error and information loss. Moreover, signals can exist in different topological structures as, for example, points, lines, surfaces and volumes. It has been challenging to analyze signals with mixed topologies (for example, point cloud with surface mesh). To this end, we develop mathematical formulations for Non-Uniform Fourier Transforms (NUFT) to directly, and optimally, sample nonuniform data signals of different topologies defined on a simplex mesh into the spectral domain with no spatial sampling error. The spectral transform is performed in the Euclidean space, which removes the translation ambiguity from works on the graph spectrum. Our representation has four distinct advantages: the process causes no spatial sampling error during initial sampling, the generality of this approach provides a unified framework for using CNNs to analyze signals of mixed topologies, it allows us to leverage state-of-the-art backbone CNN architectures for effective learning without having to design a particular architecture for a particular data structure in an ad-hoc fashion, and the representation allows weighted meshes where each element has a different weight (i.e., texture) indicating local properties. We achieve good on-par with state-of-the-art for 3D shape retrieval task, and new state-of-the-art for point cloud to surface reconstruction task. We present a unifying and novel geometry representation for utilizing Convolutional Neural Networks (CNNs) on geometries represented on weighted simplex meshes (including textured point clouds, line meshes, polygonal meshes, and tetrahedral meshes) which preserve maximal shape information based on the Fourier transformation. Most methods that leverage CNNs for shape learning preprocess these shapes into uniform-grid based 2D images (rendered multiview images) or 3D images (binary voxel or Signed Distance Function (SDF)). However, rendered 2D images do not preserve the 3D topologies of the original shapes due to occlusions and the loss of the third spatial dimension. Binary voxels and SDF representations under low resolution suffer big aliasing errors and under high resolution become memory inefficient. Loss of information in the input bottlenecks the effectiveness of the downstream learning process. Moreover, it is not clear how a weighted mesh where each element is weighted by a different scalar or vector (i.e., texture) can be represented by binary voxels and SDF. Mesh and graph based CNNs perform learning on the manifold physical space or graph spectrum, but generality across topologies remains challenging. In contrast to methods that operate on uniform sampling based representations such as voxel-based and view-based models, which suffer significant representational errors, we use analytical integration to precisely sample in the spectral domain to avoid sample aliasing errors. Unlike graph spectrum based methods, our method naturally generalize across input data structures of varied topologies. Using our representation, CNNs can be directly applied in the corresponding physical domain obtainable by inverse Fast Fourier Transform (FFT) due to the equivalence of the spectral and physical domains. This allows for the use of powerful uniform Cartesian grid based CNN backbone architectures (such as DLA BID40, ResNet ) for the learning task on arbitrary geometrical signals. Although the signal is defined on a simplex mesh, it is treated as a signal in the Euclidean space instead of on a graph, differentiating our framework from graph-based spectral learning techniques which have significant difficulties generalizing across topologies and unable to utilize state-of-the-art Cartesian CNNs. We evaluate the effectiveness of our shape representation for deep learning tasks with three experiments: a controlled MNIST toy example, the 3D shape retrieval task, and a more challenging 3D point cloud to surface reconstruction task. In a series of evaluations on different tasks, we show the unique advantages of this representation, and good potential for its application in a wider range of shape learning problems. We achieve state-of-the-art performance among non-pre-trained models for the shape retrieval task, and beat state-of-the-art models for the surface reconstruction task. The key contributions of our work are as follows:• We develop mathematical formulations for performing Fourier Transforms of signals defined on a simplex mesh, which generalizes and extends to all geometries in all dimensions. (Sec.3)• We analytically show that our approach computes the frequency domain representation precisely, leading to much lower overall representational errors. (Sec. 3)• We empirically show that our representation preserves maximal shape information compared to commonly used binary voxel and SDF representations. (Sec. 4.1)• We show that deep learning models using CNNs in conjunction with our shape representation achieves state-of-the-art performance across a range of shape-learning tasks including shape retrieval (Sec. 4.2) and point to surface reconstruction (Sec. 4.3) DISPLAYFORM0 Index of the n-th element among a total of N elements Ω j n Domain of n-th element of order j x Cartesian space coordinate vector. DISPLAYFORM1 Imaginary number unit Shape learning involves the learning of a mapping from input geometrical signals to desired output quantities. The representation of geometrical signals is key to the learning process, since on the one hand the representation determines the learning architectures, and, on the other hand, the richness of information preserved by the representation acts as a bottleneck to the downstream learning process. While data representation has not been an open issue for 2D image learning, it is far from being agreed upon in the existing literature for 3D shape learning. The varied shape representations used in 3D machine learning are generally classified as multiview images BID28 BID26 BID2, volumetric voxels BID38 BID14 BID37 BID0, point clouds BID19 BID24 BID36, polygonal meshes BID3 BID34 BID15 BID12, shape primitives BID42 BID39, and hybrid representations (Dai & Nießner, 2018).Our proposed representation is closest to volumetric voxel representation, since the inverse Fourier Transform of the spectral signal in physical domain is a uniform grid implicit representation of the shape. However, binary voxel representation suffers from significant aliasing errors during the uniform sampling step in the Cartesian space BID16. Using boolean values for de facto floating point numbers during CNN training is a waste of information processing power. Also, the primitive-in-cell test for binarization requires arbitrary grouping in cases such as having multiple points or planes in the same cell BID32. Signed Distance Function (SDF) or Truncated Signed Distance Function (TSDF) ) provides localization for the shape boundary, but is still constrained to linear surface localization due to the linear interpolation process for recovering surfaces from grids. Our proposed representation under Fourier basis can find nonlinear surface boundaries, achieving subgrid-scale accuracy (See FIG1). Cartesian CNNs are the most ubiquitous and mature type of learning architecture in Computer Vision. It has been thoroughly studied in a range of problems, including image recognition BID5 BID27 ), object detection BID21 BID21, and image segmentation BID11 ). In the spirit of 2D image-based deep learning, Cartesian CNNs have been widely used in shape learning models that adopted multiview shape representation BID28 BID26 BID2 BID29 BID17 BID33. Also, due to its straightforward and analogous extension to 3D by swapping 2D convolutional kernels with 3D counterparts, Cartesian CNNs have also been widely adopted in shape learning models using volumetric representations BID38 BID14 BID37 BID0. However, the dense nature of the operations makes it inefficient for sparse 3D shape signals. To this end, improvements to Cartesian CNNs have been made using space partitioning tree structures, such as Quadtree in 2D and Octree in 3D BID2 BID31. These Cartesian CNNs can leverage backbone CNN architectures being developed in related computer vision problems and thus achieve good performance. Since the physical domain representation in this study is based on Cartesian uniform grids, we directly use Cartesian CNNs. Graph CNNs utilize input graph structure for performing graph convolutions. They have been developed to engage with general graph structured data BID1; ). BID39 used spectral CNNs with the eigenfunctions of the graph Laplacian as a basis. However, the generality of this approach across topologies and geometris is still challenging since consistency in eigenfunction basis is implied. Specially Designed Neural Networks have been used to perform learning on unconventional data structures. For example, Qi et al. (2017a) designed a Neural Network architecture for points that achieves invariances using global pooling, with follow-up work (b) using CNNsinspired hiearchical structures for more efficient learning. BID13 performed convolution directly on the shape manifold and Cohen et al. FORMULA2 designed CNNs for the spherical domain and used it for 3D shapes by projecting the shapes onto the bounding sphere. The original work on analytical expressions for Fourier transforms of 2D polygonal shape functions is given by BID6. Improved and simpler calculation methods have been suggested in. A 3D formulation is proposed by BID41. Theoretical analyses have been performed for the Fourier analysis of simplex domains BID30 BID7 ) and BID23 designed approximation methods for Fast Fourier Transform of polynomial functions defined on simplices. describe shape with elliptic Fourier descriptors for level set-based segmentation and tracking. There has also been a substantial literature on fast non-uniform Fourier transform methods for discretely sampled signal . However we are the first to provide a simple general expression for a j-simplex mesh, an algorithm to perform the transformation, and illustrations of their applicability to deep learning problems. Almost all discrete geometric signals can be abstracted into weighted simplicial complexes. A simplicial complex is a set composed of points, line segments, triangles, and their d-dimensional counterparts. We call a simplicial complex consisting solely of j-simplices as a homogeneous simplicial j-complex, or a j-simplex mesh. Most popular geometric representations that the research community is familiar with are simplex meshes. For example, the point cloud is a 0-simplex mesh, the triangular mesh is a 2-simplex mesh, and the tetrahedral mesh is a 3-simplex mesh. A j-simplex mesh consists of a set of individual elements, each being a j-simplex. If signal is non-uniformly distributed over the simplex, we can define a piecewise constant j-simplex function over the j-simplex mesh. We call this a weighted simplex mesh. Each element has a distinct signal density. J-simplex function For the n-th j-simplex with domain Ω j n, we define a density function f j n (x). For example, for some Computer Vision and Graphics applications, a three-component density value can be defined on each element of a triangular mesh for its RGB color content. For scientific applications, signal density can be viewed as mass or charge or other physical quantity. DISPLAYFORM0 The piecewise-constant j-simplex function consisting of N simplices is therefore defined as the superposition of the element-wise simplex function. Using the linearity of the integral in the Fourier transform, we can decompose the Fourier transform of the density function on the j-simplex mesh to be a weighted sum of the Fourier transform on individual j-simplices. DISPLAYFORM1 We present a general formula for performing the Fourier transform of signal over a single j-simplex. We provide detailed derivation and proof for j = 0, 1, 2, 3 in the supplemental material. DISPLAYFORM0 We define γ j n to be the content distortion factor, which is the ratio of content between the simplex over the domain Ω j n and the unit orthogonal j-simplex. Content is the j-dimensional analogy of the 3-dimensional volume. The unit orthogonal j-simplex is defined as a j-simplex with one vertex at the Cartesian origin and all edges adjacent to the origin vertex to be pairwise orthogonal and to have unit length. Therefore from Equation the final general expression for computing the Fourier transform of a signal defined on a weighted simplex mesh is: DISPLAYFORM1 For computing the simplex content, we use the Cayley-Menger Determinant for a general expression: DISPLAYFORM2 For the matrixB j n, each entry d 2 mn represents the squared distance between nodes m and n. The matrix is of size (j + 2) × (j + 2) and is symmetrical. Since the unit orthogonal simplex has content of C j I = 1/j!, the content distortion factor γ j n can be calculated by: DISPLAYFORM3 Auxiliary Node Method: Equation provides a mean of computing the Fourier transform of a simplex with uniform signal density. However, how do we compute the Fourier transform of polytopes (i.e., polygons in 2D, polyhedra in 3D) with uniform signal density efficiently? Here, we introduce the auxiliary node method (AuxNode) that utilizes signed content for efficient computing. We show that for a solid j-polytope represented by a watertight (j − 1)-simplex mesh, we can compute the Fourier transform of the entire polytope by traversing each of the elements in its boundary (j − 1)-simplex mesh exactly once BID41 ).The auxiliary node method performs Fourier transform over the the signed content bounded by an auxilliary node (a convenient choice being the origin of the Cartesian coordinate system) and each (j − 1)-simplex on the boundary mesh. This forms an auxiliary j-simplex: DISPLAYFORM4 where N is the number of (j − 1)-simplices in the boundary mesh. However due to the overlapping of these auxiliary j-simplices, we need a means of computing the sign of the transform for the overlapping regions to cancel out. Equation provides a general expression for computing the unsigned transform for a single j-simplex. It is trivial to show that since the ordering of the nodes does not affect the determinant in Equation FORMULA6, it gives the unsigned content value. Therefore, to compute the Fourier transform of uniform signals in j-polytopes represented by its watertight (j − 1)-simplex mesh using the auxiliary node method, we modify Equation: s n γ j n is the signed content distortion factor for the n th auxiliary j-simplex where s n ∈ {−1, 1}. For practical purposes, assume that the auxiliary j-simplex is in R d where d = j. We can compute the signed content distortion factor using the determinant of the Jacobian matrix for parameterizing the auxiliary simplex to a unit orthogonal simplex: DISPLAYFORM5 DISPLAYFORM6 (a) Experiment setup 4 8 12 16 4 8 12 16 4 8 12 16 4 8 12 16 20 24 28 20 24 28 20 24 28Since this method requires the boundary simplices to be oriented, the right-hand rule can be used to infer the correct orientation of the boundary element. For 2D polygons, it requires that the watertight boundary line mesh be oriented in a counter-clockwise fashion. For 3D polytopes, it requires that the face normals of boundary triangles ing from the right-hand rule be consistently outward-facing. Algorithmic implementation: Several efficiencies can be exploited to achieve fast runtime and high robustness. First, since the general expression Equation involves division and is vulnerable to division-by-zero errors (that is not a singularity since it can be eliminated by taking the limit), add a minor random noise to vertex coordinates as well as to the k = 0 frequency mode for robustness. Second, to avoid repeated computation, the value should be cached in memory and reused, but caching all σ and e −iσ values for all nodes and frequencies is infeasible for large mesh and/or high resolution output, thus the Breadth-First-Search (BFS) algorithm should be used to traverse the vertices for efficient memory management. In this section, we will discuss the experiment setup, and we defer the details of our model architecture and training process to the supplementary material since it is not the focus of this paper. We use the MNIST experiment as a first example to show that shape information in the input significantly affects the efficacy of the downstream learning process. Since the scope of this research is on efficiently learning from nonuniform mesh-based representations, we compare our method with the state of the art in a slightly different scenario by treating MNIST characters as polygons. We choose this experiment as a first toy example since it is easy to control the learning architecture and input resolution to highlight the effects of shape representation on deep learning performance. We pre-process the original MNIST raw pixel images into polygons, which are represented by watertight line meshes on their boundaries. The polygonized digits are converted into (n × n) binary pixel images and into distance functions by uniformly sampling the polygon at the (n × n) sample locations. For NUFT, we first compute the lowest (n × n) Fourier modes for the polygonal shape function and then use an inverse Fourier transform to acquire the physical domain image. We also compare the with the raw pixel image downsampled to different resolutions, which serves as an oracle for information ceiling. Then we perform the standard MNIST classification experiment on these representations with varying resolution n and with the same network architecture. The experiment are presented in FIG2. It is evident that binary pixel representation suffers the most information loss, especially at low resolutions, which leads to rapidly declining performance. Using the distance function representation preserves more information, but underperforms our NUFT representation. Due to its efficient information compression in the Spectral domain, NUFT even outperforms the downsampled raw pixel image at low resolutions. Shape retrieval is a classic task in 3D shape learning. SHREC17 BID24 which is based on the ShapeNet55 Core dataset serves as a compelling benchmark for 3D shape retrieval performance. We compare the retrieval performance of our model utilizing the NUFT-surface (NUFT-S) and NUFT-volume (NUFT-V) at various resolutions against state-of-the-art shape retrieval algorithms to illustrate its potential in a range of 3D learning problems. We performed the experiments on the normalized dataset in SHREC17. Our model utilizes 3D DLA BID40 as a backbone architecture. Results: Results from the experiment are tabulated in TAB1. For the shape retrieval task, most state-of-the-art methods are based on multi-view representation that utilize a 2D CNN pretrained on additional 2D datasets such as ImageNet. We have achieved on par with, though not better than, state-of-the-art pretrained 2D models. We outperform other models in this benchmark that have not been pre-trained on additional data. We also compared NUFT-volume and NUFT-surface representations in FIG3. Interestingly NUFT-volume and NUFT-surface representations lead to similar performances under the same resolution. Table 3: Quantitative comparison of surface reconstruction methods. The metrics above are distances, hence lower value represents better performance. Best is highlighted in bold. We achieve better than the current stateof-the-art method by a sizable margin, and our are robust to noise in the input. We further illustrate the advantages of our representation with a unique yet important task in computational geometry that has been challenging to address with conventional deep learning techniques: surface reconstruction from point cloud. The task is challenging for deep learning in two aspects: First, it requires input and output signals of different topologies and structures (i.e., input being a point cloud and output being a surface). Second, it requires precise localization of the signal in 3D space. Using our NUFT-point representation as input and NUFT-surface representation as output, we can frame the task as a 3D image-to-image translation problem, which is easy to address using analogous 2D techniques in 3D. We use the U-Net BID22 architecture and train it with a single L 2 loss between output and ground truth NUFT-surface representation. Experiment Setup: We train and test our model using shapes from three categories of ShapeNet55 Core dataset (car, sofa, bottle). We trained our model individually for these three categories. As a pre-processing step we removed faces not visible from the exterior and simplified the mesh for faster conversion to NUFT representation. For the input point cloud, we performed uniform point sampling of 3000 points from each mesh and converted the points into the NUFT-point representation (128 3). Then, we converted the triangular mesh into NUFT-surface representation (128 3). At test time, we post-process the output NUFT-surface implicit function by using the marching cubes algorithm to extract 0.5 contours. Since the extracted mesh has thickness, we further shrink the mesh by moving vertices to positions with higher density values while preserving the rigidity of each face. Last but not least, we qualitatively and quantitatively compare the performance by showing the reconstructed mesh against from the traditional Poisson Surface Reconstruction (PSR) method BID4 at various tree depths (5 and 8) and the Deep Marching Cubes (DMC) algorithm BID9. For quantitative comparison, we follow the literature BID25 and use Chamfer distance, Accuracy and Completeness as the metrics for comparison. For comparison with BID9, we also test the model with noisy inputs (Gaussian of sigma 0.15 voxel-length under 32 resolution), computed distance metrics after normalizing the models to the range of. Table 3 for quantitative comparisons with competing algorithms on the same task, and Figures 5 and 6 for visual comparisons. GT stands for Ground Truth. We achieve new state-of-the-art in the point to surface reconstruction task, due to the good localization properties of the NUFT representations and its flexibility across geometry topologies. We present a general representation for multidimensional signals defined on simplicial complexes that is versatile across geometrical deep learning tasks and maximizes the preservation of shape information. We develop a set of mathematical formulations and algorithmic tools to perform the transformations efficiently. Last but not least, we illustrate the effectiveness of the NUFT representation with a well-controlled example (MNIST polygon), a classic 3D task (shape retrieval) and a difficult and mostly unexplored task by deep learning (point to surface reconstruction), achieving new state-of-the-art performance in the last task. In , we offer an alternative representation for performing CNN based learning on geometrical signals that shows great potential in various 3D tasks, especially tasks involving mixed-topology signals. Without loss of generality assume that the j-simplex is defined in R d space where d ≥ j, since it is not possible to define a j-simplex in a space with dimensions lower than j. For most cases below (except j = 0) we will parameterize the original simplex domain to a unit orthogonal simplex in R j (as shown in Figure 7). Denote the original coordinate system in R d as x and the new coordinate system in the parametric R j space as p. Choose the following parameterization scheme: DISPLAYFORM0 By performing the Fourier transform integral in the parametric space and restoring the by the content distortion factor γ j n we can get equivalent as the Fourier transform on the original simplex domain. Content is the generalization of volumes in arbitrary dimensions (i.e. unity for points, length for lines, area for triangles, volume for tetrahedron). Content distortion factor γ j n is the ratio between the content of the original simplex and the content of the unit orthogonal simplex in parametric space. The content is signed if switching any pair of nodes in the simplex changes the sign of the content, and it is unsigned otherwise. See subsection 3.2 for means of computing the content and the content distortion factor. Points have spatial position x but no size (length, area, volume), hence it can be mathematically modelled as a delta function. The delta function (or Dirac delta function) has point mass as a function equal to zero everywhere except for zero and its integral over entire real space is one, i.e.: DISPLAYFORM0 For unit point mass at location x n: DISPLAYFORM1 Indeed, for 0-simplex, we have recovered the definition of the Discrete Fourier Transform (DFT). For a line with vertices at location x 1, x 2 ∈ R d, by parameterizing it onto a unit line, we get: DISPLAYFORM0 = iγ Figure 7: Schematic of example for 2-simplex. Original j-simplex is parameterized to a unit orthogonal j-simplex in R j space for performing the integration. Parameterization incurs a content distortion factor γ j n which is the ratio between the original simplex and the unit-orthogonal simplex in parametric space. DISPLAYFORM1 For a triangle with vertices x 1, x 2, x 3 ∈ R d, parameterization onto a unit orthogonal triangle gives: DISPLAYFORM0 = γ 2 n e −iω·x3 1 0 dp e −ipω·(x1−x3) DISPLAYFORM1 A.4 TETRAHEDRON: 3-SIMPLEX For a tetrahedron with vertices x 1, x 2, x 3, x 4 ∈ R d, parameterization onto a unit orthogonal tetrahedron gives: DISPLAYFORM2 Besides evaluating and comparing the shape representation schemes in the context of machine learning problems, we evaluate the different representation schemes in terms of its geometrical shape information. To evaluate geometrical shape information, we first convert the original polytopal shapes into the corresponding representations and then reconstruct the shape from these representations using interpolation based upsampling followed by contouring methods. Finally, we use the mesh Intersection over Union (mesh-IoU) metric between the original and constructed mesh to quantify geometrical shape information. Mesh boolean and volume computation for 2D polygons and 3D triangular mesh can be efficiently performed with standard computational geometry methods. In three dimensions contouring and mesh extraction can be performed using marching cubes algorithm for mesh reconstruction. For binarized representation, we perform bilinear upsampling (which does not affect the final ) followed by 0.5-contouring. For our NUFT representation, we perform spectral domain upsampling (which corresponds to zero-padding of higher modes in spectral domain), followed by 0. Training Details: We train the model with batch size of 64, learning rate of 0.01 and learning rate step size of 10. We use the SGD optimizer with momentum of 0.9, weight decay of 1 × 10 −4 for 15 epochs. Model Architecture: We use DLA34 architecture with all 2D convolutions modified to be 3D convolutions. It consists of levels, with filter numers. Training Details We train the model with batch size of 64, learning rate of 1 × 10 −3, learning rate step size of 30. We use the Adam optimizer with momentum of 0.9, and weight decay of 1 × 10 −4 for 40 epochs. Model Architecture We use a modified 3D version of the U-Net architecture consisting of 4 down convolutions and 4 up-convolutions with skip layers. Number of filters for down convolutions are and double of that for up convolutions. Training Details We train the model using Adam optimizer with learning rate 3 × 10 −4 for 200 epochs. We use NUFT point representation as input and a single L 2 loss between output and ground truth (NUFT surface) to train the network. We train and evaluate the model at 128 3 resolution.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We use non-Euclidean Fourier Transformation of shapes defined by a simplicial complex for deep learning, achieving significantly better results than point-based sampling techiques used in current 3D learning literature.

Represent the short summary for finding the corresponding scientific article (long)

We present a new deep architecture, VarPSOM, and its extension to time series data, VarTPSOM,  which achieve superior clustering performance compared to current deep clustering methods on static and temporal data.