Question: Represent the scientific article to find a short TLDR

Question: Differentiable planning network architecture has shown to be powerful in solving transfer planning tasks while possesses a simple end-to-end training feature. Many great planning architectures that have been proposed later in literature are inspired by this design principle in which a recursive network architecture is applied to emulate backup operations of a value iteration algorithm. However existing frame-works can only learn and plan effectively on domains with a lattice structure, i.e. regular graphs embedded in a certain Euclidean space. In this paper, we propose a general planning network, called Graph-based Motion Planning Networks (GrMPN), that will be able to i) learn and plan on general irregular graphs, hence ii) render existing planning network architectures special cases. The proposed GrMPN framework is invariant to task graph permutation, i.e. graph isormophism. As a , GrMPN possesses the generalization strength and data-efficiency ability. We demonstrate the performance of the proposed GrMPN method against other baselines on three domains ranging from 2D mazes (regular graph), path planning on irregular graphs, and motion planning (an irregular graph of robot configurations). Reinforcement learning (RL) is a sub-field of machine learning that studies about how an agent makes sequential decision making to interact with an environment. These problems can in principle be formulated as Markov decision process (MDP). (Approximate) Dynamic programming methods such as value iteration or policy iterations are often used for policy optimization. These dynamic programming approaches can also be leveraged to handle learning, hence referred as model-based RL . Model-based RL requires an estimation of the environment model hence is computationally expensive, but it is shown to be very data-efficient. The second common RL paradigm is model-free which does not require a model estimation hence has a lower computation cost but less data-efficiency . With a recent marriage with deep learning, deep reinforcement learning (DRL) has achieved many remarkable successes on a wide variety of applications such as game , robotics, chemical synthesis , news recommendation etc. DRL methods also range from model-based (; a) to model-free approaches. On the other hand, transfer learning across tasks has long been desired because it is much more challenging in comparison to single-task learning. Recent work has proposed a very elegant idea that suggests to encode a differentiable planning module in a policy network architecture. This planning module can emulate the recursive operation of value iterations, called Value Iteration Networks (VIN). Using this network, the agent is able to evaluate multiple future planning steps for a given policy. The planning module is designed to base on a recursive application of convolutional neural networks (CNN) and max-pooling for value function updates. VIN not only allows policy optimization with more data-efficiency, but also enables transfer learning across problems with shared transition and reward structures. VIN has laid foundation for many later differentiable planning network architectures such as QMDP-Net , planning under uncertainty , Memory Augmented Control Network (MACN) , Predictron , planning networks etc. However, these approaches including VIN is limited to learning with regular environment structures, i.e. the transition function forms an underlying 2D lattice structure. Recent works have tried to mitigate this issue by resorting to graph neural networks. These work exploit geometric intuition in environments which have irregular structures such as generalized VIN , planning on relational domains , , automated planning for scheduling , etc. The common between these approaches are in the use of graph neural networks to process irregular data structures like graphs. Among these frameworks, only GVIN is able to emulate the value iteration algorithm on irregular graphs of arbitrary sizes, e.g. generalization to arbitrary graphs. GVIN has a differentiable policy network architecture which is very similar to VIN. GVIN can also have a zero-shot planning ability on unseen graphs. However, GVIN requires domain knowledge to design a graph convolution which might limit it to become a universal graph-based path planning framework. In this paper, we aim to demonstrate different formulations for value iteration networks on irregular graphs. These proposed formulations are based on different graph neural network models. These models are capable of learning optimal policies on general graphs where their transition and reward functions are not provided a priori and yet to be estimated. These models are known to be invariant to graph isomorphism, therefore they are able to have a generalization ability to graphs of different sizes and structures. As a , they enjoy the ability of zero-shot learning to plan. Specifically, it is known that Bellman equations are written as the form of message passing, therefore we propose using message passing neural networks (MPNN) to emulate the value iteration algorithm on graphs. We will show two most general formulations of graph-based value iteration network that are based on two general-purpose approaches in the MPNN family: Graph Networks (GN) and Graph Attention Networks (GAT) . In particular, our contributions are three-fold: • We develop a MPNN based path planning network (GrMPN) which can learn to plan on general graphs, e.g. regular and irregular graphs. GrMPN is an differentiable end-to-end planning network architecture trained via imitation learning. We implement GrMPN via two formulations that are based on GN and GAT. • GrMPN is a general graph-based value iteration network that will render existing graphbased planning algorithms special cases. GrMPN is invariant to graph isomorphism which enables transfer planning on graphs of different structure and size. • We will demonstrate the efficacy of GrMPN which achieves state of the art on various domains including 2D maze with regular graph structures, irregular graphs, and motion planning problems. We show that GrMPN outperforms existing approaches in terms of data-efficiency, performance and scalability. This section provides on Markov decision process (MDP), value iteration algorithm, value iteration networks (VIN) and graph neural networks (GNN). A MDP is defined as M = (S, A, P, R), where S and A represent state and action spaces. P defines a transition function P(s, a, s) = P (s |s, a), where s, s ∈ S, a ∈ A. A planning algorithm, e.g. dynamic programming , aims to find an optimal policy π: S → A so that a performance measure V π (s) = E (t γ t r t |s 0 = s) is maximized, for all state s ∈ S; where γ ∈ is a discount factor. The expectation is w.r.t stochasticity of P and R. Value iteration (VI) is one of dynamic programming algorithms that can plan on M. It starts by updating the value functions V (s), ∀s ∈ S iteratively via the Bellman backup operator T, V (k) = T V (k−1) as follows: where k is an update iteration index. T is applied iteratively until V (k) (·) converges to optimal values. It is proved that the operator T is a Lipschitz map with a factor of γ. In other words, as k → ∞, V (k) converges to a fixed-point value function V *. As a , the optimal policy π * can be computed as, π * (s) = arg max a∈A [R(s, a) + γ s P (s |s, a)V * (s)]. Q-value functions are also defined similarly as Q(s, a) = E (t γ t r t |s 0 = s, a 0 = a). In addition, we have the relation between V and Q value functions as V (s) = max a Q(s, a). For goal-oriented tasks, the reward function R(s, a) can be designed to receive low values at intermediate states and high values at goal states s *. Value iteration network Planning on large MDPs might be very computationally expensive, hence transfer-planning would be desirable, especially for tasks sharing similar structures of P and R. Value Iteration Networks (VIN) is an differentiable planning framework that can i) do transfer-planning for goal-oriented tasks with different goal states, ii) and learn the shared underlying MDP M between tasks, i.e. learning the transition P and reward R functions. Let's assume that we want to find an optimal plan on a MDP M with unknown P and R. VIN's policy network with embedded approximate reward and transition functionsR andP is trained endto-end through imitation learning. R andP are assumed to be from an unknown MDPM whose optimal policy can form useful features about the optimal policy in M. Based on observation feature φ(s) on state s, the relation between M andM is denoted asR = f R (φ(·)) andP = f P (φ(·)). More specifically, inputs are 2D images, e.g. of the m × m maze with start and goal states; outputs are optimal paths going from start to goal. VIN embeds value iteration as a recursive application of convolutions and max-pooling over the feature channels. VIN consists of a convolutional layer Q a ∈ m×m with |A| channels. The trainable parameters are W R a,i,j and W P a,i,j with |A| channels, which account for the reward and transition embeddings. The recursive process contains two following convolution and max-pooling operations, Q where the convolution operators on R.H.S in the first equation is written as: where i, j are cell index in the maze. VIN has later inspired many other differentiable planning algorithms. For example, VIN's idea can again be exploited for diffirentiable planning architectures for planning on partially observable environments such as QMDP-Net , , Memory Augmented Control Network (MACN) . A related differentiable planning network is also used in the Predictron framework where its core planning module aims to estimate a Markov reward process that can be rolled forward for many imagined planning steps. A notable extension of VIN is proposed by (b), called Gated path planning networks (GPPN), in which they use LSTM to replace the recursive VIN update, i.e. These algorithms show great success at path planning on many different grid-based navigation tasks in which the states are either fully or partially observable. However the underlying state space must assume regular lattices in order to exploit local connectivity through the help of convolution operations of CNN. This limits their applications to domains whose state spaces might be in the forms of irregular graphs. There is recent effort considering planning and reinforcement learning whose state transitions form a general graph. Niu et. al. propose such a graph-based model-based deep reinforcement learning framework that generalizes VIN to differentiable planning on graphs, called generalized value iteration (GVIN). GVIN takes a graph with a certain start node and a goal node as input, and output an optimal plan. GVIN can learn an underlying MDP and an optimal planning policy via either imitation learning or reinforcement learning. Inspired by VIN, GVIN applies recursive graph convolution and max-pooling operators to emulate the value iteration algorithm on general graphs. With a specially designed convolution kernel, GVIN can also transfer planning to unseen graphs of arbitrary size. Specifically, an input is a directed, weighted spatial graph (one-hot vector labels the goal state) describes one task. GVIN constructs convolution operators W P a as a function of the input graph G. The reward graph signal is denoted asR = f R (G, v *), where f R is a CNN in VIN, but an identity function in GVIN. The value functions V (v) with v ∈ V on graph nodes can be computed recursively as follows, Q While VIN uses CNN to construct W P a that could only capture 2D lattice structures, GVIN designs directional and spatial kernels to construct W P a that try to capture invariant translation on irregular graphs. However we will show that these kernels are not enough to capture invariance to graph isomorphism, which leads to a poor performance in domains with a complex geometric structure. In particular, though it works well on multiple navigation tasks, GVIN is shown to be sensitive to the choice of hyperparameters and specially designed convolution kernels, e.g. directional discretization. Graph neural networks (GNN) have received much attention recently as they can process data on irregular domains such as graphs or sets. The general idea of GNN is to compute an encoded feature h i for each node v i based on the structure of the graph, node v i and edge e ij features, and previous encoded features as, where f is a parametric function, and N (i) denotes the set of neighbour nodes of node i. After computing h i (probably apply f for k iterations), an additional function is used to compute the output at each node,, where g is implemented using another neural network, called a read-out function. Graph convolution network Many earliest work on GNN propose extending traditional CNNs to handle convolution operations on graphs through the use of spectral methods (; ;). For example, graph convolution networks (GCN) is based on fundamental convolution operations on spectral domains. GCN must assume graphs of same size. Therefore, these methods which rely on the computation of graph eigenvectors are either computationally expensive or not able to learn on graphs of arbitrary sizes. Many later introduced graph convolutional networks on spatial domain such as Neural FPs , PATCHY-SAN , DCNN , etc., are able to learn on graphs of arbitrary sizes. However they are limited to either the choice of a subset of node neighbours or random walks of k-hop neighborhouds. These drawbacks limit graph convolution based methods to applications on large-scale graphs with highly arbitrary sizes and structures, hence not favourable for transfer planning in MDPs. Graph attention network (GAT) GAT is inspired by the attention mechanism by modifying the convolution operation in GCN in order to make learning more efficient and scalable to domains of large graphs. Specifically, the encoding at each node is recursively computed as, h, where σ is an activation function, and α (l) ij is the attention coefficients which are computed as where a is weight vector, and k denotes the embedding layer k whose weights are W (k). Message passing neural network (MPNN) MPNN uses the mechanism of message passing to compute graph embedding. In particular, the calculation of a feature on each node involves two phases: i) message passing and readout. The message passing operation at a node is based on its state and all messages received from neighbor nodes. The readout is based on the node state and the calculated message. These phases are summarised as follows m MPNN is a unified framework for graph convolution and other existing graph neural networks back to that time, e.g. Graph convolution network and Laplacian Based Methods (; ; ;), Gated Graph Neural Networks (GG-NN) , Interaction Nework (IN) , Molecular Graph Convolutions , Deep Tensor Neural Networks (Schütt et al., 2017). Gilmer et.al. has made great effort in converting these frameworks to become a MPNN variant. MPNN is designed similarly to GG-NN in which GRU is applied to implement the recursive message operation, but different at message and output functions. Specifically, the message sent from node j to i is implemented as f (h i, h j, e ij) = A(e ij)h j, where A(e ij) is implemented as a neural network that maps edge feature e ij to an Another variant of the message funcion that is additionally based on the receiver node feature h i was also implemented in the paper. Updating message with all received information h i, h j, e ij is inspired by . MPNN has shown the state-of-the-art performance in prediction tasks on large graph dataset, e.g. molecular properties. Graph network (GN) Graph networks (GN) ) is a general framework that combines all previous graph neural networks. The update operations of GN involve nodes, edges and global graph features. Therefore it renders MPNN, GNN, GCN, GAT as special cases. Specifically, if we denote an additional global graph feature as u, the updates of GN which consist of three update functions g, and three aggregation functions f. These functions are implemented based on the message passing mechanism. The aggregation functions are 1) m i = f e→v ({e ij} j∈N (i) ) aggregate messages sent from edges to compute information of node i; 2) m e = f e→u ({e ij} j∈N (i),∀i ) aggregate messages sent from all edges to the global node u; 3) aggregate messages sent from all nodes to the global node u. These aggregation functions must be invariant to the order of nodes, which is critical to the Weisfeiler-Lehman (WL) graph isomorphism test and regarded as an important requirement for graph representation learning. Using aggregated information, the three update functions to node, edge and global features are defined as follows e The aggregation functions could be element-wise summation, averages, or max/min operations. The update functions could use general neural networks. The use of edge features and the global graph node makes GN distinct from MPNN. In addition, the use of a recursive update from immediate neighbors is in contrast to multi-hop updates of many spectral methods. In this section, we propose to use a general graph neural network to construct inductive biases for "learning to plan", called graph-based motion planning network (GrMPN). Similar to GVIN, our framework is trained on a set of motion planning problems as inputs and associated optimal paths (or complete optimal policies) as outputs. The inputs are general graphs without knowing its reward function and transition. At test time, GrMPN takes i) a test problem defined as a general graph and ii) a pair of starting node and a goal node. The target is to return an optimal plan for the test problem., where G i is a general graph G = (V, E), and τ * i is an optimal path (or an optimal policy) which can be generated by an expert as a demonstration. The target is to learn an optimal policy for a new graph G with a starting node and a goal node. This learning problem can be either formulated as imitation learning or reinforcement learning . While it is straightforward to train with RL, within the scope of this paper we only focus on the formulation of imitation learning. General GrMPN framework We propose a general framework for graph-based value iteration that is based on the principle of message passing. First, we also design a feature-extract function to learn a reward function r: r = f R (G; v *) given a graph G and a goal node v *. We use a similar setting from GVIN for f R (a more general architecture is depicted in Appendix A.1. Second, GrMPN also consists of the following recurrent application of graph operations at all nodes i and edges ij. q ai) where k is the processing step index; v i is the node feature (which also contains the node's value function); f e→v, g e, g v are defined as aggregation and update functions. Note that we use q ai as edge features. If the transition is stochastic, we can use |A| channels on edges. GrMPN can be implemented based on GN. GrMPN does not represent the global node u. It consists of the representation of nodes and edges, hence uses one aggregation function and two update functions, as described below: Note that for brevity the above updates assume deterministic actions, similar to the setting in VIN and GVIN. The message aggregation is equivalent to a sum over actions a s p(s |s, a)V (s), similar to the implementation of GPPN (b). The algorithm of GrMPN via Graph Networks (GrMPN-GN) is summarized in Algorithm 1. In general, many graph neural networks can be used to represent the graph-based value iteration module. As MPNN, GGNN and other similar approaches are special cases of GN, therefore we can Input: graph G = {V, E}, a goal node v * easily rewrite GrMPN to become an application of these approaches. In this section, we draw a connection of GrMPN to GAT, and show how GAT can also be used to represent a graph-based VI module. We use multi-heads to represent |A| channels for the Q value functions. Each head represents a value function corresponding to an action Q a, V where σ is a Maxout activation function or a over possible actions (in our implementation we chose the latter); and α a ij is the attention coefficients of channel a which are computed as We denote this formulation as GrMPN-GAT. We can also make a connection between GrMPN-GN and GrMPN-GAT by reformulating the updates and coefficients of GATPPN to message passing in which edge attentions become edge updates. The edge feature now have the attention coefficients α as additional information. GrMPN-GAT is rewritten as a GN module with attentions as: where e ij is the feature of the edge i-j that contains attention coefficients and q ij (equivalently to the Q-value function q ai). The algorithm of GrMPN-GAT can be described similar to GrMPN-GN in Algorithm 1. We evaluate the proposed framework GrMPN-GAT and GrMPN-GN on 2D mazes, irregular graphs, and motion planning problems. These problems range from simple regular graphs (2D mazes) to irregular graphs of simple (irregular grid) and complex geometric structures (motion planning). Through these experiments we want to confirm the following points: 1. GrMPN frameworks based on general graph neural networks can not only perform comparably to VIN for lattice structures, but also with more data-efficiency, because GrMPN-GAT and GrMPN-GN are invariant to graph isomorphism. 2. GrMPN is able to generalize well to unseen graphs of arbitrary sizes and structures, because they exploit graph isomorphism in a principled way. In particular, GrMPN handles longrange planning well by providing a great generalization ability a) across nodes in graphs b) across graphs. Therefore they are able to cope with planning on larger problems with high complexity, hence significantly improve task performance and data-efficiency. 3. GrMPN-GAT exploiting attended (weighted) updates for value functions, i.e. non-local updates, would outperform GrMPN-GN which only based on uniform sum over actions. In all experiments, we follow the standard encode-decode graph network design. Firstly, each graph component (node, edge) feature is passed through a two-layered MLP encoder with ReLu activation and 16 hidden units each. To increase the robustness, this two-layered network is also used in other graph network components, include both the graph block module (GrMPN-GAT or GrMPN-GN) and the decoder network. Notably, in the lattice 20× 20 experiment, as the number of nodes is significantly large, we instead increase the hidden units up to 32. We use an additional node encoding f v, to compute note features v in the motion planning problems (see Appendix A.1 a full policy network architecture). Finally, we use the standard RMSProp algorithm with a learning rate of 0.001 in all experiments. The numbers of message passing step k, is set differently in different experiments. Specifically, k is set respectively equal to 10, 15, 20 for 12 × 12, 16 × 16 and 20 × 20 2D mazes. Meanwhile, k = 20 in all other irregular graph experiments, except for the last motion planing task tested on a roadmap of 500 configuration where we set k = 40 to handle large graphs. We use other standard settings as used in the papers of VIN and GVIN (Note that VIN set the recurrent parameter as: k=40 on 2D mazes, k=200 on irregular graphs, and motion planning). We use three different metric which are also used in different work: i) %Accuracy (Acc) is the percentage of nodes whose predicted actions are optimal, ii) %Success (Succ) is the percentage of nodes whose predicted path reach the goal, and iii) path difference is the Euclidean distance over all nodes between a predicted path vs. optimal path. Training: While extensions to RL training is straightforward, we are only focused on imitation learning. Therefore we use the same objective function used by VIN and GVIN, i.e. a cross-entropy loss for the supervised learning problem with dataset {v, a * = π * (v)} where v is a state node with an optimal action a * demonstrated by an expert. In this experiment we carry out evaluations on 2D mazes which have a regular graph structure. We compare GrMPN against VIN, GVIN, and GPPN. The environment and experiment settings are set similar to VIN, GVIN, and GPPN. We use the same script used in VIN, GVIN, and GPPN to generate graphs. For each each graph, we generated seven optimal trajectories corresponding to different start and goal nodes. Note that only GPPN requires a complete optimal policy which gives an optimal action at every graph node. This setting makes GPPN have a little advantage. We train and test on the same graph with sizes 12 × 12, 16 × 16, 20 × 20. The number of generated graphs for training is chosen from small to large with values {200, 1000, 5000}. The size of testing data is fixed to 1000 graphs of corresponding size. The shown in Table 1 tells that GrMPN-GAT and GrMPN-GN not only outperform other baselines but also more data-efficient. GPPN has a slightly better performance with a large amount of data. We note that GPPN must assume the data consists of optimal policies instead of a small set of optimal demonstration trajectories as in VIN, GVIN and ours. GVIN is a graph-based VIN but it relies on specially designed convolution kernels that are based on the choice of discretised directions. Therefore GVIN is not able to perform as well as GrMPN-GAT and GrMPN-GN which are based on principled graph networks and known to be invariant to graph isomorphism. GrMPN-GAT and GrMPN-GN show significant better in terms of data-efficiency on large domains 20 × 20. On these large domains, learning algorithms often require a large amount of data. However GrMPN-GAT and GrMPN-GN can still learn well with a limited amount of data. This shows how important invariance to graph isormophism is for learning on graphs. Performance of GrMPN-GAT on bigger domain is better than small domains, because the amount of nodes involved in training is bigger in large graphs. We show more ablation in Appendix. This experiment uses the same script used by GVIN, which is based on Networkx , to create synthetic graphs that are with random coordinates from box 2 in 2D space. We vary the parameters of the generation program to create three types of irregular graphs: Dense, Sparse, and Treelike. For Tree-like graphs, we use use the Networkx's function, geographical threshold graph by setting the connectedness probability between nodes to a small value. We create Tree-like graphs which are not considered in GVIN, because there are two main challenges on these graphs. First, with the same number of nodes and amount of generated graphs, tree-like graphs would have much fewer nodes for training. Second, Tree-like graphs in a major issue which are ideal to evaluate generalization for long-range planning which requires propagation of value functions across nodes in graphs well. We generate 10000 graphs, with varying number of nodes {10, 100}. The label for each graph is an optimal policy, i.e. an optimal action at every graph node. Training is 6/7 of the generated data, while testing is 1/7. The comparing are described in Tables 2 (on Dense), 3 (on Sparse), and 7 (on Tree-like). Testing is performed on irregular graphs of different size: 100 and 150 nodes on Dense, 100 nodes on Sparse. The show that GrMPN methods perform comparably with GVIN on Dense graphs in terms of Success rate, but slightly better in terms of Accuracy and Distance difference. On Sparse graphs, GrMPN-GAT and GrMPN-GN based on the principled of message passing are able to have fast updates across nodes. Sampling-based methods such as probabilistic roadmaps (PRM) have been shown to be every efficient in practice. PRM is one of the most widely used techniques in robotic motion planning, especially for applications in navigation. In such an application, a motion planning algorithm must find an optimal path that must satisfy i) the environment's geometry constraints, i.e. collision-free path, and ii) the robot system constraint, e.g. differential constraints. PRM is multiple-query methods that are very useful in highly structured environments such as large buildings. We evaluate GrMPN on two motion planning problems: 2D navigation with a holonomic mobile robot and manipulation with a simulated 7-DoF Baxter robot arm. We aim to improve PRM by bringing it closer to an online-planning method through transfer planning. In this section, we show that GrMPN would outperform GVIN under such tasks of a complex geometric structure. Setting: Note that the input of the simulated 7-DoF Baxter robot forms a kinematic tree. Therefore we propose to use three alternative encoding layers f v to comppute node features: GN, GCN or MLP. For the 2D navigation with a mobile robot, we only use a MLP to encode robot locations. In addition, we use a simple MLP layer of 16 nodes to encode the one-hot goal value. Then, the outputs from these two encoders are concatenated and used as node features. This architecture renders our implementation a hierarchical graph-based planning method. Data generation: For the mobile robot navigation, we use a standard PRM algorithm to construct 2000 roadmaps for training, each with 200 robot configurations. A graph node is represented as a robot configuration which is a generalized coordinate (x, y) where x, y are 2D translation coordinates. We generate two different test sets consisting of 1000 roadmaps: 200 and 500 configurations. Each generation time uses a different setting of environment obstacles. For each generated graph, we use the Dijkstra algorithm to provide one optimal trajectory corresponding to one random pair of start and goal states. For the simulated 7-DoF Baxter robot arm, we use the same scrift as in to generate different environment settings (with different obstacles), and set different start and goal configurations. For each environment setting, the roadmap and the optimal trajectories from 20 randomly selected start nodes to the goal are then found by using PRM * and the Dijkstra algorithm provided by the OMPL library (Ş ucan et al., 2012). In total, we generate 280 roadmaps, each with 300 configurations. The test set contains 80 roadmaps, each with about 1200 configurations. Analysis: Table 4 show test performance on 2D navigation with a mobile robot. It shows that GrMPN methods not only outperform GVIN but also possess a great generalization ability. We additionally evaluate the generalization ability of GrMPN methods by using the trained model using Tree-like data as described in Irregular Graphs section to test on the created roadmap test set. The distance difference (Diff) computes the cost difference between the predicted path and the optimal path planned by Dijkstra. We skip reporting on GVIN on large testing graphs (500 configuration nodes) due to its degraded performance. The trained model using Tree-like graph data could also generalize well on unseen graphs generated by PRM on different environments. In addition, they can generalize to much bigger graphs (with 500 nodes). This suggests GrMPN is able to do zero-shot learning to plan. Table 5 show the test performance on the motion planning task with a simulated 7-DoF Baxter arm. We skip reports on GVIN due to its poor performance and scalability to learning on large graphs. The show that GrMPN methods are able to do motion planning on this complex geometric domain with high accuracy and success rates. Table 5: Simulated 7-DoF Baxter arm motion planning: Test performance with different uses of node encoding (MLP-, GCN-, GN-) and graph-based planning (GrMPN-GAT and GrMPN-GN). Figure 1: GrMPN based on graph neural networks. A APPENDIX Our proposed graph-based motion planning network is inspired by VIN and GVIN, and depicted in Fig. 1. The in Figs. 2 and 3 show ablation of GrMPN-GAT and GrMPN-GN on different number of graph processing steps. The figure show the value function maps after training. The show how value functions on a test graph (after training) are computed depending on the value k of processing steps. More nodes would be updated with a larger number of processing step which corresponding to more batches of value iterations updates. This ablation also shows that GrMPN-GAT is able to generalize across nodes better than GrMPN-GN. This generalization ability would significantly help with long-range planning problems. The information on generated graphs is summarized in Table 6. As seen in Fig. 4, GVIN is not able to spread the update to nodes that are far from the goal node. This figure also shows that GrMPN-GAT has a slightly better generalization ability across nodes than GrMPN-GN. The value functions of GrMPN-GN have more un-updated nodes (see the color of nodes that are far from the goal node as labeled in black) than that of GrMPN-GAT. This explains why GrMPN-GAT performs slightly better than GrMPN-GN. The in Table 7 tell that VIN is not able to cope with very sparse graphs and long-range planning. This shows GVIN has a weak generalization ability. Data generation: We use a standard PRM algorithm to construct 2000 graph maps for training, each with 200 robot configurations. A graph node is represented as a robot configuration which is a generalized coordinate (x, y) where x, y are 2D translation coordinates. We generate two different test sets consisting of 1000 roadmaps: 200 and 500 configurations. Each generation time uses a different setting of environment obstacles. For each generated graph, we use the Dijkstra algorithm to provide one optimal trajectory corresponding to one random pair of start and goal states, for an example in Fig. 6. For the simulated 7-DoF Baxter robot arm, we use the same scrift as in to generate different environment settings (with different obstacles), and set different start and goal configurations. An example of a generated environment is depicted in Fig. 5. Figures 7 and 8 show that GVIN is not able to update value functions of nodes far from the goal, while GrMPN-GAT and GrMPN-GN can generalize value updates well to such nodes. The color map in Figure 8 also suggests that GrMPN-GAT slightly has wider value propagation, which means better generalization for long-range planning and across task graphs.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose an end-to-end differentiable planning network for graphs. This can be applicable to many motion planning problems

represent the tldr for finding the corresponding scientific article

------

Fewshot example: "This paper proposes a simple procedure for evaluating compositional structure in learned representations, and uses the procedure to explore the role of compositionality in four learning problems." == "Many machine learning algorithms represent input data with vector embeddings or discrete codes. When inputs exhibit compositional structure (e.g. objects built from parts or procedures from subroutines), it is natural to ask whether this compositional structure is reflected in the the inputs’ learned representations. While the assessment of compositionality in languages has received significant attention in linguistics and adjacent fields, the machine learning literature lacks general-purpose tools for producing graded measurements of compositional structure in more general (e.g. vector-valued) representation spaces. We describe a procedure for evaluating compositionality by measuring how well the true representation-producing model can be approximated by a model that explicitly composes a collection of inferred representational primitives. We use the procedure to provide formal and empirical characterizations of compositional structure in a variety of settings, exploring the relationship between compositionality and learning dynamics, human judgments, representational similarity, and generalization. Figure 1: Representations arising from a communication game. In this game, an observation (b) is presented to a learned speaker model (c), which encodes it as a discrete character sequence (d) to be consumed by a listener model for some downstream task. The space of inputs has known compositional structure (a). We want to measure the extent to which this structure is reflected (perhaps imperfectly) in the structure of the learned codes. The success of modern representation learning techniques has been accompanied by an interest in understanding the structure of learned representations. One feature shared by many humandesigned representation systems is compositionality: the capacity to represent complex concepts (from objects to procedures to beliefs) by combining simple parts BID18. While many machine learning approaches make use of human-designed compositional analyses for representation and prediction BID44 BID16, it is also natural to ask whether (and how) compositionality arises in learning problems where compositional structure has not been built in from the start. Consider the example in Figure 1, which shows a hypothetical character-based encoding scheme learned for a simple communication task (similar to the one studied by). Is this encoding scheme compositional? That is, to what extent can we analyze the agents' messages as being built from smaller pieces (e.g. pieces xx meaning blue and bb meaning triangle)?A large body of work, from early experiments on language evolution to recent deep learning models BID24 ), aims to answer questions like this one. But existing solutions rely on manual (and often subjective) analysis of model outputs BID32, or at best automated procedures tailored to the specifics of individual problem domains BID7. They are difficult to compare and difficult to apply systematically. We are left with a need for a standard, formal, automatable and quantitative technique for evaluating claims about compositional structure in learned representations. The present work aims at first steps toward meeting that need. We focus on an oracle setting where the compositional structure of model inputs is known, and where the only question is whether this structure is reflected in model outputs. This oracle evaluation paradigm covers most of the existing representation learning problems in which compositionality has been studied. The first contribution of this paper is a simple formal framework for measuring how well a collection of representations (discrete-or continuous-valued) reflects an oracle compositional analysis of model inputs. We propose an evaluation metric called TRE, which provides graded judgments of compositionality for a given set of (input, representation) pairs. The core of our proposal is to treat a set of primitive meaning representations as hidden, and optimize over them to find an explicitly compositional model that approximates the true model as well as possible. For example, if the compositional structure that describes an object is a simple conjunction of attributes, we can search for a collection of "attribute vectors" that sum together to produce the observed object representations; if it is a sparse combination of (attribute, value) pairs we can additionally search for "value vectors" and parameters of a binding operation; and so on for more complex compositions. Having developed a tool for assessing the compositionality of representations, the second contribution of this paper is a survey of applications. We present experiments and analyses aimed at answering four questions about the relationship between compositionality and learning:• How does compositionality of representations evolve in relation to other measurable model properties over the course of the learning process? (Section 4) • How well does compositionality of representations track human judgments about the compositionality of model inputs? (Section 5) • How does compositionality constrain distances between representations, and how does TRE relate to other methods that analyze representations based on similarity? (Section 6) • Are compositional representations necessary for generalization to out-of-distribution inputs?(Section 7)We conclude with a discussion of possible applications and generalizations of TRE-based analysis. Arguments about whether distributed (and other non-symbolic) representations could model compositional phenomena were a staple of 1980s-era connectionist-classicist debates. BID42 provides an overview of this discussion and its relation to learnability, as well as a concrete implementation of a compositional encoding scheme with distributed representations. Since then, numerous other approaches for compositional representation learning have been proposed, with BID30 BID43 and without BID15 BID22 ) the scaffolding of explicit composition operations built into the model. The main experimental question is thus when and how compositionality arises "from scratch" in the latter class of models. In order to answer this question it is first necessary to determine whether compositional structure is present at all. Most existing proposals come from linguistics and and philosophy, and offer evaluations of compositionality targeted at analysis of formal and natural languages BID8 BID28. Techniques from this literature are specialized to the details of linguistic representations-particularly the algebraic structure of grammars BID31. It is not straightforward to apply these techniques in more general settings, particularly those featuring non-string-valued representation spaces. We are not aware of existing work that describes a procedure suitable for answering questions about compositionality in the general case. Machine learning research has responded to this absence in several ways. One class of evaluations BID32 BID11 derives judgments from ad-hoc manual analyses of representation spaces. These analyses provide insight into the organization of representations but are time-consuming and non-reproducible. Another class of evaluations BID6 BID0 BID4 ) exploits task-specific structure (e.g. the ability to elicit pairs of representations known to feature particular relationships) to give evidence of compositionality. Our work aims to provide a standard and scalable alternative to these model-and task-specific evaluations. Other authors refrain from measuring compositionality directly, and instead base analysis on measurement of related phenomena, for which more standardized evaluations exist. Examples include correlation between representation similarity and similarity of oracle compositional analyses BID7 and generalization to structurally novel inputs BID26. Our approach makes it possible to examine the circumstances under which these surrogate measures in fact track stricter notions of compositionality; similarity is discussed in Sec. 6 and generalization in Sec. 7.A long line of work in natural language processing BID13 BID2 BID12 BID19 focuses on learning composition functions to produce distributed representations of phrases and sentences-that is, for purposes of modeling rather than evaluation. We use one experiment from this literature to validate our own approach (Section 5). On the whole, we view work on compositional representation learning in NLP as complementary to the framework presented here: our approach is agnostic to the particular choice of composition function, and the aforementioned references provide well-motivated choices suitable for evaluating data from language and other sources. Indeed, one view of the present work is simply as a demonstration that we can take existing NLP techniques for compositional representation learning, fit them to representations produced by other models (even in non-linguistic settings), and view the ing training loss as a measure of the compositionality of the representation system in question. Consider again the communication task depicted in Figure 1. Here, a speaker model observes a target object described by a feature vector. The speaker sends a message to a listener model, which uses the message to complete a downstream task-for example, identifying the referent from a collection of distractors based on the content of the message BID17 ). Messages produced by the speaker model serve as representations of input objects; we want to know if these representations are compositional. Crucially, we may already know something about the structure of the inputs themselves. In this example, inputs can be identified via composition of categorical shape and color attributes. How might we determine whether this oracle analysis of input structure is reflected in the structure of representations? This section proposes an automated procedure for answering the question. Representations A representation learning problem is defined by a dataset X of observations x (Figure 1b); a space Θ of representations θ (Figure 1d); and a model f: X → Θ (Figure 1c). We assume that the representations produced by f are used in a larger system to accomplish some concrete task, the details of which are not important for our analysis. Derivations The technique we propose additionally assumes we have prior knowledge about the compositional structure of inputs. In particular, we assume that inputs can be labeled with treestructured derivations d (Figure 1a), defined by a finite set D 0 of primitives and a binary bracketing operation ·, ·, such that if DISPLAYFORM0 Compositionality In intuitive terms, the representations computed by f are compositional if each f (x) is determined by the structure of D(x). Most discussions of compositionality, following BID31, make this precise by defining a composition operation θ a * θ b → θ in the space of representations. Then the model f is compositional if it is a homomorphism from inputs to representations: we require that for any x with DISPLAYFORM1 In the linguistic contexts for which this definition was originally proposed, it is straightforward to apply. Inputs x are natural language strings. Their associated derivations D(x) are syntax trees, and composition of derivations is syntactic composition. Representations θ are logical representations of meaning (for an overview see BID47 . To argue that a particular fragment of language is compositional, it is sufficient to exhibit a lexicon D 0 mapping words to their associated meaning representations, and a grammar for composing meanings where licensed by derivations. Algorithms for learning grammars and lexicons from data are a mainstay of semantic parsing approaches to language understanding problems like question answering and instruction following BID49 BID9 BID1 .But for questions of compositionality involving more general representation spaces and more general analyses, the above definition presents two difficulties: In the absence of a clearly-defined syntax of the kind available in natural language, how do we identify lexicon entries: the primitive parts from which representations are constructed? What do we do with languages like the one in Figure 1d, which seem to exhibit some kind of regular structure, but for which the homomorphism condition given in Equation 1 cannot be made to hold exactly?Consider again the example in Figure 1. The oracle derivations tell us to identify primitive representations for dark, blue, green, square, and triangle. The derivations then suggest a process for composing these primitives (e.g. via string concatenation) to produce full representations. The speaker model is compositional (in the sense of Equation 1) as long as there is some assignment of representations to primitives such that for each model input, composing primitive representations according to the oracle derivation reproduces the speaker's prediction. In Figure 1 there is no assignment of strings to primitives that reproduces model predictions exactly. But predictions can be reproduced approximately-by taking xx to mean blue, aa to mean square, etc. The quality of the approximation serves as a measure of the compositionality of the true predictor: predictors that are mostly compositional but for a few exceptions, or compositional but for the addition of some noise, will be well-approximated on average, while arbitrary mappings from inputs to representations will not. This suggests that we should measure compositionality by searching for representations that allow an explicitly compositional model to approximate the true f as closely as possible. We define our evaluation procedure as follows:Tree Reconstruction Error (TRE)First choose: DISPLAYFORM2, a compositional approximation to f with parameters η, as: DISPLAYFORM3 f η has one parameter vector η i for every d i in D 0; these vectors are members of the representation space Θ.Given a dataset X of inputs x i with derivations d i = D(x i), compute: DISPLAYFORM4 Then we can define datum-and dataset-level evaluation metrics: DISPLAYFORM5 DISPLAYFORM6 TRE and compositionality How well does the evaluation metric TRE(X) capture the intuition behind Equation 1? The definition above uses parameters η i to witness the constructability of representations from parts, in this case by explicitly optimizing over those parts rather than taking them to be given by f. Each term in Equation 2 is analogous to an instance of Equation 1, measuring how wellf η * (x i), the best compositional prediction, matches the true model prediction f (x i). In the case of models that are homomorphisms in the sense of Equation 1, TRE reduces to the familiar case: DISPLAYFORM7 Learnable composition operators The definition of TRE leaves the choice of δ and * up to the evaluator. Indeed, if the exact form of the composition function is not known a priori, it is natural to define * with free parameters (as in e.g. BID2, treat these as another learned part off, and optimize them jointly with the η i . However, some care must be taken when choosing * (especially when learning it) to avoid trivial solutions:Remark 2. Suppose D is injective; that is, every x ∈ X is assigned a unique derivation. Then there is always some * that achieves TRE(X) = 0: DISPLAYFORM8 as in the preceding definition, and setf = f.In other words, some pre-commitment to a restricted composition function is essentially inevitable: if we allow the evaluation procedure to select an arbitrary composition function, the will be trivial. This paper features experiments with * in both a fixed functional form and a learned parametric one. Implementation details For models with continuous Θ and differentiable δ and *, TRE(X) is also differentiable. Equation 2 can be solved using gradient descent. We use this strategy in Sections 4 and 5. For discrete Θ, it may be possible to find a continuous relaxation with respect to which δ(θ, ·) and * are differentiable, and gradient descent again employed. We use this strategy in Section 7 (discussed further there). An implementation of an SGD-based TRE solver is provided in the accompanying software release. For other problems, task-specific optimizers (e.g. machine translation alignment models; BID4 or general-purpose discrete optimization toolkits can be applied to Equation 2.The remainder of the paper highlights ways of using TRE to answer questions about compositionality that arise in machine learning problems of various kinds. We begin by studying the relationship between compositionality and learning dynamics, focusing on the information bottleneck theory of representation learning proposed by BID45 . This framework proposes that learning in deep models consists of an error minimization phase followed by a compression phase, and that compression is characterized by a decrease in the mutual information between inputs and their computed representations. We investigate the hypothesis that the compression phase finds a compositional representation of the input distribution, isolating decision-relevant attributes and discarding irrelevant information. FIG1). We predict classifiers in a meta-learning framework BID39 BID37: for each sub-task, the learner is presented with two images corresponding to some compositional visual concept (e.g. "digit 8 on a black " or "green with heavy stroke") and must determine whether a held-out image is an example of the same visual concept. Given example images x 1 and x 2, a test image x *, and label y *, the model computes: DISPLAYFORM0 We use θ as the representation of a classifier for analysis. The model is trained to minimize the logistic loss between logitsŷ and ground-truth labels y *. More details are given in Appendix A.Compositional structure Visual concepts used in this task are all single attributes or conjunctions of attributes; i.e. their associated derivations are of the form attr or attr 1, attr 2. Attributes include color, digit color, digit identity and stroke type. The composition function * is addition and the distance δ(θ, θ) is cosine similarity 1 − θ θ /(θ θ).Evaluation The training dataset consists of 9000 image triplets, evenly balanced between positive and negative classes, with a validation set of 500 examples. At convergence, the model achieves validation accuracy of 75.2% on average over ten training runs. (Perfect accuracy is not possible because the true classifier is not fully determined by two training examples). We explore the relationship between the information bottleneck and compositionality by comparing TRE(X) to the mutual information I(θ; x) between representations and inputs over the course of training. Both quantities are computed on the validation set, calculating TRE(X) as described in Section 3 and I(θ; X) as described in BID41. (For discussion of limitations of this approach to computing mutual information between inputs and representations, see .) FIG2 shows the relationship between TRE(X) and I(θ; X). Recall that small TRE is indicative of a high degree of compositionality. It can be seen that both mutual information and reconstruction error are initially low (because representations initially encode little about distinctions between inputs). Both increase over the course of training, and decrease together after mutual information reaches a maximum FIG2. This pattern holds if we plot values from multiple training runs at the same time FIG2 ), or if we consider only the postulated compression phase FIG2 ). These are consistent with the hypothesis that compression in the information bottleneck framework is associated with the discovery of compositional representations. Next we investigate a more conventional representation learning task. High-dimensional embeddings of words and phrases are useful for many natural language processing applications BID46, and many techniques exist to learn them from unlabeled text BID14 BID29. The question we wish to explore is not whether phrase vectors are compositional in aggregate, but rather how compositional individual phrase representations are. Our hypothesis is that bigrams whose representations have low TRE are those whose meaning is essentially compositional, and well-explained by the constituent words, while bigrams with large reconstruction error will correspond to non-compositional multi-word expressions BID33 ).This task is already well-studied in the natural language processing literature BID36, and the analysis we present differs only in the use of TRE to search for atomic representations rather than taking them to be given by pre-trained word representations. Our goal is to validate our approach in a language processing context, and show how existing work on compositionality (and representations of natural language in particular) fit into the more general framework proposed in the current paper. We train embeddings for words and bigrams using the CBOW objective of BID29 using the implementation provided in FastText BID5 with 100-dimensional vectors and a context size of 5. Vectors are estimated from a 250M-word subset of the Gigaword dataset BID34. More details are provided in Appendix A.Compositional structure We want to know how close phrase embeddings are to the composition of their constituent word embeddings. We define derivations for words and phrases in the natural way: single words w have primitive derivations d = w; bigrams w 1 w 2 have derivations of the form w 1, w 2. The composition function is again vector addition and distance is cosine distance. (Future work might explore learned composition functions as in e.g. BID21, for future work.) We compare bigram-level judgments of compositionality computed by TRE with a dataset of human judgments about noun-noun compounds BID35. In this dataset, humans rate bigrams as compositional on a scale from 0 to 5, with highly conventionalized phrases like gravy train assigned low scores and graduate student assigned high ones. Results We reproduce the of BID36 within the tree reconstruction error framework: for a given x, TRE(x) is anticorrelated with human judgments of compositionality (ρ = −0.34, p < 0.01). Collocations rated "most compositional" by our approach (i.e. with lowest TRE) are: application form, polo shirt, research project; words rated "least compositional" are fine line, lip service, and nest egg. The next section aims at providing a formal, rather than experimental, characterization of the relationship between TRE and another perspective on the analysis of representations with help from oracle derivations. BID7 introduce a notion of topographic similarity, arguing that a learned representation captures relevant domain structure if distances between learned representations are correlated with distances between their associated derivations. This can be viewed as providing a weak form of evidence for compositionality-if the distance function rewards pairs of representations that share overlapping substructure (as might be the case with e.g. string edit distance), edit distance will be expected to correlate with some notion of derivational similarity.In this section we aim to clarify the relationship between the two evaluations. To do this we first need to equip the space of derivations described in Section 3 with a distance function. As the derivations considered in this paper are all tree-structured, it is natural to use a simple tree edit distance BID3 for this purpose. We claim the following: Proposition 1. Letf =f η * be an approximation to f estimated as in Equation 2, with all TRE(x) ≤ for some. Let ∆ be the tree edit distance (defined formally in Appendix B, Definition 2), and let δ be any distance on Θ satisfying the following properties: DISPLAYFORM0, where 0 is the identity element for *. DISPLAYFORM1 (This condition is satisfied by any translation-invariant metric.)Then ∆ is an approximate upper bound on δ: for any DISPLAYFORM2 In other words, representations cannot be much farther apart than the derivations that produce them. Proof is provided in Appendix B.We emphasize that small TRE is not a sufficient condition for topographic similarity as defined by BID7: very different derivations might be associated with the same representation (e.g. when representing arithmetic expressions by their ). But this does demonstrate that compositionality imposes some constraints on the inferences that can be drawn from similarity judgments between representations. In our final set of experiments, we investigate the relationship between compositionality and generalization. Here we focus on communication games like the one depicted in Figure 1 and in more detail in FIG3. As in the previous section, existing work argues for a relationship between compositionality and generalization, claiming that agents need compositional communication protocols to generalize to unseen referents BID26 BID11. Here we are able to evaluate this claim empirically by training a large number of agents from random initial conditions, measuring the compositional structure of the language that emerges, and seeing how this relates to their performance on both familiar and novel objects. A speaker model observes a pair of target objects, and sends a description of the objects (as a discrete code) to a listener model. The listener attempts to reconstruct the targets, receiving fractional reward for partially-correct predictions. Our experiment focuses on a reference game BID20. Two policies are trained: a speaker and a listener. The speaker observes pair of target objects represented with a feature vector. The speaker then sends a message (coded as a discrete character sequence) to the listener model. The listener observes this message and attempts to reconstruct the target objects by predicting a sequence of attribute sets. If all objects are predicted correctly, both the speaker and the listener receive a reward of 1 (partial credit is awarded for partly-correct objects; FIG3).Because the communication protocol is discrete, policies are jointly trained using a policy gradient objective BID48. The speaker and listener are implemented with RNNs; details are provided in Appendix A.Compositional structure Every target referent consists of two objects; each object has two attributes. The derivation associated with each communicative task thus has the tree structure attr 1a, attr 1b, attr 2a, attr 2b. We hold out a subset of these object pairs at training time to evaluate generalization: in each training run, 1/3 of possible reference candidates are never presented to the agent at training time. Where the previous examples involved a representation space of real embeddings, here representations are fixed-length discrete codes. Moreover, the derivations themselves have a more complicated semantics than in Sections 4 and 5: order matters, and a commutative operation like addition cannot capture the distinction between green, square, blue, triangle and green, triangle, blue, square. We thus need a different class of composition and distance operations. We represent each agent message as a sequence of one-hot vectors, and take the error function δ to be the 1 distance between vectors. The composition function has the form: DISPLAYFORM0 with free composition parameters η * = {A, B} in Equation 2. These matrices can redistribute the tokens in θ and θ across different positions of the input string, but cannot affect the choice of the tokens themselves; this makes it possible to model non-commutative aspects of string production. To (b) However, compositional languages also exhibit lower absolute performance (r = 0.57, p < 1e−9). Both facts remain true even if we restrict analysis to "successful" training runs in which agents achieve a reward > 0.5 on held-out referents (r = 0.6, p < 1e−3 and r = 0.38, p < 0.05 respectively). Figure 6: Fragment of languages ing from two multiagent training runs. In the first section, the left column shows the target referent, while the remaining columns show the message generated by speaker in the given training run after observing the referent. The two languages have substantially different TRE, but induce similar listener performance (Train and Test reward).compute TRE via gradient descent, we allow the elements of D 0 to be arbitrary vectors (intuitively assigning fractional token counts to string indices) rather than restricting them to one-hot indicators. With this change, both δ and * have subgradients and can be optimized using the same procedure as in preceding sections. Results We train 100 speaker-listener pairs with random initial parameters and measure their performance on both training and test sets. Our suggest a more nuanced view of the relationship between compositionality and generalization than has been argued in the existing literature. TRE is significantly correlated with generalization error (measured as the difference between training accuracies, FIG4). However, TRE is also significantly correlated with absolute model reward (FIG4)-"compositional" languages more often from poor communication strategies than successful ones. This is largely a consequence of the fact that many languages with low TRE correspond to trivial strategies (for example, one in which the speaker sends the same message regardless of its observation) that in poor overall performance. Moreover, despite the correlation between TRE and generalization error, low TRE is by no means a necessary condition for good generalization. We can use our technique to automatically mine a collection of training runs for languages that achieve good generalization performance at both low and high levels of compositionality. Examples of such languages are shown in Figure 6. We have introduced a new evaluation method called TRE for generating graded judgments about compositional structure in representation learning problems where the structure of the observations is understood. TRE infers a set of primitive meaning representations that, when composed, approximate the observed representations, then measures the quality of this approximation. We have applied TRE-based analysis to four different problems in representation learning, relating compositionality to learning dynamics, linguistic compositionality, similarity and generalization. Many interesting questions regarding compositionality and representation learning remain open. The most immediate is how to generalize TRE to the setting where oracle derivations are not available; in this case Equation 2 must be solved jointly with an unsupervised grammar induction problem BID25. Beyond this, it is our hope that this line of research opens up two different kinds of new work: better understanding of existing machine learning models, by providing a new set of tools for understanding their representational capacity; and better understanding of problems, by better understanding the kinds of data distributions and loss functions that give rise to compositionalor non-compositional representations of observations. Code and data for all experiments in this paper are provided at https://github.com/jacobandreas/tre. Thanks to Daniel Fried and David Gaddy for feedback on an early draft of this paper. The author was supported by a Facebook Graduate Fellowship at the time of writing. Few-shot classification The CNN has the following form:Conv(out=6, kernel=5) ReLU MaxPool(kernel=2) Conv(out=16, kernel=5) ReLU MaxPool(kernel=2) Linear(out=128) ReLU Linear(out=64) ReLUThe model is trained using ADAM BID23 ) with a learning rate of.001 and a batch size of 128. Training is ended when the model stops improving on a held-out set. Word embeddings We train FastText BID5 on the first 250 million words of the NYT section of Gigaword BID34. To acquire bigram representations, we pre-process this dataset so that each occurrence of a bigram from the BID35 dataset is treated as a single word for purposes of estimating word vectors. Communication The encoder and decoder RNNs both use gated recurrent units BID10 with embeddings and hidden states of size 256. The size of the discrete vocabulary is set to 16 and the maximum message length to 4. Training uses a policy gradient objective with a scalar baseline set to the running average reward; this is optimized using ADAM BID23 ) with a learning rate of.001 and a batch size of 256. Each model is trained for 500 steps. Models are trained by sampling from the decoder's output distribution, but greedy decoding is used to evaluate performance and produce Figure 6. First, some definitions:Definition 1. The size of a derivation is given by: DISPLAYFORM0 Definition 2. The tree edit distance between derivations is defined by: Proof. For d ∈ D 0 this follows immediately from Condition 2 in the proposition. For composed derivations it follows from Condition 3 taking θ k = θ = 0 and induction on |d|. DISPLAYFORM1"

We propose Dual Actor-Critic algorithm, which is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation. The algorithm achieves the state-of-the-art performances across several benchmarks.


Question: Represent the article for finding a single-sentence TLDR

Question: We describe techniques for training high-quality image denoising models that require only single instances of corrupted images as training data. Inspired by a recent technique that removes the need for supervision through image pairs by employing networks with a "blind spot" in the receptive field, we address two of its shortcomings: inefficient training and poor final denoising performance. This is achieved through a novel blind-spot convolutional network architecture that allows efficient self-supervised training, as well as application of Bayesian distribution prediction on output colors. Together, they bring the self-supervised model on par with fully supervised deep learning techniques in terms of both quality and training speed in the case of i.i.d. Gaussian noise. Denoising, the removal of noise from images, is a major application of deep learning. Several architectures have been proposed for general-purpose image restoration tasks, e.g., U-Nets BID13, hierarchical residual networks BID11, and residual dense networks BID17. Traditionally, the models are trained in a supervised fashion with corrupted images as inputs and clean images as targets, so that the network learns to remove the corruption. BID9 introduced NOISE2NOISE training, where pairs of corrupted images are used as training data. They observe that when certain statistical conditions are met, a network faced with the impossible task of mapping corrupted images to corrupted images learns, loosely speaking, to output the "average" image. For a large class of image corruptions, the clean image is a simple per-pixel statistic -such as mean, median, or mode -over the stochastic corruption process, and hence the restoration model can be supervised using corrupted data by choosing the appropriate loss function to recover the statistic of interest. While removing the need for clean training images, NOISE2NOISE training still requires at least two independent realizations of the corruption for each training image. While this eases data collection significantly compared to noisy-clean pairs, large collections of (single) poor images are still much more widespread. This motivates investigation of self-supervised training: how much can we learn from just looking at bad data? While foregoing supervision would lead to the expectation of some regression in performance, can we make up for it by making stronger assumptions about the corruption process? In this paper, we show that under the assumption of additive Gaussian noise that is i.i.d. between pixels, no concessions in denoising performance are necessary. We draw inspiration from the recent NOISE2VOID (N2V) training technique of BID7. The algorithm needs no image pairs, and uses just individual noisy images as training data, assuming that the corruption is zero-mean and independent between pixels. The method is based on blind-spot networks where the receptive field of the network does not include the center pixel. This allows using the same noisy image as both training input and training target -because the network cannot see the correct answer, using the same image as target is equivalent to using a different noisy realization. This approach is self-supervised in the sense that the surrounding context is used to predict the value of the output pixel without a separate reference image BID3.The networks used by BID7 do not have a blind spot by design, but are trained to ignore the center pixel using a masking scheme where only a few output pixels can contribute to the loss function, reducing training efficiency considerably. We remedy this with a novel architecture that allows efficient training without masking. Furthermore, the existence of the blind spot leads to poor denoising quality. We derive a scheme for combining the network output with data in the blind spot, bringing the denoising quality on par with conventionally trained networks. In our blind-spot network architecture, we effectively construct four denoiser network branches, each having its receptive field restricted to a different direction. A single-pixel offset at the end of each branch separates the receptive field from the center pixel. The are then combined by 1×1 convolutions. In practice, we run four rotated versions of each input image through a single receptive field -restricted branch, yielding a simpler architecture that performs the same function. This also implicitly shares the convolution kernels between the branches and thus avoids the four-fold increase in the number of trainable weights. Our convolutional blind-spot networks are designed by combining multiple branches that each have their receptive field restricted to a half-plane (FIG0) that does not contain the center pixel. The principle of limiting the receptive field has been used in PixelCNN (van den a) image synthesis networks, where only pixels synthesized before the current pixel are allowed in the receptive field. We combine the four branches with a series of 1×1 convolutions to obtain a receptive field that can extend arbitrarily far in every direction but does not contain the center pixel. In order to transform a restoration network into one with a restricted receptive field, we modify each individual layer so that its receptive field is fully contained within one half-plane, including the center row/column. The receptive field of the ing network includes the center pixel, so we offset the feature maps by one pixel before combining them. Layers that do not extend the receptive field, e.g., concatenation, summation, 1×1 convolution, etc., can be used without modifications. Convolution layers. To restrict the receptive field of a zero-padding convolution layer to extend only, say, upwards, the easiest solution is to offset the feature maps downwards when performing the convolution operation. For an h × w kernel size, a downwards offset of k = h/2 pixels is equivalent to using a kernel that is shifted upwards so that all weights below the center line are zero. Specifically, we first append k rows of zeros to the top of input tensor, then perform the convolution, and finally crop out the k bottom rows of the output. Downsampling and upsampling layers. Many image restoration networks involve downsampling and upsampling layers, and by default, these extend the receptive field in all directions. Consider, e.g., a bilinear 2 × 2 downsampling step followed immediately by a nearest-neighbor 2 × 2 upsampling step. The contents of every 2 × 2 pixel block in the output now correspond to the average of this block in the input, i.e., information has been transferred in every direction within the block. We fix this problem by again applying an offset to the data. It is sufficient to restrict the receptive field for the pair of downsampling and upsampling layers, which means that only one of the layers needs to be modified, and we have chosen to attach the offsets to the downsampling layers. For a 2 × 2 bilinear downsampling layer, we can restrict the receptive field to extend upwards only by padding the input tensor with one row of zeros at top and cropping out the bottom row before performing the actual downsampling operation. In their basic form, blind-spot networks suffer from the inability to utilize the data at the center pixel at test time; yet, clearly, the observed value carries information about the underlying clean signal. For training it is mandatory to disconnect the information flow from pixel position to itself, but there is no such restriction when using the network to restore novel images after it has been trained. We capitalize on this by training the network to predict, based on the context, a distribution of values instead of a single mean prediction, and applying maximum a posteriori estimation at test time. In Bayesian training BID12 BID8 BID5, the network predicts output distributions using a negative log-likelihood loss function. We model the data using multivariate Gaussian distributions: For images with c color components, we have the denoising network output a vector of means µ y and a covariance matrix Σ y for each pixel. For convenience, we parameterize the c × c inverse per-pixel covariance matrix as Σ −1 y = A y T A y, where A y is an upper triangular matrix. This choice ensures that Σ y is positive semidefinite with non-negative diagonal entries, as required for a covariance matrix. For RGB images, the network thus outputs a total of nine values per pixel: the three-component mean µ y and the six nonzero elements of A y.Let f (y; µ y, Σ y) denote the probability density of a multivariate Gaussian distribution N (µ y, Σ y) at target pixel color y, i.e., exp[− DISPLAYFORM0 Under our parameterization, the corresponding negative log-likelihood loss to optimize during training is DISPLAYFORM1 where C is a constant term that can be discarded. Because A y is a triangular matrix, its determinant |A y | is the product of its diagonal elements. To avoid numerical issues, we clamp this determinant to a small positive epsilon ( = 10 −8) so that the logarithm is always well-defined. We assume that all of our images are corrupted by additive uniform Gaussian noise N (0, σ 2 I) with a known standard deviation σ. Using noisy targets means that there is a "baseline" level of noise in the network output distributions that we must discount. Thanks to the blind spot, the network output is independent of the noise in the center pixel, so their (co-)variances are additive. We can therefore calculate Σ p = Σ y − σ 2 I to determine the actual uncertainty Σ p of the network. To avoid negative variances due to approximation errors, the diagonal elements of Σ p can be clamped to zero. Let us now derive our maximum a posteriori (MAP) denoising procedure. For each pixel, our goal is to find the most likely clean valuex given our knowledge of the noisy valuex and the output distribution predicted by the network based on the blind-spot neighborhood Ω. It follows that DISPLAYFORM2 where we have first applied Bayes' theorem to obtain the MAP objective P (x|x)P (x|Ω), and then expressed the associated probabilities as pdfs of Gaussian distributions. In the first term we have exploited the symmetry of the Gaussian distribution, and as the prior term P (x|Ω) we use the prediction of the network with the baseline uncertainty removed. Following BID0, the mean, and consequently the argmax, of this product of two Gaussian distributions iŝ DISPLAYFORM3 For the baseline experiments, as well as for the backbone of our blind-spot networks, we use the same U-Net BID13 architecture as BID9, see their appendix for details. The only differences are that we have layers DEC CONV1A and DEC CONV1B output 96 feature maps like the other convolution layers at the decoder stage, and layer DEC CONV1C is removed. After combining the four receptive field restricted branches, we thus have 384 feature maps. These are fed into three successive 1×1 convolutions with 384, 96, and n output channels, respectively, where n is the number of output components for the network. All convolution layers except the last 1×1 convolution use leaky ReLU with α = 0.1 . All networks were trained using Adam with default parameters BID6, learning rate λ = 0.0003, and minibatch size of 4. As training data, we used random 256×256 crops from the 50K images in the ILSVRC2012 (Imagenet) validation set. The training continued until 1.2M images were shown to the network. All training and test images were corrupted with Gaussian noise, σ = 25. Table 1 shows the denoising quality in dB for the four test datasets used. From the BSD300 dataset we use the 100 validation images only. Similar to BID7, we use the grayscale version of the BSD68 dataset -for this case we train a single-channel (c = 1) denoiser using only the luminance channel of the training images. All our blind-spot noise-to-noise networks use the convolutional architecture (Section 2) and are trained without masking. In BSD68 our simplified L2 variant closely matches the original NOISE2VOID training, suggesting that our network with an architecturally enforced blind spot is approximately as capable as the masking-based network trained by BID7. We see that the denoising quality of our Full setup (Section 3) is on par with baseline of N2N and N2C, and clearly surpasses standard blind-spot denoising (L2) that does not exploit the information in the blind spot. Doing the estimation separately for each color BID9 and BID7. Full is our blind-spot training and denoising method as described in Section 3. Per-comp. is an ablated setup where each color component is treated as an independent univariate Gaussian, highlighting the importance of expressing color outputs as multivariate distributions. L2 refers to training using the standard L2 loss function and ignoring the center pixel when denoising. Columns N2N and N2C refer to NOISE2NOISE training of BID9 and traditional supervised training with clean targets (i.e., noise-to-clean), respectively. Results within 0.05 dB of the best for each dataset are shown in boldface. channel (Per-comp.) performs significantly worse, except in the grayscale BSD68 dataset where it is equivalent to the Full method. FIG1 shows example denoising . Our Full setup produces images that are virtually identical to the N2N baseline both visually and in terms of PSNR. The ablated Per-comp. setup tends to produce color artifacts, demonstrating the shortcomings of the simpler per-component univariate model. Finally, the L2 variant that ignores the center pixel during denoising produces visible checkerboard patterns, some of which can also be seen in the images of BID7. We have shown that self-supervised training -looking at noisy images only, without the benefit of seeing the same image under different noise realizations -is sufficient for learning deep denoising models on par with those that make use of another realization as a training target, be it clean or corrupted. Currently this comes at the cost of assuming pixel-wise independent noise with a known analytic likelihood model. PixelCNNs BID16 a; BID14 generate novel images in a scanline order, one pixel at a time, by conditioning the possible pixel colors using all previous, already generated pixels. The training uses masked convolutions that prevent looking at pixels that would not have been generated yet -one good implementation of masking (van den a) combines a vertical half-space (previous scanlines) with a horizontal line (current scanline). In our application we use four half-spaces to exclude the center pixel only. Regrettably the term "blind spot" has a slightly different meaning in PixelCNNs: van den BID15 uses it to denote valid input pixels that the network in question fails to see due to poor design, whereas we follow the naming convention of BID7 so that a blind spot is always intentional. Applying Bayesian statistics to denoising has a long history. Non-local means BID1, BM3D BID2, and WNNM BID4 identify a group of similar pixel neighborhoods and estimate the center pixel's color from those. This is conceptually similar to our solution, which uses a convolutional network to represent the mapping from neighborhoods to the distilled outputs. Both approaches need only the noisy images, but while the explicit block-based methods determine a small number of neighborhoods from the input image alone, our blind-spot training can implicitly identify and regress an arbitrarily large number of neighborhoods from a collection of noisy training data.

Represent the TLDR for finding the corresponding scientific article

We learn high-quality denoising using only single instances of corrupted images as training data.

Represent the short summary for finding the corresponding scientific article (long)

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.