Question: Represent the scientific article to find a short TLDR

Question: Reinforcement learning methods that continuously learn neural networks by episode generation with game tree search have been successful in two-person complete information deterministic games such as chess, shogi, and Go. However, there are only reports of practical cases and there are little evidence to guarantee the stability and the final performance of learning process. In this research, the coordination of episode generation was focused on. By means of regarding the entire system as game tree search, the new method can handle the trade-off between exploitation and exploration during episode generation. The experiments with a small problem showed that it had robust performance compared to the existing method, Alpha Zero. The that computer programs beat professional human players on chess, shogi and Go was a huge achievement in computer science. In particular, the development of highly general methods totally changed our perspective about two-person complete information deterministic games. Then, has this field already finished? My answer is no. To deal with many games, more robust methods are required to free humans from hyperparameter tuning. Moreover, the challenge to the god of games won't be finished and we want algorithms that can achieve better final performance. This study attempts to bring suggestions for recent achievements in two-player complete information deterministic games from classical game tree search context. More specifically, this is a new approach in which the reinforcement learning system using game tree search itself is handled as game tree search. Turn-based complete information environments this report deals with is called Markov decision process. When the state s t is observed at the time t and the action a t is performed, the environment returns immediate reward r t and the next state s t+1. In general, reinforcement learning is a framework which improves agent's policy p and value estimation v in a given environment. While p and v can be stored in a table for all states in a small environment, function approximation is commonly used for larger problems. Model-free reinforcement learning using a deep neural network (DNN) has been achieving great resutls after the success on several video game environments. This study is aimed at two-person complete information deterministic games. While various modelfree reinforcement learning algorithms can be applied to these games, we can also apply modelbased reinforcement learning because agents can keep the state transition model of the environment. In particular, we can apply forward planning to the two-person complete information deterministic game using the fact that the progress of the game is represented as a tree structure. Ideally, the winning player and the optimal action in any two-player complete information deterministic game in which a finite number of states can appear can be determined by performing minimax search on the game tree within finite time. Actually, it is not necessary to search all states in order to determine the winner at the initial state. We can use alpha-beta search which reduces useless search. However, if the game is not very easy, it is not realistic to complete alpha-beta search that takes exponential time against the size of the game. For this reason, various methods have been developed so far that can be executed in a realistic time and achieve sufficient performance. Roughly, both the algorithmic improvement of the search and the reduction of the search amount by the function approximation have been much effective. For example, Monte Carlo tree search (MCTS), which is the base of this research, performs effective forward search using function approximations. With the good property that the performance improves monotonously the the time being spent is, MCTS is widely used in this field. There are a number of variants of Monte Carlo tree search, but the algorithm PUCT in algorithm 1 2 is described in this report. In that pseudo code, it is assumed that the turn player is changed every turn, and the discount factor is 1, and the immediate reward is obtained only at the terminal state. The Monte Carlo tree search starts from an empty game-tree, and the new node which represents one state is added to expand the game tree in each simulation. After expanding tree, the new state value which approximation function (e.g. neural nets) has returned is fed back to the upper nodes to update the estimated action value of each node. The core of the Monte Carlo tree search is how to decide action response on the game tree, where the bandit algorithm is used. The bandit algorithm is an algorithm for maximizing the cumulative reward in a problem setting where we select one of multiple slot machines that we do not know the true average reward in one turn. It is required for bandit algorithms to handle the trade-off between exploration and exploitation. In the Monte Carlo tree search, by using this bandit algorithm for selection phase at each node, the best action is asymptotically proven and selected at each node, and as a , it can asymptotically find the optimal action. PUCT uses the following PUCB formula 1 as the bandit algorithm. In previous research, the output of Monte Carlo tree search usually only consists the best action or the new policy probability distribution at the root node. However, the new estimated value at the root node (v s in algorithm 2) is also returned in algorithm 2 in order to use it in this research. Although it is a detailed point, there is important point that is heavily related to the experimental of this report, which is that some noise is added to the policy by applying AddNoise function only at the root node shown in the algorithm 1. This is because the root node is recognized not only as bandits but also as best arm identification problems. Therefore the action selection should be more exploratory to find good actions which have been evaluated poorly. In previous method, noises generated by the Dirichlet distribution are added as described in 2. This study follows that. Also from the viewpoint of reinforcement learning, it is useful for agents to be able to perform the game tree search using the true environment model. This is because agents can usually obtain a Algorithm 2: PUCT Data: state s 0, neural net net, total simulation count N Result: posterior estimation of policy and value at s 0 begin while better strategy and a more accurate value estimation by performing the game tree search. Therefore, the game tree search can be seen as a kind of operator to improve there estimates. Various reinforcement learning studies using this property have been done. In particular Alpha Zero and EXpert ITeration are the great success, which enables us to handle various games. These algorithms continuously update approximation function by making training targets by using Monte Carlo tree search as such operator. The procedure in Alpha Zero is followed in this research. The algorithm 3 describes the single thread version of Alpha Zero. Algorithm 3: Alpha Zero (single thread version) Data: total epochs E, games per epoch G, each simulation count N Result: trained neural net In the previous method, the variety of generated episodes is dependent on random action selection after Monte Carlo tree search. However, if there is a state where the policy calculated by the neural network is too sharp, it might be difficult to reverse action evaluation in a limited number of simulations. Therefore, there may be is several weak states even after long-time learning process. The experimental in this study suggest that such phenomena can be actually observed. proposed the preprocess and the post-process to the simulation counts in order to improve explorability at the root node. His suggested that this method can accelarate the learning speed in leaning Go. In this report, from the other perspective, a method to apply Monte Carlo tree search to the whole episode generation procedure is proposed. The output of the Monte Carlo tree search is a new probability distribution and a new state value estimation. Therefore, the Monte Carlo tree search itself can be also regarded as a function of policy and value estimation. The fact that the Monte Carlo tree search itself can be regarded as a function of policy and value estimation implies that we can execute Monte Carlo tree search which uses this function. In the proposed method, the Monte Carlo tree search is performed on the master side that organizes episode generation. This is hereinafter referred as a master game tree. The value of the table 1 is stored in each node of the master game tree. Thus, the values stored in each node in the master game tree has a one-to-one correspondence with the values stored in the ordinal Monte Carlo tree search node. Therefore the master game tree can be implemented as same as the ordinal one. It is desired that the master game tree converges to the optimal strategy asymptotically by assigning episode generation by this master game tree. It seems that theoretical convergence will be guaranteed if same classic bandit algorithm with no prior information is used in each action selection on the master game tree. However, through the success of research on the Monte Carlo tree search so far, function approximation is also used on the master game tree side in the proposed implementation in this report. It means that PUCT is also applied to the master game tree. Since the proposed method uses the master game tree so as to refine the performance of Monte Carlo tree search by the other Monte Carlo tree search, this report named it MbM (MCTS-by-MCTS). The difference between this master game tree search and the ordinal Monte Carlo tree search is that the accuracy of the policy and the value estimation returned by neural nets is increased through proceeding training. Therefore, by updating the p and v, which were fixed after the node creation timing in ordinal PUCT, bandits on the nodes that has already created can behave as if they had created recently. Instead of replacing p or v with new ones, the weighted average between old ones and new ones are calculated after each episode in the proposed method. It helps absorb the fluctuation of the of each trial of the Monte Carlo tree search. This weighting ratio can be increased as the generation of neural networks progresses to emphasize newer estimation. In this report, however, a simple average is used. Specifically, the master game tree is updated as in the algorithm 4. The state value estimate of the newly added node and the difference between the old and the new state value estimation inside the tree are summed and propagated to the upper nodes. Using this update rule, the entire proposed method is discribed in the algorithm 5. The NextPathByMasterTree function, which generates the opening action sequence by descending the master game tree, is almost the same as the operation of descending the tree in the algorithm 1. When going down the master game tree, the noise is always added to p s in the algorithm 1 in the regular version of the proposed method. This point will be discussed at the later part of this report. In the experiment, in the tic-tac-toe where the solution is already sought for the small size, the existing method AlphaZero and the proposed method learned for 20,000 battles, The following points were compared by the obtained neural network policy. • Ratio of defeating against random players • Ratio of defeating to perfect players Tic-tac-toe is obviously very simple task, and it is easy to obtain a perfect strategy by minimax search. Only by function approximation using neural network, however, it may be necessary to generate a certain number of episodes and train from them to obtain good policy. Still, it is so easy problem that we desire that neural nets can output the optimal strategy. The experiments were performed with the following settings. In the proposed method, in the bandit on the master game tree side, the same as the root node on the episode generation side. We tried two ways of adding Dirichlet noise to the policy and not adding it. Especially when adding Dirichlet noise to the policy, if the other parameters are the same in Alpha Zero and the MbM, episode generation become more exploratory in the proposed method than Alpha Zero. The performance can vary greatly depending on the effect (actually shown in the experimental in this paper). Therefore, the initial values of episode generation temperature parameters were varied as 0.3, 0.6, 1.2, and 2.4, respectively into both the previous method and the proposed method. Other experimental settings and hyperparameters are shown below. For details, please refer to the online experimental code. 1. • attenuation rate of generation temperature: 0.8 • C in PUCB: 2.0 • noise to the policy α = 0.1, ϵ = 0.25 • the number of epochs: 200 • the number of episodes per epoch: 100 • input feature: 2 planes (the position mine, the position of opponents) • detail of neural nets: (3x3 convolution with 32 filters + BN), (Wide Resnet x 4 layers), (1x1 convolution with 2 filters + BN + fully connected + softmax in policy head), (1x1 convolution with 2filters + BN + fully connected + tanh) • detail of training data: (batch size 32 x the number of episodes before x 50 loop) per epoch, • detail of optimization: SGD (learning rate=1e-3, momentum=0.75 weight decay=1e-4) The experimental are plotted in the figure 1 2, comparing the temperature conditions for each opponent (random player, perfect player) in the previous method AlphaZero and the proposed method MbM. From the of AlphaZero in the FIg. 1 2, Alpha Zero has not reached the optimal strategy within 200 epochs when the temperature of episode generation is low. In comparison, the of MbM in Fig. 1 2 shows that almost optimal strategies seem to be obtained under all conditions. Compared through those , the performance of proposed methods are equal to or better than the best one of AlphaZero. In the of the previous experiment, the proposed method MbM showed good performance. However, compared to the of Alpha Zero, there seems to be the possibility that the difference of performance caused only by the explorability of episode generation by adding noise to the policy when going down the master game tree. Additional experiment was done with the following two modified versions of the proposed method under the temperature is low (t = 0.3 or t = 0.6). • MbM-NoNoise: no noise is added to the policy • MbM-Relaxation: applying conversion to the policy in order to soften sharpness of it. In the latter one, the policy p is modified as p ←− (p + 0.1)/(∑ (p + 0.1)). The figure shows learning up to 100 epochs under each condition and comparing the with those in previous experiment. The in Fig. 3 show that MbM-NoNoise, which does not add noise, is even worse than Alpha Zero. MbM-UniformRelaxation, which the fixed amount of diversity is added to the policy, was not so good as MbM but pretty better than Alpha Zero. This point will be discussed in the final section. In this study, we examined a very simple task, Tic-tac-toe. First of all, it was shown that obtaining the optimal strategy is sometimes difficult depending on the parameters. The suggest that reinforcement learning methods like Alpha Zero often suffer from naiveness against exploration. In the proposed method, it is possible to vary the beginning of the game in the episode generation by the master game tree. The suggest that the proposed method has ability to control the search for the beginning of the game by adding proper noise. On the other hand, when PUCT using the strategy as it is applied to the master game tree (MbMNoNoise), the performance was lower than the baseline. The reason of this is that the policy has converged earlier than the effect of exploration in the master game tree. Due to this point, it was not effective. In this report, PUCT is applied to the master game tree as same as ordinal game tree. However, it is necessary to examine a mechanism that makes it more exploratory. Lastly, in this study, we verified only one of the simplest games, Tic-tac-toe. From the experimental in this paper, it is expected that the proposed method can produce robust with respect to temperature parameters even for larger games. It will also be necessary to verify whether the speed of improvement in real time is better that previous methods. I hope that the combination of tree search and reinforcement learning will be used for a wider range of domains if there exists the method in which both stableness and speed are better performance.

Apply Monte Carlo Tree Search to episode generation in Alpha Zero

Adaptation of an RL agent in a target environment with unknown dynamics is fast and safe when we transfer prior experience in a variety of environments and then select risk-averse actions during adaptation.


Question: represent the scientific article to find a short tldr:

Question: Message-passing neural networks (MPNNs) have been successfully applied in a wide variety of applications in the real world. However, two fundamental weaknesses of MPNNs' aggregators limit their ability to represent graph-structured data: losing the structural information of nodes in neighborhoods and lacking the ability to capture long-range dependencies in disassortative graphs. Few studies have noticed the weaknesses from different perspectives. From the observations on classical neural network and network geometry, we propose a novel geometric aggregation scheme for graph neural networks to overcome the two weaknesses. The behind basic idea is the aggregation on a graph can benefit from a continuous space underlying the graph. The proposed aggregation scheme is permutation-invariant and consists of three modules, node embedding, structural neighborhood, and bi-level aggregation. We also present an implementation of the scheme in graph convolutional networks, termed Geom-GCN, to perform transductive learning on graphs. Experimental show the proposed Geom-GCN achieved state-of-the-art performance on a wide range of open datasets of graphs. Message-passing neural networks (MPNNs), such as GNN , ChebNet , GG-NN , GCN , are powerful for learning on graphs with various applications ranging from brain networks to online social network . In a layer of MPNNs, each node sends its feature representation, a "message", to the nodes in its neighborhood; and then updates its feature representation by aggregating all "messages" received from the neighborhood. The neighborhood is often defined as the set of adjacent nodes in graph. By adopting permutation-invariant aggregation functions (e.g., summation, maximum, and mean), MPNNs are able to learn representations which are invariant to isomorphic graphs, i.e., graphs that are topologically identical. Although existing MPNNs have been successfully applied in a wide variety of scenarios, two fundamental weaknesses of MPNNs' aggregators limit their ability to represent graph-structured data. Firstly, the aggregators lose the structural information of nodes in neighborhoods. Permutation invariance is an essential requirement for any graph learning method. To meet it, existing MPNNs adopt permutation-invariant aggregation functions which treat all "messages" from neighborhood as a set. For instance, GCN simply sums the normalized "messages" from all one-hop neighbors . Such aggregation loses the structural information of nodes in neighborhood because it does not distinguish the "messages" from different nodes. Therefore, after such aggregation, we cannot know which node contributes what to the final aggregated output. Without modeling such structural information, as shown in and, the existing MPNNs cannot discriminate between certain non-isomorphic graphs. In those cases, MPNNs may map non-isomorphic graphs to the same feature representations, which is obviously not desirable for graph representation learning. Unlike MPNNs, classical convolutional neural networks (CNNs) avoid this problem by using aggregators (i.e., convolutional filters) with a structural receiving filed defined on grids, i.e., a Euclidean space, and are hence able to distinguish each input unit. As shown by our experiments, such structural information often contains clues regarding topology patterns in graph (e.g., hierarchy), and should be extracted and used to learn more discriminating representations for graph-structured data. Secondly, the aggregators lack the ability to capture long-range dependencies in disassortative graphs. In MPNNs, the neighborhood is defined as the set of all neighbors one hop away (e.g., GCN), or all neighbors up to r hops away (e.g., ChebNet). In other words, only messages from nearby nodes are aggregated. The MPNNs with such aggregation are inclined to learn similar representations for proximal nodes in a graph. This implies that they are probably desirable methods for assortative graphs (e.g., citation networks and community networks ) where node homophily holds (i.e., similar nodes are more likely to be proximal, and vice versa), but may be inappropriate to the disassortative graphs where node homophily does not hold. For example, shows disassortative graphs where nodes of the same class exhibit high structural similarity but are far apart from each other. In such cases, the representation ability of MPNNs may be limited significantly, since they cannot capture the important features from distant but informative nodes. A straightforward strategy to address this limitation is to use a multi-layered architecture so as to receive "messages" from distant nodes. For instance, due to the localized nature of convolutional filters in classical CNNs, a single convolutional layer is similarly limited in its representational ability. CNNs typically use multiple layers connected in a hierarchical manner to learn complex and global representations. However, unlike CNNs, it is difficult for multi-layer MPNNs to learn good representations for disassortative graphs because of two reasons. On one hand, relevant messages from distant nodes are mixed indistinguishably with a large number of irrelevant messages from proximal nodes in multi-layer MPNNs, which implies that the relevant information will be "washed out" and cannot be extracted effectively. On the other hand, the representations of different nodes would become very similar in multi-layer MPNNs, and every node's representation actually carries the information about the entire graph. In this paper, we overcome the aforementioned weaknesses of graph neural networks starting from two basic observations: i) Classical neural networks effectively address the similar limitations thanks to the stationarity, locality, and compositionality in a continuous space; ii) The notion of network geometry bridges the gap between continuous space and graph . Network geometry aims to understand networks by revealing the latent continuous space underlying them, which assumes that nodes are sampled discretely from a latent continuous space and edges are established according to their distance. In the latent space, complicated topology patterns in graphs can be preserved and presented as intuitive geometry, such as subgraph , community (, and hierarchy (; . Inspired by those two observations, we raise an enlightening question about the aggregation scheme in graph neural network. • Can the aggregation on a graph benefit from a continuous latent space, such as using geometry in the space to build structural neighborhoods and capture long-range dependencies in the graph? To answer the above question, we propose a novel aggregation scheme for graph neural networks, termed the geometric aggregation scheme. In the scheme, we map a graph to a continuous latent space via node embedding, and then use the geometric relationships defined in the latent space to build structural neighborhoods for aggregation. Also, we design a bi-level aggregator operating on the structural neighborhoods to update the feature representations of nodes in graph neural networks, which are able to guarantee permutation invariance for graph-structured data. Compared with exist-ing MPNNs, the scheme extracts more structural information of the graph and can aggregate feature representations from distant nodes via mapping them to neighborhoods defined in the latent space. We then present an implementation of the geometric aggregation scheme in graph convolutional networks, which we call Geom-GCN, to perform transductive learning, node classification, on graphs. We design particular geometric relationships to build the structural neighborhood in Euclidean and hyperbolic embedding space respectively. We choose different embedding methods to map the graph to a suitable latent space for different applications, where suitable topology patterns of graph are preserved. Finally, we empirically validate and analyze Geom-GCN on a wide range of open datasets of graphs, and Geom-GCN achieved the state-of-the-art . In summary, the contribution of this paper is three-fold: i) We propose a novel geometric aggregation scheme for graph neural network, which operates in both graph and latent space, to overcome the aforementioned two weaknesses; ii) We present an implementation of the scheme, Geom-GCN, for transductive learning in graph; iii) We validate and analyze Geom-GCN via extensive comparisons with state-of-the-art methods on several challenging benchmarks. In this section, we start by presenting the geometric aggregation scheme, and then outline its advantages and limitations compared to existing works. As shown in Fig. 1, the aggregation scheme consists of three modules, node embedding (panel A1 and A2), structural neighborhood (panel B1 and B2), and bi-level aggregation (panel C). We will elaborate on them in the following. Figure 1: An illustration of the geometric aggregation scheme. A1-A2 The original graph is mapped to a latent continuous space. B1-B2 The structural neighborhood. All adjacent nodes lie in a small region around a center node in B1 for visualization. In B2, the neighborhood in the graph contains all adjacent nodes in graph; the neighborhood in the latent space contains the nodes within the dashed circle whose radius is ρ. The relational operator τ is illustrated by a colorful 3 × 3 grid where each unit is corresponding to a geometric relationship to the red target node. C Bi-level aggregation on the structural neighborhood. Dashed and solid arrows denote the low-level and high-level aggregation, respectively. Blue and green arrows denote the aggregation on the neighborhood in the graph and the latent space, respectively. A. Node embedding. This is a fundamental module which maps the nodes in a graph to a latent continuous space. Let G = (V, E) be a graph, where each node v ∈ V has a feature vector x v and each edge e ∈ E connects two nodes. Let f: v → z v be a mapping function from a node in graph to a representation vector. Here, z v ∈ R d can also be considered as the position of node v in a latent continuous space, and d is the number of dimensions of the space. During the mapping, the structure and properties of graph are preserved and presented as the geometry in the latent space. For instance, hierarchical pattern in graph is presented as the distance to the original in embedding hyperbolic space . One can employ various embedding methods to infer the latent space (; . B. Structural neighborhood. Based on the graph and the latent space, we then build a structural neighborhood, N (v) = ({N g (v), N s (v)}, τ ), for the next aggregation. The structural neighborhood consists of a set of neighborhood {N g (v), N s (v)}, and a relational operator on neighborhoods τ. The neighborhood in the graph, N g (v) = {u|u ∈ V, (u, v) ∈ E}, is the set of adjacent nodes of v. The neighborhood in the latent space, N s (v) = {u|u ∈ V, d(z u, z v) < ρ}, is the set of nodes from which the distance to v is less than a pre-given parameter ρ. The distance function d(·, ·) depends on the particular metric in the space. Compared with N g (v), N s (v) may contain nodes which are far from v in the graph, but have a certain similarity with v, and hence are mapped together with v in the latent space though preserving the similarity. By aggregating on such neighborhood N s (v), the long-range dependencies in disassortative graphs can be captured. The relational operator τ is a function defined in the latent space. It inputs an ordered position pair (z v, z u) of nodes v and u, and outputs a discrete variable r which indicates the geometric relationship from v to u in the latent space. where R is the set of the geometric relationships. According to the particular latent space and application, r can be specified as an arbitrary geometric relationship of interest. A requirement on τ is that it should guarantee that each ordered position pair has only one geometric relationship. For example, τ is illustrated in Fig. 1B by a colorful 3 × 3 grid in a 2-dimensional Euclidean space, in which each unit is corresponding to a geometric relationship to node v. C. Bi-level aggregation. With the structural neighborhood N (v), we propose a novel bi-level aggregation scheme for graph neural network to update the hidden features of nodes. The bi-level aggregation consists of two aggregation functions and operates in a neural network layer. It can extract effectively structural information of nodes in neighborhoods as well as guarantee permutation invariance for graph. Let h In the low-level, the hidden features of nodes that are in the same neighborhood i and have the same geometric relationship r are aggregated to a virtual node via the aggregation function p. The features of the virtual node are e v,l+1 (i,r), and the virtual node is indexed by (i, r) which is corresponding to the combination of a neighborhood i and a relationship r. It is required to adopt a permutation-invariant function for p, such as an L p -norm (the choice of p = 1, 2, or ∞ in average, energy, or max pooling). The low level aggregation is illustrated by dashed arrows in Fig. 1C. In the high-level, the features of virtual nodes are further aggregated by function q. The inputs of function q contain both the features of virtual nodes e v,l+1 (i,r) and the identity of virtual nodes (i, r). That is, q can be a function that take an ordered object as input, e.g., concatenation, to distinguish the features of different virtual nodes, thereby extracting the structural information in the neighborhoods explicitly. The output of high-level aggregation is a vector m, are given by a non-linear transform, wherein W l is a learnable weight matrix on the l-th layer shared by all nodes, and σ(·) is a non-linear activation function, e.g., a ReLU. Permutation invariance is an essential requirement for aggregators in graph neural networks. Thus, we then prove that the proposed bi-level aggregation, Eq. 1, is able to guarantee invariance for any permutation of nodes. We firstly give a definition for permutation-invariant mapping of graph. Definition 1. Let a bijective function ψ: V → V be a permutation for nodes, which renames v ∈ V as ψ(v) ∈ V. Let V and E be the node and edge set after a permutation ψ, respectively. A mapping of graph, φ(G), is permutation-invariant if, given any permutation ψ, we have Proof. Let G be an isomorphic graph of G after a permutation ψ, as defined in Definition 1. If φ 2 (G) is permutation-invariant, we have φ 2 (G) = φ 2 (G). Therefore, the entire composite function Theorem 1. Given a graph G = (V, E) and its structural neighborhood N (v), ∀v ∈ V, the bi-level aggregation, Eq. 1, is a permutation-invariant mapping of graph. Proof. The bi-level aggregation, Eq. 1, is a composite function, where the low-level aggregation is the input of the high-level aggregation. Thus, Eq. 1 is permutation-invariant if the low-level aggregation is permutation-invariant according to Lemma 1. We then prove that the low-level aggregation is permutation-invariant. The low-level aggregation consists of 2×|R| sub-aggregations, each of which is corresponding to the nodes in a neighborhood i and with a relationship r to v. Firstly, the input of each sub-aggregations is permutation-invariant because both i ∈ {g, s} and r ∈ R are determined by the given structural neighborhood N (v), ∀v ∈ V, which is constant for any permutation. Secondly, Eq. 1 adopts a permutation-invariant aggregation function p for the sub-aggregations. Thus the low-level aggregation is permutation-invariant. We now discuss how the proposed geometric aggregation scheme overcomes the two aforementioned weaknesses, i.e., how it effectively models the structural information and captures the long-range dependencies, in comparison to some closely related works. To overcome the first weakness of MPNNs, i.e., losing the structural information of nodes in neighborhoods, the proposed scheme explicitly models the structural information by exploiting the geometric relationship between nodes in latent space and then extracting the information effectively by using the bi-level aggregations. In contrast, several existing works attempt to learn some implicit structure-like information to distinguish different neighbors when aggregating features. For example, GAT , LGCL and GG-NN learn weights on "messages" from different neighbors by using attention mechanisms and node and/or edge attributes. CCN utilizes a covariance architecture to learn structure-aware representations. The major difference between these works and ours is that we offer an explicit and interpretable way to model the structural information of nodes in neighborhood, with the assistance of the geometry in a latent space. We note that our work is orthogonal with existing methods and thus can be readily incorporated to further improve their performance. In particular, we exploit geometric relationships from the aspect of graph topology, while other methods focus on that of feature representation-the two aspects are complementary. For the second weakness of MPNNs, i.e., lacking the ability to capture long-range dependencies, the proposed scheme models the long-range dependencies in disassortative graphs in two different ways. First of all, the distant (but similar) nodes in the graph can be mapped into a latent-spacebased neighborhood of the target node, and then their useful feature representations can be used for aggregations. This way depends on an appropriate embedding method, which is able to preserve the similarities between the distant nodes and the target node. On the other hand, the structural information enables the method to distinguish different nodes in a graph-based neighborhood (as mentioned above). The informative nodes may have some special geometric relationships to the target node (e.g., a particular angle or distance), whose relevant features hence will be passed to the target node with much higher weights, compared to the uninformative nodes. As a , the long-range dependencies are captured indirectly through the whole message propagation process in all graph-based neighborhoods. In literature, a recent method JK-Nets captures the long-range dependencies by skipping connections during feature aggregations. In literature, and construct several non-isomorphic example graphs that cannot be distinguished by the aggregators (e.g., mean and maximum) in existing MPNNs. We present a case study to illustrate how to distinguish the non-isomorphic example graphs once the structural neighborhood is applied. We take two non-isomorphic graphs in as an example, where each node has the same feature a and after any mapping f (a) remains the same across all nodes, as shown in Fig. 2 (left). Then the aggregator, e.g., mean or maximum, over f (a) remains f (a), and hence the final representations of the nodes are the same. That is, mean and maximum aggregators fail to distinguish the two different graphs. In contrast, the two graphs become distinguishable once we apply a structural neighborhood in aggregation. With the structural neighborhood, the nodes have different geometric relationships to the center node V 1 in the structural neighborhood, as shown in Fig. 2 (right). Taking aggregation for V 1 as an example, we can adopt different mapping function f r, r ∈ R to the neighbors with different geometric relationship r to V 1. Then, the aggregator in two graph have different inputs, {f 2 (a), f 8 (a)} in the left graph and {f 2 (a), f 7 (a), f 9 (a)} in the right graph. Finally, the aggregator (mean or maximum) will output different representations for the node V 1 in the two graphs, thereby distinguishing the topological difference between the two graphs. In this section, we present Geom-GCN, a specific implementation of the geometric aggregation scheme in graph convolutional networks, to perform transductive learning in graphs. To implement the general aggregation scheme, one needs to specify its three modules: node embedding, structural neighborhood, and bi-level aggregation function. Node embedding is the fundamental. As shown in our experiments, a common embedding method which only preserves the connection and distance pattern in a graph can already benefit the aggregation. For particular applications, one can specify embedding methods to create suitable latent spaces where particular topology patterns (e.g., hierarchy) are preserved. We employ three embedding methods, Isomap , Poincare embedding , and struc2vec , which in three Geom-GCN variants: Geom-GCN-I, Geom-GCN-P, and Geom-GCN-S. Isomap is a widely used isometry embedding method, by which distance patterns (lengths of shortest paths) are preserved explicitly in the latent space. Poincare embedding and struc2vec can create particular latent spaces that preserve hierarchies and local structures in a graph, respectively. We use an embedding space of dimension 2 for ease of explanation. The structural neighborhood N (v) = ({N g (v), N s (v)}, τ ) of node v includes its neighborhoods in both the graph and latent space. The neighborhood-in-graph N g (v) consists of the set of v's adjacent nodes in the graph, and the neighborhood-in-latent-space N s (v) those nodes whose distances to v are less than a parameter ρ in the latent space. We determine ρ by increasing ρ from zero until the average cardinality of N s (v) equals to that of N g (v), ∀v ∈ V -i.e., when the average neighborhood sizes in the graph and latent spaces are the same. We use Euclidean distance in the Euclidean space. In the hyperbolic space, we approximate the geodesic distance between two nodes via their Euclidean distance in the local tangent plane. Here we simply implement the geometric operator τ as four relationships of the relative positions between two nodes in a 2-D Euclidean or hyperbolic space. Particularly, the relationship set R = {upper left, upper right, lower left, lower right}, and a τ (z v, z u) is given by Table 1. Note that, we adopt the rectangular coordinate system in the Euclidean space and angular coordinate in the hyperbolic space. By this way, the relationship "upper" indicates the node nearer to the origin and thus lie in a higher level in a hierarchical graph. One can design a more sophisticated operator τ, such as borrowing the structure of descriptors in manifold geometry , thereby preserving more and richer structural information in neighborhood. lower left lower right Finally, to implement the bi-level aggregation, we adopt the same summation of normalized hidden features as GCN as the aggregation function p in the low-level aggregation, where deg(v) is the degree of node v in graph, and δ(·, ·) is a Kronecker delta function that only allows the nodes with relationship r to v to be included. The features of all virtual nodes e v,l+1 (i,r) are further aggregated in the high-level aggregation. The aggregation function q is a concatenation || for all layers except the final layer, which uses mean for its aggregation function. Then, the overall bi-level aggregation of Geom-GCN is given by where we use ReLU as the non-linear activation function σ(·) and W l is the weight matrix to estimate by backpropagation. We validate Geom-GCN by comparing Geom-GCN's performance with the performance of Graph Convolutional Networks (GCN) and Graph Attention Networks (GAT) . Two state-of-the-art graph neural networks, on transductive node-label classification tasks on a wide variety of open graph datasets. We utilize nine open graph datasets to validate the proposed Geom-GCN. An overview summary of characteristics of the datasets is given in Table 2. Citation networks. Cora, Citeseer, and Pubmed are standard citation network benchmark datasets . In these networks, nodes represent papers, and edges denote citations of one paper by another. Node features are the bag-of-words representation of papers, and node label is the academic topic of a paper. WebKB. WebKB 1 is a webpage dataset collected from computer science departments of various universities by Carnegie Mellon University. We use the three subdatasets of it, Cornell, Texas, and Wisconsin, where nodes represent web pages, and edges are hyperlinks between them. Node features are the bag-of-words representation of web pages. The web pages are manually classified into the five categories, student, project, course, staff, and faculty. Actor co-occurrence network. This dataset is the actor-only induced subgraph of the film-directoractor-writer network . Each nodes correspond to an actor, and the edge between two nodes denotes co-occurrence on the same Wikipedia page. Node features correspond to some keywords in the Wikipedia pages. We classify the nodes into five categories in term of words of actor's Wikipedia. Wikipedia network. Chameleon and squirrel are two page-page networks on specific topics in Wikipedia . In those datasets, nodes represent web pages and edges are mutual links between pages. And node features correspond to several informative nouns in the Wikipedia pages. We classify the nodes into five categories in term of the number of the average monthly traffic of the web page. As mentioned in Section 3, we construct three Geom-GCN variants by using three embedding methods, Isomap (Geom-GCN-I), Poincare (Geom-GCN-P), and struc2vec (Geom-GCN-S). We specify the dimension of embedding space as two, and use the relationship operator τ defined in Table 1, and apply mean and concatenation as the low-and high-level aggregation function, respectively. With the structural neighborhood, we perform a hyper-parameter search for all models on validation set. For fairness, the size of search space for each method is the same. The searching hyperparameters include number of hidden unit, initial learning rate, weight decay, and dropout. We fix the number of layer to 2 and use Adam optimizer for all models. We use ReLU as the activation function for Geom-GCN and GCN, and ELU for GAT. The final hyper-parameter setting is dropout of p = 0.5, initial learning rate of 0.05, patience of 100 epochs, weight decay of 5E-6 (WebKB datasets) or 5E-5 (the other all datasets). In GCN, the number of hidden unit is 16 (Cora), 16 (Citeseer), 64 (Pubmed), 32 (WebKB), 48 (Wikipedia), and 32 (Actor). In Geom-GCN, the number of hidden unit is 8 times as many as the number in GCN since Geom-GCN has 8 virtual nodes. For each attention head in GAT, the number of hidden unit is 8 (Citation networks), 32 (WebKB), 48 (Wikipedia), and 32 (Actor). GAT has 8 attention heads in layer one and 8 (Pubmed) or 1 (the all other datasets) attention heads in layer two. For all graph datasets, we randomly split nodes of each class into 60%, 20%, and 20% for training, validation and testing. With the hyper-parameter setting, we report the average performance of all models on the test sets over 10 random splits. Results are summarized in Table 3. The reported numbers denote the mean classification accuracy in percent. In general, Geom-GCN achieves state-of-the-art performance. The best performing method is highlighted. From the , Isomap embedding (Geom-GCN-I) which only preserves the connection and distance pattern in graph can already benefit the aggregation. We can also specify an embedding method to create a suitable latent space for a particular application (e.g., disassortative graph or hierarchical graph), by doing which a significant performance improvement is achieved (e.g., Geom-GCN-P). The proposed Geom-GCN aggregates "message" from two neighborhoods which are defined in graph and latent space respectively. In this section, we present an ablation study to evaluate the contribution from each neighborhood though constructing new Geom-GCN variants with only one neighborhood. For the variants with only neighborhood in graph, we use "g" as a suffix of their name (e.g., Geom-GCN-I-g), and use suffix "s" to denote the variants with only neighborhood in latent space (e.g., Geom-GCN-I-s). Here we set GCN as a baseline so that the contribution can be measured via the performance improvement comparing with GCN. The are summarized in Table 4, where positive improvement is denoted by an up arrow ↑ and negative improvement by a down arrow ↓. The best performing method is highlighted. We also design an index denoted by β to measure the homophily in a graph, Number of v's neighbors who have the same label as v Number of v's neighbors. A large β value implies that the homophily, in term of node label, is strong in a graph, i.e., similar nodes tend to connect together. From Table 4, one can see that assortative graphs (e.g., citation networks) have a much larger β than disassortative graphs (e.g., WebKB networks). Table 4 exhibits three interesting patterns: i) Neighborhoods in graph and latent space both benefit the aggregation in most cases; ii) Neighborhoods in latent space have larger contributions in disassortative graphs (with a small β) than assortative ones, which implies relevant information from disconnected nodes is captured effectively by the neighborhoods in latent space; iii) To our surprise, several variants with only one neighborhood (in Table 4) achieve better performances than the variants with two neighborhoods (in Tabel 3). We think the reason is that Geom-GCN with two neighborhoods aggregate more irrelevant "messages" than Geom-GCN with only one neighborhood, and the irrelevant "messages" adversely affect the performance. Thus, we believe an attention mechanism can alleviate this issue-which we will study as future work. The structural neighborhood in Geom-GCN is very flexible, where one can combine arbitrary embedding space. To study which combination of embedding spaces is desirable, we construct new Geom-GCN variants by adopting neighborhoods built by different embedding space. For the variants adopted Isomap and poincare embedding space to build neighborhood in graph and in latent space respectively, we use Geom-GCN-IP to denote it. The naming rule is the same for other combinations. The performances of all variants are summarized in Table 5. One can observe that several combinations achieve better performance than Geom-GCN with neighborhoods built by only one embedding space (in Table 3); and there are also many combinations that have bad performance. Thus, we think it's significant future work to design an end-to-end framework that can automatically determine the right embedding spaces for Geom-GCN. for each virtual node (i.e., (i, r)), and 2|R| is the number of virtual nodes. Geom-GCN has 2|R| times complexity than GCN whose time complexity is O(n × m). We also compare the real running time (500 epochs) of GCN, GAT, and Geom-GCN on all datasets with the hyper-parameters described in Section 4.2. Results are shown in Fig. 3 (a). One can see that GCN is the fastest, and GAT and Geom-GCN are on the same level. An important future work is to develop accelerating technology so as to solve the scalability of Geom-GCN. (a) (b) Figure 3: (a) Running time comparison. GCN, GAT, and Geom-GCN both run 500 epochs, and y axis is the log seconds. GCN is the fastest, and GAT and Geom-GCN are on the same level. (b) A visualization for the feature representations of Cora obtained from Geom-GCN-P in a 2-D space. Node colors denote node labels. There are two obvious patterns, nodes with the same label exhibit a spatial clustering and all nodes distribute radially. The radial pattern indicates graph's hierarchy learned by Poincare embedding. To study what patterns are learned in the feature representations of node by Geom-GCN, we visualize the feature representations extracted by the last layer of Geom-GCN-P on Cora dataset by mapping it into a 2-D space though t-SNE , as shown in Fig. 3 (b). In the figure, the nodes with the same label exhibit spatial clustering, which could shows the discriminative power of Geom-GCN. That all nodes distribute radially in the figure indicates the proposed model learn graph's hierarchy by Poincare embedding. We tackle the two major weaknesses of existing message-passing neural networks over graphslosses of discriminative structures and long-range dependencies. As our key insight, we bridge a discrete graph to a continuous geometric space via graph embedding. That is, we exploit the principle of convolution: spatial aggregation over a meaningful space-and our approach thus extracts or "recovers" the lost information (discriminative structures and long-range dependencies) in an embedding space from a graph. We proposed a general geometric aggregation scheme and instantiated it with several specific Geom-GCN implementations, and our experiments validated clear advantages over the state-of-the-art. As future work, we will explore techniques for choosing a right embedding method-depending not only on input graphs but also on target applications, such as epidemic dynamic prediction on social contact network .

For graph neural networks, the aggregation on a graph can benefit from a continuous space underlying the graph.

Represent the short summary for finding the corresponding scientific article (long)

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question: represent the scientific article to find a short tldr

Question: We consider the task of program synthesis in the presence of a reward function over the output of programs, where the goal is to find programs with maximal rewards. We introduce a novel iterative optimization scheme, where we train an RNN on a dataset of K best programs from a priority queue of the generated programs so far. Then, we synthesize new programs and add them to the priority queue by sampling from the RNN. We benchmark our algorithm called priority queue training (PQT) against genetic algorithm and reinforcement learning baselines on a simple but expressive Turing complete programming language called BF. Our experimental show that our deceptively simple PQT algorithm significantly outperforms the baselines. By adding a program length penalty to the reward function, we are able to synthesize short, human readable programs. Automatic program synthesis is an important task with many potential applications. Traditional approaches (e.g., BID29 ; BID1) typically do not make use of machine learning and therefore require domain specific knowledge about the programming languages and hand-crafted heuristics to speed up the underlying combinatorial search. To create more generic programming tools without much domain specific knowledge, there has been a surge of recent interest in developing neural models that facilitate some form of memory access and symbolic reasoning (e.g., BID37 ; BID33 ; BID21 ; BID49 ;). Despite several appealing contributions, none of these approaches is able to synthesize source code in an expressive programming language. More recently, there have been several successful attempts at using neural networks to explicitly induce programs from input-output examples BID8 BID2 BID35 and even from unstructured text BID35, but often using restrictive programming syntax and requiring supervisory signal in the form of ground-truth programs or correct outputs. By contrast, we advocate the use of an expressive programming language called BF 1, which has a simple syntax, but is Turing complete. Moreover, we aim to synthesize programs under the reinforcement learning (RL) paradigm, where only a solution checker is required to compute a reward signal. Furthermore, one can include a notion of code length penalty or execution speed into the reward signal to search for short and efficient programs. Hence, the problem of program synthesis based on reward is more flexible than other formulations in which the desired programs or correct outputs are required during training. To address program synthesis based on a reward signal, we study two different approaches. The first approach is a policy gradient (PG) algorithm BID44, where we train a recurrent neural network (RNN) to generate programs one token at a time. Then, the program is executed and scored, and a reward feedback is sent back to the RNN to update its parameters such that over time better programs are produced. The second approach is a deceptively simple optimization algorithm called priority queue training (PQT). We keep a priority queue of K best programs seen during training and train an RNN with a log-likelihood objective on the top K programs in the queue. We then sample new programs from the RNN, update the queue, and iterate. We also compare against a genetic algorithm (GA) baseline which has been shown to generate BF programs BID3. Surprisingly, we find that the PQT approach significantly outperforms the GA and PG methods. We assess the effectiveness of our method on the BF programming language. The BF language is Turing complete, while comprising only 8 operations. The minimalist syntax of the BF language makes it easier to generate a syntactically correct program, as opposed to more higher level languages. We consider various string manipulation, numerical, and algorithmic tasks. Our demonstrate that all of the search algorithms we consider are capable of finding correct programs for most of the tasks, and that our method is the most reliable in that it finds solutions on most random seeds and most tasks. The key contributions of the paper include,• We propose a learning framework for program synthesis where only a reward function is required during training (the ground-truth programs or correct outputs are not needed).Further, we advocate to use a simple and expressive programming language, BF, as a benchmark environment for program synthesis (see also BID3).• We propose an effective search algorithm using a priority queue and an RNN.• We propose an experimental methodology to compare program synthesis methods including genetic algorithm and policy gradient. Our methodology measures the success rates of each synthesis method on average and provides a standard way to tune the hyper-parameters. With this methodology, we find that a recurrent network trained with priority queue training outperforms the baselines. Our method shares the same goal with traditional techniques in program synthesis and inductive programming BID40 BID7 BID29 BID1. These techniques have found many important applications in practice, ranging from education to programming assistance BID14. In machine learning, probabilistic program induction has been used successfully in many settings, such as learning to solve simple Q&A BID26, and learning with very few examples BID24.There has been a surge of recent interest in using neural networks to induce and execute programs either implicitly or explicitly BID12 BID48 BID20 BID21 BID23 BID33 BID37 BID0 BID2 BID6 BID11 BID22 BID34 BID39 BID49 BID27 BID5 BID9 BID10 BID17 BID15 BID30 BID35 BID36 BID42 BID47. For example, there have been promising on the task of binary search BID31, sorting an array of numbers BID37 BID9, solving simple Q&A from tables BID33 BID5, visual Q&A BID0 BID17, filling missing values in tables BID10, and querying tables. There are several key components that highlight our problem formulation in the context of previous work. First, our approach uses a Turing complete language instead of a potentially restricted domain-specific language. Second, it does not need existing programs or even the stack-trace of existing programs. Third, it only assumes the presence of a verifier that scores the outputs of hypothesis programs, but does not need access to correct outputs. This is important for domains where finding the correct outputs is hard but scoring the outputs is easy. Finally, our formulation does not need to modify the internal workings of the programming language to obtain a differentiable error signal. The PG approach adopted in this paper for program synthesis is closely related to neural architecture search BID50 and neural combinatorial optimization BID4, where variants of PG are used to train an RNN and a pointer network BID43 to perform combinatorial search. BID31 applies PG to program synthesis, but they differ from us in that they train RNNs that implicitly model a program by consuming inputs and emitting machine instructions as opposed to explicit programs. Our PG baseline resembles such previous techniques. The PQT algorithm presented here is partly inspired by, where they use a priority queue of top-K programs to augment PG with off-policy training. PQT also bears resemblance to the cross-entropy method (CEM), a reinforcement learning technique which has been used to play games such as Tetris BID41.Our use of BF programming language enables a comparison between our technique and a concurrent work by BID3 on the use of genetic algorithms for program synthesis in the BF language. However, our method for program synthesis has important benefits over BID3 including better performance and the potential for transfer learning, which is possible with neural networks BID19. We also make the observation that PQT alone is stable and effective, without needing to use PG. We implement a generative model of programs as an RNN that emits a strings of BF language one character at a time. Figure 2 depicts the RNN model, which enables sampling a sequence of BF characters in an autoregressive fashion, where one feeds the previous prediction as an input to the next time step. The input to the first time step is a special START symbol. The RNN stops when it generates a special EOS symbol, indicating the end of sequence, or if the length of the program exceeds a pre-specified maximum length. The predictions at each timestep are sampled from a multinomial distribution (a softmax layer with shared weights across timesteps). The joint probability of the program sequence is the product of the probabilities of all of the tokens. We study two training algorithms, which are also compatible and can be combined. These are policy gradient, and priority queue training. We treat the RNN program synthesizer as a policy π(a 1:T ; θ) parametrized by θ, where a 1:T ≡ (a 1, . . ., a T) denotes a sequence of T actions, each of which represents a symbol in the BF langauge (and optionally an EOS symbol). The policy is factored using the chain rule as π(a 1:T ; θ) = t π(a t | a 1:t−1 ; θ) where each term in the product is given by the RNN as depicted in Figure 2. Typically an RL agent receives a reward after each action. However in our setup, we cannot score the code until completion of the program (either by emitting the EOS symbol or by hitting the maximum episode length), and accordingly, only a terminal reward of r(a 1:T) is provided. The goal is to learn a policy that assigns a high probability to plausible programs. We use the well known policy gradient approach: the REINFORCE algorithm BID44.As suggested by REINFORCE, we optimize the parameters θ to maximize the following expected reward objective, DISPLAYFORM0 We perform stochastic gradient descent in O ER to iteratively refine θ. To estimate the gradient of, we draw N Monte Carlo samples from π θ denoted {a DISPLAYFORM1 and compute the gradient of the policy as, DISPLAYFORM2 where N is the number of episodes sampled from the policy in one mini-batch, and T i denotes the number of actions in the i th episode. The gradient in Equation FORMULA2 is an unbiased estimate of the policy's true gradient, but it suffers from high variance in practice. The term b, known as a baseline, subtracted from the rewards serves as a control variate to reduce the variance of this estimator BID44 . We use an exponential moving average over rewards as the baseline. Our key technical contribution in the paper involves training an RNN with a continually updated buffer of top-K best programs (i.e., a priority queue of maximum size K). The queue is initialized empty, and after each gradient update, it is provided with new sampled programs, keeping only the programs that fall within the K highest rewarded programs. We use supervised learning to maximize the probability of the programs in the top-K buffer denoted {ã DISPLAYFORM0 under the current policy. In this way, the RNN and priority queue bootstrap off each other, with the RNN finding better programs through exploration, and the priority queue providing better training targets. The objective for PQT is simply log-likelihood in the form, DISPLAYFORM1 When PG and PQT objectives are combined, their respective gradients can simply be added together to arrive at the joint gradient. In the joint setting the priority queue component has a stabilizing affect and helps reduce catastrophic forgetting in the policy. This approach bears some similarity to the RL approaches adopted by Google's Neural Machine Translation and Neural Symbolic Machines .Entropy exploration. We also regularize the policy by adding an entropy term which aims to increase the uncertainty of the model and encourage exploration. This prevents the policy from assigning too much probability mass to any particular sequence, thereby encouraging more diversity among sampled programs. This regularizer has been prescribed initially by BID45 and more recently adopted by BID28 ; BID32 . We use the entropy regularizer for both PG and PQT.The most general form of the objective can be expressed as the sum of all of these components into one quantity. We assign different scalar weights to the PG, PQT, and entropy terms, and the gradient of the overall objective is expressed as, DISPLAYFORM2 where entropy DISPLAYFORM3 The optimization goal is the maximize. Any specific term can be remove by settings its corresponding λ to 0. When we train vanilla PG, λ TOPK = 0, and when we train with PQT, λ ER = 0.Distributed training. To speed up the training of the program synthesizers, we also make use of an asynchronous distributed setup, where a parameter server stores the shared model parameters for a number of synthesizer replicas. Each synthesizer replica samples a batch of episodes from its local copy of the policy and computes the gradients. Then, the gradients are sent to the parameter server, which asynchronously updates the shared parameters BID28. The replicas periodically update their local policy with up-to-date parameters from the parameter server. Also, to make the implementation of distributed PQT simple, each replica has its own priority queue of size K. We use 32 replicas for all of the experiments. We assess the effectiveness of our program synthesis setup by trying to discover BF programs. In what follows, we first describe the BF programming language. Then we discuss the tasks that were considered, and we present our experimental protocol and . BF is a minimalist Turing complete language consisting of only 8 low-level operations, each represented by a char from +-<>[].,. See TAB0 for operation descriptions. Operations are executed from left to right and square brackets enable looping, which is the only control flow available in the language. BF programs operate on a memory tape and internally manipulate a data pointer. The data pointer is unbounded in the positive direction but is not permitted to be negative. Memory values are not accessed by address, but relatively by shifting the data pointer left or right, akin to a Turing machine. Likewise, to change a value in memory, only increment and decrement operations are available (overflow and underflow is allowed). We include an execution demo in Appendix 6.1 to further aid understanding of the language. BF programs are able to read from an input stream and write to an output stream (one int at a time). This is how inputs and outputs are passed into and out of a BF program in our tasks. In our implementation, will write zeros once the end of the input stream is reached, and many synthesized programs make use of this feature to zero out memory. Memory values are integers, typically 1 byte and so they are often interpreted as chars for string tasks. In our BF implementation the interpreter is given a task-dependent base B so that each int is in Z B, i.e., the set of integers modulo base B. By default B = 256, unless otherwise specified. In our main experiments programs are fixed length, and most characters end up being useless no-ops. There are many ways to make no-ops in BF, and so it is very easy to pad out programs. For example, the move left operation < when the data pointer is at the leftmost position is a no-op. Unmatched braces when strict mode is off are also no-ops. Putting opposite operations together, like +-or <>, work as no-op pairs. Notice that there is only one type of syntax error in BF: unmatched braces. BF's extremely simple syntax is another advantage for program synthesis. For languages with more complex syntax, synthesizing at the character level would be very difficult due to the fact that most programs will not run or compile, thus the reward landscape is even sparser. We gave our BF interpreter a flag which turns off "strict mode" so that unmatched braces are just ignored. We found that turning off strict mode makes synthesis easier. To evaluate a given program under a task, the code is executed on one or many test cases sampled from the task (separate execution for each test input). Each test case is scored based on the program's output and the scores are summed to compute the final reward for the program. More formally, let T be the task we want to synthesize code for, and let P be a candidate program. We treat P as a function of input I so that output Q = P (I). Inputs and outputs are lists of integers (in base B). In principle for any NP task a polynomial time reward function can be computed. A trivial reward function would assign 1.0 to correct outputs and 0.0 to everything else. However, such 0/1 reward functions are extremely sparse and non-smooth (i.e., a code string with reward 1.0 may be completely surrounded by strings of reward 0.0 in edit-distance space). In practice a somewhat graded reward function must be used in order for learning to take place. The following formulation presents a possible way to compute rewards for these tasks or other similar tasks, but our method does not depend on any particular form of the reward (as long as it is not too sparse). Because all of the tasks we chose in our experiments are in the polynomial time class, the easiest way for us to score program outputs is just to directly compare to the correct output Q * for a given input I. We use a continuous comparison metric between Q and Q * to reduce reward sparsity. To evaluate a given program P under task T, a set of test cases is sampled {(I 1, Q * 1),..., (I n, Q * n)} from T. We leave open the option for T to produce static test cases, i.e., the same test cases are provided each time, or stochastic test cases drawn from a distribution each time. In practice, we find that static test cases are helpful for learning most tasks (see Section 4.3).For simplicity we define a standardized scoring function S(Q, Q *) for all our tasks and test cases (see Appendix 6.2 for details). Total reward for the program is DISPLAYFORM0 where ζ is a constant scaling factor, which can differ across tasks. We use ζ to keep rewards approximately in the range [−1, 1]. R tot is given to the agent as terminal reward. When generating variable length programs we give preference to shorter programs by adding a program length bonus to the total reward: R tot + 1 − |P | /MaxProgramLength. If the BF interpreter is running in strict mode and there is a syntax error (i.e., unmatched braces) we assign the program a small negative reward. We also assign negative reward to programs which exceed 5000 execution steps to prevent infinite loops. Note that R tot is the terminal reward for the agent. We assess the effectiveness of our priority queue training method against the following baselines:• Genetic algorithm (GA) implemented based on BID3.2 See Appendix 6.3 for more details regarding our implementation.• Policy gradient (PG) as described in Section where λ TOPK = 0.• Policy gradient (PG) combined with priority queue training (PQT), where λ TOPK > 0.We are operating under the RL paradigm, and so there is no test/evaluation phase. We use a set of benchmark coding tasks (listed in Appendix 6.4) to compare these methods, where different models are trained on each task. Simply the best program found during training is used as the final program for that task. In order to tune each method, we propose to carry out experiments in two phases. First, the hyperparameters will be tuned on a subset of the tasks (reverse and remove-char). Next, the best hyperparameters will be fixed, and each synthesis method will be trained on all tasks. To compare performance between the synthesis methods, we measure the success rate of each method after a predetermined number of executed programs. More details will be described in Section 4.4.We tune the hyperparameters of the synthesis methods on reverse and remove-char tasks. We use grid search to find the best hyperparameters in a given set of possible values. The tuning space is as follows. For PG, learning rate ∈ {10 −5, 10 −4, 10 −3} and entropy regularizer ∈ {0.005, 0.01, 0.05, 0.10}. For PQT, learning rate and entropy regularizer are searched in the same spaces, and we also allow the entropy regularizer to be 0; PQT loss multiplier (λ TOPK from Equation 4) is searched in {1.0, 10.0, 50.0, 200.0}. For GA, population size ∈ {10, 25, 50, 100, 500}, crossover rate ∈ {0.2, 0.5, 0.7, 0.9, 0.95} and mutation rate ∈ {0.01, 0.03, 0.05, 0.1, 0.15}.For PQT we set K = 10 (maximum size of the priority queue) in all experiments. In early experiments we found 10 is a nice compromise between a very small queue which is too easy for the RNN to memorize, and a large queue which can dilute the training pool with bad programs. • PG+PQT: entropy regularizer = 0.01, learning rate = 10 −4, λ TOPK = 50.0.• PQT: entropy regularizer = 0.01, learning rate = 10 −4, λ TOPK = 200.0.• GA: population size = 100, crossover rate = 0.95, mutation rate = 0.15.Other model and environment choices. For PG and PQT methods we use the following architecture: a 2-layer LSTM RNN BID16 with 35 units in each layer. We jointly train embeddings for the program symbols of size 10. The outputs of the top LSTM layer are passed through a linear layer with 8 or 9 outputs (8 BF ops plus an optional EOS token) which are used as logits for the softmax policy. We train on minibatches of size 64, and use 32 asynchronous training replicas. Additional hyperparameter values: gradient norm clipping threshold is set at 50, parameter initialization factor 3 is set at 0.5, RMSProp is used as the optimizer, and decay for the exponential moving average baseline is set at 0.99.We explored a number of strategies for making test cases. For the reverse task we tried:1. One random test case of random length.2. Five random test cases of random length.3. Five random test cases of lengths 1 through 5. However, solutions were only found when we used static test cases (option 4).In the experiments below, all programs in each task are evaluated on the same test cases. The test inputs are randomly generated with a fixed seed before any training happens. By default each task has 16 test cases, with a few exceptions noted in Appendix 6.4. For the two tuning tasks we continue to use a small number of hand crafted test cases. A potential problem with using the test cases for code synthesis is that the synthesized code can overfit, i.e., the code can contain hard-coded solutions for test inputs. In the experiments below we also run synthesized code on a large set of held-out eval test cases. These eval test cases are also randomly generated with a fixed seed, and the total number of test cases (train and eval) for each task is 1000. Success rates on training test cases and all test cases are reported in TAB2. We do manual inspection of code solutions in Table 4 to identify overfitting programs, which are highlighted in red. In the following, we show success rates on tuning tasks and held-out tasks for all algorithms. Again, our metric to compare these methods is the success rate of each method at finding the correct program after a predetermined maximum number of executed programs. For all tasks, we ran training 25 times independently to estimate success rates. A training run is successful if a program is found which solves all the test cases for the task. Training is stopped when the maximum number of programs executed (NPE) is reached, and the run is considered a failure. For tuning, we use maximum NPE of 5M. For evaluation we use a maximum NPE of 20M.Our genetic algorithm is best suited for generating programs of fixed length, meaning all code strings considered are of some fixed length preset by the experimenter. In all the experiments presented below, the EOS token is disabled for all algorithms, so that there are 8 possible code characters. Program length is 100 characters for all experiments, and the search space size is 8 100 ≈ 10 90.In TAB1, we report the success rates from tuning of the genetic algorithm, policy gradient, priority queue training, and policy gradient with priority queue training. The for these tuning tasks are different from the same tasks in TAB2, due to the smaller NPE used in tuning, and the fact that we tune on a different set of hand-selected test cases. There is also sensitivity of each synthesis method to initialization, sampling noise, and asynchronous weight updates which accounts for differences between multiple runs of the same tasks. In TAB2, we report the success rates of the same algorithms plus uniform random search. We include success rates for training and eval test cases. We also do an aggregate comparison between columns by taking the average at the bottom. As can be seen from the table, PQT is clearly better than PG and GA according to training and eval averages. PG+PQT is on par with PQT alone. The eval success rates are lower than the training success rates in many cases due to overfitting programs. DISPLAYFORM0 2.5 / 1.0 8.6 / 4.3 3.3 / 1.5 13.0 / 5.7 12.9 / 5.4 In this section we have our method generate shortest possible code string. Code shortening, sometimes called code golf, is a common competitive exercise among human BF programmers. We use PG+PQT to generate programs of variable length (RNN must output EOS token) with a length bonus in the reward to encourage code simplification (see 4.2). We train each task just once, but with a much larger maximum NPE of 500M. We do not stop training early, so that the agent can iterate on known solutions. We find that alternative hyperparameters work better for code shortening, with λ ENT = 0.05 and λ TOPK = 0.5.In Table 4 we show simplified programs for coding tasks where a solution was found. Table 4: Synthesized BF programs for solved tasks. All programs were discovered by the agent as-is, and no code characters were removed or altered in anyway. Notice that some programs overfit their tasks. For example, cascade is correct only for up to length 6 (provided test cases are no longer than this). We highlighted in red all the tasks with code solutions that overfit, which we determined by manual inspection. In this paper, we considered the task of learning to synthesize programs for problems where a reward function is defined. We use an RNN trained with our priority queue training method. We experimented with BF, a simple Turing-complete programming language, and compared our method against a genetic algorithm baseline. Our experimental showed that our method is more stable than vanilla policy gradient or a genetic algorithm. That PQT works as a standalone search algorithm is surprising, and future work is needed in order to better explain it. We can speculate that it is implementing a simple hill climbing algorithm where the buffer stores the best known samples, thereby saving progress, while the RNN functions as an exploration mechanism. Even more surprising is that this algorithm is able to bootstrap itself to a solution starting from an empty buffer and a randomly initialized RNN. We believe that our coding environment complements the PQT algorithm, since finding code with non-zero reward through purely random search is feasible. 6.1 BF EXECUTION DEMO Figure 2: In the following figure we step through a BF program that reverses a given list. The target list is loaded into the input buffer, and the programs output will be written to the output buffer. Each row depicts the state of the program and memory before executing that step. Purple indicates that some action will be taken when the current step is executed. We skip some steps which are easy to infer. Vertical ellipses indicate the continuation of a loop until its completion. Our implementation of S(Q, Q *) is computed from a non-symmetric distance function d(Q, Q *) which is an extension of Hamming distance H(list 1, list 2) for non-equal length lists (note that arguments to H must be the same length). Hamming distance provides additional information, rather than just saying values in Q and Q * are equal or not equal. Further since BF operates on values only through increment and decrement operations this notation of distance is very useful for conveying to the agent information about how many inc or dec ops to use in various places. This serves to make the reward space smoother and less sparse. We want a distance of 0 to in the maximum score and a large distance to in a small or negative score. Thus we define: DISPLAYFORM0 where ∅ is the empty list, l = |Q|, l * = |Q * |, and B is the integer base (number of possible ints at each position). We define our distance function: DISPLAYFORM1 Essentially d(Q, Q *) adds maximum char distance to the Hamming distance for each missing position or each extra position, depending on whether Q is shorter or longer than Q * respectively. S(Q, Q *) subtracts the distance d(Q, Q *) from the distance of an empty list to Q *, which is equal to B |Q * |. A genetic algorithm (GA) simulates sexual reproduction in a population of genomes in order to optimize a fitness function. To synthesize BF programs, we let a genome be one code string. The GA is initialized with a population of randomly chosen code strings. For each iteration a new population of children is sampled from the existing population. Each new population is called a generation. GA samples a new population from the current population with 3 steps: 1) parent selection, 2) mating, and 3) mutation. Many algorithms have been developed for each of these steps which have varying effects on the GA as a whole. We describe our algorithm for each step:1. Parent selection: Randomly sample a set of parents from the population. We use roulette selection (a.k.a. fitness proportionate selection) where parents are chosen with probability proportional to their fitness.2. Mating: Choose pairs of parents and perform an operation ing in two children to replace the parents. We use single point crossover where a position in the genome (code string) for the first parent is sampled uniformly and the two parents' genetic material is swapped after that point to create two new children. Crossover is performed with probability p crossover.3. Mutation: With some probability make small random modifications to each child. We use the primaryobjects mutation function. This function iterates through each code token and with probability p mutate chooses among 4 possible mutation operations to apply: insert random token, replace with random token, delete token, shift and rotate either left or right. When inserting the last token is removed, and when deleting a random token is added at the end so that none of the mutation operations change the number of tokens in the list. We tune p mutate, p crossover, and the population size (see Experimental Setup). For each task, the genome size is set to the maximum code length, since GA operates on fixed length genomes. For input of length N, return the (n − 1) th input, then the 0 th, then (n − 2), 1, (n − 3), 2,... unriffle Inverse of the riffle function. middle-char For input of length N, return the value at position f loor(N/2). remove-last Remove the last character from the list. remove-last-two Remove the last two characters from the list. echo-alternating Return every even indexed value, followed by every odd indexed value. echo-half Return the first half of the input. lengthReturn the length of the list. echo-nth-seq For M input sequences each seperated by a 0, return the n th sequence, where n is given as the first value in the input. substringReturn a sub-range of the input list, given a starting index i and length l. divide-2 Return input value divided by two (integer division). dedup Return input list, in which all duplicate adjacent values removed.

Represent the TLDR for finding the corresponding scientific article

We use a simple search algorithm involving an RNN and priority queue to find solutions to coding tasks.