Question: Represent the scientific article to find a short TLDR

Question: Efficiently learning to solve tasks in complex environments is a key challenge for reinforcement learning (RL) agents. We propose to decompose a complex environment using a task-agnostic world graphs, an abstraction that accelerates learning by enabling agents to focus exploration on a subspace of the environment. The nodes of a world graph are important waypoint states and edges represent feasible traversals between them. Our framework has two learning phases: 1) identifying world graph nodes and edges by training a binary recurrent variational auto-encoder (VAE) on trajectory data and 2) a hierarchical RL framework that leverages structural and connectivity knowledge from the learned world graph to bias exploration towards task-relevant waypoints and regions. We show that our approach significantly accelerates RL on a suite of challenging 2D grid world tasks: compared to baselines, world graph integration doubles achieved rewards on simpler tasks, e.g. MultiGoal, and manages to solve more challenging tasks, e.g. Door-Key, where baselines fail. Many real-world applications, e.g., self-driving cars and in-home robotics, require an autonomous agent to execute different tasks within a single environment that features, e.g. high-dimensional state space, complex world dynamics or structured layouts. In these settings, model-free reinforcement learning (RL) agents often struggle to learn efficiently, requiring a large amount of experience collections to converge to optimal behaviors. Intuitively, an agent could learn more efficiently by focusing its exploration in task-relevant regions, if it has knowledge of the high-level structure of the environment. We propose a method to 1) learn and 2) use an environment decomposition in the form of a world graph, a task-agnostic abstraction. World graph nodes are waypoint states, a set of salient states that can summarize agent trajectories and provide meaningful starting points for efficient exploration (; ;). The directed and weighted world graph edges characterize feasible traversals among the waypoints. To leverage the world graph, we model hierarchical RL (HRL) agents where a high-level policy chooses a waypoint state as a goal to guide exploration towards task-relevant regions, and a low-level policy strives to reach the chosen goals. Our framework consists of two phases. In the task-agnostic phase, we obtain world graphs by training a recurrent variational auto-encoder (VAE) (; ;) with binary latent variables over trajectories collected using a random walk policy and a curiosity-driven goal-conditioned policy . World graph nodes are states that are most frequently selected by the binary latent variables, while edges are inferred from empirical transition statistics between neighboring waypoints. In the task-specific phase, taking advantage of the learned world graph for structured exploration, we efficiently train an HRL model . In summary, our main contributions are: • A task-agnostic unsupervised approach to learn world graphs, using a recurrent VAE with binary latent variables and a curiosity-driven goal-conditioned policy. • An HRL scheme for the task-specific phase that features multi-goal selection (Wide-thenNarrow) and navigation via world graph traversal. 4. On its traversal course to wide goal, agent hits final target and exits.: waypoints selected by the manager: waypoints initiates traversal: trajectories directly from worker actions: exit point: agent: final goal from manager close to selected waypoints: trajectories from world graph traversal Figure 1: Top Left: overall pipeline of our 2-phase framework. Top Right (world graph discovery): a subgraph exemplifies traversal between waypoint states (in blue), see Section 3 for more details. Bottom (Hierarhical RL): an example rollout from our proposed HRL policy with Wide-then-Narrow Manager instructions and world graph traversals, solving a challenging Door-Key task, see Section 4 for more details. • Empirical evaluations on multiple tasks in complex 2D grid worlds to validate that our framework produces descriptive world graphs and significantly improves both sample efficiency and final performance on these tasks over baselines, especially thanks to transfer learning from the unsupervised phase and world graph traversal. An understanding of the environment and its dynamics is essential for effective planning and control in model-based RL. For example, a robotics agent often locates or navigates by interpreting a map (; ;). Our exploration strategy draws inspiration from active localization, where robots are actively guided to investigate unfamiliar regions . Besides mapping, recent works (; ;) learn to represent the world with generative latent states (; ; Racanière et al., 2017). If the latent dynamics are also extrapolated, the latent states can assist planning (a;) or model-based RL . While also aiming to model the world, we approach this as abstracting both the structure and dynamics of the environment in a graph representation, where nodes are states from the environment and edges encode actionable efficient transitions between nodes. Existing works (; ; ;) have shown benefits of such graph abstractions but typically select nodes only subject to a good coverage the observed state space. Instead, we identify a parsimonious subset of states that can summarize trajectories and provide more useful intermediate landmarks, i.e. waypoints, for navigating complex environments. Our method for estimating waypoint states can be viewed as performing automatic (sub)goal discovery. Subgoal and subpolicy learning are two major approaches to identify a set of temporally-extended actions, "skills", that allow agents to efficiently learn to solve complex tasks. Subpolicy learning identifies policies useful to solve RL tasks, such as option-based methods and subtask segmentations . Subgoal learning, on the other hand, identifies "important states" to reach (Şimşek et al., 2005). Previous works consider various definitions of "important" states: frequently visited states during successful task completions , states introducing the most novel information , bottleneck states connecting densely-populated regions (;Şimşek et al., 2005), or environment-specific heuristics . Our work draws intuition from unsupervised temporal segmentation and imitation learning . We define "important" states (waypoints) as the most critical states in recovering action sequences generated by some agent, which indicates that these states contain the richest information about the executed policy . We propose a method for learning a world graph G w, a task-agnostic abstraction of an environment that captures its high-level structure and dynamics. In this work, the primary use of world graphs is to accelerate reinforcement learning of downstream tasks. The nodes of G w, denoted by a set of waypoints states s p ∈ V p, are generically "important" for accomplishing tasks within the environment, and therefore useful as starting points for exploration. Our method identifies such waypoint states from interactions with the environment. In addition, we embed feasible transitions between nearby waypoint states as the edges of G w. In this work, we define important states in the context of learning G w (see Section 2 for alternative definitions). That is, we wish to discover a small set of states that, when used as world graph nodes, concisely summarize the structure and dynamics of the environment. Below, we describe 1) how to collect state-action trajectories and an unsupervised learning objective to identify world graph nodes, and 2) how the graph's edges (i.e., how to transition between nodes) are formed from trajectories. The structure and dynamics of an environment are implicit in the state-action trajectories observed during exploration. To identify world graph nodes from such data, we train a recurrent variational autoencoder (VAE) that, given a sequence of state-action pairs, identifies a subset of the states in the sequence from which the full action sequence can be reconstructed (Figure 2). In particular, the VAE infers binary latent variables that controls whether each state in the sequence is used by the generative decoder, i.e., whether a state is "important" or not. Binary Latent VAE The VAE consists of an inference, a generative and a prior network. These are structured as follows: the input to the inference network q φ is a trajectory of state-action pairs observed from the environment τ ={(s t, a t)} T t=0, with s={s t} T t=0 and a={a t} T t=0 denoting the state and action sequences respectively. The output of the inference network is the approximated posterior over a sequence z={z t} T t=0 of binary latent variables, denoted as q φ (z|a, s). The generative network p θ computes a distribution over the full action sequence a using the masked state sequence, where s t is masked if z t =0 (we fix z 0 =z T =1 during training), denoted as p θ (a|s, z). Finally, a state-conditioned p ψ (z t |s t) given by the prior network p ψ for each s t encodes the empirical average probability that state s t is activated for reconstruction. This choice encourages inference to select within a consistent subset of states for use in action reconstruction. In particular, the waypoint Algorithm 1: Identifying waypoint states V p and learning a goal-conditioned policy π g Result: Waypoint states V p and a goal-conditioned policy π g Initialize network parameters for the recurrent variational inference model V Initialize network parameters for the goal-conditioned policy π g Initialize V p with the initial position of the agent, i.e. V p = {s 0 =} while VAE reconstruction error has not converged do for n ← 1 to N do Sample random waypoint s p ∈ V p Navigate agent to s p and perform T -step rollout using a randow walk policy: T Navigate agent to s p and perform T -step rollout using π g with goal g n: τ Re-label π g rewards with action reconstruction error as curiosity bonus: end Perform policy gradient update of π g using τ π and r π Update V using τ r and τ π Update V p as set of states with largest prior mean αs αs+βs. end states V p are chosen as the states with the largest prior means and during training, once every few iterations, V p is updated based on the current prior network. Objective Formally, we optimize the VAE using the following evidence lower bound (ELBO): To ensure differentiablity, we apply a continuous relaxation over the discrete z t. We use the Beta distribution p ψ (z t) = Beta(α t, β t) for the prior and the Hard Kumaraswamy distribution q ψ (z t |a, z) = HardKuma(α t,β t) for the approximate posterior, which resembles the Beta distribution but is outside the exponential family . This choice allows us to sample 0s and 1s without sacrificing differentiability, accomplished via the stretch-and-rectify procedure and the reparametrization trick . Lastly, to prevent the trivial solution of using all states for reconstruction, we use a secondary objective L 0 to regularize the L 0 norm of z at a targeted value µ 0 , the desired number of selected states out of T steps, e.g. for when T = 25, we set µ 0 = 5, meaning ideally 5 out of 25 states are activated for action reconstruction. Another term L T to encourage temporal separation between selected states by targeting the number of 0/1 switches among z at 2µ 0: See Appendix A for details on training the VAE with binary z t, including integration of the Hard Kumaraswamy distribution and how to regularize the statistics of z. Naturally, the latent structure learned by the VAE depends on the trajectories used to train it. Hence, collecting a rich set of trajectories is crucial. Here, we propose a strategy to bootstrap a useful set of trajectories by alternately exploring the environment based on the current iteration's V p and updating the VAE and V p, repeating this cycle until the action reconstruction accuracy plateaus (Algorithm 1). During exploration, we use action replay to navigate the agent to a state drawn from the current iteration's V p. Although resetting via action replay assumes our underneath environment to be deterministic, in cases where this resetting strategy is infeasible, it may be modified so long as to allow the exploration starting points to expand as the agent discovers more of its environment. For each such starting point, we collect two rollouts. In the first rollout, we perform a random walk to explore the nearby region. In the second rollout, we perform actions using a goal-conditioned policy π g (GCP), setting the final state reached by the random walk as the goal. Both rollouts are used for trianing the VAE and the latter is also used for training π g. GCP provides a venue to integrate intrinsic motivation, such as curiosity (; ; ;) to generate more diverse rollouts. Specifically, we use the action reconstruction error of the VAE as an intrinsic reward signal when training π g. This choice of curioisty also prevents the VAE from collapsing to the simple behaviors of a vanilla π g. The final stage is to construct the edges of G w, which should ideally capture the environment dynamics, i.e. how to transition between waypoint states. Once VAE training is complete and V p is fixed, we collect random walk rollouts from each of the waypoints s p ∈ V p to estimate the underlying adjacency matrix . More precisely, we claim a directed edge s p → s q if there exists a random walk trajectory from s p to s q that does not intersect a third waypoint. We also consider paths taken by π g (starting at s p and setting s q as the goal) and keep the shortest observed path from s p to s q as a world graph edge transition. We use the action sequence length of the edge transition between adjacent waypoints as the weight of the edge. As shown experimentally, a key benefit of our approach is the ability to plan over G w. To navigate from one waypoint to another, we can use dynamic programming to output the optimal traversal of the graph. World graphs present a high-level, task-agnostic abstraction of the environment through waypoints and feasible transition routes between them. A key example of world graph applications for taskspecific RL is structured exploration: instead of exploring the entire environment, RL agents can use world graphs to quickly identify task-relevant regions and bias low-level exploration to these regions. Our framework to leverage world graphs for structured exploration consists of two parts: 1. Hierarchical RL wherein the high-level policy selects subgoals from V p. 2. Traversals using world graph edges. Formally, an RL agent learning to solve a task is formulated as a Markov Decision Process: at time t, the agent is in a state s t, executes an action a t via a policy π(a t |s t) and receives a rewards r t. The agent's goal is to maximize its cumulative expected return R = E (st,at)∼π,p,p0 t≥0 γ t r t, where p(s t+1 |s t, a t), p 0 (s 0) are the transition and initial state distributions. To incorporate world graphs with RL, we use a hierarchical approach based on the Feudal Network (FN) , depicted in Figure 3. A standard FN Collect randomly spawned balls, each ball gives +1 reward. To end an episode, the agent has to exit at a designated point. Balls are located randomly, dense reward. Agents receive a single reward r ≤ 1 proportional to the number of balls collected upon exiting. Balls are located randomly, sparse reward. Spawn lava blocks at random locations each time step that immediately terminates the episode if stepped on. Stochastic environment. Multiple objects: lava and balls are randomly located, dense reward. Agent has to pick up a key to open a door (reward +1) and reach the exit point on the other side (reward +1). Walls, door and key are located randomly. Agents have additional actions: pick and toggle. Table 1: An overview of tasks used to evaluate the benefit of using world graphs. Visualizations can be found in Appendix D. decomposes the policy of the agent into two separate policies that receive distinct streams of reward: a high-level policy ("Manager") learns to propose subgoals; a low-level policy ("Worker") receives subgoals from the Manager as inputs and is rewarded for taking actions in the environment that reach the subgoals. The Manager receives the environment reward defined by the task and therefore must learn to emit subgoals that lead to task completion. The Manager and Worker do not share weights and operate at different temporal resolutions: the Manager only outputs a new subgoal if either the Worker reaches the chosen one or a subgoal horizon c is exceeded. For all our experiments, policies are trained using advantage actor-critic (A2C), an on-policy RL algorithm (; ; b). To ease optimization, the feature extraction layers of the Manager and Worker that encode s t are initialized with the corresponding layers from π g, the GCP learned during world graph discovery phase. More details are in Appendix B. To incorporate the world graph, we introduce a Manager policy that factorizes subgoal selection as follows: a wide policy π w (g w t |s t) selects a waypoint state as the wide goal g w ∈ V p, and a narrow policy π n (g n t |s t, g The wide-then-narrow subgoal format simplifies the search space for the Manager policy. Using waypoints as wide goals also makes it possible to leverage the edges of the world graph for planning and executing the planned traversals. This process breaks down as follows: 1. When to Traverse: When the agent encounters a waypoint state s t ∈ V p, a "traversal" is initiated if s t has a feasible connection in G w to the active wide goal g w t . 2. Planning: Upon triggering a traversal, the optimal traversal route from the initiating state to g w t is estimated from the G w edge weights using classic dynamic programming planning . This yields a sequence of intermediate waypoint states. 3. Execution: Execution of graph traversals depends on the nature of the environment. If deterministic, the agent simply follows the action sequences given by the edges of the traversal. Otherwise, the agent uses the pretrained GCP π g to sequentially reach each of the intermediate waypoint states along the traversal (we fine-tune π g in parallel where applicable). If the agent fails to reach the next waypoint state within a certain time limit, it stops its current pursuit and a new (g w, g n) pair is received from the Manager. World graph traversal allows the Manager to assign task-relevant wide goals g w that can be far away from the agent yet still reachable, which consequentially accelerates learning by focusing exploration around the task-relevant region near g w. We now assess each component of our framework on a set of challenging 2D grid worlds. Our ablation studies demonstrate the following benefits of our framework: • It improves sample efficiency and performance over the baseline HRL model. • It benefits tasks varying in envirionment scale, task type, reward structure, and stochasticity. • The identified waypoints provide superior world representations for solving downstream tasks, as compared to graphs using randomly selected states as nodes. Implementation details, snippets of the tasks and mazes are in Appendix C-D. For our ablation studies, we construct 2D grid worlds of increasing sizes (small, medium and large) along with challenging tasks with different reward structures, levels of stochasticity and logic (summarized in Table 1). In all tasks, every action taken by the agent receives a negative reward penalty. We follow a rigorous evaluation protocol (; ;): each experiment is repeated with 3 training seeds. 10 additional validation seeds are used to pick the model with the best reward performance. This model is then tested on 100 testing seeds. We report mean reward and standard deviation. We ablate each of the following components in our framework and compare against non-hierarchical (A2C) and hierarchical baselines (FN): 1. initializing the feature extraction layers of the Manager and Worker from π g, 2. applying Wide-then-Narrow Manager (WN) goal instruction, and 3. allowing the Worker to traverse along G w. Results are shown in Table 2. In sum, each component improves performance over the baselines. Wide and narrow goals Using two goal types is a highly effective way to structure the Manager instructions and enables the Worker to differentiate the transition and local task-solving phases. We note that for small MultiGoal, agents do not benefit much from G w traversal: it can rely solely on the guidance from WN goals to master both phases. However with increasing maze size, the Worker struggles to master traversals on its own and thus fails solving the tasks. World Graph Traversal As conjectured in Section 4.3, the performance gain of our framework can be explained by the larger range and more targeted exploration strategy. In addition, the Worker We see that 1) traversal speeds up convergence, 2) V rand gives higher variance and slightly worse performance than Vp. Right: comparing with or without πg initialization on Vp, all models use WN. We see that initializing the task-specific phase with the task-agnostic goal-conditioned policy significantly boosts learning. does not have to learn long distance transitions with the aid of G w traversals. Figure 4 confirms that G w traversal speeds up convergence and its effect becomes more evident with larger mazes. Note that the graph learning stage only need 2.4K iterations to converge. Even when taking these additional environment interactions into account, G w traversal still exhibits superior sample efficiency, not to mention that the graph is shared among all tasks. Moreover, solving Door-Key involves a complex combination of sub-tasks: find and pick up the key, reach and open the door and finally exit. With limited reward feedback, this is particularly difficult to learn. The ability to traverse along G w enables longer-horizon planning on top of the waypoints, thanks to which the agents boost the success rate on medium Door-Key from 0.56±0.02 to 0.75±0.06. To highlight the benefit of establishing the waypoints learned by the VAE as nodes for G w, we compare against using a G w constructed around randomly selected states (V rand). The edges of the random-node graph are formed in the same way as described in Section 3.3 and its feature extractor is also initialized from π g. Although granting knowledge acquired during the unsupervised phase to V rand is unfair to V p, deploying both initialization and traversal while only varying V rand and V p isolates the effect from the nodes to the best extent. The comparative (in Table 3, learning curves for MultiGoal in Figure 4) suggest V p generally outperforms V rand. Door-Key is the only task in which the two matches. However, V rand exhibits a large variance, implying that certain sets of random states can be suitable for this task, but using learned waypoints gives strong performance more consistently. Initialization with GCP Initializing the weights of the Worker and Manager feature extractors from π g (learned during the task-agnostic phase) consistently benefits learning. In fact, we observe that models starting from scratch fail on almost all tasks within the maximal number of training iterations, unless coupled with G w traversal, which is still inferior to using π g -initialization. Particularly, for the small MultiGoal-Stochastic environment, there is a high chance that a lava square blocks traversal; therefore, without the environment knowledge from π g transferred by weight initialization, the interference created by the episode-terminating lava prevents the agent from learning the task. We have shown that world graphs are powerful environment abstractions, which, in particular, are capable of accelerating reinforcement learning. Future works may extend their applications to more challenging RL setups, such as real-world multi-task learning and navigation. It is also interesting to generalize the proposed framework to learn dynamic world graphs for evolving environments, and applying world graphs to multi-agent problems, where agents become part of the world graphs of other agents. As illustrated in the main text, the main objective for the recurrent VAE is the following evidence lower bound with derivation: log p(a|s) = log p(a|s, z)dz = log p(a|s, z)p(z|s) q(z|a, s) q(z|a, s) dz = log p(a|s, z) p(z|s) q(z|a, s) q(z|a, s)dz The inference network q ψ takes in the trajectories of state-action pairs τ and at each time step approximates the posterior of the corresponding latent variable z t. The prior network p ψ takes the state s t at each time step and outputs the state-conditioned prior p ψ (s t). We choose Beta as the prior distribution and the Hard Kuma as the approximated posterior to relax the discrete latent variables to continuous surrogates. The Kuma distribution Kuma(α, β) highly resembles the Beta Distribution in shape but does not come from the exponential family. Similar to Beta, the Kuma distribution also ranges from bimodal (when α ≈ β) to unimodal (α/β → 0 or α/β → ∞). Also, when α = 1 or β = 1, Kuma(α, β) = Beta(α, β). We observe empirically better performance when we fix β = 1 for the Kuma approximated posterior. One major advantage of the Kuma distribution is its simple Cumulative Distribution Function (CDF): It is therefore amendable to the reparametrization trick (; ;) by sampling from uniform distribution u ∼ U: Lastly, the KL-divergence between the Kuma and Beta distributions can be approximated in closed form : where Ψ is the Digamma function, γ the Euler constant, and the approximation uses the first few terms of the Taylor series expansion. We take the first 5 terms here. Next, we make the Kuma distribution "hard" by following the steps in. First stretch the support to (r = 0 − 1, l = 1 + 2), 1, 2 > 0, and the ing CDF distribution takes the form: Then, the non-eligible probabilities for 0's and 1's are attained by rectifying all samples below 0 to 0 and above 1 to 1, and other value as it is, that is Lastly, we impose two additional regularization terms L and L T on the approximated posteriors. As described in the main text, L prevents the model from selecting all states to reconstruct {a t} by restraining the expected L 0 norm of z = (z 1 · · · z T −1) to approximately be at a targeted value µ 0 . In other words, this objective adds the constraint that there should be µ 0 of activated z t = 1 given a sequence of length T. The other term L T encourages temporally isolated activation of z t, meaning the number of transition between 0 and 1 among z t's should roughly be 2µ 0. Note that both expectations in Equation 2 have closed forms for HardKuma. Lagrangian Relaxation. The overall optimization objective consists of action sequence reconstruction, KL-divergence between the posterior and prior, L 0 and L T (Equation 12). We tune the objective weights λ i using Lagrangian relaxation (; ;), treating λ i's as learnable parameters and performing alternative optimization between λ i's and the model parameters. We observe that as long as their initialization is within a reasonable range, λ i's converge to a local optimum: We observe this approach to produce efficient and stable mini-batch training. Optimizing composite neural networks like HRL is sensitive to weight initialization , due to its complexity and lack of clear supervision at various levels. Therefore, taking inspiration from prevailing pre-training procedures in computer vision and NLP , we take advantage of the weights learned by π g during world graph discovery when initializing the Worker and Manager policies for downstream HRL, as π g has already implicitly embodied much environment dynamics information. More specifically, we extract the weights of the feature extractor, i.e. the state encoder, and use them as the initial weights for the state encoders of the HRL policies. Our empirical demonstrate that such weight initialization consistently improves performance and validates the value of skill/knowledge transfer from GCP . Model code folder including all architecture details is shared in comment. Our models are optimized with Adam using mini-batches of size 128, thus spawning 128 asynchronous agents to explore. We use an initial learning rate of 0.0001, with = 0.001, β 1 = 0.9, β 2 = 0.999; gradients are clipped to 40 for inference and generation nets. For HardKuma, we set l = −0.1 and r = 1.1. The maximum sequence length for BiLSTM is 25. The total number of training iterations is 3600 and model usually converges around 2400 iterations. We train the prior, inference, and generation networks end-to-end. We initialize λ i's (see Lagrangian Relaxation) to be λ 1 = 0.01 (KL-divergence),λ 2 = 0.06 (L 0), λ 3 = 0.02 (L T). After each update of the latent model, we update λ i's, whose initial learning rate is 0.0005, by maximizing the original objective in a similar way as using Lagrangian Multiplier. At the end of optimization, λ i's converge to locally optimal values. For example, with the medium maze, λ 1 = 0.067 for the KL-term, λ 2 = 0.070 for the L 0 and λ 3 = 0.051 for the L T term. The total number of waypoints |V p | is set to be 20% of the size of the full state space. The procedure of the Manager and the Worker in sending/receiving orders using either traversal paths among V p from replay buffer for deterministic environments or with π g for stochastic ones follows: 1. The Manager gives a wide-narrow subgoal pair (g w, g n). 2. The agent takes action based on the Worker policy π ω conditioned on (g w, g n) and reaches a new state s. If s ∈ V p, g w has not yet met, and there exists a valid path basing on the edge paths from the world graph s → g w, agent then either follows replay actions or π g to reach g w. If π g still does not reach desired destination in a certain steps, then stop the agent wherever it stands; also π g can be finetuned here. 3. The Worker receives positive reward for reaching g w for the first time. 4. If agent reaches g n, the Worker also receives positive rewards and terminates this horizon. 5. The Worker receives negative for every action taken except for during traversal; the Manager receives negative reward for every action taken including traversal. 6. When either g n is reached or the maximum time step for this horizon is met, the Manager renews its subgoal pair. The training of the Worker policy π ω follows the same A2C algorithm as π g. The training of the Manager policy π m also follows a similar procedure but as it operates at a lower temporal resolution, its value function regresses against the t m -step discounted reward where t m covers all actions and rewards generated from the Worker. When using the Wide-then-Narrow instruction, the policy gradient for the Manager policy π m becomes: E (st,at)∼π,p,p0 [A m,t ∇ log (π ω (g w,t |s t) π n (g n,t |s t, g w,t, s w,t))] + ∇ [H (π ω) + H (π n (·|g w,t))], where A m,t is the Manager's advantage at time t. Also, for Manager, as the size of the action space scales linearly with |S|, the exact entropy for the π m can easily become intractable. Essentially there are O |V| × (N 2) possible actions. To calculate the entropy exactly, all of them has to be summed, making it easily computationally intractable: H = w∈V wn∈sw π n (w n |s w, s t)π ω (w|s t) log ∇π n (w n |s w, s t)π ω (w|s t). Thus in practice we resort to an effective alternative H (π ω) + H (π n (·|g w,t)). Psuedo-code for Manager training is in Algorithm 2. For training the HRL policies, we inherit most hyperparameters from those used when training π g, as the Manager and the Worker both share similar architectures with π g. The hyperparameters used when training π g follow those from. Because the tasks used in HRL experiments are more difficult than the generic goal-reaching task, we set the maximal number of training iterations to 100K abd training is stopped early if model performance reaches a plateau. The rollout steps for each iteration is 60. Hyperparameters specific to HRL are the horizon c = 20 and the size of the Manager's local attention range (that is, the neighborhood around g w within which g n is selected), which are N = 5 for small and medium mazes, and N = 7 for the large maze.

Represent the short summary for finding the corresponding scientific article (long)

We learn a task-agnostic world graph abstraction of the environment and show how using it for structured exploration can significantly accelerate downstream task-specific RL.

We view exploration in RL as a problem of matching a marginal distribution over states.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We introduce the notion of property signatures, a representation for programs and program specifications meant for consumption by machine learning algorithms. Given a function with input type τ_in and output type τ_out, a property is a function of type: (τ_in, τ_out) → Bool that (informally) describes some simple property of the function under consideration. For instance, if τ_in and τ_out are both lists of the same type, one property might ask ‘is the input list the same length as the output list? ’. If we have a list of such properties, we can evaluate them all for our function to get a list of outputs that we will call the property signature. Crucially, we can ‘guess’ the property signature for a function given only a set of input/output pairs meant to specify that function. We discuss several potential applications of property signatures and show experimentally that they can be used to improve over a baseline synthesizer so that it emits twice as many programs in less than one-tenth of the time. Program synthesis is a longstanding goal of computer science research (; ; ; Shaw; ;), arguably dating to the 1940s and 50s . Deep learning methods have shown promise at automatically generating programs from a small set of input-output examples (; ; b; 2019b). In order to deliver on this promise, we believe it is important to represent programs and specifications in a way that supports learning. Just as computer vision methods benefit from the inductive bias inherent to convolutional neural networks , and likewise with LSTMs for natural language and other sequence data , it stands to reason that ML techniques for computer programs will benefit from architectures with a suitable inductive bias. We introduce a new representation for programs and their specifications, based on the principle that to represent a program, we can use a set of simpler programs. This leads us to introduce the concept of a property, which is a program that computes a boolean function of the input and output of another program. For example, consider the problem of synthesizing a program from a small set of input-output examples. Perhaps the synthesizer is given a few pairs of lists of integers, and the user hopes that the synthesizer will produce a sorting function. Then useful properties might include functions that check if the input and output lists have the same length, if the input list is a subset of the output, if element 0 of the output list is less than element 42, and so on. The outputs of a set of properties can be concatenated into a vector, yielding a representation that we call a property signature. Property signatures can then be used for consumption by machine learning algorithms, essentially serving as the first layer of a neural network. In this paper, we demonstrate the utility of property signatures for program synthesis, using them to perform a type of premise selection as in. More broadly, however, we envision that property signatures could be useful across a broad range of problems, including algorithm induction , improving code readability , and program analysis . More specifically, our contributions are: • We introduce the notion of property signatures, which are a general purpose way of featurizing both programs and program specifications (Section 3). • We demonstrate how to use property signatures within a machine-learning based synthesizer for a general-purpose programming language. This allows us to automatically learn a useful set of property signatures, rather than choosing them manually (Sections 3.2 and 4). • We show that a machine learning model can predict the signatures of individual functions given the signature of their composition, and describe several ways this could be used to improve existing synthesizers (Section 5). • We perform experiments on a new test set of 185 functional programs of varying difficulty, designed to be the sort of algorithmic problems that one would ask on an undergraduate computer science examination. We find that the use of property signatures leads to a dramatic improvement in the performance of the synthesizer, allowing it to synthesize over twice as many programs in less than one-tenth of the time (Section 4). An example of a complex program that was synthesized only by the property signatures method is shown in Listing 1. For our experiments, we created a specialized programming language, called Searcho 1 (Section 2), based on strongly-typed functional languages such as Standard ML and Haskell. Searcho is designed so that many similar programs can be executed rapidly, as is needed during a large-scale distributed search during synthesis. We release 2 the programming language, runtime environment, distributed search infrastructure, machine learning models, and training data from our experiments so that they can be used for future research. Listing 1: A program synthesized by our system, reformatted and with variables renamed for readability. This program returns the sub-list of all of the elements in a list that are distinct from their previous value in the list. In Inductive Program Synthesis, we are given a specification of a program and our goal is to synthesize a program meeting that specification. Inductive Synthesis is generally divided into Programming by Example (PBE) and Programming by Demonstration (PBD). This work is focused on PBE. In PBE, we are given a set of input/output pairs such that for each pair, the target program takes the input to the corresponding output. Existing PBE systems include , , and. A PBE specification might look like Listing 2: 1 io_pairs = [,,,] Listing 2: An example PBE specification. for which a satisfying solution would be the function squaring its input. Arbitrarily many functions satisfy this specification. It is interesting but out of scope 3 to think about ways to ensure that the synthesis procedure recovers the'best' or'simplest' program satisfying the specification. Much (though not all) work on program synthesis is focused on domain specific languages that are less than maximally expressive (; ; ;). We would like to focus on the synthesis of programs in a Turing complete language, but this presents technical challenges: First, general purpose languages such as C++ or Python are typically quite complicated and sometimes not fully specified; this makes it a challenge to search over partial programs in those languages. Second, sandboxing and executing code written in these languages is nontrivial. Finally, searching over and executing many programs in these languages can be quite slow, since this is not what they were designed for. For these reasons, we have created a general-pupose, Turing complete programming language and runtime. The programming language is called Searcho and it and its runtime have been designed specifically with program synthesis in mind. The language can roughly be thought of as a more complicated version of the simply typed lambda calculus or as a less complicated version of Standard ML or OCaml. 4 Searcho code is compiled to bytecode and run on the Searcho Virtual Machine. Code is incrementally compiled, which means that the standard library and specification can be compiled once and then many programs can be pushed on and popped off from the stack in order to check them against the specification. Searcho is strongly typed with algebraic datatypes 5 Searcho includes a library of 86 functions, all of which are supported by our synthesizer. This is a significantly larger language and library than have been used in previous work on neural program synthesis. We have also implemented a baseline enumerative synthesizer. The main experiments in this paper will involve plugging the outputs of a machine learning model into the configuration for our baseline synthesizer to improve its performance on a set of human-constructed PBE tasks. Consider the PBE specification in Listing 3: 1, 2345, 34567], Listing 3: An example PBE Specification. We can see that the function concatenating the input list to its reverse will satisfy the specification, but how can we teach this to a computer? we take the approach of training a machine learning model to do premise selection for a symbolic search procedure. But how do we get a representation of the specification to feed to the model? , the model acts only on integers and lists of integers, constrains all integers to lie in [−256, 256], has special-case handling of lists, and does not deal with polymorphic functions. It would be hard to apply this technique to the above specification, since the first example contains unbounded integers, the second example contains a different type than the first 6, and the third and fourth examples contain recursive data structures (lists of characters and lists of integers respectively). Thankfully, we can instead learn a representation that is composed of the outputs of multiple other programs running on each input/output pair. We will call these other programs properties. Consider the three properties in Listing 4. 1 all_inputs_in_outputs ins outs = all (map (\x -> x in outs) ins) 2 ouputs_has_dups ins outs = has_duplicates (outs) 3 input_same_len_as_output ins outs = (len ins) == (len outs) Listing 4: Three function projections that can act on the specification from Listing 3. 4 In this paper, we will present illustrative programs in Haskell syntax to make them more broadly readable. Searcho programs will be presented in Searcho syntax, which is similar. 5 Types have been shown to substantially speed up synthesis. See e.g. Figure 6 of. 6 So any function satisfying the spec will be parametrically polymorphic. Each of these three programs can be run on all 4 of the input output pairs to yield a Boolean. The first always returns True for our spec, as does the second. The third always returns False on the given examples, although note that it would return True if the examples had contained the implicit base case of the empty list. Thus, we can write that our spec has the'property signature' [True, True, False]. How is this useful? From the first property we can infer that we should not throw away any elements of the input list. From the third we might guess that we have to add or remove elements from the input list. Finally, the second might imply that we need to create copies of the input elements somehow. This does not narrow our search down all the way, but it narrows it down quite a lot. Since the properties are expressed in the same language as the programs we are synthesizing, we can emit them using the same synthesizer. Later on, we will describe how we enumerate many random properties and prune them to keep only the useful ones. The property signatures that we consider in our experiments contain thousands of values. Since the output of these properties is either always True, always False, or sometimes True and sometimes False, a neural network can learn embeddings for those three values and it can be fed a vector of such values, one for each applicable property, as the representation of a program specification. Now we describe our representation for a program f:: τ in → τ out. Each property is a program p:: (τ in, τ out) → Bool that represents a single "feature" of the program's inputs and outputs which might be useful for its representation. 7 In this section, we assume that we have determined a sequence P = [p 1 . . . p n] of properties that are useful for describing f, and we wish to combine them into a single representation of f. Later, we will describe a learning principle for choosing relevant properties. We want the property signature to summarize the output of all the properties in P over all valid inputs to f. To do this, we first extend the notion of property to a set of inputs in the natural way. If S is a set of values of type τ in and p ∈ P, we define p(S) = {p(x, f (x)) | x ∈ S}. Because p(S) is a set of booleans, it can have only three possible values, either p(S) = {True}, or p(S) = {False}, or p(S) = {True, False}, corresponding respectively to the cases where p is always true, always false, or neither. To simplify notation slightly, we define the function Π as Π({True}) = AllTrue, Π({False}) = AllFalse, and Π({True, False}) = Mixed. Finally, we can define the property signature sig(P, f) for a program f and a property sequence P as where V (τ in) is the possibly infinite set of all values of type τ in. Computing the property signature for f could be intractable or undecidable, as it might require proving difficult facts about the program. Instead, in practice, we will compute an estimated property signature for a small set of input-output pairs S io. The estimated property signature summarizes the actions of P on S io rather than on the full set of inputs V (τ in). Formally, the estimated property signature is This estimate gives us an under-approximation of the true signature of f in the following sense: If we have sig(P, S) = Mixed, we must also have sig(P, f) = Mixed. If sig(P, S) = AllTrue, then either sig(P, f) = AllTrue or sig(P, f) = Mixed, and similarly with AllFalse. Estimated property signatures are particularly useful for synthesis using PBE, because we can compute them from the input-output pairs that specify the synthesis task, without having the definition of f. Thus we can use estimated property signatures to'featurize' PBE specifications for use in synthesis. How do we choose a set of properties that will be useful for synthesis? Given a training set of random programs with random input/output examples, we generate many random properties. We then prune the random properties based on whether they distinguish between any of the programs. Then, given a test suite of programs, we do an additional pruning step: among all properties that give the same value for every element of the test suite, we keep the shortest property, because of Occam's razor considerations. Given these'useful' properties, we can train a premise selector to predict library function usage given properties. Specifically, from the remaining properties, we compute estimated property signatures for each function in the training set, based on its input output examples. Then we use the property signature as the input to a feedforward network that predicts the number of times each library function appears in the program. In Section 4, we will give more details about the architecture of this premise selector, and evaluate it for synthesis. For now, we point out that this premise selector could itself be used to find useful properties, by examining which properties are most useful for the model's predictions. Experiments in the next section will establish that property signatures let our baseline synthesizer emit programs it previously could not, but we think that they can have broader utility: • They allow us to represent more types of functions. Property signatures can automatically deal with unbounded data types, recursive data types, and polymorphic functions. • They reduce dependency on the distribution from which examples are drawn. If the user of a synthesizer gives example inputs distributed differently than the training data, the'estimated' properties might not change much. • They can be used wherever we want to search for functions by semantics. Imagine a search engine where users give a specification, the system guesses a property signature, and this signature guess is used to find all the pre-computed functions with similar semantics. • Synthesized programs can themselves become new properties. For example, once I learn a program for primality checking, I can use primality checking in my library of properties. We design an experiment to answer the following question: Can property signatures help us synthesize programs that we otherwise could not have synthesized? As we will show, the answer is yes! How Does the Baseline Synthesizer Work? Our baseline synthesizer is very similar to that in and works by filling in typed holes 9. That is, we infer a program type τ in → τ out from the specification and the synthesizer starts with a empty'hole' of type τ in → τ out and then fills it in all possible ways allowed by the type system. Many of these ways of filling-in will yield new holes, which can in turn be filled by the same technique. When a program has no holes, we check if it satisfies the spec. We order the programs to expand by their cost, where the cost is essentially a sum of the costs of the individual operations used in the program. At the beginning of the procedure, the synthesizer is given a configuration, which is essentially a weighted set of pool elements that it is allowed to use to fill in the holes. A pool element is a rewrite rule that replaces a hole with a type-correct Searcho program, which may itself contain its own, new holes. In our synthesizer, there is one possible pool element for each of the 86 library functions in Searcho, which calls the library function, with correctly-typed holes for each of its arguments. The configuration will specify a small subset of these pool elements to use during search. It is through the configuration that we will use machine learning to inform the search procedure, as we describe later. See Appendix A.1 for further details on this baseline system. How is the Training Data Generated? Our test corpus contains programs with 14 different types. For each of those 14 types, we randomly sample configurations and then randomly generate training programs for each configuration, pruning for observational equivalence. We generate up 10,000 semantically distinct programs for each type, though of course some function types admit less distinct programs than this (e.g. Bool → Bool). We also generate and prune random properties as described in Section 3.2. See Listing 5 for examples of useful properties that were generated. How was the Test Set Constructed? We've constructed a test set of 185 human generated programs ranging in complexity from one single line to many nested function calls with recursion. Programs in the test set include computing the GCD of two integers, computing the n-th fibonacci number, computing the intersection of two sets, and computing the sum of all pairs in two lists. We ensure that none of the test functions appear in the training set. See the open source code for more details on this. What is the Architecture of the Model? As mentioned above, we train a neural network to predict the number of times each pool element will appear in the output. This neural network is fully connected, with learned embeddings for each of the values AllTrue, AllFalse and Mixed. How does the Model Output Inform the Search Procedure? Since we have a large number of pool elements, we can't run the synthesizer with all pool elements if we want to find programs of reasonable length. This is both because we will run out of memory and because it will take too long. Thus, we randomly sample configurations with less pool elements. We then send multiple such configurations to a distributed synthesis server that tries them in parallel. When we use the model predictions, we sample pool elements in proportion to the model's predicted number of times that pool element appears. The baseline samples pool elements in proportion to their rate of appearance in the training set. We ran 3 different runs of our distributed synthesizer for 100,000 seconds with and without the aid of property signatures. The baseline synthesizer solved 28 test programs on average. With property signatures, the synthesizer solved an average of 73 test programs. See Figure 1 for more discussion. Indeed, it can be seen from the figure that not only did the synthesizer solve many more test programs using property signatures, but it did so much faster, synthesizing over twice as many programs in one-tenth of the time as the baseline. Most programs involve composing functions with other functions. Suppose that we are trying to solve a synthesis problem from a set of input/output examples, and during the search we create a partial program of the form f (g(x)) for some unknown g. Since we know f, we know its property signature. Since we have the program specification, we also have the estimated property signature for f • g:= f (g(x) ). If we could somehow guess the signature for g, we could look it up in a cache of previously computed functions keyed by signature. If we found a function matching the desired Figure 1: Comparison of synthesis with property signatures and without property signatures. The x-axis denotes time elapsed in seconds. Roughly speaking, we let the distributed synthesizer run for 1 day. The y-axis represenets the cumulative number of programs synthesized. On average, the baseline solved 28 of the test programs, while the baseline enhanced with property signatures solved 73 test programs (around 2.6 times as many programs). Both the baseline and the run with property signatures were run with three different random seeds. Altogether, this experiment provides strong evidence that property signatures can be useful. signature, we would be done. If no matching function exists in the cache, we could start a smaller search with only the signature of g as the target, then use that in our original search. We could attempt to encode the relationship between f and g into a set of formal constraints and pass that to a solver of some kind (De Moura & Bjørner, 2008), and while that is potentially an effective approach, it may be difficult to scale to a language like Searcho. Instead, we can simply train a machine learning model to predict the signature of g from the signature of f and the signature of f • g. Here we present an experiment to establish a proof of concept of this idea. First, we generated a data set of 10,000 random functions taking lists of integers to lists of integers. Then we randomly chose 50,000 pairs of functions from this list, arbitrarily designating one as f and one as g. We then computed the signatures of f, g and f • g for each pair, divided the data into a training set of 45,000 elements and a test set of 5,000 elements, and trained a small fully connected neural network to predict the signature of g from the other two signatures. On the test set, this model had 87.5% accuracy, which is substantially better than chance. We inspected the predictions made on the test set and found interesting examples like the one in Listing 6, where the model has learned to do something you might (cautiously) refer to as logical deduction on properties. This is suggestive of the expressive power of property signatures. It also points toward exciting future directions for research into neurally guided program synthesis. Listing 6: Example of successful prediction made by our composition predictor model. The property in question checks whether all the elements of the output list are members of the input list. For f, the value is AllTrue, and for f • g the value is Mixed. The model doesn't know g or its signature, but correctly predicts that the value of this property for g must be Mixed. There is substantial prior work on program synthesis in general. We can hardly do it justice here, but see some of;;; for more detailed surveys. Property Based Testing: Function properties are similar to the properties from Property Based Testing, a software testing methodology popularized by the QuickCheck library that has now spread to many contexts (; ; ; ; ;). Quickcheck properties are human-specified and operate on functions, while our properties operate on input/output pairs. Automated Theorem Proving: Synthesizing programs using machine learning is related to the idea of proving theorems using machine learning . Synthesis and theorem proving are formally related as well . Most existing work on synthesis approaches is from the perspective of programming language design. Our baseline synthesizer borrows many ideas from. use refinement types (roughly, a decidable version of dependent types -see) to give program specifications, allowing the type-checker to discard many candidate programs. Property signatures can be thought of as a compromise between refinement types and dependent types: we can write down specifications with them that would be impossible to express in refinement types, but we can only check those specifications empirically. More recently, researchers have used machine learning to synthesize and understand programs. We have mentioned , but see all of: introduces the idea of features: a predecessor to the idea of properties. Features differ from properties in that they are hand-crafted rather than learned, and that they were applied only on a limited string processing domain. The relationship between this work and merits special discussion. Aside from the inclusion of property signatures, they differ in the following ways: • We use a more expressive DSL. Their DSL only allows linear control flow with a small set of functions, whereas our language is Turing complete (it has looping, recursion, etc). We also have a larger set of allowed component functions: 86 vs. 34. • Their machine learning method does not work straightforwardly for arbitrary programs. Their training and test programs only deal with integers and lists of integers, while we have 14 different function types. It would thus not be feasible to compare the techniques on anything but a tiny subset of our existing test set. • The test cases in are generated from their enumerative synthesizer. It is therefore guaranteed that the synthesizer will be able to emit them in a reasonable amount of time during testing, so their demonstrated improvements are'merely' speed-ups. Our test cases are human generated, and over half of the programs synthesized using property signatures were not synthesized at all 10 given over a day of time. In this work, we have introduced the idea of properties and property signatures. We have shown that property signatures allow us to synthesize programs that a baseline otherwise was not able to synthesize, and have sketched out other potential applications as well. Finally, we have open sourced all of our code, which we hope will accelerate future research into ML-guided program synthesis. The top-down synthesizer that we use as a baseline in this work. In a loop until a satisfying program is found or we run out of time, we pop the lowest-cost partial program from the queue of all partial programs, then we fill in the holes in all ways allowed by the type system, pushing each new partial program back onto the queue. If there are no holes to fill, the program is complete, and we check it against the spec. The cost of a partial program is the sum of the costs of its pool elements, plus a lower bound on the cost of filling each of its typed holes, plus the sum of the costs of a few special operations such as tuple construction and lambda abstraction. This section contains details on the baseline synthesizer that did not fit into the main text. Figure 2 gives a more formal description of the basic synthesis algorithm. Listing 7 shows an example trajectory of partial program expansions. Listing 7: The trajectory the synthesizer took to generate the swap function, which just swaps the two elements of a tuple. Since it knows it needs to take a tuple of ints as an argument and return a tuple of ints, it starts with a hole of type (Int, Int) in line 1. It then converts that hole into a tuple of holes, both of type Int in line 2, fills one of the holes with a reference to one of the arguments in line 3, and fills in the final hole with a reference to the other argument in line 4. Note that this listing doesn't show all programs attempted, it just shows the sequence of partial programs that led to the final solution. We have conducted an experiment to compare premise selection using Property Signatures to the premise selection algorithm from . This required considerable modifications to the experimental procedure. First, since the premise-selection part of DeepCoder can only handle Integers and lists of Integers, we restricted the types of our training and test functions. In particular, we read through and found four function types in use: The types of f and g in 8 are taken directly from . The types of h and k are inferred from examples given in the appendix of . Their DSL does not technically have tuples, but we have wrapped the inputs of their'two-input-functions' in tuples for convienence. Second, since DeepCoder can only handle integers betwen −255 and 255, we first re-generated all of our random inputs (used for 'hashing' of generated training data) to lie in that range. We then generated random training functions of the above four types. We then made a data set of training functions associated with 5 input-output pairs, throwing out pairs where any of the outputs were outside the aforementioned range, and throwing out functions where all outputs contained some number outside that range. Third, of the examples in our test set with the right types, we modified their input output pairs in a similar way. We filtered out functions that could not be so modified. After doing so, we were left with a remaining test suite of 32 functions. Finally, we trained a model to predict functions-to-use from learned embeddings of the input-output pairs, as in DeepCoder. We didn't see a description of how functions with multiple inputs had their inputs embedded, so we elected to separate them with a special character, distinct from the null characters that are used to pad lists. Compared with the Property Signatures method, this technique in far fewer synthesized test set programs. We did 3 random restarts for each of DeepCoder, Property Signatures, and the Random Baseline (recall that the random baseline itself is already a relatively sophisticated synthesis algorithm -it's just the configurations that are random). The 3 DeepCoder runs synthesized an average of 3.33 test programs, while the Property Signature runs (trained on the same modified training data and tested on the same modified test data) synthesized 16.33. The random baseline synthesized 3 programs on average. A priori, this seems like a surprisingly large gap, but it actually fits with what we know from existing literature. observe something similar: which is that DeepCoder-esque techniques tend to generalize poorly to a a test set where the input-output pairs come from a different distribution than they do in training. This is the case in our experiment, and it will be the case in any realistic setting, since the test set will be provided by users. Property Signatures are (according to our experiments) much less sensitive to such shift. This makes intuitive sense: whether an input list is half the length of an output list (for instance) is invariant to the particular distribution of members of the list. Note that even if Property Signatures did not outperform DeepCoder on this subset of our test set, they would still constitute an improvement due to their allowing us to operate on arbitrary programs and inputs types.

Represent the TLDR for finding the corresponding scientific article

We represent a computer program using a set of simpler programs and use this representation to improve program synthesis techniques.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.