Question: represent

Question: Temporal logics are useful for describing dynamic system behavior, and have been successfully used as a language for goal definitions during task planning. Prior works on inferring temporal logic specifications have focused on "summarizing" the input dataset -- i.e., finding specifications that are satisfied by all plan traces belonging to the given set. In this paper, we examine the problem of inferring specifications that describe temporal differences between two sets of plan traces. We formalize the concept of providing such contrastive explanations, then present a Bayesian probabilistic model for inferring contrastive explanations as linear temporal logic specifications. We demonstrate the efficacy, scalability, and robustness of our model for inferring correct specifications across various benchmark planning domains and for a simulated air combat mission. In a meeting where multiple plan options are under deliberation by a team, it would be helpful for that team's resolution process if someone could intuitively explain how the plans under consideration differ from one another. Also, given a need to identify differences in execution behavior between distinct groups of users (e.g., a group of users who successfully completed a task using a particular system versus those who did not), explanations that identify distinguishing patterns between group behaviors can yield valuable analytics and insights toward iterative system refinement. In this paper, we seek to generate explanations for how two sets of divergent plans differ. We focus on generating such contrastive explanations by discovering specifications satisfied by one set of plans, but not the other. Prior works on plan explanations include those related to plan recognition for inferring latent goals through observations BID25 BID35, works on system diagnosis and excuse generation in order to explain plan failures BID29 BID10, and those focused on synthesizing "explicable" plans -i.e., plans that are self-explanatory with respect to a human's mental model BID16. The aforementioned works, however, only involve the explanation or generation of a single plan; we instead focus on explaining differences between multiple plans, which can be helpful in various applications, such as the analysis of competing systems and compliance models, and detecting anomalous behaviour of users. A specification language should be used in order to achieve clear and effective plan explanations. Prior works have considered surface-level metrics such as plan cost and action (or causal link) similarity measures to describe plan differences BID23 BID3. In this work, we leverage linear temporal logic (LTL) BID24 which is an expressive language for capturing temporal relations of state variables. We use a plan's individual satisfaction (or dissatisfaction) of LTL specifications to describe their differences. LTL specifications have been widely used in both industrial systems and planning algorithms to compactly describe temporal properties BID32. They are human interpretable when expressed as compositions of predefined templates; inversely, they can be constructed from natural language descriptions BID7 ) and serve as natural patterns when encoding high-level human strategies for planning constraints BID14.Although a suite of LTL miners have been developed for software engineering and verification purposes BID32 BID17 BID28, they primarily focus on mining properties that summarize the overall behavior on a single set of plan traces. Recently, BID22 presented SAT-based algorithms to construct a LTL specification that asserts contrast between two sets of traces. The algorithms, however, are designed to output only a single explanation, and are susceptible to failure when the input contains imperfect traces. Similar to Neider and Gavran, our problem focuses on mining contrastive explanations between two sets of traces, but we adopt a probabilistic approach -we present a Bayesian inference model that can generate multiple explanations while demonstrating robustness to noisy input. The model also permits scalability when searching in large hypothesis spaces and allows for flexibility in incorporating various forms of prior knowledge and system designer preferences. We demonstrate the efficacy of our model for extracting correct explanations on plan traces across various benchmark planning domains and for a simulated air combat mission. Plan explanations are becoming increasingly important as automated planners and humans collaborate. This first involves humans making sense of the planner's output (e.g., PDDL plans), where prior work has focused on developing user-friendly interfaces that provide graphical visualizations to describe the causal links and temporal relations of plan steps BID1 BID26 BID21. The outputs of these systems, however, require an expert for interpretation and do not provide a direct explanation as to why the planner made certain decisions to realize the outputted plan. Automatic generation of explanations has been studied in goal recognition settings, where the objective is to infer the latent goal state that best explains the incomplete sequence of observations BID25 BID30. Works on explicable planning emphasize the generation of plans that are deemed selfexplanatory, defined in terms of optimizing plan costs for a human's mental model of the world BID16. Mixed-initiative planners iteratively revise their plan generation based on user input (e.g. action modifications), indirectly promoting an understanding of differences across newly generated plans through continual user interaction BID27 BID3. All aforementioned works deal with explainability with respect to a single planning problem specification, whereas our model deals with explaining differences in specifications governing two distinct sets of plans given as input. Works on model reconciliation focus on producing explanations for planning models (i.e. predicates, preconditions and effects), instead of the realized plans. Explanations are specified in the form of model updates, iteratively bringing an incomplete model to a more complete world model. The term, "contrastive explanation," is used in these works to identify the relevant differences between the input pair of models. Our work is similar in spirit but focuses on producing a specification of differences in the constraints satisfied among realized plans. Our approach takes sets of observed plans as input rather than planning models. While model updates are an important modality for providing plan explanations, there are certain limitations. We note that an optimal plan generated with respect to a complete environment/world model is not always explicable or self-explanatory. The space of optimal plans may be large, and the underlying preference or constraint that drives the generation of a particular plan may be difficult to pre-specify and incorporate within the planning model representation. We focus on explanations stemming directly from the realized plans themselves. Environment/world models (e.g. PDDL domain files) can be helpful in providing additional context, but are not necessary for our approach. Our work leverages LTL as an explanation language. Temporal patterns can offer greater expressivity and explanatory power in describing why a set of plans occurred and how they differ, and may reveal hidden plan dynamics that cannot be captured by the use of surface-level metrics like plan cost or action similarities. Our work on using LTL for contrastive explanations directly contributes to exploring how we can answer the following roadmap questions for XAIP BID9: "why did you do that? why didn't you do something else (that I would have done)?" Prior research into mining LTL specifications has focused on generating a "summary" explanation of the observed traces. BID12 explored mining globally persistent specifications from demonstrated action traces for a finite state Markov decision process. BID17 introduced Texada, a system for mining all possible instances of a given LTL template from an output log where each unique string is represented as a new proposition. BID28 proposed a template-based probabilistic model to infer task specifications given a set of demonstrations. However, all of these approaches focus on inferring a specification that all the demonstrated traces satisfy. For contrastive explanations, presented SAT-based algorithms to infer a LTL specification that delineates between the positive and negative sets of traces. Unlike existing LTL miners, the algorithms construct an arbitrary, minimal LTL specification without requiring predefined templates. However, they are designed to output only a single specification, and can fail when the sets contain imperfect traces (i.e., if there exists no specification consistent with every single input trace.). We present a probabilistic model for the same problem and generate multiple contrastive explanations while offering robustness to noisy input. Some works have proposed algorithms to infer contrastive explanations for continuous valued time-series data based on restricted signal temporal logic (STL) grammar BID33 BID15. However, the continuous space semantics of STL and a restricted subset of temporal operators make the grammar unsuitable for use with planning domain problems. To the best of our knowledge, our proposed model is the first probabilistic model to infer contrastive LTL specifications for sets of traces in domains defined by PDDL. Linear Temporal Logic (LTL) provides an expressive grammar for describing temporal behavior BID24 ). An LTL specification ϕ is constructed from a set of propositions V, the standard Boolean operators, and a set of temporal operators. Its truth value is determined with respect to a trace, π, which is an infinite or finite sequence of truth assignments for all propositions in V. The notation π, t |= ϕ indicates that ϕ holds at time t. The trace π satisfies ϕ (denoted by π |= ϕ) iff π, 0 |= ϕ. The minimal syntax for LTL can be described as follows: DISPLAYFORM0 where p is a proposition, and ϕ 1 and ϕ 2 are valid LTL specifications. DISPLAYFORM1 Only one contiguous interval exists where pi is true. DISPLAYFORM2 If pi occurs, pj occurred in the past. Table 1: An example set of LTL templates. n T corresponds to the number of free propositions for each template. X reads as "next" where Xϕ evaluates as true at t if ϕ holds in the next time step t + 1. U reads as "until" where ϕ 1 Uϕ 2 evaluates as true at time step t if ϕ 1 is true at that time and going forward, until a time step is reached where ϕ 2 becomes true. In addition to the minimal syntax, we also use higher-order temporal operators, F (eventually), G (global), and R (release). Fϕ holds true at t if ϕ holds for some time step ≥ t. Gϕ holds true at t if ϕ holds for all time steps ≥ t. ϕ 1 Rϕ 2 holds true at time step t if either there exists a time step t 1 ≥ t such that ϕ 2 holds true until t 1 where both ϕ 1 and ϕ 2 hold true simultaneously, or no such t 1 exists and ϕ 2 holds true for all time steps ≥ t. Interpretable sets of LTL templates have been defined and successfully integrated for a variety of software verification systems BID32 BID20. Some of the widely used templates are shown in Table 1. According to BID8, a contrastive explanation describes "why event A occurred as opposed to some alternative event B." In our problem, events A and B represent two sets of plan traces (can be seen as traces generated from different systems or different group behavior). The form of why may be expressed in various ways BID18 ); our choice is to define it according to the plans' satisfaction of a constraint. Then, formally: Definition 3.1. A contrastive explanation is a constraint ϕ that it is satisfied by one set of plan traces (positive set, π A), but not by the other (negative set, π B).The constraint ϕ can be seen as a classifier trying to separate the provided positive and negative traces. Its performance measure corresponds to standard classification accuracy, computed by counting the number of traces in π A that satisfy ϕ and, conversely, the number of traces in π B where ϕ is unsatisfied. Formally, accuracy of ϕ is: DISPLAYFORM0 Accuracy is 1 for a perfect contrastive explanation, and approaches zero if both sets contains no valid trace with respect to ϕ (i.e., all traces in π A dissatisfy ϕ and all traces in π B satisfy ϕ). The input to the problem is a pair of sets of traces (π A, π B). Each π i ∈ π is a trace on the set of propositions V (we refer to V as the vocabulary). The output is a set of specifications, {ϕ}, where each ϕ achieves perfect or near-perfect contrastive explanation. This is an unsupervised classification problem. We use LTL specifications for the choice of ϕ. Planning is sequential, and so temporal patterns can offer greater expressivity and explanatory power for identifying plan differences rather than static facts. We utilize a set of interpretable LTL templates, such as those shown in Table 1.A LTL template T is instantiated with a selection of n T propositions denoted by p ∈ V n T. The candidate formula ϕ is then composed as a conjunction of multiple instantiations of a template T based on a set of selections {p} ⊆ V n T. For example, an instantiation of T ="stability" with p = [apple] is written as FG(apple). If the selected subset of DISPLAYFORM0, asserting the stability condition for all three propositions. Conjunctions provide powerful semantics with the ability to capture a notion of quantification. Formally, our LTL specification is written as follows: DISPLAYFORM1 Note that the number of free propositions, n T, varies per LTL template. The number of possible specifications for a given LTL template T is 2 |V | n T. Instead of extracting specifications narrowed down to a single template query, our hypothesis space Φ is set to include a number of predefined templates, T 1, T 2,...T k. With k representing the number of possible templates, the full hypothesis space of Φ grows with O(k · 2 |V | n T). Employing brute force enumeration to find {ϕ} that achieves the contrastive explanation criterion rapidly becomes intractable with increasing vocabulary size. We model specification learning as a Bayesian inference problem, building on the fundamental Bayes theorem: DISPLAYFORM0 Our goal is to infer ϕ * = argmax Φ P (ϕ|X). P (ϕ) represents the prior distribution over the hypothesis space, and Figure 1: A graphical model of the generative distribution. ϕ represents the latent LTL specification that we seek to infer given the evidence X (in our case, the traces).P (X | ϕ) is the likelihood of observing the evidence X = (π A, π B) given ϕ. We adopt a probabilistic generative modeling approach that has been used extensively in topic modeling BID2. Below, we describe each component of our generative model, depicted in Figure 1.Prior Function ϕ is generated by choosing a LTL template, T, the number of conjunctions, N, and then the proposition instantiations, p for each conjunct. The generative process for each of those components is as follows: DISPLAYFORM1 T is generated with respect to a categorical distribution with weights w T ∈ R k over the k possible LTL templates. w T is a hyperparameter that the designer can set to assert preferences for mining certain types of templates over others (e.g., preferring templates with "global" operators than "until" operators).The number of conjunctions, N = |{p}|, is generated using a geometric distribution with a decay rate of λ. Thus, the the probability of ϕ is reduced by λ for each addition of a conjunct, incentivizing low-complexity specifications defined in terms of having a fewer number of conjunctions (which also implies fewer total propositions). This promotes conciseness and prevents over-fitting to the input traces (i.e., to avoid restating the input as a long, convoluted LTL formula).Similar to the method used for template selection, we use a separate categorical distribution for selecting propositions p for each conjunct in ϕ. Propositions are generated with respect to the probability weights, w p ∈ R |V |, defined for all p in V. The designer can likewise control w p to favor specifications instantiated with certain types of propositions over others. w p may be interpreted as the level of saliency of propositions for an application. (For example, propositions that are landmarks for planning problems BID11, or a part of the causal links set BID31, may be deemed more important to express in plan explanations than other auxiliary state variables.) Several forms of variable importance, corresponding to the saliency of that importance in an explanation, may be applied to set w p. This opens the door to hypothesizing which propositions are most salient for a given domain, and generating explanations restricted to those propositions exclusively. The full prior function, P (ϕ), is evaluated as follows: DISPLAYFORM2 The derivation follows from the definition that T, N, {p} completely describe ϕ (i.e. P (ϕ | T, N, {p}) = 1), and the assumption that the three probability distributions are independent of each other. P (T) and P (N) are calculated using categorical and geometric distributions outlined in Equations 5 and 6, respectively. P ({p}) denotes the probability of the full set of proposition instantiations (over all conjuncts); it is calculated by the average categorical weight, w p, over all propositions. Formally: Likelihood Function The likelihood function P (X | ϕ) is the probability of observing the input sets of traces in the satisfying set π A and the non-satisfying set π B given the contrastive specification. The traces in π A and π B are generated by different solutions to the planning problem that satisfy the problem specification. As the problem specification is the only input needed to generate a set of plans, we assume that the individual traces are conditionally independent of each other, given the planning problem specification. With the conditional independence assumption, the likelihood can then be factored as follows: DISPLAYFORM3 DISPLAYFORM4 LTL satisfaction checks are conducted over all traces belonging to sets π A and π B; P (π i |ϕ) is set equal to 1 − α if π i |= ϕ, and α otherwise. Conversely, P (π j |ϕ) is set equal to 1−β if π j ϕ, and β otherwise. α and β permit non-zero probability to traces not adhering to the constrastive explanation criterion, thereby providing robustness to noisy traces and outliers. α and β may be set to different values to reflect the relative importance of the positive and negative sets (e.g., may be used to counteract imbalanced sets).In order to perform LTL satisfaction checks on a trace, we follow the method developed by BID17, in which ϕ is represented as a tree and each temporal operator is recursively evaluated according to its semantics. Since sub-trees of two different ϕ may be identical, we memoize and re-use evaluation to significantly speed up LTL satisfaction checks. Proposal Function Exact inference methods to find maximum a posterior (MAP) estimates, {ϕ *}, are intractable. Thus we implement a Markov Chain Monte Carlo method, specifically the Metropolis-Hasting (MH) algorithm BID6, to iteratively draw samples whose collection approximates the true posterior distribution. MH sampling requires a user-defined proposal function F (ϕ |ϕ) that samples a new candidate ϕ given the current ϕ. Our F behaves similar to an -greedy search, utilizing a drift kernel (i.e. a random walk) with a probability of 1-or sampling from the prior distribution (i.e. a restart) with a probability of. The drift kernel operates by performing one of the following moves on the current candidate LTL ϕ:• Remain within the current template T, add a new conjunct, and instantiate that conjunct with a randomly sampled p that is currently not in ϕ. The probability associated with this move, Q add, is equal to 1/(|V n T | − N).• Remain within the current template T and randomly remove one of the existing conjuncts. The probability associated with this move, Q remove, is equal to 1/N. The selection between these two moves is conducted uniformly, though there is no issue with allowing the designer to weight one more likely than the other. Note that the drift kernel perturbs ϕ, but stays within the current template. ϕ transitions to a new template (probabilistically) when choosing to sample from the prior. The probability distribution associated with F, denoted by Q(ϕ |ϕ), is then outlined as follows: DISPLAYFORM5, sample prior function Our proposal function F fulfills the ergodicity condition of the Markov process (the transition from any ϕ to ϕ is aperiodic and occurs within a finite number of steps), thus asymptotically guarantees the sampling process from the true posterior distribution. A new sample ϕ is accepted at every MH iteration with the following probability: DISPLAYFORM6 The set of accepted samples approximates the true posterior, and the MAP estimates (the output {ϕ *}) are determined from the relative frequencies of accepted samples. We evaluated the effectiveness of our model for inferring contrastive explanations from sets of traces generated from a number of International Planning Competition (IPC) planning domains BID19. The plan traces in π A were generated by first injecting the ground truth ϕ ground into the original PDDL domain and problem files, enforcing valid plans on the modified domain/problem files to satisfy ϕ ground. The LTL injection to create modified planning files was performed using the LTLFOND2FOND tool BID4. Second, a state-of-the-art top-k planner 1 BID13 ) was used to produce a set of distinct, valid plans and their accompanying state execution traces. Similarly, the above steps were repeated to generate execution traces for π B, wherein the negation of the ground truth specification, ¬ϕ ground, was injected to the planning files, and then a set of traces was collected. Such a setup guarantees the existence of contrastive explanation solutions on (π A, π B), which includes (but is not limited to) ϕ ground. We collected twenty traces for each set. We evaluated our model using six different IPC benchmark domains, containing problems related to mission planning, vehicle routing, and resource allocation. For each of these domains, we tested three different problem instances of increasing vocabulary size, and on twenty randomly generated ϕ ground specifications for each problem instance. For each test case, ϕ ground was randomly generated using one of the seven LTL templates listed in Table 1; thus the hypothesis space Φ was set to include all possible specifications over the predefined templates. The categorical distribution weights, w T and w p, were set to be uniform. Other hyperparameters were set as follows: α = β = 0.01, to put equal importance of positive and negative sets, λ = 0.7 to penalize ϕ for every additional conjunct, and = 0.2 to apply -greedy search in the the proposal function. We ran the MH sampler with num M H = 2, 000 iterations with the first 300 used as a burn-in period. These hyperparameters were set apriori, similar to how a wide range of probablistic graphical models are designed. However, our experimental were found to be robust to the various settings of these parameters. We evaluated our model against the SAT-based miner developed by BID22, the state-of-the-art for extracting contrastive LTL specifications. We also evaluated our model against brute force enumeration, a common approach employed by existing LTL miners used for summarization BID32 BID17. Because enumerating through full space of Φ would in a time out, we tested delimited enumeration with only a random subset of brute force samples. This baseline selects a random subset of size num brute from Φ. Then, a function proportional to the posterior distribution (numerator in Equation 4) is evaluated for each of the samples to determine {ϕ *}. num brute was set equal to num M H to enable a fair baseline in terms of having the same amount of allotted computation. TAB2 shows the inference on the tested domains and on problem instances of varying complexity (reflected by an increase in |V |). For evaluation, we measured M = |{ϕ *}|, the number of unique contrastive explanations extracted by the different approaches, along with the explanations' accuracy. Each domain-problem combination row shows the average statistics over twenty ϕ ground test cases. High M and high accuracy across all domain-problem combinations demonstrate how our probabilistic model was able to generate multiple, near-perfect contrastive explanations. The solution set {ϕ *} almost always included ϕ ground. Our model outperformed the baseline and the stateof-the-art miner by producing more contrastive explanations within an allotted amount of computation / runtime. The runtime for our model and the delimited enumeration baseline with 2,000 samples ranged between 1.2-4.7 seconds (increase in |V | only had marginal effect on the runtime). The SAT-based miner by Neider and Gavran often failed to generate a solution within a five minute cutoff (see the number of its timeout cases in the last column of TAB2). The prior work can only output a single ϕ *, which frequently took on a form of Fp i. It did not scale well to problems that required more complex ϕ as solutions. This is because increasing the "depth" of ϕ (the number of temporal / Boolean operators and propositions) exponentially increased the size of the compiled SAT problem. In our experiments, the prior work timed out for problems requiring solutions with depth ≥ 3 (note that Fp i has depth of 2). Robustness to Noisy Input In order to test robustness, we perturbed the input X by randomly swapping traces between π A and π B. For example, a noise rate of 0.2 would swap 20% of the traces, where the accuracy of ϕ ground on the perturbed data, X = (π A, π B), would evaluate to 0.8 (note that it may be possible to discover other ϕ that achieve better accuracy on X). The MAP estimates inferred from X, {ϕ *}, were evaluated on the original input X to assess any loss of ability to provide contrast. Figure 3 shows the average accuracy of {ϕ *}, evaluated on both X and X, across varying noise rate. Even at a moderate noise rate of 0.25, the inferred ϕ * s were able to maintain an average accuracy greater than 0.9 on X. Such a threshold is promising for real-world applications. The robustness did start to sharply decline as noise rate increased past 0.4. For all test cases, the Neider and Gavran miner failed to generate a solution for anything with a noise rate ≥ 0.1. Large values of M signify how there are often various ways to express how plan traces differ using the LTL semantics. Some LTL specifications are logically dependent. For example, the global template subsumes both the stability and the eventuality template. LTL specifications may also be related through substitutions of propositions. For example, on Figure 3: The accuracy of ϕ * with respect to increasing noise rate. ϕ * is inferred from the perturbed, noisy data and then is evaluated (generalized) on the original input X. Each domain subplot shows the averages across all three problem instances and all twenty ϕ ground test cases. 95% confidence intervals are displayed.problems where holding a block is a prerequisite to placing it onto a table, ϕ 1 = F(holding A) ∧ F(holding B) will be satisfied in concert with the satisfaction of ϕ 2 = F(ontable A)∧F(ontable B). For contrastive explanation, however, one needs to be mindful of both positive and negative sides of satisfaction which affect the accuracy. Relations like template subsumptions or precondition / effect pairs should not be simply favored during search without understanding that the converse may not hold and may in worse accuracy. For a contrastive ϕ, it is possible to create a new contrastive ϕ that includes stationary propositions or tautologies specific to the planning problem. For example, if ϕ 1 = F(holding A) ∧ F(holding B) is a contrastive explanation, so is ϕ 3 = F(holding A) ∧ F(holding B) ∧ F(earth is round). Our posterior distribution assigns a lower probability to ϕ 3 than ϕ 1 based on the decay rate on the number of conjunctions. Also, tautologies by themselves cannot be contrastive explanations, because they can never be dissatisfied. The output of our model appropriately excluded such vacuous explanations. TAB2 shows how M generally increased as |V | increased. This opens up interesting research avenues for determining a minimal set of {ϕ}. Assessing logical dependence or metric space between two arbitrary LTL specifications, however, is non-trivial. Evaluation on Real-world Inspired Domain We applied our inference model on a large force exercise (LFE) domain, which simulate air-combat games used to train pilots. Through the use of Joint Semi-Automated Forces environment BID0, realistic aircraft behavior and their state execution traces were collected for the mission objective of "gain and maintain air superiority." A total of 24 instances (i.e. traces) of LFEs were separated into positive and negative sets by a subject matter expert. The detail of the input was as follows: |π A |=16, |π B |=8, |V |=15, and the average length of traces involved 11 time steps. Within a second (2,000 samples), our model generated ten unique contrastive explanations, all with accuracy of 0.96. ϕ * 1 = G(attrition < 0.25) ∧ G(striker not shot) represents how friendly attrition rate should be always less than 25% and that the striker aircraft should never be shot upon. ϕ * 2 = (attrition < 0.25) U (weapon release) asserts how friendly attrition rate has to be less than 25% before releasing the weapon. The model also inferred rules of the environment, for example, asserting that propositions (attrition < 0.75) and (attrition < 0.50) precede (attrition < 0.25) (which makes sense because attrition can only increase throughout the mission). After discussion with the expert, we discovered that the model could not generate the perfect contrastive ϕ ground, because it required having multiple conjuncts that incorporate different LTL templates (which is not part of our defined solution space). Nevertheless, the generated explanations were consistent with the expert's interpretation of achieving the mission objective of air superiority. We have presented a probabilistic Bayesian model to infer contrastive LTL specifications describing how two sets of plan traces differ. Our model generates multiple contrastive explanations more efficiently than the state-of-the-art and demonstrates robustness to noisy input. It also provides a principled approach to incorporate various forms of prior knowledge or preferences during search. It can serve as a strong foundation that can be naturally extended to multiple input sets by repeating the algorithm for all pairwise or one-vs.-rest comparisons. Interesting avenues for future work include gauging the saliency of propositions, as well as deriving a minimal set of contrastive explanations. Furthermore, we seek to test the model in human-in-the-loop settings, with the goal of understanding the relationship between different planning heuristics for the saliency of propositions (e.g. landmarks and causal links) to their actual explicability when the explanation is communicated to a human.

We present a Bayesian inference model to infer contrastive explanations (as LTL specifications) describing how two sets of plan traces differ.

represent this text to find a much much longer article about it with say hundreds-thousands of words

For example, A new intrinsic reward signal based on successor features and a novel way to combine extrinsic and intrinsic reward. should be similar to Exploration in sparse reward reinforcement learning remains an open challenge. Many state-of-the-art methods use intrinsic motivation to complement the sparse extrinsic reward signal, giving the agent more opportunities to receive feedback during exploration. Commonly these signals are added as bonus rewards, which in a mixture policy that neither conducts exploration nor task fulfillment resolutely. In this paper, we instead learn separate intrinsic and extrinsic task policies and schedule between these different drives to accelerate exploration and stabilize learning. Moreover, we introduce a new type of intrinsic reward denoted as successor feature control (SFC), which is general and not task-specific. It takes into account statistics over complete trajectories and thus differs from previous methods that only use local information to evaluate intrinsic motivation. We evaluate our proposed scheduled intrinsic drive (SID) agent using three different environments with pure visual inputs: VizDoom, DeepMind Lab and DeepMind Control Suite. The show a substantially improved exploration efficiency with SFC and the hierarchical usage of the intrinsic drives. A video of our experimental can be found at https://gofile.io/?c=HpEwTd. Reinforcement learning (RL) agents learn on evaluative feedback (reward signals) instead of instructive feedback (ground truth labels), which takes the process of automating the development of intelligent problem-solving agents one step further . With deep networks as powerful function approximators bringing traditional RL into high-dimensional domains, deep reinforcement learning (DRL) has shown great potential (; ;). However, the success of DRL often relies on carefully shaped dense extrinsic reward signals. Although shaping extrinsic rewards can greatly support the agent in finding solutions and shortening the interaction time, designing such dense extrinsic signals often requires substantial domain knowledge, and calculating them typically requires ground truth state information, both of which is hard to obtain in the context of robots acting in the real world. When not carefully designed, the reward shape could sometimes serve as bias or even distractions and could potentially hinder the discovery of optimal solutions. More importantly, learning on dense extrinsic rewards goes backwards on the progress of reducing supervision and could prevent the agent from taking full advantage of the RL framework. In this paper, we consider terminal reward RL settings, where a signal is only given when the final goal is achieved. When learning with only an extrinsic terminal reward indicating the task at hand, intelligent agents are given the opportunity to potentially discover optimal solutions even out of the scope of the well established domain knowledge. However, in many real-world problems defining a task only by a terminal reward means that the learning signal can be extremely sparse. The RL agent would have no clue about what task to accomplish until it receives the terminal reward for the first time by chance. Therefore in those scenarios guided and structured exploration is crucial, which is where intrinsically-motivated exploration has recently gained great success (; b). Most commonly in current state-of-the-art approaches, an intrinsic reward is added as a reward bonus to the extrinsic reward. Maximizing this combined reward signal, however, in a mixture policy that neither acts greedily with regard to extrinsic reward max-imization nor to exploration. Furthermore, the non-stationary nature of the intrinsic signals could potentially lead to unstable learning on the combined reward. In addition, current state-of-the-art methods have been mostly looking at local information calculated out of 1-step lookahead for the estimation of the intrinsic rewards, e.g. one step prediction error , or network distillation error of the next state (b). Although those intrinsic signals can be propagated back to earlier states with temporal difference (TD) learning, it is not clear that this in optimal long-term exploration. We seek to address the aforementioned issues as follows: 1. We propose a hierarchical agent scheduled intrinsic drive (SID) that focuses on one motivation at a time: It learns two separate policies which maximize the extrinsic and intrinsic rewards respectively. A high-level scheduler periodically selects to follow either the extrinsic or the intrinsic policy to gather experiences. Disentangling the two policies allows the agent to faithfully conduct either pure exploration or pure extrinsic task fulfillment. Moreover, scheduling (even within an episode) implicitly increases the behavior policy space exponentially, which drastically differs from previous methods where the behavior policy could only change slowly due to the incremental nature of TD learning. 2. We introduce successor feature control (SFC), a novel intrinsic reward that is based on the concept of successor features. This feature representation characterizes states through the features of all its successor states instead of looking at local information only. This implicitly makes our method temporarily extended, which enables more structured and farsighted exploration that is crucial in exploration-challenging environments. We note that both the proposed intrinsic reward SFC and the hierarchical exploration framework SID are without any task-specific components, and can be incorporated into existing DRL methods with minimal computation overhead. We present experimental in three sets of environments, evaluating our proposed agent in the domains of visual navigation and control from pixels, as well as its capabilities of finding optimal solutions under distraction. Intrinsic Motivation and Auxiliary Tasks Intrinsic motivation can be defined as agents conducting actions purely out of the satisfaction of its internal rewarding system rather than the extrinsic rewards . There exist various forms of intrinsic motivation and they have achieved substantial improvement in guiding exploration for DRL, in tasks where extrinsic signals are sparse or missing altogether. proposed to evaluate curiosity, one of the most widely used kinds of intrinsic motivation, with the 1-step prediction error of the features of the next state made by a forward dynamics model. Their ICM module has been shown to work well in visual domains including first-person view navigation. Since ICM is potentially susceptible to stochastic transitions (a), Burda et al. (2018b) propose as a reward bonus the error of predicting the features of the current state output by a randomly initialized fixed embedding network. The value function is decomposed for extrinsic and intrinsic reward, but different to us a single mixture policy is learned. Another form of curiosity, learning progress or the change in the prediction error, has been connected to count-based exploration via a pseudo-count and has also been used as a reward bonus. propose to train a reachability network, which gives out a reward based on whether the current state is reachable within a certain amount of steps from any state in the current episode. Similar to our proposed SFC, their intrinsic motivation is related to choosing states that could lead to novel trajectories. However, we use two different distance metrics, theirs is explicitly learned to be proportional to the time step differences while ours is based on successor features which measures two states by the difference of the average feature activations of future trajectories. Moreover, their method rewards states with high distance to the states in the current episode while our method rewards states with high distance to the states also from past trajectories, as the successor features are trained from samples of the replay buffer. Auxiliary tasks have been proposed for learning more representative and distinguishable features. add depth prediction and loop closure prediction as auxiliary tasks for learning the features. learn separate policies for maximizing pixel changes (pixel control) and activating units of a specific hidden layer (feature control). However, their proposed UNREAL agent never follows those auxiliary policies as they are only used to learn more suitable features for the main extrinsic task. Hierarchical RL Various HRL approaches have been proposed (a; ; ;). In the context of intrinsic motivation, feature control has been adopted into a hierarchical setting , in which options are constructed for altering given features. However, they report that a flat policy trained on the intrinsic bonus achieves similar performance to the hierarchical agent. Our hierarchical design is perhaps inspired mostly by the work of. Unlike other HRL approaches that try to learn a set of options to construct the optimal policy, their proposed SAC agent aims to learn one flat policy that maximizes the extrinsic reward. While SAC schedules between following the extrinsic task and a set of pre-defined auxiliary tasks such as maximizing touch sensor readings or translation velocity, in this paper we investigate scheduling between the extrinsic task and intrinsic motivation that is general and not task-specific. A concurrent work along this line is presented by. Successor Representation The successor representation (SR) was first introduced to improve generalization in TD learning . While previous works extended SR to the deep setting for better generalized navigation and control algorithms across similar environments and changing goals (b; ;), we focus on its temporarily extended property to accelerate exploration. SR has also been investigated under the options framework. When using SR to measure the intrinsic motivation, the most relevant work to ours is that of. They also design a task-independent intrinsic reward based on SR, however they rely on the concept of count-based exploration and propose a reward bonus, that vastly differs from ours. Their bonus is inverse proportional to the norm of the SR while our formulation rewards change in the SR of two successive states. We will present our proposed method in the next section. We use the RL framework for learning and decision-making under uncertainty. It is formalized by Markov decision processes (MDPs) defined by the tuple S, A, p, r, γ. At time step t the agent samples an action a ∈ A according to policy π(·|s), which depends on its current state s ∈ S. The agent receives a scalar reward r ∈ R and transits to the next state s ∈ S. The distribution of the corresponding state, action and reward process (S t, A t, R t+1) is determined by the distribution of the initial state S 0, the transition operator p and the policy π. The goal of the agent is to find a policy that maximizes the expectation of the sum of discounted rewards T k=0 γ k R t+k+1. We seek to speed up learning in sparse reward RL, where the reward signal is uninformative for almost all transitions. We set the focus on terminal reward scenarios, where the agent only receives a single reward of +1 for successfully accomplishing the task and 0 otherwise. We will first introduce our proposed intrinsic reward successor feature control (SFC) (3.1,3.2), then present our proposed hierachical framework for accelerating intrinsically motivated exploration, which we denote as scheduled intrinsic drive (SID) (Sec.3.3, 3.4). In order to encode long-term statistics into the design of intrinsic rewards for far-sighted exploration, we build on the formulation of successor represention (SR), which introduces a temporally extended view of the states. introduced the idea of representing a state s by the occupancies of all other states from a process starting in s following a fixed policy π, where the occupancies denote the average number of time steps the state process stays in each state per episode. ). The agent starts at the red cross and transitions to an adjacent state at each time step. The goal is to explore the four rooms when no extrinsic reward is provided. In a) each state is annotated by its SD (Eq.3) to the starting state and b) shows for each state the highest possible SFC reward (Eq.4) for a one-step transition from it. Here the successor features are learned using a random walk. c) and d) show a comparison between visitation counts of each state from a random agent and an agent that uses the SFC rewards for control via Q-learning. In the latter case the successor features are learned from scratch via TD. In this environment, the agent receives high rewards for crossing bottleneck states, when the SF are learned beforehand, using a random policy. But even when the SF are learned during exploration, bottleneck states are still visited disproportionately high. Furthermore the intrinsic reward greatly improves exploration compared to a random agent. For implementation details see Appendix. D.4 features (SF) (b;) extend the concept to an arbitrary feature embedding φ: S → R m. For a fixed policy π and embedding φ the SF is defined by the |m|-dimensional vector Analogously, the SF represent the average discounted feature activations, when starting in s and following π. They can be learned by temporal difference (TD) updates SF have several interesting properties which make them appealing as a basis for an intrinsic reward signal: 1) They can be learned even in the absence of extrinsic rewards and without learning a transition model and therefore combine advantages of model-based and model-free RL . 2) They can be learned via computationally efficient TD. 3) They capture the expected feature activations for complete episodes. Therefore they contain information even of spatially and temporarily distant states which might help for effective far-sighted exploration. Given the discussion, we introduce the successor distance (SD) metric that measures the distance between states by the similarity of their SF Fig.1 a) shows an example of the successor distance metric in the tabular case. There the SD roughly correlates to the length of the shortest path between the states. Using this metric to evaluate the intrinsic motivation, one choice could be to use the SD to a fixed anchor state as the intrinsic reward, which depends heavily on the anchor position. Even when a sensible choice for the anchor can be found, e.g. the initial state of an episode, the SDs of distant states from the anchor assimilate. For a pair of states with a fixed spatial distance, their SD is higher when they are located in different rooms and the SD increases substantially when crossing rooms. Therefore the metric might capture the connectivity of the underlying state space. This observation motivates us to define the intrinsic reward successor feature control (SFC) as the squared SD of a pair of consecutive states A high SFC reward indicates a big change in the future feature activations when π is followed. We argue this big change is a strong indicator of bottleneck states, since in bottlenecks a minor change in the action selection can lead to a vastly different trajectory being taken. Fig.1b ) shows that those highly rewarding states under SFC and the true bottlenecks agree, which can be very valuable for exploration . The classical way of adding the intrinsic reward to the extrinsic reward has several drawbacks. First, the final policy is not trained to maximize the actual objective but a mixed version. Second, the intrinsic reward signal is usually changing over time. Including this non-stationary signal in the overall reward can make learning of the actual task unstable. Furthermore, the performance is often extremely sensitive to the scaling of the intrinsic reward relative to the extrinsic and hence it has to be tuned very carefully for every environment. To overcome these issues we propose scheduled intrinsic drive (SID), which learns two separate policies, one for each reward signal. During each episode the scheduler samples several times which of the two policies to follow for the next time steps. Each policy is trained off-policy from all of the transitions irrespective of which policy collected the data. As SID does not add the two reward signals no scaling parameter is needed. A policy is learned that exclusively maximizes extrinsic reward and hence neither the final policy nor the learning process is disturbed by the intrinsic reward. At the same time exploration is ensured as there is experience collected by the policy that learns from the intrinsic reward. Furthermore, scheduling can help exploration as each policy is acted on for an extended time interval, allowing long-term exploration instead of local exploration. Besides that the agent is less susceptible to always go to a nearby small reward instead of looking for other larger rewards that maybe further away. A mixture policy might be attracted to the small reward while with SID the exploration policy is followed for several timesteps which can bring the agent to new states with larger rewards that it did not know of before. We investigated several types of high-level schedulers, however, none of them consistently outperforms a random one. We present possible explanations why a random scheduler already performs well and present them in Appendix F along with the different scheduler choices we tested. Our proposed method can be combined with an any approach that allows off-policy learning. This section describes an instantiation of the SID framework when using Ape-X DQN as a basic offpolicy DRL algorithm with SFC as the intrinsic reward, which we used for all experiments. We depict this algorithm instance in Appendix Figure 8 and more details are provided in Appendix C. The algorithm is composed of: • A Q-Net {θ ϕ, θ E, θ I}: Contains a shared embedding θ ϕ and two Q-value output heads θ E (extrinsic) and θ I (intrinsic). • A SF-Net {θ φ, θ ψ}: Contains an embedding θ φ and a successor feature head θ ψ. θ φ is initialized randomly and kept fixed during training. The output of SF-Net is used to calculate the SFC intrinsic reward (Eq.4). The SF-net is trained with the samples of the replay buffer, which contains the experience generated by the behavior policy. • A high-level scheduler: Instantiated in each actor, selects which policy to follow (extrinsic or intrinsic) after a fixed number of environment steps (max episode length/M). The scheduler randomly picks one of the tasks with equal probability. • N parallel actors (N = 8): Each actor instantiates its own copy of the environment, periodically copies the latest model from the learner. We learn from K-step targets (K = 5), so each actor at each environment step stores (s t−K, a t−K, K k=1 γ k−1 r t−K+k, s t) into a shared replay buffer. Each actor will act according to either the extrinsic or the intrinsic policy based on the current task selected by its scheduler. • A learner: Learns the Q-Net (θ E and θ I are learned with the extrinsic and intrinsic reward respectively) and the SF-Net from samples (Eq.2) from the same shared replay buffer, which contains all experiences collected from following different policies. We evaluate our proposed intrinsic reward SFC and the hierarchical framework of intrinsic motivation SID in three sets of simulated environments: VizDoom , DeepMind Lab and DeepMind Control Suite . Throughout all experiments, agents receive as input only raw pixels with no additional domain knowledge or task specific information. We mainly compare the following agent configurations: M: Ape-X DQN with 8 actors, train with only the extrinsic main task reward; ICM: train a single policy with the ICM reward bonus ; RND: train a single policy with the RND reward bonus (b); Ours: with our proposed SID framework, schedule between following the extrinsic main task policy and the intrinsic policy trained with our proposed SFC reward. We carried out an ablation study, where we compare the performance of an agent with intrinsic and extrinsic reward summed up, to the corresponding SID agent for each intrinsic reward type (ICM, RND, SFC). We present the plots and discussions in Section 4.4 Appendix A. For the intrinsic reward normalization and the scaling for the extrinsic and intrinsic rewards we do a parameter sweep for each environment (Appendix C.4) and choose the best setting for each agent. We notice that our scheduling agent is much less sensitive to different scalings than agents with added reward bonus. Since our proposed SID setup requires an off-policy algorithm to learn from experiences generated by following different policies, we implement all the agents under the Ape-X DQN framework. After a parameter sweep we set the number of scheduled tasks per episode to M = 8 for our agent in all experiments, meaning each episode is divided into up to 8 sub-episodes, and for each of which either the extrinsic or the intrinsic policy is sampled as the behavior policy. Appendix C and D contain additional information about experimental setups and model training details. We start by verifying our implementation of the baseline algorithms in "DoomMyWayHome" which was previously used in several state-of-the-art intrinsic motivation papers . The agent needs to navigate based only on first-person view visual inputs through 8 rooms connected by corridors (Fig.2a), each with a distinct texture (Fig.2c). The experimental are shown in Fig.3 (left). Since our basic RL algorithm is doing off-policy learning, it has relatively decent random exploration capabilities. We see that the M agent is able to solve the task sometimes without any intrinsically generated motivations, but that all intrinsic motivation types help to solve the task more reliably and speed up the learning. Our method solve the task the fastest, but also ICM and RND learn to reach the goal reliably and efficiently. We wanted to test the agents on a more difficult VizDoom map where structured exploration would be of vital importance. We thus designed a new map which scales up the navigation task of MyWayHome. Inspired by how flytraps catch insects, we design the layout of the rooms in a geometrically challenging way that escaping from one room to the next with random actions is extremely unlikely. We show the layout of MyWayHome (Fig.2a) and FlytrapEscape (Fig.2b) with the same downscaling ratio. The maze consists of 4 rooms separated by V-shaped walls pointing inwards the rooms. The small exits of each room is located at the junction of the V-shape, which is extremely difficult to maneuver into without a sequence of precise movements. As in the MyWayHome task, in each episode, the agent starts from the red dot shown in Fig.2b with a random orientation. An episode terminates if the final goal is reached and the agent will receive a reward of +1, or if a maximum episode steps of 10,000 (2100 for MyWayHome) is reached. The task is to escape the fourth room. The experimental on FlytrapEscape are shown in Fig.3 (right). Neither M nor RND manages to learn any useful policies. ICM solves the task in sometimes, while we can clearly observe that our method efficiently explores the map and reliably learns how to navigate to the goal. We visualize the learned successor features in Appendix E and its evolution over time is shown in the video https://gofile.io/?c=HpEwTd. In the second experiment, we set out to evaluate if the agents would be able to reliably collect the faraway big reward in the presence of small nearby distractive rewards. For this experiment we use the 3D visual navigation simulator of DeepMind Lab . We constructed a challenging level "AppleDistractions" (Fig.10b) with a maximum episode length of 1350. In this level, the agent starts in the middle of the map (blue square) and can follow either of the two corridors. Each corridor has multiple sections and each section consists of two dead-ends and an entry to next section. Each section has different randomly generated floor and wall textures. One of the corridors (left) gives a small reward of 0.05 for each apple collected, while the other one (right) contains a single big reward of 1 at the end of its last section. The optimal policy would be to go for the single faraway big reward. But since the small apple rewards are much closer to the spawning location of the agent, the challenge here is to still explore other areas sufficiently often so that the optimal solution could be recovered. The are presented in Fig.4 (left). Ours received on average the highest rewards and is the only method that learns to navigate to the large reward in every run. The baseline methods get easily distracted by the small short-term rewards and do not reliably learn to navigate away from the distractions. With a separate policy for intrinsic motivation the agent can for some time interval completely "forget" about the extrinsic reward and purely explore, since it does not get distracted by the easily reachable apple rewards and can efficiently learn to explore the whole map. In the meanwhile the extrinsic policy can simultaneously learn from the new experiences and might learn about the final goal discovered by the exploration policy. This highlights a big advantage of scheduling over bonus rewards, that it reduces the probability of converging to bad local optimums. In Section 4.4 we further showed that SID is generally applicable and also helps ICM and RND in this task. To show that our methods can be used in domains other than first-person visual navigation, we evaluate on the classic control task "cartpole: swingup sparse" , using third-person view images as inputs (Fig.11). The pole starts pointing down and the agent receives a single terminal reward of +1 for swinging up the unactuated pole using only horizontal forces on the cart. Additional details are presented in Appendix D.3. The are shown in Fig.4 (right). Compared to the previous tasks, this task is easy enough to be solved without intrinsic motivation, but we can see also that all intrinsic motivation methods significantly reduce the interaction time. Ours still outperforms other agents even in the absence of clear bottlenecks which shows its general applicability, but since the task is relatively less challenging for exploration, the performance gain is not as substantial as the previous experiments. Further, we conducted an ablation study on AppleDistractions. We denote with "M+SFC", "M+RND", "M+ICM" the agents with one policy where the respective intrinsic reward is added to the extrinsic one. With "SID(M,SFC)", "SID(M,RND)", "SID(M,ICM)" the agents are named that have two policies, one for the respective intrinsic and one for the extrinsic reward and use SID to schedule between them. We note that the "SID(M,ICM)" agent corresponds to the "Ours" agent from the previous experiments. The are presented in Figure 5. Our SID(M, SFC) agent received on average the highest rewards. Furthermore, we see that scheduling helped both ICM and SFC to find the goal and not settle for the small rewards, and SID also helps improve the performance of RND. The respective reward bonus counterparts of the three SID agents were more attracted to the small nearby rewards. This behavior is expected: By scheduling, the intrinsic policy of the SID agent is assigned with its own interaction time with the environment, during which it could completely "forget" about the extrinsic rewards. The agent then has a much higher probability of discovering the faraway big reward, thus escaping the distractions of the nearby small rewards. Once the intrinsic policy collects these experiences of the big reward, the extrinsic policy can immediately learn from those since both policies share the same replay buffer. Ablations for the other environments are reported in Appendix A. In this paper, we investigate an alternative way of utilizing intrinsic motivation for exploration in DRL. We propose a hierarchical agent SID that schedules between following extrinsic and intrinsic drives. Moreover, we propose a new type of intrinsic reward SFC that is general and evaluates the intrinsic motivation based on longer time horizons. We conduct experiments in three sets of environments and show that both our contributions SID and SFC help greatly in improving exploration efficiency. We consider many possible research directions that could stem from this work, including designing more efficient scheduling strategies, incorporating several intrinsic drives (that are possibly orthogonal and complementary) instead of only one into SID, testing our framework in other control domains such as manipulation, combining the successor representation with learned feature representations and extending our evaluation onto real robotics systems. We have conducted ablation studies for all the three sets of environments to investigate the influence of scheduling on our proposed method, whether other reward types can benefit from scheduling too, and whether environment specific differences exist. We compare the performance of the following agent configurations: • Three reward bonus agents M+ICM, M+RND, M+SFC: The agent receives the intrinsic reward of ICM , RND (b) or our proposed SFC respectively as added bonus to the extrinsic main task reward and trains a mixture policy on this combined reward signal. We note that the M+ICM and M+RND agent in this section corresponds to the ICM and RND agent in all other sections respectively. The agent schedules between following the extrinsic main task policy and the intrinsic policy trained with the ICM, RND or our proposed SFC reward respectively. We note that the SID (M, SFC) agent in this section corresponds to the Ours agent in all other sections. The on AppleDistractions were shown in in Section 4.4 the main paper. In Fig.6 (left), we present the ablation study for FlytrapEscape. The agents with the ICM component perform poorly. Only 1 run of M+ICM learned to navigate to the goal, while the scheduling agent SID(M,ICM) did not solve the task even once. But for the two SFC agents, the scheduling greatly improves the performance. Although the reward bonus agent M+SFC was not successful in every run, the SID(M,SFC) agent solved the FlytrapEscape in 5 out of 5 runs. We hypothesize the reason for the superior performance of SID(M,SFC) compared to M+SFC could be the following: Before seeing the final goal for the first time, the M+SFC agent is essentially learning purely on the SFC reward, which is equivalent to the intrinsic policy of the scheduling SID(M,SFC) agent. Since SFC might preferably go to bottleneck states as the difference between the SF of the two neighboring states are expected to be relatively larger for those states. Since the extrinsic policy is doing random exploration before receiving any reward signal, it could be a good candidate to explore the next new room from the current bottleneck state onwards. Then the SFs of the new room will be learned when it is being explored, which would then guide the agent to the next bottleneck regions. Thus the SID(M,SFC) agent could efficiently explore from bottleneck to bottleneck, while the M+SFC agent could not be able to benefit from the two different behaviors under the extrinsic and intrinsic rewards and could oscillate around bottleneck states. On the other hand, scheduling did not help ICM or RND. A reason could be that ICM or RND is not especially attracted by bottleneck states so it does not help exploration if the agent spends half of the time acting randomly as the extrinsic policy had no reward yet to learn from. Also since the FlytrapEscape environment is extremely explorationchallenging, the temporally extended view of our proposed SFC might of vital importance to guide efficient exploration. In Fig.6 (right), we present the ablation study for Cartpole. We can observe that SID helps to improve the performance of both ICM and RND. As for SFC, although the reward bonus agent learns a bit faster than the SID agent, we note that actually all the three SID agent converge to more stable policies, while the reward bonus agents tend to oscillate around the optimal return. In experiments of the main paper with SID the scheduler chooses 8 times per episode which policy to follow until a new policy is chosen. We conducted a further experiment to examine how different numbers of switches per episode affects the performance of the agent. We carried out the experiment on "DoomMyWayHome" with an agent using SID with SFC for the intrinsic reward. The are show in 7 C APPENDIX: IMPLEMENTATION DETAILS This section describes implementation details and design choices. The backbone of our algorithm implementation is presented in Section 3.4 and visualized in Fig. 8. Since our algorithm requires an off-policy learning strategy, and in consideration for faster learning and less computation overhead, we use the state-of-the-art off-policy algorithm with the K-step target (K = 5) for bootstraping without off-policy correction where θ − denotes the target network parameters. We chose the number of actors the be the highest the hardware supported, which was 8. To adapt the settings from the 360 actors in the Ape-X DQN to our setting of N = 8 actors, we set a fixed i for each actor i ∈ {1, . . ., 8} as i = where α = 7 and = 0.4 are set as in the original work. For computational efficiency, we implement our own version of the prioritized experience replay. We split the replay buffer into two, with size of 40, 000 and 10, 000. Every transition is pushed to the first one, while in the second one only transitions are pushed on which a very large TD-error is computed. We store a running estimate of the mean and the standard deviation of the TD-errors and if for a transition the error is larger than the mean plus two times the standard deviation, the transition is pushed. In the learner a batch of size 128 consists of 96 transitions drawn from the normal replay buffer and 32 are drawn from the one that stores transition with high TD-error, which as a have relatively seen a higher chance of being picked. We note that previous works for learning the deep SF have included an auxiliary task of reconstruction on the features φ Kulkarni et al. (2016a); , while in this work we investigate learning ψ without this extra reconstruction stream. Instead of adapting the features φ while learning the successor features ψ, we fix the randomly initialized φ. This design follows the intuition that since SF (ψ) estimates the expectation of features (φ) under the transition dynamics and the policy being followed, more stable learning of the SF could be achieved if the features are kept fixed. The SF are learned from the same replay buffer as for training the Q-Net. Since our base algorithm is K-step Ape-X, and we follow the memory efficient implementation of the replay buffer as suggested in Ape-X, we only have access to K-step experience tuples (K = 5) for learning the SF. Therefore we calculate the intrinsic reward by applying the canonical extension of the SFC reward formulation (Eq.4) to K-step transitions The behaviour policy π associated with the SF is not given explicitly, but since the SF are learned from the replay buffer via TD learning, it is a mixture of current and past behaviour policies from all actors. Most network parameters are shared for estimating the expected discounted return of the intrinsic and extrinsic rewards. The scale of the rewards has a big influence on the scale of the gradients for the network parameters. Hence, it is important that the rewards are roughly on the same scale, otherwise effectively different learning rates are applied. The loss of the network comes from the regression on the Q-values, which approximate the expected return. So our normalization method aims to bring the discounted return of both tasks into the same range. To do so we first normalize the intrinsic rewards by dividing them by a running estimate of their standard deviation. We also keep a running estimate of the mean of this normalized reward and denote it r I. Since every time step an intrinsic reward is received we estimate the discounted return via the geometric series. We scale the extrinsic task reward that is always in {0, 1} with η, where γ I is the discount rate for the intrinsic reward. Furthermore, η is a hyperparameter which takes into account that for Q-values from states more distant to the goal the reward is discounted with the discount rate for the extrinsic reward depending on how far away that state is. In our experiments we set η = 3. We did the same search for hyperparameters and normalization technique for all algorithms that include an intrinsic reward and found out that the procedure above works best for all of them. The algorithms were evaluated on the FlytrapEscape. For η we tried the values in {0.3, 1, 3, 10}. We also tried to not normalize the rewards and just scale the intrinsic reward. To scale the intrinsic reward we tried the values {0.001, 0.01, 0.1, 1}. However, we found that as the scale of the intrinsic rewards is not the same over the whole training process this approach does not work well. We also tried to normalize the intrinsic rewards by dividing it by a running estimate of its standard deviation and then scale this quantity with a value in {0.01, 0.1, 1}. We use the same model architecture as depicted in Fig. 9 across all 3 sets of experiments. ReLU activations are added after every layer except for the last layers of each dashed blocks in the above figure. For the experiments with the ICM , we added BatchNorm before activation for the embedding of the ICM module following the original code released by the authors. Code is implemented in pytorch . For all experiments we used a stack of 4 consecutive, preprocessed observations as states. For the first-person view experiments in VizDoom and DeepMind Lab, we use an action repetition of 4, while for the classic control experiment we did not apply action repetition. In the text, we only refer to the actual environment steps (e.g. before divided by 4). The VizDoom environment produces 320 × 240 RGB images as observations. In a preprocessing step, we downscaled the images to 84 × 84 pixels and converted them to grayscale. For FlytrapEscape, we adopted the action space settings from the MyWayHome task. The action space was given by the following 5 actions: TURN LEFT, TURN RIGHT, MOVE FORWARD, MOVE LEFT, MOVE RIGHT We setup the DmLab environment to produce 84 × 84 RGB images as observations. In Fig.10 we show examplary observations of AppleDistractions. We preprocessed the images by converting the observations to grayscale. For a given enviroment seed, textures for each segment of the maze are generated at random. We used the predefined DmLab actions from. The action space was given by the following 8 actions (no shooting setting): Forward, Backward, Strafe Left, Strafe Right, Look Left, Look Right, Forward+Look Left, Forward+Look Right. We conducted the experiments for the classic control task on the'Cart-pole' domain with the'swingup sparse' task provided by the DeepMind Control Suite. Since our agents needs a discrete action space and the control suite only provides continuous action spaces, we discretized the single action dimension. The set of actions was {-0.5, -0.25, 0, 0.25, 0.5}. We configured the environment to produce 84×84 RGB pixel-only observations from the 1st camera, which is the only predefined camera that shows the full cart and pole at all times. We further convert the images to grey-scale and stack four consecutive frames as input to our network. The episode length was 200 environment steps. In the four-room domain, the agent can transition to directly connected states using the four actions'up','down','left' and'right'. For Fig.1 a),b) the successor features were calculated analytically via the formula Ψ = (I − γP) − 1, where P denotes the one-step transition matrix. The SF discount factor was set to γ = 0.95. For Fig.1 c),d) the agents performed 10000 episodes with 30 steps each. These short episodes ensure that exploration remains challenging, even in a relatively small environment. In d), the learning rate for the SF as well as the Q-table was set to 0.05. To prevent optimistic initialization effects the Q-table was initialized to 0. To generate our we used two machines that run Ubuntu 16.04. Each machine has 4 GeForce Titan X (Pascal) GPUs. On one machine we run 4 experiments in parallel, each experiment on a separate GPU. E APPENDIX: SUCCESSOR DISTANCE VISUALIZATION Figure 12: Projection of the SFs. For the purpose of visualization we discretized the map into 85 × 330 grids and position the trained agent SID(M,SFC) at each grid, then computed the successor features ψ for that location for each of the 4 orientations (0°, 90°, 180°, 270°), which ed in a 4 × 512 matrix. We then calculated the l2-difference of this matrix with a 4 × 512 vector containing the successor features of the starting position with the 4 different orientations. Shown in log-scale. As an additional evaluation, we visualize the SF of an Ours agent (i.e. trained with SID which schedules between the extrinsic policy and the SFC policy) at the end of the training (Fig.12). That means the SF are trained with the experiences from the behaviour policy of our agent. We can see that the SD from each coordinate to the starting position tends to grow as the geometric distance increases, especially for those that locate on the pathways leading to later rooms. This shows that the learned SD and the geometric distance are in good agreement and that the SF are learned as expected. Furthermore, we observe big intensity changes around the bottlenecks (the room entries) in the heatmap, which also supports the hypothesis that SFC leads the agent to bottleneck states. We believe this is the first time that SF are shown to behave in a first-person view environment as one would expect from its definition. The evolution of the SF over time is shown in the video https://gofile.io/?c=HpEwTd. When learning optimal value functions or optimal policies via TD or policy gradient with deep function approximators, optimizing with algorithms such as gradient descent means that the policy would only evolve incrementally: It is necessary that the TD-target values do not change drastically over a short period of time in order for the gradient updates to be meaningful. The common practice of utilizing a target network in off-policy DRL stabilizes the update but in the meanwhile making the policy adapt even more incrementally over each step. But intrinsically motivated exploration, or exploration in general, might benefit from an opposite treatment of the policy update. This is because the intrinsic reward is non-stationary by nature, as well as the fact that the exploration policy should reflect the optimal strategy corresponding to the current stage of learning, and thus is also non-stationary. With the commonly adopted way of using intrinsic reward as a bonus to the extrinsic reward and train a mixture policy on top, exploration would be a balancing act between the incrementally updated target values for stable learning and the dynamically adapted intrinsic signals for efficient exploration. Moreover, neither the extrinsic nor the intrinsic signal is followed for an extended amout of time. Therefore, we propose to address this issue with a hierarchical approach that by design has slowly changing target values while still allowing drastic behavior changes. The idea is to learn not a single, but multiple policies, with each one optimizing on a different reward function. To be more specific, we assume to have N tasks T ∈ T (e.g. N = 2 and T = {T E, T I} where T E denotes the extrinsic task and T I the intrinsic task) defined by N reward functions (e.g. R E and R I) that share the state and action space. The optimal policy for each of these N different MDPs can be learned with arbitrary off-policy DRL algorithms. During each episode, a high-level scheduler periodically selects a policy for the agent to follow to gather experiences, and each policy is trained with all experiences collected following those N different policies. The overall learning objective is to maximize the extrinsic reward E ω(T|St) E π T (At|St) [q T E (S t, A t |A t ∼ π T (·|S t))] (ω: the macro-policy of the scheduler). By allowing the agent to follow one motivation at a time, it is possible to have a pool of N different behavior policies without creating unstable targets for off-policy learning. By scheduling M times even during an episode, we implicitly increase the behavior policy space by exponential to N M for a single episode. Moreover, disentangling the extrinsic and intrinsic policy strictly separates stationary and non-stationary behaviors, and the different sub-objectives would each be allocated with its own interaction time, such that extrinsic reward maximization and exploration do not distract each other. We investigated several types of high-level schedulers, however, none of them consistently outperforms a random one. We suspect the reason why a random scheduler already performs very well under the SID framework, is that a highly stochastic schedule can be beneficial to make full use of the big behavior policy space. We investigated three types of high-level schedulers: • Random scheduler: Sample a task from uniform distribution every task steps. • Switching scheduler: Sequentially switches between extrinsic and intrinsic task. • Macro-Q Scheduler: Learn a scheduler that learns with macro actions and from subsampled experience tuples. In each actor, we keep an additional local buffer that stores N + 1 subsampled experiences: {s t−N m, . . ., s t−2m, s t−m, s t}. Then at each environment step, Besides the K-step experience tuple mentioned above, we also store an additional macro-transition {s t−N m, s t} along with its sum of discounted rewards to the shared replay buffer. This macro-transition is paired with the current task as its macro-action. The Macro-Q Scheduler is then learned with an additional output head attached to θ ϕ (we also tried θ φ). • Threshold-Q Scheduler: Selects task according to the Q-value output of the extrinsic task head. For this scheduler no additional learning is needed. It just selects a task based on the current Q-value of the extrinsic head θ e. We tried the following selection strategies: -Running mean: select intrinsic when the current Q-value of the extrinsic head is below its running mean, extrinsic otherwise -Heuristic median: observing that the running mean of the Q-values might not be a good statistics for selecting tasks due to the very unevenly distributed Q-values across the map, we choose a fixed value that is around the median of the Q-values (0.007), and choose intrinsic when below, extrinsic otherwise As we report in the paper, none of the above scheduler choices consistently performs better across all environments than a random scheduler. We leave this part to future work. G APPENDIX: SINGLE RUNS

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.