Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In competitive situations, agents may take actions to achieve their goals that unwittingly facilitate an opponent’s goals. We consider a domain where three agents operate: a user (human), an attacker (human or a software) agent and an observer (a software) agent. The user and the attacker compete to achieve different goals. When there is a disparity in the domain knowledge the user and the attacker possess, the attacker may use the user’s unfamiliarity with the domain to its advantage and further its own goal. In this situation, the observer, whose goal is to support the user may need to intervene, and this intervention needs to occur online, on-time and be accurate. We formalize the online plan intervention problem and propose a solution that uses a decision tree classifier to identify intervention points in situations where agents unwittingly facilitate an opponent’s goal. We trained a classifier using domain-independent features extracted from the observer’s decision space to evaluate the “criticality” of the current state. The trained model is then used in an online setting on IPC benchmarks to identify observations that warrant intervention. Our contributions lay a foundation for further work in the area of deciding when to intervene. When an agent is executing a plan to achieve some goal, it's progress may be challenged by unforeseen changes such as an unexpected modification to the environment or an adversary subverting the agent's goal. In these situations, a passive observer intervening to help the agent reach it's intended goal will be beneficial. Intervention is different from the typical plan recognition problem because we assume the observed agent pursues desirable goals while avoiding undesirable states. Therefore, the observer must monitor actions/state unobtrusively to predict trajectories of the observed agent (keyhole recognition) and assist the observed agent to safely complete the intended task or block the current step if unsafe. Consider a user checking email on a computer. An attacker who wants to steal the user's password makes several approaches: sending an email with a link to a phishing website and sending a PDF file attachment embedded with a keylogger. The user, despite being unaware of the attacker's plan, would like to complete the task of checking email safely and avoid the attacker's goal. Through learning, our observer can recognize risky actions the user may execute in the environment and ensure safety. The decision of when to intervene must be made judicially. Intervening too early may lead to wasted effort chasing down false positives, helpful warnings being ignored as a nuisances, or leaking information for the next attack. Intervening too late may in the undesirable state. Further, we are interested in assisting a human user with different skill levels, who would benefit more from customized intervention. To this end, we need to identify actions that warrant intervention over three different time horizons: critical action, which if unchecked will definitely trigger the undesirable state, mitigating action, which gives the user some time to react because the threat is not imminent and preventing actions, which allows for long term planning to help the user avoid threats. Based on the time horizon we are current in, we can then plan to correct course accordingly. In this work we focus on identifying the first horizon. Intervention is useful in both online settings, where undesirable states may arrive incrementally and in offline settings where observations are available prior to intervention. In this paper, we model online intervention in a competitive environment where three agents operate: a user (human), an attacker (human or a software) agent and an observer (a software) agent who will intervene the user. The observer passively monitors the user and the attacker competing to achieve different goals. The attacker attempts (both actively and passively) to leverage the progress made by a user to achieve its own goal. The attacker may mask domain knowledge available to the user to expand the attack vector and increase the likelihood of a successfull attack. The user is pursuing a desirable goal while avoiding undesirable states. Using domain-independant features, we train a decision tree classifier to help the observer decide whether to intervene. A variation of the relaxed plan graph BID0 ) models the desirable, undesirable and neutral states that are reachable at different depths. From the graph, we extract several domain independent features: risk, desirability, distances remaining to desirable goal and undesirable states and active landmarks percentage. We train a classifier to recognize an observation as a intervention point and evaluate the learned model on previously unseen observation traces to assess the accuracy. Furthermore, the domain independent features used in the classifier offer a mechanism to explain why the intervention occurred. In real-time, making the decision to intervene for each observation may be costly. We examine how the observer can establish a wait time without compromising accuracy. The contributions of this paper include: formalizing the online intervention problem as an intervention graph that extends the planning graph, introducing domainindependent features that estimate the criticality of the current state to cause a known undesirable state, presenting an approach that learns to classify an observation as intervention or not, incorporating salient features that are better predictors of intervention to generate explanations, and showing this approach works well with benchmarks. Before we formalize the problem, we present examples for two cases of the online intervention: the attacker is actively trying to make the user reach the undesirable state by leveraging the user's progress and the passive attacker introduces an undesirable state to the environment without the user's knowledge (i.e., a trap), where attacker masks the location of the trap and exploits the user's unfamiliarity with the domain to make the user reach the undesirable state. In both cases, the observer monitors the attacker and the users' actions. The user plans for a desirable goal state, G d. Given the unexpected modification to the domain model, executing this plan may likely cause the user to reach the undesirable state (G u). The observer is assumed to be familiar with the domain (regardless of attacker's attempts to mask information to the user) and has knowledge about commonly occurring goals such as G d and G u. The user would like to be interrupted if some action will trigger G u.Active Attacker: We use the IPC block-words domain BID5 to illustrate the active attacker's case. The observer is watching the user stacking blocks to spell a word. The domain contains 4 blocks: T, B, A, D. Figure 1 shows the undesirable state developing from initial state I. G d equals the word TAD, while G u equals the word BAD. The user can not recognize block B (indicated by dotted lines), which prevents the user from identifying states ing from performing operations on B such as stack and pick up, and therefore fail to circumvent G u on his own. The attacker will use block B to defeat the user and achieve G u.In the initial state (I), all blocks are on the table. The user's arm (solid line) and the attacker's arm (dotted line) are empty. In the next sequence of events, the observer sees that the user has picked up block A (S 1) and stacked A on D (S 2). Consider two alternative timelines T 1 and T 2 stemming from S 2. In T 1, the observer sees that the user has picked up T and the attacker has also picked up B. The next state shows that the user has stacked T on A to spell the word TAD and reached G d successfully. In timeline T 2, the attacker has succeeded in reaching G u by stacking B on A before the user stacked T on A, leveraging the user's progress. Passive Attacker: This case considers the 3x3 grid world domain BID12 shown in FIG1. The observer watches the user (white circle) navigating from a start point on the grid to reach G d point in 1-step actions. When executing a plan to reach G d, the user would like to avoid the trap at point X, G u but will not be able to do so unless the observer interrupted. Let us assume the observer sees the user's action ing in state S 1. Although the move indicates that the user is moving toward G u and G d, interruption is too early. In two alternative timelines T 1 (top right) and T 2 (bottom), the observer sees different moves. In T 1 the user has reached G d while avoiding G u, in which case the observer need not interrupt. However, in T 2 the user has reached G u, in which case it would have been helpful if the user was blocked before moving to. Our formulation of the intervention problem makes several assumptions about the three actors. Observer: intervention decisions are made in an online setting for each observation that appears incrementally and include actions executed by the attacker or the user. The goals G d or G u are known but the plans to reach G d or G u are hidden. The domains for which plan intervention problem is defined are discrete and all actions are assumed to be of unit cost. The observer has full observability in the domain and the environment is deterministic. Therefore, it can determine the actions that are immediately applicable in the current state. User: Follows a plan to reach G d, but may reach G u unwittingly. G u is hidden, but would like the observer's help to avoid G u. The user does not have full observability of the domain or the attacker's actions. Attacker: Follows a plan to reach G u. The attacker has full observability of the domain and the user's actions. Given these assumptions, the observer assesses the state after each observation. This requires the observer to hypothesize about possible interesting trajectories from current state and evaluate each trajectory in terms of their likelihood to cause G u. Following STRIPS BID4, we define a planning problem as a tuple P = F, A, I, G where F is the set of fluents, I ⊆ F is the initial state, G ⊆ F represents the set of goal states and A is the set of actions. Each action a ∈ A is a triple a = P re(a), Add(a), Del(a) that consists of preconditions, add and delete effects respectively, where P re(a), Add(a), Del(a) are all subsets of F. An action a is applicable in a state s if preconditions of a are true in s; pre(a) ∈ s. If an action a is executed in state s, it in a new state s = (s \ del(a) ∪ add(a)). The solution to P is a plan π = {a 1, . . ., a k} of length k that modifies I into G by execution of actions a 1,..., a k. The plan recognition problem defined by is a triple T = D, G, O where D = F, A, I is a planning domain, G is the set of goals, and m]. A solution to the plan recognition problem is a subset of goals G ∈ G for which an optimal plan P [G] satisfying O is produced. DISPLAYFORM0 Similarly, the plan intervention problem (I) also uses observations of actions. However, instead of using information gleaned from the observation trace to find the most likely plans (and goals), the intervention problem aims to assess the current state for it's ability to cause G u and identify whether or not the user needs to be blocked from making further progress. Unlike Ramirez and Geffeners' approach, the observations used in our solution are not noisy nor do they contain missing actions. This will be addressed in future work. Plan DISPLAYFORM1, and a decision tree classifier model M that combines a vector of domain-independant features to classify an obervation as requiring intervention or not. The extension to typical plan/goal recognition comes from the domain-independent feature vector, which will be discussed in section 3.3. A solution to I is a vector of decision points corresponding to actions in O indicating whether each action was identified as requiring intervention. To assess the criticality of the current state to cause G u, the observer enumerates action sequences that will transform the current state to G d. These action sequences and intermediate states make up the observer's decision space, which is a single-root directed acyclic connected graph S = V, E, where V is the set of vertices denoting possible states the user could be in until G d is reached, and E is the set of edges representing actions from A. We refer to this graph as the intervention graph. The root of the intervention graph indicates the current state. Leaves of the graph are goal states (i.e., G u and G d). A path from root of the tree to G u represents a candidate attack plan, while a path from root to leaf node containing G d represents a desirable plan. FIG2 illustrates the observer's decision space for unobserved actions extending from state S 1 in Figure 1. Some subtrees are hidden for simplicity. Given the initial state where all 4 blocks are on the table, the observer expects the next action to be one in the set (PICK-UP {T, D, A, B}), but B is hidden from the user. The attacker can execute any FIG3 illustrates how the user's plans to reach G d could fail in the presence of an active (left) or passive (right) attacker. In the case of an active attacker, given the assumption that the attacker does not backtrack to a previous state and only leverages progress made thus far, it can make four attempts to prevent the user from reaching G u by inserting the hidden block into the partially built stack. If the user achieves goal states 1 or 4 the user wins despite the attacker. If the observed actions indicate that the user is heading toward one of these two states, then an interrupt is unwarranted. State 3 is less ideal for the user but G u is not achieved. In state 2 the attacker has successfully reached G u. Observations leading to state 2 warrant interruption. DISPLAYFORM0 In the case of a passive attacker, the observer needs to hypothesize about likely goals of the user given the current state. FIG3 (right) shows three of many such plans the user may follow to reach G d. Paths 1, 2 and 3 all in user going past the undesirable state (marked x), and at some point in these observation sequences the user must be interrupted before G u is reached. In contrast, path 4 indicates a safe path and must not generate an interrupt. Algorithm 1 describes how the intervention graph is built. The intervention graph is similar to the relaxed planning graph (RPG), where each level consists of predicates that have been made true and actions a ∈ A whose preconditions are satisfied. Initially, before any observations have been made, the current state (i.e., root of the tree) is set to initial state I. Next, using the domain theory D, actions a ∈ A whose preconditions are satisfied at current state are added to the graph. Each action in level i spawn possible states for level i + 1. Calling the method recursively for each state until G d and G u are added to some subsequent level in the graph will generate a possible hypotheses space for the observer. As a new observation arrives, the root of the graph is changed to reflect the new state after the observation and subsequent layers are also modified to that effect. Similar to the RPG, we omit delete effects during construction. Also construction terminates once G d is reached. The graph building algorithm does not allow adding backtracking actions because it will create a cycle. for a ∈ A where P re(a) ∈ s i do 6: DISPLAYFORM0 e ← AddEdge (s, s i+1, a)10: DISPLAYFORM1 We extract a set of features from the intervention graph that help determine when to intervene. These features include: Risk, Desirability, Distance to G d, Distance to G u and Percentage of active undesirable landmarks in current state. We use these features to train a decision tree. FIG4 illustrates a fragment of the intervention graph after PICK-UP A. Following the subtree extending from action STACK A D, both G u and G d can be reached. Unexpanded subtree T 1 also contains instances where the user can reach G d safely, without reaching G u. We will use FIG4 as a running example to discuss feature computation. Risk (R) quantifies how likely the effects of current observation will lead to G u. R is also coupled with the uncertainty the observer has about the next observation. We model the uncertainty as a uniform probability distribution across the set of actions whose preconditions are satisfied in current state. We define R as the posterior probability of reaching G u while the user is trying to achieve G d. Given the intervention graph, we extract paths from root to any leaf containing the G d, including the ones in which the user has been subverted to reach G u instead. By virtue of construction termination, G d will always be a leaf. R is computed for paths leading to state in FIG3 (left) because in that state the attacker has won. In the passive attacker case any path in the intervention graph that causes the user to reach point X, before G d is reached qualifies as candidates to compute R. Let Π candidates be the plans reaching G d and let |Π candidates | = n. The plan set Π u contains action sequences that reach state G u such that, Π u ⊆ Π candidates, |Π u | = m and (m <= n). We compute posterior probability of reaching G u for a path π ∈ Π u, using chain rule in probability as, P π = k j=1 P (α j |α 1, α 2, ..., α k−1), and α j ∈ A and k is the length of path until G u is reached. Then: DISPLAYFORM0 There are six action sequences the observer might observe when the user is trying to achieve G d (n = 6) and only one of those six sequences will make the user reach G u (m = 1). Since we assumed full observability for the observer, the root of the tree (current state) is assigned the probability of 1.0. Then, actions that are immediately possible after current state (STACK A B, STACK A D, STACK A T) are each assigned probabilites following a uniform distribution across the branching factor (0.33). Then for each applicable action in the current state, the ing state gets the probability of (1.0 × 0.33 = 0.33). Similarly, we apply the chain rule of probability for each following state and action level in the graph until G u first appears in the path. In this graph, G u appears two actions later and R = 0.08 1 = 0.08. Desirability (D) measures the effect of the observed action to help the user pursue the desirable goal safely. It separates common harmless actions from avoidable ones and connects the observations to knowledge of the goals the user wants to achieve. Given Π candidates as the set of plans extracted from the intervention graph that reach G d and |Π candidates | = n. The plan set Π d contains action sequences that reach state G d without reaching G u, Π d = Π candidates \Π u, we compute posterior probability of reaching G d without reaching G u for a path π ∈ Π d, using chain rule in probability as, P π = k j=1 P (α j |α 1, α 2, ..., α k−1), and α j ∈ A and k is the length of path. Then: DISPLAYFORM1 In FIG4, there are five instances where user achieved R and D are based on probabilities indicating the confidence the observer has about the next observation. We also use simple distance measures: distance to G u (δ u) and distance to G d (δ d). Both distances are measured in the number of actions required to reach a state containing G d or G u from root in the intervention graph. DISPLAYFORM2 Distance to G u (δ u) measures the distance to state G u from the current state in terms of the number of actions. As with the computations of R and D, given Π candidates is the set of paths extracted from the intervention graph that reach G d and |Π candidates | = n. The path set Π u contains action sequences that reach state G u such that, Π u ⊆ Π candidates, |Π u | = m and (m <= n). We count s, the number of the edges (actions) before G u is reached for each path π ∈ Π u and δ u is defined as the average of the distance values given by the formula: DISPLAYFORM3 In this formula, −1 indicates that the undesirable state is not reachable from the current state. For the example problem illustrated in FIG4, δ u = 3 1 = 3. Distance to G d (δ d) measures the distance to G d from current state. The path set Π d contains action sequences that reach G d without reaching G u, Π d = Π candidates \ Π u, we count t, the number of the edges where G d is achieved without reaching G u for each path π ∈ Π d. Then, δ d is defined as the average of the distances given by the formula: DISPLAYFORM4 In this formula, −1 indicates that G d can not be reached safely from the current state. For the example problem illustrated in Figure Percentage of active attack landmarks (L ac) captures the criticality of current state toward contributing to G u. Landmarks BID6 are predicates (or actions) that must be true in every valid plan for a planning problem. We used the algorithm in BID6 to extract fact landmarks for the planning problem P = D, G u. These landmarks are referred to as attack landmarks because they establish predicates that must be true to reach G u. Landmark Generation Graph (LGG) BID6 for P for the active attacker case is shown in Figure 6. Predicates (ON B A), (ON A D) correspond to G u. Predicates that are grouped must be made true together. When the observed actions activate any attack landmarks, it signals that an undesirable state is imminent. Landmarks have been successfully used in deriving heuristics in plan recognition BID18 PICK UP A PICK UP B G u Figure 6: LGG for P. Contains verified fact landmarks for P and greedy-necessary orders. A box with multiple landmarks indicate fact landmarks that must be true together. for o ∈ O do 4: DISPLAYFORM0 Apply action probabilities to e ∈ E following p 6:Apply state probabilities to v ∈ V following p 7: DISPLAYFORM1 and generating alternative plans BID3 ). We compute a feature using attack landmarks: percentage of active attack landmarks in current state (L ac). To compute L ac for the example in FIG4, we count the number of landmark predicates that have become active (l) in the root of the intervention graph. Then, (L ac) is given by the formula: DISPLAYFORM2 In FIG4 We train the decision tree classifier in supervised learning mode to categorize observed actions into two classes: "Y" indicating that the interruption is warranted and "N", indicating that intervention is unwarranted. According to this policy, in the expanded sub-tree in FIG4 the path that reaches G u is labeled as follows: PICK-UP A (N), STACK A D (N), PICK-UP B (N), STACK B A (Y). Label for each action is indicated within brackets. We will make this labeled data set available for the community. Given a labeled observation set and corresponding feature vectors, we train the decision tree classifier with 10-fold cross validation. Then the trained model is used to predict intervention for previously unseen intervention problems. We decided to chose the decision tree as the classifier because the decision tree learned model had the highest accuracy in predicting intervention on new problems compared to the two other classifiers: random forests BID2 and Naive Bayes. To generate training data we first created twenty planning problems for each benchmark domain. Then observation traces corresponding to each problem were generated. We enforced a limit of 100 observation traces for each planning problem for grid domains. These observation traces were provided as input to Algorithm 2. The algorithm takes a PDDL domain, a set of undesirable and desirable states and a probability distribution as input and produces a relation V of observations and feature vectors. We train a decision tree classifier using the Weka 1 framework. We selected the implementation of C4.5 algorithm BID14 ) (J48), which builds a decision tree using the concept of information entropy. We chose the decision tree classifier for its ability determine salient features for intervention, which facilitates generating explanations for the user. We focus on two questions: Using domain-independent features indicative of the likelihood to reach G u from current state, can the intervening agent correctly interrupt to prevent the user from reaching G u? and If the user was not interrupted now, how can we establish a wait time until the intervention occurred before G u? To address the first question, we evaluated the performance of the learned model to predict intervention on previously unseen problems. The experiment suit consists of the two example domains from Section 2. To this we added Navigator and Ferry domains from IPC benchmarks. In Navigator domain, an agent simply moves from one point in grid to another goal destination. In the Ferry domain, a single ferry moves cars between different locations. To simulate intervention in active attacker case (the Block-Words domain), we chose word building problems. The words user and the attacker want to build are different but they have some common letters (e.g., TAD/BAD). The attacker is able to exploit the user's progress on stacking blocks to complete word the attacker wants to build. In Easy-IPC and Navigator domains, we designated certain locations on the grid as traps. The goal of the robot is to navigate to a specific point on the grid safely. In the Ferry domain a port is compromised and a ferry carrying a car there in an undesirable state. The ferry's objective is to transport cars to specified locations without passing a compromised port. In addition to the trained data set, we also generated 3 separate instances of 20 problems each (total of 60) for the benchmark domains to produce testing data for the learned model. The three instances contained intervention problems that were different the trained instances. For example, number of blocks in the domain (block-words), size of grid (navigator, easy-ipc), accessible and inaccessible paths on the grid (navigator, easy-ipc), properties of artifacts in the grid (easy-ipc). For each instance we generated 10 observation traces for each planning problem (i.e., 200 observation traces per instance). We define true-positive as the classifier correctly predicting "Y". True-negative is an instance where the classifier correctly predicts "N". False-positives are instances where classifier incorrectly predicts an observation as an interrupt. False-negatives are instances where the classifier incorrectly predicts the observation not as an interrupt. When a human user receives an interruption, the user may like to know a reason. To extract salient features for intervention, we applied a correlation based feature selection technique in data pre-processing step to identify the top four 1 http://www.cs.waikato.ac.nz/ml/weka/ best predictors. Feature selection reduces complexity of the model, makes the outcome of the model easier to interpret, and reduces over-fitting. The attribute selector in Weka uses the Pearson's correlation to measure predictive ability between nominal attributes and the class. Our feature vector consists of nominal attributes. Table 1 summarizes top 4 correlated features for each domain. Risk is the best performing feature. Distance desirable state feature is the next best choice for a feature. The percentage of active attack landmarks was the weakest predictor of intervention across all benchmark domains and was removed from training. Interrupting at each observation: Assuming the decision to intervene is made for every observation, we calculated the true-positive rate (TPR= T P T P +F N), false-positive rate (FPR= TAB3 summarizes TPR, FPR, TNR, FNR for predicting intervention in unseen observation traces. The classifier works well in identifying intervention across domains. In line with our expectation, TPR and TNR are very high (> 95%) across domains and FNR and FPR is very low(< 5%). Because the accuracy remains consistant across test instances we conclude that the model is reasonably tolerant for modifications in the domain such as grid sizes and number of objects. Delaying the interruption: In real-life, making the intervention decision for every observation may be costly. If we are intervening a human user, he may disregard frequent interruptions as noise. For this reason, we examine how to establish a wait time until intervention occurs for the first time. We used the feature (L ac) as a checkpoint for the intervening agent to wait safely without interrupting the user. We modified the observation traces to contain action sequences starting from a point where the current state contained 50% and 75% of active landmarks. For problem instances where 75% active landmark percentage was infeasible, we limited it to the maximum active landmark percentage. We used the same learned model to predict intervention for these modified traces. For each domain, row'Delayed50' in table 2 summarizes TPR, FPR, TNR, FNR for predicting interruptions for benchmark domains given that the decision is delayed until 50% <= L ac < 75%. The row'Delayed75' indicates that the decision was delayed until L ac >= 75%.Accuracy is not affected significantly by delaying the intervention from the chosen checkpoints. However, a negative effect of delaying intervention is missing true positives. We evaluated how the delay affects the percentage of true positive observations missed. TAB4 summarizes these . Intuitively, the longer the delay, a higher percentage of true positives will be missed. For the Blocks-Word domain, there is no effect between the the delay until 50% and 75%. In both cases the delaying the decision does not cause the intervening agent to miss any true positives. The most significant loss occurs in Navigator domain, where delay until 75% will cause a loss of 2%-28% while delaying until 50% is the safest choice. The Ferry domain exhibits a similar pattern where the delay until 75% landmarks become active will cause a loss of 8%-18%. We conclude that delaying interruptions can be controlled by the percentage of active landmarks in the current state and that for certain domains it is a trade off between loss of true-positives and the delay. When an observation that warrants intervention is identified intervening agent issues a warning (and an explanation) to the user. The user needs to take corrective/mitigating actions to avoid the undesirable state. The decision trees can help explain intervention. Decision trees generated for the benchmark domains are shown in FIG8. Combining the shallow trees and the definitions of the features allow us to generate a clear and succinct set of rules to explain intervention. For the Block-word domain, FIG8, the rule that explains intervention first looks at the value of Risk. If the risk is less than or equal to 0.5 then that observation does not qualify as an intervention point. By definitions, this means that from the current state there are multiple ways to reach the undesirable state, indicating the observation is a common action that can be perceived as harmless. Next, if the observation that has a risk level of grater than 0.5 (indicating there are fewer ways of reach the undesirable state and that it's imminent), next feature to look at is the distance to the undesirable state. If the distance is negative, indicating that execution of this step will trigger the undesirable state, then the observation warrants intervention. Otherwise the observation does not require intervention. With this decision tree, an explanation for intervention in Blocks-words domain can be developed as: The current step was intervened because the risk level is significant (> .5) and the effect of this observed action will trigger the undesirable state. For the passive attacker domains FIG8 -(b),(c),(d)) the learned model generated even simpler trees with only one feature being used to determine intervention. For Easy-IPC and Navigator domains, the Risk feature determines the class of an observation. This leads to generating explanations for the Easy-IPC and Navigator domains such as The current step was intervened because the risk level is significant (> .75 for Easy-IPC and > .5 for Navigator). For the Ferry domain, Distance to G d determines intervention. A negative value indicates that if the next step was executed there is no way to reach the desirable goal state without triggering the undesirable state. Thus an explanation of intervention for the Ferry domain will be: The current step was intervened because the effect of this step will make it impossible to reach the desired goal without triggering the undesirable state. Closely related areas of literature for this work is plan/goal recognition. Plan recognition is the problem of inferring the course of action (i.e., plan) an actor may take towards achieving a goal from a sequence of observations BID17 BID9. The constructed plan, if followed to completion, is expected to in states that correspond to goals of the actor, which in turn presupposes that the actor intends to achieve those goals. Plan/goal recognition approaches in the literature explore both domain-dependent and independent methods. In domain-dependent methods agents rely heavliy on domain knowledge for inference. For example, BID8 presents a solution that recognizes an agents adversarial intent by mapping observations made to date to a plan library. BID1 discuss how to construct and manipulate domain models that describe behaviors of adversaries in computer security domain and use these models to generate plans. Another approach uses Goal Driven Autonomy (GDA) that allows agents to continuously monitor the current plans execution and assess if the current state matches with expectation BID11. More recent work attempts to separate this knowledge dependency by allowing the agent to learn knowledge from observations (Jaidee, Muñoz-Avila, and W. Aha 2011). In contrast, domain-independent goal recognition that use planning to infer agents goals. Ramirez and Geffner (2009; used an existing planner to generate hypotheses from observations to infer a single agent's plan. Their approaches offer advantages of being more adaptive to input as well as exploiting existing planning systems and plan representations. Their first approach computed the set of goals that can be achieved by optimal plans that match the observations. The second approach removed the optimality constraint and computed a probability distribution across possible plans that could be generated from existing planners BID16 . BID10 introduced the worst-case distinctiveness (wcd) metric as a measurement of the ease of performing goal recognition in a domain. The wcd problem finds the longest sequence of actions an agent can execute while hiding its goal. They show that by limiting the set of available actions in the model wcd can be minimized, which will allow the agent to reveal it's goal as early as possible. In online recognition, BID18 propose an approach that combines goal-mirroring and landmarks to infer the goal of an agent. Landmarks are used to minimize the number of hypotheses the agent has to evaluate, thus improving the effeciency of the recognition process. BID13 combines Ramirez and Geffener's plan recognition approach and leverages landmarks to counterplan and block an opponent's goal achievement. The main difference between plan intervention and recognition is that, in intervention the time intervention happens is critical. In plan recognition, identifying the plan at the right time is not a priority. The user's preferences in intervention (e.g., in-time, targetted intervention vs. prolonged and incremental) and the source of uncertainty in the environment (e.g., environment, attacker) complicate the intervening agent's decisioni and can be seen as trade-offs. Furthermore, our approach complements existing approaches by using a decision tree to identify events that warrant intervention and identifying salient features that may be useful in generating explanations to plan intervention. We formalized the online plan intervention problem in a competitive domain where an attacker both actively and passively attempts to leverage progress made by a user to achieve the attacker's own conflicting goals. We introduced the intervention graph, which models the decision space of an observer, whose goal is to support the user by blocking actions that allows the attacker to achieve his goal. We trained a classifier using domain-independent features extracted from the intervention graph to evaluate the criticality of the current state. The model predicts intervention with high accuracy for the benchmark domains. Our solution suffers from state space explosion for large domains. As an solution, we suggest sampling from alternative plans generated from off-the-shelf planners. This will also allow us to compare the proposed approach with existing online goal-recognition methods. The uncertainty model can be extended to limiting the observer's ability to fully perceive the current state. We recognize the attack models (for both active and passive cases) can be expanded to different threat models. For example, the attacker can behave as truly adversarial and undo progress the user has made so far and guide the user towards an entirely different goal. We will improve on explanations by suggesting actions that will help the user avoid the undesirable state when intervention occurs, instead of delegating the responsibility of being safe to the user, and integrating causal reasoning to explanations. These extensions lay a foundation for applying classical planning techniques for decision support and assistive agents.

We introduce a machine learning model that uses domain-independent features to estimate the criticality of the current state to cause a known undesirable state.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We introduce the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended to policy gradient methods.


Question: Represent the scientific article to find a short TLDR

Question: Deep generative models can emulate the perceptual properties of complex image datasets, providing a latent representation of the data. However, manipulating such representation to perform meaningful and controllable transformations in the data space remains challenging without some form of supervision. While previous work has focused on exploiting statistical independence to \textit{disentangle} latent factors, we argue that such requirement can be advantageously relaxed and propose instead a non-statistical framework that relies on identifying a modular organization of the network, based on counterfactual manipulations. Our experiments support that modularity between groups of channels is achieved to a certain degree on a variety of generative models. This allowed the design of targeted interventions on complex image datasets, opening the way to applications such as computationally efficient style transfer and the automated assessment of robustness to contextual changes in pattern recognition systems. Deep generative models, by learning a non-linear function mapping a latent space to the space observations, have proven successful at designing realistic images in a variety of complex domains (objects, animals, human faces, interior scenes). In particular, two kinds of approaches emerged as state-of-the-art (SOTA): Generative Adversarial Networks (GAN) , and Variational Autoencoders (VAE) . Efforts have been made to have such models produce disentangled latent representations that can control interpretable properties of images . However, the ing models are not necessarily mechanistic (or causal) in the sense that interpretable properties of an image cannot be ascribed to a particular part, a module, of the network architecture. Gaining access to a modular organization of generative models would benefit the interpretability and allow extrapolations, such as generating an object in a that was not previously associated with this object, as illustrated in a preview of our experimental in Fig. 1. Such extrapolations are an integral part of human representational capabilities (consider common expressions such as "like an elephant in a china shop") and consistent with the modular organization of its visual system, comprising specialized regions encoding objects, faces and places (see e.g.). Extrapolations moreover likely support adaptability to environmental changes and robust decision making . How to leverage trained deep generative architectures to perform such extrapolations is an open problem, largely due to the non-linearities and high dimensionality that prevent interpretability of computations performed in successive layers. In this paper, we propose a causal framework to explore modularity, which relates to the causal principle of Independent Mechanisms, stating that the causal mechanisms contributing to the overall generating process do not influence nor inform each other . 1 We study the effect of direct interventions in the network from the point of view that the mechanisms involved in generating data can be modified individually without affecting each other. This principle can be applied to generative models to assess how well they capture a causal mechanism . Causality allows to assay how an outcome would have changed, had some variables taken different values, referred to as a counterfactual . We use counterfactuals to assess the role of specific internal variables in the overall functioning of trained deep generative models, along with a rigorous definition of disentanglement in a causal framework. Then, we analyze this disentanglement in implemented models based on unsupervised counterfactual manipulations. We show empirically how VAEs and GANs trained on image databases exhibit modularity of their hidden units, encoding different features and allowing counterfactual editing of generated images. Related work. Our work relates to the interpretability of convolutional neural networks, which has been intensively investigated in discriminative architectures (; ; ; b; a). Generative models require a different approach, as the downstream effect of changes in intermediate representations are high dimensional. InfoGANs. β-VAEs and other works (; ; ;) address supervised or unsupervised disentanglement of latent variables related to what we formalize as extrinsic disentanglement of transformations acting on data points. We introduce the novel concept of intrinsic disentanglement to uncover the internal organization of networks, arguing that many interesting transformations are statistically dependent and are thus unlikely to be disentangled in the latent space. This relates to who proposed a framework based on interventions on internal variables of a GAN which, in contrast to our fully unsupervised approach, requires semantic information. suggest a definition of disentanglement based on group representation theory. Compared to this proposal, our approach (introduced independently in ) is more flexible as it applies to arbitrary continuous transformations, free from the strong requirements of representation theory (see Appendix F). Finally, an interventional approach to disentanglement has also be taken by , who focuses on extrinsic disentanglement in a classical graphical model setting and develop measures of interventional robustness based on labeled data. We introduce a general framework to formulate precisely the notion of disentanglement and bridge it to causal concepts. This theory section will be presented informally to ease high level understanding. Readers interested in the mathematical aspects can refer to Appendix A where we provide all details. FRAMEWORK We consider a generative model M that implements a function g M, which maps a latent space Z to a manifold Y M where the learned data points live, embedded in ambient Euclidean space Y (Fig. 2a). A sample from the model is generated by drawing a realization z from a prior latent variable distribution with mutually independent components, fully supported in Z. We will use the term representation to designate a mapping r from Y M to some representation space R (we also call r(y) the representation of a point y ∈ Y M ). In particular, we will assume (see Def. 6 and Prop. 6 M is a representation of the data, called the latent representation. Assuming the generative model is implemented by a non-recurrent neural network, we can use a causal graphical model representation of the entailed computational graph implementing the mapping g M through a succession of operations (called functional assignments in causal language), as illustrated in Fig. 2b, that we will call Causal Generative Model (CGM). In addition to the latent representation, we can then choose a collection of possibly multi-dimensional endogenous (internal) variables represented by nodes in the causal graph, such that the mapping g M is computed by composing the endogenous variable assignment v M with the endogenous mappingg M according to the diagram 1 Note that this is not a statistical independence; the quantities transformed by the mechanisms of course do influence each other and can be statistically dependent. (a) Independence of mechanisms Endogenous variables Latent A paradigmatic choice for these variables is the collection of output activation maps of each channel in one hidden layer of a convolutional neural network, as illustrated in Fig. 2b. As for the latent case, we use mild conditions that guaranteeg M to be left-invertible, defining the internal representation of the network (see Def. 6 and Prop. 4 in Appendix A). Given the typical choice of dimensions for latent and endogenous variables, the V k's are also constrained to take values in subsets V k M of smaller dimension than their Euclidean ambient space V k. As detailed in Appendix A, we will denote the endogenous image sets of the form ) k∈E, z ∈ Z for a subset of variables indexed by E (amounting to V M when E includes all endogenous variables). This CGM framework allows defining counterfactuals in the network following. Definition 1 (Unit level counterfactual, informal). Given CGM M, the interventional model M h is obtained by replacing assignments of the subset variables E and by the vector of assignments h. Then for any latent input z, called unit, the unit-level counterfactual is the output of Def. 1 is also in line with the concept of potential outcome . Importantly, counterfactuals induce a transformation of the output of the generative model. Given an embedded CGM, we call the transformation We introduce faithfulness of a counterfactual mapping to account for the fact that not all interventions on internal variables will in an output that could have been generated by the original model. In the context of generative model, non-faithful counterfactuals generate examples that leave the support of the distribution learned from data, possibly ing in an artifactual output (assigning a large value to a neuron may saturate downstream neurons), or allowing extrapolation to unseen data. The classical notion of disentangled representation (e.g. ;), posits individual latent variables "sparsely encode real-world transformations". Although the concept of real-world transformations remains elusive, this insight, agnostic to statistical concepts, has driven supervised approaches to disentangling representations, where relevant transformations are well-identified and manipulated explicitly using appropriate datasets and training procedures. In contrast, unsupervised learning approaches to disentanglement need to learn such real-world transformations from unlabeled data. In order to address this challenge, SOTA approaches seek to encode such transformations by changes in individual latent factors, and resort to a statistical notion of disentanglement, enforcing conditional independence between latent factors . This statistical approach leads to several issues: • The i.i.d. constraints on the prior distribution of latent variables, impose statistical independence between disentangled factors on the data distribution. This is unlikely for many relevant properties, counfounded by factors of the true data generating mechanisms (e.g. skin and hair color). • Independence constraints are not sufficient to specify a disentangled representation, such that the problem remains ill-posed . As a consequence, finding an appropriate inductive bias to learn a representation that benefits downstream tasks remains an open question. • To date, SOTA unsupervised approaches are mostly demonstrated on synthetic datasets, and beyond MNIST disentangling complex real world data has been limited to the well-calibrated CelebA dataset. On complex real-world datasets, disentangled generative models exhibit visual sample quality far below non-disentangled SOTA (e.g. BigGAN exploited in our work ). We propose an non-statistical definition of disentanglement by first phrasing mathematically the transformation-based insights . Consider a transformation T acting on the data manifold Y M. As illustrated by the commutative diagram of Fig. 2c, disentanglement of such property then amounts to having T correspond to a transformation T of the latent space that would act only on a single variable z k, using transformation f, leaving the other latent variables available to encode other properties. More explicitly we have It is then natural to qualify two transformations T 1 and T 2 as disentangled (from each other), whenever they modify different components of the latent representation (see Def. 9. This amounts to saying that the transformations follow the causal principle of independent mechanisms . Due to the fact that it relies on transformation of the latent representation, that are exogenous to the CGM, we call this notion extrinsic disentanglement. This "functional" definition has the benefit of being agnostic the the subjective choice of the property to disentangle, and to the statistical notion of independence. However, we can readily notice that, if applied to the latent space (where components are i.i.d. distributed), this functional notion of disentangled transformation still entails statistical independence between disentangled factors. We thus need to exploit a different representation to uncover possibly statistically related properties, but disentangled in the sense of our definition. As illustrated in the CGM of Fig. 2b, in contrast to latent variables, properties encoded by endogenous variables of the graphical model are not necessarily statistically independent due to common latent cause, but may still reflect interesting properties of the data that can be intervened on independently, following the principle of independence of mechanisms. We thus extend our definition of disentanglement to allow transformations of the internal variables of the network as follows. intrinsically disentangled with respect to a subset E of endogenous variables, if there is transformation T acting on the internal representation space such that for any endogenous value v where T (v) only affects the variables indexed by E. on variables that remain within the support of the original marginal distribution, it is sufficient that E and its complement E do not have common latent ancestors. This indicates that finding faithful counterfactuals can be used to learn disentangled transformations. Building on Sec. 2, we define modularity as a structural property of the internal representation, allowing (with the immediately following Prop. 2) to implement arbitrary disentangled transformations. Definition 4 (Modularity). A subset of endogenous variables E is called modular whenever. If E is modular, then any transformation applied to it staying within its input domain is disentangled. The proof is a natural extension of the proof of Proposition 1. Both the Definition and the Proposition have trivial extensions to multiple modules (along the line described in Appendix F). While we have founded this framework on a functional definition of disentanglement that applies to transformations, the link made here with an intrinsic property of the trained network allows us to define a disentangled representation as follows: consider of partition of the intermediate representation in several modules, such that their Cartesian product is a factorization of V M. We can call this partition a disentangled representation since any transformation applied to a given module leads to a valid transformation in the data space (it is relatively disentangled following Def. 9). Interestingly, we obtain that a disentangled representation requires the additional introduction of a partition of the considered set of latent variables into modules. This extra requirement was not considered in classical approaches to disentanglement as it was assumed that each single scalar variables could be considered as an independent module. Our framework provides an insight relevant to artificial and biological systems: as the activity of multiple neurons can be strongly tied together, the concept of representation may not be meaningful at the "atomic" level of single neurons, but require to group them into modules forming a "mesoscopic" level, at which each group can be intervened on independently. As stated in Sec. 2, a functional definition of disentanglement, leaves unanswered how to find relevant transformations. Prop. 1 and 2 provide the following hints: Once a modular structure is found in the network, a broad class of disentangled transformations are available, Transformations that stay within their input domain are good candidates of disentanglement, Counterfactual interventions implicitly defines transformation. We follow these guidelines by assigning a constant value v 0 to a subset of endogenous variables E to define counterfactuals (i.e. h is a constant function), aiming for faithful ones by constraining v 0 to belong to V E M. To avoid characterizing V E M, we rely on sampling from the (joint) marginal distribution of the variables in E. To illustrate the procedure, we consider a standard feed-forward multilayer neural network and choose endogenous variables to be the collection of all output activations of channels of a given layer. Let E be a subset of these channels, the hybridization procedure, illustrated in Fig. 3a goes as follows. We take two independent examples of the latent variable z 1 and z 2, that will generate two original examples of the output (y 1, y 2) = (g M (z 1), g M (z 2)) (that we call Original 1 and Original 2). We also memorize the tuple v(z 2) gathering values of variables indexed by E when generating Original 2, and v(z 1) the tuple of values taken by all other endogenous variables on this layer, but when generating Original 1. Assuming the choice of E identifies a modular structure, v(z 1) and v(z 2) would encode different aspects of their corresponding generated images, such that one can generate a hybrid example mixing these features by assigning the collection of output values of layer with the concatenated tuple (v(z 1), v(z 2)) and feeding it to the downstream part of the generator network. The above counterfactual hybridization framework allows assessing how a given module E affects the output of the generator. For this purpose we quantify its causal effect by repetitively generating pairs (z 1, z 2) from the latent space, where both vectors are sampled independently of each other. We then generate and collect hybrid outputs following the above described procedure for a batch of samples and use them to estimate an influence map as the mean absolute effect: where Y (z 1) = g M (z 1) is the non-intervened output of the generator for latent input z 1. In eq. 2, the difference inside the absolute value can be interpreted as a unit-level causal effect in the potential outcome framework , and taking the expectation is analogous to computing the average treatment effect. Our approach has however two specificities: we take the absolute value of the unit-level causal effects, as their sign may not be consistent across units, the is averaged over many interventions corresponding to different values of z 2. While IM has the same dimension as the output image, we then average it across color channels to get a single grayscale heat-map pixel map. We also define a scalar quantity to quantify the magnitude of the causal effect, the individual influence of module E, by averaging IM across output pixels. A challenge with the hybridization approach is to select the subsets E to intervene on, especially with networks containing a large amount of units or channels per layer. We use a fine to coarse approach to extract such groups, that we will describe in the context of convolutional layers. First, we estimate elementary influence maps (EIM) associated to each individual output channel c of each convolutional layer of the network (i.e. we set E = {c} in eq.). Then influence maps are grouped by similarity to define modules at a coarser scale, as we will describe in detail below. Representative EIMs for channels of convolutional layers of a VAE trained on the CelebA face dataset (see section) are shown in Supplementary Fig. 6 and suggest channels are functionally segregated, with for example some influencing finer face feature (eyes, mouth,...) and others affecting the of the image or the hair. This supports the idea that individual channels can be grouped into modules that are mostly dedicated to one particular aspect of the output. In order to achieve this grouping in an unsupervised way, we perform clustering of channels using their EIMs as feature vectors as follows. We first pre-process each influence map by: performing a local averaging with a small rectangular sliding window to smooth the maps spatially, thresholding the ing maps at the 75% percentile of the distribution of values over the image to get a binary image. After flattening image dimensions, we get a (channel×pixels) matrix S which is then fed to a Non-negative Matrix Factorization (NMF) algorithm with manually selected rank K, leading to the factorization S = WH. From the two ing factor matrices, we get the K cluster template patterns (by reshaping each rows of H to image dimensions), and the weights representing the contribution of each of these pattern to individual maps (encoded in W). Each influence map is then ascribed a cluster based on which template pattern contributes to it with maximum weight. The choice of NMF is justified by its success in isolating meaningful parts of images in different components . However, we will also compare our approach to the classical k-means clustering algorithm. In order to further justify our NMF based approach, we also introduce a toy generative model. Model 1. Consider Z a vector of K i.i.d. uniformly distributed RVs. Assume a neural network with one hidden layers composed of m vector variables V k such that with H k ∈ R n, n > 1 and S a strictly increasing activation function applied entry-wise to the components of each vector (e.g. a leaky ReLU). These endogenous variables are mapped to the output with matrices W k ∈ R m×n, m > nK. Assume additionally the following random choice for the model parameters: all coefficients of H k's are sampled i.i.d. from an arbitrary distribution that has a density with respect to the Lebesgue measure, there exists K sets of indices I k over [1, m] each containing at least one element l k ∈ I k such that for all j = k, l k / ∈ I j, For a given column of W k, coefficient in I k are sampled i.i.d. from an arbitrary distribution that has a density with respect to the Lebesgue measure, while the remaining coefficients are set to zero. The specific condition on the I k's enforced in encodes the assumption that there is an area in the image that is only influenced by one of the modules. For example, assuming a simple /object module pair, it encodes that the borders of the image never belong to the object while the center of the image never belong to . For this model, we get the following identifiability . Proposition 3. For Model 1, with probability 1 we have: The partition of the hidden layer entailed by the K vectors {V k} corresponds to a disentangled representation (i.e. each vector is modular relatively to the others). This justifies the use of NMF of a thresholded version of the influence map matrix computed for individual endogenous variables (to generate a binary matrix summarizing their significant influences on each output pixel). Moreover, the application of the sliding window is justified in order to enforce the similarity between the influence maps belonging to the same module, reflected by the condition on identical support I k for all columns of W k in Model 1, and favoring low-rank matrix factorization. 4.1 DCGAN, β-VAE AND BEGAN ON THE CELEBA DATASET We first investigated modularity of genrative models trained on the CelebFaces Attributes Dataset (CelebA) .We first used a basic architecture: a plain β-VAE (https://github.com/ yzwxx/vae-celebA . We ran the full procedure described in Sec. 3, comprised of EIM calculations, clustering of channels into modules, and hybridization of generator samples using these modules. Hybridization procedures were performed by intervening on the output of the intermediate convolutional layer (indicated in Supplemental Fig. 7). The are summarized in Supplemental Fig. 8. We observed empirically that setting the number of clusters to 3 leads consistently to highly interpretable cluster templates as illustrated in the figure, with one cluster associated to the , one to the face and one to the hair. This observation was confirmed by running the following cluster stability analysis: we partition at random the influence maps in 3 subsets, and we use this partition to run the clustering twice on two thirds of the data, both runs overlapping only on one third. The obtained clusters were then matched in order to maximize the label consistency (the proportion of influence maps assigned the same label by both runs) on the overlapping subset, and this maximum consistency was used to assess robustness of the clustering across the number of clusters. The consistency are provided in Supplemental Fig. 9 and show 3 clusters is a reasonable choice as consistency is large (> 90%) and drops considerably for 4 clusters. Moreover, these also show that the NMF-based clustering outperforms clustering with the more standard k-means algorithm. In addition, we also assessed the robustness of the clustering by looking at the cosine distance between the templates associated to matching clusters, averaged across clusters. The , also provided in Supplemental Fig. 9, are consistent with the above analysis with an average cosine similarity of.9 (scalar product between the normalized feature vectors) achieved with 3 clusters (maximum similarity is 1 for perfectly identical templates). Exemplary influence maps shown in Supplemental Fig. 8 (center panel) reflect also our general observation: some maps may spread over image locations reflecting different clusters. Interestingly, applying the hybridization procedure to the ing 3 modules obtained by clustering leads to a replacement of the features associated to the module we intervene on, as shown in Supplemental Fig. 8 (center panel), while respecting the overall structure of the image (no discontinuity introduced). For example, on the middle row we see the facial features of the Original 2 samples are inserted in the Original 1 image (shown on the left), while preserving the hair. While the β-VAE is designed for extrinsic disentanglement, further work has shown that it can prove suboptimal with respect to other approaches suggesting further work could investigate whether better extrinsic disentanglement could also favor intrinsic disentanglement. It is however important to investigate intrinsic disentanglement in models for which (extrinsic) disentanglement is not enforced explicitly. This is in particular the case of most GAN-like architectures, who typically outperform VAE-like approaches in terms of sample quality in complex image datasets. Interestingly, the above could also be reproduced in the official tensorlayer DCGAN implementation, equipped with a similar architecture (https://github.com/tensorlayer/dcgan) (see Appendix E). This suggests that our approach can be applied to models that have not been optimized for disentanglement. After these experiments with basic models, we used a pretrained (https: //github.com/Heumi/BEGAN-tensorflow) Boundary Equilibrium GAN (BEGAN) , which used to set a milestone in visual quality for higher resolution face images. The good quality and higher resolution of the generated images combined with the relatively simple generator architecture of BEGAN allows us to test our hypothesis with minimal modifications of the computational graph. Most likely due to the increase in the number of layers, we observed that obtaining counterfactuals with noticeable effects required interventions on channels from the same cluster in two successive layers. The shown in Fig. 3b, obtained by intervening on layers 5 and 6, reveal a clear selective transfer of features from Original 2 to Original 1. As the model was trained on face images cropped with a tighter frame than for the above models, leaving little room for the hair and , we observe only one module associated to these features (Fig. 3b, middle row) showing a clear hair transfer. The remaining two modules are now encoding different aspects of face features: eye contour/mouth/nose for the top row and eyelids/face shape for the bottom row module. We further evaluated the relative quality of the counterfactual images with respect to the original generated images using the Frechet Inception Distance (FID) (Table 2 in the appendix), supporting that the hybridization procedure only mildly affects the image quality, in comparison to the original samples. In order to check whether our approach could scale to high resolution generative models, and generalize to complex image datasets containing a variety of objects, we used the BigGAN-deep architecture , pretrained (https://tfhub.dev/deepmind/biggan-deep-256/1) on the ImageNet dataset (http://www.image-net.org/). This is a conditional GAN architecture comprising 12 so-called Gblocks, each containing a cascade of 4 convolutional layers (see Appendix C for details). Each Gblock also receives direct input from the latent variables and the class label, and is bypassed by a skip connection. We then checked that we were able to generate hybrids by mixing the features of different classes. As for the case of BEGAN, intervening on two successive layers within a Gblock was more effective to generate counterfactuals (examples are provided for the 7th Gblock). Examples provided in Fig. 4 (cock-ostrich) show that it is possible to generate high quality counterfactuals with modified while keeping a very similar object in the foreground. In a more challenging situation, with objects of different nature (Koala-teddy bear on the same figure), meaningful combinations of each original samples are still generated: e.g. a teddy bear in a tree (bottom row), or a "teddy-koala" merging teddy texture with the color of a koala on a uniform indoor with a wooden texture (top row). In order to investigate how the generated counterfactual images can be used to probe and improve the robustness of classifiers to contextual changes, we compared the ability of several SOTA pretrained classifier available on Tensorflow-hub (https://tfhub.dev/, see Appendix C for details) to recognize one of the original classes. Fig. 5 shows the average recognition rate of the most recognized original class (teddy-bear or koala), as a function of layer depth tends overall to increase. We first observe that high recognition rates are in line with the small pixel distance between hybrids and original when intervening at layers closest to the output (right panel). Interestingly, at intermediate blocks 5-6, there is a clear contrast between classifiers, with the Inception resnet performing better than the others. Interestingly, examples of non-consensual classification in Supplementary Table 3, together with the associated hybrids (Supplementary Fig. 17) suggest different SOTA classifiers rely on different aspects of the image content to take their decision (e.g. versus object). We introduced a mathematical definition of disentanglement, related it to the causal notion of counterfactual and used it for the unsupervised characterization of the representation encoded by different groups of channels in deep generative architectures. We found evidence for interpretable modules of internal variables in four different generative models trained on two complex real world datasets. Our framework opens a way to a better understanding of complex generative architectures and applications such as the style transfer of controllable properties of generated images at low computational cost (no further optimization is required), and the automated assessment of robustness of object recognition systems to contextual changes. From a broader perspective, this research direction contributes to a better exploitation of deep neural networks obtained by costly and highly energy-consuming training procedures, by enhancing their interpretability and allowing them to be used for tasks their where not trained for. This offers a perspective on how more sustainable research in Artificial Intelligence could be fostered in the future. We rely on the assumption that a trained generator architecture can be exploited as a mechanistic model, such that parts of this model can be manipulated independently. A mathematical representation of such models can be given using structural causal models (SCMs, that rely on structural equations (SEs) of the form Y:= f (X 1, X 2, · · ·, X N,), denoting the assignment of a value to variable Y, computed from the values of other variables X k in the system under consideration, and of putative exogenous influences, imposed by factors outside the system under study. As in the above equation, we will use uppercase letters to indicate variables being the outcome of a structural assignment, while specific values taken by them will be lower case. SEs stay valid even if right-hand side variables undergo a change due to interventions (; , e.g.), and can model the operations performed in computational graphs of modern neural network implementations. Such graphs then depict SCMs made of interdependent modules, for which assignments' dependencies are represented by a directed acyclic graph G. Without loss of generality, we introduce a Causal Generative Model (CGM) M capturing the computational relations between a selected subset of variables comprising: the input latent variables {z k}, the generator's output Y (typically multidimensional), and a collection of possibly multi-dimensional endogenous (internal) variables forming an intermediate representation such that the generator's output can be decomposed into two successive steps as {Z k} → {V k} → Y. In a feed-forward neural network, one V k may for instance represent one channel of the output of a convolutional layer (e.g. after application of the ReLU non-linearity). where all Z k's are closed intervals, the CGM M = G(Z, S, G) comprises a directed acyclic graph G and a set S of N + 1 deterministic continuous structural equations that assign: The graph of an example CGM is exemplified on Fig. 2b, consisting of 3 endogenous variables, 2 latent inputs and the output. This aligns with the definition of a deterministic structural causal model by Pearl (2009, chapter 7), once our latent variables are identified with exogenous ones. CGMs have however specificities reflecting the structure of models encountered in practice. For instance, variable assignments may or may not involve latent/exogenous variables in their right-hand side, which is unusual in causal inference. This allows modeling feed-forward networks consisting in a first layer receiving latent inputs followed by a cascade of deterministic operations in downstream layers. The above definition guaranties several basic properties found in the computational graph of existing generative networks: all endogenous variables V k are unambiguously assigned once z is chosen, the output Y is unambiguously assigned once either z is chosen, or, alternatively, if an appropriate subset of V k's, such as Pa y, is assigned. This allows us to introduce several useful mappings. In an ideal case, while the support of the latent distribution covers the whole latent space Z, internal variables and outputs typically live on manifolds of smaller dimension than their ambient space. These can be defined as the images 4 of Z by operations of the graph: the out- ) k∈E, z ∈ Z for a subset of variables indexed by E, and V M when E includes all endogenous variables. Functions assigning Y from latent variables and from endogenous variables, respectively, are and we call them latent and endogenous mappings, respectively. Given the typical choice of dimensions for latent and endogenous variables, the V k's and Y are constrained to take values in subsets of their euclidean ambient space. We will assume thatg M and g M define proper embeddings, in particular implying that they are both invertible. We call a CGM satisfying these assumptions an embedded CGM. With this vocabulary we can for example verify the example of Fig. 2b contains exactly two layers (in green). Note g M andg M are well defined because the output can be unambiguously computed from their inputs by successive assignments along G, and are both surjective due to appropriate choices for domains and codomains. All defined image sets (V M, Y M, ...) are constrained by the parameters of M, and are typically not easy to characterize. For example V M is likely a strict subset of the Cartesian product k V k M. Importantly, the image set Y M of a trained model is of particular significance, as it should approximate at best the support of the data distribution we want to model. Learning the generator parameters such that Y M precisely matches the support of the target data distribution is arguably a major goal for generative models (see e.g.). As we will manipulate properties of the output, we restrict ourselves to transformations that respect the topology of Y M, and use embeddings as the basic structure for it, allowing inversion of g M. Definition 6 (Embedded CGMs). If f: X → Y is a continuous injective function with continuous inverse f Since Definition 5 imposes continuous structural equations,which is satisfied for all operations in standard generative models, injectivity of g M is the key additional requirement for embedded CGMs. Proposition 4. If Z of CGM M is compact (all Z k 's are bounded), then M is embedded if and only if g M is injective. Proof is provided in Appendix B. This implies that generative models based on uniformly distributed latent variables (the case of many GANs), provided they are injective, are embedded CGMs. While VAEs' latent space is typically not compact (due to the use of normally distributed latent variables), we argue that restricting it to a product of compact intervals (covering most of the probability mass) will in an embedded CGM that approximates the original one for most samples. Based on this precise framework, we can now provide the formal definitions and described informally in main text. The CGM framework allows defining counterfactuals in the network following. Definition 7 (Unit level counterfactual). Given CGM M, for a subset of endogenous variables E = {e 1, .., e n}, and assignment h of these variables, we define the interventional CGM M h obtained by replacing structural assignments for V |E by assignments {V e k := h k (z)} e k ∈E. Then for a given value z of the latent variables, called unit, the unit-level counterfactual is the output of Definition 7 is also in line with the concept of potential outcome . Importantly, conterfactuals induce a transformation of the output of the generative model. Definition 2 (Counterfactual mapping). Given an embedded CGM, we call the continuous map Our approach then relates counterfactuals to a form of disentanglement allowing transformations of the internal variables of the network as follows. Definition 3 (Intrinsic disentanglement). In a CGM M, endomorphism T: Y M → Y M is intrinsically disentangled with respect to a subset E of endogenous variables, if it exists a transformation T of endogenous variables such that for any latent z ∈ Z, leading to the tuple of values v ∈ V M, where T (v) only affects the variables indexed by E. In this definition, Y (v) corresponds to the unambiguous assignment of Y based on endogenous values. 5 Fig. 2d illustrates this second notion of disentanglement, where the split node indicates that the value of V 2 is computed as in the original CGM (Fig. 2b) before applying transformation T 2 to the outcome. Intrinsic disentanglement relates to a causal interpretation of the generative model's structure in the sense that it expresses a form of robustness to perturbation of one of its subsystems. Counterfactuals represent examples of such perturbations, and as such, may be disentangled given their faithfulness. Following a stated in , since Z is compact and the codomain of g M is Hausdorff (because Euclidean), then a continuous (by definition) and injective g M is an embedding. In addition, g M injective impliesg M's are injective on their respective domains V M. Moreover, the V M's being image of a compact Z by a continuous mapping (by the CGM definition), they are compact, such that the respectiveg M's are also embeddings. Part 1: Proof of the equivalence between faithful and disentangled. One conditional is trivial: if a transformation is disentangled, it is by definition an endomorphism of Y M so the counterfactual mapping must be faithful. For the second conditional, let us assume a faithful Y E h and denote the (unambiguous) map from V to the output such that T is disentangled with respect to E in M. Proof of Proposition 5. The absence of common latent ancestor between E and E ensures that values in both subsets are unambiguously assigned by non-overlapping subsets of latent variables, A and B respectively, such that we can write This implies that the image set of this layer fully covers the Cartesian product of the image sets of the two subsets of variables, i.e., and guaranties that T is and endomorphism of V M for any choice of endomorphism T E. This further implies T is well defined and an endomorphism. Due to the i.i.d. assumption for components of Z and the structure following the sufficient condition of Prop. 1, it is clear that the subsets of endogenous variables associated to each V k are modular and the associated partition of the hidden layer is a disentangled representation. The choice of increasing dimensions as well as the i.i.d. sampling of the model parameters from a distribution with a density make sure the ing mapping is injective (and hence follow the embedded CGM assumptions of Def. 6) and that counterfactual hybridization of any component of V k will in an influence map whose support covers exactly I k. Finally, the conditions on the I k's and the thresholding approach guaranties a rank K binary factorization of the matrix B, with one factor gathering the indicator vectors associated to each V k and the uniqueness of this factorization is guaranteed by classical NMF identifiability , e.g. following (Diop et al.) [Theorem III,1]. VANILLA β-VAE AND DCGAN The β-VAE architecture is presented in Supplementary Fig. 7 and is very similar to the DCGAN architecture. Hyperparameters for both structures are specified in Table 1. We used the method proposed in for CelebA dataset. We used the pre-trained model with the same architecture as was used in the paper. It consists of three blocks of convolutional layers each followed by an upsampling layer. The filter size of convolutional layers is all over the generator. There is also skip connections in the model that is argued to increase the sharpness of images. Consult (, Figure 1) for architectural details. The pretrained model is taken from Tensorflow-hub (https://tfhub.dev/, we summarize below the main aspects of the architectures. We used the BigGan-deep architecture of as a pre-trained model on 256x256 ImageNet. We did not retrain the model. The architecture consists of several ResBlocks which are the building block of the generator. Each ResBlock contains BatchNorm-ReLU-Conv Layers followed by upsampling transformations and augmented with skip connections that bring fresh signal from the input to every ResBlock. for architectural details. We argue that the notion introduced in Table 1 Figure 6: Generation of influence maps. Example of influence maps generated by a VAE on the CelebA dataset (lighter pixel indicate larger variance and thus stronger influence of the perturbations on that pixel). Table 2: FID analysis of BEGAN hybrids. Distance between different pairs of classes (R: Real data, G: Generated data, Ck: Hybrids by intervention on cluster k). The distances are is computed for balanced number of examples (10k) for each class and normalized by the FID between the real data and the generated data. It can be seen in the table that Hybrids have a small distance to the generated and also to each other. This can be interpreted as closeness of the distribution of Hybrids to that of generated data suggesting that Hybridization produces visually plausible images. Fig. 13. Columns indicate intervened Gblock (from 4 to 6), rows indicate module (as ordered in Fig. 13). The entropy is computed using the probabilistic output for the 10 classes receiving top ranking across all hybrids, normalized to provide a total probability of 1. In particular for Gblock 6 (left column) we can see that the module with poorer quality leads to larger entropy values. Interestingly, entropy values are also much smaller for hybrids based on interventions on the (more abstract level) Gblock number 4. Overall, the suggests that object texture, which is well rendered in hybrids generated from Gblock 4, is a key information for the classifier's decision. Under review as a conference paper at ICLR 2020 Figure 15: Larger collection of hybrids for the BIGAN, between classes "cock" and "ostrich". Each panel corresponds to intervening on a different Gblock (from 4 to 6, from top to bottom respectively). Modules are fixed and extracted by the NMF algorithm using 3 clusters. Each row of hybrids corresponds to interventions on one of the extracted module, numbered on the right-hand side. Fig. 15 ). The entropy is computed using the probabilistic output for the 10 classes receiving top ranking across all hybrids, normalized to provide a total probability of 1. Interestingly, large entropy is obtained for first module of Gblock 5 (middle column), consistent with the fact the intervention generates a hybrid bird, mixing shape properties of both cock and ostrich. Table 3: The classification outcome of several discriminative models for three randomly chosen koala+teddy hybrids (see Figure. 17). The purpose of this experiment is to investigate the use of the proposed intervention procedure for assessing robustness of classifiers. As can be seen in the following images, the ant hybrids are roughly a teddy bear in a koala context. An ideal classifier must be sensitive to the object present in the scene not the contextual information. A teddy bear must still be classified as a teddy bear even if it appears on a tree which is the koala environment in most of the koala images in the ImageNet dataset. It can be seen in the following table that nasnet large is more robust to the change of context compared to other classifiers. Figure 17: Three koala+teddy hybrids as the inputs to the classifiers of Table. 3

Represent the short summary for finding the corresponding scientific article (long)

We develop a framework to find modular internal representations in generative models and manipulate then to generate counterfactual examples.

Represent the TLDR for finding the corresponding scientific article

We introduce a type of neural network that is structurally resistant to adversarial attacks, even when trained on unaugmented training sets.  The resistance is due to the stability of network units wrt input perturbations.