Question: represent the text to find a very short summary (1-2 sentences max).

Question: In this paper, we ask for the main factors that determine a classifier's decision making and uncover such factors by studying latent codes produced by auto-encoding frameworks. To deliver an explanation of a classifier's behaviour, we propose a method that provides series of examples highlighting semantic differences between the classifier's decisions. We generate these examples through interpolations in latent space. We introduce and formalize the notion of a semantic stochastic path, as a suitable stochastic process defined in feature space via latent code interpolations. We then introduce the concept of semantic Lagrangians as a way to incorporate the desired classifier's behaviour and find that the solution of the associated variational problem allows for highlighting differences in the classifier decision. Very importantly, within our framework the classifier is used as a black-box, and only its evaluation is required. A considerable drawback of the deep classification paradigm is its inability to provide explanations as to why a particular model arrives at a decision. This black-box nature of deep systems is one of the main reasons why practitioners often hesitate to incorporate deep learning solutions in application areas, where legal or regulatory requirements demand decision-making processes to be transparent. A state-of-the-art approach to explain misclassification is saliency maps, which can reveal the sensitivity of a classifier to its inputs. Recent work , however, indicates that such methods can be misleading since their are at times independent of the model, and therefore do not provide explanations for its decisions. The failure to correctly provide explanations by some of these methods lies in their sensibility to feature space changes, i.e. saliency maps do not leverage higher semantic representations of the data. This motivates us to provide explanations that exploit the semantic content of the data and its relationship with the classifier. Thus we are concerned with the question: can one find semantic differences which characterize a classifier's decision? In this work we propose a formalism that differs from saliency maps. Instead of characterizing particular data points, we aim at generating a set of examples which highlight differences in the decision of a black-box model. Let us consider the task of image classification and assume a misclassification has taken place. Imagine, for example, that a female individual was mistakenly classified as male, or a smiling face was classified as not smiling. Our main idea is to articulate explanations for such misclassifications through sets of semantically-connected examples which link the misclassified image with a correctly classified one. In other words, starting with the misclassified point, we change its features in a suitable way until we arrive at the correctly classified image. Tracking the black-box output probability while changing these features can help articulate the reasons why the misclassification happened in the first place. Now, how does one generate such a set of semantically-connected examples? Here we propose a solution based on a variational auto-encoder framework. We use interpolations in latent space to generate a set of examples in feature space connecting the misclassified and the correctly classified points. We then condition the ing feature-space paths on the black-box classifier's decisions via a user-defined functional. Optimizing the latter over the space of paths allows us to find paths which highlight classification differences, e.g. paths along which the classifier's decision changes only once and as fast as possible. A basic outline of our approach is given in Fig. 1. In what follows we introduce and formalize the notion of stochastic semantic paths -stochastic processes on feature (data) space created by decoding latent code interpolations. We formulate the corresponding path integral formalism which allows for a Lagrangian formulation of the problem, viz. how to condition stochastic semantic paths on the output Figure 1: Auto-Encoding Examples Setup: Given a misclassified point x 0 and representatives x −T, x T, we construct suitable interpolations (stochastic processes) by means of an Auto-Encoder. Sampling points along the interpolations produces a set of examples highlighting the classifier's decision making. probabilities of black-box models, and introduce an example Lagrangian which tracks the classifier's decision along the paths. We show the explanatory power of our approach on the MNIST and CelebA datasets. We are concerned with the problem of explaining a particular decision of a black-box model. Many recent works discuss the roll and provide definitions of explanations in the machine learning context (; ; ;). Here we follow and, in broad terms, to explain we mean to provide textual or visual artifacts that provide qualitative understanding of the relationship between the data points and the model prediction. Attempts to clarify such a broad notion of explanation require the answers to questions such as what were the main factors in a decision?, as well as would changing a certain factor have changed the decision? . To provide an answer to such questions, one must be able to define a clear notion of factors. One can think of factors as the minimal set of coordinates that allows us to describe the data points. This definition mirrors the behavior and purpose of the variational auto-encoder (VAE) code -by training an auto-encoder one can find a code which describes a particular data point. Our role here is to provide a connection between these latent codes and the classifier's decision. Changes on the code should change the classification decision in a user-defined way. Defining such a code will allow us to formalize the framework required to provide an answer to question and above. we require explanations to be model-agnostic, i.e independent of the classifier's inner workings, interpretable, and expressing local fidelity. Following the discussion above, we use the variational auto-encoder (VAE) formalism to introduce a notion of semantics useful to qualitatively explain the decisions of a black-box classifier. Let us denote the feature (data) space by X and the latent linear space of codes (describing the data) by Z, where usually dim(Z) dim(X). We consider a latent variable generative model whose distribution P θ (X) on X is defined implicitly through a two-step generation process: one first samples a code Z from a fixed prior distribution P (Z) on Z and then (stochastically) maps Z to feature space through a (decoder) distribution P θ (X|Z), the latter being parametrized by neural networks with parameters θ. This class of models are generically train by minimizing specific distances between the empirical data distribution P D (X) and the model distribution P θ (X). VAE approaches this problem by introducing an encoder distribution Q φ (Z|X), parametrized by neural networks with parameters φ, which approximates the true posterior distribution P θ (Z|X) and minimizing a variational upper bound on the Kullback-Leibler divergence D KL between P θ (X) and P D (X). This bound reads where p θ (x|z) denotes the decoder's density and yields the likelihood function of the data given the code 1. Once the model is trained one can think of the inferred latent code as containing some highlevel description of the input data. Below we will use such inferred code to modify in a controlled way the features of a given input data point. We define a defendant black-box model b(l, x) as a classifier which yields the probability that the data point x ∈ X in feature (data) space belongs to the class l ∈ L, where L is a set of classes. Assume the model b(l, x) is expected to perform by its users or clients, in a dataset D = {(l i, x i)}, where x i ∈ X and l i ∈ L is the label that x i belongs to. 2 Suppose now that the following litigation case emerges. The black-box model b has assigned the data point x 0 to the class l 0. Accordingly, a plaintiff presents a complaint as the point x 0 should have been classified as l t. Furthermore, assume we are given two additional representative data points x −T, x T which have been correctly classified by the black-box model to the classes l −T, l T, respectively -as expected by e.g. the plaintiff, the defendant (if agreed), or the institution upon which the complain or litigation case is presented (say, the court). With this set-up in mind, we propose that an explanation why x 0 was misclassified can be articulated through an example set E = {x −T, . . ., x 0, . . ., x T}, where x t ∼ P θ (X|Z = z t). Here P θ (X|Z = z t) is a given decoder distribution and the index t runs over semantic changes (properly defined below) that highlight classification decisions. This example set constitutes the context revealing how factor changes impact the classification decision (see Section 2). One expects that human oriented explanations are semantic in character. One can understand the expression bigger eyes will change the classification. As opposed to changes in some specific pixels 3. The index t would run over these changes e.g. would make the eyes bigger. In this section we first formalize the notion of semantic change by introducing the concept of (stochastic) semantic interpolations in feature space X. This will allow us to generate examples which provide local fidelity, as the examples are smooth modifications of the latent code associated to the plaintiff data point x 0. We then define a collection of probability measures over semantic paths in X. These measures will be used later in Section 6 to constrain the paths to be explanatory with respect to the classifier's decision. One of the main motivations behind the VAE formalism is the ability of the inferred latent code z to provide semantic high-level information over the data set. If one is to generate examples which have characteristics common to two different data points, say x 0 and x T from the litigation case, one can perform interpolations between the latent codes of these points, that is z 0 and z T, and then decode the points along the interpolation. A main observation is that these interpolations in latent space can be used to induce certain interpolating stochastic processes on feature space 4 X. We refer to these as stochastic semantic processes. In what follows, we first focus on linear latent interpolations, i.e. and construct an interpolating stochastic semantic process X t on X by using the decoder distribution P θ (X|Z = z(t)). In practice, the generation process of such stochastic interpolations consists then of three steps: (i) sample Q φ (Z|X) at the end points x 0 and x T using the reparametrization trick , (ii) choose a set of points z t along the line connecting z 0 and z T and (iii) decode the z t by sampling P θ (X|Z = z t). A formal description of this procedure is given below, in subsection 5.2, and an impression of the stochastic process thus constructed is presented in Fig. 1b. We observe that for every sequence of points {t i} n i=0 there is a natural measure on piecewise linear paths starting at x 0 ∈ X and terminating at x T ∈ X. More precisely, we define the probability of a piecewise linear path x(t) with nodes x 1, x 2..., x n ∈ X as dP t0,...,tn (x(t)): where q φ, p θ label the densities of Q φ, P θ, respectively, and where z(t) is defined by eq. 5. In other words, for every pair of points x 0 and x T in feature space, and its corresponding code, the decoder P θ (X|Z) induces a measure over the space of paths {x(t)|x = x 0, x(T) = x T }. Formally speaking, the collection of measures dP t0,...,tn given by different choices of points {t i} n i=0 in defines a family of consistent measures (cf. Definition 2 in the Appendix, Subsection D.1). This implies that these different measures are assembled into a stochastic process on feature space X over the continuous interval [0, T]: Proposition 1. The collection of measures prescribed by induces a corresponding continuous-time stochastic process. Moreover, under appropriate reconstruction assumptions on the auto-encoder mappings P θ, Q φ, the sample paths are interpolations, that is, start and terminate respectively at x 0, x T almost surely. The statement goes along the lines of classical on existence of product measures. For the sake of completeness we provide all the necessary technical details in the Appendix, Subsection D. Another important remark is that the stochastic semantic process construction in Proposition 1 is just one way to define such a process -there are other natural options, e.g. in terms of explicit transition kernels or Itô processes. Having described a procedure to sample stochastic semantic processes in X, we need to discover autoencoding mappings (P θ, Q φ) that give rise to reasonable and interesting stochastic paths. Specifically, to generate examples which are able to explain the defendant black-box model b(l, x) in the current litigation case (Section 4), one needs to ensure that semantic paths between the data points x 0 and x T highlight classification differences, i.e. classifications of the model along this path are far apart in the plaintiff pair of labels. Thus, to design auto-encoding mappings P θ, Q φ accordingly, we propose an optimization problem of the form min where X t is a stochastic semantic process and S P θ,Q φ is an appropriately selected functional that extracts certain features of the black-box model b(l, x). The minimization problem can be seen in the context of Lagrangian mechanics. For a given stochastic semantic process X t, and given initial and final feature "states" x 0 and x T, we introduce the following function, named the model-b semantic Lagrangian which gives rise to the semantic model action: In mechanics, the optimization given by suitable Lagrangians delivers physically meaningful paths, e.g. those specified by the equations of motion . In our case, a guiding intuition is that the semantic Lagrangian should reflect how the black-box model takes decisions along the path 6 X t, starting at x 0 and ending at x T. In this way, the minimization of the semantic action (i.e. finding minimizing paths X t) should make such classification aspects prominent along the example set. Our problem, viz. to find encoding mappings P θ, Q φ which yield explainable semantic paths with respect to a black-box model, is then a constrain optimization problem whose total objective function we write as where L VAE is given by eq., S[x(t)] corresponds to the Lagrangian action and λ is an hyper parameter controlling the action' scale. The average over the paths is taken with respect to the stochastic paths and the corresponding measure dP [x(t)] from Proposition 1, that is, the path integral where x k t labels the tth point along the the kth path, sampled as described in Section 5, n is the number of points on each path, K is the total number of paths, and the estimator on the right hand side corresponds to an explicit average over paths 7. Algorithm 1: PATH Auto-Encoder 1 while φ and θ not converged do 2 Draw {x 1, ..., x n} from the training set Generate Latent Interpolations Sample k Paths in Feature Space Evaluate Semantic Action for each path k 12 and average over k Update P θ and Q φ by descending: In practice, both L VAE and the action term are optimized simultaneously. Note that the VAE loss function L VAE is trained on the entire data set on which the black-box performs. The action term, in contrast, only sees the x 0 and x T points. This can be seen explicitly in Algorithm 1, which shows an overview of the auto-encoder pair training algorithm. Let us finally note that, drawing analogies with the adversarial formalism , the defendant black-box model plays the role of a fixed discriminator, not guiding the example generation, but the interpolations among these examples. There are plenty of options for Lagrangian functionals that provide reasonable (stochastic) example-paths -roughly speaking, we attempt to define an objective value for a certain subjective notion of explanations. In what follows we illustrate one particular such Lagrangian 8 MINIMUM HESITANT PATH We want to find an example path such that the classifier's decisions along it changes as quickly as possible, as to highly certain regions in X. In Figure 2: Probability Paths for the litigation case l 0 = 2, l T = 7. Y axis corresponds to classification probability and x axis corresponds to interpolation index. Interpolation images for a specific paths are presented below the x axis. other words, the path is forced to stay in regions where the black-box produces decisions with maximum/minimum probability. An intuitive way to enforce this is via the simple Lagrangian where l 0, l T are the labels of the litigation case in question. Roughly speaking, given the appropriate initial conditions, the paths that minimize the action associated to L 1 are paths that attempt to keep L 1 close to 1 over almost the entire interpolation interval. Additionally we require b(l T, x(t)) to be a monotonous function along the interpolating path x(t). Furthermore, in accordance with Proposition 1 we require certain level of reconstruction at the end points. To enforce these conditions we introduce the regularizers r m, r e which are described in detail in subsection D.4 of the Appendix. The total objective function is therefore where λ, λ m, λ e are hyper-parameters and S 1 is the action associated to the minimum hesitant Lagrangian L 1 in eq.. We evaluate our method in two real-world data sets: MNIST, consisting of 70k Handwriting digits, and CelebA , with roughly 203k images. We use a vanilla version of the VAE with Euclidean latent spaces Z = R dz and an isotropic Gaussian as a prior distribution P (Z) = N (Z|0, I dz). We used Gaussian encoders, i.e. Q φ (Z|X) = N (Z|µ φ (X), Σ φ (X)), where µ φ, σ φ are approximated with neural networks of parameters φ, and Bernoulli decoders P θ (X|Z). We compare the standard VAE, VAE-EDGE (VAE augmented with the edge loss r e) and PATH-VAE (our full model, eq.). The black-box classifier b(l, x) is defined as a deep network with convolutional layers and a final soft-max output layer for the labels. Details of the specifics of the architectures as well as training procedure are left to the Appendix. For MNIST we studied a litigation case wherein l −T, l T = 2, 7 and l 0 = 2, whereas its true label (i.e. that of x 0) is l t = 7 (see Section 4). The are presented in Fig. 2. VAE delivers interpolations which provide uninformative examples, i.e. the changes in the output probability b(l 0, x) cannot be associated with changes in feature space. In stark contrast, PATH-VAE causes the output probability to change abruptly. This fact, together with the corresponding generated examples, allows us to propose explanations of the form: what makes the black-box model classify an image in the path as two or seven, is the shifting up of the lower stroke in the digit two as to coincide with the middle bar of the digit seven. Similarly, the upper bar of the digit seven (especially the upper left part) has a significant decision weight. In order to provide a more quantitative analysis we demonstrate the capability of our methodology to control the path action while retaining the reconstruction capacity. Hereby, we use not only the VAE as the underlying generative model, but also Wasserstein Auto-Encoder (WAE) and Adversarial Auto-Encoder (AAE) , i.e. we simply change L VAE in eq. with the loss of WAE or AAE. The theoretical details and corresponding architectures are presented in the Appendix. We present, in Fig. 3, the action values defined over random litigation end pairs (x −T, x T). The PATH version of the model indeed yields lower action values. Furthermore, these models tend to reduce the variance within the different paths. This is expected since there is one path that minimizes the action, hence, the distribution will try to arrive at this specific path for all samples. In order to compare with other explanation models, we define a saliency map with the interpolations obtained in our methodology. We defined the interpolation saliency as the sum over the differences between interpolation images weighted with the probability change of the black-box classifier through the interpolation path. We see in Fig. 4 the comparisons among different methods. While the standard methods only show local contributions to a classification probability, our saliency maps show the minimum changes that one is to apply to the dubious image in order to change the decision to the desired label. Our approach reveals that the curvature of the lower bar is decisive to be classified as a two, while the style of the upper bar is important to be classified as a seven. Further, we provide a sanity check analysis by studying the rank correlation between original saliency map and the one obtained for a randomized layers of the black-box classifier, shown in Fig. 5. As desired, our proposed saliency map decorrelates with the randomized version. For the CelebA dataset we use a black-box classifier based on the ResNet18 architecture . We investigate two specific misclassifications. In the first case, a smile was not detected (Fig. 6 a). Here we only interpolate between the misclassified image (left) and a correctly classified one (right), of the same person. Interpolations obtained by VAE are not informative: specific changes in feature space corresponding to changes in the probability cannot be detected since the latter changes rather slowly over the example path. This observation also holds true for the VAE-EDGE model, except that the examples are sharper. Finally, our PATH-VAE model yields a sharp change in the probability along with a change of the visible teeth (compare the third and fifth picture in the example path), revealing that this feature (i.e. teeth visibility) could constitute, from a human standpoint, a decisive factor in the probability of detecting a smile for the given black-box model. It is important to note that these observations represent one of many possible path changes which could change the classifier decision. This is constrained by the current realization and representative end points. The important is that our methods are able to shape the behavior of the classifier along the path. Further experimental examples are provided in Section C of the Appendix. The bulk of the explanation literature for deep/black-box models relies on input dependent methodologies. Gradient Based approaches Figure 6: Probability Paths for the case of detecting a smile in images of celebrities. Y axis corresponds to classification probability and x axis corresponds to interpolation index. Interpolation images for a specific paths are presented below the x axis. The images are vertically aligned with a corresponding tick in the x-axis determining the interpolation index of the image score for a given input example and class label by computing the gradient of the classifier with respect to each input dimension. Generalizations of this approach address gradient saturation by incorporating gradients' values in the saliency map or integrating scaled versions of the input . Ad hoc modifications of the gradient explanation via selection of the required value , , as well as direct studies of final layers of the convolutions units of the classifiers , are also provided. In contrast to gradient based approaches, other categories of explanatory models rely on reference based approaches which modify certain inputs with uninformative reference values . Bayesian approaches treat inputs as hidden variables and marginalize over the distribution to obtain the saliency of the input . More recent generalizations exploit a variational Bernoulli distribution over the pixels values . Other successful methodologies include substitution of black-box model with locally interpretable linear classifiers. This is further extended to select examples from the data points in such a way that the latter reflect the most informative components in the linear explanations, . Studies of auto-encoder interpolations seek to guarantee reconstruction quality. In the authors characterize latent space distortions compared to the input space through a stochastic Riemannian metric. Other solutions include adversarial cost on the interpolations such as to improve interpolation quality compared to the reconstructions, . Examples which are able to deceive the classifier's decisions have been widely studied in the framework of adversarial examples . These methodologies, however, do not provide interpretable explanations or highlight any semantic differences that lead to the classifier's decisions. Finally, the Auto-Encoder framework can also naturally be seen as a tool for dimensionality reduction. Geometrically speaking, assuming that the data set approximately lies along a manifold embedded in feature space X, one can interpret the encoder, decoder as the coordinate map (chart) and its inverse. From this point of view, our approach above translates to finding coordinate charts with additional constraints on mapping the segments from z 0 to z T to appropriate (stochastic) curves between x 0 and x T. In the present work we provide a novel framework to explain black-box classifiers through examples obtained from deep generative models. To summarize, our formalism extends the auto-encoder framework by focusing on the interpolation paths in feature space. We train the auto-encoder, not only by guaranteeing reconstruction quality, but by imposing conditions on its interpolations. These conditions are such that information about the classification decisions of the model B is encoded in the example paths. Beyond the specific problem of generating explanatory examples, our work formalizes the notion of a stochastic process induced in feature space by latent code interpolations, as well as quantitative characterization of the interpolation through the semantic Lagrangian's and actions. Our methodology is not constrained to a specific Auto-Encoder framework provided that mild regularity conditions are guaranteed for the auto-encoder. There was no preprocessing on the 28x28 MNIST images. The models were trained with up to 100 epochs with mini-batches of size 32 -we remark that in most cases, however, acceptable convergence occurs much faster, e.g. requiring up to 15 epochs of training. Our choice of optimizer is Adam with learning rate α = 10 −3. The weight of the KL term of the VAE is λ kl = 1, the path loss weight is λ p = 10 3 and the edge loss weight is λ e = 10 −1. We estimate the path and edge loss during training by sampling 5 paths, each of those has 20 steps. Encoder Architecture Both the encoder and decoder used fully convolutional architectures with 3x3 convolutional filters with stride 2. Conv k denotes the convolution with k filters, FSConv k the fractional strides convolution with k filters (the first two of them doubling the resolution, the third one keeping it constant), BN denotes batch normalization, and as above ReLU the rectified linear units, FC k the fully connected layer to R k. The pre-processing of the CelebA images was done by first taking a 140x140 center crop and then resizing the image to 64x64. The models are trained with up to 100 epochs and with mini-batches of size 128. Our choice of optimizer is Adam with learning rate α = 10 −3. The weight of the KL term of the VAE is λ kl = 0.5, the path loss weight is λ p = 0.5 and the edge loss weight is λ e = 10 − 3. We estimate the path and edge loss during training by sampling 10 paths, each of those has 10 steps. Encoder Architecture Decoder Architecture Both the encoder and decoder used fully convolutional architectures with 3x3 convolutional filters with stride 2. Conv k denotes the convolution with k filters, FSConv k the fractional strides convolution with k filters (the first two of them doubling the resolution, the third one keeping it constant), BN denotes batch normalization, and as above ReLU the rectified linear units, FC k the fully connected layer to R k. C FURTHER Interpolation between 2 and 7. It is seen that the Path-VAE interpolation optimizes both probabilities (P and P) according to the chosen Lagrangian -in this case the minimum hesitant L 1. Briefly put, the construction we utilize makes use of the well-known notion of consistent measures, which are finite-dimensional projections that enjoy certain restriction compatibility; afterwards, we show existence by employing the central extension of Kolmogorov-Daniell. We start with a couple of notational remarks. Definition 1. Let S, F be two arbitrary sets. We denote that is, the set of all maps F → S. Definition 2. Let (S, B) be a measurable space and let G ⊆ F ⊆ [0, T] for some positive number T. We define the restriction projections π F,G by Moreover, for each F ⊆ [0, T] the restriction projections induce the σ-algebra B F which is the smallest σ-algebra on S F so that all projections are measurable. In particular, the projections π F,G are measurable with respect to B F, B G. } is called consistent if it is push-forward compatible with respect to the restriction projection mappings, i.e. be an arbitrary finite set. The mapping defines a consistent collection of finite measures. Proof. Let us fix Without loss of generality, it suffices to check consistency for the pair (F 1, F 2). We have where we have used L 1 -finiteness and integrated out the s variable via Fubini's theorem. Note also, that by the definitions above χ π for any fixed s ∈ X. We briefly recall the following classical due to Kolmogorov and Daniell: Theorem 1 (Theorem 2.11, Bär & Pfäffle ). Let (S, B(S)) be a measurable space with S being compact and metrizable and let I be an index set. Assume that for each J ∈ Fin(I) there exists a measure µ J on S J, B J, such that the following compatibility conditions hold: Here π J1: S J2 → S J1 denotes the canonical projection (obtained by restriction). Then, there exists a unique measure µ on (S I, B I) such that for all J ∈ Fin(I) one has We recall that a well-known way to construct the classical Wiener measure and Brownian motion is precisely via the aid of Theorem 1 . We are now in a position to construct the following stochastic process. Proposition 3. There exists a continuous-time stochastic process Moreover, for small positive numbers, δ we have X 0 ∈ B δ (x 0) with probability at least (1 −), provided the reconstruction error of encoding/decoding process is sufficiently small. In particular, if x 0 stays fixed after the application of encoder followed by decoder, then X 0 = x 0 almost surely. A similar statement holds also for the terminal point X t and x T respectively. Proof. By applying Theorem 1 to the collection of consistent finite measures prescribed by Proposition 2 we obtain a measure µ on the measurable space (S [0,T], B [0,T] ). Considering the probability space (S [0,T], B [0,T], µ) we define stochastic process It follows from the construction and the Theorem of Kolmogorov-Daniell that P ((X t1, X t2, . . ., X tn) ∈ A) is expressed in the required way. This shows the first claim of the statement. Now, considering a small ball B δ (x 0) we have Here, the function R(x *, U) measures the probability that the input x * is decoded in the set U. Thus, if the reconstruction error gets smaller, R converges to 1. This implies the second statement. Finally, if we assume that the auto-encoder fixes x 0 in the sense above, we similarly get D.2 CONCERNING THE REGULARITY OF SAMPLE PATHS An important remark related to the the variational problem is the following: one could develop plenty of meaningful functionals S P θ,Q φ that involve taking velocities or higher derivatives -thus one is supposed to work over spaces of curves with certain regularity assumptions. However, as stated above we are working over stochastic paths X t whose regularity is, in general, difficult to guarantee. A straightforward way to alleviate this issue is to consider a "smooth" version of the curve X t -e.g. by sampling X t through a decoder with controllable or negligible variance or by means of an appropriate smoothing. Furthermore, one could also approach such stochastic variational analysis via Malliavin calculus -however, we do not pursue this direction in the present work. We now briefly discuss a few remarks about the regularity of the stochastic semantic process from Proposition 1. First, we state a well-known of Kolmogorov and Chentsov: Theorem 2 (Theorem 2.17, Bär & Pfäffle ). Let (M, ρ) be a metric measure space and let X t, t ∈ [0, T] be a stochastic process. Suppose that there exists positive numbers a, b, C, with the property, ∀s, t, |s − t| < Then, there exists a version Y t, t ∈ [0, T] of the stochastic process X t whose paths are α-Hölder continuous for any α ∈ (0, b/a). Thus, roughly speaking, an estimate on E [ρ(X s, X t) a ] can be regarded as a measure of the extent to which Theorem 2 fails. To give an intuitive perspective, let us consider the stochastic process given by Proposition 1 and, considering only the points X s, X s+δ for a small positive number δ, let us write the expectation in as: where we have used the standard Euclidean distance. To estimate the integral further, let us for simplicity assume that the encoder is deterministic and the decoder is defined via a Gaussian Ansatz of the type µ(z) + σ(z) ⊗ for a normal Gaussian variable. Thus the last integral can be written as: where we denote the covariance matrix at time s by Σ s. Now, if Σ s+δ becomes sufficiently small as δ converges to 0, then the exponential factor will dominate and thus holds. In other words, Hölder regularity of the process is verified provided that p θ (x|z) becomes localized in x and converges to a Dirac measure (similarly to the case of the heat kernel propagator and Brownian motion). From this point of view, the variance of the decoder can be considered as an indicator of how far the stochastic process is from being Hölder continuous. Below we discuss two other stochastic process constructions, one of which is built upon Itô diffusion processes and enjoys further path-regularity properties. We briefly recall that, among other aspects, Lagrangian theory suggests a framework for optimization of functionals (Lagrangians) defined over appropriate function spaces. Critical points of Lagrangians are identified by means of the corresponding Euler-Lagrange equations . To obtain the Euler-Lagrange equations for the Lagrangians in we compute in a straightforward manner the first variation where φ: [0, T] → T X is a compactly supported deformation 9. This produces the following conditions: Assuming the Hessian is not identically vanishing along the curve, the critical points of the variational problem are given by the condition (∇B) (l T |x(t)) = αẋ(t). In addition to following the geometry of the black box B, one could also impose a natural condition that the stochastic paths minimize distances on the manifold in feature space that the auto-encoder pair induces. We recall from basic differential geometry that the image of the decoder as a subset of the feature space is a submanifold with a Riemannian metric g induced by the ambient Euclidean metric in the standard way (for we refer to do). In the simple case of a deterministic auto-encoder, one can think of g as the matrix J T J where J denotes the Jacobian of the decoder -thus g gives rise to scalar product g(X, Y):= XJ T JY. In the stochastic case, one can use suitable approximations to obtain g in a similar manner -e.g. in the authors decompose the decoder into a deterministic and a stochastic part, whose Jacobians J 1, J 2 are summed as J T 1 J 1 + J T 2 J 2 to obtain the matrix g. Now, having Riemannian structure (i.e. the notion of a distance) on the data submanifold, geodesic curves naturally arise as minimizers of a suitable distance functional, namely: where the norm · g is computed with respect to the Riemannian metric g, that is g(·, ·). We note that the utilization of geodesics for suitable latent space interpolations was thoroughly discussed in. As mentioned already, we would like that classifier's probabilities change in a monotonous fashion along the paths -these paths are preferable in the sense that they provide examples following a particular trend along the disputed labels. We enforce such monotonic behaviour along the paths with the term r m:= 1 K(n − 1) with n the number of points along the path and K the number of paths. Further, and in accordance with Proposition 1, one can also require that the auto-encoder reconstructs the endpoints with sufficiently large accuracy. We enforce this requirement with the edge term r e:= i (|b(l i, x i)) − b(l i,x i)| + c(x i,x i)), i = 0, T, −T,where c measures the reconstruction error 10 andx i ∼ P θ (X|Z = z i), with z i ∼ Q φ (Z|X = x i) and x i the data points at i = 0, T, −T. In contrast to VAE, within the WAE framework one only needs to be able to sample from Q φ (Z|X) and P θ (X|Z) -i.e. their density is not needed. WAE is trained by minimizing a (penalized) optimal transport divergence -the Wasserstein distance, between the input data distribution P D (X) and the implicit latent variable model P θ (X). As in VAE, the latter is defined by first sampling Z from P (Z) and then mapping Z to X through the decoder P θ (X|Z). The loss function of WAE is given by where c is a distance function and D Z is an arbitrary divergence between the prior P (Z) and the agregate posterior Q φ (Z) = E P D (X) [Q φ (Z|X)], weighted by a positive hyperparameter λ. Minimizing Equation 56 corresponds to minimizing the Wasserstein distance if the decoder is deterministic (i.e. P θ (X|Z = z) = δ g θ (z) ∀z ∈ Z, with the map g θ: Z → X ) and the distance term is optimized. If the decoder is stochastic Equation 56 yields an upper bound on the Wasserstein

Represent the TLDR for finding the corresponding scientific article.

We generate examples to explain a classifier desicion via interpolations in latent space. The variational auto encoder cost is extended with a functional of the classifier over the generated example path in data space.

We introduce a statistical approach to assessing neural network robustness that provides an informative notion of how robust a network is, rather than just the conventional binary assertion of whether or not of property is violated.


Question: Represent the article for finding a single-sentence TLDR.

Question: The soundness and optimality of a plan depends on the correctness of the domain model. In real-world applications, specifying complete domain models is difficult as the interactions between the agent and its environment can be quite complex. We propose a framework to learn a PPDDL representation of the model incrementally over multiple planning problems using only experiences from the current planning problem, which suits non-stationary environments. We introduce the novel concept of reliability as an intrinsic motivation for reinforcement learning, and as a means of learning from failure to prevent repeated instances of similar failures. Our motivation is to improve both learning efficiency and goal-directedness. We evaluate our work with experimental for three planning domains. Planning requires as input a model which describes the dynamics of a domain. While domain models are normally hand-coded by human experts, complex dynamics typical of real-world applications can be difficult to capture in this way. This is known as the knowledge engineering problem BID3. One solution is to learn the model from data which is then used to synthesize a plan or policy. In this work, we are interested in applications where the training data has to be acquired by acting or executing an action. However, training data acquired in a planning problem could be insufficient to infer a complete model. While this is mitigated by including past training data from previous planning problems, this would be ill-suited for nonstationary domains where distributions of stochastic dynamics shift over time. Furthermore, the computation time increases with the size of the training data. Following these observations, we present an incremental learning model (ILM) which learns action models incrementally over planning problems, under the framework of reinforcement learning. PPDDL, a planning language modelling probabilistic planning problems BID20 ) (see Figure 1), is used for planning, and a rules-based representation (see FIG0) is used for the learning process. A parser translates between these two representations. Action models that were learned previously are provided to subsequent planning problems and are improved upon acquiring new training data; past training data are not used. We denote the models provided as prior action models. These could also be hand-coded, incomplete models serving as prior knowledge. Using prior knowledge has two advantages: it biases the learning towards the prior action models, and it reduces the amount of exploration required. While the learning progress cannot be determined without the true action models, we can estimate it empirically based on the of learning and acting. This empirical estimate, or reliability, is used to guide the search in the space of possible models during learning and as an intrinsic motivation in reinforcement learning. When every action is sufficiently reliable, we instead exploit with Gourmand, a planner that solves finite-horizon Markov Decision Processes (MDP) problems online BID9.Another major contribution of our work is its ability to learn from failure. Actions fail to be executed if their preconditions are not satisfied in the current state. This is common when the model is incorrect. Failed executions can have dire consequences in the real-world or cause irreversible changes such that goal states cannot be reached. ILM records failed executions and prevents any further attempts that would lead to similar failure. This reduces the number of failed executions and increases the efficiency of exploration. The rest of the paper is organized as follows. First, we review related work and then present the necessary . Next, we provide details of ILM. Lastly, we evaluate ILM in three planning domains and discuss the significance of various algorithmic features introduced in this paper. We extend the rules learner from BID15 that learns a set of relational rules to represent an action which can have probabilistic effects. A relational representation allows generalization of the state space unlike propositional rules which are used in BID14. Our training data consists of state transitions (s t, a t, s t+1) where s t is the pre-state, a t is the grounded action, and s t+1 is the post-state. This requirement is stricter than BID19 BID22 BID2 which learns from sequences of actions with no information on intermediate states. However, these works learn deterministic actions whereas we are interested in probabilistic actions. BID13 BID12 but do not address the incremental nature of reinforcement learning. BID5 BID18 ) learn deterministic action models incrementally while BID16 learns probabilistic action models. Our work is most similar to the latter which revises relational rules representing an action whenever contradicting examples are received. They do not store all the examples but rather track how well each rule explains the examples. On the other hand, we address incremental learning over planning problems where only current training data is used. Furthermore, our approach could consider prior knowledge in the form of incomplete action models which can have extraneous predicates unlike BID23. A second area of research that is related to our work is model-based reinforcement learning. R-MAX BID0 ) is provably sample-efficient, handling the balance between exploration and exploitation implicitly by assigning the maximum reward to unknown states which are set to absorbing states. If the count, defined as the number of times an action is executed in the state, of every applicable action exceeds a threshold, then the state is known. R-MAX is impractical for planning problems with large state spaces. Hence, additional assumptions such as factored state spaces BID8, known structures of dynamic Bayesian networks (DBN) BID6, or known maximum in-degree of DBNs BID4 ) are often made. Conversely, we only assume that the arguments of actions are known (e.g., we know moveCar has ? from and ?to as its arguments). We also use an empirical estimate for the learning progress, which we call reliability, as intrinsic motivation. Reliability is also used to quantify prior knowledge which other works on intrinsic motivation do not address BID1 BID7.Background PPDDL. Action models described in PPDDL are defined by their preconditions and effects, typically restricted to conjunctions of predicates. An example is shown in Figure 1. An action is applicable if its precondition is true in the current state, and executing it changes the state according to its effects which can be deterministic or probabilistic. Rules. For learning action models, we use a rules-based representation as it is well-suited to the incremental nature of reinforcement learning BID16 ). An action is described by a set of rules R where a rule r ∈ R has three parts: the name of the action, the precondition, and the effect. An example is shown in FIG0. The key difference between PPDDL and rules representations are the addition of noise effect in the latter which serves to avoid modelling a multitude of rare effects which could increase the complexity of synthesizing a plan. When a rare effect occurs, it is often better to replan. Multiple rules are required to represent disjunctive preconditions or effects. A rule covers a state-action pair (s, a) if it represents a and is applicable in s. Every state-action pair in the training data is covered by at most one rule which is called the unique covering rule, denoted as r (s,a). A propositional rule is obtained from the grounding of a relational rule by assigning an object or value to every argument in the rule (e.g. grounding moveCar(?loc1, ?loc2) to moveCar(l31, l13)). Actions are grounded in a similar fashion. MDPs model fullyobservable problems with uncertainty. A finite-horizon MDP is a tuple of the form (S, A, T, R, G, s 0, H) where S is a set of states, A is the set of actions, T: S ×A×S → is the transition function, R: S × A → R specifies rewards for performing actions, G is the set of goal states, s 0 is the initial state, and H is the number of decision epochs or planning horizon. The objective is to find a policy which maximizes the sum of expected rewards. Reinforcement Learning. When transition functions in MDPs are not known, model-based reinforcement learning can be used to learn them and perform sequential decisionmaking. This is the same as learning action models as they can be translated to transition functions BID20. Reinforcement learning deals with the balance between exploration and exploitation. Exploration seeks meaningful experiences from which action models are learned while exploitation synthesizes a policy using the models. We propose a new approach to incremental learning across planning problems called ILM. ILM has two main components: a rules learner and a reinforcement learning framework. We first introduce the concept of reliability which is used in both components followed by the extension made to the rules learner from BID15. Lastly, we provide details of the framework. The reliability of learned action models are empirical estimates of its learning progress. Reliability serves two purposes. We extend the rules learner from BID15 to consider the prior action model and its reliability to learn new rules. In reinforcement learning, less reliable actions are preferred during exploration. Reliability is defined as: DISPLAYFORM0 where o is an action, EX is the exposure, SU is the success rate, V O is the volatility, α s, α v, and γ are scaling parameters, n is the number of updates, and o 0 is the prior action model which can be an incomplete action model or an empty action model (no predicates in precondition and effect). Reliability is updated whenever o is executed. The initial values of SU and V O are set to zero. The reliability of the prior model is inherited with γ ∈ as the discount factor which reduces its significance given new data. Success Rate. An action with a high success rate indicates that recent executions are successful which is more likely if it has a small error. We define the success rate as: DISPLAYFORM1 where SU (o) ∈ 0, 1 1−γ, st is the execution status, and the indicator function 1 equals to 1 if the enclosing condition is true; otherwise, it is 0. The execution status is'failure' when the precondition of the action executed is not satisfied. The state is then assumed to be unchanged. The status is'partial success' if the post-state is not expected given the learned effects. SU is computed recursively with γ as the discount factor which gives less importance to past executions. Volatility. Volatility measures how much a set of rules representing an action changes after learning. A low volatility suggests that learning has converged to the true action model. Volatility is computed recursively, and is defined as: DISPLAYFORM2 where DISPLAYFORM3 is the set of rules before (after) learning, andd(R prev, R) is the normalized difference between the two sets of rules. The difference between two rules is defined as: DISPLAYFORM4 ) where superscripts p and e refer to the precondition and effect of a rule, respectively, and d − (p 1, p 2) returns the number of predicates that are in the set of predicates p 1 but not in p 2. The normalized difference is defined as: DISPLAYFORM5 |r 1 | + |r 2 | where the operator |r| refers to the number of predicates in r. The difference between two set of rules, d(R 1, R 2), is the sum of differences of pairs of rules r 1 ∈ R 1 and r 2 ∈ R 2 where the rules are paired such that the sum is minimal. Each rule is paired at most once and the number of predicates in unpaired rules are added to the sum. Exposure. Exposure measures the variability (inverse of similarity BID10) of the prestates in the training data, and is defined as: DISPLAYFORM6 where S is the set of unique pre-states in the state transitions involving o, and N s is the number of state transitions ing from successful executions. The first term is the ratio of state transitions from successful executions, penalizing those from failed executions which are less informative. Essentially, exposure is the average pairwise difference between pre-states weighted by N s. Since probabilities of effects are inferred using maximum likelihood on the N s successful state transitions, reliability considers these probabilities implicitly. Only unique pre-states are used to prevent double-counting. For example, in the Exploding Blocksworld domain, the sequence of actions pickUpFromTable(b1) and putDown(b1) can be executed repeatedly. This also causes V O to decrease and SU to increase which yields a high reliability which does not reflect the learning progress of the actions. Using exposure as a scaling factor prevents such scenarios. The rules learner from BID15 applies a search operator, selected at random, to a rule. Each search operator modifies the rule differently to yield a set of new rules. An example of a rule is shown in FIG0. A greedy search uses a score function as heuristics. We introduce a deviation penalty, P EN (R, R 0), to the score function such that the search begins from and is bounded around the prior action models, R 0, which can be a set of empty rules, or rules of incomplete action models. Hence, the learner refines R 0. The score function is defined as: DISPLAYFORM0 whereP is the probability of the effect in r (s,a) which covers the transition (s, a, s'), T is the training data, α p is a parameter, and P EN (r) penalizes complex rules to avoid over-specialization. The deviation penalty increases when R deviates further from R 0, and is defined as: DISPLAYFORM1 where α drop and α add are scaling parameters, and ∆ drop (R, R 0) and ∆ add (R, R 0) are defined as: DISPLAYFORM2 where the pairings of rules r ∈ R and r 0 ∈ R 0 are the same as the pairings in d(R, R 0).Since past training data is not used, the rules learner may consider a probabilistic effect of R 0 as noise if this effect is rarely seen in the current training data. ∆ drop increases when this happens. If the probabilistic effect is not seen at all, it will be dropped in R regardless of how large P EN (R, R 0) is. Such rules will be rejected. The deviation penalty is scaled by the reliability of the prior action model and the inverse of exposure. The intuition is that deviation should be limited if the prior action model is highly reliable, and encouraged if the training data has high variability. We begin this section by explaining the main algorithm for ILM (Algorithm 1), followed by the subroutines for reinforcement learning and learning from failure. The inputs to Algorithm 1 are the prior action models (R 0) and their reliability (RE 0), initial state (s 0), goal state (g), and the maximum number of iterations (N). EX max = 0 and tabu = ∅ for the first function call and shall be discussed later. The main loop interleaves learning, planning, and acting (lines 5 to 19). Exploration and exploitation is performed at the start of each iteration (line 6). If no action is found, then a dead-end is reached (line 7) and the algorithm terminates. When an action fails to execute, ILM learns from this failure by recording the failed instance in tabu (line 11: relevant predicates returns the set of grounded predicates in s that does not contain objects that were not in a), otherwise, synthetic state transitions (s t, a, s t) are generated (line 13) where a T is a randomly grounded action such that check tabu(s t, a, tabu) ⇒ ⊥. Failed executions are exceedingly less as failed instances are added to tabu. Reconstructing synthetic failed transitions augment the training data and aids the learning of preconditions. Learning from training data of low variability (or low exposure) could in lower correctness of learned rules. To prevent this, we delay learning until certain criteria are met (can learn in line 15): 1. If R 0 is the set of empty rules, always learn since no information can be lost. However, this risks learning incorrect preconditions or effects that can prevent the agent from reaching the goal state. 2. Otherwise, learn if there is at least one successful transition, at least one failed or synthetic transition, and EX > α EX EX max where α EX ∈.If learning is allowed, then new rules are learned (learn rules in line 16) and the values of RE, EX, V O, and SU are updated (line 17). Otherwise, only RE, EX, and SU are updated. The algorithm terminates after reaching the maximum number of iterations or when the goal is reached. It returns the learned rules, reliability, maximum exposure (EX max), and tabu. These are used as inputs to the next function call to Algorithm 1. The balance between exploration and exploitation is implemented in EE(s, g, R, RE, tabu, ζ). First, we compute the counts return R, RE, max(EX, EX max), tabu for all applicable actions in s using the context-based density formula from (Lang, Toussaint, and Kersting 2012) which performs relational generalizations -the amount of exploration is reduced as states which are unknown under propositional representations could be known under relational representations. The count-action pairs < c, o > are sorted in increasing order of c = RE(o) r∈R (s,a,s')∈T 1(r is applicable in s) in a list, L, where R are rules of o. Reliability serves as intrinsic motivation where less reliable actions are explored more. A state is known if ∀c i ∈ L (c i ≥ ζ), or if the reliability of every action exceeds a constant threshold. The second condition allows exploitation using prior action models when counts are still zero. If the state is known, exploitation is attempted using Gourmand, a planner that solves problems modelled in finite-horizon MDP online BID9. ILM can use any planner that accepts planning problems written in PPDDL. Exploitation fails if no plan is found or if the first action of the plan is in tabu. Exploration is attempted if the state is not known or exploitation fails. An action is popped off the top of L and a list of grounded actions that are applicable in s are enumerated. A grounded action that is not in tabu is selected at random and returned. If no such actions exist, then the next action is popped off until L is empty, following which random exploration is resorted to where actions are grounded without considering if preconditions are satisfied in s. If all grounded actions are in tabu, then a dead-end is reached. Learning from Failure Failed executions due to unsatisfied preconditions are recorded in tabu. Before an action a is executed in state s, Algorithm 2 checks if (s, a) is in tabu, returning False if so. We describe the algorithm with an example as shown in FIG1. A state is described by a set of predicates. We extract the set of predicates f s ⊆ s that does not have an object in its binding that is not in the argu- (l31) road(l11 l21) road(l21 l31) road(l12 l11) road(l13 l12) road(l13 l22) road(l22 l31) road(l22 l21) road(l12 l22) spareIn(l11) spareIn(l12) spareIn(l21) fs: ¬hasspare notFlattire at(l31) ft: ¬hasspare notFlattire at(?loc1) spareIn(?loc2) Perform substitution σ = {? loc1 → l31, ? loc2 → l13} on ft ft: ¬hasspare notFlattire at(l31) spareIn(l13) ments of a (line 2). We assume that the arguments of actions are known for this to be possible. f s is compared to each entry (f t, a t) in tabu (lines 3 to 7). The predicates in f t are grounded with the same substitution as the variables binding of a (line 5). Hence, the check is lifted to relational representations and is applicable even if the objects in the domain change. If f s does not have at least one predicate that is not in f t, then a is in tabu (line 6). In the example, moveCar(l31, l13) is in tabu, as are all grounded actions of moveCar that do not have road(?loc1, ?loc2) in f s. check tabu exploits experiences from failed executions which are otherwise uninformative to the rules learner as it cannot determine the reason for the failure BID17. Since every action is checked before execution, tabu will not contain identical entries. This keeps the size of tabu to a minimum which is important as the memory and time complexity is O(|tabu|). The completeness of Algorithm 2 depends on the failed instances in tabu. In the example, if tabu is ∅, then a is not in tabu, and in this case, the algorithm is incomplete. a then fails to execute following which f s is lifted with σ = {l31 → ? loc1, l13 → ? loc2} and inserted with o in tabu. Since no erroneous instance is ever added to tabu, the algorithm is sound. That is, no action that is found in tabu will succeed in execution. In one trial of experiments, ten planning problems are attempted sequentially in an order of increasing scale (see Table 1). We denote an attempt as one round. Each trial starts with no prior knowledge; the prior action models for round 1 are empty action models. Since the planning problems are probabilistic, 50 independent trials are conducted. The ma- Table 1: Number of objects in small, medium, and largescale planning problems for each of the three domains. Table 2: Algorithmic configurations for ILM, ILM-R, ILM-T, and R-MAX. DISPLAYFORM0 chine used to run the experiments was a four core Intel(R) i5-6500 with 4 GB of RAM.We used three planning domains: Tireworld and Exploding Blocksworld domains from the International Probabilistic Planning Competition BID21, and the Logistics domain. In the Tireworld domain, the car may get a flat tire when moving to another location. If the tire is flat, the car cannot move and a deadend is reached if no spare tires are available. Tireworld problems of the same scale are identical and are constructed systematically such that there are no unavoidable dead-ends BID11. In the Exploding Blocksworld domain, a block may detonate when it is put down, destroying the block or table beneath. A destroyed block or table is no longer accessible. Each block can only detonate once. We set the goal states as random configurations of three blocks. All Logistics problems have one truck per city, one airplane, and one parcel. Loading and unloading parcels may fail and the state remains unchanged. The models for all domains are stationary where probabilities of the effects of actions are kept constant in all rounds. The performance of ILM is evaluated with the correctness of the learned model and the goal-directedness. R-MAX and two variants of ILM are included for comparison. ILM-R does not use reliability; the relational count is not weighted and the deviation penalty in the score function used by the rules learner is zero. In addition, ILM-R does not delay learning (line 15 of Algorithm 1) as this requires EX max, a component of reliability. ILM-T does not learn from failure. ILM, ILM-R, and ILM-T do not use past training data while R-MAX does. The algorithmic configurations are summarized in Table 2. The correctness of a learned modelP can be defined as the average variational distance betweenP and the true model P BID15: DISPLAYFORM0 where T is the set of test examples -500 state transitions per action are generated with the true distribution. FIG2 show the variational distances for Tireworld, Exploding Blocksword, and Logistics domains. The variational distances at round 0 are of the prior action models, which are empty models for round 1.Tireworld. ILM learns action models incrementally as evident by the decrease in variational distance from rounds 1 to 10. ILM-R performed marginally worse as it learns from training data of low variability which caused the variational distances to increase in rounds 4 and 6. The utility of learning from failure is illustrated by the significantly larger variational distances for ILM-T and R-MAX. In both cases, most of the executions led to failure which are less meaningful DISPLAYFORM1 1 18 4 2 22 1 3 3 0 0 6 0 0 2 to 3 38 22 10 42 10 5 6 4 0 16 10 3 4 to 6 17 27 20 16 12 9 2 9 3 13 8 6 7 to 10 6 17 18 4 7 16 1 9 9 2 9 17 Table 3: Average number of successful trials out of 50 trials for Tireworld (T), Exploding Blocksworld (E), and Logistics (L) domains.experiences for the rules learner. Since the maximum number of iterations is only 15 (moveCar alone has 36 possible groundings for the small-scale planning problems), such inefficient exploration performs poorly. Exploding Blocksworld. The lowest variational distances are achieved with ILM from rounds 1 to 4 and with R-MAX thereafter. The latter learns from a larger training set which is important for this domain which has complex actions putOnBlock and putDown. These actions have conditional effects which are modelled as separate rules with different preconditions. Preconditions are inferred by comparing prestates in the training data. Most of the predicates in the prestates remain unchanged as an action typically changes a small subset of the state. Hence, more training data is required to learn more complex preconditions. Since the training data used by R-MAX are largely from failed experiences, it took four rounds before it outperforms ILM.Logistics. ILM had the best performance in all rounds. The large variational distances for ILM-T is due to the difficulty in learning driveTruck. This action has four arguments and there are 432 possible groundings in the smallscale planning problems. This has complications in the goaldirectedness which shall be discussed in the next subsection. The goal-directedness is evaluated by the number of successful trials which are trials where the goal state is reached. The goal-directedness for the three domains is shown in Table 3 which underlines the performance of the different algorithmic configurations. It is averaged over rounds with planning problems of the same scale. Round 1 is separated from rounds 2 and 3 to illustrate the advantage of having prior knowledge. The average number of successful trials for rounds 2 and 3 were generally larger than round 1 even though the scales of the planning problems are the same. This is because ILM exploits learned models from the previous round whereas round 1 had no such prior knowledge. Tireworld. ILM-R outperforms ILM in rounds 1 to 3. This is because the goal state can be reached by executing moveCar repeatedly as long as the tire is not flat along the way. ILM attempts exploitation more often than ILM-R Figure 5: Number of actions from exploration, exploitation, forced exploration, and random exploration that were successfully executed in the Exploding Blocksworld domain. The are the means and standard deviations of 50 trials using ILM (top) and ILM-T (bottom).as it weights relational counts with reliability. For smallscale planning problems, exploration or exploitation may not make a significant difference. When the scale increases, the number of steps between the initial state and the goal state increases and the probability of getting a flat tire along the way is higher. A dead-end is reached if the tire is flat and no spare tire is available. In such circumstances, exploitation is required and ILM outperforms ILM-R in rounds 4 to 10. ILM-T and R-MAX did not perform well as actions failed to execute most of the time. Exploding Blocksworld. Dead-ends are often the cause of failing to reach the goal state. A block could detonate with a probability of 0.2 when executing putDown or putOnBlock which destroys the table or underlying block. These irreversible changes to the state could then lead to dead-ends. Nevertheless, ILM has the most number of successful trials in all rounds. ILM-R performed much poorer than ILM as reaching the goal state with exploration alone is difficult. Even though R-MAX has lower variational distances than ILM for rounds 5 to 10, it did not outperform ILM as it does not learn from failure. Figure 5 compares the number of actions that were executed successfully in each round using ILM and ILM-T. The latter had significantly fewer number of successful executions. Figure 5 shows the frequency of exploration decreasing over the rounds while that of exploitation increased. This is expected as the action models are learned incrementally and exploitation should be used in later rounds where the variational distance is lower. FIG3 shows the use of tabu in ILM. The number of entries added to tabu declined sharply after round 1 because repeating entries are not added. The number of actions found in tabu correspond to the number of failed executions if this check was not done. This number rose for rounds 7 to 10 because the number of grounded actions increases in the largescale planning problems. Logistics. The number of successful trials increases even when the scale of the planning problem increases. In smallscale planning problems, there were few successful trials because driveTruck was not learned yet as mentioned previously. driveTruck failed to execute repeatedly till round 3 as only two out of 432 grounded actions would succeed. As a , a subset of the state space, which could include the goal state, is not reached. If states where the truck is at a location with a parcel are never reached, then loadTruck and unloadTruck could not be executed. This applies to loadAirplane and unloadAirplane if a parcel is not at an airport. For the trial shown in FIG4, the action model for driveTruck was learned from a single partially successful execution in round 3. The learned rule is shown in Figure 8. Its Action: driveTruck(?truck ? from ?to ?city) Precondition: airport(?from) ∧ truck at(?truck ?from)... ∧ in city(?to ?city) Effect: 1.0 ¬truck at(?truck ? from) ∧ truck at(?truck ? to) 0 noise Figure 8: Learned rule for driveTruck in the Logistics domain. The predicate airport(?from) in the precondition is extraneous and causes the action to be erroneously inapplicable in some parts of the state space.precondition had the extraneous predicate airport(?from). As a , driveTruck is not selected in rounds 4, 5, and 7 because this incorrect precondition is not satisfied though the action is in fact applicable. loadTruck and unloadTruck are not attempted in rounds 4 to 8 because they are in tabu. This example illustrates the adverse impact of learning extraneous predicates for preconditions. Although we delay learning when the training data has low variability, this is not done if the action models are empty. We presented a domain-independent framework, ILM, for incremental learning over multiple planning problems of a domain without the use of past training data. We introduced a new measure, reliability, which serves as an empirical estimate of the learning progress and influences the processes of learning and planning. The relational counts are weighted with reliability to reduce the amount of exploration required for reliable action models. We also extended an existing rules learner to consider prior knowledge in the form of incomplete action models. ILM learns from failure by checking if an action is in a list of state-action pairs which represents actions that have failed to execute. We evaluated ILM on three benchmark domains. Experimental showed that variational distances of learned action models decreased over each subsequent round. Learning from failure greatly reduces the number of failed executions leading to improved correctness and goal-directedness. For complex domains, more training data is required to learn action models. Using past training data would not work well for non-stationary domains and also increases the computation time for learning. The first issue could be resolved by learning distributions from the current training data only. The second issue could be resolved by maintaining a fixed size of training data by replacing older experiences while maximizing the exposure, or variability, of the training data. These will be explored in the future.

Introduce an approach to allow agents to learn  PPDDL action models incrementally over multiple planning problems under the framework of reinforcement learning.

Represent the short summary for finding the corresponding scientific article (long)

Identify decision states (where agent can take actions that matter) without reward supervision, use it for transfer.