Question: Represent the scientific article to find a short TLDR

Question: Generative models such as Variational Auto Encoders (VAEs) and Generative Adversarial Networks (GANs) are typically trained for a fixed prior distribution in the latent space, such as uniform or Gaussian. After a trained model is obtained, one can sample the Generator in various forms for exploration and understanding, such as interpolating between two samples, sampling in the vicinity of a sample or exploring differences between a pair of samples applied to a third sample. However, the latent space operations commonly used in the literature so far induce a distribution mismatch between the ing outputs and the prior distribution the model was trained on. Previous works have attempted to reduce this mismatch with heuristic modification to the operations or by changing the latent distribution and re-training models. In this paper, we propose a framework for modifying the latent space operations such that the distribution mismatch is fully eliminated. Our approach is based on optimal transport maps, which adapt the latent space operations such that they fully match the prior distribution, while minimally modifying the original operation. Our matched operations are readily obtained for the commonly used operations and distributions and require no adjustment to the training procedure. Generative models such as Variational Autoencoders (VAEs) BID7 and Generative Adversarial Networks (GANs) BID3 have emerged as popular techniques for unsupervised learning of intractable distributions. In the framework of Generative Adversarial Networks (GANs) BID3, the generative model is obtained by jointly training a generator G and a discriminator D in an adversarial manner. The discriminator is trained to classify synthetic samples from real ones, whereas the generator is trained to map samples drawn from a fixed prior distribution to synthetic examples which fool the discriminator. Variational Autoencoders (VAEs) BID7 are also trained for a fixed prior distribution, but this is done through the loss of an Autoencoder that minimizes the variational lower bound of the data likelihood. For both VAEs and GANs, using some data X we end up with a trained generator G, that is supposed to map latent samples z from the fixed prior distribution to output samples G(z) which (hopefully) have the same distribution as the data. In order to understand and visualize the learned model G(z), it is a common practice in the literature of generative models to explore how the output G(z) behaves under various arithmetic operations on the latent samples z. However, the operations typically used so far, such as linear interpolation BID3, spherical interpolation BID20, vicinity sampling and vector arithmetic BID12, cause a distribution mismatch between the latent prior distribution and the of the operations. This is problematic, since the generator G was trained on a fixed prior and expects to see inputs with statistics consistent with that distribution. To address this, we propose to use distribution matching transport maps, to obtain analogous latent space operations (e.g. interpolation, vicinity sampling) which preserve the prior distribution of samples from prior linear matched (ours) spherical (a) Uniform prior: Trajectories of linear interpolation, our matched interpolation and the spherical interp. BID20. (e) Spherical midpoint distribution BID20 Figure 1: We show examples of distribution mismatches induced by the previous interpolation schemes when using a uniform prior in two dimensions. Our matched interpolation avoids this with a minimal modification to the linear trajectory, traversing through the space such that all points along the path are distributed identically to the prior. the latent space, while minimally changing the original operation. In Figure 1 we showcase how our proposed technique gives an interpolation operator which avoids distribution mismatch when interpolating between samples of a uniform distribution. The points of the (red) matched trajectories are obtained as minimal deviations (in expectation of l 1 distance) from the the points of the (blue) linear trajectory. In the literature there are dozens of papers that use sample operations to explore the learned models BID0; BID3; BID2; BID13; BID1; BID13 to name a few), but most of them have ignored the problem of distribution mismatch. BID7 and BID10 sidestep the problem when visualizing their models, by not performing operations on latent samples, but instead restrict the latent space to 2-d and uniformly sample the percentiles of the distribution on a 2-d grid. This way, the samples have statistics that are consistent with the prior distribution. However, this approach does not scale up to higher dimensions -whereas the latent spaces used in the literature can have hundreds of dimensions. BID20 experimentally observe that there is a distribution mismatch between the norm for points drawn from uniform or Gaussian distribution and points obtained with linear interpolation (SLERP), and (heuristically) propose to use a so-called spherical linear interpolation to reduce the mismatch, obtaining higher quality interpolated samples. While SLERP has been subjectively observed to produce better looking samples than linear interpolation and is now commonly, its heuristic nature has limited it from fully replacing the linear interpolation. Furthermore, while perhaps possible it is not obvious how to generalize it to other operations, such as vicinity sampling, n-point interpolation and random walk. In Section 2 we show that for interpolation, in high dimensions SLERP tends to approximately perform distribution matching the approach taken by our framework which can explain why it works well in practice. BID6 further analyze the (norm) distribution mismatch observed by BID20 (in terms of KL-Divergence) for the special case of Gaussian priors, and propose an alternative prior distribution with dependent components which produces less (but still nonzero) distribution mismatch for linear interpolation, at the cost of needing to re-train and re-tune the generative models. In contrast, we propose a framework which allows one to adapt generic operations, such that they fully preserve the original prior distribution while being faithful to the original operation. Thus the KL-Divergence between the prior and the distribution of the from our operations is zero. The approach works as follows: we are given a'desired' operation, such as linear interpolation y = tz 1 + (1 − t)z 2, t ∈. Since the distribution of y does not match the prior distribution of z, we search for a warping f: DISPLAYFORM0 has the same distribution as z. In order to have the modificationỹ as faithful as possible to the original operation y, we use optimal transform Published as a conference paper at ICLR 2019 Operation Expression 2-point interpolation maps BID17 BID18 BID19 to find a minimal modification of y which recovers the prior distribution z. Figure 1a, where each pointỹ of the matched curve is obtained by warping a corresponding point y of the linear trajectory, while not deviating too far from the line. DISPLAYFORM1 With implicit models such as GANs BID3 and VAEs BID7, we use the data X, drawn from an unknown random variable x, to learn a generator G: DISPLAYFORM0 with respect to a fixed prior distribution p z, such that G(z) approximates x. Once the model is trained, we can sample from it by feeding latent samples z through G.We now bring our attention to operations on latent samples DISPLAYFORM1 We give a few examples of such operations in TAB0.Since the inputs to the operations are random variables, their output y = κ(z 1, · · ·, z k) is also a random variable (commonly referred to as a statistic). While we typically perform these operations on realized (i.e. observed) samples, our analysis is done through the underlying random variable y. The same treatment is typically used to analyze other statistics over random variables, such as the sample mean, sample variance and test statistics. In TAB0 we show example operations which have been commonly used in the literature. As discussed in the Introduction, such operations can provide valuable insight into how the trained generator G changes as one creates related samples y from some source samples. The most common such operation is the linear interpolation, which we can view as an operation DISPLAYFORM2 where z 1, z 2 are latent samples from the prior p z and y t is parameterized by t ∈. Now, assume z 1 and z 2 are i.i.d, and let Z 1, Z 2 be their (scalar) first components with distribution p Z. Then the first component of y t is Y t = tZ 1 + (1 − t)Z 2, and we can compute: DISPLAYFORM3 Since (1 + 2t(t − 1)) = 1 for all t ∈ \ {0, 1}, it is in general impossible for y t to have the same distribution as z, which means that distribution mismatch is inevitable when using linear interpolation. A similar analysis reveals the same for all of the operations in TAB0.This leaves us with a dilemma: we have various intuitive operations (see TAB0) which we would want to be able to perform on samples, but their ing distribution p yt is inconsistent with the distribution p z we trained G for. Due to the curse of dimensionality, as empirically observed by BID20, this mismatch can be significant in high dimensions. We illustrate this in FIG1, where we plot the distribution of the squared norm y t 2 for the midpoint t = 1/2 of linear interpolation, compared to the prior distribution z 2. With d = 100 (a typical dimensionality for the latent space), the distributions are dramatically different, having almost no common support. BID6 quantify this mismatch for Gaussian priors in terms of KL-Divergence, and show that it grows linearly with the dimension d. In Appendix A (see Supplement) we expand this analysis and show that this happens for all prior distributions with i.i.d. entries (i.e. not only Gaussian), both in terms of geometry and KL-Divergence. In order to address the distribution mismatch, we propose a simple and intuitive framework for constructing distribution preserving operators, via optimal transport: Published as a conference paper at ICLR 2019 BID20. Both linear and spherical interpolation introduce a distribution mismatch, whereas our proposed matched interpolation preserves the prior distribution for both priors. Strategy 1 (Optimal Transport Matched Operations). DISPLAYFORM0 2. We analytically (or numerically) compute the ing (mismatched) distribution p y 3. We search for a minimal modificationỹ = f (y) (in the sense that E y [c(ỹ, y)] is minimal with respect to a cost c), such that distribution is brought back to the prior, i.e. pỹ = p z.The cost function in step 3 could e.g. be the euclidean distance c(x, y) = x − y, and is used to measure how faithful the modified operator DISPLAYFORM1 Finding the map f which gives a minimal modification can be challenging, but fortunately it is a well studied problem from optimal transport theory. We refer to the modified operationỹ as the matched version of y, with respect to the cost c and prior distribution p z.For completeness, we introduce the key concepts of optimal transport theory in a simplified setting, i.e. assuming probability distributions are in euclidean space and skipping measure theoretical formalism. We refer to BID18 BID19 and BID17 for a thorough and formal treatment of optimal transport. The problem of step above was first posed by Monge and can more formally be stated as: Problem 1 (Problem 1.1). Given probability distributions p x, p y, with domains X, Y respectively, and a cost function c: X × Y → R +, we want to minimize DISPLAYFORM2 We refer to the minimizer f * X → Y of (MP) (if it exists), as the optimal transport map from p x to p y with respect to the cost c. However, the problem remained unsolved until a relaxed problem was studied by BID5: Problem 2 (Problem 1.2). Given probability distributions p x, p y, with domains X, Y respectively, and a cost function c: X × Y → R +, we want to minimize DISPLAYFORM3 where (x, y) ∼ p x,y, x ∼ p x, y ∼ p y denotes that (x, y) have a joint distribution p x,y which has (previously specified) marginals p x and p y.We refer to the joint p x,y which minimizes (KP) as the optimal transport plan from p x to p y with respect to the cost c. The key difference is to relax the deterministic relationship between x and f (x) to a joint probability distribution p x,y with marginals p x and p y for x and y. In the case of Problem 1, the minimization Published as a conference paper at ICLR 2019 might be over the empty set since it is not guaranteed that there exists a mapping f such that f (x) ∼ y. In contrast, for Problem 2, one can always construct a joint density p x,y with p x and p y as marginals, such as the trivial construction where x and y are independent, i.e. p x,y (x, y):= p x (x)p y (y).Note that given a joint density p x,y (x, y) over X × Y, we can view y conditioned on x = x for a fixed x as a stochastic function f (x) from X to Y, since given a fixed x do not get a specific function value f (x) but instead a random variable f (x) that depends on x, with f (x) ∼ y|x = x with density p y (y|x = x):= px,y(x,y)px (x). In this case we have (x, f (x)) ∼ p x,y, so we can view the Problem KP as a relaxation of Problem MP where f is allowed to be a stochastic mapping. While the relaxed problem of Kantorovich (KP) is much more studied in the optimal transport literature, for our purposes of constructing operators it is desirable for the mapping f to be deterministic as in (MP) (see Appendix C for a more detailed discussion on deterministic vs stochastic operations).To this end, we will choose the cost function c such that the two problems coincide and where we can find an analytical solution f or at least an efficient numerical solution. In particular, we note that the operators in TAB0 are all pointwise, such that if the points z i have i.i.d. components, then the y will also have i.i.d. components. If we combine this with the constraint for the cost c to be additive over the components of x, y, we obtain the following simplification: Theorem 1. Suppose p x and p y have i.i.d components and c over DISPLAYFORM4 Consequently, the minimization problems (MP) and (KP) turn into d identical scalar problems for the distributions p X and p Y of the components of x and y: DISPLAYFORM5 such that an optimal transport map T for (MP-1-D) gives an optimal transport map f for (MP) by pointwise application of T, i.e. f (x) (i):= T (x (i) ), and an optimal transport plan p X,Y for (KP-1-D)gives an optimal transport plan p x,y (x, y): DISPLAYFORM6 Proof. See Appendix. Fortunately, under some mild constraints, the scalar problems have a known solution: Theorem 2 (Theorem 2.9 in Santambrogio FORMULA3). Let h: R → R + be convex and suppose the cost C takes the form C(x, y) = h(x − y). Given an continuous source distribution p X and a target distribution p Y on R having a finite optimal transport cost in (KP-1-D), then DISPLAYFORM7 defines an optimal transport map from p X to p Y for (MP-1-D), where DISPLAYFORM8 is the Cumulative Distribution Function (CDF) of X and F DISPLAYFORM9 ≥ y} is the pseudo-inverse of F Y. Furthermore, the joint distribution of (X, T mon X→Y (X)) defines an optimal transport plan for (KP-1-D).The mapping T mon X→Y (x) in Theorem 2 is non-decreasing and is known as the monotone transport map from X to Y. It is easy to verify that T mon X→Y (X) has the distribution of Y, in particular DISPLAYFORM10 Now, combining Theorems 1 and 2, we obtain a concrete realization of the Strategy 1 outlined above. We choose the cost c such that it admits to Theorem 1, such as c(x, y):= x − y 1, and use an operation that is pointwise, so we just need to compute the monotone transport map in. That is, if DISPLAYFORM11 0.8 1 ỹ y t = 0.05 t = 0.25 t = 0.5 −3 −2.5 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 2.5 3 −3 −2 −1 0 1 2 3 ỹ y t = 0.05 t = 0.25 t = 0.5 (a) Uniform prior (b) Gaussian prioras the component-wise modification of y, i.e.ỹ DISPLAYFORM12 In FIG3 we show the monotone transport map for the linear interpolation y = tz 1 + (1 − t)z 2 for various values of t. The detailed calculations and examples for various operations are given in Appendix B, for both Uniform and Gaussian priors. To validate the correctness of the matched operators computed in Appendix B, we numerically simulate the distributions for toy examples, as well as prior distributions typically used in the literature. Priors vs. interpolations in 2-D For Figure 1, we sample 1 million pairs of points in two dimension, from a uniform prior (on [−1, 1] 2 ), and estimate numerically the midpoint distribution of linear interpolation, our proposed matched interpolation and the spherical interpolation of BID20. It is reassuring to see that the matched interpolation gives midpoints which are identically distributed to the prior. In contrast, the linear interpolation condenses more towards the origin, forming a pyramidshaped distribution (the of convolving two boxes in 2-d). Since the spherical interpolation of BID20 follows a great circle with varying radius between the two points, we see that the ing distribution has a "hole" in it, "circling" around the origin for both priors. FIG1, we sample 1 million pairs of points in d = 100 dimensions, using either i.i.d. uniform components on [−1, 1] or Gaussian N and compute the distribution of the squared norm of the midpoints. We see there is a dramatic difference between vector lengths in the prior and the midpoints of linear interpolation, with only minimal overlap. We also see that the spherical interpolation (SLERP) is approximately matching the prior (norm) distribution, having a matching first moment, but otherwise also induces a distribution mismatch. In contrast, our matched interpolation, fully preserves the prior distribution and perfectly aligns. We note that this setting (d = 100, uniform or Gaussian) is commonly used in the literature. Setup We used DCGAN BID12 generative models trained on LSUN bedrooms BID21, CelebA BID8 and LLD BID14, an icon dataset, to qualitatively evaluate. For LSUN, the model was trained for two different output resolutions, providing 64 × 64 pixel and a 128 × 128 pixel output images (where the latter is used in figures containing larger sample images). The models for LSUN and the icon dataset where both trained on a uniform latent prior distribution, while for CelebA a Gaussian prior was used. The dimensionality of the latent space is 100 for both LSUN and CelebA, and 512 for the model trained on the icon model. Furthermore we use improved Wasserstein GAN (iWGAN) with gradient penalty (Gulrajani et Table 3 : We measure over the average (normalized) perturbation ỹ − y p / y p incurred by our matched interpolation for the latent spaces used in TAB2, for p = 1, 2.2017) trained on CIFAR-10 at 32 × 32 pixels with a 128-dimensional Gaussian prior to compute inception scores. To measure the effect of the distribution mismatch, we quantitatively evaluate using the Inception score BID16. In TAB2 we compare the Inception score of our trained models (i.e. using random samples from the prior) with the score when sampling midpoints from the 2-point and 4-point interpolations described above, reporting mean and standard deviation with 50,000 samples, as well as relative change to the original model scores if they are significant. Compared to the original scores of the trained models (random samples), our matched operations are statistically indistinguishable (as expected) while the linear interpolation gives a significantly lower score in all settings (up to 29% lower).However, this is not surprising, since our matched operations are guaranteed to produce samples that come from the same distribution as the random samples. To quantify the effect our matching procedure has on the original operation, in Table 3 we compute the perturbation incurred when warping the linear interpolation y to the matched counterpartỹ for 2-point interpolation on the latent spaces used in TAB2. We compute the normalized perturbation ỹ t − y t p / y t p (with p = 1 corresponding to l 1 distance and p = 2 to l 2 distance), over N = 100000 interpolation points y t = tz 1 + (1 − t)z 2 where z 1, z 2 are sampled from the prior and t ∈ sampled uniformly. We observe that for all priors and both metrics, the perturbation is in the range 0.23 − 0.25, i.e. less than a one fourth of y t. In the following, we will qualitatively show that our matched operations behave as expected, and that there is a visual difference between the original operations and the matched counterparts. To this end, the generator output for latent samples produced with linear interpolation, SLERP (spherical linear interpolation) of BID20 and our proposed matched interpolation will be compared. We begin with the classic example of 2-point interpolation: Figure 4 shows three examples per dataset for an interpolation between 2 points in latent space. Each example is first done via linear interpolation, then SLERP and finally matched interpolation. It is immediately obvious in Figures 4a and 4b that linear interpolation produces inferior with generally more blurry, less saturated and less detailed output images. The SLERP heuristic and matched interpolation are slightly different visually, but we do not observe a difference in visual quality. However, we stress that the goal of this work is to construct operations in a principled manner, whose samples are consistent with the generative model. In the case of linear Published as a conference paper at ICLR 2019 interpolation (our framework generalizes to more operations, see below and Appendix), the SLERP heuristic tends to work well in practice but we provide a principled alternative. 4-point interpolation An even stronger effect can be observed when we do 4-point interpolation, showcased in Figure 5 (LSUN) and Figure 8 (LLD icons). The higher resolution of the LSUN output highlights the very apparent loss of detail and increasing prevalence of artifacts towards the midpoint in the linear version, compared to SLERP and our matched interpolation. Midpoints (Appendix) In all cases, the point where the interpolation methods diverge the most, is at the midpoint of the interpolation where t = 0.5. Thus we provide 25 such interpolation midpoints in Figures 11 (LLD icons) and 12 (LSUN) in the Appendix for direct comparison. Vicinity sampling (Appendix) Furthermore we provide two examples for vicinity sampling in Figures 9 and 10 in the Appendix. Analogous to the previous observations, the output under a linear operator lacks definition, sharpness and saturation when compared to both spherical and matched operators. Random walk An interesting property of our matched vicinity sampling is that we can obtain a random walk in the latent space by applying it repeatedly: we start at a point y 0 = z drawn from the prior, and then obtain point y i by sampling a single point in the vicinity of y i−1, using some fixed'step size'. We show an example of such a walk in FIG5, using = 0.5. As a of the repeated application of the vicinity sampling operation, the divergence from the prior distribution in the non-matched case becomes stronger with each step, ing in completely unrecognizable output images on the LSUN and LLD icon models. We proposed a framework that fully eliminates the distribution mismatch in the common latent space operations used for generative models. Our approach uses optimal transport to minimally modify (in l 1 distance) the operations such that they fully preserve the prior distribution. We give analytical formulas of the ing (matched) operations for various examples, which are easily implemented. The matched operators give a significantly higher quality samples compared to the originals, having the potential to become standard tools for evaluating and exploring generative models. This work was partly supported by ETH Zurich General Fund (OK) and Nvidia through a hardware grant. Published as a conference paper at ICLR 2019 We note that the analysis here can bee seen as a more rigorous version of an observation made by BID20, who experimentally show that there is a significant difference between the average norm of the midpoint of linear interpolation and the points of the prior, for uniform and Gaussian distributions. Suppose our latent space has a prior with DISPLAYFORM0 In this case, we can look at the squared norm DISPLAYFORM1 From the Central Limit Theorem (CLT), we know that as d → ∞, DISPLAYFORM2 in distribution. Thus, assuming d is large enough such that we are close to convergence, we can approximate the distribution of z 2 as N (dµ Z 2, dσ 2 Z 2). In particular, this implies that almost all points lie on a relatively thin spherical shell, since the mean grows as O(d) whereas the standard deviation grows only as O(DISPLAYFORM3 We note that this property is well known for i.i.d Gaussian entries (see e.g. Ex. 6.14 in MacKay FORMULA4). For Uniform distribution on the hypercube it is also well known that the mass is concentrated in the corner points (which is consistent with the claim here since the corner points lie on a sphere).Now consider an operator such as the midpoint of linear interpolation, y = In this case, we can compute: DISPLAYFORM4 Thus, the distribution of y 2 can be approximated with N (DISPLAYFORM5 . Therefore, y also mostly lies on a spherical shell, but with a different radius than z. In fact, the shells will intersect at regions which have a vanishing probability for large d. In other words, when looking at the squared norm y 2, y 2 is a (strong) outlier with respect to the distribution of z 2.This can be quantified in terms of KL-Divergence: DISPLAYFORM6 so D KL (z 2, y 2) grows linearly with the dimensions d. Proof. We will show it for the Kantorovich problem, the Monge version is similar. Published as a conference paper at ICLR 2019Starting from (KP), we compute DISPLAYFORM0 where the inequality in is due to each term being minimized separately. DISPLAYFORM1 where p X,Y has marginals p X and p Y. In this case P d (X, Y) is a subset of all joints p x,y with marginals p x and p y, where the pairs (DISPLAYFORM2 where the inequality in is due to minimizing over a smaller set. Since the two inequalities above are in the opposite direction, equality must hold for all of the expressions above, in particular: DISPLAYFORM3 Thus, (KP) and (KP-1-D) equal up to a constant, and minimizing one will minimize the other. Therefore the minimization of the former can be done over p X,Y with p x,y (x, y) = DISPLAYFORM4 In the next sections, we illustrate how to compute the matched operations for a few examples, in particular for linear interpolation and vicinity sampling, using a uniform or a Gaussian prior. We picked the examples where we can analytically compute the uniform transport map, but note that it is also easy to compute F DISPLAYFORM0 and (F Y (y)) numerically, since one only needs to estimate CDFs in one dimension. Since the components of all random variables in these examples are i.i.d, for such a random vector x we will implicitly write X for a scalar random variable that has the distribution of the components of x. When computing the monotone transport map T mon X→Y, the following Lemma is helpful. Lemma 1 (Theorem 2.5 in BID17). Suppose a mapping g(x) is non-decreasing and maps a continuous distribution p X to a distribution p Y, i.e. DISPLAYFORM1 then g is the monotone transport map T mon X→Y.According to Lemma 1, an alternative way of computing T mon X→Y is to find some g that is nondecreasing and transforms p X to p Y. Suppose z has uniform components Z ∼ Uniform(−1, 1). In this case, p Z (z) = 1/2 for −1 < z < 1. Now let y t = tz 1 + (1 − t)z 2 denote the linear interpolation between two points z 1, z 2, with component distribution p Yt. Due to symmetry we can assume that t > 1/2, since p Yt = p Y1−t. We then obtain p Yt as the convolution of p tZ and p (1−t)Z, i.e. p Yt = p tZ * p (1−t)Z. First we note that p tZ = 1/(2t) for −t < z < t and p (1−t)Z = 1/(2(1 − t)) for −(1 − t) < z < 1 − t. We can then compute:p Yt (y) = (p tZ * p (1−t)Z )(y)= 1 2(1 − t)(2t) DISPLAYFORM0 if y < −1 y + 1 if − 1 < y < −t + (1 − t) 2 − 2t if − t + (1 − t) < y < t − (1 − t) −y + 1 if t − (1 − t) < y < 1 0 if 1 < yThe CDF F Yt is then obtained by computing if 1 − 2t < y < 2t − 1 2(1 − t)(3t − 1) + (− 1 2 y 2 + y + 1 2 (2t − 1) 2 − (2t − 1)) if 2t − 1 < y < 1 2(1 − t)(2t) if 1 < ySince p Z (z) = 1/2 for |z| < 1, we have F Z (z) =

Represent the short summary for finding the corresponding scientific article (long)

We propose a framework for modifying the latent space operations such that the distribution mismatch between the resulting outputs and the prior distribution the generative model was trained on is fully eliminated.

It's important to consider optimization in function space, not just parameter space. We introduce a learning rule that reduces distance traveled in function space, just like SGD limits distance traveled in parameter space.


Question: Represent the article for finding a single-sentence TLDR

Question: The problem of building a coherent and non-monotonous conversational agent with proper discourse and coverage is still an area of open research. Current architectures only take care of semantic and contextual information for a given query and fail to completely account for syntactic and external knowledge which are crucial for generating responses in a chit-chat system. To overcome this problem, we propose an end to end multi-stream deep learning architecture which learns unified embeddings for query-response pairs by leveraging contextual information from memory networks and syntactic information by incorporating Graph Convolution Networks (GCN) over their dependency parse. A stream of this network also utilizes transfer learning by pre-training a bidirectional transformer to extract semantic representation for each input sentence and incorporates external knowledge through the neighbourhood of the entities from a Knowledge Base (KB). We benchmark these embeddings on next sentence prediction task and significantly improve upon the existing techniques. Furthermore, we use AMUSED to represent query and responses along with its context to develop a retrieval based conversational agent which has been validated by expert linguists to have comprehensive engagement with humans. With significant advancements in Automatic speech recognition systems and the field of natural language processing, conversational agents have become an important part of the current research. It finds its usage in multiple domains ranging from self-driving cars (b) to social robots and virtual assistants (a). Conversational agents can be broadly classified into two categories: a task oriented chat bot and a chit-chat based system respectively. The former works towards completion of a certain goal and are specifically designed for domain-specific needs such as restaurant reservations , movie recommendation , flight ticket booking systems ) among many others. The latter is more of a personal companion and engages in human-computer interaction for entertainment or emotional companionship. An ideal chit chat system should be able to perform non-monotonous interesting conversation with context and coherence. Current chit chat systems are either generative or retrieval based in nature. The generative ones tend to generate natural language sentences as responses and enjoy scalability to multiple domains without much change in the network. Even though easier to train, they suffer from error-prone responses (b). IR based methods select the best response from a given set of answers which makes them error-free. But, since the responses come from a specific dataset, they might suffer from distribution bias during the course of conversation. A chit-chat system should capture semantic, syntactic, contextual and external knowledge in a conversation to model human like performance. Recent work by proposed a memory network based approach to encode contextual information for a query while performing generation and retrieval later. Such networks can capture long term context but fail to encode relevant syntactic information through their model. Things like anaphora resolution are properly taken care of if we incorporate syntax. Our work improves upon previous architectures by creating enhanced representations of the conversation using multiple streams which includes Graph Convolution networks , Figure 1: Overview of AMUSED. AMUSED first encodes each sentence by concatenating embeddings (denoted by ⊕) from Bi-LSTM and Syntactic GCN for each token, followed by word attention. The sentence embedding is then concatenated with the knowledge embedding from the Knowledge Module (Figure 2). The query embedding passes through the Memory Module (Figure 3) before being trained using triplet loss. Please see Section 4 for more details. transformers and memory networks in an end to end setting, where each component captures conversation relevant information from queries, subsequently leading to better responses. Our contribution for this paper can be summarized as follows: • We propose AMUSED, a novel multi stream deep learning model which learns rich unified embeddings for query response pairs using triplet loss as a training metric. • We perform multi-head attention over query-response pairs which has proven to be much more effective than unidirectional or bi-directional attention. • We use Graph Convolutions Networks in a chit-chat setting to incorporate the syntactical information in the dialogue using its dependency parse. • Even with the lack of a concrete metric to judge a conversational agent, our embeddings have shown to perform interesting response retrieval on Persona-Chat dataset. The task of building a conversational agent has gained much traction in the last decade with various techniques being tried to generate relevant human-like responses in a chit-chat setting. Previous modular systems had a complex pipeline based structure containing various hand-crafted rules and features making them difficult to train. This led to the need of simpler models which could be trained end to end and extended to multiple domains. proposed a simple sequence to sequence model that could generate answers based on the current question, without needing extensive feature engineering and domain specificity. However, the responses generated by this method lacked context. To alleviate this problem, introduced a dynamic-context generative network which is shown to have improved performance on unstructured Twitter conversation dataset. To model complex dependencies between sub-sequences in an utterance, proposed a hierarchical latent variable encoder-decoder model. It is able to generate longer outputs while maintaining context at the same time. Reinforcement learning based approaches have also been deployed to generate interesting responses (a) and tend to possess unique conversational styles . With the emergence of a number of large datasets, retrieval methods have gained a lot of popularity. Even though the set of responses are limited in this scenario, it doesn't suffer from the problem of generating meaningless responses. A Sequential Matching Network proposed by performs word matching of responses with the context before passing their vectors to a RNN. Addition of external information along with the current input sentence and context improves the system as is evident by incorporating a large common sense knowledge base into an end to end conversational agent . To maintain diversity in the responses, suggests a method to combine a probabilistic model defined on item-sets with a seq2seq model. Responses like'I am fine' can make conversations monotonous; a specificity controlled model (b) in conjunction with seq2seq architecture overcomes this problem. These networks helps solve one or the other problem in isolation. To maintain proper discourse in the conversation, context vectors are passed together with input query vector into a deep learning model . A context modelling approach which includes concatenation of dialogue history has also been tried . However, the success of memory networks on Question-Answering task opened the door for its further use in conversational agents. used the same in a task oriented setting for restaurant domain and reported accuracies close to 96% in a full dialogue scenario. Zhang et al. (2018c) further used these networks in a chit chat setting on Persona-Chat dataset and came up with personalized responses. In our network, we make use of Graph Convolution Networks , which have been found to be quite effective for encoding the syntactic information present in the dependency parse of sentences. External Knowledge Bases (KBs) have been exploited in the past to improve the performances in various tasks (a; b;). The relation based strategy followed by creates a KB from dialogue itself, which is later used to improve Question-Answering .; have used KBs to generate more informative responses by using properties of entities in the graph. focused more on introducing knowledge from semantic-nets rather than general KBs. GCN for undirected graph: For an undirected graph G = (V, E), where V is the set of n vertices and E is the set of edges, the representation of the node v is given by x v ∈ R m, ∀v ∈ V. The output hidden representation h v ∈ R d of the node after one layer of GCN is obtained by considering only the immediate neighbors of the node as given by. To capture the multi-hop representation, GCN layers can be stacked on top of each other. GCN for labeled directed graph: For a directed graph G = (V, E), where V is the set of vertices we define the edge set E as a set of tuples (u, v, l(u, v) ) where there is an edge having label l(u, v) between nodes u and v. proposed the assumption that information doesn't necessarily propagate in certain directions in the directed edge, therefore, we add tuples having inverse edges (v, u, l(u, v) −1 ) as well as self loops (u, u, Ω), where Ω denotes self loops, to our edge set E to get an updated edge set E. The representation of a node x v, after the k th layer is given as: are trainable edge-label specific parameters for the layer k, N (v) denotes the set of all vertices that are immediate neighbors of v and f is any non-linear activation function (e.g., ReLU: Since we are obtaining the dependency graph from Stanford CoreNLP, some edges can be erroneous. Edgewise gating (; helps to alleviate this problem by decreasing the effects of such edges. For this, each edge (u, v, l(u, v) ) is assigned a score which is given by: u,v) ) ∈ R are trained and σ denotes the sigmoid function. Incorporating this, the final GCN embedding for a node v after n th layer is given as: This section provides details of three main components of AMUSED which can broadly be classified into Syntactic, Knowledge and Memory Module. We hypothesize that each module captures information relevant for learning representations, for a query-response pair in a chit-chat setting. Suppose that we have a dataset D consisting of a set of conversations d 1, d 2,..., d C where d c represents a single full length conversation consisting of multiple dialogues. A conversation d c is given by a set of tuples (q 1, r 1), (q 2, r 2),..., (q n, r n) where a tuple (q i, r i) denotes the query and response pair for a single turn. The context for a given query q i ∀ i ≥ 2 is defined by a list of sentences l: [q 1, r 1, ..., q i−1, r i−1]. We need to find the best response r i from the set of all responses, R. The training set D for AMUSED is defined by set of triplets (q i, r i, n i) ∀ 1 ≤ i ≤ N where N is the total number of dialogues and n i is a negative response randomly chosen from set R. Syntax information from dependency trees has been successfully exploited to improve a lot of Natural Language Processing (NLP) tasks (a;). In dialog agents, where anaphora resolution as well as sentence structure influences the responses, it finds special usage. A Bi-GRU followed by a syntactic GCN is used in this module. Each sentence s from the input triplet is represented with a list of k-dimensional GloVe embedding corresponding to each of the m tokens in the sentence. The sentence representation S ∈ R m×k is then passed to a Bi-GRU to obtain the representation S gru ∈ R m×dgru, where d gru is the dimension of the hidden state of Bi-GRU. This contextual encoding captures the local context really well, but fails to capture the long range dependencies that can be obtained from the dependency trees. We use GCN to encode this syntactic information. Stanford CoreNLP ) is used to obtain the dependency parse for the sentence s. Giving the input as S gru, we use GCN Equation 1, to obtain the syntactic embedding S gcn. , we only use three edge labels, namely forward-edge, backward-edge and self-loop. This is done because incorporating all the edge labels from the dependency graph heavily over-parameterizes the model. The final token representation is obtained by concatenating the contextual Bi-GRU representation h gru and the syntactic GCN representation h gcn. A sentence representation is then obtained by passing the tokens through a layer of word attention as used by (b;), which is concatenated with the embedding obtained from the Knowledge Module (described in Section 4.2) to obtain the final sentence representation h concat. The final sentence representation h concat of the query is then passed into Knowledge Module. It is further subdivided into two components: a pre-trained Transformer model for next dialogue prediction problem and a component to incorporate information from external Knowledge Bases (KBs). The next dialogue prediction task is described as follows: For each query-response pair in the dataset, we generate a positive sample (q, r) and a negative sample (q, n) where n is randomly chosen from the set of responses R in dataset D. , a training example is defined by concatenating q and r which are separated by a delimiter || and is given by [q||r]. The problem is to classify if the next sentence is a correct response or not. A pre-trained BERT model is used to further train a binary classifier for the next dialogue prediction task as described above. After the model is trained, the pre-final layer is considered and the vector from the special cls token is chosen as the sentence representation. The representation thus obtained would have a tendency to be more inclined towards its correct positive responses. Multi-head attention in the transformer network, along with positional embeddings during training, helps it to learn intra as well as inter sentence dependencies . The input query sentence is then passed from this network to obtain the BERT embedding, h bert. In our day-to-day conversations, to ask succinct questions, or to keep the conversation flowing, we make use of some knowledge. For example, if someone remarks that they like rock music, we can ask a question if they have listened to Nirvana. It can be done only if we know that Nirvana plays rock music. To incorporate such external information, we can make use of existing Knowledge Bases like Wikipedia, Freebase and Wikidata (Vrandečić & Krötzsch, 2014). Entities in these KBs are linked to each other using relations. We can expand the information we have about an entity by looking at its linked entities. Multiple hops of the Knowledge Graph (KG) can be used to expand knowledge. In AMUSED, we do this by passing the input query into Stanford CoreNLP to obtain entity linking information to Wikipedia. Suppose the Wikipedia page of an entity e contains links to the set of entities E. We ignore relation information and only consider one-hop direct neighbors of e. To obtain a KB-expanded embedding h kb of the input sentence, we take the average of GloVE embeddings of each entity in E. In place of Wikipedia, bigger knowledge bases like Wikidata, as well as relation information, can be used to improve KB embeddings. We leave that for future work. For effective conversations, it is imperative that we form a sense from the dialogues that have already happened. A question about' Who is the president of USA' followed by' What about France' should be self-containing. This dialogue context is encoded using a memory network . The memory network helps to capture context of the conversation by storing dialogue history i.e. both question and responses. The query representation, h concat is passed to the memory network, along with BERT embeddings h bert of the context, from the Knowledge Module (Section 4.2). In AMUSED, memory network uses supporting memories to generate the final query representation (h concat). Supporting memories contains input (m i) and output (c i) memory cells . The incoming query q i as well as the history of dialogue context l: [(q 1, r 1),.., (q i−1, r i−1)] is fed as input. The memory cells are populated using the BERT representations of context sentences l as follows: , the incoming query embedding along with input memories is used to compute relevance of context stories as a normalized vector of attention weights as a i = (< m i, h concat >), where < a, b > represents the inner product of a and b. The response from the output memory, o, is then generated as: o = i a i c i. The final output of the memory cell, u is obtained by adding o to h concat. To capture context in an iterative manner, memory cells are stacked in layers which are called as hops. The output of the memory cell after the k th hop is given by The memory network performs k such hops and the final representation h concat is given by sum of o k and u k. Triplet loss has been successfully used for face recognition . Our insight is that traditional loss metrics might not be best suited for a retrieval-based task with a multitude of valid responses to choose from. We define a Conversational Euclidean Space where the representation of a sentence is driven by its context in the dialogue along with its syntactic and semantic information. We have used this loss to bring the query and response representations closer in the conversational space. Questions with similar answers should be closer to each other and the correct response. An individual data point is a triplet which consists of a query (q i), its correct response (r i) and a negative response (n i) selected randomly. We need to learn their embeddings φ(q i) = h where α is the margin hyper-parameter used to separate negative and positive pairs. If I be the set of triplets, N the number of triplets and w the parameter set, then, triplet loss (L) is defined as: We use this dataset to build and evaluate the chit-chat system. Persona-Chat (c) is an open domain dataset on personal conversations created by randomly pairing two humans on Amazon Mechanical Turk. The paired crowd workers converse in a natural manner for 6 − 12 turns. This made sure that the data mimic normal conversations between humans which is very crucial for building such a system. This data is not limited to social media comments or movie dialogues. It contains 9907 training conversations and 1000 conversations each for testing and validation. There are a total of 131, 438 query-response pairs with a vocab size of 19262 in the dataset. We use it for training AMUSED as it provides consistent conversations with proper context. DSTC: Dialogue State Tracking Challenge dataset contains conversations for restaurant booking task. Due to its task oriented nature, it doesn't need an external knowledge module, so we train it only using memory and syntactic module and test on an automated metric. We further use Multi-Genre Natural Language Inference and Microsoft Research Paraphrase Corpus to fine-tune parts of the network i.e; Knowledge Module. It is done because these datasets resemble the core nature of our problem where in we want to predict the correctness of one sentence in response to a particular query. Pre-training BERT: Before training AMUSED, the knowledge module is processed by pre-training a bidirectional transformer network and extracting one hop neighborhood entities from Wikipedia KB. We use the approach for training as explained in Section 4.2.1. There are 104224 positive training and 27214 validation query-response pairs from Persona Chat. We perform three different operations: a) Equal sampling: Sample equal number of negative examples from dataset, b) Oversampling: Sample double the negatives to make training set biased towards negatives and c) Under sampling: Sample 70% of negatives to make training set biased towards positives. Batch size and maximum sequence length are 32 and 128 respectively. We fine-tune this next sentence prediction model with MRPC and MNLI datasets which improves the performance. Training to learn Embeddings: AMUSED requires triplets to be trained using triplet loss. A total of 131438 triplets of the form (q, r, n) are randomly split in 90:10 ratio to form training and validation set. The network is trained with a batch size of 64 and dropout of 0.5. Word embedding size is chosen to be 50. Bi-GRU and GCN hidden state dimensions are chosen to be 192 and 32 respectively. One layer of GCN is employed. Validation loss is used as a metric to stop training which converges after 50 epochs using Adam optimizer at 0.001 learning rate. As a retrieval based model, the system selects a response from the predefined answer set. The retrieval unit extracts embedding (h concat) for each answer sentence from the trained model and stores it in a representation matrix which will be utilized later during inference. First, a candidate subset A is created by sub-sampling a set of responses having overlapping words with a given user query. Then, the final output is retrieved on the basis of cosine similarity between query embedding h concat and the extracted set of potential responses (A). The response with the highest score is then labelled as the final answer and the response embedding is further added into the memory to take care of context. The model ing from oversampling method beats its counterparts by more than 3% in accuracy. It clearly indicates that a better model is one which learns to distinguish negative examples well. The sentence embeddings obtained through this model is further used for lookup in the Knowledge Module (Section 4.2) in AMUSED. We use two different automated metrics to check the effectiveness of the model and the queryresponse representations that we learnt. Next Dialogue Prediction Task: Various components of AMUSED are analysed for their performance on next dialogue prediction task. This task tell us that, given two sentences (a query and a response) and the context, whether second sentence is a valid response to the first sentence or not. Embeddings for queries and responses are extracted from our trained network and then multiple operations which include a) Concatenation, b) Element wise min and c) Subtraction are performed on those before passing them to a binary classifier. A training example consists of embeddings of two sentences from a (q, a) or (q, n) pair which are created in a similar fashion as in Section 4.2.1. Accuracy on this binary classification problem has been used to select the best network. Furthermore, we perform ablation studies using different modules to understand the effect of each component in the network. A 4 layer neural network with ReLU activation in its hidden layers and softmax in the final layer is used as the classifier. External knowledge in conjunction with memory and GCN module has the best accuracy when embeddings of query and response are concatenated together. A detailed study of performance of various components over these operations is shown in Table 1. Precision@1: This is another metric used to judge the effectiveness of our network. It is different from the next sentence prediction task accuracy. It measures that for n trials, the number of times a relevant response is reported with the highest confidence value. Table 2 reports a comparative study of this metric on 500 trials conducted for AMUSED along with for other methods. DSTC dataset is also evaluated on this metric without the knowledge module as explained in Section 5.1 Looking for exact answers might not be a great metric as many diverse answers might be valid for a particular question. So, we must look for answers which are contextually relevant for that query. Overall, we use next sentence prediction task accuracy to choose the final model before retrieval. There is no concrete metric to evaluate the performance of an entire conversation in a chit-chat system. Hence, human evaluation was conducted using expert linguists to check the quality of conversation. They were asked to chat for 7 turns and rate the quality of responses on a scale of 1−10 where a higher score is better. Similar to Zhang et al. (2018c), there were multiple parameters to rate the chat based on coherence, context awareness and non-monotonicity to measure various factors that are essential for a natural dialogue. By virtue of our network being retrieval based, we don't need to judge the responses based on their structural correctness as this will be implicit. To monitor the effect of each neural component, we get it rated by experts either in isolation or in conjunction with other components. Such a study helps us understand the impact of different modules on a human based conversation. Dialogue system proposed by Zhang et al. (2018c) is also reproduced and reevaluated for comparison. From Table 3 we can see that human evaluation follows a similar trend as the automated metric, with the best rating given to the combined architecture. In the paper, we propose AMUSED, a multi-stream architecture which effectively encodes semantic information from the query while properly utilizing external knowledge for improving performance on natural dialogue. It also employs GCN to capture long-range syntactic information and improves context-awareness in dialogue by incorporating memory network. Through our experiments and using different metrics, we demonstrate that learning these rich representations through smart training (using triplets) would improve the performance of chit-chat systems. The ablation studies show the importance of different components for a better dialogue. Our ideas can easily be extended to various conversational tasks which would benefit from such enhanced representations.

Represent the short summary for finding the corresponding scientific article (long).

This paper provides a multi -stream end to end approach to learn unified embeddings for query-response pairs in dialogue systems by leveraging contextual, syntactic, semantic and external information together.

Represent the TLDR for finding the corresponding scientific article

Combining graph neural networks and the RNN graph generative model, we propose a novel architecture that is able to learn from a sequence of evolving graphs and predict the graph topology evolution for the future timesteps


Question: Represent the scientific article to find a short TLDR

Question: We examine techniques for combining generalized policies with search algorithms to exploit the strengths and overcome the weaknesses of each when solving probabilistic planning problems. The Action Schema Network (ASNet) is a recent contribution to planning that uses deep learning and neural networks to learn generalized policies for probabilistic planning problems. ASNets are well suited to problems where local knowledge of the environment can be exploited to improve performance, but may fail to generalize to problems they were not trained on. Monte-Carlo Tree Search (MCTS) is a forward-chaining state space search algorithm for optimal decision making which performs simulations to incrementally build a search tree and estimate the values of each state. Although MCTS can achieve state-of-the-art when paired with domain-specific knowledge, without this knowledge, MCTS requires a large number of simulations in order to obtain reliable estimates in the search tree. By combining ASNets with MCTS, we are able to improve the capability of an ASNet to generalize beyond the distribution of problems it was trained on, as well as enhance the navigation of the search space by MCTS. Planning is the essential ability of a rational agent to solve the problem of choosing which actions to take in an environment to achieve a certain goal. This paper is mainly concerned with combining the advantages of forward-chaining state space search through UCT BID11, an instance of Monte-Carlo Tree Search (MCTS) BID5, with the domain-specific knowledge learned by Action Schema Networks (ASNets) BID18 ), a domain-independent learning algorithm. By combining UCT and ASNets, we hope to more effectively solve planning problems, and achieve the best of both worlds. The Action Schema Network (ASNet) is a recent contribution in planning that uses deep learning and neural networks to learn generalized policies for planning problems. A generalized policy is a policy that can be applied to any problem from a given planning domain. Ideally, this generalized policy is able to reliably solve all problems in the given domain, although this is not always feasible. ASNets are well suited to problems where "local knowledge of the environment can help to avoid certain traps" BID18. In such problems, an ASNet can significantly outperform traditional planners that use heuristic search. Moreover, a significant advantage of ASNets is that a network can be trained on a limited number of small problems, and generalize to problems of any size. However, an ASNet is not guaranteed to reliably solve all problems of a given domain. For example, an ASNet could fail to generalize to difficult problems that it was not trained on -an issue often encountered with machine learning algorithms. Moreover, the policy learned by an ASNet could be suboptimal due to a poor choice of hyperparameters that has led to an undertrained or overtrained network. Although our discussion is closely tied to ASNets, our contributions are more generally applicable to any method of learning a (generalized) policy. Monte-Carlo Tree Search (MCTS) is a state-space search algorithm for optimal decision making which relies on performing Monte-Carlo simulations to build a search tree and estimate the values of each state BID5 ). As we perform more and more of these simulations, the state estimates become more accurate. MCTS-based game-playing algorithms have often achieved state-of-the-art performance when paired with domain-specific knowledge, the most notable being AlphaGo (Silver et al. 2016). One significant limitation of vanilla MCTS is that we may require a large number of simulations in order to obtain reliable estimates in the search tree. Moreover, because simulations are random, the search may not be able to sense that certain branches of the tree will lead to sub-optimal outcomes. We are concerned with UCT, a variant of MCTS that balances the trade-off between exploration and exploitation. However, our work can be more generally used with other search algorithms. Combining ASNets with UCT achieves three goals. Learn what we have not learned: improve the capability of an ASNet to generalize beyond the distribution of problems it was trained on, and of UCT to bias the exploration of actions to those that an ASNet wishes to exploit. Improve on sub-optimal learning: obtain reasonable evaluation-time performance even when an ASNet was trained with suboptimal hyperparameters, and allow UCT to converge to the optimal action in a smaller number of trials. Be robust to changes in the environment or domain: improve performance when the test environment differs substantially from the training environment. The rest of the paper is organized as follows. Section 2 formalizes probabilistic planning as solving a Stochastic Shortest Path problem and gives an overview of ASNets and MCTS along with its variants. Section 3 defines a framework for Dynamic Programming UCT (DP-UCT) BID10. Next, Section 4 examines techniques for combining the policy learned by an ASNet with DP-UCT. Section 5 then presents and analyzes our . Finally, Section 6 summarizes our contributions and discusses related and future work. A Stochastic Shortest Path problem (SSP) is a tuple S, s 0, G, A, P, C BID2 where S is the finite set of states, s 0 ∈ S is the initial state, G ⊆ S is the finite set of goal states, A is the finite set of actions, P (s | a, s) is the probability that we transition into s after applying action a in state s, and C(s, a) ∈ (0, ∞) is the cost of applying action a in state s. A solution to an SSP is a stochastic policy π: A × S →, where π(a | s) represents the probability action a is applied in the current state s. An optimal policy π *, is a policy that selects actions which minimize the expected cost of reaching a goal. For SSPs, there always exists an optimal policy that is deterministic which may be obtained by finding the fixed-point of the state-value function V * known as the Bellman optimality equation BID2, and the actionvalue function Q *. That is, in the state s, we obtain π * by finding the action a that minimizes Q * (s, a). DISPLAYFORM0 We handle dead ends using the finite-penalty approach BID12. That is, we introduce a fixed dead-end penalty D ∈ (0, ∞) which acts as a limit to bound the maximum expected cost to reach a goal, and a give-up action which is selected if the expected cost is greater than or equal to D. The ASNet is a neural network architecture that exploits deep learning techniques in order to learn generalized policies for probabilistic planning problems BID18 ). An ASNet consists of alternating action layers and proposition layers (Figure 1), where the first and last layer are always action layers. The output of the final layer is a stochastic policy π: DISPLAYFORM0 An action layer is composed of a single action module for each ground action in the planning problem. Similarly, a proposition layer is composed of a single proposition module for each ground proposition in the problem. These modules are sparsely connected, ensuring that only the relevant action modules in one layer are connected to a proposition Figure 1: ASNet with 1 hidden layer BID18 module in the next layer. An action module in one layer is connected to a proposition module in the next layer only if the ground proposition appears in the preconditions or effects of a ground action. Similarly, a proposition module in one layer is connected to an action module in the next layer only if the ground proposition appears in the preconditions or effects of the relevant ground action. Since all ground actions instantiated from the same action schema will have the same structure, we can share the same set of weights between their corresponding action modules in a single action layer. Similarly, weights are shared between proposition modules in a single proposition layer that correspond to the same predicate. It is easy to see that by learning a set of common weights θ for each action schema and predicate, we can scale an ASNet to any problem of the same domain. ASNets only have a fixed number of layers, and are thus unable to solve all problems in domains that require arbitrarily long chains of reasoning about action-proposition relationships. Moreover, like most machine learning algorithms, an ASNet could fail to generalize to new problems if not trained properly. This could be due to a poor choice of hyperparameters, overfitting to the problems the network was trained on, or an unrepresentative training set. MCTS is a state-space search algorithm that builds a search tree in an incremental manner by performing trials until we reach some computational budget (e.g. time, memory) at each decision step BID5, at which point MCTS returns the action that gives the best estimated value. A trial is composed of four phases. Firstly, in the selection phase, MCTS recursively selects nodes in the tree using a child selection policy until it encounters an unexpanded node, i.e. a node without any children. Next, in the expansion phase, one or more child nodes of the leaf node are created in the search tree according to the available actions. Now, in the simulation phase, a simulation of the scenario is played-out from one of the new child nodes until we reach a goal or dead end, or exceed the computational budget. Finally, in the backpropagation phase, the of this trial is backpropagated through the selected nodes in the tree to update their estimated values. The updated estimates affect the child selection policy in future trials. Upper Confidence Bounds applied to Trees (UCT) BID11 ) is a variant of MCTS that addresses the trade-off between the exploration of nodes that have not been visited often, and the exploitation of nodes that currently have good state estimates. UCT treats the choice of a child node as a multi-armed bandit problem by selecting the node which maximizes the Upper Confidence Bound 1 (UCB1) term, which we detail in the selection phase in Section 3.1.Trial-Based Heuristic Tree Search (THTS) BID10 ) is an algorithmic framework that generalizes MCTS, dynamic programming, and heuristic search planning algorithms. In a THTS algorithm, we must specify five ingredients: action selection, backup function, heuristic function, outcome selection and the trial length. We discuss these ingredients and a modified version of THTS to additionally support UCT with ASNets in Section 3.Using these ingredients, BID10 create three new algorithms, all of which provide superior theoretical properties over UCT: MaxUCT, Dynamic Programming UCT (DP-UCT) and UCT*. DP-UCT and its variant UCT*, which use Bellman backups, were found to outperform original UCT and MaxUCT. Because of this, we will focus on DP-UCT, which we formally define in the next section. Our framework is a modification of DP-UCT from THTS. It is designed for SSPs with dead ends instead of finite horizon MDPs and is focused on minimizing the cost to a goal rather than maximizing rewards. It also introduces the simulation function, a generalization of random rollouts used in MCTS.We adopt the representation of alternating decision nodes and chance nodes in our search tree, as seen in THTS. DISPLAYFORM0 0 is the number of visits to the node in the first k trials, V k ∈ R + 0 is the state-value estimate based on the first k trials, and {n 1, . . ., n m} are the successor nodes (i.e. children) of n d. A chance node n c is a tuple s, a, C k, Q k, {n 1, . . ., n m}, where additionally, a ∈ A is the action, and Q k is the action-value estimate based on the first k trials. We use V k (n d) to refer to the state-value estimate of a decision node n d, a(n c) to refer to the action of a chance node n c, and so on for all the elements of n d and n c. Additionally, we use S(n) to represent the successor nodes {n 1, . . ., n m} of a search node n, and we also employ the shorthand DISPLAYFORM1 Initially, the search tree contains a single decision node n d with s(n d) = s 0, representing the initial state of our problem. UCT is described as an online planning algorithm, as it interleaves planning with execution. At each decision step, UCT returns an action either when a time cutoff is reached, or a maximum number of trials is performed. UCT then selects the chance node n c from the children of the root decision node that has the highest action-value estimate, Q k (n c), and applies its action a(n c). We sample a decision node n d from S(n c) based on the transition probabilities P (n d | n c) and set n d to be the new root of the tree. A single trial under our framework consists of the selection, expansion, simulation and backup phase. Selection Phase. As described in THTS, in this phase we traverse the explicit nodes in the search tree by alternating between action selection for decision nodes, and outcome selection for chance nodes until we reach an unexpanded decision node n d, which we call the tip node of the trial. Action selection is concerned with selecting a child chance node n c from the successors S(n d) of a decision node n d. UCT selects the child chance node that maximizes the UCB1 term, i.e. arg max nc∈S(n d) UCB1(n d, n c), where DISPLAYFORM0 B is the bias term which allows us to adjust the trade-off between exploration and exploitation. We set DISPLAYFORM1 to force the exploration of chance nodes that have not been visited. In outcome selection, we randomly sample an outcome of an action, i.e. sample a child decision node n d from the successors S(n c) of a chance node n c based on the transition probabilities P (n d | n c).Expansion Phase. In this phase, we expand the tip node n d and optionally initialize the Q-values of its child chance nodes, S(n d). Calculating an estimated Q-value requires calculating a weighted sum of the form: DISPLAYFORM2 where H is some domain-independent SSP heuristic function such as h add, h max, h pom, or h roc (Teichteil-Königsbuch, Vidal, and Infantes 2011; BID19 . This can be expensive when n c has many successor decision nodes. Simulation Phase. Immediately after the expansion phase, we transition to the simulation phase. Here we perform a simulation (also known as a rollout) of the planning problem from the tip node's state s(n d), until we reach a goal or dead-end state, or exceed the trial length. This stands in contrast to the behaviour of THTS, which lacks a simulation phase and would continuously switch between the selection and expansion phases until the trial length is reached. We use the simulation function to choose which action to take in a given state, and sample the next state according to the transition probabilities. If we complete a simulation without reaching a goal or dead end, we add a heuristic estimate H(s) to the rollout cost, where s is the final rollout state. If s is a dead end, then we set the rollout cost to be the dead-end penalty D.The trial length bounds how many steps can be applied in the simulation phase, and hence allows us to adjust the lookahead capability of DP-UCT. By setting the trial length to be very small, we can focus the search on nodes closer to the root of the tree, much like breadth-first search BID10. Following the steps above, if the trial length is 0, we do not perform any simulations and simply take a heuristic estimate for the tip node of the trial, or D if the tip node represents a dead-end. Traditional MCTS-based algorithms use a random simulation function, where each available action in the state has the same probability of being selected. However, this is not very suitable for SSPs as we can continuously loop around a set of states and never reach a goal state. Moreover, using a random simulation function requires an extremely large number of simulations to obtain good estimates for statevalues and action-values within the search tree. Because of this, the simulation phase in MCTS-based algorithms for planning is often neglected and replaced by a heuristic estimate. This is equivalent to setting the trial length to be 0, where we backup a heuristic estimate once we expand the tip node of the trial. However, there can be situations where the heuristic function is misleading or uninformative and thus misguides the search. In such a scenario, it could be more productive to use a random simulation function, or a simulation function influenced by domain-specific knowledge (i.e., the knowledge learned by an ASNet) to calculate estimates. Backup Phase. After the simulation phase, we must propagate the information we have gained from the current trial back up the search tree. We use the backup function to update the state-value estimate V k (n d) for decision nodes and the action-value estimate Q k (n c) for chance nodes. We do this by propagating the information we gained during the simulation in reverse order through the nodes in the trial path, by continuously applying the backup function for each node until we reach the root node of the search tree. Original UCT is defined with Monte-Carlo backups, in which the transition model is unknown and hence estimated based on the number of visits to nodes. However, in our work we consider the transition model to be known a priori. For that reason, DP-UCT only considers Bellman backups BID10, which additionally take the probabilities of outcomes into consideration when backing up action value estimates Q k (n c): DISPLAYFORM3. Υ k (n c) represents the child decision nodes of n c that have already been visited in the first k trials and hence have statevalue estimates. Thus,P (n d | n c) allows us to weigh the state-value estimate V k (n d) of each visited child decision node n d proportionally by its probability P (n d | n c) and that of the unvisited child decision nodes. It should be obvious that Bellman backups are derived directly from the Bellman optimality equations we presented in Section 2. Thus a flavor of UCT using Bellman backups is asymptotically optimal given a correct selection of ingredients that will ensure all nodes are explored infinitely often.4 Combining DP-UCT with ASNets Recall that an ASNet learns a stochastic policy π: A × S →, where π(a | s) represents the probability action a is applied in state s. We introduce two simulation functions which make use of a trained ASNet: STOCHASTIC AS-NETS which simply samples from the probability distribution given by π to select an action, and MAXIMUM ASNETS which selects the action with the highest probability -i.e. arg max a∈A(s) π(a | s).Since the navigation of the search space is heavily influenced by the state-value and action-value estimates we obtain from performing simulations, DP-UCT with an ASNetbased simulation function would ideally converge to the optimal policy in a smaller number of simulations compared to if we used the random simulation function. Of course, we expect this to be the case if an ASNet has learned some useful features or tricks about the environment or domain of the problem we are tackling. However, using ASNets as a simulation function may not be very robust if the learned policy is misleading and uninformative. Here, robustness indicates how well UCT can recover from the misleading information it has been provided. In this situation, DP-UCT with ASNets as a simulation function would require a significantly larger number of simulations in order to converge to the optimal policy than DP-UCT with a random simulation function. Regardless the quality of the learned policy, DP-UCT remains asymptotically optimal when using an ASNet-based simulation function if the selection of ingredients guarantees that our search algorithm will explore all nodes infinitely often. Nonetheless, an ASNet-based simulation function should only be used if its simulation from the tip node n d better approximates V * (n d) than a heuristic estimate H(s(n d)).Choosing between STOCHASTIC ASNETS and MAXI-MUM ASNETS. We can perceive the probability distribution given by the policy π of an ASNet to represent the'confidence' the network has in applying each action. Obviously, MAXIMUM ASNETS will completely bias the simulations towards what an ASNet believes is the best action for a given state. If the probability distribution is highly skewed towards a single action, then MAXIMUM ASNETS would be the better choice, as the ASNet is very'confident' in its decision to choose the corresponding action. On the other hand, if the probability distribution is relatively uniform, then STOCHASTIC ASNETS would likely be the better choice. In this situation, the ASNet may be uncertain and not very'confident' in its decision to choose among a set of actions. Thus, to determine which ASNet-based simulation function to use, we should carefully consider to what extent an ASNet is able to solve the given problem reliably. The UCB1 term allows us to balance the trade-off between exploration of actions in the search tree that have not been applied often, and exploitation of actions that we already know have good action-value estimates based on previous trials. By including an ASNet's influence within UCB1 through its policy π, we hope to maintain this fundamental trade-off yet further bias the action selection to what the ASNet believes are promising actions. Simple ASNet Action Selection. We select the child chance node n c of a decision node n d that maximizes: DISPLAYFORM0 where M ∈ R + and π(n c) = π(a(n c) | s(n c)) for the stochastic policy π learned by ASNet. Similar to UCB1, if a child chance node n c has not been visited before (i.e., C k (n c) = 0), we set SIMPLE-ASNET(n d, n c) = ∞ to force its exploration. The new parameter M, called the influence constant, allows us to control the exploitation of an ASNet's policy π for exploration and, the higher M is, the higher the influence of the ASNet in the action selection. Notice that the influence of the ASNet diminishes as we apply the action a(n c) more often because M ·π(n c)/C k (n c) decreases as the number of visits to the chance node n c increases. Moreover, since the bias provided DISPLAYFORM1 e., the original UCB1 bias term), SIMPLE-ASNET preserves the asymptotic optimality of UCB1: as C k (n c) → ∞, SIMPLE-ASNET(n d, n c) equals UCB1(n d, n c) and both converge to the optimal action-value Q * (n c) (Kocsis and Szepesvári 2006). Because of this similarity with UCB1 and their same initial condition (i.e., treating divisions by C k (n c) = 0 as ∞), we expect that SIMPLE-ASNET action selection will be robust to any misleading information provided by the policy of a trained ASNet. Nonetheless, the higher the value of the influence constant M, the more trials we require to combat any uninformative information. Ranked ASNet Action Selection. One pitfall of the infinite exploration bonus in SIMPLE-ASNET action selection when C k (n c) = 0 is that all child chance nodes must be visited at least once before we actually exploit the policy learned by an ASNet. Ideally, we should use the knowledge learned by an ASNet to select the order in which unvisited chance nodes are explored. Thus, we introduce RANKED-ASNET action selection, an extension to SIMPLE-ASNET action selection. The first condition stipulates that all chance nodes are selected and visited at least once before SIMPLE-ASNET action selection is used. Otherwise, chance nodes that have already been visited are given a value of −∞, while the values of unvisited nodes correspond to their probability in the policy π. Thus, unvisited child chance nodes are visited in decreasing order of their probability within the policy π. RANKED-ASNET action selection will allow DP-UCT to focus the initial stages of its search on what an ASNet believes are the most promising parts of the state space. Given that the ASNet has learned some useful knowledge of which action to apply at each step, we expect RANKED-ASNET action selection to require a smaller number of trials to converge to the optimal action in comparison with SIMPLE-ASNET action selection. However, RANKED-ASNET may not be as robust as SIMPLE-ASNET when the policy learned by an ASNet is misleading or uninformative. For example, if the optimal action has the lowest probability among all actions in the ASNet policy and is hence explored last, then we would require an increased number of trials to converge to this optimum. Comparison with ASNet-based Simulation Functions. DP-UCT with ASNet-influenced action selection is more robust to misleading information than DP-UCT with an ASNet-based simulation function. Since SIMPLE-ASNET and RANKED-ASNET action selection decreases the influence of a network as we apply an action it has suggested more frequently, we will eventually explore actions that may have a small probability in the policy learned by the ASNet but are in fact optimal. We would require a much larger number of trials to achieve this when using an ASNet-based simulation function, as the state-value and action-value estimates in the search tree would be directly derived from ASNet-based simulations. All experiments were performed on an Amazon Web Services EC2 c5.4x large instance with 16 CPUs and 32GB of memory. Each experiment was limited to one CPU core with a maximum turbo clock speed of 3.5 GHz. No restrictions were placed on the amount of memory an experiment used. Considered Planners. For our experiments, we consider two baseline planners: the original ASNets algorithm and UCT*. The latter is a variation of DP-UCT where the trial length is 0 while still using UCB1 to select actions, Bellman backups as the backup function, and no simulation function. UCT* was chosen as a baseline because it outperforms original DP-UCT due to its stronger theoretical properties BID10. We consider four parametrizations of our algorithms -namely, (i) Simple ASNets, (ii) Ranked ASNets, (iii) Stochastic ASNets, and (iv) Maximum ASNets -where: parametrizations (i) and (ii) are UCT* using SIMPLE and RANKED-ASNET action selection, respectively; and parametrizations (iii) and (iv) are DP-UCT with a problem-dependent trial length using STOCHASTIC and MAXIMUM ASNETS as the simulation function, respectively. ASNet Configuration. We use the same ASNet hyperparameters as described by Toyer et al. to train each network. Unless otherwise specified, we imposed a strict two hour time limit to train the network, though in most situations, the network finished training within one hour. All ASNets were trained using an LRTDP-based BID4 teacher that used LM-cut (Helmert and Domshlak 2009) as the heuristic to compute optimal policies. We only report the time taken to solve each problem for the final for an ASNet, and hence do not include the training time. DP-UCT Configuration. For all DP-UCT configurations we used h add BID3 as the heuristic function because it allowed DP-UCT to converge to a good solution in a reasonable time in our experiments, and set the UCB1 bias parameter B to √ 2. For all problems with dead ends, we enabled Q-value initialization, as it helps us avoid selecting a chance node for exploration that may lead to a dead end. We did not enable this for problems without dead ends because estimating Q-values is computationally expensive, and not beneficial in comparison to the number of trials that could have been performed in the same time frame. We gave all configurations a 10 second time cutoff to do trials and limited the maximum number of trials to 10,000 at each decision step to ensure fairness. Moreover, we set the dead-end penalty to be 500. We gave each planning round a maximum time of 1 hour, and a maximum of 100 execution steps. We ran 30 rounds per planner for each experiment. Stack Blocksworld. Stack Blocksworld is a special case of the deterministic Blocksworld domain in which the goal is to stack n blocks initially on the table into a single tower. We train an ASNet to unstack n blocks from a single tower and put them all down on the table. Since the network has never learned how to stack blocks, it completely fails at stacking the n blocks on the table into a single tower. A setting like this one-where the distributions of training and testing problems have non-overlapping support-represents a near-worst-case scenario for inductive learners like ASNets. In contrast, stacking blocks into a single tower is a relatively easy problem for UCT*. Our aim in this experiment is to show that DP-UCT can overcome the misleading information learned by ASNet policy. We train an ASNet on unstack problems with 2 to 10 blocks, and evaluate DP-UCT and ASNets on stack problems with 5 to 20 blocks. Exploding Blocksworld. This domain is an extension of deterministic Blocksworld, and is featured in the International Probabilistic Planning Competitions (IPPC). In Exploding Blocksworld, putting down a block can detonate and destroy the block or the table it was put down on. Once a block or the table is exploded, we can no longer use it; therefore, this domain contains unavoidable dead ends. A good policy avoids placing a block down on the table or down on another block that is required for the goal state (if possible). It is very difficult for an ASNet to reliably solve Exploding Blocksworld problems as each problem could have its own'trick' in order to avoid dead ends and reach the goal with minimal cost. We train an ASNet for 5 hours on a selected set of 16 problems (including those with avoidable and unavoidable dead ends) that were optimally solved by LRTDP within 2 minutes. 1 We evaluate ASNets and DP-UCT on the first eight problems from IPPC 2008 BID6. By combining DP-UCT and ASNets, we hope to exploit the limited knowledge and'tricks' learned by an ASNet on the problems it was trained on to navigate the search space. That is, we aim to learn what we have not learned, and improve suboptimal learning. CosaNostra Pizza BID18. The objective in CosaNostra Pizza is to safely deliver a pizza from the pizza shop to the waiting customer and then return to the shop. There is a series of toll booths on the two-way road between the pizza shop and the customer FIG1 ). At each toll booth, you can choose to either pay the toll operator or drive straight through without paying. We save a time step by driving straight through without paying but the operator becomes angry. Angry operators drop their toll gate on you and crush your car (leading to a dead end) with a probability of 50% when you next pass through their booth. Hence, the optimal policy is to only pay the toll operators on the trip to the customer, but not on the trip back to the pizza shop (as we will not revisit the booth). This ensures a safe return, as there will be no chance of a toll operator crushing your car at any stage. Thus, CosaNostra Pizza is an example of a problem with avoidable dead ends. An ASNet is able to learn the trick of paying the toll operators only on the trip to the customer, and scales up to large instances while heuristic search planners based on determinisation (either for search or for heuristic computation) do not scale up BID18. The reason for the underperformance of determinisation-based techniques (e.g., using h add as heuristic) is the presence of avoidable dead ends in the CosaNostra domain. Moreover, heuristics based on delete relaxation (e.g., h add) also underperform in the CosaNostra domain because they consider that the agent crosses each toll booth only once, i.e., this relaxation ignores the return path since it uses the same propositions as the path to the customer. Thus, we expect UCT* to not scale up to larger instances since it will require extremely long reasoning chains in order to always pay the toll operator on the trip to the customer; however, by combining DP-UCT with the optimal policy learned by an ASNet, we expect to scale up to much larger instances than UCT* alone. For the CosaNostra Pizza problems, we train an ASNet on problems with 1 to 5 toll-booths, and evaluate DP-UCT and ASNets on problems with 2 to 15 toll booths. Stack Blocksworld. We allocate to each execution step n/2 seconds for all runs of DP-UCT, where n is the number of blocks in the problem. We use Simple ASNets with the influence constant M set to 10, 50 and 100 to demonstrate how DP-UCT can overcome the misleading information provided by the ASNet. We do not run experiments that use ASNets as a simulation function, as that would in completely misleading state-value and action-value estimates in the search tree, meaning DP-UCT would achieve near-zero coverage. Figure 3 depicts our . ASNets achieves zero coverage, while UCT* is able to reliably achieve near-full coverage for all problems up to 20 blocks. In general, as we increase M, the coverage of Simple ASNets decays earlier as the number of blocks increases. This is not unexpected, as by increasing M, we increasingly'push' the UCB1 term to select actions that the ASNet wishes to exploit, and hence misguide the navigation of the search space. Nevertheless, Simple ASNets is able to achieve near-full coverage for problems with up to 17 blocks for M = 10, 15 blocks for M = 50, and approximately 11 blocks for M = 100. We also observed a general increase in the time taken to reach a goal as we increased M, though this was not always the case due to the noise of DP-UCT.This experiment shows that Simple ASNets is capable of learning what ASNet has not learned and being robust to changes in the environment by correcting the bad actions the ASNet suggests through search and eventually converging to the optimal solution. Exploding Blocksworld. For all DP-UCT flavors, we increased the UCB1 bias parameter B to 4 and set the maximum number of trials to 30,000 in order to promote more exploration. To combine DP-UCT with ASNets, we use Ranked ASNets with the influence constant M set to 10, 50 and 100. Note, that the coverage for Exploding Blocksworld is an approximation of the true probability of reaching the goal. Since we only run each algorithm 30 times, the are susceptible to chance. Table 1 shows our . 2 Since the training set used by ASNets was likely not representative of the evaluation problems (i.e., the IPPC 2008 problems), the policy learned by ASNets is suboptimal and failed to to reach the goal for the relatively easy problems (e.g., p04 and p07) while UCT* was able to more reliably solve these problems. By combining DP-UCT with ASNets through Ranked ASNets, we were able to either match the performance of UCT* or outperform it, even when ASNet achieved zero coverage for the given problem. However, for certain configurations, we were able to improve upon all other configurations. For p08, Ranked ASNets with M = 50 achieves a coverage of 10/30, while all other configurations of DP-UCT are only able to achieve a coverage of around 4/30. Despite the fact that the ASNet achieved zero coverage in this experiment, the general knowledge learned by the ASNet helped us navigate the search tree more effectively and efficiently, even if the suggestions provided by the ASNet are not optimal. The same reasoning applies to the for p04, where Ranked ASNets with M = 50 achieves a higher coverage than all other configurations. We have demonstrated that we can exploit the policy learned by an ASNet to achieve more promising than UCT* and the network itself, even if this policy is suboptimal. Thus, we have shown that Ranked ASNets is capable of learning what the ASNet has not learned and improving the suboptimal policy learned by the network. CosaNostra Pizza. For this experiment, we considered ASNets as both a simulation function (Stochastic and Maximum ASNets), and in the UCB1 term for action selection (Simple and Ranked ASNets with M = 100) to improve upon UCT*. The optimal policy for CosaNostra Pizza takes 3n + 4 steps, where n is the number of toll booths in the problem. We set the trial length when using ASNets as a simulation function to be 1.25 · (3n + 4), where the 25% increase gives some leeway for Stochastic ASNets. FIG3 shows our -the curves for ASNets and Maximum ASNets overlap, as well as the curves for Simple and Ranked ASNets. ASNets achieves full coverage for all problems, while UCT* alone is only able to achieve full coverage for the problems with 2 and 3 toll booths. Using ASNets in the action selection ingredient through Simple or Ranked ASNets with the influence constant M = 100 only allows us to additionally achieve full coverage for the problem with 4 toll booths. This is because Simple and Ranked ASNets guide the action selection towards the optimal action, but UCT still forces the exploration of other parts of the state space. We are able to more reliably solve CosaNostra Pizza prob-In this experiment, we have shown how using ASNets in UCB1 through SIMPLE-ASNET or RANKED-ASNET action selection can only provide marginal improvements over UCT* when the number of reachable states increases exponentially with the problem size, and the heuristic estimates are misleading. We also demonstrated how we can combat this sub-optimal performance of DP-UCT by using ASNets as a simulation function, as it allows us to more efficiently explore the search space and find the optimal actions. Thus, an ASNet-based simulation function may help DP-UCT learn what it has not learned. Tireworld is a domain with avoidable dead ends. ASNets is trivially able to find the optimal policy which always avoids dead ends. The of our new algorithms on Triangle Tireworld are very similar to the in the CosaNostra experiments, as the algorithms leverage the fact that ASNets finds the optimal generalized policy for both domains. In this paper, we have investigated techniques to improve search using generalized policies. We discussed a framework for DP-UCT, extended from THTS, that allowed us to generate different flavors of DP-UCT including those that exploited the generalized policy learned by an ASNet. We then introduced methods of using this generalized policy in the simulation function, through STOCHASTIC ASNETS and MAXIMUM ASNETS. These allowed us to obtain more accurate state-value estimates and action-value estimates in the search tree. We also extended UCB1 to bias the navigation of the search space to the actions that an ASNet wants to exploit whilst maintaining the fundamental balance between exploration and exploitation, by introducing SIMPLE-ASNET and RANKED-ASNET action selection. We have demonstrated through our experiments that our algorithms are capable of improving the capability of an ASNet to generalize beyond the distribution of problems it was trained on, as well as improve sub-optimal learning. By combining DP-UCT with ASNets, we are able to bias the exploration of actions to those that an ASNet wishes to exploit, and allow DP-UCT to converge to the optimal action in a smaller number of trials. Our experiments have also demonstrated that by harnessing the power of search, we may overcome any misleading information provided by an ASNet due to a change in the environment. Hence, we achieved the three following goals: Learn what we have not learned, Improve on sub-optimal learning, and Be robust to changes in the environment or domain. It is important to observe that our contributions are more generally applicable to any method of learning a (generalized) policy (not just ASNets), and potentially to other trialbased search algorithms including (L)RTDP.In the deterministic setting, there has been a long tradition of learning generalized policies and using them to guide heuristic Best First Search (BFS). For instance, Yoon et al. BID20 add the states ing from selecting actions prescribed by the learned generalized policy to the the queue of a BFS guided by a relaxed-plan heuristic, and de la BID7 learn and use generalized policies to generate lookahead states within a BFS guided by the FF heuristic. These authors observe that generalized policies provide effective search guidance, and that search helps correcting deficiencies in the learned policy. Search control knowledgeà la TLPlan, Talplanner or SHOP2 has been successfully used to prune the search of probabilistic planners BID13 BID17 ). More recently, BID15 have also experimented with the use of preferred actions in variants of RTDP BID1 and AO* BID14, albeit with limited success. Our work differs from these approaches by focusing explicitly on MCTS as the search algorithm and, unlike existing work combining deep learning and MCTS (e.g. AlphaGo (Silver et al. 2016)), looks not only at using neural network policies as a simulation function for rollouts, but also as a means to bias the UCB1 action selection rule. There are still many potential avenues for future work. We may investigate how to automatically learn the influence parameter M for SIMPLE-ASNET and RANKED-ASNET action selection, or how to combat bad information provided by an ASNet in a simulation function by mixing ASNet simulations with random simulations. We may also investigate techniques to interleave planning with learning by using UCT with ASNets as a'teacher' for training an AS

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

Techniques for combining generalized policies with search algorithms to exploit the strengths and overcome the weaknesses of each when solving probabilistic planning problems

Represent the short summary for finding the corresponding scientific article (long).

Parameter space noise allows reinforcement learning algorithms to explore by perturbing parameters instead of actions, often leading to significantly improved exploration performance.