Question: represent the article for finding a single-sentence tldr.

Question: Implicit probabilistic models are models defined naturally in terms of a sampling procedure and often induces a likelihood function that cannot be expressed explicitly. We develop a simple method for estimating parameters in implicit models that does not require knowledge of the form of the likelihood function or any derived quantities, but can be shown to be equivalent to maximizing likelihood under some conditions. Our holds in the non-asymptotic parametric setting, where both the capacity of the model and the number of data examples are finite. We also demonstrate encouraging experimental . Generative modelling is a cornerstone of machine learning and has received increasing attention. Recent models like variational autoencoders (VAEs) BID32 BID45 and generative adversarial nets (GANs) BID21 BID25, have delivered impressive advances in performance and generated a lot of excitement. Generative models can be classified into two categories: prescribed models and implicit models BID12 BID40. Prescribed models are defined by an explicit specification of the density, and so their unnormalized complete likelihood can be usually expressed in closed form. Examples include models whose complete likelihoods lie in the exponential family, such as mixture of Gaussians BID18, hidden Markov models BID5, Boltzmann machines BID27. Because computing the normalization constant, also known as the partition function, is generally intractable, sampling from these models is challenging. On the other hand, implicit models are defined most naturally in terms of a (simple) sampling procedure. Most models take the form of a deterministic parameterized transformation T θ (·) of an analytic distribution, like an isotropic Gaussian. This can be naturally viewed as the distribution induced by the following sampling procedure:1. Sample z ∼ N (0, I) 2. Return x:= T θ (z)The transformation T θ (·) often takes the form of a highly expressive function approximator, like a neural net. Examples include generative adversarial nets (GANs) BID21 BID25 and generative moment matching nets (GMMNs) BID36 BID16. The marginal likelihood of such models can be characterized as follows: DISPLAYFORM0 where φ(·) denotes the probability density function (PDF) of N (0, I).In general, attempting to reduce this to a closed-form expression is hopeless. Evaluating it numerically is also challenging, since the domain of integration could consist of an exponential number of disjoint regions and numerical differentiation is ill-conditioned. These two categories of generative models are not mutually exclusive. Some models admit both an explicit specification of the density and a simple sampling procedure and so can be considered as both prescribed and implicit. Examples include variational autoencoders BID32 BID45, their predecessors BID38 BID10 and extensions BID11, and directed/autoregressive models, e.g., BID42 BID6 BID33 van den ). Maximum likelihood BID19 BID17 is perhaps the standard method for estimating the parameters of a probabilistic model from observations. The maximum likelihood estimator (MLE) has a number of appealing properties: under mild regularity conditions, it is asymptotically consistent, efficient and normal. A long-standing challenge of training probabilistic models is the computational roadblocks of maximizing the log-likelihood function directly. For prescribed models, maximizing likelihood directly requires computing the partition function, which is intractable for all but the simplest models. Many powerful techniques have been developed to attack this problem, including variational methods BID31, contrastive divergence BID26 ), score matching BID30 and pseudolikelihood maximization BID9, among others. For implicit models, the situation is even worse, as there is no term in the log-likelihood function that is in closed form; evaluating any term requires computing an intractable integral. As a , maximizing likelihood in this setting seems hopelessly difficult. A variety of likelihood-free solutions have been proposed that in effect minimize a divergence measure between the data distribution and the model distribution. They come in two forms: those that minimize an f -divergence, and those that minimize an integral probability metric BID41. In the former category are GANs, which are based on the idea of minimizing the distinguishability between data and samples (; BID24 . It has been shown that when given access to an infinitely powerful discriminator, the original GAN objective minimizes the Jensen-Shannon divergence, the − log D variant of the objective minimizes the reverse KL-divergence minus a bounded quantity, and later extensions BID43 minimize arbitrary f -divergences. In the latter category are GMMNs which use maximum mean discrepancy (MMD) BID22 as the witness function. In the case of GANs, despite the theoretical , there are a number of challenges that arise in practice, such as mode dropping/collapse BID21, vanishing gradients ) and training instability BID21. A number of explanations have been proposed to explain these phenomena and point out that many theoretical rely on three assumptions: the discriminator must have infinite modelling capacity BID21, the number of samples from the true data distribution must be infinite ) and the gradient ascent-descent procedure BID4 ) can converge to a global pure-strategy Nash equilibrium BID21. When some of these assumptions do not hold, the theoretical guarantees do not necessarily apply. A number of ways have been proposed that alleviate some of these issues, e.g., (; ; BID13 BID15 BID28), but a way of solving all three issues simultaneously remains elusive. In this paper, we present an alternative method for estimating parameters in implicit models. Like the methods above, our method is likelihood-free, but can be shown to be equivalent to maximizing likelihood under some conditions. Our holds when the capacity of the model is finite and the number of data examples is finite. The idea behind the method is simple: it finds the nearest sample to each data example and optimizes the model parameters to pull the sample towards it. The direction in which nearest neighbour search is performed is important: the proposed method ensures each data example has a similar sample, which contrasts with an alternative approach of pushing each sample to the nearest data example, which would ensure that each sample has a similar data example. The latter approach would permit all samples being similar to one data example. Such a scenario would be heavily penalized by the former approach. The proposed method could sidestep the three issues mentioned above: mode collapse, vanishing gradients and training instability. Modes are not dropped because the loss ensures each data example has a sample nearby at optimality; gradients do not vanish because the gradient of the distance between a data example and its nearest sample does not become zero unless they coincide; training is stable because the estimator is the solution to a simple minimization problem. By leveraging recent advances in fast nearest neighbour search algorithms BID34, this approach is able to scale to large, high-dimensional datasets.longer the case, due to recent advances in nearest neighbour search algorithms BID34.Note that the use of Euclidean distance is not a major limitation of the proposed approach. A variety of distance metrics are either exactly or approximately equivalent to Euclidean distance in some non-linear embedding space, in which case the theoretical guarantees are inherited from the Euclidean case. This encompasses popular distance metrics used in the literature, like the Euclidean distance between the activations of a neural net, which is often referred to as a perceptual similarity metric (; BID14 . The approach can be easily extended to use these metrics, though because this is the initial paper on this method, we focus on the vanilla setting of Euclidean distance in the natural representation of the data, e.g.: pixels, both for simplicity/clarity and for comparability to vanilla versions of other methods that do not use auxiliary sources of labelled data or leverage domain-specific prior knowledge. For distance metrics that cannot be embedded in Euclidean space, the analysis can be easily adapted with minor modifications as long as the volume of a ball under the metric has a simple dependence on its radius. There has been debate BID29 over whether maximizing likelihood of the data is the appropriate objective for the purposes of learning generative models. Recall that maximizing likelihood is equivalent to minimizing D KL (p data p θ), where p data denotes the empirical data distribution and p θ denotes the model distribution. One proposed alternative is to minimize the reverse KLdivergence, D KL (p θ p data), which is suggested BID29 to be better because it severely penalizes the model for generating an implausible sample, whereas the standard KL-divergence, D KL (p data p θ), severely penalizes the model for assigning low density to a data example. As a , when the model is underspecified, i.e. has less capacity than what's necessary to fit all the modes of the data distribution, minimizing D KL (p θ p data) leads to a narrow model distribution that concentrates around a few modes, whereas minimizing D KL (p data p θ) leads to a broad model distribution that hedges between modes. The success of GANs in generating good samples is often attributed to the former phenomenon.This argument, however, relies on the assumption that we have access to an infinite number of samples from the true data distribution. In practice, however, this assumption rarely holds: if we had access to the true data distribution, then there is usually no need to fit a generative model, since we can simply draw samples from the true data distribution. What happens when we only have the empirical data distribution? Recall that D KL (p q) is defined and finite only if p is absolutely continuous w.r.t. q, i.e.: q(x) = 0 implies p(x) = 0 for all x. In other words, D KL (p q) is defined and finite only if the support of p is contained in the support of q. Now, consider the difference between D KL (p data p θ) and D KL (p θ p data): minimizing the former, which is equivalent to maximizing likelihood, ensures that the support of the model distribution contains all data examples, whereas minimizing the latter ensures that the support of the model distribution is contained in the support of the empirical data distribution, which is just the set of data examples. In other words, maximum likelihood disallows mode dropping, whereas minimizing reverse KL-divergence forces the model to assign zero density to unseen data examples and effectively prohibits generalization. Furthermore, maximum likelihood discourages the model from assigning low density to any data example, since doing so would make the likelihood, which is the product of the densities at each of the data examples, small. From the modelling perspective, because maximum likelihood is guaranteed to preserve all modes, it can make use of all available training data and can therefore be used to train high-capacity models that have a large number of parameters. In contrast, using an objective that permits mode dropping allows the model to pick and choose which data examples it wants to model. As a , if the goal is to train a high-capacity model that can learn the underlying data distribution, we would not be able to do so using such an objective because we have no control over which modes the model chooses to drop. Put another way, we can think about the model's performance along two axes: its ability to generate plausible samples (precision) and its ability to generate all modes of the data distribution (recall). A model that successfully learns the underlying distribution should score high along both axes. If mode dropping is allowed, then an improvement in precision may be achieved at the expense of lower recall and could represent a move to a different point on the same precision-recall curve. As a , since sample quality is an indicator of precision, improvement in sample quality in this setting may not mean an improvement in density estimation performance. On the other hand, if mode dropping is disallowed, since full recall is always guaranteed, an improvement in precision is achieved without sacrificing recall and so implies an upwards shift in the precision-recall curve. In this case, an improvement in sample quality does signify an improvement in density estimation performance, which may explain sample quality historically was an important way to evaluate the performance of generative models, most of which maximized likelihood. With the advent of generative models that permit mode dropping, however, sample quality is no longer a reliable indicator of density estimation performance, since good sample quality can be trivially achieved by dropping all but a few modes. In this setting, sample quality can be misleading, since a model with low recall on a lower precision-recall curve can achieve a better precision than a model with high recall on a higher precision-recall curve. Since it is hard to distinguish whether an improvement in sample quality is due to a move along the same precision-recall curve or a real shift in the curve, an objective that disallows mode dropping is critical tool that researchers can use to develop better models, since they can be sure that an apparent improvement in sample quality is due to a shift in the precision-recall curve.. Let F θ (·) be the cumulative distribution function (CDF) ofr θ and Ψ(z):= min θ E R θ |p θ = z.If P θ satisfies the following:• p θ (x) is differentiable w.r.t. θ and continuous w.r.t. x everywhere.• ∀θ, v, there exists θ such that p θ (x) = p θ (x + v) ∀x.• For any θ 1, θ 2, there exists θ 0 such that DISPLAYFORM0 MNIST TFD DBN BID7 138 ± 2 1909 ± 66 SCAE BID7 121 ± 1.6 2110 ± 50 DGSN 214 ± 1.1 1890 ± 29 GAN BID21 225 ± 2 2057 ± 26 GMMN BID36 147 ± 2 2085 ± 25 IMLE (Proposed) 257 ± 6 2139 ± 27 Table 1: Log-likelihood of the test data under the Gaussian Parzen window density estimated from samples generated by different methods.• DISPLAYFORM0, where B θ * (τ) denotes the ball centred at θ * of radius τ.• Ψ(z) is differentiable everywhere.• DISPLAYFORM1... DISPLAYFORM2 Now, we examine the restrictiveness of each condition. The first condition is satisfied by nearly all analytic distributions. The second condition is satisfied by nearly all distributions that have an unrestricted location parameter, since one can simply shift the location parameter by v. The third condition is satisfied by most distributions that have location and scale parameters, like a Gaussian distribution, since the scale can be made arbitrarily low and the location can be shifted so that the constraint on p θ (·) is satisfied. The fourth condition is satisfied by nearly all distributions, whose density eventually tends to zero as the distance from the optimal parameter setting tends to infinity. The fifth condition requires min θ E R θ |p θ = z to change smoothly as z changes. The final condition requires the two n-dimensional vectors, one of which can be chosen from a set of d vectors, to be not exactly orthogonal. As a , this condition is usually satisfied when d is large, i.e. when the model is richly parameterized. There is one remaining difficulty in applying this theorem, which is that the quantity 1/Ψ (p θ * (x i))p θ * (x i), which appears as an coefficient on each term in the proposed objective, is typically not known. If we consider a new objective that ignores the coefficients, i.e. n i=1 E R θ i, then minimizing this objective is equivalent to minimizing an upper bound on the ideal objective, DISPLAYFORM3 The tightness of this bound depends on the difference between the highest and lowest likelihood assigned to individual data points at the optimum, i.e. the maximum likelihood estimate of the parameters. Such a model should not assign high likelihoods to some points and low likelihoods to others as long as it has reasonable capacity, since doing so would make the overall likelihood, which is the product of the likelihoods of individual data points, low. Therefore, the upper bound is usually reasonably tight. We trained generative models using the proposed method on three standard benchmark datasets, MNIST, the Toronto Faces Dataset (TFD) and CIFAR-10. All models take the form of feedforward neural nets with isotropic Gaussian noise as input. For MNIST, the architecture consists of two fully connected hidden layers with 1200 units each followed by a fully connected output layer with 784 units. ReLU activations were used for hidden layers and sigmoids were used for the output layer. For TFD, the architecture is wider and consists of two fully connected hidden layers with 8000 units each followed by a fully connected output layer with 2304 units. For both MNIST and TFD, the dimensionality of the noise vector is 100. For CIFAR-10, we used a simple convolutional architecture with 1000-dimensional Gaussian noise as input. The architecture consists of five convolutional layers with 512 output channels and a kernel size of 5 that all produce 4 × 4 feature maps, followed by a bilinear upsampling layer that doubles the width and height of the feature maps. There is a batch normalization layer followed by leaky ReLU activations with slope −0.2 after each convolutional layer. This design is then repeated for each subsequent level of resolution, namely 8 × 8, 16 × 16 and 32 × 32, so that we have 20 convolutional layers, each with output 512 channels. We then add a final output layer with three output channels on top, followed by sigmoid activations. We note that this architecture has more capacity than typical architectures used in other methods, like BID44. This is because our method aims to capture all modes of the data distribution and therefore needs more modelling capacity than methods that are permitted to drop modes. sirable properties of the generative model that do not affect performance on the task. Intrinsic evaluation metrics measure performance without relying on external models or data. Popular examples include estimated log-likelihood ) and visual assessment of sample quality. While recent literature has focused more on the latter and less on the former, it should be noted that they evaluate different properties -sample quality reflects precision, i.e.: how accurate the model samples are compared to the ground truth, whereas estimated log-likelihood focuses on recall, i.e.: how much of the diversity in the data distribution the model captures. Consequently, both are important metrics; one is not a replacement for the other. As pointed out by , "qualitative as well as quantitative analyses based on model samples can be misleading about a model's density estimation performance, as well as the probabilistic model's performance in applications other than image synthesis." Two models that achieve different levels of precision may simply be at different points on the same precision-recall curve, and therefore may not be directly comparable. Models that achieve the same level of recall, on the other hand, may be directly compared. So, for methods that maximize likelihood, which are guaranteed to preserve all modes and achieve full recall, both sample quality and estimated log-likelihood capture precision. Because most generative models traditionally maximized likelihood or a lower bound on the likelihood, the only property that differed across models was precision, which may explain why sample quality has historically been seen as an important indicator of performance. However, in heterogenous experimental settings with different models optimized for various objectives, sample quality does not necessarily reflect how well a model learns the underlying data distribution. Therefore, under these settings, both precision and recall need to be measured. While there is not yet a reliable way to measure recall (given the known issues of estimated log-likelihoods in high dimensions), this does not mean that sample quality can be a valid substitute for estimated log-likelihoods, as it cannot detect the lack of diversity of samples. A secondary issue that is more easily solvable is that samples presented in papers are sometimes cherry-picked; as a , they capture the maximum sample quality, but not necessarily the mean sample quality. To mitigate these problems to some extent, we avoid cherry-picking and visualize randomly chosen samples, which are shown in FIG0. We also report the estimated log-likelihood in Table 1. As mentioned above, both evaluation criteria have biases/deficiencies, so performing well on either of these metrics does not necessarily indicate good density estimation performance. However, not performing badly on either metric can provide some comfort that the model is simultaneously able to achieve reasonable precision and recall. As shown in FIG0, despite its simplicity, the proposed method is able to generate reasonably good samples for MNIST, TFD and CIFAR-10. While it is commonly believed that minimizing reverse KL-divergence is necessary to produce good samples and maximizing likelihood necessarily leads to poor samples BID23, the suggest that this is not necessarily the case. Even though Euclidean distance was used in the objective, the samples do not appear to be desaturated or overly blurry. Samples also seem fairly diverse. This is supported by the estimated log-likelihood in Table 1. Because the model achieved a high score on that metric on both MNIST and TFD, this suggests that the model did not suffer from significant mode dropping. In FIG3 in the supplementary material, we show samples and their nearest neighbours in the training set. Each sample is quite different from its nearest neighbour in the training set, suggesting that the model has not overfitted to examples in the training set. Next, we visualize the learned manifold by walking along a geodesic on the manifold between pairs of samples. More concretely, we generate five samples, arrange them in arbitrary order, perform linear interpolation in latent variable space between adjacent pairs of samples, and generate an image from the interpolated latent variable. As shown in FIG2, the images along the path of interpolation appear visually plausible and do not have noisy artifacts. In addition, the transition from one image to the next appears smooth, including for CIFAR-10, which contrasts with findings in the literature that suggest the transition between two natural images tends to be abrupt. This indicates that the support of the model distribution has not collapsed to a set of isolated points and that the proposed method is able to learn the geometry of the data manifold, even though it does not learn a distance metric explicitly. Finally, we illustrate the evolution of samples as training progresses in FIG1. As shown, the samples are initially blurry and become sharper over time. Importantly, sample quality consistently improves over time, which demonstrates the stability of training. While our sample quality may not be state-of-the-art, it is important to remember that these are obtained under the setting of full recall. So, this does not necessarily mean that our method models the underlying data distribution less accurately than other methods that achieve better sample quality, as some of them may drop modes and therefore achieve less than full recall. As previously mentioned, this does not suggest a fundamental tradeoff between precision and recall that cannot be overcome -on the contrary, our method provides researchers with a way of designing models that can improve the precision-recall curve without needing to worry that the observed improvements are due to a movement along the curve. With refinements to the model, it is possible to move the curve upwards and obtain better sample quality at any level of recall as a consequence. This is left for future work; as this is the initial paper on this approach, its value stems from the foundation it lays for a new research direction upon which subsequent work can be built, as opposed to the current themselves. For this paper, we made a deliberate decision to keep the model simple, since non-essential practically motivated enhancements are less grounded in theory, may obfuscate the key underlying idea and could impart the impression that they are critical to making the approach work in practice. The fact that our method is able to generate more plausible samples on CIFAR-10 than other methods at similar stages of development, such as the initial versions of GAN BID21 and PixelRNN (van den), despite the minimal sophistication of our method and architecture, shows the promise of the approach. Later iterations of other methods incorporate additional supervision in the form of pretrained weights and/or make task-specific modifications to the architecture and training procedure, which were critical to achieving state-of-the-art sample quality. We do believe the question of how the architecture should be refined in the context of our method to take advantage of task-specific insights is an important one, and is an area ripe for future exploration. In this section, we consider and address some possible concerns about our method. It has been suggested BID29 that maximizing likelihood leads to poor sample quality because when the model is underspecified, it will try to cover all modes of the empirical data distribution and therefore assign high density to regions with few data examples. There is also empirical evidence BID23 for a negative correlation between sample quality and log likelihood, suggesting an inherent trade-off between maximizing likelihood and achieving good sample quality. A popular solution is to minimize reverse KL-divergence instead, which trades off recall for pre-cision. This is an imperfect solution, as the ultimate goal is to model all the modes and generate high-quality samples. Note that this apparent trade-off exists that the model capacity is assumed to be fixed. We argue that a more promising approach would be to increase the capacity of the model, so that it is less underspecified. As the model capacity increases, avoiding mode dropping becomes more important, because otherwise there will not be enough training data to fit the larger number of parameters to. This is precisely a setting appropriate for maximum likelihood. As a , it is possible that a combination of increasing the model capacity and maximum likelihood training can achieve good precision and recall simultaneously. When the model has infinite capacity, minimizing distance from data examples to their nearest samples will lead to a model distribution that memorizes data examples. The same is true if we maximize likelihood. Likewise, minimizing any divergence measure will lead to memorization of data examples, since the minimum divergence is zero and by definition, this can only happen if the model distribution is the same as the empirical data distribution, whose support is confined to the set of data examples. This implies that whenever we have a finite number of data examples, any method that learns a model with infinite capacity will memorize the data examples and will hence overfit. To get around this, most methods learn a parametric model with finite capacity. In the parametric setting, the minimum divergence is not necessarily zero; the same is true for the minimum distance from data examples to their nearest samples. Therefore, the optimum of these objective functions is not necessarily a model distribution that memorizes data examples, and so overfitting will not necessarily occur. observes that the data distribution and the model distribution are supported on low-dimensional manifolds and so they are unlikely to have a non-negligible intersection. They point out D KL (p data p θ) would be infinite in this case, or equivalently, the likelihood would be zero. While this does not invalidate the theoretical soundness of maximum likelihood, since the maximum of a non-negative function that is zero almost everywhere is still well-defined, it does cause a lot of practical issues for gradient-based learning, as the gradient is zero almost everywhere. This is believed to be one reason that models like variational autoencoders BID32 BID45 use a Gaussian distribution with high variance for the conditional likelihood/observation model rather than a distribution close to the Dirac delta, so that the support of the model distribution is broadened to cover all the data examples.This issue does not affect our method, as our loss function is different from the log-likelihood function, even though their optima are the same (under some conditions). As the , the gradients of our loss function are different from those of log-likelihood. When the supports of the data distribution and the model distribution do not overlap, each data example is likely far away from its nearest sample and so the gradient is large. Moreover, the farther the data examples are from the samples, the larger the gradient gets. Therefore, even when the gradient of log-likelihood can be tractably computed, there may be situations when the proposed method would work better than maximizing likelihood directly. We presented a simple and versatile method for parameter estimation when the form of the likelihood is unknown. The method works by drawing samples from the model, finding the nearest sample to every data example and adjusting the parameters of the model so that it is closer to the data example. We showed that performing this procedure is equivalent to maximizing likelihood under some conditions. The proposed method can capture the full diversity of the data and avoids common issues like mode collapse, vanishing gradients and training instability. The method combined with vanilla model architectures is able to achieve encouraging on MNIST, TFD and CIFAR-10. Before proving the main , we first prove the following intermediate : • There is a bounded set S ⊆ Ω such that bd(S) ⊆ Ω, θ * ∈ S and ∀f i, ∀θ ∈ Ω \ S, f i (θ) > f i (θ *), where bd(S) denotes the boundary of S. DISPLAYFORM0 • DISPLAYFORM1... DISPLAYFORM2 Proof. Let S ⊆ Ω be the bounded set such that bd(S) ⊆ Ω, θ * ∈ S and ∀f i, ∀θ ∈ Ω \ S, f i (θ) > f i (θ *). Consider the closure of S:= S ∪ bd(S), denoted asS. Because S ⊆ Ω and bd(S) ⊆ Ω, S ⊆ Ω. Since S is bounded,S is bounded. BecauseS ⊆ Ω ⊆ R d and is closed and bounded, it is compact. is differentiable on Ω and hence continuous on Ω. By the compactness ofS and the continuity of DISPLAYFORM0 since Φ is strictly increasing. Because Φ (·) > 0, w i > 0 and so DISPLAYFORM1 At the same time, since θ * ∈ S ⊂S, by definition ofθ, DISPLAYFORM2. Combining these two facts yields DISPLAYFORM3 Since the inequality is strict, this implies that θ /∈ Ω \ S, and soθ DISPLAYFORM4 In addition, becauseθ is the minimizer of DISPLAYFORM5 On the other hand, since Φ is differentiable on V and f i (θ) ∈ V for all θ ∈ Ω, Φ (f i (θ)) exists for all θ ∈ Ω. So, DISPLAYFORM6 Combining this with the fact that θ * is the minimizer of DISPLAYFORM7 Because ∀θ ∈ Ω, if θ = θ *, ∃j ∈ Sinceθ is a critical point on Ω, we can conclude that θ * =θ, and so θ * is a minimizer of N i=1 w i Φ(f i (·)) on Ω. Since any other minimizer must be a critical point and θ * is the only critical point, θ * is the unique minimizer. So, arg min θ∈Ω Let > 0 be arbitrary. Since p(·) is continuous at x 0, by definition, ∀˜ > 0 ∃δ > 0 such that ∀u ∈ B x0 (δ), |p(u) − p(x 0)| <˜. Letδ > 0 be such that ∀u ∈ B x0 (δ), p(x 0) − < p(u) < p(x 0) +. We choose δ =δ. Let 0 <h < δ be arbitrary. Since p(x 0)− < p(u) < p(x 0)+ ∀u ∈ B x0 (δ) = B x0 (δ) ⊃ B x0 (h), DISPLAYFORM8

represent the tldr for finding the corresponding scientific article
Examples:

"Node sequence embedding mechanism that captures both graph and text properties." == "Effectively capturing graph node sequences in the form of vector embeddings is critical to many applications. We achieve this by (i) first learning vector embeddings of single graph nodes and (ii) then composing them to compactly represent node sequences. Specifically, we propose SENSE-S (Semantically Enhanced Node Sequence Embedding - for Single nodes), a skip-gram based novel embedding mechanism, for single graph nodes that co-learns graph structure as well as their textual descriptions. We demonstrate that SENSE-S vectors increase the accuracy of multi-label classification tasks by up to 50% and link-prediction tasks by up to 78% under a variety of scenarios using real datasets. Based on SENSE-S, we next propose generic SENSE to compute composite vectors that represent a sequence of nodes, where preserving the node order is important. We prove that this approach is efficient in embedding node sequences, and our experiments on real data confirm its high accuracy in node order decoding. Accurately learning vector embeddings for a sequence of nodes in a graph is critical to many scenarios, e.g., a set of Web pages regarding one specific topic that are linked together. Such a task is challenging as: (i) the embeddings may have to capture graph structure along with any available textual descriptions of the nodes, and moreover, (ii) nodes of interest may be associated with a specific order. For instance, (i) for a set of Wikipedia pages w.r.t. a topic, there exists a recommended reading sequence; (ii) an application may consist of a set of services/functions, which must be executed in a particular order (workflow composability); (iii) in source routing, the sender of a packet on the Internet specifies the path that the packet takes through the network or (iv) the general representation of any path in a graph or a network, e.g., shortest path. Node sequence embedding, thus, requires us to (i) learn embeddings for each individual node of the graph and (ii) compose them together to represent their sequences. To learn the right representation of individual nodes and also their sequences, we need to understand how these nodes are correlated with each other both functionally and structurally. A lot of work has only gone into learning single node embeddings (i.e., where node sequence length is 1), as they are essential in feature representations for applications like multi-label classification or link prediction. For instance, algorithms in BID22, BID4, BID28 and others try to extract features purely from the underlying graph structure; algorithms in BID12, BID19 and others learn vector representations of documents sharing a common vocabulary set. However, many applications would potentially benefit from representations that are able to capture both textual descriptions and the underlying graph structure simultaneously. For example, classification of nodes in a network not only depends on their inter-connections (i.e., graph structure), but also nodes' intrinsic properties (i.e., their textual descriptions); for product recommendations, if the product is new, it may not have many edges since not many users have interacted with it; however, using the textual descriptions along with the graph structure allows for efficient bootstrapping of the recommendation service. For general case of sequence lengths greater than 1, despite the importance in applications like workflow composability described above, there is generally a lack of efficient solutions. Intuitively, we can concatenate or add all involved node vectors; however, such a mechanism either takes too much space or loses the sequence information; thus unable to represent node sequences properly. We aim to learn node sequence embeddings by first first addressing the single node embedding issue, as a special case of node sequence embedding, by considering both the textual descriptions and the graph structure. We seek to answer two questions: How should we combine these two objectives? What framework should we use for feature learning? Works that jointly address these two questions either investigate them under different problem settings BID1 BID32, under restricted learning models BID13, ignore the word context within the document BID16, do not co-learn text and graph patterns or only consider linear combinations of text and graph BID3; this is elaborated further in Section 2. In contrast, we propose a generic neural-network-based model called SENSE-S (Semantically Enhanced Node Sequence Embeddings -for Single nodes) for computing vector representations of nodes with additional semantic information in a graph. SENSE-S is built on the foundation of skip-gram models. However, SENSE-S is significantly different from classic skipgram models in the following aspects: (i) For each word φ in the textual description of node v in the given graph, neighboring words of φ within v's textual description and neighboring nodes of v within the graph are sampled at the same time.(ii) The text and graph inputs are both reflected in the output layer in the form of probabilities of co-occurrence (in graph or text). (iii) Moreover, this joint optimization problem offers an opportunity to leverage the synergy between the graph and text inputs to ensure faster convergence. We evaluate the generated vectors on (i) to show that our SENSE-S model improves multi-label classification accuracy by up to 50% and (ii) Physics Citation dataset BID14 to show that SENSE-S improves link prediction accuracy by up to 78% over the state-of-the-art. Next, we propose SENSE for general feature representation of a sequence of nodes. This problem is more challenging in that (i) besides the original objectives in SENSE-S, we now face another representation goal, i.e., sequence representation while preserving the node order; (ii) it is important to represent the sequence in a compact manner; and (iii) more importantly, given a sequence vector, we need to be able to decipher which functional nodes are involved and in what order. To this end, we develop efficient schemes to combine individual vectors into complex sequence vectors that address all of the above challenges. The key technique we use here is vector cyclic shifting, and we prove that the different shifted vectors are orthogonal with high probability. This sequence embedding method is also evaluated on the Wikispeedia and Physics Citation datasets, and the accuracy of decoding a node sequence is shown to be close to 100% when the vector dimension is large. We overview the most related works by categorizing them as follows:Learning vector representation from text: Vector representation of words (Schütze, 1993) has been a long standing research topic. It has received significant attention in the recent times due to the advances in deep neural networks BID0 BID18. In particular, these neural-network-based schemes outperform n-gram-based techniques BID10 BID9 significantly as they are able to learn the similarities between words. Furthermore, paragraph2vec BID12 extends the well-established word2vec BID19 to learn representations of chunks of text. Learning vector representation from graphs: Lot of research has gone into learning graph representations by translating the network/graph into a set of words or documents BID22 BID23 BID24 BID29 BID28 BID4. Generic models incorporating both edge weight and direction information for graph embeddings are proposed in BID39, BID28 and BID4. Specifically, node2vec BID4 advances the state-of-the-art in this area by designing flexible node sampling methodologies to allow feature vectors to exhibit different properties. Subgraph2vec BID20 extends these schemes to learn vector representations of subgraphs. BID26 proposes techniques to represent graph sequences under the assumption that each node is represented by a random binary vector. Learning graph representation with auxiliary information: Broadly speaking, our work falls into the category of node embedding in graphs with auxiliary information. BID5, BID33, BID7 and others address the case where nodes are associated with labels. BID21 studies graph embedding when node/edge attributes are continuous. BID1 investigates phrase ambiguity resolution via leveraging hyperlinks. However, all these works operate under information or network constraints. On the other hand, BID32, BID36 and explore embedding strategies in the context of knowledge graphs, where the main goal is to maintain the entity relationships specified by semantic edges. In contrast, we consider a simpler network setting where only nodes are associated with semantic information. EP BID3 and GraphSAGE BID6 learn embeddings for structured graph data. However, the textual similarities are only captured by linear combinations. Planetoid computes node embeddings under semi-supervised settings; metapath2vec BID2 learns embeddings for heterogeneous networks (node can be author or paper); and Graph-Neural-Network-based embeddings are explored in BID11 and BID17. However, these papers do not explicitly learn graph structure and text patterns simultaneously, and thus are complementary to SENSE. SNE BID16, SNEA, TADW BID34, HSCA BID37, AANE BID8, ANRL BID38 and PLANE BID13 are more related to our work. However, unlike SENSE, these do not consider the relative context of the words w.r.t. the document. Furthermore, the objective of PLANE is to maximize the likelihood that neighboring nodes have similar embeddings, which is not always the case in practice because neighboring nodes may be semantically different; more critically, it relies on strong assumptions of statistical distributions of words and edges in the network. In this regard, we propose a generic embedding scheme that jointly considers network topology as well as the nodes' semantic information. To embed a general node sequence, we first consider a special case where each node sequence contains only one node. Such single node embedding is referred to as SENSE-S, which jointly learns node representations along with textual descriptions in graphs. Let G = (V, E) denote a given directed or undirected graph, where V is the set of nodes and E the set of edges. Each node in V is associated with a text description. We aim to embed each node v in V into a feature vector that captures both graphical (neighboring node inter-connections) and textual (semantic meaning of v) properties. Specifically, let φ denote a word in the text description of node v. Suppose we obtain a set of neighboring nodes of v in graph G via a specific node sampling strategy, e.g., biased random walk BID4, and a set of neighboring words of φ in the text description of v by a sliding window over consecutive words BID19. We then define N G (v) as a probabilistic event of observing the set of neighboring nodes of v in G (under the chosen model) and N T (φ|v) as an event of observing the set of neighboring words of φ in the text description of v. DISPLAYFORM0 be the embedding function that maps each node v in V into a d-dimensional vector. Our goal is to find function f that maximizes: DISPLAYFORM1 Since events N G (v) and N T (φ|v) are independent, can be rewritten as: DISPLAYFORM2 where w v is the number of words in the description of v. Given a word in a node, jointly captures the node neighborhood in the graph and the word neighborhood in text. We build SENSE-S on the foundation of the skip-gram model BID19. In particular, BID4 and BID22 leverage this skip-gram model to learn vector representation of nodes in a graph by performing biased random walks on the graph and treating each walk as equivalent to a sentence, aiming to predict a neighboring node given the current node in a graph. On the other hand, BID12 extends skip-gram models to learn embeddings of various chunks of text, e.g., sentences, paragraphs or entire documents, via sampling the neighboring words over a sliding window within the text. Motivated by the effectiveness of these models, we build two SENSE-S models, SENSE-S (add) and SENSE-S (concat), as detailed below. Let w denote the number of words (some uninformative words are skipped) in text descriptions across all nodes in the given graph G, i.e., w is the size of the entire vocabulary, and n the number of nodes in G. Then our model is formulated as a neural network, as shown in Figure 1. In this model, each input is a two-tuple (φ, v), i.e., word φ is picked from the description of node v. Then (φ, v) is mapped to two one-hot vectors, w-dimensional word vector φ and n-dimensional vertex vector v, where only the entries corresponding to φ and v are set to 1 and others are set to 0. Then as in typical fully connected neural network architectures, in the first layer, φ T and v T are multiplied (implemented as look-up for efficiency) by two matrices M w×d and M n×d, respectively. The ing vectors are then added together and multiplied by another matrix M d×(w+n) in the second layer, i.e., a (w DISPLAYFORM0 is obtained. Finally, unlike typical skip-gram models where all entries in the output vectors of the second layer are processed by a softmax function, we decompose vector h into two sub-vectors, h 1 consisting of the first w entries and h 2 consisting of the rest. Then h 1 and h 2 are fed to separate softmax functions, yielding h 1 and h 2, respectively (see Figure 1). The reason for this decomposition operation is that we use h 1 to represent the probability vector of neighboring words of φ in the description of node v, and h 2 to represent the probability vector of neighboring nodes of v in the graph. Using this neural network architecture, we aim to learn the values of all entries in the matrices M w×d, M n×d and M d×(w+n) such that the entries (i.e., probabilities) in h 1 and h 2 corresponding to neighboring words of φ (within the text description of node v) and neighboring nodes of v (within the given graph) are maximized; see the objective in. We use matrix M n×d as our final semantically augmented node embeddings, i.e., each row in M n×d corresponds to a node embedding vector. Note that a unique property of SENSE-S is that it is a conjoined model where the textual and graphical inputs both contribute to the learning of node embeddings. Moreover, there is an add operation for combining these features, and thus this model is called SENSE-S (add). This is in contrast to the concatenation operation that is used in a different implementation of SENSE-S; see below. FIG3. Clearly, SENSE-S (concat) is quite similar to SENSE-S (add), except that (i) the ing vectors generated by the first layer are concatenated, and (ii) the matrix dimension in the second layer becomes (2d) × (w + n). In SENSE-S, our focus has been on computing semantically enhanced embeddings for individual nodes. In this section, we propose the general SENSE to represent any set of nodes following a specified order using the node vectors generated by SENSE-S, called node sequence embedding. Given the original graph G = (V, E), let S = v 1 → v 2 → · · · → v q be a node sequence constructed with v i ∈ V (i = 1, 2, . . ., q). Note that S may contain repeated nodes, e.g., some functions need to be executed more than once in one application. Intuitively, node sequence S can be represented by a d × q matrix with column i being the vector representation of node v i. However, such a representation is costly in space. Alternatively, representing S by the vector sum q i=1 v i (v i corresponds to v i) in missing the node order information. Hence, in this section, we seek to find a low-dimensional vector representation such that (i) node properties in the original network G and (ii) the node order in S are both preserved. In this section, all node vectors are unit vectors obtained by normalizing the node vectors generated by SENSE-S; this property is critical in our node sequence embedding method (see Section 4.2).Node Sequence Vector Construction: Given a node sequence S = v 1 → v 2 → · · · → v q following in order from node v 1 to node v q, let v i be the unit node vector of node v i. We first perform positional encoding, via cyclic shift function. Specifically, given vector v of dimension-d and non-negative integer m, we define v. Note that for repeated nodes in S, they are cyclically shifted by different positions depending on the specific order in the node sequence. In S, by imposing the restriction q d, we ensure that wraparounds do not occur while shifting the vector, because it may lead to ambiguity of node positions within a node sequence. Simple as this embedding approach may seem, we show that it exhibits the following advantages. First, the dimension of node sequence vectors remains the same as that of the original node vectors. Second, given a node sequence vector S, we are able to infer which nodes are involved in S and their exact positions in S as explained below. Node Sequence Vector Decoding: The method for determining which nodes are included (and in which order) in a given node sequence vector is referred to as node sequence vector decoding. The basic idea in node sequence vector decoding is rooted in Theorem 2 (Section 4.2), which implies that using cyclic shifting, we essentially enable a preferable property that v. By this property, we make the following claim, assuming that all node vectors are unit vectors. Claim 1. Given a node sequence vector S, node v, whose node vector is v, is at the k-th position of this node sequence if the inner product S · v (k−1) ≈ 1.Claim 1 provides an efficient way to decode a given node sequence. In particular, to determine whether (and where) a node resides in a node sequence, it only takes quadratic complexity O(d 2).Example: Suppose we have a node sequence S = 1 → 2 → 3 consisting of three nodes. By SENSE-S and our encoding method, we construct its node sequence vector as S = v1 +v2 +v3. Then each node can compute the inner product of its cyclic shifted vector with S. If the is approximately 1, then its position in this node sequence is uniquely determined. For instance, DISPLAYFORM0 2 ≈ 0 + 1 + 0 = 1 (see Theorem 2). Thus, given the encoded node sequence S, we know node 2 is at the second position. We now present our theoretical , to support Claim 1. Proofs are presented in the appendix. = 0 happens with high probability when the vector dimension is sufficiently large, although v a and v b are originally related. This section evaluates SENSE under varying scenarios, with multiple datasets and applications. We evaluate SENSE on the following datasets that contain graph information along with textual descriptions:: it contains both Wikipedia plain text articles (text descriptions) and hyper links between articles (graph structure). It is a directed graph with 4, 604 nodes (each is a Wikipedia article) and 119, 882 hyper links. Citation Network BID14: it contains both textual descriptions (title, authors and abstract of the paper) and graphical structure (citations). It is a directed graph with 27, 770 nodes (papers) and 352, 807 edges (citations). To train our SENSE-S model, we first define the loss function based on. Suppose the vocabulary size associated with each node v ∈ V is similar, i.e., we assume ∀v ∈ V, w v ≈ c (c is a constant). Then let DISPLAYFORM0 We define the loss function as DISPLAYFORM1 We then use stochastic gradient descent to minimize our loss function. Let η be the learning rate. At each iteration, we update the model parameters by adding a fraction of −∇L, i.e.,−η∇L = cη∇F G + η∇F T. Since the per-node vocabulary size is much larger than 1, the node neighborhood sampling via random walk is much less frequent than the textual neighborhood sampling. Therefore we want to inject more input data consisting of only the nodes' graphical neighborhood information. To do so, we adjust the model parameter update rule −η∇L as DISPLAYFORM2 where β 1 and β 2 are the equivalent learning rates for graph inputs and text inputs, respectively. We first evaluate SENSE-S, which is compared against the following baseline solutions. To compare with schemes that use graph information alone, we use node2vec since it has been shown to be flexible to capture different graph properties. To compare against schemes that use textual information alone, we use semantic vectors from paragraph2vec since it outperforms other schemes such as Recursive Neural Tensor Networks BID27 ) for tasks like Sentiment Analysis. As in SENSE-S, we also study two implementations of paragraph2vec, i.e., addition and concatenation operations at the hidden layer, referred to as paragraph2vec (add) and paragraph2vec (concat), respectively. For joint text/graph learning, we compare with the following: 1) Initialize with semantic vectors: We learn embeddings using node2vec, but rather than using random initialization, we initialze the vectors using paragraph2vec.2) Initialize with graphical vectors: Here, we learn final embeddings using paragraph2vec, but initialize them with node2vec, i.e., just reverse of the scheme above.3) Iterative Vectorization: The above approaches only leverage semantic or graphical vectors for initializations. Here, we try to capture both iteratively. Specifically, in one iteration, we compute node embedding via node2vec with the embeddings from the previous iteration as initializations; the corresponding are then fed to paragraph2vec as initializations to further compute node embeddings, after which we go to the next iteration. We repeat this process multiple times (5 times in our experiment) to get the final embeddings. Here, we simply concatenate the vectors obtained from paragraph2vec and node2vec and use them as our node embedding vectors. Test splits of data. Document length = first 500 characters (n2v: node2vec, p2v (add)/(cat): paragraph2vec (add)/(concat), n2v w/ p2v: node2vec with paragraph2vec (add) initialization, p2v w/ n2v: paragraph2vec (add) with node2vec initialization, IteraVec: Iterative Vectorization, cat(p2v,n2v): concatenation of p2v and n2v vectors).Experimental Setup: We first learn the vector representations and then use these vector representations for two different tasks: (i) Multi-label classification: Wikipedia pages are classified into different categories, such as history, science, people, etc. This ground truth information is included in the Wikispeedia dataset (which is not used while learning the vectors). There are 15 different top level categories, and our multi-label classification task tries to classify a page into one or more of these categories based on the vectors obtained from different algorithms. We train the OneVsRestClassifier (SVM, linear kernel) from scikit-learn for this task.(ii) Link prediction: Since no category information is available for the Citation Network, we evaluate for link prediction. In particular, 1% of existing citations are removed, after which vectors are learned on this network. We use these removed links as positive samples for link prediction. For negative samples, we randomly sample the same number of pairs which are not linked via a citation in the original network. To obtain the similarity features w.r.t. a pair of nodes, after experimenting with several alternatives, we chose the element-wise absolute difference and train SVM classifier (linear kernel) for link prediction. Parameter settings: (i) We perform κ random walks starting from each node in the graph (κ is 10 for Wikispeedia and 3 for Citation Network, since Citation Network is larger); (ii) each walk is of length 80 as recommended by BID4; (iii) we use sliding window of size 5 for neighboring word sampling; (iv) the default node vector dimension is 128; (v) multi-label classification error is the misclassification rate over the test set; (vi) link prediction error is the percentage of incorrect link predictions over all pairs of papers in the test set; (vii) learning rates β 1 and β 2 in are selected based on the validation set, and the error is reported on the test set. The error of multi-label classification is reported in FIG4, where the first 500 characters of each Wikipedia page are selected as its textual description. Third, among the schemes that utilize both textual and graphical information, SENSE-S (add) and SENSE-S (concat) consistently perform the best. This is because we train the network to co-learn the textual and graphical information to ensure that both objectives converge. This is in contrast with other schemes where the two objectives, due to the loose coupling, are not guaranteed to converge. Finally, SENSE-S (add) (in FIG4 (c)) outperforms node2vec by over 40%, paragraph2vec (add) by over 55% and the closest baseline scheme that leverages both text and graph by 30%. This confirms the benefit of co-learning features from textual as well as graphical information under the SENSE-S architecture. We also see similar trends using the first 1, 000 characters and omit the for brevity. The error of link prediction in the Citation Network is reported in FIG5. We fix train:valid:test to 60%:20%:20% (based on number of links) and use the first 500 characters as text description. We make several interesting observations. First, schemes that use graph information alone (node2vec) substantially outperform schemes that use text descriptions alone (paragraph2vec) for link prediction task. Intuitively, this is because the neighborhood information, which is important for link prediction task, is captured effectively by the node embeddings obtained from node2vec-like techniques. Second, even in cases where the difference in accuracy using the two different sources of information is large, SENSE-S (add) and (cat) are robust, and can effectively extract useful information from text descriptions to further reduce the error of link prediction that uses graph structure alone. Finally, this is significant because it demonstrates the effectiveness of SENSE-S in a variety of scenarios, including cases where the two sources of information may not be equally valuable. We now evaluate the accuracy of encoding/decoding by SENSE for node sequences. We evaluate on three experiments via constructing node sequences in the following different ways: Experiment 1: the node at every position in a sequence is chosen uniformly at random from 4, 604 Wikispeedia nodes. Note that as mentioned earlier, this node sequence may contain repeated nodes (which is allowed), and may or may not be a subgraph of the original graph. Experiment 2: the node sequences are constructed by performing random walks on the Wikispeedia graph. Experiment 3: the node sequences are constructed by performing random walks on the Physics Citation Network. Note that in both Experiments 2 and 3, adjacent nodes in the node sequences will have related vector embeddings. Next, for such constructed node sequences, their vector representations are computed by SENSE-S. Given these sequence vectors, we then decode the node at each position. We evaluate the decoding accuracy under different sequence lengths and vector dimensions, as reported in FIG6.From FIG6 (a), we make the following observations. First, when the node sequence length is small, all vector dimensions lead to almost 100% decoding accuracy. Second, as node sequence length increases, the decoding accuracy declines sharply especially in cases where the node vector dimensions are relatively small, i.e., 128 and 256. This is because, by Theorem 2, correlations between the involved node vectors cause inevitable errors. Such error accumulates when the length of the node sequence is large. Nevertheless, with sufficiently large node vector dimension, i.e., 1024, even long sequences can be decoded perfectly, and with 512, we can decode a workflow of length 10 with over 90% accuracy. Interestingly, FIG6 and (c) also show similar trends. This is significant, as it shows that even if node sequences are constructed from correlated node vectors, i.e., picked from same graph neighborhood, the decoding still achieves high accuracy. This is because, as shown in Theorem 2, after cyclic shifting, the ing vectors are orthogonal with high probability when the vector dimension is large (even if these vectors are originally related). Finally, in Figures 5 (a) and (b) the decoding algorithm needs to find the best match from among 4, 592 nodes (Wikispeedia network). In contrast, in FIG6 (c), it is much more challenging to find the match among 27, 770 nodes. Yet, we are able to decode with high accuracy with theoretical guarantees. We presented SENSE that learns semantically enriched vector representations of graph node sequences. To achieve this, we first developed SENSE-S that learns single node embeddings via a multi-task learning formulation that jointly learns the co-occurrence probabilities of nodes within a graph and words within a node-associated document. We evaluated SENSE-S against state-ofthe-art approaches that leverage both graph and text inputs and showed that SENSE-S improves multi-label classification accuracy in Wikispeedia dataset by up to 50% and link prediction over Physics Citation network by up to 78%. We then developed SENSE that is able to employ provable schemes for vector composition to represent node sequences using the same dimension as the individual node vectors from SENSE-S. We demonstrated that the individual nodes within the sequence can be inferred with a high accuracy (close to 100%) from such composite SENSE vectors. A LEMMA FOR THE PROOF OF THEOREM 2 Proof. Since both x and y are unit vectors, we have x · y = ||x||2||y||2 cos θ = cos θ, where θ is the angle between x and y. Since x and y are not correlated and both x and y are uniformly distributed across the sphere surface, θ is also uniformly distributed, and thus E[x · y] = 0.As x · y is purely determined by the angle θ between x and y, without loss of generality, we select y ="

We develop a new likelihood-free parameter estimation method that is equivalent to maximum likelihood under some conditions

represent the short summary for finding the corresponding scientific article (long)

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent

Question: While most approaches to the problem of Inverse Reinforcement Learning (IRL) focus on estimating a reward function that best explains an expert agent’s policy or demonstrated behavior on a control task, it is often the case that such behavior is more succinctly represented by a simple reward combined with a set of hard constraints. In this setting, the agent is attempting to maximize cumulative rewards subject to these given constraints on their behavior. We reformulate the problem of IRL on Markov Decision Processes (MDPs) such that, given a nominal model of the environment and a nominal reward function, we seek to estimate state, action, and feature constraints in the environment that motivate an agent’s behavior. Our approach is based on the Maximum Entropy IRL framework, which allows us to reason about the likelihood of an expert agent’s demonstrations given our knowledge of an MDP. Using our method, we can infer which constraints can be added to the MDP to most increase the likelihood of observing these demonstrations. We present an algorithm which iteratively infers the Maximum Likelihood Constraint to best explain observed behavior, and we evaluate its efficacy using both simulated behavior and recorded data of humans navigating around an obstacle. Advances in mechanical design and artificial intelligence continue to expand the horizons of robotic applications. In these new domains, it can be difficult to design a specific robot behavior by hand. Even manually specifying a task for a reinforcement-learning-enabled agent is notoriously difficult . Inverse Reinforcement Learning (IRL) techniques can help alleviate this burden by automatically identifying the objectives driving certain behavior. Since first being introduced as Inverse Optimal Control by , much of the work on IRL has focused on learning environmental rewards to represent the task of interest (; ; ;). While these types of IRL algorithms have proven useful in a variety of situations (; ; ;, their basis in assuming that reward functions fully represent task specifications makes them ill suited to problem domains with hard constraints or non-Markovian objectives. Recent work has attempted to address these pitfalls by using demonstrations to learn a rich class of possible specifications that can represent a task . Others have focused specifically on learning constraints, that is, behaviors that are expressly forbidden or infeasible (; Pérez-; ; ;). Such constraints arise in safety-critical systems, where requirements such as an autonomous vehicle avoiding collisions with pedestrians are more naturally expressed as hard constraints than as soft reward penalties. It is towards the problem of inferring such constraints that we turn our attention. In this work, we present a novel method for inferring constraints, drawing primarily from the Maximum Entropy approach to IRL described by. We use this framework to reason about the likelihood of observing a set of demonstrations given a nominal task description, as well as about their likelihood if we imposed additional constraints on the task. This knowledge allows us to select a constraint, or set of constraints, which maximizes the demonstrations' likelihood and best explains the differences between expected and demonstrated behavior. Our method improves on prior work by being able to simultaneously consider constraints on states, actions and features in a Markov Decision Process (MDP) to provide a principled ranking of all options according to their effect on demonstration likelihood. A formulation of the IRL problem was first proposed by as the Inverse problem of Optimal Control (IOC). Given a dynamical system and a control law, the author sought to identify which function(s) the control law was designed to optimize. This problem was brought into the domain of MDPs and Reinforcement Learning (RL) by , who proposed IRL as the task of, given an MDP and a policy (or trajectories sampled according to that policy), find a reward function with respect to which that policy is optimal. One of the chief difficulties in the problem of IRL is the fact that a policy can be optimal with respect to a potentially infinite set of reward functions. The most trivial example of this is the fact that all policies are optimal with respect to a null reward function that always returns zero. Much of the subsequent work in IRL has been devoted to developing approaches that address this ambiguity by imposing additional structure to make the problem well-posed . approach the problem by employing the principle of maximum entropy , which allows the authors to develop an IRL algorithm that produces a single stochastic policy that matches feature counts without adding any additional constraints to the produced behavior. This so called Maximum Entropy IRL (MaxEnt) provides a framework for reasoning about demonstrations from experts who are noisily optimal. The induced probability distribution over trajectories forms the basis for our efforts in identifying the most likely behavior-modifying constraints. While Markovian rewards do often provide a succinct and expressive way to specify the objectives of a task, they cannot capture all possible task specifications. highlight the utility of non-Markovian Boolean specifications which can describe complex objectives (e.g. do this before that) and compose in an intuitive way (e.g. avoid obstacles and reach the goal). The authors of that work draw inspiration from the MaxEnt framework to develop their technique for using demonstrations to calculate the posterior probability that an agent is attempting to satisfy a Boolean specification. A subset of these types of specifications that is of particular interest to us is the specification of constraints, which are states, actions, or features of the environment that must be avoided. explore how to infer trajectory feature constraints given a nominal model of the environment (lacking the full set of constraints) and a set of demonstrated trajectories. The core of their approach is to sample from the set of trajectories which have better performance than the demonstrated trajectories. They then infer that the set of possible constraints is the subset of the feature space that contains the higher-reward sampled trajectories, but not the demonstrated trajectories. Intuitively, they reason that if the demonstrator could have passed through those features to earn a higher reward, but did not, then there must have been a previously unknown constraint preventing that behavior. However, while their approach does allow for a cost function to rank elements from the set of possible constraints, the authors do not offer a mechanism for determining what cost function will best order these constraints. Our approach to constraint inference from demonstrations addresses this open question by providing a principled ranking of the likelihood of constraints. We adapt the MaxEnt framework to allow us to reason about how adding a constraint will affect the likelihood of demonstrated behaviors, and we can then select the constraints which maximize this likelihood. We consider feature-space constraints as in , and we explicitly augment the feature space with state-and action-specific features to directly compare the impacts of state-, action-, and feature-based constraints on demonstration likelihood. Following the formulation presented by , we base our work in the setting of a (finite-state) Markov Decision Process (MDP). We define an MDP M as a tuple (S, {A s}, {P s,a}, D 0, φ, R) where S is a finite set of discrete states; {A s} is a set of the sets of actions available to be taken for each state s, such that A s ⊆ A, where A is a finite set of discrete actions; {P s,a} is a set of state transition probability distributions such that P s,a (s) = P (s |s, a) is the probability of transitioning to state s after taking action a from state s; D 0: S → is an initial state distribution; φ: S × A → R k + is a mapping to a k-dimensional space of non-negative features; and R: S × A → R is a reward function. A trajectory ξ through this MDP is a sequence of states s t and actions a t such that s 0 ∼ D 0 and state s i+1 ∼ P si,ai. Actions are chosen by an agent navigating the MDP according to a, potentially time-varying, policy π such that π(·|s, t) is a probability distribution over actions in A s. We denote a finite-time trajectory of length T + 1 by ξ = {s 0:T, a 0:T}. At every time step t, a trajectory will accumulate features equal to φ(s t, a t). We use the notation φ i (·, ·) to refer to the i-th element of the feature map, and we use the label φ i to denote the i-th feature itself. We also introduce an augmented indicator feature mapping φ 1: S × A → {0, 1} n φ, where n φ = k + |S| + |A|. This augmented feature map uses binary variables to indicate the presence of a feature and expands the feature space by adding binary features to track occurrences of each state and action, such that Typically, agents are modeled as trying to maximize, or approximately maximize, the total reward earned for a trajectory ξ, given by R(ξ) = T t=0 γ t R(s t, a t), where γ ∈ is a discount factor. Therefore, an agent's policy π is closely tied to the form of the MDP's reward function. Conventional IRL focuses on inferring a reward function that explains an agent's policy, revealed through the behavior observed in a set of demonstrated trajectories D. However, our method for constraint inference poses a different challenge: given an MDP M, including a reward function, and a set of demonstrations D, find the most likely set of constraints C * that modify M. We define our notion of constraints in the following section. Constraints are those behaviors that are not disallowed explicitly by the structure of the MDP, but which would be infeasible or prohibited for the underlying system being modeled by the MDP. This sort of discrepancy can occur when a generic or simplified MDP is designed without exact knowledge of specific constraints for the modeled system. For instance, for a generic MDP modeling the behavior of cars, we might want to include states for speeds up to 500km/h and actions for accelerations up to 12m/s 2. However, for a specific car on a specific roadway, the set of states where the vehicle travels above 100km/h may be prohibited because of a speed limit, and the set of actions where the vehicle accelerates above 4m/s 2 may be infeasible because of the physical limitations of the vehicle's engine. Therefore, any MDP trajectory of this specific car system would not contain a state-action pair which violates these legal and physical limits. Figure 1 shows an example of constraints driving behavior. We define a constraint set C i ⊆ S × A as a set of state-action pairs that violate some specification of the modeled system. We consider three general classes of constraints: state constraints, action constraints, and feature constraints. A state constraint set C si = {(s, a) | s = s i } includes all state-action pairs such that the state component is s i. An action constraint set C ai = {(s, a) | a = a i } includes all state-action pairs such that the action component is a i. A feature constraint set C φi = {(s, a) | φ i (s, a) > 0} includes all state-action pairs that produce a non-zero value for feature φ i. If we augment the set of features as described in, it is straightforward to see that state and action constraints become special cases of feature constraints, where It is also evident that we can obtain compound constraints, respecting two or more conditions, by taking the union of constraint sets C i to obtain C = i C i. We need to be able to reason about how adding a constraint to an MDP will influence the behavior of agents navigating that environment. If we impose a constraint on an MDP, then none of the state-action pairs in that constraint set may appear in a trajectory of the constrained MDP. To enforce this condition, we must restrict the actions available in each state so that it is not possible for an agent to produce one of the constrained state-action pairs. For a given constraint C, we can replace the set of available actions A s in every state s with an alternative set A Performing such substitutions for an MDP M will lead to a modified MDP M C such that. The question then arises as to the how we should treat states with empty action sets A C s = ∅. Since an agent arriving in such an empty state would have no valid action to select, any trajectory visiting an empty state must be deemed invalid. Indeed, such empty action sets will be produced for any state For MDPs with deterministic transitions, agents know precisely which state they will arrive in following a certain action. Therefore, any agent respecting constraints will not take an action that leads to an empty state, since doing so will lead to constraint violations. If we consider the set of empty states S empty, then for the purposes of reasoning about an agent's behavior, we can impose an additional constraint set C empty = {(s, a) | ∃ s empty ∈ S empty: P s,a (s empty) = 1}. In this work, we will always implicitly add this constraint set, such that M C will be equivalent to M C∪Cempty, and we recursively add these constraints until reaching a fixed point. For MDPs with stochastic transitions, the semantics of an empty state are less obvious and could lend themselves to multiple interpretations depending on the nature of the system being modeled. We offer a possible treatment in the appendix. The nominal MDPs to which we will add constraints fall into two broad categories, which we denote as generic and baseline. The car MDP described at the beginning of Section 3.2 is an example of a generic nominal model: its state and action spaces are broad enough to encompass a wide range of car models, and we can use this nominal model to infer constraints that specialize the MDP to a specific car and task. For a generic nominal MDP, the reward function may also come from a generic, simplified task, such as "minimize time to the goal" or "minimize energy usage." A baseline nominal MDP is a snapshot of a system from a point in time where it was well characterized. With a baseline nominal MDP, the constraints that we infer will represent changes to the system with respect to this baseline. In this case, the nominal reward function can be learned using existing IRL techniques with demonstrated behavior from the baseline model. We take this approach in our human obstacle avoidance example in Section 4.2: we use demonstrations of humans walking through the empty space to learn a nominal reward, then we can detect the presence of a new obstacle in the space from subsequent demonstrations. Our goal is to find the constraints C * which are most likely to have been added to a nominal MDP M, given a set of demonstrations D from an agent navigating the constrained MDP. Let us define P M to denote probabilities given that we are considering MDP M. Our problem then becomes to select the constraints that maximize P M (C | D). If we assume a uniform prior over possible constraints, then we know from Bayes' Rule that P M (C | D) ∝ P M (D | C). Therefore, in order to find the constraints that maximize P M (C | D), we can solve the equivalent problem of finding which constraints maximize the likelihood of the given demonstrations. In this section, we present our approach to solving maximum likelihood constraint inference via solving demonstration likelihood maximization. Under the maximum entropy model presented by , the probability of a certain finite-length trajectory ξ being executed by an agent traversing a deterministic MDP M is exponentially proportional to the reward earned by that trajectory. where Z is the partition function, 1 M (ξ) indicates if the trajectory is feasible for this MDP, and β ∈ [0, ∞) is a parameter describing how closely an agent adheres to the task of optimizing the reward function (as β → ∞, the agent becomes a perfect optimizer, and as β → 0, the agent's actions become perfectly random). In the sequel, we assume that a given reward function will appropriately capture the role of β, so we omit β from our notation without loss of generality. In the case of finite horizon planning, the partition function will be the sum of the exponentially weighted rewards for all feasible trajectories on MDP M of length no greater than the planning horizon. We denote this set of trajectories by Ξ M. Because adding constraints C modifies the set of feasible trajectories, we express this dependence as Assuming independence among demonstrated trajectories, the probability of observing a set D of N demonstrations is given by the product Our goal is to maximize the demonstration probability given by. Because we take the reward function and demonstrations as given, our only available decision variable in this maximization is the constraint set C which alters the indicator 1 M C and partition function Z(C). where C ⊆ 2 S×A is the hypothesis space of possible constraints. From the form of, it is clear that to solve, we must choose a constraint set that does not invalidate any demonstrated trajectory while simultaneously minimizing the value of Z(C). Consider the set of trajectories that would be made infeasible by augmenting the MDP with constraint C, which we denote as Ξ The value of Z(C) is minimized when we maximize the sum of exponentiated rewards of these infeasible trajectories. Considering the form of the trajectory probability given by, we can see that this sum is proportional to the total probability of observing a trajectory from This insight leads us to the final form of the optimization In order to solve, we must reason about the probability distribution of trajectories on the original MDP M, then find the constraint C such that Ξ − M C contains the most probability mass while not containing any demonstrated trajectories. We highlight here that the fact that the chosen C must not conflict with any demonstration is an important condition: all provided demonstrations must perfectly respect a constraint in order for it to be learned, otherwise a less restrictive set of constraints may be learned instead. Future work will look to relax this requirement in order to learn about constraints that are generally respected by a set of demonstrations, without needing to first isolate just the successful, constraint-respecting demonstrations. While equation is derived for deterministic MDPs, if we can assume, as proposed by , that for a given stochastic MDP, the stochastic outcomes have little effect on an agent's behavior and the partition function, then the solution to will also approximate the optimal constraint selection for that MDP. However, in order to fully address the stochastic case, we would need to reformulate our approach based on maximum causal entropy . We save this extension for future work. In order for the solutions to to be meaningful, we must be careful with our choice of the constraint hypothesis space C. For instance, if we let C = 2 S×A, then the optimal solution will always be to choose the most restrictive C to constrain all state-action pairs not observed in the demonstration set. One approach to avoid this trivial solution is to use domain knowledge of the modeled system to restrict C to a library of plausible or common constraints. construct such a library by using reachability theory to calculate a family of likely unsafe sets. Another approach, which we will explore in this work, is the use of minimal constraint sets for our hypothesis space. These minimal sets constrain a single state, action, or feature, and were introduced in Section 3.2 as C si, C ai, and C φi, respectively. By iteratively selecting minimal constraint sets, it is possible to gradually grow the full estimated constraint set and avoid over-fitting to the demonstrations. Section 3.4 details our approach for selecting the most likely minimal constraint, and Section 3.5 details our approach for iteratively growing the estimated constraint set. As detailed in Section 3.3, the most likely constraint set is the one whose eliminated trajectories Ξ − M C have the highest probability of being demonstrated on the original, unconstrained MDP. Therefore, to find the most likely of the minimal constraints, we must find the expected proportion of trajectories which will contain any state or action, or accrue any feature. By using our augmented indicator feature map from, we can reduce this problem to only examine feature accruals. present their forward-backward algorithm for calculating expected feature counts for an agent following a policy in the maximum entropy setting. This algorithm nearly suffices for our purposes, but it computes the expectation of the total number of times a feature will be accrued (i.e. how often will this feature be observed per trajectory), rather than the expectation of the number of trajectories that will accrue that feature at any point. To address this problem, we present a modified form of the "forward" pass as Algorithm 1. Our algorithm tracks state visitations as well as feature accruals at each state, which allows us to produce the same maximum entropy distribution over trajectories as while not counting additional accruals for trajectories that have already accrued a feature. The input of Algorithm 1 includes the MDP itself, a time horizon, and a time-varying policy. This policy should capture the expected behavior of the demonstrator on the nominal MDP M, and can be computed via the "backward" part of the algorithm from. The output of Algorithm 1, Φ [1,T], is an n φ × T array such that the t-th column Φ t is a vector whose i-th entry is the expected proportion of trajectories to have accrued the i-th feature by time t. In particular, the i-th element of Φ T is equal to P M (Ξ − M C i), which allows us to now directly select the most likely constraint according to. When using minimal constraint sets as the constraint hypothesis space, it is possible that the most likely constraint still does not provide a satisfactory explanation for the demonstrated behavior. In this case, it can be beneficial to combine minimal constraints. Input: MDP M, constraint hypothesis space C, empirical probability distribution is framed as finding the combination of constraint sets that "covers" the most probability mass, then the problem becomes a direct analog for the classic maximum coverage problem. While this problem is known to be NP-hard, there exist a simple greedy algorithm with known suboptimality bounds . We present Algorithm 2 as our approach for adapting this greedy heuristic to solve the problem of constraint inference. At each iteration, we grow our estimated constraint set by augmenting it with the constraint set in our hypothesis space that covers the most currently uncovered probability mass. By analog to the maximum coverage problem, we derive the following bound on the suboptimality of our approach. Theorem 1. Let C nc be the set of all constraints C nc such that C nc = nc i=1 C i for C i ∈ C, and let C * nc be the solution to using C nc as the constraint hypothesis space. It follows, then, that at the end of every iteration i of Algorithm 2, This bound is directly analogous to the suboptimality bound for the greedy solution to the maximum coverage problem proven by . For space, the proof is included in the appendix. Rather than selecting the number of constraints n c to be used ahead of time, we check a condition based on KL divergence to decide if we should continue to add constraints. The quantity D KL P D || P M C * provides a measure of how well the distribution over trajectories induced by our inferred constraints, P M C *, agrees with the empirical probability distribution over trajectories observed in the demonstrations, P D. The threshold parameter d DKL is chosen to avoid over-fitting to the demonstrations, combating the tendency to select additional constraints that may only marginally better align our predictions with the demonstrations. We consider the grid world MDP presented in Figure 2. The environment consists of a 9-by-9 grid of states, and the actions are to move up, down, left, right, or diagonally by one cell. The objective is to move from the starting state in the bottom-left corner (s 0) to the goal state in the bottom-right corner (s G). Every state-action pair produces a distance feature, and the MDP reward is negative distance, which encourages short trajectories. There are additionally two more features, denoted green and blue, which are produced by taking actions from certain states, as shown in Figure 2. The true MDP, from which agents generate trajectories, is shown in Figure 2a, including its constraints. The nominal, more generic MDP shown in Figure 2b is what we take as M for applying the iterative maximum likelihood constraint inference in Algorithm 2, with feature accruals estimated using Algorithm 1. While Figures 2c through 2e show the iteratively estimated constraints, which align with the true constraints, it is interesting to note that not all constraints present in the true MDP are identified. For instance, it is so unlikely that an agent would ever select the up-left diagonal action, that the fact that demonstrated trajectories did not contain that action is unsurprising and does not make that action an estimated constraint. Figure 3 shows how the performance of our approach varies based on the number of available demonstrations and the selection for the threshold d DKL. The false positive rate shown in Figure 3a is the proportion of selected constraints which are not constraints of the true system. We can observe two trends in this data that we would expect. First, lower values of d DKL lead to greater false positive rates since they allow Algorithm 2 to continue iterating and accept constraints that do less to align expectations and demonstrations. Second, having more demonstrations available provides more information and reduces the rate of false positives. Further, Figure 3b shows that more demonstrations also allows the behavior predicted by constraints to better align with the observations. It is interesting to note, however, that with fewer than 10 demonstrations and a very low d DKL, we may produce very low KL divergence, but at the cost of a high false positive rate. This phenomenon highlights the role of selecting d DKL to avoid over-fitting. The threshold d DKL = 0.1 achieves a good balance of producing few false positives with sufficient examples while also producing lower KL divergences, and we used this threshold to produce the in Figures The shaded region represents an obstacle in the human's environment, and the red "X"s represent learned constraints. In our second example, we analyze trajectories from humans as they navigate around an obstacle on the floor. We map these continuous trajectories into trajectories through a grid world where each cell represents a 1ft-by-1ft area on the ground. The human agents are attempting to reach a fixed goal state (s G) from a given initial state (s 0), as shown in Figure 4. We performed MaxEnt IRL on human demonstrations of the task without the obstacle to obtain the nominal distance-based reward function. We restrict ourselves to estimating only state constraints, as we do not supply our algorithm with knowledge of any additional features in the environment and we assume that the humans' motion is unrestrained. Demonstrations were collected from 16 volunteers, and the of performing constraint inference are shown in Figure 4. Our method is able to successfully predict the existence of a central obstacle. While we do not estimate every constrained state, the constraints that we do estimate make all of the obstacle states unlikely to be visited. In order to identify those states as additional constraints, we would have to decrease our d DKL threshold, which could also lead to more spurious constraint selections, such as the three shown in Figure 4. We have presented our novel technique for learning constraints from demonstrations. We improve upon previous work in constraint-learning IRL by providing a principled framework for identifying the most likely constraint(s), and we do so in a way that explicitly makes state, action, and feature constraints all directly comparable to one another. We believe that the numerical presented in Section 4 are promising and highlight the usefulness of our approach. Despite its benefits, one drawback of our approach is that the formulation is based on, which only exactly holds for deterministic MDPs. As mentioned in Section 3.3, we plan to investigate the use of a maximum causal entropy approach to address this issue and fully handle stochastic MDPs. Additionally, the methods presented here require all demonstrations to contain no violations of the constraints we will estimate. We believe that softening this requirement, which would allow reasoning about the likelihood of constraints that are occasionally violated in the demonstration set, may be beneficial in cases where trajectory data is collected without explicit labels of success or failure. Finally, the structure of Algorithm 1, which tracks the expected features accruals of trajectories over time, suggests that we may be able to reason about non-Markovian constraints by using this historical information to our advantage. Overall, we believe that our formulation of maximum likelihood constraint inference for IRL shows promising and presents attractive avenues for further investigation. For MDPs with non-deterministic transitions, the semantics of an empty state are less obvious and could lend themselves to multiple interpretations depending on the nature of the system being modeled. In our context, we use constraints to describe how observed behaviors from demonstrations differ from possible behaviors allowed by the nominal MDP structure. We therefore assume that any demonstrations provided are, by the fact that they were selected to be provided, consistent with the system's constraints, including avoiding empty states. This assumption implies that any stochastic state transitions that would have led to an empty state will not be observed in trajectories from the demonstration set. The omission of these transitions means that, for a given (s, a), if P s,a (S empty) = p, then a proportion p of these (s, a) pairs which occur as an agent navigates the environment will be excluded from demonstrations. Therefore, as we modify the MDP to reason about demonstrated behavior, we need updated transition probabilities which eliminate the probability mass of transitioning to empty states, an event which will never be observed in a demonstration. Such modified probabilities can be given as We must also capture the change to observed state-action pair frequencies by understanding that any observed policy π C will be related to an agent's actual policy π according to π C (a|s, t) = π(a|s, t)(1 − P s,a (S empty)) a ∈A C s π(a |s, t)(1 − P s,a (S empty)). It is important to note that the modifications presented in and for non-deterministic MDPs are not meant to directly reflect the reality of the underlying system (we wouldn't expect the actual transition dynamics to change, for instance), but to reflect the apparent behavior that we would expect to observe in the subset of trajectories that would be selected as demonstrations. We further note that applying these modifications to deterministic MDPs will in the same expected behavior as augmenting the constraint set with C empty. Theorem 2. Let C nc be the set of all constraints C nc such that C nc = nc i=1 C i for C i ∈ C, and let C * nc be the solution to using C nc as the constraint hypothesis space. It follows, then, that at the end of every iteration i of Algorithm 2, Proof. The problem of finding C * nc is analogous to solving the maximum coverage problem, where the set of elements to be covered is the set of trajectories {ξ | ∃ C ∈ C : ξ ∈ Ξ − M C and D ∩ Ξ − M C = ∅} and the weight of each element ξ is P M (ξ). Because Algorithm 2 constructs C * iteratively by taking the union of the previous value of C * and the set C i ∈ C which solves, the value of C * at the end of the i-th iteration is analogous to the greedy solution of the maximum coverage problem with n c = i. Therefore, we can directly apply the suboptimality bound for the greedy solution proven in to arrive at our given bound on eliminated probability mass.

Represent the TLDR for finding the corresponding scientific article

Our method infers constraints on task execution by leveraging the principle of maximum entropy to quantify how demonstrations differ from expected, un-constrained behavior.

Represent the short summary for finding the corresponding scientific article (long)!

We introduce the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended to policy gradient methods.