Question: Represent the article for finding a single-sentence TLDR

Question: We introduce our Distribution Regression Network (DRN) which performs regression from input probability distributions to output probability distributions. Compared to existing methods, DRN learns with fewer model parameters and easily extends to multiple input and multiple output distributions. On synthetic and real-world datasets, DRN performs similarly or better than the state-of-the-art. Furthermore, DRN generalizes the conventional multilayer perceptron (MLP). In the framework of MLP, each node encodes a real number, whereas in DRN, each node encodes a probability distribution. The field of regression analysis is largely established with methods ranging from linear least squares to multilayer perceptrons. However, the scope of the regression is mostly limited to real valued inputs and outputs BID4 BID14. In this paper, we perform distribution-todistribution regression where one regresses from input probability distributions to output probability distributions. Distribution-to-distribution regression (see work by BID17) has not been as widely studied compared to the related task of functional regression BID3. Nevertheless, regression on distributions has many relevant applications. In the study of human populations, probability distributions capture the collective characteristics of the people. Potential applications include predicting voting outcomes of demographic groups BID5 and predicting economic growth from income distribution BID19. In particular, distribution-to-distribution regression is very useful in predicting future outcomes of phenomena driven by stochastic processes. For instance, the Ornstein-Uhlenbeck process, which exhibits a mean-reverting random walk, has wide-ranging applications. In the commodity market, prices exhibit mean-reverting patterns due to market forces BID23. It is also used in quantitative biology to model phenotypic traits evolution BID0.Variants of the distribution regression task have been explored in literature BID18. For the distribution-to-distribution regression task, BID17 proposed an instance-based learning method where a linear smoother estimator (LSE) is applied across the inputoutput distributions. However, the computation time of LSE scales badly with the size of the dataset. To that end, BID16 developed the Triple-Basis Estimator (3BE) where the prediction time is independent of the number of data by using basis representations of distributions and Random Kitchen Sink basis functions. BID9 proposed the Extrapolating the Distribution Dynamics (EDD) method which predicts the future state of a time-varying probability distribution given a sequence of samples from previous time steps. However, it is unclear how it can be used for the general case of regressing distributions of different objects. Our proposed Distribution Regression Network (DRN) is based on a completely different scheme of network learning, motivated by spin models in statistical physics and similar to artificial neural networks. In many variants of the artificial neural network, the network encodes real values in the nodes BID21 BID10 BID1. DRN is novel in that it generalizes the conventional multilayer perceptron (MLP) by encoding a probability distribution in each node. Each distribution in DRN is treated as a single object which is then processed by the connecting weights. Hence, the propagation behavior in DRN is much richer, enabling DRN to represent distribution regression mappings with fewer parameters than MLP. We experimentally demonstrate that compared to existing methods, DRN achieves comparable or better regression performance with fewer model parameters. Figure 1: (Left) An example DRN with multiple input probability distributions and multiple hidden layers mapping to an output probability distribution. (Right) A connection unit in the network, with 3 input nodes in layer l − 1 connecting to a node in layer l. Each node encodes a probability distribution, as illustrated by the probability density function P (l) k. The tunable parameters are the connecting weights and the bias parameters at the output node. and Y i are univariate continuous distributions with compact support, the regression task is to learn the function f which maps the input distributions to the output distribution. DISPLAYFORM0 No further assumptions are made on the form of the distribution. It is trivial to generalize our method to regress to multiple output distributions but for simplicity of explanation we shall restrict to single output regressions in the following discussions. Fig. 1 illustrates how the regression in Eq. is realized. DRN generalizes the traditional neural network structure by encoding each node with a probability distribution and connecting the nodes with real-valued weights. The input data consists of one or more probability distributions which are fed into the first layer and propagated layerwise through the hidden layers. We emphasize our network is not a Bayesian network even though each node encodes a probability. Unlike bayes net where the conditional probability among variables are learnt by maximizing the likelihood over observed data, DRN regresses probability distributions using a feedforward network, similar to MLP. At each node in the hidden layer, the probability distribution is computed from the probability distributions of the incoming nodes in the previous layer and the network parameters consisting of the weights and bias parameters (see right of Fig. 1). Pk represents the probability density function (pdf) of the k th node in the l th layer and P DISPLAYFORM0 k ) is the density of the pdf when the node variable is s DISPLAYFORM1 Before obtaining the probability distribution P (l) k, we first compute its unnormalized formP DISPLAYFORM2 k is computed by marginalizing over the product of the unnormalized conditional probabilitỹ Q(s DISPLAYFORM3) and the incoming node probabilities. DISPLAYFORM4 represent the variables of the lower layer nodes and E is the energy given a set of node variables, which we define later in Eq.. The unnormalized conditional probability has the same form as the Boltzmann distribution in statistical mechanics, except that the partition function is omitted. This omission reduces the computational complexity of our model through factorization, shown later in Eq..Our energy function formulation is motivated by work on spin models in statistical physics where spin alignment to coupling fields and critical phenomena are studied BID12 BID8 BID27. Energy functions are also used in other network models where a scalar energy is associated to each configuration of the nodes BID24 BID11. In such energybased models, the parameters are learnt such that the observed configurations of the variables have lower energies than unobserved ones. However, the energy function used in DRN is part of the forward propagation process and is not directly optimized. For a given set of node variables, the energy function is DISPLAYFORM5 ki is the weight connecting the i th node in the lower layer to the upper layer node. b respectively. The support length of the distribution is given by ∆. All terms in Eq. are normalized by the support length so that the energy function is invariant with respect to the support. Eq. can be factorized such that instead of having multidimensional integrals, there are n univariate integrals: DISPLAYFORM6 k ) captures the bias terms of the energy function in Eq.. DISPLAYFORM7 Finally, the probability distribution from Eq. is normalized. DISPLAYFORM8 The propagation of probability distributions within a connection unit forms the basis for forward propagation. Forward propagation is performed layerwise from the input layer using Eq. to. The forward propagation in DRN has some important properties. Fig. 2 illustrates the propagation behavior for a connection unit with one input node where the bias values b DISPLAYFORM0 a,k and b DISPLAYFORM1 q,k are set as zero. DISPLAYFORM2 Figure 2: Propagation behavior for a connection unit with one input node. The biases are set as zero in these examples. When weight is zero, the output distribution is flat. Positive weights causes the output distribution to have the same peak position as the input distribution while negative weights causes the output pdf to'repel' away from the input peak. When the weight is a sufficiently large positive number, the propagation tends towards the identity mapping. When the weight is zero, the output distribution is flat and the output distribution is independent of the input. With a positive weight, the output distribution is'attracted' to the peak of the input distribution whereas a negative weight causes the output distribution to be'repelled' away from the input peak. In addition, the weight magnitude represents the strength of the'attraction' or'repulsion'. When the weight is a sufficiently large positive number, the propagation tends towards the identity mapping (top right example in Fig. 2). The implication is that like in neural networks, a deeper network should have at least the same complexity as a shallow one, as the added layers can produce the identity function. Conversely, a small positive weight causes the output peak to be at the same position as the input peak, but with more spread (second example on left column of Fig. 2).The remaining absolute and quadratic bias terms in Eq. have a similar role as the bias in a traditional neural network. Depending on the bias values b q,k respectively. The weight and bias values play a similar role as the inverse temperature in the Boltzmann distribution in statistical physics BID12 BID27. The cost function of the network given a set network parameters is measured by the Jensen-Shannon (JS) divergence between the label (Y i) and predicted DISPLAYFORM0 and D KL is the Kullback-Liebler divergence. The Jensen-Shannon divergence is a suitable cost function as it is symmetric and bounded. The network cost function C net is the average D JS over all M training data: DISPLAYFORM1 In our experiments, the integrals in Eq. and are performed numerically. This is done through discretization from continuous probability density functions (pdf) to discrete probability mass functions (pmf). Given a continuous pdf with finite support, the range of the continuous variable is partitioned into q equal widths and the probability distribution is binned into the q states. The estimation error arising from the discretization step will decrease with larger q. The network cost is a differentiable function over the network parameters. We derive the cost gradients similar to backpropagation in neural networks BID22. We use chain rule to derive at each node a q-by-q matrix which denotes the derivative of the final layer node distribution with respect to the current node distribution. DISPLAYFORM0 where DISPLAYFORM1 is the final layer output probability distribution. From the derivative DISPLAYFORM2 We evaluate DRN on synthetic and real-world datasets and compare its performance to the state-ofthe-art 3BE method and a fully-connected multilayer perceptron (MLP). For each of the datasets, DRN achieves similar or higher accuracy with fewer model parameters. In MLP, each discretized probability mass function is represented by q nodes. The MLP consists of fully connected hidden layers with ReLU units and a softmax final layer, and is optimized with mean squared error using Adam. Unlike DRN and MLP where the distribution pdfs are directly used by the methods, 3BE assumes the input and output distributions are observed through i.i.d. samples. Hence, for the first two datasets we provide 3BE with sufficient samples from the underlying distribution such that errors from density estimation are minimal. The first experiment involves a synthetic dataset similar to the one used by BID17 DISPLAYFORM0 The function h transforms the means and standard deviations using the non-linear function shown in Fig. 3a. The transformation is such that the two gaussian means will remain in their respective ranges. The sample input-output data pairs in Fig. 3b shows the complexity of the regression task with various behavior like peak splitting and peak spreading. 1000 training data and 1000 testing data were created to evaluate the regression methods. For DRN and MLP, the pdfs are discretized into q = 100 states and for 3BE, 10,000 samples from each data distribution are generated. While 3BE gives a continuous distribution as the output, DRN and MLP output the discrete pmf and require conversion to continuous pdf. Following BID18, the regression performance on the test set is measured by the L2 loss between the continuous predicted distribution,Ŷ (s) and the true distribution. We study how the regression accuracy varies with respect to the number of model parameters. For DRN and MLP, the number of parameters are varied using different depths and widths of the networks and for 3BE, we vary the number of Random Kitchen Sink features. We present the detailed DRN architecture in Appendix B. Fig. 4a shows the L2 loss on the test set as we vary the number of model parameters. Note that the x-axis is presented on the log scale. DRN's test performance is comparable to the other methods and uses fewer model parameters to attain reasonable performance. We note there is little overfitting for the three methods, as shown in the plots comparing train and test loss in Fig. 4b, though 3BE starts to exhibit overfitting when the number of model parameters approaches 10,000. Because of the Boltzmann distribution term (ref. Eq. 3), DRN models the diffusion process very well. For this experiment, we evaluate our model on data generated from the stochastic OrnsteinUhlenbeck (OU) process BID25 which combines the notion of random walk with a drift towards a long-term mean. The OU process has wide-ranging applications. In the commodity market, prices exhibit mean-reverting patterns due to market forces and hence modelling the prices with the OU process helps form valuation strategies BID23 BID28.The OU process is described by a time-varying gaussian pdf. With the long-term mean set at zero, the pdf has a mean of µ(t) = y exp(−θt) and variance of σ 2 (t) = DISPLAYFORM0. t represents time, y is the initial point mass position, and D and θ are the diffusion and drift coefficients respectively. The regression task is to map from an initial gaussian distribution at t init to the ing distribution after some time step ∆t. The gaussian distributions are truncated with support of. With different sampled values for y ∈ [0.3, 0.9] and t init ∈ [0.01, 2], pairs of distributions are created for ∆t = 1, D = 0.003 and θ = 0.1. For DRN and MLP, q = 100 was used for discretization of the pdfs while 10,000 samples were taken for each distribution to train 3BE. We compare the number of model parameters required to achieve a small L2 test loss with 100 training data. We also increased the training size to 1000 and attained similar . TAB0 and FIG5 show that a simple DRN of one input node connecting to one output node with 5 parameters performs similarly as MLP and 3BE. MLP requires 1 fully-connected hidden layer with 3 nodes, with a total of 703 network parameters. 3BE requires 64 projection coefficients for both input and output distributions and 17 Random Kitchen Sink features, ing in 272 model parameters. The regression by DRN on two random test samples are shown in FIG5 and we see that DRN is able to demonstrate the OU process. FIG5 shows the 5 DRN parameters after training. The values of these parameters are interpreted as follows. The weight parameter is positive, hence the output peak position is positively correlated to the input peak position. Moreover, w = 75.3 is such that the network mimics the diffusion property of the OU process. The bias position λ a is negative and its magnitude is 5 times the distribution support, causing the output peak to be displaced leftwards of the input peak. These two observations reflect the random walk and mean-reverting properties of the OU process. Figure 6: Single-layer network used in DRN for the stock dataset with 7 model parameters (3 weights, 4 bias parameters). We demonstrate that DRN can be useful for an important real-world problem and outperforms 3BE and MLP in terms of prediction accuracy. With greater integration of the global stock markets, there is significant co-movement of stock indices BID6 BID2. In a study by BID26, it was found that the previous day stock returns of the Nikkei and Dow Jones Industrial Average (Dow) are good predictors of the FTSE return. Modelling the co-movement of global stock indices has its value as it facilitates investment decisions. Stock indices are weighted average of the constituent companies' prices in a stock exchange, and existing research has primarily focused on the movement of returns of the indices. However, for our experiment, we predict the future distribution of returns over the constituent companies in the index as it provides more information than just a weighted average. Our regression task is as follows. Given the current day's distribution of returns of constituent companies in FTSE, Dow and Nikkei, predict the distribution of returns for constituent companies in FTSE k days later. The logarithmic return for the company's stock at day t is given by ln(V t /V t−1), where V t and V t−1 represent its closing price at day t and t − 1 respectively. The stock data consists of 9 years of daily returns from January 2007 to December 2015. To adapt to changing market conditions, we use a sliding-window training scheme where the data is split into windows of training, validation and test sets and moved foward in time BID7. A new window is created and the network is retrained after every 300 days (which is the size of test set). For each test set, the previous 500 and 100 days were used for training and validation. To reduce the noise in the data, we performed exponential window averaging on the price series for each stock with a window of 50 days following common practice BID15. The logarithmic returns of the constituent company stocks form the samples for the distributions of the returns. For DRN and MLP, the pdf is estimated using kernel density estimation with a gaussian kernel function with bandwidth of 0.001 and q = 100 was used for discretization of the pdf. The authors of 3BE have extended their method for multiple input functions (see joint motion prediction experiment in BID16). We followed their method and concatenated the basis coefficients obtained from the three input distributions. In addition, for 3BE we scale the return samples to before applying cosine basis projection. The predicted distribution is then scaled back to the original range for quantification of the regression performance. First, we performed evaluations for the task of predicting the next-day distributions. As we do not have the underlying true pdf for this real-world dataset, the regression performance is measured by the log-likelihood of the test samples. TAB1 shows the test log-likelihoods, where higher loglikelihood is favorable. Interestingly, the single-layer network in DRN (see Fig. 6) was sufficient to perform well, using just 7 network parameters. In comparison, MLP and 3BE require 4110 and 8100 parameters respectively. To visualize the regression on the test set, we compare for each day the first two moments (mean and variance) of the predicted distribution and the ground truth (see 1-day ahead panels of FIG6). Each point represents one test data and we show the Pearson correlation coefficients between the predicted and labelled moments. DRN has the best regression performance as the points lie closest to the diagonal line where the predicted and labelled moments are equal, and its correlation values are highest. As an extension, we predict the FTSE returns distribution several days ahead. The second and third rows of FIG6 and FIG6 show the moment plots for 5 and 10 days ahead respectively. Expectedly, the performance deterioriates as the number of days increases. Still, DRN outperforms the rest as shown by the moment plots and the correlation values. FIG7 summarizes the by showing the average absolute error of the mean and variance as the number of days-ahead increases. For all experiments, DRN consistently has the lowest error. Finally, we conducted experiments on a real-world cell dataset similar to the one used in BID17. The dataset is a time-series of images of NIH3T3 fibroblast cells. There are 277 time frames taken at 5-minute intervals, containing 176 to 222 cells each. In each frame, we measured the long and short-axis nuclear length of the cells and scaled the lengths to. At each time-frame, given the distribution of long-axis length, we predict the distribution of the short-axis length. The first 200 frames were used for training and last 77 for testing. For DRN and MLP, the pdf is estimated using kernel density estimation with a gaussian kernel of bandwidth 0.02 and q = 100 was used for discretization. We compare the log-likelihood on test data in TAB3. DRN had the best loglikelihood with a simple network of one input node connecting to one output node. In contrast, MLP and 3BE used more model parameters but achieved lower log-likelihoods. This validated DRN's advantage at learning distribution regressions on real-world data with fewer model parameters. The distribution-to-distribution regression task has many useful applications ranging from population studies to stock market prediction. In this paper, we propose our Distribution Regression Network which generalizes the MLP framework by encoding a probability distribution in each node. Our DRN is able to learn the regression mappings with fewer model parameters compared to MLP and 3BE. MLP has not been used for distribution-to-distribution regression in literature and we have adapted it for this task. Though both DRN and MLP are network-based methods, they encode the distribution very differently. By generalizing each node to encode a distribution, each distribution in DRN is treated as a single object which is then processed by the connecting weight. Thus, the propagation behavior in DRN is much richer, enabling DRN to represent the regression mappings with fewer parameters. In 3BE, the number of model parameters scales linearly with the number of projection coefficients of the distributions and number of Random Kitchen Sink features. In our experiments, DRN is able to achieve similar or better regression performance using less parameters than 3BE. Furthermore, the runtime for DRN is competitive with other methods (see comparison of mean prediction times in Appendix C).For future work, we look to extend DRN for variants of the distribution regression task such as distribution-to-real regression and distribution classification. Extensions may also be made for regressing multivariate distributions. In this section, show the DRN network architecture used for the synthetic dataset presented in Fig. 4a. There is one input node and one output node connected by a number of hidden layers of arbitrary width. All layers are fully-connected. We compare the mean prediction time per data for DRN and the baseline methods. All runs were conducted on the CPU. For the synthetic dataset, we have shown the test loss for varying parameter sizes. For a fair comparison of runtime, for each method we chose a model size which gave a test L2 loss of about 0.37. For all the datasets, MLP has the fastest prediction time, followed by DRN and then 3BE.

A learning network which generalizes the MLP framework to perform distribution-to-distribution regression

represent this text to find a much much longer article about it with say hundreds-thousands of words

For instance, <<A new approach that learns a representation for describing transition models in complex uncertaindomains using relational rules. >>  to "We present a representation for describing transition models in complex uncertain domains using relational rules. For any action, a rule selects a set of relevant objects and computes a distribution over properties of just those objects in the ing state given their properties in the previous state. An iterative greedy algorithm is used to construct a set of deictic references that determine which objects are relevant in any given state. Feed-forward neural networks are used to learn the transition distribution on the relevant objects' properties. This strategy is demonstrated to be both more versatile and more sample efficient than learning a monolithic transition model in a simulated domain in which a robot pushes stacks of objects on a cluttered table. Many complex domains are appropriately described in terms of sets of objects, properties of those objects, and relations among them. We are interested in the problem of taking actions to change the state of such complex systems, in order to achieve some objective. To do this, we require a transition model, which describes the system state that from taking a particular action, given the previous system state. In many important domains, ranging from interacting with physical objects to managing the operations of an airline, actions have localized effects: they may change the state of the object(s) being directly operated on, as well as some objects that are related to those objects in important ways, but will generally not affect the vast majority of other objects. In this paper, we present a strategy for learning state-transition models that embodies these assumptions. We structure our model in terms of rules, each of which only depends on and affects the properties and relations among a small number of objects in the domain, and only very few of which may apply for characterizing the effects of any given action. Our primary focus is on learning the kernel of a rule: that is, the set of objects that it depends on and affects. At a moderate level of abstraction, most actions taken by an intentional system are inherently directly parametrized by at least one object that is being operated on: a robot pushes a block, an airport management system reschedules a flight, an automated assistant commits to a venue for a meeting. It is clear that properties of these "direct" objects are likely to be relevant to predicting the action's effects and that some properties of these objects will be changed. But how can we characterize which other objects, out of all the objects in a household or airline network, are relevant for prediction or likely to be affected?To do so, we make use of the notion of a deictic reference. In linguistics, a deictic (literally meaning "pointing") reference, is a way of naming an object in terms of its relationship to the current situation rather than in global terms. So, "the object I am pushing," "all the objects on the table nearest me," and "the object on top of the object I am pushing" are all deictic references. This style of reference was introduced as a representation strategy for AI systems by BID0, under the name indexical-functional representations, for the purpose of compactly describing policies for a video-game agent, and has been in occasional use since then. We will learn a set of deictic references, for each rule, that characterize, relative to the object(s) being operated on, which other objects are relevant. Given this set of relevant objects, the problem of describing the transition model on a large, variable-size domain, reduces to describing a transition model on fixed-length vectors characterizing the relevant objects and their properties and relations, which we represent and learn using standard feed-forward neural networks. Next, we briefly survey related work, describe the problem more formally, and then provide an algorithm for learning both the structure, in terms of deictic references, and parameters, in terms of neural networks, of a sparse relational transition model. We go on to demonstrate this algorithm in a simulated robot-manipulation domain in which the robot pushes objects on a cluttered table. Rule learning has a long history in artificial intelligence. The novelty in our approach is the combination of learning discrete structures with flexible parametrized models in the form of neural networks. Rule learning We are inspired by very early work on rule learning by BID4, which sought to find predictive rules in simple noisy domains, using Boolean combinations of binary input features to predict the effects of actions. This approach has a modern re-interpretation in the form of schema networks BID5. The rules we learn are lifted, in the sense that they can be applied to objects, generally, and are not tied to specific bits or objects in the input representation and probabilistic, in the sense that they make a distributional prediction about the outcome. In these senses, this work is similar to that of and methods that build on it (BID9, BID8, BID6 .) In addition, the approach of learning to use deictic expressions was inspired by Pasula et al. and used also by BID7 in the form of object-oriented reinforcement learning and by BID2. BID2, however, relies on a full description of the states in ground first-order logic and does not have a mechanism to introduce new deictic references to the action model. Our representation and learning algorithm improves on the Pasula et al. strategy by using the power of feed-forward neural networks as a local transition model, which allows us to address domains with real-valued properties and much more complex dependencies. In addition, our EM-based learning algorithm presents a much smoother space in which to optimize, making the overall learning faster and more robust. We do not, however, construct new functional terms during learning; that would be an avenue for future work for us. Graph network models There has recently been a great deal of work on learning graph-structured (neural) network models BID1. There is a way in which our rule-based structure could be interpreted as a kind of graph network, although it is fairly non-standard. We can understand each object as being a node in the network, and the deictic functions as being labeled directed hyper-edges (between sets of objects). Unlike the typical graph network models, we do not condition on a fixed set of neighboring nodes and edges to compute the next value of a node; in fact, a focus of our learning method is to determine which neighbors (and neighbors of neighbors, etc.) to condition on, depending on the current state of the edge labels. This means that the relevant neighborhood structure of any node changes dynamically over time, as the state of the system changes. This style of graph network is not inherently better or worse than others: it makes a different set of assumptions (including a strong default that most objects do not change state on any given step and the dynamic nature of the neighborhoods) which are particularly appropriate for modeling an agent's interactions with a complex environment using actions that have relatively local effects. We assume we are working on a class of problems in which the domain is appropriately described in terms of objects. This method might not be appropriate for a single high-dimensional system in which the transition model is not sparse or factorable, or can be factored along different lines (such as a spatial decomposition) rather than along the lines of objects and properties. We also assume a set of primitive actions defined in terms of control programs that can be executed to make actual changes in the world state and then return. These might be robot motor skills (grasping or pushing an object) or virtual actions (placing an order or announcing a meeting). In this section, we formalize this class of problems, define a new rule structure for specifying probabilistic transition models for these problems, and articulate an objective function for estimating these models from data. A problem domain is given by tuple D = (Υ, P, F, A) where Υ is a countably infinite universe of possible objects, P is a finite set of properties P i: Υ → R, i ∈ [N P] = {1, · · ·, N P}, and F is a finite set of deictic reference functions DISPLAYFORM0 where ℘ (Υ) denotes the powerset of Υ. Each function F i ∈ Υ maps from an ordered list of objects to a set of objects, and we define it as DISPLAYFORM1 where the relation f i: Υ mi+1 → {True, False} is defined in terms of the object properties in P. For example, if we have a location property P loc and m i = 1, we can define f i (o, o 1) = 1 Ploc(o)−Ploc(o1) <0.5 so that the function F i associated with f i maps from one object to the set of objects that are within 0.5 distance of its center; here 1 is an indicator function. Finally, A is a set of action templates DISPLAYFORM2 where Ψ is the space of executable control programs. Each action template is a function parameterized by continuous parameters α i ∈ R di and a tuple of n i objects that the action operates on. In this work, we assume that P, F and A are given. A problem instance is characterized by I = (D, U), where D is a domain defined above and U ⊂ Υ is a finite universe of objects with |U| = N U. For simplicity, we assume that, for a particular instance, the universe of objects remains constant over time. In the problem instance I, we characterize a state s in terms of the concrete values of all properties in P on all objects in U; that is, DISPLAYFORM0 A problem instance induces the definition of its action space A, constructed by applying every action template A i ∈ A to all tuples of n i elements in U and all assignments α i to the continuous parameters; namely, DISPLAYFORM1 In many domains, there is substantial uncertainty, and the key to robust behavior is the ability to model this uncertainty and make plans that respect it. A sparse relational transition model (SPARE) for a domain D, when applied to a problem instance I for that domain, defines a probability density function on the ing state s ing from taking action a in state s. Our objective is to specify this function in terms of domain elements P, R, and F in such a way that it will apply to any problem instance, independent of the number and properties of the objects in its universe. We achieve this by defining the transition model in terms of a set of transition rules, T = {T k} K k=1 and a score function C: T × S → N. The score function takes in as input a state s and a rule T ∈ T, and outputs a non-negative integer. If the output is 0, the rule does not apply; otherwise, the rule can predict the distribution of the next state to be p(s | s, a; T). The final prediction of SPARE is DISPLAYFORM0 whereT = arg max T ∈T C(T, s) and the matrix DISPLAYFORM1 ) is the default predicted covariance for any state that is not predicted to change, so that our problem is well-formed in the presence of noise in the input. Here I N U is an identity matrix of size N U, and diag ([σ DISPLAYFORM2 represents a square diagonal matrix with σ i on the main diagonal, denoting the default variance for property P i if no rule applies. Note that the transition rules will be learned from past experience with a loss function specified in Section 3.3. In the rest of this section, we formalize the definition of transition rules and the score function. Transition rule T = (A, Γ, ∆, φ θ, v default) is characterized by an action template A, two ordered lists of deictic references Γ and ∆ of size N Γ and N ∆, a predictor φ θ and the default variances DISPLAYFORM3 for each property P i under this rule. The action template is defined as operating on a tuple of n object variables, which we will refer to as DISPLAYFORM4 A reference list uses functions to designate a list of additional objects or sets of objects, by making deictic references DISPLAYFORM5 Figure 2: Instead of directly mapping from current state s to next state s, our prediction model uses deictic references to find subsets of objects for prediction. In the left most graph, we illustrate what relations are used to construct the input objects with two rules for the same action template, DISPLAYFORM6 default ), where the reference list DISPLAYFORM7 1 to the target object o 2 and added input features computed by an aggregator g on o 3, o 6 to the inputs of the predictor of rule T 1. Similarly for DISPLAYFORM8, the first deictic reference selected o 3 and then γ2 is applied on o 3 to get o 1. The predictors φ θ are neural networks that map the fixed-length input to a fixed-length output, which is applied to a set of objects computed from a relational graph on all the objects, derived from the reference list DISPLAYFORM9, to compute the whole next state s. Because δ DISPLAYFORM10 θ is only predicting a single property, we use a "de-aggregator" function h to assign its prediction to both objects o 4, o 6. based on previously designated objects. In particular, Γ generates a list of objects whose properties affect the prediction made by the transition rule, while ∆ generates a list of objects whose properties are affected after taking an action specified by the action template A.We begin with the simple case in which every function returns a single object, then extend our definition to the case of sets. Concretely, for the t-th element DISPLAYFORM11 where F ∈ F is a deictic reference function in the domain, m is the arity of that function, and integer k j ∈ [n+t−1] specifies that object O n+t in the object list can be determined by applying function F to objects (O kj) m j=1. Thus, we get a new list of objects, DISPLAYFORM12. So, reference γ 1 can only refer to the objects (O i) n i=1 that are named in the action, and determines an object O n+1. Then, reference γ 2 can refer to objects named in the action or those that were determined by reference γ 1, and so on. When the function DISPLAYFORM13 ) ∈ Γ returns a set of objects rather than a single object, this process of adding more objects remains almost the same, except that the O t may denote sets of objects, and the functions that are applied to them must be able to operate on sets. In the case that a function F returns a set, it must also specify an aggregator, g, that can return a single value for each property P i ∈ P, aggregated over the set. Examples of aggregators include the mean or maximum values or possibly simply the cardinality of the set. For example, consider the case of pushing the bottom (block A) of a stack of 4 blocks, depicted in FIG1. Suppose the deictic reference is F =above, which takes one object and returns a set of objects immediately on top of the input object. Then, by applying F =above starting from the initial set O 0 = {A}, we get an ordered list of sets of objects DISPLAYFORM14 Returning to the definition of a transition rule, we now can see informally that if the parameters of action template A are instantiated to actual objects in a problem instance, then Γ and ∆ can be used to determine lists of input and output objects (or sets of objects). We can use these lists, finally, to construct input and output vectors. The input vector x consists of the continuous action parameters α of action A and property P i (O t) for all properties P i ∈ P and objects O t ∈ O NΓ that are selected by Γ in arbitrary but fixed order. In the case that O t is a set of size greater than one, the aggregator associated with the function F that computed the reference is used to compute P i (O t). Similar for the desired output construction, we use the references in the list ∆, initializeÔ = O, and gradually add more objects to construct the output set of objectsÔ =Ô (N∆). The output vector is y = [P (ô)]ô ∈Ô,P ∈P where ifô is a set of objects, we apply a mean aggregator on the properties of all the objects inô. The predictor φ θ is some functional form φ (such as a feed-forward neural network) with parameters (weights) θ that will take values x as input and predict a distribution for the output vector y. It is difficult to represent arbitrarily complex distributions over output values. In this work, we restrict ourselves to representing a Gaussian distributions on all property values in y, encoded with a mean and independent variance for each dimension. Now, we describe how a transition rule can be used to map a state and action into a distribution over the new state. A transition rule T = (A, Γ, ∆, φ θ, v default) applies to a particular state-action (s, a) pair if a is an instance of A and if none of the elements of the input or output object lists is empty. To construct the input (and output) list, we begin by assigning the actual objects o 1,..., o n to the object variables O 1,..., O n in action instance a, and then successively computing references γ i ∈ Γ based on the previously selected objects, applying the definition of the deictic reference F in each γ i to the actual values of the properties as specified in the state s. If, at any point, a γ i ∈ Γ or δ i ∈ ∆ returns an empty set, then the transition rule does not apply. If the rule does apply, and successfully selects input and output object lists, then the values of the input vector x can be extracted from s, and predictions are made on the mean and variance values Pr(DISPLAYFORM15 θ2 (x) be the vector entry corresponding to the predicted Gaussian parameters of property P i of j-th output object setô j and denote s[o, P i] as the property P i of object o in state s, for all o ∈ U. The predicted distribution of the ing state p(s | s, a; T) is computed as follows: DISPLAYFORM16 where v i ∈ v default is the default variance of property P i in rule T. There are two important points to note. First, it is possible for the same object to appear in the object-list more than once, and therefore for more than one predicted distribution to appear for its properties in the output vector. In this case, we use the mixture of all the predicted distributions with uniform weights. Second, when an element of the output object list is a set, then we treat this as predicting the same single property distribution for all elements of that set. This strategy has sufficed for our current set of examples, but an alternative choice would be to make the predicted values be changes to the current property value, rather than new absolute values. Then, for example, moving all of the objects on top of a tray could easily specify a change to each of their poses. We illustrate how we can use transition rules to build a SPARE in Fig. 2.For each transition rule T k ∈ T and state s ∈ S, we assign the score function value to be 0 if T k does not apply to state s. Otherwise, we assign the total number of deictic references plus one, N Γ + N ∆ + 1, as the score. The more references there are in a rule that is applicable to the state, the more detailed the match is between the rules conditions and the state, and the more specific the predictions we expect it to be able to make. We frame the problem of learning a transition model from data in terms of conditional likelihood. The learning problem is, given a problem domain description D and a set of experience E tuples, DISPLAYFORM0, find a SPARE T that minimizes the loss function: DISPLAYFORM1 Note that we require all of the tuples in E to belong to the same domain D, and require for any (DISPLAYFORM2 and s (i) belong to the same problem instance, but individual tuples may be drawn from different problem instances (with, for example, different numbers and types of objects). In fact, to get good generalization performance, it will be important to vary these aspects across training instances. We describe our learning algorithm in three parts. First, we introduce our strategy for learning φ θ, which predicts a Gaussian distribution on y, given x. Then, we describe our algorithm for learning reference lists Γ and ∆ for a single transition rule, which enable the extraction of x and y from E. Finally, we present an EM method for learning multiple rules. For a particular transition rule T with associated action template A, once Γ and ∆ have been specified, we can extract input and output features x and y from a given set of experience samples E. We would like to learn the transition rule's predictor φ θ to minimize Eq.. Our predictor takes the form φ θ (x) = N (µ θ (x), Σ θ (x)) and a neural network is used to predict both the mean µ θ (x) and the diagonal variance Σ θ (x). We directly optimize the negative data-likelihood loss function DISPLAYFORM0 Let E T ∈ E be the set of experience tuples to which rule T applies. Then once we have θ, we can optimize the default variance of the rule DISPLAYFORM1 It can be shown that these loss-minimizing values for the default predicted variances v default are the empirical averages of the squared deviations for all unpredicted objects (i.e., those for which φ θ does not explicitly make predictions), where averages are computed separately for each object property. We use θ, v default ← LEARNDIST(D, E, Γ, ∆) to refer to this learning and optimization procedure for the predictor parameters and default variance. Algorithm 1 Greedy procedure for constructing Γ. DISPLAYFORM0 train model using Γ 0 = ∅, save loss L 0 i ← 1 4: DISPLAYFORM0 for all γ ∈ R i do 7: DISPLAYFORM1 else breakIn the simple setting where only one transition rule T exists in our domain D, we show how to construct the input and output reference lists Γ and ∆ that will determine the vectors x and y. Suppose for now that ∆ and v default are fixed, and we wish to learn Γ. Our approach is to incrementally build up Γ by adding DISPLAYFORM2 ) tuples one at a time via a greedy selection procedure. Specifically, let R i be the universe of possible γ i, split the experience samples E into a training set E train and a validation set E val, and initialize the list Γ to be Γ 0 = ∅. For each i, compute γ i = arg min γ∈Ri L(T γ ; D, E val), where L in Eq. evaluates a SPARE T γ with a single transition rule T = (A, Γ i−1 ∪ {γ}, ∆, φ θ, v default ), where θ and v default are computed using the LEARNDIST described in Section 4.1 2. If the value of the loss function L(T γi ; D, E val) is less than the value of L(T γi−1 ; D, E val), then we let Γ i = Γ i−1 ∪{γ i} and continue. Else, we terminate the greedy selection process with Γ = Γ i−1, since further growing the list of deictic references hurts the loss. We also terminate the process when i exceeds some predetermined maximum allowed number of input deictic references, N Γ. Pseudocode for this algorithm is provided in Algorithm 1.In our experiments we set ∆ = Γ and construct the lists of deictic references using a single pass of the greedy algorithm described above. This simplification is reasonable, as the set of objects that are relevant to predicting the transition outcome often overlap substantially with the objects that are affected by the action. Alternatively, we could learn ∆ via an analogous greedy procedure nested around or, as a more efficient approach, interleaved with, the one for learning Γ. Our training data in robotic manipulation tasks are likely to be best described by many rules instead of a single one, since different combinations of relations among objects could be present in different states. For example, we may have one rule for pushing a single object and another rule for pushing a stack of objects. We now address the case where we wish to learn K rules from a single experience set E, for K > 1. We do so via initial clustering to separate experience samples into K clusters, one for each rule to be learned, followed by an EM-like approach to further separate samples and simultaneously learn rule parameters. To facilitate the learning of our model, we will additionally learn membership probabilities Z = ((z i,j) DISPLAYFORM0, where z i,j represents the probability that the i-th experience sample is assigned to transition rule T j, and DISPLAYFORM1. We initialize membership probabilities via clustering, then refine them through EM.Because the experience samples E may come from different problem instances and involve different numbers of objects, we cannot directly run a clustering algorithm such as k-means on the (s, a, s) samples themselves. Instead we first learn a single transition rule T = (A, Γ, ∆, φ θ, v default) from E using the algorithm in Section 4.2, use the ing Γ and ∆ to transform E into x and y, and then run k-means clustering on the concatenation of x, y, and values of the loss function when T is used to predict each of the samples. For each experience sample, the squared distance from the sample to each of the K cluster centers is computed, and membership probabilities for the sample to each of the K transition rules to be learned are initialized to be proportional to the (multiplicative) inverses of these squared distances. Before introducing the EM-like algorithm that simultaneously improves the assignment of experience samples to transition rules and learns details of the rules themselves, we make a minor modification to transition rules to obtain mixture rules. Whereas a probabilistic transition rule has been defined as T = (A, Γ, ∆, φ θ, v default), a mixture rule is T = (A, π Γ, π ∆, Φ), where π Γ represents a distribution over all possible lists of input references Γ (and similarly for π ∆ and ∆), of which there are a finite number, since the set of available reference functions F is finite, and there is an upper bound N Γ on the maximum number of references Γ may contain. For simplicity of terminology, we refer to each possible list of references Γ as a shell, so π Γ is a distribution over possible shells. DISPLAYFORM2 is a collection of κ transition rules (i.e., predictors φ θ (k), each with an associated Γ (k), ∆ (k), and v (k) default ). To make predictions for a sample (s, a) using a mixture rule, predictions from each of the mixture rule's κ transition rules are combined according to the probabilities that π Γ and π ∆ assign to each transition rule's Γ (k) and ∆ (k). Rather than having our EM approach learn K transition rules, we instead learn K mixture rules, as the distributions π Γ and π ∆ allow for smoother sorting of experience samples into clusters corresponding to the different rules, in contrast to the discrete Γ and ∆ of regular transition rules. As before, we focus on the case where for each mixture rule, DISPLAYFORM3, and π Γ = π ∆ as well. Our EM-like algorithm is then as follows:1. For each j ∈ [K], initialize distributions π Γ = π ∆ for mixture rule T j as follows. First, use the algorithm in Section 4.2 to learn a transition rule on the weighted experience samples E Zj with weights equal to the membership probabilities DISPLAYFORM4. In the process of greedily assembling reference lists Γ = ∆, data likelihood loss function values are computed for multiple explored shells, in addition to the shell Γ = ∆ that was ultimately selected. Initialize π Γ = π ∆ to distribute weight proportionally, ac- DISPLAYFORM5, with the summation taken over all explored shells Γ, is a normalization factor so that the total weight assigned by π Γ to explored shells is 1−. The remaining probability weight is distributed uniformly across unexplored shells., where we have dropped subscripting according to j for notational simplicity: DISPLAYFORM0 default ) using the procedure in Section 4.2 on the weighted experience samples E Zj, where we choose DISPLAYFORM1 to be the list of references with k-th highest weight according to π Γ = π ∆. (b) Update π Γ = π ∆ by redistributing weight among the top κ shells according to a voting procedure where each training sample "votes" for the shell whose predictor minimizes the validation loss for that sample. In other words, the i-th experience sample DISPLAYFORM2 ). Then, shell weights are assigned to be proportional to the sum of the sample weights (i.e., membership probability of belonging to this rule) of samples that voted for each particular shell: the number of votes received by the k-th shell is V (k) = |E| i=1 1 v(i)=k · z i,j, for indicator function 1 and k ∈ [κ]. Then, π Γ (k), the current k-th highest value of π Γ, is updated to become V (k)/ξ, where ξ is a normalization factor to ensure that π Γ remains a valid probability distribution. (Specifically, ξ = ( DISPLAYFORM3 Step 2a, in case the κ shells with highest π Γ values have changed, in preparation for using the mixture rule to make predictions in the next step.3. Update membership probabilities by scaling by data likelihoods from using each of the K rules to make predictions: DISPLAYFORM4 is the data likelihood from using mixture rule T j to make predictions for the i-th experience sample E (i), and DISPLAYFORM5 ) is a normalization factor to maintain K j=1 z i,j = 1. 4. Repeat Steps 2 and 3 some fixed number of times, or until convergence. We apply our approach, SPARE, to a challenging problem of predicting pushing stacks of blocks on a cluttered table top. We describe our domain, the baseline that we compare to and report our . In our domain D = (Υ, P, F, A), the object universe Υ is composed of blocks of different sizes and weight, the property set P includes shapes of the blocks (width, length, height) and the position of the block ((x, y, z) location relative to the table). We have one action template, push(α, o), which pushes toward a target object o with parameters α = (x g, y g, z g, d), where (x g, y g, z g) is the 3D position of the gripper before the push starts and d is the distance of the push. The orientation of the gripper and the direction of the push are computed from the gripper location and the target object location. We simulate this 3D domain using the physically realistic PyBullet (simulator. In real-world scenarios, an action cannot be executed with the exact action parameters due to the inaccuracy in the motor and hence in our simulation, we add Gaussian noise on the action parameters during execution to imitate this effect. We consider the following deictic references in the reference collection F: identity (O), which takes in an object O and returns O; above (O), which takes in an object O and returns the object immediately above O; below (O), which takes in an object O and returns the object immediately below O; nearest (O), which takes in an object O and returns the object that is closest to O. Neural network (NN) We compare to a neural network function approximator that takes in as input the current state s ∈ R N P ×N U and action parameter α ∈ R N A, and outputs the next state s ∈ R N P ×N U. The list of objects that appear in each state is ordered: the target objects appear first and the remaining objects are sorted by their poses (first sort by x coordinate, then y, then z).Graph NN We compare to a fully connected graph NN. Each node of the graph corresponds to an object in the scene, and the action α is concatenated to the state of each object. Bidirectional edges connect every node in the graph. The graph NN consists of encoders for the nodes and edges, propagation networks for message passing, and a node decoder to convert back to predict the mean and variance of the next state of each object. As a sanity check, we start from a simple problem where a gripper is pushing a stack of three blocks with two extra blocks on the table. We randomly sampled 1250 problem instances by drawing random block shapes and locations from a uniform distribution within a range while satisfying the condition that the stack of blocks is stable and the extra blocks do not affect the push. In each problem instance, we uniformly randomly sample the action parameters and obtain the training data, a collection of tuples of state, action and next state, where the target object of the push action is always the one at the bottom of the stack. We held out 20% of the training data as the validation set. We found that our approach is able to reliably select the correct combinations of the references that select all the blocks in the problem instance to construct inputs and outputs. In FIG3, we show how the performance varies as deictic references are added during a typical run of this experiment. The solid purple curves show training performance, as measured by data likelihood on the validation set, while the dashed purple curve shows performance on a held-out test set with 250 unseen problem instances. As expected, performance improves noticeably from the addition of the first two deictic references selected by the greedy selection procedure, but not from the 4th. The brown curve shows the learned default standard deviations, used to compute data likelihoods for features of objects not explicitly predicted by the rule. As expected, the learned default standard deviation drops as deictic references are added, until it levels off after the third reference is added since at that point the set of references captures all moving objects in the scene. Sensitivity analysis on the number of objects We compare our approach to the baselines in terms of how sensitive the performance is to the number of objects that exist in the problem instance. We continue the setting where a stack of three blocks lie on a table, with extra blocks that may affect the prediction of the next state. FIG3 shows the performance, as measured by the log data likelihood, as a function of the number of extra blocks. For each number of extra blocks, we used 1250 training problem instances with 20% as the validation set and 250 testing problem instances. When there are no extra blocks, SPARE learns a single rule whose x and y contain the same information as the inputs and outputs for the baselines. As more objects are added to the table, NN's performance drops as the presence of these additional objects appear to complicate the scene and NN is forced to consider more objects when making its predictions. SPARE outperforms graph NN, as the good predictions for the extra blocks contribute to the log data likelihood. Note that, performance aside, NN is limited to problems for which the number of objects in the scenes is fixed, as the it requires a fixed-size input vector containing information about all objects. Our SPARE approach does not have this limitation, and could have been trained on a single, large dataset that is the combination of the datasets with varying numbers of extra objects. However, we did not do this in our experiments for the sake of providing a more fair comparison against NN.Sample efficiency We evaluate our approach on more challenging problem instances where the robot gripper is pushing blocks on a cluttered table top and there are two additional blocks on the table that do not interfere or get affected by the pushing action. FIG3 (c) plots the data likelihood as a function of the number of training samples. We evaluate with training samples varying from 100 to 1250 and in each setting, the test dataset has 250 samples. Both our approach and the baselines benefit from having more training samples, but our approach is much more sample efficient and achieves good performance within only 500 training samples. Learning multiple transition rules Now we put our approach in a more general setting where multiple transition rules need to be learned for prediction of the next state. Our approach adopts an EM-like procedure to assign each training sample its distribution on the transition rules and learn each transition rule with re-weighted training samples. First, we construct a training dataset and 70% of it is on pushing 4-block stack. Our EM approach is able to concentrate to the 4-block case as shown in FIG4. The three curves correspond to the three stack heights in the original dataset, and each shows the average weight assigned to the "target" rule among samples of that stack height, where the target rule is the one that starts with a high concentration of samples of that particular height. At iteration 0, we see that the rules were initialized such that samples were assigned 70% probability of belonging to specific rules, based on stack height. As the algorithm progresses, the samples separate further, suggesting that the algorithm is able to separate samples into the correct groups. Conclusion These demonstrate the power of combining relational abstraction with neural networks, to learn probabilistic state transition models for an important class of domains from very little training data. In addition, the structural nature of the learned models will allow them to be used in factored search-based planning methods that can take advantage of sparsity of effects to plan efficiently. Finally, the end purpose of obtaining a transition model for robotic actions is to enable planning to achieve high-level goals. Due to time constraints, this work did not assess the effectiveness of learned template-based models in planning, but this is a promising area of future work as the assumption that features of objects for which a template makes no explicit prediction do not change meshes well with STRIPS-style planners, as they make similar assumptions. We here provide details on our experiments and more in experiments. Experimental details Our experiments on SPARE in this paper used nueral network predictors for making mean predictions and variance predictions, described in Section 4.1. Each network was implemented as a fully-connected network with two hidden layers of 64 nodes each in Keras, used ReLU activations between layers, and the Adam optimizer with default parameters. Predictors for the templates approach were trained for 1000 epochs each with a decaying learning rate starting at 1e-2 and decreasing by a factor of 0.6 every 100 epochs. The baseline NN predictor was implemented in exactly the same way. For the GNN, we used a node encoder and edge encoder to map to latent spaces of 16 dimensions. The propagation networks consisted of 2 fully connected layers of 16 units each, and the decoder mapped back to 6 dimensions: 3 for the mean, and 3 for the variance. The GNN was trained using a decaying learning rate starting at 1e-2, and decreasing by a factor of 0.5 every 100 epochs. A total of 900 epochs were used. States were parameterized by the (x, y, z) pose of each object in the scene, ordered such that the target object of the action always appeared first, and other objects appeared in random order (except for the baseline). Action parameters included the (x, y, z) starting pose of the robotic gripper, as well as a "push distance" parameter that controls how long the push action lasts. Actions were implemented to be stochastic by adding some amount of noise to the target location of each push, potentially reflecting inaccuries in robotic control. We use the clustering-based approaches for initializing membership probabilities presented in Section 4.3. In this section, we how well our clustering approach performs. TAB0 shows the sample separation achieved by the discrete clustering approach, where samples are assigned solely to their associated clusters found by k-means, on the push dataset for stacks of varying height. Each column corresponds to the one-third of samples which involve stacks of a particular height. Entries in the table show the proportion of samples of that stack height that have been assigned to each of the three clusters, where in generating these data the clusters were ordered so that the predominantly 2-block sample cluster came first, followed by the predominantly 3-block cluster, then the 4-block cluster. Values in parentheses are standard deviations across three runs of the experiment. As seen in the table, separation of samples into clusters is quite good, though 22.7% of 3-block samples were assigned to the predominantly 4-block cluster, and 11.5% of 2-block samples were assigned to the predominantly 3-block cluser. The sample separation evidenced in TAB0 is enough such that templates trained on the three clusters of samples reliably select deictic references that consider the correct number of blocks, i.e., the 2-block cluster reliably learns a template which considers the target object and the object above the target, and similiarly for the other clusters and their respective stack heights. However, initializing samples to belong solely to a single cluster, rather than initializing membership probabilities, is unlikely to be robust in general, so we turn to the proposed clustering-based approaches for initializing membership probabilities instead. TAB1 is analogous to TAB0 in structure, but shows sample separation for sample membership probabilities initialized to be proportional to the inverse distance from the sample to each of the cluster centers found by k-means. TAB2 is the same, except with membership probabilities initialized to be proportional to the square of the inverse distance to cluster centers. Sample separation is better in the case of squared distances than non-squared distances, but it's unclear whether this generalizes to other datasets. For our specific problem instance, the log data likelihood feature turns out to be very important for the success of these clustering-based initialization approaches. For example, Table 4 shows analogous to those in TAB2, where the only difference is that all log data likelihoods were multiplied by five before being passed as input to the k-means clustering algorithm. Comparing the two tables, this scaling of data likelihood to become relatively more important as a feature in better data separation. This suggests that the relative importance between log likelihood and the other input features is a parameter of these clustering approaches that should be tuned. Effect of object ordering on baseline performance The single-predictor baseline used in our experiments receives all objects in the scene as input, but this leaves open the question of in what order these objects should be presented. Because the templates approach has the target object of the action specified, in the interest of fairness this information is also provided to the baseline by having the target object always appear first in the ordering. As there is in general no clear ordering for the remainder of the objects, we could present them in a random order, but perhaps sorting the objects according to position (first by x-coordinate, then y, then z) could in better predictions than if objects are completely randomly ordered. Table 4: Sample separation from clustering-based initialization of membership probabilities, where probabiliites are assigned to be proportional to squared inverse distance to cluster centers, and log data likelihood feature used as part of k-means clustering has been multiplied by a factor of five. Standard deviations are reported in parentheses. To analyze the effect of object ordering on baseline performance, we run the same experiment where a push action is applied to the bottom-most of a stack of three blocks, and there exist some number of additional objects on the table that do not interfere with the action in any way. Figure 6 shows our . We test three object orderings: random ("none"), sorted according to object position ("xtheny"), and an ideal ordering where the first three objects in the ordering are exactly the three objects in the stack ordered from bottom up ("stack"). As expected, in all cases, predicted log likelihood drops as more extra blocks are added to the scene, and the random ordering performs worst while the ideal ordering performs best. Figure 6: Effect of object ordering on baseline performance, on task of pushing a stack of three blocks on a table top, where there are extra blocks on the table that do not interfere with the push."

We detect statistical interactions captured by a feedforward multilayer neural network by directly interpreting its learned weights.


Question: Represent the scientific article to find a short TLDR

Question: Existing sequence prediction methods are mostly concerned with time-independent sequences, in which the actual time span between events is irrelevant and the distance between events is simply the difference between their order positions in the sequence. While this time-independent view of sequences is applicable for data such as natural languages, e.g., dealing with words in a sentence, it is inappropriate and inefficient for many real world events that are observed and collected at unequally spaced points of time as they naturally arise, e.g., when a person goes to a grocery store or makes a phone call. The time span between events can carry important information about the sequence dependence of human behaviors. In this work, we propose a set of methods for using time in sequence prediction. Because neural sequence models such as RNN are more amenable for handling token-like input, we propose two methods for time-dependent event representation, based on the intuition on how time is tokenized in everyday life and previous work on embedding contextualization. We also introduce two methods for using next event duration as regularization for training a sequence prediction model. We discuss these methods based on recurrent neural nets. We evaluate these methods as well as baseline models on five datasets that resemble a variety of sequence prediction tasks. The experiments revealed that the proposed methods offer accuracy gain over baseline models in a range of settings. Event sequence prediction is a task to predict the next event 1 based on a sequence of previously occurred events. Event sequence prediction has a broad range of applications, e.g., next word prediction in language modeling BID10, next place prediction based on the previously visited places, or next app to launch given the usage history. Depending on how the temporal information is modeled, event sequence prediction often decomposes into the following two categories: discrete-time event sequence prediction and continuous-time event sequence prediction. Discrete-time event sequence prediction primarily deals with sequences that consist of a series of tokens (events) where each token can be indexed by its order position in the sequence. Thus such a sequence evolves synchronously in natural unit-time steps. These sequences are either inherently time-independent, e.g, each word in a sentence, or ed from sampling a sequential behavior at an equally-spaced point in time, e.g., busy or not busy for an hourly traffic update. In a discrete-time event sequence, the distance between events is measured as the difference of their order positions. As a consequence, for discrete-time event sequence modeling, the primary goal is to predict what event will happen next. Continuous-time event sequence prediction mainly attends to the sequences where the events occur asynchronously. For example, the time interval between consecutive clinical visits of a patient may potentially vary largely. The duration between consecutive log-in events into an online service can change from time to time. Therefore, one primary goal of continuous-time event sequence prediction is to predict when the next event will happen in the near future. Although these two tasks focus on different aspects of a future event, how to learn a proper representation for the temporal information in the past is crucial to both of them. More specifically, even though for a few discrete-time event sequence prediction tasks (e.g., neural machine translation), they do not involve an explicit temporal information for each event (token), a proper representation of the position in the sequence is still of great importance, not to mention the more general cases where each event is particularly associated with a timestamp. For example, the next destination people want to go to often depends on what other places they have gone to and how long they have stayed in each place in the past. When the next clinical visit BID3 will occur for a patient depends on the time of the most recent visits and the respective duration between them. Therefore, the temporal information of events and the interval between them are crucial to the event sequence prediction in general. However, how to effectively use and represent time in sequence prediction still largely remains under explored. A natural and straightforward solution is to bring time as an additional input into an existing sequence model (e.g., recurrent neural networks). However, it is notoriously challenging for recurrent neural networks to directly handle continuous input that has a wide value range, as what is shown in our experiments. Alternatively, we are inspired by the fact that humans are very good at characterizing time span as high-level concepts. For example, we would say "watching TV for a little while" instead of using the exact minutes and seconds to describe the duration. We also notice that these high-level descriptions about time are event dependent. For example, watching movies for 30 minutes might feel much shorter than waiting in the line for the same amount of time. Thus, it is desirable to learn and incorporate these time-dependent event representations in general. Our paper offers the following contributions:• We propose two methods for time-dependent event representation in a neural sequence prediction model: time masking of event embedding and event-time joint embedding. We use the time span associated with an event to better characterize the event by manipulating its embedding to give a recurrent model additional resolving power for sequence prediction.• We propose to use next event duration as a regularizer for training a recurrent sequence prediction model. Specifically, we define two flavors of duration-based regularization: one is based on the negative log likelihood of duration prediction error and the other measures the cross entropy loss of duration prediction in a projected categorical space.• We evaluated these proposed methods as well as several baseline methods on five datasets (four are public). These datasets span a diverse range of sequence behaviors, including mobile app usage, song listening pattern, and medical history. The baseline methods include vanilla RNN models and those found in the recent literature. These experiments offer valuable findings about how these methods improve prediction accuracy in a variety of settings. In recent years, recurrent neural networks (RNN) especially with Long-Short Term Memory (LSTM) BID9 have become popular in solving a variety of discretetime event sequence prediction problems, including neural machine translation BID0, image captioning BID21 and speech recognition BID16. In a nutshell, given the sequence of previously occurred events, {e 1, e 2, ..., e t}, the conditional probability P(e t+1 |{e 1, e 2, ..., e t}) = P(e t+1 |h t, θ) of the next event e t+1 is estimated by using a recurrent neural network with parameters θ and the hidden state vector h t = f (h t−1, e t, θ) which is assumed to encode the information of the past events. To feed an event into a recurrent neural network, the event, often described as a categorical variable, needs to be represented in a continuous vector space. A common way to achieve this is to use embedding BID1 x t = 1(e t)E x where 1(e t) is a one-hot vector. For the jth event in the vocabulary V, e j, its one-hot vector has 0s for all the entries except the jth entry being 1. E x ∈ R |V |×E is the embedding matrix, where |V | is the number of unique events (the vocabulary size) and E is the embedding dimension. The use of embedding provides a dense representation for an event that improves learning BID18. Through training, the embedding vector of an event encodes its meaning relative to other events. Events that are similar tend to have embedding vectors closer to each other in the embedding space than those that are not. On the other hand, temporal point processes are mathematical abstractions for the continuous-time event sequence prediction task by explicitly modeling the inter-event interval as a continuous random variable. Since the occurrence of an event may be triggered by what happened in the past, we can essentially specify different models for the timing of the next event given what we have already known so far. Very recently, BID6 BID13 BID19 b) focus on expanding the flexibility of temporal point processes using recurrent neural networks where the prediction of the next event time is based on the current hidden state h t of RNN. However, all of these work use the direct concatenation between the inter-event interval and the respective event embedding as the input to the recurrent layer where the representation of the temporal information is limited. Because it is not clear how to properly represent time as input, in this work, we intend to let the model learn a proper representation for encoding temporal information in a sequence, similar to learning embeddings for words. Rather than proposing a new model, our approach should be considered an "embedding" approach for time that can be used by general event sequence prediction models, including models proposed previously BID6 BID13. There are two notions about time spans in a sequential behavior: duration and intervals. Duration is how long an event lasts, e.g., listening to music for an half hour, and an interval is the time span between two adjacent events. To unify both types of time spans, we treat the idle period when no event is occurring (e.g., the person is not using any app for an app usage sequence) as a special event. Thus, duration becomes an inherent property of an event-the interval between two events is the duration of an idle event (see FIG0). With this, h t = f (h t−1, e t, d t ; θ) where d t is the duration of event e t. We here propose two methods to bring continuous time, d t, into a neural sequence prediction model. Both achieve timedependent event representation by manipulating event embedding vectors using time. Our methods are schematically illustrated in Figure 2. Recent work by BID4 revealed that in neural machine translation the embedding vector of a word encodes multiple meanings of the word. As a , it requires a recurrent layer to sacrifice its capacity to disambiguate a word based on its context, instead of focusing on its main task for learning the higher-level compositional structure of a sentence. To address this problem, they used a mask computed based on all the words in a sentence to contextualize the embedding of a target word. Based on this recent work, we propose a method to learn a time mask to "contextualize" event embedding, by which we hope a time-dependent embedding would give the recurrent layer additional resolving power. Similar to the word mask proposed by Choi et al. BID4, we first compute a time context vector for duration, c d. DISPLAYFORM0 φ is a nonlinear transformation of d t and is implemented as a feedforward neural network parameterized by θ. d t is log transformed before it is fed to φ to effectively cover the wide numerical range of duration values, e.g., it can range from seconds to hours for app usage events. Figure 2: A time-dependent RNN for event sequence prediction. d t is used to generate time-dependent event embedding. Next event duration can be used as a regularizer, which can be applied to the recurrent layer and/or any post recurrent layer. We compute a time mask by linearly transforming c d with weights W d ∈ R C×E and bias b d ∈ R E, which is followed by a sigmoid nonlinear activation, σ, to generate a mask m d ∈ R E and R E →. C is the size of the time context vector, and E is the event embedding dimension. DISPLAYFORM1 We then apply the mask to an event embedding by performing an element-wise multiplication,, between the embedding vector and the mask. Finally, the product is fed to the recurrent layer. DISPLAYFORM2 3.2 EVENT-TIME JOINT EMBEDDING Humans developed many ways to tokenize continuous time in everyday life. For example, we would say "talk to someone briefly" instead of using exact minutes and seconds to characterize the length of the conversation. Such a kind of tokenization is extensively used in natural languages. In addition, our perception about the duration also depends on the specific event that we are experiencing. Based on these intuitions, we propose a method to first encode the duration of an event using soft one-hot encoding and then use the encoding to form the joint embedding with the event. To do so, we first project the scalar duration value onto a vector space, where W d ∈ R 1×P is the weight matrix, b d ∈ R P is the bias vector, and P is the projection size. DISPLAYFORM3 We then compute the soft one-hot encoding, s d, of a duration value by applying a softmax function to the projection vector, p d. Softmax has been typically used in the output layer BID7 and in the attention mechanisms BID0 BID21 for selecting one out of many. The ith entry of the encoding vector is calculated as the following and p d i is the ith entry in p d. DISPLAYFORM4 All the entries in the soft one-hot encoding are positive. Similar to a regular one-hot encoding, DISPLAYFORM5 We then project the soft one-hot encoding onto a time embedding space, g d. It has the same dimension as the event embedding. E s ∈ R P ×E is the embedding matrix. DISPLAYFORM6 Embedding for a regular one-hot encoding essentially takes a single row of the embedding matrix that is corresponding to the non-zero entry as the embedding vector. In contrast, embedding for a soft one-hot encoding computes a weighted sum over all the rows in the embedding matrix. Finally, we form the joint embedding of an event and its duration by taking the mean of their embedding vectors, which is then fed to the recurrent layer. DISPLAYFORM7 4 NEXT EVENT DURATION AS A REGULARIZER While our goal here is to predict next event, it can help learning by introducing an additional loss component based on the prediction of the next event duration (see Figure 2). The duration prediction of the next event at step t, d t+1, is computed from a linear transformation of the recurrent layer. A loss defined on the prediction error of d t+1 provides additional information during back propagation, acting like a regularizer. Optionally, one can use the concatenation of the recurrent layer output and a hidden layer on the path for event prediction to regularize more layers. We discuss two alternatives for the loss function over d t+1. A common way for the loss over a continuous value is to use the squared error. Here, it is (d t+1 − d t+1) 2 where d t+1 is the observed duration of the next event. However, such a loss needs to be at the same scale as that of of event prediction, which is typically a log likelihood of some form. Hinton and Van Camp (Hinton & van) have shown that minimizing the squared error can be in fact formulated as maximizing the probability density of a zero-mean Gaussian distribution. Note that this does not require duration to obey a Gaussian distribution but rather the prediction error. We define our regularizer, R N t, as the negative log likelihood of duration prediction error at step t. DISPLAYFORM0 The variance, σ i, is seeded with an initial value (e.g., the variance of duration values in the training data) and updated iteratively during training based on the duration prediction error distribution of the learned model at each update i. In Section 3.2, we proposed to use softmax to project a continuous duration value onto a categorical space. Using the same technique, by projecting both d t+1 and d t+1 onto a categorical space, we can then compute a cross entropy loss based on the two projections as another regularizer R X t. DISPLAYFORM0 P roj is the softmax projection process we defined in Equation 4 and 5, P roj k is the kth entry in the projection vector. When event-time joint embedding and R X t are both used, the embedding and the regularizer can use the same projection function, i.e., sharing the same projection weights (Equation 4). In this section, we evaluate the effectiveness of our proposed approaches on the following five real-world datasets across a diverse range of domains.• Electrical Medical Records. MIMIC II medical dataset is a collection of de-identified clinical visit records of Intensive Care Unit patients for seven years. The filtered dataset released by BID6 include 650 patients and 204 diseases. The goal is to predict which major disease will happen to a given patient.• Stack Overflow Dataset. The Stack Overflow dataset includes two years of user awards on a question-answering website. The awarded badges are treated as the events. BID6 collected 6,000 users with a total of 480,000 events. The goal is to predict the next badge a user will receive.• Financial Transaction Dataset. BID6 ) collected a long stream of high frequency transactions for a single stock from NYSE where the events correspond to the "buy" and "sell" actions. The task is to predict the next action a user might take.• App Usage Dataset. Mobile users often use a large number of apps, ranging from tens to hundreds. It is time consuming to find a target app on mobile devices. One promising way to address this problem is to predict the next app a user will use based on their app usage history. Being able to predict next apps also allows the mobile platform to preload an app in memory to speed up its startup. We have collected 5,891 app usage sequences comprising of 2.8 million app usage events. The task is to predict the next app that will be used for a given user.• Music Recommendation. The music dataset represents the longitudinal listening habits of 992 users BID12 BID2 involving millions of listening events. The goal is to predict the next five unique songs that the user has not listened given the user's listen history. For the MIMIC II, Stack Overflow, and Financial data, we follow BID6 to pre-process the data and seek to predict every single held-out event from the history. We evaluate the prediction accuracy with the binary 0-1 loss. For the app usage data, to avoid users who participated in the data collection only briefly, we exclude sequences that have fewer than 50 app launches or if the time span of the sequence is shorter than a week. This ed in 5,891 app usage sequences, one from each unique user. These sequences include 2,863,095 app usage events and the longest sequence spanned 551 days. We split the dataset on users into the training (80%), validation (10%) and test (10%) such that each user is only in one of these partitions. Hence there is no intersection of users between training, validation and test sets. For an event that has fewer than 5 occurrences in the training dataset, we assign it the OOV id for out of vocabulary. In total, there are 7,327 events in the vocabulary, including 7,325 unique apps, the idle event and the OOV (out of vocabulary). In practice, predicting the next 5 apps is often desired so we use Precision@K to evaluate the performance. For the music recommendation, each listen event has a timestamp. We removed sequences that are shorter than 50 and songs that have fewer than 50 listens. We thus generate a collection of examples where each example consists of a listen history and a set of 5 unique songs to recommend. To do so, we split each original listen sequence into segments. We first take the 40 events out in order from the beginning of the sequence as the listen history, and then take more events out from the beginning of the sequence until we find 5 unique songs that have not occurred in the listen history. We do so repeatedly to extract each example until we exhaust all the original sequences. This data processing ed in 221,920 sequence examples with 71,619 unique songs (the vocabulary size). We then allocate these sequence examples for the training (80%), validation (10%) and test (10%). Because the original dataset does not have the duration information for each listen event, we did not inject the additional idle event in the sequence to differentiate duration versus intervals. Because in practice, the ranking order of the recommended music often matters, we further use MAP@K and Prevision@K to evaluate the performance. We compare with the following five models: NoTime in which a simple LSTM sequence model is used; TimeConcat in which we feed time (log transformed) directly into the recurrent layer along the event embedding; TimeMask (Section 3.1) and TimeJoint (Section 3.2) for generating time-dependent event embedding as input to the recurrent layer; and RMTPP for the model introduced previously by BID6. Moreover, we also include four regularized models based on R FORMULA3 ). For the App Usage and Music Recommendation experiments, we use a two-layer hierarchical softmax BID14 for the output layer due to the large vocabulary size, while we use a full sofmax for the rest experiments. For the MIMIC II, Stack Overflow, and Financial data, we follow BID6 for RMTPP's model parameters. For the app usage data, we determined the parameters of each model based on the training and the validation datasets on a distributed parallel tuning infrastructure. We used LSTM units BID9 for the recurrent layer, and Rectified Linear Units (ReLu) BID15 for the activation function in the nonlinear projection layer. The event embedding dimension, the number of LSTM units, and the nonlinear projection layer size are all set to 128. For the music recommendation data, we use a setting similar to the app prediction experiment where we chose the embedding size as 128 and LSTM size as 256. We did not use the nonlinear projection layer after the LSTM layer for this task because it does not seem to help. We implemented all the models in TensorFlow . For the experiments based on MIMIC II, Stack Overflow and Financial Transaction datasets, we use the same training and testing strategy of BID6. For App Usage and Music Recommendation tasks, we selected the model architecture and hyper parameters with early stopping based on the validation dataset of each task, and report the performance of each model based on the test dataset. For the App Usage experiment, we used truncated back-propagation through time with the number of unroll to be 30. We used an adaptive gradient descent optimizer BID22, using a learning rate of 0.024 with a threshold for gradient clipping of 1.0, and a batch size of 32. We decided not to use dropout as it did not seem to improve accuracy on this task. For the Music Recommendation experiment, we used the full sequence back-propagation through time with 2% dropout ratio on the recurrent layer for better generalization. We used the Adam optimizer by BID11 for adaptive learning with a learning rate of 0.00005 and a gradient clipping threshold at 1.0. The mini-batch size is 256.We trained the models by minimizing the cross-entropy loss, plus the regularization loss if the duration regularizer is used, over all the sequences in the training dataset. The training for App Usage and Music Recommendation was conducted on a distributed learning infrastructure BID5 with 50 GPU cores where updates are applied asynchronously across multiple replicas. Effectiveness of Temporal Representation. FIG2 presents the comparisons between all the models on three released public datasets. We can observe a consistent performance gain with using the proposed methods for time-dependent event embedding compared to the NoTime baseline and the simple TimeConcat approach. TimeJoint significantly outperformed all other methods on both the Stack Overflow and the Financial dataset, with p<0.05 using Paired T-test. But none of the methods for using time is able to improve accuracy on the MIMIC II dataset. This indicates that using time might not always help. However, when it does, our methods such as TimeJoint enable more efficient representation of time than simply using the scalar value of time in RNN models. Our methods also outperformed RMTPP for event prediction. The performance gain of our models are more pronounced on the App Usage and Music Recommendation datasets as shown in TAB3 and 2. TimeJoint seems to outperform the rest on most measures and TimeMask also performs well compared to other previous methods. We also notice that using time directly without representing them appropriately in RNN, i.e., TimeConcat, can sometime hurt the performance. Effectiveness of Event Duration Regularization. We demonstrate the performance boosting gained from our proposed temporal regularization in TAB3 and 2, respectively. We can observe that our proposed regularizers can bring additional performance gain on many cases. In particular, the crossentropy regularizer, R X t, is able to give consistent performance gain with the temporal embedding approaches. Learned Time Representation. Our motivation in this work is to let the model learn a proper representation of time from data. We here briefly discuss what the TimeJoint approach learns about how to project a scalar value of time into a soft one-hot encoding 4. It seems that for small time periods, e.g., shorter than 20 seconds for the Next App prediction task, more dimensions are needed to express the differences of continuous time values. As the time period grows, we need less dimensions for representing time, e.g., two of the curves have converged to the same small values. We proposed a set of methods for leveraging the temporal information for event sequence prediction. Based on our intuition about how humans tokenize time spans as well as previous work on contextual representation of words, we proposed two methods for time-dependent event representation. They transform a regular event embedding with learned time masking and form time-event joint embedding based on learned soft one-hot encoding. We also introduced two methods for using next duration as a way of regularization for training a sequence prediction model. Experiments on a diverse range of real data demonstrate consistent performance gain by blending time into the event representation before it is fed to a recurrent neural network.

Proposed methods for time-dependent event representation and regularization for sequence prediction; Evaluated these methods on five datasets that involve a range of sequence prediction tasks.

Represent the TLDR for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.