Question: Represent the article for finding a single-sentence TLDR

Question: Many irregular domains such as social networks, financial transactions, neuron connections, and natural language structures are represented as graphs. In recent years, a variety of graph neural networks (GNNs) have been successfully applied for representation learning and prediction on such graphs. However, in many of the applications, the underlying graph changes over time and existing GNNs are inadequate for handling such dynamic graphs. In this paper we propose a novel technique for learning embeddings of dynamic graphs based on a tensor algebra framework. Our method extends the popular graph convolutional network (GCN) for learning representations of dynamic graphs using the recently proposed tensor M-product technique. Theoretical that establish the connection between the proposed tensor approach and spectral convolution of tensors are developed. Numerical experiments on real datasets demonstrate the usefulness of the proposed method for an edge classification task on dynamic graphs. Graphs are popular data structures used to effectively represent interactions and structural relationships between entities in structured data domains. Inspired by the success of deep neural networks for learning representations in the image and language domains, recently, application of neural networks for graph representation learning has attracted much interest. A number of graph neural network (GNN) architectures have been explored in the contemporary literature for a variety of graph related tasks and applications (; ; ;). Methods based on graph convolution filters which extend convolutional neural networks (CNNs) to irregular graph domains are popular (; ;). Most of these GNN models operate on a given, static graph. In many real-world applications, the underlining graph changes over time, and learning representations of such dynamic graphs is essential. Examples include analyzing social networks , predicting collaboration in citation networks , detecting fraud and crime in financial networks , traffic control , and understanding neuronal activities in the brain . In such dynamic settings, the temporal interdependence in the graph connections and features also play a substantial role. However, efficient GNN methods that handle time varying graphs and that capture the temporal correlations are lacking. By dynamic graph, we mean a sequence of graphs (V, A (t), X (t) ), t ∈ {1, 2, . . ., T}, with a fixed set V of N nodes, adjacency matrices A (t) ∈ R N ×N, and graph feature matrices X (t) ∈ R N ×F where X (t) n: ∈ R F is the feature vector consisting of F features associated with node n at time t. The graphs can be weighted, and directed or undirected. They can also have additional properties like (time varying) node and edge classes, which would be stored in a separate structure. Suppose we only observe the first T < T graphs in the sequence. The goal of our method is to use these observations to predict some property of the remaining T − T graphs. In this paper, we use it for edge classification. Other potential applications are node classification and edge/link prediction. In recent years, tensor constructs have been explored to effectively process high-dimensional data, in order to better leverage the multidimensional structure of such data . Tensor based approaches have been shown to perform well in many image and video processing ap- plications;;;; ). A number of tensor based neural networks have also been investigated to extract and learn multi-dimensional representations, e.g. methods based on tensor decomposition , tensor-trains , and tensor factorized neural network . Recently, a new tensor framework called the tensor M-product framework (; ;) was proposed that extends matrix based theory to high-dimensional architectures. In this paper, we propose a novel tensor variant of the popular graph convolutional network (GCN) architecture , which we call TensorGCN. It captures correlation over time by leveraging the tensor M-product framework. The flexibility and matrix mimeticability of the framework, help us adapt the GCN architecture to tensor space. Figure 1 illustrates our method at a high level: First, the time varying adjacency matrices A (t) and feature matrices X (t) of the dynamic graph are aggregated into an adjacency tensor and a feature tensor, respectively. These tensors are then fed into our TensorGCN, which computes an embedding that can be used for a variety of tasks, such as link prediction, and edge and node classification. GCN architectures are motivated by graph convolution filtering, i.e., applying filters/functions to the graph Laplacian (in turn its eigenvalues) , and we establish a similar connection between TensorGCN and spectral filtering of tensors. Experimental on real datasets illustrate the performance of our method for the edge classification task on dynamic graphs. Elements of our method can also be used as a preprocessing step for other dynamic graph methods. The idea of using graph convolution based on the spectral graph theory for GNNs was first introduced by. then proposed Chebnet, where the spectral filter was approximated by Chebyshev polynomials in order to make it faster and localized. presented the simplified GCN, a degree-one polynomial approximation of Chebnet, in order to speed up computation further and improve the performance. There are many other works that deal with GNNs when the graph and features are fixed/static; see the review papers by and and references therein. These methods cannot be directly applied to the dynamic setting we consider. devised the Graph Convolutional Recurrent Network for graphs with time varying features. However, this method assumes that the edges are fixed over time, and is not applicable in our setting. proposed a method called EdgeConv, which is a neural network (NN) approach that applies convolution operations on static graphs in a dynamic fashion. Their approach is not applicable when the graph itself is dynamic. develop a temporal GCN method called T-GCN, which they apply for traffic prediction. Their method assumes the graph remains fixed over time, and only the features vary. The set of methods most relevant to our setting of learning embeddings of dynamic graphs use combinations of GNNs and recurrent architectures (RNN), to capture the graph structure and handle time dynamics, respectively. The approach in uses Long Short-Term Memory (LSTM), a recurrent network, in order to handle time variations along with GNNs. They design architectures for semi-supervised node classification and for supervised graph classification. presented a variant of GCN called EvolveGCN, where Gated Recurrent Units (GRU) and LSTMs are coupled with a GCN to handle dynamic graphs. This paper is currently the stateof-the-art. However, their approach is based on a heuristic RNN/GRU mechanism, which is not theoretically viable, and does not harness a tensor algebraic framework to incorporate time varying information. present a tensor NN which utilizes the M-product tensor framework. Their approach can be applied to image and other high-dimensional data that lie on regular grids, and differs from ours since we consider data on dynamic graphs. Here, we cover the necessary preliminaries on tensors and the M-product framework. For a more general introduction to tensors, we refer the reader to the review paper by. In this paper, a tensor is a three-dimensional array of real numbers denoted by boldface Euler script letters, e.g. X ∈ R I×J×T. Matrices are denoted by bold uppercase letters, e.g. X; vectors are denoted by bold lowercase letter, e.g. x; and scalars are denoted by lowercase letters, e.g. x. An element at position (i, j, t) in a tensor is denotes by subscripts, e.g. X ijt, with similar notation for elements of matrices and vectors. A colon will denote all elements along that dimension; X i: denotes the ith row of the matrix X, and X::k denotes the kth frontal slice of X. The vectors X ij: are called the tubes of X. The framework we consider relies on a new definition of the product of two tensors, called the M-product (; ;). A distinguishing feature of this framework is that the M-product of two three-dimensional tensors is also three-dimensional, which is not the case for e.g. tensor contractions . It allows one to elegantly generalize many classical numerical methods from linear algebra, and has been applied e.g. in neural networks , imaging; ), facial recognition, and tensor completion and denoising (; ;). Although the framework was originally developed for three-dimensional tensors, which is sufficient for our purposes, it has been extended to handle tensors of dimension greater than three . The following definitions 3.1-3.3 describe the M-product. Definition 3.1 (M-transform). Let M ∈ R T ×T be a mixing matrix. The M-transform of a tensor X ∈ R I×J×T is denoted by X × 3 M ∈ R I×J×T and defined elementwise as We say that X × 3 M is in the transformed space. may also be written in matrix form as, where the unfold operation takes the tubes of X and stack them as columns into a T × IJ matrix, and fold(unfold(X)) = X. Appendix A provides illustrations of how the M-transform works. Definition 3.2 (Facewise product). Let X ∈ R I×J×T and Y ∈ R J×K×T be two tensors. The I×J×T and Y ∈ R J×K×T be two tensors, and let M ∈ R T ×T be an invertible matrix. The M-product, denoted by X Y ∈ R I×K×T, is defined as In the original formulation of the M-product, M was chosen to be the Discrete Fourier Transform (DFT) matrix, which allows efficient computation using the Fast Fourier Transform (FFT) (; ; . The framework was later extended for arbitrary invertible M (e.g. discrete cosine and wavelet transforms) . A benefit of the tensor M-product framework is that many standard matrix concepts can be generalized in a straightforward manner. Definitions 3.4-3.7 extend the matrix concepts of diagonality, identity, transpose and orthogonality to tensors (; . Definition 3.5 (Identity tensor). LetÎ ∈ R N ×N ×T be defined facewise asÎ::t = I, where I is the matrix identity. The M-product identity tensor I ∈ R N ×N ×T is then defined as Definition 3.6 (Tensor transpose). The transpose of a tensor X is defined as X def = Y × 3 M −1, where Y::t = (X × 3 M)::t for each t ∈ {1, . . ., T}. Definition 3.7 (Orthogonal tensor). A tensor X ∈ R N ×N ×T is said to be orthogonal if X X = X X = I. Leveraging these concepts, a tensor eigendecomposition can now be defined (; : Definition 3.8 (Tensor eigendecomposition). Let X ∈ R N ×N ×T be a tensor and assume that each frontal slice (X × 3 M)::t is symmetric. We can then eigendecompose these as (X × 3 M)::t = Q::tD::tQ::t, whereQ::t ∈ R N ×N is orthogonal andD::t ∈ R N ×N is diagonal (see e.g. Theorem 8.1.1 in). The tensor eigendecomposition of X is then defined as Our approach is inspired by the first order GCN by for static graphs, owed to its simplicity and effectiveness. For a graph with adjacency matrix A and feature matrix X, a GCN layer takes the form Y = σ(ÃXW), wherẽ is the matrix identity, W is a matrix to be learned when training the NN, and σ is an activation function, e.g., ReLU. Our approach translates this to a tensor model by utilizing the M-product framework. We first introduce a tensor activation functionσ which operates in the transformed space. Definition 4.1. Let A ∈ R I×J×T be a tensor and σ an elementwise activation function. We define the activation functionσ asσ(A) We can now define our proposed dynamic graph embedding. Let A ∈ R N ×N ×T be a tensor with frontal slices A::t =Ã (t), whereÃ (t) is the normalization of A (t). Moreover, let X ∈ R N ×F ×T be a tensor with frontal slices X::t = X (t). Finally, let W ∈ R F ×F ×T be a weight tensor. We define our dynamic graph embedding as Y = A X W ∈ R N ×F ×T. This computation can also be repeated in multiple layers. For example, a 2-layer formulation would be of the form One important consideration is how to choose the matrix M which defines the M-product. For time-varying graphs, we choose M to be lower triangular and banded so that each frontal slice (A × 3 M)::t is a linear combination of the adjacency matrices A::max(1,t−b+1),..., A::t, where we refer to b as the "bandwidth" of M. This choice ensures that each frontal slice (A × 3 M)::t only contains information from current and past graphs that are close temporally. Specifically, the entries of M are set to otherwise, which implies that k M tk = 1 for each t. Another possibility is to treat M as a parameter matrix to be learned from the data. In order to avoid over-parameterization and improve the performance, we choose the weight tensor W (at each layer), such that each of the frontal slices of W in the transformed domain remains the same, i.e., (W × 3 M)::t = (W × 3 M)::t ∀t, t. In other words, the parameters in each layer are shared and learned over all the training instances. This reduces the number of parameters to be learned significantly. An embedding Y ∈ R N ×F ×T can now be used for various prediction tasks, like link prediction, and edge and node classification. In Section 5, we apply our method for edge classification by using a model similar to that used by: Given an edge between nodes m and n at time t, the predictive model is where (Y × 3 M) m:t ∈ R F and (Y × 3 M) n:t ∈ R F are row vectors, U ∈ R C×2F is a weight matrix, and C the number of classes. Note that the embedding Y is first M-transformed before the matrix U is applied to the appropriate feature vectors. This, combined with the fact that the tensor activation functions are applied elementwise in the transformed domain, allow us to avoid ever needing to apply the inverse M-transform. This approach reduces the computational cost, and has been found to improve performance in the edge classification task. Here, we present the that establish the connection between the proposed TensorGCN and spectral convolution of tensors, in particular spectral filtering and approximation on dynamic graphs. This is analogous to the graph convolution based on spectral graph theory in the GNNs by , , and. All proofs are provided in Appendix D. Let L ∈ R N ×N ×T be a form of tensor Laplacian defined as L def = I − A. Throughout the remainder of this subsection, we will assume that the adjacency matrices A (t) are symmetric. Following the work by, three-dimensional tensors in R M ×N ×T can be viewed as operators on N × T matrices, with those matrices "twisted" into tensors in R N ×1×T. With this in mind, we define a tensor variant of the graph Fourier transform. Definition 4.4 (Tensor-tube M-product). Let X ∈ R I×J×T and θ ∈ R 1×1×T. Analogously to the definition of the matrix-scalar product, we define X θ via (X θ) ij: Definition 4.5 (Tensor graph Fourier transform). Let X ∈ R N ×F ×T be a tensor. We define a tensor graph Fourier transform F as F (X) This is analogous to the definition of the matrix graph Fourier transform. This defines a convolution like operation for tensors similar to spectral graph convolution . Each lateral slice X:j: is expressible in terms of the set {Q :n:} N n=1 as follows: where each (Q X :j:) n1: ∈ R 1×1×T can be considered a tubal scalar. In fact, the lateral slices Q:n: form a basis for the set R N ×1×T with product; see Appendix D for further details. Definition 4.6 (Tensor spectral graph filtering). Given a signal X ∈ R N ×1×T and a function g: R 1×1×T → R 1×1×T, we define the tensor spectral graph filtering of X with respect to g as where In order to avoid the computation of an eigendecomposition, use a polynomial to approximate the filter function. We take a similar approach, and approximate g(D) with an M-product polynomial. For this approximation to make sense, we impose additional structure on g. Assumption 4.7. Assume that g: where f is defined elementwise as Proposition 4.8. Suppose g satisfies Assumption 4.7. For any ε > 0, there exists an integer K and a set {θ where · is the tensor Frobenius norm, and where As in the work of , a tensor polynomial approximation allows us to approximate X filt in without computing the eigendecomposition of L: All that is necessary is to compute tensor powers of L. We can also define tensor polynomial analogs of the Chebyshev polynomials and do the approximation in in terms of those instead of the tensor monomials D k. This is not necessary for the purposes of this paper. Instead, we note that if a degree-one approximation is used, the computation in becomes, which is analogous to the parameter choice made in the degree-one approximation by , we get If we let X contain F signals, i.e., X ∈ R N ×F ×T, and apply F filters, becomes where Θ ∈ R F ×F ×T. This is precisely our embedding model, with Θ replaced by a learnable parameter tensor W. Here, we present for edge classification on four datasets 1: The Bitcoin Alpha and OTC transaction datasets , the Reddit body hyperlink dataset , and a chess dataset . The bitcoin datasets consist of transaction histories for users on two different platforms. Each node is a user, and each directed edge indicates a transaction and is labeled with an integer between −10 and 10 which indicates the senders trust for the receiver. We convert these labels to two classes: positive (trustworthy) and negative (untrustworthy). The Reddit dataset is build from hyperlinks from one subreddit to another. Each node represents a subreddit, and each directed edge is an interaction which is labeled with −1 for a hostile interaction or +1 for a friendly interaction. We only consider those subreddits which have a total of 20 interactions or more. In the chess dataset, each node is a player, and each directed edge represents a match with the source node being the white player and the target node being the black player. Each edge is labeled −1 for a black victory, 0 for a draw, and +1 for a white victory. The data is temporally partitioned into T graphs, with each graph containing data from a particular time window. Both T and the time window length can vary between datasets. For each node-time pair (n, t) in these graphs, we compute the number of outgoing and incoming edges and use these two numbers as features. The adjacency tensor A is then constructed as described in Section 4. The T frontal slices of A are divided into S train training slices, S val validation slices, and S test testing slices, which come sequentially after each other; see Figure 2 and Table 2. Since the adjacency matrices corresponding to graphs are very sparse for these datasets, we apply the same technique as and add the entries of each frontal slice A::t to the following l − 1 frontal slices A::t,..., A::(t+l−1), where we refer to l as the "edge life." Note that this only affects A, and that the added edges are not treated as real edges in the classification problem. The bitcoin and Reddit datasets are heavily skewed, with about 90% of edges labeled positively, and the remaining labeled negatively. Since the negative instances are more interesting to identify (e.g. to prevent financial fraud or online hostility), we use the F1 score to evaluate the experiments on these datasets, treating the negative edges as the ones we want to identify. The classes are more well-balanced in the chess dataset, so we use accuracy to evaluate those experiments. We choose to use an embedding Y train = A::(1:Strain) X::(1:Strain) W for training. When computing the embeddings for the validation and testing data, we still need S train frontal slices of A, which we get by using a sliding window of slices. This is illustrated in Figure 2, where the green, blue and red blocks show the frontal slices used when computing the embeddings for the training, validation and testing data, respectively. The embeddings for the validation and testing data are Y val = A::(Sval+1:Strain+Sval) X::(Sval+1:Strain+Sval) W and Y test = A::(Sval+Stest+1:T) X::(Sval+Stest+1:T) W, respectively. Preliminary experiments with 2-layer architectures did not show convincing improvements in performance. We believe this is due to the fact that the datasets only have two features, and that a 1-layer architecture therefore is sufficient for extracting relevant information in the data. For training, we use the cross entropy loss function: where f (m, n, t) ∈ R C is a one-hot vector encoding the true class of the edge (m, n) at time t, and α ∈ R C is a vector summing to 1 which contains the weight of each class. Since the bitcoin and Reddit datasets are so skewed, we weigh the minority class more heavily in the loss function for those datasets, and treat α as a hyperparameter; see Appendix C for details. The experiments are implemented in PyTorch with some preprocessing done in Matlab. Our code is available at [url redacted for review]. In the experiments, we use an edge life of l = 10, a bandwidth b = 20, and F = 6 output features. Since the graphs in the considered datasets are directed, we also investigate the impact of symmetrizing the adjacency matrices, where the symmetrized version of an adjacency matrix A is defined as A sym def = 1/2(A + A). We compare our method with three other methods. The first one is a variant of the WD-GCN by , which they specify in Equation (8a) of their paper. For the LSTM layer in their description, we use 6 output features instead of N. This is to avoid overfitting and make the method more comparable to ours which uses 6 output features. For the final layer, we use the same prediction model as that used by for edge classification. The second method is a 1-layer variant of EvolveGCN-H by. The third method is a simple baseline which uses a 1-layer version of the GCN by. It uses the same weight matrix W for all temporal graphs. Both EvolveGCN-H and the baseline GCN use 6 output features as well. Table 3 shows the when the adjacency matrices have not been symmetrized. In this case, our method outperforms the other methods on the two bitcoin datasets and the chess dataset, with WD-GCN performing best on the Reddit dataset. Table 4 shows the for when the adjacency matrices have been symmetrized. Our method outperforms the other methods on the Bitcoin OTC dataset and the chess dataset, and performs similarly but slightly worse than the best performing methods on the Bitcoin Alpha and Reddit datasets. Overall, it seems like symmetrizing the adjacency matrices leads to lower performance. We have presented a novel approach for dynamic graph embedding which leverages the tensor Mproduct framework. We used it for edge classification in experiments on four real datasets, where it performed competitively compared to state-of-the-art methods. Future research directions include further developing the theoretical guarantees for the method, investigating optimal structure and learning of the transform matrix M, using the method for other prediction tasks, and investigating how to utilize deeper architectures for dynamic graph learning. We provide some illustrations that show how the M-transform in Definition 3.1 works. Recall that X × 3 M = fold(M unfold(X)). The matrix X is first unfolded into a matrix, as illustrated in Figure 3. This unfolded tensor is then multiplied from the left by the matrix M, as illustrated in Figure 4; the figure also illustrates the banded lower triangular structure of M. Finally, the output matrix is folded back into a tensor. The fold operation is defined to be the inverse of the unfold operation. • The Bitcoin Alpha dataset is available at https://snap.stanford.edu/data/soc-sign-bitcoin-alpha.html. • The Bitcoin OTC dataset is available at https://snap.stanford.edu/data/soc-sign-bitcoin-otc.html. • The Reddit dataset is available at https://snap.stanford.edu/data/soc-RedditHyperlinks.html. Note that we use the dataset with hyperlinks in the body of the posts. • The chess dataset is available at http://konect.uni-koblenz.de/networks/chess. When partitioning the data into T graphs, as described in Section 5, if there are multiple data points corresponding to an edge (m, n) for a given time step t, we only add that edge once to the corresponding graph and set the label equal to the sum of the labels of the different data points. For example, if bitcoin user m makes three transactions to n during time step t with ratings 10, 2, −1, then we add a single edge (m, n) to graph t with label 10 + 2 − 1 = 11. For training, we run gradient descent with a learning rate of 0.01 and momentum of 0.9 for 10,000 iterations. For each 100 iterations, we compute and store the performance of the model on the validation data. As mentioned in Section 5, the weight vector α in the loss function is treated as a hyperparameter in the bitcoin and Reddit experiments. Since these datasets all have two edge classes, let α 0 and α 1 be the weights of the minority (negative) and majority (positive) classes, respectively. Since these parameters add to 1, we have α 1 = 1 − α 0. For all methods, we repeat the bitcoin and Reddit experiments once for each α 0 ∈ {0.75, 0.76, . . ., 0.95}. For each model and dataset, we then find the best stored performance of the model on the validation data across all α 0 values. We then treat the corresponding model as the trained model, and report its performance on the testing data in Tables 3 and 4. The for the chess experiment are computed in the same way, but only for a single vector α = . Throughout this section, · will denote the Frobenius norm (i.e., the square root of the sum of the elements squared) of a matrix or tensor, and · 2 will denote the matrix spectral norm. We first provide a few further that clarify the algebraic properties of the M-product. Let R 1×1×T denote the set of 1 × 1 × T tensors. Similarly, let R N ×1×T denote the set of N × 1 × T tensors. Under the M-product framework, the set R 1×1×T play a role similar to that played by scalars in matrix algebra. With this in mind, the set R N ×1×T can be seen as a length N vector consisting of tubal elements of length T. Propositions D.1 and D.2 make this more precise. Proposition D.1 (Proposition 4.2 in). The set R 1×1×T with product, which is denoted by (, R 1×1×T), is a commutative ring with identity. Proposition D.2 (Theorem 4.1 in). The set R N ×1×T with product, which is denoted by (, R N ×1×T), is a free module over the ring (, R 1×1×T). A free module is similar to a vector space. Like a vector space, it has a basis. Proposition D.3 shows that the lateral slices of Q in the tensor eigendecomposition form a basis for (, R N ×1×T), similarly to how the eigenvectors in a matrix eigendecomposition form a basis. Proposition D.3. The lateral slices Q:n: ∈ R N ×1×T of Q in Definition 3.8 form a basis for (, R N ×1×T). Proof. Let X ∈ R N ×1×T. Note that Since each frontal face of Q × 3 M is an invertible matrix, this implies that each frontal face of S × 3 M is zero, and hence S = 0. So the lateral slices of Q are also linearly independent in (, R N ×1×T).

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a novel tensor based method for graph convolutional networks on dynamic graphs

Represent the short summary for finding the corresponding scientific article (long)

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question:
represent the article for finding a single-sentence tldr
For example, 'Imitation Learning (IL) is an appealing approach to learn desirable autonomous behavior. However, directing IL to achieve arbitrary goals is difficult. In contrast, planning-based algorithms use dynamics models and reward functions to achieve goals. Yet, reward functions that evoke desirable behavior are often difficult to specify. In this paper, we propose "Imitative Models" to combine the benefits of IL and goal-directed planning. Imitative Models are probabilistic predictive models of desirable behavior able to plan interpretable expert-like trajectories to achieve specified goals. We derive families of flexible goal objectives, including constrained goal regions, unconstrained goal sets, and energy-based goals. We show that our method can use these objectives to successfully direct behavior. Our method substantially outperforms six IL approaches and a planning-based approach in a dynamic simulated autonomous driving task, and is efficiently learned from expert demonstrations without online data collection. We also show our approach is robust to poorly-specified goals, such as goals on the wrong side of the road. Imitation learning (IL) is a framework for learning a model to mimic behavior. At test-time, the model pursues its best-guess of desirable behavior. By letting the model choose its own behavior, we cannot direct it to achieve different goals. While work has augmented IL with goal conditioning , it requires goals to be specified during training, explicit goal labels, and are simple (e.g., turning). In contrast, we seek flexibility to achieve general goals for which we have no demonstrations. In contrast to IL, planning-based algorithms like model-based reinforcement learning (MBRL) methods do not require expert demonstrations. MBRL can adapt to new tasks specified through reward functions . The "model" is a dynamics model, used to plan under the user-supplied reward function. Planning enables these approaches to perform new tasks at test-time. The key drawback is that these models learn dynamics of possible behavior rather than dynamics of desirable behavior. This means that the responsibility of evoking desirable behavior is entirely deferred to engineering the input reward function. Designing reward functions that cause MBRL to evoke complex, desirable behavior is difficult when the space of possible undesirable behaviors is large. In order to succeed, the rewards cannot lead the model astray towards observations significantly different than those with which the model was trained. Our goal is to devise an algorithm that combines the advantages of MBRL and IL by offering MBRL's flexibility to achieve new tasks at test-time and IL's potential to learn desirable behavior entirely from offline data. To accomplish this, we first train a model to forecast expert trajectories with a density function, which can score trajectories and plans by how likely they are to come from the expert. A probabilistic model is necessary because expert behavior is stochastic: e.g. at an intersection, the expert could choose to turn left or right. Next, we derive a principled probabilistic inference objective to create plans that incorporate both the model and arbitrary new tasks. Finally, we derive families of tasks that we can provide to the inference framework. Our method can accomplish new tasks specified as complex goals without having seen an expert complete these tasks before. We investigate properties of our method on a dynamic simulated autonomous driving task (see Fig. 1). Videos are available at https://sites.google.com/view/imitative-models. Our contributions are as follows: Figure 1: Our method: deep imitative models. Top Center. We use demonstrations to learn a probability density function q of future behavior and deploy it to accomplish various tasks. Left: A region in the ground plane is input to a planning procedure that reasons about how the expert would achieve that task. It coarsely specifies a destination, and guides the vehicle to turn left. Right: Goal positions and potholes yield a plan that avoids potholes and achieves one of the goals on the right. 1. Interpretable expert-like plans with minimal reward engineering. Our method outputs multistep expert-like plans, offering superior interpretability to one-step imitation learning models. In contrast to MBRL, our method generates expert-like behaviors with minimal reward engineering. 2. Flexibility to new tasks: In contrast to IL, our method flexibly incorporates and achieves goals not seen during training, and performs complex tasks that were never demonstrated, such as navigating to goal regions and avoiding test-time only potholes, as depicted in Fig. 1. 3. Robustness to goal specification noise: We show that our method is robust to noise in the goal specification. In our application, we show that our agent can receive goals on the wrong side of the road, yet still navigate towards them while staying on the correct side of the road. 4. State-of-the-art CARLA performance: Our method substantially outperforms MBRL, a custom IL method, and all five prior CARLA IL methods known to us. It learned near-perfect driving through dynamic and static CARLA environments from expert observations alone. We begin by formalizing assumptions and notation. We model continuous-state, discrete-time, Partially-Observed Markov Decision Processes (POMDPs). For brevity, we call the components of state of which we have direct observations the agent's "state", although we explicitly assume these states do not represent the full Markovian world state. Our agent's state at time t is s t ∈ R D; t = 0 refers to the current time step, and φ is all of the agent's observations. Variables are bolded. Random variables are capitalized. Absent subscripts denote all future time steps, e.g. S. = S 1:T ∈ R T ×D. We denote a probability density function of a random variable S as p(S), and its value as p(s). = p(S = s). To learn agent dynamics that are possible and preferred, we construct a model of expert behavior. We fit an "Imitative Model" q(S 1:T |φ) = T t=1 q(S t |S 1:t−1, φ) to a dataset of expert trajectories drawn from a (unknown) distribution of expert behavior s i ∼ p(S|φ i). By training q(S|φ) to forecast expert trajectories with high likelihood, we model the scene-conditioned expert dynamics, which can score trajectories by how likely they are to come from the expert. After training, q(S|φ) can generate trajectories that resemble those that the expert might generatee.g. trajectories that navigate roads with expert-like maneuvers. However, these maneuvers will not have a specific goal. Beyond generating human-like behaviors, we wish to direct our agent to goals and have the agent automatically reason about the necessary mid-level details. We define general tasks by a set of goal variables G. The probability of a plan s conditioned on the goal G is modelled by a posterior p(s|G, φ). This posterior is implemented with q(s|φ) as a learned imitation prior and p(G|s, φ) as a test-time goal likelihood. We give examples of p(G|s, φ) after deriving a maximum a posteriori inference procedure to generate expert-like plans that achieve abstract goals: We perform gradient-based optimization of Eq. 1, and defer this discussion to Appendix A. Next, we discuss several goal likelihoods, which direct the planning in different ways. They communicate goals they desire the agent to achieve, but not how to achieve them. The planning procedure determines how to achieve them by producing paths similar to those an expert would have taken to reach the given goal. In contrast to black-box one-step IL that predicts controls, our method produces interpretable multi-step plans accompanied by two scores. One estimates the plan's "expertness", the second estimates its probability to achieve the goal. Their sum communicates the plan's overall quality. Our approach can also be viewed as a learning-based method to integrate mid-level and high-level controllers together, where demonstrations from both are available at train-time, only the highlevel controller is available at test-time, and the high-level controller can vary. The high-level controller's action specifies a subgoal for the mid-level controller. A density model of future trajectories of an expert mid-level controller is learned at train-time, and is amenable to different types of direction as specified by the high-level controller. In this sense, the model is an "apprentice", having learned to imitate mid-level behaviors. In our application, the high-level controller is composed of an A * path-planning algorithm and one of a library of components that forms goal likelihoods from the waypoints produced by A *. Connecting this to related approaches, learning the midlevel controller (Imitative Model) resembles offline IL, whereas inference with an Imitative Model resembles trajectory optimization in MBRL, given goals provided by the high-level controller. Constraint-based planning to goal sets (hyperparameter-free): Consider the setting where we have access to a set of desired final states, one of which the agent should achieve. We can model this by applying a Dirac-delta distribution on the final state, to ensure it lands in a goal set G ⊂ R D: δ s T (G)'s partial support of s T ∈ G ⊂ R D constrains s T and introduces no hyperparameters into p(G|s, φ). For each choice of G, we have a different way to provide high-level task information to the agent. The simplest choice for G is a finite set of points: a (A) Final-State Indicator likelihood. We applied (A) to a sequence of waypoints received from a standard A * planner (provided by the CARLA simulator), and outperformed all prior dynamic-world CARLA methods known to us. We can also consider providing an infinite number of points. Providing a set of line-segments as G yields a (B) Line-Segment Final-State Indicator likelihood, which encourages the final state to land along one of the segments. Finally, consider a (C) Region Final-State Indicator likelihood in which G is a polygon (see Figs. 1 and 4). Solving Eq. 1 with (C) amounts to planning the most expert-like trajectory that ends inside a goal region. Appendix B provides derivations, implementation details, and additional visualizations. We found these methods to work well when G contains "expert-like" final position(s), as the prior strongly penalizes plans ending in non-expert-like positions. Unconstrained planning to goal sets (hyperparameter-based): Instead of constraining that the final state of the trajectory reach a goal, we can use a goal likelihood with full support (s T ∈ R D), centered at a desired final state. This lets the goal likelihood encourage goals, rather than dictate them. If there is a single desired goal (G = {g T}), the (D) Gaussian Final-State likelihood p(G|s, φ) ← N (g T ; s T, I) treats g T as a noisy observation of a final future state, and encourages the plan to arrive at a final state. We can also plan to K successive states G = (g T −K+1, . . ., g T) with a (E) Gaussian State Sequence: p(G|s, φ) ← T k=T −K+1 N (g k ; s k, I) if a program wishes to specify a desired end velocity or acceleration when reaching the final state g T (Fig. 2). Alternatively, a planner may propose a set of states with the intention that the agent should reach any one of them. This is ) and is useful if some of those final states are not reachable with an expert-like plan. Unlike A-C, D-F introduce a hyperparameter " ". However, they are useful when no states in G correspond to observed expert behavior, as they allow the imitation prior to be robust to poorly specified goals. Costed planning: Our model has the additional flexibility to accept arbitrary user-specified costs c at test-time. For example, we may have updated knowledge of new hazards at test-time, such as a given map of potholes or a predicted cost map. Cost-based knowledge c(s i |φ) can be incorporated as an (G) Energy-based likelihood: p(G|s, φ) ∝ T t=1 e −c(st|φ) . This can be combined with other goal-seeking objectives by simply multiplying the likelihoods together. Examples of combining G (energy-based) with F (Gaussian mixture) were shown in Fig. 1 and are shown in Fig. 3. Next, we describe instantiating q(S|φ) in CARLA . Designing general goal likelihoods can be considered a form of reward engineering if there are no restrictions on the goal likelihoods. This connection is best seen in (G), which has an explicit cost term. One reason why it is easier to design goal likelihoods than to design reward functions is that the task of evoking most aspects of goal-driven behavior is already learned by the prior q(s|φ), which models desirable behavior. This is in contrast to model-free RL, which entirely relies on the reward design to evoke goal-driven behavior, and in contrast to model-based RL, which heavily relies on the reward design to evoke goal-driven behavior, as its dynamics model learns what is possible, rather than what is desirable. Additionally, it is easy to design goal likelihoods when goals provide a significant amount of information that obviates the need to do any manual tuning. The main assumption is that one of the goals in the goal set is reachable within the model's time-horizon. In our autonomous driving application, we model the agent's state at time t as s t ∈ R D with D = 2; s t represents our agent's location on the ground plane. The agent has access to environment perception φ ← {s −τ :0, χ, λ}, where τ is the number of past positions we condition on, χ is a high-dimensional observation of the scene, and λ is a low-dimensional traffic light signal. χ could represent either LIDAR or camera images (or both), and is the agent's observation of the world. In our setting, we featurize LIDAR to χ = R 200×200×2, with χ ij representing a 2-bin histogram of points above and at ground level in a 0.5m 2 cell at position (i, j). CARLA provides ground-truth s −τ:0 and λ. Their availability is a realistic input assumption in perception-based autonomous driving pipelines. Model requirements: A deep imitative model forecasts future expert behavior. It must be able to compute q(s|φ)∀s ∈ R T ×D. The ability to compute ∇ s q(s|φ) enables gradient-based optimization for planning. provide a recent survey on forecasting agent behavior. As many forecasting methods cannot compute trajectory probabilities, we must be judicious in choosing q(S|φ). A model that can compute probabilities R2P2 , a generative autoregressive flow . We extend R2P2 to instantiate the deep imitative model q(S|φ). R2P2 was previously used to forecast vehicle trajectories: it was not demonstrated or developed to plan or execute controls. Although we used R2P2, other futuretrajectory density estimation techniques could be used -designing q(s|φ) is not the primary focus of this work. In R2P2, q θ (S|φ) is induced by an invertible, differentiable function:; f θ warps a latent sample from a base distribution Z ∼ q 0 = N (0, I) to S. θ is trained Figure 5: Illustration of our method applied to autonomous driving. Our method trains an imitative model from a dataset of expert examples. After training, the model is repurposed as an imitative planner. At test-time, a route planner provides waypoints to the imitative planner, which computes expert-like paths to each goal. The best plan is chosen according to the planning objective and provided to a low-level PID-controller in order to produce steering and throttle actions. This procedure is also described with pseudocode in Appendix A. to maximize q θ (S|φ) of expert trajectories. f θ is defined for 1..T as follows: where µ θ (S 1:t−1, φ) = 2S t−1 −S t−2 +m θ (S 1:t−1, φ) encodes a constant-velocity inductive bias. The m θ ∈ R 2 and σ θ ∈ R 2×2 are computed by expressive neural networks. The ing trajectory distribution is complex and multimodal (Appendix C.1 depicts samples). Because traffic light state was not included in the φ of R2P2's "RNN" model, it could not react to traffic lights. We created a new model that includes λ. It fixed cases where q(S|φ) exhibited no forward-moving preference when the agent was already stopped, and improved q(S|φ)'s stopping preference at red lights. We used T = 40 trajectories at 10Hz (4 seconds), and τ = 3. Fig. 12 in Appendix C depicts the architecture of µ θ and σ θ. We now instantiate a complete autonomous driving framework based on imitative models to study in our experiments, seen in Fig. 5. We use three layers of spatial abstraction to plan to a faraway destination, common to autonomous vehicle setups: coarse route planning over a road map, path planning within the observable space, and feedback control to follow the planned path . For instance, a route planner based on a conventional GPS-based navigation system might output waypoints roughly in the lanes of the desired direction of travel, but not accounting for environmental factors such as the positions of other vehicles. This roughly communicates possibilities of where the vehicle could go, but not when or how it could get to them, or any environmental factors like other vehicles. A goal likelihood from Sec. 2.2 is formed from the route and passed to the planner, which generates a state-space plan according to the optimization in Eq. 1. The ing plan is fed to a simple PID controller on steering, throttle, and braking. Pseudocode of the driving and inference algorithms are given in Algs 1 and 2. The PID algorithm is given in Appendix A. G ← ROUTEPLAN(φ) {Generate goals from a route} 4: φ ← ENVIRONMENT(u) {Execute control} 8: end for 9: end while 3 RELATED WORK A body of previous work has explored offline IL (Behavior Cloning -BC) in the CARLA simulator (; ; ; ; . These BC Algorithm 2 IMITATIVEPLAN(q θ, f, p, G, φ) 1: Initialize z 1:T ∼ q 0 2: while not converged do 3: approaches condition on goals drawn from a small discrete set of directives. Despite BC's theoretical drift shortcomings , these methods still perform empirically well. These approaches and ours share the same high-level routing algorithm: an A * planner on route nodes that generates waypoints. In contrast to our approach, these approaches use the waypoints in a Waypoint Classifier, which reasons about the map and the geometry of the route to classify the waypoints into one of several directives: {Turn left, Turn right, Follow Lane, Go Straight}. One of the original motivations for these type of controls was to enable a human to direct the robot . However, in scenarios where there is no human in the loop (i.e. autonomous driving), we advocate for approaches to make use of the detailed spatial information inherent in these waypoints. Our approach and several others we designed make use of this spatial information. One of these is CIL-States (CILS): whereas the approach in uses images to directly generate controls, CILS uses identical inputs and PID controllers as our method. With respect to prior conditional IL methods, our main approach has more flexibility to handle more complex directives post-training, the ability to learn without goal labels, and the ability to generate interpretable planned and unplanned trajectories. These contrasting capabilities are illustrated in Table 1. Our approach is also related to MBRL. MBRL can also plan with a predictive model, but its model only represents possible dynamics. The task of evoking expert-like behavior is offloaded to the reward function, which can be difficult and time-consuming to craft properly. We know of no MBRL approach previously applied to CARLA, so we devised one for comparison. This MBRL approach also uses identical inputs to our method, instead to plan a reachability tree over an dynamic obstacle-based reward function. See Appendix D for further details of the MBRL and CILS methods, which we emphasize use the same inputs as our method. Several prior works (; ;) used imitation learning to train policies that contain planning-like modules as part of the model architecture. While our work also combines planning and imitation learning, ours captures a distribution over possible trajectories, and then plan trajectories at test-time that accomplish a variety of given goals with high probability under this distribution. Our approach is suited to offline-learning settings where interactively collecting data is costly (time-consuming or dangerous). However, there exists online IL approaches that seek to be safe (; ;). We evaluate our method using the CARLA driving simulator . We seek to answer four primary questions: Can we generate interpretable, expert-like plans with offline learning and minimal reward engineering? Neither IL nor MBRL can do so. It is straightforward to interpret the trajectories by visualizing them on the ground plane; we thus seek to validate whether these plans are expert-like by equating expert-like behavior with high performance on the CARLA benchmark. Can we achieve state-of-the-art CARLA performance using resources commonly available in real autonomous vehicle settings? There are several differences between the approaches, as discussed in Sec 3 and shown in Tables 1 and 2. Our approach uses the CARLA toolkit's resources that are commonly available in real autonomous vehicle settings: waypoint-based routes (all prior approaches use these), LIDAR and traffic-light observations (both are CARLAprovided, but only the approaches we implemented use it). Furthermore, the two additional methods of comparison we implemented (CILS and MBRL) use the exact same inputs as our algorithm. These reasons justify an overall performance comparison to answer: whether we can achieve state-ofthe-art performance using commonly available resources. We advocate that other approaches also make use of such resources. How flexible is our approach to new tasks? We investigate by applying each of the goal likelihoods we derived and observing the ing performance. How robust is our approach to error in the provided goals? We do so by injecting two different types of error into the waypoints and observing the ing performance. We begin by training q(S|φ) on a dataset of 25 hours of driving we collected in Town01, detailed in Appendix C.2. Following existing protocol, each test episode begins with the vehicle starting in one of a finite set of starting positions provided by the CARLA simulator in Town01 or Town02 maps in one of two settings: static-world (no other vehicles) or dynamic-world (with other vehicles). We ran the same benchmark 3 times across different random seeds to quantify means and their standard errors. We construct the goal set G for the Final-State Indicator (A) directly from the route output by CARLA's waypointer. B's line segments are formed by connecting the waypoints to form a piecewise linear set of segments. C's regions are created a polygonal goal region around the segments of (B). Each represents an increasing level of coarseness of direction. Coarser directions are easier to specify when there is ambiguity in positions (both the position of the vehicle and the position of the goals). Further details are discussed in Appendix B.3. We use three metrics: (a) success rate in driving to the destination without any collisions (which all prior work reports); (b) red-light violations; and (c) proportion of time spent driving in the wrong lane and off road. With the exception of metric (a), lower numbers are better. Results: Towards questions and (expert-like plans and flexibility), we apply our approach with a variety of goal likelihoods to the CARLA simulator. Towards question, we compare our methods against CILS, MBRL, and prior work. These are shown in Table 3. The metrics for the methods we did not implement are from the aggregation reported in. We observe our method to outperform all other approaches in all settings: static world, dynamic world, training conditions, and test conditions. We observe the Goal Indicator methods are able to perform well, despite having no hyperparameters to tune. We found that we could further improve our approach's performance if we use the light state to define different goal sets, which defines a "smart" waypointer. The settings where we use this are suffixed with "S." in the Tables. We observed the planner prefers closer goals when obstructed, when the vehicle was already stopped, and when a red light was detected; we observed the planner prefers farther goals when unobstructed and when green lights or no lights were observed. Examples of these and other interesting behaviors are best seen in the videos on the website (https://sites.google.com/view/imitative-models). These behaviors follow from the method leveraging q(S|φ)'s internalization of aspects of expert behavior in order to reproduce them in new situations. Altogether, these provide affirmative answers to questions and. Towards question, these show that our approach is flexible to different directions defined by these goal likelihoods. Towards questions (flexibility) and (noise-robustness), we analyze the performance of our method when the path planner is heavily degraded, to understand its stability and reliability. We use the Gaussian Final-State Mixture goal likelihood. Navigating with high-variance waypoints. As a test of our model's capability to stay in the distribution of demonstrated behavior, we designed a "decoy waypoints" experiment, in which half of the waypoints are highly perturbed versions of the other half, serving as distractions for our Gaussian Final-State Mixture imitative planner. We observed surprising robustness to decoy waypoints. Examples of this robustness are shown in Fig. 6. In Table 4, we report the success rate and the mean number of planning rounds for failed episodes in the " 1 /2 distractors" row. These numbers indicate our method can execute dozens of planning rounds without decoy waypoints causing a catastrophic failure, and often it can execute the hundreds necessary to achieve the goal. See Appendix E for details. Navigating with waypoints on the wrong side of the road. We also designed an experiment to test our method under systemic bias in the route planner. Our method is provided waypoints on the wrong side of the road (in CARLA, the left side), and tasked with following the directions of these waypoints while staying on the correct side of the road (the right side). In order for the value of q(s|φ) to outweigh the influence of these waypoints, we increased the hyperparameter. We found our method to still be very effective at navigating, and report in Table 4. We also investigated providing very coarse 8-meter wide regions to the Region Final-State likelihood; these always include space in the wrong lane and off-road (Fig. 11 in Appendix B.4 provides visualization). Nonetheless, on Town01 Dynamic, this approach still achieved an overall success rate of 48%. Taken together towards question, our indicate that our method is fairly robust to errors in goal-specification. To further investigate our model's flexibility to test-time objectives (question 3), we designed a pothole avoidance experiment. We simulated potholes in the environment by randomly inserting them in the cost map near waypoints. We ran our method with a test-time-only cost map of the simulated potholes by combining goal likelihoods (F) and (G), and compared to our method that did not incorporate the cost map (using (F) only, and thus had no incentive to avoid potholes). We recorded the number of collisions with potholes. In Table 4, our method with cost incorporated avoided most potholes while avoiding collisions with the environment. To do so, it drove closer to the centerline, and occasionally entered the opposite lane. Our model internalized obstacle avoidance by staying on the road and demonstrated its flexibility to obstacles not observed during training. Fig. 7 shows an example of this behavior. See Appendix F for details of the pothole generation. We proposed "Imitative Models" to combine the benefits of IL and MBRL. Imitative Models are probabilistic predictive models able to plan interpretable expert-like trajectories to achieve new goals. Inference with an Imitative Model resembles trajectory optimization in MBRL, enabling it to both incorporate new goals and plan to them at test-time, which IL cannot. Learning an Imitative Model resembles offline IL, enabling it to circumvent the difficult reward-engineering and costly online data collection necessities of MBRL. We derived families of flexible goal objectives and showed our model can successfully incorporate them without additional training. Our method substantially outperformed six IL approaches and an MBRL approach in a dynamic simulated autonomous driving task. We showed our approach is robust to poorly specified goals, such as goals on the wrong side of the road. We believe our method is broadly applicable in settings where expert demonstrations are available, flexibility to new situations is demanded, and safety is paramount. Future work could investigate methods to handle both observation noise and out-of-distribution observations to enhance the applicability to robust real systems -we expand on this issue in Appendix E. Finally, to facilitate more general planning, future work could extend our approach to explicitly reason about all agents in the environment in order to inform a closed-loop plan for the controlled agent. In Algorithm 1, we provided pseudocode for receding-horizon control via our imitative model. In Algorithm 2 we provided pesudocode that describes how we plan in the latent space of the trajectory. In Algorithm 3, we detail the speed-based throttle and position-based steering PID controllers. Algorithm 3 PIDCONTROLLER(φ = {s 0, s −1, . . .}, s Since s 1:T = f (z 1:T) in our implementation, and f is differentiable, we can perform gradient descent of the same objective in terms of z 1:T, as shown in Algorithm 2.Since q is trained with z 1:T ∼ N (0, I), the latent space is likelier to be better numerically conditioned than the space of s 1:T, although we did not compare the two approaches formally. We implemented the following optimizations to improve this procedure's output and practical run time. 1) We started with N = 120 different z initializations, optimized them in batch, and returned the highest-scoring value across the entire optimization. 2) We observed the ing planning procedure to usually converge quickly, so instead of specifying a convergence threshold, we simply ran the optimization for a small number of steps, M = 10, and found that we obtained good performance. Better performance could be obtained by performing a larger number of steps. We now derive an approach to optimize our main objective with set constraints. Although we could apply a constrained optimizer, we find that we are able to exploit properties of the model and constraints to derive differentiable objectives that enable approximate optimization of the corresponding closed-form optimization problems. These enable us to use the same straightforward gradient-descent-based optimization approach described in Algorithm 2. Shorthand notation: In this section we omit dependencies on φ for brevity, and use short hand µ t. = µ θ (s 1:t−1) and Σ t. = Σ θ (s 1:t−1). For example, q(s t |s 1:t−1) = N (s t ; µ t, Σ t). Let us begin by defining a useful delta function: which serves as our goal likelihood when using goal with set constraints: p(G|s 1:T) ← δ S T (G). We now derive the corresponding maximum a posteriori optimization problem: arg max q(s t |s 1:t−1) = arg max arg max By exploiting the fact that q(s T |s 1:T −1) = N (s T ; µ T, Σ T), we can derive closed-form solutions for when G has special structure, which enables us to apply gradient descent to solve this constrainedoptimization problem (examples below). With a closed form solution to equation 6, we can easily compute equation 5 using unconstrained-optimization as follows: The solution to equation 6 in the case of a single desired goal g ∈ R D is simply: More generally, multiple point goals help define optional end points for planning: where the agent only need reach one valid end point (see Fig. 8 for examples), formulated as: We can form a goal-set as a finite-length line segment, connecting point a ∈ R D to point b ∈ R D: The solution to equation 6 in the case of line-segment goals is: Proof: To solve equation 15 is to find which point along the line g line (u) maximizes N (·; µ T, Σ T) subject to the constraint 0 ≤ u ≤ 1: Since L u is convex, the optimal value u * is value closest to the unconstrained arg max of L u (u), subject to 0 ≤ u ≤ 1: We now solve for u * R: which gives us: B.1.3 MULTIPLE-LINE-SEGMENT GOAL-SET: More generally, we can combine multiple line-segments to form piecewise linear "paths" we wish to follow. By defining a path that connects points line-segment, select the optimal segment i * = arg max i L i u, and use the segment i *'s solution to u * to compute s * T. Examples shown in Fig. 9. Instead of a route or path, a user (or program) may wish to provide a general region the agent should go to, and state within that region being equally valid. Polygon regions (including both boundary and interior) offer closed form solution to equation 6 and are simple to specify. A polygon can be specified by an ordered sequence of vertices (p 0, p 1, . .., p N) ∈ R N ×2. Edges are then defined as the sequence of line-segments between successive vertices (and a final edge between first and last vertex): ((p 0, p 1),..., (p N −1, p N), (p N, p 0) ). Examples shown in Fig. 10 and 11. Solving equation 6 with a polygon has two cases: depending whether µ T is inside the polygon, or outside. If µ T lies inside the polygon, then the optimal value for s * T that maximizes N (s * T ; µ T, Σ T) is simply µ T: the mode of the Gaussian distribution. Otherwise, if µ T lies outside the polygon, then the optimal value s * T will lie on one of the polygon's edges, solved using B.1.3. The waypointer uses the CARLA planner's provided route to generate waypoints. In the constrainedbased planning goal likelihoods, we use this route to generate waypoints without interpolating between them. In the relaxed goal likelihoods, we interpolate this route to every 2 meters, and use the first 20 waypoints. As mentioned in the main text, one variant of our approach uses a "smart" waypointer. This waypointer simply removes nearby waypoints closer than 5 meters from the vehicle when a green light is observed in the measurements provided by CARLA, to encourage the agent to move forward, and removes far waypoints beyond 5 meters from the vehicle when a red light is observed in the measurements provided by CARLA. Note that the performance differences between our method without the smart waypointer and our method with the smart waypointer are small: the only signal in the metrics is that the smart waypointer improves the vehicle's ability to stop for red lights, however, it is quite adept at doing so without the smart waypointer. Given the in-lane waypoints generated by CARLA's route planner, we use these to create Point goal sets, Line-Segment goal sets, and Polygon Goal-Sets, which respectively correspond to the (A) Final-State Indicator, (B) Line-Segment Final-State Indicator, and (C) Final-State Region Indicator described in Section 2.2. For (A), we simply feed the waypoints directly into the Final-State Indicator, which in a constrained optimization to ensure that. We also included the vehicle's current position in the goal set, in order to allow it to stop. The gradient-descent based optimization is then formed from combining Eq. 8 with Eq. 12. The gradient to the nearest goal of the final state of the partially-optimized plan encourage the optimization to move the plan closer to that goal. We used K = 10. We applied the same procedure to generate the goal set for the (B) Line Segment indicator, as the waypoints returned by the planner are ordered. Finally, for the (C) Final-State Region Indicator (polygon), we used the ordered waypoints as the "skeleton" of a polygon that surrounds. It was created by adding a two vertices for each point v t in the skeleton at a distance 1 meter from v t perpendicular to the segment connecting the surrounding points (v t−1, v t+1). This ed in a goal set G polygon ⊃ G line-segment, as it surrounds the line segments. The (F) Gaussian Final-State Mixture goal set was constructed in the same way as (A), and also used when the pothole costs were added. For the methods we implemented, the task is to drive the furthest road location from the vehicle's initial position. Note that this protocol more difficult than the one used in prior work;;;; , which has no distance guarantees between start positions and goals, and often in shorter paths. Visualizations of examples of our method deployed with different goal likelihoods are shown in Fig. 8, Fig. 9, Fig. 10, and Fig. 11. The architecture of q(S|φ) is shown in Table 5. C.2 DATASET Before training q(S|φ), we ran CARLA's expert in the dynamic world setting of Town01 to collect a dataset of examples. We have prepared the dataset of collected data for public release upon publication. We ran the autopilot in Town01 for over 900 episodes of 100 seconds each in the presence of 100 other vehicles, and recorded the trajectory of every vehicle and the autopilot's LIDAR observation. We randomized episodes to either train, validation, or test sets. We created sets of 60,701 train, 7586 validation, and 7567 test scenes, each with 2 seconds of past and 4 seconds of future position information at 10Hz. The dataset also includes 100 episodes obtained by following the same procedure in Town02. We designed a conditional imitation learning baseline that predicts the setpoint for the PID-controller. Each receives the same scene observations (LIDAR) and is trained with the same set of trajectories as our main method. It uses nearly the same architecture as that of the original CIL, except it outputs setpoints instead of controls, and also observes the traffic light information. We found it very effective for stable control on straightaways. When the model encounters corners, however, prediction is more difficult, as in order to successfully avoid the curbs, the model must implicitly plan a safe path. We found that using the traffic light information allowed it to stop more frequently. Static-world To compare against a purely model-based reinforcement learning algorithm, we propose a model-based reinforcement learning baseline. This baseline first learns a forwards dynamics model s t+1 = f (s t−3:t, a t) given observed expert data (a t are recorded vehicle actions). We use an MLP with two hidden layers, each 100 units. Note that our forwards dynamics model does not imitate the expert preferred actions, but only models what is physically possible. Together with the same LIDAR map χ our method uses to locate obstacles, this baseline uses its dynamics model to plan a reachability tree through the free-space to the waypoint while avoiding obstacles. The planner opts for the lowest-cost path that ends near the goal C(s 1:T ; g T) = ||s T − g T || 2 + T t=1 c(s t), where cost of a position is determined by c(s t) = 1.51(s t < 1 meters from any obstacle) + 0.751(1 <= s t < 2 meters from any obstacle) +... We plan forwards over 20 time steps using a breadth-first search over CARLA steering angle {−0.3, −0.1, 0., 0.1, 0.3}, noting valid steering angles are normalized to [−1, 1], with constant throttle at 0.5, noting the valid throttle range is. Our search expands each state node by the available actions and retains the 50 closest nodes to the waypoint. The planned trajectory efficiently reaches the waypoint, and can successfully plan around perceived obstacles to avoid getting stuck. To convert the LIDAR images into obstacle maps, we expanded all obstacles by the approximate radius of the car, 1.5 meters. We use the same setup as the Static-MBRL method, except we add a discrete temporal dimension to the search space (one R 2 spatial dimension per T time steps into the future). All static obstacles remain static, however all LIDAR points that were known to collide with a vehicle are now removed: and replaced at every time step using a constant velocity model of that vehicle. We found that the main failure mode was due to both to inaccuracy in constant velocity prediction as well as the model's inability to perceive lanes in the LIDAR. The vehicle would sometimes wander into the opposing traffic's lane, having failed to anticipate an oncoming vehicle blocking its path. In the decoy waypoints experiment, the perturbation distribution is N (0, σ = 8m): each waypoint is perturbed with a standard deviation of 8 meters. One failure mode of this approach is when decoy waypoints lie on a valid off-route path at intersections, which temporarily confuses the planner about the best route. Additional visualizations are shown in Fig. 14. Besides using our model to make a best-effort attempt to reach a user-specified goal, the fact that our model produces explicit likelihoods can also be leveraged to test the reliability of a plan by evaluating whether reaching particular waypoints will in human-like behavior or not. This capability can be quite important for real-world safety-critical applications, such as autonomous driving, and can be used to build a degree of fault tolerance into the system. We designed a classification experiment to evaluate how well our model can recognize safe and unsafe plans. We planned our model to known good waypoints (where the expert actually went) and known bad waypoints (off-road) on 1650 held-out test scenes. We used the planning criterion to classify these as good and bad plans and found that we can detect these bad plans with 97.5% recall and 90.2% precision. This indicates imitative models could be effective in estimating the reliability of plans. We determined a threshold on the planning criterion by single-goal planning to the expert's final location on offline validation data and setting it to the criterion's mean minus one stddev. Although a more intelligent calibration could be performed by analyzing the information retrieval statistics on the offline validation, we found this simple calibration to yield reasonably good performance. We used 1650 test scenes to perform classification of plans to three different types of waypoints 1) where the expert actually arrived at time T (89.4% reliable), 2) waypoints 20m ahead along the waypointer-provided route, which are often near where the expert arrives (73.8% reliable) 3) the same waypoints from 2), shifted 2.5m off of the road (2.5% reliable). This shows that our learned model exhibits a strong preference for valid waypoints. Therefore, a waypointer that provides expert waypoints via 1) half of the time, and slightly out-of-distribution waypoints via 3) in the other half, an "unreliable" plan classifier achieves 97.5% recall and 90.2% precision. The existence of both observation noise and uncertain/out-of-distribution observations is an important practical issue for autonomous vehicles. Although our current method only conditions on our current observation, several extensions could help mitigate the negative effects of both and. For, a Bayesian filtering formulation is arguably most ideal, to better estimate (and track) the location of static and dynamic obstacles under noise. However, such high-dimensional filtering are often intractable, and might necessitate approximate Bayesian deep learning techniques, RNNs, or frame stacking, to benefit from multiple observations. Addressing would ideally be done by placing a prior over our neural network weights, to derive some measure of confidence in our density estimation of how expert each plan is, such that unfamiliar scenes generate large uncertainty on our density estimate that we could detect, and react cautiously (pessimistically) to. One way to address the situation if the distributions are very different is to adopt an ensembling approach in order for the method to determine when the inputs are out of distribution -the ensemble will usually have higher variance (i.e. disagree) when each element of the ensemble is provided with an out-of-distribution input. For instance, this variance could be used as a penalization in the planning criterion. As discussed, our model assumes access to the traffic-light state provided by the simulator, which we call λ. However, access to this state would be noisy in practice, because it relies on a sensor-based (usually image-based) detection and classification module. We performed an experiment to assess robustness to noise in λ: we simulated noise in λ by "flipping" the light state with 20% probability, corresponding to a light state detector that has 80% accuracy on average. "Flipping" means that if the light is green, then changingλ to indicate red, and if the light is red, then changing λ to indicate green. We performed this following the experimental method of "Region Final-St. Indicator S." in dynamic Town02, and ran it with three separate seeds. The means and their standard errors are reported in Table 6. The we draw is that the approach can still achieve success most of the time, although it tends to violate red-lights more often. Qualitatively, we observed the ing behavior near intersections to sometimes be "jerky", with the model alternating between stopping and non-stopping plans. We hypothesize that the model itself could be made more robust if the noise in λ was also present in the training data. Table 6: We evaluate the effect of noise in the traffic-light state (λ) on CARLA's Dynamic Navigation task. Noise in the light state predictably degrades overall and red-light performance, but not to the point of preventing the method from operating at all. Success Ran Red Light Wrong lane Off road Region Final-St. Indicator S. (original) 88% ± 3.3 2.57% ± 0.04 0.49% ± 0.32 2.6% ± 1.06 Region Final-St. Indicator S. (noisy λ) 76% ± 5.0 34.8% ± 2.4 0.15% ± 0.04 1.79% ± 0.34 We simulated potholes in the environment by randomly inserting them in the cost map near each waypoint i with offsets distributed N i (µ= [−15m, 0m], Σ = diag([1, 0.01])), (i.e. mean-centered on the right side of the lane 15m before each waypoint). We inserted pixels of root cost −1e3 in the cost map at a single sample of each N i, binary-dilated the cost map by 1 /3 of the lane-width (spreading the cost to neighboring pixels), and then blurred the cost map by convolving with a normalized truncated Gaussian filter of σ = 1 and truncation width 1. See Fig. 15 for a visualization of our baseline methods. In order to tune the hyperparameter of the unconstrained likelihoods, we undertook the following binary-search procedure. When the prior frequently overwhelmed the posterior, we set ← 0.2, to yield tighter covariances, and thus more penalty for failing to satisfy the goals. When the posterior frequently overwhelmed the prior, we set ← 5, to yield looser covariances, and thus less penalty for failing to satisfy the goals. We executed this process three times: once for the "Gaussian Final-State Mixture" experiments (Section 4), once for the "Noise Robustness" Experiments (Section 4.1), and once for the pothole-planning experiments (Section 4.2). Note that the Constrained-Goal Likelihoods introduced no hyperparameters to tune.' should have a representation like 'In this paper, we propose Imitative Models to combine the benefits of IL and goal-directed planning: probabilistic predictive models of desirable behavior able to plan interpretable expert-like trajectories to achieve specified goals.' but very far from 'We view exploration in RL as a problem of matching a marginal distribution over states.'.

Question: Our main motivation is to propose an efficient approach to generate novel multi-element stable chemical compounds that can be used in real world applications. This task can be formulated as a combinatorial problem, and it takes many hours of human experts to construct, and to evaluate new data. Unsupervised learning methods such as Generative Adversarial Networks (GANs) can be efficiently used to produce new data. Cross-domain Generative Adversarial Networks were reported to achieve exciting in image processing applications. However, in the domain of materials science, there is a need to synthesize data with higher order complexity compared to observed samples, and the state-of-the-art cross-domain GANs can not be adapted directly. In this contribution, we propose a novel GAN called CrystalGAN which generates new chemically stable crystallographic structures with increased domain complexity. We introduce an original architecture, we provide the corresponding loss functions, and we show that the CrystalGAN generates very reasonable data. We illustrate the efficiency of the proposed method on a real original problem of novel hydrides discovery that can be further used in development of hydrogen storage materials. In modern society, a big variety of inorganic compositions are used for hydrogen storage owing to its favorable cost BID4. A vast number of organic molecules are applied in solar cells, organic light-emitting diodes, conductors, and sensors BID25. Synthesis of new organic and inorganic compounds is a challenge in physics, chemistry and in materials science. Design of new structures aims to find the best solution in a big chemical space, and it is in fact a combinatorial optimization problem. In this work, we focus on applications of hydrogen storage, and in particular, we challenge the problem to investigate novel chemical compositions with stable crystals. Traditionally, density functional theory (DFT) plays a central role in prediction of chemically relevant compositions with stable crystals BID22. However, the DFT calculations are computationally expensive, and it is not acceptable to apply it to test all possible randomly generated structures. A number of machine learning approaches were proposed to facilitate the search for novel stable compositions BID3. There was an attempt to find new compositions using an inorganic crystal structure database, and to estimate the probabilities of new candidates based on compositional similarities. These methods to generate relevant chemical compositions are based on recommender systems BID10. The output of the recommender systems applied in the crystallographic field is a rating or preference for a structure. A recent approach based on a combination of machine learning methods and the high-throughput DFT calculations allowed to explore ternary chemical compounds BID21, and it was shown that statistical methods can be of a big help to identify stable structures, and that they do it much faster than standard methods. Recently, support vector machines were tested to predict crystal structures BID16 showing that the method can reliably predict the crystal structure given its composition. It is worth mentioning that data representation of observations to be passed to a learner, is critical, and data representations which are the most suitable for learning algorithms, are not necessarily scientifically intuitive BID23.Deep learning methods were reported to learn rich hierarchical models over all kind of data, and the GANs BID8 ) is a state-of-the-art model to synthesize data. Moreover, deep networks were reported to learn transferable representations BID18. The GANs were already exploited with success in cross-domain learning applications for image processing BID13 BID12.Our goal is to develop a competitive approach to identify stable ternary chemical compounds, i.e., compounds containing three different elements, from observations of binary compounds. Nowadays, there does not exist any approach that can be applied directly to such an important task of materials science. The state-of-the-art GANs are limited in the sense that they do not generate samples in domains with increased complexity, e.g., the application where we aim to construct crystals with three elements from observations containing two chemical elements only. An attempt to learn many-to-many mappings was recently introduced by BID0, however, this promising approach does not allow to generate data of a higher-order dimension. Our contribution is multi-fold:• To our knowledge, we are the first to introduce a GAN to solve the scientific problem of discovery of novel crystal structures, and we introduce an original methodology to generate new stable chemical compositions; • The proposed method is called CrystalGAN, and it consists of two cross-domain GAN blocks with constraints integrating prior knowledge including a feature transfer step; • The proposed model generates data with increased complexity with respect to observed samples; • We demonstrate by numerical experiments on a real challenge of chemistry and materials science that our approach is competitive compared to existing methods; • The proposed algorithm is efficiently implemented in Python, and it will be publicly available shortly, as soon as the contribution is de-anonymized. This paper is organized as follows. We discuss the related work in Section 2. In Section 3, we provide the formalisation of the problem, and introduce the CrystalGAN. The of our numerical experiments are discussed in Section 4. Concluding remarks and perspectives close the paper. Our contribution is closely related to the problems of unsupervised learning and cross-domain learning, since our aim is to synthesize novel data, and the new samples are supposed to belong to an unobserved domain with an augmented complexity. In the adversarial nets framework, the deep generative models compete with an adversary which is a discriminative model learning to identify whether an observation comes from the model distribution or from the data distribution BID7. A classical GAN consists of two models, a generator G whose objective is to synthesize data and a discriminator D whose aim is to distinguish between real and generated data. The generator and the discriminator are trained simultaneously, and the training problem is formulated as a two-player minimax game. A number of techniques to improve training of GANs were proposed by; BID9; BID19.Learning cross domain relations is an active research direction in image processing. Several recent papers BID13 BID0 discuss an idea to capture some particular characteristics of one image and to translate them into another image. A conditional GAN for image-to-image translation is considered by. An advantage of the conditional model is that it allows to integrate underlying structure into the model. The conditional GANs were also used for multi-model tasks BID15 ). An idea to combine observed data to produce new data was proposed in BID26, e.g., an artist can mix existing pieces of music to create a new one. First domain, H is hydrogen, and A is a metal BH:Second domain, H is hydrogen, and B is another metal DISPLAYFORM0 Generator function that translates input features xAH from (domain) AH to BH GBHA 1:Generator function that translates input features xBH from (domain) BH to AH DAH and DBH:Discriminator functions of AH domain and BH domain, respectively AHB1:xAHB 1 is a sample generated by generator function GAHB 1 BHA1:yBHA 1 is a sample produced by generator function GBHA 1 AHBA1 and BHAB1: Data reconstructed after two generator translations AHBg and BHAg:Data obtained after feature transfer step from domain AH to domain BH, and from domain BH to domain AH, respectively Input data for the second step of CrystalGAN GAHB 2:Generator function that translates xAHB g Features generated in the first step from AHBg to AHB2 GBHA 2:Generator function that translates yBHA g Data generated in first step from BHAg to BHA2 DAHB and DBHA:The discriminator functions of domain AHBg and domain BHAg, respectively AHB2:xAHB 2 is a sample generated by the generator function GAHB 2 BHA2:yBHA 2 is a sample produced by the generator function GBHA 2 AHBA2 and BHAB2:Data reconstructed as a of two generators translations AHB2 and BHA2:Final new data (to be explored by human experts) An approach to learn high-level semantic features, and to train a model for more than a single task, was introduced by BID18. In particular, it was proposed to train a model to jointly learn several complementary tasks. This method is expected to overcome the problem of overfitting to a single task. An idea to introduce multiple discriminators whose role varies from formidable adversary to forgiving teacher was discussed by BID5.Several GANs were adapted to some materials science and chemical applications. So, ObjectiveReinforced GANs that perform molecular generation of carbon-chain sequence taking into consideration some desired properties, were introduced in BID20, and the method was shown to be efficient for drug discovery. Another avenue is to integrate rule-based knowledge, e.g., molecular descriptors with the deep learning. ChemNet BID6 ) is a deep neural network pre-trained with chemistry-relevant representations obtained from prior knowledge. The model can be used to predict new chemical properties. However, as we have already mentioned before, none of these methods generates crystal data of augmented complexity. In this section, we introduce our approach. The CrystalGAN consists of three procedures:1. First step GAN which is closely related to the cross-domain GANs, and that generates pseudo-binary samples where the domains are mixed. 2. Feature transfer procedure constructs higher order complexity data from the samples generated at the previous step, and where components from all domains are well-separated. 3. Second step GAN synthesizes, under geometric constraints, novel ternary stable chemical structures. First, we describe a cross-domain GAN, and then, we provide all the details on the proposed CrystalGAN. We provide all notations used by the CrystalGAN in TAB0. The GANs architectures for the first and the second steps are shown on FIG0. We now propose a novel architecture based on the cross-domain GAN algorithms with constraint learning to discover higher order complexity crystallographic systems. We introduce a GAN model to find relations between different crystallographic domains, and to generate new materials. To make the paper easier to follow, without loss of generality, we will present our method providing a specific example of generating ternary hydride compounds of the form "A (a metal) -H (hydrogen) -B (a metal)".Our observations are stable binary compounds containing chemical elements A+H which is a composition of some metal A and the hydrogen H, and B+H which is a mixture of another metal B with the hydrogen. So, a machine learning algorithm has access to observations {(x AHi)} DISPLAYFORM0 Our goal is to generate novel ternary, i.e. more complex, stable data x AHB (or y BHA) based on the properties learned from the observed binary structures. Below we describe the architecture of the CrystalGAN. Our approach consists of two consecutive steps with a feature transfer procedure inbetween. The first step of CrystalGAN generates new data with increased complexity. The adversarial network takes DISPLAYFORM0, and synthesizes DISPLAYFORM1 and FIG0 summarizes the first step of CrystalGAN. DISPLAYFORM2 The reconstruction loss functions take the following form: DISPLAYFORM3 Ideally, L R AH = 0, L R BH = 0, and x AHBA1 = x AH, y BHAB1 = y BH, and we minimize the distances d(x AHBA1, x AH) and d(y BHAB1, y BH).The generative adversarial loss functions of the first step of CrystalGAN aim to control that the original observations are reconstructed as accurate as possible: DISPLAYFORM4 DISPLAYFORM5 The generative loss functions contain the two terms defined above: DISPLAYFORM6 The discriminative loss functions aim to discriminate the samples coming from AH and BH: DISPLAYFORM7 DISPLAYFORM8 Now, we have all elements to define the full generative loss function of the first step: DISPLAYFORM9 where λ 1, λ 2, λ 3, and λ 4 are real-valued hyper-parameters that control the ratio between the corresponding terms, and the hyper-parameters are to be fixed by cross-validation. The full discriminator loss function of this step L D1 is defined as follows: DISPLAYFORM10 The first step generates pseudo-binary samples M H, where M is a new discovered domain merging A and B properties. Although these can be interesting for human experts, the samples generated by the first step are not easy to interpret, since the domains A and B are completely mixed in these samples, and there is no way to deduce characteristics of two separate elements coming from these domains. So, we need a second step which will generate data of a higher order complexity from two given domains. We transfer the attributes of A and B elements, this procedure is also shown on FIG0, in order to construct a new dataset that will be used as a training set in the second step of the CrystalGAN.In order to prepare the datasets to generate higher order complexity samples, we add a placeholder. (E.g., for domain AH, the fourth matrix is empty, and for domain BH, the third matrix is empty.) The second step GAN takes as input the data generated by the first step GAN and modified by the feature transfer procedure. The of the second step are samples which describe ternary chemical compounds that are supposed to be stable from chemical viewpoint. The geometric constraints control the quality of generated data. A crystallographic structure is fully described by a local distribution. This distribution is determined by distances to all nearest neighbors of each atom in a given crystallographic structure. We enforce the second step GAN with the following geometric constraints which satisfy the geometric conditions of our scientific domain application. The implemented constraints are also shown on FIG0. DISPLAYFORM0 be the set of distances of the first neighbors of all atoms in a crystallographic structure. There are two geometric constraints to be considered while generating new data. The first geometric (geo) constraint is defined as follows: DISPLAYFORM1 where d 1 is the minimal distance between two first nearest neighbors in a given crystallographic structure. The second geometric constraint takes the following form: DISPLAYFORM2 where d 2 is the maximal distance between two first nearest neighbors. The loss function of the second step GAN is augmented by the following geometric constraints: DISPLAYFORM3 Given x AHBg and y BHAg from the previous step, we generate: DISPLAYFORM4 The reconstruction loss functions are given: DISPLAYFORM5 DISPLAYFORM6 The generative adversarial loss functions are given by: DISPLAYFORM7 DISPLAYFORM8 The generative loss functions of the this step are defined as follows: DISPLAYFORM9 DISPLAYFORM10 The losses of the discriminator of the second step can be defined: DISPLAYFORM11 DISPLAYFORM12 Now, we have all elements to define the full generative loss function: DISPLAYFORM13 where λ 1, λ 2, λ 3, λ 4, λ 5, and λ 6 are real-valued hyper-parameters that control the influence of the terms. The full discriminative loss function of the second step L D2 takes the form: DISPLAYFORM14 To summarise, in the second step, we use the dataset issued from the feature transfer as an input containing two domains x AHBg and y BHAg. We train the cross-domain GAN taking into consideration constraints of the crystallographic environment. We integrated geometric constraints proposed by crystallographic and materials science experts to satisfy environmental constraints, and to increase the rate of synthesized stable ternary compounds. The second step is drafted on FIG0. Crystallographic structures can be represented using the POSCAR files which are input files for the DFT calculations under the VASP code BID14. These are coordinate files, they contain the lattice geometry and the atomic positions, as well as the number (or the composition) and the nature of atoms in the crystal unit cell. We use a dataset constructed from BID2 ) by experts in materials science. Our training data set contains the POSCAR files, and the proposed CrystalGAN generates also POSCAR files. Such a file contains three matrices: the first one is abc matrix, corresponding to the three lattice vectors defining the unit cell of the system, the second matrix contains atomic positions of H atom, and the third matrix contains coordinates of metallic atom A (or B).The information from the files is fed into 4-dimensional tensors. An example of a POSCAR file, and its corresponding representation for the GANs is shown on FIG1. On the same figure on the right we show the corresponding structure in 3D. Note that we increase the data complexity by the feature transfer procedure by adding placeholders. Our training dataset includes 1,416 POSCAR files of binary hydrides divided into 63 classes where each class is represented as a 4-dimensional tensor. Each class of binary M H hydride contains two elements: the hydrogen H and another element M from the periodic table. In our experiments, after discussions with materials science researchers, we focused on exploration of ternary compositions "Palladium -Hydrogen -Nickel" from the binary systems observations of "Palladium -Hydrogen" and "Nickel -Hydrogen". So, AH = PdH, and BH = NiH. We also considered another task to generate ternary compounds "Magnesium -Hydrogen -Titanium".In the CrystalGAN, we need to compute all the distances of the nearest neighbors for each generated POSCAR file. The distances between hydrogen atoms H in a given crystallographic structure should respect some geometric rules, as well as the distances between the atoms A−B, A−A', and B −B. We applied the geometric constraints on the distances between the neighbors (for each atom in a crystallographic structure) introduced in the previous section. Note that the distances A−H and B−H are not penalized by the constraints. In order to compute the distances between all nearest neighbors in the generated data, we used the pythonic library Pymatgen BID17 specifically developed for material analysis. For all experiments in this paper, the distances are fixed by our colleagues in crystallographic and materials science to d 1 = 1.8Å (angstrom, 10 −10 meter) and d 2 = 3Å. We set all the hyperparameters by cross validation, however, we found that a reasonable performance is reached when all λ i have similar values, and are quite close to 1. We use the standard AdamOptimizer with learning rate α = 0.0001, and β 1 = 0.5. The number of epochs is set to 1000 (we verified that the functions converge). The mini-batch size equals 35. DISPLAYFORM0 without constraints with constraints Pd -Ni -H 0 0 4 9 Mg -Ti -H 0 0 2 8 Table 2: Number of ternary compositions of good quality generated by the tested methods. Each block of the CrystalGAN architecture (the generators and the discriminators) is a multi-layer neural network with 5 hidden layers. Each layer contains 100 units. We use the rectified linear unit (ReLU) as an activation function of the neural network. All these parameters were fixed by cross-validation (for both chosen domains "Palladium -Hydrogen" and "Nickel -Hydrogen").Our code is implemented in Python (TensorFlow). We run the experiments using GPU with graphics card NVIDIA Quadro M5000. In our numerical experiments, we compare the proposed CrystalGAN with a classical GAN, the DiscoGAN BID13, and the CrystalGAN but without the geometric constraints. All these GANs generate POSCAR files, and we evaluate the performance of the models by the number of generated ternary structures which satisfy the geometric crystallographic environment. Table 2 shows the number of successes for the considered methods. The classical GAN which takes Gaussian noise as an input, does not generate acceptable chemical structures. The DiscoGAN approach performs quite well if we use it to generate novel pseudo-binary structures, however, it is not adapted to synthesize ternary compositions. We observed that the CrystalGAN (with the geometric constraints) outperforms all tested methods. From multiple discussions with experts in materials science and chemistry, first, we know that the number of novel stable compounds can not be very high, and it is already considered as a success if we synthesize several stable structures which satisfy the constraints. Hence, we can not really reason in terms of accuracy or error rate which are widely used metrics in machine learning and data mining. Second, evaluation of a stable structure is not straightforward. Given a new composition, only the of density functional theory (DFT) calculations can provide a whether this composition is stable enough, and whether it can be used in practice. However, the DFT calculations are computationally too expensive, and it is out of question to run them on all data we generated using the CrystalGAN. It is planned to run the DFT calculations on some pre-selected generated ternary compositions to take a final decision on practical utility of the chemical compounds. Our goal was to develop a principled approach to generate new ternary stable crystallographic structures from observed binary, i.e. containing two chemical elements only. We propose a learning method called CrystalGAN to discover cross-domain relations in real data, and to generate novel structures. The proposed approach can efficiently integrate, in form of constraints, prior knowledge provided by human experts. CrystalGAN is the first GAN developed to generate scientific data in the field of materials science. To our knowledge, it is also the first approach which generates data of a higher-order complexity, i.e., ternary structures where the domains are well-separated from observed binary compounds. The CrystalGAN was, in particular, successfully tested to tackle the challenge to discover new materials for hydrogen storage. Currently, we investigate different GANs architectures, also including elements of reinforcement learning, to produce data even of a higher complexity, e.g., compounds containing four or five chemical elements. Note that although the CrystalGAN was developed and tested for applications in materials science, it is a general method where the constraints can be easily adapted to any scientific problem.

Represent the TLDR for finding the corresponding scientific article

"Generating new chemical materials using novel cross-domain GANs."

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.