Question: Represent the article for finding a single-sentence TLDR:

Question: Deep learning natural language processing models often use vector word embeddings, such as word2vec or GloVe, to represent words. A discrete sequence of words can be much more easily integrated with downstream neural layers if it is represented as a sequence of continuous vectors. Also, semantic relationships between words, learned from a text corpus, can be encoded in the relative configurations of the embedding vectors. However, storing and accessing embedding vectors for all words in a dictionary requires large amount of space, and may stain systems with limited GPU memory. Here, we used approaches inspired by quantum computing to propose two related methods, word2ket and word2ketXS, for storing word embedding matrix during training and inference in a highly efficient way. Our approach achieves a hundred-fold or more reduction in the space required to store the embeddings with almost no relative drop in accuracy in practical natural language processing tasks. Modern deep learning approaches for natural language processing (NLP) often rely on vector representation of words to convert discrete space of human language into continuous space best suited for further processing through a neural network. For a language with vocabulary of size d, a simple way to achieve this mapping is to use one-hot representation -each word is mapped to its own row of a d × d identity matrix. There is no need to actually store the identity matrix in memory, it is trivial to reconstruct the row from the word identifier. Word embedding approaches such as word2vec or GloVe use instead vectors of dimensionality p much smaller than d to represent words, but the vectors are not necessarily extremely sparse nor mutually orthogonal. This has two benefits: the embeddings can be trained on large text corpora to capture the semantic relationship between words, and the downstream neural network layers only need to be of width proportional to p, not d, to accept a word or a sentence. We do, however, need to explicitly store the d × p embedding matrix in GPU memory for efficient access during training and inference. Vocabulary sizes can reach d = 10 5 or 10 6 , and dimensionality of the embeddings used in current systems ranges from p = 300 to p = 1024 . The d × p embedding matrix thus becomes a substantial, often dominating, part of the parameter space of a learning model. In classical computing, information is stored in bits -a single bit represents an element from the set B = {0, 1}, it can be in one of two possible states. A quantum equivalent of a bit, a qubit, is fully described by a single two-dimensional complex unit-norm vector, that is, an element from the set C 2. A state of an n-qubit quantum register corresponds to a vector in C 2 n. To have exponential dimensionality of the state space, though, the qubits in the register have to be interconnected so that their states can become entangled; a set of all possible states of n completely separated, independent qubits can be fully represented by C 2n instead of C 2 n. Entanglement is a purely quantum phenomenon -we can make quantum bits interconnected, so that a state of a two-qubit system cannot be decomposed into states of individual qubits. We do not see entanglement in classical bits, which are always independent -we can describe a byte by separately listing the state of each of the eight bits. We can, however, approximate quantum register classically -store vectors of size m using O (log m) space, at the cost of losing the ability to express all possible m-dimensional vectors that an actual O (log m)-qubit quantum register would be able to represent. As we show in this paper, the loss of representation power does not have a significant impact on NLP machine learning algorithms that use the approximation approaches to store and manipulate the high-dimensional word embedding matrix. Here, we used approaches inspired by quantum computing to propose two related methods, word2ket and word2ketXS, for storing word embedding matrix during training and inference in a highly efficient way 1. The first method operates independently on the embedding of each word, allowing for more efficient processing, while the second method operates jointly on all word embeddings, offering even higher efficiency in storing the embedding matrix, at the cost of more complex processing. Empirical evidence from three NLP tasks shows that the new word2ket embeddings offer high space saving rate at little cost in terms of accuracy of the downstream NLP model. Consider two separable 2 Hilbert spaces V and W. A tensor product space of V and W, denoted as V ⊗ W, is a separable Hilbert space H constructed using ordered pairs v ⊗ w, where v ∈ V and w ∈ W. In the tensor product space, the addition and multiplication in H have the following properties The inner product between v ⊗ w and v ⊗ w is defined as a product of individual inner products v ⊗ w, v ⊗ w = v, v w, w. It immediately follows that ||v ⊗ w|| = ||v|| ||w||; in particular, a tensor product of two unit-norm vectors, from V and W, respectively, is a unit norm vector in V ⊗ W. The Hilbert space V ⊗ W is a space of equivalence classes of pairs v ⊗ w; for example {cv} ⊗ w and v ⊗ {cw} are equivalent ways to write the same vector. A vector in a tensor product space is often simply called a tensor. Let {ψ j} and {φ k} be orthonormal basis sets in V and W, respectively. From eq. 1 and 2 we can see that where δ z is the Kronecker delta, equal to one at z = 0 and to null elsewhere. That is, the set {ψ j ⊗ φ k} jk forms an orthonormal basis in V ⊗ W, with coefficients indexed by pairs jk and numerically equal to the products of the corresponding coefficients in V and W. We can add any pairs of vectors in the new spaces by adding the coefficients. The dimensionality of V ⊗ W is the product of dimensionalities of V and W. We can create tensor product spaces by more than one application of tensor product, H = U ⊗V ⊗W, with arbitrary bracketing, since tensor product is associative. Tensor product space of the form is said to have tensor order 3 of n. 1 In Dirac notation popular in quantum mechanics and quantum computing, a vector u ∈ C 2 n is written as |u, and called a ket. 2 That is, with countable orthonormal basis. 3 Note that some sources alternatively call n a degree or a rank of a tensor. Here, we use tensor rank to refer to a property similar to matrix rank, see below. Consider H = V ⊗ W. We have seen the addition property v ⊗ w + v ⊗ w = {v + v} ⊗ w and similar property with linearity in the first argument -tensor product is bilinear. We have not, however, seen how to express v ⊗ w + v ⊗ w as φ ⊗ ψ for some φ ∈ V, ψ ∈ W. In many cases, while the left side is a proper vector from the tensor product space, it is not possible to find such φ and ψ. The tensor product space contains not only vectors of the form v ⊗ w, but also their linear combinations, some of which cannot be expressed as φ ⊗ ψ. For example, can be decomposed as cannot; no matter what we choose as coefficients a, b, c, d, we have since we require ac = 1/ √ 2, that is, a = 0, c = 0, and similarly bd = 1/ √ 2, that is, b = 0, c = 0, yet we also require bd = ad = 0, which is incompatible with a, b, c, d = 0. For tensor product spaces of order n, that is, is a tensor of rank 2. Tensors with rank greater than one are called entangled. Maximum rank of a tensor in a tensor product space of order higher than two is not known in general (Buczyński &). A p-dimensional word embedding model involving a d-token vocabulary is 5 a mapping f: p, that is, it maps word identifiers into a p-dimensional real Hilbert space, an inner product space with the standard inner product ·, · leading to the L 2 norm. Function f is trained to capture semantic information from the language corpus it is trained on, for example, two words i, j with f (i), f (j) ∼ 0 are expected to be semantically unrelated. In practical implementations, we represent f as a collection of vectors f i ∈ R p indexed by i, typically in the form of d × p matrix M, with embeddings of individual words as rows. We propose to represent an embedding v ∈ R p of each a single word as an entangled tensor. Specifically, in word2ket, we use tensor of rank r and order n of the form where v jk ∈ R q. The ing vector v has dimension p = q n, but takes rnq = O (rq log q log p) space. We use q ≥ 4; it does not make sense to reduce it to q = 2 since a tensor product of two vectors in R 2 takes the same space as a vector in R 4, but not every vector in R 4 can be expressed as a rank-one tensor in R 2 ⊗ R 2. If the downstream computation involving the word embedding vectors is limited to inner products of embedding vectors, there is no need to explicitly calculate the q n -dimensional vectors. Indeed, we have (see eq. 2) Thus, the calculation of inner product between two p-dimensional word embeddings, v and w, represented via word2ket takes O (rq log q log p) time and O additional space. In most applications, a small number of embedding vectors do need to be made available for processing through subsequent neural network layers -for example, embeddings of all words in all sentences in a batch. For a batch consisting of b words, the total space requirement is O (bp + rq log q log p), instead of O (rp) in traditional word embeddings. Reconstructing a single p-dimensional word embedding vector from a tensor of rank r and order n takes O (rn log 2 p) arithmetic operations. To facilitate parallel processing, we arrange the order-n tensor product space into a balanced tensor product tree (see Figure 1), with the underlying vectors v jk as leaves, and v as root. For example, for n = 4, Instead of performing n multiplications sequentially, we can perform them in parallel along branches of the tree, reducing the length of the sequential processing to O (log 2 n). Typically, word embeddings are trained using gradient descent. The proposed embedding representation involves only differentiable arithmetic operations, so gradients with respect to individual elements of vectors v jk can always be defined. With the balanced tree structure, word2ket representation can be seen as a sequence of O (log 2 n) linear layers with linear activation functions, where n is already small. Still, the gradient of the embedding vector v with respect to an underlying tunable parameters v lk involves products ∂ k n j=1 v jk /∂v lk = j =l v jk, leading to potentially high Lipschitz constant of the gradient, which may harm training. To alleviate this problem, at each node in the balanced tensor product tree we use LayerNorm . Let A: V → U be a linear operator that maps vectors from Hilbert space V into vector in Hilbert space U; that is, for v, v, ∈ V, α, β ∈ R, the vector A(αv + βv) = αAv + βAv is a member of U. Let us also define a linear operator B: W → Y. A mapping A ⊗ B is a linear operator that maps vectors from V ⊗ W into vectors in U ⊗ Y. We define A ⊗ B: V ⊗ W → U ⊗ Y through its action on simple vectors and through linearity for ψ j ∈ V and φ k ∈ U. Same as for vectors, tensor product of linear operators is bilinear In finite-dimensional case, for n × n matrix representation of linear operator A and m × m matrix representing B, we can represent A ⊗ B as an mn × m n matrix composed of blocks a jk B. We can see a p-dimensional word embedding model involving a d-token vocabulary as a linear operator F: R d → R p that maps the one-hot vector corresponding to a word into the corresponding word embedding vector. Specifically, if e i is the i-th basis vector in R d representing i-th word in the vocabulary, and v i is the embedding vector for that word in R p, then the word embedding linear operator is T. If we store the word embeddings a d × p matrix M, we can then interpret that matrix's transpose, M T, as the matrix representation of the linear operator F. Consider q and t such that q n = p and t n = d, and a series of n linear operators In word2ketXS, we represent the d × p word embedding matrix as where F jk can be represented by a q × t matrix. The ing matrix F has dimension p × d, but takes rnqt = O (rq log qt log t log p log d) space. Intuitively, the additional space efficiency comes from applying tensor product-based exponential compression not only horizontally, individually to each row, but horizontally and vertically at the same time, to the whole embedding matrix. We use the same balanced binary tree structure as in word2ket. To avoid reconstructing the full embedding matrix each time a small number of rows is needed for a multiplication by a weight matrix in the downstream layer of the neural NLP model, which would eliminate any space saving, we use lazy tensors . If A is an m × n matrix and matrix B is p × q, then ij th entry of A ⊗ B is equal to As we can see, reconstructing a row of the full embedding matrix involves only single rows of the underlying matrices, and can be done efficiently using lazy tensors. In order to evaluate the ability of the proposed space-efficient word embeddings in capturing semantic information about words, we used them in three different downstream NLP tasks: text summarization, language translation, and question answering. In all three cases, we compared the accuracy in the downstream task for the proposed space-efficient embeddings with the accuracy achieved by regular embeddings, that is, embeddings that store p-dimensional vectors for d-word vocabulary using a single d × p matrix. In text summarization experiments, we used the GIGAWORD text summarization dataset using the same preprocessing as, that is, using 200K examples in training. We used an encoder-decoder sequence-to-sequence architecture with bidirectional forwardbackward RNN encoder and an attention-based RNN decoder , as implemented in. In both the encoder and the decoder we used internal layers with dimensionality of 256 and dropout rate of 0.2, and trained the models, starting from random weights and embeddings, for 20 epochs. We used the validation set to select the best model epoch, and reported on a separate test set. We used Rouge 1, 2, and L scores . In addition to testing the regular dimensionality of 256, we also explored 400, and 8000, but kept the dimensionality of other layers constant. The in Table 1 show that word2ket can achieve 16-fold reduction in trainable parameters at the cost of a drop of Rouge scores by about 2 points. As expected, word2ketXS is much more spaceefficient, matching the scores of word2ket while allowing for 34,000 fold reduction in trainable parameters. More importantly, it offers over 100-fold space reduction while reducing the Rouge scores by only about 0.5. Thus, in the evaluation on the remaining two NLP tasks we focused on word2ketXS. The second task we explored is German-English machine translation, using the IWSLT2014 (DE-EN) dataset of TED and TEDx talks as preprocessed in . We used the same sequence-to-sequence model as in GIGAWORD summarization task above. We used BLEU score to measure test set performance. We explored embedding dimensions of 100, 256, 400, 1000, and 8000 by using different values for the tensor order and the dimensions of the underlying matrices F jk. The in Table 2 show a drop of about 1 point on the BLEU scale for 100-fold reduction in the parameter space, with drops of 0.5 and 1.5 for lower and higher space saving rates, respectively. The third task we used involves the Stanford Question Answering Dataset (SQuAD) dataset. We used the DrQA's model , a 3-layer bidirectional LSTMs with 128 hidden units for both paragraph and question encoding. We trained the model for 40 epochs, starting from random weights and embeddings, and reported the test set F1 score. DrQA uses an embedding with vocabulary size of 118,655 and embedding dimensionality of 300. As the embedding matrix is larger, we can increase the tensor order in word2ketXS to four, which allows for much higher space savings. Results in Table 3 show a 0.5 point drop in F1 score with 1000-fold saving of the parameter space required to store the embeddings. For order-4 tensor word2ketXS, we see almost 10 5 -fold space saving rate, at the cost of a drop of F1 by less than two points, that is, by a relative drop of less than 3%. We also investigated the computational overhead introduced by the word2ketXS embeddings. For tensors order 2, the training time for 40 epochs increased from 5.8 for the model using regular embedding to 7.4 hours for the word2ketXS-based model. Using tensors of order 4, to gain additional space savings, increased the time to 9 hours. Each run was executed on a single NVIDIA Tesla V100 GPU card, on a machine with 2 Intel Xeon Gold 6146 CPUs and 384 GB RAM. While the training time increased, as shown in Fig. 3, the dynamics of model training remains largely unchanged. The of the experiments show substantial decreases in the memory footprint of the word embedding part of the model, used in the input layers of the encoder and decoder of sequence-tosequence models. These also have other parameters, including weight matrices in the intermediate layers, as well as the matrix of word probabilities prior to the last, softmax activation, that are not compressed by our method. During inference, embedding and other layers dominate the memory footprint of the model. Recent successful transformer models like BERT by , GPT-2 by, RoBERTa by and Sparse Transformers by require hundreds of millions of parameters to work. In RoBERTa BASE, 30% of the parameters belong to the word embeddings. During training, there is an additional memory need to store activations in the forward phase in all layers, to make them available for calculating the gradients in the backwards phase. These often dominate the memory footprint during training, but one can decrease the memory required for Figure 2: Dynamics of the test-set F1 score on SQuAD dataset using DrQA model with different embeddings, for rank-2 order-2 word2ketXS, for rank-1 order-4 word2ketXS, for for the regular embedding. Given the current hardware limitation for training and inference, it is crucial to be able to decrease the amount of memory these networks requires to work. A number of approaches have been used in lowering the space requirements for word embeddings. Dictionary learning and word embedding clustering approaches have been proposed. Bit encoding has been also proposed. An optimized method for uniform quantization of floating point numbers in the embedding matrix has been proposed recently . To compress a model for low-memory inference, used pruning and quantization for lowering the number of parameters. For low-memory training sparsity and low numerical precision approaches were proposed. In approximating matrices in general, Fourier-based approximation methods have also been used ). None of these approaches can mach space saving rates achieved by word2ketXS. The methods based on bit encoding, such as;; are limited to space saving rate of at most 32 for 32-bit architectures. Other methods, for example based on parameter sharing or on PCA, can offer higher saving rates, but their storage requirement is limited by d + p, the vocabulary size and embedding dimensionality. In more distantly related work, tensor product spaces have been used in studying document embeddings, by using sketching of a tensor representing n-grams in the document.

Represent the short summary for finding the corresponding scientific article (long)

We use ideas from quantum computing to proposed word embeddings that utilize much fewer trainable parameters.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

It's the quantum algorithm for Expectation Maximization. It's fast: the runtime depends only polylogarithmically on the number of elements in the dataset.


Question: represent the article for finding a single-sentence tldr

Question: One of the distinguishing aspects of human language is its compositionality, which allows us to describe complex environments with limited vocabulary. Previously, it has been shown that neural network agents can learn to communicate in a highly structured, possibly compositional language based on disentangled input (e.g. hand- engineered features). Humans, however, do not learn to communicate based on well-summarized features. In this work, we train neural agents to simultaneously develop visual perception from raw image pixels, and learn to communicate with a sequence of discrete symbols. The agents play an image description game where the image contains factors such as colors and shapes. We train the agents using the obverter technique where an agent introspects to generate messages that maximize its own understanding. Through qualitative analysis, visualization and a zero-shot test, we show that the agents can develop, out of raw image pixels, a language with compositional properties, given a proper pressure from the environment. One of the key requirements for artificial general intelligence (AGI) to thrive in the real world is its ability to communicate with humans in natural language. Natural language processing (NLP) has been an active field of research for a long time, and the introduction of deep learning BID18 enabled great progress in NLP tasks such as translation, image captioning, text generation and visual question answering; BID13 BID10; BID19 BID0. However, training machines in a supervised manner with a large dataset has its limits when it comes to communication. Supervised methods are effective for capturing statistical associations between discrete symbols (i.e. words, letters). The essence of communication is more than just predicting the most likely word to come next; it is a means to coordinate with others and potentially achieve a common goal BID1 BID7 ).An alternative path to teaching machines the art of communication is to give them a specific task and encourage them to learn how to communicate on their own. This approach will encourage the agents to use languages grounded to task-related entities as well as communicate with other agents, which is one of the ways humans learn to communicate BID5. Recently, there have been several notable works that demonstrated the emergence of communication between neural network agents. Even though each work produced very interesting of its own, in all cases, communication was either achieved with a single discrete symbol (as opposed to a sequence of discrete symbols) BID8 BID17 or via a continuous value (; BID12 . Not only is human communication un-differentiable, but also using a single discrete symbol is quite far from natural language communication. One of the key features of human language is its compositional nature; the meaning of a complex expression is determined by its structure and the meanings of its constituents BID9 . More recently, BID22 and BID16 trained the agents to communicate in grounded, compositional language. In both studies, however, inputs given to the agents were hand-engineered features (disentangled input) rather than raw perceptual signals that we receive as humans. In this work, we train neural agents to simultaneously develop visual perception from raw image pixels, and learn to communicate with a sequence of discrete symbols. Unlike previous works, our setup poses greater challenges to the agents since visual understanding and discrete communication have to be induced from scratch in parallel. We place the agents in a two-person image description game, where images contain objects of various color and shape. Inspired by the pioneering work of BID3, we employ a communication philosophy named obverter to train the agents. Having its root in the theory of mind and human language development BID21, the obverter technique motivates an agent to search over messages and generate the ones that maximize their own understanding. The contribution of our work can be summarized as follows:• We train artificial agents to learn to disentangle raw image pixels and communicate in compositional language at the same time.• We describe how the obverter technique, a differentiable learning algorithm for discrete communication, could be employed in a communication game with raw visual input.• We visualize how the agents are perceiving the images and show that they learn to disentangle color and shape without any explicit supervision other than the communication one.• Experiment suggest that the agents could develop, out of raw image input, a language with compositional properties, given a proper pressure from the environment (i.e. the image description game).Finally, while our exposition follows a multi-agent perspective, it is also possible to interpret our in the single-agent setting. We are effectively learning a neural network that is able to learn disentangled compositional representations of visual scenes, without any supervision. Subject to the constraints imposed by their environment, our agents learn disentangled concepts, and how to compose these to form new concepts. This is an important milestone in the path to AGI. 2.1 THE TWO-PERSON IMAGE DESCRIPTION GAME Speaker Listener ABACD 1Speaker Listener ABACD 0Figure 1: The two-person image description game. Speaker observes an image and generates a message (i.e. a sequence of discrete symbols). The listener, after observing a separate image and the message, must correctly decide whether it is seeing the same object as the speaker (left side; output 1) or not (right side; output 0). (blue, red, white, gray, yellow, green, cyan, magenta), and five shapes (box, sphere, cylinder, capsule, ellipsoid), giving us total 40 combinations. We choose a straightforward image description game with two factors (color and shape) so that we can perform extensive analysis on the outcome confidently, based on full control of the experiment. In a single round of the two-person image description game, one agent becomes the speaker and the other the listener. The speaker is given a random image, and generates a message to describe it. The listener is also given a random image, possibly the same image as the speaker's. After hearing the message from the speaker, the listener must decide if it is seeing the same object as the speaker (Figure 1). Note that an image is the raw pixels given to the agents, and an object is the thing described by the image. Therefore two different images can depict the same object. In each round the agents change roles of being the speaker and the listener. We generated synthetic images using Mujoco physics simulator 1. The example images are shown in FIG0. Each image depicts a single object with a specific color and shape in 128×128 resolution. There are eight colors (blue, red, white, gray, yellow, green, cyan, magenta) and five shapes (box, sphere, cylinder, capsule, ellipsoid), giving us 40 combinations. We generated 100 variations for each of the 40 object type. Note that the position of the object varies in each image, changing the object size and the orientation. Therefore even if the speaker and the listener are given the same object type, the actual images are very likely to be different, preventing the agents from using pixel-specific information, rather than object-related information to win the game. Figure 3: Agent model architecture. The visual module processes the image, and the language module generates or consumes messages. The decision module accepts embeddings from both modules and produces the output. The solid arrows indicate modifying the output from the previous layer. The dotted arrows indicate copying the output from the previous layer. Aside from using disentangled input, another strong assumption made in previous works BID3 BID22 was that the agents had access to the true intention of the speaker. In BID3, the listener was trained to modify its RNN hidden vector as closely to the speaker's intention (meaning vector; please see TAB7 in Appendix A) as possible. In BID22, each agent had an auxiliary task to predict the goals of all other agents. In both cases, the true meaning/goal vector was used to update the model parameters, exposing the disentangled information to the agents. In order to relax this assumption and encourage the agents to develop communication with minimal guidance, our model uses no other signal than whether the listener made a correct decision. Figure 3 depicts the agent model architecture. We use a convolutional neural network followed by a fully-connected layer to process the image. A single RNN, specifically the gated recurrent units (GRU), is used for both generating and consuming messages (the message generation using the obverter strategy is described in the next section). When consuming a message, the image embedding from the visual module and the message embedding from the language module are concatenated and processed by another fully-connected layers (i.e. decision module) with the sigmoid outputŷ, 0 being "My (listener) image is different from the speaker's" and 1 being "My image is the same as the speaker's". Further details of the model architecture (e.g. number of layers) are described in Appendix C. Although our work is inspired by (see Appendix A for the description of BID3), obverter technique is a general message generation philosophy used/discussed in a number of communication and language evolution studies BID11;; BID14, which has its root in the theory of mind. Theory of mind observes that a human has direct access only to one's own mind and not to the others'. Therefore we typically assume that the mind of others is analogous to ours, and such assumption is reflected in the functional use of language BID5. For example, if we want to convey a piece of information to the listener 2, it is best to speak in a way that maximizes the listener's understanding. However, since we cannot directly observe the listener's state of mind, we cannot exactly solve this optimization problem. Therefore we posit that the listener's mind operates in a similar manner as ours, and speak in a way that maximizes our understanding, thus approximately solving the optimization problem. This is exactly what the obverter technique tries to achieve. When an agent becomes the teacher (i.e. speaker), the model parameters are fixed. The image is converted to an embedding via the visual module. After initializing its RNN hidden layer to zeros, the teacher at each timestep evaluatesŷ for all possible symbols and selects the one that maximizesŷ. The RNN hidden vector induced by the chosen symbol is used in the next timestep. This is repeated untilŷ becomes bigger than the predefined threshold, or the maximum message length is reached (see Appendix D for algorithm). Therefore the teacher, through introspection, greedily selects characters at each timestep to generate a message such that the consistency between the image and the message is as clear to itself as possible. When an agent becomes the learner (i.e. listener), its parameters are updated via back-propagating the cross entropy loss between its outputŷ and the true label y. Therefore the agents must learn to communicate only from the true label indicating whether the teacher and the learner are seeing the same object. We remind the reader that only the learner's RNN parameters are updated, and the teacher uses its fixed RNN. Therefore an agent uses only one RNN for both speaking and listening, guaranteeing self-consistency (see Appendix B for a detailed comparison between the obverter technique and the RL-based approach). Furthermore, because the teacher's parameters are fixed, message generation can easily be extended to be more exploratory. Although in this work we deterministically selected a character in each timestep, one can, for example, sample characters proportionally toŷ and still use gradient descent for training the agents. Using a more exploratory message generation strategy could help us discover a more optimal communication language when dealing with complex tasks. Another feature of the obverter technique is that it observes the principle of least effort . Because the teacher stops generating symbols as soon asŷ reaches the threshold, it does not waste any more effort trying to perfect the message. The same principle was implemented in one way or another in previous works, such as choosing the shortest among the generated strings BID14 or imposing a small cost for generating a message BID22. During the early stages of research, we noticed that randomly sampling object pairs (one for the teacher, one for the learner) lead to agents focusing only on colors and ignoring shapes. When the teacher's object is fixed, there are 40 (8 colors × 5 shapes) possibilities on the learner's side. If the teacher only talks about the color of the object, the learner can correctly decide for 36 out of 40 possible object types. The learner makes incorrect decisions only when the teacher and the learner are given objects of the same color but different shapes, ing in 90% accuracy on average. This is actually what we observed; the accuracy plateaued between 0.9 and 0.92 during the training, and the messages were more or less the same for objects with the same color. Therefore when constructing a mini-batch of images, we set 25% to be the object pairs of the same color and shape, 30% the same shape but different colors, 20% the same color but different shapes. The remaining 25% object pairs were picked randomly 3.Vocabulary size (i.e. number of unique symbols) and the maximum message length were also influential to the final outcome. We noticed that a larger vocabulary and a longer message length helped the agents achieve a high communication accuracy more easily. But the ing messages were more challenging to analyze for compositional patterns. In all our experiments we used 5 and 20 respectively for the vocabulary size and the maximum message length, similar to what BID3 used. This suggests that the environment plays as important, if not more, role as the model architecture in the emergence of complex communication as discussed by previous studies BID15 BID4 BID16 and should be a main consideration for future efforts. Further details regarding hyperparameters are described in Appendix E. In this section, we first study the convergence behavior during the training phase. Then we analyze the language developed by the agents in terms of compositionality. As stated in the introduction, in compositional language, the meaning of a complex expression is determined by its structure and the meanings of its constituents. With this definition in mind, we focus on two aspects of the inter-agent communication to evaluate its compositional properties: the structure (i.e. grammar) of the communication, and zero-shot performance (i.e. generalizing to novel stimuli). These two aspects, which are both necessary conditions for any language to be considered compositional, have been used by previous works to study the compositional nature of artificial communication BID3 BID22 BID16.To evaluate the structure of the messages, we study the evolution of the communication as training proceeds, and try to derive a grammar for expressing colors and shapes. To evaluate the zero-shot capabilities, we test if the agents can compose consistent messages for objects they have not seen during the training. Moreover, we visualize the image embeddings from the visual modules of both agents to understand how they are recognizing colors and shapes, the of which, for a better view of the figures, are provided in Appendix H. Figure 4: Progress during the training (best seen in color). (Top) We plot the training accuracy, training loss, average message length and average message distinctness in each round. (Bottom) We plot the perplexities and the Jaccard similarity of the messages spoken by both agents in each round. Note that the average message length and the perplexities are divided by 20 to match the y-axis range with other metrics. Figure 4 shows the convergence behavior during the training. Training accuracy was calculated by rounding the learner's sigmoid output by 0.5. Message distinctness was calculated by dividing the number of unique messages in the mini-batch by the size of the mini-batch. Ideally there should be, on average, 40 distinct messages in the mini-batch of 50 images, therefore giving us 0.8 distinctness. Every 10 round, both agents were given the same 1, 000 randomly sampled images to generate 1, 000 message pairs. Then perplexity was calculated for each object type and averaged, thus indicating the average number of distinct messages used by the agents to describe a single object type (note that perplexities in the plot was divided by 20). Jaccard similarity between both agents' messages was also calculated for each object type and averaged. At the beginning, the listener (i.e. learner) always decides it is not seeing the same object as the speaker, giving us 0.75 accuracy 4. But after 7, 000 rounds, accuracy starts to go beyond 0.9. Loss is negatively correlated with accuracy until round 15, 000, where it starts to fluctuate. Accuracy, however, remains high due to how accuracy is measured; by rounding the learner's output by 0.5. Although we could occasionally observe some patterns in the messages when both accuracy and loss were high, a lower loss generally ed in a clearer communication structure (i.e. grammar) and better zero-shot performance. The loss fluctuation also indicates some instability in the training process, which is a potential direction for future work. Message distinctness starts at near 0, indicating the agents are generating the same message for all object types. After round 7, 000, where both message distinctness and message length reach their maximum, both start to decrease. But message distinctness never goes as high as the ideal 0.8, meaning that the agents are occasionally using the same message for different object types, as will be shown in the following section. Both perplexities and Jaccard similarity show seemingly meaningless fluctuation at early rounds. After round 7, 000, perplexities and Jaccard similarity show negatively correlated behavior, meaning that not only is each agent using consistent messages to describe each object type, but also both agents are using very similar messages to describe each object type. We found perplexity and Jaccard similarity to be an important indicator of the degree of the communication structure. During rounds 7, 000 ∼ 8, 000, performance was excellent in terms of loss and accuracy, but perplexity was high and Jaccard similarity low, indicating the agents were assigning incoherent strings to each object type just to win the game. Similar behavior was observed in the early stages of language evolution simulation in BID14 where words represented some meanings but had no structure (i.e. protolanguage). It seems that artificial communication acquires compositional properties after the emergence of protolanguage regardless of whether the input is entangled or disentangled. We choose agents from different training rounds to highlight how the language becomes more structured over time. TAB1 shows agents' messages in the beginning (round 40), when the training accuracy starts pushing beyond 90% (round 6, 940), when agents settle on a common language (round 16, 760).In round 40, both agents are respectively producing the same message for all object types as mentioned in section 3.1. We might say the messages are structured, but considering that the listener always answers 0 in early rounds, we cannot say the agents are communicating. In round 6, 940, which is roughly when the agents begin to communicate more efficiently, training accuracy is significantly higher than round 40. However, perplexities show that both agents are assigning many names to a single object type (40-80 names depending on the object type), indicating that the agents are focusing on pixel-level differences between images of the same object type. TAB1 shows, as an example, the messages used by both agents to describe the red sphere. Due to high perplexity, it is difficult to capture the underlying grammar of the messages even with regular expression. Furthermore, as Jaccard similarity indicates, both agents are generating completely different messages for the same object type. In round 16, 760, as the perplexities and Jaccard similarity tell us, the agents came to share a very narrow set of names for each object type (1-4 names depending on the object type). Moreover, the names of the same-colored objects and same-shaped objects clearly seem to follow a pattern. Overall, each of the three phases (round 40, round 6,940, round 16,760) seem to represent the development of visual perception, learning to communicate, and emergence of structure. We found the messages in round 16, 760 could be decomposed in a similar manner as Table 6 in Appendix A. The top of TAB2 shows a possible decomposition of the messages from round 16, 760 and the bottom shows the rules for each color and shape derived from the decomposition. According to our analysis, the agents use the first part of the message (i.e. prefix) to specify a shape, and the second part (i.e. suffix) to specify a color. However, they use two different strings to specify a shape. For example, the agents use either aaaa or bbbbb to describe a box. The strings used for specifying colors show slightly weaker regularity. For example, red is always described by either the suffix c or suffix e, but magenta is described by the suffix bb, bd, and sometimes b or bc.ā used for gray objects represents deletion of the prefix a. Note that removing prefixes beyond their length causes the pattern to break. For example, gray box, gray sphere and gray cylinder use the sameāāa to express the color, but gray capsule and gray ellipsoid use irregular suffixes. Despite some irregularities and one exceptional case (cyan box), the messages provide strong evidence that the agents learned to properly recognize color and shape from raw pixel input (see Appendix H for studying what the visual module learned), mapped each color and shape to prefixes and suffixes, and are able to compose meaningful messages to describe a given image to one another. Communication accuracy for each object type is described in Appendix F. Communication examples and their analysis are given in Appendix G. If the agents have truly learned to compose a message that can be divided into a color part and a shape part, then they should be able to accurately describe an object they have not seen before, which is another necessary condition for a compositional language. Therefore, we hold out five objects (the shaded cells in TAB4) from the dataset during the training and observe how agents describe five novel objects during the test phase. The agents were chosen from round 19, 980, which showed a high accuracy (97.8%), low perplexities (1.48, 1.65) and a high Jaccard similarity (0.75). TAB4 shows a potential decomposition of the messages used by the agents (original messages are described by TAB12 in Appendix I). We can observe that there is clearly a structure in the communication, although some messages show somewhat weaker patterns compared to when the agents were trained with all object types (even when we consider the effects ofb andē. However, the messages describing the held-out object types show clear structure with the exception of yellow ellipsoid. In order to assess the communication accuracy when held-out objects are involved, we conducted another test with the agents from round 19, 980. Each held-out object was given to the speaker, the listener, or both. In the first two cases, the held-out object was paired with all 40 object types and each pair was tested 10 times. In the last case, the held-out object was tested against itself 10 times. In all cases, the agents switched roles after 5 times. Table 4 shows communication accuracies for each case. We can see the agents can successfully communicate most of the time even when given novel objects. The last column shows that the listener is not simply producing 0 to maximize its chance to win the game. It is also notable that the objects described withoutb orē show better performance in general. We noticed the communication accuracy for held-out objects seems relatively weak considering the messages used to describe them strongly showed structure. TAB4 . This, however, from the grammar (i.e. structure) being not as straightforward as TAB2, especially with short messages (i.e. frequent use ofb andē). The same tendency can be observed for non-held-out objects as described by the per-object communication accuracy Table 9 in Appendix J. Table 4: Communication accuracy when agents were given objects not seen during the training. From the grammar analysis in the previous section, we have shown that the emerged language strongly follows a well-defined grammar. In the zero-shot test, the agents demonstrated that they can successfully describe novel object, although not perfectly, by also following a similar grammar. Both are, as stated in the beginning of section 3, necessary conditions for any communication to be considered compositional. Therefore we can safely conclude that the emerged language in this work possesses some qualifications to be considered compositional. In this work, we used the obverter technique to train neural network agents to communicate in a two-person image description game. Through qualitative analysis, visualization and the zero-shot test, we have shown that even though the agents receive raw perception in the form of image pixels, under the right environment pressures, the emerged language had properties consistent with the ones found in compositional languages. As an evaluation strategy, we followed previous works and focused on assessing the necessary conditions of compositional languages. However, the exact definition of compositional language is still somewhat debatable, and, to the best of our knowledge, there is no reliable way to mathematically quantify the degree of compositionality of an arbitrary language. Therefore, in order to encourage active research and discussion among researchers in this domain, we propose for future work, a quantitatively measurable definition of compositionality. We believe compositionality of a language is not binary (e.g. language A is compositional/not compositional), but a spectrum. For example, human language has some aspects that are compositional (e.g., syntactic constructions, most morphological combinations) and some that are not (e.g., irregular verb tenses in English, character-level word composition). It is also important to clearly define grounded language and compositional language. If one agent says abc (eat red apple) and another says cba (apple red eat), and they both understand each other, are they speaking compositional language? We believe such questions should be asked and addressed to shape the definition of compositionality. In addition to the definition/evaluation of compositional languages, there are numerous directions of future work. Observing the emergence of a compositional language among more than two agents is an apparent next step. Designing an environment to motivate the agents to disentangle more than two factors is also an interesting direction. Training agents to consider the context (i.e. pragmatics), such as giving each agent several images instead of one, is another exciting future work. A EMERGENCE OF GRAMMAR, BID3 In BID3, the author successfully trained neural agents to develop a structured (i.e. grammatical) language using disentangled meaning vectors as the input. Using 10 subject vectors and 10 predicate vectors, all represented as explicit binary vectors, total 100 meaning vectors could be composed TAB7 ). Each digit in the subject vector 5a serves a clear role, respectively representing speaker(sp), hearer(hr), other(ot), and plural(pl). The predicate vector values, on the other hand, are randomly chosen so that each predicate vector will have three 1's and three 0's. The combination of ten subject vectors and ten predicate vectors allows 100 meaning vectors. The author used twenty neural agents for the experiment. Each agent was implemented with the vanilla recurrent neural networks (RNN), where the hidden vector h's size was 10, same as the size of the meaning vector m in order to treat h as the agent's understanding of m. In each training round a single learner (i.e. listener) and ten teachers (i.e. speaker) were randomly chosen. Each teacher, given all 100 m's in random order, generates a message s 5 for each m and sends it to the learner. The messages are generated using the obverter techinque, which is described in Algorithm 1. The learner is trained to minimize the mean squared error (MSE) between h (after consuming the s) and m. After the learner has learned from all ten teachers, the next round begins, repeating the process until the error goes below some threshold. Algorithm 1: Message generation process used in BID3. DISPLAYFORM0 9 Append i to s; DISPLAYFORM1 Terminate;When the training was complete, the author was able to find strong patterns in the messages used by the agents (Table 6). Note that the messages using predicates tired, scared, sick and happy especially follow a very clear pattern. Batali also conducted a zero-shot test where the agents were trained without the diagonal elements in Table 6 and tested with all 100 meaning vectors. The agents were able to successfully communicate even when held-out meaning vectors were used, but the Table 6: (Top) Messages used by a majority of the population for each of the given meanings.(Bottom) A potential analysis of the system in terms of a root plus modifications. Italic symbols are used to specify predicates and roman symbols are used to specify subjects. Messages in parentheses cannot be made to fit into this analysis.messages used for the held-out meaning vectors did not show as strong compositional patterns as the non-zero-shot case. The obverter technique allows us to generate messages that encourage the agents to use a shared language, even a highly structured one, via using a single RNN for both speaking and listening. This is quite different from other RL-based related works BID17 BID22 BID8 BID12 BID16 where each agent has separate components (e.g. two RNNs) for generating messages and consuming messages. This is necessary typically because the message generation module and the message consumption module have different input/output requirements. The message generation module accepts some input related to the task (e.g. goal description vector, question embedding, or image embedding) and generates discrete symbols. The message consumption module, on the other hand, accepts discrete symbols (i.e. the message) and generates some output related to the task (e.g. some prediction or some action to take). Therefore, when a neural agent speaks in the RL-based approach, its message generation process is completely separated from its own listening process, but tied to the listening process of another agent (i.e. listener) 6. This means an agent may not have internal consistency; what an agent speaks may not make sense to itself. However, agents in the RL-based setting do converge on a common language because, during the training, the error signal flows from the listener to the speaker directly. Obverter approach, on the other hand, requires that each agent has a single component for both message generation and message consumption. This single component accepts discrete symbols and generates some output related to the task. This guarantees internal consistency because an agent's message generation process is tied to its own message consumption process; it will only generate messages that make sense to itself. In the obverter setting, the error signal does not flow between agents directly, but agents converge on a common language by taking turns to be the listener; the listener tries to understand what the speaker says, so that when the listener becomes the speaker, it can generate messages that make sense to itself and, at the same time, will be understood by the former speaker (now listener).The advantage of obverter approach over RL-based approach is that it is motivated by the theory of mind and more resembles the acquisition/development process of human language. Having a single mechanism for both speaking and listening, and training oneself to be a good listener leads to the emergence of self-consistent, shared language. However, obverter technique requires that all agents perform the same task, which means all agents must have identical model architectures. This is because, during the message generation process, the speaker internally simulates what the listener will go though when it hears the message. Therefore we cannot play an asymmetrical game such as where the speaker sees only one image and generates a message but the listener is given multiple images and must choose one after hearing the message. RL-based approaches do not have this problem since there are separate modules for speaking and listening. We believe obverter technique could be the better choice for certain tasks regarding human mind emulation. But it certainly is not the tool for every occasion. The RL-based approach is a robust tool for any general task that may or may not involve human-like communication. We conclude this section with a possible future research direction that combines the strengths of both approaches to enable communication in more interesting and complicated tasks. We used TensorFlow and the Sonnet library for all implementation. We used an eight-layer convolutional neural network. We used 32 filters with the kernel size 3 for every layer. The strides were for each layer. We used rectified linear unit (ReLU) as the activation function for every layer. Batch normalization was used for every layer. We did not use the bias parameters since we used Batch normalization. For padding, we used the TensorFlow VALID padding option for every layer. The fully connected layer that follows the convolutional neural network was of 256 dimensions, with ReLU as the activation function. We used a single layer Gated Recurrent Units (GRU) to implement the language module. The size of the hidden layer was 64. We used a two-layer feedforward neural network. The first layer reduces the dimensionality to 128 with ReLU as the activation function, then the second layer generates a scalar value with sigmoid as the activation function. Both agents' model parameters are randomly initialized. The training process consists of rounds where teacher/learner roles are changed, and each round consists of multiple games where learner's model parameters are updated. In each game, the teacher, given a mini-batch of images, generates corresponding messages. The learner, given a separate mini-batch of images and the messages from the teacher, decides whether it is seeing the same object type as the teacher. Learner's model parameters are updated to minimize the cross entropy loss. After playing a predefined number of games, we move on to the next round where two agents change their roles. Algorithm 2: Message generation process used in our work. Table 7: Accuracy when each object type is given to the speaker. DISPLAYFORM0 We conducted a separate test with the agents from round 16, 760 to assess the communication accuracy for each object type. The agents were given 1, 600 total object pairs (40 × 40). Each object pair was tested 10 times, where after 5 times the agents switched speaker/listener roles. The average accuracy was 95.4%, and only 88 out of 1, 600 object pairs were communicated with accuracy lower than 0.8. Table 7 describes the accuracy when each object type was given to the speaker. We can observe that the accuracy is higher for objects that are described with less overlapping messages. For example, yellow box is communicated with the accuracy of 98%, and it is described with aaaaaa, which is not used for any other object types. Gray box, on the other hand, is communicated with accuracy 93%. It is described with aaa, which is also used for yellow capsule and green sphere, both of which are communicated with low accuracies as well. Figure 5 provides ten examples of communication when the speaker is given a blue box and the listener is given various object types. The listener's belief (i.e. score) that it is seeing the same image as the speaker changes each time it consumes a symbol. It is notable that most of the time the score jumps between 0 and 1, rather than gradually changing in between. This is natural given that messages that differ by only a single character can mean different objects (e.g. blue box and blue cylinder). This phenomenon can also be seen in human language. For example, blue can and blue cat differ by a single alphabet, but the semantics are completely different. Object types that are described by similar messages as blue box, such as blue cylinder and magenta box cause marginal confusion to the listener such that prediction scores for both objects are not complete zeros. There are also cases where two completely different objects are described by the same message as mentioned in Section 3.1. From TAB2 we can see that blue box and cyan cylinder are described by the same message bbbbbbb{b,d}, although the messages were composed using different rules. Therefore the listener generates high scores for both objects, occasionally losing the game when the agents are given this specific object pair (1 out of 40 chance). This can be seen as a side effect coming from the principle of least effort which motivates the agents to win the game most of the time while minimizing the effort to generate messages. Section 3.2 provides strong evidence that the agents are properly recognizing the color and shape of an object. In this section, we study the visual module of both agents to study how they are processing Table 9: Accuracy when each object type is given to the speaker. Shaded cells indicate the objects not seen during the training. In the same manner as Appendix F, we conducted a separate test with the agents from round 19, 980 to assess the communication accuracy for each object type. The agents were given 1, 600 total object pairs (40 × 40). Each object pair was tested 10 times, where after 5 times the agents switched speaker/listener roles. The average accuracy was 94.73%, and 103 out of 1, 600 object pairs were communicated with accuracy lower than 0.8. Table 9 describes the accuracy when each object type was given the speaker. Shaded cells indicate objects not seen during the training. Here we can observe the same tendency as the one seen in Appendix F; the accuracy is higher for objects that are described with less overlapping messages. Lets assume agent0 is aware of red circle, blue square and green triangle. If agent0 came upon a blue circle for the first time and had to describe it to agent1, the efficient way would be to say blue circle. But it could also say blue not square not triangle. If agent1 had a similar knowledge as agent0 did, then both agents would have a successful communication. However, it is debatable whether saying blue not square not triangle is as compositional as blue circle.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We train neural network agents to develop a language with compositional properties from raw pixel input.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce and analyze the phenomenon of "hallucinations" in NMT, or spurious translations unrelated to source text, and propose methods to reduce its frequency.


Question: Represent the scientific article to find a short TLDR!

Question: The ability to forecast a set of likely yet diverse possible future behaviors of an agent (e.g., future trajectories of a pedestrian) is essential for safety-critical perception systems (e.g., autonomous vehicles). In particular, a set of possible future behaviors generated by the system must be diverse to account for all possible outcomes in order to take necessary safety precautions. It is not sufficient to maintain a set of the most likely future outcomes because the set may only contain perturbations of a dominating single outcome (major mode). While generative models such as variational autoencoders (VAEs) have been shown to be a powerful tool for learning a distribution over future trajectories, randomly drawn samples from the learned implicit likelihood model may not be diverse -- the likelihood model is derived from the training data distribution and the samples will concentrate around the major mode of the data. In this work, we propose to learn a diversity sampling function (DSF) that generates a diverse yet likely set of future trajectories. The DSF maps forecasting context features to a set of latent codes which can be decoded by a generative model (e.g., VAE) into a set of diverse trajectory samples. Concretely, the process of identifying the diverse set of samples is posed as DSF parameter estimation. To learn the parameters of the DSF, the diversity of the trajectory samples is evaluated by a diversity loss based on a determinantal point process (DPP). Gradient descent is performed over the DSF parameters, which in turn moves the latent codes of the sample set to find an optimal set of diverse yet likely trajectories. Our method is a novel application of DPPs to optimize a set of items (forecasted trajectories) in continuous space. We demonstrate the diversity of the trajectories produced by our approach on both low-dimensional 2D trajectory data and high-dimensional human motion data. Forecasting future trajectories of vehicles and human has many useful applications in autonomous driving, virtual reality and assistive living. What makes trajectory forecasting challenging is that the future is uncertain and multi-modal -vehicles can choose different routes and people can perform different future actions. In many safety-critical applications, it is important to consider a diverse set of possible future trajectories, even those that are less likely, so that necessary preemptive actions can be taken. For example, an autonomous vehicle should understand that a neighboring car can merge into its lane even though the car is most likely to keep driving straight. To address this requirement, we need to take a generative approach to trajectory forecasting that can fully characterize the multimodal distribution of future trajectories. To capture all modes of a data distribution, variational autoencoders (VAEs) are well-suited generative models. However, random samples from a learned VAE model with Gaussian latent codes are not guaranteed to be diverse for two reasons. First, the sampling procedure is stochastic and the VAE samples can fail to cover some minor modes even with a large number of samples. Second, since VAE sampling is based on the implicit likelihood function encoded in the training data, if most of the training data is centered around a specific mode while other modes have less data (Fig. 1 (a) ), the VAE samples will reflect this bias and concentrate around the major mode (Fig. 1 (b) ). To tackle this problem, we propose to learn a diversity sampling function (DSF) that can reliably generate a diverse set of trajectory samples (Fig. 1 (c) ). The proposed DSF is a deterministic parameterized function that maps forecasting context features (e.g., past trajectories) to a set of latent codes. The latent codes are decoded by the VAE docoder into a set of future trajectory samples, denoted as the DSF samples. In order to optimize the DSF, we formulate a diversity loss based on a determinantal point process (DPP) to evaluate the diversity of the DSF samples. The DPP defines the probability of choosing a random subset from the set of trajectory samples. It models the negative correlations between samples: the inclusion of a sample reduces the probability of including a similar sample. This makes the DPP an ideal tool for modeling the diversity within a set. In particular, we use the expected cardinality of the DPP as the diversity measure, which is defined as the expected size of a random subset drawn from the set of trajectory samples according to the DPP. Intuitively, since the DPP inhibits selection of similar samples, if the set of trajectory samples is more diverse, the random subset is more likely to select more samples from the set. The expected cardinality of the DPP is easy to compute and differentiable, which allows us to use it as the objective to optimize the DSF to enable diverse trajectory sampling. Our contributions are as follows: We propose a new forecasting approach that learns a diversity sampling function to produce a diverse set of future trajectories; We propose a novel application of DPPs to optimize a set of items (trajectories) in continuous space with a DPP-based diversity measure; Experiments on synthetic data and human motion validate that our method can reliably generate a more diverse set of future trajectories compared to state-of-the-art generative models. Trajectory Forecasting has recently received significant attention from the vision community. A large portion of previous work focuses on forecasting 2D future trajectories for pedestrians (; ; ;) or vehicles (a). Some approaches use deterministic trajectory modeling and only forecast one future trajectory;. As there are often multiple plausible future trajectories, several approaches have tried to forecast distributions over trajectories; ). Recently, Rhinehart et al. (2018; propose a generative model that can accurately forecast multi-modal trajectories for vehicles. also use egocentric videos to predict the future trajectories of the camera wearer. Some work has investigated forecasting higher dimensional trajectories such as the 3D fullbody pose sequence of human motions. Most existing work takes a deterministic approach and forecasts only one possible future motion from past 3D poses (; ; ; b), static images or egocentric videos . Differently, some probabilistic approaches use conditional variational autoencoders (cVAEs) to generate multiple future motions. In constrast to previous work, our approach can generate a diverse set of future motions with a limited number of samples. Diverse Solutions have been sought after in a number of problems in computer vision and machine learning. A branch of these methods aiming for diversity stems from the M-Best MAP problem , including diverse M-Best solutions and multiple choice learning ). Alternatively, previous work has used submodular function maximization to select a diverse subset of garments from fashion images . Determinantal point processes (DPPs) (; are efficient probabilistic models that can measure both the diversity and quality of items in a subset, which makes it a natural choice for the diverse subset selection problem. DPPs have been applied for document and video summarization , recommendation systems , object detection , and grasp clustering . have also used DPPs to mitigate mode collapse in generative adversarial networks (GANs). The work most related ours is , which also uses the cardinality of DPPs as a proxy for user engagement. However, there are two important differences between our approach and theirs. First, the context is different as they use the cardinality for a subset selection problem while we apply the cardinality as an objective of a continuous optimization problem in the setting of generative models. Second, their main motivation behind using the cardinality is that it aligns better with the user engagement semantics, while our motivation is that using cardinality as a diversity loss for deep neural networks is more stable due to its tolerance of similar trajectories, which are often produced by deep neural networks during stochastic gradient descent. The aim of multi-modal trajectory forecasting is to learn a generative model over future trajectories. Variational autoencoders (VAEs) are a popular choice of generative models for trajectory forecasting ) because it can effectively capture all possible future trajectories by explicitly mapping each data point to a latent code. VAEs model the joint distribution p θ (x, z) = p(z)p θ (x|z) of each data sample x (e.g., a future trajectory) and its corresponding latent code z, where p(z) denotes some prior distribution (e.g., Gaussians) and p θ (x|z) denotes the conditional likelihood model. To calculate the marginal likelihood p θ (x) = p θ (x, z)/p θ (z|x), one needs to compute the posterior distribution p θ (z|x) which is typically intractable. To tackle this, VAEs use variational inference which introduces an approximate posterior q φ (z|x) and decomposes the marginal log-likelihood as where L(x; θ, φ) is the evidence lower bound (ELBO) defined as During training, VAEs jointly optimize the recognition model (encoder) q φ (z|x) and the likelihood model (decoder) p θ (x|z) to maximize the ELBO. In the context of multi-modal trajectory forecasting, one can generate future trajectories from p(x) by drawing a latent code z from the prior p(z) and decoding z with the decoder p θ (x|z) to produce a corresponding future trajectory x. Our core technical innovation is a method to learn a diversity sampling function (DSF) that can generate a diverse set of future trajectories. To achieve this, we must equip ourselves with a tool to evaluate the diversity of a set of trajectories. To this end, we make use of determinantal point processes (DPPs) to model the diversity within a set. DPPs promote diversity within a set because the inclusion of one item makes the inclusion of a similar item less likely if the set is sampled according to a DPP. Formally, given a set of items (e.g., data points) Y = {x 1, . . ., x N}, a point process P on the ground set Y is a probability measure on 2 Y, i.e., the set of all subsets of Y. P is called a determinantal point process if a random subset Y drawn according to P has where Y ⊆ Y, I is the identity matrix, L ∈ R N ×N is the DPP kernel, a symmetric positive semidefinite matrix, and The DPP kernel L is typically constructed by a similarity matrix S, where S ij defines the similarity between two items x i and x j. If we use the inner product as the similarity measure, L can be written in the form of a Gram matrix L = S = X T X where X is the stacked feature matrix of Y. As a property of the Gram matrix, det (L Y) equals the squared volume spanned by vectors x i ∈ Y. With this geometric interpretation in mind, one can observe that diverse sets are more probable because their features are more orthogonal, thus spanning a larger volume. In addition to set diversity encoded in the similarity matrix S, it is also convenient to introduce a quality vector r = [r 1, . . ., r N] to weigh each item according to some unary metric. For example, the quality weight might be derived from the likelihood of an item. To capture both diversity and quality of a subset, the DPP kernel L is often decomposed in the more general form: With this decomposition, we can see that both the quality vector r and similarity matrix S contribute to the DPP probability of a subset Y: Due to its ability to capture the global diversity and quality of a set of items, we choose DPPs as the probabilistic approach to evaluate and optimize the diversity of the future trajectories drawn by our proposed diversity sampling function. Safety-critical applications often require that the system can maintain a diverse set of outcomes covering all modes of a predictive distribution and not just the most likely one. To address this requirement, we propose to learn a diversity sampling function (DSF) to draw deterministic trajectory samples by generating a set of latent codes in the latent space of a conditional variational autoencoder (cVAE) and decoding them into trajectories using the cVAE decoder. The DSF trajectory samples are evaluated with a DPP-based diversity loss, which in turn optimizes the parameters of the DSF for more diverse trajectory samples. Formally, the future trajectory x ∈ R T ×D is a random variable denoting a D dimensional feature over a future time horizon T (e.g., a vehicle trajectory or a sequence of humanoid poses). The context ψ = {h, f} provides the information to infer the future trajectory x, and it contains the past trajectory h ∈ R H×D of last H time steps and optionally other side information f, such as an obstacle map. In the following, we first describe how we learn the future trajectory model p θ (x|ψ) with a cVAE. Then, we introduce the DSF and the DPP-based diversity loss used to optimize the DSF. In order to generate a diverse set of future trajectory samples, we need to learn a generative trajectory forecasting model p θ (x|ψ) that can cover all modes of the data distribution. Here we use cVAEs (other proper generative models can also be used), which explicitly map data x with the encoder q φ (z|x, ψ) to its corresponding latent code z and reconstruct the data from the latent code using the decoder p θ (x|z, ψ). By maintaining this one-on-one mapping between the data and the latent code, cVAEs attempt to capture all modes of the data. As discussed in Sec. 3.1, cVAEs jointly optimize the encoder and decoder to maximize the variational lower bound: We use multivariate Gaussians for the prior, encoder and decoder:, and p θ (x|z, ψ) = N (x;x, αI). Both the encoder and decoder are implemented as neural networks. The encoder network f φ outputs the parameters of the posterior distribution: (µ, σ) = f φ (x, ψ). The decoder network g θ outputs the reconstructed future trajectoryx: Detailed network architectures are given in Appendix B.1. Based on the Gaussian parameterization of the cVAE, the objective in Eq. 6 can be rewritten as where we take V samples from the posterior q φ (z|x, ψ), D z is the number of latent dimensions, and β = 1/α is a weighting factor. The training procedure for the cVAE is detailed in Alg. 2 (Appendix A). Once the cVAE model is trained, sampling from the learned future trajectory model p θ (x|ψ) is efficient: we can sample a latent code z according to the prior p(z) and use the decoder p θ (x|z, ψ) to decode it into a future trajectory x. Algorithm 1 Training the diversity sampling function (DSF) S γ (ψ) Initialize γ randomly 4: while not converged do 5: Generate latent codes Z = {z 1, . . ., z N} with the DSF S γ (ψ) Generate the trajectory ground set Y = {x 1, . . ., x N} with the decoder g θ (z, ψ) 8: Compute the similarity matrix S and quality vector r with Eq. 8 and 9 9: Compute the DPP kernel Calculate the diversity loss L diverse Update γ with the gradient ∇L diverse 12: end for 13: end while As mentioned before, randomly sampling from the learned cVAE model according to the implicit likelihood function p θ (x|ψ), i.e., sampling latent codes from the prior p(z), does not guarantee that the trajectory samples are diverse: major modes (those having more data) with higher likelihood will produce most of the samples while minor modes with lower likelihood will have almost no sample. This prompts us to devise a new sampling strategy that can reliably generate a diverse set of samples covering both major and minor modes. We propose to learn a diversity sampling function (DSF) S γ (ψ) that maps context ψ to a set of latent codes Z = {z 1, . . ., z N}. The DSF is implemented as a γ-parameterized neural network which takes ψ as input and outputs a vector of length N · D z (see Appendix B.1 for network details). The latent codes Z are decoded into a diverse set of future trajectories Y = {x 1, . . ., x N}, which are denoted as the DSF trajectory samples. We note that N is the sampling budget. To solve for the parameters of the DSF, we propose a diversity loss based on a DPP defined over Y. In this section, we first describe how the DPP kernel L is defined, which involves the construction of the similarity matrix S and quality vector r. We then discuss how we use the DPP kernel L to formulate a diversity loss to optimize the parameters of the DSF. Recall that the DPP kernel is defined as L = Diag(r) · S · Diag(r), where r defines the quality of each trajectory and S measures the similarity between two trajectories. The DPP kernel L(γ) is a function of γ as it is defined over the ground set Y output by the DSF S γ (ψ). Similarity. We measure the similarity S ij between two trajectories x i and x j as where d x is the Euclidean distance and k is a scaling factor. This similarity design ensures that 0 ≤ S ij ≤ 1 and S ii = 1. It also makes S positive definite since the Gaussian kernel we use is a positive definite kernel. Quality. It may be tempting to use p(x|ψ) to define the quality of each trajectory sample. However, this likelihood-based measure will clearly favor major modes that have higher probabilities, making it less likely to generate samples from minor modes. This motivates us to design a quality metric that treats all modes equally. To this end, unlike the similarity metric which is defined in the trajectory space, the quality of each sample is measured in the latent space and is defined as Geometrically, let R be the radius of a sphere Φ containing most samples from the Gaussian prior p(z). We treat samples inside Φ equally and only penalize samples outside Φ. In this way, samples from major modes are not preferred over those from minor modes as long as they are inside Φ, while samples far away from the data manifold are heavily penalized as they are outside Φ. The radius R is determined by where ρ percent of the Gaussian samples lie within, and we set ρ = 90. To compute R, we use the percentage point function of the chi-squared distribution which models the distribution over the sum of squares of independent standard normal variables. The base quality ω is a hyperparameter which we set to 1 during training in our experiments. At test time, we can use a larger ω to encourage the DPP to select more items from the ground set Y. The hyperparameter ρ (or R) allows for the trade-off between diversity and quality. When R is small, the quality metric is reduced to a pure likelihood-based metric (proportional to the latent likelihood), which will prefer samples with high likelihood and in a less diverse sample set. When R is large, most samples will have the same quality, and the ing samples will be highly diverse but less likely. In practice, the choice of R should be application dependent, as one could imagine autonomous vehicles would need to consider more diverse scenarios including those less likely ones to ensure robustness. We note that after the diverse samples are obtained, it is possible to reassign the quality score for each sample based on its likelihood to allow users to prioritize more likely samples. Diversity Loss. To optimize the DSF S γ (ψ), we need to define a diversity loss that measures the diversity of the trajectory ground set Y generated by S γ (ψ). An obvious choice for the diversity loss would be the negative log likelihood However, there is a problem with directly using the DPP log likelihood. The log likelihood heavily penalizes repeated items: if two trajectories inside Y are very similar, their corresponding rows in L will be almost identical, making det(L(γ)) = λ 1 λ 2... λ N ≈ 0 (λ n is the n-th eigenvalue). In practice, if the number of modes in the trajectory distribution p(x|ψ) is smaller than |Y|, Y will always have similar trajectories, thus making det(L(γ)) always close to zero. In such cases, optimizing the negative log likelihood causes numerical issues, which is observed in our early experiments. Instead, the expected cardinality of the DPP is a better measure for the diversity of Y, which is defined as Intuitively, since the DPP discourages selection of similar items, if Y is more diverse, a random subset Y drawn according to the DPP is more likely to select more items from Y, thus having larger cardinality. The expected cardinality can be computed as (Eq. 15 and 34 in): The main advantage of the expected cardinality is that it is well defined even when the ground set Y has duplicated items, since it does not require all eigenvalues of L to be non-zero as the log likelihood does. Thus, our diversity loss is defined as L diverse (γ) = −tr I − (L(γ) + I) −1. The training procedure for S γ (ψ) is outlined in Alg. 1. Inference. At test time, given current context ψ,we use the learned DSF S γ (ψ) to generate the future trajectory ground set Y. In some cases, Y may still contain some trajectories that are similar to others. In order to obtain a diverse set of trajectories without repetition, we aim to perform MAP inference for the DPP to find the most diverse subset Y * = arg max Y ∈Y P L(γ) (Y). A useful property of DPPs is that the log-probability function is submodular . Even though submodular maximization is NP-hard, we use a greedy algorithm which is a popular optimization procedure that works well in practice. As outlined in Alg. 3, the output set Y f is initialized as ∅, and at each iteration, the trajectory which maximizes the log probability is added to Y f, until the marginal gain becomes negative or Y f = Y. The primary focus of our experiments is to answer the following questions: Are trajectory samples generated with our diversity sampling function more diverse than samples from the cVAE and other baselines? How does our method perform on both balanced and imbalanced data? Is our method general enough to perform well on both low-dimensional and high-dimensional tasks? Figure 2: In real data, contexts (past trajectories) are seldom the same due to noise. Metrics. A problem with trajectory forecasting evaluation is that in real data each context ψ (i) usually only has one future trajectory x (i), which means we only have one sample from a multi-modal distribution. Let us consider a scenario of three data examples as shown in Fig. 2 (red, purple, blue). The contexts (past trajectories) of the three examples are instances of the same trajectory but they are slightly different due to noise. As these three contexts have the same semantic meaning, they should share the future trajectories, e.g., the purple and blue future trajectories are also valid for the red context. If we evaluate each example (x (i), ψ (i) ) only with its own future trajectory x (i), a method can achieve high scores by only forecasting the mode corresponding to x (i) and dropping other modes. This is undesirable because we want a good method to capture all modes of the future trajectory distribution, not just a single mode. To allow for multi-modal evaluation, we propose collecting multiple future trajectories for each example by clustering examples with similar contexts. Specifically, we augment each data example (x (i), ψ (i) ) with a future trajectory set.., M } and metrics are calculated based on X (i) instead of x (i), i.e., we compute metrics for each x ∈ X (i) and average the .;; ). However, these two metrics do not penalize repeated samples. To address this, we introduce two new metrics ASD and FSD to evaluate the similarity between samples in the set of forecasted trajectories. Larger ASD and FSD means the forecasted trajectories are more non-repetitive and diverse. Baselines. We compare our Diversity Sampler Function (DSF) with the following baselines: cVAE: a method that follows the original sampling scheme of cVAE by sampling latent codes from a Gaussian prior p(z). MCL: an approach that uses multiple choice learning to optimize the sampler S γ (ψ) with the following loss: L mcl = minx ∈Y x − x 2, where x is the ground truth future trajectory. R2P2: a method proposed in that uses a reparametrized pushforward policy to improve modeling of multi-modal distributions for vehicle trajectories. cGAN: generative adversarial networks conditioned on the forecasting context. We implement all baselines using similar networks and perform hyperparameter search for each method for fair comparisons. For methods whose samples are stochastic, we use 10 random seeds and report the average for all metrics. We first use synthetic data to evaluate our method's performance for low-dimensional tasks. We design a virtual 2D traffic scene where a vehicle comes to a crossroad and can choose three different future routes: forward, left, and right. We consider two types of synthetic data: Balanced data, which means the probabilities of the vehicle choosing one of the three routes are the same; Table 1: Quantitative on synthetic data (numbers scaled by 10) when N = 10. Imbalanced data, where the probabilities of the vehicle going forward, left and right are 0.8, 0.1, 0.1, respectively. We synthesize trajectory data by simulating the vehicle's behavior and adding Gaussian noise to vehicle velocities. Each data example (x (i), ψ (i) ) contains future trajectories of 3 steps and past trajectories of 2 steps. We also add an obstacle map around the current position to the context ψ (i). In total, we have around 1100 training examples and 1000 test examples. Please refer to Appendix B for more implementation details. Table 1 summarizes the quantitative for both balanced and imbalanced data when the sampling budget N is 10. We can see that our method DSF outperforms the baselines in all metrics in both test settings. As shown in Fig. 3, our method generates more diverse trajectories and is less affected by the imbalanced data distribution. The trajectory samples of our method are also less repetitive, a feature afforded by our DPP formulation. Fig. 4 shows how ADE changes as a function of the sampling budget N. Table 2: Quantitative on for human motion forecasting when N = 10. To further evaluate our method's performance for more complex and high-dimensional tasks, we apply our method to forecast future human motions (pose sequences). We use motion capture to obtain 10 motion sequences including different types of motions such as walking, turning, jogging, bending, and crouching. Each sequence is about 1 minute long and each pose consists of 59 joint angles. We use past 3 poses (0.1s) to forecast next 30 poses (1s). There are around 9400 training examples and 2000 test examples where we use different sequences for training and testing. More implementation details can be found in Appendix B. We present quantitative in Table 2 and we can see that our approach outperforms other methods in all metrics. As the dynamics model used in R2P2 does not generalize well for high-dimensional human motion, we find the model fails to converge and we do not compare with it in this experiment. Fig. 4 shows that our method achieves large improvement when the sampling budget is big (N = 50). We also present qualitative in Fig. 5, where we show the starting pose and the final pose of all 10 forecasted motion samples for each method. We can clearly see that our method generates more diverse future human motions than the baselines. Please refer to Appendix C and our video for additional qualitative . In this section, we perform additional experiments on a large human motion dataset (3.6 million frames), Human3.6M , to evaluate the generalization ability of our approach. We predict future motion of 2 seconds based on observed motion of 0.5 seconds. Please refer to Appendix B.3 for implementation details. We also use a new selection of baselines including several variants of our method (DSF) and the cVAE to validate several design choices of our method, including the choice of the expected cardinality over the negative log likelihood (NLL) of the DPP as the diversity loss. Specifically, we use the following new baselines: DSF-NLL: a variant of DSF that uses NLL as the diversity loss instead of the expected cardinality. DSF-COS: a DSF variant that uses cosine similarity to build the similarity matrix S for the DPP kernel L. DSF-NLL: a variant of the cVAE that samples 100 latent codes and performs DPP MAP inference on the latent codes to obtain a diverse set of latent codes, which are then decoded into trajectory samples. We present quantitative in Table 3 when the number of samples N is 10 and 50. The baseline DSF-COS is able to achieve very high diversity (ASD and FSD) but its samples are overly diverse and have poor quality which is indicated by the large ADE and FDE. Compared with DSF-NLL, Table 3: Quantitative on Human3.6M for N = 10 and N = 50. X means the method is unable to learn a model due to numerical instability. our method achieves better diversity (ASD and FSD) and similar ADE and FDE when the number of samples is small (N = 10). For a larger number of samples (N = 50), NLL becomes unstable even with a large (1e-3) added to the diagonal. This behavior of NLL, i.e., stable for small N but unstable for large N, matches our intuition that NLL becomes unstable when samples become similar (as discussed in Sec. 4.2), because when there are more samples, it is easier to have similar samples during the SGD updates of the DSF network. The baseline cVAE-LDPP also performs worse than DSF in all metrics even though it is able to outperfom the cVAE. We believe the reason is that diversity in sample space may not be well reflected in the latent space due to the non-linear mapping from latent codes to samples induced by deep neural networks. We proposed a novel forecasting approach using a DSF to optimize over the sample space of a generative model. Our method learns the DSF with a DPP-based diversity measure to generate a diverse set of trajectories. The diversity measure is a novel application of DPPs to optimize a set of items in continuous space. Experiments have shown that our approach can generate more diverse vehicle trajectories and human motions compared to state-of-the-art baseline forecasting approaches. 2: Output: cVAE encoder network f φ (x, ψ) and decoder network g θ (z, ψ) 3: Initialize φ and θ randomly 4: while not converged do 5: Compute parameters (µ, σ) of the posterior distribution q φ (z|x, ψ) using f φ (x, ψ) Sample V Gaussian noises {1, . . ., V} from N (0, I) Transform noises to latent samples from q φ (z|x, ψ): Decode latent samples into reconstructed trajectories {x 1, . . .,x V} using g θ (z, ψ) Calculate the cVAE loss L cvae according to Eq. 6 11: Update φ and θ with ∇ φ L cvae and ∇ θ L cvae 12: end for 13: end while Figure 6: Network architectures for synthetic data and human motion. Top: for synthetic data, we use a CNN to process the obstacle map f and directly flatten trajectories x and h into vectors. The reconstructed trajectoryx is decoded with an MLP. Bottom: for human motion, we use Bi-LSTMs to extract temporal features for x and h and decode the reconstructed trajectoryx with a forward LSTM. Synthetic data. Fig. 6 (Top) shows the network architecture for synthetic data. The number of latent dimensions is 2. By default, we use ReLU activation for all networks. The future trajectory x ∈ R 3×2 consists of 3 future positions of the vehicle. The context ψ contains past trajectories h ∈ R 2×2 of 2 time steps and a obstacle map f ∈ {0, 1} 28×28 spanning a 4 × 4 area around the current position of the vehicle (the road width is 2). For the encoder, we use a convolutional neural network (CNN) with three 32-channel convolutional layers to process f. The first two layers have kernel size 4 and stride 2 while the last layer has kernel size 6 and stride 1. The obtained CNN features are concatenated with flattened x and h into a unified feature, which is feed into a multilayer perceptron (MLP). The MLP has one 128-dim hidden layer and two heads outputing the mean µ and variance σ of the latent distribution. For the decoder, we concatenate the CNN feature from f with the latent code z ∈ R 2 and flattened h into a unified feature. The feature is passed through an MLP with one 128-dim hidden layer which outputs the reconstructed future trajectorỹ x ∈ R 3×2. For the diversity sampler function (DSF), we concatenate the CNN feature from f with the flattened h and pass it through an MLP with one 128-dim hidden layer to obtain a set of latent codes {z 1, . . ., z N} which are represented by a vector of length 2N. Human motion. Fig. 6 (Bottom) shows the network architecture for synthetic data. The number of latent dimensions is 8. The future trajectory x ∈ R 30×59 consists of future poses of 30 time steps (1s). The context ψ contains past poses h ∈ R 3×59 of 3 time steps (0.1s). Each pose consists of 59 joint angles. For the encoder, we use two 128-dim bidirectional LSTMs (Bi-LSTMs) and mean pooling to obtain the temporal features for x and h. We then concatenate the temporal features into a unified feature and feed it into an MLP with two hidden layers and two heads to obtain the mean µ and variance σ of the latent distribution. For the decoder, we reuse the Bi-LSTM of the encoder for the context h and a 128-dim forward LSTM to decode the future trajectoryx. At each time step t, the forward LSTM takes as input the previous posex t−1 (h H for t = 0), the latent code z ∈ R 8 and the temporal features from h, and outputs a 128-dim feature. The feature is then passed through an MLP with two hidden layers to generate the reconstructed posex t. For the DSF, we use a different 128-dim Bi-LSTM to obtain the temporal feature for h, which is feed into an MLP with a 128-dim hidden layer to produce a set of latent codes {z 1, . . ., z N} which are represented by a vector of length 8N. When training the cVAE model using Eq. 7, we take V = 1 sample from the posterior q φ (z|x, ψ). The weighting factor β for the KL term is set to 0.1 for synthetic data and 1e-4 for human motion. We use Adam to jointly optimize the encoder and decoder. The learning rate is set to 1e-4 and we use a mini batch size of 32 for synthetic data. We optimize the model for 500 epochs for synthetic data and 100 epochs for human motion. When training the DSF, the scale factor k for the similarity matrix S is set to 1 for synthetic data and 1e-2 for human motions. For both synthetic data and human motions, we use Adam with learning rate 1e-4 to optimize the DSF for 20 epochs. Recall that in the metrics section (Sec. 5.1), we need the grouping threshold ε to build the ground truth future trajectory set X (i) = {x (j) | ψ (j) − ψ (i) ≤ ε, j = 1,..., M }. For synthetic data, ε is set to 0.1 and we only use past trajectories h to compute the distance between contexts. For human motion, ε is set to 0.5. Following previous work (; ;), we convert the motion sequences in the dataset into sequences of 3D joint positions, and adopt a 17-joint skeleton. We train on five subjects (S1, S5, S6, S7, S8), and test on two subjects (S9 and S11). We use the same network architectures (Fig.6 (Bottom)) in this experiment as the one used in the human motion forecasting experiment above. The number of latent dimensions is 128. When training the cVAE model, the weighting factor β is set to 0.1. We sample 5000 training examples every epoch and optimize the cVAE for 500 epochs using Adam and a learning rate of 1e-4. We set the batch size to 64 for the optimization. The scale factor k for the similarity matrix S of the DPP kernel is set to 5. When learning the DSF, we use a batch size of 64 and sample 1000 training examples every epoch and optimize the DSF for 20 epochs using Adam and a learning rate of 1e-3. When computing the metrics, we set the grouping threshold ε to 0.1. We also show additional qualitative for human motion forecasting in Fig. 7. The quality and diversity of the forecasted motions are best seen in our video 2. Figure 7: Additional visualization for human motion forecasting. The left shows the starting pose, and on the right we show for each method the final pose of 10 forecasted motion samples.

represent the short summary for finding the corresponding scientific article (long)

We learn a diversity sampling function with DPPs to obtain a diverse set of samples from a generative model.

Represent the short summary for finding the corresponding scientific article (long)

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.