Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In open-domain dialogue intelligent agents should exhibit the use of knowledge, however there are few convincing demonstrations of this to date. The most popular sequence to sequence models typically “generate and hope” generic utterances that can be memorized in the weights of the model when mapping from input utterance(s) to output, rather than employing recalled knowledge as context. Use of knowledge has so far proved difficult, in part because of the lack of a supervised learning benchmark task which exhibits knowledgeable open dialogue with clear grounding. To that end we collect and release a large dataset with conversations directly grounded with knowledge retrieved from Wikipedia. We then design architectures capable of retrieving knowledge, reading and conditioning on it, and finally generating natural responses. Our best performing dialogue models are able to conduct knowledgeable discussions on open-domain topics as evaluated by automatic metrics and human evaluations, while our new benchmark allows for measuring further improvements in this important research direction. Arguably, one of the key goals of AI, and the ultimate the goal of natural language research, is for humans to be able to talk to machines. In order to get close to this goal, machines must master a number of skills: to be able to comprehend language, employ memory to retain and recall knowledge, to reason about these concepts together, and finally output a response that both fulfills functional goals in the conversation while simultaneously being captivating to their human speaking partner. The current state-of-the-art approaches, sequence to sequence models of various kinds BID20 BID23 BID17 BID21 attempt to address some of these skills, but generally suffer from an inability to bring memory and knowledge to bear; as indicated by their name, they involve encoding an input sequence, providing limited reasoning by transforming their hidden state given the input, and then decoding to an output. To converse intelligently on a given topic, a speaker clearly needs knowledge of that subject, and it is our contention here that more direct knowledge memory mechanisms need to be employed. In this work we consider setups where this can be naturally measured and built. We consider the task of open-domain dialogue, where two speakers conduct open-ended chit-chat given an initial starting topic, and during the course of the conversation the topic can broaden or focus on related themes. During such conversations, an interlocutor can glean new information and personal points of view from their speaking partner, while providing similarly themselves. This is a challenging task as it requires several components not found in many standard models. We design a set of architectures specifically for this goal that combine elements of Memory Network architectures BID19 to retrieve knowledge and read and condition on it, and Transformer architectures BID21 to provide state-of-the-art text representations and sequence models for generating outputs, which we term Transformer Memory Networks. As, to our knowledge, no public domain dataset of requisite scale exists, we build a supervised dataset of human-human conversations using crowd-sourced workers, first crowd-sourcing 1365 diverse discussion topics and then conversations involving 201, 999 utterances about them. Each topic is connected to Wikipedia, and one of the humans (the wizard) is asked to link the knowledge they use to sentences from existing articles. In this way, we have both a natural way to train a knowledgeable conversation agent, by employing a memory component that can recall and ground on this existing text, and a natural way to evaluate models that we build, by assessing their ability at locating and using such knowledge. Our Transformer Memory Network architectures, both in retrieval and generative versions, are tested in this setup using both automatic metrics and human evaluations. We show their ability to execute engaging knowledgeable conversations with humans, compared to a number of baselines such as standard Memory Networks or Transformers. Our new benchmark, publicly in ParlAI (http:// parl.ai/projects/wizard of wikipedia/), aims to encourage and measure further improvements in this important research direction. Many existing dialogue tasks do not study the use of knowledge explicitly. For example, popular chit-chat datasets such as Open-Subtitles BID23, Persona-Chat BID26 and Twitter BID18 have tested the ability of sequence-to-sequence models that attend over the recent dialogue history, but do not attempt to recall long-term knowledge beyond encoding it directly into the weights of the feed-forward network. In the area of goal-directed dialogue, separate from open domain chit-chat, such as airline BID6 or restaurant booking BID9 BID24, knowledge conditioning is typically employed by allowing access to a database through API calls or otherwise. In contrast, our work investigates unstructured knowledge across a large, diverse set of topics potentially spanning all of Wikipedia. In question answering one does not produce a dialogue response based on a conversation history, but a factual answer based on a question. In that case, it is clear that retrieving and conditioning knowledge is vital. For example, in SQuAD neural models have been developed that attend to a given paragraph from Wikipedia to answer questions BID15, or Open-SQuAD which extends this to answering without being given the paragraph, instead performing retrieval over the entirety of Wikipedia BID3. Recently, the QuAC dataset investigates similar themes, but as a sequence of questions and answers in dialogue form instead BID4. In this work we do not address question answering, but focus on natural human dialogues which contain a diverse set of utterances, not just questions and answers. The closest work to ours lies in the area of non-goal directed dialogue incorporating knowledge. The work of BID5 employed Memory Networks to perform dialogue discussing movies in terms of recommendation and open-ended discussion from Reddit, conditioning on a structured knowledge base. also links Reddit to structured knowledge. Both BID14 and BID8 use unstructured text instead, as we do: the former to discuss news articles using Wikipedia summaries as knowledge, and the latter to discuss local businesses in two-turn dialogues using Foursquare tips as knowledge. BID8 uses an extended Encoder-Decoder where the decoder is provided with an encoding of the context along with the external knowledge encoding. Neither involves dialogue authored with the given knowledge, so it is unclear when knowledge is useful or not. In contrast, in our task, we know the Wikipedia articles and sentences that ground crowdworkers dialogues. Model-wise, BID14 uses a Bag-of-Words Memory Network type fact encoder and an RNN decoder. Our work compares Memory Networks BID19 and Transformers which have been shown to be on-par or superior to RNN encoder-decoders BID21, and develops an architecture that combines these approaches. Concurrently with our work BID13 proposed a dataset based on the closed domain of movie chats. Our paper shows models working on full multi-turn dialogue in an open-domain setting, which to our knowledge was not shown before. We consider the following general open-domain dialogue setting: two participants engage in chitchat, with one of the participants selecting a beginning topic, and during the conversation the topic is allowed to naturally change. The two participants, however, are not quite symmetric: one will play the role of a knowledgeable expert (which we refer to as the wizard) while the other is a curious learner (the apprentice).Apprentice At each stage of the conversation the apprentice talks to the wizard freely, playing the role of a curious learner, eager to chat. Their goal is to go into depth about a chosen topic that interests themselves or their partner, while keeping the conversation engaging and fun. Note that the instruction to delve deeply into a topic makes this different to more "shallow" chit-chat tasks. In this task the use of knowledge is emphasized more. Wizard The wizard is given the following instructions: "You have just met the other person, who seems quite curious, and you are eager to discuss a topic with them!" Their goal is to inform their conversation partner about a topic that one of them will choose. Crucially, the wizard has access to an information retrieval system that shows them paragraphs from Wikipedia possibly relevant to the conversation, which are unobserved by the apprentice. Before each conversation turn the wizard can read these paragraphs and then potentially base their next reply on that observed knowledge. Note, the wizard is particularly instructed not to simply parrot this knowledge, but to use it to craft a relevant reply, and to present any relevant knowledge in a fun and engaging way, if possible. The flow of the conversation thus takes place as follows.1. Either the wizard or apprentice is picked to choose the topic and speak first. The other player receives the topic information, and the conversation begins.2. When the apprentice sends the wizard a message, the wizard is shown relevant knowledge (described below), and chooses a relevant sentence in order to construct a response, or else chooses the no sentence used option.3. The Wizard responds to the apprentice basing their response on their chosen sentence.4. The conversation repeats until one of the conversation partners ends the chat (after a minimum of 4 or 5 turns each, randomly chosen beforehand).After collecting data of such wizard-apprentice conversations between humans, the goal is to then replace the human wizard with a learned agent that will speak to a human apprentice instead, similar to the procedure in Wizard of Oz experiments BID0.Topics We crowd-sourced a set of 1365 natural, open-domain dialogue topics, each linked to a Wikipedia article. These include diverse topics such as commuting, Gouda cheese, music festivals, podcasts, bowling, and Arnold Schwarzenegger. Knowledge Retrieval At each step of the dialogue the wizard has access to a set of passages of knowledge which may be relevant to the given dialogue context. While this is a potentially learnable part of the model, we required for this to be fixed so that we could present the to the annotator when collecting the dataset. We thus used exactly the same retriever that is commonly used for the Open-SQuAD dataset in BID3. It uses a simple inverted index lookup followed by term vector model scoring. Articles and queries are compared as TF-IDF weighted bag-of-word and n-gram vectors, using the hashing trick. We retrieve the top 7 articles (first paragraph only) for the last two turns of dialogue (by wizard and apprentice) and the article (first 10 sentences only) for the original topic, and present these articles to the wizard as knowledge context, along with their titles. Note that while this system is used to build the dataset, a superior method can in principle be learned and used by a model at test time. Knowledge Selection and Response Generation During data collection, the wizard can click on any of the retrieved article titles in the dialogue UI to expand that article, at which point they can click on a sentence that is most relevant to the response they want to make (only one article, and one sentence can be selected on any turn, for simplicity). If they see no relevant article or sentence they can choose no sentence used instead. The wizard then enters their response to the apprentice. An image of the Wizard's UI is shown in Appendix A.1. In this work we consider learning dialogue models to replace the wizard in our learning tasks, i.e. the knowledgeable speaker. The dialogue model thus can have access to a knowledge source, in this case Wikipedia, to ground the conversation with. We thus develop extensions of the Memory Network BID19 and Transformer BID21 models that can (i) retrieve from a large memory relevant information conditioned on the dialogue history, (ii) carefully read and attend over the retrieved set of knowledge, and then (iii) generate the next dialogue utterance. This model is then used consecutively on each turn to form an entire dialogue with a user. We develop two classes of models capable of leveraging knowledge: (i) retrieval models that produce an output among a set of candidate responses (the set of utterances from the training set); and (ii) generative models that generate word-by-word (using a beam).The input to either model is the same: at each dialogue turn where the model is intended to make a response, it is given the current dialogue context x 1,..., x t of t dialogue turns, where x 1 is always the initial starting topic (e.g. "Kurt Cobain"), and the remaining turns swap between the two speakers. The goal at each stage is to output the next utterance x t+1.Knowledge Retrieval We assume a large knowledge base (memory) m 1,..., m N which is hierarchically organized into documents consisting of paragraphs and sentences. As it is infeasible for current neural attention techniques to operate on this scale, we use standard information retrieval (IR) techniques (c = IR(x, m)) as a first step to return a smaller set of candidates m c1,..., m c K for fine-grained selection. In our experiments, we use the IR system provided to the human annotators during dataset creation, detailed in Section 3. The retriever operates on the topic (x 1) and the last two turns (x t and x t−1) if they exist, effectively calling the IR system three times with three different queries. Empirically, this provided better performance compared to merging into one query, likely because it can address quite different topics. We retrieve the top 7 articles (first paragraph only) for each lookup and then flatten all the into separate sentences (i.e. remove the organization of sentences belonging to articles), but prepend every sentence with its article title. In this way the candidates m c1,..., m c K given to the neural model in the next stage can be attended to independently without having to deal with hierarchical issues. Knowledge Attention We use an attention mechanism to perform fine-grained selection of which knowledge sentences will be used to produce the next turn of dialogue. Each sentence in the memory is independently encoded with a Transformer encoder BID21, and the same Trans- former is used to encode the dialogue context x. We then perform standard dot-product attention between the memory candidates and the dialogue context. Utterance Prediction Given the hidden state derived from the memory attention process described above, the final stage is to predict the output utterance that will form the next dialogue turn. We consider different variants of the two stages above, knowledge attention and utterance prediction, when considering retrieval and generative variants of our models. We will now detail these in turn. This model encodes each knowledge sentence m c1,..., m c K and the dialogue context x with a Transformer, as described above. The final input encoding is calculated by performing dot-product attention over enc(m c1),..., enc(m c K) and adding the ing weighted sum of these vectors to enc(x) to get the representation rep LHS (m c1, . . ., m c K, x). The candidate responses r 1,..., r L are encoded with a separate Transformer to get rep RHS (r i) for each i. We choose as a response r where DISPLAYFORM0 The model is trained to minimize the cross-entropy loss, where the negative candidates for each example are the responses to the other examples in the batch BID10. We consider two versions: a Two-stage and an End-to-end version. Both models find the most relevant piece of knowledge m best, and then perform an encoding step by concatenating it with the dialogue context, allowing the decoder to attend over both the knowledge and dialogue when formulating its response. We employ a beam search of 5 to select our best response. All generative models employ BPE encoding BID16, which we found effective at enabling generators to copy rare words from Wikipedia sentences BID7.In the End-to-end version, a shared Transformer encoder is used to encode all candidates m ci and the dialogue history. The encoded candidates are flattened into vectors using the normalization from BID2 (summing, and normalizing by the square root of the sentence length in order to balance short and long sentences) to produce an attention prediction over the memory. The full sequence encoding of the single highest selected knowledge m best is concatenated with the encoding of the dialogue, and passed into a Transformer decoder. An illustration of our End-to-end model is shown in FIG0. We train the model to minimize the negative log-likelihood of the response utterance. We can add additional supervision by forcing the knowledge selection to correctly choose the same knowledge candidate as the human wizard in the training set by adding an additional crossentropy loss over the knowledge attention, modulated by a weight λ: DISPLAYFORM0 In the Two-stage version, we employ two separately trained models for each of these two tasks, knowledge selection and utterance prediction. As the knowledge selection step creates a hard deci- sion influencing the output of the generator, we find maximizing the performance of this component to be vital. We can also improve performance of the decoder by employing knowledge dropout (K.D.), wherein we artificially prevent the model from attending to knowledge a fraction of the time during training. We find this helps the generator be more resilient to errors at the knowledge selection stage, and makes training faster. K. D. is a novel technique we propose here, however it is similar to many other dropout techniques, e.g. feature dropout used in BID25. We describe each of our experimental setups and . We first investigate the ability of our models to select knowledge appropriately, and then consider the full task of dialogue with knowledge. Before looking at the full Wizard dialogue task, we assess the ability of models to predict the knowledge selected by human wizards in the dataset given the dialogue history. This will inform us of the feasibility of this task and the best models to use in a two-stage architecture. We compare Transformers against various baselines including a random baseline; an Information Retrieval (IR) baseline, which uses simple word overlap; and a Bag-of-Words Memory Network BID19. Where noted, the Transformer is pretrained on Reddit data BID12, and fine-tuned for our task. The are shown in TAB1. Transformers work best, as long as they are pretrained on a large dataset (Reddit), while multi-tasking on SQuAD provides marginal impact. Further analysis of this task using other models is provided in Appendix B.1. We use the best performing Transformer model reported here for our two-stage generative Memory Network in the full dialogue task. We evaluate our models on the full task of dialogue generation given knowledge in two settings: given the gold knowledge sentence chosen by a human, or where the model needs to predict which knowledge to use. We separately describe experiments for retrieval and generative models. We use similar baselines as in the knowledge selection experiments, but now also apply Transformer Memory Networks, which attend over knowledge. Models are evaluated measuring Recall@1 when ranking the gold response among 99 randomly chosen candidates, and unigram F1 of the model's prediction with the gold response. The are shown in TAB2. We find that the addition of knowledge improves all models (improving Bow MemNet from 56 to 71 R@1 and the Transformer MemNet from 79 to 87 R@1) for predicted knowledge. Performance improves dramatically when models are provided gold knowledge, but otherwise retain similar trends. Generative Experiments We compare our generative End-to-end and Two-stage Transformer Memory Network models to two more baselines: repeating the last utterance, and a generative Transformer model trained to respond to dialogue but without access to knowledge. Models are evaluated using perplexity (PPL) of the gold response and unigram F1. The are given in TAB3. Our experiments show that both the End-to-end and Two-stage models employ the knowledge in their response predictions, as they outperform their counterpart Transformer without knowledge, and demonstrate substantial improvements when provided the gold knowledge. While the Two-stage model produces significantly stronger perplexity and F1 scores using the predicted knowledge, the End-to-end model outperforms the Two-stage model in the gold knowledge experiments. This suggests that the Two-stage model benefits from the strong knowledge selection module (Section 5.1), but that the End-to-end model is better at employing the selected knowledge. Furthermore, we find that the additional knowledge selection supervision (auxiliary loss) in the End-to-end model improves it on every metric, suggesting that tightly integrating these tasks is beneficial. Knowledge dropout (K. D.) also helps (compare last two rows). More evidenceTopic: E-book (seen) Human: i do not know why, but I have never gotten into E-books Model: Same here. I have a kindle, but I don't use it for books. It's nicer to hold a real book and turn pages. Human: yes, i like the physical feel and smell of a real book Model: I actually had a Nook e-book where you had to download the e-book after selecting it online. I grew out of it, I prefer paper books also. Human: i tried e-books but I do not like reading on a screen, its disorientated for me Model: Yes, reading a book on a tablet just isn't the same! Human: plus I like owning the physical copy of a real book, I feel like I own it Model: I prefer print and newsreels. for this is shown in Appendix B.1. Lastly, we note that both Two-stage models give higher F1 scores than any of the retrieval models shown in TAB2. We perform human evaluation of our models using crowd-sourced workers. Humans are paired with our models and asked to chat about a specific topic (given a choice of 2-3 topics) for 3-5 dialogue turns. Following their conversation, the humans are asked to rate their dialogue partner on a scale of 1-5, with the rating indicating how much they "liked" the conversation (5 is best), which we refer to as the engagingness rating. Using the collected conversations, we also calculate a metric we call the Wiki F1 sore: the F1 overlap of the model's utterances with the first 10 sentences of the Wikipedia page for the chosen topic as a proxy for how much knowledge the model exhibits. We seek a model that can be simultaneously engaging and knowledgeable, hence we would like to maximize both these metrics 1. For comparison, we also collect 100 human-human conversations, with only one human choosing the topic and performing evaluation. In total, we collect a total of 546 conversations with ratings from 464 distinct workers. These are shown in TAB4.We find that the retrieval models significantly outperform the generative models on the human engagingness evaluation(Student's t-test, p < .05). The human engagingness differences between retriever models with and without knowledge are not significant, but note they both trend toward use of knowledge due to the candidate sentences retrieved, with the knowledgeable version obtaining significantly higher Wiki F1 scores in both seen and unseen test sets. For the generative models, we find human engagingness ratings are significantly improved by the use of knowledge (p < .01). The significantly higher Wiki F1 scores indicate that (i) these models convey more knowledge than their counterparts without knowledge conditioning; and (ii) on both seen and unseen sets they convey more knowledge than the retrieval models. In particular, on unseen data the gap between retrieval and generative models is larger. This is understandable, as retrieval models are limited to producing a response from the training set where the unseen topic did not appear. There is still a considerable gap to human ratings of other humans compared to all our models (first row of TAB4). FIG2 shows example conversations with the retrieval and generative models. Additional analysis and examples can be found in Appendix B.3 and C. In this work we build dialogue agents which are able to employ large memory systems containing encyclopedic knowledge about the world in order to conduct engaging open-domain conversations. We develop a set of architectures, Transformer Memory Network models, that are capable of retrieving and attending to such knowledge and outputting a response, either in retrieval or generative modes. To train and evaluate such models, we collect the Wizard of Wikipedia dataset, a large collection of open-domain dialogues grounded by Wikipedia knowledge, and demonstrated the effectiveness of our models in automatic and human experiments. Our new publicly available benchmark aims to encourage further model exploration, and we expect such efforts will in significant advances in this important research direction. There is much future work to be explored using our task and dataset. Some of these include: (i) bridging the gap between the engagingness of retrieval responses versus the ability of generative models to work on new knowledge and topics, (ii) learning to retrieve and reason simultaneously rather than using a separate IR component; and (iii) investigating the relationship between knowledge-grounded dialogue and existing QA tasks which also employ such IR systems. The aim is for those strands to come together to obtain an engaging and knowledgeable conversational agent. Examples of collected conversations from the dataset, where both wizard and apprentice are humans. The wizard has access to an information retrieval system over Wikipedia, so that they can ask and answer questions, and make statements relevant to the discussion. For each utterance, knowledge retrieval is performed based on dialogue history, giving ∼61 knowledge candidates per turn, with wizards clicking no sentence used 6.2% of the time. Assuming that a question contains a question mark or begins with'how','why','who','where','what' or'when', in the dataset Apprentices ask questions in 13.9% of training set utterances, and answer questions (i.e., the Wizard has asked a question) 39.5% of the time, while saying new or follow-on statements (neither asking nor answering a question) 49.3% of the time. Hence, the wizard and apprentice conduct conversations with a variety of dialogue acts. To choose between topics that are natural we employed the existing Persona-Chat dataset BID26 where crowdworkers where asked to create personas of typical speakers. There are ∼1000 personas, each of which consists of 4-5 sentences describing that person's interests, e.g. "I love watching Game of Thrones", "I like to eat cheetos" and "I recently got a cat". These can thus naturally be seen as topics of interest, and using another set of annotators we mapped each sentence to 1 or more relevant Wikipedia pages, if possible, e.g. "Ariel is my favorite princess" was labeled with the Wikipedia page for The Little Mermaid. As some sentences are harder to connect with Wikipedia, e.g. "I am witty", they are left unlabeled. We thus obtain 1,431 topics in total to use for our task. We retain the persona topic sets and thus present 2-3 related topic choices as conversation starters per dialogue during data collection. B ADDITIONAL EXPERIMENTS B.1 KNOWLEDGE SELECTION Table 6: Test performance of the Knowledge Selection Tasks. We also tested the performance of our models trained to do the full dialogue task (see Section 5.2) on the knowledge selection task. For our retrieval system, this refers to the performance of the knowledge attention. The show that our retrieval system could be improved, and the auxiliary loss clearly helps the generative models. We perform an analysis of the dialogues produced from the human evaluation experiments detailed in Section 5.3. We sample 20 conversations from each experimental setting, split between seen and unseen. Conversations are re-tokenized and lowercased to reduce superficial differences between models, and then analyzed in a single-blind setup. We note of common errors and behaviors exhibited in each of the different conversations. In general, the human-human conversations are starkly different than any of the bot conversations -humans tend to have more small talk, or use the topic of discussion as a mere icebreaker, with neither human behaving as a wizard. This is in contrast to human-human conversations from the Wizard dataset itself, where one human has access to Wikipedia, and the conversation becomes more grounded in factual sentences. Similarly, all models attempt to play the role of wizard and produce more factual sentences too. In some rounds, humans treat the bot as a sort of question-answer machine, suggesting that the models could be improved by additionally employing SQuAD-like training data. The retriever without knowledge is particularly prone to non sequiturs, or rapidly changing the subject. During unseen conversations, it is especially likely to discuss something other than the chosen topic. In contrast, the retriever with knowledge tends to stick to the chosen topic strongly, but has difficulty if the human changes the subject. Frequently in unseen topics, the retriever with Table 7: Retrieval methods on the full Wizard task. In addition to the models we tested in the paper, we also tested a two-stage retrieval system in which we used our best-performing model on the knowledge selection task to choose a single knowledge sentence to condition on for the dialogue retrieval task. This outperformed our best retrieval method in terms of F1 but not not in terms of Recall@1. Furthermore, we compared these to a two-stage retrieval system in which the dialogue retrieval module is optimized for seeing the gold chosen knowledge sentence. The performance of this system on the gold knowledge task suggests that the retrieval system could be improved by increasing performance on the knowledge selection subtask. Table 8: Human Experiments. We calculate the Wiki F1 score for the wizard and apprentice as they appear in the dataset for the sake of comparison to our human evaluations. Note that this differed from the human-human evaluation set-up in the sense that the wizard had direct access to Wikipedia passages in the UI, which explains the higher values of Wiki F1 both for the wizard (who uses Wikipedia) and for the apprentice (who would likely reference that use). knowledge produces similar, but factually inaccurate answers to user queries. For example, when one user asks about parts of Ireland to visit, the model enumerates a list of locations in Greece. Nonetheless, its repertoire of available responses often include inviting responses, allowing the bot to have a more natural conversational flow. Selected conversations with the retriever with knowledge may be found in FIG5, for both seen and unseen topics. The generator without knowledge is particularly prone to many of the typical behaviors of seq2seq systems BID11 BID22, including local repetition ("cookies are made of flour, oil, oil, and oil"), global repetition (producing the near same utterance for multiple turns), or inconsistencies in its personality (saying it both likes and dislikes movies). The generator with knowledge has significantly fewer issues with repetition, as it errs on the side of copying large fragments from the Wikipedia knowledge. The generator with knowledge can also act as a selfish conversationalist, choosing to respond or detail information without inviting a response. Although it generally produces accurate statements, it sometimes produces statements using an incorrect date, name or word. It also frequently produces formulaic responses, like "I don't know, but I do know that [Wikipedia excerpt]". Nonetheless, we find the generator with knowledge is able to successfully generalize to unseen topics using the knowledge from Wikipedia. Selected conversations with the generator with knowledge may be found in Figure 5.

represent the short summary for finding the corresponding scientific article (long)

We build knowledgeable conversational agents by conditioning on Wikipedia + a new supervised task.

Represent the short summary for finding the corresponding scientific article (long)

We show language understanding via reading is promising way to learn policies that generalise to new environments.


Question: represent the scientific article to find a short tldr

Question: We formulate a new problem at the intersection of semi-supervised learning and contextual bandits, motivated by several applications including clinical trials and dialog systems. We demonstrate how contextual bandit and graph convolutional networks can be adjusted to the new problem formulation. We then take the best of both approaches to develop multi-GCN embedded contextual bandit. Our algorithms are verified on several real world datasets. We formulate the problem of Online Partially Rewarded (OPR) learning. Our problem is a synthesis of the challenges often considered in the semi-supervised and contextual bandit literature. Despite a broad range of practical cases, we are not aware of any prior work addressing each of the corresponding components. Online: data incrementally collected and systems are required to take an action before they are allowed to observe any feedback from the environment. Partially: oftentimes there is no environment feedback available, e.g. a missing label Rewarded: instead of the true label, we can only hope to observe feedback indicating whether our prediction is good or bad (1 or 0 reward), the latter case obscuring the true label for learning. Practical scenarios that fall under the umbrella of OPR range from clinical trials to dialog orchestration. In clinical trials, reward is partial, as patients may not return for followup evaluation. When patients do return, if feedback on their treatment is negative, the best treatment, or true label, remains unknown. In dialog systems, a user's query is often directed to a number of domain specific agents and the best response is returned. If the user provides negative feedback to the returned response, the best available response is uncertain and moreover, users can also choose to not provide feedback. In many applications, obtaining labeled data requires a human expert or expensive experimentation, while unlabeled data may be cheaply collected in abundance. Learning from unlabeled observations is the key challenge of semi-supervised learning BID2. We note that the problem of online semi-supervised leaning is rarely considered, with few exceptions BID14 BID13. In our setting, the problem is further complicated by the bandit-like feedback in place of labels, rendering existing semi-supervised approaches inapplicable. We will however demonstrate how one of the recent approaches, Graph Convolutional Networks (GCN) BID9, can be extended to our setting. The multi-armed bandit problem provides a solution to the exploration versus exploitation tradeoff while maximizing cumulative reward in an online learning setting. In Linear Upper Confidence Bound (LINUCB) BID10 BID4 and in Contextual Thompson Sampling (CTS) BID0, the authors assume a linear dependency between the expected reward of an action and its context. However, these algorithms assume that the bandit can observe the reward at each iteration. Several authors have considered variations of partial/corrupted rewards BID1 BID6, but the case of entirely missing rewards has not been studied to the best of our knowledge. The rest of the paper is structured as follows. In section 2, we formally define the Online Partially Rewarded learning setup and present two extensions to GCN to suit our problem setup. Section 3 presents quantitative evidence of these methods applied to four datasets and analyses the learned latent space of these methods. We first formally define each of the OPR keywords:Online: at each step t = 1,..., T we observe observation x t and seek to predict its labelŷ t using x t and possibly any information we had obtained prior to step t. Partially: after we make a predictionŷ t, the environment may not provide feedback (we will use -1 to encode its absence) and we must proceed to step t + 1 without knowledge of the true y t.Rewarded: suppose there are K possible labels y t ∈ {1, . . ., K}. The environment at step t will not provide true y t, but instead a response h t ∈ {−1, 0, 1}, where h t = 0 indicatesŷ t = y t and h t = 1 indicatesŷ t = y t (-1 indicates missing response). UpdateÂ with new edges using x t Update GCN weights W DISPLAYFORM0 Retrieve GCN embeddings g(X) (k) 8: DISPLAYFORM1 (k) t 10: DISPLAYFORM2 (k) t Predictŷ t = argmax k (µ k + σ k) and observe h t 14: DISPLAYFORM0 ∀k, y DISPLAYFORM1 Append t to each C k and 1 to r ·,k ifŷ t = k and 0 otherwise DISPLAYFORM2 Append t to Cŷ t, output ofŷ t -th GCN to r ·,ŷtRewarded Online GCN (ROGCN) is a natural extension of GCN, adapted to the online, partially rewarded setting along with a potential absence of true graph information. We assume availability of a small portion of data and labels (size T 0) available at the start, X 0 ∈ R T0×D and y 0 ∈ {−1, 1, . . ., K} T0. When there is no graph available we can construct a k-NN graph (k is a parameter chosen a priori) based on similarities between observations -this approach is common in convolutional neural networks on feature graphs BID8 BID5 and we adopt it here for graph construction between observations X 0 to obtain graph adjacency A 0. Using X 0, y 0, A 0, we can train GCN with L hidden units (a parameter chosen a priori) to obtain initial estimates of hidden layer weights W 1 ∈ R D×L and softmax weights W 2 ∈ R L×K. Next we start to observe the stream of data -as new observation x t arrives, we add it to the graph and data matrix, and append -1 (missing label) to y. Then we run additional training steps of GCN and output a prediction to obtain environment response h t ∈ {−1, 0, 1}. Here 1 indicates correct prediction, hence we include it to the set of available labels for future predictions; 0 indicates wrong prediction and -1 an absence of a response, in the later two cases we continue to treat the label of x t as missing. ROGCN is unable to learn from missclassified observations and has to treat them as missing labels. The bandit perspective allows one to learn from missclassfied observations, i.e. when the environment response h t = 0, and the neural network perspective facilitates learning better features such that linear classifier is sufficient. This observation motivates us to develop a more sophisticated synthesis of GCN and LINUCB approaches, where we can combine advantages of both perspectives. Notice that if K = 2, a h t = 0 environment response identifies the correct class, hence the OPR reduces to online semi-supervised learning for which GCN can be trivially adjusted using ideas from ROGCN. To take advantage of this for K > 2, we propose to use a suite of class specific GCNs, where the hidden layer representation from the k-th class GCN, i.e. g(X)(k) =Â ReLU(ÂXW (k) 1 ) and g(X) (k) t denotes the embedding of observation x t, is used as context by the contextutal bandit for the predictions of the k-th arm. Based on the environment response to the prediction, we update the labels and the reward information to reflect a correct, incorrect, or a missing environment response. The reward is imputed from the corresponding GCN when the response is missing. As we add new observation x t+1 to the graph and update weights of the GCNs, the embedding of the previous observations x 1,..., x t evolves. Therefore instead of dynamically updating bandit parameters, we maintain a set of indices for each of the arms C k = {t :ŷ t = k or h t = 1} and use observations and responses from only these indices to update the corresponding bandit parameters. Similar to ROGCN, we can use a small amount of data X 0 and labels y 0 converted to binary labels y DISPLAYFORM0 T0 (as before -1 encodes missing label) for each class k to initialize GCNs weights DISPLAYFORM1 for k = 1,..., K. We present the GCNUCB in Algorithm 1, where r t,k ∈ denotes the reward observed or imputed at step t for arm k as described in the algorithm. In this section we compare baseline method LINUCB which ignores the data with missing rewards to ROGCN and GCNUCB. We consider four different datasets: CNAE-9 and Internet Advertisements from the the UCI Machine Learning Repository 1, Cora 2, and Warfarin BID12. Cora is naturally a graph structured data which can be utilized by ROGCN and GCNUCB. For other datasets we use a 5-NN graph built online from the available data as follows. Suppose at step t we have observed data points x i ∈ R D for i = 1,..., t. Weights of the similarity graph computed as follows: DISPLAYFORM0. As it was done by we set σ = A ij is the diagonal matrix of node degrees. For pre-processing we discarded features with large magnitudes (3 features in Internet Advertisements and 2 features in Warfarin) and row normalized all observations to have unit l 1 norm. For all the methods that use GCN, we use 16 hidden units for GCN, and use Adam optimizer with a learning rate of 0.01, and regularization strength of 5e-4, along with a dropout of 0.5. To simulate the OPR setting, we randomly permute the order of the observations in a dataset and remove labels for 25% and 75% of the observations chosen at random. For all methods we consider initial data X 0 and y 0 to represent a single observation per class chosen randomly (T 0 = K). At a step t = T 0 + 1,..., T each algorithm is given a feature vector x t and is ought to make a predictionŷ t. The environment response h t ∈ {−1, 0, 1} is then observed and algorithms moves onto step t + 1. To compare performance of different algorithms at each step t we compareŷ t to true label y t available from the dataset (but concealed from the algorithms themselves) to evaluate running accuracy. DISPLAYFORM1 For GCNUCB we use baseline LINUCB for first 300 steps, and for both we use explorationexploitation trade-off parameter α = 0.25. Results are summarized in TAB0. Since ordering of the data can affect the problem difficulty, we performed 10 data resampling for each setting to obtain error margins. GCNUCB outperforms the LINUCB baseline and ROGCN in all of the ex-periments, validating our intuition that a method synthesizing the exploration capabilities of bandits coupled with the effective feature representation power of neural networks is the best solution to the OPR problem. We see the greatest increase in accuracy between GCNUCB and the alternative approaches on the Cora dataset which has a natural adjacency matrix. This suggests that GCNUCB has a particular edge in OPR applications with graph structure. Such problems are ubiquitous. Consider our motivating example of dialog systems -for dialog systems deployed in social network or workplace environments, there exists graph structure between users, and user information can be considered alongside queries for personalization of responses. Visualizing GCNUCB context space. Recall that the context for each arm of GC-NUCB is provided by the corresponding binary GCN hidden layer. The motivation for using binary GCNs to provide the context to LINUCB is the ability of GCN to construct more powerful features using graph convolution and neural networks expressiveness. To see how this procedure improves upon the baseline LINUCB utilizing input features as context, we project the context and the corresponding bandit weight vectors, θ 1,..., θ K, for both LINUCB and GCNUCB to a 2-dimensional space using t-SNE BID11. In this experiment we analyzed CNAE-9 dataset with 25% missing labels. Recall that the bandit makes prediction based on the upper confidence bound of the regret: argmax k (θ k x k,t + σ k) and that x k,t = x t ∀k = 1,..., K for LINUCB and x k,t = g(X) (k) t for GCNUCB. To better visualize the quality of the learned weight vectors, for this experiment we set α = 0 and hence σ k = 0 ing in a greedy bandit, always selecting an arm maximizing expected reward θ k x t,k. In this case, a good combination of contexts and weight vectors is the one where observations belonging to the same class are well clustered and corresponding bandit weight vector is directed at this cluster. For LINUCB FIG2, 68% accuracy) the bandit weight vectors mostly point in the direction of their respective context clusters, however the clusters themselves are scattered, thereby inhibiting the capability of LINUCB to effectively distinguish between different arms given the context. In the case of GCNUCB (Figure 2, 77% accuracy) the context learned by each GCN is tightly clustered into two distinguished regions -one with context for corresponding label and binary GCN when it is the correct label (points with bolded colors), and the other region with context for the label and GCN when a different label is correct (points with faded colors). The tighter clustered contexts allow GCNUCB to effectively distinguish between different arms by assigning higher expected reward to contexts from the correct binary GCN than others, thereby ing in better performance of GCNUCB than other methods.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Synthesis of GCN and LINUCB algorithms for online learning with missing feedbacks

represent the short summary for finding the corresponding scientific article (long)
To give you a sense - "We propose a model that learns to discover informative frames in a future video sequence and represent the video via its keyframes." should be close to "To flexibly and efficiently reason about temporal sequences, abstract representations that compactly represent the important information in the sequence are needed. One way of constructing such representations is by focusing on the important events in a sequence. In this paper, we propose a model that learns both to discover such key events (or keyframes) as well as to represent the sequence in terms of them. We do so using a hierarchical Keyframe-Inpainter (KeyIn) model that first generates keyframes and their temporal placement and then inpaints the sequences between keyframes. We propose a fully differentiable formulation for efficiently learning the keyframe placement. We show that KeyIn finds informative keyframes in several datasets with diverse dynamics. When evaluated on a planning task, KeyIn outperforms other recent proposals for learning hierarchical representations. When thinking about the future, humans focus their thoughts on the important things that may happen (When will the plane depart?) without fretting about the minor details that fill each intervening moment (What is the last word I will say to the taxi driver?). Because the vast majority of elements in a temporal sequence contains redundant information, a temporal abstraction can make reasoning and planning both easier and more efficient. How can we build such an abstraction? Consider the example of a lead animator who wants to show what happens in the next scene of a cartoon. Before worrying about every low-level detail, the animator first sketches out the story by keyframing, drawing the moments in time when the important events occur. The scene can then be easily finished by other animators who fill in the rest of the sequence from the story laid out by the keyframes. In this paper, we argue that learning to discover such informative keyframes from raw sequences is an efficient and powerful way to learn to reason about the future. Our goal is to learn such an abstraction for future image prediction. In contrast, much of the work on future image prediction has focused on frame-by-frame synthesis . This strategy puts an equal emphasis on each frame, irrespective of the redundant content it may contain or its usefulness for reasoning relative to the other predicted frames. Other recent work has considered predictions that "jump" more than one step into the future, but these approaches either used fixed-offset jumps or used heuristics to select the predicted frames (; ;). In this work, we propose a method that selects the keyframes that are most informative about the full sequence, so as to allow us to reason about the sequence holistically while only using a small subset of the frames. We do so by ensuring that the full sequence can be recovered from the keyframes with an inpainting strategy, similar to how a supporting animator finishes the story keyframed by the lead. One possible application for a model that discovers informative keyframes is in long-horizon planning. Recently, predictive models have been employed for model-based planning and control ). However, they reason about every single future time step, limiting their applicability to short horizon tasks. In contrast, we show that a model that reasons about the future using a small set of informative keyframes enables visual predictive planning for horizons much greater than previously possible by using keyframes as subgoals in a hierarchical planning framework. Figure 1: Keyframing the future. Instead of predicting one frame after the other, we propose to represent the sequence with the keyframes that depict the interesting moments of the sequence. The remaining frames can be inpainted given the keyframes. To discover informative frames in raw sequence data, we formulate a hierarchical probabilistic model in which a sequence is represented by a subset of its frames (see Fig. 1). In this two-stage model, a keyframing module represents the keyframes as well as their temporal placement with stochastic latent variables. The images that occur at the timepoints between keyframes are then inferred by an inpainting module. We parametrize this model with a neural network and formulate a variational lower bound on the sequence log-likelihood. Optimizing the ing objective leads to a model that discovers informative future keyframes that can be easily inpainted to predict the full future sequence. Our contributions are as follows. We formulate a hierarchical approach for the discovery of informative keyframes using joint keyframing and inpainting (KEYIN). We propose a soft objective that allows us to train the model in a fully differentiable way. We first analyze our model on a simple dataset with stochastic dynamics in a controlled setting and show that it can reliably recover the underlying keyframe structure on visual data. We then show that our model discovers hierarchical temporal structure on more complex datasets of demonstrations: an egocentric gridworld environment and a simulated robotic pushing dataset, which is challenging for current approaches to visual planning. We demonstrate that the hierarchy discovered by KEYIN is useful for planning, and that the ing approach outperforms other proposed hierarchical and non-hierarchical planning schemes on the pushing task. Specifically, we show that keyframes predicted by KEYIN can serve as useful subgoals that can be reached by a low-level planner, enabling long-horizon, hierarchical control. Hierarchical temporal structure. Hierarchical neural models for efficiently modeling sequences were proposed in;. These approaches were further extended to predict with an adaptive step size so as to leverage the natural hierarchical structure in language data (; Kádár et al., 2018). However, these models rely on autoregressive techniques for text generation and applying them to structured data, such as videos, might be impractical. The video processing community has used keyframe representations as early as 1991 in the MPEG codec . adapted this algorithm in the context of neural compression; however, these approaches use constant offsets between keyframes and thus do not fully reflect the temporal structure of the data. Recently, several neural methods were proposed to leverage such temporal structure. and propose models that find and predict the least uncertain "bottleneck" frames. construct a representation that can be used to predict any number of frames into the future. In contrast, we propose an approach for hierarchical video representation that discovers the keyframes that best describe a certain sequence. In parallel to our work, propose a related method for video segmentation via generative modeling. focus on using the discovered task boundaries for training hierarchical RL agents, while we show that our model can be used to perform efficient hierarchical planning by representing the sequence with only a small set of keyframes. Also concurrently, propose a similar method to KEYIN for learning temporal abstractions. focuses on learning hierarchical state-space models, we propose a model that operates directly in the observation space and performs joint keyframing and inpainting. Video modeling. Early approaches to probabilistic video modeling include autoregressive models that factorize the distribution by considering pixels sequentially ). To reason about the images in the video holistically, latent variable approaches were developed based on variational inference (; ;), including (; ;) and large-scale models such as . is a recently proposed approach that uses exact inference based on normalizing flows . We build on existing video modeling approaches and show how they can be used to learn temporal abstractions with a novel keyframe-based generative model. Visual planning and model predictive control. We build on recent work that explored applications of learned visual predictive models to planning and control. Several groups (; ;) have proposed models that predict the consequences of actions taken by an agent given its control output. Recent work (; ; has shown that visual model predictive control based on such models can be applied to a variety of different settings. In this work, we show that the hierarchical representation of a sequence in terms of keyframes improves planning performance in the hierarchical planning setting. Figure 2: A probabilistic model for jointly keyframing and inpainting a future sequence. First, a sequence of keyframes K 1:N is generated, as well as corresponding temporal indices τ 1:N, defining the structure of the underlying sequence. In the second stage, for each pair of keyframes K n and K n+1, the frames I τ n :τ n+1 −1 are inpainted. Our goal is to develop a model that generates sequences by first predicting key observations and the time steps when they occur and then filling in the remaining observations in between. To achieve this goal, in the following we (i) define a probabilistic model for joint keyframing and inpainting, and (ii) show how a maximum likelihood objective leads to the discovery of keyframe structure. We first describe a probabilistic model for joint keyframing and inpainting of a sequence I 1:T. The model consists of two parts: the keyframe predictor and the sequence inpainter (see Fig. 2). The keyframe predictor takes in C conditioning frames I co and produces N keyframes K 1:N as well as the corresponding time indices τ 1:N: From each pair of keyframes, the sequence inpainter generates the sequence of frames in between: which completes the generation of the full sequence. The inpainter additionally observes the number of frames it needs to generate τ n+1 − τ n. The temporal spacing of the most informative keyframes is data-dependent: shorter keyframe intervals might be required in cases of rapidly fluctuating motion, while longer intervals can be sufficient for steadier motion. Our model handles this by predicting the keyframe indices τ and inpainting τ n+1 −τ n frames between each pair of keyframes. We parametrize the prediction of τ n in relative terms by predicting offsets δ n: τ n = τ n−1 + δ n. To produce a complex multimodal distribution over K we use a per-keyframe latent variable z with prior distribution p(z) and approximate posterior q(z|I, I co). 1 We construct a variational lower bound on the likelihood of both I and K as follows 2: In practice, we use a weight β on the KL-divergence term, as is common in amortized variational inference (; ;). If a simple model is used for inpainting, most of the representational power of the model has to come from the keyframe predictor. We use a relatively powerful latent variable model for the keyframe predictor and a simpler Gaussian distribution produced with a neural network for inpainting. Because of this structure, the keyframe predictor has to predict keyframes that describe the underlying sequence well enough to allow a simpler inpainting process to maximize the likelihood. We will show that pairing a more flexible keyframe predictor with a simpler inpainter allows our model to discover semantically meaningful keyframes in video data. Ground Truth Sequence For each predicted keyframeK n we compute a target imagẽ K n as the sum of the ground truth images weighted with the corresponding distribution over index τ n. Finally, we compute the reconstruction loss between the estimated imagê K n and the soft targetK n. Our model can dynamically predict the keyframe placement τ n. However, learning a distribution over the discrete variable τ n is challenging due to the expensive evaluation of the expectation over p(τ n |z 1:n, I co) in the objective in Eq. 3. To be able to evaluate this term efficiently and in a differentiable manner while still learning the keyframe placement, we propose a continuous relaxation of the objective. The placement distribution τ n defines a probability for each predicted frame to match to a certain frame in the ground truth sequence. Instead of sampling from this distribution to pick a target frame we produce a soft target for each predicted frame by computing the expected target frame, i.e. the weighted sum of all frames in the true sequence, each multiplied with the probability of matching to the predicted frame. When the entropy of τ n converges to zero, the continuous relaxation objective is equivalent to the original, discrete objective. Keyframe targets. To produce a keyframe target,K n, we linearly interpolate between the ground truth images according to the predicted distribution over the keyframe's temporal placement τ n: K n = t τ n t I t, where τ n t is the probability that the n th keyframe occurs at timestep t. This process is depicted in Fig. 3. We parametrize temporal placement prediction in terms of offsets δ with a maximum offset of J. Because of this, the maximum possible length of the predicted sequence is N J. It is desirable for J to be large enough to be able to capture the distribution of keyframes in the data, but this may lead to Figure 4: Sequences generated by KEYIN and a method with constant temporal keyframe offset (Jumpy) on Brownian Motion data. Generation is conditioned on the first five frames. The first half of the sequence is shown. Movement direction changes are marked red in the ground truth sequence and predicted keyframes are marked blue. We see that KEYIN can correctly reconstruct the motion as it selects an informative set of keyframes. The sequence generated by the Jumpy method does not reproduce the direction changes since they cannot be inferred from the selected keyframes. the generation of sequences longer than the target N J > T. To correctly compute the value of the relaxed objective in this case, we discard predicted frames at times > T and normalize the placement probability output by the network so that it sums to one over the first T steps. Specifically, for each keyframe we compute this probability as c n: The loss corresponding to the last two terms of Eq. then becomes: Inpainting targets. To complete our relaxed objective, for each ground truth frame, we produce a target image composed from the inpainted frames. 4 We note that as offsets δ have a maximum range of J, and in general have non-zero probability on each timestep, the inpainting network needs to produce J framesÎ n 1:J between each pair of keyframes K n, K n+1. As in the previous section, the targets for ground truth images are given as an interpolation between generated images weighted by the probability of the predicted frameÎ n j being matched to ground truth frame I t: Here, m n j,t is the probability that the j-th predicted image in segment n has an offset of t from the beginning of the predicted sequence, which can be computed from τ n. To obtain a probability distribution over produced frames, we normalize the with n,j m n j,t. The full loss for our model is: We show how to instantiate KEYIN with deep neural networks and train it on high-dimensional observations, such as images. We further describe an effective training procedure for KEYIN. We use a common encoder-recurrent-decoder architecture . Video frames are first processed with a convolutional encoder module to produce image embeddings ι t = CNN enc (I t). Inferred frame embeddingsι are decoded with a convolutional decoderÎ The keyframe predictor p(K 1:N, τ 1:N |z 1:N, I co) is parametrized with a Long Short-Term Memory network . To condition the keyframe predictor on past frames, we initialize its state with the final state of another LSTM that processes the conditioning frames. Similarly, we parametrize the sequence inpainter p(I τ n :τ n+1 |K n, K n+1, τ n+1 − τ n) with an LSTM. We condition the inpainting on both keyframe embeddings,κ n−1 andκ n, as well as the temporal offset between the two, δ n, by passing these inputs through a multi-layer perceptron that produces the initial state of the inpainting LSTM. The generation is conditioned on a single ground truth frame. Twelve of the 30 predicted frames are shown. We observe that for each transition between pushes and each action of the Gridworld agent our network predicts a keyframe either exactly at the timestep of the event or one timestep apart. Note, although agent position is randomized, objects not visible in the first image can be predicted in Gridworld because the maze is fixed across episodes. We use a Gaussian distribution with identity variance as the output distribution for both the keyframe predictor and the inpainting model and a multinomial distribution for δ n. We parametrize the inference q(z 1:N |I −C+1:T) with an LSTM with attention over the entire input sequence. The inference distribution is a diagonal covariance Gaussian, and the prior p(z 1:N) is a unit Gaussian. Further details of the inference procedure are given in Sec. B and Fig. 8 of the Appendix. We train our model in two stages. First, we train the sequence inpainter to inpaint between ground truth frames sampled with random offsets, thus learning interpolation strategies for a variety of different inputs. In the second stage, we train the keyframe predictor using the loss from Eq. 5 by feeding the predicted keyframe embeddings to the inpainter. In this stage, the weights of the inpainter are frozen and are only used to backpropagate errors to the rest of the model. We found that this simple two-stage procedure improves optimization of the model. We use L1 reconstruction losses to train the keyframe predictor. We found that this and adding a reconstruction loss on the predicted embeddings of the keyframes, weighted with a factor β κ, improved the ability of the model to produce informative keyframes. Target embeddings are computed using the same soft relaxation used for the target keyframes. More details of the loss computation are given in Sec. E and Algorithm 1 of the Appendix. We evaluate the quality of KEYIN's representation for future sequences by addressing the following questions: (i) Can it discover and predict informative keyframes? (ii) Can it model complex data distributions? (iii) Is the discovered hierarchy useful for long-horizon hierarchical planning? Datasets. We evaluate our model on three datasets containing structured long-term behavior. The Structured Brownian motion (SBM) dataset consists of binary image sequences of size 32 × 32 pixels in which a ball randomly changes directions after periods of straight movement of six to eight frames. The Gridworld Dataset consists of 20k sequences of an agent traversing a maze with different objects. The agent sequentially navigates to objects and interacts with them following a task sketch. We use the same maze for all episodes and randomize the initial position of the agent and the task sketch. We use 64 × 64 pixel image observations and further increase visual complexity by constraining the field of view to a 5 × 5-cells egocentric window. The Pushing Dataset consists of 50k sequences of a robot arm pushing a puck towards a goal on the opposite side of a wall. Each sequence consists of six consecutive pushes. We vary start and target position of the puck, as well as the placement of the wall. The demonstrations were generated with the MuJoCo simulator at a resolution of 64 × 64 pixels. For more details on the data generation process, see Sec. D of the Appendix. Further details about the experimental setup are given in Sec. C of the Appendix. 6.1 KEYFRAME DISCOVERY To evaluate KEYIN's ability to discover keyframes, we train KEYIN on all three datasets with N = 6, which can be interpreted as selecting the N most informative frames from a sequence. We show qualitative examples of keyframe discovery for the SBM dataset in Fig. 4 and for the Gridworld and Pushing datasets in Fig. 5. On all datasets the model discovers meaningful keyframes which mark direction changes of the ball, transitions between pushes or interactions with objects, adapting its keyframe prediction patterns to the data. Consequently, the inpainter network is able to produce frames of high visual quality. Misplaced keyframes yield blurry interpolations, as can be seen for the jumpy prediction in Fig. 4. This suggests that keyframes found by KEYIN describe the overall sequences better. Ground Truth Generated We see that our model covers both modes of the distribution, producing both trajectories that go to the right and to the left of the obstacle. To show that KEYIN discovers informative keyframes, we compare keyframe predictions against an alternative approach that measures the surprise associated with observing a frame given the previous frames. This approach selects keyframes as the N frames with the largest peaks in "surprise" as measured by the KL-divergence D KL [q(z t |I 1:t)||p(z t)] between the prior and the posterior of a stochastic predictor based on (see Sec. F and Algorithm 2 of the Appendix for details). We provide comparisons to alternative formulations of surprise in Appendix Sec. F, Tab. 3. For quantitative analysis, we define approximate ground truth keyframes to be the points of direction change for the SBM dataset, the moments when the robot lifts its arm to transitions between pushes, or when the agent interacts with objects in the gridworld. We report F1 scores that capture both the precision and recall of keyframe discovery. We additionally compare to random keyframe placement, and a learned but static baseline that is the same for all sequences. The evaluation in Tab. 1 shows that KEYIN discovers better keyframes than alternative methods. The difference is especially large on the more complex Pushing and Gridworld datasets. The surprise-based method does not reason about which frames are most helpful to reconstruct the entire trajectory and thus is unable to discover the correct structure on the more complex datasets. In addition to the F1 scores, we report temporal distance between predicted and annotated keyframes in Appendix, Tab. 4, also indicating that KEYIN is better able to discover the temporal structure in both datasets. Even though the focus of this work is on discovering temporal structure via keyframing and not on improving video prediction quality, we verify that KEYIN can represent complex data distributions in terms of discovered keyframes and attains high diversity and visual quality. We show sample generations from our model on the Pushing and Gridworld datasets on the supplementary website 5. We see that KeyIn is able to faithfully model complex distributions of video sequences. We further visualize multiple sampled Pushing sequences from our model conditioned on the same start position in Fig. 6, showing that KEYIN is able to cover both modes of the demonstration distribution. We further show that KEYIN compares favorably to prior work on video prediction metrics on sequence modeling in Tab. 5 of the Appendix, and outperforms prior approaches in terms of keyframe modeling in Appendix, Tab. 6. 6.3 ROBUSTNESS OF KEYFRAME DETECTION In the previous sections, we showed that when the sequence can indeed be summarized with N keyframes, KEYIN predicts the keyframes that correspond to our notion of salient frames. However, what happens if we train KEYIN to select a larger or a smaller amount of keyframes? To evaluate this, we measure KEYIN recall with extra and precision with fewer available keyframes. We note that high precision is unachievable in the first case and high recall is unachievable in the second case, since these problems are misspecified. As these numbers are not informative, we do not report them. In Tab. 2, we see that KEYIN is able to find informative keyframes even when N does not exactly match the structure of the data. We further qualitatively show that KEYIN selects a superset or a subset of the original keyframes respectively in Sec. G. This underlines that our method's ability to discover keyframe structure is robust to the choice of the number of predicted keyframes. As a first step towards analyzing the robustness of KEYIN under more realistic conditions we report keyframe discovery when trained and tested on sequences with additive Gaussian noise, a noise characteristic commonly found in real-world camera sensors. We find that KEYIN is still able to discover the temporal structure on both the Pushing and the Gridworld dataset. For qualitative and quantitative , see Appendix Fig. 11 and Tab. 7. We have seen that KEYIN can find frames that correspond to an intuitive notion of keyframes. This demonstrates that the keyframes discovered by KEYIN do indeed capture an abstraction that compactly describes the sequence. In light of this, we hypothesize that an informative set of keyframes contains sufficient information about a sequence to effectively follow the trajectory it shows. To test this, we use the inferred keyframes as subgoals for hierarchical planning in the pushing environment. During task execution, we first plan a sequence of keyframes that reaches the target using our learned keyframe predictor. Specifically, we generate keyframe trajectories from our model by sampling latent variables z from the prior and using them to roll out the keyframe prediction model. We optimize for a sequence of latent variables z that in a keyframe trajectory which reaches the goal using the Cross-Entropy Method . We then execute the plan by using the keyframes as subgoals for a low-level planner. This planner reaches each subgoal via model predictive control using ground truth dynamics, again employing CEM for optimization of the action trajectory. This planning procedure is illustrated in Fig. 7 (left). For more details, see Sec. I and Algs. 3 and 4 of the Appendix. We find that KEYIN is able to plan coherent subgoal paths towards the final goal that often lead to successful task execution (executions are shown on the supplementary website 6). To quantitatively evaluate the keyframes discovered, we compare to alternative subgoal selection schemes: fixed time offset (Jumpy, similar to), a method that determines points of peak surprise (Surprise, see Sec. 6.1), and a bottleneck-based subgoal predictor (time-agnostic prediction or). We additionally compare to an approach that plans directly towards the final goal using the low-level planner (Flat). We evaluate all methods with the shortest path between the target and the actual position of the object after the plan is executed. All compared methods use the same low-level planner as we only want to measure the quality of the predicted subgoals. As shown in Fig. 7 (right), our method outperforms all prior approaches. TAP shows only a moderate increase in performance over the Flat planner, which we attribute to the fact that it fails to predict good subgoals and often simply predicts the final image as the bottleneck. This is likely due to the relatively large stochasticity of our dataset and the absence of the clear bottlenecks that TAP is designed to find. Our method outperforms the planners that use Jumpy and Surprise subgoals. This further confirms that KEYIN is able to produce keyframes that are informative about the underlying trajectory, such that planning toward these keyframes makes it easier to follow the trajectory. We presented KEYIN, a method for representing a sequence by its informative keyframes by jointly keyframing and inpainting. KEYIN first generates the keyframes of a sequence and their temporal placement and then produces the full sequence by inpainting between keyframes. We showed that KEYIN discovers informative keyframes on several datasets with stochastic dynamics. Furthermore, by using the keyframes for planning, we showed our method outperforms several other hierarchical planning schemes. Our method opens several avenues for future work. First, an improved training procedure that allows end-to-end training is desirable. Second, more powerful hierarchical planning approaches can be designed using the keyframe representation to scale to long-term real-world tasks. Finally, the proposed keyframing method can be applied to a variety of applications, including video summarization, video understanding, and multi-stage hierarchical video prediction. We include video on the supplementary website at https://sites.google.com/ view/keyin. The website includes inference samples, prior samples, and exectued trajectories for all methods. We found simple attention over LSTM outputs to be an effective inference procedure. Our approximate inference network LSTM inf outputs (κ inf t, ζ t) t≤T, where κ inf is an embedding used to compute an attention weight and the ζ t are values to be attended over. We compute the posterior distribution over z t using a key-value attention mechanism : µ n, σ n = (t a n,t ζ t)/ t a n,t. The distance metric, d, is the standard inner product. The architecture used for keyframe inference, including the attention mechanism, is depicted in Supplemental Fig. 8. We set the prediction horizon to T = 30 frames and predict N = 6 segments with J = 10 frames each for the SBM dataset and 6 frames each for the Pushing dataset. We pre-train the interpolator on segments of two to eight frames for Structured Brownian motion data, and two to six frames for Pushing data. The weight on the KL-divergence term for the interpolator VAE is 1e−3. For training the keyframe predictor, we set β K = 0, β κ = 1, β = 5e−2. The hyperparameters were hand-tuned. We activate the generated images with a sigmoid and use BCE losses on each color channel to avoid saturation. The convolutional encoder and decoder both have three layers for the Structured Brownian motion dataset and four layers for the Pushing dataset. We use a simple two-layer LSTM with a 256-dimensional state in each layer for all recurrent modules. Each LSTM has a linear projection layer before and after it that projects the observations to and from the correct dimensions. We use the Adam optimizer with β 1 = 0.9 and β 2 = 0.999, batch size of 30, and a learning rate of 2e−4. For more details please refer to the appendix. Each network was trained on a single high-end NVIDIA GPU. We trained the interpolator for 100K iterations, and the keyframe predictor for 200K iterations. The toal training time took about a day. In the Pushing environment, we use a held-out test set of 120 of sequences. The Structured Brownian Motion dataset is generated automatically and is potentially infinite. We used 1000 testing samples on the Structured Brownian Motion generated using a different random seed. The data collection for our pushing dataset was performed in an environment simulated in. In the environment, a robot arm initialized at the center of the table pushes an object to a goal position located at the other side of a wall-shaped obstacle. The demonstrations followed a rule-based algorithm that first samples subgoals between the initial position of the object and the goal and then runs a deterministic pushing procedure to the subgoals in order. The ground truth keyframes of the demonstrations were defined by frames at which subgoals were completed. We subsampled demonstration videos by a factor of two when saving them to the dataset, dropping every other frame in the trajectory and averaging actions of every two consecutive frames. For all datasets we generated for this environment following a rule-based algorithm, we only kept successful demonstrations and dropped the ones that fail to push the object to the goal position within a predefined horizon. We describe the details of the continuous relaxation loss computation in Algorithm 1. Note that we efficiently implement computing of the cumulative distributions τ as a convolution, which allows us to vectorize much of the computation. Computational complexity of the proposed implementation scales linearly with the number of keyframes N, and number of allowed frames per segment J and number of ground truth frames T. The final complexity is O(N T J), which we find in practice to be negligible compared to the time needed for the forward and backward pass. Standard stochastic video prediction methods do not attempt to estimate keyframes, as they are designed to densely estimate future videos frame-by-frame. Accordingly, they cannot be used directly as baselines for keyframe prediction methods, such as KEYIN. observe that the variance of the learned prior of a stochastic video prediction model tends to spike before an uncertain event happens. We exploit this observation to find the points of high uncertainty for our strong Surprise baseline. We use the KL divergence between the prior and the approximate posterior KL[q(z t |I 1:t)||p(z t)] to measure the surprise. This quantity can be interpreted as the number of bits needed to encode the latent variable describing the next state, it will be larger if the next state is more stochastic. We train a stochastic video prediction network with a fixed prior with the same architectures of encoder, decoder, and LSTM as our model. We found that selecting the peaks of suprise works the best for finding true keyframes. The procedure we use to select the keyframes is described in Algorithm 2. In order to find the keyframes in a sequence sampled from the prior, we run the inference network on the generated sequence. Parameters: Number of ground truth frames T, Number of keyframes N Input: Ground truth frames I 1:T, Generated framesÎ t i, generated offset distributions δ n Convert the distributions of interframe offsets δ n to keyframe timesteps τ n. For the first keyframe, Compute further τ n with chain rule. This can be efficiently computed via convolution: end for Compute probabilities of keyframes being within the predicted sequence: c n = t≤T τ n t. Compute soft keyframe targets:K n = t τ n t I t. Compute the keyframe loss: Get probabilities of segments ending after particular frames: e Compute the sequence loss: Algorithm 2 Get the surprise measure: Find the set of peak surprise points S where: s t > s t+1 ∧ s t < s t−1. if |S| < M then add M − |S| maximum surprise points to S. end if Return: The M keyframes from S with maximum surprise. We show qualitative of training KEYIN with N = 4, 6 (optimal number), and 8 on the SBM dataset in Fig. 9. We observe that if we train KEYIN to select a smaller or a larger number of keyframes than needed, it learns to predict a subset or a superset of the true keyframes, respectively. This property follows from the structure of the model, which encourages the model to predict the keyframes that allow the full sequence to be inpainted. When too few keyframes are available, the model will be unable to put keyframes at all important times, but those it picks must still be good for inpainting. When more keyframes are available than necessary, the model can place the additional keyframes at any time, as only a subset of the keyframes are needed to ensure good inpainting. Sample L sequences of latent variables: Figure 12: The top row shows the planned subgoals using KEYIN. The bottom row shows snapshots from a successful trajectory between the start state on the left and goal state (depicted transparently in each frame). The low-level execution closely follows the given subgoals and successfully reaches the goal. To apply the KEYIN model for planning, we use the approach for visual planning outlined in Algorithm 1. At the initial timestep, we use the cross-entropy method (CEM) to select subgoals for the task. To do so, we sampleM latent sequences z 0 from Figure 11: Example keyframe detections on noisy sequences. Red frames mark annotated keyframes. Top: Pushing dataset. Bottom: Gridworld dataset. Each triplet depicts top: ground truth sequence with additive Gaussian noise, middle: predicted keyframes at the predicted time steps, bottom: predicted full sequence. KEYIN is reliably able to detect keyframes and reconstruct the full sequence. Table 7: F1 score and distance to closest annotated / predicted keyframe when trained and tested on sequences with additive Gaussian noise. KEYIN is able to reliably find keyframes on both datasets even when trained and tested on noisy sequences. Even though the F1 score is lower on the Pushing dataset, the distances indicate that the discovered keyframes are well aligned with the annotated keyframes even under noise. the prior N (0, I) and use the keyframe model to retrieveM corresponding keyframe sequences, each with L frames. We define the cost of an image trajectory as the distance between the target image and the final image of each keyframe sequence defined under a domain-specific distance function (see below). In the update step of the CEM algorithm, we rank the trajectories based on their cost and fit a diagonal Gaussian distribution to the latents z that generated thẽ M = rM best sequences, where r is the elite ratio. We repeat the procedure above for a total of N iterations. We define the cost between two frames used during planning as the Euclidean distance between the center pixels of the object in both frames. We recover the center pixel via color-based segmentation of the object. While this cost function is designed for the particular planning environment we are testing on, our algorithm can be easily extended to use alternative, more domain-agnostic cost formulations that have been proposed in the literature; Ebert et al. (2017; . After subgoals are selected, we use a CEM based planner to produce rollout trajectories. Similar to the subgoal generation procedure, at each time step, we initially sample M action sequences u 0 from the prior N (0, I) and use the ground truth dynamics of the simulator to retrieve M corresponding image sequences, each with l frames 7. We define the cost of an image trajectory as the distance between the target image and the final image of each trajectory. In the update step, we rank the trajectories based on their cost and fit a diagonal Gaussian distribution to the actions u that generated the M = rM best sequences. After sampling a new set of actions u n+1 from the fitted Gaussian distributions we repeat the procedure above for a total of N iterations. Finally, we execute the first action in the action sequence corresponding to the best rollout of the final CEM iteration. The action at the next time step is chosen using the same procedure with the next observation as input and reinitialized action distributions. The algorithm terminates when the specified maximal number of planning steps T max has been executed or the distance to the goal is below a set threshold."

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.