Question: represent the scientific article to find a short tldr

Question: The demand for abstractive dialog summary is growing in real-world applications. For example, customer service center or hospitals would like to summarize customer service interaction and doctor-patient interaction. However, few researchers explored abstractive summarization on dialogs due to the lack of suitable datasets. We propose an abstractive dialog summarization dataset based on MultiWOZ. If we directly apply previous state-of-the-art document summarization methods on dialogs, there are two significant drawbacks: the informative entities such as restaurant names are difficult to preserve, and the contents from different dialog domains are sometimes mismatched. To address these two drawbacks, we propose Scaffold Pointer Network (SPNet) to utilize the existing annotation on speaker role, semantic slot and dialog domain. SPNet incorporates these semantic scaffolds for dialog summarization. Since ROUGE cannot capture the two drawbacks mentioned, we also propose a new evaluation metric that considers critical informative entities in the text. On MultiWOZ, our proposed SPNet outperforms state-of-the-art abstractive summarization methods on all the automatic and human evaluation metrics. Summarization aims to condense a piece of text to a shorter version, retaining the critical information. On dialogs, summarization has various promising applications in the real world. For instance, the automatic doctor-patient interaction summary can save doctors' massive amount of time used for filling medical records. There is also a general demand for summarizing meetings in order to track project progress in the industry. Generally, multi-party conversations with interactive communication are more difficult to summarize than single-speaker documents. Hence, dialog summarization will be a potential field in summarization track. There are two types of summarization: extractive and abstractive. Extractive summarization selects sentences or phrases directly from the source text and merges them to a summary, while abstractive summarization attempts to generate novel expressions to condense information. Previous dialog summarization research mostly study extractive summarization . Extractive methods merge selected important utterances from a dialog to form summary. Because dialogs are highly dependant on their histories, it is difficult to produce coherent discourses with a set of non-consecutive conversation turns. Therefore, extractive summarization is not the best approach to summarize dialogs. However, most modern abstractive methods focus on single-speaker documents rather than dialogs due to the lack of dialog summarization corpora. Popular abstractive summarization dataset like CNN/Daily Mail is on news documents. AMI meeting corpus is the common benchmark, but it only has extractive summary. In this work, we introduce a dataset for abstractive dialog summarization based on MultiWOZ . Seq2Seq models such as Pointer-Generator have achieved high-quality summaries of news document. However, directly applying a news summarizer to dialog in two drawbacks: informative entities such as place name are difficult to capture precisely and contents in different domains are summarized unequally. To address these problems, we propose Scaffold Pointer Network (SPNet). SPNet incorporates three types of semantic scaffolds in dialog: speaker role, semantic slot, and dialog domain. Firstly, SPNet adapts separate encoder to attentional Seq2Seq framework, producing distinct semantic representations for different speaker roles. Then, our method inputs delexicalized utterances for producing delexicalized summary, and fills in slot values to generate complete summary. Finally, we incorporate dialog domain scaffold by jointly optimizing dialog domain classification task along with the summarization task. We evaluate SPNet with both automatic and human evaluation metrics on MultiWOZ. SPNet outperforms Pointer-Generator and Transformer on all the metrics. 2 first applied modern neural models to abstractive summarization. Their approach is based on Seq2Seq framework and attention mechanism , achieving state-of-the-art on Gigaword and DUC-2004 dataset. proposed copy mechanism in summarization, demonstrating its effectiveness by combining the advantages of extractive and abstractive approach. applied pointing as copy mechanism and use coverage mechanism to discourage repetition. Most recently, reinforcement learning (RL) has been employed in abstractive summarization. RL-based approaches directly optimize the objectives of summarization . However, deep reinforcement learning approaches are difficult to train and more prone to exposure bias . Recently, pre-training methods are popular in NLP applications. BERT and GPT have achieved state-of-the-art performance in many tasks, including summarization. For instance, proposed a method to pre-train hierarchical document encoder for extractive summarization. proposed two strategies to incorporate a pre-trained model (GPT) to perform the abstractive summarizer and achieved a better performance. However, there has not been much research on adapting pre-trained models to dialog summarization. Dialog summarization, specifically meeting summarization, has been studied extensively. Previous work generally focused on statistical machine learning methods in extractive dialog summarization: used skip-chain conditional random fields (CRFs) as a ranking method in extractive meeting summarization. compared support vector machines (SVMs) with LDA-based topic models for producing decision summaries. However, abstractive dialog summarization was less explored due to the lack of a suitable benchmark. Recent work (; ;) created abstractive dialog summary benchmarks with existing dialog corpus. annotated topic descriptions in AMI meeting corpus as the summary. However, topics they defined are coarse, such as "industrial designer presentation". They also proposed a model with a sentence-gated mechanism incorporating dialog acts to perform abstractive summarization. first built a model to summarize audio-visual meeting data with an abstractive method. However, previous work has not investigated the utilization of semantic patterns in dialog, so we explore it in-depth in our work. As discussed above, state-of-the-art document summarizers are not applicable in conversation settings. We propose Scaffold Pointer Network (SPNet) based on Pointer-Generator . SPNet incorporates three types of semantic scaffolds to improve abstractive dialog summarization: speaker role, semantic slot and dialog domain. We first introduce Pointer-Generator . It is a hybrid model of the typical Seq2Seq attention model and pointer network . Seq2Seq framework encodes source sequence and generates the target sequence with the decoder. The input sequence is fed into the encoder token by token, producing the encoder hidden states h i in each encoding step. The decoder receives word embedding of the previous word and generates a distribution to decide the target element in this step, retaining decoder hidden states s t. In Pointer-Generator, attention distribution a t is computed as in: where W h, W s, v and b attn are all learnable parameters. With the attention distribution a t, context vector h * t is computed as the weighted sum of encoder's hidden states. Context vector is regarded as the attentional information in the source text: Pointer-Generator differs from typical Seq2Seq attention model in the generation process. The pointing mechanism combines copying words directly from the source text with generating words from a fixed vocabulary. Generation probability p gen is calculated as "a soft switch" to choose from copy and generation: where x t is the decoder input, w h *, w s, w x and b ptr are all learnable parameters. σ is sigmoid function, so the generation probability p gen has a range of. The ability to select from copy and generation corresponds to a dynamic vocabulary. Pointer network forms an extended vocabulary for the copied tokens, including all the out-of-vocabulary(OOV) words appeared in the source text. The final probability distribution P (w) on extended vocabulary is computed as follows: where P vocab is the distribution on the original vocabulary, V, V, b and b are learnable parameters used to calculate such distribution. Our Scaffold Pointer Network (depicted in Figure 1) is based on Pointer-Generator . The contribution of SPNet is three-fold: separate encoding for different roles, incorporating semantic slot scaffold and dialog domain scaffold. Our encoder-decoder framework employs separate encoding for different speakers in the dialog. User utterances x usr t and system utterances x sys t are fed into a user encoder and a system encoder separately to obtain encoder hidden states h. The attention distributions and context vectors are calculated as described in section 3.1. In order to merge these two encoders in our framework, the decoder's hidden state s 0 is initialized as: The pointing mechanism in our model follows the Equation 3, and we obtain the context vector h * t: We integrate semantic slot scaffold by performing delexicalization on original dialogs. Delexicalization is a common pre-processing step in dialog modeling. Specifically, delexicalization replaces the slot values with its semantic slot name(e.g. replace 18:00 with [time]). It is easier for the language modeling to process delexicalized texts, as they have a reduced vocabulary size. But these generated sentences lack the semantic information due to the delexicalization. Some previous dialog system Figure 1: SPNet overview. The blue and yellow box is the user and system encoder respectively. The encoders take the delexicalized conversation as input. The slots values are aligned with their slots position. Pointing mechanism merges attention distribution and vocabulary distribution to obtain the final distribution. We then fill the slots values into the slot tokens to convert the template to a complete summary. SPNet also performs domain classification to improve encoder representation. research ignored this issue or completed single delexicalized utterance as generated response. We propose to perform delexicalization in dialog summary, since delexicalized utterances can simplify dialog modeling. We fill the generated templates with slots with the copy and pointing mechanism. We first train the model with the delexicalized utterance. Attention distribution a t over the source tokens instructs the decoder to fill up the slots with lexicalized values: Note that w slot specifies the tokens that represents the slot name (e.g. [hotel place], [time] ). Decoder directly copies lexicalized value value(w i) conditioned on attention distribution a t i. If w is not a slot token, then the probability P (w) is calculated as Equation 4. We integrate dialog domain scaffold through a multi-task framework. Dialog domain indicates different conversation task content, for example, booking hotel, restaurant and taxi in MultiWOZ dataset. Generally, the content in different domains varies so multi-domain task summarization is more difficult than single-domain. We include domain classification as the auxiliary task to incorporate the prior that different domains have different content. Feedback from the domain classification task provides domain specific information for the encoder to learn better representations. For domain classification, we feed the concatenated encoder hidden state through a binary classifier with two linear layers, producing domain probability d. The i th element d i in d represents the probability of the i th domain: where U, U, b d and b d are all trainable parameters in the classifier. We denote the loss function of summarization as loss 1 and domain classification as loss 2. Assume target word at timestep t is w * t, loss 1 is the arithmetic mean of the negative log likelihood of w * t over the generated sequence: The domain classification task is a multi-label binary classification problem. We use binary cross entropy loss between the i th domain labeld i and predict probability d i for this task: where |D| is the number of domains. Finally, we reweight the classification loss with hyperparameter λ and the objective function is: 4 EXPERIMENTAL SETTINGS We validate SPNet on MultiWOZ-2.0 dataset . MultiWOZ consists of multi-domain conversations between a tourist and a information center clerk on varies booking tasks or domains, such as booking restaurants, hotels, taxis, etc. There are 10,438 dialogs, spanning over seven domains. 3,406 of them are single-domain (8.93 turns on average) and 7,302 are multidomain (15.39 turns on average). During MultiWOZ data collection, instruction is provided for crowd workers to perform the task. We use the instructions as the dialog summary, and an example data is shown in Table 2. Dialog domain label is extracted from existing MultiWOZ annotation. In the experiment, we split the dataset into 8,438 training, 1,000 validation, and 1,000 testing. ROUGE is a standard metric for summarization, designed to measure the surface word alignment between a generated summary and a human written summary. We evaluate our model with ROUGE-1, ROUGE-2 and ROUGE-L. They measure the word-overlap, bigram-overlap, and longest common sequence between the reference summary and the generated summary respectively. We obtain ROUGE scores using the files2rouge package 1. However, ROUGE is insufficient to measure summarization performance. The following example shows its limitations: In this case, the summary has a high ROUGE score, as it has a considerable proportion of word overlap with the reference summary. However, it still has poor relevance and readability, for leaving out one of the most critical information: [time]. ROUGE treats each word equally in computing n-gram overlap while the informativeness actually varies: common words or phrases (e.g. "You are going to") significantly contribute to the ROUGE score and readability, but they are almost irrelevant to essential contents. The semantic slot values (e.g. [restaurant name], [time] ) are more essential compared to other words in the summary. However, ROUGE did not take this into consideration. To address this drawback in ROUGE, we propose a new evaluation metric: Critical Information Completeness (CIC). Formally, CIC is a recall of semantic slot information between a candidate summary and a reference summary. CIC is defined as follows: Models ROUGE-1 ROUGE-2 ROUGE-L CIC base (Pointer-Gen) 62 where V stands for a set of delexicalized values in the reference summary, Count match (v) is the number of values co-occurring in the candidate summary and reference summary, and m is the number of values in set V. In our experiments, CIC is computed as the arithmetic mean over all the dialog domains to retain the overall performance. CIC is a suitable complementary metric to ROUGE because it accounts for the most important information within each dialog domain. CIC can be applied to any summarization task with predefined essential entities. For example, in news summarization the proper nouns are the critical information to retain. We implemented our baselines with OpenNMT framework . We delexicalize utterances according to the belief span annotation. To maintain the generalizability of SPNet, we combine the slots that refer to the same information from different dialog domains into one slot (e.g. time). Instead of using pre-trained word embeddings like GloVe , we train word embeddings from scratch with a 128-dimension embedding layer. We set the hidden states of the bidirectional LSTM encoders to 256 dimensions, and the unidirectional LSTM decoder to 512 dimension. Our model is optimized using Adam with a learning rate of 0.001, β 1 = 0.9, β 2 = 0.999. We reduce the learning rate to half to avoid overfitting when the validation loss increases. We set the hyperparameter λ to 0.5 in the objective function and the batch size to eight. We use beam search with a beam size of three during decoding. We use the validation set to select the model parameter. Our model with and without multi-task takes about 15 epochs and seven epochs to converge, respectively. To demonstrate SPNet's effectiveness, we compare it with two state-of-the-art methods, PointerGenerator and Transformer . Pointer-Generator is the stateof-the-art method in abstractive document summarization. In inference, we use length penalty and coverage penalty mentioned in. The hyperparameters in the original implementation were used. Transformer uses attention mechanisms to replace recurrence for sequence transduction. Transformer generalizes well to many sequence-to-sequence problems, so we adapt it to our task, following the implementation in the official OpenNMT-py documentation. We show all the models' in Table 1. We observe that SPNet reaches the highest score in both ROUGE and CIC. Both Pointer-Generator and Transformer achieve high ROUGE scores, but a relative low CIC scores. It suggests that the baselines have more room for improvement on preserving critical slot information. All the scaffolds we propose can be applied to different neural network models. In this work we select Pointer-Generator as our base model in SPNet because we observe that Transformer only has a small improvement over Pointer-Generator but is having a higher cost on training time and computing resources. We observe that SPNet outperforms other methods in all the automatic evaluation metrics with a big margin, as it incorporates all the three semantic scaffolds. Semantic slot contributes the most to SPNet's increased performance, bringing the largest increase on all automatic evaluation metrics. Conversation Client: I'm looking for a place to stay. It needs to be a guesthouse and include free wifi. Help desk: There are 23 hotels that meet your needs. Would you like to narrow your search by area and/or price range? Client: I would like for it to be cheap and include free parking. Help desk: There are nine guesthouse hotels in various areas. What part of town are you hoping for? Client: Nothing in particular. I just need it booked for 6 people for a total of 4 nights starting from sunday. I'd also like the reference number, please. Help desk: Sure! We will book you in a hotel with 4 stars. The reservation will be for 6 people for 4 nights. How many rooms would you like to reserve? Client: Well, for six people I suppose I will be needing three rooms, for four nights, starting Sunday. Help desk: You're booked at the Alexander Bed and Breakfast, 517a coldham lane, for 6 people for four nights starting Sunday. Your reference number is JXVKZ7KV. Client: Thank you! I'm also looking for a restaurant. Ideally an Italian place in the same price range in the centre. Help desk: I found Ask restaurant would you like to book that? Client: Is the name of the restaurant Ask restaurant? Help desk: Yes, the restaurant is called Ask restaurant. Would you like to book a table there? Client: Yes. Book it for the same group of people on the same day at 18:45. Can I get the reference number please? Help desk: Great! Your booking for 6 people at the restaurant Ask on Sunday at 18:45 was successful! Your reference number is Y2Y8QYBY. Client: I am all set. Thank you. Help desk: Thank you for using the Cambridge TowneInfo Help Desk! Summaries Pointer-Generator: You are traveling to Cambridge and looking forward to try local restaurants. You are looking for a place to stay. The hotel should include free wifi and should be in the type of guesthouse. The hotel does n't need to include internet. If there is no such hotel, how about one that has free wifi. Once you find the hotel you want to book it for 6 people and 4 nights starting from Sunday. If the booking fails how about 1 nights. Make sure you get the reference number. You are also looking for information particular restaurant. Its name is called Ask restaurant restaurant. Make you you the price range and postcode. You are looking for information in Cambridge. You are looking for a place to stay. The hotel should be in the type of guest house and should include free wifi. The hotel should be in the cheap price range. Once you find the hotel you want to book it for 6 people and 4 nights starting from Sunday. Make sure you get the reference number. You are also looking for a restaurant. The restaurant should be in the same price range as the hotel and should serve Italian food. The restaurant should be in the same area as the hotel. Once you find the restaurant you want to book a table for the same group of people at 18:45 on the same day. Make sure you get the reference number. You are planning your trip in Cambridge. You are looking for a place to stay. The hotel should include free wifi and should be in the type of guest house. The hotel should be in the cheap price range and should include free parking. Once you find the hotel you want to book it for 6 people and 4 nights starting from Sunday. Make sure you get the reference number. You are also looking for a restaurant. The restaurant should be in the same price range as the hotel and should be in the centre. The restaurant should serve italian food. Once you find the restaurant you want to book a table for the same group of people at 18:45 on the same day. Make sure you get the reference number. Table 2: An example dialog and Pointer-Generator, SPNet and ground truth summaries. We underline semantic slots in the conversation. Red denotes incorrect slot values and green denotes the correct ones. We also perform human evaluation to verify if our method's increased performance on automatic evaluation metrics entails better human perceived quality. We randomly select 100 test samples from MultiWOZ test set for evaluation. We recruit 150 crowd workers from Amazon Mechanical Turk. For each sample, we show the conversation, reference summary, as well as summaries generated by Pointer-Generator and SPNet to three different participants. The participants are asked to score each summary on three indicators: relevance, conciseness and readability on a 1 to 5 scale, and rank the summary pair (tie allowed). We present human evaluation in Table 3. In the scoring part, our model outperforms PointerGenerator in all three evaluation metrics. SPNet scored better than Pointer-Generator on relevance and readability. All generated summaries are relatively concise; therefore, they score very similar in conciseness. Ground truth is still perceived as more relevant and readable than SPNet . However, ground truth does not get a high absolute score. From the feedback of the evaluators, we found that they think that the ground truth has not covered all the necessary information in the conversation, and the description is not so natural. This motivates us to collect a dialog summarization dataset with high-quality human-written summaries in the future. Results in the ranking evaluation show more differences between different summaries. SPNet outperforms Pointer-Generator with a large margin. Its performance is relatively close to the ground truth summary. Relevance Conciseness Readability Ground truth 3.83 3.67 3.87 Pointer-Gen 3 Table 3: The upper is the scoring part and the lower is the the ranking part. SPNet outperforms Pointer-Generator in all three human evaluation metrics and the differences are significant, with the confidence over 99.5% in student t test. In the ranking part, the percentage of each choice is shown in decimal. Win, lose and tie refer to the state of the former summary in ranking. Table 2 shows an example summary from all models along with ground truth summary. We observe that Pointer-Generator ignores some essential fragments, such as the restaurant booking information (6 people, Sunday, 18:45). Missing information always belongs to the last several domains (restaurant in this case) in a multi-domain dialog. We also observe that separately encoding two speakers reduces repetition and inconsistency. For instance, Pointer-Generator's summary mentions "free wifi" several times and has conflicting requirements on wifi. This is because dialogs has information redundancy, but single-speaker model ignores such dialog property. Our method has limitations. In the example shown in Table 2, our summary does not mention the hotel name (Alexander Bed and Breakfast) and its address (517a Coldham Lane) referred in the source. It occurs because the ground truth summary doe not cover it in the training data. As a supervised method, SPNet is hard to generate a summary containing additional information beyond the ground truth. However, in some cases, SPNet can also correctly summarize the content not covered in the reference summary (see Table 6 in Appendix). Furthermore, although our SPNet achieves a much-improved performance, the application of SPNet still needs extra annotations for semantic scaffolds. For a dialog dataset, speaker role scaffold is a natural pattern for modeling. Most multi-domain dialog corpus has the domain annotation. While for texts, for example news, its topic categorization such as sports or entertainment can be used as domain annotation. We find that semantic slot scaffold brings the most significant improvement, but it is seldom explicitly annotated. However, the semantic slot scaffold can be relaxed to any critical entities in the corpus, such as team name in sports news or professional terminology in a technical meeting. We adapt a dialog generation dataset, MultiWOZ to an abstractive dialog summarization dataset. We propose SPNet, an end-to-end model that incorporates the speaker role, semantic slot and dialog domain as the semantic scaffolds to improve abstractive summary quality. We also propose an automatic evaluation metric CIC that considers semantic slot relevance to serve as a complementary metric to ROUGE. SPNet outperforms baseline methods in both automatic and human evaluation metrics. It suggests that involving semantic scaffolds efficiently improves abstractive summarization quality in the dialog scene. Moreover, we can easily extend SPNet to other summarization tasks. We plan to apply semantic slot scaffold to news summarization. Specifically, we can annotate the critical entities such as person names or location names to ensure that they are captured correctly in the generated summary. We also plan to collect a human-human dialog dataset with more diverse human-written summaries. A SUPPLEMENT TO CASE STUDY Supplement Summary Transformer: You are planning your trip in Cambridge. You are looking for a place to stay. The hotel doesn't need to include internet and should include free parking. The hotel should be in the type of guesthouse. If there is no such hotel, how about one that is in the moderate price range? Once you find the hotel, you want to book it for 6 people and 4 nights starting from Sunday. Make sure you get the reference number. You are also looking forward to dine. The restaurant should be in the centre. Make sure you get the reference number. Which one is better in your opinion, summary 1 (Pointer-Generator) or summary 2 (SPNet) or tie? Evaluator: Summary 2 (SPNet) What is the reason for your rank? We recommend you to talk about it from relevance, conciseness and readability. Evaluator: Summary 2 is more coherent and relevant. It is just as concise but more readable. Which one is better in your opinion, summary 2 (SPNet) or summary 3 (Ground Truth) or tie? Evaluator: Tie What is the reason for your rank? We recommend you to talk about it from relevance, conciseness and readability. Evaluator: Summary 2 and 3 are almost identical. They are equally coherent, relevant, and readable. Help desk: Their postcode is cb11lh and if you want a reservation just let me know the time, party size, and day and I'll make it for you. Client: Not now, thanks. Here's a weird one, though -are there any expensive 0-star hotels in the center of town? Help desk: I'm sorry, but there are no expensive 0-star hotels in the area. Can I help you find something else? Client: Do you have a hotel in the moderate price range? Help desk: The Cityroomz Hotel is moderatly priced, and has free internet and parking. Would you like to make a reservation now? Client: That sounds nice, yes please book a room for 3 guests staying for 5 nights starting on Sunday. Help desk: I'm sorry, those particular dates are filled, if you'd like I could try to make the trip shorter or maybe change days? Client: How about just for the one night? Help desk: Alright, I was able to book your hotel room at the Cityroomz, for 3 people, for one night. Your reference number is E62IXZGB. Is there anything else I can do? Client: I will also need a taxi to go from the restaurant to the hotel, I want to leave the restaurant at 02:15. Help desk: I have booked your taxi. It will be a blue volvo and the contact number is 07729446320. Client: Great. Thanks. That's all I need today. Help desk: You're welcome. Have a wonderful visit. Pointer-Generator: You are traveling to Cambridge. You are looking for a restaurant. The restaurant should serve 1 food and should be in the moderate price range. Make sure you get postcode and address. You are also looking for information in the centre. The hotel should have a star of 0 and should include free wifi. The hotel doesn't need to include internet. If there is no such hotel, how about one that doesn't have free parking? Once you find the hotel, you want to book it for 3 people and 5 nights starting from Sunday. If the booking fails, how about 1 night? Make sure you get the reference number. You are traveling to Cambridge and looking forward to try local restaurants. You are looking for a place to dine. The restaurant should be in the moderate price range and should be in the centre. Make sure you get postcode. You are also looking for a place to stay. The hotel should be in the type of hotel and should have 0-star. The hotel should be in the same area as the restaurant and should be in the same price range as the restaurant. Once you find the hotel, you want to book it for 3 people and 5 nights starting from Sunday. If the booking fails, how about 1 night? Make sure you get the reference number. You also want to book a taxi to commute between the 2 places. You want to leave the restaurant by 02:15. Make sure you get contact number and car type. You are traveling to Cambridge and looking forward to try local restaurants. The restaurant should be in the centre and should be in the moderate price range. Make sure you get postcode. You are also looking for a place to stay. The hotel should be in the expensive price range and should have a star of 0. The hotel should be in the same area as the restaurant. If there is no such hotel, how about one that is in the moderate price range? Once you find the hotel you want to book it for 3 people and 5 nights starting from Sunday. If the booking fails how about 1 night. Make sure you get the reference number. You also want to book a taxi to commute between the 2 places. You want to leave the restaurant by 02:15. Make sure you get contact number and car type. Table 5: An example dialog and Pointer-Generator, SPNet and ground truth summaries. The dialog spans over three domains: restaurant, hotel and taxi. We underline semantic slots in the conversation. Red denotes incorrect slot values and green denotes the correct ones.

Represent the TLDR for finding the corresponding scientific article

We propose a novel end-to-end model (SPNet) to incorporate semantic scaffolds for improving abstractive dialog summarization.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

Language generation using seq2seq models which produce word embeddings instead of a softmax based distribution over the vocabulary at each step enabling much faster training while maintaining generation quality


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Knowledge bases (KB), both automatically and manually constructed, are often incomplete --- many valid facts can be inferred from the KB by synthesizing existing information. A popular approach to KB completion is to infer new relations by combinatory reasoning over the information found along other paths connecting a pair of entities. Given the enormous size of KBs and the exponential number of paths, previous path-based models have considered only the problem of predicting a missing relation given two entities, or evaluating the truth of a proposed triple. Additionally, these methods have traditionally used random paths between fixed entity pairs or more recently learned to pick paths between them. We propose a new algorithm, MINERVA, which addresses the much more difficult and practical task of answering questions where the relation is known, but only one entity. Since random walks are impractical in a setting with unknown destination and combinatorially many paths from a start node, we present a neural reinforcement learning approach which learns how to navigate the graph conditioned on the input query to find predictive paths. On a comprehensive evaluation on seven knowledge base datasets, we found MINERVA to be competitive with many current state-of-the-art methods. Automated reasoning, the ability of computing systems to make new inferences from observed evidence, has been a long-standing goal of artificial intelligence. We are interested in automated reasoning on large knowledge bases (KB) with rich and diverse semantics BID44 BID1 BID5. KBs are highly incomplete BID26, and facts not directly stored in a KB can often be inferred from those that are, creating exciting opportunities and challenges for automated reasoning. For example, consider the small knowledge graph in Figure 1. We can answer the question " Who did Malala Yousafzai share her Nobel Peace prize with?" from the following reasoning path: Malala Yousafzai → WonAward → Nobel Peace Prize 2014 → AwardedTo → Kailash Satyarthi. Our goal is to automatically learn such reasoning paths in KBs. We frame the learning problem as one of query answering, that is to say, answering questions of the form (Malala Yousafzai, SharesNobelPrizeWith, ?).From its early days, the focus of automated reasoning approaches has been to build systems that can learn crisp symbolic logical rules BID24 BID34. Symbolic representations have also been integrated with machine learning especially in statistical relational learning BID29 BID15 BID21 BID22, but due to poor generalization performance, these approaches have largely been superceded by distributed vector representations. Learning embedding of entities and relations using tensor factorization or neural methods has been a popular approach BID31 BID2 , inter alia), but these methods cannot capture chains of reasoning expressed by KB paths. Neural multi-hop models BID30 BID17 BID47 address the aforementioned problems to some extent by operating on KB paths embedded in vector space. However, these models take as input a set of paths which are gathered by performing random walks Figure 1: A small fragment of a knowledge base represented as a knowledge graph. Solid edges are observed and dashed edges are part of queries. Note how each query relation (e.g. SharesNobelPrizeWith, Nationality, etc.) can be answered by traversing the graph via "logical" paths between entity'Malala Yousafzai' and the corresponding answer.independent of the query relation. Additionally, models such as those developed in BID30; BID9 use the same set of initially collected paths to answer a diverse set of query types (e.g. MarriedTo, Nationality, WorksIn etc.).This paper presents a method for efficiently searching the graph for answer-providing paths using reinforcement learning (RL) conditioned on the input question, eliminating any need for precomputed paths. Given a massive knowledge graph, we learn a policy, which, given the query (entity 1, relation, ?), starts from entity 1 and learns to walk to the answer node by choosing to take a labeled relation edge at each step, conditioning on the query relation and entire path history. This formulates the query-answering task as a reinforcement learning (RL) problem where the goal is to take an optimal sequence of decisions (choices of relation edges) to maximize the expected reward (reaching the correct answer node). We call the RL agent MINERVA for "Meandering In Networks of Entities to Reach Verisimilar Answers."Our RL-based formulation has many desirable properties. First, MINERVA has the built-in flexibility to take paths of variable length, which is important for answering harder questions that require complex chains of reasoning BID42. Secondly, MINERVA needs no pretraining and trains on the knowledge graph from scratch with reinforcement learning; no other supervision or fine-tuning is required representing a significant advance over prior applications of RL in NLP. Third, our path-based approach is computationally efficient, since by searching in a small neighborhood around the query entity it avoids ranking all entities in the KB as in prior work. Finally, the reasoning paths found by our agent automatically form an interpretable provenance for its predictions. The main contributions of the paper are: (a) We present agent MINERVA, which learns to do query answering by walking on a knowledge graph conditioned on an input query, stopping when it reaches the answer node. The agent is trained using reinforcement learning, specifically policy gradients (§ 2). (b) We evaluate MINERVA on several benchmark datasets and compare favorably to Neural Theorem Provers (NTP) BID39 and Neural LP, which do logical rule learning in KBs, and also state-of-the-art embedding based methods such as DistMult BID54 and ComplEx BID48 and ConvE BID12. (c) We also extend MINERVA to handle partially structured natural language queries and test it on the WikiMovies dataset (§ 3.3) BID25.We also compare to DeepPath BID53 which uses reinforcement learning to pick paths between entity pairs. The main difference is that the state of their RL agent includes the answer entity since it is designed for the simpler task of predicting if a fact is true or not. As such their method cannot be applied directly to our more challenging query answering task where the second entity is unknown and must be inferred. Nevertheless, MINERVA outperforms DeepPath on their benchmark NELL-995 dataset when compared in their experimental setting (§ 3.2.2). We formally define the task of query answering in a KB. Let E denote the set of entities and R denote the set of binary relations. A KB is a collection of facts stored as triplets (e 1, r, e 2) where e 1, e 2 ∈ E and r ∈ R. From the KB, a knowledge graph G can be constructed where the entities e 1, e 2 are represented as the nodes and relation r as labeled edge between them. Formally, a knowledge graph is a directed labeled multigraph G = (V, E, R), where V and E denote the vertices and edges of the graph respectively. Note that V = E and E ⊆ V × R ×V. Also, following previous approaches BID2 BID30 BID53, we add the inverse relation of every edge, i.e. for an edge (e 1, r, e 2) ∈ E, we add the edge (e 2, r −1, e 1) to the graph. (If the set of binary relations R does not contain the inverse relation r −1, it is added to R as well.)Since KBs have a lot of missing information, two natural tasks have emerged in the information extraction community -fact prediction and query answering. Query answering seeks to answer questions of the form (e 1, r, ?), e.g. Toronto, locatedIn,?, whereas fact prediction involves predicting if a fact is true or not, e.g. (Toronto, locatedIn, Canada)?. Algorithms for fact prediction can be used for query answering, but with significant computation overhead, since all candidate answer entities must be evaluated, making it prohibitively expensive for large KBs with millions of entities. In this work, we present a query answering model, that learns to efficiently traverse the knowledge graph to find the correct answer to a query, eliminating the need to evaluate all entities. Query answering reduces naturally to a finite horizon sequential decision making problem as follows:We begin by representing the environment as a deterministic partially observed Markov decision process on a knowledge graph G derived from the KB (§2.1). Our RL agent is given an input query of the form e 1q, r q,?. Starting from vertex corresponding to e 1q in G, the agent follows a path in the graph stopping at a node that it predicts as the answer (§ 2.2). Using a training set of known facts, we train the agent using policy gradients more specifically by REINFORCE with control variates (§ 2.3). Let us begin by describing the environment. Our environment is a finite horizon, deterministic partially observed Markov decision process that lies on the knowledge graph G derived from the KB. On this graph we will now specify a deterministic partially observed Markov decision process, which is a 5-tuple (S, O, A, δ, R), each of which we elaborate below. States. The state space S consists of all valid combinations in E × E × R × E. Intuitively, we want a state to encode the query (e 1q, r q), the answer (e 2q), and a location of exploration e t (current location of the RL agent). Thus overall a state S ∈ S is represented by S = (e t, e 1q, r q, e 2q) and the state space consists of all valid combinations. Observations. The complete state of the environment is not observed. Intuitively, the agent knows its current location (e t) and (e 1q, r q), but not the answer (e 2q), which remains hidden. Formally, the observation function O: S → E × E × R is defined as O(s = (e t, e 1q, r q, e 2q)) = (e t, e 1q, r q).Actions. The set of possible actions A S from a state S = (e t, e 1q, r q, e 2q) consists of all outgoing edges of the vertex e t in G. Formally A S = {(e t, r, v) ∈ E: S = (e t, e 1q, r q, e 2q), r ∈ R, v ∈ V } ∪ {(s, ∅, s)}. Basically, this means an agent at each state has option to select which outgoing edge it wishes to take having the knowledge of the label of the edge r and destination vertex v. During implementation, we unroll the computation graph up to a fixed number of time steps T. We augment each node with a special action called'NO OP' which goes from a node to itself. Some questions are easier to answer and needs fewer steps of reasoning than others. This design decision allows the agent to remain at a node for any number of time steps. This is especially helpful when the agent has managed to reach a correct answer at a time step t < T and can continue to stay at the'answer node' for the rest of the time steps. Alternatively, we could have allowed the agent to take a special'STOP' action, but we found the current setup to work sufficiently well. As mentioned before, we also add the inverse relation of a triple, i.e. for the triple (e 1, r, e 2), we add the triple (e 2, r −1, e 1) to the graph. We found this important because this actually allows our agent to undo a potentially wrong decision. Transition. The environment evolves deterministically by just updating the state to the new vertex incident to the edge selected by the agent. The query and answer remains the same. Formally, the transition function is δ: S × A → S defined by δ(S, A) = (v, e 1q, r q, e 2q), where S = (e t, e 1q, r q, e 2q) and A = (e t, r, v)).Rewards. We only have a terminal reward of +1 if the current location is the correct answer at the end and 0 otherwise. To elaborate, if S T = (e t, e 1q, r q, e 2q) is the final state, then we receive a reward of +1 if e t = e 2q else 0., i.e. R(S T) = I{e t = e 2q}. To solve the finite horizon deterministic partially observable Markov decision process described above, we design a randomized non-stationary history-dependent policy π = (d 1, d 2, ..., d T−1), where d t: H t → P(A S t) and history H t = (H t−1, A t−1, O t) is just the sequence of observations and actions taken. We restrict ourselves to policies parameterized by long short-term memory network (LSTM) BID19 ).An agent based on LSTM encodes the history H t as a continuous vector h t ∈ R 2d. We also have embedding matrix r ∈ R |R|×d and e ∈ R |E|×d for the binary relations and entities respectively. The history embedding for H t = (H t−1, A t−1, O t) is updated according to LSTM dynamics: DISPLAYFORM0 where a t−1 ∈ R d and o t ∈ R d denote the vector representation for action/relation at time t − 1 and observation/entity at time t respectively and [;] denote vector concatenation. To elucidate, a t−1 = r A t−1, i.e. the embedding of the relation corresponding to label of the edge the agent chose at time t − 1 and o t = e e t if O t = (e t, e 1q, r q) i.e. the embedding of the entity corresponding to vertex the agent is at time t. Based on the history embedding h t, the policy network makes the decision to choose an action from all available actions (A S t) conditioned on the query relation. Recall that each possible action represents an outgoing edge with information of the edge relation label l and destination vertex/entity d. So embedding for each A ∈ A S t is [r l ; e d], and stacking embeddings for all the outgoing edges we obtain the matrix A t. The network taking these as inputs is parameterized as a two-layer feedforward network with ReLU nonlinearity which takes in the current history representation h t and the embedding for the query relation r q and outputs a probability distribution over the possible actions from which a discrete action is sampled. In other words, DISPLAYFORM1 Note that the nodes in G do not have a fixed ordering or number of edges coming out from them. The size of matrix A t is |A S t | × 2d, so the decision probabilities d t lies on simplex of size |A S t |. Also the procedure above is invariant to order in which edges are presented as desired and falls in purview of neural networks designed to be permutation invariant BID56. Finally, to summarize, the parameters of the LSTM, the weights W 1, W 2, the corresponding biases (not shown above for brevity), and the embedding matrices form the parameters θ of the policy network. For the policy network (π θ) described above, we want to find parameters θ that maximize the expected reward: DISPLAYFORM0 where we assume there is a true underlying distribution (e 1, r, e 2) ∼ D. To solve this optimization problem, we employ REINFORCE as follows:• The first expectation is replaced with empirical average over the training dataset.• For the second expectation, we approximate by running multiple rollouts for each training example. The number of rollouts is fixed and for all our experiments we set this number to 20.• For variance reduction, a common strategy is to use an additive control variate baseline BID18 BID14 BID13. We use a moving average of the cumulative discounted reward as the baseline. We tune the weight of this moving average as a hyperparameter. Note that in our experiments we found that using a learned baseline performed similarly, but we finally settled for cumulative discounted reward as the baseline owing to its simplicity.• To encourage diversity in the paths sampled by the policy at training time, we add an entropy regularization term to our cost function scaled by a constant (β). We now present empirical studies for MINERVA in order to establish that (i) MINERVA is competitive for query answering on small (Sec. 3.1.1) as well as large KBs (Sec. 3.1.2), (ii) MINERVA is superior to a path based models that do not search the KB efficiently or train query specific models (Sec. 3.2), (iii) MINERVA can not only be used for well formed queries, but can also easily handle partially structured natural language queries (Sec 3.3), (iv) MINERVA is highly capable of reasoning over long chains, and (v) MINERVA is robust to train and has much faster inference time (Sec. 3.5). To gauge the reasoning capability of MINERVA, we begin with task of query answering on KB, i.e. we want to answer queries of the form (e 1, r, ?). Note that, as mentioned in Sec. 2, this task is subtly different from fact checking in a KB. Also, as most of the previous literature works in the regime of fact checking, their ranking includes variations of both (e 1, r, x) and (x, r, e 2). However, since we do not have access to e 2 in case of question answering scenario the same ranking procedure does not hold for us -we only need to rank on (e 1, r, x). This difference in ranking made it necessary for us to re-run all the implementations of previous work. We used the implementation or the best pre-trained models (whenever available) of BID39 and BID12. For MINERVA to produce a ranking of answer entities during inference, we do a beam search with a beam width of 50 and rank entities by the probability of the trajectory the model took to reach the entity and remaining entities are given a rank of ∞.Method We compare MINERVA with various state-of-the-art models using HITS@1,3,10 and mean reciprocal rank (MRR), which are standard metrics for KB completion tasks. In particular we compare against embedding based models -DistMult BID54, ComplEx BID48 and ConvE BID12. For ConvE and ComplEx, we used the implementation released by BID12 1 on the best hyperparameter settings reported by them. For DistMult, we use our highly tuned implementation (e.g. which performs better than the state-of-the-art of BID46). We also compare with two recent work in learning logical rules in KB namely Neural Theorem Provers (NTP) BID39 and NeuralLP. BID39 also reports a NTP model which is trained with an additional objective function of ComplEx (NTP-λ). For these models, we used the implementation released by corresponding authors 2 3, again on the best hyperparameter settings reported by them. Table 3: Query answering on KINSHIP and UMLS datasets. Dataset We use three standard datasets: COUNTRIES BID3, KINSHIP, and UMLS BID21. The COUNTRIES dataset ontains countries, regions, and subregions as entities and is carefully designed to explicitly test the logical rule learning and reasoning capabilities of link prediction models. The queries are of the form LocatedIn(c, ?) and the answer is a region (e.g. LocatedIn(Egypt, ?) with the answer as Africa). The dataset has 3 tasks (S1-3 in table 2) each requiring reasoning steps of increasing length and difficulty (see BID39 for more details about the tasks). Following the design of the COUNTRIES dataset, for task S1 and S2, we set the maximum path length T = 2 and for S3, we set T = 3. The Unified Medical Language System (UMLS) dataset, is from biomedicine. The entities are biomedical concepts (e.g. disease, antibiotic) and relations are like treats and diagnoses. The KINSHIP dataset contains kinship relationships among members of the Alyawarra tribe from Central Australia. For these two task we use maximum path length T = 2. Also, for MINERVA we turn off entity in in these experiments. Observations For the COUNTRIES dataset, in TAB1 we report a stronger metric -the area under the precision-recall curve -as is common in the literature. We can see that MINERVA compares favorably or outperforms all the baseline models except on the task S2 of COUNTRIES, where the ensemble model NTP-λ and ConvE outperforms it, albeit with a higher variance across runs. Our gains are much more prominent in task S3, which is the hardest among all the tasks. The Kinship and UMLS datasets are small KB datasets with around 100 entities each and as we see from Table 3, embedding based methods (ConvE, ComplEx and DistMult) perform much better than methods which aim to learn logical rules (NTP, NeuralLP and MINERVA). On Kinship, MINERVA outperforms both NeuralLP and NTP and matches the HITS@10 performance of NTP on UMLS. Unlike COUNTRIES, these datasets were not designed to test the logical rule learning ability of models and given the small size, embedding based models are able to get really high performance. Combination of both methods gives a slight increase in performance as can be seen from the of NTP-λ. However, when we initialized MINERVA with pre-trained embeddings of ComplEx, we did not find a significant increase in performance. Dataset Next we evaluate MINERVA on three large KG datasets -WN18RR, FB15K-237 and NELL-995. The WN18RR BID12 and FB15K-237 BID46 datasets are created from the original WN18 and FB15K datasets respectively by removing various sources of test leakage, making the datasets more realistic and challenging. The NELL-995 dataset released by BID53 has separate graphs for each query relation, where a graph for a query relation can have triples from the test set of another query relation. For the query answering experiment, we combine all the graphs and removed all test triples (and the corresponding triples with inverse relations) from the graph. We also noticed that several triples in the test set had an entity (source or target) that never appeared in the graph. Since, there will be no trained embeddings for those entities, we removed them from the test set. This reduced the size of test set from 3992 queries to 2818 queries. We observe that on FB15K-237, however, embedding based methods dominate over MINERVA and NeuralLP. Upon deeper inspection, we found that the query relation types of FB15K-237 knowledge graph differs significantly from others. Analysis of query relations of FB15k-237: We analyzed the type of query relation types on the FB15K-237 dataset. Following BID2, we categorized the query relations into (M)any to 1, 1 to M or 1 to 1 relations. An example of a M to 1 relation would be'/people/profession' (What is the profession of person 'X'?). An example of 1 to M relation would be /music/instrument/instrumentalists ('Who plays the music instrument X?') or'/people/ethnicity/people' ('Who are people with ethnicity X?'). From a query answering point of view, the answer to these questions is a list of entities. However, during evaluation time, the model is evaluated based on whether it is able to predict the one target entity which is in the query triple. Also, since MINERVA outputs the end points of the paths as target entities, it is sometimes possible that the particular target entity of the triple does not have a path from the source entity (however there are paths to other 'correct' answer entities). TAB10 (in appendix) shows few other examples of relations belonging to different classes. Following BID2, we classify a relation as 1-to-M if the ratio of cardinality of tail to head entities is greater than 1.5 and as M-to-1 if it is lesser than 0.67. In the validation set of FB15K-237, 54% of the queries are 1-to-M, whereas only 26% are M-to-1. Contrasting it with NELL-995, 27% are 1-to-M and 36% are M-to-1 or UMLS where only 18% are 1-to-M. Table 10 (in appendix) shows few relations from FB15K-237 dataset which have high tail-to-head ratio. The average ratio for 1-TO-M relations in FB15K-237 is 13.39 (substantially higher than 1.5). As explained before, the current evaluation scheme is not suited when it comes to 1-to-M relations and the high percentage of 1-to-M relations in FB15K-237 also explains the sub optimal performance of MINERVA.We also check the frequency of occurrence of various unique path types. We define a path type as the sequence of relation types (ignoring the entities) in a path. Intuitively, a predictive path which generalizes across queries will occur many number of times in the graph. Figure 2 shows the plot. As we can see, the characteristics of FB15K-237 is quite different from other datasets. For example, in NELL-995, more than 1000 different path types occur more than 1000 times. WN18RR has only 11 different relation types which means there are only 11 3 possible path types of length 3 and even fewer number of them would be predictive. As can be seen, there are few path types which occur more than 10 4 times and around 50 of them occur more than 1000 times. However in FB15K-237, Figure 2: Count of number of unique path types of length 3 which occur more than'x' times in various datasets. For example, in NELL-995 there are more than 10 3 path types which occur more than 10 3 times. However, for FB15k-237, we see a sharp decrease as'x' becomes higher, suggesting that path types do not repeat often.which has the highest number of relation types, we observe a sharp decrease in the number of path types which occur a significant number of times. Since MINERVA cannot find path types which repeat often, it finds it hard to learn path types that generalize. In this experiment, we compare to a model which gathers path based on random walks and tries to predict the answer entity. Neural multi-hop models BID30 BID47, operate on paths between entity pairs in a KB. However these methods need to know the target entity in order to pre-compute paths between entity pairs. BID17 is an exception in this regard as they do random walks starting from a source entity'e 1' and then using the path, they train a classifier to predict the target answer entity. However, they only consider one path starting from a source entity. In contrast, BID30; BID47 use information from multiple paths between the source and target entity. We design a baseline model which combines the strength of both these approaches. Starting from'e 1', the model samples (k = 100) random paths of up to a maximum length of T = 3. Following BID30, we encode each paths with an LSTM followed by a max-pooling operation to featurize the paths. This feature is concatenated with the source entity and query relation vector which is then passed through a feed forward network which scores all possible target entities. The network is trained with a multi-class cross entropy objective based on observed triples and during inference we rank target entities according to the model score. The PATH-BASELINE column of table 4 shows the performance of this model on the three datasets. As we can see MINERVA outperforms this baseline significantly. This shows that a model which predicts based on a set of randomly sampled paths does not do as well as MINERVA because it either loses important paths during random walking or it fails to aggregate predictive features from all the k paths, many of which would be irrelevant to answer the given query. The latter is akin to the problem with distant supervision BID27, where important evidence gets lost amidst a plethora of irrelevant information. However, by taking each step conditioned on the query relation, MINERVA can effectively reduce the search space and focus on paths relevant to answer the query. We also compare MINERVA with DeepPath which uses RL to pick paths between entity pairs. For a fair comparison, we only rank the answer entities against the negative examples in the dataset used in their experiments 5 and report the mean average precision (MAP) scores for each query relation. DeepPath feeds the paths its agent gathers as input features to the path ranking algorithm (PRA) BID22, which trains a per-relation classifier. But unlike them, we train one model which learns for all query relations so as to enable our agent to leverage from correlations and more data. If our agent is not able to reach the correct entity or one of the negative entities, the corresponding entities gets a score of negative infinity. If MINERVA fails to reach any of the entities in the set of correct and negative entities. then we fall back to a random ordering of the entities. As show in Queries in KBs are structured in the form of triples. However, this is unsatisfactory since for most real applications, the queries appear in natural language. As a first step in this direction, we extend MINERVA to take in "partially structured" queries. We use the WikiMovies dataset BID25 which contains questions in natural language albeit generated by templates created by human annotators. An example question is "Which is a film written by Herb Freed?". WikiMovies also has an accompanying KB which can be used to answer all the questions. We link the entity occurring in the question to the KB via simple string matching. To form the vector representation of the query relation, we design a simple question encoder which computes the average of the embeddings of the question words. The word embeddings are learned from scratch and we do not use any pretrained embeddings. We compare our with those reported in TAB7. For this experiment, we found that T = 1 sufficed, suggesting that WikiMovies is not the best testbed for multihop reasoning, but this experiment is a promising first step towards the realistic setup of using KBs to answer natural language question. While chains in KB need not be very long to get good empirical BID30 BID9, in principle MINERVA can be used to learn long reasoning chains. To evaluate the same, we test our model on a synthetic 16-by-16 grid world dataset created by, where the task is to navigate to a particular cell (answer entity) starting from a random cell (start entity) by following a set of directions (query relation). The KB consists of atomic triples of the form (, North,) -entity is north of entity. The queries consists of a sequence of directions (e.g. North, SouthWest, East). The queries are classified into classes based on the path lengths. FIG2 shows the accuracy on varying path lengths. Compared to Neural LP, MINERVA is much more robust to queries, which require longer path, showing minimal degradation in performance for even the longest path in the dataset. Training time. Figure 5 plots the HITS@10 scores on the development set against the training time comparing MINERVA with DistMult. It can be seen that MINERVA converges to a higher score much faster than DistMult. It is also interesting to note that even during the early stages of the training, MINERVA has much higher performance than that of DistMult, as during these initial stages, MINERVA would just be doing random walks in the neighborhood of the source entity (e 1). This implies that MINERVA's approach of searching for an answer in the neighborhood of e 1 is a much more efficient and smarter strategy than ranking all entities in the knowledge graph (as done by DistMult and other related methods).Inference Time. At test time, embedding based methods such as ConvE, ComplEx and DistMult rank all entities in the graph. Hence, for a test-time query, the running time is always O (|E|) where R denotes the set of entities (= nodes) in the graph. MINERVA, on the other hand is efficient at inference time since it has to essentially search for answer entities in its local neighborhood. The many cost at inference time for MINERVA is to compute probabilities for all outgoing edges along the path. Thus inference time of MINERVA only depends on degree distribution of the graph. If we assume the knowledge graph to obey a power law degree distribution, like many natural graphs, then for MINERVA the average inference time can be shown to be O(α α−1), when the coefficient of the power law α > 1. The median inference time for MINERVA is O for all values of α. Note that these quantities are independent of size of entities |E|. For instance, on the test dataset of WN18RR, the wall clock inference time of MINERVA is 63s whereas that of a GPU implementation of DistMult, which is the simplest among the lot, is 211s. Similarly the wall-clock inference time on the test set of NELL-995 for a GPU implementation of DistMult is 115s whereas that of MINERVA is 35s. Query based Decision Making. At each step before making a decision, our agent conditions on the query relation. Figure 4 shows examples, where based on the query relation, the probabilities are peaked on different actions. For example, when the query relation is WorksFor, MINERVA assigns a much higher probability of taking the edge CoachesTeam than AthletePlaysInLeague. We also see similar behavior on the WikiMovies dataset where the query consists of words instead of fixed schema relation. Model Robustness. TAB8 also reports the mean and standard deviation across three independent runs of MINERVA. We found it easy to obtain/reproduce the highest scores across several runs as can be seen from the low deviations in scores. Similarly inverse relation gives the agent the ability to recover from a potentially wrong decision it has taken before. Example (ii) shows such an example, where the agent took a incorrect decision at the first step but was able to revert the decision because of the presence of inverted edges. Learning vector representations of entities and relations using tensor factorization BID31 BID32 BID2 BID38 BID33 BID54 or neural methods BID43 BID46 BID49 has been a popular approach to reasoning with a knowledge base. However, these methods cannot capture more complex reasoning patterns such as those found by following inference paths in KBs. Multi-hop link prediction approaches BID22 BID30 BID17 BID47 BID9 ) address the problems above, but the reasoning paths that they operate on are gathered by performing random walks independent of the type of query relation. BID22 further filters paths from the set of sampled paths based on the restriction that the path must end at one of the target entities in the training set and are within a maximum length. These constraints make them query (i) Can learn general rules: DISPLAYFORM0 (ii) Can learn shorter path: Richard F. Velky Published as a conference paper at ICLR 2018 dependent but they are heuristic in nature. Our approach eliminates any necessity to pre-compute paths and learns to efficiently search the graph conditioned on the input query relation. Inductive Logic Programming (ILP) BID29 aims to learn general purpose predicate rules from examples and knowledge. Early work in ILP such as FOIL BID36, PROGOL BID28 are either rule-based or require negative examples which is often hard to find in KBs (by design, KBs store true facts). Statistical relational learning methods BID15 BID21 BID41 along with probabilistic logic BID37 BID4 combine machine learning and logic but these approaches operate on symbols rather than vectors and hence do not enjoy the generalization properties of embedding based approaches. There are few prior work which treat inference as search over the space of natural language. BID35 propose a task (WikiNav) in which each the nodes in the graph are Wikipedia pages and the edges are hyperlinks to other wiki pages. The entity is to be represented by the text in the page and hence the agent is required to reason over natural language space to navigate through the graph. Similar to WikiNav is Wikispeedia BID51 in which an agent needs to learn to traverse to a given target entity node (wiki page) as quickly as possible. BID0 propose natural logic inference in which they cast the inference as a search from a query to any valid premise. At each step, the actions are one of the seven lexical relations introduced by BID23.Neural Theorem Provers (NTP) BID39 and Neural LP are methods to learn logical rules that can be trained end-to-end with gradient based learning. NTPs are constructed by Prolog's backward chaining inference method. It operates on vectors rather than symbols, thereby providing a success score for each proof path. However, since a score can be computed between any two vectors, the computation graph becomes quite large because of such soft-matching during substitution step of backward chaining. For tractability, it resorts to heuristics such as only keeping the top-K scoring proof paths trading-off guarantees for exact gradients. Also the efficacy of NTPs has yet to be shown on large KBs. Neural LP introduces a differential rule learning system using operators defined in TensorLog BID7. It has a LSTM based controller with a differentiable memory component BID16 BID45 and the rule scores are calculated via attention. Even though, differentiable memory allows end to end training, it necessitates accessing the entire memory, which can be computationally expensive. RL approaches capable of hard selection of memory BID57 are computationally attractive. MINERVA uses a similar hard selection of relation edges to walk on the graph. More importantly, MINERVA outperforms both these methods on their respective benchmark datasets. DeepPath BID53 uses RL based approaches to find paths in KBs. However, the state of their MDP requires the target entity to be known in advance and hence their path finding strategy is dependent on knowing the answer entity. MINERVA does not need any knowledge of the target entity and instead learns to find the answer entity among all entities. DeepPath, additionally feeds its gathered paths to Path Ranking Algorithm BID22, whereas MINERVA is a complete system trained to do query answering. DeepPath also uses fixed pretrained embeddings for its entity and relations. Lastly, on comparing MINERVA with DeepPath in their experimental setting on the NELL dataset, we match their performance or outperform them. MINERVA is also similar to methods for learning to search for structured prediction BID8 BID10 BID11 BID40 BID6. These methods are based on imitating a reference policy (oracle) which make near-optimal decision at every step. In our problem setting, it is unclear what a good reference policy would be. For example, a shortest path oracle between two entities would be unideal, since the answer providing path should depend on the query relation. We explored a new way of automated reasoning on large knowledge bases in which we use the knowledge graphs representation of the knowledge base and train an agent to walk to the answer node conditioned on the input query. We achieve state-of-the-art on multiple benchmark knowledge base completion tasks and we also show that our model is robust and can learn long chains-ofreasoning. Moreover it needs no pretraining or initial supervision. Future research directions include applying more sophisticated RL techniques and working directly on textual queries and documents. Table 10: Few example 1-to-M relations from FB15K-237 with high cardinality ratio of tail to head. Experimental Details We choose the relation and embedding dimension size as 200. The action embedding is formed by concatenating the entity and relation embedding. We use a 3 layer LSTM with hidden size of 400. The hidden layer size of MLP (weights W 1 and W 2) is set to 400. We use Adam BID20 with the default parameters in REINFORCE for the update. In our experiments, we tune our model over two hyper parameters, viz., β which is the entropy regularization constant and λ which is the moving average constant for the REINFORCE baseline. The table 11 lists the best hyper parameters for all the datasets. The NELL dataset released by BID53 includes two additional tasks for which the scores were not reported in the paper and so we were unable to compare them against DeepPath. Nevertheless, we ran MINERVA on these tasks and report our in

We present a RL agent MINERVA which learns to walk on a knowledge graph and answer queries

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: There are many differences between convolutional networks and the ventral visual streams of primates. For example, standard convolutional networks lack recurrent and lateral connections, cell dynamics, etc. However, their feedforward architectures are somewhat similar to the ventral stream, and warrant a more detailed comparison. A recent study found that the feedforward architecture of the visual cortex could be closely approximated as a convolutional network, but the ing architecture differed from widely used deep networks in several ways. The same study also found, somewhat surprisingly, that training the ventral stream of this network for object recognition ed in poor performance. This paper examines the performance of this network in more detail. In particular, I made a number of changes to the ventral-stream-based architecture, to make it more like a DenseNet, and tested performance at each step. I chose DenseNet because it has a high BrainScore, and because it has some cortex-like architectural features such as large in-degrees and long skip connections. Most of the changes (which made the cortex-like network more like DenseNet) improved performance. Further work is needed to better understand these . One possibility is that details of the ventral-stream architecture may be ill-suited to feedforward computation, simple processing units, and/or backpropagation, which could suggest differences between the way high-performance deep networks and the brain approach core object recognition. For most cortical areas, these are layers L2/3, L4, L5, and L6. For V1, these are L2/3(blob), L2/3(interblob), L4B, L4Cα, L4Cβ. For LGN, they are parvo, magno, and koniocellular divisions. reported validation accuracy of 79% on CIFAR-10, for a similar ventral-stream sub-network that 49 omitted connections with FLNe<0.15, and was trained for 50 epochs with the Adam update algorithm. In the present study, networks were trained for 300 epochs, using SGD with momentum 0.9, starting 51 with learning rate 0.1 and reducing it by 10x every 100 epochs. This ed in validation accuracy 52 of 84.59%. A standard DenseNet (github.com/kuangliu/pytorch-cifar) was trained using the same 53 procedure, ing in validation accuracy of 95.36%. To understand the basis of this performance gap, I created hybrid networks, with features of both the 55 ventral-stream network (VSN) and DenseNet. The VSN has a wide range of kernel sizes, optimized 56 to fill realistic receptive field sizes. In the first hybrid (H1), all kernel sizes of the VSN were set 57 to 3x3. The VSN also has a wide range of sparsity, with some connections consisting mostly of 58 zeros. In the second hybrid network (H2), in addition to using 3x3 kernels, I eliminated pixel-wise 59 sparsity, and limited channel-wise sparsity so that at least half of the input channels were used in 60 each connection. Thirdly (H3), I replaced each layer with a two-layer bottleneck module, specifically 61 a 1x1-kernel layer followed by 3x3 layer with four times fewer channels. The number of channels Most of these modifications improved performance, when applied cumulatively to make the ventral- stream increasingly similar to a DenseNet (Table 2,. In the monkey inferotemporal cortex data, mean selectivity is 3.5, and mean sparseness is 12.51. The ventral-stream model has much higher means, and DenseNet has much lower means. C. Representational dissimilarity, using a subset of images from. The plotted values are percentiles of one minus the Pearson correlations between responses to different stimuli. Monkey cell data shows relatively low values (high similarity) throughout the lower-right quarter of this matrix (spanning non-animal natural and artificial images), but neither of the deep networks does. efficiently, or the connection pattern of the ventral stream may be better suited to extracting certain 135 feature combinations in unsupervised learning than to communicating gradients through many layers.

represent the short summary for finding the corresponding scientific article (long)

An approximation of primate ventral stream as a convolutional network performs poorly on object recognition, and multiple architectural features contribute to this.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.