CliCR: A Dataset of Clinical Case Reports for Machine Reading Comprehension
Simon Šuster Walter Daelemans


Proton Colliders at the Energy Frontier
    Michael Benedikt and Frank Zimmermann 

CERN, CH-1211 Geneva 23, Switzerland
    January 31, 2017
================================================================================


We present a new dataset for machine comprehension in the medical domain. Our dataset uses clinical case reports with around 100,000 gap-filling queries about these cases. We apply several baselines and state-of-the-art neural readers to the dataset, and observe a considerable gap in performance (20% F1) between the best human and machine readers. We analyze the skills required for successful answering and show how reader performance varies depending on the applicable skills. We find that inferences using domain knowledge and object tracking are the most frequently required skills, and that recognizing omitted information and spatio-temporal reasoning are the most difficult for the machines.





§ INTRODUCTION

Machine comprehension is a task in which a system reads a text passage and then answers questions about it. The progress in machine comprehension heavily depends on the introduction of new datasets <cit.>, which encourages the development of new algorithms and deepens our understanding of the (linguistic) challenges that can or can not be tackled well by these algorithms.  Recently, a number of reading comprehension datasets have been proposed (<ref>), differing in various aspects such as mode of construction, answer-query formulation and required understanding skills. Most are open-domain datasets built from news, fiction and Wikipedia texts. For specialized domains, however, large machine comprehension datasets are extremely scarce <cit.>, and the required comprehension skills poorly understood. With our work we hope to narrow this gap by proposing a new resource for reading comprehension in the clinical domain, and by analyzing the different types of comprehension skills that are triggered while answering <cit.>.



Machine comprehension for healthcare and medicine has received little attention so far, although it offers great potential for practical use.
A typical application would be clinical decision support, where given a massive amount of text, a clinician asks questions about either external, medical knowledge (reading literature) or about particular patients (reading electronic health records). Currently, patient-specific questions are tackled by manually browsing or searching those records. This task can be facilitated by summarization and QA systems <cit.>, and we believe, by fine-grained machine reading. Reading comprehension systems that perform on a finer level could play an important role especially when combined with document retrieval to perform machine reading at scale, such as in the models of  <cit.> and <cit.> for the general domain.=-1

For our dataset, we construct queries, answers and supporting passages from BMJ Case Reports, the largest online repository of such documents. A case report is a detailed description of a clinical case that focuses on rare diseases, unusual presentation of common conditions and novel treatment methods. Each report contains a Learning points section, summarizing the key pieces of information from that report. The learning points are typically paraphrased portions of passage text and do not match passage sentences exactly. We use these learning points to create queries by blanking out a medical entity. To counteract potential errors and inconsistencies due to automated dataset creation, we perform several checks to improve the quality of the dataset (<ref>). Our dataset contains around 100,000 queries on 12,000 case reports, has long support passages (around 1,500 tokens on average) and includes answers which are single- or multi-word medical entities. We show an example from the dataset in Figure <ref>. 


We examine the performance on the dataset in two ways. First, we report machine performance for several baselines and neural readers. To enable a more flexible answer evaluation, we expand the answers with their respective synonyms from a medical knowledge base, and additionally supplement the standard evaluation metrics with BLEU and embedding-based methods. We investigate different ways of representing medical entities in the text and how this affects the neural readers. We obtain the best results with a recurrent neural network (RNN) with gated attention <cit.>, but a simple approach based on embedding similarity proves to be a strong baseline as well. Second, we look at how well humans perform on this task, by asking both a medical expert and a novice to answer a portion of the validation set. When categorizing the skills necessary to find the right answer, we observe that a large number of comprehension skills get activated and that prior knowledge in the form of the ability to perform lexico-grammatical inferences matters the most. This suggests that for our dataset and possibly for domain-specific datasets more generally, more background knowledge should be incorporated in machine comprehension models. The current gap between the best machine and the best human performance is nearly 20% F1, which leaves ample space for further study of machine readers on our dataset. In brief, the contributions of our paper are:


  * We propose a large dataset for reading comprehension in the medical domain, using clinical case descriptions.=-1

  * We carry out an empirical analysis of [a)] 
  * system and human performance on reading comprehension, and 
  * comprehension skills that are required for answering the queries correctly and that allow us to position the dataset according to its difficulty on each of the skills.




§ RELATED DATASETS



Numerous general-domain datasets have been recently created to allow machine comprehension using data-intensive methods. These datasets were collected from
Wikipedia <cit.>, web search queries <cit.>,
news articles <cit.>,
books <cit.> and English exams <cit.>. 
In Table <ref>, we compare our dataset to several domain-specific datasets  for machine comprehension. In Quasar-S, the queries are constructed from definitions of software entity tags in a community QA website, while in our case the queries are more varied and explicitly relate to the supporting passages. SciQ is a dataset of science exam questions, in which question-answer pairs are used to retrieve the text passages. For each question, four candidate answers are available. In our dataset, the number of candidate answer is much higher as the candidate answers come from the relatively long passages. Other datasets mentioned in the table are smaller, so they could not be used as training sets for statistical NLP models.

Cloze datasets require the reader to fill in gaps by relying on accompanying text.
Representative datasets are Children's Book Test <cit.> and Book Test <cit.>, in which queries are created by removing a word or a named entity from the running text in a book; and <cit.>, who similarly to us blank out entities in abstractive CNN and Daily Mail summaries, but who are only concerned with short proper nouns and short passages. Who-did-what <cit.> requires the reader to select the person name from a short candidate list that best answers the query about a news event. They do not use summaries for query formation but remove a named entity from the initial sentence in a news article, and then perform information retrieval to find independent passages relevant to the query. Another cloze dataset for language understanding is ROCStories <cit.>, but it is targeted more towards script knowledge evaluation, and only contains five-sentence stories. Another related task is predicting rare entities only, with a focus on improving a reading comprehension system with external knowledge sources <cit.>.

Another popular way of creating datasets for reading comprehension is crowdsourcing <cit.>. These datasets exist primarily for the general domain; for specialized domains where background knowledge is crucial, crowdsourcing is intuitively less suitable <cit.>, although some positive precedent exists for example in crowdsourcing annotations of radiology reports <cit.>. 
Compared to automated dataset construction, crowdsourcing is more likely to provide high-quality queries and answers. On the other hand, human question generation may also lead to less varied datasets as questions would tend to be of wh- type; for cloze datasets, the questions may be more varied and might require readers to possess a different set of skills.[Support for this is given in <cit.>, who show that Who-did-what dataset, for example, requires on average a larger number of reading skills than SQuAD <cit.> and MCTest <cit.>.] 



§ DATASET DESIGN

We collected the articles from BMJ Case Reports[<http://casereports.bmj.com/>]. The data span the years 2005–2016 and amount to almost 12 thousand reports. We removed the HTML boilerplate from the crawled reports using jusText[<https://pypi.python.org/pypi/jusText>], segmented and tokenized the texts with cTakes <cit.>, and annotated the medical entities using Clamp <cit.>. We apply two simple heuristics to refine the recognized entities and to decrease their sparsity. Namely, we move the function words (determiners and pronouns) from the beginning of the entity outside of it, and we adjust the entity boundary so that it does not include a parenthetical at the end of the entity. Clamp assigns entities following the i2b2-2010 shared task specifications <cit.>. For each entity, a concept unique identifier (CUI) is also available, which links it to the UMLS® Metathesaurus® <cit.>. To check the quality of the recognized entities, we carried out a small manual analysis on 250 entities. We found that in 89% of cases, the boundaries were correct and defined a true entity. Wrongly recognized cases occurred mostly when two entities were coordinated and recognized as one; when a verb was wrongly included in the entity; or when a pre-modifier was left out.



 §.§ Query construction
 We create a query by replacing a medical entity in one learning point with a blank. For example, in a report describing comorbid disorders of ADHD, we could obtain the following query:

Patients with ADHD have higher incidence of .

The missing entity enuresis is taken as the correct answer. Even though one query corresponds to at most one learning point, there can be more than one query built from a learning point. Occasionally, a learning point contains an exact repetition from the passage. These instances would be trivial to answer, so we remove them. We count as an exact match every instance whose longer side to left/right of the query blank coincides with a part in the passage text. This curation step reduces the dataset size by 5%.
More commonly, the learning points are paraphrases of crucial parts of the passage. Sometimes, the entity answering the query is expressed differently in the passage. For example, in place of enuresis, the passage might include its synonym bedwetting. We manage these cases in two ways, by extending the set of answers for a certain query (<ref>), and adding a semantic relatedness metric to the standard evaluation (<ref>).




 §.§ Answer set
 We account for lexical variation of the ground-truth answers (compared to mentions in the passages) by extending each original ground-truth answer a to a set of ground-truth answers A using a knowledge base. Since our entity recognizer already provides the CUI labels, we can use them to obtain the list of alternative word and phrase forms (synonyms, abbreviations and acronyms) from UMLS®.

Similarly to previous work <cit.>, for certain queries none of the answers in A occurs verbatim in the passage. We have found upon manual inspection that this is mostly due to lexical variation that is not captured by answer extension, and to a lesser degree, due to the introduction of entirely new information in the learning point and the entity recognition errors. In the empirical part, we use for training only the instances for which at least one answer occurs in the passage, but we evaluate on all instances in the validation and test sets, including those for which A∩ E=∅, where E is the set of all entities in the passage. This mimics a likely real-life scenario where the set of ground-truth answers is a priori unknown.=-1



 §.§ Task formulation

The reading comprehension problem in our case can be represented as a tuple (q,p,A), where q is the query, built from a learning point; the passage p is the entire report excluding the Learning points section; and A is the set of ground-truth entities answering q. In defining the task, it is important to consider how to take into account entity annotation and how to define the answer output space. We look at these more closely in the rest of this section.


Whenever the entities are marked in the passage, the system can learn to exploit this cue to find the answers more easily <cit.>. Although this simplifies the task, it also makes it less realistic as the entities may not be recognized at test time. Realizing that the presence of entities makes the task easier for the machines, <cit.> anonymize the entities, also with a goal of discouraging language model solutions to the queries. In our case, it is not clear how relevant the anonymization is since we deal with medical entities, which have different properties than proper name entities <cit.>. We explore different entity-annotation choices in the empirical part, where we refer to them as Ent (entities marked) and Anonym (entities marked but anonymized). We further examine a more challenging setup in which the reader can not rely on entity markers as they are not present in the passage (NoEnt). In all cases, the reader chooses an answer among the candidates E collected from all entities in the passage.[The candidate answers could in principle be obtained also in some other way, so we do not list them in our dataset.] 
Multi-word entities, which are common in our dataset, are treated as a single token by Ent and Anonym.=-1







§ DATASET ANALYSIS

We now describe the dataset in more detail, starting with the general statistics summarized in Table <ref>.
 It is worth pointing out that the support passages are rather long, which stems from the data origin (journal articles). We show the passage length distribution in Figure <ref>, which has the average length of 1,466 tokens. Furthermore, passages are rich with medical entities. There is little repetition of answers—the total of around 100,000 queries are answered by 50,000 distinct entities. Upon extending the answer set with UMLS® we introduce on average four alternative answers for each original one. In 59% of instances, the answer entity is found verbatim in the relevant passage.  The answers can belong to any of the problem, treatment or test categories (Table <ref>), and usually consist of multiple words (Figure <ref>). The diversity of medical specialties represented in the articles is shown in Figure <ref>.








 §.§ Analysis of comprehension skills

We estimate the types of skills required in answering by following the categorization of <cit.>. We include the skill  definitions with examples from our dataset in <ref>. We annotated 100 instances in the validation set (with ground-truth answers provided), which yielded on average 2.85 skills per query. The distribution of the required skills is shown in Figure <ref>. 
In comparison to the general-domain datasets (SQuAD, Who-did-what), our dataset and QA4MRE (which is also a domain-specific dataset, but with human-generated questions) require more bridging inferences (inferences using background knowledge about the domain), spatio-temporal reasoning and coreference resolution. In our dataset, meta knowledge and object tracking are required more often than in any other dataset. This can be explained by the data origin and the nature of queries. In the case reports, a prominent topic can be discussed which the author refers to in the query, but the query itself is never answered in the passage (meta knowledge). Furthermore, the authors often enumerate medical entities in the query, which leads to the frequent use of object tracking. The queries which were unanswerable are marked as none. The fraction of these cases was around 16%. 



In our experience, the annotation of skills proved quite challenging due to certain confusables. For example, object tracking and coreference both need to maintain the link between objects; object tracking, which includes establishing set relations and membership, may be overlaid with the schematic clause relation skill (subordination); and bridging inference can overlap with coreference resolution. Nevertheless, we adhered to this classification of skills to increase comparability  to other datasets included in Figure <ref>. 



§ METHODS



 §.§ Baselines

Our simplest baselines that we apply on the test set include choosing a random entity (rand-entity) and selecting the most frequent passage entity (maxfreq-entity) as the answer. We also include a distance-based method that uses word embeddings (sim-entity). Here, we vectorize the passage and the query, and then choose that entity from the passage whose representation has the highest cosine similarity to the query representation:


    sim-entity=_i∈ Ecos(∑_j∈ C_i c_j,∑_k∈ Q q_k),

where c,q∈ℝ^d.
The multiset C_i contains the words {x_i-n, …, x_i-1, x_i+1, …, x_i+n} surrounding the passage entity i∈ E. We define Q, the context words of the query, likewise. 
To find out how well the queries can be answered without reading the passage, we also predict the most likely continuation with a language model (lang-model). We trained a 4-gram Kneser-Ney model on CliCR training data (with multi-word entities represented as a single token) using SRILM <cit.>.




 §.§ Neural readers

We apply two types of bidirectional RNNs to our data. Following <cit.>, we distinguish between aggregation readers and explicit reference readers, which differ in their formulation of the attention mechanism and how it is being used for answer prediction. 



  
Stanford Attentive (SA) Reader The model proposed by <cit.> is an aggregation reader based on the Attentive Reader <cit.>.  It predicts the answer using:

    â = _i∈ E e_o(i)^T o,

where e_o(i) is the answer's output embedding and o is the passage representation obtained by weighting every token representation in the passage with attention: o=∑_t α_t h_t. The attention mechanism is used here to measure the compatibility between token (h_t) and query (q) representations with a bilinear form, α_t = softmax_t h_t^TW_α q. At prediction time, attention should highlight that position t in the passage where the answer occurs. Note that the prediction relies on the aggregate representation o, hence the name  of the reader category. As we see in (<ref>), the prediction score does not allow accounting for multi-word entities, unless they are treated as a single token. Returning to our different set-ups based on entity annotation (<ref>), this means that we can apply SA reader with Ent and Anonym set-ups, but not with NoEnt, where multi-word answers should be allowed.



  
Gated-Attention (GA) Reader <cit.> investigate neural readers with a fine-grained attention mechanism that learns token representations for the passage that are also conditional on the query, but are in addition refined through multiple hops of the network. 
The model predicts the answer using attention weights with explicit reference to answer positions in the passage:

    â = _i∈ E∑_t∈ R(i,p)α_t,
 where R is the set of indices in passage p at which a token from the candidate i occurs. This operation is also called the pointer sum attention <cit.>. Since the model marks the references for each token in the answer separately, it allows us to investigate also the NoEnt set-up.[We assume the candidate entities are known in advance.] 

We train each reader with the best hyper-parameters found on the validation set using random search <cit.>, and evaluate it on the test part of the dataset. We provide more details about parameter optimization in <ref>. The models use word embeddings pre-trained on biomedical texts. 



 §.§ Embedding data and pre-training

We induce the word embeddings on a combination of the CliCR training corpus and PubMed abstracts with open-access PMC articles available until 2015 (segmented and tokenized), amounting to over 9 billion tokens <cit.>. 
Considering the large effect of hyper-parameter selection on the quality of word embeddings <cit.>, we optimize the embedding hyper-parameters also using random search.



§ EVALUATION


A model f takes as input a passage–query pair and outputs an answer â.[In our case, the answer is a word or a word phrase representing a medical entity. Alternatively, one could also take the UMLS® CUI identifier as the answering unit. However, in that case, it would mean that sometimes the original word phrase is lost. This is because entity linking with CUIs can be noisy, and only a part of a word phrase may be linked to the ontology. In the current setup, we are able to keep both the original word phrase as well as the extended answers. The CUI information is still an integral part of the answer field in our dataset, so it can be used by other researchers if preferred.] We carry out the evaluation with different metrics described below. The final score m for a metric v is obtained by averaging over the test set:

    m_v(f) = 1/|D_test|∑_(p,q,A)∈ D_testmax_a∈ A v(f(p,q), a).

Since there are multiple correct answers A, we take the highest scoring answer â at each instance, as done in <cit.>. Note that in the dataset we do not supply the candidate answers; in the experiments, we constrain the candidates to the set of entities in the passage.

The two standardly used metrics for machine comprehension evaluation are the exact match (EM) and the F1 score. For EM, the predicted and the ground truth answers must match precisely, safe for articles, punctuation and case distinction (same for other metrics). F1 metric is applied per instance and measures the overlap between the prediction â and the ground truth a, which are treated as bags of words.[In precision, the number of correct words is divided by the number of all predicted words. In recall, the former is divided by the number of words in the ground-truth answer.] 
While these two metrics are arguably sufficient in news-style machine comprehension where the entities are proper nouns which allow for little variation and synonymy, in our case the medical entities are often mostly common nouns modified by specifiers and qualifiers. To take into account potentially large lexical and word-order variation, we use two additional metrics.
First, we measure BLEU <cit.> for n-grams of length 2 (shortly, B-2) and 4 (B-4) using the package by <cit.>, with which we aim to capture contiguity of tokens in longer answers. Second, it may occur that answers contain no word overlap yet still be good candidates because of their semantical relatedness, as in  renal failure–kidney breakdown. We take this into account by using an embedding metric (E-avg), in which we construct mean vectors for both ground-truth and system answer sequences, and then compare them with the cosine similarity. This and other embedding metrics for evaluation were previously studied in dialog-system research <cit.>.



§ RESULTS AND ANALYSIS



We show the results in Table <ref>. We see that answer prediction based on contextual representation of queries and passages (sim-entity) achieves a strong base performance that is only outperformed by GA reader. The language model performs poorly on EM and F1, but the embedding-metric score is higher, likely reflecting the fact that the predicted answers—though mostly incorrect—are related to the ground-truth answers. The poor performance means that based on queries alone (without reading the passage), it is difficult to provide accurate answers. 
The GA reader performs well across all entity set-ups, even when the entities are not marked in the passage. Interestingly, the exact match and BLEU scores in this case are much lower compared to other entity set-ups. Upon inspecting the predicted answers more closely, we have observed that GA-NoEnt tends to predict longer answers than GA-Ent/Anonym. For example, the average predicted answer length for GA-NoEnt was as high as 3.7 tokens, whereas for the other two set-ups and the ground-truth answers the numbers range between 2.3 and 2.5. A plausible explanation for this lies in how GA reaches its prediction (<ref>), which is by accumulating the attention weights without normalizing. This would then drive the model to prefer longer answers. For example, for the ground-truth entity chest CT, GA-NoEnt predicts interval CT scans of the chest. Although all neural models use pre-trained word embeddings, for Ent and Anonym the multi-word entities do not have pre-trained embeddings since our embeddings are induced on the word level. This may partly explain the competitive performance of NoEnt compared to Ent. We leave the integration of entity embeddings for the future work. 



The results for SA reader are far below the performance of GA reader. We also see that it performs much better on anonymized entities than on non-anonymized ones. This is in line with <cit.> who find that SA reader suffers a drop of 19 points in exact match on Who-did-what dataset when anonymization is not done. A possible explanation is that anonymization reduces the output space to only several hundred entity candidates for which the output embedding needs to be trained. When we do not use anonymization, the set of output entities increases to the set of all entity types found in all passages, which is several orders of magnitude more. While this effect also occurs for GA reader, it is less pronounced because GA reader scores words in the passage and does not need to learn separate answer word embeddings.



 §.§ Human performance

To measure the accuracy of human answering, we have used the same sample of data instances as used for the analysis of skills.[Human answers were collected before the skill analysis.] The queries were answered separately by a novice reader (linguistics background, little-to-none medical knowledge) and by an expert reader (both linguistics and medical background). The annotators needed around 15 minutes on average to read the passage and answer the query. The results are shown at the bottom of Table <ref>. The expert scores higher across all evaluation metrics, with as much as a 7-point advantage in % F1. This advantage is largely coming from the better performance on those instances where bridging inferences are required (the average F1 score was 10 points higher on these queries), which suggests that domain knowledge is beneficial in the comprehension task. For a novice in a specialized domain, it is harder to build a good situation model that would lead to successful comprehension since it requires more effort—active, strategic processing and establishing ontological relationships in that specific domain. For an expert reader this process is more automatized <cit.>.

We can see from the table that the best human performance is well below its theoretical upper bound of 100% F1. An important part of explanation for this lies in the automated dataset construction, which leaves certain queries unanswerable, especially when the authors do not refer to a part in the article but introduce completely new information. Another reason is the problem of answer openness: Typically more than one correct answer is possible and the answers can be correct to various degrees, which we aimed to capture with the use of the embedding metric in the evaluation. Nevertheless, the gap between the best human and machine F1 score is large (around 20 points), leaving considerable space for future applications of machine readers on our dataset.[For comparison, the gap for SQuAD was 12.2 and for NewsQA 19.8 <cit.>.]



 §.§ Breakdown of results by skill



To see how the answering performance relates to the skill requirements, we have analyzed the part of the validation set annotated with the skills by averaging F1 values for all instances with a particular skill. In this way, we are able to break down both human and machine performance skill-wise, as shown in Figure <ref>. Because of the small sample size, the results should only be taken as a general indication. The most difficult cases for the GA reader are those annotated with none (unanswerable) and ellipsis (recognizing implicit and omitted information), ignoring analogy for which we only have a single annotated case. Furthermore, spatio-temporal reasoning, elaboration (inferences using general knowledge) and bridging—which is also the most commonly required skill—are the next most difficult ones. The human scores are mostly much higher, which is especially apparent for spatio-temporal reasoning, logical skills and the skill involving punctuation. 
Our findings align with those of <cit.> on the Lambada dataset <cit.>: Although they used a different categorization of comprehension skills, they also find that GA reader has most difficulties with elaboration (which they refer to as external knowledge), followed by coreference resolution.



§ CONCLUSION AND FUTURE WORK

We have introduced a new dataset for domain-specific reading comprehension in which we have constructed around 100,000 cloze queries from clinical case reports. We analyzed the dataset in terms of the skills required for successful comprehension, and applied various baseline methods and state-of-the-art neural readers. We showed that a large gap still exists between the best machine reader and the expert human reader. One direction for future research is improving the reading models on the queries that are currently the most challenging, i.e. those requiring world and background domain knowledge. Better representing background knowledge by inducing embeddings for entities or otherwise integrating ontological knowledge is in our opinion a promising avenue for future research.





§ ACKNOWLEDGMENTS

We would like to thank Madhumita Sushil and the anonymous reviewers for useful comments. We are also grateful to BMJ Case Reports for allowing the collection of case reports. This work was carried out in the framework of the Accumulate IWT SBO project (nr. 150056), funded by the government agency for Innovation by Science and Technology. We also acknowledge the support of the Nvidia GPU Grant Program.


acl_natbib



§ TRAINING DETAILS AND HYPER-PARAMETER OPTIMIZATION



We train the word embeddings using word2vec <cit.>, and optimize the window size, the model type (CBOW, skip-gram), the dimensionality and the number of negative samples using random search. For the embedding baseline sim-entity, the evaluation was carried out 20 times on the validation part of our dataset, and we chose the parameter configuration that led to the highest-performing embedding model as measured by F1. We find that higher embedding dimensionality works better, that CBOW obtains somewhat better scores than Skipgram, and that medium-sized word windows work best. The best configuration: 'win_size': 5, 'min_freq': 200, 'model': 'cbow', 'dimension': 750, 'neg_samples': 5. The difference between the lowest and the highest scoring model was 3.4 F1. At prediction time (equation (<ref>)) we set the window size to 3, which worked best on the validation set.






For inclusion in the neural readers, it would be impractical to use the high embedding dimensionality found in the hyper-parameter search from the previous paragraph, so we fix the input embedding dimensionality to 200, as done in <cit.> to keep the training time practical. We optimize the remaining embedding hyper-parameters just like above. The best parameters were: 'win_size': 4, 'min_freq': 200, 'model': 'cbow', 'dimension': 200, 'neg_samples': 9.


For SA reader, we optimized the hidden state size and the dropout rate using 20 different random configurations. The best values were 70 and 0.57, respectively. We explore the same parameters for the GA reader, but add to the search space the feature that indicates the presence of a passage token in the query, which was found useful in the NoEnt set-up.

The best hidden state number and dropout rate were 64 and 0.5, respectively. 
We used the default values for all the remaining hyper-parameters.



§ LIST OF SKILLS WITH SELECTED EXAMPLES

In annotating the skills, we followed the categorization by <cit.>:


  * Object tracking: tracking or grasping multiple objects; it is a version of list/enumeration skill used in previous skill classifications

  * Mathematical reasoning: whenever a mathematical operation is involved in finding the answer

  * Coreference resolution: direct reference to an object, includes anaphoras. These include inferential processes based on background knowledge or context.

  * Logical reasoning: conditionals, quantifiers, negation, transitivity

  * Analogy: metaphors, metonymy

  * Causal relation: explicit expression such as "why", "the reason of"

  * Spatio-temporal relations

  * Ellipsis: recognizing implicit or omitted information

  * Bridging: inference through grammatical and lexical knowledge (synonymy, idioms etc). 

This link however is not automatic or stereotypical, as in the category of elaboration.

  * Elaboration: inference through commonsense reasoning. 

Note that unlike in the previous category, there is no direct way in which grammatical, lexical or ontological knowledge could help.

  * Meta-knowledge: knowing about the text genre and the main topic being discussed assists in comprehending. In our dataset, knowing the way the queries are constructed (Learning points) is sometimes beneficial.

  * Schematic clause relation: complex sentences that include coordination or subordination

  * Punctuation: understanding parentheses, dashes, quotations, colons etc.


In the following examples, we mark the medical entities in blue, and italicize the parts in the passage that are crucial for answering. Whenever we shorten a part of the passage, we use [...].



 §.§ Bridging inference

passage

We report a case of a 72 - year - old Caucasian woman with pl-7 positive antisynthetase syndrome . Clinical presentation included interstitial lung disease , myositis , ‘ mechanic 's hands ’ and dysphagia . As lung injury was the main concern , treatment consisted of  and  . Complete remission with reversal of pulmonary damage was achieved , as reported by CT scan , pulmonary function tests and functional status . [...]

query

Therefore , in severe cases an aggressive treatment , combining  and glucocorticoids as used in systemic vasculitis , is suggested .

answer

cyclophoshamide

explanation The reader needs to have the background knowledge that prednisolone is a glucocorticoid, then it becomes obvious that the answer is cyclophoshamide.



 §.§ Object tracking

passage[...] The patient was managed with supportive measures and the National Poisons Information Service was contacted . 
A toxicology consultant was involved in view of the unusual mode of administration . 
Although there was no precedent on how to treat a significant rectal overdose of amitriptyline , it was advised that the patient be administered a  and if failed to adequately remove the tablets then the patient should be given  with 2 litre of Klean - Prep via a nasogastric tube . It was also advised that we admit the patient to a high dependency unit and manage him according to the usual protocol for a tricyclic overdose if complications arose . [...]
query

It seems reasonable to attempt careful removal of the drug from the rectum and if that fails to consider  and whole bowel irrigation .

answer

phosphate enemas

explanation The query mentions removal (A), then  (B) and whole bowel irrigation (C). In the passage, one needs to track those elements and choose the right one. This skill should be considered whenever the gap is part of an enumeration or is mentioned as a part of another entity.



 §.§ Meta knowledge

query

bedaquiline , a new agent with bactericidal and sterilising activity against mycobacterium tuberculosis , is effective against  when given together with a background regimen , and is well tolerated and safe if there is awareness of drug interactions and precautions are taken to avoid potential qt prolongation .

answer

tuberculosis

explanation
The right answer can be inferred from several parts in the passage (not shown), or even from the title or the query. The query, though, is nowhere in the document explicitly answered.




Error analysis

ga-reader, No-Ent, 
a_hat=treatment
requires bridging, but GA reader gets it wrong.
bcr-2013-200832.4

oral neurovascular hamartoma : an extraordinary verdict in the oral cavity summary  the presence of a neurovascular hamartoma within the oral cavity is truly a rare entity . scarcely reported in the literature , these hamartomas are smooth , pinkish masses and are painless , and therefore difficult to diagnose . they are benign in nature and apply pressure to their surroundings . the histological diagnosis remains the gold standard as it comprises of neural tissue and vascular components . treatment is surgical excision with adequate margins . recurrence is reported in cases of incomplete resection .  background  the term hamartoma is derived from the greek word ‘ hamartion ’ which literally means bodily defect . it was initially used by albrecht in 1904.1 it refers to a benign malformation in different areas of the body where growth occurs . the histology consists of an abnormal mixture of cells and tissues localised at the particular site secondary to a developmental error . they do not develop as part of an inflammatory or neoplastic process and have a self limiting proliferation . 2 on the other hand , they do not have a tendency to regress spontaneously .  while considering the oral cavity , the hamartoma might result secondary to the proliferation of smooth muscles , skeletal muscles , fat , salivary tissues , vessels and lymphatics . an oral neurovascular hamartoma is rare because of proliferation of neural tissue solely or in combination with vascular elements .  described below is an unusual case of an oral hamartoma with neurovascular components .  case presentation  a 14 - year - old boy presented to the outpatient department with right cheek swelling for the past 1.5 years . the swelling had been gradually increasing over a period of time and eventually became impossible to ignore . he also reported of decreased mouth opening and a burning sensation within the oral cavity on eating spicy food for 2 years . he did have a significant history of consuming betel nut since the age of 11 years , discontinuing the use once the mouth opening started reducing . his right cheek also had decreased sensation , but there were no symptoms of fever or pain .  the rest of the history was unremarkable .  on examination , the boy appeared to be physically fit with average height and weight . he had a diffuse 2 × 1.5 cm sized right cheek swelling about two finger breadths below the zygoma . the swelling was soft in consistency , non-tender , non-mobile and non-translucent . the swelling would become firm on clenching of teeth and mobile on bimanual examination .  examination of the oral cavity showed significant submucous fibrosis with poor oral hygiene and fullness in the right buccal mucosa . neck examination was unremarkable as was the systemic examination .  as part of his workup , his complete blood picture was normal . the patient brought with him a ct scan of poor quality and thick cuts . the lesion was not clearly outlined but owing to financial constraint the scan was not repeated ( figure 1 ) . no lymphadenopathy was identified on the scan , however .  investigations  histological findings confirmed the diagnosis . the tissue specimen showed haphazard distribution of mature skeletal muscles of varying sizes and nerve fibres closely intertwined with each other on h     amp ; e staining ( figure 2 ) . the specimen was also evaluated with immunehistochemical stains , which showed positive reactivity with s - 100 ( figure 3 ) . the lesion was concluded to be neurovascular hamartoma of the oral cavity .  differential diagnosis  treatment  performing a 2 cm incision over the right buccal mucosa , below the stenson 's duct , the dissection was continued using dissection scissors . on reaching the masseter muscle and locating the mass adjacent to the ramus of the mandible , the mass was removed with wide margins ( figure 4 ) . the cavity was closed with proper tissue reapproximation using absorbable sutures . postoperatively , the patient was provided with ice packs to prevent extensive swelling and started on intravenous augmentin 1.2 g / q8h and analgesics for 48 h after which he was switched to oral augmentin in a dose of 625 mg thrice a day and oral analgesics .  outcome and follow - up  discussion  hamartomas arising within the oral cavity are unusual and may show a variety of clinical presentations and histological and growth patterns . the biological behaviour is benign with reasons for development vague . while epithelial and mesenchymal hamartomas involving the oral cavity are rare , one can not deny that the presence of a neural or a neurovascular component in a hamartoma is even more infrequent . the literature has only reported a handful of these cases .  in 2007 , takeyama et al5 reported of a hamartoma on the hard palate associated with corpus callosum agenesis , microphthalmia and skin malformation . prior to that , goldsmith et al6 described a case of leiomyomatous hamartoma of the posterior tongue leading to dysphagia in a 16 - year - old , whereas miyamoto et al7 documented a lingual hamartoma in a child with a cleft palate . al habeeb et al8 reported a solitary but cutaneous neural hamartoma in a middle - aged individual . the lesion had histological characteristics comprising of numerous mature nerve bundles within the papillary dermis . prior to that , a case of congenital neural hamartoma on the leg was reported by argenyi et al . vascular hamartomas are most common of all in the head and neck . 10  as well as being sporadic , hamartomas can present as a part of a syndrome . epithelial hamartomas can be a part of cronkhite – canada syndrome or orofacial digital syndrome , 11 whereas neuromuscular and vascular hamartomas may become apparent in crohn 's disease . 12  oral neurovascular hamartomas present as smooth surfaced exophytic lesions usually less than 1 cm in diameter . being pedunculated or wide based , their colour may range from pink to yellow . they are painless lesions leading to general discomfort in the patient .  histologically , there is no clear boundary between the lesions and the surrounding normal structures as they are non-encapsulated and poorly circumscribed . composed of hypocellular connective tissue containing aggregates of loose vessels closely packed with small to medium sized nerve bundles , they are not easily distinguishable from the surrounding non-hamartomatous tissues . these nerve bundles are reactive with s - 100 protein . this feature can be easily used as a criterion for diagnosis while examining . 13 this reactivity was also pivotal in our case to reach the final diagnosis . distinction from quite a few oral benign lesions is still hard . oral neurovascular hamartomas resemble traumatic hamartomas and can be distinguished based on the fact that the blood vessels in the former are very closely entwined with the neural component . in traumatic neuromas the two components are separate .  once the diagnosis is confirmed the clinical management comprises of surgical excision . local excision is curative ; a management plan similar to most benign oral lesions . a recurrence only follows an incomplete excision and one should not mistake it with malignant transformation . 14 close clinical follow - up is absolutely essential . the literature does not say much regarding local recurrence of oral neurovascular hamartomas but oral mesenchymal hamartomas arising from the cheek have a reported local recurrence rate of 20 
cal presentation and treatment plan .

@placeholder cures the disease and recurrence is only in cases of incomplete resection .

surgical excision




§ APPENDIX: EXAMPLES

Query uses paraphrasing, with a guideline tone:

D: ... He was transfused and a portovenous abdominopelvic CT scan was performed to define the portal venous anatomy and source of umbilical bleeding ...

Q: A portovenous abdominopelvic CT scan should be performed in patients with cirrhosis and portal hypertension who have recurrent umbilical bleeding , to identify @placeholder and site of bleeding .

A: ...

Requiring inference:

D: Diacon et al22 had also found that BDQ , when given together with a background regimen , led to faster culture conversion and higher rates of culture conversion at 6 months of treatment compared to a background regimen and placebo .

This suggests BDQ 's potential capacity to significantly reduce the treatment period and the debilitating and dangerous adverse effects associated with some of the existing second - line anti-TB drugs , as well as to improve adherence , achieve higher cure rates , and preserve the activity of the background regimen .

Q: Bedaquiline , a new agent with bactericidal and sterilising activity against Mycobacterium tuberculosis , is effective against @placeholder when given together with a background regimen , and is well tolerated and safe if there is awareness of drug interactions and precautions are taken to avoid potential QT prolongation .

Hard to answer right: Answer could be inferred from "anti-TB" mention, or from the reoccuring TB entity in the document and the title.

D: ...

Q: Early diagnosis will increase the quality of life of patients with @placeholder and improve the prognosis .

Here, it's necessary to understand what is the main topic of the case; the entity occurs several times in the text, but the query itself may not be mentioned (in any form) in the document

D: [...] A high level of suspicion is necessary to make an early diagnosis and avoid the irreversible changes in lung architecture that determine a persistent respiratory incapacity , worse prognosis and deterioration of functional status .

Q: A high level of suspicion is necessary to make an early diagnosis and start @placeholder before irreversible damage takes place .

A: appropriate treatment


Need domain knowledge that prednisolone is a glucocorticoid ("bridging"):

D: summary we report a case of a 72 - year - old caucasian woman with @pl_-_7_positive_antisynthetase_syndrome . clinical presentation included @interstitial_lung_disease , @myositis , UNK mechanic "s hands UNK and @dysphagia . as @lung_injury was the main concern , @treatment consisted of @prednisolone and @cyclophosphamide . complete remission with reversal of @pulmonary_damage was achieved , as reported by @ct_scan , @pulmonary_function_tests and functional status . background case presentation a 72 - year - old caucasian woman with no @previous_significant_medical_history , medicated with @simvastatin 40 mg [...]

Q: therefore , in severe cases an @aggressive_treatment , combining @placeholder and @glucocorticoids as used in @systemic_vasculitis , is suggested .

A: cyclophosphamide





§ APPENDIX: ANNOTATING SKILLS


This analysis is to a large extent inspired by the categories listed in <cit.>.

