We use four systems to evaluate the difficulty of this dataset.  The first two are an information retrieval system and a word-association method, following the designs of BIBREF26 Clark2016CombiningRS. These are naive baselines that do not parse the question, but nevertheless may find some signal in a large corpus of text that helps guess the correct answer. The third is a CCG-style rule-based semantic parser written specifically for friction questions (the QuaRel INLINEFORM0 subset), but prior to data being collected. The last is a state-of-the-art neural semantic parser. We briefly describe each in turn. 
 Question: Which off-the-shelf tools do they use on QuaRel?
information retrieval system word-association method  CCG-style rule-based semantic parser written specifically for friction questions state-of-the-art neural semantic parser