What can AI do for me: Evaluating Machine Learning Interpretations in
  Cooperative Play
  Machine learning is an important tool for decision making, but its ethical
and responsible application requires rigorous vetting of its interpretability
and utility: an understudied problem, particularly for natural language
processing models. We propose an evaluation of interpretation on a real task
with real human users, where the effectiveness of interpretation is measured by
how much it improves human performance. We design a grounded, realistic
human-computer cooperative setting using a question answering task, Quizbowl.
We recruit both trivia experts and novices to play this game with computer as
their teammate, who communicates its prediction via three different
interpretations. We also provide design guidance for natural language
processing human-in-the-loop settings.
