Mitigating belief projection in explainable artificial intelligence via
  Bayesian Teaching
  State-of-the-art deep-learning systems use decision rules that are
challenging for humans to model. Explainable AI (XAI) attempts to improve human
understanding but rarely accounts for how people typically reason about
unfamiliar agents. We propose explicitly modeling the human explainee via
Bayesian Teaching, which evaluates explanations by how much they shift
explainees' inferences toward a desired goal. We assess Bayesian Teaching in a
binary image classification task across a variety of contexts. Absent
intervention, participants predict that the AI's classifications will match
their own, but explanations generated by Bayesian Teaching improve their
ability to predict the AI's judgements by moving them away from this prior
belief. Bayesian Teaching further allows each case to be broken down into
sub-examples (here saliency maps). These sub-examples complement whole examples
by improving error detection for familiar categories, whereas whole examples
help predict correct AI judgements of unfamiliar cases.
