Identifying Reasoning Flaws in Planning-Based RL Using Tree Explanations
  Enabling humans to identify potential flaws in an agent's decision making is
an important Explainable AI application. We consider identifying such flaws in
a planning-based deep reinforcement learning (RL) agent for a complex real-time
strategy game. In particular, the agent makes decisions via tree search using a
learned model and evaluation function over interpretable states and actions.
This gives the potential for humans to identify flaws at the level of reasoning
steps in the tree, even if the entire reasoning process is too complex to
understand. However, it is unclear whether humans will be able to identify such
flaws due to the size and complexity of trees. We describe a user interface and
case study, where a small group of AI experts and developers attempt to
identify reasoning flaws due to inaccurate agent learning. Overall, the
interface allowed the group to identify a number of significant flaws of
varying types, demonstrating the promise of this approach.
