Learning User-Interpretable Descriptions of Black-Box AI System
  Capabilities
  Several approaches have been developed to answer specific questions that a
user may have about an AI system that can plan and act. However, the problems
of identifying which questions to ask and that of computing a
user-interpretable symbolic description of the overall capabilities of the
system have remained largely unaddressed. This paper presents an approach for
addressing these problems by learning user-interpretable symbolic descriptions
of the limits and capabilities of a black-box AI system using low-level
simulators. It uses a hierarchical active querying paradigm to generate
questions and to learn a user-interpretable model of the AI system based on its
responses. In contrast to prior work, we consider settings where imprecision of
the user's conceptual vocabulary precludes a direct expression of the agent's
capabilities. Furthermore, our approach does not require assumptions about the
internal design of the target AI system or about the methods that it may use to
compute or learn task solutions. Empirical evaluation on several game-based
simulator domains shows that this approach can efficiently learn symbolic
models of AI systems that use a deterministic black-box policy in fully
observable scenarios.
