The Difficulty of Novelty Detection in Open-World Physical Domains: An
  Application to Angry Birds
  Detecting and responding to novel situations in open-world environments is a
key capability of human cognition. Current artificial intelligence (AI)
researchers strive to develop systems that can perform in open-world
environments. Novelty detection is an important ability of such AI systems. In
an open-world, novelties appear in various forms and the difficulty to detect
them varies. Therefore, to accurately evaluate the detection capability of AI
systems, it is necessary to investigate the difficulty to detect novelties. In
this paper, we propose a qualitative physics-based method to quantify the
difficulty of novelty detection focusing on open-world physical domains. We
apply our method in a popular physics simulation game, Angry Birds. We conduct
an experiment with human players with different novelties in Angry Birds to
validate our method. Results indicate that the calculated difficulty values are
in line with the detection difficulty of the human players.
