Accounting for the Neglected Dimensions of AI Progress
  We analyze and reframe AI progress. In addition to the prevailing metrics of
performance, we highlight the usually neglected costs paid in the development
and deployment of a system, including: data, expert knowledge, human oversight,
software resources, computing cycles, hardware and network facilities,
development time, etc. These costs are paid throughout the life cycle of an AI
system, fall differentially on different individuals, and vary in magnitude
depending on the replicability and generality of the AI solution. The
multidimensional performance and cost space can be collapsed to a single
utility metric for a user with transitive and complete preferences. Even absent
a single utility function, AI advances can be generically assessed by whether
they expand the Pareto (optimal) surface. We explore a subset of these
neglected dimensions using the two case studies of Alpha* and ALE. This
broadened conception of progress in AI should lead to novel ways of measuring
success in AI, and can help set milestones for future progress.
