Achilles Heels for AGI/ASI via Decision Theoretic Adversaries
  As progress in AI continues to advance, it is crucial to know how advanced
systems will make choices and in what ways they may fail. Machines can already
outsmart humans in some domains, and understanding how to safely build ones
which may have capabilities at or above the human level is of particular
concern. One might suspect that artificially generally intelligent (AGI) and
artificially superintelligent (ASI) systems should be modeled as as something
which humans, by definition, can't reliably outsmart. As a challenge to this
assumption, this paper presents the Achilles Heel hypothesis which states that
even a potentially superintelligent system may nonetheless have stable
decision-theoretic delusions which cause them to make obviously irrational
decisions in adversarial settings. In a survey of relevant dilemmas and
paradoxes from the decision theory literature, a number of these potential
Achilles Heels are discussed in context of this hypothesis. Several novel
contributions are made toward understanding the ways in which these weaknesses
might be implanted into a system.
