Intelligence and Unambitiousness Using Algorithmic Information Theory
  Algorithmic Information Theory has inspired intractable constructions of
general intelligence (AGI), and undiscovered tractable approximations are
likely feasible. Reinforcement Learning (RL), the dominant paradigm by which an
agent might learn to solve arbitrary solvable problems, gives an agent a
dangerous incentive: to gain arbitrary "power" in order to intervene in the
provision of their own reward. We review the arguments that generally
intelligent algorithmic-information-theoretic reinforcement learners such as
Hutter's (2005) AIXI would seek arbitrary power, including over us. Then, using
an information-theoretic exploration schedule, and a setup inspired by causal
influence theory, we present a variant of AIXI which learns to not seek
arbitrary power; we call it "unambitious". We show that our agent learns to
accrue reward at least as well as a human mentor, while relying on that mentor
with diminishing probability. And given a formal assumption that we probe
empirically, we show that eventually, the agent's world-model incorporates the
following true fact: intervening in the "outside world" will have no effect on
reward acquisition; hence, it has no incentive to shape the outside world.
