Optimal Policies Tend to Seek Power
  Some researchers speculate that intelligent reinforcement learning (RL)
agents would be incentivized to seek resources and power in pursuit of their
objectives. Other researchers are skeptical, because RL agents need not have
human-like power-seeking instincts. To clarify this debate, we develop the
first formal theory of the statistical tendencies of optimal policies. In the
context of Markov decision processes, we prove that certain environmental
symmetries are sufficient for optimal policies to tend to seek power over the
environment. These symmetries exist in many environments in which the agent can
be shut down or destroyed. We prove that in these environments, most reward
functions make it optimal to seek power by keeping a range of options available
and, when maximizing average reward, by navigating towards larger sets of
potential terminal states.
