SafeLife 1.0: Exploring Side Effects in Complex Environments
  We present SafeLife, a publicly available reinforcement learning environment
that tests the safety of reinforcement learning agents. It contains complex,
dynamic, tunable, procedurally generated levels with many opportunities for
unsafe behavior. Agents are graded both on their ability to maximize their
explicit reward and on their ability to operate safely without unnecessary side
effects. We train agents to maximize rewards using proximal policy optimization
and score them on a suite of benchmark levels. The resulting agents are
performant but not safe -- they tend to cause large side effects in their
environments -- but they form a baseline against which future safety research
can be measured.
