AGI Agent Safety by Iteratively Improving the Utility Function
  While it is still unclear if agents with Artificial General Intelligence
(AGI) could ever be built, we can already use mathematical models to
investigate potential safety systems for these agents. We present an AGI safety
layer that creates a special dedicated input terminal to support the iterative
improvement of an AGI agent's utility function. The humans who switched on the
agent can use this terminal to close any loopholes that are discovered in the
utility function's encoding of agent goals and constraints, to direct the agent
towards new goals, or to force the agent to switch itself off. An AGI agent may
develop the emergent incentive to manipulate the above utility function
improvement process, for example by deceiving, restraining, or even attacking
the humans involved. The safety layer will partially, and sometimes fully,
suppress this dangerous incentive. The first part of this paper generalizes
earlier work on AGI emergency stop buttons. We aim to make the mathematical
methods used to construct the layer more accessible, by applying them to an MDP
model. We discuss two provable properties of the safety layer, and show ongoing
work in mapping it to a Causal Influence Diagram (CID). In the second part, we
develop full mathematical proofs, and show that the safety layer creates a type
of bureaucratic blindness. We then present the design of a learning agent, a
design that wraps the safety layer around either a known machine learning
system, or a potential future AGI-level learning system. The resulting agent
will satisfy the provable safety properties from the moment it is first
switched on. Finally, we show how this agent can be mapped from its model to a
real-life implementation. We review the methodological issues involved in this
step, and discuss how these are typically resolved.
