Risks from Learned Optimization in Advanced Machine Learning Systems
  We analyze the type of learned optimization that occurs when a learned model
(such as a neural network) is itself an optimizer - a situation we refer to as
mesa-optimization, a neologism we introduce in this paper. We believe that the
possibility of mesa-optimization raises two important questions for the safety
and transparency of advanced machine learning systems. First, under what
circumstances will learned models be optimizers, including when they should not
be? Second, when a learned model is an optimizer, what will its objective be -
how will it differ from the loss function it was trained under - and how can it
be aligned? In this paper, we provide an in-depth analysis of these two primary
questions and provide an overview of topics for future research.
