Augmented Utilitarianism for AGI Safety
  In the light of ongoing progresses of research on artificial intelligent
systems exhibiting a steadily increasing problem-solving ability, the
identification of practicable solutions to the value alignment problem in AGI
Safety is becoming a matter of urgency. In this context, one preeminent
challenge that has been addressed by multiple researchers is the adequate
formulation of utility functions or equivalents reliably capturing human
ethical conceptions. However, the specification of suitable utility functions
harbors the risk of "perverse instantiation" for which no final consensus on
responsible proactive countermeasures has been achieved so far. Amidst this
background, we propose a novel socio-technological ethical framework denoted
Augmented Utilitarianism which directly alleviates the perverse instantiation
problem. We elaborate on how augmented by AI and more generally science and
technology, it might allow a society to craft and update ethical utility
functions while jointly undergoing a dynamical ethical enhancement. Further, we
elucidate the need to consider embodied simulations in the design of utility
functions for AGIs aligned with human values. Finally, we discuss future
prospects regarding the usage of the presented scientifically grounded ethical
framework and mention possible challenges.
