Impossibility and Uncertainty Theorems in AI Value Alignment (or why
  your AGI should not have a utility function)
  Utility functions or their equivalents (value functions, objective functions,
loss functions, reward functions, preference orderings) are a central tool in
most current machine learning systems. These mechanisms for defining goals and
guiding optimization run into practical and conceptual difficulty when there
are independent, multi-dimensional objectives that need to be pursued
simultaneously and cannot be reduced to each other. Ethicists have proved
several impossibility theorems that stem from this origin; those results appear
to show that there is no way of formally specifying what it means for an
outcome to be good for a population without violating strong human ethical
intuitions (in such cases, the objective function is a social welfare
function). We argue that this is a practical problem for any machine learning
system (such as medical decision support systems or autonomous weapons) or
rigidly rule-based bureaucracy that will make high stakes decisions about human
lives: such systems should not use objective functions in the strict
mathematical sense.
  We explore the alternative of using uncertain objectives, represented for
instance as partially ordered preferences, or as probability distributions over
total orders. We show that previously known impossibility theorems can be
transformed into uncertainty theorems in both of those settings, and prove
lower bounds on how much uncertainty is implied by the impossibility results.
We close by proposing two conjectures about the relationship between
uncertainty in objectives and severe unintended consequences from AI systems.
