Model-based Utility Functions
  Orseau and Ring, as well as Dewey, have recently described problems,
including self-delusion, with the behavior of agents using various definitions
of utility functions. An agent's utility function is defined in terms of the
agent's history of interactions with its environment. This paper argues, via
two examples, that the behavior problems can be avoided by formulating the
utility function in two steps: 1) inferring a model of the environment from
interactions, and 2) computing utility as a function of the environment model.
Basing a utility function on a model that the agent must learn implies that the
utility function must initially be expressed in terms of specifications to be
matched to structures in the learned model. These specifications constitute
prior assumptions about the environment so this approach will not work with
arbitrary environments. But the approach should work for agents designed by
humans to act in the physical world. The paper also addresses the issue of
self-modifying agents and shows that if provided with the possibility to modify
their utility functions agents will not choose to do so, under some usual
assumptions.
