Servant of Many Masters: Shifting priorities in Pareto-optimal
  sequential decision-making
  It is often argued that an agent making decisions on behalf of two or more
principals who have different utility functions should adopt a {\em
Pareto-optimal} policy, i.e., a policy that cannot be improved upon for one
agent without making sacrifices for another. A famous theorem of Harsanyi shows
that, when the principals have a common prior on the outcome distributions of
all policies, a Pareto-optimal policy for the agent is one that maximizes a
fixed, weighted linear combination of the principals' utilities.
  In this paper, we show that Harsanyi's theorem does not hold for principals
with different priors, and derive a more precise generalization which does
hold, which constitutes our main result. In this more general case, the
relative weight given to each principal's utility should evolve over time
according to how well the agent's observations conform with that principal's
prior. The result has implications for the design of contracts, treaties, joint
ventures, and robots.
