Counterexamples to the reward hypothesis On Sutton and Barto's RL book, the reward hypothesis is stated as 


  that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal (called reward)


Are there examples of tasks where the goals and purposes cannot be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal?

All I can think of are tasks with subjective rewards, like "writing good music", but I am not convinced because maybe this is actually definable (perhaps by some super-intelligent alien) and we just aren't smart enough yet. Thus, I'm especially interested in counterexamples that logically or provably fail the hypothesis.
What if a scalar reward is insufficient, or its unclear on how to collapse a multi-dimensional reward to a single dimension. Example, for someone eating a burger, both taste and cost are important. Agents may prioritize taste and cost differently, so its not clear on how to aggregate the two. It is also not clear on how a subjective categorical taste value can be combined with a numerical cost.