Title: Multi-Observation Elicitation,
Abstract: We study loss functions that measure the accuracy of a prediction based on
multiple data points simultaneously. To our knowledge, such loss functions have
not been studied before in the area of property elicitation or in machine
learning more broadly. As compared to traditional loss functions that take only
a single data point, these multi-observation loss functions can in some cases
drastically reduce the dimensionality of the hypothesis required. In
elicitation, this corresponds to requiring many fewer reports; in empirical
risk minimization, it corresponds to algorithms on a hypothesis space of much
smaller dimension. We explore some examples of the tradeoff between
dimensionality and number of observations, give some geometric
characterizations and intuition for relating loss functions and the properties
that they elicit, and discuss some implications for both elicitation and
machine-learning contexts.