Title: Simple Surveys: Response Retrieval Inspired by Recommendation Systems,
Abstract: In the last decade, the use of simple rating and comparison surveys has
proliferated on social and digital media platforms to fuel recommendations.
These simple surveys and their extrapolation with machine learning algorithms
shed light on user preferences over large and growing pools of items, such as
movies, songs and ads. Social scientists have a long history of measuring
perceptions, preferences and opinions, often over smaller, discrete item sets
with exhaustive rating or ranking surveys. This paper introduces simple surveys
for social science application. We ran experiments to compare the predictive
accuracy of both individual and aggregate comparative assessments using four
types of simple surveys: pairwise comparisons and ratings on 2, 5 and
continuous point scales in three distinct contexts: perceived Safety of Google
Streetview Images, Likeability of Artwork, and Hilarity of Animal GIFs. Across
contexts, we find that continuous scale ratings best predict individual
assessments but consume the most time and cognitive effort. Binary choice
surveys are quick and perform best to predict aggregate assessments, useful for
collective decision tasks, but poorly predict personalized preferences, for
which they are currently used by Netflix to recommend movies. Pairwise
comparisons, by contrast, perform well to predict personal assessments, but
poorly predict aggregate assessments despite being widely used to crowdsource
ideas and collective preferences. We demonstrate how findings from these
surveys can be visualized in a low-dimensional space that reveals distinct
respondent interpretations of questions asked in each context. We conclude by
reflecting on differences between sparse, incomplete simple surveys and their
traditional survey counterparts in terms of efficiency, information elicited
and settings in which knowing less about more may be critical for social
science.