Title: Crowd ideation of supervised learning problems,
Abstract: Crowdsourcing is an important avenue for collecting machine learning data,
but crowdsourcing can go beyond simple data collection by employing the
creativity and wisdom of crowd workers. Yet crowd participants are unlikely to
be experts in statistics or predictive modeling, and it is not clear how well
non-experts can contribute creatively to the process of machine learning. Here
we study an end-to-end crowdsourcing algorithm where groups of non-expert
workers propose supervised learning problems, rank and categorize those
problems, and then provide data to train predictive models on those problems.
Problem proposal includes and extends feature engineering because workers
propose the entire problem, not only the input features but also the target
variable. We show that workers without machine learning experience can
collectively construct useful datasets and that predictive models can be
learned on these datasets. In our experiments, the problems proposed by workers
covered a broad range of topics, from politics and current events to problems
capturing health behavior, demographics, and more. Workers also favored
questions showing positively correlated relationships, which has interesting
implications given many supervised learning methods perform as well with strong
negative correlations. Proper instructions are crucial for non-experts, so we
also conducted a randomized trial to understand how different instructions may
influence the types of problems proposed by workers. In general, shifting the
focus of machine learning tasks from designing and training individual
predictive models to problem proposal allows crowdsourcers to design
requirements for problems of interest and then guide workers towards
contributing to the most suitable problems.