Title: Caveat Emptor, Computational Social Science: Large-Scale Missing Data in a Widely-Published Reddit Corpus,
Abstract: As researchers use computational methods to study complex social behaviors at
scale, the validity of this computational social science depends on the
integrity of the data. On July 2, 2015, Jason Baumgartner published a dataset
advertised to include ``every publicly available Reddit comment'' which was
quickly shared on Bittorrent and the Internet Archive. This data quickly became
the basis of many academic papers on topics including machine learning, social
behavior, politics, breaking news, and hate speech. We have discovered
substantial gaps and limitations in this dataset which may contribute to bias
in the findings of that research. In this paper, we document the dataset,
substantial missing observations in the dataset, and the risks to research
validity from those gaps. In summary, we identify strong risks to research that
considers user histories or network analysis, moderate risks to research that
compares counts of participation, and lesser risk to machine learning research
that avoids making representative claims about behavior and participation on
Reddit.