Title: Single versus Double Blind Reviewing at WSDM 2017,
Abstract: In this paper we study the implications for conference program committees of
using single-blind reviewing, in which committee members are aware of the names
and affiliations of paper authors, versus double-blind reviewing, in which this
information is not visible to committee members. WSDM 2017, the 10th ACM
International ACM Conference on Web Search and Data Mining, performed a
controlled experiment in which each paper was reviewed by four committee
members. Two of these four reviewers were chosen from a pool of committee
members who had access to author information; the other two were chosen from a
disjoint pool who did not have access to this information. This information
asymmetry persisted through the process of bidding for papers, reviewing
papers, and entering scores. Reviewers in the single-blind condition typically
bid for 22% fewer papers, and preferentially bid for papers from top
institutions. Once papers were allocated to reviewers, single-blind reviewers
were significantly more likely than their double-blind counterparts to
recommend for acceptance papers from famous authors and top institutions. The
estimated odds multipliers are 1.63 for famous authors and 1.58 and 2.10 for
top universities and companies respectively, so the result is tangible. For
female authors, the associated odds multiplier of 0.78 is not statistically
significant in our study. However, a meta-analysis places this value in line
with that of other experiments, and in the context of this larger aggregate the
gender effect is also statistically significant.