Equity and Fairness of Single- versus Double-Blind Review
Abstract: Blind review is often seen as a solution to inequities and unfairness of selection processes for employment and credit applications, legal cases, and peer review of scientific work. To test this conjecture, we conducted a large-scale, high-stakes field experiment comparing double-blind review—where both reviewer and author identities are withheld during peer-review—and single-blind—where only the reviewer identities are withheld. We compared them on not only equity (bias), but also fairness (validity, reliability). Each submission (N = 530) received both review types. Single-blind reviews were biased for senior coauthors, while double-blind reviews were slightly biased against women authors. Yet, neither author characteristics nor review process consistently predicted talk quality or popularity. Reviews did predict subsequent publication. Single- and double-blind reviews were moderately reliable, yet the overlap of top submissions for talks between the two was only 40%. This horse-race comparison between single- and double-blind review in situ helps establish some evidence-based principles for using merit-based reviews. For instance, our results coupled with survey data showing scientists’ preference for double-blind review suggest an equitable and fair approach for selecting submissions for scientific conferences is an informed lottery: double-blind review identifies meritorious submissions, then randomly select presentations from this set. Preprint: https://psyarxiv.com/q2tkw/