Title: Large Scale Replication Projects in Contemporary Psychological Research,
Abstract: Replication is complicated in psychological research because studies of a
given psychological phenomenon can never be direct or exact replications of one
another, and thus effect sizes vary from one study of the phenomenon to the
next--an issue of clear importance for replication. Current large scale
replication projects represent an important step forward for assessing
replicability, but provide only limited information because they have thus far
been designed in a manner such that heterogeneity either cannot be assessed or
is intended to be eliminated. Consequently, the non-trivial degree of
heterogeneity found in these projects represents a lower bound on
heterogeneity. We recommend enriching large scale replication projects going
forward by em- bracing heterogeneity. We argue this is key for assessing
replicability: if effect sizes are sufficiently heterogeneous--even if the sign
of the effect is consistent--the phenomenon in question does not seem
particularly replicable and the theory underlying it seems poorly constructed
and in need of enrichment. Uncovering why and revising theory in light of it
will lead to improved theory that explains heterogeneity and in- creases
replicability. Given this, large scale replication projects can play an
important role not only in assessing replicability but also in advancing
theory.