Title: 5 Replicability | Reproducibility and Replicability in Science | The National Academies Press
Headings: Reproducibility and Replicability in Science
(2019)

Reproducibility and Replicability in Science (2019)
5
Replicability
ASSESSING REPLICABILITY
THE EXTENT OF NON-REPLICABILITY
Assessments of Replicability
Perspectives of Researchers Who Have Studied Replicability
Surveys
Retraction Trends
SOURCES OF NON-REPLICABILITY
Non-Replicability That Is Potentially Helpful to Science
BOX 5-1
Varied Sources of Non-Replication
Shaken or Stirred
The Lifespan of Worms
Unhelpful Sources of Non-Replicability
Publication Bias
Misaligned Incentives
Inappropriate Statistical Inference
Poor Study Design
Errors
Incomplete Reporting of a Study
Fraud and Misconduct
BOX 5-2
A Note on Generalizability

Content: 10.17226/25303. ×
Save
Cancel
However, a successful replication does not guarantee that the original scientific results of a study were correct, nor does a single failed replication conclusively refute the original claims. A failure to replicate previous results can be due to any number of factors, including the discovery of an unknown effect, inherent variability in the system, inability to control complex variables, substandard research practices, and, quite simply, chance. The nature of the problem under study and the prior likelihoods of possible results in the study, the type of measurement instruments and research design selected, and the novelty of the area of study and therefore lack of established methods of inquiry can also contribute to non-replicability. Because of the complicated relationship between replicability and its variety of sources, the validity of scientific results should be considered in the context of an entire body of evidence, rather than an individual study or an individual replication. Moreover, replication may be a matter of degree, rather than a binary result of “success” or “failure.” 1 We explain in Chapter 7 how research synthesis, especially meta-analysis, can be used to evaluate the evidence on a given question. ASSESSING REPLICABILITY
How does one determine the extent to which a replication attempt has been successful? When researchers investigate the same scientific question using the same methods and similar tools, the results are not likely to be identical—unlike in computational reproducibility in which bitwise agreement between two results can be expected (see Chapter 4 ). We repeat our definition of replicability, with emphasis added: