Scientific discovery in a model-centric framework: Reproducibility, innovation, and epistemic diversity
Berna Devezer Luis G. Nardin Bert Baumgaertner Erkan Buzbas







Scientific discovery in a model-centric framework: Reproducibility, innovation, and epistemic diversity 




Berna Devezer1,5*,
Luis G. Nardin2,5,
Bert Baumgaertner3,5,
Erkan Buzbas4,5


1 Dept. of Business, University of Idaho, Moscow, ID, USA


2 Dept. of Informatics, Brandenburg University of Technology, Cottbus, Germany


3 Dept. of Politics and Philosophy, University of Idaho, Moscow, ID, USA


4 Dept. of Statistical Science, University of Idaho, Moscow, ID, USA


5 Center for Modeling Complex Interactions, University of Idaho, Moscow, ID, USA









These authors contributed equally to this work.

















* bdevezer@uidaho.edu






§ ABSTRACT

Consistent confirmations obtained independently of each other lend credibility to a scientific result. We refer to results satisfying this consistency as reproducible and assume that reproducibility is a desirable property of scientific discovery. Yet seemingly science also progresses despite irreproducible results, indicating that the relationship between reproducibility and other desirable properties of scientific discovery is not well understood. These properties include early discovery of truth, persistence on truth once it is discovered, and time spent on truth in a long-term scientific inquiry. We build a mathematical model of scientific discovery that presents a viable framework to study its desirable properties including reproducibility. In this framework, we assume that scientists adopt a model-centric approach to discover the true model generating data in a stochastic process of scientific discovery. We analyze the properties of this process using Markov chain theory, Monte Carlo methods, and agent-based modeling. We show that the scientific process may not converge to truth even if scientific results are reproducible and that irreproducible results do not necessarily imply untrue results. The proportion of different research strategies represented in the scientific population, scientists' choice of methodology, the complexity of truth, and the strength of signal contribute to this counter-intuitive finding. Important insights include that innovative research speeds up the discovery of scientific truth by facilitating the exploration of model space and epistemic diversity optimizes across desirable properties of scientific discovery.






§ AUTHOR SUMMARY


We mathematically model a virtual scientific community to understand how reproducibility is related to scientific discovery of truth. Studying this community over time, we find that irreproducible results may not be untrue and reproducible results do not guarantee that the scientific process will converge to truth. A combination of research strategies, statistical methods, and the state of truth explains this finding. We show that innovative research speeds up the discovery of truth and that the pursuit of diverse research approaches in the scientific population optimizes across desirable properties of scientific discovery.




§ INTRODUCTION

Consistent confirmations obtained independently of each other lend credibility to a scientific result <cit.>. We refer to this notion of multiple confirmations as reproducibility of scientific results. Ioannidis <cit.> argued that a research claim is more likely to be false than true, partly due to the prevalent use of statistical significance and null hypothesis significance testing as method of inference. Recent theoretical research explored aspects of scientific practice contributing to irreproducibility. McElreath and Smaldino <cit.> modeled a population of scientists testing a variety of hypotheses and tracking positive and negative published findings to investigate how the evidential value of replication studies changed with the base rate of true hypotheses, statistical power, and false positive rate. Other studies found that current incentive structures may lead to degradation of scientific practice <cit.>. Publication bias was also proposed to contribute to the transitioning of incorrect findings from claim to fact <cit.>. These studies focus on how structural incentives and questionable research practices (QRPs) influence reproducibility of scientific results within a hypothesis-centric framework, and how to improve statistical practices and publication norms to increase reproducibility. Under limitations of hypothesis testing <cit.>, however, understanding salient properties of the scientific process is challenging, especially for fields that progress by building, comparing, selecting, and re-building models.

In this work, we make three contributions to the literature on meta-research. First, we present a model-centric mathematical framework modeling scientists' convergence to truth in the process of scientific discovery. We identify, mathematically define and study the relationship between key properties of this process such as early discovery of truth, persistence on truth once it is discovered, time spent on truth in a long-term scientific inquiry, and rate of reproducibility. Second, in a system without QRPs or structural incentives, we study how the diversity of research strategies in the scientific population, the complexity of truth, and the noise-to-signal ratio in the true data generating model affect these properties. Third, we study the scientific process where scientists engage in model comparison instead of statistical hypothesis testing. Model comparison aims to select a useful model that approximates the true model generating the data and it has long been a cornerstone in many scientific disciplines because of its generality. Our model-centric view allows us to study the process of scientific discovery under uncertainty, bypassing the complications inherited from hypothesis testing <cit.>.




§ A MODEL-CENTRIC META-SCIENTIFIC FRAMEWORK

We adopt a notion of confirmation of results in idealized experiments and build a mathematical framework of scientific discovery based on this notion.

Model, idealized experiment, replication experiment, and reproducibility.
We let K be the background knowledge on a natural phenomenon of interest, M be a prediction in the form of a probability model parameterized by θ∈Θ, that is in principle testable using observables, and D be the data generated by the true model. The degree of confirmation of M by D is assessed by S, a fixed and known method. We define ξ, an idealized experiment, as (M,θ,D,S,K).

In an idealized experiment ξ, the data D confirms the model M if (M|D,K)>(M|K), where (M|D,K) and (M|K) are probabilities of M after and before observing the data, respectively. By Bayes's Theorem, (M|D,K)/(M|K) is proportional to the likelihood (D|M,K). Large (D|M,K) implies high degree of confirmation of M. Complex models, however, have a tendency to inflate (D|M,K) and hence (M|D,K). As a measure against overfitting, modern model comparison statistics S are not only based on P(D|M,K) but also penalize the complexity of M to prevent inflating the likelihood under complex models. For several well-known S, smaller S(M) means the model M is more favorable in a set of models, and we follow this convention here.

In a scientific inquiry, a novel prediction is often tested against a status quo consensus. We formulate this situation by denoting the novel prediction as a proposed model M_P which is tested against the global model M_G, the scientific consensus. Conditional on the data, S(M_P)<S(M_G) means that the proposed model is more favorable than M_G. In this case, M_P becomes the new scientific consensus, otherwise the global model remains as the scientific consensus.

The description of scientific inquiry given in the last paragraph and reproducibility of results in a replication experiment as follows. If ξ_1 given by (M_P,θ, D_1, S, K_1) is tested against M_G, then the experiment ξ_2 immediately following ξ_1 is a replication experiment for ξ_1 if and only if ξ_2 is given by (M_P,θ, D_2, S, K_2) and it is tested against the same M_G as ξ_1. That is, the replication experiment proposes the same model, uses the same methods, and is tested against the same global model as the original experiment. Of two elements that differ between the original experiment and the replication experiment, the first is D_2, which is the data that is generated in the replication experiment independent of the data D_1 of the original experiment. The second is the background information K_2 which includes all the information necessary to replicate ξ_1. In particular, K_2 includes the knowledge that M_P was the proposed model in ξ_1, it was tested against M_G, and the outcome of this test—whether M_P was updated to consensus or M_G remained as the consensus. We say that the replication experiment ξ_2 reproduces the results of ξ_1 if the results of ξ_1 and ξ_2 are the same in terms of updating the consensus. There are two mutually exclusive ways that ξ_2 can reproduce the results of ξ_1:
1) If the proposed model in ξ_1 won against the global model, then this must also be the case in ξ_2, that is S(M_P)<S(M_G) in both experiments. 2) If the proposed model in ξ_1 lost against the global model, then this must also be the case in ξ_2, that is S(M_P)>S(M_G) in both experiments. Otherwise, we say that ξ_2 fails to reproduce the results of ξ_1. These definitions of replication experiment and reproducibility of results formalize necessary open science practices for potential reproducibility of results: Information about the proposed and global models in the original experiment and the results of this experiment which we capture by K_2 must be transferred to ξ_2.


Stochastic process of scientific discovery.
We assume an infinite population of scientists who conduct a sequence of idealized experiments to find the true model generating the data (see S1_Appendix for mathematical framework). In the population, we consider various types of scientists, each with a mathematically well-defined research strategy for proposing models. Scientists search for a true model in a set of linear models. Linear models were chosen because they can accommodate a variety of designs with straightforward statistical analysis, and their complexity is mathematically tractable (S2_Appendix). We define model complexity as a function of the number of model parameters and interaction terms, and visualize it by representing each model with a unique geometry on an equilateral hexagon inscribed in its tangent circle (Fig. <ref>A).





We assume a discrete time process with t = 0, 1, ⋯, where at each time step an experiment ξ^(t) is conducted by a scientist randomly selected from a population of scientists with equal probability. The experiment entails proposing a model M_P^(t) as a candidate for the true data generating mechanism. The probability of proposing a particular model is determined by the scientist's research strategy and the global model M_G^(t)—the current scientific consensus. The scientist compares the global model against the proposed model using new data D^(t) generated from the true model and a model comparison statistic S. The model with favorable statistic is set as the global model for the next time step M_G^(t+1). Because the probability of proposing a model is independent of the past and the transition from M_G^(t) to M_G^(t+1) admits the (first order) Markov property, the stochastic process representing the scientific process is a Markov chain. This mechanism represents how scientific consensus is updated in light of new evidence. We study the mathematical properties of this process for different scientist populations representing a variety of research strategies.

Introducing replication experiments to the process fundamentally alters the probability mechanism of updating global models: By definition, a replication experiment depends on the experiment conducted at the previous time step via K_2. Hence the stochastic process is a higher order Markov chain (see <cit.>) and we turn to an agent-based model (ABM) <cit.> to analyze the process with replication. Our ABM is a forward-in-time, simulation-based, individual-level implementation of the scientific process where agents represent scientists (S3_Algorithm).

We assume that reproducibility is a desirable property of scientific discovery. However, arguably early discovery of truth, persistence on truth once it is discovered, and long time spent on truth in a long-term scientific inquiry are also desirable properties of scientific discovery since they would characterize a resource efficient and epistemically accurate scientific process. We seek insight into the drivers of these properties and the relationship among them, which we motivate with the following questions (see S4_Appendix for mathematical definitions). How quickly does scientific community discover the true model? We assess this property by the mean first passage time to the true model and view it as a key indicator of resource efficiency in the process of scientific discovery. How “sticky” is the true model as global model? We define the stickiness of the true model as the mean probability of staying in the true model once it becomes global model. How long does scientific community stay on the true model? The stationary probability that the true model is global model has the interpretation of the long-term stay of the scientific community on the true model. How reproducible are the results of experiments? We track replication experiments to calculate the rate of reproducibility when the true model is global model, as well as when it is not. We study the answers to these questions as a function of the following aspects of model-centric approach to scientific discovery: the proportion of research strategies in the scientific population, the complexity of the true model generating the data, the ratio of error variance to deterministic expectation in the true model (i.e., noise-to-signal ratio), and the model comparison statistic.

Scientist types. We include distinct research strategies to explore the effect of epistemic diversity in the scientific population. We define simple research strategies where our scientists do not have a memory of their past decisions and they do not interact directly with each other, but only via the global model. Nonetheless, the research strategies we include in our model seem reasonably realistic to us in capturing the essence of some well-known research approaches. Figure <ref> illustrates our stochastic process of scientific discovery for a specific scientist population.




For the process with no replication, we define three types of scientists: Tess, Mave, Bo (S5_Appendix). Tess, the theory tester, uniformly randomly selects a proposal model that is only one main effect predictor away from the current global model. We impose a hierarchical constraint on interaction terms in the sense that when Tess proposes to drop a predictor from the current global model, all higher order interactions including this term are dropped too. We think of Tess' strategy as a refinement of an existing theory by testing current consensus against models that are close to it. Mave, the maverick does not build off of the current consensus but she ignores it. She advances novel ideas and uniformly randomly selects a model from the set of all models. The novelty-seeking aspect of her strategy is similar to a maverick strategy proposed in prior research on epistemic landscapes and the division of cognitive labor <cit.>. However, in contrast to that strategy, Mave does not actively aim to avoid previously tested models and she acts independently of the current scientific consensus. Bo, the boundary tester systematically tests the boundaries of the current global model. She selects a model that adds interactions to the current global model to explore the conditions under which the current global model holds. After her, if the tested boundary has been confirmed by the data, the global model may earn a new predictor representing a main interaction and the lower order interactions associated with it that are not currently in the model. If the tested boundary has not been confirmed by the data, the global model does not change.


For the process with replication, we introduce Rey, the replicator who conducts a replication experiment which is the exact same experiment conducted by her precedent, but with new data. She reproduces the results of the preceding experiment if she confirms as global model the same model confirmed as global model by her precedent. Thus, Rey compares the same pair of proposed model (M_P) and global model (M_G) as that of her predecessor, using new data, and the replication is successful if two conditions are satisfied: 1) either the results of her predecessor and the replication experiment are both judged favorable against the global model (M_G), or they both are not, leaving scientific consensus unchanged, 2) sufficient information about the the result of her predecessor's experiment is available through the background knowledge of the replication experiment to assess if the first condition holds. This second condition implies that a replication experiment requires open science practices for transferring sufficient information about the experiment in the previous time step to the replication experiment in order for the latter to assess the reproducibility of the result.

Scientist populations. We assess the effect of each strategy by considering populations of scientists in which Rey, Tess, Bo, and Mave are represented at varying proportions ([S6_Table]S6 Table). Of particular interest to us are homogeneous populations where the dominant scientist type comprises 99% of the scientist population and epistemically diverse populations where all scientist types are represented in equal proportion.


Accumulation of evidence. The background knowledge that a scientist brings to an experiment consist of the global model, all other models in the system, predictors, and their parameters as well as the results from the previous experiment if the current experiment is a replication experiment. Scientific evidence in our model accumulates through experiments and all the evidence is counted at the end of a long run. Data set in each experiment has a weight of one and the sample proportion of experiments which reproduce a result converges to the true value of reproducibility rate by the Law of Large Numbers.


Model comparison criteria. We adopt two well-known likelihood-based criteria for model comparison and show how these interact with the behavior of scientists in the population: the Schwarz Criterion (SC) <cit.> and the Akaike's Information Criterion (AIC) <cit.>. A smaller value of these statistics indicate a better model performance than a larger value. When the true model generating the data is in the universe of candidate models, SC is statistically consistent and selects the true model with probability 1 as n→∞ (S5_Appendix).

Maximum number of factors in the model. For computational feasibility, we fix the number of factors in the linear model to 3, which results in 14 models. Each of these 14 models refers to its linear structure. In this sense, there are infinitely many probability distributions that can be fully specified within a model. For the system without replication, we analyze all 14 models as true models. The most complex model has 7 predictors (Fig. <ref>A), including three main effects, three 2–way interactions, and one 3–way interaction. We fix the sample size to 100 and calibrate the ratio of the error variance σ^2 to expected value of the model at the mean value of the predictors (y|μ_x) where μ_x=(x). We fix σ^2:(y|μ_x) to (1:4) (and include results for (1:1), (4:1) in [S13_Figure]S13 Figure-[S22_Figure]S22 Figure; S5_Appendix).

Design of ABM experiments. For the system with replication, we use three true models representing a gradient of complexity (Fig. <ref>A TM_1,TM_2,TM_3). We set up a completely randomized factorial simulation experiment: 3 true models, 3 σ^2:(y|μ_x) levels at (1:4),(1:1),(4:1), 5 scientist populations ([S6_Table]S6 Table), and 2 model comparison statistics (AIC, SC). We run each experimental condition as an ABM simulation for 11000 iterations and replicated 100 times, each using a different random seed. We discard the first 1000 iterations as burn-in, except when analyzing the mean first passage time to true model. Code and data are given in S7_CodeAndData.



§ A BRIEF DISCUSSION OF OUR MODELING CHOICES, ASSUMPTIONS, AND THEIR IMPLICATIONS


Our model-centric framework facilitates investigating the consequences of the process of scientific discovery, including reproducibility, as a system-wide phenomenon. System-wide reproducibility and its relationship to scientific discovery are largely unexplored topics. Navigating through numerous potential variables and parameters to create a realistic system rich in behavior whose outcomes are easily interpretable is challenging. Our model aims to create such a system by making design choices and simplifying assumptions. Among many results that we obtain, we report some intuitive results as sanity checks. These results connect our idealized system to reality. However, we highlight the results that seem counter-intuitive to us because we find them to be interesting patterns warranting further investigation. The implications and limitations of each specific result are discussed in the Results section.

Here, we qualitatively clarify the implications and limitations of our system and emphasize the assumptions which constitute its salient features for our results to hold. We anchor our system firmly against the backdrop of guarantees provided by statistical theory to avoid over-generalization.

What statistical theory offers in isolation.
A well-known statistical inference mode is comparing a set of hypotheses represented as probabilistic models, with the goal of selecting a model. A statistical method selects the model which fits the stochastic observations best according to some well-defined measure. Consider the following three conditions:



  * There exists a true model generating the observations and it is in the search set.

  * The signal in the observations is detectable.

  * A reliable method whose assumptions are met is used to perform inferences.


If these conditions are met, then the statistical theory guarantees that under repeated testing with independent observations, the true model is selected with highest frequency. This frequency approaches to a constant value determined by conditions (1),(2), and (3). The practical implication of this guarantee is that the results under the true model are reproducible with a constant rate.

We now contemplate on the consequences of violating conditions (1)-(3).
If condition (1) is not met, then the true model cannot be selected. In this case, well-established methods select the model that is closest to the true model and in the set with highest frequency. As we discuss in research strategies below, this is a situation where if results are reproducible they are not true.

Conditions (2) and (3) work in conjunction. A method is reliable with respect to the strength of the signal it is designed to detect. There are a variety of ways to evaluate the reliability of statistical methods. Hypothesis tests use specificity and sensitivity. Modern model selection methods often invoke an information-theoretic measure. Intuitively, we expect a statistical test designed to detect the bias in a coin to perform well even with small sample size if the coin is heavy on heads because the data structure is simple and the signal strong. We would be fortunate, however, if a model selection method can discern between two complex models close to each other with the same sample size. If we violate condition (2) or (3), then we have an unreliable method to detect the strength of the signal. In this case, even if the true model generating the observations is in the set, we might not be able to select it with high frequency due to the mismatch between the performance of the method and the strength of the signal (see also <cit.> for a discussion of how method choice might affect reproducibility).

When conditions (1)-(3) are met, statistical guarantees hold in the absence of external factors that are not part of the data generating mechanism and the inference process. To quote Lindley <cit.>: “Statisticians tend to study problems in isolation, with the result that combinations of statements are not needed, and it is in the combinations that difficulties can arise [...]” Scientific claims often are accompanied by statistical evidence to support them. However, we doubt that in practice scientific discovery is always based on evidence using statistical methods whose assumptions are satisfied. A variety of external factors such as choices made in theory building, design of experiments, data collection, and analyses might affect system-wide properties in scientific discovery. Our work is motivated to develop intuition on how some of these external factors affect the guarantees made by statistical theory. In particular, we introduce external factors which violate conditions (1)-(3), and produce counter-intuitive results. We explicitly discuss two factors that have major effects on our outcomes next.

Research strategies as an external factor and their potential counter-intuitive effect on reproducibility.
When scientists aim to discover a true model among a large number of candidate models, reducing the search space is critical. Our system introduces one external factor to statistical theory as research strategies which determine the models to be tested at each step of the scientific process thereby serving as a means to reduce the search space. However, by choosing models to reduce the search space, the research strategies also affect the frequency of testing each model. As a consequence of affecting frequencies of tests, these strategies may alter the results guaranteed by statistical theory in many ways. Results depend on how frequently these strategies are employed by the scientific community and how frequently they propose each model. In this sense, the strategies determine the opportunity given to each model to show its value.

To clarify the effect that strategies can have on reproducibility, we give an extreme example. Consider a search space with only three possible models. We pursue the bizarre research strategy to always test two of these models against each other, neither of which is the true model generating the data. Then:



  * The true model will never be selected because it is never tested.

  * Between the two models tested, the model that is closer to the true model will be selected with higher frequency than the model further.

  * The result stated in item (2) is reproducible.


This toy example shows that we can follow strategies which produce results that are reproducible but not true. In this work, we further show that counter-intuitive results like this can arise under mild research strategies that modify the search space in subtle ways. However, our results do not mean that true results are not reproducible. In fact, this is a sanity check that we have in our system: provided the three conditions of the previous section are met, true results are reproducible.

System updating as an external factor and its effect on reproducibility.
A second external factor that our system introduces is the temporal characteristic of the scientific discovery. Probabilistic uncertainty dictates that one instance of statistical inference cannot be conclusive even if the true model is included in the test set and it produces highly reliable data. Thus, repeated testing through time using independent data sets calls for a temporal stochastic process. A state variable defines this process whose outcome is determined as a function of this state variable with respect to a reasonable measure of success.

The natural state variable in our system is the model selected in each test. We think of this model as a pragmatic consensus of the scientific community at any given time in the process of scientific discovery. When another model is proposed, it is tested against this consensus.

There are difficulties in defining a reasonable success measure for models, however. A pragmatic consensus of the scientific community is presumably a model which withstands testing against other models to some degree. The consensus is expected to survive even if it is not selected, say, a few times. A tally of each model against every other model can be kept introducing a system memory. This tally, can be used as prior evidence in the next testing instance of particular models. Introducing memory into the temporal process is technically easy. The real difficulty is how to choose the success measure. A decision rule about when a model should lose confidence and be replaced by another model is needed. Consider the following example: Consensus model A and model B were compared two times each winning once. In the third comparison, model B wins. Should we abandon model A and make model B consensus? If not, how many more times should model B win against model A before we are willing to replace model A?

We find these questions challenging, but they help illustrate our point. One of several well-known rules from decision theory can be implemented to update the consensus. No matter which rule is chosen, however, it will affect system-wide properties including the reproducibility of results. In this sense, a decision rule is another external factor: Precisely because the rule dictates when a model becomes consensus, it has the power to alter the frequency of statistical results in the process, otherwise obtained in isolation.

Even without the complication of a decision rule, scientific strategies make our system complex. They produce a diverse array of results whose implications we do not fully understand. Hence, we left the complication of model memory out of our system by choosing to update the consensus with the selected model at each test. This corresponds to a memoryless 0-1 decision rule. We admit that this memoryless property of our model is unsatisfactory and might not reflect a realistic representation of the scientific process. We caution the reader to interpret our results with care on this aspect. On the other hand, we are interested in system-wide and aggregate results of our model through time. That is, we look at the rate of reproducibility and other properties of scientific discovery in a given process by integrating across many independent iterations of tests and systems. Thus, our sanity checks still apply. For example, we expect (and find) the scientific consensus to converge to the true model once it is discovered, and the true model to be sticky and reproducible. While the memorylessness of our system might prevent the scientific process being realistically captured at any given point in the process, on the aggregate we are able to observe certain realistic patterns.




§ RESULTS




First, we present results in a system with no replicator where properties of our scientific process can be obtained for all true models in our model space using Markov chain theory and computationally efficient Monte Carlo methods (S8_Appendix). We use this computational advantage to gain insight into process properties and to inform ABM experiments for the system with replication, in which exploring all model space is computationally unfeasible. Second, we present results from these ABM experiments ([S9_Table]S9 Table).



 §.§ Results in a system with no replication
  We examine stickiness of the true model, time spent at the true model, and mean first passage time to the true model for populations composed of different proportions of Tess, Mave, and Bo. Our interest is in how the proportion of different research strategies represented in scientist populations influences these desirable properties of scientific discovery. A key feature of the theoretical calculations we present in this section is the implementation of soft research strategies where all scientists are allowed to propose a model not consistent with their strategy with a small probability. Technically, this feature guarantees that the transition probability matrix of the Markov chain is well connected. In the system with replicator, we investigate hard research strategies—where scientists are allowed to propose only models consistent with their strategy—in addition to soft strategies. We compare results across four scientific populations, two model comparison statistics—Akaike's Information Criterion (AIC) and Schwarz Criterion (SC), and all possible true models in our model space (Fig. <ref>A). We fix a low error variance to model expectation ratio (1:4).

Stickiness, the probability of the true model staying as global model once it is hit, is high under low error both for AIC (Fig. <ref>B) and SC (Fig. <ref>C). This result serves as a sanity check for our theoretical model. Once the true model becomes the consensus, it stays as such most of the time. The stickiness of the true model increases with complexity, except for the Bo-dominant population. For Bo-dominant population, stickiness decreases with complexity, except for the most complex true model which Bo cannot overfit.

Even though the true model is sticky under low error, and hence, tends to stay as global model once it is hit, the system still spends considerable time at models that are not true. For example, Bo-dominant population overfits unduly complex models, spending only 25% of the time at the true model under AIC and 48% under SC ([S10_Figure]S10 Figure, [S11_Figure]S11 Figure). This population spends most time in models more complex than the true model. Out of 14 true models in our model space, under AIC 4 are not among Bo-dominant population's top 3 most visited models, and under SC 4 are not the most visited model. This is a consequence of Bo's strategy to add only interaction predictors to her proposed models, which regularly pits the global model against more complex models. Bo's poor performance is striking because boundary conditions show whether the relationship between variables holds across the values of other variables and hence, boundary testing is a widely used strategy for theory development in many disciplines <cit.>. In comparison with Bo-dominant population, Tess- and Mave-dominant populations spend more time in the true model, regardless of its complexity (47% and 41% under AIC and 67% and 72% under SC, respectively). Overall, the theory testing and maverick strategies maximize the probability of spending time at the true model under AIC and SC, respectively.


For the epistemically diverse population, the true model is in top 3 most visited models irrespective of its complexity ([S12_Figure]S12 Figure). Because boundary testing strategy is ineffective in capturing the true model, the presence of Bos causes the epistemically diverse population to spend less time at the true model (36% under AIC and 62% under SC) than Tess- or Mave-dominant populations. However, the effect of overfitting complex models by Bo is alleviated in this population due to the presence of other research strategies in the population and does not prevent the epistemically diverse population from consistently recovering the true model. In essence, epistemic diversity protects against ineffective research strategies.


We assessed the speed of scientific discovery by the mean first passage time to the true model. In this system without replication, where the transition matrix is well connected due to the implementation of soft research strategies, the true model is hit quickly across all populations (between 3.39 and 6.43 mean number of steps) when noise-to-signal ratio is low. Increasing the proportion of boundary testers in the population, however, slows down the discovery of the true model. Further, the model comparison statistic interacts with scientist populations with respect to the speed of discovery. Under AIC, Tess-dominant population is the fastest to find the true model (Fig. <ref>A, Tess). In comparison, as shown by red region in Fig. <ref>A, Bo-dominant population is slow to discover the true model. Tess's speeding up the discovery of truth is also reflected in the epistemically diverse population. Using SC as opposed to AIC as the model comparison statistic decreases differences across populations (Fig. <ref>B), increasing the speed of discovery for all populations. The fastest population to hit the true model is the epistemically diverse population under SC. We find that the speed of discovery slows down considerably as the noise-to-signal ratio is increased to (4:1)
([S22_Figure]S22 Figure).




These results from the system with no replication show that while the true model is sticky and reached quickly under low error in a well connected system, the scientific population still spends considerable time in false models over the long run. Moreover, proportion of research strategies in scientific populations, true model complexity, and model comparison statistic have an effect on all of these properties. Overall, Bo-dominant population performs poorly in most scenarios whereas Tess- and Mave-dominant populations excel in different scenarios. Epistemically diverse population minimizes the risk of worst outcomes. These patterns that we described change substantially as the ratio of error variance to model expectation in the system increases ([S13_Figure]S13 Figure-[S22_Figure]S22 Figure). We now discuss the implications and limitations of results presented so far for the scientific practice.

Implications and limitations. When the truth exists and is accessible, we find that scientific process indeed discovers and sticks to it in most situations. The exceptions to this result come from 1) research strategies that search the model space in a biased manner and fail to test the true model against alternatives often enough, and 2) large error in the data generating process. In practice, (1) might be realized when there is no overarching theoretical framework guiding the search of model space but instead folk theories or intuitions are used to reduce the possibilities <cit.>. Further, (2) is a real challenge, especially in disciplines where data do not carry a strong signal (e.g., low estimated effect sizes in psychology literature) or whose methods are not sufficiently fine-tuned to detect the signal (e.g., high measurement error). While these are implications that might hold qualitatively for real-life scientific practice, we caution the readers to not over-generalize specific parameters such as the stickiness of a true model and the proportion of time spent at the true model. These quantities depend on the parameters of our system, such as the number of models in the universe, and the linear models framework.




 §.§ Results in a system with replication

 In addition to the properties analyzed in the previous section, in the system with replication we can also analyze the rate of reproducibility since we introduce a replicator in the system. One of our goals is to understand the relationship between reproducibility and other desirable properties of scientific discovery. Informed by the findings reported in the previous section, we run the ABM under three true models of varying complexity, three levels of error variance to model expectation ratio, five scientific populations, and two model comparison statistics ([S9_Table]S9 Table). Moreover, ABM allows us to implement hard research strategies where scientists propose only models complying with their strategies and all models incompatible with a given research strategy have zero probability of being proposed by the scientists pursuing that strategy (S5_Appendix). Thus, connectedness among models is restricted in this framework for all scientists with the exception of Mave whose research strategy allows her to propose any model at any given time, and who thereby maintains a soft research strategy. When the transition matrix is highly connected, the discovery of truth is fast, as shown in the previous section. In the current section, we explore how the speed of discovery changes under restricted connectedness for different scientist populations.



  §.§.§ Reproducible results do not imply convergence to scientific truth
 We first explored the relationship between the rate of reproducibility and other desirable properties of scientific discovery, and found that this relationship must be interpreted with caution. In our framework, we defined the rate of reproducibility as the probability of the global model staying the same after a replication experiment. We show that the rate of reproducibility has no causal effect on other desirable properties of scientific discovery including: the probability that a model is selected as the global model in the long run, the mean first time to hit a model, and stickiness of a model (see S23_Appendix for mathematical proof). Thus, although multiple confirmations of a result in a scientific inquiry lend credibility to that result, withstanding the test of multiple confirmations is not sufficient for convergence to scientific truth.

On the other hand, desirable properties of scientific discovery and the rate of reproducibility might be correlated. Whether there is any correlation depends on the research strategies and their frequency in the population (see S23_Appendix for mathematical explanation). We present scatter plots (Fig. <ref>A and Fig. <ref>B) as evidence for the complexity of these correlations across scientist populations. From these scatter plots and Fig. <ref>C, we see, for example, that Bo-dominant populations reach perfect rate of reproducibility while spending little time at the true model (as assessed by the very low Spearman rank-order correlation coefficient, r_SR=-0.06), which confirms that high rate of reproducibility does not imply true results.

Across all simulations, as the rate of reproducibility increases, scientist populations do not necessarily spend more time on the true model, as indicated by a lack of correlation between rate of reproducibility and time spent at the true model (r_SR=-0.02, Fig. <ref>A). Further, as the rate of reproducibility increases, the discovery of truth slows down rather than speeding up as shown by a positive but small correlation (r_SR=0.26, Fig. <ref>B). Crucially, both of these correlations are driven by the research strategy dominant in the population and should only be taken as evidence for the complexity of these relationships between rate of reproducibility and other desirable properties of scientific discovery (see Table <ref> in S24_Appendix for all correlation coefficients per scientist population). For example, Bo-dominant population reaches almost perfect reproducibility (Fig. <ref>A-C, red) while taking a long time to hit the true model and spending short time at it. On the other hand, Mave-dominant population hits the true model quickly and spends long time there (Fig. <ref>A-C, blue), but it has a much lower rate of reproducibility than Bo-dominant population.




Implications and limitations. This counter-intuitive result on reproducibility is due to violating assumptions of statistical methods. Statistical theory guarantees to find the true data generating mechanism as the best fit for the observed data, if a reliable method operates in the absence of external factors. The research strategies implemented in our ABM include critical external factors that determine how the model space is searched. For Bo-dominant populations, we get high levels of reproducibility and low level of actual discovery for the same reason: Models proposed by Bos consistently result in fitting overly-complex models to data but lead to reproducible inferences since the comparisons are often between models that are 1) not true and 2) far from each other in the sense of statistical distance. As a result, even though the true model is not proposed, and hence not selected as the global model, the method consistently favors the same untrue model when a specific comparison is repeated with independent data.

Mave-dominant populations search the full model space and consequently discover the true model quickly. Their rate of reproducibility is lower relative to Bo-dominant populations. This is because randomly proposed models are typically not true, but also, Maves do not have a biased strategy of proposing models that are far away from the global model. Hence, Mave's comparisons do not always favor a specific model especially when models close to each other are tested.

The effects discussed in this section are marginal main effects of scientist populations over other factors that we vary in our ABM, including levels of noise-to-signal ratio. Due to this marginalization, the mean noise-to-signal ratio that affects the results is higher in our ABM than the system without replication. Nonetheless, our sanity checks still capture the salient properties of scientific discovery well. For example, the basic expectation that a successful scientific endeavor will move us closer to truth is captured. Fig. <ref>C confirms that most scientific populations (with the exception of Bo-dominant) spend considerable time at the true model. In all populations, the true model is sticky. The rate of reproducibility under the true model is higher than the overall rate of reproducibility across all models that become the global model. The scientific community ultimately discovers the true model with varying speed, depending on dominant strategies represented in the population. We also find that the rate of reproducibility is highly positively correlated with stickiness of the true model in most scientist populations except Bo-dominant (Table <ref> in S24_Appendix). The rate of reproducibility is also positively correlated with time spent at true model for most populations, although these correlations are expectedly lower because they are unconditional on the first time to discovery. Speed of discovery has low correlation with the rate of reproducibility. This makes intuitive sense because speed of discovery is largely determined by how the model space is searched (i.e., an external factor) whereas the rate of reproducibility is collectively determined by all variables in the system.

Against this backdrop, we speculate how these counter-intuitive findings might extend to the practice of science beyond our theoretical framework. What we observe appears akin to the tension noted by Shiffrin, Börner, and Stigler <cit.> regarding the risk of obstructing scientific exploration by imposing restrictions on how science should be conducted. Indeed, we show that exploratory strategies (represented by Maves in our system) are needed to speed up scientific discovery. But then we also need scientists testing theory and running replication studies (e.g, Tess and Reys) to establish which discoveries are true. If we restrict exploration to allow only research strategies that search the model space in an extremely biased manner (e.g, Bos), we may lock ourselves in a vicious circle of never making a discovery. The reason is that we may be able to obtain high rate of reproducibility as an artifact of this research strategy. Our cases may be extreme and in reality we might expect diverse scientist populations to emerge naturally. However, past research suggests that if incentive structures reward Bo-like strategies due to high rates of reproducibility they report, these strategies may be widely selected for in scientific populations <cit.> thereby resulting in canonization of false results <cit.>.




  §.§.§ Innovation speeds up scientific discovery
 Mave-dominant population is the fastest to hit the true model (Fig. <ref>C [S25_Figure]S25 Figure A) regardless of the true model complexity and the error variance to model expectation ratio (Fig. <ref>D and Fig. <ref>E). Further, for epistemically diverse population in which all scientist types are equally represented, the proportion of mavericks is sufficient to garner this desirable result. The reason is that Mave provides connectedness in transitioning from model to model via her soft research strategy even when all other scientists represented in the scientific population pursue hard research strategies. All other homogeneous populations take a long time to reach the truth due to pursuing hard research strategies. For example, the estimate for mean first passage time to the true model for Bo-dominant population is 1592.5 steps (Fig. <ref>C). We also ran the ABM with soft research strategies and include the results regarding speed of discovery in Table <ref> (S26_Appendix) as further confirmation that connectedness among models leads to faster discovery in Tess- and Bo-dominant populations, besides Mave-dominant and epistemically diverse populations.

Implications and limitations. The idea that innovative research plays a significant role in scientific discovery is intuitive and hardly new <cit.>. Our results qualify this idea in a particular way: Innovation leads to fast discovery, which is a property determined by the stochastic process governing the connectedness of models. We should note that the memorylessness property of our system might have exaggerated the role of Maves in making a quick discovery. If all scientists carry a tally of past results and adjust their strategies accordingly, it is possible that the model space could be explored more efficiently by scientist types other than Mave. What is needed in essence is not Maves necessarily but a way to guarantee high connectedness among models in the search space and an efficient search algorithm. Arguably the role of innovative, exploratory research is more critical early on in the research cycle and once we are in the vicinity of truth, limited scientific resources might be better spent elsewhere (e.g., confirmatory research or pursuit of other research questions).




  §.§.§ Epistemic diversity optimizes the process of scientific discovery

We looked at which scientific population optimizes across all desirable properties of scientific discovery. Figure <ref>C summarizes the sample median and interquartile range for the time spent at, the stickiness of, and the mean first passage time to the true model, as well as the rate of reproducibility (also see [S25_Figure]S25 Figure). These statistics show the advantage of an epistemically diverse population of scientists on the efficiency of scientific discovery. Homogeneous populations with one dominant research strategy tend to perform poorly in at least one of these desirable properties. For example, Rey-dominant population has low median rate of reproducibility. Mave-dominant population has low median rate of reproducibility and high variability in time spent at the true model. Tess-dominant population has high variability in mean first passage time to the true model and the rate of reproducibility. Bo-dominant population has low median time spent at the true model, low median stickiness, and high variability in mean first passage time to the true model. In contrast to all these examples, epistemically diverse population always performs better than the worst homogeneous population with respect to all properties and further, it has low variability. Thus, epistemic diversity serves as a buffer against weaknesses of each research strategy, consistent with results from the system with no replication. We conclude that among the scientist populations we investigate, epistemic diversity optimizes the properties of scientific discovery that we specified as desirable.


Implications and limitations. We believe that the importance of epistemic diversity is intuitive, yet, it cannot be emphasized enough. Our definition of epistemic diversity is limited to the representation of the four research strategies that we included in our system. In reality, there are numerous philosophical (e.g., logical positivist, post-modernist), research methodological (e.g., empirical experimentation, computer simulations, ethnography), and statistical (e.g., frequentist, likelihoodist, Bayesian) approaches to conducting science and our model is agnostic as to what kind or what degree of epistemic diversity would optimize scientific discovery. We merely find that the role of epistemic diversity in scientific population is akin to diversifying an investment portfolio to reduce risk while trying to optimize returns.




  §.§.§ Methodological choices affect time spent at scientific truth
 The choice of method may appear to be perfunctory if multiple methods perform well. However, violating the assumptions of a method affects the results of an analysis performed with that method. The effects of the model comparison statistic in our system, where a comparison of misspecified models is routinely performed, is not trivial <cit.>. When true model complexity is low, using SC for model comparison increases the time spent at the true model compared to AIC ([S27_Figure]S27 FigureA). As model complexity increases, however, this difference disappears and further, AIC has lower variability. When the ratio of error variance to model expectation is low, SC leads to a longer time spent at the true model. As the ratio of error variance to model expectation increases, AIC and SC spend comparable amount of time at the true model, but AIC has smaller variability ([S27_Figure]S27 FigureB). Averaged over all other parameters, SC spends longer time at the true model than AIC (medians = 27.05% and 19.83%, respectively), but with greater variability (IQR = 66.03% and 33.80%, respectively).


Implications and limitations. The finding that methodological tools might affect scientific progress is factually known <cit.> and being studied extensively by statisticians and meta-scientists alike. Model comparison methods such as AIC and SC as well as all other statistical inference methods work best when their assumptions are met and might lead to invalid inferences under assumption violations. An unsurprising implication of our findings is that statistical theory should inform statistical practice even in the absence of well-known procedural violations such as p-hacking.




§ CONCLUSION


 We studied the process of scientific discovery and reproducibility in a meta-scientific framework using a model-centric approach.
We have chosen a model-centric approach because 1) it translates to scientific models directly, 2) it is a generic mode of inference encompassing hypothesis testing, and 3) model selection methods bypass difficulties associated with classical hypothesis testing.

Our scientists engage in straightforward research strategies and do not commit experimenter bias, learn from their own or others' experiences, engage in hypothesis testing, or commit measurement errors. Further, they are not prone to QRPs or structural incentives. We also assume that there exists a true model that our scientist population attempts to discover and that this true model is within the search space readily available to the scientist population. These factors that we have abstracted away are potential avenues for future research, particularly for complex social dynamics, but our goal here was to explore how the process of scientific discovery works in an idealized framework. We did, however, provide sanity checks to make sure that our system behaves in meaningful ways with respect to what we would expect from a well-functioning scientific process.
Our study shows that even in this idealized framework, the link between reproducibility and the convergence to a scientific truth is not straightforward. A dominant research strategy producing highly reproducible results might select untrue models and steer the scientific community away from the truth. Reproducible false results may also arise due to bias in methods and instruments used, as discussed by Baumgaertner and colleagues <cit.>. While both reproducibility and convergence to a scientific truth are presumably desirable properties of scientific discovery, they are not equivalent concepts.
In our system inequivalence of these concepts is explained by a combination of research strategies, statistical methods, noise-to-signal ratio, and the complexity of truth. This finding further indicates that issues regarding reproducibility or validity of scientific results should not be reduced down to QRPs or structural incentives. Considering such methodological and institutional factors, however, would add additional layers of complication, moving us even further away from the guarantees provided by statistical theory.
Not all our results are as counter-intuitive however. On a positive note, we find that the process of scientific discovery is rendered efficient if the transitions between models in the model space are easy. Scientist populations that expedite transitions via  promoting innovative research or pursuing flexible strategies will discover the truth quickly. In real life, we surmise that the model space might be much larger and the true model–if it exists–might not necessarily be easily accessible in the search space. Therefore, an outstanding challenge for science appears to be to attain a scientific population that can realize optimum connectedness in the model space to expedite the discovery of truth.
Recently, Shiffrin, Börner, and Stigler <cit.> have warned against “one size fits all” type of approaches in science and scientific reforms, advising a nuanced approach instead (p.2638). Complementary to their perspective, our results also advise against homogeneity in scientific practice. We find that a diversity of strategies in the scientific population optimizes across desirable properties of scientific discovery—a finding consistent with the cognitive division of labor literature <cit.>. If populations are largely homogeneous, with one research strategy dominant over others, then the scientific population tends to perform poorly on at least one of the desirable properties which might mean forsaking reproducibility or delaying discovery.
We find that the choice of statistics relative to true model complexity has non-trivial effects on our results. This is corroborated by recent statistical theory <cit.>. The difficulty is that the complexity of the true model is often unknown to scientists who make not only their statistical inference but also their methodological choices under uncertainty. We believe that model complexity may have differential effects on the desirable properties of scientific discovery depending on the choice of statistic.
Our model, as any other model, is an abstraction of reality and we believe that we have captured salient features of the scientific process of interest to our research questions. Main limitations of our framework are the lack of capacity to learn and memorylessness of scientists. The replicator only provides meta-level information about the scientific process and does not contribute directly to the accumulation of scientific knowledge. Incorporating past reproducibility of specific results in decision making strategies might allow the replicator to make substantial contributions to scientific discovery. A realistic implementation of this aspect requires our virtual scientists to adopt machine learning algorithms that can heuristically teach them to become intelligent agents.

Our research also raises questions with regard to reproducibility of scientific results. If reproducibility can be uncorrelated with other possibly desirable properties of scientific discovery, optimizing the scientific process for reproducibility might present trade-offs against other desirable properties. How should scientists resolve such trade-offs? What outcomes should scientists aim for to facilitate an efficient and proficient scientific process? We leave such considerations for future work.





§ SUPPORTING INFORMATION





















  
*S1 Appendix.

A stochastic process of scientific discovery.
Consider an infinite population of scientists conducting a sequence of idealized experiments ξ^(t):=(M_P^(t),θ, D^(t),S,K^(t)), indexed by time t=1,2,⋯ where M_P^(t) belongs to a set of probability structures ={M_1,M_2,⋯,M_L} known to all scientists. Further, assume that there are A distinct scientist types in the population, each with a well-defined research strategy R∈={R_o,R_1,⋯,R_A} of proposing a model in their experiment. These strategies depend on the type of scientist and a global model M_G^(t)∈, K^(t), which represents the consensus of the scientist population at time t. The population of scientists aims to find the true model M_T ∈. A scientist selected to conduct an experiment at time t, uses her background knowledge K^(t) to propose a new candidate model M_P^(t). Specifically, we define K^(t) as a probability distribution (M_P|R^(t),M_G^(t)), where {M_P,M_G^(t)}∈^2, and R^(t)∈.

The initial conditions of our stochastic process include the true model M_T, true parameter values θ_T of M_T, an initial global model M_G^(0), a method for model selection S, and the sample size of the data n. At each time step, an idealized experiment ξ^(t) is performed and new data D^(t) of size n is generated independent of everything else from distribution M_T(θ_T). Each experiment is performed by a scientist randomly selected from A types in the population using the categorical distribution with probabilities (p_1,p_2,⋯,p_A). The selected scientist proposes a model M_P^(t) with probability (M_P|R^(t),M_G^(t)) conditional on a research strategy fully specified by her type and the current global model. Given the data D^(t), model scores under the proposed model and the current global model are calculated as S(M_P^(t)) and S(M_G^(t)), respectively. The model with favorable score (i.e., smaller for both AIC and SC) is set as the new global model M_G^(t+1). This mechanism represents how scientific consensus is updated in light of new evidence.

A defining property of our stochastic process with no replication is that K^(t) depends only on quantities at time t. If R_a∈ℛ depends only on M_G^(t) for all a, the transition from M_G^(t) to M_G^(t+1) admits the Markov property and the stochastic process representing the scientific process is a Markov chain with transition probabilities given by


    (M_G^(t+1)=M_ℓ|M_G^(t)=M_i)=
    ∑_a=1^A(S(M_ℓ)<S(M_i))(M_ℓ|R_a,M_i)(R_a).


On the right hand side of Eq. (<ref>), the last term is the probability of selecting a scientist with research strategy R_a independent of all else, the middle term is the probability of proposing the model M_ℓ given the current global model M_i and the scientist type a with research strategy R_a selected. The probability (S(M_ℓ)<S(M_i)) depends on M_T via D^(t) generated and it is obtained by


∫_Θ∫_𝒟(S(M_ℓ)<S(M_i)|D)(D|θ)(θ)dDdθ,


where (θ) is the probability of parameter, (D|θ) is the likelihood of the data, and (S(M_ℓ)<S(M_i)|D) is the probability that the proposed model M_ℓ has a more favorable score than M_i conditional on data. By convention we set (S(M_ℓ)<S(M)_i)=1 when ℓ=i, and (M_ℓ|R_a,M_i)>0
for all a,i,ℓ so that transition probabilities are nonzero for all models and scientist types. This second condition guarantees that our Markov chain is ergodic, which implies that it has a unique stationary distribution—its limiting distribution for visiting a model.


When there are no replication experiments in the system, K^(t) is defined as (M_P|R^(t),M_G^(t)) which states that conditional on R^(t) and M_G^(t), the probability of proposing a model is independent of the past time steps. Let R_o∈ℛ be the replicator strategy. Given the proposed and global models at time t-1, the replicator strategy at time t, R_o^(t), is to perform an experiment at time t, using the exact same proposed and global models as those at time t-1, but with new data D^(t) generated under M_T(θ_T). Since R_o∈ R depends on M_G^(t-1), the transition from M_G^(t) to M_G^(t+1) does not admit the Markov property anymore and the stochastic process representing the scientific process is a higher order Markov chain. The transition probabilities of the Markov chain at time t can be expressed by conditioning on whether a scientist chosen at a given time is a replicator:


    (R^(t)≠ R_o)(M_G^(t+1)|M_G^(t))+
          (R^(t)= R_o)[(R^(t-1)≠ R_o)(M_G^(t+1)|M_G^(t),
    M_G^(t-1))+⋯+
          (R^(1)= R_o)[(R^(0)≠ R_o)(M_G^(t+1)|M_G^(t), M_G^(t-1),⋯,M_G^(0))]⋯].


In Eq. (<ref>), the first term in the sum is the joint probability of choosing a scientist who is not a replicator at time t and the transition probability from global model at time t to global model at time t+1. Since the scientists are chosen independently of all else, the joint probability is written as the product of choosing a scientist who is not a replicator at time t, given by (R^(t)≠ R_o), and the probability of transition to the global model at time t+1 is given by Eq. (<ref>). The second term in the sum is the joint probability of choosing a scientist who is a replicator at time t and the transition probabilities to a model. We write the second term as the product of (R^(t)=R_o) and the transition probabilities when a replicator is chosen. If the scientist at time t is a replicator, she replicates the experiment at time step t-1, which might be a replication experiment itself. Therefore, the transition probabilities to a model within the first brackets is a sum of two probabilities. The first term is the joint probability of choosing a scientist who is not a replicator and the transition probability in that case, and the second term is the probability of choosing a replicator given by (R^(t-1)=R_o) at time step t-1, and the transition probability in that case. This is a recursive equation, in the sense that the transition probabilities at time t depend on the transition probabilities at time t-1. An implication is that the transition probabilities at time t are path dependent. Therefore, when a replicator scientist is included in the population, we have a higher order Markov chain, whose long term dynamics are feasible to obtain with a forward simulation method.

For the process with replicator, we lift the assumption (M_ℓ|R_a,M_i)>0 for all a,i,ℓ that we imposed in the process without a replicator. This assumption increases the connectivity of the transition probability matrix, which makes calculations in the long-term behavior of the Markov chain straightforward. Due to our new process not admitting the Markov property, these calculations are irrelevant in the analysis of the process with a replicator. Therefore, we drop the assumption of transitioning from a model to any other model to be nonzero. Removing this assumption allows us to define scientist types that visit only the subset of all models consistent with a specific research strategy. This property of the process renders the effects of each research strategy on the process outcomes well-pronounced.



  
*S2 Appendix.
 
Description of example system of linear models.
We define , the family of linear models as


    {𝐲=𝐗β+ϵ|β_i⋯ jv≠ 0 ⇒ (β_i≠0,⋯,β_v≠0, ⋯, β_i⋯ j≠0)}.


Here, 𝐲 is n × 1 vector of response variables, 𝐗 is n × p matrix of predictor variables, β is p × 1 vector of model parameters, and ϵ is n × 1 vector of random errors satisfying the Gauss-Markov conditions (ϵ_i)=0, Var(ϵ_i)=σ^2, and Cov(ϵ_i,ϵ_j)=0 for all i,j.  contains L linear models with up to k factors subject to the constraint β_i⋯ jv≠ 0 ⇒ (β_i≠0, ⋯, β_i⋯ j≠0) which guarantees that if a model contains a v–way (v≤ k) interaction term of v factors in the model, then all (v-1)–way, (v-2)–way, ⋯, 2–way interactions of those factors and their main effects are included in the model. We include the predictor x_1 and the response variable y in all models reflecting our assumption that all scientists in the community focus on a research question that involves at least one common factor of interest and a common response variable.





Let M_i ∈ be a model with p_i parameters of which v_ℓ_i is the highest order interaction term with order ℓ_i denoting the order, #v_ℓ_i is the cardinality of v_ℓ_i. Let M_i ≻ M_j denote that M_i is more complex than M_j.  We define the model complexity as a partial ordering obeying three conditions:


  * If p_i>p_j then M_i≻ M_j.

  * If p_i=p_j and ℓ_i>ℓ_j, then M_i≻ M_j.

  * If p_i=p_j, ℓ_i=ℓ_j, and #v_ℓ_i>#v_ℓ_j, then M_i≻ M_j.

Otherwise, we say that complexities of M_i and M_j are indistinguishable.




  
* S3 Algorithm.
 
Agent-based model algorithm.
We first present the general algorithm in the agent-based model and then discuss specific values used in the article.
 
 

We choose M_G^(0) randomly with equal probability from models in . Θ determines θ_min, θ_max and θ_T is chosen uniformly randomly on this interval. The parameters of the categorical distribution used in step 4 is determined by the proportion of scientists in the population.



  
*S4 Appendix.
 
 Properties of scientific discovery.


How quickly does scientific community discover the true model? When there are no replicators in the model, 𝐏={(M_ℓ|M_i)} is the transition probability matrix given by Eq. (<ref>). We assess the speed with which scientific community discovers the true model by the mean first passage (or hit) time to the true model. This is the first time to reach M_T, given that we start the system from M_i. The mean first passage time to the true model which we denote by τ_i,T, is the expected value of this first passage time to select the true model as the global model for the first time in the process, given that the process starts from a known model M_i. By theory of Markov chains we have


    τ_i,T=1+∑_ℓ=1
    M_ℓ≠ M_T^L(M_ℓ|M_i)τ_ℓ,T,  i=1,2,⋯, L.


Given the transition probabilities (M_ℓ|M_i), the solution to this system of L linear equations is readily obtained by a method to solve linear equations.

When there are replicators in the model, we assess the speed to discover the true model by the mean number of steps for the system to update the global model to the true model for the first time. The first the time true model is global model is the mean first passage time to the true model described in the process of scientific discovery with no replication, but it is unconditional on the starting model.

How “sticky” is the true model as global model? We define stickiness of a model as the mean—over proposed models—probability of staying in model M_i conditional on the current global model M_i. When there are no replicators in the model, by theory of Markov chains, it is given by


    (M_G^(t+1)=M_i|M_G^(t)=M_i)=∑_a=1^A∑_ℓ=1^L(S(M_i)<S(M_ℓ))
    (M_ℓ|R_a,M_i)(R_a),


where (R_a) and (M_ℓ|R_a,M_i) are same as in Eq. (<ref>), and (S(M_i)<S(M_ℓ)) is the probability that the current global model M_i has a more favorable score than the proposed model M_ℓ. The right hand side of Eq. (<ref>) can be calculated for all L models given M_T. This probability captures the tendency of scientific community to stay at a visited model. Calculated for M_G^(t+1)=M_G^(t)=M_T, it gives the stickiness of the true model.

When there are replicators in the model, we measure the tendency of the scientific community to maintain consensus on the true model by the proportion of time that true model remains as global model given that it was already the global model. Stickiness is the complement of the probability that a global model switches from true model to another model.


How long does scientific community stay on the true model?
By theory of ergodic Markov chains, the probability that each model is selected as the global model converges to a constant value independent of the current global model M_G^(t). The limiting probabilities of a model being selected as a global model reflect the long-term behavior of the scientific community and the proportion of time spent on each model. These limiting probabilities are given by lim_t→∞𝐏^t.

When there are replicators in the model, we assess the mean time the scientific community spends on a model (and in particular the true model) as a consensus by the proportion of times that a model is selected as the global model. We use the proportion of times true model is selected as global model as a proxy for the limiting probabilities of selecting the true model as the global model in the process of scientific discovery with no replication.

How reproducible are the results of experiments? Under ABM, we define the rate of reproducibility as the expectation of Bernoulli distributed random variable 𝐈_{M_G^(t+1)=M_G^(t)|R_o^(t)},

which is the indicator function that takes the value 1 if the global model at time t+1 is the same as the global model at t given that a replication experiment is performed at time t. The parameter of the Bernoulli distribution is the probability of reproducibility
(M_G^(t+1)=M_G^(t)|R_o^(t)). Further, we define the rate of reproducibility when the true model is the global model by
𝐈_{M_G^(t+1)=M_G^(t)|R_o^(t),M_G^(t)=M_T} which also determines the rate of reproducibility when the true model is not the global model
𝐈_{M_G^(t+1)=M_G^(t)|R_o^(t),M_G^(t)≠ M_T}. We estimate these rates of reproducibility by Monte Carlo integration of ABM simulations.


  §.§.§ Monte Carlo estimates of rates of reproducibility


The random variable 𝐈_{M_G^(t+1)=M_G^(t)|R_o^(t)}
which takes the value 1 if the global model at time t+1 is equal to global model at time t given that at time t we have chosen a replicator, is a Bernoulli distributed random variable. Its mean is given by
(𝐈_{M_G^(t+1)=M_G^(t)|R_o^(t)}) whose Monte Carlo estimate is given by


    (𝐈_{M_G^(t+1)=M_G^(t)|R_o^(t)})=1/V∑_v=1^V
    𝐈_{M_G^(t+1)=M_G^(t)|R_o_v^(t)},

where R_o_v^(t) is the vth instance a replicator is chosen. The rate of reproducibility when the true model is global model is estimated by

    (𝐈_{M_G^(t+1)=M_G^(t)|R_0^(t),M_G^(t)=M_T})=1/V_T∑_v=1^V_T𝐈_{M_G^(t+1)=M_G^(t)|R_0_v^(t),M_G^(t)=M_T}


which also implies

    (𝐈_{M_G^(t+1)=M_G^(t)|R_0^(t),M_G^(t)≠ M_T})=1/V_N∑_v=1^V_N𝐈_{M_G^(t+1)=M_G^(t)|R_0_v^(t),M_G^(t)≠ M_T}.

Here, V_T is the number of times a replicator is selected when the true model is global model and V_N is the number of times a replicator is selected when the true model is not global model, and V_T+V_N=V.




  
*S5 Appendix.

Predictor variables, error variance, and sample size.
We uniformly randomly generate the value of the jth factor at the ith level x_ij on the set {1,2,⋯, 100} for all i and j. We fix the sample size n=100 and calibrate the ratio of the error variance σ^2 to expected value of the model at the mean value of the predictors (y|μ_x), where μ_x=(x). We use a standardized linear regression model so that if the ith observed response is y_i^*, then we set y_i=(y_i^*-y̅^*)/s^*, where y̅^* is the sample mean and s^* is the sample standard deviation of y_i^*, i=1,2,⋯, n. Implementing standardized linear regression model allows us to precisely specify σ^2:(y|μ_x). We fix three levels for σ^2:(y|μ_x). (1:4), (1:1), and (4:1).
We set the true regression coefficients to be Dirichlet distributed with unit parameters or equivalently, uniformly distributed on the interval (0,1) with the constraint that they sum to 1. Thus, all regression coefficients have the same mean effect size. We set the correlation between the first factor x_1 and other factors to a small value of 0.2 to avoid orthogonality between predictors which is an idealized case that we believe rarely achievable in practice. We performed additional analyses with higher correlation between predictor variables and found that correlation between predictors does not affect the results in our system unless it is extremely high and causes multicollinearity.

Model selection criteria.
SC is defined by plog(n)-log(D|θ̂,M), where p is the number of model parameters, n is the sample size, and θ̂ is the maximum likelihood estimate of model parameters under M. Akaike's Information Criterion (AIC) is defined by 2p-log(D|θ̂,M). For both SC and AIC, a smaller value indicates a better model performance than a larger value. The maximized likelihood log(D|θ̂,M) rewards model fit equally in SC and AIC. The first term in these formulas penalize the model complexity, with SC penalizing complex models more heavily for n≥8 (with log(n)) than AIC as the sample size increases. SC has the desirable property that when the true model generating the data is in the universe of candidate models, it selects the true model with probability 1 as n→∞, in other words it is consistent. For model M fitted to D, we calculate the model selection criteria SC and AIC in a computationally efficient way.

Description of scientists and scientist populations.
We let ={R_Rey, R_Tess, R_Mave, R_Bo} and define the transition probabilities for these strategies as follows. For R_Rey,


    (M_P^(t)=M_j| M_P^(t-1)=M_i,M_G^(t)=M_G^(t-1)) =
      
        1           if  i=j,
    
        0           otherwise.


For R_Tess,


    (M_P^(t)=M_i|M_G^(t)) =
      
        1/(m+1)           if  M_i  is one of  m  models
         one term away from  M_G^(t),
    
        m_o      otherwise.

If R_Rey∈  then m_o=0 and  m_o=1/[(L-m)(m+1)] otherwise.

For R_Mave, (M_P^(t)=M_i|M_G^(t)) =1/L, 
i=1,2,⋯,L. For R_Bo,


    (M_P^(t)=M_i|M_G^(t)) =
      
        1/(m+1)           if  M_i  is one of  m  models one
         interaction larger than  M_G^(t),
    
        m_o      otherwise.


If R_Rey∈  then m_o=0 and  m_o=1/[(L-m)(m+1)] otherwise.





  
*S7 Code and Data.

The code to perform the simulations and analyze the data generated in this project, and a summary data set are included as a Git repository at <https://github.com/gnardin/CRUST>. Refer to the description in the main page of this repository for further instructions and details of the implementation.




  
*S8 Appendix.

Monte Carlo Estimates of Model Comparisons.
Let 𝐲 denote n × 1 vector of responses generated by the true model and 𝐗 be n × p matrix of predictors, where p is the number of parameters in the fitted model. Under the Gauss-Markov assumptions (ϵ_i)=0,  Var(ϵ_i)=σ^2,  Cov(ϵ_i,ϵ_j) for all i,j we denote the vector of joint maximum likelihood estimates for the regression coefficients by β̂. By definition of Schwarz Criterion


    S(M)=2plog(n)-2log(|,β̂).

If Akaike's Information Criterion is used, 2p replaces 2plog(n).

The loglikelihood in the second term is equal to n times the log of the residual sums of squares and it can be written as


    log(|,β̂)=nlog( ^' -^'(^')^-1^')+C,


where C is a term dependent only on M_T. For transition probabilities, we are interested in (S(M_i)<S(M_j)). We have


    S(M_i)-S(M_j)   =   (p_i-p_j)log(n)+nlog(^'_i(_i^'_i)^-1_i^')
          -nlog(^'_j (_j^'_j)^-1_j^'),


where subscripts i,j now denote quantities that depend on model M_i and M_j.

The random variable 𝐈_{S(M_i)-S(M_j)<0|M_T} is Bernoulli distributed with probability of success (S(M_i)-S(M_j)<0) which is equal to its expectation (𝐈_{S(M_i)-S(M_j)<0|M_T}) whose Monte Carlo estimate is given by


    (𝐈_{S(M_i)-S(M_j)<0|M_T})=1/V∑_v=1^V
    𝐈_{S(M_i)-S(M_j)<0|_v}.


An estimate of (S(M_i)-S(M_j)<0) is obtained using Eq. (<ref>) conditional on true model M_T and its predictors X_T as follows. First, generate the set of k predictor variables and build _i and _j for M_i and M_j respectively. Then generate β_T_v, v=1,2,⋯, V independently of each other. Finally, simulate _v|_T,β_T_v from the normal distribution with expected value (_v)=_Tβ_T_v and variance σ^2.  Each realization (_1,_2,⋯,_v) is used in Eq. (<ref>) to assess S(M_i)-S(M_j)<0 and the estimate is obtained using the mean in Eq. (<ref>).


































  
S23 Appendix.
 Reproducibility does not imply discovery of truth.

The rate of reproducibility is the probability of the global model staying the same model after a replication experiment.
In this section, we show that the rate of reproducibility has no effect on other desirable properties of scientific discovery including: the probability that a model is selected as the global model in the long run, the mean first time to hit a model, and stickiness of a model.
The key observation for the result is that all of these desirable properties of scientific discovery are properties of the stochastic process of scientific discovery defined by the Markov chain. A Markov chain is characterized solely by its transition probability matrix. Below, we show that the rate of reproducibility, does not affect the transition probability matrix in our model. Therefore, any property of the Markov chain is not affected by the rate of reproducibility. To keep the notation tractable and without loss of generality we assume that Rey, the replicator is not chosen consecutively in the process. Generalization to the case where Rey is chosen consecutively is by induction and using Eq. (<ref>).

When Rey, the replicator, is in the system, the model is a second order Markov chain, which has transitions specified over two time steps. We let the triplet of indices (i,j,ℓ) to be associated with a transition in this second order Markov chain, where i is the beginning state and ℓ is the final transition state through state j. We let R_Rey to be the replicator strategy and R_Rey^' to be any non-replicator strategy. P(R_Rey) = 1-P(R_Rey^') is the probability that the second step is a replication experiment. The probability that the global model transitions to M_ℓ in the second step given that M_i was the beginning global model can be written as


    P(R_Rey^')∑_j=1^Lp_ijℓ+P(R_Rey)∑_j=1^Lq_ijℓ,


where p_ijℓ and q_ijℓ are transition probabilities when a replicator is not selected and is selected, respectively. The first term in Eq. (<ref>) does not involve a replicator so to establish whether the rate of reproducibility affects transition probabilities it is sufficient to focus on the second term.

There are only two nonzero terms in ∑_j=1^Lq_ijℓ. The first term is

    q_iiℓ=P(S(M_ℓ)<S(M_i))P(S(M_ℓ)<S(M_i))P(M_ℓ|M_i),

where on the right hand side: M_ℓ is proposed with probability P(M_ℓ|M_i) and wins against M_i with probability P(S(M_ℓ)<S(M_i))
at the first step, and in the replication (second step) it wins again with probability P(S(M_ℓ)<S(M_i)) to stay as the global model.

The second term nonzero term is

    q_iℓℓ=P(S(M_ℓ)<S(M_i))P(S(M_ℓ)>S(M_i))P(M_ℓ|M_i),

where on the right hand side: M_ℓ is proposed with probability P(M_ℓ|M_i) and loses against M_i with probability P(S(M_ℓ)>S(M_i)) at the first step, and in the replication (second step) it wins against M_i with probability P(S(M_ℓ)<S(M_i)) to become the global model. Here, q_iiℓ and q_iℓℓ are the probabilities of reproducing and not reproducing a result. Further, only their sum contributes to the Markov transition probability matrix (Eq. (<ref>)). We write

    ∑_j=1^Lq_ijℓ   =q_iiℓ+q_iℓℓ
       =P(S(M_ℓ)<S(M_i))P(S(M_ℓ)<S(M_i))P(M_ℓ|M_i)
       +P(S(M_ℓ)<S(M_i))P(S(M_ℓ)>S(M_i))P(M_ℓ|M_i)
       =P(S(M_ℓ)<S(M_i))P(M_ℓ|M_i)[P(S(M_ℓ)<S(M_i))+P(S(M_ℓ)>S(M_i))],

where the first and second term within brackets determine whether a result is reproduced or not reproduced, respectively, and sum to 1. Therefore we have


    ∑_j=1^Lq_ijℓ
    =P(S(M_ℓ)<S(M_i))P(M_ℓ|M_i),


which shows that whether a result is reproduced has no bearing on the calculation of transition probability from M_i to M_ℓ. Since the transition probability matrix characterizes a Markov chain, the rate of reproducibility cannot affect other desirable properties of scientific discovery including: the probability that a model is selected as the global model in the long run, the mean first time to hit a model, and the stickiness of a model.

On the other hand, we do not claim that there cannot be correlation between desirable properties of scientific discovery and the rate of reproducibility. In fact, whether there is correlation depends on the strategies and their frequency in the population, and this also can be seen from our mathematical result. For transition to M_ℓ the rate of reproducibility is proportional to ∑_i≠ℓP(S(M_ℓ)<S(M_i)) and the transition probability to M_ℓ is proportional to P(S(M_ℓ)<S(M_i)). If the effect of the first term P(R_Rey^')∑_j=1^Lp_ijℓ in Eq. (<ref>) and the second term P(M_ℓ|M_i) in Eq. <ref> are small then there might be considerable correlation between desirable properties of scientific discovery and the rate of reproducibility. This relationship depends on a number of factors including the exact strategies of scientists in the population and the frequency of these scientists, and is very complex.



  
*S24 Appendix. 
Correlations per scientist population.








  
*S26 Appendix.
ABM with soft research strategies.



Mean first passage time to true model is faster for Tess- and Bo-dominant populations when scientists in the ABM simulations pursue soft (vs. hard) research strategies, consistent with results from Markov chain without replicator. The median of the mean first passage time for Mave-dominant population is the same as the median we reported for ABM with hard research strategies because the maverick strategy has a well connected transition matrix, leading to quick discovery of the true model. Further, the representation of Maves in the epistemically diverse population makes this population also well connected, in ABM with both hard and soft research strategies. Rey-dominant populations are the slowest to discover the truth because when replicators comprise 99% of the population, there are not many new models being proposed and same experiments are repeated many times slowing down the discovery of the true model.















§ ACKNOWLEDGMENTS

This research made use of the resources of the High Performance Computing Center at Idaho National Laboratory, which is supported by the Office of Nuclear Energy of the US DoE under Contract No. DE-AC07-05ID14517. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. 














10

Ramsey1931
Ramsey FP.
Truth and probability (1926).
In: The Foundations of Mathematics and other Logical Essays. London:
  Routledge and Kegan Paul Ltd; 1931. p. 156–198.

Popper1959
Popper KR.
The logic of scientific discovery.
London: Hutchinson & Co.; 1959.

Kyburg1964
Kyburg HE, Smokier HE, editors.
Studies in subjective probability.
New York, NY: Wiley; 1964.

Schmidt2009
Schmidt S.
Shall we really do it again? The powerful concept of replication is
  neglected in the social sciences.
Review of General Psychology. 2009;13(2):90–100.
doi:10.1037/a0015108.

Ioannidis2005
Ioannidis JPA.
Why most published research findings are false.
PLOS Medicine. 2005;2(8):e124.
doi:10.1371/journal.pmed.0020124.

McElreath2015
McElreath R, Smaldino PE.
Replication, communication, and the population dynamics of scientific
  discovery.
PLOS ONE. 2015;10(8):e0136088.
doi:10.1371/journal.pone.0136088.

Smaldino2016
Smaldino PE, McElreath R.
The natural selection of bad science.
Royal Society Open Science. 2016;3(9):160384.
doi:10.1098/rsos.160384.

Higginson2016
Higginson AD, Munafò MR.
Current incentives for scientists lead to underpowered studies with
  erroneous conclusions.
PLOS Biology. 2016;14(11):e2000995.
doi:10.1371/journal.pbio.2000995.

Nissen2016
Nissen SB, Magidson T, Gross K, Bergstrom CT.
Publication bias and the canonization of false facts.
eLife. 2016;5:e21451.
doi:10.7554/elife.21451.

Gelman2017
Gelman A, Carlin J.
Some natural solutions to the p-value communication problem—and why
  they won't work.
Journal of the American Statistical Association.
  2017;112(519):899–901.
doi:10.1080/01621459.2017.1311263.

Bensonetal2017
Benson AR, Gleich DF, Lim LH.
The spacey random walk: A stochastic process for higher-order data.
SIAM Review. 2017;59(2):321–345.
doi:10.1137/16m1074023.

Epstein2006
Epstein JM.
Generative social science: Studies in agent-based computational
  modeling.
Princeton University Press; 2006.

Gilbert2008
Gilbert N.
Agent-based models. vol. 153.
SAGE Publications, Inc; 2008.

Weisberg2009
Weisberg M, Muldoon R.
Epistemic landscapes and the division of cognitive labor.
Philosophy of Science. 2009;76(2):225–252.
doi:10.1086/644786.

Muldoon2013
Muldoon R.
Diversity and the division of cognitive labor.
Philosophy Compass. 2013;8(2):117–125.
doi:10.1111/phc3.12000.

Alexander2015
Alexander JM, Himmelreich J, Thompson C.
Epistemic landscapes, optimal search, and the division of cognitive
  labor.
Philosophy of Science. 2015;82(3):424–453.
doi:10.1086/681766.

Schwarz1978
Schwarz G.
Estimating the dimension of a model.
The Annals of Statistics. 1978;6(2):461–464.

Baumgaertner2018
Baumgaertner B, Devezer B, Buzbas EO, Nardin LG.
A model-centric analysis of openness, replication, and reproducibility,
ArXiv e-prints. 2018;Nov:arXiv:1811.04525.

Lindley2000
Lindley DV.
Philosophy of Statistics.
Journal of the Royal Statistical Society: Series D (The Statistician). 2000;49(3):293–337.

Akaike1973
Akaike H.
Information theory and an extension of the maximum likelihood
  principle.
In: Petrov BN, Csaki F, editors. Proceedings of the 2nd International
  Symposium on Information Theory. Budapest: Akademiai Kiado; 1973. p.
  267–281.

Akaike1974
Akaike H.
A new look at the statistical model identification.
IEEE Transactions on Automatic Control. 1974;19(6):716–723.
doi:10.1109/tac.1974.1100705.

Mule2017
Gonzalez-Mulé E, Aguinis H.
Advancing theory by assessing boundary conditions with
  metaregression: A critical review and best-practice recommendations.
Journal of Management. 2017;44(6):2246–2273.
doi:10.1177/0149206317710723.

Whetten1989
Whetten DA.
What constitutes a theoretical contribution?
Academy of Management Review. 1989;14(4):490–495.

Lv2013
Lv J, Liu JS.
Model selection principles in misspecified models.
Journal of the Royal Statistical Society: Series B (Statistical
  Methodology). 2013;76(1):141–167.
doi:10.1111/rssb.12023.

Shiffrin2018
Shiffrin RM, Börner K, Stigler SM.
Scientific progress despite irreproducibility: A seeming paradox.
Proceedings of the National Academy of Sciences.
  2018;115(11):2632–2639.
doi:10.1073/pnas.1711786114.

Zollman2009
Zollman KJS.
The epistemic benefit of transient diversity.
Erkenntnis. 2009;72(1):17–35.
doi:10.1007/s10670-009-9194-6.

Muthukrishna2019
Muthukrishna M, Henrich J.
A problem in theory.
Nature Human Behaviour. 2019;
https://doi.org/10.1038/s41562-018-0522-1

Box1976
Box GEP.
Science and statistics.
Journal of the American Statistical Association. 1976;71(356):791–799.

Nelder1986
Nelder JA.
Statistics, science and technology.
Journal of the Royal Statistical Society: Series A (General). 1986;149(2):109–121.




