An analysis of the effects of sharing research data, code, and preprints on citations
Giovanni Colavizza Lauren Cadwallader Marcel LaFlamme Grégory Dozot Stéphane Lecorney Daniel Rappo Iain Hrynaszkiewicz

[
    [
    25 April 2024
=================



Calls to make scientific research more open have gained traction with a range of societal stakeholders. Open Science practices include but are not limited to the early sharing of results via preprints and openly sharing outputs such as data and code to make research more reproducible and extensible. Existing evidence shows that adopting Open Science practices has effects in several domains. In this study, we investigate whether adopting one or more Open Science practices leads to significantly higher citations for an associated publication, which is one form of academic impact. We use a novel dataset known as Open Science Indicators, produced by PLOS and DataSeer, which includes all PLOS publications from 2018 to 2023 as well as a comparison group sampled from the PMC Open Access Subset. In total, we analyze circa 122’000 publications. We calculate publication and author-level citation indicators and use a broad set of control variables to isolate the effect of Open Science Indicators on received citations. We show that Open Science practices are adopted to different degrees across scientific disciplines. We find that the early release of a publication as a preprint correlates with a significant positive citation advantage of about 20.2% (± .7) on average. We also find that sharing data in an online repository correlates with a smaller yet still positive citation advantage of 4.3% (± .8) on average. However, we do not find a significant citation advantage for sharing code. Further research is needed on additional or alternative measures of impact beyond citations. Our results are likely to be of interest to researchers, as well as publishers, research funders, and policymakers.









§ INTRODUCTION


Arising from a diverse set of cultural and technological projects at the turn of the twenty-first century <cit.>, contemporary calls to make scientific research more open point toward a no less diverse range of outcomes. One influential definition characterizes Open Science as “transparent and accessible knowledge that is shared and developed through collaborative networks” <cit.>, encompassing knowledge objects or outputs as well as processes <cit.>. Another developed by UNESCO defines Open Science as “an inclusive construct that combines various movements and practices aiming to make multilingual scientific knowledge openly available, accessible and reusable for everyone, to increase scientific collaborations and sharing of information for the benefits of science and society, and to open the processes of scientific knowledge creation, evaluation and communication to societal actors beyond the traditional scientific community” <cit.>. 

While acknowledging this diversity of ambitions, in what follows we focus on practices resulting in what UNESCO terms “open scientific knowledge” <cit.>: that is, the making of scientific publications and the materials that underpin them available to all, free of charge. These Open Science practices include but are not limited to Open Access publication; the early sharing of results, for example via the use of preprints; openly sharing outputs such as data, code, and protocols to make research more reproducible and extensible; and fostering rigor and transparency in study design, for example via study registration. While the uptake of these practices by researchers varies by field, career stage, and region, their prevalence is growing overall <cit.>. Drivers of this growth include publisher and funder policies, training and infrastructure support, and cultural change <cit.>. The proliferation of policies for Open Science has led to a greater need to monitor the effects of these policies on Open Science <cit.>, although comprehensive solutions for measuring Open Science are lacking. Still, researchers, technology providers, research funders, institutions and publishers have begun to monitor the prevalence of Open Science practices (<https://open-science-monitoring.org/monitors/>). These efforts provide new evidence and data sources from which to understand if and how Open Science practices are being adopted, and to explore the extent to which these practices confer effects, impacts or benefits as a result of their adoption.

In this article, we contribute to an emerging strand of research assessing the impact of Open Science practices. We focus on a set of measurable Open Science practices that include data sharing, code sharing, and preprint posting. More specifically, we ask whether adopting any combination of these practices leads to a significantly higher citation impact for an associated publication when compared to similar publications for which authors have not adopted Open Science practices. We answer this question by leveraging a novel dataset known as Open Science Indicators, which is produced by the nonprofit Open Access publisher PLOS in partnership with DataSeer (<https://dataseer.ai>) <cit.>, and by adapting a previously released workflow to mine citation data from the PMC Open Access Subset <cit.>. An important aspect of our contribution is the assessment of Open Science practices in combination, rather than individually as is usually the case in previous work.





Open Science has been defined in a variety of ways. For example, some emphasize the movement behind Open Science as well as the outcomes it aims to create. An influential definition states that “Open Science is transparent and accessible knowledge that is shared and developed through collaborative networks” <cit.>. More generally, Open Science must be considered as a “broad term that refers to many different concepts, ranging from scientific philosophies and cultural norms, such as the ownership of scientific methods and the principle that scientific output should be evaluated on its merit, to actual specific practices that operationalise such norms” <cit.>. While acknowledging this diversity of complementary viewpoints, here we take up the perspective of Open Science as a set of practices. Open Science practices may include removing pay-walls to scientific results, for example via open access and preprints; openly sharing scientific outcomes such as data and code to make research more easily reproducible and replicable; fostering scientific rigour for example via study pre-registration. Such practices have been widely supported across fields of research <cit.>, and stem from two broad objectives of Open Science: facilitating access to scientific knowledge to accelerate and democratise research, and increasing its rigour by making the research procedures transparent and inspectable <cit.>. 

There is increasing evidence that adopting Open Science practices has a positive impact on several dimensions. First of all research visibility, as measured for example by received citations and media attention <cit.>. Open Science has been instrumental in accelerating progress on certain scientific findings <cit.>, in making results more transparent <cit.>, and in opening up the Pandora box of the replication crisis in certain fields of research <cit.>. Furthermore, open access has seen a rapid, wider adoption <cit.>, with beneficial effects for the open knowledge commons <cit.>. Despite these encouraging results and the broad support from all stakeholders and the public, Open Science has also been criticised for not necessarily meeting its ambitious goals <cit.>.



 §.§ Code and Data Sharing

Several studies converge on the importance of data (and code) sharing for scientific advancement but diverge to some extent in their findings on current practices and their impact. Some work has gone into assessing the ethical and practical aspects of data sharing in specific domains, from genomics <cit.> to the humanities <cit.>, emphasizing the importance of balancing research-added value with participant privacy and required effort. Overall, data sharing is seen as important to facilitate re-use and increase the visibility of data-driven research. Evidence shows that the novel combination of datasets leads to higher impact and visibility <cit.>. The pros and cons of data sharing have been assessed via numerous surveys. Work on code sharing in computational biology found that while researchers value code sharing for understanding articles, they are hesitant to incur additional costs associated with sharing tools <cit.>. At the same time, and despite a strong data-sharing policy by PLOS, adoption of best practices in data sharing by researchers – via data repositories – remains limited, indicating satisfaction with current data-sharing abilities and a reluctance to seek new solutions <cit.>. Nevertheless, data sharing in data repositories seems to correlate positively with citation impact, as some results show <cit.>. Crucially, data-sharing practices vary significantly across disciplines, and more journals may need to enforce data-sharing policies at manuscript acceptance to improve rates of data sharing when data supporting publications are available “upon [reasonable] request” <cit.>.



 §.§ Preprints

The emerging role of preprints in scientific communication is instead explored more systematically through various studies. Preprint adoption has seen a rapid increase over time (63-fold over the past 30 years) <cit.>. Preprints are also shown to lead to substantial advantages in terms of higher visibility and dissemination, altmetric and citation impact <cit.>. This was shown specifically during COVID-19 <cit.>. Early career researchers particularly benefit from using preprints to increase their visibility <cit.>. Nevertheless, significant progress remains to be made in Open Science and the adoption of preprints specifically <cit.>, which are still used by a minority of researchers overall. For example, Open Science Communities (OSCs) are crucial in the transition to Open Science practices. Despite the rise in Open Science infrastructure, these practices are not yet the norm. OSCs play a pivotal role in making Open Science knowledge more accessible and facilitating communication among scholars and policymakers <cit.>. Together, these studies highlight the growing importance and benefits of preprinting in accelerating the dissemination and impact of research, with particular emphasis on its role during public health emergencies like the COVID-19 pandemic. They also underscore the challenges and varying adoption rates across disciplines and career stages, indicating the need for continued advocacy and support for preprint practices in the scientific community.





§ STATE OF THE ART


There is evidence that adopting Open Science practices has effects or impacts in several domains: academic, societal, and economic <cit.>. In terms of academic or research impacts, Open Science practices are associated with increased visibility and reuse, as measured for example by the diversity of citations and media attention received by Open Access articles <cit.>. Open Science has been instrumental in accelerating progress on certain scientific problems <cit.>, in making results more transparent <cit.>, and in addressing what has been termed the replication crisis in certain fields of research <cit.>. Societal benefits identified in a systematic scoping review include enabling broader participation in research, by supporting citizen science and educational initiatives. However, evidence of societal benefits to policy, health, or trust in research is to date more limited <cit.>. Economic benefits, identified from economic modeling studies and case studies, include cost and labor savings from Open Access and open (or FAIR) data, as well as increased innovation <cit.>. However, there is a lack of causal evidence for and prospective studies of these benefits. Open Science practices have also been linked to negative impacts including imposing additional costs <cit.>, reinforcing existing inequalities <cit.>, and homogenizing diverse research traditions <cit.>.



 §.§ Data and code sharing


Researchers who adopt Open Science practices may see increased use and impact of their work, which can support career progression. Several studies examine the importance of data (and code) sharing for scientific advancement but diverge to some extent in their findings. Evidence shows that the novel combination of datasets leads to higher impact and visibility <cit.>. Several studies in specific research disciplines have found correlations between sharing research and increased citations of articles that share data <cit.>. Implementation of journal policies requiring data sharing has also been correlated with increased citations <cit.>. Researchers can share data in several different ways but sharing data privately, upon request, and via supporting information files with publications are the most common approaches – despite being considered suboptimal <cit.>. Sharing research data in a public repository is considered best practice for data sharing but may require additional effort compared to other approaches <cit.>. However, in previous work, we found that, relative to sharing data upon request or as supporting information files, data sharing in repositories was correlated with a 25.36% citation advantage on average <cit.>.

While we can hypothesize that, similar to data sharing, code sharing would promote the reuse of published research that shares code, there is less evidence about whether code sharing is correlated with any effect on citations. Studies of a single journal <cit.> or a small number of journals in a single field <cit.> have found mixed effects. A larger-scale study showed a correlation between links to methods including (but not limited to) code and increased citation, especially when links were still active <cit.>. Another found that monthly citations of articles increased after their associated code repositories were made public <cit.>.



 §.§ Preprints


There is evidence for advantages in terms of visibility for peer-reviewed publications that were previously posted as preprints, as measured by increased citations and altmetrics <cit.>. This effect was examined in detail during the COVID-19 pandemic <cit.>, when media coverage of health-related preprints also saw a significant uptick <cit.>. Other forms of impact associated with preprint posting include receiving additional feedback, which research has shown to be constructive if variable in frequency <cit.>. Studies examining the adoption of preprints by career stage have suggested that they have particular advantages for early-career researchers in terms of career development <cit.>. At the same time, concerns over how preprints may introduce unvetted findings into the scientific record have pointed to the need for nuanced approaches to evidence synthesis <cit.> and the communication of retractions <cit.>.



§ METHODS AND DATA


To make this study entirely reproducible, we focus only on Open Access publications and release all of the accompanying code. We strictly follow and expand upon a published methodology <cit.>. This methodology entails selecting a set of publications of interest, calculating publication and author-level citation counts using a larger Open Access collection, and modeling the effect of interest as independent variables. We use PLOS' Open Science Indicators version 5 as a starting point <cit.>. The OSI publication count totals N = 124'274. We also use the PMC Open Access Subset, with all publications up to October 2023 included <cit.>. The PMC Open Access Subset is used to calculate citation counts for publications and authors. Citation counts calculated using the PMC Open Access Subset have been shown to track global citation counts, and thus to be appropriate when the relative rather than absolute counts are of interest <cit.>. Publications missing a known identifier (DOI, PubMed reference number, PMCID, or a publisher-specific ID), a publication date, and at least one reference are discarded. These often are editorials, letters, or similar article types. The final PMC Open Access Subset publication count totals M = 5'020'948. After an initial analysis, a limited amount of OSI publications are also discarded for being absent in the PMC Open Access Subset or identified as editorials or reviews (i.e., not research articles). Of the 124'376 publications in OSI, 121’999 (98.1%) are processed, matched, and used for the modeling analysis that follows.

We use a linear model for quantifying the relative effect of Open Science Indicators on citation counts, as follows:



  
Dependent variable. Citation counts for each publication are calculated using the full PMC Open Access Subset dataset (M publications above). Citations are based on identifiers, hence only references that include a valid ID are considered. Under these limits, we calculate total citation counts and use this as our main dependent variable. We also calculate citations given within a certain time window from publication (1, 2, and 3 years, also considering the month of publication). This is done in order to conduct a robustness check using citation counts over the same citation accrual time as the dependent variable (e.g., the three-year window for a publication published in June 2015 runs to June 2018 excluded).



  
Independent variables. We use a set of control variables for modeling. Firstly, publication-level variables are commonly considered in similar studies <cit.>. We include the year of publication, to account for citation inflation over time; the month of publication (missing values are set to a default value of 6, that is June), to account for the advantage of publications published early in the year that have more time to accrue citations; the number of authors and the total number of references (including those without a known identifier), both usually correlated with citation impact. We also use the Australian and New Zealand Standard Research Classification (ANZSRC) Fields of Research classification system at the publication level, to account for disciplinary variation in the adoption of Open Science practices. We use the broadest level provided, that of the Division, to avoid data sparsity. In the dataset, 22 divisions are found. We group the least frequent five categories into a single category, since they all belong to the Arts and Humanities. We therefore end up with 18 distinct categories that are encoded as dummy variables to account for the fact that a publication can belong to multiple categories. See Table <ref> for a list of the categories used from the division-level ANZSRC Fields of Research.



The reputation of authors before publication has also been linked to the citation success of a paper <cit.>. To control for this, we have to identify individual authors, a challenging task in itself <cit.>. We focus on a publication-level aggregated indicator of author popularity: the mean H-index of a publication’s authors at the time of publication, calculated from the PMC Open Access Subset. In so doing, we minimize the impact of errors arising from disambiguating author names <cit.>, which would have been higher if we had used measures based on individual observations such as the maximum H-index. We therefore use a simple disambiguation technique when compared to the current state of the art, and consider two author mentions to refer to the same individual if both full name and surname are found to be identical within all of the PMC Open Access Subset. We acknowledge the limitations of this method in possibly merging different authors with the same name and surname. We identify 8'481'129 seemingly distinct authors in this way.

We finally consider the following journal-level variables: if a publication is published by PLOS (any journal), and if a publication is published in PLOS ONE specifically. Given the preponderance of PLOS publications (101'366, or nearly 85% of publications overall), and specifically PLOS ONE publications (83'843, or nearly 70% of publications overall) in the dataset, we do not use any other journal-level variable. 

A set of descriptive statistics for the numerical variables in use is reported in Tables <ref> and <ref>, while their correlations are illustrated in Figure <ref>. The models we test, besides OLS linear regression and robust linear regression, include ANOVA, Tobit, and GLM with negative binomial, zero-inflated negative binomial, lognormal, and Pareto 2 family distributions. These largely support the findings using linear regression and robust linear regression, which are easier to interpret. Therefore, results from other models are omitted here and can be reproduced using the accompanying codebase. Robust linear regression results differ little from simple linear regression, as is expected given the log transformations we systematically apply on skewed numerical variables, but they are provided for comparison.









§ RESULTS


We start by providing a brief descriptive overview of the Open Science Indicators in the target corpus and then proceed to the modeling section.



 §.§ Overview of the Open Science Indicators dataset


As mentioned previously, the OSI dataset we use for analysis comprises 121'999 articles. The majority are articles published in PLOS journals, with the largest proportion originating from PLOS ONE. The remaining articles have been taken from 1232 different journals published by a range of publishers. Rates of adoption for each Open Science practice can be calculated from the dataset to give an overall impression of the degree to which Open Science is practiced. Table <ref> outlines the overall rates of adoption for the main Open Science practices in the dataset and shows that data (in a repository) and code sharing have a relatively low adoption rate across the dataset.



In OSI, the average rates of adoption for Open Science practices observed in the dataset have been increasing over time with changes between 5% and 15% from 2018 to 2023. Data sharing in any form has seen a 5% increase from 2018 to 2023, with data sharing in repositories and online increasing by 9% and 10% respectively. Code sharing (out of all publications) has increased by 6% over the same time period and preprint posting by 15%. Whilst data and code sharing show positive trends over time, the trend for preprint posting shows a large increase between 2018 and 2019 and again from 2019 to 2020, followed by a plateauing since 2021. These trends are also seen when the PLOS cohort and the PMC Open Access Subset cohort are considered separately, although the PMC Open Access Subset cohort shows an increase in preprints in 2023 compared to 2022 which is not present in the PLOS data. We show the general adoption trends in Figure <ref>.



The prevalence of different Open Science practices varies by field of research (following the divisions presented in <ref>). For example, both Division 3 (Health Sciences) and Division 8 (Engineering) have the lowest data sharing rate at 60%, whilst Division 16 (Language, Communication and Culture) has the highest data sharing rate at 82%. Similar degrees in variation are seen for the other indicators with data sharing in a repository ranging from 14% to 43%, code sharing from 7% to 36%, and preprint posting from 10% to 33%. Such wide variation in OSI adoption across divisions suggest that research fields face different challenges in adopting Open Science practices, and some practices may not be equally useful or relevant across fields. In Figure <ref>, we show trends for the main OSIs across all Divisions, as described in Table <ref>.



Please refer to the PLOS' Open Science Indicators version 5 documentation for further details <cit.>.



 §.§ Modeling


The base model results we discuss are provided in Table <ref>. It contains the basic author, publication, and journal-level variables we discussed above. It does not contain publication-level division classification. The most complete model we discuss is instead provided in Table <ref>. Here, we use all the previous variables from the base model and add the publication-level division classification as dummy variables (division 1 to 18, see Table <ref>. Several more models were tested, primarily as robustness checks, and are discussed in the Appendix. 

The base model is described in Equation <ref>, and the full model is described in Equation <ref>. Variable transformations are shown, numerical variables are given in Italics, and categorical variables are in regular text. Variables are grouped along lines. An illustration of the assumed causal dependency graph among variable groups is given in Figure <ref>. In the same figure, the variables for which we used a log scaling to limit the effects of outliers are flagged as such. These include the dependent variable (), which is always used on a log scale.


    log(n_cit_tot + 1) =    log(n_authors + 1) + log(n_references + 1) + p_year + p_month + 
       log(h_index_mean + 1) + 
       is_plos + is_plos_one + 
       data_shared + data_location + repositories_data + 
       code_shared + code_location + 
       preprint_match



    log(n_cit_tot + 1) =    log(n_authors + 1) + log(n_references + 1) + p_year + p_month + 
       log(h_index_mean + 1) + 
       is_plos + is_plos_one + 
       data_shared + data_location + repositories_data + 
       code_shared + code_location + 
       preprint_match + 
       ∑_i=1^18I(division = i)




Starting with the base model in Table <ref>, we provide results for an OLS model and a robust linear model as a comparison. The results are aligned and show a relatively high explained variance with the base model having R^2 = .408. The model shows expected trends, as previously discussed. For example, the higher the year the lower the total citation count on average (about -30% per year increase), or the higher the average H-index of the authors, the higher the citation counts of the paper (this can be interpreted as an elasticity in a log-log model, therefore a 1% increase in the average H-index leads to a .141% increase in the number of citations, on average). More of interest to us are the OSIs. These show that there is a significant and positive effect of preprints (20.4%) and of sharing data via an online repository (3.9%). These percentage changes for log-linear relationships are calculated as follows: (exp(.186) - 1) × 100 ≈ 20.4%. These effects are cumulative, so a publication with both a preprint and data shared in a repository would be associated with an average citation increase of 24.3%. On the other hand, the OSI for code sharing did not yield a statistically significant positive citation effect. Our next question is whether these results hold when we account for the large disciplinary variations in the adoption of Open Science practices, which we assess next.




@5ptlD..-3 D..-3  

  Results for the base model. 
   
  


 
     2cDependent variable: 
 
2-3 

     2cn_cit_tot_log 
 

     1cOLS     1crobust 
 
     1c     1clinear 
 

     1c(1)     1c(2)
 

 
 n_authors_log     0.265^***     0.254^*** 
 
      (0.005)     (0.004) 
 
          
 
 n_references_tot_log     0.192^***     0.198^*** 
 
      (0.005)     (0.005) 
 
          
 
 p_year     -0.357^***     -0.370^*** 
 
      (0.001)     (0.001) 
 
          
 
 p_month     -0.037^***     -0.039^*** 
 
      (0.001)     (0.001) 
 
          
 
 h_index_mean_log     0.141^***     0.141^*** 
 
      (0.003)     (0.003) 
 
          
 
 C(is_plos)True     0.095^***     0.107^*** 
 
      (0.009)     (0.008) 
 
          
 
 C(is_plos_one)True     -0.347^***     -0.351^*** 
 
      (0.007)     (0.007) 
 
          
 
 C(Data_Shared)True     -0.011     -0.012 
 
      (0.034)     (0.033) 
 
          
 
 C(Data_Location)Online     -0.0002     -0.001 
 
      (0.034)     (0.034) 
 
          
 
 C(Data_Location)Supplementary Information     0.024     0.027 
 
      (0.034)     (0.033) 
 
          
 
 C(Repositories_data_bool)True     0.038^***     0.038^*** 
 
      (0.008)     (0.008) 
 
          
 
 C(Code_Shared)True     0.057     0.064 
 
      (0.107)     (0.104) 
 
          
 
 C(Code_Location)Online     -0.105     -0.127 
 
      (0.107)     (0.105) 
 
          
 
 C(Code_Location)Supplementary Information     -0.130     -0.132 
 
      (0.108)     (0.105) 
 
          
 
 C(Preprint_Match)True     0.186^***     0.159^*** 
 
      (0.006)     (0.006) 
 
          
 
 Constant     720.843^***     746.799^*** 
 
      (2.705)     (2.638) 
 
          
 

 
Observations     1c121,999     1c121,999 
 
R^2     1c0.408      
 
Adjusted R^2     1c0.408      
 
Residual Std. Error (df = 121983)     1c0.775     1c0.723 
 
F Statistic     1c5,615.550^*** (df = 15; 121983)      
 

 
Note:      2r^*p<0.1; ^**p<0.05; ^***p<0.01 
 
 

The full model in Table <ref> adds the ANZSRC divisions as 18 dummy variables. The model shows an even higher explained variance with R^2 = 0.426. The full model shows trends that largely confirm the results from the base model. We consolidate our estimate for the citation impact of OSI indicators as follows. We find that the early release of a publication as a preprint correlates with a significant positive citation advantage of about 20.2% (± .7) on average. We also find that sharing data in an online repository is associated with a smaller yet still positive citation advantage of 4.3% (± .8) on average. These effects are cumulative, so a publication with both a preprint and data shared in a repository would be associated with an average citation increase of 24.5%. We do not find a significant effect for sharing code, and we detect significant variations across disciplines in average citation impact. All the remaining coefficients are confirmed in sign and, with minor variation, in magnitude.




@5ptlD..-3 D..-3  

  Results for the full model. 
   
  


 
     2cDependent variable: 
 
2-3 

     2cn_cit_tot_log 
 

     1cOLS     1crobust 
 
     1c     1clinear 
 

     1c(1)     1c(2)
 

 
 n_authors_log     0.207^***     0.194^*** 
 
      (0.005)     (0.005) 
 
          
 
 n_references_tot_log     0.246^***     0.252^*** 
 
      (0.005)     (0.005) 
 
          
 
 p_year     -0.357^***     -0.368^*** 
 
      (0.001)     (0.001) 
 
          
 
 p_month     -0.037^***     -0.038^*** 
 
      (0.001)     (0.001) 
 
          
 
 h_index_mean_log     0.119^***     0.120^*** 
 
      (0.003)     (0.003) 
 
          
 
 C(is_plos)True     0.058^***     0.070^*** 
 
      (0.009)     (0.008) 
 
          
 
 C(is_plos_one)True     -0.304^***     -0.306^*** 
 
      (0.007)     (0.007) 
 
          
 
 C(Data_Shared)True     0.002     -0.005 
 
      (0.033)     (0.033) 
 
          
 
 C(Data_Location)Online     0.010     0.015 
 
      (0.034)     (0.033) 
 
          
 
 C(Data_Location)Supplementary Information     0.020     0.027 
 
      (0.033)     (0.033) 
 
          
 
 C(Repositories_data_bool)True     0.042^***     0.041^*** 
 
      (0.008)     (0.008) 
 
          
 
 C(Code_Shared)True     0.099     0.107 
 
      (0.105)     (0.103) 
 
          
 
 C(Code_Location)Online     -0.110     -0.134 
 
      (0.106)     (0.103) 
 
          
 
 C(Code_Location)Supplementary Information     -0.147     -0.154 
 
      (0.106)     (0.104) 
 
          
 
 C(Preprint_Match)True     0.184^***     0.155^*** 
 
      (0.006)     (0.006) 
 
          
 
 C(division_1)True     0.126^***     0.128^*** 
 
      (0.006)     (0.006) 
 
          
 
 C(division_2)True     0.023^***     0.031^*** 
 
      (0.006)     (0.006) 
 
          
 
 C(division_3)True     0.099^***     0.089^*** 
 
      (0.006)     (0.006) 
 
          
 
 C(division_4)True     0.018^*     0.030^*** 
 
      (0.009)     (0.009) 
 
          
 
 C(division_5)True     -0.086^***     -0.084^*** 
 
      (0.010)     (0.010) 
 
          
 
 C(division_6)True     -0.075^***     -0.087^*** 
 
      (0.010)     (0.010) 
 
          
 
 C(division_7)True     0.031^***     0.020^* 
 
      (0.011)     (0.011) 
 
          
 
 C(division_8)True     -0.186^***     -0.172^*** 
 
      (0.011)     (0.011) 
 
          
 
 C(division_9)True     -0.301^***     -0.294^*** 
 
      (0.012)     (0.011) 
 
          
 
 C(division_10)True     0.035^**     0.045^*** 
 
      (0.014)     (0.013) 
 
          
 
 C(division_11)True     -0.059^***     -0.046^*** 
 
      (0.015)     (0.015) 
 
          
 
 C(division_12)True     -0.235^***     -0.238^*** 
 
      (0.015)     (0.015) 
 
          
 
 C(division_13)True     -0.137^***     -0.133^*** 
 
      (0.018)     (0.018) 
 
          
 
 C(division_14)True     -0.147^***     -0.138^*** 
 
      (0.021)     (0.021) 
 
          
 
 C(division_15)True     -0.037     -0.017 
 
      (0.023)     (0.023) 
 
          
 
 C(division_16)True     -0.057^**     -0.064^*** 
 
      (0.025)     (0.024) 
 
          
 
 C(division_17)True     -0.173^***     -0.164^*** 
 
      (0.026)     (0.025) 
 
          
 
 C(division_18)True     -0.111^***     -0.112^*** 
 
      (0.015)     (0.014) 
 
          
 
 Constant     721.097^***     744.009^*** 
 
      (2.698)     (2.635) 
 
          
 

 
Observations     1c121,999     1c121,999 
 
R^2     1c0.426      
 
Adjusted R^2     1c0.426      
 
Residual Std. Error (df = 121965)     1c0.764     1c0.714 
 
F Statistic     1c2,739.270^*** (df = 33; 121965)      
 

 
Note:      2r^*p<0.1; ^**p<0.05; ^***p<0.01 
 




§ DISCUSSION


This study offers a comprehensive analysis of the citation impact of Open Science practices, drawing on a dataset of about 122'000 research articles and using both descriptive and regression analysis. Our findings reveal a consistent citation advantage for articles whose authors adopted Open Science practices, including data sharing in online repositories and preprint posting. This correlation suggests that Open Science practices may significantly enhance the visibility and academic impact of research findings. However, the Open Science practice of sharing code does not seem to lead to a citation advantage in our sample.



 §.§ Limitations

Several limitations of our study should be acknowledged. First, while our dataset is extensive, it is heavily weighted toward publications by the Open Access publisher PLOS, and as such it may not fully capture the diversity of research across all fields, potentially limiting the generalizability of our findings. Furthermore, PLOS champions Open Science practices, and the stance that a publisher takes in this regard may have an influence on the observed effects. In particular, PLOS requires all authors, with limited exceptions, to share the research data supporting their articles as a condition of publication, with the use of data repositories as the preferred approach. This is reflected in the higher overall rates of data sharing, and higher rates of data repository use in PLOS articles compared to comparators in the OSI dataset. As data sharing is the norm in PLOS articles and as the use of repositories is not uncommon, a citation advantage for the use of data repositories may be smaller in PLOS articles compared to non-PLOS articles. Posting preprints, however, is an optional practice for researchers publishing with PLOS and most other journals. Code sharing, similarly, is optional in most of the journals in our sample, with rare exceptions such as PLOS Computational Biology, where this practice is mandatory <cit.>.

Additionally, the observational nature of our study precludes definitive conclusions about causality. The observed citation advantage might be influenced by other factors not accounted for in our analysis, such as the intrinsic quality of the research or access to research funding. 



 §.§ Extension of previous research

The model-explained variance in our results is globally high with respect to similar studies. For instance, there is previous work showing a positive correlation between citation and altmetric impact of publications, and the posting of preprints <cit.>. The extent of the citation advantage, previously found to be as much as fivefold, is known to vary according to the timing of preprint posting, the discipline, and the preprint server used, among other factors. The smaller magnitude of the effect we find relative to previous studies may relate to the broader range of preprint servers that our sample considers.

Using similar methods to ours, previous work also found a correlation between articles that include statements linking to data in a repository and a citation advantage of up to 25% <cit.>. We confirm this finding in our study, finding a positive correlation between sharing data in a repository and citation impact. Yet the effect we find is considerably smaller in magnitude. This might be caused by the smaller and more uniform dataset that we use here, which includes all PLOS publications and a smaller comparator set, while this previous work used all PLOS and BMC articles and a dataset of over half a million publications. Other studies have also found a positive citation impact of the use of discipline-specific repositories <cit.>.

While previous work <cit.> has found as much as a threefold citation advantage for code sharing, we did not confirm this finding in our sample. Following <cit.>, it is possible that outside of fields like computer science authors are more likely to cite or link to shared code directly rather than citing the research paper with which it was associated.

 



 §.§ Implications for future research

Our data and code are shared openly to enable independent replication of our results and extension of our findings as larger or different, comparable sources of data on the adoption of Open Science practices become available. This includes future versions of the PLOS OSI dataset, as well as outputs from other Open Science monitoring initiatives, such as the French Open Science Monitor [<https://data.enseignementsup-recherche.gouv.fr/explore/dataset/open-access-monitor-france>.] or OpenAIRE [<https://monitor.openaire.eu>.].

As Open Science practices and policies continue to develop, future research could explore longitudinal changes in citation patterns. Further studies could also investigate the relationship between additional Open Science practices and citation impact, extending our understanding of how different aspects of openness contribute to research visibility. Moreover, it would be valuable to examine the impact of Open Science practices on other domains of research dissemination and engagement, such as open commons (e.g., Wikipedia), public policy influence, collaboration networks, and public engagement. We might hypothesize, for example, that non-citation measures of impact – such as forks and downloads – may be more relevant for the sharing of code and software. Contemporary calls for the reform of research assessment (such as <https://coara.eu>) emphasize valuing more diverse research outputs and contributions, as well as more diverse measures of impact. These developments underscore the importance of future research exploring the association of Open Science practices with effects other than citations.



§ CONCLUSION


In summary, our study contributes to the growing body of literature on the effects or impacts of Open Science by quantifying the citation impact of data sharing, code sharing, and preprint posting. Our results could be readily extended with additional data on Open Science practices detected in a larger sample of non-PLOS Open Access publications. We advocate for further empirical research to build on these findings, particularly work that focuses on causal mechanisms, discipline-specific effects, and broader impacts beyond citation metrics.


We now briefly discuss our results in view of the existing literature. There is previous work showing a positive association between citation and altmetric impact, and the early release of preprints <cit.>. This association varies according to the timing of preprint release, the discipline, and the preprint server used, among other factors. The model explained variance is relatively high for this kind of study, yet it leaves significant room for expanding the scope of the models with more, complementary data. 

Using similar methods to ours, previous work also found an association between articles that include statements that link to data in a repository and up to 25.4% (± 1.1%) higher citation impact on average <cit.>. We confirm this finding in our study, finding a positive association between sharing data in a repository and citation impact. Nevertheless, the effect we find is considerably smaller in magnitude. This might be caused by the smaller and more uniform dataset that we use here, which includes all PLOS publications and a smaller control, while this previous work used all PLOS and BMC articles and a dataset of over half a million publications. Other studies also found a positive citation impact at the journal level, overall giving the impression of a trend <cit.>.



§ CONCLUSION


This study presents a comprehensive analysis of the citation impact of open science practices, drawing on a dataset of about 122'000 research articles and using descriptive and regression analysis. Our findings reveal a consistent citation advantage for articles that adhere to open science practices, including data sharing in online repositories and preprint publication. This correlation suggests that open science practices may significantly enhance the visibility and dissemination of research findings. Nevertheless, the open science practice of sharing code does not seem to lead to a further citation advantage in our sample.

However, several limitations of our study should be acknowledged. Firstly, while our dataset is extensive, it centers on the open-access publisher PLOS, and as such it may not fully capture the diversity of research across all disciplines, potentially limiting the generalizability of our findings. Furthermore, PLOS champions open science practices, therefore the detected effects might be larger when publications from other publishers are considered. Additionally, the observational nature of our study precludes definitive conclusions about causality. The observed citation advantage might be influenced by other factors not accounted for in our analysis, such as the intrinsic quality of the research or access to research funding.

Another consideration is the evolving landscape of open science. As practices and policies continue to develop, future research could explore longitudinal changes in citation patterns. Further studies could also investigate the relationship between specific open science practices and citation impact, providing a more granular understanding of how different aspects of openness contribute to research visibility.

Moreover, it would be valuable to examine the impact of open science practices on other dimensions of research dissemination and engagement, such as open commons (e.g., Wikipedia), public policy influence, collaboration networks, and public engagement. We might hypothesise, for example, that non-citation measures of impact may be more relevant for sharing of code and software – such as forks and downloads – reflecting different disciplinary norms of research discovery and reuse. Such investigations could provide a more holistic view of the benefits and potential trade-offs associated with open science practices.

In summary, our study contributes to the growing body of literature on open science by highlighting its potential benefits in terms of citation impact and could be readily extended with additional data on the open science practices detected in a larger sample of non-PLOS open access publications. We advocate for further empirical research to build on these findings, particularly focusing on causal mechanisms, discipline-specific effects, and broader impacts beyond citation metrics.




§ DATA AND CODE AVAILABILITY


OSI dataset: <https://doi.org/10.6084/m9.figshare.21687686.v5>

Code (GitHub): <https://github.com/MediaComem/das-public>

Data (Zenodo): <https://zenodo.org/doi/10.5281/zenodo.10134811>



§ FUNDING


PLOS provided funding for data acquisition, modeling, and analysis, and had a role in the study design, analysis, and preparation of the manuscript. PLOS also provided support in the form of salaries for authors LC, ML, and IH.



§ COMPETING INTERESTS


Three of the authors (LC, ML, and IH) were at the time of publication employed by PLOS, the publisher of PLOS ONE.



§ AUTHORS’ CONTRIBUTIONS



    
  * GC: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Supervision, Writing – original draft, Writing – review & editing.
    
  * LC: Formal Analysis, Visualization, Writing – original draft.
    
  * ML: Writing – original draft, Writing – review & editing.
    
  * GD: Data curation, Software, Writing – review & editing.
    
  * SL: Supervision, Writing – review & editing.
    
  * DR: Supervision, Writing – review & editing.
    
  * IH: Conceptualization, Funding acquisition, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing.




§ ACKNOWLEDGEMENTS


We thank Tim Vines, Scott Kerr, Souad McIntosh, and the team at DataSeer for their collaboration in enabling the OSI dataset to be used for this analysis. We also thank Ross Gray at PLOS for reviewing the data and code from our study.



§ APPENDIX


We show in this Appendix results for a few more models in order to further confirm our results. Firstly, a base model adding code generated as a variable shows a small yet significant negative effect related to it (Table <ref>). This effect goes away when controlling for disciplines, therefore we consider it spurious. When considering OSI interactions (Table <ref>), we find a further negative effect provided by code generated and code shared. This surprising result may be an artifact of the dataset, that we are unsure how to explain. Next, we show how different preprint servers are associated with varying degrees of citation impact (Table <ref>). Lastly, we check a full model using as dependent variables the citation counts up to 1 year after publication (Table <ref>). We still find the same results as using the full citation counts, albeit with a smaller magnitude as expected.


@5ptlD..-3 D..-3  

  Results for the base model with code generated OSI. 
  
  


 
     2cDependent variable: 
 
2-3 

     2cn_cit_tot_log 
 

     1cOLS     1crobust 
 
     1c     1clinear 
 

     1c(1)     1c(2)
 

 
 n_authors_log     0.266^***     0.255^*** 
 
      (0.005)     (0.004) 
 
          
 
 n_references_tot_log     0.195^***     0.200^*** 
 
      (0.005)     (0.005) 
 
          
 
 p_year     -0.357^***     -0.370^*** 
 
      (0.001)     (0.001) 
 
          
 
 p_month     -0.037^***     -0.039^*** 
 
      (0.001)     (0.001) 
 
          
 
 h_index_mean_log     0.141^***     0.142^*** 
 
      (0.003)     (0.003) 
 
          
 
 C(is_plos)True     0.095^***     0.106^*** 
 
      (0.009)     (0.008) 
 
          
 
 C(is_plos_one)True     -0.348^***     -0.352^*** 
 
      (0.007)     (0.007) 
 
          
 
 C(Data_Shared)True     -0.005     -0.007 
 
      (0.034)     (0.033) 
 
          
 
 C(Data_Location)Online     -0.003     -0.003 
 
      (0.034)     (0.034) 
 
          
 
 C(Data_Location)Supplementary Information     0.020     0.024 
 
      (0.034)     (0.033) 
 
          
 
 C(Repositories_data_bool)True     0.041^***     0.040^*** 
 
      (0.008)     (0.008) 
 
          
 
 C(Code_Generated)True     -0.022^***     -0.017^*** 
 
      (0.005)     (0.005) 
 
          
 
 C(Code_Shared)True     0.070     0.075 
 
      (0.107)     (0.104) 
 
          
 
 C(Code_Location)Online     -0.111     -0.132 
 
      (0.107)     (0.105) 
 
          
 
 C(Code_Location)Supplementary Information     -0.137     -0.138 
 
      (0.108)     (0.105) 
 
          
 
 C(Preprint_Match)True     0.188^***     0.160^*** 
 
      (0.006)     (0.006) 
 
          
 
 Constant     720.947^***     746.881^*** 
 
      (2.705)     (2.638) 
 
          
 

 
Observations     1c121,999     1c121,999 
 
R^2     1c0.409      
 
Adjusted R^2     1c0.408      
 
Residual Std. Error (df = 121982)     1c0.775     1c0.723 
 
F Statistic     1c5,266.481^*** (df = 16; 121982)      
 

 
Note:      2r^*p<0.1; ^**p<0.05; ^***p<0.01 
 



@5ptlD..-3 D..-3  

  Results for the base model with interactions among OSI. 
    


 
     2cDependent variable: 
 
2-3 

     2cn_cit_tot_log 
 

     1cOLS     1crobust 
 
     1c     1clinear 
 

     1c(1)     1c(2)
 

 
 n_authors_log     0.266^***     0.254^*** 
 
      (0.005)     (0.004) 
 
          
 
 n_references_tot_log     0.195^***     0.200^*** 
 
      (0.005)     (0.005) 
 
          
 
 p_year     -0.357^***     -0.370^*** 
 
      (0.001)     (0.001) 
 
          
 
 p_month     -0.037^***     -0.039^*** 
 
      (0.001)     (0.001) 
 
          
 
 h_index_mean_log     0.141^***     0.142^*** 
 
      (0.003)     (0.003) 
 
          
 
 C(is_plos)True     0.096^***     0.107^*** 
 
      (0.009)     (0.008) 
 
          
 
 C(is_plos_one)True     -0.350^***     -0.352^*** 
 
      (0.007)     (0.007) 
 
          
 
 C(Data_Shared)True     -0.007     -0.009 
 
      (0.034)     (0.033) 
 
          
 
 C(Data_Location)Online     -0.003     -0.003 
 
      (0.034)     (0.034) 
 
          
 
 C(Data_Location)Supplementary Information     0.022     0.025 
 
      (0.034)     (0.033) 
 
          
 
 C(Repositories_data_bool)True     0.044^***     0.041^*** 
 
      (0.009)     (0.009) 
 
          
 
 C(Preprint_Match)True     0.190^***     0.160^*** 
 
      (0.007)     (0.007) 
 
          
 
 C(Code_Generated)True     -0.015^***     -0.010^* 
 
      (0.005)     (0.005) 
 
          
 
 C(Code_Shared)True     0.144     0.147 
 
      (0.108)     (0.105) 
 
          
 
 C(Code_Location)Online     -0.120     -0.142 
 
      (0.107)     (0.105) 
 
          
 
 C(Code_Location)Supplementary Information     -0.166     -0.166 
 
      (0.108)     (0.105) 
 
          
 
 C(Repositories_data_bool)True:C(Preprint_Match)True     -0.007     0.002 
 
      (0.012)     (0.012) 
 
          
 
 C(Code_Generated)True:C(Code_Shared)True     -0.079^***     -0.077^*** 
 
      (0.017)     (0.017) 
 
          
 
 Constant     720.952^***     746.876^*** 
 
      (2.705)     (2.638) 
 
          
 

 
Observations     1c121,999     1c121,999 
 
R^2     1c0.409      
 
Adjusted R^2     1c0.409      
 
Residual Std. Error (df = 121980)     1c0.775     1c0.723 
 
F Statistic     1c4,683.261^*** (df = 18; 121980)      
 

 
Note:      2r^*p<0.1; ^**p<0.05; ^***p<0.01 
 



@5ptlD..-3 D..-3  

  Results for the base model with preprint servers (considering only those mentioned in 500 or more publications part of the dataset). 
   


 
     2cDependent variable: 
 
2-3 

     2cn_cit_tot_log 
 

     1cOLS     1crobust 
 
     1c     1clinear 
 

     1c(1)     1c(2)
 

 
 n_authors_log     0.259^***     0.250^*** 
 
      (0.005)     (0.004) 
 
          
 
 n_references_tot_log     0.201^***     0.204^*** 
 
      (0.005)     (0.005) 
 
          
 
 p_year     -0.360^***     -0.372^*** 
 
      (0.001)     (0.001) 
 
          
 
 p_month     -0.037^***     -0.039^*** 
 
      (0.001)     (0.001) 
 
          
 
 h_index_mean_log     0.143^***     0.143^*** 
 
      (0.003)     (0.003) 
 
          
 
 C(is_plos)True     0.094^***     0.106^*** 
 
      (0.009)     (0.009) 
 
          
 
 C(is_plos_one)True     -0.344^***     -0.346^*** 
 
      (0.007)     (0.007) 
 
          
 
 C(Data_Shared)True     -0.010     -0.013 
 
      (0.034)     (0.033) 
 
          
 
 C(Data_Location)Online     -0.001     0.002 
 
      (0.034)     (0.034) 
 
          
 
 C(Data_Location)Supplementary Information     0.027     0.030 
 
      (0.034)     (0.033) 
 
          
 
 C(Repositories_data_bool)True     0.043^***     0.041^*** 
 
      (0.009)     (0.008) 
 
          
 
 C(Code_Shared)True     0.060     0.059 
 
      (0.107)     (0.105) 
 
          
 
 C(Code_Location)Online     -0.107     -0.119 
 
      (0.108)     (0.105) 
 
          
 
 C(Code_Location)Supplementary Information     -0.131     -0.125 
 
      (0.108)     (0.106) 
 
          
 
 C(Preprint_Match)True     0.689^**     0.264 
 
      (0.345)     (0.338) 
 
          
 
 C(Preprint_Server)bioRxiv     0.189^***     0.191^*** 
 
      (0.027)     (0.027) 
 
          
 
 C(Preprint_Server)Journal of Medical Internet Research     0.517^***     0.480^*** 
 
      (0.038)     (0.037) 
 
          
 
 C(Preprint_Server)medRxiv     0.470^***     0.381^*** 
 
      (0.030)     (0.029) 
 
          
 
 C(Preprint_Server)N/A     0.721^**     0.309 
 
      (0.346)     (0.339) 
 
          
 
 C(Preprint_Server)Protocols.io     -0.047     -0.040 
 
      (0.043)     (0.042) 
 
          
 
 C(Preprint_Server)PsyArXiv     0.181^***     0.139^*** 
 
      (0.039)     (0.038) 
 
          
 
 C(Preprint_Server)Research Square     0.191^***     0.194^*** 
 
      (0.029)     (0.028) 
 
          
 
 Constant     726.760^***     751.681^*** 
 
      (2.754)     (2.695) 
 
          
 

 
Observations     1c120,195     1c120,195 
 
R^2     1c0.413      
 
Adjusted R^2     1c0.413      
 
Residual Std. Error (df = 120172)     1c0.772     1c0.721 
 
F Statistic     1c3,838.518^*** (df = 22; 120172)      
 

 
Note:      2r^*p<0.1; ^**p<0.05; ^***p<0.01 
 



@5ptlD..-3 D..-3  

  Results for the full model with dependent variable as citation data for 1 year from publication. 
   


 
     2cDependent variable: 
 
2-3 

     2cn_cit_1_log 
 

     1cOLS     1crobust 
 
     1c     1clinear 
 

     1c(1)     1c(2)
 

 
 n_authors_log     0.131^***     0.109^*** 
 
      (0.004)     (0.003) 
 
          
 
 n_references_tot_log     0.113^***     0.117^*** 
 
      (0.004)     (0.004) 
 
          
 
 p_year     0.020^***     0.015^*** 
 
      (0.001)     (0.001) 
 
          
 
 p_month     -0.013^***     -0.013^*** 
 
      (0.001)     (0.001) 
 
          
 
 h_index_mean_log     0.052^***     0.050^*** 
 
      (0.003)     (0.002) 
 
          
 
 C(is_plos)True     0.129^***     0.149^*** 
 
      (0.007)     (0.006) 
 
          
 
 C(is_plos_one)True     -0.303^***     -0.302^*** 
 
      (0.006)     (0.005) 
 
          
 
 C(Data_Shared)True     0.015     0.015 
 
      (0.025)     (0.023) 
 
          
 
 C(Data_Location)Online     0.007     0.008 
 
      (0.025)     (0.023) 
 
          
 
 C(Data_Location)Supplementary Information     0.003     0.005 
 
      (0.025)     (0.023) 
 
          
 
 C(Repositories_data_bool)True     0.012^*     0.011^* 
 
      (0.006)     (0.006) 
 
          
 
 C(Code_Shared)True     0.084     0.075 
 
      (0.081)     (0.075) 
 
          
 
 C(Code_Location)Online     -0.091     -0.092 
 
      (0.081)     (0.075) 
 
          
 
 C(Code_Location)Supplementary Information     -0.101     -0.087 
 
      (0.082)     (0.075) 
 
          
 
 C(Preprint_Match)True     0.133^***     0.099^*** 
 
      (0.004)     (0.004) 
 
          
 
 C(division_1)True     0.047^***     0.049^*** 
 
      (0.004)     (0.004) 
 
          
 
 C(division_2)True     0.019^***     0.029^*** 
 
      (0.005)     (0.004) 
 
          
 
 C(division_3)True     0.003     -0.001 
 
      (0.005)     (0.004) 
 
          
 
 C(division_4)True     -0.015^**     -0.007 
 
      (0.007)     (0.006) 
 
          
 
 C(division_5)True     -0.037^***     -0.030^*** 
 
      (0.008)     (0.007) 
 
          
 
 C(division_6)True     -0.041^***     -0.044^*** 
 
      (0.008)     (0.007) 
 
          
 
 C(division_7)True     0.006     0.0003 
 
      (0.009)     (0.008) 
 
          
 
 C(division_8)True     -0.073^***     -0.054^*** 
 
      (0.009)     (0.008) 
 
          
 
 C(division_9)True     -0.119^***     -0.104^*** 
 
      (0.009)     (0.008) 
 
          
 
 C(division_10)True     0.045^***     0.056^*** 
 
      (0.010)     (0.009) 
 
          
 
 C(division_11)True     -0.026^**     -0.019^* 
 
      (0.012)     (0.011) 
 
          
 
 C(division_12)True     -0.056^***     -0.057^*** 
 
      (0.012)     (0.011) 
 
          
 
 C(division_13)True     -0.037^**     -0.030^** 
 
      (0.014)     (0.013) 
 
          
 
 C(division_14)True     -0.072^***     -0.062^*** 
 
      (0.016)     (0.015) 
 
          
 
 C(division_15)True     -0.024     -0.012 
 
      (0.018)     (0.017) 
 
          
 
 C(division_16)True     0.010     0.0002 
 
      (0.019)     (0.018) 
 
          
 
 C(division_17)True     -0.056^***     -0.045^** 
 
      (0.020)     (0.018) 
 
          
 
 C(division_18)True     -0.032^***     -0.033^*** 
 
      (0.011)     (0.010) 
 
          
 
 Constant     -40.101^***     -29.688^*** 
 
      (2.385)     (2.198) 
 
          
 

 
Observations     1c106,733     1c106,733 
 
R^2     1c0.120      
 
Adjusted R^2     1c0.120      
 
Residual Std. Error (df = 106699)     1c0.542     1c0.522 
 
F Statistic     1c442.350^*** (df = 33; 106699)      
 

 
Note:      2r^*p<0.1; ^**p<0.05; ^***p<0.01 
 









