Peer review: a simplified approach. Chance and consensus in peer review. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. Effects of training on quality of peer review: randomised controlled trial. Evaluation of cardiovascular grant-in-aid applications by peer review: influence of internal and external reviewers and committees. Effect of structured workshop training on subsequent performance of journal peer reviewers. The effects of blinding on the quality of peer review. A randomized trial.
One hundred and thirteen grant applications were initially reviewed with only a brief outline of the applicant and the proposal and were later reviewed in more detail with the reports of outside experts. The detailed discussion and referees' reports had little impact on the original ratings. An experiment in which 150 proposals submitted to the National Science Foundation were evaluated independently by a new set of reviewers indicates that getting a research grant depends to a significant extent on chance. The degree of disagreement within the population of eligible reviewers is such that whether or not a proposal is funded depends in a large proportion of cases upon which reviewers happen to be selected for it. No evidence of systematic bias in the selection of NSF reviewers was found. Anxiety about bias, lack of accountability, and poor quality of peer review has led to questions about the imbalance in anonymity between reviewers and authors.
                To evaluate the effect on the quality of peer review of blinding reviewers to the authors' identities and requiring reviewers to sign their reports.
                Randomized controlled trial.
                A general medical journal.
                A total of 420 reviewers from the journal's database.
                We modified a paper accepted for publication introducing 8 areas of weakness. Reviewers were randomly allocated to 5 groups. Groups 1 and 2 received manuscripts from which the authors' names and affiliations had been removed, while groups 3 and 4 were aware of the authors' identities. Groups 1 and 3 were asked to sign their reports, while groups 2 and 4 were asked to return their reports unsigned. The fifth group was sent the paper in the usual manner of the journal, with authors' identities revealed and a request to comment anonymously. Group 5 differed from group 4 only in that its members were unaware that they were taking part in a study.
                The number of weaknesses in the paper that were commented on by the reviewers.
                Reports were received from 221 reviewers (53%). The mean number of weaknesses commented on was 2 (1.7, 2.1, 1.8, and 1.9 for groups 1, 2, 3, and 4 and 5 combined, respectively). There were no statistically significant differences between groups in their performance. Reviewers who were blinded to authors' dentities were less likely to recommend rejection than those who were aware of the authors' identities (odds ratio, 0.5; 95% confidence interval, 0.3-1.0).
                Neither blinding reviewers to the authors and origin of the paper nor requiring them to sign their reports had any effect on rate of detection of errors. Such measures are unlikely to improve the quality of peer review reports. To determine the effects of training on the quality of peer review.
                Single blind randomised controlled trial with two intervention groups receiving different types of training plus a control group.
                Reviewers at a general medical journal. Interventions Attendance at a training workshop or reception of a self taught training package focusing on what editors want from reviewers and how to critically appraise randomised controlled trials.
                Quality of reviews of three manuscripts sent to reviewers at four to six monthly intervals, evaluated using the validated review quality instrument; number of deliberate major errors identified; time taken to review the manuscripts; proportion recommending rejection of the manuscripts.
                Reviewers in the self taught group scored higher in review quality after training than did the control group (score 2.85 v 2.56; difference 0.29, 95% confidence interval 0.14 to 0.44; P = 0.001), but the difference was not of editorial significance and was not maintained in the long term. Both intervention groups identified significantly more major errors after training than did the control group (3.14 and 2.96 v 2.13; P < 0.001), and this remained significant after the reviewers' performance at baseline assessment was taken into account. The evidence for benefit of training was no longer apparent on further testing six months after the interventions. Training had no impact on the time taken to review the papers but was associated with an increased likelihood of recommending rejection (92% and 84% v 76%; P = 0.002).
                Short training packages have only a slight impact on the quality of peer review. The value of longer interventions needs to be assessed. To determine what factors of the peer review process played a significant role in the assignment of scores for scientific merit by a major cardiovascular research finding agency. Specific variables studied included scores of internal and external reviewers, committee assignment and year of application.
                Retrospective analysis of research proposals submitted to the Heart and Stroke Foundation of Ontario from 1990 to 1994.
                Of a total database of 804 grant-in-aid (operating or project) proposals submitted to the Heart and Stroke Foundation of Ontario from 1990 to 1994, 779 had complete data and were included in the study. Variables analyzed were scores of internal and external reviewers, final committee score, committee assignment and year of application.
                There was poor correlation (r = 0.113) between external reviewers of grant proposals and only slightly better correlation between internal reviewers (r = 0.331). When mean internal and external scores were analyzed, correlation improved (r = 0.451) but agreement beyond chance was only moderate (weighed Cohen's kappa = 0.532). Regression analysis established that mean internal score had a greater impact on final committee score (r2 = 0.607, P < 0.001) than mean internal score (r2 = 0.348, P < 0.001). When review committee was entered as a dummy variable into a regression statement, it was statistically significant (P < 0.001) but explained less than 8% of variance in the final committee score. Mean scores showed a small 'upward creep' over the five-year period but explained less than 4% of variance in final committee score (r2 = 0.039, P < 0.001).
                In the peer review system studied, the scores of internal reviewers were more closely correlated to final committee score for scientific merit than those of external reviewers. Nevertheless, final committee scores were significantly different from either internal or external scores, suggesting that the process of committee discussion is an important, and highly influential, step in the peer review process. Study objective: We sought to determine whether peer reviewers who attend a formal interactive training session produce better reviews.
                Peer reviewers were invited to attend a formal, 4-hour, highly interactive workshop on peer review. Attendees received a sample manuscript to read and review in writing in advance. The workshop included presentations on analyzing a study and the journal's expectations for a quality review, discussion of the sample manuscript's flaws and how to address them in a review, discussion of the reviews written by the attendees, and discussion of real reviews of other manuscripts illustrating key points. The performance of attendees on the basis of standard editor quality ratings (1 to 5) was assessed for the 2 years after workshop attendance. Control reviewers matched for previous review quality and volume were selected from nonattendees of the workshop. In study 1, all average reviewers received a standard written invitation. In study 2, 75 randomly selected average reviewers were personally and actively recruited with intensive follow-up by means of e-mail and telephone calls in an effort to reduce self-selection bias.
                In study 1, 25 reviewers volunteered for the course, were eligible for study, attended, and were compared with 25 matched control reviewers. Of attendees filling out evaluations, 19% thought it somewhat and 81% thought it very helpful. All thought it would improve their subsequent reviews, and 85% thought it would improve their review ratings. The mean change in rating after the workshop was 0.11 (95% confidence interval [CI] -0.25 to 0.48) for control reviewers and 0.10 (95% CI -0.20 to 0.39) for attendees. In study 2, of 75 reviewers intensively recruited, only 12 (41%) of those who said they would attend did. All of the participants thought the workshop would improve their performance and ratings. Test scores at the end of the workshop improved in 73% of participants compared with scores on pretests. The control reviewers' average rating changed by -0.10 (95% CI -0.49 to 0.29) versus 0.06 (95% CI -0.34 to 0.23) for attendees.
                Among invited peer reviewers, voluntary attendance at a highly structured and interactive workshop was low and did not improve the quality of subsequent reviews, contrary to the predictions of attendees. Efforts to aggressively recruit average reviewers to a second workshop were time consuming, had low success rates, and showed a similar lack of effect on ratings, despite improvement in scores on a test instrument. Workshop teaching formats, although traditional, are of unproven efficacy. Peer reviewers are blinded sometimes to authors' and institutions' names, but the effects of blinding on review quality are not known. We, therefore, conducted a randomized trial of blinded peer review. Each of 127 consecutive manuscripts of original research that were submitted to the Journal of General Internal Medicine were sent to two external reviewers, one of whom was randomly selected to receive a manuscript with the authors' and institutions' names removed. Reviewers were asked, but not required, to sign their reviews. Blinding was successful for 73% of reviewers. Quality of reviews was higher for the blinded manuscripts (3.5 vs 3.1 on a 5-point scale). Forty-three percent of reviewers signed their reviews, and blinding did not affect the proportion who signed. There was no association between signing and quality. Our study shows that, in our setting, blinding improves the quality of reviews and that research on the effects of peer review is possible.
There is little empirical evidence on the effects of grant giving peer review. No studies assessing the impact of peer review on the quality of funded research are presently available. Experimental studies assessing the effects of grant giving peer review on importance, relevance, usefulness, soundness of methods, soundness of ethics, completeness and accuracy of funded research are urgently needed. Practices aimed to control and evaluate the potentially negative effects of peer review should be implemented meanwhile.