Definition of Peer Review

There is no written definition of peer review that applies across the federal government. Peers generally are considered to be scientists or engineers who have qualifications and expertise equivalent to those of the researcher whose work they review. However, the office advocates the use of peer review and provides guidance to agencies on the use of peer review to assess the quality of research. Officials at the Office of Science and Technology Policy said that agencies’ peer review practices should be flexible and tailored to agency missions and type of research, and that specific uniform practices should not be dictated for every agency or all federally funded research. “Science agencies must devise assessment strategies that are appropriate to the nature of scientific processes and to the enabling role of fundamental science in support of over-arching national goals... should be designed to...respond to surprises, pursue detours, and revise program agendas in response to new scientific information and technical opportunities essential to the future well-being of all our people.”
All the agencies that we contacted identified policies, orders, or other internal guidance regarding the conduct of peer review. Reviews of Research Proposals

All the agencies conduct peer reviews to help determine which competitive research proposals to fund. In some circumstances, agencies use these same types of peer review processes to assess the merit of research that is not funded through competitive selection, generally this research is internal to the agencies. These reviews are generally conducted by supervisors or managers and are, therefore, not independent reviews of the research. These reviews occur at both the project and program level. Research Not Subject to Review

While the agencies said that they conduct either peer reviews or other quality reviews for almost all of their research, there are small amounts of research that may not be reviewed. Scope and Methodology

To define what is meant by peer review and to describe the federal government’s peer review policy, we reviewed studies of government peer review, previous GAO reports, and documentation provided by the 12 agencies included in our review. To describe the peer review practices of 12 agencies, we obtained and compared descriptive information on peer review at each agency to identify what the various practices were and to determine whether the practices were uniform among and within the agencies. Peer Review Practices

ARS has a multilayered system of complimentary peer reviews that includes reviews of the technical merit of planned research projects prior to funding, reviews of research results prior to the publication of the results, and reviews of ongoing research programs. Peer review will probably be defined as a method for conducting merit review that uses people with qualifications and expertise to conduct research similar to that being reviewed. Reviews of Publications

According to FS officials, publications play a key role in the quality assurance and quality control process for research and development. Other Agency Quality Assurance Reviews

Since all NOAA research is evaluated by some type of peer review process, agency officials provided no examples of other quality assurance reviews. Peer Review is also used to evaluate unsolicited proposals.
Why GAO Did This Study

Pursuant to a congressional request, GAO studied the peer review and other quality assurance processes that federal agencies use in conducting scientific research and development, focusing on: (1) defining what is meant by peer review; (2) describing the federal government's peer review policy; (3) describing the peer review practices of 12 federal agencies that conduct scientific research; (4) describing other agencies' quality assurance reviews; and (5) identifying which research is not subject to review.

What GAO Found

GAO noted that: (1) there is no written governmentwide definition of peer review; (2) officials at the Office of Science and Technology Policy (OSTP) and at the agencies GAO contacted generally concur that peer review is defined as a process that includes an independent assessment of the technical, or scientific merit of research by peers who are scientists with knowledge and expertise equal to that of the researchers whose work they review; (3) there is no uniform federal policy for conducting peer reviews; (4) through annual budget guidance to federal agencies, OSTP and the Office of Management and Budget encourage funding of research projects that are peer reviewed over those that are not reviewed; (5) officials at OSTP said that peer review practices should not be dictated uniformly for every agency or for all types of federally funded research; (6) rather, the practices should be tailored to agency missions and type of research; (7) each of the 12 agencies that GAO contacted had a variety of policies, orders, or other internal guidance regarding the conduct of peer review; (8) to varying degrees, the 12 agencies use peer review to: (a) assess the merit of competitive and noncompetitive research proposals; (b) determine whether to continue or renew research projects; (c) evaluate the results of the research prior to the publication of those results; (d) establish annual budget priorities for research programs; and (e) evaluate program and scientist performance; (9) all of the agencies use peer review to assess competitive research proposals; (10) the methods for conducting peer reviews vary among and within the agencies; (11) most of the agencies that GAO reviewed also use reviews by agency supervisors or program managers to assess the quality of research proposals, to check the quality of in-progress research, and to evaluate program performance; (12) generally, these quality assurance reviews are not considered independent assessments--a key criterion in the peer review process; (13) these quality assurance reviews occur at both the project and program levels; (14) while agencies reported that almost all research is reviewed either through peer reviews or other quality assurance reviews, a small amount of research may not be reviewed by the agencies in certain circumstances; and (15) examples of research that may be funded without being reviewed include projects that are congressionally mandated or projects that use widely accepted methodologies.