	Background

FHWA is the DOT agency responsible for federal highway programs— including distributing billions of dollars in federal highway funds to the  states—and developing federal policy regarding the nation’s highways. Research Community Promotes Use of Best Practices for Developing Research Agendas and Evaluating Research Outcomes

Leading organizations that conduct scientific and engineering research,  other federal agencies with research programs, and experts in research  and technology have identified and use best practices for developing  research agendas and evaluating research outcomes. Although the  uncertain nature of research outcomes over time makes it difficult to set  specific, measurable program goals and evaluate results, the best practices  we identified are designed to ensure that the research objectives are  related to the areas of greatest interest and concern to research users and  that research is evaluated according to these objectives. Developing Research Agendas Through the Involvement of External Stakeholders

External stakeholder involvement is particularly important for FHWA  because its research is expected to improve the construction, safety, and  operation of transportation systems that are primarily managed by others,  such as state departments of transportation. In its 1999 report, the Committee on Science, Engineering,  and Public Policy also stated that expert review is widely used to evaluate:  (1) the quality of current research as compared with other work being  conducted in the field, (2) the relevance of research to the agency’s goals  and mission, and (3) whether the research is at the “cutting edge.”

	External Stakeholders’ Involvement in Developing FHWA’s Research Agendas Has Been Limited

Although FHWA engages external stakeholders in elements of its research  and technology program, the agency currently does not follow the best  practice of engaging external stakeholders on a consistent and transparent  basis in setting its research agendas. As we reported in May 2002, FHWA acknowledges  that its approach to preparing research agendas is inconsistent and that  the associate administrators of FHWA’s program offices primarily use  input from the agency’s program offices, resource centers, and division  offices. It appears that FHWA has committed to  taking the necessary steps to adopt the best practice of developing a  systematic process for involving external stakeholders in the agenda  setting process. However, because FHWA’s plan has not been finalized, we cannot  comment on its potential effectiveness in involving external stakeholders. FHWA Lacks a Systematic Approach to Evaluating Research Outcomes

As we reported last year, FHWA does not have an agency wide systematic  process to evaluate whether its research projects are achieving intended  results that uses such techniques as peer review. In our May 2002 report, we recommended that FHWA develop a systematic  process for evaluating significant ongoing and completed research that  incorporates peer review or other best practices in use at federal agencies  that conduct research. According to FHWA’s assistant  director for Research, Technology, and Innovation Deployment, the  agency is using the results of this report to develop its own systematic  approach for evaluating its research and technology program. FHWA is still in the process of developing, defining, and adopting a  framework for measuring performance.
Why GAO Did This Study

Improvement and innovation based on highway research have long been important to the highway system. The Federal Highway Administration (FHWA) is the primary federal agency involved in highway research. Throughout the past decade, FHWA received hundreds of millions of dollars for its surface transportation research program, including nearly half of the Department of Transportation's approximate $1 billion budget for research in fiscal year 2002. Given the expectations of highway research and the level of resources dedicated to it, it is important to know that FHWA is conducting high quality research that is relevant and useful. In May 2002, GAO issued a report on these issues and made recommendations to FHWA, which the agency agreed with, aimed at improving its processes for setting research agendas and evaluating its research efforts. GAO was asked to testify on (1) best practices for developing research agendas and evaluating research outcomes for federal research programs; (2) how FHWA's processes for developing research agendas align with these best practices; and (3) how FHWA's processes for evaluating research outcomes align with these best practices.

What GAO Found

Leading organizations, federal agencies, and experts that conduct scientific and engineering research use best practices designed to ensure that research objectives are related to the areas of greatest interest to research users and that research is evaluated according to these objectives. Of the specific best practices recommended by experts--such as the Committee on Science, Engineering, and Public Policy and the National Science Foundation--GAO identified the following practices as particularly relevant for FHWA: (1) developing research agendas in consultation with external stakeholders to identify high-value research and (2) using a systematic approach to evaluate research through such techniques as peer review. FHWA's processes for developing its research agendas do not always consistently include stakeholder involvement. External stakeholder involvement is important for FHWA because its research is to be used by others that manage and construct transportation systems. FHWA acknowledges that its approach for developing research agendas lacks a systematic process to ensure that external stakeholders are involved. In response to GAO's recommendation, FHWA has drafted plans that take the necessary steps toward developing a systematic process for involving external stakeholders. While the plans appear responsive to GAO's recommendation, GAO cannot evaluate their effectiveness until they are implemented. FHWA does not have a systematic process that incorporates techniques such as peer review for evaluating research outcomes. Instead, the agency primarily uses a "success story" approach to communicate about those research projects that have positive impacts. As a result, it is unclear the extent to which all research projects have achieved their objectives. FHWA acknowledges that it must do more to measure the performance of its research program, however, it is still in the process of developing a framework for this purpose. While FHWA's initial plans appear responsive to GAO's recommendation, GAO cannot evaluate their effectiveness until they are implemented.