Federal Agencies Devote Variable but Small Amounts of Resources to Evaluating Program Results, Primarily for Internal Use

The resources allocated to conducting systematic assessments of program results (or evaluation studies) were small and unevenly distributed across the 23 agencies (departments and independent agencies) we surveyed. Program evaluation was reported to be used more often for general program improvement than for direct congressional oversight. Reforms May Increase Interest In, But Complicate The Process of, Obtaining Results

Some of our interviewees thought that recent governmentwide reforms would increase interest in learning the results of federal programs and policies but would also complicate the task of obtaining that information. A few evaluation officials were concerned that a large investment would be required to produce valid and reliable outcome (rather than process) data. While performance measures were seen as useful for program management, some state agency and legislative staff also saw a continuing need for evaluations to assess policy impact or address problems of special interest or “big-picture” concerns, such as whether a government program should be continued or privatized. Agencies Suggested Varied Strategies for Obtaining Program Results Information

		Leveraging Limited Resources

Given constraints on federal budgets, some officials we interviewed in general did not expect federal evaluation resources to rise to meet demand, so they described efforts to leverage and prioritize available resources. A few agencies reported that they were adapting the elements of their existing program information systems to yield information on program results. But in other agencies, evaluation officials and external experts thought that their systems were primarily focused on program process, rather than results. The evaluation official said that structural changes to, and a major investment in, their data systems will be required to provide valid and meaningful data on results. Creating New Evaluation Designs and Partnerships

As program responsibility shifts to state and local entities, evaluation officials and others we interviewed described the need for study designs that can handle greater contextual complexity, new ways to measure outcomes, and the need to build partnerships with the programs’ stakeholders. Comparison of current federal program evaluation resources with the anticipated challenges leads us to several conclusions. First, federal agencies’ evaluation resources have important roles to play in responding to increased demand for information on program results, but—as currently configured and deployed—they are likely to be challenged to meet these future roles. Second, in the future, carefully targeting and reshaping the use of federal evaluation resources and leveraging federal and nonfederal resources show promise for addressing the most important questions about program results. “Evaluation in the Federal Government: Changes, Trends, and Opportunities.” Entire issue. New Directions for Program Evaluation.
Why GAO Did This Study

GAO reviewed federal agencies' efforts to provide information on federal program results, focusing on: (1) the current resources and roles for program evaluation in federal agencies; (2) the anticipated effects of governmentwide reforms and other initiatives on evaluation of federal programs; and (3) potential strategies for agencies to respond to the anticipated effects and provide information on program results.

What GAO Found

GAO noted that: (1) existing federal evaluation resources--at least as currently configured and deployed--are likely to be challenged to meet increasing demands for program results information; (2) agencies reported devoting variable but relatively small amounts of resources to evaluating program results; (3) morever, agencies reported that the primary role of program evaluation was internally focused on program improvement, rather than direct congressional or other external oversight; (4) interest in the program by high-level officials was most often cited as a criterion for initiating evaluation work; a small portion of studies were said to be conducted for a congressional committee or in response to a legislative mandate; (5) some of the evaluation officials and experts that GAO interviewed anticipated not only increased interest in learning the results of federal programs and policies but also additional complications in obtaining that information; (6) some evaluation officials from states with performance measurement experience noted that effectiveness evaluations would continue to be needed to assess policy impact and address problems of special interest or larger policy issues, such as the need for any government intervention at all in an area; (7) to meet the anticipated increase in demand for program results information as well as the associated technical challenges, some evaluation officials GAO interviewed described efforts to leverage both federal and nonfederal resources; (8) however, some agencies anticipated that major investments in their data systems would be required to produce reliable data on program outcomes; and, in a prior study, program officials were concerned that reliance on less rigorous methods would not provide an accurate picture of program effectiveness; (9) moreover, while some federal evaluation officials envisioned providing increased technical assistance to state and local evaluators, a few state evaluation officials suggested an alternative strategy for the federal government; (10) GAO drew several conclusions from its comparison of current federal evaluation resources with the anticipated challenges to meeting increased demand for information on program results; (11) federal evaluation resources have important roles to play in responding to increased demand for information on program results, but--at least as currently configured and deployed--they are likely to be challenged to meet that demand; and (12) in the future, carefully targeting federal agencies' evaluation resources shows promise for addressing key questions about program results.