Moreover, a similar number of federal managers (40 percent) reported  that they did not know if an evaluation had been completed. Most Managers Who Had Evaluations Reported That Evaluations Helped Them Assess and Improve Programs

For the 37 percent of federal managers who had evaluations, the survey  asked to what extent those evaluations had contributed to a variety of  program management and policy making activities. Eighty to 81 percent  of these managers reported that evaluations contributed to a moderate or  greater extent to implementing changes to improve program management  or performance and in assessing program effectiveness or value. The evaluators we interviewed provided several examples of how  evaluations had contributed to their modifying existing or developing new  programs. The only factor that more than one-fourth of the managers reported as  having hindered the agency’s use of evaluation to a great or very great  extent was lack of resources to implement the evaluation findings (33  percent). Potential Barriers Associated with Study Features or Agency Evaluation Capacity

Of seven potential barriers to use concerning the studies or an agency’s  capacity or support for using evaluations, none were generally considered  significant by the 37 percent of federal managers who reported having  evaluations:  concern about the credibility (validity or reliability) of study results (49  percent rated small or no extent); difficulty generalizing the results to other persons or localities (53 difficulty obtaining study results in time to be useful (58 percent); difficulty determining how to use evaluation findings to improve the  program (54 percent);  lack of staff knowledgeable about interpreting or analyzing program  evaluation results (50 percent); difficulty accepting evaluation findings that do not conform to  expectations (54 percent);  lack of ongoing top executive commitment or support for using  evaluation to make program or funding decisions (55 percent). Managers’ Limited Knowledge of Congressional Support for Evaluation

Only 18 percent of managers reported that lack of ongoing congressional  commitment to use evaluation to make program or funding decisions was  a barrier to use to a great or very great extent; however, more (39  percent) reported not being able to judge whether this was a barrier. Lessons for Facilitating Agencies’ Use of Evaluation Results

The evaluators we interviewed emphasized three basic strategies to  facilitate evaluation’s influencing program management and policy: demonstrate leadership support of evaluation for accountability and build a strong body of evidence, and engage stakeholders throughout the evaluation process. It is also a strategy for  ensuring that information is available for input to fast-breaking policy  discussions. The evaluators described providing assistance, training, and incentives to  program staff and service providers to conduct and use evaluations. Concluding Observations

Agencies’ lack of evaluations may be the greatest barrier to their ability to  inform program management and policy making. Yet, just over a third of  federal managers reported that an evaluation had been completed in the  past 5 years on any of the programs, operations, or projects that they  were involved in. Seeking out in advance the interests and concerns of key program  stakeholders, including the Congress, can help ensure that agency  evaluations provide the information necessary for effective management  and congressional oversight. Two of the barriers to evaluation use  managers most frequently cited in our survey concerned addressing  issues important to decision makers and resolving differences of opinion  among stakeholders. Comprehensive program evaluations that examine the coverage and  effectiveness of a cluster of federal programs and policies aimed at  achieving similar outcomes could be key in coordinating and streamlining  programs so as to reduce duplication and overlap. Agency Comments

We requested comments on a draft of this report from the Secretaries of  Agriculture, Health and Human Services, and Labor and the Director of  the Office of Management and Budget. The agencies and OMB staff  provided technical comments that we incorporated as appropriate. We are sending copies of this report to the Secretaries of Agriculture,  Health and Human Services, and Labor; to the Director of the Office of  Management and Budget; and to appropriate congressional committees. Human Services Research  Dissemination: What Works? Performance Measurement and Evaluation: Definitions and  Relationships. Government Performance: Lessons Learned for the Next Administration  on Using Performance Information to Improve Results.
Why GAO Did This Study

The GPRA Modernization Act of 2010 (GPRAMA) aims to ensure that agencies use performance information in decision making and holds them accountable for achieving results and improving government performance. GPRAMA requires GAO to evaluate the act's implementation; this report is one of a series to assess its initial implementation. GAO examined the extent of agencies' use of program evaluations--a particular form of performance information, factors that may hinder their use in program management and policy making, and strategies that may facilitate their use.
GAO surveyed a stratified random sample of 4,391 federal civilian managers and supervisors to obtain their perspectives on several results-oriented management topics, including the extent of and barriers to their evaluation use. GAO also interviewed the Office of Management and Budget and evaluators on barriers to evaluation use and strategies to facilitate it at five agencies selected for their evaluation experience in the Departments of Agriculture, Health and Human Services, and Labor. These officials' views cannot be generalized but provide useful insights.

What GAO Found

In a governmentwide survey, GAO found that most federal managers lack recentevaluations of their programs. Thirty-seven percent reported that an evaluation had been completed within the past 5 years of any program, operation, or project they were involved in. Another 40 percent of managers reported that they did not know if an evaluation had been completed. However, 80 percent of managers who did have evaluations reported that those evaluations contributed to a moderate or greater extent to improving program management or performance and to assessing program effectiveness or value. Fewer reported that evaluations contributed moderately or more to allocating resources within a program (67 percent) or streamlining programs (61 percent).
Of the 37 percent of federal managers who had evaluations, the factor most often rated as having hindered use to a great or very great extent was lack of resources to implement the evaluation findings (33 percent). The next most frequently reported barriers related to program context, such as resolving differences of opinion among program stakeholders (23 percent). Other issues were not considered significant barriers by these managers, such as the lack of credibility or timeliness of study results, lack of leadership commitment or support for using evaluations, or difficulty accepting unexpected findings. Managers reported limited knowledge of congressional support for using results; 39 percent reported not being able to judge whether this was a barrier.
The agency evaluators GAO interviewed noted that it usually takes a number of studies, rather than just one, to influence change in programs or policies. They described using evaluations to modify existing or develop new programs and share what works with their program partners. They emphasized three basic strategies to facilitate evaluation influence:
1. Demonstrate leadership support of evaluation for accountability and improvement by promoting capacity building and the use of evidence and funding evaluation offices to promote and support the use of evidence.
2. Build a strong body of evidence by attending to rigor in whatever methods are used and accumulating a knowledge base from which to respond to varied questions over time or fast-breaking policy discussions.
3. Engage stakeholders throughout the evaluation process--developing relationships to gain their input to planning and buy-in; providing assistance, training, and incentives; and disseminating usable messages.
GAO observes that
Agencies' lack of evaluations may be the greatest barrier to their informing program managers and policy makers.
Seeking out in advance the interests and concerns of program stakeholders, including Congress, can help ensure that evaluations will provide the information necessary for effective management and oversight.
Comprehensive evaluations that examine the coverage and effectiveness of federal programs and policies aimed at achieving similar outcomes could be key to coordinating and streamlining programs to reduce duplication and overlap.

What GAO Recommends

GAO is not making recommendations.
The Departments of Agriculture, Health and Human Services, and Labor, and Office of Management and Budget staff provided technical comments on a draft of this report that we incorporated as appropriate.