Most Federal Agencies Reported Access to and Commitment to Using Evidence but Fewer Reported Congressional Requests for Program Evaluation

GPRA represents a central component of the enabling environment for  U.S. government evaluation capacity by providing, for over 20 years, a  statutory framework for performance management and accountability  across the government. About half  the agencies (11) reported committing resources to obtain evaluations by  establishing a central office responsible for evaluating agency programs,  operations, or projects. Most of these offices  were said to be independent of program offices in making decisions about  evaluation design, conduct, and reporting and to have access to analytic  expertise through external experts or contractors, but about half were  reported to have a stable source of funding (6). About one-quarter  of agencies reported having agency-wide written policies or guidance for  key issues addressed in those guides: ensuring internal or external evaluator independence and objectivity; ensuring completeness and transparency of evaluation reports; selecting and prioritizing evaluation topics;  consulting program staff and subject matter experts;  selecting evaluation approaches and methods;  timely, public dissemination of evaluation findings and  recommendations; or  tracking implementation of evaluation findings. Two-thirds of the agencies reported evaluation coverage of less  than half their performance goals; including 7 that reported having  evaluations for none of their performance goals. Five agencies  reported using evaluation to support all these activities to a moderate or  greater extent on average. In fact, in its May 2012 memorandum, OMB  encouraged agencies to designate a high-level official responsible for  evaluation who can
Conduct or oversee rigorous and objective studies;
Provide independent input to agency policymakers on resource  “Develop and manage the agency’s research agenda;  allocation and to program leaders on program management;
Attract and retain talented staff and researchers, including through  flexible hiring authorities such as the Intergovernmental Personnel  Act; and
Refine program performance measures, in collaboration with program  managers and the Performance Improvement Officer.”
In addition, 4 of 11 agencies with a central office responsible for  evaluation reported that this office started conducting evaluations after  2010. Half the Agencies Reported Increasing Their Use of Evaluations as Evidence in Decision Making after 2010

In line with the increases reported in capacity building activities and  organizational resources, about half the agencies reported that their use  of evaluation as supportive evidence had increased at least somewhat  since 2010 (only a few reported great increases). PIOs Reported Hiring, Professional Networking, and Consulting with Experts Were Very Useful in Improving the Capacity to Conduct Evaluations

Our survey asked the PIOs about the usefulness of 14 different actions or  resources for improving their capacity to conduct evaluations, drawn from  the literature and some GPRAMA provisions related to building agency  capacity. PIOs Reported Engaging Program Staff and Holding Goal Leaders Accountable Were Very Useful for Building Capacity to Use Evaluations in Decision Making

Previously, we found that experienced evaluators emphasized three basic  strategies to facilitate evaluation’s influence on program management and  policy: demonstrate leadership support of evaluation for accountability  and program improvement, build a strong body of evidence, and engage  stakeholders throughout the evaluation process. However, a third of PIOs stated that the agency  did not use this vehicle. Those with centralized evaluation authority  reported greater evaluation coverage and use in decision making, but  additional effort will be required to expand agencies’ evaluation capacity  beyond those that already possess evaluation expertise. In addition to  hiring and training staff and consulting experts, promoting information  sharing through informal and formal evaluation professionals’ networks  offers promise for building agencies’ capacity to conduct evaluation in a  constrained budget environment. Engaging program staff, regularly  reviewing progress on agency priority goals, and holding goal leaders  accountable can help build agency use of evaluation in decision making,  as our survey results show. Engaging congressional and other stakeholders in evaluation  planning might increase their interest in evaluation as well as their  adoption of evaluation findings and recommendations. As we have noted before,  congressional committees can also communicate their interest in  evaluation in a variety of ways to encourage agencies to produce  credible, relevant studies that inform decision making:  consult with agencies on proposed revisions to their strategic plans  and priority goals, as GPRAMA requires them to do every 2 years, to  ensure that agency missions are focused, goals are specific and  results-oriented, and strategies and funding expectations are  appropriate and reasonable;  request agency evaluations to address specific questions about the  implementation and results of major program or policy reforms, in time  to consider their results in program reauthorization; and  review agencies’ annual evaluation plans or agendas to ensure that  they address issues that will inform budgeting, reauthorization, and  ongoing program management. Agency Comments

We requested comments on a draft of this report from the Director of the  Office of Management and Budget, whose staff provided technical  comments that we incorporated as appropriate, and from the Director of  the Office of Personnel Management, who provided none. The survey gave us information about agencies’ evaluation  resources, policies, and activities, and the activities and resources they  have found useful in building their evaluation capacity. Influencing  Change: Building Evaluation Capacity to Strengthen Governance. Managing for Results: A Guide for Using the GPRA Modernization Act to  Help Inform Congressional Decision Making.
Why GAO Did This Study

To improve federal government performance and accountability, GPRAMA aims to ensure that agencies use performance information in decision making and holds them accountable for achieving results. The Office of Management and Budget (OMB) has encouraged agencies to strengthen their program evaluations– systematic studies of program performance–and expand their use in management and policy making. This report is one of a series in which GAO, as required by GPRAMA, examines the act's implementation. GAO examined federal agencies' capacity to conduct and use program evaluations and the activities and resources, including some related to GPRAMA, agencies found useful for building that capacity.
GAO reviewed the literature to identify the key components and measures of evaluation capacity. GAO surveyed the PIOs of the 24 federal agencies subject to the Chief Financial Officers Act regarding their organizations' characteristics, expertise, and policies, and their observations on the usefulness of various resources and activities for building evaluation capacity. All 24 responded. GAO also interviewed OMB and Office of Personnel Management (OPM) staff about their capacity-building efforts.

What GAO Found

In a governmentwide survey of agency Performance Improvement Officers (PIO), GAO found uneven levels of evaluation expertise, organizational support within and outside the organization, and use across the government. The Government Performance and Results Act of 1993 (GPRA) is a key component of the enabling environment for federal evaluation capacity, having established a solid foundation of agency performance reporting and leadership commitment to using evidence in decision making. However, only half the agencies reported congressional interest in or requests for program evaluation studies.
Eleven of the 24 agencies reported committing resources to obtain evaluations by establishing a central office responsible for evaluation of agency programs, operations, or projects, although only half these offices were reported to have a stable source of funding. Seven agencies reported having a high-level official responsible for oversight of evaluation. A quarter of agencies reported having agency-wide policies or guidance concerning key issues in study design, evaluator independence and objectivity, report transparency, or implementing findings. Two-thirds of the agencies reported evaluation coverage of less than half their performance goals. Over a third reported using evaluations to a moderate or greater extent as evidence in support of budget or policy changes or program management. Those agencies with centralized evaluation authority reported greater evaluation coverage and use of the results in decision making.
Since the GPRA Modernization Act of 2010 (GPRAMA) was passed, 2 to 4 agencies established a central evaluation office or leader. Half the agencies reported increased efforts to improve their evaluation capacity through hiring, training, conference participation, and consulting experts, but 4 to 5 reported declines in hiring and conference participation. About half reported increased use of evaluations as supporting evidence for management and policy decisions.
About a quarter of PIOs were not familiar with their agencies' various capacity- building activities but many of those that did respond rated hiring, professional networking, consulting with experts, reviewing progress on priority goals, and holding goal leaders accountable under GPRAMA most useful for building capacity to conduct evaluations. They rated engaging program staff in evaluation design, conduct, and reporting, and the GPRAMA priority goal review and accountability provisions most useful for building capacity to use evaluation.
Based on our survey results, GAO observes that
Promoting information sharing in professional networks and engaging program managers and staff in evaluation studies and priority goal reviews offer promise for building capacity in a constrained budget environment.
Engaging congressional and other stakeholders in evaluation planning might increase their interest in and adoption of evaluation recommendations.
Congressional committees can communicate their interest in evaluation by consulting with agencies on their strategic plans and priority goals, reviewing agency annual evaluation plans to ensure they address issues that will inform congressional decision making, and requesting evaluations to address specific questions of interest.

What GAO Recommends

GAO is not making recommendations.
OMB staff provided technical comments on a draft of this report that were incorporated as appropriate. OPM provided no comments.