Having developed informal processes over time,  senior officials, and evaluation and program office staff have a common  understanding of how they will develop, review, and select evaluations. In addition to consulting internal program staff, most of the agencies we  examined consult external groups to obtain ideas for evaluation proposals. Generally, the agencies review and approve evaluation proposals in two  steps. First, evaluation or program staff members develop ideas or brief  concept papers for initial feedback from senior officials. Only CDC’s divisions hold formal, ranked competitions to review and  select proposals. These plans align well with the  American Evaluation Association’s (AEA) recommendation, made in a  recent policy paper on federal government evaluation, that federal  agencies prepare annual and multiyear evaluation plans to guide program  decision-making and consult with the Congress and nonfederal  stakeholders in defining program and policy objectives, critical operations,  and definitions of success. Common Types of Criteria Agencies Use to Prioritize Evaluation Proposals

We found these mature agencies remarkably similar in the four general  criteria they used for selecting evaluations to conduct during the next  fiscal year: strategic priorities, program concerns, critical unanswered  questions, and the feasibility of conducting a valid evaluation study. The first criterion, strategic priorities, represents major program or policy  areas identified as a focus of concern and reflected in a new initiative or  increased level of effort. Prioritization Depends on Funding Sources and Agency Circumstances

Although the agencies generally followed a similar process in developing  their evaluation agendas, some agency characteristics or conditions  appeared to influence their choices and may be important for other  agencies to consider as they develop their own evaluation agendas. The  four conditions we identified as strongly influencing the evaluation  planning process were  1.  the location of evaluation funding, whether with the program or  2.  the scope of the evaluation unit’s responsibility within the agency;  3.  how much the evaluators rely on program partners; and  4.  the extent and form of congressional oversight over agency program  evaluations. Where evaluation funds and authority are tied to the program, and funds  are available, evaluation staff generally choose not which programs to  evaluate but which research questions to answer. Being responsible for a wide range of analytic activities also influenced an  evaluation office’s choice of evaluations. The evaluation  offices in ACF, Education, and HUD all noted their responsibility to  conduct congressionally mandated evaluation studies in describing the  criteria they used in evaluation planning. However, developing a multiyear evaluation  plan could help ensure that all an agency’s programs are examined over  time. To produce an effective evaluation agenda, agencies may want to follow  the general model we identified at the agencies we reviewed: professional  evaluators lead an iterative process of identifying important policy and  program management questions, vetting initial ideas with the evaluations’  intended users, and scrutinizing the proposed portfolio of studies for  relevance and feasibility within available resources. To ensure that their evaluations provide the information necessary for  effective management and legislative oversight, evaluation offices are  likely to need to seek out in advance the interests and concerns of key  program and congressional stakeholders, especially program partners, and  discuss preliminary proposals with the intended users. Agency Comments

The Departments of Health and Human Services and Housing and Urban  Development provided comments on a draft of this report, which are  reprinted in appendixes I and II. This is consistent with our concluding observation that  agencies may need to adapt the general model—including where to locate  evaluation offices—to match their own organizational and financial  circumstances. HUD agreed with our description of how it plans evaluations but was  concerned that the report did not place enough emphasis on the  appropriations process as a major influence on what projects it funds and  when it can begin the contracting process. Education, HHS, and HUD also provided technical comments that were  incorporated where appropriate throughout the text. Program Evaluation: A Variety of Rigorous Methods Can Help Identify  Effective Interventions.
Why GAO Did This Study

Amid efforts to improve performance and constrain spending, federal agencies are being asked to expand the use of rigorous program evaluation in decision-making. In addition to performance data, indepth program evaluation studies are often needed for assessing program impact or designing improvements. Agencies can also use their evaluation resources to provide information needed for effective management and legislative oversight. GAO was asked to study federal agencies with mature evaluation capacity to examine (1) the criteria, policies, and procedures they use to determine programs to review, and (2) the influences on their choices. GAO reviewed agency materials and interviewed officials on evaluation planning in four agencies in three departments with extensive evaluation experience: Education, Health and Human Services (HHS), and Housing and Urban Development (HUD). HHS and HUD agreed with the description of how they plan evaluations. HHS noted that the optimal location of evaluation units will vary with the circumstances and purpose of evaluations. HUD felt the draft report did not emphasize enough the influence of the appropriations process. GAO has added text to note its influence on evaluation planning. Education provided technical comments.

What GAO Found

Although no agency GAO reviewed had a formal policy describing evaluation planning, all followed a generally similar model for developing and selecting evaluation proposals. Agencies usually planned an evaluation agenda over several months in the context of preparing spending plans for the coming fiscal year. Evaluation staff typically began by consulting with a variety of stakeholders to identify policy priorities and program concerns. Then with program office staff they identified the key questions and concerns and developed initial proposals. Generally, the agencies reviewed and selected proposals in two steps: develop ideas to obtain initial feedback from senior officials and develop full-scale evaluation proposals for review and approval. The four general criteria these mature agencies use to plan evaluations were remarkably similar: (1) strategic priorities representing major program or policy area concerns or new initiatives, (2) program-level problems or opportunities, (3) critical unanswered questions or evidence gaps, and (4) the feasibility of conducting a valid study. The agencies' procedures differed on some points. External parties' participation in evaluation planning may reflect these agencies' common reliance on nonfederal program partners. Only the offices GAO reviewed in HHS' Centers for Disease Control and Prevention held formal competitions to rank-order proposals before submitting them for approval; in other agencies, senior officials assessed proposals in a series of discussions. When evaluation authority and funds are tied to the program, evaluators generally choose not which programs to evaluate but which research questions to answer. Sometimes this resulted in a program's never being evaluated. Evaluation units at higher organizational levels conducted a wider range of analytic activities, consulted more formally with program offices, and had less control over approvals. The Congress influences an agency's program evaluation choices through legislating evaluation authority, mandating studies, making appropriations, and conducting oversight. GAO concludes that (1) all four criteria appear key to setting an effective evaluation agenda that provides credible, timely answers to important questions; (2) most agencies could probably apply the general model in which professional evaluators iteratively identify key questions in consultation with stakeholders and then scrutinize and vet research proposals; (3) agencies could adapt the model and decide where to locate evaluation units to meet their own organizational and financial circumstances and authorities; and (4) agencies' reaching out to key program and congressional stakeholders in advance of developing proposals could help ensure that their evaluations will be used effectively in management and legislative oversight.

What GAO Recommends

GAO makes no recommendations.