All Programs Responded to OMB’s Recommendations, but Only Half Completed Evaluations

About half of the programs we reviewed (11 of the 20) had completed an  evaluation by June 2005—2 years after the fiscal year 2004 PART reviews  and recommendations were published. Four evaluations were in progress,  while 3 were still in the planning stage. Measurement Challenges Delayed Evaluation Starts

To evaluate program effectiveness, agencies needed to identify  appropriate measures of the outcomes they intended to achieve and  credible data sources for those measures. Some agency officials thought  that evaluations should not be conducted for all programs but should be  targeted instead to areas of uncertainty. However, officials in all four agencies  indicated that the visibility of a PART recommendation and associated  OMB pressure brought management attention, and sometimes funds, to  getting the evaluations done. Centralization helped  the agencies to leverage their evaluation expertise throughout the agency  and helped them prioritize spending on the evaluations they considered  most important. Where OMB and Program Managers Do Not Share Expectations, Evaluations May Not Meet OMB Needs

Because the OMB evaluation recommendations were fairly general,  agencies had flexibility in interpreting the information OMB expected and  the evaluations to fund. Some program managers disagreed with OMB on  the scope and purpose of their evaluations, their quality, and the  usefulness of evaluations conducted by independent third parties. Program managers concerned about an increased focus on process said  that they were more interested in learning how to improve program  performance than in meeting an OMB checklist. Since a few programs did  not discuss their evaluation plans with OMB, it is not certain whether OMB  will accept their ongoing evaluations. In reviewing the vaccination program, OMB did not accept the several  research and evaluation studies offered, since they did not meet all key  dimensions of “scope.” OMB acknowledged that the program had  conducted several management evaluations to see whether the program  could be improved but found their coverage narrow and concluded “there  have previously been no comprehensive evaluations looking at how well  the program is structured/managed to achieve its overall goals.” OMB also  did not accept an external Institute of Medicine evaluation of how the  government could improve its ability to increase immunization rates  because the evaluation report had not looked at the effectiveness of the  individual federal vaccine programs or how this program complemented  the other related programs. Conclusions

The PART review process has stimulated agencies to increase their  evaluation capacity and available information on program results. Agencies are  likely to design evaluations to meet their own needs—that is, in-depth  analyses that inform program improvement. If OMB wants evaluations  with a broader scope, such as information that helps determine a  program’s relevance or value, it will need to take steps to shape both  evaluation design and execution. Because agency evaluation resources tend to be limited, they are most  usefully focused on illuminating important areas of uncertainty. While  regular performance reporting is key to good program management and  oversight, requiring all federal programs to conduct frequent evaluation  studies is likely to result in many superficial reviews that will have little  utility and that will overwhelm agency evaluation capacity. Recommendations for Executive Action

In light of our findings and conclusions in this report, we are making the  following recommendations to OMB reiterating and expanding on  recommendations in our previous report:  OMB should encourage agencies to discuss their plans for program  evaluations—especially those in response to an OMB recommendation— with OMB and with congressional and other program stakeholders to  ensure that their findings will be timely, relevant, and credible and that  they will be used to inform policy and management decisions. Department of Health and Human Services Agency Reports

317 Immunization Program: RTI International.
Why GAO Did This Study

The Office of Management and Budget (OMB) designed the Program Assessment Rating Tool (PART) as a diagnostic tool to draw on program performance and evaluation information for forming conclusions about program benefits and recommending adjustments to improve results. To assess progress in improving the evidence base for PART assessments, GAO was requested to examine (1) agencies' progress in responding to OMB's recommendations to evaluate programs, (2) factors facilitating or impeding agencies' progress, and (3) whether agencies' evaluations appear to be designed to yield the information on program results that OMB expects.

What GAO Found

GAO examined agency progress on 20 of the 40 evaluations OMB recommended in its PART reviews at four federal agencies: the Department of Energy, Department of Health and Human Services, Department of Labor, and Small Business Administration. About half the programs GAO reviewed had completed an evaluation in the 2 years since those PART reviews were published; 4 more were in progress and 3 were still being planned. Program restructuring canceled plans for the remaining 2 evaluations. Several agencies struggled to identify appropriate outcome measures and credible data sources before they could evaluate program effectiveness. Evaluation typically competed with other program activities for funds, so managers may be reluctant to reallocate funds to evaluation. Some agency officials thought that evaluations should be targeted to areas of policy significance or uncertainty. However, all four agencies indicated that the visibility of an OMB recommendation brought agency management attention--and sometimes funds--to get the evaluations done. Moreover, by coordinating their evaluation activities, agencies met these challenges by leveraging their evaluation expertise and strategically prioritizing their evaluation resources to the studies that they considered most important. Because the OMB recommendations were fairly general, agencies had flexibility in interpreting the kind of information OMB expected. Some program managers disagreed with OMB on the purpose of their evaluations, their quality, and the usefulness of "independent" evaluations by third parties unfamiliar with their programs. Agency officials concerned about an increased focus on process said that they were more interested in learning how to improve program results than in meeting an OMB checklist. Since a few programs did not discuss their evaluation plans with OMB, it is not certain whether OMB will find their ongoing evaluations useful during the programs' next PART review. GAO concludes that the PART review process stimulated agencies to increase their evaluation capacity and available information on program results. Further, agencies are likely to design evaluations to meet their own needs--that is, in-depth analyses that inform program improvement. If OMB wants evaluations with a broader scope, such as information that helps determine a program's relevance or value, it will need to take steps to shape both evaluation design and execution. Finally, because agency evaluation resources tend to be limited, they are most usefully focused on important areas of uncertainty. Regular performance reporting is key to good management, but requiring all federal programs to conduct frequent evaluation studies is likely to result in superficial reviews of little utility and to overwhelm agency evaluation capacity.