The Department  of Education, HHS, and Department of Justice support six initiatives  similar to the Coalition’s to identify effective social interventions. Reviewers Defined the Top Tier Criteria through Case Discussion

The Top Tier initiative’s public materials are less transparent about the  process and criteria used to determine whether an intervention met the  Top Tier standard than about candidate selection. The Coalition reports that it plans to supplement guidance over time by  accumulating case decisions rather than developing more detailed  guidance on what constitutes sizable and sustained effects. Top Tier Follows Rigorous Standards but Is Limited for Identifying Effective Interventions

The Top Tier initiative’s criteria for assessing evaluation quality conform  to general social science research standards, but other features of the  overall process differ from common practice for drawing conclusions  about intervention effectiveness from a body of research. The major difference in rating study quality between the Top Tier initiative  and the six other initiatives is a product of the top tier standard as set out  in certain legislative provisions: the other initiatives accept well-designed,  well-conducted quasi-experimental studies as credible evidence. Broad Scope Fails to Focus on Effectiveness in Achieving Specific Outcomes

The Top Tier initiative’s topic selection, emphasis on long-term effects,  and narrow evidence criteria combine to provide limited information on  the effectiveness of approaches for achieving specific outcomes. In addition, the  exclusivity of its top tier standard also diverges from the more common  approach of rating the credibility of study findings along a continuum and  resulted in the panel’s recommending only 6 of 63 interventions for ages 0– 18 reviewed as providing “sizable, sustained effects on important life  outcomes.” Thus, although they are not their primary audience, the Top  Tier initiative provides practitioners with limited guidance on what works. Randomized Experiments Can Provide the Most Credible Evidence of Effectiveness under Certain Conditions

In our review of the literature on program evaluation methods, we found  general agreement that well-conducted randomized experiments are best  suited for assessing intervention effectiveness where multiple causal  influences lead to uncertainty about what has caused observed results but,  also, that they are often difficult to carry out. Randomized experiments are  considered best suited for interventions in which exposure to the  intervention can be controlled and the treatment and control groups’  experiences remain separate, intact, and distinct throughout the study. Rigorous Alternatives to Random Assignment Are Available

In our review of the literature on evaluation research methods, we  identified several alternative methods for assessing intervention  effectiveness when random assignment is not considered appropriate— quasi-experimental comparison group studies, statistical analyses of  observational data, and in-depth case studies. In general, they involve  collecting additional data and targeting comparisons to help rule out  plausible alternative explanations of the observed results. Reliable assessments of the credibility of evaluation results  require expertise in research design and measurement, but their reliability  can be improved by providing detailed guidance and training. Requiring evidence from randomized experiments as sole proof of an  intervention’s effectiveness is likely to exclude many potentially effective  and worthwhile practices for which random assignment is not practical. Deciding to adopt an intervention involves additional considerations— cost, ease of use, suitability to the local community, and available  resources. Agency and Third- Party Comments

The Coalition for Evidence-Based Policy provided written comments on a  draft of this report, reprinted in appendix II. The Coalition and the Departments of Education and Health  and Human Services provided technical comments that were incorporated  as appropriate throughout the text. Improving  Evaluation of Anticrime Programs. Abstinence Education: Efforts to Assess the Accuracy and Effectiveness  of Federally Funded Programs.
Why GAO Did This Study

Recent congressional initiatives seek to focus funds for certain federal social programs on interventions for which randomized experiments show sizable, sustained benefits to participants or society. The private, nonprofit Coalition for Evidence-Based Policy undertook the Top Tier Evidence initiative to help federal programs identify interventions that meet this standard. The Government Accountability Office (GAO) was asked to examine (1) the validity and transparency of the Coalition's process, (2) how its process compared to that of six federally supported efforts to identify effective interventions, (3) the types of interventions best suited for assessment with randomized experiments, and (4) alternative rigorous methods used to assess effectiveness. GAO reviewed documents, observed the Coalition's advisory panel deliberate on interventions meeting its top tier standard, and reviewed other documents describing the processes the federally supported efforts had used. GAO reviewed the literature on evaluation methods and consulted experts on the use of randomized experiments. The Coalition generally agreed with the findings. The Departments of Education and Health and Human Services provided technical comments on a draft of this report. The Department of Justice provided no comments.

What GAO Found

The Coalition's Top Tier Evidence initiative criteria for assessing evaluation quality conform to general social science research standards, but other features of its overall process differ from common practice for drawing conclusions about intervention effectiveness. The Top Tier initiative clearly describes how it identifies candidate interventions but is not as transparent about how it determines whether an intervention meets the top tier criteria. In the absence of detailed guidance, the panel defined sizable and sustained effects through case discussion. Over time, it increasingly obtained agreement on whether an intervention met the top tier criteria. The major difference in rating study quality between the Top Tier and the six other initiatives examined is a product of the Top Tier standard as set out in certain legislative provisions: the other efforts accept well-designed, well-conducted, nonrandomized studies as credible evidence. The Top Tier initiative's choice of broad topics (such as early childhood interventions), emphasis on long-term effects, and use of narrow evidence criteria combine to provide limited information on what is effective in achieving specific outcomes. The panel recommended only 6 of 63 interventions reviewed as providing "sizeable, sustained effects on important outcomes." The other initiatives acknowledge a continuum of evidence credibility by reporting an intervention's effectiveness on a scale of high to low confidence. The program evaluation literature generally agrees that well-conducted randomized experiments are best suited for assessing effectiveness when multiple causal influences create uncertainty about what caused results. However, they are often difficult, and sometimes impossible, to carry out. An evaluation must be able to control exposure to the intervention and ensure that treatment and control groups' experiences remain separate and distinct throughout the study. Several rigorous alternatives to randomized experiments are considered appropriate for other situations: quasi-experimental comparison group studies, statistical analyses of observational data, and--in some circumstances--in-depth case studies. The credibility of their estimates of program effects relies on how well the studies' designs rule out competing causal explanations. Collecting additional data and targeting comparisons can help rule out other explanations. GAO concludes that (1) requiring evidence from randomized studies as sole proof of effectiveness will likely exclude many potentially effective and worthwhile practices; (2) reliable assessments of evaluation results require research expertise but can be improved with detailed protocols and training; (3) deciding to adopt an intervention involves other considerations in addition to effectiveness, such as cost and suitability to the local community; and (4) improved evaluation quality would also help identify effective interventions.