Most of the Reviewed Overall, we found that 10 of the 15 evaluations that we reviewed could not NIJ Outcome Evaluations Could Not Produce Sufficiently Sound Information on Program Outcomes

produce sufficiently sound information about program outcomes. Six evaluations began with sufficiently sound designs, but encountered  implementation problems that would render their results inconclusive. An  additional 4 studies had serious methodological problems that from the  start limited their ability to produce reliable and valid results. Five studies  appeared to be methodologically rigorous in both their design and  implementation. Funding for these methodologically sound studies totaled about $7.5  million, or nearly 50 percent of the approximately $15.4 million spent on  the studies we reviewed. Funding for these studies with  implementation problems totaled about $3.3 million, or about 21 percent  of the approximately $15.4 million spent on the studies we reviewed. Funding for these studies that began with serious methodological  problems totaled about $4.7 million, or about 30 percent of the  approximately $15.4 million spent on the studies we reviewed. These 3 studies totaled about $3.7 million. However, DOJ program administrators  told us that they found some of the process and implementation findings  from the completed studies to be useful. We recognize that it is very difficult to design and execute outcome  evaluations that produce meaningful and definitive results. However, it is  too soon to tell whether and to what extent these efforts will lead to NIJ  funding more rigorous effectiveness evaluations, and result in NIJ  obtaining evaluative information that can better assist policy makers in  making decisions about criminal justice funding priorities. We are reporting on (1) the  methodological quality of a sample of completed and ongoing NIJ outcome  evaluation grants and (2) the usefulness of the evaluations in producing  information on program outcomes. Our review covered outcome evaluation grants managed by NIJ from 1992  through 2002. Because of  our interest in the effectiveness of criminal justice programs, we limited  our review of the usefulness of NIJ outcome evaluations to evaluations of  DOJ programs, or evaluations funded by DOJ program offices, and did not  examine the 3 other completed NIJ outcome evaluations that focused on  programs funded by agencies other than DOJ. These were  to be obtained from official agency records. Assessment of evaluation 	 The study has several limitations.
Why GAO Did This Study

Policy makers need valid, reliable, and timely information on the outcomes of criminal justice programs to help them decide how to set criminal justice funding priorities. In view of previously reported problems with selected outcome evaluations managed by the National Institute of Justice (NIJ), GAO assessed the methodological quality of a sample of completed and ongoing NIJ outcome evaluation grants.

What GAO Found

From 1992 through 2002, NIJ managed 96 evaluation studies that sought to measure the outcomes of criminal justice programs. Spending on these evaluations totaled about $37 million. Our methodological review of 15 of the 96 studies, totaling about $15 million and covering a broad range of criminal justice issues, showed that sufficiently sound information about program effects could not be obtained from 10 of the 15. Five studies, totaling about $7.5 million (or 48 percent of the funds spent on the studies we reviewed), appeared to be methodologically rigorous in both design and implementation, enabling meaningful conclusions to be drawn about program effects. Six studies, totaling about $3.3 million (or 21 percent of the funds spent on the studies we reviewed), began with sound designs but encountered implementation problems that would render their results inconclusive. An additional 4 studies, totaling about $4.7 million (or 30 percent of the funds spent on the studies we reviewed), had serious methodological limitations that from the start limited their ability to produce reliable and valid results. Although results from 5 completed studies were inconclusive, DOJ program administrators said that they found some of the process and implementation findings from them to be useful. We recognize that optimal conditions for the scientific study of complex social programs almost never exist, making it difficult to design and execute outcome evaluations that produce definitive results. However, the methodological adequacy of NIJ studies can be improved, and NIJ has taken several steps--including the formation of an evaluation division and funding feasibility studies--in this direction. It is too soon to tell whether these changes will lead to evaluations that will better inform policy makers about the effectiveness of criminal justice programs.