The CFSR Is a Valuable yet Substantial Undertaking, but Data Enhancements Could Improve Its Reliability

ACF and many state officials perceive the CFSR as a valuable process— highlighting many areas needing improvement—and a substantial  undertaking, but some state officials and child welfare experts told us that  data enhancements could improve its reliability. Further, 26 of the 36 states responding to a relevant question in  our survey commented that they generally or completely agreed with the  results of the final CFSR report, even though none of the 41 states with  final CFSR reports released through 2003 has achieved substantial  conformity on all 14 outcomes and systemic factors. In addition, both ACF  and the states have dedicated substantial financial and staff resources to  the process. However, several state officials and child welfare experts we  interviewed questioned the accuracy of the data used to compile state  profiles and establish the national standards. While ACF officials in the  central office contend that stakeholder interviews and case reviews  compliment the data profiles, many state officials and experts reported  that additional data from the statewide assessment could bolster the  evaluation of state performance. ACF staff  in 8 of the 10 regions considered the CFSR a helpful tool to improve  outcomes for children. All of the states we visited experienced discrepant findings between the  aggregate data from the statewide assessment and the information  obtained from the on-site review, which complicated ACF’s determination  of states’ performance. Further, at least 9 states responding to our  survey indicated that insufficient time, funding, and staff, as well as high  caseloads, were the greatest challenges to PIP implementation. As states  progress in PIP implementation, some ACF officials expressed a need for  more guidance on how to monitor state accomplishments, and both ACF  and state officials were uncertain about how the estimated financial  penalties would be applied if states fail to achieve the goals described in  their plans. ACF’s Focus Rests Almost Exclusively on Implementing the CFSR

To implement the CFSRs, ACF has focused its activities almost entirely on  the four phases of the review process. However, staff in several regions  report limitations in providing assistance to states in helping them to meet  key federal goals. While ACF officials in the central office said that the CFSR has become the  primary method for evaluating states’ performance, they acknowledged  that regional staff might still be adjusting to the new way ACF oversees  child welfare programs. We are sending copies of this report to the Secretary of Health and Human  Services, state child welfare directors, and other interested parties. Appendix I: Objectives, Scope, and Methodology

	Objectives

The objectives of our study were to report on (1) ACF’s and the states’  experiences preparing for and conducting the statewide assessments and  on-site reviews; (2) ACF’s and the states’ experiences developing, funding,  and implementing items in the PIP; and (3) additional efforts that ACF has  taken beyond the CFSR to ensure that all states meet federal goals of  safety, permanency, and well-being.
Why GAO Did This Study

In 2001, the Department of Health and Human Services' (HHS) Administration for Children and Families (ACF) implemented the Child and Family Services Reviews (CFSR) to increase states' accountability. The CFSR uses states' data profiles and statewide assessments, as well as interviews and an on-site case review, to measure state performance on 14 outcomes and systemic factors, including child well-being and the provision of caseworker training. The CFSR also requires progress on a program improvement plan (PIP); otherwise ACF may apply financial penalties. This report examines (1) ACF's and the states' experiences preparing for and conducting the statewide assessments and on-site reviews; (2) ACF's and the states' experiences developing, funding, and implementing items in PIPs; and (3) any additional efforts that ACF has taken beyond the CFSR to help ensure that all states meet federal goals related to children's safety, permanency, and well-being.

What GAO Found

ACF and many state officials perceive the CFSR as a valuable process and a substantial undertaking, but some data enhancements could improve its reliability. ACF staff in 8 of the 10 regions considered the CFSR a helpful tool to improve outcomes for children. Further, 26 of 36 states responding to a relevant question in our survey commented that they generally or completely agreed with the results of the final CFSR report, even though none of the 41 states with final CFSR reports released through 2003 has achieved substantial conformity on all 14 outcomes and systemic factors. Additionally, both ACF and the states have dedicated substantial financial and staff resources to the process. Nevertheless, several state officials and child welfare experts we interviewed questioned the accuracy of the data used in the review process. While ACF officials contend that stakeholder interviews and case reviews complement the data profiles, many state officials and experts reported that additional data from the statewide assessment could bolster the evaluation of state performance. Program improvement planning is under way, but uncertainties have affected the development, funding, and implementation of state PIPs. Officials from 3 of the 5 states we visited said ACF's PIP-related instructions were unclear, and at least 9 of the 25 states reporting on PIP implementation in our survey said that insufficient funding and staff were among the greatest challenges. While ACF has provided some guidance, ACF and state officials remain uncertain about PIP monitoring efforts and how ACF will apply financial penalties if states fail to achieve their stated PIP objectives. Since 2001, ACF's focus has been almost exclusively on the CFSRs and regional staff report limitations in providing assistance to states in helping them to meet key federal goals. While staff from half of ACF's regions told us they would like to provide more targeted assistance to states, and state officials in all 5 of the states we visited said that ACF's existing technical assistance efforts could be improved, ACF officials acknowledged that regional staff might still be adjusting to the new way ACF oversees child welfare programs.