The CFSR Is a Valuable Yet Substantial Undertaking, but Data Enhancements Could Improve Its Reliability

ACF and many state officials perceive the CFSR as a valuable process— highlighting many areas needing improvement—and a substantial  undertaking, but some state officials and child welfare experts told us that  data enhancements could improve its reliability. ACF staff in 8 of the 10  regions considered the CFSR a helpful tool to improve outcomes for  children. Further, 26 of the 36 states responding to a relevant question in  our survey commented that they generally or completely agreed with the  results of the final CFSR report, even though none of the 41 states with  final CFSR reports released through 2003 has achieved substantial  conformity on all 14 outcomes and systemic factors. In addition, both ACF  and the states have dedicated substantial financial and staff resources to  the process. However, several state officials and child welfare experts we  interviewed questioned the accuracy of the data used to compile state  profiles and establish the national standards. While ACF officials in the  central office contend that stakeholder interviews and case reviews  compliment the data profiles, many state officials and experts reported  that additional data from the statewide assessment could bolster the  evaluation of state performance. ACF and the States Report That Reviews Have Been a Substantial Undertaking

Given the value that ACF and the states have assigned to the CFSR  process, both have spent substantial financial resources and staff time to  prepare for and implement the reviews. All of the states we visited experienced discrepant findings between the  aggregate data from the statewide assessment and the information  obtained from the on-site review. As  states progress in PIP implementation, some ACF officials expressed a  need for more guidance on how to monitor state accomplishments, and  both ACF and state officials were uncertain about how the estimated  financial penalties would be applied if states fail to achieve the goals  described in their plans. Officials in the  states we visited echoed survey respondents’ concerns with officials from  3 of the 5 states informing us that ACF had given states different  instructions regarding acceptable PIP format and content. ACF’s Focus Rests Almost Exclusively on Implementing the CFSR

To implement the CFSRs, ACF has focused its activities almost entirely on  the CFSR review process, and regional staff report limitations in providing  assistance to states in helping them to meet key federal goals. To ensure that ACF uses the best available data in measuring state  performance, we recommended in our April 2004 report that the Secretary  of HHS expand the use of additional data states may provide in their  statewide assessments and consider alternative data sources when  available, such as longitudinal data that track children’s placements over  time, before making final CFSR determinations. In addition, to ensure that  ACF regional offices and states fully understand the PIP development,  approval, and monitoring processes, and that regional offices fully  understand ACF’s prioritization of the CFSR as the primary mechanism for  child welfare oversight, we recommended that the Secretary of HHS issue  clarifying guidance on the PIP process and evaluate states’ and regional  offices’ adherence to this instruction and provide guidance to regional  offices explaining how to better integrate the many training and technical  assistance activities for which they are responsible, such as participation  in state planning meetings and the provision of counsel to states on  various topics, with their new CFSR responsibilities. In response to the  first recommendation, HHS acknowledged that the CFSR is a new process  that continues to evolve, and also noted several steps it has taken to  address the data quality concerns we raise in our report. Child and Family Services Reviews: Better Use of Data and Improved  Guidance Could Enhance HHS’s Oversight of State Performance. District of Columbia Child Welfare: Long-Term Challenges to Ensuring  Children’s Well- Being.
Why GAO Did This Study

In 2001, the Department of Health and Human Services' (HHS) Administration for Children and Families (ACF) implemented the Child and Family Services Reviews (CFSR) to increase states' accountability. The CFSR uses states' data profiles and statewide assessments, as well as interviews and an on-site case review, to measure state performance on 14 outcomes and systemic factors, including child well-being and the provision of caseworker training. The CFSR also requires progress on a program improvement plan (PIP); otherwise ACF may apply financial penalties. This testimony is based on our April 2004 report and addresses (1) ACF's and the states' experiences preparing for and conducting the statewide assessments and on-site reviews; (2) ACF's and the states' experiences developing, funding, and implementing items in PIPs; and (3) any additional efforts that ACF has taken beyond the CFSR to improve state performance. For the April 2004 report, we surveyed all 50 states, the District of Columbia, and Puerto Rico regarding their experiences throughout the CFSR process, visited 5 states to obtain first-hand information, and conducted a content analysis of all 31 available PIPs as of January 1, 2004. We also interviewed HHS officials--including those in all 10 regional offices--and key child welfare experts.

What GAO Found

ACF and many state officials perceive the CFSR as a valuable process and a substantial undertaking, but some data enhancements could improve its reliability. ACF staff in 8 of the 10 regions considered the CFSR a helpful tool to improve outcomes for children. Further, 26 of 36 states responding to a relevant question in our survey commented that they generally or completely agreed with the results of the final CFSR report, even though none of the 41 states with final CFSR reports released through 2003 has achieved substantial conformity on all 14 outcomes and systemic factors. Additionally, both ACF and the states have dedicated substantial financial and staff resources to the process. Nevertheless, several state officials and child welfare experts we interviewed questioned the accuracy of the data used in the review process. While ACF officials contend that stakeholder interviews and case reviews complement the data profiles, many state officials and experts reported that additional data from the statewide assessment could bolster the evaluation of state performance. Program improvement planning is under way, but uncertainties have affected the development, funding, and implementation of state PIPs. Officials from 3 of the 5 states we visited said ACF's PIP-related instructions were unclear, and at least 9 states reported in our survey that challenges to implementing their plans include insufficient funding, staff, and time. While ACF has provided some guidance, ACF and state officials remain uncertain about PIP monitoring efforts and how ACF will apply financial penalties if states fail to achieve their stated PIP objectives. Since 2001, ACF's focus has been almost exclusively on the CFSRs and regional staff report limitations in providing assistance to states in helping them to meet key federal goals. While staff from half of ACF's regions told us they would like to provide more targeted assistance to states, and state officials in all 5 of the states we visited said that ACF's existing technical assistance efforts could be improved, ACF officials acknowledged that regional staff might still be adjusting to the new way ACF oversees child welfare programs. In the April 2004 report, we recommended that the Secretary of HHS ensure that ACF uses the best available data to measure state performance. We also recommended that the Secretary clarify PIP guidance and provide guidance to regional officials on how to better integrate their many oversight responsibilities. In commenting on a draft of the April 2004 report, HHS acknowledged that the CFSR is a new process that continues to evolve, and noted several steps it has taken to address the data quality concerns we raised in that report.