In fiscal year 2015, federal spending under demonstrations represented a  third of all Medicaid spending nationwide. 1.) CMS officials told us that in spring 2017, CMS began  requiring states to submit a comprehensive evaluation report for  demonstrations in its high priority policy areas for evaluation at the end of  each demonstration cycle, rather than after the expiration of the  demonstration. However, CMS had  not established written procedures for implementing this new policy. Further, CMS’s efforts  to improve the quality of state-led evaluations and its plan to require final  reports after each demonstration cycle are consistent with evaluation  guidance from the American Evaluation Association that recommends  that federal agencies conduct evaluations of public programs and policies  throughout the programs’ life cycles, not just at their end, and that  agencies use evaluations to improve programs and assess their  effectiveness. Further, though CMS has begun indicating to  states, including those with comprehensive demonstrations, that the  agency may allow less rigorous evaluations for certain types of  demonstration approaches, CMS has not established timeframes for  issuing the criteria defining those conditions. Ongoing Federal Evaluations Led by CMS Have Been Limited by Data Challenges and It Is Uncertain When Results Will Be Available

Data and other challenges have significantly limited the scope and  progress of CMS’s large, multi-state evaluation and the agency’s  evaluation of Indiana’s demonstration. Further, CMS has not released  available evaluation results from the multi-state evaluation nor set  timeframes for making these and future federal evaluation findings public. These data  challenges included limitations in the quality of CMS data and delays  obtaining data directly from states. These limitations caused CMS to  narrow the evaluation’s scope, often by reducing the number of state  demonstrations evaluated or limiting what was being examined. (See table 3.) Timeframes for the completion  and release of final evaluation results are even more uncertain, both  because of the delays in the evaluation progress and because CMS has  no standard policy for timeframes for releasing evaluation results. CMS officials stated that federal evaluations provide a  unique cross-state perspective that states typically do not have the  capacity to provide in their own state-led evaluations; however, if these  reports are not made public in a timely fashion, opportunities may be  missed to inform federal and state policymakers and other stakeholders  on the effects of Medicaid demonstrations. Conclusions

Section 1115 demonstrations have long been an important tool for  providing states with the flexibility to test new approaches to providing  and financing Medicaid coverage. CMS’s efforts since 2014 to improve the usefulness of  evaluations in informing state and federal Medicaid policy decisions have  promise. However, CMS has not  established written procedures for requiring final, comprehensive  evaluation reports at the end of each cycle for all demonstrations, a key  step in improving the usefulness of state-led evaluations. By  not making the results of the federal evaluations public in a timely  manner, CMS is missing an opportunity to inform important policy  discussions happening at the state and federal levels. (Recommendation 1)
The Administrator of CMS should issue written criteria for when CMS  will allow limited evaluation of a demonstration or a portion of a  demonstration, including defining conditions, such as what it means  for a demonstration to be longstanding or noncomplex, as applicable. HHS  concurred with all three recommendations. Regarding our third recommendation that  CMS establish and implement a policy for publicly releasing findings from  federal evaluations of demonstrations, HHS said that CMS is in the  process of establishing such a policy. At that time, we will send copies of this report to the  Secretary of Health and Human Services, appropriate congressional  committees, and other interested parties. Appendix I: Characteristics of Selected States’ 1115 Demonstrations

The Medicaid section 1115 demonstrations (referred to as  demonstrations) in our eight selected states varied in terms of the number  of years the demonstrations had been in effect and cost, among other  things.
Why GAO Did This Study

Demonstrations—which represented roughly a third of the more than $300 billion in federal Medicaid spending in 2015—are a powerful tool to test new approaches to providing coverage and delivering Medicaid services that could reduce costs and improve beneficiaries' outcomes. Evaluations are essential to determining whether demonstrations are having their intended effects. States are required to evaluate their demonstrations and CMS can initiate its own federal evaluations of demonstrations.
GAO was asked to examine evaluations of demonstrations, including how the results have been used to inform Medicaid policy. This report examines (1) state-led evaluations and (2) federal evaluations. GAO reviewed evaluation documentation for eight states with high demonstration expenditures that varied in the number of years their demonstrations had been in effect and by geography. GAO also reviewed documentation for the ongoing federal evaluations and interviewed state and federal Medicaid officials. GAO assessed evaluation practices against federal standards for internal control and leading evaluation guidelines.

What GAO Found

Under section 1115 of the Social Security Act, the Secretary of Health and Human Services (HHS) may approve Medicaid demonstrations to allow states to test new approaches to providing coverage and for delivering services that can transform large portions of states' programs. However, GAO found that selected states' evaluations of these demonstrations often had significant limitations that affected their usefulness in informing policy decisions. The limitations included gaps in reported evaluation results for important parts of the demonstrations. (See table.) These gaps resulted, in part, from HHS's Centers for Medicare & Medicaid Services (CMS) requiring final, comprehensive evaluation reports after the expiration of the demonstrations rather than at the end of each 3- to 5-year demonstration cycle. CMS has taken a number of steps since 2014 to improve the quality of state-led evaluations, and in October 2017, officials stated that the agency planned to require final reports at the end of each demonstration cycle for all demonstrations. However, the agency has not established written procedures for implementing such requirements, which could allow for gaps to continue. CMS also plans to allow states to conduct less rigorous evaluations for certain types of demonstrations but has not established criteria defining under what conditions limited evaluations would be allowed.
Federal evaluations led by CMS have also been limited due to data challenges that have affected the progress and scope of the work. For example, delays obtaining data directly from states, among other things, led CMS to considerably reduce the scope of a large, multi-state evaluation, which was initiated in 2014 to examine the impact of state demonstrations in four policy areas deemed to be federal priorities. Though CMS has made progress in obtaining needed data, it is uncertain when results from the multi-state and other federal evaluations will be available to policymakers because CMS has no policy for making results public. By not making these results public in a timely manner, CMS is missing an opportunity to inform important federal and state policy discussions.

What GAO Recommends

GAO recommends that CMS: (1) establish written procedures for requiring final evaluation reports at the end of each demonstration cycle, (2) issue criteria for when it will allow limited evaluations of demonstrations, and (3) establish a policy for publicly releasing findings from federal evaluations of demonstrations. HHS concurred with these recommendations.