3. CMS Used Input from External Entities on the Development and Implementation of the Five-Star System, Which Included Three Key Methodological Decisions

CMS developed and implemented the Five-Star System largely during an  8-month period with input from long-term care stakeholders, its Five-Star  System contractor, and members of a technical expert panel. CMS’s contractor established this panel, composed of nine members that  the contractor and CMS identified as experts in long-term care and that  included researchers and an industry representative, to help guide the  development and implementation of the Five-Star System. CMS made numerous methodological decisions during the development  of the Five-Star System, including three key methodological decisions. The other two key methodological decisions pertain to how to  create ratings that account for variation in the type of care provided  across nursing homes. One key methodological decision that CMS made was  deciding how to best combine the health inspection, quality measure,  and staffing component ratings to create an overall rating. CMS Typically Considers Modifying the Five-Star System in Response to Stakeholders, after Routine Monitoring, and When New Data Sources Become Available

CMS generally considers modifying the Five-Star System in response to  (1) methodological issues raised by stakeholders, (2) its routine  monitoring of the system, and (3) the availability of new data sources. CMS officials explained that when a methodological issue is raised by  long-term care stakeholders, they review the Five-Star System to  determine whether modifications should be made. Officials further said  that each issue raised does not always result in modifications to the Five- Star System, although some minor modifications have been made. CMS also considers making modifications to the Five-Star System based  on its periodic analysis of the system’s rating trends; however, to date, no  modifications have been made based on these analyses. intended to help CMS evaluate the Five-Star System and determine if  modifications are needed. However, CMS has not taken steps to ensure that these efforts  will help CMS achieve its goals for the Five-Star System—to inform  consumers and improve provider quality. These  plans include evaluating its usability, adding information on nursing home  capability, revising the staffing component, and developing additional  quality measures. Specifically, while CMS officials have given us broad estimates for  when they anticipate some of these efforts to be undertaken, CMS does  not have planning documents or strategies that outline specific milestones  and timelines associated with implementing the agency’s planned efforts  to improve the Five-Star System. We have found, in our prior work, that  developing and using specific milestones and timelines to guide and  gauge progress toward achieving an agency’s desired results is a leading  practice for effective strategic planning and management. CMS Has Not Established How Its Planned Efforts Will Help Achieve Goals

CMS has not established, through planning activities or resulting planning  documents, how its planned efforts to improve the Five-Star System will  help CMS achieve the goals of the system—to inform consumers and  improve provider quality. As a result, CMS may not be identifying and  prioritizing its efforts in a manner that best ensures that the goals are  being achieved. Recommendations for Executive Action

In order to strengthen CMS’s efforts to improve the Five-Star System, we  recommend that the Administrator of CMS use strategic planning  practices to: establish—through planning documents—how its planned efforts will  help CMS achieve the goals of the Five-Star System, and develop milestones and timelines for each of its planned efforts. Appendix I: Methodology for Identifying Key Methodological Decisions Made during Five- Star Quality Rating System Development

To identify the key methodological decisions made during the  development of the Five-Star Quality Rating System (Five-Star System),  defined as those that caused the most intense discussion and review  according to at least six members of the Centers for Medicare & Medicaid  Services’ (CMS) technical expert panel, we solicited the views of panel  members through a series of interviews and questionnaires. 2. Appendix II: Overview of CMS’s Five-Star Quality Rating System Methodology

In the Five-Star System, nursing homes are assigned ratings for three  components—health inspections, staffing, and quality measures—and an  overall rating. These ratings range from one star to five stars, with more  stars indicating higher quality.
Why GAO Did This Study

In 2008, in an effort to provide helpful information to consumers and improve provider quality, the Centers for Medicare & Medicaid Services (CMS) developed and implemented the Five-Star Quality Rating System (Five-Star System). The Five-Star System assigns each nursing home an overall rating and three component ratingshealth inspections, staffing, and quality measuresbased on the extent to which the nursing home meets CMSs quality standards and other measures. The rating scale ranges from one to five stars, with more stars indicating higher quality.
The Patient Protection and Affordable Care Act directed GAO to review CMSs Five-Star System. This report examines (1) how CMS developed and implemented the Five-Star System and what key methodological decisions were made during development, (2) the circumstances under which CMS considers modifying the Five-Star System, and (3) the extent to which CMS has established plans to help ensure it achieves its goals for the Five-Star System. To conduct this work, GAO reviewed CMS documents, interviewed CMS officials and others, and assessed whether CMS uses certain strategic planning practices.

What GAO Found

CMS developed and implemented the Five-Star System largely during an 8-month period in 2008 with input from long-term care stakeholders, CMSs Five-Star System contractor, and members of a technical expert panela panel composed of nine individuals that CMS identified as experts in long-term care research. CMS made numerous methodological decisions during the development of the Five-Star System, including three key methodological decisions. GAO defines key methodological decisions as those that at least six technical expert panel membersof the nine that GAO contactedrecalled as eliciting the most intense review and discussion during the development of the Five-Star System. One key methodological decision was how to combine the component ratings to create an overall rating. The other two key methodological decisions pertained to how to create ratings that account for variation in the type of care provided across nursing homes.
CMS generally considers modifying the Five-Star System in response to (1) methodological issues raised by stakeholders, (2) its routine monitoring of the system, and (3) the availability of new data sources. CMS officials explained that when a methodological issue is raised by long-term care stakeholders, they review the Five-Star System to determine whether modifications should be made. Officials said that each issue raised does not always result in modifications to the Five-Star System, although some minor modifications have been made. CMS also considers making modifications to the Five-Star System based on its periodic analyses of trends of the system; however, to date, no modifications have been made based on these analyses. Lastly, CMS is currently determining how to modify the staffing and quality measure ratings of the Five-Star System based on newly available data.
CMS has several planned efforts intended to improve the Five-Star System, including evaluating the usability of the system, adding nursing home capability information, revising the staffing component, and developing additional quality measures. However, CMS lacks GAO-identified leading strategic planning practicesthe use of milestones and timelines to guide and gauge progress toward achieving desired results and the alignment of activities, resources, and goalsthat could help the agency to more efficiently and effectively accomplish its planned efforts intended to improve the Five-Star System. While CMS officials have given us broad estimates for when they anticipate some of these efforts to be implemented, CMS does not have milestones and timelines associated with implementing the efforts, which could help ensure that appropriate progress is made towards implementation. In addition, CMS has not established, through planning documents, how its planned efforts to improve the Five-Star System will help CMS achieve the goals of the systemto inform consumers and improve provider quality. As a result, CMS may not be identifying and prioritizing its intended improvements in a manner that best ensures that the goals are being achieved.

What GAO Recommends

GAO recommends that the Administrator of CMS use strategic planning to establish how its planned efforts will help meet the goals of the Five-Star System, and develop milestones and timelines for each of its planned efforts. CMS agreed with these recommendations.