Under its contract,  NQF was scheduled to make a final recommendation to CMS prior to the  national reporting of quality indicators. Appropriateness of Quality Indicators Proposed for Public Reporting Is Unresolved

CMS’s initiative to augment existing public data on nursing home quality  has considerable merit but more time is needed to assure that the  indicators  proposed by CMS for public reporting are appropriate in terms  of their validity and reliability. Our review of available portions  of the Abt report, however, raised  serious questions about whether testing  and validation of the selected indicators has been sufficient to move  forward with national reporting at this time. CMS’s decision to implement national reporting in November 2002 is  troubling, given the issues raised by our  review of the available portions of  Abt’s validation report. CMS Has Not Addressed Concerns About the Underlying Accuracy of MDS Data Used to Develop Quality Indicators

Two different Abt studies have presented CMS with conflicting messages  about the accuracy of MDS data. Our analysis of the  data currently available in the six pilot states demonstrated the potential  for public confusion over both the quality indicators themselves and  inconsistencies with other available data on deficiencies identified during  nursing home surveys—which, to date, are the primary source of public  data on nursing home quality. Generally, hotline  staff did not express a preference for using either nursing home surveys or  quality indicators in choosing a nursing home. Thus, CMS focused the pilot evaluation on identifying improvements that  could be incorporated into the initiative’s design prior to the scheduled  national implementation in November 2002. NQF was asked to delay recommending a set of indicators for  national reporting until 2003, in part, to provide sufficient time for it to  review Abt’s report. CMS’s own evaluation of the  pilot, designed to help refine the initiative, was limited to fit CMS’s  timetable for the initiative and the preliminary finding were not available  until October 2002, leaving little time to incorporate the results into the  planned national rollout. We also have serious concerns about the  potential for public confusion over quality data, highlighting the need for  clear descriptions of the data’s limitations and easy access to informed  experts at both the Medicare and QIO hotlines. CMS has not yet  demonstrated its readiness to meet these consumer needs either directly or  through the QIOs. Recommendations for Executive Action

To ensure that publicly reported quality indicator data accurately reflect  the status of quality in nursing homes and fairly compare homes to one  another, we recommend that the Administrator of CMS delay the  implementation of nationwide reporting of quality indicators until  there is greater assurance  that the quality indicators are appropriate for  public reporting—including the validity of the indicators selected and  the use of an appropriate risk-adjustment methodology—based on input  from the NQF and other experts and, if necessary, additional analysis  and testing; and  a more thorough evaluation of the pilot is completed to help improve the  initiative’s effectiveness, including an assessment of the presentation of  information on the Web site and the resources needed to assist  consumers’ use of the information. Although CMS stated it would  use our report to help improve the initiative over time, it intends to move  forward with national implementation in November 2002 as planned. It  stated that “waiting for more reliability, more validity, more accuracy, and  more usefulness will delay needed public accountability, and deprive  consumers, clinicians, and providers of important information they can use  now.”  The NQF commented that it unequivocally supports CMS’s plans to  publicly report quality indicators but indicated that the initiative would  benefit from a short-term postponement of 3 to 4 months to achieve a  consensus on a set of indicators and to provide additional time to prepare  the public on how to use and interpret the data.
Why GAO Did This Study

GAO was asked to review the Centers for Medicare & Medicaid Services (CMS) initiative to publicly report additional information on its "Nursing Home Compare" Web site intended to help consumers choose a nursing home. GAO examined CMS's development of the new nursing home quality indicators and efforts to verify the underlying data used to calculate them. GAO also reviewed the assistance CMS offered the public in interpreting and comparing indicators available in its six-state pilot program, launched in April 2002, and its own evaluation of the pilot. The new indicators are scheduled to be used nationally beginning in November 2002.

What GAO Found

CMS's initiative to augment existing public data on nursing home quality has considerable merit, but its planned November 2002 implementation does not allow sufficient time to ensure the indicators are appropriate and useful to consumers. CMS's plan urges consumers to consider nursing homes with positive quality indicator scores, in effect, attempting to use market forces to encourage nursing homes to improve the quality of care. However, CMS is moving forward without adequately resolving important open issues on the appropriateness of the indicators chosen for national reporting or the accuracy of the underlying data. To develop and help select the quality indicators, CMS hired two organizations with expertise in health care data--Abt Associates, Inc. and the National Quality Forum (NQF). Abt identified a list of potential quality indicators and tested them to verify that they represented the actual quality of care individual nursing homes provide. GAO's review of the available portions of the report raised serious questions about the basis for moving forward with national reporting at this time. NQF, which was created to develop and implement a national strategy for measuring health care quality, was hired to review Abt's work and identify core indicators for national reporting. To allow sufficient time to review Abt's validation report, NQF agreed to delay its recommendations for national reporting until 2003. CMS limited its own evaluation of its six-state pilot program for the initiative so that the November 2002 implementation date could be met. Early results were expected in October 2002, leaving little time to incorporate them into the national rollout. Despite the lack of a final report from NQF and an incomplete pilot evaluation. CMS has announced a set of indicators it will begin reporting nationally in November 2002. GAO has serious concerns about the potential for public confusion by the quality information published, especially if there are significant changes to the quality indicators due to the NQF's review. CMS's proposed reporting format implies a precision in the data that is lacking at this time. While acknowledging this problem, CMS said it prefers to wait until after the national rollout to modify the presentation of the data. GAO's analysis of data currently available from the pilot states demonstrated there was ample opportunity for the public to be confused, highlighting the need for clear descriptions of the data's limitations and easy access to impartial experts hired by CMS to operate consumer hotlines. CMS has not yet demonstrated its readiness to meet these consumer needs either directly or through the hotlines fielding public questions about confusing or conflicting quality data. CMS acknowledged that further work is needed to refine its initiative, but believes that its indicators are sufficiently valid, reliable, and accurate to move forward with national implementation in November 2002 as planned.