CMS Has Processes for Checking Data Accuracy but Has No Ongoing Process to Check Completeness

CMS has processes for ensuring the accuracy of the quality data submitted  by hospitals for the APU program, but has no ongoing process to assess  whether hospitals are submitting complete data. Specifically, it reabstracts the quality data from  medical records for a sample of five patients per quarter for each hospital  and compares its results to the quality data submitted by hospitals. The  data are deemed to be accurate if there is 80 percent or greater agreement  between these two sets of results, a standard that hospitals had to meet for  the APU-measure set to qualify for their full annual payment update for  fiscal year 2006. However, these analyses did not address non-Medicare patient  records and the approach that CMS took in these analyses was not capable  of detecting incomplete data for all hospitals. We found a high overall baseline level of accuracy when we examined  CMS’s assessment of the data submitted by hospitals for the first two  calendar quarters of 2004. The median accuracy score exceeded   90 percent, which was well above the 80 percent accuracy threshold set by  CMS, and about 90 percent of hospitals met or exceeded that threshold for  both the first and the second calendar quarters of 2004. However, for approximately one-fourth to one-third of  all the hospitals that CMS assessed for accuracy, the statistical margin of  error for their accuracy score included both passing and failing accuracy  levels. Consequently, for these hospitals, the small number of cases that  CMS examined was not sufficient to establish with statistical certainty  whether the hospital met the threshold level of data accuracy. No Data Were Available to Provide Baseline Assessment of Completeness of Hospital Quality Data

There were no data available from which to estimate a baseline level of  completeness for the first two calendar quarters of data submitted for the  APU program. In contrast to the system of quarterly reabstractions  performed by CDAC to check the accuracy of quality data submitted by  hospitals, CMS did not conduct any corresponding assessment of the  extent to which all hospitals submitted data on all the cases, or a  representative sample of such cases, that met CMS’s eligibility criteria for  the first two calendar quarters of 2004. Other Reporting Systems Use Various Methods to Ensure Data Accuracy and Completeness, Notably an Independent Audit

Other reporting systems that collect clinical performance data have  adopted various methods to ensure data accuracy and completeness, and  officials from these systems stressed the importance of including an  independent audit in these activities. Most other reporting systems that conduct independent audits incorporate  three methods as part of their process that CMS does not use in its  independent audit. Specifically, they (1) include an on-site visit as part of  their independent audit, (2) focus their audits on a selected number of  facilities or reporting entities, and (3) review a minimum of 50 patient  medical records per reporting entity during the auditing process. Officials at other reporting systems we interviewed and an expert in the  field stressed the importance of the independent audit. Recommendations for Executive Action

In order for CMS to help ensure the reliability of the quality data it uses to  produce information on hospital performance, we recommend that the  CMS Administrator undertake the following three actions:  focusing on the subset of hospitals for which it is statistically uncertain if  they met CMS’s accuracy threshold in one or more previous quarters,  increase the number of patient records reabstracted by CDAC in a  subsequent quarter so that the proportion of hospitals with statistically  uncertain results is reduced;  require hospitals to certify that they took steps to ensure that they  submitted data on all eligible patients, or a representative sample thereof;  and    assess the level of incomplete data submitted by hospitals for the APU  program to determine the magnitude of underreporting, if any, in order to  refine how completeness assessments may be done in future reporting  efforts. GAO staff who made major contributions to this report  are listed in appendix V.

Appendix I: Scope and Methodology

To determine the processes used by the Centers for Medicare & Medicaid  Services (CMS) to ensure the accuracy and completeness of data  submitted by hospitals for the Annual Payment Update program (APU  program), we interviewed both CMS officials and staff at DynKePRO— which operates the Clinical Data Abstraction Center (CDAC)—and the  Iowa Foundation for Medical Care (IFMC), two contractors that perform  data collection and data quality monitoring tasks for the APU program.
Why GAO Did This Study

The Medicare Modernization Act of 2003 directed that hospitals lose 0.4 percent of their Medicare payment update if they do not submit clinical data for both Medicare and non-Medicare patients needed to calculate hospital performance on 10 quality measures. The Centers for Medicare & Medicaid Services (CMS) instituted the Annual Payment Update (APU) program to collect these data from hospitals and report their rates on the measures on its Hospital Compare Web site. For hospital quality data to be useful to patients and other users, they need to be reliable, that is, accurate and complete. GAO was asked to (1) describe the processes CMS uses to ensure the accuracy and completeness of data submitted for the APU program, (2) analyze the results of CMS's audit of the accuracy of data from the program's first two calendar quarters, and (3) describe processes used by seven other organizations that assess the accuracy and completeness of clinical performance data.

What GAO Found

CMS has contracted with an independent medical auditing firm to assess the accuracy of the APU program data submitted by hospitals, but has no ongoing process in place to assess the completeness of those data. CMS's independent audit checks accuracy by comparing the quality data submitted by hospitals from the medical records for a sample of five patients per calendar quarter for each hospital to the quality data that the contractor has reabstracted from the same records. The data are deemed to be accurate if there is 80 percent or greater agreement between these two sets of results. CMS has established no ongoing process to check data completeness. For the payment updates for fiscal years 2005 and 2006, CMS compared the number of cases submitted by a hospital to the number of Medicare claims that hospital submitted. However, these analyses did not address non-Medicare patient records, and the approach that CMS took in these analyses was not capable of detecting incomplete data for all hospitals. Although GAO found a high overall baseline level of accuracy when it examined CMS's assessment of the data submitted for the first two quarters of the APU program, the results are statistically uncertain for up to one-third of hospitals, and a baseline level of data completeness cannot be determined. The median accuracy score of 90 to 94 percent--depending on the calendar quarter and measures used--was well above the 80 percent accuracy threshold set by CMS, and about 90 percent of hospitals met or exceeded that threshold for both the first and the second calendar quarters of 2004. However, for approximately one-fourth to one-third of all the hospitals that CMS assessed for accuracy, the statistical margin of error for their accuracy score included both passing and failing accuracy levels. Consequently, for these hospitals, the small number of cases that CMS examined was not sufficient to establish with statistical certainty whether they met the accuracy threshold set by CMS. With respect to completeness of data, CMS did not assess the extent to which all hospitals submitted data on all eligible patients, or a representative sample thereof, for the two baseline quarters. As a result, there were no data from which to derive an assessment of the baseline level of completeness of the quality data that hospitals submitted for the APU program. Other reporting systems that collect clinical performance data have adopted a range of activities to ensure data accuracy and completeness, which include some methods employed by all, such as checking the data electronically to identify missing data. Officials from some of the other reporting systems and an expert in the field stressed the importance of including an independent audit in the methods used by organizations to check data accuracy and completeness. Most of the other reporting systems incorporate three methods into their process that CMS does not use in its independent audit. Specifically, most include an on-site visit in their independent audit, focus their audits on a selected number of facilities, and review a minimum of 50 patient medical records during the audit.