NQF Made Progress on Projects under Each of the Contract Activities as of August 31, 2011

From January 14, 2010, through August 31, 2011, NQF has made  progress on 60 of the 63 projects under the activities required under its  contract with HHS. NQF has made progress on three projects related  to retooling—that is, converting previously endorsed quality measures  to an electronic format that is compatible with electronic health  records (EHR). Other Health Care Quality Measurement Activity. NQF Did Not Meet Expected Time Frames and Exceeded Cost Estimates

Our review of NQF documents found that NQF had not met or did not  expect to meet time frames on more than half of the projects under the  contract activities that were completed or ongoing, as of August 2011. For example, NQF was expected to complete its  initial retooling of 113 endorsed quality measures into electronic  formats by September 2010, but this effort was not completed until  December 2010. HHS officials stated that the first set of 44  retooled measures submitted had errors that required correction. For example, the delays in projects related to the EHR contract  activity, including expanding the scope of the retooling project,  contributed to NQF exceeding its cost estimate of about $3.8 million for  the entire EHR contract activity by about $560,000 in the second contract  year. While HHS monitored NQF’s progress  and approved changes to the time frames and cost estimates for the  projects under the contract activities, HHS did not use available tools for  monitoring that are required under NQF’s contract. These tools could  have helped to provide an opportunity for HHS to make any appropriate  changes to NQF’s projects. HHS officials told us that,  prior to August 2011, they had not enforced a contractual requirement for  NQF to submit—nor had it received from NQF—a financial graph in its  monthly progress reports that provides information comparing NQF’s  monthly incurred costs for each of the contract activities with initial cost  estimates. Specifically, NQF reported about $12.8 million in  total costs and fixed fees for the contract activities it performed during the  second contract year—January 14, 2010, through January 13, 2011. HHS Used or Planned to Use Some NQF Measures Received under the Contract but Has Not Comprehensively Planned for Other Measurement Needs to Implement PPACA

For its various programs or initiatives, HHS has used or planned to use  about one-half of the quality measures that NQF has endorsed,  maintained, or retooled under the contract, as of August 31, 2011, and  HHS officials expect to evaluate if and how the remaining measures will  be used. HHS officials told us that these 44 measures are  being used but have not yet been tested to assess the feasibility of  implementing them in the electronic format. For example, HHS officials told us that they will  consider implementation of most of the retooled measures in future  stages of the EHR Incentive Program. HHS Does Not Have a Comprehensive Plan for Determining How It Will Use NQF’s Work under the Contract to Implement PPACA Requirements Related to Quality Measurement

Although HHS has taken steps to determine how it can use the measures  received under the contract with NQF, the agency does not have a  comprehensive plan for determining how it will use the remainder of the  work conducted under NQF’s contract to implement PPACA  requirements, including plans for additional quality measures that need to  be endorsed during the remaining contract years. Without such a  plan, HHS may be limited in its efforts to prioritize which specific  measures it needs to develop and have endorsed by NQF for its health  care quality programs and initiatives established by PPACA. As a result,  HHS may be unable to ensure that the agency receives the quality  measures needed to meet PPACA requirements and specified time  frames related to quality measurement. For more than half of  the projects, including all five projects in the endorsement activity, NQF  did not meet or did not expect to meet the initial time frames approved by  HHS. Until the  testing is completed, HHS runs the risk that some of the retooled  measures may not work as intended when implemented in electronic  format for performance measurement, which is a concern because use of  these measures is an important component of HHS’s long-term goal for  providers to use health information technology (IT) to exchange  information and improve the quality of care. Recommendations for Executive Action

To help ensure that HHS receives the quality measures it needs to  effectively implement its quality measurement programs and initiatives  within required time frames, we recommend that the Secretary of HHS  take the following three actions: use monitoring tools required under the NQF contract to obtain  detailed and timely information on NQF’s performance and use that  information to inform any appropriate changes to time frames,  projects, and cost estimates for the remaining contract years; ensure that testing of the electronic versions of the measures retooled  by NQF that are being used or are planned for use in the Medicare  and Medicaid EHR Incentive programs is completed in a timely  manner to help identify potential errors and address issues of  implementation; and develop a comprehensive plan that identifies the quality measurement  needs of HHS programs and initiatives, including PPACA  requirements, and provides a strategy for using the work NQF  performs under the contract to help meet these needs. HHS neither agreed nor disagreed with our recommendations  and provided general comments. NQF concurred with many of the  findings in the report and provided clarification and additional context on  the findings and recommendations.
Why GAO Did This Study

The Medicare Improvements for Patients and Providers Act of 2008 (MIPPA) directed the Department of Health and Human Services (HHS) to enter into a 4-year contract with an entity to perform various activities related to health care quality measurement. In January 2009, HHS awarded a contract to the National Quality Forum (NQF), a nonprofit organization that endorses health care quality measuresthat is, recognizes certain ones as national standards. In 2010, the Patient Protection and Affordable Care Act (PPACA) established additional duties for NQF. This is the second of two reports MIPPA required GAO to submit on NQFs contract with HHS. In this reportwhich covers NQFs performance under the contract from January 14, 2010, through August 31, 2011GAO examines (1) the status of projects under NQFs required contract activities and (2) the extent to which HHS used or planned to use the measures it has received from NQF under the contract to meet its quality measurement needs, as of August 2011. GAO interviewed NQF and HHS officials, reviewed relevant laws, and reviewed HHS and NQF documents.

What GAO Found

NQF has made progress on projects under its contract activities, as of August 2011. Specifically, NQF has completed or made progress on 60 of 63 projects. For example, NQF has completed projects to endorse measures related to various topics, including nursing homes. However, for more than half of the projects, NQF did not meet or did not expect to meet the initial time frames approved by HHS. For example, NQF completed one project to retool measuresthat is, convert previously endorsed quality measures to an electronic format. While the retooling project was expected to be completed by September 2010, its completion was delayed by 3 months. NQF and HHS officials identified various reasons that contributed to this delay, including an expansion of the projects scope and complexity. As a result of the delay, HHS did not have all the retooled measures it expected to include in its Electronic Health Records (EHR) Incentive Program. The delay of this project was also a contributing factor to NQF exceeding its estimated cost for its entire contract activity related to EHR by about $560,000 in the second contract year January 14, 2010, through January 13, 2011. While HHS monitored NQFs progress through monthly progress reports and approved changes to time frames and costs, HHS did not use all of the tools for monitoring that are required under the contract. Specifically, HHS did not conduct an annual performance evaluation to assess timeliness and cost issues that could have helped to inform NQFs future scope of work. Until August 2011, HHS did not enforce the provision for NQF to submit a financial graph to compare monthly costs for each contract activity with cost estimates, which is information not included in monthly progress reports. These tools could have provided additional, more detailed information to help identify instances in which NQF might have been at risk of not meeting time frames or exceeding cost estimates, which could have provided HHS an opportunity to make any appropriate changes to NQFs activities.
HHS had used or planned to use about half of the measures164 of 344that it received from NQF under the contract, as of August 2011. For example, HHS used 44 measures that NQF retooled under the contract in its EHR Incentive Program. HHS officials stated that the 44 measures used in the program contained errors, which required corrections. HHS officials also have not yet tested the retooled measures to assess the feasibility of implementing them in the electronic format; therefore, HHS runs the risk that some of these measures may not work as intended when implemented. HHS officials told GAO they expect to evaluate if and how they could use all of the remaining measures HHS received under the contract. However, HHS has not determined how PPACA requirements for quality measurement may have changed its needs for endorsed quality measures. As a result, HHS has not established a comprehensive plan that identifies its measurement needs and time frames for obtaining endorsed measures and that accounts for relevant PPACA requirements. Without such a plan, HHS may be limited in its efforts to prioritize which specific measures it needs to develop and to have endorsed by NQF during the remainder of the NQF contract. As a result, HHS may be unable to ensure that the agency receives the quality measures needed to meet PPACA requirements, including time frames for implementing quality measurement programs.

What GAO Recommends

GAO recommends HHS: (1) use all monitoring tools required under the contract to help address NQFs performance, (2) complete testing of retooled measures, and (3) comprehensively plan for its quality measurement needs. HHS neither agreed nor disagreed with these recommendations. NQF concurred with many of the findings in the report and provided additional context.