Weaknesses in CMS Survey Process Contributed to Understatement, but Long-Term Effect of New Survey Methodology Is Not Yet Known

Over a third of both surveyors and state agency directors responding to  our questionnaire identified weaknesses in the federal government’s  nursing home survey process that contributed to the understatement of  deficiencies. One such weakness identified by both  surveyors and directors was the number of survey tasks that need to be  completed. Forty percent of surveyors and  58 percent of state agency directors reported that additional training on  how to apply CMS guidance was needed. State agency directors and surveyors indicated that CMS’s written  guidance for certain federal nursing home quality standards could be  improved and that revised investigative protocols were helpful. Since the evaluation did not find improved accuracy, CMS  concluded that non-QIS factors, including survey guidance clarification  and surveyor training and supervision, would help improve survey  accuracy. Workforce Shortages Sometimes Contributed to Understatement

Although a small percentage of state agency directors reported that  workforce shortages always or frequently contributed to the  understatement of nursing home deficiencies in their states, 36 percent  indicated that workforce shortages sometimes contributed to  understatement (see table 4). On average, 30 percent  of surveyors had less than 2 years’ experience (see table 5); however the  percentage of inexperienced surveyors ranged from 10 to 82 percent  across states who reported this information. In addition, 46 percent of state  agency directors reported that time pressures always, frequently, or  sometimes contributed to understatement in their states. Initial Surveyor Training. About 29 percent of surveyors in  high-understatement states reported that initial training was not sufficient  to cite appropriate scope and severity levels, compared with 16 percent of  surveyors in low-understatement states (see table 7). Ongoing Training. Supervisory Reviews Often Focused on Higher-Level Deficiencies

According to state agency directors’ responses to our questionnaire, states  generally focused supervisory review on surveys with higher-level  deficiencies, rather than on the surveys with deficiencies at the potential  for more than minimal-harm level (D through F)—the deficiencies most  likely to be understated. Survey methodology weaknesses. State Agency Practices and External Pressure May Compromise Survey Accuracy and Lead to Understatement in a Few States

Nursing home surveyors and state agency directors in a minority of states  told us that in isolated cases issues such as a state agency practice of  noncitation, external pressure from the nursing home industry, and an  unbalanced IDR process may have led to the understatement of  deficiencies. Surveyors Reported Noncitation Practices in a Small Number of States

Approximately 20 percent of surveyors nationwide and over 40 percent of  surveyors in five states reported that their state agency had at least one of  the following noncitation practices: (1) not citing certain deficiencies,   (2) not citing deficiencies above a certain scope and severity level, and   (3) allowing nursing homes to correct deficiencies without receiving a  citation (see fig. Concerns about CMS’s Survey Process. For example, surveyors who reported that survey teams had  too many new surveyors, more often also reported either frequent changes  to or removals of deficiencies during supervisory reviews—indicating that  states with inexperienced workforces may rely more heavily on  supervisory reviews. In a few states,  noncitation practices, challenging relationships with the industry or  legislators, or unbalanced IDR processes—those that surveyors regard as  favoring nursing home operators over resident welfare—may have had a  negative effect on survey quality and resulted in the citation of fewer  nursing home deficiencies than was warranted. Twelve state agency directors reported on our questionnaire  experiencing some kind of external pressure. However, our May 2008 report noted several improvements CMS  had made since fiscal years 2002 and 2003 in federal comparative surveys  intended to make them more comparable to the state surveys they are  assessing; these improvement include; (1) reducing the time between the  state and federal surveys to ensure that they more accurately capture the  conditions at the time of the state survey, (2) including at least half of the  residents from state survey investigative samples to allow for a more clear- cut determination of whether the state survey should have cited a  deficiency, and (3) using the same number of federal surveyors as the  corresponding state survey, again to more closely mirror the conditions  under which the state survey was conducted. (For response rates by state, see table 9.) GAO-10-37R.
Why GAO Did This Study

Under contract with the Centers for Medicare and Medicaid Services (CMS), states conduct surveys at nursing homes to help ensure compliance with federal quality standards. Over the past decade, the Government Accountability Office (GAO) has reported on inconsistencies in states' assessment of nursing homes' quality of care, including understatement--that is, when state surveys fail to cite serious deficiencies or cite them at too low a level. In 2008, GAO reported that 9 states had high and 10 had low understatement based on CMS data for fiscal years 2002 through 2007. This report examines the effect on nursing home deficiency understatement of CMS's survey process, workforce shortages and training, supervisory reviews of surveys, and state agency practices. GAO primarily collected data through two Web-based questionnaires sent to all eligible nursing home surveyors and state agency directors, achieving 61 and 98 percent response rates, respectively.

What GAO Found

A substantial percentage of both state surveyors and directors identified general weaknesses in the nursing home survey process, that is, the survey methodology and guidance on identifying deficiencies. On the questionnaires, 46 percent of surveyors and 36 percent of directors reported that weaknesses in the traditional survey methodology, such as too many survey tasks, contributed to understatement. Limited experience with a new data-driven survey methodology indicated possible improvements in consistency; however, an independent evaluation led CMS to conclude that other tools, such as survey guidance clarification and surveyor training and supervision, would help improve survey accuracy. According to questionnaire responses, workforce shortages and greater use of surveyors with less than 2 years' experience sometimes contributed to understatement. Nearly three-quarters of directors reported that they always or frequently experienced a workforce shortage, while nearly two-thirds reported that surveyor inexperience always, frequently, or sometimes led to understatement. Substantial percentages of directors and surveyors indicated that inadequate training may compromise survey accuracy and lead to understatement. According to about 29 percent of surveyors in 9 high understatement states compared to 16 percent of surveyors in 10 low understatement states, initial surveyor training was not sufficient to cite appropriate scope and severity--a skill critical in preventing understatement. Furthermore, over half of directors identified the need for ongoing training for experienced surveyors on both this skill and on documenting deficiencies, a critical skill to substantiate citations. CMS provides little guidance to states on supervisory review processes. In general, directors reported on our questionnaire that supervisory reviews occurred more often on surveys with higher-level rather than on those with lower-level deficiencies, which were the most frequently understated. Surveyors who reported that survey teams had too many new surveyors also reported frequent changes to or removal of deficiencies, indicating heavier reliance on supervisory reviews by states with inexperienced surveyors. Surveyors and directors in a few states informed us that, in isolated cases, state agency practices or external pressure from stakeholders, such as the nursing home industry, may have led to understatement. Forty percent of surveyors in five states and four directors reported that their state had at least one practice not to cite certain deficiencies. Additionally, over 40 percent of surveyors in four states reported that their states' informal dispute resolution processes favored concerns of nursing home operators over resident welfare. Furthermore, directors from seven states reported that pressure from the industry or legislators may have compromised the nursing home survey process, and two directors reported that CMS's support is needed to deal with such pressure. If surveyors perceive that certain deficiencies may not be consistently upheld or enforced, they may choose not to cite them.