A QRT conducts individual quality reviews of claims processors’  work for performance assessment purposes. In 2013, VBA revised its approach and began using  questionnaires as its primary means for assessing consistency. VBA’s Approach to Measuring and Reporting Accuracy of Claim Decisions Has Limitations

		VBA Does Not Follow Accepted Statistical Practices and Thus Generates Imprecise Accuracy Data

When calculating accuracy rates, VBA does not always follow generally  accepted statistical practices. For example, VBA does not weight the  results of its STAR reviews to reflect its approach to selecting claims by  regional office, which can affect the accuracy of estimates. 1). 2). According to our analysis of fiscal year 2013  regional office workload and accuracy results, VBA could reduce the  overall number of claims it reviews annually by about 39 percent (over  5,000 claims) and still achieve its desired precision for its regional office  accuracy estimates. More efficient sampling could allow VBA to select  fewer cases for review and free up limited resources for other important  quality assurance activities, such as additional targeted accuracy reviews  on specific types of error-prone or complex claims. Such underrepresentation may inflate  VBA’s reported accuracy rate because redistributed claims have  historically had lower accuracy rates than non-redistributed claims. Further, VA has not explained in public reports that its accuracy  measures are estimates that have distinct confidence intervals and  limitations. VBA Has Enhanced and Coordinated Its Quality Assurance Activities, Though Gaps in Implementation May Limit Their Effectiveness

		VBA Has Taken Steps to Enhance and Coordinate Key Quality Assurance Activities

In addition to its STAR reviews, VBA’s quality assurance framework  includes other complementary activities, which have been enhanced to  help meet its goal of 98 percent accuracy in fiscal year 2015. In addition, at the four offices we contacted,  quality reviewers are available to answer questions and provide guidance  to claims processors as needed. VBA also has taken steps to coordinate its quality assurance efforts in  several ways, such as systematically disseminating information on  national accuracy and consistency results and trends to regional office  management and QRTs, which in turn share this information with claims  processing staff. Pre-testing is a generally accepted practice in sound  questionnaire development for examining the clarity of questions or the  validity of the questionnaire results. Guidance: Federal internal control standards highlight the need for  pertinent information being captured and distributed in a form that  allows people to perform their duties efficiently. This, in turn, could affect the accuracy with  which they decide claims. Finally, VBA’s efforts to evaluate the effectiveness of its quality assurance  activities have been limited. Also, VBA identified 13 regional offices whose  issue-based accuracy rates improved between the first and third quarters  of fiscal year 2014, attributing these improvements to actions taken by   However, it was not clear  quality assurance staff in fiscal year 2014.from the documentation VBA provided whether and how it monitored the  effectiveness of these actions for all regional offices. With respect to  consistency studies, VBA also has not evaluated—and lacks plans to  evaluate—the efficacy of using consistency questionnaires relative to the  more resource-intensive IRR studies. VBA has enhanced and coordinated other aspects of its quality  assurance framework, but shortcomings in implementation and evaluation  detract from their overall effectiveness. Recommendations for Executive Action

To help improve the quality of VBA’s disability compensation claim  decisions, we recommend that the Secretary of Veterans Affairs direct the  Under Secretary for Benefits to:
Leverage appropriate expertise to help VBA do each of the following: weight its accuracy estimates to reflect the sample design for determine and report the confidence intervals associated with its  reported accuracy estimates; and re-examine its approach to calculating the regional office sample  size for STAR. Review the multiple sources of policy guidance VBA provides to  determine ways to consolidate them or otherwise improve their  availability and accessibility for use by staff in regional offices. Regarding our recommendation to take steps to ensure that any  future upgrades to local data systems allow QRTs to pause the claims  process when errors are detected and enable QRTs to better track  error trends, VA stated that VBA is designing a new database that will  incorporate all types of quality reviews (i.e., regional office reviews,  STAR, and consistency studies) and provide VBA with more data  analysis capabilities. Appendix I: Objectives, Scope and Methodology

The objectives of this report were to examine (1) the extent to which the  Veterans Benefits Administration (VBA) effectively measures and reports  the accuracy of compensation claim decision-making, and (2) whether  VBA’s other quality assurance activities are coordinated and effective. Review of Systematic Technical Accuracy Review (STAR)

To assess VBA’s measurement and reporting of the accuracy of  compensation claim decision-making, we focused on the STAR process  for reviewing disability compensation claims that VBA identifies as rating- related—that is, requiring a decision on the claimant’s eligibility for  benefits and the monthly benefit amount. Coordination and Effectiveness of Quality Assurance Activities

To determine whether VBA’s quality assurance activities are coordinated  and effective, we reviewed VBA quality assurance policies, reports, and  guidance to identify key quality assurance activities. At each office, we spoke with service  center managers and quality assurance staff, as well as representatives  of local veteran service organizations.
Why GAO Did This Study

With a backlog of disability compensation claims, VBA faces difficulties in improving the accuracy and consistency of the claim decisions made by staff in its 57 regional offices. To help achieve its goal of 98 percent accuracy by fiscal year 2015, VBA recently implemented a new way of measuring accuracy and changed several quality assurance activities to assess the accuracy and consistency of decisions and to provide feedback and training to claims processors. GAO was asked to examine VBA's quality assurance activities.
This report evaluates (1) the extent to which VBA effectively measures and reports the accuracy of its disability compensation claim decisions and (2) whether VBA's other quality assurance activities are coordinated and effective. GAO analyzed VBA claims and STAR accuracy data from fiscal year 2013 (the most recent fiscal year for which complete data are available); reviewed relevant federal laws, VBA guidance, and other documents relevant to quality assurance activities; and interviewed VBA staff from headquarters and four VBA regional offices (selected to achieve variety in geography, workload, and accuracy rates), as well as veteran service organization officials.

What GAO Found

The Veterans Benefits Administration (VBA)—within the Department of Veterans Affairs—measures and reports the accuracy of its disability compensation claim decisions in two ways: (1) by claim and (2) by disabling condition, though its approach has limitations. When calculating accuracy rates for either measure through its Systematic Technical Accuracy Review (STAR), VBA does not always follow generally accepted statistical practices, resulting in imprecise performance information. For example, VBA does not adjust its accuracy estimates to reflect that it samples the same number of claims for review from each regional office—despite their varying workloads—and thus produces imprecise estimates of national and regional accuracy. Further, VBA reviews about 39 percent (over 5,000) more claims nationwide than is necessary to achieve its desired precision in reported accuracy rates, thereby diverting limited resources from other important quality assurance activities, such as targeted reviews of error-prone cases. In addition to issues with its statistical practices, VBA's process for selecting claims for STAR review creates an underrepresentation of claims that are moved between regional offices, which may inflate accuracy estimates because these claims have had historically lower accuracy rates. Finally, VBA has not clearly explained in public reports the differences in how its two accuracy measures are calculated or their associated limitations, as suggested by best practices for federal performance reporting.
VBA has taken steps to enhance and coordinate its other quality assurance activities, but GAO found shortcomings in how VBA is implementing and evaluating these activities. To improve local accuracy, VBA created regional office quality review teams (QRTs) with staff dedicated primarily to performing local accuracy reviews. QRTs assess individual claims processor performance and conduct special reviews to forestall certain types of errors. In addition, VBA began using questionnaires for assessing decision-making consistency, which are more efficient to administer than VBA's prior approach to conducting consistency studies. VBA also coordinates quality assurance efforts by disseminating national accuracy and consistency results, trends, and related guidance to regional offices for use in training claims processors. Further, VBA uses STAR results to inform other quality assurance activities, such as focusing certain QRT reviews on commonly made errors. However, GAO identified implementation shortcomings that may detract from the effectiveness of VBA's quality assurance activities. For example, contrary to accepted practices for ensuring the clarity and validity of questionnaires, VBA did not pre-test its consistency questionnaires to ensure the clarity of questions or validity of the expected results, although VBA officials indicated that they plan to do so for future questionnaires. In contrast with federal internal control standards that call for capturing and distributing information in a form that allows people to efficiently perform their duties, staff in the four regional offices that we visited had trouble finding the guidance they needed to do their work, which could affect the accuracy as well as the speed with which staff decide claims. Federal standards also call for knowing the value of efforts such as quality assurance activities and monitoring their performance over time; however, VBA has not evaluated the effect of its special QRT reviews or certain consistency studies on improving targeted accuracy rates, and lacks clear plans to do so.

What GAO Recommends

GAO is making eight recommendations to VA to improve its measurement and reporting of accuracy, review the multiple sources of policy guidance available to claims processors, enhance local data systems, and evaluate the effectiveness of quality assurance activities. VA concurred with all of GAO's recommendations.