VBA staff in 57 regional offices process disability compensation claims. Under the STAR  program, VBA reviews a stratified random sample of completed claims,  and certified reviewers use a checklist to assess specific aspects of each  claim. Specifically, for each of the 57 regional offices, completed claims  are randomly sampled each month and the data are used to produce  estimates of the accuracy of all completed claims. The QRT also conducts in- process reviews before claims are finalized to help prevent inaccurate  decisions by identifying specific types of common errors. VBA’s Approach to Measuring and Reporting Accuracy of Claim Decisions Has Limitations

When calculating accuracy rates, VBA does not follow generally accepted  statistical practices. In addition, VBA’s accuracy reporting lacks methodological details that  would help users understand the distinction between its two accuracy  measures and their associated limitations. 1). Finally, VBA’s approach to measuring accuracy is inefficient because it  reviews more claims than needed to estimate accuracy. According to VBA, this uniform approach allows the  agency to achieve a desired level of precision of its accuracy estimates  for each regional office. According to our analysis of fiscal year 2013 regional office  workload and accuracy results, VBA could reduce the overall number of  claims it reviewed annually by about 39 percent (over 5,000 claims) and  still achieve its desired precision for its regional office accuracy estimates. More efficient sampling could allow VBA to select  fewer cases for review and free up limited resources for other important  quality assurance activities, such as additional targeted accuracy reviews  on specific types of error-prone or complex claims. VBA Has Enhanced and Coordinated Its Quality Assurance Activities, Although Gaps in Effectiveness Exist

		VBA Enhanced Other Quality Assurance Activities, but Shortcomings Exist and Effectiveness is Unclear

In addition to its STAR reviews, VBA’s quality assurance framework  includes other complementary activities, some of which have been  enhanced to help meet its 98-percent goal for fiscal year 2015. VBA  officials told GAO they do not know how many regional offices include or  exclude claims processed during overtime, or the extent to which  excluding cases worked during overtime occurs nationally. VBA’s efforts to assess consistency of claims decisions have also  expanded in recent years. Although VBA has enhanced its approach to measuring consistency, VA  officials told us that consistency questionnaires to date have been  developed and implemented without any prior pre-testing which would  allow VA to examine the clarity of questions or the validity of the expected  questionnaire results. Specifically,  VBA officials told us that, although they have not seen an increase in the  national accuracy rate in the current fiscal year, the number of errors  related to claim development—specifically, to ensuring the claim reflected  sufficient / appropriate medical examinations)—has declined, showing  the success of QRT efforts in targeting these errors through in-process  reviews and providing related training. In our ongoing work, we plan to further review how VBA uses the results  of its consistency studies to improve quality and whether and how VBA  could further leverage its vast amount of claims data to identify error  trends and opportunities for improvement. VBA Has Taken Steps to Coordinate Quality Assurance Activities, but Key Supports Have Not Been Updated

VA has taken steps to coordinate its quality assurance efforts, in part, by  systematically disseminating information on national accuracy and  consistency trends to regional office management and QRTs, which in  turn used this information to provide feedback. At the same time, staff in all four offices said that key  supports were not sufficiently updated to help quality review staff and  claims processors efficiently and effectively do their jobs. Staff at these  offices consistently described problems with data systems, central  guidance and training. As a result, staff must search numerous sources of  guidance to locate current policy, which is time-consuming and  difficult. VBA officials acknowledged that there are several ways it  provides guidance to regional offices—such as guidance letters;  periodic quality calls (and notes from those calls); various bulletins;  and training materials maintained on VBA’s Intranet site—and that this  could be confusing to staff. This makes it difficult to provide claims processors  with the information they need to avoid future errors. VBA has bolstered regional attention to quality and  national consistency efforts, but several shortcomings—such as excluding  claims worked during overtime from regional office quality reviews, or the  lack of pretesting of consistency questionnaires—may detract from their  overall effectiveness. Moreover, VBA does not systematically track the  impact of these efforts on accuracy rates, again creating a gap in  information that could help focus future quality assurance efforts.
Why GAO Did This Study

With a growing workload of disability claims due, in part, to recent wars, VBA faces difficulties in improving the accuracy and consistency of its claims decisions by VBA staff in 57 regional offices. To help achieve its goal of 98 percent accuracy by fiscal year 2015, VBA recently implemented a new way of measuring accuracy, and changed several quality assurance activities to assess the accuracy and consistency of claims decisions and to provide feedback and training to claims processors.
In this statement, GAO discusses initial observations from its ongoing review of VBA's quality assurance efforts, addressing the extent to which (1) VBA effectively measures and reports the accuracy of its compensation claim decisions; and (2) VBA's other quality assurance activities are complementary and coordinated. For this work, GAO analyzed STAR accuracy and VBA claims data from fiscal year 2013 (the most recent fiscal year for which complete data are available); reviewed relevant federal laws, regulations, guidance and other documents relevant to quality assurance activities; and interviewed VBA staff from headquarters and four VBA regional offices (selected to achieve variety in geography, size of workload and reported accuracy rates), and Veterans Service Organization officials. GAO has no recommendations at this time. GAO plans to issue its final report later in 2014, along with any related recommendations.

What GAO Found

The Veterans Benefit Administration (VBA) within the Department of Veterans Affairs (VA) now measures and reports the accuracy of its claim decisions in two ways—by claim and by medical issue—but its approach has limitations. When calculating accuracy rates for either measure through its Systematic Technical Accuracy Review (STAR) program, VBA does not always follow generally accepted statistical practices. For example, VBA does not adjust its accuracy estimates to reflect that it samples the same number of claims for review from each regional office despite their varying workloads, and thus, produces imprecise estimates of national and regional accuracy. Further, because VBA does not clearly explain differences in how its two accuracy measures are calculated or their associated limitations, reported information about accuracy performance lacks clarity and may be confusing. Finally, VBA reviews 39 percent (over 5,000) more claims nationwide than necessary to achieve its desired precision in reported accuracy rates, thereby diverting limited resources from other important quality assurance activities, such as targeted reviews of error-prone cases. VBA could achieve its desired precision by reviewing fewer claims for offices with smaller workloads or higher performance levels.
Overall, VBA has made enhancements to its other quality assurance activities, although GAO identified implementation shortcomings that may detract from their effectiveness. To improve local quality, VBA recently created quality review teams (QRTs) comprised of certified staff in each regional office. QRTs review a sample of claims before they are finalized to identify specific error types and help prevent inaccurate decisions. QRTs also assess individual staff performance; however, claims processed during overtime were excluded from such reviews in three of the four offices GAO visited, which may undermine regional quality for a portion of claims. VBA officials told GAO they do not know the extent to which this occurs nationally. Recent VBA data shows that about 10 percent of claims were processed during overtime. Also, to help ensure claims processors make consistent decisions when presented with the same evidence, VBA began using a questionnaire approach to test for consistency. Although these questionnaires allow VBA to reach more staff and require fewer resources to administer, VBA did not pre-test them to ensure the clarity of questions or validity of the expected results. For each enhancement, their effectiveness is unclear because VBA has done little to date to assess their impact on improving accuracy.
VBA has also taken steps to coordinate its quality assurance efforts, but has not maintained centralized guidance and other key supports that might help prevent future errors. VBA coordinates quality assurance efforts by disseminating national accuracy and consistency results, trends and related guidance to regional offices, which use this information to train staff. Further, VBA uses the results of STAR reviews for other quality assurance activities, such as focusing QRT in-process reviews on commonly made errors. However, regional office staff told GAO that there are multiple sources of guidance, that searching these sources is time-consuming and difficult, and that VBA's policy manual and national training are not sufficiently updated to help claims processors avoid future errors. In addition, regional office data systems do not allow managers to readily track error trends, resulting in ad-hoc and inefficient work-arounds.