Operational test and  evaluation is intended to evaluate a system’s effectiveness and suitability  under realistic combat conditions before full-rate production or  deployment occurs. Inherent Tension Exists in Operational Test Oversight, but Few Programs Experienced Significant Disputes with DOT&E

Different objectives and incentives exist between the acquisition and  testing communities, which can potentially fuel tension over what is  needed to accomplish operational testing for programs. According to  DOD officials, differences usually are resolved in a reasonable and timely  manner, with modest adjustments often required in the course of  developing and executing a test approach. However, sometimes  acquisition and test officials have differences about operational testing  requirements, methods, costs, or results that develop into significant  disputes that are more difficult to resolve. Acquisition and test officials in  the military services identified a small number of cases where significant  disputes occurred among programs receiving DOT&E oversight during  fiscal years 2010 through 2014. Of the 454 programs on DOT&E’s  oversight list during that period, officials identified 42 programs—less than  10 percent—that had significant operational test disputes with DOT&E.According to military service and DOT&E officials, while these disputes  can require additional time and effort to work through, they generally do  get resolved. A Variety of Factors Contributed to Disputes We Reviewed, but Only a Few Cases Had Significant Cost or Schedule Impacts

To gain a better understanding of the circumstances of disputes and  assess the merits of disputes for cases where opinions varied, we  selected and performed an in-depth review of 10 cases from among the  42 programs that had disputes. The 10 cases had a  variety of factors that contributed to disputes between DOT&E and  military service officials over operational testing, but only a few had  considerable cost or schedule impacts. The factors typically involved the  adequacy of proposed testing and differences over test requirements,  assets, and the reporting of test results. In general, we found that DOT&E  had valid and substantive operational test-related concerns for each  program reviewed. Most of the disputes we reviewed had been, or are expected to be,  resolved in favor of DOT&E’s concerns, and with limited cost and  schedule impacts to the programs. In a few of these cases, military  service officials acknowledged that benefits were achieved from resolving  the disputes, such as a reduction in the scope of operational testing and  better understanding of system requirements. However, resolution of  disputes for three programs—DDG-51 Flight III Destroyer, F-35 Joint  Strike Fighter, and CVN 78 aircraft carrier—had considerable potential  cost or schedule effects that required formal involvement from senior  DOD leadership. For the first two programs, hundreds of millions of  dollars in additional costs are associated with resolving their disputes. For  CVN 78, the dispute, which remains unresolved, has ramifications for the  Navy’s carrier deployment schedule and whether a key test to assess the  survivability of the carrier will be deferred by several years. For the other seven case studies included in our review, cost and  schedule effects in resolving their disputes were more limited, and in  some cases, were not related to operational testing requirements. Appendix I: Objectives, Scope, and Methodology

The Joint Explanatory Statement to Accompany the National Defense  Authorization Act for Fiscal Year 2015 directed GAO to review the  oversight activities of the Department of Defense’s (DOD) Office of the  Director, Operational Test and Evaluation (DOT&E), including how they  may affect acquisition programs. Our objectives for this review were to  examine (1) the extent to which there have been any significant disputes  between DOT&E and acquisition programs over operational testing, and  (2) the circumstances and impact of identified operational test-related  disputes. In addition to  our interview activities, we formally solicited input and received responses  from officials within the military service acquisition executive offices, test  and evaluation offices, and operational test agencies that identified  programs that had experienced any of the following circumstances since  fiscal year 2010:  significant delays in obtaining DOT&E’s approval of test and  evaluation master plans (TEMPs);  significant delays in obtaining DOT&E approval of operational test  plans to support initial operational test and evaluation (IOT&E);  significant disputes over the test assets needed to conduct IOT&E;  significant disputes with DOT&E related to what requirements were to  be tested during IOT&E, such as testing key performance parameters  only versus testing to intended mission capabilities in an operational  environment;  significant disagreements over the characterization of IOT&E results  that led to a delay in reaching a full-rate production decision;  significant disputes related to the need to conduct live fire testing or  the extent of testing;  substantially increased costs for operational test completion that were  judged unwarranted by programs or the military services; or  significant disputes over any other elements associated with DOT&E’s  oversight role, such as decisions by DOT&E to add programs to its  oversight list, DOT&E’s activities related to operational assessments,  or the need to conduct follow-on operational testing and evaluation. Our case study  programs are representative of the types of disputes identified overall by  military service officials.
Why GAO Did This Study

DOD conducts extensive operational testing and evaluation of its military systems prior to full-rate production and fielding. DOT&E plays an integral role in operational test and evaluation by issuing policy and procedures, overseeing operational test planning, and independently evaluating and reporting test results. At times, DOT&E and acquisition programs may disagree about what is needed to adequately demonstrate operational capability, which sometimes may affect programs' cost or schedule.
The Joint Explanatory Statement to Accompany the National Defense Authorization Act for Fiscal Year 2015 directed GAO to review DOT&E's oversight activities. This report examines (1) the extent to which DOD acquisition programs have had significant disputes, if any, with DOT&E over operational testing, and (2) the circumstances and impact of identified disputes. GAO evaluated documentation and interviewed officials from DOT&E, other DOD test organizations, and the acquisition community. GAO also conducted 10 case studies from among 42 programs identified by military service officials as having had significant disputes with DOT&E. GAO analyzed information received from acquisition and testing officials to verify the merits and degree of those disputes. Based on this assessment, GAO selected case studies that were representative of the most significant disputes identified across the military services.

What GAO Found

The Director, Operational Test and Evaluation (DOT&E) provided oversight for 454 Department of Defense (DOD) acquisition programs from fiscal years 2010-2014. Military service officials identified 42 programs from that period that they believed had significant disputes with DOT&E over operational testing—that is, disputes that may have led to cost and schedule impacts for programs. Operational testing is intended to evaluate a system's capability in realistic combat conditions before full-rate production or full deployment. Acquisition programs and DOT&E have different objectives and incentives, which can potentially fuel tension between the two over what is needed to accomplish operational testing for programs. According to military service officials, the tension is generally manageable and differences usually are resolved in a reasonable and timely manner, with modest adjustments often required in the course of developing and executing a test approach. However, sometimes differences about operational testing requirements, methods, costs, or results develop into significant disputes and are more difficult to resolve. Acquisition and test officials from the military services identified only a limited number of cases—less than 10 percent of programs receiving DOT&E operational test oversight since fiscal year 2010—that they believed had experienced significant operational testing disputes with DOT&E. Officials noted that although these disputes can require additional time and effort to work through, they generally get resolved.
In an in-depth review of 10 case studies selected from among the 42 programs with significant disputes, GAO identified a variety of factors that contributed to disputes between the acquisition programs and DOT&E, but only a few cases that involved considerable cost or schedule impacts. Key factors involved the adequacy of proposed testing and differences over test requirements, assets, and the reporting of test results. In general, GAO found that DOT&E had valid and substantive concerns about operational test and evaluation for each of the 10 cases reviewed. However, military service officials indicated to GAO that testing advocated by DOT&E was, in some instances, beyond what they believed was necessary and lacked consideration for programs' test resource limitations. Many of the disputes GAO reviewed were, or are expected to be, resolved in DOT&E's favor with limited cost and schedule impacts to the programs. In a few cases, military service officials acknowledged that benefits were achieved from resolving the disputes, such as a reduction in the scope of operational testing and better understanding of system requirements. Resolution of disputes for three programs—DDG-51 Flight III Destroyer, F-35 Joint Strike Fighter, and CVN 78 aircraft carrier—had considerable potential or realized cost or schedule effects and required formal involvement from senior DOD leadership. For the first two programs, hundreds of millions of dollars in additional costs were associated with resolving their disputes. For CVN 78, the dispute—which remains unresolved— involves the Navy's carrier deployment schedule and whether survivability testing will be deferred by several years. For the other seven case study programs that GAO reviewed, the cost and schedule effects tied to dispute resolution were more limited, and in some instances, not related to operational testing requirements.

What GAO Recommends

GAO is not making any recommendations in this report.