1.1.) Later, OT&E focuses on the system’s effectiveness and suitability. Deficiencies discovered during developmental and operational testing affect a system’s cost, schedule, and performance. Because of these concerns, we initiated this review to identify (1) the extent to which software-related problems affect the performance of defense acquisition programs during OT&E, (2) pervasive barriers in the acquisition process that limit the effectiveness of test and evaluation of software-intensive systems, and (3) DOD’s efforts to resolve software test and evaluation problems. OT&E Often Identifies Immature Software-Intensive Systems

To ensure no surprises during OT&E, defense systems are expected to be subjected to rigorous DT&E. DOD has yet to develop and implement the management processes and tools required to improve the reliability of its data. The individual military services have tried to improve their software development processes, but a DOD-wide, coordinated approach is lacking. Senior OSD officials told us that they believe the creation of a single OSD-level office for software would, in part, help to resolve long-standing software problems. Because of the acquisition community’s bias toward hardware, DOD has not adequately ensured that software was fully mature and had undergone thorough and rigorous DT&E before systems were certified as ready for OT&E. Recommendations

To realize lasting improvements in test and evaluation of software-intensive systems and to enhance the life-cycle affordability of such programs, we recommend that the Secretary of Defense issue and implement a software test and evaluation policy that defines testing requirements for software maturity, regression testing, and the use of temporary software fixes during testing; strengthen controls to ensure that operational testing does not begin until results of development test and evaluation demonstrate an appropriate level of software maturity; require program management officials to define exit criteria for certifying a systems’ readiness for operational testing at the beginning of full-scale development (i.e., milestone II); and require the services to develop a common core set of management metrics for software (i.e., cost, schedule, and quality) for major defense programs early in the development cycle to be approved at milestone II.
Why GAO Did This Study

GAO reviewed the Department of Defense's (DOD) efforts to improve its software acquisition processes and test and evaluation of software-intensive systems, focusing on: (1) how software-related problems affect defense acquisition programs during operational test and evaluation (OT&E); and (2) barriers that limit OT&E effectiveness.

What GAO Found

GAO found that: (1) DOD software-intensive systems often fail to meet their performance expectations, experience cost overruns and schedule delays, and are not ready for OT&E because they are not subject to rigorous developmental testing and evaluation and software development is often not fully mature when OT&E begins; (2) DOD has not adequately addressed barriers that inhibit software acquisition and OT&E effectiveness; (3) DOD has not adequately addressed the importance of software development, lacks a consistent policy that defines and coordinates software maturity and operational testing readiness, has not adequately defined and managed software requirements, and lacks reliable cost, schedule, and performance data; (4) DOD has not implemented previous recommendations to ensure that quality software is adequately developed and tested and software-intensive systems are effective; (5) although the military services have tried to improve their software development processes, they lack a DOD-wide coordinated strategy; and (6) the creation of a single executive-level office for software development and testing would help to resolve long-standing software problems.