The more managers know about software development  processes and metrics, the better equipped they are to acquire software. Completion of these steps leads to a product design. The Software Capability Maturity Model®, for example, focuses on  improving software development processes. Successful Outcomes Are Largely the Result of Creating the Right Environment, Disciplined Processes, and Useful Metrics

Leading software companies we visited have been successful at software  development largely because they establish a manageable product  development environment, disciplined processes, and strong metrics to  manage program outcomes. Within this environment, these companies use a structured  management review process, and at the end of each of four key  development phases—requirements, design, coding, and testing—the  companies conduct reviews so that the development team does not  progress to the next phase unless it attains a certain level of knowledge. We found that leading developers are relentless in their efforts to collect  metrics to improve project outcomes and processes. Outcomes on DOD’s Software-Intensive Acquisitions Were Influenced by Environment, Processes, and Metrics

In our reviews of five major DOD software-intensive weapon system  acquisitions, we found mixed results. When DOD managers had a smaller,  more evolutionary product with manageable requirements, used  disciplined development process with gated reviews, and collected and  used metrics to manage software development progress—such as the  Tactical Tomahawk and the F/A-18-C/D programs—they delivered their  product with less cost increase and less schedule delay. Table 2 illustrates how an evolutionary environment, effective process  management, and use of meaningful metrics correlate with cost and  schedule outcomes experienced by each program. Outcomes Were Poor for Programs That Did Not Use an Evolutionary Approach, Disciplined Processes, and Meaningful Metrics

The F/A-22, SBIRS, and Comanche are complex programs that attempted  to achieve quantum leaps in performance requiring extensive use of  software rather than follow an evolutionary approach to software  development. The Air Force’s F/A-22, originally planned to be an air dominance aircraft,  will also have air-to-ground attack capability. However, additional steps must be  taken. The plans could be strengthened by adding  specific criteria to ensure that  requirements’ baselines based on systems engineering are documented  and agreed to by both the acquirer and developer before a program’s  initiation and that cost/benefit analyses are required when new  requirements are proposed;    software developers and acquirers make efforts to continually improve    gated reviews and deliverables are integrated into the development    developers collect and analyze metrics, including earned value to obtain  knowledge about development progress and to manage risk. This is a positive step. As DOD works to finalize its software process  improvement plans, it has the opportunity to put in place those practices  that have proven successful in achieving improved outcomes for software- intensive systems. To ensure DOD has the knowledge it needs to oversee software-intensive  acquisitions, we recommend that acquirers require software contractors to  collect and report metrics related to cost, schedule, size, requirements,  tests, defects, and quality to program offices on a monthly basis and before  program milestones and that acquirers should ensure that contractors  have an earned value management system that reports cost and schedule  information at a level of work that provides information specific to  software development. (c) Department of Defense Guidance—The Assistant Secretary of Defense for  Command, Control, Communications, and Intelligence, in consultation with the Under  Secretary of Defense for Acquisition, Technology, and Logistics, shall—  (1) prescribe uniformly applicable guidance for the administration of all of the  programs established under subsection (a) and take such actions as are necessary  to ensure that the military departments and Defense Agencies comply with the  guidance; and  (2) assist the Secretaries of the military departments and the heads of the Defense  Agencies to carry out such programs effectively by—  (A) ensuring that the criteria applicable to the selection of sources provides  added emphasis on past performance of potential sources, as well as on the  maturity of the software products offered by the potential sources; and  (B) identifying, and serving as a clearinghouse for information regarding, best  practices in software development and acquisition in both the public and private  sectors. Best Practices: DOD Can Help Suppliers Contribute More to Weapon  System Programs.
Why GAO Did This Study

The Department of Defense (DOD) has been relying increasingly on computer software to introduce or enhance performance capabilities of major weapon systems. To ensure successful outcomes, software acquisition requires disciplined processes and practices. Without such discipline, weapon programs encounter difficulty in meeting cost and schedule targets. For example, in fiscal year 2003, DOD might have spent as much as $8 billion to rework software because of quality-related issues. GAO was asked to identify the practices used by leading companies to acquire software and to analyze the causes of poor outcomes of selected DOD programs. GAO also was asked to evaluate DOD's efforts to develop programs for improving software acquisition processes and to assess how those efforts compare with leading companies' practices.

What GAO Found

Software developers and acquirers at firms that GAO visited use three fundamental management strategies to ensure the delivery of high-quality products on time and within budget: working in an evolutionary environment, following disciplined development processes, and collecting and analyzing meaningful metrics to measure progress. When these strategies are used together, leading firms are better equipped to improve their software development processes on a continuous basis. An evolutionary approach sets up a more manageable environment--one in which expectations are realistic and developers are permitted to make incremental improvements. The customer benefits because the initial product is available sooner and at a lower, more predictable cost. This avoids the pressure to incorporate all the desired capabilities into a single product right away. Within an evolutionary environment, there are four phases that are common to software development: setting requirements, establishing a stable design, writing code, and testing. At the end of each of these phases, developers must demonstrate that they have acquired the right knowledge before proceeding to the next development phase. To provide evidence that the right knowledge was captured, leading developers emphasize the use of meaningful metrics, which helps developers, managers, and acquirers to measure progress. These metrics focus on cost, schedule, the size of a project, performance requirements, testing, defects, and quality. In a review of five DOD programs, GAO found that outcomes were mixed for software-intensive acquisitions. The F/A-18 C/D, a fighter and attack aircraft, and the Tactical Tomahawk missile had fewer additional cost and schedule delays. For these programs, developers used an evolutionary approach, disciplined processes, and meaningful metrics. In contrast, the following programs, which did not follow these management strategies, experienced schedule delays and cost growth: F/A-22, an air dominance aircraft; Space- Based Infrared System, a missile-detection satellite system; and Comanche, a multimission helicopter. In response to congressional requirements, DOD, the military services, and the Missile Defense Agency have taken positive steps to improve the environment for acquiring software-intensive systems. However, their plans to implement software process improvement programs are not yet complete and more work is required to ensure controls that would help managers increase the chances of successful acquisition outcomes. Such controls include documenting baseline requirements agreements between the developer and acquirer that leverage systems engineering knowledge, meeting with the developer for periodic reviews (gates) during the development process, and obtaining meaningful metrics from the developer to manage the program. Furthermore, there are no assurances that program managers will be held accountable for using the plans once they are completed.