Progressively more detailed descriptions of the system design are  established in what systems engineering literature refers to as  configuration baselines. Four Factors Frame the Challenge Posed by Requirements, and Detailed Systems Engineering Is Key to Knowing Whether and How the Challenge Can Be Met

Our analysis of nine selected DOD programs, supported by our previous  work examining best practices for product development, identified four  key factors that provide insight into the challenge posed by a system’s  top-level capability requirements and the related risk. Systems engineering is the primary  means for determining whether and how the challenge posed by a  program’s requirements can be met with available resources. It is a  disciplined learning process that translates capability requirements into  specific design features and thus identifies key risks to be resolved. Our  prior best practices work has indicated that if detailed systems  engineering is done before the start of product development, the program  can resolve these risks through trade-offs and additional investments,  ensuring that risks have been sufficiently retired or that they are clearly  understood and adequately resourced if they are being carried forward. This practice was also emphasized by Congress in the  Weapon System Acquisition Reform Act of 2009, which required  programs to hold a preliminary design review or obtain a waiver prior to  starting a development program. Our analysis of development cost and  schedule data for the nine case studies shows that, as of December  2015, about half of the programs had reported relatively good  development cost and schedule outcomes, while the others had  experienced significant cost and schedule growth (see table 2). Finally, the three programs in our sample that began  development with challenging requirements and had done little systems  engineering generally reported poor outcomes. The Air Force planned an  incremental acquisition approach focused on integrating mature military  technologies onto a commercial aircraft derivative design. With all of the  detailed systems engineering done before development started, risks  were understood. P-8A Increment I experienced some cost  and schedule growth. When development began, DOD planned to achieve full  capability using a single-step acquisition approach and the F-35’s critical  technologies were immature. A particular challenge for Congress is the fact that committees must often  consider requests to authorize and fund a new program well ahead of the  start of product development—the point at which key business case  information would be presented. The nine case studies we examined for this report suggest that  understanding the dynamic between program requirements, risks, and the  requisite systems engineering analysis can facilitate early oversight. Specifically, when the top-level requirements for what could be a new  major weapon system are initially identified in a draft Capability  Development Document, the challenge those requirements pose and how  it relates to the four factors we identified—acquisition approach,  technology status, design maturity, and system interdependence—can be  observed. While congressional insight at the time of the funding decision is limited,  DOD policy directs program managers to provide certain documents to  DOD decision makers that when taken together could provide a picture of  a proposed program’s risk factors and systems engineering status. However, it is not clear whether Congress is  given the same information DOD decision makers are given when DOD  submits its budget request for funding to begin a new program. The case studies in this report illustrate a strong relationship  between requirements, systems engineering, and program outcomes. Matter for Congressional Consideration

To enhance program oversight and provide more robust input to budget  deliberations, Congress should consider requiring DOD to report on each  major acquisition program’s systems engineering status in the  department’s annual budget request, beginning with the budget  requesting funds to start development. In its comments, DOD agreed that early systems  engineering reduces risk and establishes a solid foundation for program  success. Appendix I: Objectives, Scope and Methodology

This report (1) identifies a framework for assessing the challenges and  risks associated with program requirements and the extent of systems  engineering done before product development begins; (2) illustrates the  relationship between systems engineering and program outcomes; and  (3) assesses implications for program management and oversight. To assess the relationship between systems engineering and program  outcomes, we analyzed requirements, cost, and schedule documentation  for each of our case study programs, and then met with relevant program  officials and prime contractors to obtain relevant program documents,  data, and historical information.
Why GAO Did This Study

Cost and schedule growth in DOD major defense acquisition programs persist, and some acquisition reform proponents believe such growth is due to unplanned changes in program requirements (commonly referred to as "requirements creep"). GAO found in June 2015 that cost and schedule growth are often more directly related to a lack of systems engineering, which, if done, would reduce risk by introducing discipline and rigor into the process of defining and understanding a program's initial requirements.
House Armed Services Committee Report 114-102 contained a provision for GAO to review the DOD requirements process. This report (1) identifies a framework for assessing the challenge posed by weapon system requirements and the extent of systems engineering done before product development begins; (2) illustrates the relationship between systems engineering and program outcomes; and (3) assesses implications for program oversight. GAO analyzed a non-generalizable sample of nine case studies. GAO assessed the extent to which systems engineering was conducted before development by reviewing program requirements and analyzing cost and schedule documentation for each case study. GAO also reviewed prior GAO work and interviewed DOD officials.

What GAO Found

GAO's analysis of nine case studies identified four factors that frame the challenge posed by a given weapon system's requirements: acquisition approach, technology status, design maturity, and system interdependency. Systems engineering is the primary means for determining whether and how that challenge can be met. It is a disciplined learning process that translates requirements into specific design features and thus identifies key risks to be resolved. GAO's prior best practices work has found that if detailed systems engineering is done early, a program can resolve such risks through trade-offs and additional investments before a program starts. A key point in systems engineering where this match can be assessed is the preliminary design. As shown below, establishing a preliminary design through early detailed systems engineering portends better program outcomes than doing so after program start.
GAO's analysis of selected Department of Defense (DOD) programs illustrates the relationship among the four factors, systems engineering, and program outcomes. Programs with modest requirements and early detailed systems engineering had better outcomes. For example, the Small Diameter Bomb Increment I program, which delivered within cost and schedule estimates, had an incremental approach, mature technologies, a derivative design, and detailed systems engineering before development began. Programs that began with more challenging requirements and insufficient systems engineering reported worse outcomes. For example, the F-35 Lightning II, which has encountered significant cost and schedule problems, began development with a single-step approach, a highly complex design, immature technologies, and little systems engineering.
Understanding the dynamic between a program's requirements, risks, and the requisite systems engineering effort has important implications for oversight. A particular challenge is that Congress often must consider requests to authorize and fund a new program in advance of the start of product development, when the business case would be better established. DOD policy requires that DOD decision makers have information about a proposed program's risk factors and systems engineering status, in a systems engineering plan, at the start of a new program. However, it is not clear whether Congress also has this information at that time. A systems engineering plan could provide more robust information to Congress when considering a budget request to start a new program. In commenting on a draft of this report DOD disagreed.

What GAO Recommends

To support oversight and inform budget decisions, Congress should consider requiring DOD to report on systems engineering status of each major acquisition program in the department's annual budget request.