Most Programs Have Not Conducted a Robust Assessment of Alternatives

Most of the programs we reviewed either did not conduct an AOA or  conducted an AOA that focused on a narrow scope of alternatives and did  not adequately assess and compare technical and other risks of each  alternative. While many factors can affect program cost and schedule  outcomes, we found that programs that conducted a limited assessment of  alternatives before the start of system development tended to experience  poorer outcomes than the programs that conducted more robust AOAs. Many AOAs Have Not Adequately Assessed Risks for the Alternatives

DOD acquisition policy requires that AOAs assess the technical risk of  alternatives, but it does not provide criteria and guidance for how and to  what extent technical risks should be addressed and it does not specify  that other types of risks should be assessed. We found that  programs with AOAs that conducted a more comprehensive assessment of  risks tended to have better cost and schedule outcomes than those that did  not (see table 5). Choosing an Alternative Too Early and Conducting AOAs under Compressed Time Frames and without Effective Guidance Limit the Scope and Quality of AOAs

We identified several factors that may have limited the effectiveness of  AOAs and their ability to identify the most promising option and  contribute to a sound business case for starting a weapon system program:  (1) service sponsors lock into a solution early on when a capability need is  first validated through DOD’s requirements process and before an AOA is  conducted; (2) AOAs are conducted under compressed time frames in  order to meet a planned milestone review or fielding date and their results  come too late to inform key trade off decisions; and (3) DOD does not  always provide guidance for conducting individual AOAs. The identification of a potential concept is  intended to provide a general approach for addressing the gap and set the  stage for a more in-depth assessment of alternatives to be conducted in the  AOA. In 9 of the 22 programs we reviewed that  had AOAs, the timing of the AOAs was compressed or concurrent with  other planning activities. Ultimately, the Future Combat Systems AOA was completed 1  month after the operational requirements were validated and the same  month that the program was approved to begin system development,  which precluded trade off discussions among cost, performance, and risks  from taking place. The  guidance is intended to ensure that the services are examining a sufficient  number of alternatives that take into consideration joint plans and  interoperability, but to also ensure that AOAs are analyzing key risks such  as technology, cost, and schedule. Recent DOD Policy Changes Could Improve AOA Effectiveness

In December 2008, DOD revised its acquisition policy and introduced  several initiatives based in part on direction from Congress that could  provide a better foundation for establishing knowledge-based business  cases for initiating weapon system programs. Improving the effectiveness of AOAs will depend on DOD’s  ability to ensure that its policy changes are consistently implemented and  reflected in decisions on individual weapon system programs. DOD’s revised policies, for example, may help mitigate service sponsors  from locking into a solution too early in the process by eliminating the  functional solutions analysis in a capability proposal, which identified a  preferred solution and influenced the scope of alternatives in an AOA. Many of these problems could be avoided if programs started with sound,  knowledge-based business cases. The  revisions, for example, should help ensure that DOD direction is provided  before AOAs are started and that AOAs are conducted at an early point in  the acquisition process where their results can inform key decisions  affecting program initiation. Appendix I: Scope and Methodology

To assess whether analyses of alternatives (AOA) have been effective in  identifying the most promising options and providing a sound rationale for  program initiation, we analyzed data and documents for Acquisition  Category (ACAT) I programs that have been initiated between fiscal years  (FY) 2003 and 2008 and were in the Department of Defense’s (DOD) FY  2008 Major Defense Acquisition Program (MDAP) list.
Why GAO Did This Study

Department of Defense (DOD) weapon programs often experience significant cost and schedule problems because they are allowed to start with too many technical unknowns and not enough knowledge about the development and production risks they entail. GAO was asked to review the department's Analysis of Alternatives (AOA) process--a key first step in the acquisition process intended to assess the operational effectiveness, costs, and risks of alternative weapon system solutions for addressing a validated warfighting need. This report (1) examines whether AOAs have been effective in identifying the most promising options and providing a sound rationale for weapon program initiation, (2) determines what factors have affected the scope and quality of AOAs, and (3) assesses whether recent DOD policy changes will enhance the effectiveness of AOAs. To meet these objectives, GAO efforts included collecting information on AOAs from 32 major defense acquisition programs, reviewing guidance and other documents, and interviewing subject matter experts.

What GAO Found

Although an AOA is just one of several inputs required to initiate a weapon system program, a robust AOA can be a key element to ensure that new programs have a sound, executable business case. Many of the AOAs that GAO reviewed did not effectively consider a broad range of alternatives for addressing a warfighting need or assess technical and other risks associated with each alternative. For example, the AOA for the Future Combat System program, one of DOD's large and most complex development efforts, analyzed the operational performance and cost of its alternatives but failed to compare the technical feasibility and risks, assuming that the technologies would perform as forecasted. Without a sufficient comparison of alternatives and focus on technical and other risks, AOAs may identify solutions that are not feasible and decision makers may approve programs based on limited knowledge. While many factors can affect cost and schedule outcomes, we found that programs that had a limited assessment of alternatives tended to have poorer outcomes than those that had more robust AOAs. The narrow scope and limited risk analyses in AOAs can be attributed in part to program sponsors choosing a solution too early in the process, the compressed timeframes that AOAs are conducted under, and the lack of guidance for conducting AOAs. While AOAs are supposed to provide a reliable and objective assessment of viable weapon solutions, we found that service sponsors sometimes identify a preferred solution or a narrow range of solutions early on, before an AOA is conducted. The timing of AOAs has also been problematic. Some AOAs are conducted under compressed timeframes in order to meet a planned milestone or weapon system fielding date and are conducted concurrently with other key activities required to become a program of record. This can short-change a comprehensive assessment of risks and preclude effective cost, schedule, and performance trade offs from taking place prior to beginning development. Furthermore, while DOD has an opportunity to influence the scope and quality of AOAs, it has not always provided guidance for conducting individual AOAs. Recognizing the need for more discipline in weapon systems acquisition, DOD recently revised its overall acquisition and requirements policies. If implemented properly, the revised policies could provide a better foundation for planning and starting new programs with sound, knowledge-based business cases. Included in the revised acquisition policy are several mechanisms to improve the AOA process. For example, the policy revisions should help ensure that DOD direction is provided before AOAs are started and that they are conducted at an early point in the acquisition process where their results can inform decisions affecting program initiation. While these policy changes are promising, DOD must ensure that they are consistently implemented and reflected in decisions on individual programs. Furthermore, more specific criteria and guidance for how AOAs should be conducted may need to be developed to ensure they meet their intended objectives and provide an in-depth assessment of alternatives.