NASA is now implementing the recommendations  from this study within the Constellation Program, which includes three  major development projects—the Ares I Crew Launch Vehicle, the Orion  Crew Exploration Vehicle, and the Ares V Cargo Launch Vehicle as shown  in figure 1. To reduce cost and minimize risk in developing these projects, NASA  planned to maximize the use of heritage systems and technology. NASA has completed the system  requirements review for each project and is in the midst of finalizing the  system definition reviews. NASA is using its Web-based Integrated Risk Management Application to  help monitor and mitigate the risks with the Ares I and Orion development  efforts and for the overall Constellation Program. Requirements Setting is a Primary Challenge for Both the Ares I and Orion Projects

Although project level requirements were baselined at both systems  requirements reviews, continued uncertainty about the systems’  requirements have led to considerable unknowns as to whether NASA’s  plans for the Ares I and Orion vehicles can be executed within schedule  goals, as well as what these efforts will ultimately cost. Such uncertainty  has created knowledge gaps that are affecting many aspects of both  projects. Both the Orion and Ares I vehicles  have a history of weight and mass growth, and NASA is still defining the  mass, loads, and weight requirements for both vehicles. With regard to the Orion project, there is currently no industry  capability for producing a thermal protection system of the size required  by the Orion. The table 1 describes these and other  examples of knowledge gaps in the development of the Ares I and Orion  vehicles. NASA is working under a self-imposed deadline to deliver the  new launch vehicles no later than 2015 in order to minimize the gap in  human spaceflight between the Space Shuttle’s retirement in 2010 and the  availability of new transportation vehicles. Orion

Cost and schedule reporting on the Orion project indicates that the Orion  project’s efforts to mature requirements and design and to resolve weight  issues is placing pressure on the Orion schedule. Test Facilities for Ares I and Orion Insufficient

According to NASA, at this time, existing test facilities are insufficient to  adequately test the Ares I and Orion systems. To address this issue, NASA is in the process of constructing a new  altitude test facility at Stennis Space Center for the J-2X. Also, current  facilities are inadequate to replicate the Orion vibration and acoustic  environment. Our prior  work has shown that investment decisions of this magnitude need to be  based on an established and executable business case and that there are  several key indicators that Congress could be informed of to assess  progress throughout development. Our previous work on government-funded space systems has shown that  weight growth is often not anticipated even though it is among the highest  drivers of cost growth for space systems. The complexity of software development on a system, often denoted by  the number of lines of code on a system, can also be used as an indicator  to monitor whether a program will meet cost and schedule goals. Indicators that Can be Used to Assess Knowledge Gap at Key  Junctures  Additionally, since the mid-1990s, GAO has studied the best practices of  leading commercial companies. We have highlighted the three critical junctures at which developers must  have knowledge to make large investment decisions—the preliminary  design review, the critical design review, and the production review—and  the numerous key indicators that can be used to increase the chances of  successful outcomes. One approach  to ensure that technology readiness is reliably assessed is to use  independent testing;    Project requirements are defined and informed by the systems    Cost and schedule estimates established for the project are based on  knowledge from the preliminary design using systems engineering  tools;    Additional resources are in place, including needed workforce, and a  decision review is conducted following completion of the preliminary  design review. A critical enabler for success in this phase of development is performance  and requirements flexibility. To provide oversight at this juncture, NASA could provide  Congress with information to verify that the following indicators have  been met:    At least 90 percent of engineering drawings are complete;    All subsystem and system design reviews have been completed;    The design meets requirements demonstrated through modeling,  simulation, or prototypes;    Stakeholders’ concurrence that drawings are complete and producible    Failure modes and effects analysis have been completed;    Key system characteristics are identified;    Critical manufacturing processes are identified;    Reliability targets are established and a growth plan based on  demonstrated reliability rates of components and subsystems is  developed; and    A decision review is conducted following the completion of the critical  design review. In addition, we have reported that this  type of approach is being embraced by the Ares I project.
Why GAO Did This Study

The National Aeronautics and Space Administration (NASA) is in the midst of two new development efforts as part of the Constellation Program--the Ares I Crew Launch Vehicle and the Orion Crew Exploration Vehicle. These projects are critical to the success of the overall program, which will return humans to spaceflight after Space Shuttle retirement in 2010. To reduce the gap in human spaceflight, NASA plans to launch Ares I and Orion in 2015--5 years after the Shuttle's retirement. GAO has issued a number of reports and testimonies that touch on various aspects of NASA's Constellation Program, particularly the development efforts underway for the Orion and Ares I projects. These reports and testimonies have questioned the affordability and overall acquisition strategy for each project. NASA has revised the Orion acquisition strategy and delayed the Ares I preliminary design review based on GAO's recommendations in these reports. In addition, GAO continues to monitor these projects on an ongoing basis at the request of members of Congress. Based on this work, GAO was asked to testify on the types of challenges that NASA faces in developing the Ares I and Orion vehicles and identify the key indicators that decision makers could use to assess risks associated with common trouble spots in development. The information in this testimony is based on work completed in accordance with generally accepted government auditing standards.

What GAO Found

NASA is currently working toward preliminary design reviews for the Ares I and Orion vehicles. While this is a phase for discovery and risk reduction, there are considerable unknowns as to whether NASA's plans for these vehicles can be executed within schedule goals and what these efforts will ultimately cost. This is primarily because NASA is still in the process of defining many performance requirements. Such uncertainties could affect the mass, loads, and weight requirements for the vehicles. NASA is aiming to complete this process in 2008, but it will be challenged to do so given the level of knowledge that still needs to be attained. The challenges NASA is facing pose risks to the successful outcome of the projects. For example: both vehicles have a history of weight issues; excessive vibration during launch threatens system design; Uncertainty about how flight characteristics will be impacted by a fifth segment added to the Ares I launch vehicle; Ares I upper stage essentially requires development of a new engine; no industry capability currently exists for producing the kind of heat shields that the Orion will need for protecting the crew exploration vehicle when it reenters Earth's atmosphere; and existing test facilities are insufficient for testing Ares I's new engine, for replicating the engine's vibration and acoustic environment, and for testing the thermal protection system for the Orion vehicle. All these unknowns, as well as others, leave NASA in the position of being unable to provide firm cost estimates for the projects at this point. Meanwhile, tight deadlines are putting additional pressure on both the Ares I and Orion projects. Future requirements changes raise risks that both projects could experience cost and schedule problems. GAO's past work on space systems acquisition and the practices of leading developers identifies best practices that can provide decision makers with insight into the progress of development at key junctures, facilitate Congressional oversight, and support informed decision making. This work has also identified common red flags throughout development, which decision makers need to keep in mind when assessing the projects. They include: Key indicators: Weight growth is often among the highest drivers of cost growth. Unanticipated software complexity, often indicated by increases in the number of lines of code, can portend cost and schedule growth. Key junctures: The preliminary design review, critical design review, and production review are key junctures that involve numerous steps and help focus the agency on realistic accomplishments within reachable goals. A disciplined approach aligned with key indicators can provide the knowledge needed to make informed investment decisions at each review.