NASA Projects Have Not Consistently Implemented Key EVM Practices and Most Did Not Have Access to Reliable EVM Data

Our assessment of 10 major spaceflight projects showed that NASA has  not yet fully implemented EVM and thus is not taking full advantage of an  important tool that could help reduce acquisition risk. GAO found that the  projects had shortfalls in two of the three fundamental practices that we  assessed. Specifically, we found that half of the projects did not use an  EVM system that was certified as compliant with the ANSI/EIA-748  standard. However, the rigor of both the formal and  informal surveillance reviews is questionable given the numerous EVM  data anomalies we found in the monthly EVM reports. Out of the 10 projects we reviewed, we found that just 3 projects  had reliable EVM data while the remaining 7 had only partially reliable  data. Unreliable EVM Data Limit NASA’s Ability to Measure Project Performance

Only 3 of the 10 projects we reviewed, MAVEN, RBSP, and OCO-2,  produced fully reliable data for managing the project and reporting status. Cultural, Technical, and Other Challenges Seen as Impediments to EVM Implementation

NASA EVM focal points, headquarters officials, project representatives,  and program executives cited cultural and technical challenges, as well  as other challenges, as impediments to the effective use of EVM at the  agency. NASA’s culture traditionally has focused on solving science and  engineering challenges and not on monitoring cost and schedule data  such as data produced by an effective EVM system. In addition, though  NASA has not conducted an EVM skills gap analysis, NASA  representatives said it is a challenge for the agency to implement EVM  effectively due to a lack of sufficient staff with the skills and experience to  analyze EVM data. Without a sufficient number of trained staff to analyze  contractor data and implement in-house EVM efforts, NASA will likely  continue to struggle to effectively use EVM as a valuable project  management tool. NASA Policy Is in Line with Best Practices but Implementation Remains the Challenge

NASA has undertaken several initiatives aimed at improving the agency’s  use of EVM. For example, NASA strengthened its spaceflight  management policy to require projects to comply with the 32 ANSI/EIA- 748 guidelines and has developed the processes and tools for projects to  meet this requirement through its new EVM system. While these are  positive steps, the policy continues to lack a requirement for rigorous  oversight or surveillance of how projects are implementing EVM and  NASA does not require projects to use the new EVM system to implement  the EVM requirement of the revised policy. Strong Leadership Needed to Fully Implement EVM

Over the years, NASA has attempted to address its EVM shortcomings  through a series of policy changes, but these efforts have failed to  adequately address the cultural resistance to implementing EVM  highlighted by many of the NASA officials we interviewed. However, NASA is  not making full use of a key tool that could help it address the cost and  schedule issues that have kept NASA acquisition management on GAO’s  high risk list for more than 20 years. Recommendations for Executive Action

To improve NASA management and oversight of its spaceflight projects,  we recommend that the NASA Administrator direct the appropriate offices  to take the following four actions:
Establish a time frame by which all new spaceflight projects will be  required to implement NASA’s newly developed EVM system, unless  the project is proposing to use a certified system, to ensure that in- house efforts are compliant with ANSI/EIA-748. Conduct an EVM skills gap analysis to identify areas requiring  augmented capability across the agency. In its written  comments, reproduced in appendix IV, NASA’s Chief Engineer stated that  the agency concurred with two recommendations and partially concurred  with two other recommendations. To determine the steps that NASA is taking to improve its use of earned  value management, we examined the results of NASA’s EVM capability  pilot projects and draft policies and guidance and compared these with  best practices in EVM as discussed in GAO’s Cost Estimating and  Assessment Guide, the ANSI/EIA-748 standard, and OMB Circular A-11,  Preparation, Submission, and Execution of the Budget and the Capital  Programming Guide. As  part of our analysis, we assessed the projects’ implementation of three  fundamental earned value management (EVM) practices that we believe  are necessary for maintaining a reliable EVM system—using a certified  American National Standards Institute (ANSI) and Electronic Industries  Alliance (EIA) compliant system, performing surveillance, and conducting  integrated baseline reviews. EVM Surveillance Is Being Performed

JPL also has a formal surveillance plan in place for monitoring the EVM  data.
Why GAO Did This Study

NASA historically has experienced cost growth and schedule slippage in its portfolio of major projects and has taken actions to improve in this area, including adopting the use of EVM. EVM is a tool developed to help project managers monitor risks. GAO was asked to examine (1) the extent to which NASA is using EVM to manage its major space flight acquisitions, (2) the challenges that NASA has faced in implementing an effective EVM system, and (3) NASA's efforts to improve its use of EVM. To address these questions, GAO obtained contractor and project EVM data and used established formulas and tools to analyze the data and assess NASA's implementation of EVM on 10 major spaceflight projects; interviewed relevant NASA headquarters, center and mission directorate officials on their views on EVM; and reviewed prior reports on EVM and organizational transformations. GAO compared NASA policies and guidance on EVM to best practices contained in GAO's cost estimating best practices guide.

What GAO Found

The National Aeronautics and Space Administration's (NASA) 10 major spaceflight projects discussed in this report have not yet fully implemented earned value management (EVM). As a result, NASA is not taking full advantage of opportunities to use an important tool that could help reduce acquisition risk. GAO assessed the 10 projects against three fundamental EVM practices that, according to GAO's best practices cost guide, are necessary for maintaining a reliable EVM system. GAO found shortfalls in two of three fundamental practices. Specifically, we found that
More than half of the projects did not use an EVM system that was fully certified as compliant with the industry EVM standard.
Only 4 of the 10 projects established formal surveillance reviews, which ensure that key data produced by the system was reliable. The remaining 6 projects provided evidence of monthly EVM data reviews; however, the rigor of both the formal and informal surveillance reviews is questionable given the numerous data anomalies GAO found.
GAO also found that 3 projects had reliable EVM data while 7 had only partially reliable data. For the EVM data to be considered reliable per best practices it must be complete and accurate with all data anomalies explained.
NASA EVM focal points, headquarters officials, project representatives, and program executives cited cultural and other challenges as impediments to the effective use of EVM at the agency. Traditionally, NASA's culture has focused on managing science and engineering challenges and not on monitoring cost and schedule data, like an effective EVM system produces. As a result, several representatives said this information traditionally has not been valued across the agency. This sentiment was also echoed in a NASA study of EVM implementation. Also cited as a challenge to the effective use of EVM was NASA's insufficient number of staff with the skills to analyze EVM data. Without a sufficient number of staff with such skills, NASA's ability to conduct a sound analysis of the EVM data is limited. However, NASA has not conducted an EVM skills gap analysis to determine the extent of its workforce needs.
NASA has undertaken several initiatives aimed at improving the agency's use of EVM. For example, NASA strengthened its spaceflight management policy to reflect the industry EVM standard and has developed the processes and tools for projects to meet these standards through its new EVM system. While these are positive steps, the revised policy contains only the minimum requirements for earned value management. For example, it lacks a requirement for rigorous surveillance of how projects are implementing EVM and also does not require use of the agency's newly developed EVM system to help meet the new requirements. NASA has attempted to address EVM shortcomings through policy changes over the years, but these efforts have failed to adequately address the cultural resistance to implementing EVM.

What GAO Recommends

GAO recommends that NASA establish a time frame for requiring new spaceflight projects to implement its new EVM system; conduct an EVM skills gap assessment; develop a change management plan for EVM; and strengthen its EVM requirements by requiring projects to implement formal EVM surveillance. NASA concurred with two recommendations and partially concurred with two others citing resource constraints. Despite NASA's plans to address some issues GAO identified, not addressing all key issues lessens the usefulness of EVM at NASA.