Most Selected MAIS Programs Had Changes in Their Planned Cost or Schedule Estimates, and Half Met Their Technical Performance Targets

The majority of the 18 selected MAIS programs that we reviewed had  experienced changes in their planned cost estimates and in their planned  schedule estimates when comparing the first acquisition program  baseline to the most recent acquisition program baseline estimate. Further, nine of the programs had met all of their  technical performance targets, while four of the programs had partially  met the performance targets. The remaining five programs had not yet  conducted testing activities. In addition, 11 of the 16 programs had experienced cost increases. Among the 11  programs that reported an increase, the average increase was $457.2  million. Two of the Five MAIS Programs Had Not Fully Implemented All Practices for Managing Requirements

Leading requirements management practices include establishing an  agreed-upon set of requirements, ensuring traceability between  requirements and work products, and managing any changes to the  requirements in collaboration with stakeholders. Three of the five programs had fully implemented the requirements  management practices, while the other two had partially implemented  some practices. Two of the Five MAIS Programs Had Not Fully Implemented All Practices for Managing Risk

According to leading practices, an effective risk management process  identifies potential problems before they occur, so that risk-handling  activities may be planned and invoked, as needed, across the life of the  project in order to mitigate the potential for adverse impacts. Three of the five programs had fully implemented the risk management  practices, while two had partially implemented some practices. On the other hand, the program did not define thresholds or bounds for  mitigation or management action. Four of the Five MAIS Programs Had Fully Implemented All Practices for Managing Systems Testing and Integration

According to leading practices for managing systems testing and  integration activities, roles and responsibilities should be established and  test-related plans, schedules, and reports should be developed. Although the program had  established roles and responsibilities to manage testing and integration  activities, a chief developmental tester to oversee testing activities had  not been identified. While a number of leading  practices for risk, requirements, and systems testing and integration had  been fully or partially implemented by five programs that we reviewed in- depth—the Air Force’s AOC WS Inc 10.2, BITI Wired and JMS Inc 2  programs; and the Army’s GCSS-A Inc 1 and IPPS-A Inc 2 programs— three programs lacked practices to better manage the development of  systems. In addition, the  program office for AOC-WS Inc 10.2 did not have an overall Risk  Mitigation Plan to guide the implementation of individual risk mitigation  and contingency planning activities. Unless this position is filled, it will be difficult for the program  to effectively manage risks and verify compliance with certain operational  requirements throughout the acquisition. Recommendations for Executive Action

To help improve the management of DOD’s MAIS programs, we are  recommending that the Secretary of Defense take the following three  actions: direct the Secretary of the Army to direct the program manager for  GCSS-A Inc 1 to establish standard operating procedures for  managing risks that include guidance for establishing thresholds and  bounds for key risk areas. Appendix I: Objectives, Scope, and Methodology

The National Defense Authorization Act for Fiscal Year 2012 mandated  that we select, assess, and report on Department of Defense’s (DOD)  major automated information system (MAIS) programs annually through  March 2018. Our objectives were to: (1) describe the extent to which selected MAIS  programs have changed their planned cost and schedule estimates and  met technical performance targets and (2) assess the extent to which  selected MAIS programs have used leading IT acquisition practices,  including requirements and risk management, and systems testing and  integration. The IPPS-A Inc 2 program was still early in development and system  performance data was not available.
Why GAO Did This Study

DOD's MAIS programs include systems that are intended to help the department sustain its key operations. The National Defense Authorization Act for Fiscal Year 2012 includes a provision for GAO to select, assess, and report on the department's MAIS programs annually through March 2018. This is GAO's fifth report and (1) describes the extent to which selected MAIS programs have changed their planned cost and schedule estimates and met technical performance targets and (2) assesses the extent to which selected MAIS programs have used leading IT acquisition practices, including risk management.
GAO selected and reviewed cost, schedule, and performance data for 18 of DOD's MAIS programs that were non-classified and had an acquisition performance baseline. In addition, GAO performed an in-depth review of 5 of the programs, comparing selected IT management practices used by them to leading practices for requirements and risk management and systems testing and integration. The five selected programs were from at least two military services and had not been assessed by GAO in the past year. GAO also interviewed relevant program officials.

What GAO Found

Most of the 18 selected Department of Defense (DOD) major automated information system (MAIS) programs that GAO reviewed had experienced changes in their planned cost and schedule estimates and half of the programs had met their technical performance targets. Specifically, 16 programs experienced changes in their cost estimates ranging from a 39 percent decrease ($1.47 billion) to a 469 percent increase ($1.63 billion). The average cost increase was $457.2 million among the 11 programs reporting an increase. Fourteen programs experienced schedule delays, which ranged from 2 months to over 13 years. Finally, half of the MAIS programs fully met all of their technical performance targets. Of the remaining nine programs, 4 four had partially met their target because each was still conducting tests. The other five programs were in the early stages of system development and had not begun testing.
In addition, for the five MAIS programs GAO selected for in-depth review, all had either fully or partially applied leading practices for managing requirements, risks, and for conducting systems testing and integration.
Managing requirements. Three of the five programs had fully implemented the practices for managing requirements, while the other two had partially implemented some practices. Leading practices in this area include establishing requirements and ensuring traceability between requirements and work products.
Managing risks. Three of the five programs had fully implemented the risk management practices, while two had partially implemented some practices. An effective risk management process identifies potential problems before they occur. For example, one Army program did not have standard operating procedures for defining thresholds or bounds to manage risk. Unless such procedures are defined, the program will not have the tools needed to define risk management activities, including whether and how certain risks are prioritized. Further, programs should include practices to identify potential problems so that risk-handling activities may be planned and invoked across the project to mitigate the potential for adverse impacts. However, one Air Force program did not develop an overall risk mitigation plan to guide the implementation of individual risk mitigation activities. Without an overall risk plan to guide individual development efforts, those efforts cannot be managed cohesively.
Testing and integration. Four of the five programs had fully implemented practices for systems testing and integration. Programs should, among other activities, establish roles and responsibilities to manage testing and integration activities, including a chief developmental tester to oversee testing activities. However, one Air Force program reported difficulty in hiring a qualified individual to perform these duties. Until this position is filled, the program may not effectively manage risks and verify compliance with system acquisition and operational requirements.

What GAO Recommends

GAO recommends that DOD improve the management of specific MAIS programs, including establishing procedures for defining risk thresholds, developing an overall risk mitigation plan, and filling a key test management position. DOD concurred with all of the recommendations.