Previous  GAO reports as well as other DOD and industry studies have also found  that user involvement is critical to the success of any software  development effort. For example, we previously reported that obtaining  frequent feedback is linked to reducing risk, improving customer  commitment, and improving technical staff motivation. Most programs had plans to incorporate these elements of  user engagement throughout their software development efforts, but they  often did not follow those plans due, in part, to the lack of specific  guidance on user involvement and feedback. Selected DOD Programs Often Did Not Effectively Engage Users

The four programs we reviewed often did not demonstrate key  characteristics of effective user engagement as summarized below:
Early engagement. OCX involved users early and JMS planned to  involve users early but, in practice, did not do so; SBIRS and MUOS  did not plan to involve users early in software development. JMS, OCX, and SBIRS all planned to  continually involve users but, in practice, did not fully do so; MUOS  did not plan to do so. OCX and SBIRS  have provided users opportunities to provide such feedback but only  years into software development; JMS and MUOS did not provide  opportunities for feedback. Feedback incorporated into subsequent development. JMS, OCX,  and SBIRS all planned to incorporate user feedback but, in practice,  have not done so throughout development; MUOS did not plan to do  so during software development. As a result of programs’ shortcomings with user involvement and  feedback, programs risk delivering systems that do not meet user needs. The MUOS program  office did not provide users an opportunity to give feedback based on  actual working software. DOD officials have acknowledged these challenges and have  recently begun recommending steps to address them. This was due, in part, to the lack of  specific guidance on the timing, frequency, and documentation for user  involvement and feedback. The lack of user engagement has contributed  to systems that were later found to be operationally unsuitable. Selected programs have also faced challenges in delivering software in  shorter time frames, and in using commercial software, applying outdated  software tools and metrics, and having limited knowledge and training in  newer software development techniques. Recommendations for Executive Action

We are making the following two recommendations to DOD:  The Secretary of Defense should ensure the department’s guidance that  addresses software development provides specific, required direction on  when and how often to involve users so that such involvement is early  and continues through the development of the software and related  program components. Appendix I: Objectives, Scope, and Methodology

Senate and House reports accompanying the National Defense  Authorization Act for Fiscal Year 2017 contained provisions for GAO to  review challenges in software-intensive Department of Defense (DOD)  space systems, among other things. This report addresses, for selected  software-intensive space programs, (1) the extent to which these  programs have involved users and delivered software using newer  development approaches; and (2) what software-specific management  challenges, if any, these programs have faced. From these 15 programs, 8 were found to have had cost growth or  schedule delays attributed, in some part, to software development. Family of Advanced Beyond Line-of-Sight Terminals (FAB-T); Air
Next Generation Operational Control System (OCX); Air Force MDAP  Joint Space Operations Center Mission System Increment 2 (JMS);  Air Force MAIS
Mobile User Objective System (MUOS); Navy MDAP
Space-Based Infrared System (SBIRS); Air Force MDAP  We were unable to assess FAB-T software issues with the same level of  detail as the other programs we reviewed because, despite prior software  challenges, the program stated it does not have documentation that  separately tracks software-related requirements or efforts. To determine how effectively selected DOD software-intensive space  programs have involved users and adopted newer software development  approaches, we reviewed applicable DOD policies, guidance, and federal  statute that identify characteristics of user engagement. We reviewed defect metrics and reports on amounts of new,  reused, inherited, and commercial software; test and evaluation reports;  program management reports; and external program assessments.
Why GAO Did This Study

Over the next 5 years, DOD plans to spend over $65 billion on its space system acquisitions portfolio, including many systems that rely on software for key capabilities. However, software-intensive space systems have had a history of significant schedule delays and billions of dollars in cost growth.
Senate and House reports accompanying the National Defense Authorization Act for Fiscal Year 2017 contained provisions for GAO to review challenges in software-intensive DOD space programs. This report addresses, among other things, (1) the extent to which these programs have involved users; and (2) what software-specific management challenges, if any, programs faced.
To do this work, GAO reviewed four major space defense programs with cost growth or schedule delays caused, in part, by software. GAO reviewed applicable statutes and DOD policies and guidance that identified four characteristics of effective user engagement. GAO reviewed program documentation; and interviewed program officials, contractors, and space systems users. GAO also analyzed program metrics, test and evaluation reports, and external program assessments.

What GAO Found

The four major Department of Defense (DOD) software-intensive space programs that GAO reviewed struggled to effectively engage system users. These programs are the Air Force's Joint Space Operations Center Mission System Increment 2 (JMS), Next Generation Operational Control System (OCX), Space-Based Infrared System (SBIRS); and the Navy's Mobile User Objective System (MUOS). These ongoing programs are estimated to cost billions of dollars, have experienced overruns of up to three times originally estimated cost, and have been in development for periods ranging from 5 to over 20 years. Previous GAO reports, as well as DOD and industry studies, have found that user involvement is critical to the success of any software development effort. For example, GAO previously reported that obtaining frequent feedback is linked to reducing risk, improving customer commitment, and improving technical staff motivation. However, the programs GAO reviewed often did not demonstrate characteristics of effective user engagement that are identified in DOD policy and statute:
Early engagement. OCX involved users early; JMS planned to but, in practice, did not; SBIRS and MUOS did not plan to involve users early.
Continual engagement. JMS, OCX, and SBIRS all planned to continually involve users but, in practice, did not fully do so; MUOS did not plan to do so.
Feedback based on actual working software. OCX and SBIRS provided users opportunities to give such feedback but only years into software development; JMS and MUOS did not provide opportunities for feedback.
Feedback incorporated into subsequent development. JMS, OCX, and SBIRS all planned to incorporate user feedback but, in practice, have not done so throughout development; MUOS did not plan to do so.
As reflected above, actual program efforts to involve users and obtain and incorporate feedback were often unsuccessful. This was due, in part, to the lack of specific guidance on user involvement and feedback. Although DOD policies state that users should be involved and provide feedback on software development projects, they do not provide specific guidance on the timing, frequency, and documentation of such efforts. Without obtaining user feedback and acceptance, programs risk delivering systems that do not meet users' needs. In selected instances, the lack of user involvement has contributed to systems that were later found to be operationally unsuitable.
The programs GAO reviewed also faced software-specific challenges in using commercial software, applying outdated software tools, and having limited knowledge and training in newer software development techniques. For example, programs using commercial software often underestimated the effort required to integrate such software into an overall system. Secondly, selected programs relied on obsolete software tools that they were accustomed to using but which industry had since replaced. Finally, GAO found that two of the reviewed programs lacked knowledge of more modern software development approaches. DOD has acknowledged these challenges and has efforts underway to address each of them.

What GAO Recommends

GAO is making two recommendations that DOD ensure its guidance that addresses software development provides specific, required direction on the timing, frequency, and documentation of user involvement and feedback. DOD concurred with the recommendations.