JROC Did Not Always Consider Trade-offs or Influence Trade-off Decisions

The JROC considered trade-offs made by the military services before  validating requirements for four of the seven proposed programs it  reviewed in fiscal year 2010, and provided input to the military services on  the cost, schedule, and performance objectives for two of the seven  programs. As a  result, a program can spend significant time in technology development  before the JROC gets to formally weigh in on these trade-offs through the  JCIDS process. JCIDS Reviews Are Not Aligned with the Most Significant Trade-off Decisions

The JROC does not formally review the trade-off decisions made as a  result of an AOA until a proposed program’s CDD enters the JCIDS  process. According to DOD officials, the most significant trade-offs are  made by the military services between ICD and CDD reviews during the  AOA, which is intended to compare the operational effectiveness, cost,  and risks of a number of alternative potential solutions. A significant amount of time and  resources can be expended before the JROC gets to weigh in on these  trade-offs during CDD reviews. We found the estimates presented  to the JROC were often unreliable when assessed against best practices  criteria. In most cases, the military services had not effectively  conducted uncertainty and sensitivity analyses, which establish  confidence levels for resource estimates, based on the knowledge  available, and examine the effects of changing assumptions and ground  rules. The lack of this information affects the JROC’s efforts to  ensure that programs are fully funded and its ability to consider the  resource implications of cost, schedule, and performance trade-offs. JROC Received Resource Estimates That Did Not Meet Best Practices

The JROC first receives resource estimates for proposed programs when it  reviews CDDs, and when we reviewed the CDD resource estimates  presented to the JROC in fiscal year 2010, we found that they were  generally unreliable when assessed against our best practices criteria. JROC Did Not Consistently Prioritize Requirements and Capability Gaps

The JROC does not currently prioritize requirements, consider  redundancies across proposed programs, or prioritize and analyze  capability gaps in a consistent manner. As a result, the Joint Staff is  missing an opportunity to improve military service and departmentwide  portfolio management efforts. According to Army, Air Force, and Navy officials,  having a better understanding of warfighter priorities from the JROC  would be useful to inform both portfolio management efforts and service  budgets. A DOD review team examining the JCIDS process is considering  changes that would address the prioritization of requirements. Recommendations for Executive Action

To enhance the JROC’s role in DOD-wide efforts to deliver better value to  the taxpayer and warfighter, we recommend that the Vice Chairman of the  Joint Chiefs of Staff, as chairman of the JROC, take the following five  actions:    Establish a mechanism to review the final AOA report prior to  Milestone A to ensure that trade-offs have been considered and to  provide military advice on these trade-offs and the proposed materiel  solution to the Milestone Decision Authority. Appendix I: Scope and Methodology

To conduct our work, we reviewed relevant sections of Title 10 of the U.S.  Code, the Weapon Systems Acquisition Reform Act of 2009 (WSARA), and  the National Defense Authorization Act for Fiscal Year 2008 to establish  the role of the Joint Requirements Oversight Council (JROC) in  considering trade-offs among cost, schedule, and performance objectives;  reviewing the estimated level of resources needed to fulfill these  requirements; and prioritizing requirements. We then focused our analysis on the unclassified  requirement documents reviewed by the JROC and JCB in fiscal year 2010  which identified capability gaps or defined performance requirements for  new major defense acquisition programs: 13 ICDs and 7 CDDs.
Why GAO Did This Study

The Weapon Systems Acquisition Reform Act of 2009 (WSARA) directed the Joint Requirements Oversight Council (JROC) to ensure trade-offs among cost, schedule, and performance objectives are considered as part of its requirements review process. WSARA also directed GAO to assess the implementation of these requirements. This report addresses (1) the extent to which the JROC has considered trade-offs within programs, (2) the quality of resource estimates presented to the JROC, and (3) the extent to which the JROC is prioritizing requirements and capability gaps. To do so, GAO analyzed requirement documents reviewed by the JROC in fiscal year 2010, which identified capability gaps or performance requirements for new major defense acquisition programs. GAO also assessed resource estimates presented to the JROC against best practices criteria in the GAO Cost Estimating and Assessment Guide.

What GAO Found

The JROC considered trade-offs made by the military services before validating requirements for four of the seven proposed programs it reviewed in fiscal year 2010. According to DOD officials, the most significant trade-offs are made by the military services during the analysis of alternatives (AOA), which occurs between the JROC's review of an Initial Capabilities Document (ICD) and its review of a Capability Development Document (CDD). The AOA is intended to compare the operational effectiveness, cost, and risks of a number of alternative potential solutions. The JROC does not formally review the trade-off decisions made as a result of an AOA until it reviews a proposed program's CDD. As a result, the JROC does not have an opportunity to provide military advice on trade-offs and the proposed solution before it is selected, and a significant amount of time and resources can be expended in technology development before the JROC gets to formally weigh in. The military services did not consistently provide high-quality resource estimates to the JROC for proposed programs in fiscal year 2010. GAO found the estimates presented to the JROC were often unreliable when assessed against best practices criteria. In most cases, the military services had not effectively conducted uncertainty and sensitivity analyses or examined the effects of changing assumptions and ground rules, all of which could further the JROC's efforts to ensure that programs are fully funded and provide a sound basis for making cost, schedule, and performance trade-offs. The JROC does not currently prioritize requirements, consider redundancies across proposed programs, or prioritize and analyze capability gaps in a consistent manner. As a result, the Joint Staff is missing an opportunity to improve the management of DOD's joint portfolio of weapon programs. According to Army, Air Force, and Navy officials, having a better understanding of warfighter priorities from the JROC would be useful to inform both portfolio management efforts and service budgets. A DOD review team examining the JROC's requirements review process is considering changes that would address the prioritization of requirements on a departmentwide basis.

What GAO Recommends

GAO recommends that the JROC establish a mechanism to review AOA results earlier in the acquisition process, require higher quality resource estimates from requirements sponsors, prioritize requirements across proposed programs, and address potential redundancies during requirements reviews. The Joint Staff partially concurred with GAO's recommendations and generally agreed with their intent, but differed with GAO on how to implement them.