MDA is responsible for developing a number of systems, known as elements, with the purpose of defending against ballistic missile attacks. MDA’s mission is to combine these elements into an integrated system- of-systems, known as the Ballistic Missile Defense System. Specifically, the goal of the BMDS is to combine the abilities of two or more elements to achieve objectives that would not have been possible for any individual element. These emergent abilities are known as “integrated capabilities” or “BMDS-level capabilities.” Table 1 provides a list and description of elements included in our review. When MDA was established in 2002, it was granted exceptional flexibilities to set requirements and manage the acquisition of the BMDS—developed as a single program—that allow MDA to expedite the fielding of assets and integrated ballistic missile defense capabilities. These flexibilities allow MDA to diverge from DOD’s traditional acquisition life cycle and defer the application of acquisition policies and laws designed to facilitate oversight and accountability until a mature capability is ready to be handed over to a military service for production and operation. Some of the laws and policies include such things as: obtaining the approval of a higher-level acquisition executive before making changes to an approved baseline, reporting certain increases in unit cost measured from the original or current baseline, obtaining an independent life-cycle cost estimate prior to beginning system development and/or production and deployment, and regularly providing detailed program status information to Congress, including specific costs, in Selected Acquisition Reports. In response to concerns related to MDA’s flexibilities, Congress and DOD have taken a number of actions. For example, Congress enacted legislation in 2008 requiring MDA to establish cost, schedule, and performance baselines—starting points against which to measure progress—for each element that has entered the equivalent of system development or is being produced or acquired for operational fielding. MDA reported its newly established baselines to Congress for the first time in its June 2010 BMDS Accountability Report. Since that time, Congress has provided more detailed requirements for the content of these baselines. Additionally, to enhance oversight of the information provided in the BMDS Accountability Report, MDA continues to incorporate suggestions and recommendations from us. However, not all of our recommendations have been fully implemented. Because MDA is not a military service, it does not abide by the same policies that the services use for delivering capabilities. Instead, a process exists whereby MDA declares an asset or capability ready for delivery for potential operational use. During this process, MDA communicates the capabilities and limitations of its delivery, and provides evidence supporting these assertions. Representatives from the receiving military service or combatant command then have the ability to assess this evidence and decide whether to accept the new capability. Because the military services conduct minimal missile defense testing of their own, this process is one of the only ways to convey vital performance information. The accuracy of this information is especially important as it informs training materials, doctrine, and deployment decisions. Typically, MDA makes capability deliveries through approved changes to its Operational Capacity Baseline (OCB). Proposed changes to the baseline are coordinated with the warfighter, including the affected combatant commands. Subsequently, the combatant commands assess these element capabilities to determine whether to accept them. This process is used for the vast majority of deliveries, including relatively minor ones such as software patches and updates. In recent years, MDA has declared major capabilities ready for delivery through a process that culminates in the issuance of a Technical Capability Declaration (TCD). According to MDA officials, the primary purpose of a TCD is to allow MDA’s senior management to manage the delivery of integrated, BMDS-level capabilities that require more than one element to function; however, TCDs have also been issued in response to mandates from the President. Though MDA has flexibilities in managing the acquisition process, it must follow the same contracting regulations that apply to DOD, including the Federal Acquisition Regulation and the Department of Defense Federal Acquisition Regulation Supplement (DFARS). We reviewed MDA’s use of a particular type of contract action that authorizes a contractor to begin work before contract terms, specifications, or price have been agreed upon. These “undefinitized contract actions” are permitted by the DFARS, with certain limitations. Undefinitized contract actions are generally used when negotiation of a definitive contract action is not possible in sufficient time to meet the government’s requirements and the government’s interest demands that the contractor be given a binding commitment so that contract performance can begin immediately. Under the DFARS, undefinitized contract actions must include a specific “not-to-exceed” price. Once the action’s terms, specifications, and price have been agreed upon or determined, a process known as definitization, the contract action converts to a “definitive” contract. Under the DFARS, undefinitized contract actions must contain definitization schedules that provide for definitization by the earlier of (1) 180 days after issuance or (2) the date on which the amount of funds obligated under the action is more than 50 percent of the not-to-exceed price. Once the government has received a qualifying proposal from the contractor, however, the government can extend the undefinitized period another 180 days. Similarly, the government may obligate up to 75 percent of the not-to-exceed price, if the contractor submits the qualifying proposal before 50 percent of the not-to-exceed price has been obligated. The amount of funds obligated should be consistent with the contractor’s requirements for the undefinitized period. Figure 1 shows the expected time frame and amount the government should spend within a specified period. The BMDS is a system of systems that cannot be completely assessed using intercept flight tests that are operationally representative because of the system’s scope and complexity and safety constraints. Consequently, MDA, independent DOD testing organizations, and the warfighter must rely heavily on representations of the integrated BMDS called models and simulations in ground testing, rather than live tests, to test the operational performance of the whole BMDS against attacks with more threats represented. In ground testing, each BMDS element is represented by a model and connected to a computer framework. During ground test execution, a model of threat ballistic missiles is applied to the framework and stimulates the modeled representations of BMDS elements to react. The resulting simulation models a BMDS engagement. Figure 1 illustrates the BMDS ground test sequence. To ensure that BMDS models and simulations accurately represent the real-world operational BMDS capabilities and that the limitations of the model are understood, they are verified, validated, and accredited. The verification, validation, and accreditation process is designed to identify and gather evidence needed to certify that the model and its associated data used in ground testing are acceptable for operational testing. No model is completely representative of the real world so the verification, validation, and accreditation process is used to assess the extent to which it reflects the operational performance of the BMDS in the real world, and how any modeling deficiencies impacted ground test results. Any modeling limitations identified in the verification, validation, and accreditation process restrict the extent to which ground test data can be used for BMDS assessment. For example, limitations in modeled sensor tracking of the threat restrict the extent to which tracking data can be relied on for interpreting operational real-world performance. Figure 2 illustrates the verification, validation, and accreditation process. The BMDS Operational Test Agency (OTA) is responsible for analyzing the verification and validation data for the models used in operational BMDS tests and provides accreditation recommendations to the Commanding General, Army Test and Evaluation Command, an independent accreditation authority for operational testing. In this role, the BMDS OTA develops accreditation criteria and assesses if the model can be used for operational assessments against these criteria. The BMDS OTA is also responsible for analyzing the extent to which the threat model, once it is applied to the ground testing framework, can be traced back to the threat model that MDA developed and the intelligence community’s description of the threat. In fiscal year 2017, MDA made some progress delivering assets, including BMDS-level capabilities and conducting tests. However, MDA did not meet many of its goals as expressed in the Ballistic Missile Defense System Accountability Report for fiscal year 2017, its integrated master test plan, and master integration plan. Specifically, MDA continued to deliver interceptors for three elements and successfully conducted its first test against an intercontinental ballistic missile target. In addition, MDA announced the delivery of one package of integrated BMDS-level capabilities through a technical capability declaration (TCD), which had been delayed from the previous year, and planned to complete the delivery of another set of capabilities by March 2018. MDA, however, did not complete its goals for delivering assets, specifically for the THAAD interceptors or conducting planned testing for Aegis BMD. We also identified several deficiencies in MDA’s processes for communicating progress in delivering integrated capabilities. MDA made progress delivering assets against its backlogs from fiscal year 2016, while its test program achieved several notable milestones. MDA also delivered several new integrated capabilities, though not always on time and often with reduced content compared to what was planned to be delivered. In addition, not all deliveries and testing objectives were met, and MDA made a number of changes, additions, and deletions to its test and capability delivery schedule during the year. Elements: While BMDS elements made progress delivering assets, including some that were delayed from fiscal year 2016, MDA did not meet all of its asset delivery goals as planned. For a summary of MDA’s major asset deliveries for fiscal year 2017, see table 2 below. Both the Aegis Standard Missile-3 (SM-3) Block IB and Ground-based Midcourse Defense (GMD) programs succeeded in achieving their asset delivery goals for the fiscal year, although both included acceptance of assets delayed from prior fiscal years. Specifically, due to quality issues and design problems discovered during testing, production on the Aegis SM-3 Block IB interceptor was temporarily halted in fiscal year 2016, and as a result MDA fell short of its deliveries for that year by 15 interceptors. To make up for this, MDA rolled over an additional 15 interceptor deliveries into fiscal year 2017, for a total delivery of 55 interceptors. In addition, MDA achieved its goal of delivering 44 ground-based interceptors by the end of calendar year 2017. However, some programs that achieved their milestones continued to employ high-risk approaches to acquisition, which we have recommended MDA reduce in previous reports. In addition, MDA maintains an ambitious schedule for key programs, such as for GMD’s Redesigned Kill Vehicle program. For more information regarding specific programs, see appendixes II through X. Other MDA elements missed asset delivery milestones. The Command, Control, Battle Management, and Communications (C2BMC) software spiral (or version) 8.2-1 was previously due to be delivered in October 2017, but was delayed again from its new date of December 2017 to second quarter of 2018. This spiral will play an important role in several tests of integrated capabilities, such as FTM-29, which was executed in January 2018. The Terminal High Altitude Area Defense (THAAD) program’s delivery of interceptor Lot 6 was scheduled to be delivered by the end of June 2017, but has since been delayed to the second quarter of 2018. THAAD officials stated this delay was due to a component production issue as well as the addition of 12 additional interceptors to the fiscal year 2017 procurement. Additionally, the Army and MDA have reached an impasse regarding the transfer of the THAAD program from MDA to the Army. MDA and the Army have been directed by the Deputy Secretary of Defense to develop a memorandum of agreement that would guide the transfer of the THAAD and AN/TPY-2 programs to the Army, and the National Defense Authorization Act for fiscal year 2018 requires the Secretary of Defense to transfer the acquisition authority of all missile defense programs that have received full-rate production authority, which includes THAAD, to the military departments not later than the date the President’s fiscal year 2021 budget is submitted. The Army, however, has identified a $10.1 billion requirements gap, and the Secretary of the Army issued a memo that he would non-concur with the transfer of the THAAD program in its current state. There is currently no plan or timeline to resolve the issue. We will continue to follow this issue in our future work. Finally, additional delays to the construction of the Aegis Ashore facility in Poland resulted in significant schedule compression, reducing the time allotted for installation and checkout activities from 16.5 months to 9.5 months. MDA initially maintained that the site would be delivered on schedule, but early in fiscal year 2018 the agency announced that the site would not be delivered until at least December 2019. Integrated BMDS Capability Increments: MDA also encountered challenges delivering packages of integrated capabilities, which it refers to as “increments.” Increment deliveries signify delivery of integrated BMDS-level capabilities, which are designed to significantly improve effectiveness and efficiency of the BMDS over its constituent elements working independently. MDA planned to deliver two increments in 2017, but both were delayed, and some constituent capabilities were removed and are planned to be delivered in future increments. For instance, MDA was late in delivering Increment 3, known as “Discrimination Improvements for Homeland Defense – Near Term.” We previously reported on schedule slips to this increment from its initial September 2016 delivery date to December 2016. However, program documentation indicates that MDA encountered further challenges in fiscal year 2017 that required an additional delay to March 2017. According to MDA officials, this most recent delay was driven by additional time needed to analyze testing results. However, we found that GMD had experienced development delays for some software upgrades leading up to assessment and integration activities. Moreover, MDA’s Increment 4, known as “Enhanced Homeland Defense,” was not completed in December 2017 as planned, because a C2BMC and a key GMD upgrade initially planned to support four BMDS-level capabilities intended for this increment would not be available until the second quarter of fiscal year 2018. MDA officials told us that they will rely on the current GMD software version, which lacks some key improvements, until this upgrade is delivered. Additionally, MDA significantly reduced the content of its BMDS cyberdefense capability planned for Increment 4. MDA documentation originally planned to deliver this capability with 10 elements and, prior to testing, the BMDS OTA declared four elements to be priorities. Of these four, MDA has conducted the assessment for only three. The remaining BMDS elements will deliver cyberdefense capabilities in future increment deliveries. MDA’s plans for delivery of future capabilities continue to be volatile. For example, plans for Increment 6 in fiscal year 2021, which will include delivering a new radar and kill vehicle for GMD, now require its capabilities to be broken up into three sub-increments delivered across several years, some as late as 2023, with multiple new capabilities added and several others deferred to Increment 7. Many of these delays continue to postpone achievement of BMDS integration, needed to improve performance against realistic attacks with multiple ballistic missiles. Most recently, MDA again delayed a capability designed to improve automated coordination between regional BMD shooters—that is, Aegis BMD, THAAD, and Patriot. While initially planned for delivery in 2015 with Increment 2, in fiscal year 2017, the capability was further delayed, from 2020 to 2023. In addition, a further integration capability that would centralize and automate command decisions across the BMDS will not be available until December 2025. See figure 3 for more information on how capabilities have been delayed within and across increments. Testing: MDA successfully completed most of its planned tests in fiscal year 2017 and achieved several notable milestones, though MDA continued to add, alter, delete, or delay parts of its test schedule throughout the year. Within the elements included in this report, MDA had nine tests in its fiscal year 2017 test plan, of which it conducted six as planned. MDA also added three additional tests to its plan over the course of the year. A summary of these tests can be found in table 3. Many of these tests are notable firsts for MDA, though others indicate continuing challenges. FTG-15 was a success, in which a Ground-Based Interceptor with a Configuration-2 booster and a CE-II Block I Exo-atmospheric Kill Vehicle intercepted for the first time an intercontinental ballistic missile with threat representative characteristics. In addition, this was the first use of the new booster avionics and upgrades to the software. The success of this test was necessary to deliver Increment 4’s requirements for Enhanced Homeland Defense. However, Department of Defense operational testing officials stated that the complexity and objectives of the test had been scaled back from what MDA originally planned. SFTM-01 was a success, in which an Aegis BMD SM-3 Block IIA missile intercepted a medium-range ballistic missile target. This was the first intercept test for the Aegis BMD SM-3 Block IIA. SFTM-02 was a failure, as the Aegis BMD SM-3 Block IIA interceptor failed to intercept its medium-range ballistic missile target. MDA officials stated that the interceptor acted “as designed” during the test, and the Navy is considering whether changes to its tactics, techniques, and procedures may be warranted. MDA officials maintained that this developmental test existed in part for risk- reduction ahead of fiscal year 2018’s FTM-29, in which the Aegis BMD SM-3 Block IIA would have to intercept an intermediate-range ballistic missile for the first time. Despite the failure, MDA has chosen not to reschedule and has instead re-assigned SFTM-02’s objectives to FTM-29. FTT-18 was a success, in which a THAAD battery intercepted an intermediate-range ballistic missile target. This test was originally planned for several years ago, as part of the 2015 delivery of Increment 2, and has been delayed in part due to issues with range availability. This is the first demonstration of THAAD against an intermediate-range ballistic missile target despite a THAAD battery having been delivered to Guam for this mission in 2013. FET-01, previously known as FTT-15, was a success, demonstrating THAAD’s ability to intercept a target in the endo-atmospheric stage of flight. MDA re-classified the test a “Flight Experiment” midway through fiscal year 2017 to reflect its more observational and experimental nature. The test objectives for FET-01 have changed several times, and while the final iteration of test objectives did not include intercept as an objective, an intercept against a medium-range ballistic missile target was achieved nonetheless. When MDA declares a capability ready for delivery to warfighters, it communicates the capabilities and limitations of the delivered asset. This information is critical for allowing warfighters to make informed decisions about whether to accept the capability, how to prepare for its deployment, and how to plan for its use. Typically this process occurs through the Operational Capacity Baseline (OCB) change process, which is structured around the delivery of new capabilities to individual elements. Alternately, as noted above, when MDA declares a key integrated, BMDS-level capability ready for delivery, it does so through a process which culminates in the issuance of a Technical Capability Declaration (TCD). The TCD is a memorandum signed by the Director, MDA and is usually reserved for significant new capabilities such as: those mandated by the President; or delivery of integrated BMDS-level capabilities that require more than one element to function. This last category of capabilities is especially important as, according MDA’s charter, the BMDS is intended to be an integrated and interoperable system. Integration is important in order to counter the larger-scale and more complex attacks that are likely to occur during a conflict. We have reported since 2014 that MDA has taken steps to improve the management and reporting of integrated capabilities, and to increase the level of BMDS integration. While MDA has recently made some progress in the area of integrated capabilities, the majority of MDA’s capability deliveries continue to be made at the element level. Until recently, MDA has done little to document the requirements and process for issuing a TCD, resulting in an inconsistent and, at times, ad- hoc process. We found inconsistencies in MDA’s decisions regarding which integrated, BMDS-level capabilities MDA would deliver through a TCD, and which it would not. For example, since 2015, the agency planned to deliver 14 integrated, BMDS-level capabilities, but delivered only 7 through the TCD process. According to MDA’s prior capability delivery documents, several of these excluded capabilities were intended to be part of the formal TCD delivery during the planning stage, but were dropped at some later point. According to MDA officials, those deliveries were made when all their constituent elements were delivered via the OCB process. MDA officials acknowledged that distinctions between requirements for element-level deliveries and BMDS-level capabilities were not readily apparent in their policy and took steps in fiscal year 2017 to do so. MDA issued a memorandum on Technical Capability Declaration Planning and Definitions in June 2017 to help distinguish element-level OCB deliveries and deliveries of integrated BMDS capabilities that would occur via TCD. This document established several definitions and requirements such as assigning responsibilities, establishing lines of authority, and defining some requirements that are not found in the other guidance document that MDA uses to govern TCD. The June 2017 memorandum also identified which capabilities through 2023 that MDA will deliver via a TCD, and identified some ways to add a new capability to the list of those receiving a TCD. While MDA’s new policy represents a substantial improvement in the management of the TCD process, it does not address several important problems with the TCD process. Specifically, although MDA has identified capabilities that it plans to deliver using a TCD, it does not identify any criteria or reasoning that guided this determination. It also does not explain the criteria MDA will apply to future capabilities under consideration for a TCD, leaving open the possibility of the same inconsistent application MDA has used in the past. Moreover, the capabilities it identified for a TCD are only a subset of all planned integrated, BMDS-level capabilities. Consequently, only some integrated capabilities are currently planned to be delivered to the warfighter with comprehensive information about their performance and limitations at the BMDS level. Unless MDA requires that all integrated capabilities are delivered via the TCD process, as the BMDS becomes more integrated, military services and other decision makers will have reduced insight into the capabilities and limitations of the BMDS as a whole. MDA’s June 2017 policy also establishes some processes governing the requirements for, and development of, test plans in support of a TCD, but it does not address some of the most problematic aspects of this process to date. Specifically, the new policy requires convening an Assessment Requirements Review board to develop a baseline for a planned TCD, determine what capabilities will be included, and identify what test plans will be necessary to generate the “body of evidence” that MDA will provide in support of the TCD’s assertions regarding capabilities and limitations. However, we found that Assessment Requirements Reviews can be held shortly before the planned delivery date—which affords no opportunity to build the test plan around the requirements identified in the review. MDA held Assessment Requirement Reviews in preparation for two of the previous three TCDs. The timing of these reviews in relation to the date of the TCD’s issuance suggests that they had little influence on MDA’s actual test plans. MDA officials stated that an Assessment Requirement Review is ideally held 18 months to 2 years prior to the issuance of the related TCD. However, we found that, for recently issued TCDs, the reviews were held much closer to the beginning of testing and the planned TCD delivery. For example, for the TCD issued in December 2017 that delivered 44 ground-based interceptors, MDA held this review less than 8 weeks in advance. Figure 4 depicts the timeline of the Assessment Requirements Review as compared to the start of testing for the TCD and the TCD delivery date. Because these reviews identify requirements that must be tested, the Assessment Requirements Review would ideally inform MDA’s test plans so that each component of the integrated capability could be adequately tested by the planned delivery date. But because the policy does not give exact requirements, process, and key milestones necessary to issue a TCD, MDA is able to hold an Assessment Requirements Review that merely acknowledges the results of tests already completed. These practices are consistent with our prior findings on MDA, which identified a lack of a management framework for delivering integrated capabilities, and showed that the lack of this framework resulted in concurrency, fragmentation of development activities, and delays for some originally planned capabilities. According to DOD’s guidance on acquisition and testing, a program’s test and evaluation strategy should begin with a review of requirements so that management can devise a test and evaluation strategy that generates the knowledge necessary to inform the acquisition and operational decisions of a program. Holding the Assessment Requirement Review so close to the planned delivery date affords no opportunity to build the test plan around the requirements identified in the review, and instead only ratifies the results of a test plan that was not necessarily developed with these requirements in mind. Undefinitized contract actions are authorized when the negotiation of a definitive contract is not possible in sufficient time to meet the government’s requirements and government interests demand that the contractor be given a binding commitment so that contract performance can begin immediately, and are subject to certain limitations. Our analysis of MDA contracting from fiscal year 2013 to fiscal year 2017 shows that the combined not-to-exceed price of all undefinitized contract actions entered in a given year, and the average time it takes to definitize undefinitized contract actions, have increased. GAO has reported that while this type of contract action may be necessary under certain circumstances, it is considered risky in part because the government may incur unnecessary costs if requirements change before the contract is definitized. Though MDA reports on its contracting activities in its annual BMDS Accountability Report, its reporting on details unique to undefinitized contract actions is often inconsistent or even absent. MDA’s Acquisition Management Instruction 5013.02-INS states that undefinitized contract actions will be used only on “an extremely limited basis” and only when negotiating contract terms before the contractor begins work is not feasible, such as when delay “would adversely impact mission accomplishment.” Our prior work, as well as that of the DOD inspector general, has found that this type of contract action is considered risky in part because the government may incur unnecessary costs if requirements change before the contract is definitized. Under undefinitized contract actions, substantial funds may be obligated before essential questions of contract scope and system design have been settled. Over the past 5 years, the average length of the undefinitized period and not-to-exceed price for MDA’s undefinitized contract actions have increased. Since 2013, MDA has entered into 11 undefinitized contract actions as shown in table 4. MDA’s use of undefinitized contract actions has fluctuated between one and five instances per year. The combined not-to-exceed price of all such contract actions entered into each year has increased, however, from $2.5 million in fiscal year 2013 to $1.4 billion in fiscal year 2017 as shown in figure 5. The average time to definitize these contract actions has steadily increased as well, from 78 days in fiscal year 2013, to over 600 days in fiscal year 2016 (see figure 6). Two undefinitized contracts were awarded in fiscal year 2017 and both exceeded 180 days without definitization. The value of MDA’s undefinitized contract actions entered into in a given year, as measured by their combined not-to-exceed prices, has risen significantly. The length of the undefinitized period has also risen on average. Together, these figures show that MDA may be initiating contractor work with incomplete knowledge of the requirements or costs involved. With regard to the increasing duration of the undefinitized period, MDA contracting officials told us that when they do not achieve definitization within 180 days, it is often because the contractor’s proposal is not adequately supported by a sound estimate, and negotiation past 180 days is necessary to achieve a fair and reasonable price. They added that the task of making this determination is made more complicated by the highly developmental nature of the work that MDA often conducts. For example, the 2015 undefinitized contract action for Aegis BMD SM-3 Block IIA test interceptors remained undefinitized for 629 days. According to MDA officials, this delay was due in part to the difficulty of accurately estimating costs on a highly developmental project. MDA officials reported having to develop a substantial knowledge base and consult closely with other DOD entities that would have insight into the costs of similar projects, after the undefinitized contract action was entered into. Using an undefinitized contract action in this case, however, was not without risk to the government. MDA made major financial commitments to a program before it fully understood the requirements or the costs. To mitigate the risks related to these contract actions, MDA’s Instruction requires all undefinitized contract actions to be supported by a determination and findings that articulates the requirement to begin performance prior to a negotiated agreement, the not-to-exceed price and the definitization schedule. The DFARS and MDA instruction require all undefinitized contract actions to be approved by the Director, MDA. MDA officials told us that they interpret the MDA Instruction to require that the Director, MDA, sign determination and findings documents in support of undefinitized contract actions. In addition, MDA contracting officials stated that to further mitigate the risks related to undefinitized contract actions, they, as a matter of practice, strive to obligate only the minimum amount of funding necessary to achieve definitization. Officials indicated that doing so limits the cost risk for the government, and forces programs to think carefully about what work needs to be done prior to definitization and its likely costs. While the Director, MDA is required to sign the determination and findings document, in one instance, this document specifically authorized the program to amend the requirements and not-to-exceed price without further formal approval from the Director, MDA. This specific undefinitized contract action was the largest MDA has entered into since fiscal year 2013. MDA entered into the undefinitized contract action in May 2017, authorizing the design, development, and initial production of the GMD program’s Redesigned Kill Vehicle (RKV), with a not-to-exceed price of $1.088 billion. This undefinitized contract action will allow MDA to continue with the RKV program despite significant cost, schedule, and performance risks, some of which the determination and findings document for the RKV undefinitized contract action acknowledged. When MDA released its acquisition strategy for the RKV in 2015, it predicted the phase covered by this contract action would cost approximately $800 million, covering initial testing and production of up to eight RKVs for initial fielding. Officials stated that the current contract action, with a not-to-exceed price of $1.088 billion, is for only four interceptors, although since it is undefinitized, that is subject to change. If the RKV program definitizes this contract action according to its schedule in May 2018, after 12 months, this will result in the definitization of the contract action with less than a year remaining before the program’s critical design review. In other words, the government will have agreed on contract terms, including costs, after much of the design work and related costs have been incurred. As of February 2018, MDA reports obligating $324 million, or 30 percent of the not-to-exceed price, to this undefinitized contract action. This is in excess of the $244 million planned for the undefinitized period at the time of award. As part of MDA’s annual BMDS Accountability Report, MDA reports on its planned performance and schedule for the coming fiscal year across several baselines, one of which is dedicated to contracting performance. MDA provides these baselines in response to statutory requirements. By establishing these baselines and then reporting any deviations in cost, schedule, or performance as a program proceeds, the BMDS Accountability Report provides information for oversight by identifying areas of program risk and their causes to decision makers. Baselines also help ensure that the full financial commitment is considered before embarking on major development efforts. These reports contain some information on undefinitized contracts. However, the information is often inconsistently presented and distributed throughout the report. Information specific to undefinitized contract actions is often absent, such as the following: the definitization schedule (that is, the expected time frame for finalizing contract terms); the amount of funds obligated to the action for the undefinitized period; or any changes to the above that have occurred since award of the action. As a result, decision makers in Congress have limited insight into how MDA is handling the risks that come with undefinitized contract actions, or how the programs enacting these contracts are performing. For example, these reports do not typically disclose how much has been obligated under an undefinitized contract action, or if this amount has increased since the contract was awarded. They do not report if the not-to-exceed value has been revised, or if the current definitization schedule adheres to the schedule determined at the time of award. Despite taking steps to improve the realism of the models it uses for ground testing, MDA continues to face challenges with its models. As a result, decision makers lack key information about BMDS performance, which could lead to miscalculations about how best to employ the BMDS and where to focus future capability development and investment. Specifically, MDA continues to encounter challenges with ensuring that its models and simulations are accredited for operational testing when they are used to test BMDS capabilities, resulting in uncertain performance outcomes in assessments supporting BMDS deliveries. Additionally, accreditation status and modeling limitations for these assessments are not communicated to most decision makers in Congress and some in the DOD and executive branch, limiting their insight into the data limitations underlying their decisions to make investments in and employ the BMDS. Finally, MDA’s assessment of the resources needed to validate and accredit its current models does not match requested funding for this effort. Since MDA cannot conduct enough system-level flight testing of the entire BMDS to completely assess BMDS performance, BMDS decision makers within MDA, DOD, Congress, and the executive branch use information from model-based ground tests to evaluate the operational effectiveness of the BMDS. The results from these model-based operational tests inform many acquisition and operational decisions, including: capability delivery, asset fielding, and interceptor inventory. Model-based testing also informs the warfighter’s tactics, techniques, and procedures to maximize BMDS effectiveness such as how many interceptors they will fire at a threat; and the capability gap analysis, the basis for warfighter requests for new capabilities. Recognizing the importance of models and simulations, MDA has taken steps to improve its ability to provide realistic modeled representations of the integrated BMDS necessary to assess operational performance. For instance: In 2009, MDA adjusted its test baseline, known as the Integrated Master Test Plan, and refocused its testing on collecting data needed for model development and accreditation. In 2016, MDA developed an update to a framework that is used to integrate the modeled representations of BMDS elements for assessments, and in 2017 continued an effort to develop digital end- to-end models and simulations to increase modeling capabilities and to expand the scope of BMDS assessments in the future. In 2017, MDA increased its collaboration with BMDS OTA to prioritize modeling needs and to address them. Despite these steps, MDA continues to deliver assets and capabilities using models that have not been accredited. In April 2016 and May 2017, we found that MDA had delivered EPAA Phase 2 capabilities in December 2015 using models that have not been accredited to support the delivery. MDA continued this practice by delivering two sets of BMDS-level capabilities since 2015, relying on operational tests conducted with models that were not accredited for use in such an assessment. The next delivery, expected at the end of the second quarter of fiscal year 2018, has also been tested using mostly unaccredited models. Relying on models that are not accredited for operational assessment increases the risk that modeling errors are not discovered, and a single undetected modeling error can distort the assessment results for the entire BMDS. DOD’s acquisition instruction requires that models and simulations used in operational assessments be verified, validated, and accredited. Although, as noted above, MDA is generally exempt from DOD acquisition policies, its own modeling and simulation policy requires that models and simulations used in operational assessments be verified, validated, and accredited for that use. Moreover, experts at DOD, MDA, and other institutions we interviewed agree that models should be verified, validated, and accredited to ensure that decisions based on models are informed by the correct data, and that the limitations of that data are understood. Additionally, according to DOD officials, defense acquisition programs that follow DOD acquisition regulations verify, validate, and accredit their models before operational assessments. However, our analysis indicates that the accreditation of many MDA models for operational assessment is, in most cases, not completed in time to support testing. In fact, many of them are not complete even after a capability has been delivered. Additionally, BMDS OTA officials said that models that are not accredited before delivery are not generally accredited later on. Figure 7 shows the percentage of accredited models that were used in the operational assessment of each BMDS capability delivery in 2015 through 2017. BMDS models are not accredited for operational assessment in large part for three reasons: (1) MDA does not provide sufficient evidence to the BMDS OTA for accreditation, (2) some models do not accurately represent BMDS performance in the real world, and (3) the threat model used to stimulate the test cannot be traced to the original intelligence community assessment. These challenges affect assessments across the entire BMDS engagement, from detection and processing of the threat to the intercept. While modeling uncertainty in any one of these areas affects uncertainty for the BMDS as a whole, factored together this uncertainty is magnified. Lack of Data: In some cases, MDA did not provide the BMDS OTA data needed to accredit the models used in operational ground testing, even though it is a signatory to the BMDS OTA’s accreditation plan. This plan identifies the data needed to achieve accreditation and directs that these data should be provided at least 60 days prior to official operational ground testing. MDA officials noted that the BMDS OTA recently changed its data requirements for accreditation and that they were unable to meet the new requirements in time to inform the capability deliveries shown above. However, we have found that MDA has encountered similar challenges since 2009. In fact, according to BMDS OTA officials, MDA has never completely provided the needed data on time and often missed numerous subsequent deadlines. In many cases, MDA failed to deliver the required data even after it tested and delivered its capabilities, and in some instances the data MDA provided did not meet the BMDS OTA’s requirements. As we have previously reported, disruptions to MDA’s testing program—such as flight test failures and delays—reduce the amount of real-world data that is available to accredit models. We also found that MDA proceeded with model-based ground tests and capability deliveries without leveraging the knowledge it planned to obtain from these tests. For example, in 2016 and 2017, we found that MDA delivered the European Phased Adaptive Approach Phase 2, even though key models, such as the model for Aegis Ashore, were unaccredited. Additionally, in other instances, MDA lacks technical data and other model information that is needed for accreditation, especially for models representing older systems. In 2017, as noted above, MDA and the BMDS OTA increased their collaboration to improve model accreditation status and, in 2017, co- developed a list of prioritized modeling deficiencies. Additionally, MDA is making progress in gathering and providing model data for operational assessment accreditation. MDA officials reported that based on this increasing collaboration, they expect that more models will be accredited in 2018. It is unlikely, however, that all models will achieve accreditation prior to the upcoming December 2018 delivery of the European Phased Adaptive Approach Phase 3. Modeling Deficiencies: Another reason that some models are not accredited for operational use is that certain models contain deficiencies, such as optimistic representations of BMDS performance and simplistic representations of BMDS environments. In these cases, while MDA initially supplied BMDS OTA with the relevant data, the model’s performance failed to meet the criteria for accreditation. Subsequently, MDA did not provide supporting rationale to explain these failures, or to explain how the modeling issues skewed the overall performance results. For example, in 2016, the BMDS OTA compared modeled sensor tracking data used in recent ground tests to real-world sensor tracking data and found that the models representing some radars performed better than the real-world radar. These modeling deficiencies can affect other BMDS elements that rely on sensor data and can artificially inflate BMDS performance. In one case, Aegis BMD’s launch-on-remote capabilities were over-estimated. As a result, the BMDS OTA could not accredit the models, and thus verify that ground test results that support Aegis’s launch-on-remote capability and other tested capabilities are credible and reliable. MDA is working to address this issue and it is too early to assess progress. Additionally, some models used in operational assessments are overly simplistic. For example, modeled representations of the battle scene in moments after intercept do not display the resulting complex scene that is caused by the large quantity of missile and interceptor debris. This deficiency limits insight into how the BMDS will perform during realistic ballistic missile attacks that could require follow-on interceptors to be launched, and how the BMDS will determine that the incoming threats have been destroyed. According to BMDS OTA and MDA officials, MDA’s efforts to develop digital models can help in this area, by providing more processing power and great scalability for engagement complexity; however, the capability is not expected to be mature until 2021 or later. Threat Models Cannot Be Traced Back to Underlying Threat Assessments: The value of ground test-generated data is dependent on the quality of the threat model that stimulates the test. However, the BMDS OTA has never been able to accredit threat models before operational testing, and in some cases, after testing. As is the case with other models, in some cases, the BMDS OTA does not receive data needed to accredit the models from MDA in a timely manner. Additionally, the BMDS OTA cannot trace the threat model used in ground testing to the threat model that MDA developed based on the intelligence community’s threat assessment. For example, according to BMDS OTA officials, during a past ground test event, a model representing a BMDS element rejected the intended threat model and instead ran its own internal threat model. As a result, the test did not reflect real world conditions where the entire BMDS would be exposed to the same threat stimulus. BMDS OTA officials said that MDA’s ground test architecture is not designed to generate the data needed to confirm that all elements are reacting to the same model during ground testing, meaning that unbeknownst to testers, other BMDS elements could also reject the approved threat model during testing. These deficiencies introduce ambiguity into the test results including the extent to which the BMDS operated as an integrated system of systems against a common threat set. BMDS OTA officials said that MDA is currently working on a pathfinder activity to help understand and rectify the traceability issue. Although the warfighter and other decision makers inside DOD, Congress, and the executive branch rely on models to provide information about BMDS effectiveness, MDA’s capability delivery documentation does not include information about the quality of modeling data. Specifically, MDA’s TCD memos and OCB change packages, which describe technical capabilities delivered to the warfighter and their limitations, do not discuss the extent to which the models used to assess the new capability are verified, validated, and accredited for assessment, or how ground test results were affected by model limitations. As a result, decision makers do not have complete information about the validity of the capability assertions in these documents and how much confidence should be placed in reported BMDS performance. According to Standards for Internal Control in the Federal Government, decision makers need access to reliable and timely information to make operational decisions. Additionally, according to DOT&E guidance, in cases where models and simulations cannot be validated and accredited, any modeling results should be caveated with a clear explanation of which areas of performance assessment could be affected by the lack of accreditation. Lack of such information could lead to miscalculations about how best to employ the BMDS or uninformed decisions about where to focus future capability development and investment. While the BMDS OTA has recently begun to brief some combatant commands on how modeling limitations impact the warfighters’ understanding of delivered capabilities, these briefings are not readily available to other stakeholders and decision makers, such as cognizant congressional committees or others in DOD and the executive branch. In its report accompanying a bill for the National Defense Authorization Act for Fiscal Year 2017, the House Armed Services Committee requested that MDA brief the House and Senate Armed Services Committees on the accreditation status of models used in testing indicating that congressional decision makers benefit from such information. Including information about model accreditation and limitations in TCD and OCB packages would ensure decision makers in DOD, Congress, and the executive branch have the same necessary information to inform their decisions. Moving forward, the Director, MDA will have to make difficult decisions on balancing funding priorities, including the need to adequately fund the validation and accreditation of models. MDA has started to make progress validating and accrediting existing models using DOT&E and OTA recommended criteria. However, MDA’s assessment of the resources needed to validate and accredit its current models and simulations does not match funding levels it requested for this effort. MDA determined that it needs an additional $99 million for fiscal years 2017- 2022 to accredit BMDS models and simulations. MDA requested $395.7 million from 2017-2022 to meet modeling and simulation needs. Figure 8 shows MDA’s fiscal year 2018 funding request for model development and the additional funding, over the 5 year period, that would be required to verify, validate, and accredit its models. Additionally, funding is not requested for the verification, validation, and accreditation of some models used in BMDS assessments because MDA officials said that they do not have written agreements with the military services that operate these elements defining funding and technical requirements for this purpose. Specifically, while the Army and the Air Force develop and accredit models to support their missions for the Patriot, the Space-based Infrared System, and the BMDS communication systems, these models have to be modified to accurately represent their BMDS roles for BMDS operational assessments. While MDA does fund the development of the Space-based Infrared System and BMDS communication models for use in BMDS assessment, it does not fund the verification, validation, and accreditation of these models or the Patriot model. Additionally, MDA officials report that it currently has no written agreements with the Army or the Air Force to define funding and technical requirements for these models for BMDS assessment. Because these requirements are not formally agreed upon and communicated between MDA and the Services, the verification, validation, and accreditation of these models is often unfunded, further complicating MDA’s and the BMDS OTA’s verification, validation, and accreditation analyses. Standards for Internal Control in the Federal Government states that organizations should assign responsibility and delegate authority to achieve their objectives. Additionally, in our prior work we found that all acquisitions efforts should have well defined roles and responsibilities for all stakeholders. Although MDA and the BMDS OTA were able to accredit the Space-based Infrared System and BMDS communications models in 2017, future upgrades to these BMDS elements will require verification, validation, and accreditation to ensure that they continue to accurately reflect the real-world system. Moreover, DOD and Congress have instructed the transfer of missile defense programs that have received full-rate production authority, which would include THAAD and Aegis BMD, to the military services for operations, which may increase the scope of this issue. Even though these systems will no longer be under MDA management, they will still be part of the BMDS and, like the Space-based Infrared System and Patriot, will require model updates to reflect changes to the tactical systems. However, as noted above, there are currently no agreements between MDA and the services to fund these modeling requirements, increasing the risk that model upgrades will not be implemented, thus preventing their verification, validation and accreditation for operational testing. If MDA and the services do not agree to the technical and funding requirements for models of elements used in BMDS testing but operated by the services before the elements are transferred, disagreements will likely continue to impede the verification, validation, and accreditation of those models, decreasing confidence in test results and understanding of how the real-world BMDS will operate. MDA continues to make mixed progress in delivering assets and integrated capabilities. Moreover, its processes for communicating the extent and limitations of these capabilities can be improved. While MDA met several significant milestones in fiscal year 2017, MDA failed to deliver either of its two most recent packages of integrated capabilities on time, and its plans for future capabilities, even in the near term, continue to be characterized by a high degree of fluidity. MDA has recently taken steps to document in policy its processes for communicating the extent and limitations of deliveries of integrated capabilities. However, these policies still do not clearly specify the exact requirements, process, and key milestones needed to complete some capability deliveries. Moreover, they do not require that all integrated BMDS capabilities are delivered using a process that describes their performance and limitations at the level of the BMDS, rather than at the element level, increasing the risk of delivered capabilities not being communicated properly to their end users: the warfighter. In addition, while no contracting strategy can be completely risk-free, trends in the not-to-exceed prices and duration of MDA’s undefinitized contract actions indicate a troubling pattern. Making major commitments to large developmental programs before important questions of scope and price have been determined exposes the government to increasing amounts of risk. MDA’s inconsistent and incomplete reporting on its use of undefinitized contract actions makes it even more difficult for Congress and decision makers to exercise oversight and track these risks. Finally, deficiencies and limitations in the models used to support operational testing of the BMDS, including the lack of accreditation, provides decision makers with some flawed information about BMDS performance. Because flight tests cannot provide complete information on BMDS performance, it is important that ground tests can be relied upon to provide accurate and representative data. This flawed information could lead to miscalculations about how best to employ the BMDS and uninformed decisions about where to focus future capability development and investment. If steps are not taken to improve BMDS models and to communicate their status and limitations clearly to decision makers, there is a risk that the BMDS will not perform as expected when needed to defend the United States at home, its regional allies, and deployed forces. We are making the following six recommendations to the Under Secretary of Defense for Research and Engineering: The Under Secretary of Defense for Research and Engineering should ensure that the Director, MDA, takes the following actions: The Director, MDA should revise MDA policies to require that all integrated capabilities—capabilities that require integration of two or more elements—be included in a Technical Capability Declaration. (Recommendation 1) The Director, MDA should clarify, in written policy, the exact requirements, process, and key milestones necessary to issue a Technical Capability Declaration, including a requirement that the Assessment Requirements Review be held in such a time frame that it can provide meaningful input to MDA’s test plans. (Recommendation 2) The Director, MDA should include information on current undefinitized contract actions in the BMDS Accountability Report, including the not-to- exceed price, the definitization schedule, the amount of funds obligated for the undefinitized period, and any changes since the contract action was entered into. (Recommendation 3) The Director, MDA should ensure that models used for operational tests are validated and accredited for such assessments. To help achieve this, MDA should provide the BMDS Operational Test Agency all evidence previously agreed to and needed to accredit models before ground testing events, as specified in the BMDS OTA accreditation plan. (Recommendation 4) The Director, MDA should include in capability delivery packages, such as the Technical Capability Declaration memos and Operational Capability Baseline change packages, the following: a. The verification, validation, and accreditation status of the models used in operational ground tests; and b. Modeling and simulation limitations that affect operational ground test results. (Recommendation 5) The Director, MDA and the Secretaries of the Armed Services responsible for operating BMDS elements should develop written agreements as soon as feasible for modeling and simulations technical and funding requirements for any BMDS elements that are service- operated but represented in BMDS performance assessments. (Recommendation 6) DOD provided written comments on a draft of this report. DOD’s comments are reprinted in Appendix I and summarized below. DOD and MDA also provided technical comments which were incorporated as appropriate. In its response, DOD concurred with five out of six of our recommendations, and partially concurred with one. In addition, DOD recommends the closure of five recommendations. However, we believe that it is premature to close out four of the five recommendations until all of its planned actions are fully implemented. For the remaining recommendation, we agree with DOD and will undertake the steps to close out the recommendation. DOD partially concurred with our first recommendation to revise MDA policy to require all integrated capabilities—capabilities requiring the integration of two or more elements— be declared and delivered via the Technical Capability Declaration (TCD) process. While DOD agreed with the intent of this recommendation, DOD stated that the Director, MDA will determine which major integrated capabilities should be delivered via the TCD process. The Department also noted that the agency developed a list of such capabilities that it will update annually. These actions are an improvement over the current process, but they do not meet the full intent of our recommendation. Specifically, the list of future TCDs that MDA produced is not inclusive of all future integrated capabilities. In addition, MDA’s policy does not articulate definitive standards for identifying capabilities requiring a TCD and leaves this decision to the discretion of the Director, MDA. As we’ve identified in this report, some capabilities have been deleted from or added to planned TCDs without explanation. The new policy leaves open the possibility of continued inconsistent application of the TCD process. This poses the risk that not all integrated capabilities will be delivered to warfighters with comprehensive information about their performance and limitations at the BMDS level. We continue to believe that in order for the agency to meet the full intent of our recommendation, it should establish in policy a clear, definitive standard for which capabilities require a TCD for delivery. In addition, DOD recommends the closure of the first two recommendations to (1) revise MDA’s policies to require that all integrated capabilities be included in a TCD; and (2) clarify the exact requirements, process, and key milestones necessary to issue a TCD as it contends that its new Policy Memorandum 90 meets the intent of our recommendation. This memorandum is dated March 28, 2018 and was provided to us on May 8, 2018. As such, we have not had an opportunity to fully assess the memorandum and the process laid out in it. However, as noted above, this new Policy Memorandum 90 leaves open the possibility of continued inconsistent application of the TCD process. This poses the risk that not all integrated capabilities will be delivered to warfighters with comprehensive information about their performance and limitations at the BMDS level. In order for the agency to meet the full intent of our recommendation, MDA should establish in policy a clear, definitive standard for which capabilities require a TCD for delivery. In addition, DOD writes that the same Policy Memorandum 90 satisfies the second recommendation to clarify the exact requirements, process, and key milestones necessary to issue a TCD. We believe it necessary to wait until MDA delivers a TCD in accordance with the new parameters set out in the memorandum before this recommendation can be closed. For the third recommendation to include information on current undefinitized contract actions in the BMDS Accountability Report, DOD states that the BMDS Accountability Report for 2018, approved by the Director, MDA on March 9, 2018 provides the information necessary for closure. We concur with this assessment will take the necessary steps to close this recommendation. In responding to our fourth recommendation requiring the Director, MDA to ensure that models used for operational tests are validated and accredited for such assessments, DOD states that MDA is actively working with the BMDS Operational Test Agency (BMDS OTA) to resolve any issues associated with, and the reporting of, modeling limitations. However, as we found in this report, according to BMDS OTA officials, MDA has never completely provided the needed data on time and often missed numerous subsequent deadlines to support the validation and verification of its models from BMDS OTA. Consequently, we believe it is premature to close out the fourth recommendation, but we will continue to track MDA’s progress and timeliness in providing the evidence previously agreed to and needed to accredit models before ground testing events. In responding to our fifth recommendation to include the verification, validation and accreditation status used in operational ground tests in capability delivery packages, such as TCDs and Operational Capability Baseline change packets, DOD states that MDA has made significant progress over the last year in achieving the BMDS OTA accreditation of MDA’s models and simulations. In addition, it states that the addition of MDA policy will ensure the verification, validation and accreditation status of each model will be discussed and assessed by the Operational Capability Baseline Working Group for each capability delivery package. We agree that MDA has made significant progress over the last year, however, we believe it premature to close out the recommendation until BMDS OTA can ensure that the status of the models used, as stated in our recommendation, are included in subsequent capability delivery packages such as the Technical Capability Declaration memos and Operational Capability Baseline change packages. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Undersecretary of Defense for Research and Engineering, and to the Director, MDA. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix XI. Aegis Ballistic Missile Defense is the naval component of the Missile Defense Agency’s (MDA) Ballistic Missile Defense System. It consists of the Aegis Ballistic Missile Defense Weapon System (AWS), including a radar and Standard Missile-3 (SM-3) interceptors. MDA is developing the Aegis BMD weapons system in versions called spirals that expand on preceding capabilities. Deliveries of the spirals are planned to support MDA’s capabilities for Regional and Homeland defense. Specifically, MDA delivered Aegis BMD 5.0 Capability Upgrade (5.0CU) in fiscal year 2016 for the European Phase Adaptive Approach (EPAA) Phase 2, but had not verified its full capability before delivery. In fiscal year 2017, the program delivered Aegis BMD 4.1 on ships with legacy hardware in order to provide similar ballistic missile defense capabilities to those of Aegis BMD 5.0 CU. MDA plans to deliver additional upgrades for such ships in 2019 and 2023. Additionally, the program is developing Aegis BMD 5.1 with upgrades for EPAA Phase 3, planned for December 2018. The Aegis BMD program also plans to deliver additional upgrades in 2023, called Aegis BMD 6.0, capitalizing on Navy’s upgrades to the Aegis radar. For specifics on Aegis Ashore and the Aegis BMD SM-3 interceptors, see appendixes III, IV and V, respectively. Table 5 provides key fiscal year 2017 AWS program facts. MDA resolved software challenges and testing delays for Aegis BMD 5.0 CU and delivered Aegis BMD 4.1, expanding the number of ships with EPAA Phase 2 missile defense capabilities. While MDA delivered initial Aegis BMD capabilities for EPAA Phase 2 with AWS 4.0.2 prior to the December 2015 Technical Capability Delivery (TCD), planned capabilities would not be available until the subsequent versions—Aegis BMD 5.0CU and 4.1—completed development and fielding. However, both encountered technical challenges and schedule slips, as well as testing delays. In fiscal year 2017, MDA continued work on Aegis BMD 5.0 CU and 4.1 and overcame some of these challenges. Specifically: Aegis BMD 5.0 CU: MDA completed Aegis BMD 5.0 CU certification late in fiscal year 2017, resolving prior technical challenges and testing delays. Specifically, MDA implemented fixes to significant defects that were in the software at the time of initial delivery. Additionally, in December 2016 and August 2017, MDA flight tested fleet and ship self-defense capability against medium-range ballistic missiles in terminal phase of flight –a capability initially planned for December 2015. Aegis BMD 4.1: MDA also delivered Aegis BMD 4.1 in August 2017, after multiple schedule slips. While initially scheduled for delivery in support of the EPAA Phase 2 TCD, the spiral was first delayed to the middle of fiscal year 2016 due to technical and cost challenges. Subsequently, activities for Aegis BMD 4.1 were suspended in 2016 to reassess the program and delivery was delayed to September 2019, to align it with a related Navy effort. In fiscal year 2017, MDA resumed activities for Aegis BMD 4.1, and certified the delivery of ballistic missile defense capabilities in August 2017. These ballistic missile defense capabilities are currently being integrated with the Navy’s larger Aegis combat system, into a single computer program called Aegis Baseline 5.4, which is still scheduled for delivery in September 2019. According to MDA’s program management documentation, Aegis BMD 5.1 is on track for delivery in support of EPAA Phase 3 in December 2018, as the program overcame or reduced key risks. For example, despite a lack of schedule margin, the program met a key software development milestone in June 2017, and delivered it for system-level ground tests, which will assess integrated BMDS capabilities for EPAA Phase 3. It also met all objectives in a fiscal year 2017 flight test. Additionally, the program reduced the ongoing programmatic risk to Aegis BMD 5.1 that could affect its interoperability with other elements. However, testing to demonstrate the risk has been resolved is not yet complete. According to the Aegis BMD program management documentation, upgrades to the Aegis communication management system, which are managed by the Navy, lag behind MDA’s Aegis BMD 5.1 development schedule. The lag in development could result in integration challenges between these upgrades, and could impact Aegis integration with other BMD elements, including the capability to intercept threats entirely on tracks from forward based radars – called Engage on Remote. In fiscal year 2017, MDA and the Navy took steps to mitigate this risk. However, MDA has yet to demonstrate the fixes in a flight test. Moreover, MDA documentation indicates that if issues are discovered, they could impact the Aegis BMD 5.1 mission and could result in interoperability restrictions against Aegis BMD 5.1. Lastly, Engage on Remote could also be affected if development challenges with C2BMC, which forwards threat track data from forward based sensors to Aegis BMD, are not mitigated. For more information on the C2BMC element, see Appendix VI. In fiscal year 2017, MDA continued to develop Aegis BMD capabilities that are expected to be deployed in 2023. Specifically, MDA continued developing and maturing capabilities for an effort it started at the end of fiscal year 2016 called Aegis BMD 6.0. Aegis BMD 6.0 is planned to provide capabilities against more threat types, larger raids, better discrimination, and improved communication with its interceptors. Additionally, it takes advantage of the Navy’s effort to replace the Aegis SPY-1 radar with a more capable SPY-6, and to overhaul the entire Aegis combat system. While it is early in development, MDA has begun identifying knowledge gaps that could diminish planned capabilities and took initial steps to address disconnects between Navy’s effort and its own. According to program management documentation, MDA plans to develop an Aegis BMD 6.0 acquisition baseline late in fiscal year 2018. The acquisition baseline is expected to detail Aegis BMD 6.0 planned capabilities, its schedule, and cost. MDA is also planning additional upgrades to Aegis BMD 4.1, called Aegis BMD 4.2. Specifically, MDA plans to collaborate with the Navy to integrate and field refurbished and upgraded SPY-1 Antennas onto legacy ships. This modification improves radar sensitivity resulting in improved tracking capabilities and is planned for delivery in fiscal year 2023. MDA plans to begin developing and maturing technologies for this upgrade in fiscal year 2019, and baseline the effort at the end of fiscal year 2020. Aegis Ashore is a land-based, or ashore, version of the ship-based Aegis Ballistic Missile Defense (BMD). Aegis Ashore is designed to track and intercept ballistic missiles in the middle of their flight using Aegis BMD Standard Missile-3 (SM-3) interceptors. Key components include a vertical launching system, interceptors, and an enclosure, called a deckhouse, that contains the SPY-1 radar and command and control system. DOD deployed an Aegis Ashore test facility in Hawaii in April 2014. The test facility has been used to flight test Aegis Ashore, and in some cases, Aegis BMD SM-3 interceptors. MDA deployed its first operational site in Romania in fiscal year 2016 as part of the European Phased Adaptive Approach (EPAA) Phase 2, and is currently constructing a second site in Poland for delivery in 2018 as part of EPAA Phase 3. Both operational sites are intended to provide additional coverage for the defense of Europe. Aegis Ashore will share many components with the sea-based Aegis BMD and will use future versions of the Aegis weapon system currently in development, including the Aegis BMD SM-3 Block IIA interceptor. The Missile Defense Agency (MDA) plans to equip Aegis Ashore with a modified version of the Aegis weapon system software that will share many components with the sea-based Aegis BMD. For further details on the Aegis Weapon System and Aegis BMD interceptors, see appendixes II, IV, and V. Table 6 provides key fiscal year 2017 Aegis Ashore program facts. Construction of the Aegis Ashore site in Poland has not overcome an initial delay that was largely due to contractor performance issues. MDA and the Army Corps of Engineers, which manages military construction at the site, took a number of measures to mitigate or reverse these delays, including working to modify the Army Corps of Engineers’ contract to permit joint occupancy of the site for a longer duration, and for the contractor to provide more granular project data to the Army Corps of Engineers. Also, the contractor has moved key personnel on site, and added a second shift. Program officials stated that they have withheld some award fees from the contractor over these issues. Program documentation states the contractor continues to be late in submitting documentation needed to move forward. If this and other issues are not corrected, it will increase the risk of further schedule slips. To make up for these delays, MDA introduced increasing levels of concurrency into its schedule, and shortened key phases of the delivery process. MDA has reduced the time allotted for Installation and Checkout activities from 16.5 months to 6.5 months. These activities would occur concurrently with the final phases of construction at the site. For example, installation of the deckhouse at the Poland site was scheduled for the end of the fourth quarter, fiscal year 2017, but was delayed to the end of the first quarter, fiscal year 2018. Despite this, installation and checkout activities still began in the fourth quarter of fiscal year 2017. The Navy’s systems testing procedures, which are needed prior to operational acceptance of the site, will have occurred mostly concurrently with the final stages of MDA’s construction and installation work on the site. MDA maintained through all of fiscal year 2017 that the site would be ready for delivery in December 2018 as scheduled. Program documentation noted that further program concurrency presented risks not only to the Aegis Ashore program, but to multiple elements relying on timely delivery of the site, up to and including the scheduled EPAA Phase 3 declaration. Early in fiscal year 2018, MDA announced that construction of the Poland site would not be completed until at least December 2019. Both Aegis Ashore sites in Europe have faced continuing challenges in several areas. For example, attrition problems have complicated efforts to keep the Poland site’s construction on schedule. These problems led to several persistent vacancies in important positions during the fiscal year. At one point in fiscal year 2017, the program lacked a full-time onsite program manager or dedicated government safety engineer, as well as other important positions. These roles had been, up to that point, filled by deputies in an acting capacity or were divided among others. MDA officials have also pointed to morale problems at the Poland site, where conditions for sailors are relatively austere. The Romania site has required more post-delivery support from MDA than was originally planned, largely due to quality and design issues in several areas. This post-construction wrap-up work was accounted for in MDA’s plans, but was originally planned to be complete by late fiscal year 2016. However, MDA has continued to provide warranty-like support in areas such as water supply, seismic-activity certification, and compatible electrical systems. Program officials stated that many of these issues arose from having to adapt Aegis systems to Romanian infrastructure, which in some cases proved to be a more complicated task than expected. Despite the issues encountered, the Romania site has remained operational throughout all of this work. The Aegis BMD Standard Missile-3 (SM-3) Block IB is a ship- and shore- based missile defense system interceptor designed to intercept short- to intermediate-range ballistic missiles during the middle stage of their flight. The Aegis BMD SM-3 interceptor has multiple versions in development or production: the SM-3 Blocks IA, IB, and IIA. Compared to the Aegis BMD SM-3 Block IA, the Block IB features an enhanced target seeker for increased discrimination, an advanced signal processor for engagement coordination, an improved throttleable divert and attitude control system for adjusting its course, and increased range. The Aegis BMD SM-3 Block IB interceptor is linked with the Aegis Ballistic Missile Defense (BMD) Weapons System and Aegis Ashore. For additional information about the Aegis Weapon Systems, see Appendix II and for Aegis Ashore, see appendix III. Recent technical and production problems have continually delayed a decision to authorize full production of the Aegis BMD SM-3 Block IB due to reliability concerns. Since fiscal year 2015, Aegis BMD SM-3 Block IB production has been delayed by several technical issues. In response to a GAO recommendation, program officials in 2015 delayed the decision to enter full-rate production until they could implement further testing and design changes. In fiscal year 2016, two failures during testing forced a suspension of interceptor deliveries, causing the program to miss its delivery target for the year. Table 7 provides key fiscal year 2017 Aegis BMD SM-3 Block IB program facts. The Aegis BMD SM-3 Block IB experienced two failures in fiscal year 2016, the investigation of which forced a temporary suspension of interceptor deliveries. As a result, MDA delivered only 33 interceptors out of a planned 47 for the year. MDA added the remaining interceptors to its planned delivery for fiscal year 2017, resulting in a target of 54 interceptors. The program successfully delivered 55 interceptors over the course of the year, and thus made up for the existing backlog. The program tracked two technical risks during fiscal year 2017, one of which it succeeded in removing, and another which will not be implemented into the production process until the third quarter of fiscal year 2018. According to MDA officials, the program successfully managed the transition of the production of the system’s Throttleable Divert and Attitude Control System to a new facility without experiencing significant delays or quality issues. In the other case, a component that was implicated in a previous test failure is currently undergoing a redesign. Program officials stated that they plan to have the new design certified by the second quarter of fiscal year 2018, and incorporated into the production line by the end of the third quarter. As we reported last year, problems testing a redesigned third-stage rocket motor on the Aegis BMD SM-3 Block IB forced the program to postpone its planned full production decision until the second quarter of fiscal year 2017, and successive delays have ensued. Though the tests validating the redesign were successful, the Undersecretary of Defense for Acquisition, Technology, and Logistics issued an Acquisition Decision Memorandum in February 2017 requesting an additional flight test in fiscal year 2017 as well as supporting analyses from the Director, Operational Test and Evaluation and the office of Cost Assessment and Program Evaluation. The memorandum issued these requirements in support of a planned full production decision in the first quarter of fiscal year 2018. Full-rate production for the Aegis BMD SM-3 Block IB was initially scheduled for fourth quarter, fiscal year 2012. MDA had one Aegis BMD SM-3 IB flight test scheduled for fiscal year 2017 at that time (FTM-24), and added another (FTM-26) in response to the Acquisition Decision Memorandum’s requirement, but neither were held as scheduled. MDA delayed FTM-24 to fiscal year 2020, in order to first analyze the new target missile’s performance to ensure it would work within the parameters of the test. While FTM-24’s delay was due to its very specific test design, its timing in fiscal year 2017 would have afforded additional information about the reliability of the interceptor that will not now be available before the full production decision. MDA deleted FTM-26 several months after adding it, and moved its objectives to coincide with NATO’s Formidable Shield – 17 naval exercises which took place in early fiscal year 2018 (wherein the system did achieve a successful intercept). As a result of the delay in conducting a test for production-readiness, the program is currently planning on a production decision in second quarter, fiscal year 2018. The Aegis Ballistic Missile Defense (BMD) Standard Missile-3 (SM-3) interceptor has multiple versions in development or production: the Aegis BMD SM-3 Blocks IA, IB, and IIA. The latest version, the Aegis BMD SM- 3 Block IIA interceptor, provides increased speed and range, more sensitive seeker technology, and an advanced kinetic warhead than previous versions. It is expected to defend against short-, medium-, and intermediate-range ballistic missiles. Additionally, most of the Aegis BMD SM-3 Block IIA components will differ from other the prior versions, and therefore requires new technology to be developed specifically for it. For additional information on the Aegis BMD SM-3 Block IB interceptor, see appendix IV. Initiated in 2006 as a cooperative development program with Japan, the Aegis BMD SM-3 Block IIA program was added as a capability to support the European Phased Adaptive Approach (EPAA) Phase 3 architecture to defend against longer-range threats. The Aegis BMD SM-3 Block IIA interceptor is planned to be fielded with Aegis Weapons System 5.1. For additional information on Aegis Weapons System, see appendix II. Table 8 provides key fiscal year 2017 Aegis BMD SM-3 Block IIA program facts. The first intercept flight test using the Aegis BMD SM-3 Block IIA interceptor, SFTM-01, was conducted in February 2017. It was originally scheduled for fiscal year 2016, but was delayed to evaluate technical issues discovered during previous tests. During this test, the Aegis BMD SM-3 Block IIA interceptor successfully engaged a medium-range ballistic missile (MRBM) target. The next intercept flight test, SFTM-02, occurred 4 months later, in June 2017. However, the interceptor failed to reach its MRBM target during this test. MDA convened a failure review board to identify the cause of the failure, and concluded that the failure was not attributable to a fault in the design or performance of the interceptor itself. The Navy is currently considering changes to its tactics, techniques, and procedures to address the findings from the failure review board. Two prior non-intercept tests using the Aegis BMD SM-3 Block IIA interceptor, although considered successful, showed potential design issues with the missile’s guidance system, which steers the interceptor to the target. Consequently, the program decided to develop a replacement component. The redesigned component passed initial acceptance testing and the program plans to employ it during FTM-29, which is scheduled for the second quarter of fiscal year 2018. The program continues to experience unit cost growth due to several factors, including decreases in the total amount being procured and increases in shipping costs. According to MDA officials, shipping costs grew because MDA underestimated the cost to ship missile components manufactured in Japan to the US on US-flagged ships. MDA officials stated that they did not adequately account for these costs when establishing the original baseline cost. Since 2014 the program’s unit cost has increased by almost 60 percent, from $24 million in fiscal year 2014 to $39 million in fiscal year 2017. Program officials stated that they do not expect either of these issues to lead to further cost growth in the future. The Aegis BMD SM-3 Block IIA program has limited schedule margin to address any issues prior to operational testing to meet the EPAA Phase 3 declaration by the first quarter of fiscal year 2019. For the EPAA Phase 3 declaration, the Aegis BMD SM-3 Block IIA interceptor must demonstrate the ability to intercept an intermediate-range ballistic missile (IRBM) target using remote sensor data. The program has one flight test, FTM-29, prior to its operational flight test. This test was originally scheduled for the first quarter of fiscal year 2018, but was delayed to the second quarter, and the launch site for the test was moved to the land-based Aegis Ashore facility in Hawaii. Adapting the Aegis BMD SM-3 Block IIA interceptor for a land-based test delayed this test further, from the first quarter to the second quarter of fiscal year 2018. Despite these delays, the dates for the operational test of the Aegis BMD SM-3 Block IIA—FTO-03 E1—and the EPAA Phase 3 declaration remain unchanged: the third quarter of fiscal year 2018 and first quarter of fiscal year 2019, respectively. That leaves the program approximately 3 to 5 months to resolve any issues discovered during FTM-29, prior to the operational test, which is needed to support the EPAA Phase 3 declaration. In addition, FTM-29 will be the Aegis BMD SM-3 Block IIA interceptor’s first test against an IRBM, first test of its ability to engage a target using remote sensor data, and the first test with to incorporate the new missile guidance system component incorporated. As a result of the complex test environment and limited time between tests, any significant failure during FTM-29 could lead to a delay in the EPAA Phase 3 declaration. Appendix VI: Command, Control, Battle Management, and Communications (C2BMC) C2BMC is a global system of hardware—workstations, servers, and network equipment—and software that integrates all missile defense elements of the Ballistic Missile Defense System (BMDS). Specifically, it allows users to plan operations, see the battle develop, and manage BMDS sensors. As the integrator, C2BMC enables the defense of a larger area than the individual BMDS elements operating independently and against more missiles simultaneously, thereby potentially conserving interceptor inventory. MDA is developing C2BMC in spirals, or software and hardware upgrades designed to improve various aspects of the integrated BMDS performance. MDA fielded Spiral 6.4 in 2011 and plans to complete the fielding of Spiral 8.2-1 by March 2018. The program is working on efforts for additional capabilities in the future. Table 9 provides an overview of C2BMC spiral development and table 10 provides key fiscal year 2017 C2BMC program facts. At the beginning of 2017, MDA completed the Spiral 6.4 assessment, which was designed to enable capabilities for Increment 3, Near Term Discrimination Improvements for Homeland Defense. The spiral performed nominally during testing, providing discrimination tasking from a forward-positioned radar for long-range threats, multiple-radar discrimination tasking of a threat, and several fixes related to sequencing and timing of messages. These tests provided performance data, which informed MDA’s Technical Capability Delivery for Increment 3 in March 2017. Despite this success, however, the spiral continues to have cyber vulnerabilities that place the BMDS operations in certain geographic areas at risk. For example, Spiral 6.4 has been in use since 2011, and its operating system (Windows XP) as well as other supporting software products will remain in the field well past their end of life cycle and support by vendors. According to program documentation, upgrading these systems before they are replaced by subsequent spirals is cost prohibitive, but program documentation does not indicate the cost. While MDA is in the process of fielding Spiral 8.2-1 to replace Spiral 6.4 in the Strategic, Northern and Pacific Commands by March 2018, Spiral 6.4 will remain operational at the European and Central Commands until the delivery of Spiral 8.2-3 in early fiscal year 2019. According to fiscal year 2017 MDA program reviews, the likelihood that critical cyber vulnerabilities are discovered is low for the remaining two years, and, according to MDA, no fielded system has been exploited to date. However, known vulnerabilities have been exploited in lab experiments. Moreover, MDA program documentation from fiscal year 2017 acknowledges that new security deficiencies could still be discovered, and if those or known deficiencies are exploited, mission capabilities like BMD planning, radar control, track reporting, and situational awareness may be significantly degraded. MDA collaborated with Combatant Commands to monitor and minimize the risks. In fiscal year 2017, MDA mitigated developmental risks necessary to complete the development and testing of C2BMC Spiral 8.2-1 in fiscal year 2018. Spiral 8.2-1—planned to support Enhanced Homeland Defense capabilities—was initially planned for delivery by December 2017, but, according to MDA officials, the delivery was delayed to allow additional time for assessment of results from BMD system-level ground test campaign called Ground Test (GT)-07a. Prior to GT-07a, the program identified risks that could affect interoperability with other elements and threat tracking, but, according to recent program documentation, MDA implemented fixes to many of them before the testing began. At the time of our assessment, MDA’s analysis was ongoing. However, MDA plans to complete its fielding by March 2018. MDA has begun testing Spiral 8.2-3, which is planned for fielding throughout fiscal year 2019, but it continues to face technical challenges and cost risk. This spiral is to replace Spiral 8.2-1 at the Strategic, Northern and Pacific Command, and Spiral 6.4 at European and Central Commands. According to MDA, the spiral is designed to enable a five-fold increase in the size of area that can be defended by the BMDS, and is an integral part of EPAA Phase 3. However, the program continues to track a prior risk and identified a new risk to an element level C2BMC capability needed for EPAA Phase 3 called Engage on Remote. Specifically, program documentation indicates that processing of data about threat missile flight paths, known as threat tracks, has issues that could reduce the likelihood of the successful engagements utilizing Aegis Ballistic Missile Defense in Engage on Remote scenarios. C2BMC has faced similar challenges with threat tracking capabilities for prior spirals, which required delays certain aspects of integration with Aegis BMD until fixes were implemented. MDA is implementing fixes to these issues in Spiral 8.2-3, which once fielded should resolve these integration issues, but it still needs to assess them in the ongoing test campaign for EPAA Phase 3. Since 2016, MDA Spiral 8.2-3 costs have increased by about 20 percent, from $68 million to $82 million. According to MDA documentation, the increase is due to several factors, including higher than expected costs for architecture and system engineering, as well as testing and integration requirements, and additional requirements for cybersecurity, which increased algorithm complexity required for Engage on Remote. MDA officials stated that some of the cost increases for cybersecurity were driven by DOD-wide cyber requirements, implemented in March 2014. Further cost increases, according to MDA, were driven by a warfighter request for geographic redundancy. While the original concept for 8.2-3 had the suites for Central and European Command at each location, MDA met the warfighter request by installing the suites at different locations so that losing a single node would not result in the loss of all capability for the warfighter. According to the C2BMC program, implementation of this requirement cost about $6.4 million. MDA identified element requirements for Spiral 8.2-5, which is planned for delivery in fiscal year 2021. This Spiral will integrate the Long Range Discriminating Radar and provide additional BMDS-level planning, track processing, and battle management capabilities. While MDA currently plans to hold the Preliminary Design Review by March 2018 and may report its acquisition baseline for the first time in the subsequent BMDS Accountability Report, program management documentation has already identified two specific challenges: Program documentation indicates that the Northern Command has concerns about performance issues associated with threat track processing, called System Track, for GMD engagements. While this is a key C2BMC function, track processing has been a challenge for other spirals supporting prior and upcoming regional and homeland defense capabilities. MDA is currently working with stakeholders to address this issue. The program also identified disconnects between LRDR, GMD and C2BMC, which are driving up element development and test costs, and delayed some capabilities initially planned to be delivered along with the LRDR. MDA developed a mitigation plan and established a working group to coordinate with stakeholders to address these issues. Appendix VII: Ground-based Midcourse Defense (GMD) The GMD system is a missile defense interceptor system designed to defend the United States against a limited intermediate and intercontinental ballistic missile attack from countries such as North Korea and Iran. To counter such threats to the homeland, GMD, in conjunction with a network of ground-, sea-, and space-based sensors, launches interceptors from missile fields based in Fort Greely, Alaska and Vandenberg Air Force Base, California. After launching from in-ground silos, the interceptor boosts towards the incoming enemy missile and releases an Exoatmospheric Kill Vehicle (EKV) to find and destroy the threat. GMD also has ground support and fire control systems that the warfighter relies upon to operate the system. Since the program’s initiation in 1996, DOD has spent over $45 billion developing, operating, and maintaining the GMD system, including: fielding ground station assets and a fleet of 44 interceptors; upgrading, redesigning, refurbishing, and retrofitting the system; successfully performing 5 out of 9 intercept tests and 3 out of 3 non-intercept tests; and developing Multi Object Kill Vehicle technology. Three of the intercept tests failed because of problems with the EKV while one of the tests failed because of a target failure, which is not associated with the GMD system. MDA has efforts ongoing to address concerns with the existing fleet of interceptors and increase protection to the U.S. homeland. In March 2013, the Secretary of Defense directed MDA to increase the number of fielded GMD interceptors from 30 to 44 by the end of 2017. To achieve this fielding goal, MDA performed a limited redesign of the CE-II, called the CE-II Block I, to fix known issues, address obsolescence, and improve producibility and cost. MDA also performed an extensive upgrade to the boost vehicle to improve reliability and address obsolescence issues. Although the CE-II Block I will address some concerns with the CE-II design, MDA determined a more complete redesign of the CE-II was needed. MDA subsequently developed an acquisition strategy and began developing the new kill vehicle, called the Redesigned Kill Vehicle (RKV). The RKV is intended to be more reliable, producible, testable, and cost effective. Table 11 provides key fiscal year 2017 GMD program facts. Fiscal year 2017 was a seminal year for the GMD program, as it achieved a number of major accomplishments. Over the past several years, the GMD program developed the newest interceptor version equipped with the CE-II Block I EKV and C2 boost vehicle. The program conducted its first successful flight test of this interceptor in May 2017 when it successfully intercepted a target representative of an intercontinental ballistic missile—another first for the GMD system. MDA proceeded to produce and field eight of these new interceptors and complete the refurbishment of Missile Field 1 in Fort Greely, Alaska, enabling the program to meet its directive from the Secretary of Defense to field 44 interceptors by the end of 2017. The program also fielded a software upgrade to the fire control segment of the GMD ground station, which included some improvements for battle management and discrimination. In addition, the program completed a preliminary design review for the RKV in March 2017. The program was able to execute all of these activities while also maintaining 24/7 availability of the system to the warfighter during a heightened period of North Korean missile testing. In total, the GMD program’s total cost has increased to over $67 billion and that total is likely to continue to increase as MDA defines future capability increments. In March 2013, we reported the total cost of the GMD program was estimated to be approximately $41 billion. Since that time, MDA defined new capability increments that included major GMD initiatives, such as the RKV and Multi Object Kill Vehicle efforts, which increased the program’s total cost. GMD is now the fourth most expensive DOD weapon system among a portfolio of 78 major defense acquisition programs, totaling approximately $1.5 trillion. As seen in table 12 below, only the F-35 and two naval programs are projected to cost more than the GMD system, demonstrating the department’s level of resources committed to defending the U.S. homeland against a long range ballistic missile attack. In November 2017, the President submitted to Congress an amendment to the fiscal year 2018 budget request for DOD to, among other things, increase current missile defense capacity, expand the sensor network, and accelerate technology development efforts. According to MDA, the request for additional funds was in direct response to recent demonstrations of advanced and accelerated capabilities by North Korea. MDA’s justification materials for the budget amendment includes an addition $774 million for GMD to build a new, 20-silo missile field at Fort Greely, begin procuring four additional interceptors, continue booster development, accelerate RKV development, and to add a non-intercept target to an initial RKV flight test. In total, MDA now plans to spend over $14 billion on GMD over the next six years with 64 total interceptors fielded by 2023. The new direction of the GMD program reflects a decision by the Director, MDA to set aside a strategy approved in 2016 by the prior Director for the government to take on the role of system integrator. Since the late 1990s, Boeing has been the GMD prime contractor, performing the role of system integrator. In 2011, Boeing competitively won a follow-on GMD development and sustainment contract that runs through December 2018. According to MDA, the government serving as the system integrator provides several benefits, such as eliminating organizational conflicts of interest issues—where industry tests and delivers assets based on requirements it wrote—and providing an unbiased assessment of system performance. However, a subsequent review team identified gaps and risks with implementing the strategy and the agency determined that transitioning to the new strategy at a time of heightened threat activity created unacceptable levels of risk for defending the U.S. homeland. On January 31, 2018, MDA awarded a sole-source contract modification to Boeing to extend the current development and sustainment contract. The contract modification has a total value of $6.56 billion and includes the accelerated delivery of a new 20-silo missile field, development of a new boost vehicle and the RKV, procurement of 20 new interceptors, and ground system upgrades. MDA faced a difficult choice, as both options included advantages and disadvantages. Under the prior approved strategy, MDA expected to achieve cost savings through competition. According to MDA, the sole- sourced labor rates for new development efforts under the recently modified development and sustainment contract have proven to be significantly higher than originally planned. In addition, MDA stated that the contract modification process is also often very lengthy, making it difficult for the agency to respond to the rapidly changing threat environment. MDA also stated that the lack of competition makes it challenging for the government to achieve favorable contract outcomes. Conversely, the government taking on the role of system integrator would make it responsible for managing multiple contracts. MDA plans to implement measures to mitigate some of the current challenges with extending the development and sustainment contract and ultimately provide the GMD program with a level of continuity during the current period of heightened threat activity. In October 2017, MDA informed the Under Secretary of Defense for Acquisition, Technology and Logistics (USD(AT&L)) that it had revised the RKV acquisition plan that was previously established in 2015 and approved by the USD(AT&L). This revision, in response to the advancement of the North Korean missile threat, accelerates the RKV’s development by concurrently performing development and production and reducing the number of necessary flight tests. MDA removed the previously-established alignment between flight tests and production decisions, which enables the program to begin production well before the system’s design is stabilized. In addition, MDA now plans to contract for production, on a sole source basis with the current GMD prime contractor, rather than through a full and open competition. According to MDA, the acceleration plan does not change the content of the RKV’s development plan and the program will continue to execute the same engineering processes including hardware qualifications essential to delivering the RKV. However, MDA’s revision of the RKV acquisition plan is more likely to prolong the effort rather than accelerate it. Our prior best practice work has shown that finding a balance between resources available (i.e., time and funding) and needed operational attributes (i.e., reliability and effectiveness) and obtaining buy-in from across the department is essential for program success. Although some risk may be necessary, programs that rely on heightened levels of concurrent development and production, starting production before stabilizing the design, and other risky practices greatly increase the likelihood a program will fail to deliver reliable, effective capabilities in an accelerated manner. The revised RKV plan no longer includes some of the key best practices, such as alignment between testing and production decisions included in the 2015 RKV plan. In addition, MDA has already experienced development delays and was operating on the threshold schedule of the prior acquisition plan, with no additional margin for delays. Moreover, MDA did not vet the revised plan in a similar manner to that of the 2015 RKV acquisition plan, which Congress required to be subject to approval by the USD(AT&L) and include rigorous elements for systems engineering, design, integration, development, testing and evaluation. The revised plan is also inconsistent with the acquisition best practice to “fly before you buy”, as MDA will begin production based on the results of design reviews rather than flight testing. In May 2017, we recommended that the Secretary of Defense require the Director of DOD’s Office of Cost Assessment and Program Evaluation (CAPE) perform a comprehensive review of the RKV acquisition strategy and provide any recommendations to the Secretary of Defense that the Director deems necessary and appropriate to obtain CAPE’s concurrence for the RKV program’s acquisition strategy. DOD did not concur with our recommendation, stating that CAPE and other organizations had previously reviewed the strategy prior to USD(AT&L)’s approval. As we noted in our report, CAPE raised serious concerns about the plan and expected MDA would encounter development delays. MDA justified the prior RKV plan, in part, so that it could begin urgently replacing the less reliable CE-Is as expeditiously as possible, which were fielded between 2004 and 2007. Under the newly accelerated plan, MDA does not plan to begin replacing the CE-I interceptors until after it has fielded the additional 20 RKV-equipped GBIs in 2024. However, GBIs only have an initial service life of 20 years and MDA previously decided not to make any upgrades to the CE-I because of initial plans to begin replacing them with RKVs in 2020. We continue to believe that DOD should implement our recommendation in order to ensure that MDA’s plans for the RKV are viable and meet the needs of the warfighter. A family of satellite-, sea-, and land-based radars provides worldwide sensor coverage to enable the Ballistic Missile Defense System (BMDS) to effectively detect and track threat missiles through all phases of their trajectory. Land-based BMDS sensors include the Army/Navy Transportable Radar Surveillance and Control Mode-2 (AN/TPY-2), Upgraded Early Warning Radars (UEWR), and the future Long Range Discrimination Radar (LRDR). Figure 16 below illustrates the locations of select BMDS sensors world-wide. AN/TPY-2 is a transportable X-band high resolution radar that is capable of tracking all classes of ballistic missiles. AN/TPY-2 in the forward-based mode is capable of detecting and tracking missiles in all stages of flight to support Aegis BMD and GMD engagements and provides threat missile data to C2BMC. AN/TPY-2 in the terminal mode can track missiles in the later stages of flight to support THAAD engagements. Five AN/TPY-2 radars for use in forward-based mode are deployed to support regional defense: two in U.S. European Command, two in U.S. Pacific Command, and one in U.S. Central Command. Two AN/TPY-2 radars for use in terminal mode is also deployed to U.S. Pacific Command. UEWR are U.S. Air Force early warning radars that are upgraded and integrated into the BMDS to provide sensor coverage for critical early warning, tracking, object classification, and cueing data. Upgraded Early Warning Radars are located in Beale, California; Fylingdales, United Kingdom; and Thule, Greenland. MDA awarded a contract to upgrade the early warning radars in Clear, Alaska and at Cape Cod, Massachusetts, and both of these assets are approaching their operational acceptance for use in the BMDS. The upgrades to the Clear and Cape Cod Early Warning Radar sites are joint MDA / Air Force projects. Both organizations are contributing funding to these sites. LRDR is being designed as an S-band radar intended to address the need for persistent, precision tracking and discrimination capability in the Pacific sensor architecture. MDA anticipates the addition of LRDR will optimize the employment of the Ground-based Midcourse Defense (GMD) interceptors and address evolving threats. The radar will be located at Clear Air Force Station, Alaska with initial operational capability planned for 2020. Table 13 provides key fiscal year 2017 Sensors program facts. To address future requirements and as part of its spiral development process, AN/TPY-2 transitioned from its Increment 2 software development phase to its Configuration 3 software development phase. The transition results in Configuration 3 subsuming all unfinished Increment 2 content including 44 percent of development costs ($60 million), 31 percent of productions costs ($61 million), 88 percent of operations and support costs ($2,281 million), and 100 percent of disposal costs ($30 million). Four Knowledge Points and Technical Performance Metrics for the program were also carried over from Increment 2. New capabilities were also added in Configuration 3 including electronic protection and discrimination improvements. Additionally, the Conditional Materiel Release of software upgrade CX 2.1.0 was delayed from the first quarter of fiscal year 2017 to the first quarter of fiscal year 2018. To mitigate this delay, MDA executed an Urgent Software Release for CX 2.1.1 to support the fielding of Command, Control and Battle Management (C2BMC) S6.4-3 in December 2016. The UEWR is executing a concurrent development approach to improve UEWR Object Classification (OC), Data Processor/Signal Processor (DP/SP), and Bias Correction capabilities, and to certify the UEWR Clear and Cape Cod sites for use in the BMDS. Because of this concurrent development, a delay in the Beale UEWR’s operational acceptance for the OC and DP/SP program has had cascading effects on the same upgrades for the Clear, Cape Cod, and Fylingdales UEWRs in addition to the BMDS Certification for the Clear UEWR, delaying the use of these key radar capabilities. These delays are shown in figure 17 below: The delay in Beale’s Operational Acceptance was due in part to the following: The contractor, Raytheon, delivered unacceptable UEWR technical orders that required rework. Development and operational testing supporting the operational acceptance were delayed because the operators required remediation of all emergency operational maintenance issues found on the operational UEWRs. Some UEWR software required fixes to address deficiencies. Other programs were competing for test time on needed equipment. The delay in operational acceptance will affect the delivery of Bias Correction for the Clear and Cape Cod UEWRs in addition to the delivery of and Data Processor/ Signal Processor improvements to support the missile defense mission of Beale, Clear, Cape Cod, and Fylingdales UEWRs. It has also delayed the BMDS Certification of the Clear UEWR. Because the program currently has sufficient schedule margin before the Cape Cod BMDS Certification, the delays have not yet affected the missile defense mission for that radar. The program office is working with Raytheon on a recovery plan to address the Technical Order issues and other issues that arose from the developmental and operational testing conducted in July 2017. We have previously reported that concurrent development increases program risk for cost and schedule delays caused by redesigns and retrofits needed after testing has occurred. In fiscal year 2017, MDA made progress towards stabilizing LRDR’s design, by completing a preliminary design review in March 2017 and a critical design review in September 2017. The program also began production of long lead radar electronic components and awarded a military construction contract for the Mission Control Facility. However, the program has experienced challenges integrating multiple facilities- related projects, which require synchronization between MDA, the U.S. Air Force, the U.S. Army Corps of Engineers, and contractors. For example, in fiscal year 2017, the LRDR program began demolishing a decommissioned, Cold War-era radar, which sits on the planned LRDR site at Clear Air Force Station. The program discovered that the radar’s foundation and surrounding soil contained steel and concrete coated with polychlorinated biphenyl (PCB), which was a common industrial material used at the time of the radar’s construction in the late 1950s. PCBs do not readily break down once in the environment and have been demonstrated to cause a variety of adverse health effects. In April 2017, the U.S. Army Corps of Engineers modified its contract for the removal of the PCB- contaminated foundation and soil and plans to complete excavation and removal by early fiscal year 2018. Demolition is now expected to be completed in 2019 and the additional costs for these complications are not covered under the program’s resource baseline. In June 2017, the LRDR program initiated a power study with the commercial power supplier for the LRDR radar. The program expects to complete the study on LRDR’s power demands on the commercial electrical grid, as well as assess updated U.S. Northern Command concept of operations to determine the extent, if any, of system capabilities, limitations, and mitigations. During the LRDR critical design review, MDA officials stated that U.S. Air Force informed the agency that it required 24/7 availability of the radar if it is to become operational. According to MDA officials, the current LRDR design, with its reliance on commercial power and limited back-up generators, would not provide that capability. MDA officials stated the program plans to increase the number of back-up generators, which may increase the military construction costs and annual operational expense of the radar. A November 2016 study of LRDR’s power system performed for MDA by a contractor indicated that agreements with the commercial power provider place limitations on the warfighter’s ability to operate the radar without consulting the commercial power provider in advance and that emergency activation of the radar could result in other commercial power provider customers having their power supply temporarily switched off if the generators were not brought online in time. The LRDR program has also encountered design challenges with the radar’s circuit card assemblies, as the planned design included the use of pure tin parts, which are susceptible to corrosion. Lockheed Martin, the prime contractor for the LRDR program, plans to replace some of the pure tin parts with parts that have a lead-based finish, as available. The program does not anticipate there to be enough of these parts available and estimated that redesigning the pure tin parts would result in an approximate 9-month delay. For those parts that cannot be readily replaced, Lockheed Martin plans to use corrosion mitigation techniques, such as applying conformal coating to the circuit card assemblies and applying lead solder. Although MDA maintains that these mitigation techniques will ensure corrosion-free operations, government and industry studies show that such mitigations reduce, but do not eliminate the risk, Lockheed Martin is conducting on-site inspections and providing additional information on the historical use of pure tin parts in similar systems and anticipates being able to clear the unmitigated, pure tin parts through the MDA’s Parts, Materials, and Processes control board. The Missile Defense Agency’s (MDA) Targets and Countermeasures procures missiles to serve as targets during the developmental and operational testing of independent or integrated ballistic missile defense system (BMDS) elements. Specifically, this program supplies MDA with short-, medium-, intermediate-, and intercontinental-range targets to test, verify, and validate the BMDS elements’ performance in threat relevant environments. As targets are solely test assets, they are not operationally fielded. The number of targets that the program supplies vary based on each element’s requirements and testing schedule. While some targets have been used for years, others have been recently added or are now being developed to more closely represent current and future threats. The quality and availability of these targets is instrumental to the execution of MDA’s flight test schedule. Table 14 provides key fiscal year 2017 Targets and Countermeasure program facts. Despite challenges MDA has previously experienced using new targets during intercept flight tests, in fiscal year 2017, the program successfully flew the first intercontinental ballistic missile (ICBM) range target to support a critical intercept test for the GMD element. The GMD element provides the warfighter capability to engage and destroy intermediate- and intercontinental-range ballistic missile threats for the protection of the United States. In March 2013, the Secretary of Defense announced plans to increase the number of deployed GMD interceptors called Ground- based interceptors (GBI) from 30 to 44 by the end of 2017. To do this, a test—FTG-15—was needed to collect data on the GBI’s new booster design and demonstrate its performance against a target at the ICBM threat range before completing this mandated fielding goal. The successful flight of the ICBM target, the GBI’s performance against the target, and other information gathered during this test will provide the warfighter with a better understanding of the GBI’s capabilities and limitations. For further details on the GMD element, see appendix VII. The Targets and Countermeasures program is planning to contract for up to 12 additional medium range ballistic missile (MRBM) T1/T2 targets despite cost growth, schedule delays, and the lack of demonstrated performance. In fiscal year 2014, the program competitively awarded the initial contract for 6 MRBM T1/T2 targets with an option for an additional 12, for a total of 18. According to program officials, the contract was structured with a fixed price for the target and incentives to ensure successful execution during testing. However, the contractor has been underperforming since the award. First, this target’s costs have continued to significantly increase as some MDA officials originally warned. One of MDA’s reasons for selecting the current MRBM T1/T2 contractor was because it offered a lower price. However, some officials within MDA objected to this award due to the near certainty that the contractor would overrun costs. Since then, both MDA and Defense Contract Management Agency (DCMA) officials have acknowledged that the contractor did not adequately account for the costs associated with this target. Consequently, this target’s costs have been volatile, and despite changes and rebaselines, the contractor has been unable to meet projections. In fiscal year 2017, the program conducted another review to address significant cost growth and set new projections, and despite a relatively steady period of performance against these new projections, DCMA officials believe that this contractor will continue to have increasing costs. In addition, the first delivery of this target has been delayed almost five years beyond the original plan primarily due to contractor performance issues. There was an initial delay because the contract was awarded later than planned due to an investigation of an unsubstantiated procurement integrity allegation. However, since then, contractor performance issues have further delayed the first target delivery, necessitating several substitute targets for tests in the interim. Finally, since the program will not fly the first target in a test until the second quarter of fiscal year 2019, the target’s performance has yet to be demonstrated. Hence, buying an additional 12 targets without confirmation of the target’s performance is a significant risk for the program, as even one failure would delay all future tests with this target, and ultimately, the entire test program. Appendix X: Terminal High Altitude Area Defense (THAAD) THAAD is a rapidly-deployable ground-based system able to defend against short-, medium-, and intermediate- range ballistic missile attacks during the middle and end stages of a missile’s flight. THAAD is organized as a battery that consists of interceptors, launchers, an Army Navy / Transportable Surveillance (AN/TPY-2) radar, a fire control and communications system, and other support equipment. The first two batteries were originally conditionally accepted by the Army for operational use. Since then, THAAD received urgent materiel release approval from the Commanding General of the United States Army Aviation and Missile Command to enable an earlier delivery of equipment for THAAD batteries one through six for operational use to meet the Army’s request to support urgent warfighter needs. The MDA plans to continue THAAD production through fiscal year 2024, for a total of 7 batteries, 503 interceptors, and 7 radars. MDA has two THAAD acquisition efforts—THAAD 1.0 and THAAD 2.0. THAAD 1.0 is for the production of the batteries, interceptors, and supporting hardware and provides the warfighter with initial integrated defense against short- and medium-range threats in one region. Appendix X: Terminal High Altitude Area Defense (THAAD) THAAD 2.0 is primarily software enhancements that expand THAAD’s ability to defend against threats in multiple regions and at different ranges, and adds debris mitigation and other upgrades. Table 15 provides key fiscal year 2017 THAAD program facts. THAAD successfully completed two tests. In FTT-18 (previously scheduled for fiscal year 2015), THAAD successfully intercepted an Intermediate Range Ballistic Missile (IRBM)-representative target, demonstrating THAAD’s capability against IRBM threats. THAAD has been deployed to Guam since 2013 to defend against IRBM threats, but this is the first time it has demonstrated that capability in a flight test. According to program officials, for the second planned flight test originally named FTT-15, MDA changed the name to Flight Experiment THAAD (FET)-01 to more accurately reflect the experimental purpose of the test. However, an intercept was formerly a primary test objective in FTT-15, but this objective was removed before the test name was changed to FET-01. In FET-01, although not a primary objective, THAAD did complete an intercept of a medium-range ballistic missile target with countermeasures. Despite the intercept, the test revealed significant operational limitations. THAAD delayed the delivery of several key hardware and software deliveries that will impact warfighter capabilities. Figure 20 shows the delayed hardware and software deliveries. Appendix X: Terminal High Altitude Area Defense (THAAD) Appendix X: Terminal High Altitude Area Defense (THAAD) interceptors due to a production issue that had cascading schedule effects on interceptor production and delivery. In May 2017, we found that THAAD interceptor production was halted due to a parts quality issue discovered when a connector in the interceptor failed multiple testing iterations. Upon investigation, the contractor learned that one of its sub-contractors changed the manufacturing process on the connector without informing Lockheed Martin. According to program officials, Lockheed Martin halted interceptor delivery but continued interceptor production. The connector was redesigned and incorporated into 20 interceptors, which again failed testing before being deployed. After a second redesign the connector passed testing and interceptor delivery resumed in April 2017. As of December 2017, there were 16 interceptors that had been produced but not yet fitted with the redesigned connector. Program officials report that the delay should result in about 2 months of delivery delays of the last interceptor lot currently under contract. According to program officials, to prevent similar problems from occurring again, the government revised the Parts Materials and Processes Control plan to provide improved guidance and clarity related to parts selection and change control; added additional criteria to annual audits to enhance review of supplier parts management, materials, and processes; and tightened controls on suppliers to report any significant changes. Appendix X: Terminal High Altitude Area Defense (THAAD) THAAD, that would be the Army) not later than the date the President’s fiscal year 2021 budget is submitted. However, in a memo from the Secretary of the Army, the Army said that it would non-concur with a transfer of the THAAD program in its current state because it cannot meet the Army’s global mission requirements. To meet global mission requirements for the THAAD mission, the Army requires about $10.1 billion of additional hardware, life-cycle sustainment funding, and AN/TPY-2 upgrades. MDA is willing to transfer to the Army the THAAD program of record as is. An official from the Army said that this impasse has existed before, but that the recent reprioritization of the THAAD mission contributed to it. For further details on the AN/TPY-2 program, see appendix VIII. Table 16 below shows the difference between the THAAD program of record and the Army’s requirements. Appendix X: Terminal High Altitude Area Defense (THAAD) MDA’s requirements and warfighter requirements exist and can lead to situations such as this impasse. Consequently, we recommended that the Secretary of Defense require MDA to develop a plan to transition operational requirements analysis currently performed within MDA’s Achievable Capabilities List to the U.S. Combatant Commanders, with U.S. Strategic Command as the lead entity and, in the interim, require MDA to obtain their concurrence of the Achievable Capabilities List prior to its release. The Department of Defense (DOD) did not agree with our recommendation. However, as evidenced by the discrepancy between the Army’s and MDA’s requirements for the THAAD and AN/TPY-2 program, the difference between MDA’s requirements and those of the warfighter will continue to present substantial problems to DOD in executing the missile defense mission, and we continue to believe that our recommendation should be implemented. In addition to the contact named above, LaTonya Miller, Assistant Director; Matthew Ambrose; Kristine Hassinger; Helena Johnson; Joe Kirschbaum; Wiktor Niewiadomski; Steven Stern; Brian Tittle; Hai V. Tran; Alyssa Weir; Tonya Woodbury; and Robin Wilson made key contributions to this report. Missile Defense: Some Progress Delivering Capabilities, but Challenges with Testing Transparency and Requirements Development Need to Be Addressed. GAO-17-381. Washington, D.C.: May 2017. Missile Defense: Opportunities Exist to Reduce Acquisition Risk and Improve Reporting on System Capabilities. Washington, D.C.: Mar. 2015. Missile Defense: Mixed Progress in Achieving Acquisition Goals and Improving Accountability. GAO-14-351. Washington, D.C.: Apr. 2014. Missile Defense: Opportunity to Refocus on Strengthening Acquisition Management. GAO-13-432. Washington, D.C.: Apr. 2013. Missile Defense: Opportunity Exists to Strengthen Acquisitions by Reducing Concurrency. GAO-12-486. Washington, D.C.: Apr. 2012. Missile Defense: Actions Needed to Improve Transparency and Accountability. GAO-11-372. Washington, D.C.: Mar. 2011. Defense Acquisitions: Missile Defense Transition Provides Opportunity to Strengthen Acquisition Approach. GAO-10-311. Washington, D.C.: Feb. 2010 Defense Acquisitions: Production and Fielding of Missile Defense Components Continue with Less Testing and Validation Than Planned. GAO-09-338. Washington, D.C.: Mar. 2009 Defense Acquisitions: Progress Made in Fielding Missile Defense, but Program is Short of Meeting Goals. GAO-08-448. Washington, D.C.: Mar. 2008 Defense Acquisitions: Missile Defense Acquisitions Strategy Generates Results but Delivers Less at a Higher Cost. GAO-07-387. Washington, D.C.: Mar. 2007 Defense Acquisitions: Missile Defense Agency Fields Initial Capability but Falls Short of Original Goals. GAO-06-327. Washington, D.C.: Mar. 2006. Defense Acquisitions: Status of Ballistic Missile Defense Program in 2004. GAO-05-243. Washington, D.C.: Mar. 2005 Missile Defense: Actions Are Needed to Enhance Testing and Accountability. GAO-04-409. Washington, D.C.: Apr. 2004.
Since 2002, MDA has been developing a Ballistic Missile Defense System that can identify and intercept enemy threats. MDA has received approximately $132 billion and is planning to spend an additional $47.8 billion through fiscal year 2022 to continue its efforts. The National Defense Authorization Act for Fiscal Year 2012 included a provision that GAO annually assess and report on the extent to which MDA has achieved its acquisition goals and objectives. This report addresses (1) the progress MDA made in achieving fiscal year 2017 goals; (2) the extent to which MDA uses contracting vehicles known as undefinitized contract actions; and (3) the extent to which models provide credible information about the system's operational performance. To do this work, GAO reviewed planned fiscal year 2017 baselines and other documentation and assessed them against baseline reviews and GAO's acquisition best practices guides. In addition, GAO interviewed relevant officials. In fiscal year 2017, the Missile Defense Agency (MDA) made mixed progress in achieving its delivery and testing goals. MDA continued to deliver assets to the military services. However, system-level integrated capabilities, such as some discrimination and integrated cyber defense improvements, were delayed and delivered with performance limitations. Several programs achieved notable firsts, including the first intercept of an Intercontinental Ballistic Missile. However, one program experienced a failure, and other tests were delayed or deleted. Moreover, GAO found challenges in MDA's processes for communicating the extent and limitations of integrated capabilities when they are delivered. As a result, warfighters do not have full insight into the capabilities MDA delivers. GAO found that the average length of the undefinitized period and the not-to-exceed price of MDA's undefinitized contract actions, which authorize contractors to begin work before an agreement on terms, specifications, or price have been agreed upon, have increased over the past 5 years. While MDA policy permits use of undefinitized contracts on a limited basis, GAO and others have found that they can place unnecessary cost risks on the government. MDA does not completely assess BMDS performance using traditional flight tests. Instead, MDA relies on models, some of which produce data with limited credibility. According to Department of Defense and MDA policy, models used to operationally assess weapons systems must be accredited to ensure they reflect the real-world system. In addition, using unaccredited models increases the risk that test results could be distorted, and leaves decision makers without key information on how the system will perform. While MDA has taken steps to improve its models, it has used many models in system operational ground tests that were not certified for that use (see figure). Additionally, MDA does not communicate model limitations to some decision makers. Percentage of Accredited Models Used in Operational Assessments of Ballistic Missile Defense System Capability Deliveries, 2015 through 2017 GAO is making six recommendations to, among other things, improve the way MDA communicates capability deliveries; better report information about MDA's use of undefinitized contract actions; and address the challenges MDA has encountered with certifying its test models and communicating limitations of those models. DOD partially concurred with the first recommendation and concurred with the other five. GAO continues to believe the recommendations are valid as discussed in the report.