In March 2015, NNSA issued a new unclassified strategic plan for NNSA  efforts to address future nuclear proliferation and terrorism threats for  fiscal years 2016 to 2020 in response to an August 2014 interim report by  the Secretary of Energy Advisory Board Task Force on Nuclear  Nonproliferation. The DOE Secretary  agreed with this recommendation and asked NNSA to begin drafting such  a report. NNSA used these methods  throughout the OTH initiative process to identify and assess future  threats, and they included a literature review, structured interviews and  workshops, and a peer review of OTH findings by external subject matter  experts. This NIS specific phase of OTH work included the  use of several established methods consistent with those in GAO- developed guidance, including a literature review encompassing several  dozen U.S. government documents and other publications, two  workshops, and structured interviews with participants from the U.S.  government, nongovernmental organizations, and industry. The Way NNSA Implemented Established Methods Had Several Limitations

Through reviews of NNSA documentation and interviews with NNSA  officials, we identified several limitations in the way NNSA implemented  the established methods it used (i.e., the literature review, structured  interviews, and peer review by external subject matter experts) through  the OTH initiative to identify and assess future proliferation-related threats  and trends, which raise concerns about the quality and usefulness of the  analyses produced through the process. We found, based on our interviews with NNSA officials, that the OTH  structured interview process conducted in the first OTH phase had  limitations including the following:
According to NNSA officials, NNSA did not document the rationale for  selecting individuals to be interviewed to show that the people  selected as data sources provided a balanced range of views, were  sufficiently knowledgeable, and that their opinions may be considered  candid and accurate, consistent with common methodological  practice. According to NNSA officials, NNSA did not have detailed records or  analyses of these interviews, although this is called for by established  methods and common methodological practices. We have  found that such practices also help ensure the reliability and validity of the  information being collected. According to NNSA officials, NNSA did not document the results of  the peer review process, such as through a summary document or  record of analysis, consistent with established peer review standards  that state the importance of receiving written comments from all peer  reviewers and producing a record of these comments. Taken together, the limitations we found in the OTH process raise  concerns about the quality of the analyses produced and about the  usefulness of the OTH initiative—as it has been implemented so far—as  a DNN planning tool. It Is Not Clear How Recent NNSA Nonproliferation Organization and Planning Decisions Were Informed by the Over the Horizon Initiative

The OTH initiative was intended to help inform DNN management in its  decisions about the future DNN organization and resources needed to  address future threats, but it is not clear how information generated by the  OTH initiative informed the January 2015 DNN reorganization. According to  NNSA officials, the OTH initiative is in the process of being integrated into  a new DNN strategic planning and integration function that will be led out  of the office of the DNN Deputy Administrator, and which will manage  integration of OTH studies into strategic planning documents, such as  future versions of the NNSA strategic plan to reduce global nuclear  threats. In January 2015, NNSA reorganized the DNN  programs into a new program office structure by consolidating the  previous five DNN program offices into four offices. NNSA officials in the DNN Deputy Administrator’s office also stated that  there were other drivers behind the reorganization, and that it was not  solely a threat-based reorganization. The Extent to Which OTH Threat Assessments Informed DOE’s and NNSA’s Plan for Countering Nuclear Threats Is Unclear

The OTH initiative was established to institutionalize long-term DNN  planning and inform development of DNN strategic plans. Conclusions

DNN programs remain critical to addressing the serious threats  associated with the proliferation of nuclear and radiological weapons. Recommendation for Executive Action

To ensure that any future NNSA effort—through the OTH initiative or  another process—to assess proliferation threats and the implications for  DNN produces high-quality information, we recommend that the NNSA  Administrator better implement established methods, including literature  reviews, structured interviews, and peer reviews. Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) describe how the National Nuclear Security  Administration (NNSA) assessed potential future proliferation threats  through its “Over the Horizon” (OTH) initiative and assess the limitations,  if any, in the process used by the initiative and (2) examine the extent to  which NNSA used information about the potential future proliferation  threats assessed through the OTH initiative in DNN organization and  planning decisions. In addition, we reviewed  the March 2015 NNSA strategic plan and draft classified appendix  describing the planned activities of DNN and other NNSA programs to  prevent, counter, and respond to nuclear terrorism and proliferation  threats over the next 5 years, and we interviewed NNSA officials about  the plan and the extent to which it was informed by information generated  through the OTH initiative. 1. 2. 3.
Why GAO Did This Study

The threat posed by the proliferation of nuclear and radiological weapons remains one of the most pressing U.S. national security challenges, and these threats are evolving. Information produced by the OTH initiative about future proliferation threats is intended to support long-term DNN planning and other DNN management decisions.
The Joint Explanatory Statement Accompanying the National Defense Authorization Act for Fiscal Year 2014 includes a provision for GAO to assess NNSA's nuclear nonproliferation programs. This report (1) describes how NNSA assessed potential proliferation threats through its OTH initiative and assesses the limitations, if any, in the process used by the initiative and (2) examines the extent to which NNSA used information about the potential threats assessed through the OTH initiative in DNN organization and planning decisions.
GAO analyzed NNSA documentation on the OTH initiative and NNSA planning documents. GAO also interviewed DOE and NNSA officials, DOE national laboratory representatives, and external subject matter experts and external validators in peer review groups involved in OTH activities.

What GAO Found

The National Nuclear Security Administration (NNSA)—a semiautonomous agency within the Department of Energy (DOE)—has used a variety of established methods in its “Over the Horizon” (OTH) initiative to assess potential proliferation threats, but the implementation of these methods had several limitations. NNSA used established methods in its OTH initiative, including: (1) a literature review of studies on proliferation threats; (2) structured interviews with participants from the U.S. government, nongovernmental organizations, and industry; and (3) a peer review of OTH findings by external subject matter experts. Although NNSA used established methods, the way NNSA implemented them had several limitations. For example, NNSA officials did not document the rationale for selecting individuals for structured interviews to show that those selected provided a balanced range of views and were sufficiently knowledgeable. NNSA officials also did not have detailed records or analyses of these interviews. Established methods and common methodological practices call for structured interviews to be documented and analyzed. GAO has found that such practices help ensure the reliability and validity of the information collected. Another limitation was that the peer review was not conducted in a way consistent with established standards. For example, NNSA officials did not document the results of the peer review, as called for by established peer review standards, and some reviewers told GAO that it was unclear how their comments had been incorporated into a 2012 OTH report. These limitations raise concerns about the quality of the analyses produced, and about the usefulness of the OTH initiative, as it has been implemented so far, as a planning tool in NNSA's Office of Defense Nuclear Nonproliferation (DNN).
It is unclear how information generated by the OTH initiative informed recent organization and planning decisions in the DNN office, even though NNSA established the initiative partly to support decision making on such matters. For instance, in January 2015, NNSA reorganized DNN programs by consolidating five DNN program offices into four offices. The extent to which the OTH initiative informed the reorganization is unclear because NNSA officials could not provide documentation or examples of links between OTH findings and elements of the reorganization. Instead, NNSA officials told GAO that the OTH initiative was one of several sources that informed the reorganization and that there were other reasons for it, including consolidating similar functions to achieve efficiencies. In addition, in March 2015, NNSA issued an unclassified strategic plan that was presented as defining and describing NNSA program missions—including DNN programs—to prevent, counter, and respond to future nuclear proliferation and terrorism threats. However, the extent to which the OTH initiative informed the plan was unclear because the plan and the OTH initiative covered different time frames, and because the plan and a draft classified appendix contained conflicting information about the role of the initiative in the plan's development. GAO is not making a recommendation on these matters because NNSA is creating a new strategic planning function that will oversee the OTH process and manage integration of OTH and other long-range studies into future versions of the NNSA strategic plan on nuclear threats.

What GAO Recommends

In conducting any future proliferation threat assessments, through the OTH initiative or another process, NNSA should better implement established methods, including literature reviews, structured interviews, and peer reviews. NNSA agreed with this recommendation and is taking action to address it.