DNDO conducted its formal tests in two phases. According to DNDO, these  special tests were recommended by subject matter experts outside the  ASP program to address the limitations of the original NTS test plan,  including    available time and funding resources,    special nuclear material sources, and  the number of test configurations that could be incorporated in  the test plan, including source isotope and activity, shielding  materials and thicknesses, masking materials, vehicle types,  and measurement conditions. DNDO’s Test Methods Raise Concerns Regarding the Reliability of Test Results

Based on our analysis of DNDO’s test plan, the test results, and  discussions with experts from four national laboratories, we are  concerned that DNDO used biased test methods that enhanced the  performance of the ASPs. Specifically, we are concerned because almost all of the materials, and  most combinations of materials, DNDO used in the formal tests were  identical to those that the ASP contractors had specifically set their ASPs  to identify during the dry runs and dress rehearsals. It is highly unlikely  that such favorable circumstances would present themselves under real  world conditions. Based on our  analysis, the masking materials DNDO used at NTS did not sufficiently test  the performance limits of the ASPs. DOE national laboratory officials  raised similar concerns to DNDO after reviewing a draft of the test plan in  November 2006. Yet, DNDO’s  own test plan for “ASP Special Testing” states, “The DNDO ASP NTS Test  Plan was designed to… measure capabilities and limitations in current  ASP systems.”
In addition, the NTS tests did not objectively test the ASPs against the  currently deployed radiation detection system. DNDO did not include this  critical step in its formal tests. In regards to the special tests DNDO conducted, based on what DNDO has  told us and our own evaluation of the special test plan, we note that    because DNDO did not consult NIST on the design of the blind tests,  we do not know the statistical significance of the results, and  the tests were not entirely blind because some of the nuclear materials  used in the blind tests were also used to calibrate the ASPs on a daily  basis. We raised our  concerns about this issue and those of DOE and CBP to DNDO’s attention  on multiple occasions. On August 30, 2007, the DHS  Undersecretary for Management recommended that the Secretary of  Homeland Security delay Secretarial Certification of ASPs for an  additional two months. In our view, the tests conducted by DNDO at the Nevada Test Site between  February and March 2007 used biased test methods and were not an  objective assessment of the ASPs’ performance capabilities. As a result of DNDO’s test methods and the limits of the tests—including a  need to meet a secretarial certification deadline and the limited  configurations of special nuclear material sources, masking, and shielding  materials used—we believe that the results of the tests conducted at the  Nevada Test Site do not demonstrate a “significant increase in operational  effectiveness” relative to the current detection system, and cannot be  relied upon to make a full-scale production decision. Combating Nuclear Smuggling: DHS’s Cost-Benefit Analysis to Support  the Purchase of New Radiation Detection Portal Monitors Was Not Based  on Available Performance Data and Did Not Fully Evaluate All the  Monitors’ Costs and Benefits. Ports of Entry.
Why GAO Did This Study

The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are key elements in our national defenses against such threats. DHS has sponsored testing to develop new monitors, known as advanced spectroscopic portal (ASP) monitors. In March 2006, GAO recommended that DNDO conduct a cost-benefit analysis to determine whether the new portal monitors were worth the additional cost. In June 2006, DNDO issued its analysis. In October 2006, GAO concluded that DNDO did not provide a sound analytical basis for its decision to purchase and deploy ASP technology and recommended further testing of ASPs. DNDO conducted this ASP testing at the Nevada Test Site (NTS) between February and March 2007. GAO's statement addresses the test methods DNDO used to demonstrate the performance capabilities of the ASPs and whether the NTS test results should be relied upon to make a full-scale production decision.

What GAO Found

Based on our analysis of DNDO's test plan, the test results, and discussions with experts from four national laboratories, we are concerned that DNDO's tests were not an objective and rigorous assessment of the ASPs' capabilities. Our concerns with the DNDO's test methods include the following: (1) DNDO used biased test methods that enhanced the performance of the ASPs. Specifically, DNDO conducted numerous preliminary runs of almost all of the materials, and combinations of materials, that were used in the formal tests and then allowed ASP contractors to collect test data and adjust their systems to identify these materials. It is highly unlikely that such favorable circumstances would present themselves under real world conditions. (2) DNDO's NTS tests were not designed to test the limitations of the ASPs' detection capabilities--a critical oversight in DNDO's original test plan. DNDO did not use a sufficient amount of the type of materials that would mask or hide dangerous sources and that ASPs would likely encounter at ports of entry. DOE and national laboratory officials raised these concerns to DNDO in November 2006. However, DNDO officials rejected their suggestion of including additional and more challenging masking materials because, according to DNDO, there would not be sufficient time to obtain them based on the deadline imposed by obtaining Secretarial Certification by June 26. 2007. By not collaborating with DOE until late in the test planning process, DNDO missed an important opportunity to procure a broader, more representative set of well-vetted and characterized masking materials. (3) DNDO did not objectively test the performance of handheld detectors because they did not use a critical CBP standard operating procedure that is fundamental to this equipment's performance in the field. Because of concerns raised that DNDO did not sufficiently test the limitations of ASPs, DNDO is attempting to compensate for weaknesses in the original test plan by conducting additional studies--essentially computer simulations. While DNDO, CBP, and DOE have now reached an agreement to wait and see whether the results of these studies will provide useful data regarding the ASPs' capabilities, in our view and those of other experts, computer simulations are not as good as actual testing with nuclear and masking materials.