The Department of Homeland Security Appropriations Act for Fiscal Year  2007 states that “none of the funds appropriated … shall be obligated for  full scale procurement of  monitors until the Secretary of Homeland  Security has certified … that a significant increase in operational  effectiveness will be achieved.” Congress enacted a similar requirement in  DHS’s fiscal year 2008 appropriation. DNDO conducted  the tests at NTS in two phases. DNDO’s Phase 3 Test Report Frequently Overlooks the Limitations Associated with the Tests’ Small Sample Sizes

Because the limitations of the Phase 3 test results are not properly  discussed in the Phase 3 test report, the report does not accurately portray  the results from the Phase 3 tests and could be misleading. The purpose of  the Phase 3 tests was to identify areas in which the ASPs needed  improvement. The narrative of the report often presents each test  result as a single value, although, because of the limited number of test runs,  the results would be more thoroughly and appropriately stated as a range of  potential values. In addition, the report’s narrative sometimes omits key  facts that conflict with DNDO’s analysis of the results. To allow for more substantial variation among test cases,  the test plan called for a smaller number of trials over a larger range of objects and  configurations rather than running sufficient number of repetitions for each  individual test case to provide higher statistically significan results.” (p. 2)
In these comparisons , results are  small sample sizes induce large uncertainties in the estimates of the probabilities  being compared [for example: n ≤ 5].” (p.9) “For  at 2 mph, the ASP system performances are  statistically indistinguishable.” (p.13)  “For shielded , performance for all three systems is  statistically indistinguishable with probabilities of correct alarm varying  approximately between 0.84 and .0.92.” (p.11)
The statements imply that the performances of the ASPs were similar  because the results were “statistically indistinguishable.” However, given  the small number of test runs, it is impossible to determine with a high  degree of confidence whether or not the performances were actually  similar. Phase 3 Test Results Provide Little Evidence as To Whether ASPs Represent an Improvement Over Currently Deployed Technology

In our view, it is not appropriate to use the Phase 3 test report in  determining whether the ASPs represent a significant improvement over  currently deployed radiation equipment because the limited number of test  runs does not support many of the comparisons of ASP performance made  in the Phase 3 report. This is true even if the ASP was successful  every time a test was conducted. For example, as noted in the Phase 3 test  report, if the ASP correctly identified a source material 100 percent of the  time, but the test was run only five times, the most DNDO can estimate is  that the ASP should be able to correctly identify the source no worse than  about 60 percent of the time. The Phase 3 test results do not help to determine an ASP’s “true” level of  performance because DNDO did not design the tests to assess ASP  performance with a high degree of statistical confidence. On this  point, the special test report acknowledges that “the special tests were  never intended to demonstrate conformity of the  systems against  specific performance requirements.”
From the special tests, SNL concluded  1. However, SNL does not comment on the biased testing methods we  identified during the Phase 1 ASP tests at the Nevada Test Site in 2007. Testing radiation detection equipment to understand its  capabilities and limitations is an important part of preventing nuclear  smuggling. The Phase 3 test results are relevant to DNDO’s original objective  for the Phase 3 tests—to identify areas in which the ASPs needed  improvement. Overall, because of the  limitations discussed in this report, it is our view that neither the Phase 3  tests nor the special tests should serve as a basis for the Secretary of DHS  whether the ASPs represent “a significant increase in operational  effectiveness” over current radiation detection equipment. However, if the Secretary of DHS intends to consider the results of the  Phase 3 tests, along with other test data information, in making a  certification decision regarding ASPs, then we recommend that the  Secretary take the following actions:    Direct the Director of DNDO to revise and clarify the Phase 3 test report to  more fully disclose and articulate the limitations present in the Phase 3  tests—particularly the limitations associated with making comparisons  between detection systems from a small number of test runs. Using an example from the Phase 3 report, if DNDO notes that  an ASP successfully identified a specific source material 34 percent of the  time during the tests, it should also indicate that, given the small number  of test runs, DNDO can only estimate that the ASP would be able to  correctly identify the specific source material between 10 and 65 percent  of the time. We continue to believe that the Phase 3 tests should be used only  for the intended purpose stated in its test plan—to improve the software of  ASPs.
Why GAO Did This Study

The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are part of the U.S. defense against such threats. In 2007, Congress required that funds for new advanced spectroscopic portal (ASP) monitors could not be spent until the Secretary of DHS certified that these machines represented a significant increase in operational effectiveness over currently deployed portal monitors. In addition to other tests, DNDO conducted the Phase 3 tests on ASPs to identify areas in which the ASPs needed improvement. GAO was asked to assess (1) the degree to which the Phase 3 test report accurately depicts the test results and (2) the appropriateness of using the Phase 3 test results to determine whether ASPs represent a significant improvement over current radiation detection equipment. GAO also agreed to provide its observations on special tests conducted by Sandia National Laboratories (SNL).

What GAO Found

Because the limitations of the Phase 3 test results are not appropriately stated in the Phase 3 test report, the report does not accurately depict the results from the tests and could potentially be misleading. In the Phase 3 tests, DNDO performed a limited number of test runs. Because of this, the test results provide little information about the actual performance capabilities of the ASPs. The report often presents each test result as a single value; but considering the limited number of test runs, the results would be more appropriately stated as a range of potential values. For example, the report narrative states in one instance that an ASP could identify a source material during a test 50 percent of the time. However, the narrative does not disclose that, given the limited number of test runs, DNDO can only estimate that the ASP would correctly identify the source from about 15 percent to about 85 percent of the time--a result that lacks the precision implied by DNDO's narrative. DNDO's reporting of the test results in this manner makes them appear more conclusive and precise than they really are. The purpose of the Phase 3 tests was to conduct a limited number of test runs in order to identify areas in which the ASP software needed improvement. While aspects of the Phase 3 report address this purpose, the preponderance of the report goes beyond the test's original purpose and makes comparisons of the performance of the ASPs with one another or with currently deployed portal monitors. In GAO's view, it is not appropriate to use the Phase 3 test report in determining whether the ASPs represent a significant improvement over currently deployed radiation equipment because the limited number of test runs do not support many of the comparisons of ASP performance made in the Phase 3 report. As the report shows, if an ASP can identify a source material every time during a test, but the test is run only five times, the only thing that can be inferred with a high level of statistical confidence is that the probability of identification is no less than about 60 percent. Although DNDO states in the Phase 3 test report that the results will be relevant to the Secretary's certification that the ASPs represent a significant increase in operational effectiveness, it does not clarify in what ways the results will be relevant. Furthermore, DNDO offers no explanation as to why it changed its view from the Phase 3 test plan, which states that these tests will not be used to support a certification decision. The goal of SNL's special tests was, among other things, to identify potential vulnerabilities in the ASPs by using different test scenarios from those that DNDO planned to use in other ASP tests. SNL concluded in its test report that the ASPs' software and hardware can be improved and that rigor could be added to DNDO's testing methods. Furthermore, the report acknowledges that (1) a specific objective of the testing at the Nevada Test Site was to refine and improve the ASP's performance and (2) the special tests were never intended to demonstrate conformity of the ASPs with specific performance requirements. In GAO's view, these statements appear to accurately describe the purpose, limitations, and results of the special tests.