	Improved Testing Rigor Discussed in Our May 2009 Report Demonstrates Limitations of ASPs

Our May 2009 report on the then-current round of ASP testing found that  DHS increased the rigor of ASP testing over that of previous tests, and that  a particular area of improvement was in the performance testing at the  Nevada Test Site, where DNDO compared the capability of ASP and  current-generation equipment to detect and identify nuclear and  radiological materials. Nevertheless, based on the  following factors, in our report we questioned whether the benefits of the  new portal monitors justify the high cost:    The DHS criteria for a significant increase in operational effectiveness. Our chief concern with the criteria is that they require only a marginal  improvement over current-generation portal monitors in the detection of  certain weapons-usable nuclear materials during primary screening. DNDO considers detection of such materials to be a key limitation of  current-generation portal monitors. DNDO officials expect they can achieve small improvements in  sensitivity through energy windowing, but DNDO has not yet completed  efforts to fine-tune the PVTs’ software. The results of performance testing at the Nevada Test  Site showed that the new portal monitors detected certain nuclear  materials better than PVTs when shielding approximated DOE threat  guidance, which is based on light shielding. DNDO’s plans for computer simulations. As of May 2009, DNDO did not  plan to complete injection studies—computer simulations for testing the  response of ASPs and PVTs to simulated threat objects concealed in cargo  containers—prior to the Secretary of Homeland Security’s decision on  certification even though delays to the ASP test schedule have allowed  more time to conduct the studies. DNDO had not updated its cost- benefit analysis to take into account the results of ASP testing. An updated  analysis that takes into account the testing results, including injection  studies, might show that DNDO’s plan to replace existing equipment with  ASPs is not justified, particularly given the marginal improvement in  detection of certain nuclear materials required of ASPs and the potential  to improve the current-generation portal monitors’ sensitivity to nuclear  materials, most likely at a lower cost. Our May report recommended that the Secretary of Homeland Security  direct DNDO to (1) assess whether ASPs meet the criteria for a significant  increase in operational effectiveness based on a valid comparison with  PVTs’ full performance potential and (2) revise the schedule for ASP  testing and certification to allow sufficient time for review and analysis of  results from the final phases of testing and completion of all tests,  including injection studies. We further recommended that, if ASPs are  certified, the Secretary direct DNDO to develop an initial deployment plan  that allows CBP to uncover and resolve any additional problems not  identified through testing before proceeding to full-scale deployment. DHS  agreed to a phased deployment that should allow time to uncover ASP  problems but disagreed with GAO’s other recommendations, which we  continue to believe remain valid. Results from July 2009 Testing Raise Continuing Issues

The results of DNDO’s most recent round of field validation testing, which  it undertook in July 2009, after our May report was released, raise new  issues. In July 2009, DNDO resumed the field testing of ASPs at four CBP  ports of entry that it initiated in January 2009 but suspended because of  serious performance problems. However, the July tests also revealed ASP  performance problems, including two critical performance deficiencies. First, the ASP monitors had an unacceptably high number of false positive  alarms for the detection of certain high-risk nuclear materials. According  to CBP officials, these false alarms are very disruptive in a port  environment in that any alarm for this type of nuclear material would  cause CBP to take enhanced security precautions because such materials  (1) could be used in producing an improvised nuclear device and (2) are  rarely part of legitimate or routine cargo. In addition to these key performance problems, the ASP was not able to  reduce referrals to secondary inspection by 80 percent as required by the  DHS criteria for a significant increase in operational effectiveness. Specifically, to address the problem of false positive alarms indicating the  presence of certain nuclear materials, according to DNDO officials, DNDO  has modified the ASP to make this equipment less sensitive to these  nuclear materials. As we reported earlier this  year, the results of the testing at the Nevada Test Site demonstrated that  the ASPs represented a marginal improvement in detecting certain nuclear  materials. By reducing the sensitivity to these materials and not retesting  the modified ASPs against actual nuclear materials, it is uncertain exactly  what improvement in detecting certain nuclear materials these costly  portal monitors are providing. In our view, ASPs being modified to diminish  their capabilities to detect certain nuclear materials raises questions about  whether energy windowing might be able to achieve a similar level of  performance against these same materials from the PVTs that are already  in place.
Why GAO Did This Study

The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are key elements in the nation's defenses against such threats. DHS has sponsored testing to develop new monitors, known as advanced spectroscopic portal (ASP) monitors, to replace radiation detection equipment being used at ports of entry. DNDO expects that ASPs may offer improvements over current-generation portal monitors, particularly the potential to identify as well as detect radioactive material and thereby to reduce both the risk of missed threats and the rate of innocent alarms, which DNDO considers to be key limitations of radiation detection equipment currently used by Customs and Border Protection (CBP) at U.S. ports of entry. However, ASPs cost significantly more than current generation portal monitors. Due to concerns about ASPs' cost and performance, Congress has required that the Secretary of Homeland Security certify that ASPs provide a significant increase in operational effectiveness before obligating funds for full-scale ASP procurement. In May 2009, GAO issued a report (GAO-09-655) on the status of the ongoing ASP testing round. This testimony (1) discusses the principal findings and recommendations from GAO's May report on ASP testing and (2) updates those findings based on information from DNDO and CBP officials on the results of testing conducted since the report's issuance. DHS, DNDO, and CBP's oral comments on GAO's new findings were included as appropriate.

What GAO Found

GAO's May 2009 report on ASP testing found that DHS increased the rigor in comparison with previous tests and thereby added credibility to the test results. However, GAO's report also questioned whether the benefits of the ASPs justify its high cost. In particular, the DHS criteria for a significant increase in operational effectiveness require only a marginal improvement in the detection of certain weapons-usable nuclear materials, which DNDO considers a key limitation of current-generation portal monitors. The marginal improvement required of ASPs is particularly notable given that DNDO has not completed efforts to fine-tune current-generation equipment to provide greater sensitivity. Moreover, the test results showed that ASPs performed better than current-generation portal monitors in detection of such materials concealed by light shielding approximating the threat guidance for setting detection thresholds, but that differences in sensitivity were less notable when shielding was slightly below or above that level. Finally, DNDO had not yet updated its cost-benefit analysis to take into account the results of ASP testing and did not plan to complete computer simulations that could provide additional insight into ASP capabilities and limitations prior to certification even though test delays have allowed more time to conduct the simulations. DNDO officials believed the other tests were sufficient for ASPs to demonstrate a significant increase in operational effectiveness. GAO recommended that DHS assess ASPs against the full potential of current-generation equipment and revise the program schedule to allow time to conduct computer simulations and to uncover and resolve problems with ASPs before full-scale deployment. DHS agreed to a phased deployment that should allow time to uncover ASP problems but disagreed with the other recommendations, which GAO believes remain valid. The results of DNDO's most recent round of field testing raise continuing issues. In July 2009, DNDO resumed the field testing of ASPs that it initiated in January 2009 but suspended because of serious performance problems. However, the July tests also revealed critical performance deficiencies. For example, the ASP had a high number of false positive alarms for the detection of certain nuclear materials. According to CBP, these false alarms are very disruptive in a port environment because any alarm for this type of nuclear material causes CBP to take enhanced security precautions. To address these false alarms, DNDO plans to modify the ASP to make these monitors less sensitive to these nuclear materials and thereby diminishing the ASPs' capability. As GAO reported earlier this year, previous testing results demonstrated that the ASPs represented a marginal improvement in detecting these materials. By reducing the sensitivity to nuclear materials even further, it is uncertain exactly what improvement in detecting these materials the ASPs are providing or whether DNDO might be able to achieve a similar level of performance as the modified ASPs by improving the current-generation portal monitors that are already in place. In addition, the July 2009 testing also identified a critical equipment failure, including an alert malfunction, which DNDO is taking steps to resolve for future testing.