	DHS’s Criteria for Significant Increase in Operational Effectiveness Require a Marginal Improvement in the Detection of Certain Nuclear Materials and a Large Reduction in Innocent Alarms

Although the DHS criteria for primary screening require an improved  ability to detect certain nuclear materials at operational thresholds, ASPs  could meet the criteria for improvement while still failing to detect  anything more than lightly shielded material. DNDO officials acknowledge  that passive radiation detection equipment, which includes both the new  and current-generation portal monitors, is capable of detecting certain  nuclear materials only when this material is unshielded or lightly  shielded. For this reason, the DOE threat guidance used to set PVTs’  detection threshold is based on the equipment’s limited sensitivity to  anything more than lightly shielded nuclear material rather than on the  assumption that smugglers would take effective shielding measures. Other aspects of the criteria for a significant increase in operational  effectiveness require that ASPs either provide more than a marginal  improvement in addressing other limitations of current-generation  equipment or at least maintain the same level of performance in areas in  which the current-generation equipment is considered adequate:    The primary screening requirement for an 80 percent reduction in the rate  of innocent alarms could result in hundreds of fewer secondary screenings  per day, thereby reducing CBP’s workload and delays to commerce. However, the testing still had  limitations, and the preliminary results are mixed. Improvements to Testing Provided Credibility to Test Results

As we testified in September 2008, DHS’s improvements to the 2008 round  of ASP testing addressed concerns we raised about previous tests. In contrast, the plan for the 2008 performance test  stipulated that there would be no system contractor involvement in test  execution, and no ASP contractors were at the test location on the day we  observed performance testing. The performance test results cannot be generalized beyond the limited set  of scenarios tested. Other phases of testing, particularly integration testing, uncovered multiple  problems meeting requirements for successfully integrating the new  technology into operations at ports of entry. CBP officials anticipate that they will continue to uncover problems  during the first few years of use if the new technology is deployed in the  field. Schedule Delays Have Allowed More Time for Analysis and Review of Test Results, but DNDO’s Latest Schedule Does Not Include Computer Simulations to Provide Additional Insight into ASP Capabilities

Delays to the schedule for the 2008 round of testing have allowed more  time for analysis and review of results, particularly from performance  testing conducted at the Nevada Test Site. DHS’s most recent schedule  anticipated a decision on ASP certification as early as May 2009, but DHS  has not updated its schedule for testing and certification since suspending  field validation in February 2009 due to ASP performance problems. Problems uncovered during testing of ASPs’ readiness to be integrated  into operations at U.S. ports of entry have caused the greatest delays to  date and have allowed more time for DNDO to analyze and review the  results of performance testing. However, DNDO does not plan to complete  the studies prior to the Secretary of Homeland Security’s decision on  certification even though DNDO and other officials have indicated that the  studies could provide additional insight into the capabilities and  limitations of advanced portal monitors. DNDO has the data needed to conduct the studies. Preliminary results of DNDO’s  performance testing show that ASPs outperformed the PVTs in detection  of such materials during runs with light shielding, but ASPs’ performance  rapidly deteriorated once shielding was slightly increased. First, the DHS criteria for a significant increase in operational  effectiveness do not take into account recent efforts to improve the  current-generation portal monitors’ sensitivity to nuclear materials  through the “energy windowing” technique, most likely at a much lower  cost. Appendix I: Scope and Methodology

To evaluate the degree to which Department of Homeland Security’s  (DHS) criteria for a significant increase in operational effectiveness  address the limitations of the current generation of radiation detection  equipment, we clarified the intent of the criteria through the Domestic  Nuclear Detection Office’s (DNDO) written answers to our questions and  through interviews with U.S. Customs and Border Protection (CBP)  officials.
Why GAO Did This Study

The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is testing new advanced spectroscopic portal (ASP) radiation detection monitors. DNDO expects ASPs to reduce both the risk of missed threats and the rate of innocent alarms, which DNDO considers to be key limitations of radiation detection equipment currently used by Customs and Border Protection (CBP) at U.S. ports of entry. Congress has required that the Secretary of DHS certify that ASPs provide a significant increase in operational effectiveness before obligating funds for full-scale procurement. GAO was asked to review (1) the degree to which DHS's criteria for a significant increase in operational effectiveness address the limitations of existing radiation detection equipment, (2) the rigor of ASP testing and preliminary test results, and (3) the ASP test schedule. GAO reviewed the DHS criteria, analyzed test plans, and interviewed DHS officials.

What GAO Found

The DHS criteria for a significant increase in operational effectiveness require a minimal improvement in the detection of threats and a large reduction in innocent alarms. Specifically, the criteria require a marginal improvement in the detection of certain weapons-usable nuclear materials, considered to be a key limitation of current-generation portal monitors. The criteria require improved performance over the current detection threshold, which for certain nuclear materials is based on the equipment's limited sensitivity to anything more than lightly shielded materials, but do not specify a level of shielding that smugglers could realistically use. In addition, DNDO has not completed efforts to improve current-generation portal monitors' performance. As a result, the criteria do not take the current equipment's full potential into account. With regard to innocent alarms, the other key limitation of current equipment, meeting the criteria could result in hundreds fewer innocent alarms per day, thereby reducing CBP's workload and delays to commerce. DHS increased the rigor of ASP testing in comparison with previous tests. For example, DNDO mitigated the potential for bias in performance testing (a concern GAO raised about prior testing) by stipulating that there would be no ASP contractor involvement in test execution. Such improvements added credibility to the test results. However, the testing still had limitations, such as a limited set of scenarios used in performance testing to conceal test objects from detection. Moreover, the preliminary results are mixed. The results show that the new portal monitors have a limited ability to detect certain nuclear materials at anything more than light shielding levels: ASPs performed better than current-generation portal monitors in detection of such materials concealed by light shielding approximating the threat guidance for setting detection thresholds, but differences in sensitivity were less notable when shielding was slightly below or above that level. Testing also uncovered multiple problems in ASPs meeting the requirements for successful integration into operations at ports of entry. CBP officials anticipate that, if ASPs are certified, new problems will appear during the first few years of deployment in the field. While DNDO's schedule underestimated the time needed for ASP testing, test delays have allowed more time for review and analysis of results. DNDO's original schedule anticipated completion in September 2008. Problems uncovered during testing of ASPs' readiness to be integrated into operations at U.S. ports of entry caused the greatest delays to this schedule. DHS's most recent schedule anticipated a decision on ASP certification as early as May 2009, but DHS recently suspended field validation due to ASP performance problems and has not updated its schedule for testing and certification. In any case, DNDO does not plan to complete computer simulations that could provide additional insight into ASP capabilities and limitations prior to certification even though delays have allowed more time to conduct the simulations. DNDO officials believe the other tests are sufficient for ASPs to demonstrate a significant increase in operational effectiveness.