	The Latest Round of Testing Highlights the Limitations of ASPs

Our report on the latest round of ASP testing found that DHS increased the  rigor of ASP testing in comparison with previous tests and that a particular  area of improvement was in the performance testing at the Nevada Test  Site, where DNDO compared the capability of ASP and current-generation  equipment to detect and identify nuclear and radiological materials. Nevertheless, based on the following factors, we continue to question  whether the benefits of the new portal monitors justify the high cost:    The DHS criteria for a significant increase in operational effectiveness. DNDO considers detection of such materials to be a  key limitation of current-generation portal monitors. The marginal improvement in detection of such materials  required of ASPs is particularly notable given that DNDO has not  completed efforts to fine-tune PVTs’ software and thereby improve  sensitivity to nuclear materials. DNDO has not yet updated its  cost-benefit analysis to take into account the results of the latest round of  ASP testing. An updated analysis that takes into account the results from  the latest round of testing, including injection studies, might show that  DNDO’s plan to replace existing equipment with ASPs is not justified,  particularly given the marginal improvement in detection of certain  nuclear materials required of ASPs and the potential to improve the  current-generation portal monitors’ sensitivity to nuclear materials, most  likely at a lower cost. DHS  agreed to a phased deployment that should allow time to uncover ASP  problems but disagreed with GAO’s other recommendations, which we  continue to believe remain valid. Procurement Decisions for New Technologies Require Rigorous Testing and Thorough Analysis of Results

The challenges DNDO has faced in developing and testing ASPs illustrate  the importance of following existing DHS policies as well as best practices  for investments in complex homeland security acquisitions and for testing  of new technologies. In addition,  many major investments, including DNDO’s ASP program, had not met the  department’s requirements for basic acquisition documents necessary to  inform the investment review process. As a result, DHS had not  consistently provided the oversight needed to identify and address cost,  schedule, and performance problems in its major investments. We found that  testing programs designed to validate a product’s performance against  increasing standards for different stages in product development are a best  practice for acquisition strategies for new technologies. In the case of  ASPs, the push to replace existing equipment with the new portal monitors  led to a testing program that until recently lacked the necessary rigor. Even for the most recent round of testing, DNDO’s schedule consistently  underestimated the time required to conduct tests, resolve problems  uncovered during testing, and complete key documents, including final  test reports. Appendix I: Key Findings and Recommendations from Related GAO Products on Testing and Development of ASPs

Combating Nuclear Smuggling: DHS Has Made Progress  Deploying Radiation Detection Equipment at U.S. Ports-of-Entry,  but Concerns Remain. Prototypes of advanced spectroscopic portals (ASP) were  expected to be significantly more expensive than current-generation portal  monitors but had not been shown to be more effective. GAO-09-655. With regard to ASP  testing, we found that DHS increased the rigor in comparison with  previous tests, thereby adding credibility to the test results, but that  preliminary results were mixed. The results showed that the new portal  monitors performed better than current-generation portal monitors in  detection of certain nuclear materials concealed by light shielding  approximating the threat guidance for setting detection thresholds, but  differences in sensitivity were less notable when shielding was slightly  below or above that level. Finally, we found that DNDO did not plan to complete  computer simulations that could provide additional insight into ASP  capabilities and limitations prior to certification even though delays to  testing allowed more time to conduct the simulations. We recommended that the Secretary of Homeland  Security direct the Director of DNDO to assess whether ASPs meet the  criteria for a significant increase in operational effectiveness based on a  valid comparison with current-generation portal monitors’ full  performance potential and revise the schedule for ASP testing and  certification to allow sufficient time for review and analysis of results from  the final phases of testing and completion of all tests, including computer  simulations.
Why GAO Did This Study

The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are key elements in the nation's defenses against such threats. DHS has sponsored testing to develop new monitors, known as advanced spectroscopic portal (ASP) monitors, to replace radiation detection equipment being used at ports of entry. DNDO expects that ASPs may offer improvements over current-generation portal monitors, particularly the potential to identify as well as detect radioactive material and thereby to reduce both the risk of missed threats and the rate of innocent alarms, which DNDO considers to be key limitations of radiation detection equipment currently used by Customs and Border Protection (CBP) at U.S. ports of entry. However, ASPs cost significantly more than current generation portal monitors. Due to concerns about ASPs' cost and performance, Congress has required that the Secretary of Homeland Security certify that ASPs provide a significant increase in operational effectiveness before obligating funds for full-scale ASP procurement. This testimony addresses (1) GAO findings on DNDO's latest round of ASP testing, and (2) lessons from ASP testing that can be applied to other DHS technology investments. These findings are based on GAO's May 2009 report GAO-09-655 and other related reports.

What GAO Found

GAO's report on the latest round of ASP testing found that DHS increased the rigor in comparison with previous tests and thereby added credibility to the test results. However, GAO's report also questioned whether the benefits of the ASPs justify the high cost. In particular, the DHS criteria for a significant increase in operational effectiveness require only a marginal improvement in the detection of certain weapons-usable nuclear materials, which DNDO considers a key limitation of current-generation portal monitors. The marginal improvement required of ASPs is particularly notable given that DNDO has not completed efforts to fine-tune current-generation equipment to provide greater sensitivity. Moreover, the preliminary test results show that ASPs performed better than current-generation portal monitors in detection of such materials concealed by light shielding approximating the threat guidance for setting detection thresholds, but that differences in sensitivity were less notable when shielding was slightly below or above that level. Finally, DNDO has not yet updated its cost-benefit analysis to take into account the results of the latest round of ASP testing and does not plan to complete computer simulations that could provide additional insight into ASP capabilities and limitations prior to certification even though test delays have allowed more time to conduct the simulations. DNDO officials believe the other tests are sufficient for ASPs to demonstrate a significant increase in operational effectiveness. GAO recommended that DHS assess ASPs against the full potential of current-generation equipment and revise the program schedule to allow time to conduct computer simulations and to uncover and resolve problems with ASPs before full-scale deployment. DHS agreed to a phased deployment that should allow time to uncover ASP problems but disagreed with the other recommendations, which GAO believes remain valid. The challenges DNDO has faced in developing and testing ASPs illustrate the importance of following best practices for investments in complex homeland security acquisitions and for testing of new technologies. GAO recently found that many major DHS investments, including DNDO's ASP program, had not met the department's requirements for basic acquisition documents necessary to inform the investment review process, which has adopted many acquisition best practices. As a result, DHS had not consistently provided the oversight needed to identify and address cost, schedule, and performance problems in its major investments. A primary lesson to be learned regarding testing is that the push to replace existing equipment with the new portal monitors led to an ASP testing program that until recently lacked the necessary rigor. Even for the most recent round of testing, DNDO's schedule consistently underestimated the time required to conduct tests and resolve problems uncovered during testing. In contrast, GAO has previously found that testing programs designed to validate a product's performance against increasing standards for different stages in product development are a best practice for acquisition strategies for new technologies. Aspects that improved the latest round of ASP testing could also, if properly implemented, provide rigor to DHS's testing of other advanced technologies.