The performance tests determined  the probability of detection and identification of radiation sources,  including special nuclear material that could be used to make a nuclear  weapon. CBP made  three attempts to complete this testing—in January and February 2009,  July and August 2009, and October and November 2010. Test Results Show That ASP Did Not Meet Criteria to Pass Field Validation Testing, Leading DHS to Cancel the Program

Because of unsatisfactory test results, ASP did not pass field validation  testing, which led DHS to cancel the program. The  Acting Director also stated that the cost to replace current RPMs with  ASP for secondary screening was relatively low—approximately $350  million—compared with the original total program cost of $2 billion to $3  billion to use ASP for primary and secondary screening. ASP did not meet these criteria in part because of its inability to turn on at  the beginning of each day and to continue operating long enough to  complete a full day of testing. In addition, when ASP was operating, it  produced an excessive number of alarms that incorrectly indicated  naturally occurring radioactive material in the cargo might be masking  special nuclear material. Accordingly, it is unclear  why such speeds cannot be achieved at CBP land border crossings and  seaports but can be achieved at truck weigh stations. Lessons Learned Reviews Improve Future Acquisition Efforts, but DHS Does Not Have Processes Ensuring Such Reviews

Conducting lessons learned reviews when programs are cancelled  benefits organizations by identifying things that worked well and did not  work well in order to improve future acquisitions programs, according to  experts we consulted; however, DHS does not have processes in place to  ensure such reviews are conducted and reports documenting the results  of the review are disseminated. When we asked these experts for their views on lessons  learned reviews and reports for cancelled acquisition programs, they  generally agreed on the following observations:
Lessons learned reviews help to determine the reasons why programs  were cancelled including problems with the technology, management,  or setting requirements. Similar to the  views of the experts identified by the National Academies, the DHS  acquisition guidance states that (1) the objective of the reviews is to share  lessons learned throughout the department to increase the probability of  success for future acquisition programs and (2) the reviews are to take  place immediately after programs are cancelled. In the case of the ASP program, a lessons learned review was  conducted, and a lessons learned report was submitted to PARM and  disseminated within CBP and DNDO in November 2012 at the explicit  direction of DHS’s Under Secretary for Management. However, before the Under Secretary’s directive, there  was confusion about whether such a review should be conducted for  cancelled programs, as well as uncertainty about whether the program  had been cancelled. DNDO officials told us in March 2012 that, even  though they considered the program to have been cancelled by the  Secretary in October 2011, they did not intend to conduct a lessons  learned review because such reviews were not needed for cancelled  programs. Another lesson  learned from the ASP program is that the acquisition officials and the end  users must work closely together to ensure the needed capability meets  operational requirements. However, DNDO prepared the lessons learned  report and disseminated it 2 years after the final ASP testing, and about a  year after the Secretary of Homeland Security notified Congress of her  decision to cancel the program. The amount of time between the final field validation testing  of the ASP, the program’s cancellation, and the conducting of a lessons  learned review and dissemination of a lessons learned report suggests  that timely identification of lessons learned was not a DHS priority. DHS’s comments also refer to additional guidance called the  Department of Homeland Security (DHS) Capital Planning and  Investment Control (CPIC) Guide. These experts were identified on the basis on their knowledge of leading  practices in large-scale engineering and acquisition programs. Nuclear Detection: Domestic Nuclear Detection Office Should Improve  Planning to Better Address Gaps and Vulnerabilities. Combating Nuclear Smuggling: DHS’s Decision to Procure and Deploy  the Next Generation of Radiation Detection Equipment Is Not Supported  by Its Cost-Benefit Analysis. Combating Nuclear Smuggling: DNDO Has Not Yet Collected Most of the  National Laboratories’ Test Results on Radiation Portal Monitors in  Support of DNDO’s Testing and Development Program.
Why GAO Did This Study

Preventing terrorists from smuggling radiological or nuclear material into the United States to carry out an attack is a national priority. DHS's DNDO develops and deploys radiation detection equipment to assist other federal agencies, such as CBP, in intercepting illicit radiological or nuclear materials that could be used to make a radiological dispersive device (dirty bomb) or a crude nuclear bomb. CBP uses RPMs at nearly all land border crossings and seaports to detect radiation in trucks and cargo. DHS recently cancelled acquisition of ASPs, which was originally envisioned as costing from $2 billion to $3 billion.
GAO was asked to provide updated information on the ASP program. This report examines, among other things, (1) the results of ASP testing conducted in 2009 and 2010 that led to DHS's decision to cancel the ASP program and (2) the benefits of lessons learned reviews and how DHS captures any lessons learned when programs are cancelled. GAO reviewed testing and acquisition documents and interviewed key agency officials, as well as seven experts the National Academies identified for their knowledge of leading practices in large-scale engineering and acquisition programs.

What GAO Found

The advanced spectroscopic portal monitor (ASP)--a next-generation radiation portal monitor (RPM) for screening trucks and cargo containers--did not pass field validation tests conducted in 2009 and 2010. The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) intended to replace many currently deployed RPMs and handheld radiation detectors used by U.S. Customs and Border Protection (CBP) with ASPs. However, in the tests, ASP did not meet key requirements to detect radiation and identify its source. For example, ASP triggered too many false alarms from benign, naturally occurring radioactive material in common items such as kitty litter and granite, and it sometimes would not turn on or continue operating long enough to complete a day of testing. In addition, GAO's review identified analytical weaknesses related to the testing and program cancellation, including inconsistencies in DNDO's analysis of the settings used for testing the ASP. The final field validation test was conducted in November 2010, and the Secretary of Homeland Security notified Congress of her decision to cancel the program in October 2011.
Conducting lessons learned reviews when programs are cancelled benefits organizations by identifying things that worked well and did not work well in order to improve future acquisitions programs, according to experts GAO consulted. However, DHS does not have processes in place to ensure such reviews are conducted or that the results are disseminated. Experts identified by the National Academies told GAO that lessons learned reviews help identify reasons why programs were cancelled. The experts also said lessons learned reviews should be required and conducted promptly, and the results should be disseminated. At the direction of DHS management, DNDO reviewed the ASP program and submitted and disseminated a lessons learned report in November 2012. This report cited 32 lessons learned including having program officials work closely with end users to ensure equipment meets operational requirements. DHS guidance calls for lessons learned reviews immediately after programs are cancelled and states that the lessons learned are to be shared throughout the department, but this guidance is not a requirement. Before DHS's directive, there was confusion about whether a lessons learned review was needed for the ASP program, and DNDO officials did not intend to conduct such a review. Moreover, DHS officials were unable to provide examples of previous lessons learned reports from other cancelled programs. DHS officials also said they have no process for disseminating such reports but are planning one.

What GAO Recommends

DHS should require lessons learned reviews and develop processes to ensure such reviews are done in a timely manner and the results disseminated throughout the department. DHS agreed with all of GAO's recommendations and has planned and taken some actions to address them.