Envisioned as a laboratory-in-a-box, the  autonomous detection system would automatically collect air samples,  conduct analysis to detect the presence of biothreat agents every 4 to 6  hours, and communicate the results to public health officials via an  electronic network without manual intervention. DHS’s April 24, 2014, ADM announced the  cancellation of the Gen-3 acquisition and made Gen-2 the official program  of record for aerosol biological threat detection. DHS Cannot Make Informed Decisions about Upgrades or Enhancements Because It Lacks Reliable Information about Gen-2’s Capability to Detect an Attack

DHS lacks reliable information about BioWatch Gen-2’s technical  capabilities to detect a biological attack and therefore lacks the basis for  informed cost-benefit decisions about possible upgrades or  enhancements to the system. However, these studies have not directly and comprehensively  assessed the capabilities of the Gen-2 system. DHS Has Not Developed Performance Requirements That Would Allow Conclusions about Gen-2’s Ability to Detect Attacks

DHS commissioned four key tests of Gen-2’s technical performance  characteristics, but has not developed and validated performance  requirements that would enable it to interpret the test results and draw  conclusions about the ability of an array of detectors in an operational  environment to detect attacks. In particular, they  said that these modeling and simulation studies support their assertion  that the Gen-2 system can detect catastrophic attacks, defined as attacks  large enough to cause 10,000 casualties. Limitations and Uncertainties in Test Results

We found limitations and uncertainties in the four key tests of the Gen-2  system’s performance characteristics—in particular, in the use of test  chambers instead of operational environments and the use of simulants in  place of live biothreat agents. As noted earlier, it is not possible to test the  BioWatch system directly by releasing live biothreat agents into the air in  operational environments. When comparing DHS’s actions and decisions regarding the planned  acquisition and testing of Gen-3, we found that DHS’s actions partially  aligned with these best practices. This led to the  cancellation of the Gen-3 acquisition in April 2014. Practice 6: Test to Build in Resilience, Especially in the Development Stages

DHS’s Actions Partially Aligned with Best Practice 6  DHS performed some testing that could help build resilience into the  system during Phase I testing, but it could improve resilience testing with  more rigorous methods. DHS took steps to refine the sensitivity requirement for Gen- 3, but it was the modeling study, rather than the developmental testing  protocol, that led to the change. 3. By applying the lessons learned that  DHS identified as a result of evaluating the Gen-3 acquisition, as well as  those identified through application of these best practices for testing,  DHS can continue to engage in a continuous cycle of improvement for its  testing and acquisition of detection technologies as it considers upgrades  or enhancements to the Gen-2 system. Further, an autonomous detection system would  have to address several likely challenges, including minimizing possible  false positives, securing a networked detection and communication  system, and operating under various environmental conditions. According to DHS officials, autonomous detection  is among the technologies being considered. Clean-up effort  reduction is thus uncertain because of an attack’s unpredictability. Recommendations for Executive Action

To help ensure that biosurveillance-related funding is directed to  programs that can demonstrate their intended capabilities, and to help  ensure sufficient information is known about the current Gen-2 system to  make informed cost-benefit decisions about possible upgrades and  enhancements to the system, the Secretary of Homeland Security should  direct the Assistant Secretary for Health Affairs and other relevant officials  within the Department to not pursue upgrades or enhancements to the  current BioWatch system until OHA: establishes technical performance requirements, including limits of  detection, necessary for a biodetection system to meet a clearly  defined operational objective for the BioWatch program by detecting  attacks of defined types and sizes with specified probabilities; assesses the Gen-2 system against these performance requirements  to reliably establish its capabilities; and produces a full accounting of statistical and other uncertainties and  limitations in what is known about the system’s capability to meet its  operational objectives. We stated that (1) the modeling and simulation studies did not  incorporate specific, best available test results (for example, particular  estimates of the system’s limits of detection) to draw specific conclusions  about the BioWatch Gen-2 system’s capability to detect attacks of defined  types and sizes, and (2) the modeling and simulation studies did not  incorporate uncertainties in the empirical test results that are important for  understanding the precision or confidence in the modeling and simulation  results. If you or your staff members have any questions about this report, please  contact Tim Persons at (202) 512-6412 or personst@gao.gov or Chris  Currie at (404) 679-1875 or curriec@gao.gov. GAO staff who made key contributions to this  report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

The objectives of this report were to discuss: (1) the extent to which the  Department of Homeland Security (DHS) has assessed the technical  capability of the currently deployed system (Gen-2) to detect a biological  attack; (2) the extent to which DHS adhered to best practices for  developmental testing during Gen-3 Phase I, and what lessons can be  learned; and (3) the most mature technology for an autonomous detection  system, as well as what the potential benefits and likely challenges would  be if DHS were to pursue an autonomous detection system for the  BioWatch program in the near future.
Why GAO Did This Study

DHS's BioWatch program aims to provide early indication of an aerosolized biological weapon attack. Until April 2014, DHS pursued a next-generation autonomous detection technology (Gen-3), which aimed to enable collection and analysis of air samples in less than 6 hours, unlike the current system (Gen-2), which requires manual intervention and can take up to 36 hours to detect the presence of biological pathogens. DHS is taking steps to address the capability gap that resulted from the cancellation of Gen-3 by exploring other technology upgrades and improvements to the Gen-2 system.
GAO was asked to review (1) the technical capabilities of the currently deployed BioWatch system, (2) the Gen-3 testing effort, and (3) characteristics of autonomous detection as a possible option to replace the current BioWatch system. GAO analyzed key program documents, including test plans, test results, and modeling studies. GAO assessed Gen-3 testing against best practices, reviewed relevant literature, and discussed the BioWatch program and testing efforts with key agency officials and national laboratories staff.

What GAO Found

The Department of Homeland Security (DHS) lacks reliable information about BioWatch Gen-2's technical capabilities to detect a biological attack and therefore lacks the basis for informed cost-benefit decisions about upgrades to the system. DHS commissioned several tests of the technical performance characteristics of the current BioWatch Gen-2 system, but has not developed performance requirements that would enable it to interpret the test results and draw conclusions about the system's ability to detect attacks. Although DHS officials said that the system can detect catastrophic attacks, which they define as attacks large enough to cause 10,000 casualties, they have not specified the performance requirements necessary to reliably meet this operational objective. In the absence of performance requirements, DHS officials said computer modeling and simulation studies support their assertion. However, none of these studies were designed to incorporate test results from the Gen-2 system and comprehensively assess the system against the stated operational objective. Additionally, DHS has not prepared an analysis that combines the modeling and simulation studies with the specific Gen-2 test results to assess the system's capabilities to detect attacks. Finally, we found limitations and uncertainties in the four key tests of the Gen-2 system's performance. Because it is not possible to test the BioWatch system directly by releasing live biothreat agents into the air in operational environments, DHS relied on chamber testing and the use of simulated biothreat agents, which limit the applicability of the results. These limitations underscore the need for a full accounting of statistical and other uncertainties, without which decision makers lack a full understanding of the Gen-2 system's capability to detect attacks of defined types and sizes and cannot make informed decisions about the value of proposed upgrades.
The actions and decisions DHS made regarding the acquisition and testing of a proposed next generation of BioWatch (Gen-3) partially aligned with best practices GAO previously identified for developmental testing of threat detection systems. For example, best practices indicate that resilience testing, or testing for vulnerabilities, can help uncover problems early. While DHS took steps to help build resilience into the Gen-3 testing, future testing could be improved by using more rigorous methods to help predict performance in different operational environments. DHS canceled the Gen-3 acquisition in April 2014, but GAO identified lessons DHS could learn by applying these best practices to the proposed Gen-2 upgrades.
According to experts and practitioners, the polymerase chain reaction (PCR), which detects genetic signatures of biothreat agents, is the most mature technology to use for an autonomous detection system. DHS is considering autonomous detection as an upgrade to Gen-2, because according to DHS, it may provide benefits such as reduction in casualties or clean-up costs. But the extent of these benefits is uncertain because of several assumptions, such as the speed of response after a detection, that are largely outside of DHS's control. As a result, the effectiveness of the responseâ€”and the number of lives that could be savedâ€”is uncertain. Further, an autonomous detection system must address several likely challenges, including minimizing possible false positive readings, meeting sensitivity requirements, and securing information technology networks.

What GAO Recommends

GAO recommends DHS not pursue upgrades or enhancements for Gen-2 until it reliably establishes the system's current capabilities. GAO also recommends DHS incorporate best practices for testing in conducting any system upgrades. DHS generally concurred with GAO's recommendations.
or Chris Currie at (404) 679-1875 or curriec@gao.gov .