Afterward, over several years, outside contractors’ laboratories developed  and validated several genetic tests to analyze the B. anthracis samples  for the presence of certain genetic mutations. This possible  explanation of genetic similarity between spores in the letters and  in RMR-1029 was not rigorously explored during the course of the  investigation, further complicating the interpretation of the  apparent association between the B. anthracis genotypes  discovered in the attack letters and those found in RMR-1029.”

	Genetic Tests Generally Were Verified and Validated but Lack of a Validation Framework Limited Statistical Confidence for Interpreting Results

We found that the genetic tests used to screen the FBI’s repository of B.  anthracis samples demonstrated through the verification and validation  testing that they generally met the FBI’s minimum validation  requirements. Validation Test Results Met the FBI Requirements but Did Not Demonstrate a Level of Statistical Confidence

We found that (1) the genetic tests used to screen the FBI’s repository of  B. anthracis samples met the FBI’s validation requirements, (2) the  validation tests were not required to and did not demonstrate a level of  statistical confidence for interpreting the validation test results, and (3)  some information on the sensitivity and specificity of the genetic tests was  not characterized until after validation (postvalidation testing). When we  compared the FBI’s statistical approach to these six characteristics, we  found that three could be improved to strengthen the significance of its  evidence for future investigations. We found that the FBI has taken some  steps to include such expertise in future investigations by building formal  forensic statistical expertise both internally and externally. The FBI’s Statistical Approach Could Have Been Improved for Three of the Six Characteristics of a Statistical Framework

We believe that the six general characteristics described above make up  a comprehensive statistical framework that could have allowed the FBI to  quantify significance and probative value of the scientific evidence  collected in a statistically meaningful way and could have strengthened  the evidence it collected. The FBI’s Research Did Not Provide Full Understanding of the Methods and Environmental Conditions That Give Rise to Genetic Mutations

Although the specific genetic mutations used as genetic markers to  determine a match or exclusion were adequately characterized, the FBI  did not conduct studies to understand the methods and environmental  conditions that gave rise to the mutations. By instituting more rigorous controls over sample  identification and collection for future investigations, the FBI can improve  the completeness and accuracy of a repository. In addition, we identified a key scientific gap that is related to  the verification and validation of the genetic tests and the statistical  analyses—that is, the significance of using genetic mutations in B.  anthracis as genetic markers for analyzing evidentiary samples. DHS has  funded some research on this gap but this research is not yet complete,  and it is not yet known whether it will fully address the gap. This gap is related to the  significance of using genetic mutations as genetic markers for analyzing  evidentiary samples to determine their origins. Such research is a step in the right direction, since the FBI has  indicated that it is likely to use genome sequencing methods in future  investigations to analyze evidence. The  framework should be applied at the outset of an investigation involving an  intentional release of B. anthracis, or any other microbial pathogen. Developing such a framework could also be facilitated by a written plan. Key contributors to  the report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

The scope of our work was limited to a review of the scientific methods  employed to validate the genetic tests used to screen the FBI’s repository  of Ames B. anthracis samples, the procedures used to identify and collect  samples of Ames B. anthracis in the creation of the FBI’s repository, and  the statistical analyses and interpretation of the results of the genetic  tests. We did not address any other scientific methods or any of the  traditional investigative techniques used to support the FBI’s conclusions  in this case, and we take no position on the FBI’s conclusions when it  closed its investigation in 2010. What are the characteristics of an adequate statistical approach for  analyzing the repository samples and to what extent was the  statistical approach used adequate? If not adequate, how could this  approach be improved for future efforts? To determine any remaining scientific concerns and uncertainties  regarding the validation of the genetic tests and statistical approaches  that would need to be addressed in future analyses, we reviewed relevant  federal agencies’ and their contractors’ documents, published literature,  and industry documentation on the validation of polymerase chain- reaction based tests, such as those for detecting rare variants, and  related scientific concerns and uncertainties that could affect a future  investigation.
Why GAO Did This Study

In 2001, the FBI investigated an intentional release of B. anthracis , a bacterium that causes anthrax, which was identified as the Ames strain. Subsequently, FBI contractors developed and validated several genetic tests to analyze B. anthracis samples for the presence of certain genetic mutations. The FBI had previously collected and maintained these samples in a repository.
GAO was asked to review the FBI's genetic test development process and statistical analyses. This report addresses (1) the extent to which these genetic tests were scientifically verified and validated; (2) the characteristics of an adequate statistical approach for analyzing samples, whether the approach used was adequate, and how it could be improved for future efforts; and (3) whether any remaining scientific concerns regarding the validation of genetic tests and statistical approaches need to be addressed for future analyses. GAO reviewed agency and contractor documentation, conducted literature reviews, and conducted statistical analyses of the repository data. GAO's review focused solely on two aspects of the FBI's scientific evidence: the validation of the genetic tests and the statistical approach for the analyses of the results. GAO did not review and is not taking a position on the conclusions the FBI reached when it closed its investigation in 2010.

What GAO Found

After the 2001 Anthrax attacks, the genetic tests that were conducted by the Federal Bureau of Investigation's (FBI) four contractors were generally scientifically verified and validated, and met the FBI's criteria. However, GAO found that the FBI lacked a comprehensive approach—or framework—that could have ensured standardization of the testing process. As a result, each of the contractors developed their tests differently, and one contractor did not conduct verification testing, a key step in determining whether a test will meet a user's requirements, such as for sensitivity or accuracy. Also, GAO found that the contractors did not develop the level of statistical confidence for interpreting the testing results for the validation tests they performed. Responses to future incidents could be improved by using a standardized framework for achieving minimum performance standards during verification and validation, and by incorporating statistical analyses when interpreting validation testing results.
GAO identified six characteristics of a statistical framework that can be applied for analyzing scientific evidence. When GAO compared the approach the FBI used to this framework, it found that that the FBI's approach could have been improved in three of six areas. First, the FBI's research did not provide a full understanding of the methods and conditions that give rise to genetic mutations used to differentiate between samples of B. anthracis . Second, the FBI did not institute rigorous controls over the sampling procedures it used to build the repository of B. anthracis samples. Third, the FBI did not include measures of uncertainty to strengthen the interpretation of the scientific evidence. GAO found that since 2001 the FBI has taken some steps to build formal forensic statistical expertise. The FBI's approach to future incidents could benefit from including such expertise early in an investigation.
The lack of an understanding of how bacteria change (mutate) in their natural environment and in a laboratory is a key scientific gap that remains and could affect testing conducted in future incidents. Specifically, the significance of using such mutations as genetic markers for analyzing evidentiary samples to determine their origins is not clear. This gap affects both the development of genetic tests targeting such mutations and statistical analyses of the results of their use on evidentiary samples. The Department of Homeland Security is currently funding some research on genetic changes in bacteria and genome sequencing methods, among others. Such research is a step in the right direction since the FBI is planning to use genome sequencing methods in future investigations. However, because this research may not be complete for several more years, the extent to which it will close this gap is not known.

What GAO Recommends

GAO recommends that the FBI develop a framework for validation and statistical approaches for future investigations. The FBI agreed with our recommendations.