Behavior Detection and Analysis: TSA’s BDA program employs  behavior detection officers (BDO) at passenger screening checkpoints to  identify potential threats by observing individuals for certain behavioral  indicators—behaviors indicative of stress, fear, or deception. TSA Has Effectiveness Data on Some Countermeasures That Show Mixed Results, But Does Not Measure Deterrence

		Data on the Effectiveness of Selected Countermeasures in Detecting and Disrupting Threats to Aviation Security Vary in Extent and Reliability

TSA collected fiscal year 2015 data on the effectiveness of four of the six  countermeasures we selected—passenger prescreening, checkpoint  screening, checked baggage screening, and explosives detection  canines—in detecting or disrupting threats to passenger aviation  security. For  FAMS, TSA officials explained that it is very difficult to empirically  measure the effectiveness of federal air marshals and the program has  no efforts underway to collect such data. This official further stated that developing a method to assess deterrence  for this purpose would be challenging but feasible. TSA Can Compare the Effectiveness of Certain Combinations of Aviation Security Countermeasures, but Does Not Systematically Analyze Cost and Effectiveness Tradeoffs Across All Countermeasures TSA Has a Tool to Assess the Security Effectiveness of Alternate Combinations of Some Countermeasures

In 2014, TSA’s ORCA began using a Risk and Trade Space Portfolio  Analysis Tool (RTSPA) to analyze the security effectiveness of alternate  combinations of some aviation security countermeasures for the purpose  of informing TSA acquisition and deployment decisions. TSA Has Not Systematically Analyzed Potential Cost and Effectiveness Tradeoffs Across the Entire System of Aviation Security Countermeasures

TSA does not have any efforts underway to systematically evaluate the  potential cost and effectiveness tradeoffs across the full aviation security  system. DHS policy and TSA’s strategic plan call for the systematic evaluation of  the costs and effectiveness of TSA’s chosen mix of aviation security  countermeasures. Conclusions

Since the terrorist attacks of September 11, 2001, TSA has spent billions  of dollars on a range of aviation security programs with the goal of  detecting, disrupting, and deterring threats. Specifically, TSA has some information on how well it can  detect and disrupt threats and is taking steps to improve this information,  but does not have information on its ability to deter attacks—a key  component of TSA’s goal. For example, in fiscal year 2015, TSA spent  approximately $800 million on FAMS—a program with a focus on  deterring attacks on aircraft—yet the agency has no information on its  effectiveness in doing so. Specifically, TSA does not have any efforts underway to evaluate the  potential cost and effectiveness tradeoffs across the full aviation security  system because, according to a senior TSA official, the aviation security  system is constantly evolving in response to emerging threats, and  assessing a system in flux is challenging. The Administrator of TSA should explore and pursue methods to  assess the deterrent effect of TSA’s passenger aviation security  countermeasures; such an effort should identify FAMS—a  countermeasure with a focus on deterring threats—as a top priority to  address. (Recommendation 1)  2. With regard to our first recommendation that TSA explore and pursue  methods to assess the deterrent effect of its passenger aviation security  countermeasures, DHS concurred, noting that this may require proxy or  output measures and assumptions about potential adversary choices. DHS also concurred with our second recommendation that TSA  systematically evaluate the potential cost and effectiveness tradeoffs  across countermeasures. Appendix I: Transportation Security Administration (TSA) Information on the Direct Costs of Selected Countermeasures

As part of this review, we analyzed TSA’s fiscal year 2015 cost data for  six selected aviation security countermeasures—passenger prescreening  (Secure Flight), checkpoint screening, checked baggage screening,  explosives detection canines, the Behavior Detection and Analysis (BDA)  program, and the U.S. Federal Air Marshal Service (FAMS). We selected  these six passenger aviation security countermeasures because they  involved direct interaction with passengers, their belongings, or their  personal information and are largely operated and funded by TSA. Annual Proficiency Reviews. Specific  details about TSA’s detection standards and the results of OOI’s covert  tests are omitted because the information is classified.
Why GAO Did This Study

Since the attacks of September 11, 2001, TSA has spent billions of dollars on aviation security programs. However, recent attacks involving aircraft and airports in other countries underscore the continued threat to aviation and the need for an effective aviation security program.
GAO was asked to review TSA's passenger aviation security countermeasures. This report examines the extent to which TSA has (1) information on the effectiveness of selected passenger aviation security countermeasures and (2) systematically analyzed the cost and effectiveness tradeoffs among countermeasures.
GAO reviewed TSA documentation on the effectiveness of six passenger aviation security countermeasures in fiscal year 2015—the most recent year for which data were available. GAO selected these countermeasures because they involve direct interaction with passengers, their belongings, or their personal information, and are largely operated and funded by TSA. GAO also reviewed TSA documents and interviewed TSA officials regarding efforts to systematically analyze cost and effectiveness tradeoffs across countermeasures.

What GAO Found

The Transportation Security Administration (TSA) has data on the effectiveness of some, but not all of its passenger aviation security countermeasures. Specifically, TSA has data on passenger prescreening, checkpoint and checked baggage screening, and explosives detection canines. Further, TSA is taking steps to improve the quality of this information. However, it does not have effectiveness data for its Behavior Detection and Analysis (BDA) program and the U.S. Federal Air Marshal Service (FAMS). For BDA—a program to identify potential threats by observing passengers for behaviors indicative of stress, fear, or deception—in July 2017, GAO reported that (1) TSA does not have valid evidence supporting most of its behavioral indicators, and (2) TSA should continue to limit future funding for its behavior detection activities until it can provide such evidence. For FAMS—a program that deploys armed law enforcement officers on certain flights at an annual cost of about $800 million for fiscal year 2015—officials reported that one of the primary security contributions is to deter attacks. However, TSA does not have information on its effectiveness in doing so, nor does it have data on the deterrent effect resulting from any of its other aviation security countermeasures. While officials stated that deterrence is difficult to measure, the Government Performance and Results Act of 1993, as updated, provides that agencies are to assess the effectiveness of their programs. Further, the Office of Management and Budget and GAO have suggested approaches for measuring deterrence. Developing such methods for TSA countermeasures, especially for an effort such as FAMS in which the primary goal is deterrence, would enable TSA to determine whether its substantial investment is yielding results.
TSA has a tool to compare the security effectiveness of some aviation security countermeasures, but has no efforts underway to systematically evaluate potential cost and effectiveness tradeoffs across all countermeasures. In 2014, the agency developed a tool to analyze the security effectiveness of alternate combinations of some countermeasures for the purpose of informing acquisition and deployment decisions, but does not have a tool to assess such tradeoffs across the entire system of countermeasures. TSA officials explained that the aviation security system is constantly evolving, and assessing a system in flux is challenging. However, DHS policy and TSA's strategic plan call for the systematic evaluation of costs and effectiveness of TSA's chosen mix of aviation security countermeasures. Without such an analysis, TSA is not well positioned to strike an appropriate balance of costs, effectiveness, and risk.
This is a public version of a classified report that GAO issued in August 2017. Information that TSA deemed classified or sensitive security information, such as the results of TSA's covert testing and details about TSA's screening procedures, have been omitted.

What GAO Recommends

GAO recommends that TSA (1) explore and pursue methods to assess the deterrent effect of TSA's passenger aviation security countermeasures, with FAMS as a top priority to address, and (2) systematically evaluate the potential cost and effectiveness tradeoffs across aviation security countermeasures. DHS concurred with these recommendations.