According to TSA’s strategic plan and other program  guidance for the BDA program released in December 2012, the goal of  the agency’s behavior detection activities, including the SPOT program, is  to identify high-risk passengers based on behavioral indicators that  indicate “mal-intent.” For example, the strategic plan notes that in concert  with other security measures, behavior detection activities “must be  dedicated to finding individuals with the intent to do harm, as well as  individuals with connections to terrorist networks that may be involved in  criminal activity supporting terrorism.”
TSA developed its primary behavior detection activity, the SPOT  program, in 2003 as an added layer of security to identify potentially high- risk passengers through behavior observation and analysis techniques. From fiscal year 2007—when the SPOT  program began deployment nationwide—through fiscal year 2012, about  $900 million has been expended on the program, as shown in figure 2. While the April 2011 SPOT validation study was a  useful initial step and, in part, addressed issues raised in our May 2010  report, it does not demonstrate the effectiveness of the SPOT indicators  because of methodological weaknesses in the study. Further, TSA  program officials and BDOs we interviewed agree that some of the  behavioral indicators used to identify passengers for additional screening  are subjective. The meta- analyses, or reviews that synthesize the findings of other studies, we  reviewed collectively included research from more than 400 separate  studies on detecting deception, and found that the ability of human  observers to accurately identify deceptive behavior based on behavioral  cues or indicators is the same as or slightly better than chance (54  percent). Subjective Interpretation of Behavioral Indicators and Variation in Referral Rates Raise Questions about the Use of Indicators; TSA Plans to Study Indicators

			BDO Interpretation of Some Behavioral Indicators Is Subjective; TSA Plans Study

BDA officials at headquarters and BDOs we interviewed in four airports  said that some of the behavioral indicators are subjective, and TSA has  not demonstrated that BDOs can consistently interpret behavioral  indicators, though the agency has efforts under way to reduce subjectivity  in the interpretation by BDOs. Our analysis of SPOT referral data from fiscal years 2011 and 2012  indicates that SPOT and LEO referral rates vary significantly across  BDOs at some airports, which raises questions about the use of  behavioral indicators by BDOs. TSA Has Limited Information to Evaluate SPOT Program Effectiveness but Plans to Collect Additional Performance Data

TSA has limited information to evaluate SPOT program effectiveness  because the findings from the April 2011 validation comparison study are  inconclusive because of methodological weaknesses in the study’s  overall design and data collection. Specifically,  in December 2009, TSA initially began collecting data from 24 airports  whose participation in the study was determined by the local TSA  officials. More than 7 months later, TSA added another 18 airports to the  study when it determined that enough data were not being collected on  the randomly selected passengers at participating airports to reach the  study’s required sample size. Because commercial aviation activity and the  demographics of the traveling public are not constant throughout the year,  this uneven data collection may have conflated the effect of random  versus SPOT selection methods with differences in the rates of high-risk  passengers when TSA used either method. In response to an additional recommendation in our May 2010 report to  develop a plan for outcome-based performance measures, TSA  completed a performance metrics plan in November 2012, which details  the performance measures required for TSA to determine whether the  agency’s behavior detection activities are effective, and identifies the  gaps that exist in its current data collection efforts. Until TSA can provide  scientifically validated evidence demonstrating that behavioral indicators  can be used to identify passengers who may pose a threat to aviation  security, the agency risks funding activities that have not been  determined to be effective. Matter for Congressional Consideration

To help ensure that security-related funding is directed to programs that  have demonstrated their effectiveness, Congress should consider the  findings in this report regarding the absence of scientifically validated  evidence for using behavioral indicators to identify aviation security  threats when assessing the potential benefits of behavior detection  activities relative to their cost when making future funding decisions  related to aviation security. Recommendation for Executive Action

To help ensure that security-related funding is directed to programs that  have demonstrated their effectiveness, we recommend that the Secretary  of Homeland Security direct the TSA Administrator to limit future funding  support for the agency’s behavior detection activities until TSA can  provide scientifically validated evidence that demonstrates that behavioral  indicators can be used to identify passengers who may pose a threat to  aviation security. However, we disagree  with this statement. We did not substantiate these specific claims. To what extent does available evidence support the use of behavioral  indicators to identify aviation security threats? We selected this nonprobability  sample based on the airports’ size and participation in behavior detection  programs. Data to Assess Effectiveness of SPOT

To determine the extent to which TSA has data necessary to assess the  effectiveness of the SPOT program in identifying threats to aviation  security, we reviewed the validation study’s findings comparing  passengers selected by SPOT with randomly selected passengers,  analyzed TSA plans and analyses designed to measure SPOT’s  effectiveness, and analyzed data on SPOT referrals and LEO arrests. These responses are not  generalizable to the entire BDO population at SPOT airports. We  did not substantiate these specific claims.
Why GAO Did This Study

TSA began deploying the SPOT program in fiscal year 2007--and has since spent about $900 million--to identify persons who may pose a risk to aviation security through the observation of behavioral indicators. In May 2010, GAO concluded, among other things, that TSA deployed SPOT without validating its scientific basis and SPOT lacked performance measures. GAO was asked to update its assessment. This report addresses the extent to which (1) available evidence supports the use of behavioral indicators to identify aviation security threats and (2) TSA has the data necessary to assess the SPOT program's effectiveness. GAO analyzed fiscal year 2011 and 2012 SPOT program data. GAO visited four SPOT airports, chosen on the basis of size, among other things, and interviewed TSA officials and a nonprobability sample of 25 randomly selected BDOs. These results are not generalizable, but provided insights.

What GAO Found

Available evidence does not support whether behavioral indicators, which are used in the Transportation Security Administration's (TSA) Screening of Passengers by Observation Techniques (SPOT) program, can be used to identify persons who may pose a risk to aviation security. GAO reviewed four meta-analyses (reviews that analyze other studies and synthesize their findings) that included over 400 studies from the past 60 years and found that the human ability to accurately identify deceptive behavior based on behavioral indicators is the same as or slightly better than chance. Further, the Department of Homeland Security's (DHS) April 2011 study conducted to validate SPOT's behavioral indicators did not demonstrate their effectiveness because of study limitations, including the use of unreliable data. Twenty-one of the 25 behavior detection officers (BDO) GAO interviewed at four airports said that some behavioral indicators are subjective. TSA officials agree, and said they are working to better define them. GAO analyzed data from fiscal years 2011 and 2012 on the rates at which BDOs referred passengers for additional screening based on behavioral indicators and found that BDOs' referral rates varied significantly across airports, raising questions about the use of behavioral indicators by BDOs. To help ensure consistency, TSA officials said they deployed teams nationally to verify compliance with SPOT procedures in August 2013. However, these teams are not designed to help ensure BDOs consistently interpret SPOT indicators.
TSA has limited information to evaluate SPOT's effectiveness, but plans to collect additional performance data. The April 2011 study found that SPOT was more likely to correctly identify outcomes representing a high-risk passenger--such as possession of a fraudulent document--than through a random selection process. However, the study results are inconclusive because of limitations in the design and data collection and cannot be used to demonstrate the effectiveness of SPOT. For example, TSA collected the study data unevenly. In December 2009, TSA began collecting data from 24 airports, added 1 airport after 3 months, and an additional 18 airports more than 7 months later when it determined that the airports were not collecting enough data to reach the study's required sample size. Since aviation activity and passenger demographics are not constant throughout the year, this uneven data collection may have conflated the effect of random versus SPOT selection methods. Further, BDOs knew if passengers they screened were selected using the random selection protocol or SPOT procedures, a fact that may have introduced bias into the study. TSA completed a performance metrics plan in November 2012 that details the performance measures required for TSA to determine whether its behavior detection activities are effective, as GAO recommended in May 2010. However, the plan notes that it will be 3 years before TSA can begin to report on the effectiveness of its behavior detection activities. Until TSA can provide scientifically validated evidence demonstrating that behavioral indicators can be used to identify passengers who may pose a threat to aviation security, the agency risks funding activities that have not been determined to be effective. This is a public version of a sensitive report that GAO issued in November 2013. Information that TSA deemed sensitive has been redacted.

What GAO Recommends

Congress should consider the absence of scientifically validated evidence for using behavioral indicators to identify threats to aviation security when assessing the potential benefits and cost in making future funding decisions for aviation security. GAO included this matter because DHS did not concur with GAOs recommendation that TSA limit future funding for these activities until it can provide such evidence, in part because DHS disagreed with GAOs analysis of indicators. GAO continues to believe the report findings and recommendation are valid.