DHS Is Taking Action to Validate the Scientific Basis of TSA’s SPOT Program but Opportunities Exist to Help Inform Future Program Decisions

Although DHS is in the process of validating the way in which the SPOT  program utilizes the science of behavior detection in an airport  environment, TSA deployed SPOT nationwide before first determining  whether there was a scientifically valid basis for using behavior and  appearance indicators as a means for reliably identifying passengers as  potential threats in airports. TSA stated  that no other large-scale U.S. or international screening program  incorporating behavior- and appearance-based indicators has ever been  rigorously scientifically validated. However, S&T’s current research plan is  not designed to fully validate whether behavior detection and appearances  can be effectively used to reliably identify individuals in an airport  terminal environment who pose a risk to the aviation system. Our prior work has  recommended the use of such independent panels for comprehensive,  objective reviews of complex issues. Specifically, these data could have been compared to data collected during  the SPOT pilots to determine if SPOT was more effective than random  screening in detecting passengers who pose a potential risk to aviation  security. However, TSA does not link goals in the SPOT  strategic plan with other related strategic documents, such as the Aviation  Implementation Plan of DHS’s Transportation Systems Sector-Specific  Plan and the Passenger Checkpoint Screening Program Strategic Plan. More Fully and Consistently Utilizing Available Information Technology Could Enhance TSA’s Ability to Identify Threats to the Aviation System

Inconsistencies in the use of available information technology to aid in the  collection and recording of data on passengers by BDOs during referrals to  LEOs, lack of guidance on, or a mechanism for, BDOs to request the TSA’s  Transportation Security Operations Center to run the names of passengers  exhibiting suspicious behaviors against law enforcement and intelligence   databases, and the Center’s not checking all of the databases available to  it—have limited TSA’s ability to identify potential terrorist threats to the  aviation system. Developing additional  guidance in the SPOT operating procedures could help improve  consistency in the extent to which BDOs utilize Transportation Security  Operations Center resources. We did not  analyze the extent to which the law enforcement and intelligence  databases available to TSA may contain overlapping information. TSA Lacks Program Effectiveness Measures for SPOT but Is Taking Steps to Improve Evaluation Capabilities

TSA has established some performance measures by tracking SPOT  referral and arrest data, but lacks the measures needed to evaluate the  effectiveness of the SPOT program and, as a result, has not been able to  fully assess SPOT’s contribution to improving aviation security. Concurrent with the DHS S&T Directorate study of SPOT, and an  independent panel assessment of the soundness of the methodology of the  S&T study, we recommend that the TSA Administrator take the following  six actions to ensure the program’s effective implementation:    To provide additional assurance that TSA utilizes available resources to  support the goals of deterring, detecting, and preventing security  threats to the aviation system, TSA should:    Provide guidance in the SPOT Standard Operating Procedures or  other TSA directive to BDOs, or other TSA personnel, on inputting  data into the Transportation Information Sharing System and set  milestones and a time frame for deploying Transportation  Information Sharing System access to SPOT airports so that TSA  and intelligence community entities have information from all SPOT  LEO referrals readily available to assist in “connecting the dots” and  identifying potential terror plots. To better measure the effectiveness of the program and evaluate the  performance of BDOs, TSA should:    Establish a plan that includes objectives, milestones, and time  frames to develop outcome-oriented performance measures to help  refine the current methods used by Behavior Detection Officers for  identifying individuals who may pose a risk to the aviation system. Appendix I: Scope and Methodology

To determine the extent to which the Transportation Security  Administration (TSA) determined whether the Screening of Passengers By  Observation Techniques (SPOT) program had a scientifically-validated  basis for identifying passengers before deploying it, we reviewed literature  on behavior analysis by subject matter experts, interviewed seven experts  in behavior analysis, interviewed other federal agencies and entities about  how they use behavior detection techniques, and analyzed relevant reports  and books on the topic. These included a 2008 study by the National  Research Council of the National Academy of Sciences that has a  discussion regarding deception and behavioral surveillance, as well as  other issues related to behavioral analysis. However, these visits provided helpful insight into the operation of SPOT  at airports.
Why GAO Did This Study

To enhance aviation security, the Transportation Security Administration (TSA) began initial testing in October 2003 of its Screening of Passengers by Observation Techniques (SPOT) program. Behavior Detection Officers (BDO) carry out SPOT's mission to identify persons who pose a risk to aviation security by focusing on behavioral and appearance indicators. GAO was asked to review the SPOT program. GAO analyzed (1) the extent to which TSA validated the SPOT program before deployment, (2) implementation challenges, and (3) the extent to which TSA measures SPOT's effect on aviation security. GAO analyzed TSA documents, such as strategic plans and operating procedures; interviewed agency personnel and subject matter experts; and visited 15 SPOT airports, among other things. Although the results from these visits are not generalizable, they provided insights into SPOT operations.

What GAO Found

Although the Department of Homeland Security (DHS) is in the process of validating some aspects of the SPOT program, TSA deployed SPOT nationwide without first validating the scientific basis for identifying suspicious passengers in an airport environment. A scientific consensus does not exist on whether behavior detection principles can be reliably used for counterterrorism purposes, according to the National Research Council of the National Academy of Sciences. According to TSA, no other large-scale security screening program based on behavioral indicators has ever been rigorously scientifically validated. DHS plans to review aspects of SPOT, such as whether the program is more effective at identifying threats than random screening. Nonetheless, DHS's current plan to assess SPOT is not designed to fully validate whether behavior detection can be used to reliably identify individuals in an airport environment who pose a security risk. For example, factors such as the length of time BDOs can observe passengers without becoming fatigued are not part of the plan and could provide additional information on the extent to which SPOT can be effectively implemented. Prior GAO work has found that independent expert review panels can provide comprehensive, objective reviews of complex issues. Use of such a panel to review DHS's methodology could help ensure a rigorous, scientific validation of SPOT, helping provide more assurance that SPOT is fulfilling its mission to strengthen aviation security. TSA is experiencing implementation challenges, including not fully utilizing the resources it has available to systematically collect and analyze the information obtained by BDOs on passengers who may pose a threat to the aviation system. TSA's Transportation System Operations Center has the resources to investigate aviation threats but generally does not check all law enforcement and intelligence databases available to it to identify persons referred by BDOs. Utilizing existing resources would enhance TSA's ability to quickly verify passenger identity and could help TSA to more reliably "connect the dots." Further, most BDOs lack a mechanism to input data on suspicious passengers into a database used by TSA analysts and also lack a means to obtain information from the Transportation System Operations Center on a timely basis. TSA states that it is in the process of providing input capabilities, but does not have a time frame for when this will occur at all SPOT airports. Providing BDOs, or other TSA personnel, with these capabilities could help TSA "connect the dots" to identify potential threats. Although TSA has some performance measures related to SPOT, it lacks outcome-oriented measures to evaluate the program's progress toward reaching its goals. Establishing a plan to develop these measures could better position TSA to determine if SPOT is contributing to TSA's strategic goals for aviation security. TSA is planning to enhance its evaluation capabilities in 2010 to more readily assess the program's effectiveness by conducting statistical analysis of data related to SPOT referrals to law enforcement and associated arrests.