To mitigate  those concerns, TSA began installing automated target recognition (ATR)  software on deployed AIT systems in July 2011.with ATR (AIT-ATR) automatically interpret the image and display  anomalies on a generic outline of a passenger instead of displaying  images of actual passenger bodies. TSA Has Taken Steps to Improve Oversight of Secure Flight, but Could Take Further Action to Measure Program Performance and Address Screening Errors

In September 2014, we reported on three issues affecting the  effectiveness of TSA’s Secure Flight program—(1) the need for additional  performance measures to capture progress toward Secure Flight program  goals, (2) Secure Flight system matching errors, and (3) mistakes  screening personnel have made in implementing Secure Flight at the  screening checkpoint. Secure Flight system matching errors: In September 2014, we found  that TSA lacked timely and reliable information on all known cases of  Secure Flight system matching errors, meaning instances where Secure  Flight did not identify passengers who were actual matches to these lists. DHS concurred with our recommendation, and as of April  2015, TSA had developed such a mechanism. However, TSA has not yet  demonstrated how it will use the information to improve the performance  of the Secure Flight system. TSA information from May 2012  through February 2014 that we assessed indicates that screening  personnel made errors at the checkpoint in screening passengers  consistent with their Secure Flight determinations. However, we  found that TSA did not have a systematic process for evaluating the root  causes of these screening errors across airports, which could allow TSA  to identify trends across airports and target nationwide efforts to address  these issues. Therefore, we recommended in September 2014  that TSA develop a process for evaluating the root causes of screening  errors at the checkpoint and then implement corrective measures to  address those causes. DHS concurred with our recommendations and  has developed a process for collecting and evaluating data on the root  causes of screening errors. However, as of April 2015, TSA had not yet  shown that the agency has implemented corrective measures to address  the root causes. TSA Performance Assessments of AIT- ATR Did Not Account for All Factors Affecting the System

In March 2014, we reported that, according to TSA officials, checkpoint  security is a function of technology, people, and the processes that  govern them, however we found that TSA did not include each of those  factors in determining overall AIT-ATR system performance. Specifically, we found that TSA evaluated the technology’s performance  in the laboratory to determine system effectiveness. However, laboratory  test results provide important insights but do not accurately reflect how  well the technology will perform in the field with actual human operators. AIT-ATR system effectiveness relies on both the  technology’s capability to identify threat items and its operators to resolve  those threat items. Given that TSA was seeking to procure the second generation of AIT  systems, known as AIT-2, we reported that DHS and TSA would be  hampered in their ability to ensure that future AIT systems meet mission  needs and perform as intended at airports unless TSA evaluated system  effectiveness based on both the performance of the AIT-2 technology and  screening officers who operate the technology. TSA  concurred and reported taking steps to address this recommendation. However, TSA has not yet  provided sufficient documentation showing that this recommendation has  been fully addressed. TSA Has Not Tested the Overall Effectiveness of Its Managed Inclusion Process, But Plans to Conduct Such Testing

In December 2014, we reported that, according to TSA officials, TSA  tested the security effectiveness of the individual components of the  Managed Inclusion process—such as BDOs and ETD devices—before  implementing Managed Inclusion, and TSA determined that each layer  alone provides an effective level of security. As a result, we  recommended that to ensure TSA’s planned testing yields reliable results,  the TSA Administrator take steps to ensure that TSA’s planned  effectiveness testing of the Managed Inclusion process adheres to  established evaluation design practices. Specifically,  DHS stated that TSA plans to use a test and evaluation process—which  calls for the preparation of test and evaluation framework documents  including plans, analyses, and a final report describing the test results— for its planned effectiveness testing of Managed Inclusion. Key contributors for the previous work that this  testimony is based on are listed in each product.
Why GAO Did This Study

Since the attacks of September 11, 2001 exposed vulnerabilities in the nation's aviation system, billions of dollars have been spent on a wide range of programs designed to enhance aviation security. Securing commercial aviation remains a daunting task, and continuing fiscal pressure highlights the need for TSA to determine how to allocate its finite resources for the greatest impact. GAO previously reported on TSA's oversight of its aviation security programs, including the extent to which TSA has the information needed to assess the programs.
This testimony focuses on TSA's oversight of aviation security measures including, among other things (1) Secure Flight, (2) Advanced Imaging Technology, and (3) Managed Inclusion. This statement is based on reports and testimonies issued from December 2011 through May 2015. For prior work, GAO analyzed TSA documents and interviewed TSA officials, among other things.

What GAO Found

The Transportation Security Administration (TSA) has taken steps to improve oversight of Secure Flight—a passenger prescreening program that matches passenger information against watch lists and assigns each passenger a risk category—but could take further action to address screening errors. In September 2014, GAO reported that TSA lacked timely and reliable information on system matching errors—instances where Secure Flight did not identify passengers who were actual matches to watch lists. GAO recommended that TSA systematically document such errors to help TSA determine if actions can be taken to prevent similar errors from occurring. The Department of Homeland Security (DHS) concurred and has developed a mechanism to do so, but has not yet shown how it will use this information to improve system performance. In September 2014, GAO also found that screening personnel made errors in screening passengers at the checkpoint at a level consistent with their Secure Flight risk determinations and that TSA did not have a systematic process for evaluating the root causes of these errors across airports. GAO recommended that TSA develop a process for evaluating the root causes and implement corrective measures to address them. DHS concurred and has developed such a process but has not yet demonstrated implementation of corrective measures.
In March 2014, GAO found that TSA performance assessments of certain full-body scanners used to screen passengers at airports did not account for all factors affecting the systems. GAO reported that the effectiveness of Advanced Imaging Technology (AIT) systems equipped with automated target recognition software (AIT-ATR)—which displays anomalies on a generic passenger outline instead of actual passenger bodies—relied on both the technology's capability to identify potential threat items and its operators' ability to resolve them. However, GAO found that TSA did not include these factors in determining overall AIT-ATR system performance. GAO also found that TSA evaluated the technology's performance in the laboratory—a practice that does not reflect how well the technology will perform with actual human operators. In considering procurement of the next generation of AIT systems (AIT-2), GAO recommended that TSA measure system effectiveness based on the performance of both the technology and the screening personnel. DHS concurred and in January 2015 reported that it has evaluated the AIT-2 technology and screening personnel as a system but has not yet provided sufficient documentation of this effort.
In December 2014, GAO found that TSA had not tested the effectiveness of its overall Managed Inclusion process—a process to assess passenger risk in real time at the airport and provide expedited screening to certain passengers—but had plans to do so. Specifically, GAO found that TSA had tested the effectiveness of individual components of the Managed Inclusion process, such as canine teams, but had not yet tested the effectiveness of the overall process. TSA officials stated that they had plans to conduct such testing. Given that GAO has previously reported on TSA challenges testing the effectiveness of its security programs, GAO recommended that TSA ensure its planned testing of the Managed Inclusion process adhere to established evaluation design practices. DHS concurred and has plans to use a test and evaluation process for its planned testing of Managed Inclusion.

What GAO Recommends

GAO has previously made recommendations to DHS to strengthen TSA's oversight of aviation security programs. DHS generally agreed and has actions underway to address them. Consequently, GAO is not making any new recommendations in this testimony.