Additionally, TSA used those requirements to evaluate the next  generation of AIT systems, referred to as AIT-2. TSA Does Not Collect or Analyze Data That Could Enhance System Performance

TSA does not collect or analyze three types of available information that  could be used to enhance the effectiveness of the entire AIT-ATR system. Third, TSA assesses the overall AIT-ATR  system performance using laboratory test results that do not reflect the  combined performance of the technology, the personnel that operate it,  and the process that governs AIT-related security operations. TSA is not enforcing compliance with its directive, and as a result, data on  SO performance are not being consistently collected or reported by  approximately half of airports with AIT-ATR systems. For example,  according to TSA data, we found that TSA personnel at almost half of the  airports with AIT-IO or AIT-ATR systems did not report any IED  checkpoint drill results on those systems from March 2011 through  February 2013. According to TSA  officials, the agency does not ensure compliance with the directive at  every airport because it is unclear which office within the Office of  Security Operations should oversee enforcing the operational directive. Data on IED checkpoint drills could provide insight into how well SOs  resolve anomalies detected by the AIT systems, information that could be  used to help strengthen the existing screening process. TSA officials agreed that it is important to analyze performance by  including an evaluation of the technology, operators, and processes, and  stated that TSA is planning to assess the performance of all layers of  security. TSA Has Enhanced Privacy by Upgrading All Deployed Systems with ATR Software, but Has Not Met Its Goals Pertaining to Enhancing AIT Capabilities

TSA has met milestones as documented in its roadmap pertaining to the  installation of ATR software upgrades that were intended to address  privacy concerns and improve operational efficiency for all deployed AIT  systems in accordance with the statutory deadline included as part of the Federal Aviation Administration Modernization and Reform Act of 2012.However, it did not meet proposed milestones documented in its AIT  roadmap to provide enhanced capabilities to meet the agency’s mission  needs. As shown in figure 2, TSA began operational test and  evaluation for Tier II upgrades 17 months after the expected start date  articulated in its October 2012 roadmap. TSA did not incorporate  available information from the national laboratories and vendors into its  updated roadmap. Recommendations for Executive Action

To help ensure that TSA improves SO performance on AIT-ATR systems  and uses resources effectively, the Administrator of the Transportation  Security Administration should take the following two actions:  clarify which office is responsible for overseeing TSA’s IED screening  checkpoint drills operational directive, direct the office to ensure  enforcement of the directive in conducting these drills, and analyze  the data to identify any potential weaknesses in the screening  process, and establish protocols that facilitate the capturing of operational data on  secondary screening of passengers at the checkpoint to determine  the extent to which AIT-ATR system false alarm rates affect  operational costs once AIT-ATR systems are networked together. To help ensure that TSA invests in screening technology that meets  mission needs, the Administrator of the Transportation Security  Administration should ensure that the following two actions are taken  before procuring AIT-2 systems: measure system effectiveness based on the performance of the AIT-2  technology and screening officers who operate the technology, while  taking into account current processes and deployment strategies, and use scientific evidence and information from DHS’s Science and  Technology Directorate, and the national laboratories, as well as  information and data provided by vendors to develop a realistic  schedule with achievable milestones that outlines the technological  advancements, estimated time, and resources needed to achieve  TSA’s Tier IV end state. To what extent does TSA collect and analyze available information  that could be used to enhance the performance of AIT systems  equipped with ATR (AIT-ATR)? To what extent has TSA made progress toward enhancing AIT  capabilities to detect concealed explosives and other threat items, and  what challenges, if any, remain? To determine the extent to which TSA collects and analyzes available  information to improve the performance of screening officers (SO)  responsible for resolving anomalies identified by ATR software, we  analyzed improvised explosive device (IED) checkpoint drills conducted  by TSA personnel at airports that submitted data to TSA from March 1,  2011, through February 28, 2013, under TSA’s IED checkpoint drill  operational directive. TSA’s IED checkpoint drill operational directive  requires personnel at airports to conduct drills to assess Transportation  Security Officer (TSO) compliance with TSA’s screening standard  operating procedures (SOP) and to train TSOs to better resolve  anomalies identified by AIT-ATR systems.
Why GAO Did This Study

TSA accelerated the deployment of AIT systems, or full-body scanners, in response to the December 25, 2009, attempted terrorist attack on Northwest Airlines Flight 253. Pursuant to the Federal Aviation Administration Modernization and Reform Act of 2012, TSA was mandated to ensure that AIT systems were equipped with ATR software, which displays generic outlines of passengers rather than actual images, by June 1, 2013. All deployed AIT systems were equipped with ATR software by the deadline. GAO was asked to evaluate TSA's AIT-ATR systems' effectiveness. This report addresses the extent to which (1) TSA collects and analyzes available information that could be used to enhance the effectiveness of the AIT-ATR system and (2) TSA has made progress toward enhancing AIT capabilities to detect concealed explosives and other threat items, and any challenges that remain. GAO analyzed testing results conducted by the Transportation Security Laboratory and TSA personnel at airports and interviewed DHS and TSA officials. This is a public version of a classified report that GAO issued in December 2013. Information DHS and TSA deemed classified or sensitive has been omitted, including information and recommendations related to improving AIT capabilities.

What GAO Found

The Department of Homeland Security's (DHS) Transportation Security Administration (TSA) does not collect or analyze available information that could be used to enhance the effectiveness of the advanced imaging technology (AIT) with automated target recognition (ATR) system. Specifically, TSA does not collect or analyze available data on drills using improvised explosive devices (IED) at the checkpoint that could provide insight into how well screening officers (SO) resolve anomalies, including objects that could pose a threat to an aircraft, identified by AIT systems, because it does not enforce compliance with its operational directive. TSA's operational directive requires personnel at airports to conduct drills to assess SO compliance with TSA's screening standard operating procedures and to train SOs to better resolve anomalies identified by AIT-ATR systems. GAO found that TSA personnel at about half of airports with AIT systems did not report any IED checkpoint drill results on those systems from March 2011 through February 2013. According to TSA, it does not ensure compliance with the directive at every airport because it is unclear which office should oversee enforcing the directive. Without data on IED checkpoint drills, TSA lacks insight into how well SOs resolve anomalies detected by AIT systems, information that could be used to help strengthen existing screening processes. Potential weaknesses in the screening process could be caused by TSA not clarifying which office is responsible for overseeing TSA's operational directive, directing that office to ensure enforcement of the directive in conducting these drills, and analyzing the data. Further, when determining AIT-ATR system effectiveness, TSA uses laboratory test results that do not reflect the combined performance of the technology, the personnel who operate it, and the process that governs AIT-related security operations. TSA officials agreed that it is important to analyze performance by including an evaluation of the technology, operators, and processes and stated that TSA is planning to assess the performance of all layers of security. By not measuring system effectiveness based on the performance of the technology and SOs who operate the technology or taking into account current processes and deployment strategies, DHS and TSA are not ensuring that future procurements meet mission needs.
TSA completed the installation of ATR software upgrades intended to address privacy concerns for all deployed AIT systems; however, it has not met proposed milestones for enhancing capabilities as documented in its AIT roadmap—a document that contains milestones for achieving enhanced capabilities to meet the agency's mission needs. For example, TSA began operational test and evaluation for Tier II upgrades 17 months after the expected start date. Moreover, TSA did not use available scientific research or information from experts from the national laboratories or vendors on the technological challenges that it faces in developing requirements and milestones, because, according to TSA, it relied on time frames proposed by vendors. Thus, TSA cannot ensure that its roadmap reflects the true capabilities of the next generation of AIT systems by using scientific evidence and information from DHS's Science and Technology Directorate, the national laboratories, and vendors to develop a realistic schedule with achievable milestones that outlines the technological advancements, estimated time, and resources needed to achieve enhanced capabilities as outlined in TSA's roadmap.

What GAO Recommends

GAO recommends that TSA, among other things, clarify which office should oversee its operational directive, better measure system effectiveness, and develop a realistic schedule before procuring future generations. TSA concurred with GAO's recommendations.