	Background

The Aviation and Transportation Security Act (ATSA) established TSA as  the primary federal agency with responsibility for securing the nation’s  civil aviation system. AIT systems equipped with ATR (AIT-ATR) automatically  interpret the image and display anomalies on a generic outline of a  passenger instead of displaying images of actual passenger bodies. TSA Has Not Consistently Evaluated the Overall Effectiveness of New Technologies, Programs, and Processes Using Robust Methods of Testing and Evaluation

In our 2014 reviews of TSA’s AIT-ATR systems and Managed Inclusion  process, we found that TSA had conducted some testing before adopting  the new technology and process, but it had not fully demonstrated their  effectiveness. With regard to the AIT-ATR system, in March 2014, we reported that,  according to TSA officials, checkpoint security is a function of technology,  people, and the processes that govern them; however, we found that TSA  did not include each of those factors in determining overall AIT-ATR  system performance. AIT-ATR system effectiveness relies on  both the technology’s capability to identify threat items and its operators  to resolve those threat items. Given that TSA was seeking to procure the second generation of AIT  systems, known as AIT-2, we reported that DHS and TSA would be  hampered in their ability to ensure that future AIT systems meet mission  needs and perform as intended at airports unless TSA evaluated system  effectiveness based on both the performance of the AIT-2 technology and  screening officers who operate the technology. TSA concurred and has addressed this recommendation. Specifically, in  June 2015, TSA provided documentation showing that, while conducting  operational testing of the AIT-2 system, the agency considered screening  officer performance and measured AIT-2 system effectiveness based on  both the performance of the AIT-2 technology and the screening officers  who operate it. This should help TSA assess whether this screening  system will meet mission needs and perform as intended. TSA Has Not Consistently Established Performance Measures That Fully Reflect Program Goals

In 2014, we reported on two instances in which TSA’s performance  measures made it difficult to assess TSA’s performance in meeting its  goals. In September 2014, we found that Secure Flight had established program  goals that reflect new program functions since implementation began in  2009 to identify additional types of high-risk and also low-risk passengers;  however, the program performance measures in place at that time did not  allow TSA to fully assess its progress toward achieving all of its goals. For example, one program goal was to accurately identify passengers on  various watch lists. To assess performance toward this goal, Secure  Flight collected various types of data, including the number of passengers  TSA identifies as matches to high- and low-risk lists, but did not have  measures to assess the extent of system matching errors—for example,  the extent to which Secure Flight is missing passengers who are actual  matches to these lists. DHS concurred with our recommendation, and,  according to TSA officials, as of April 2015, TSA’s Office of Intelligence  and Analysis was evaluating its current Secure Flight performance goals  and measures and determining what new performance measures should  be established to fully measure progress against program goals. TSA Has Not Consistently Used Program Data to Identify Opportunities for Improvement

We have also reported on findings related to program data—such as  canine program assessment data and Secure Flight screening error  data—that TSA collected but had not analyzed, missing opportunities to  refine and further improve TSA programs. In January 2013, we reported that TSA collected and used key canine  program data in support of its NEDCTP program, but could better analyze  these data to identify program trends. For example, we found that in  reviewing short notice assessments (covert tests), TSA did not analyze  the results beyond the pass and fail rates. We recommended that TSA regularly analyze available  data to identify program trends and areas that are working well and those  in need of corrective action to guide program resources and activities. DHS concurred with our recommendation and has taken  actions to address them. Key contributors for the previous  work that this testimony is based on are listed in each product.
Why GAO Did This Study

Since the attacks of September 11, 2001 exposed vulnerabilities in the nation's aviation system, billions of dollars have been spent on a wide range of programs designed to enhance aviation security. Continuing fiscal pressure highlights the need for DHS's TSA, the primary federal agency responsible for aviation security, to allocate its finite resources for the greatest impact.
This testimony addresses the extent to which TSA has (1) evaluated the overall effectiveness of new technologies, programs, and processes using robust methods of testing and evaluation, (2) established performance measures that fully reflect program goals, and (3) used program data to identify opportunities for improvement. This statement is based on findings from GAO reports and testimonies issued from January 2013 through June 2015, with selected updates conducted from April 2015 through October 2015 to, among other things, determine progress made in implementing previous GAO recommendations. For prior work, GAO analyzed TSA policy documents and interviewed TSA officials, among other things. For the updates, GAO reviewed documents and followed up with TSA officials about actions to address GAO recommendations.

What GAO Found

Evaluation of new technologies, programs, and processes . GAO has found that TSA has not consistently evaluated the overall effectiveness of new technologies before adopting them. For example, in March 2014, GAO found that TSA testing of certain Advanced Imaging Technology (AIT) systems—also referred to as full-body scanners—used to screen passengers at airports did not account for all factors affecting the systems. GAO reported that the effectiveness of AIT systems equipped with automated target recognition software (AIT-ATR)—which display anomalies on a generic passenger outline—relied on both the technology's capability to identify potential threat items and its operators' ability to resolve them. However, GAO found that TSA did not include operators' ability in determining overall AIT-ATR system performance. GAO recommended that TSA, in considering procurement of the next generation of AIT systems (AIT-2), measure system effectiveness based on the performance of both the technology and the screening personnel. The Department of Homeland Security (DHS) concurred and, in June 2015, TSA provided documentation showing that, while conducting operational testing of the AIT-2 system, the agency considered screening officer performance and measured AIT-2 system effectiveness based on both the performance of the AIT-2 technology and the screening officers who operate it. This should help TSA assess whether this screening system will meet mission needs and perform as intended.
Establishment of performance measures. GAO has found that TSA has not consistently established performance measures that fully reflect program goals. For example, in September 2014, GAO found that TSA's performance measures for Secure Flight—a passenger prescreening program—did not allow TSA to fully assess its progress toward achieving all of its goals. For example, one program goal was to accurately identify passengers on various watch lists, but TSA did not have measures to assess the extent of system matching errors, such as the extent to which Secure Flight is missing passengers who are actual matches to these lists. GAO recommended that TSA develop such measures. DHS concurred, and, as of April 2015, TSA was evaluating its current Secure Flight performance goals and measures and determining what new performance measures should be established to fully measure progress against program goals.
Use of program data. GAO has also reported on findings related to program data that TSA collected but had not analyzed, missing opportunities to refine and further improve TSA programs. For example, in January 2013, GAO reported that TSA collected and used key program data in support of its National Explosives Detection Canine Team Program, but could better analyze these data to identify program trends. For example, GAO found that in reviewing the results of certain covert tests, TSA did not analyze the results beyond the pass and fail rates, missing an opportunity to identify corrective actions. GAO recommended that TSA regularly analyze available data to identify program trends and areas that are working well and those in need of corrective action to guide program resources and activities. TSA concurred with GAO's recommendation and has taken actions to address this, including requiring analysis of the reasons for certain failed assessments.

What GAO Recommends

GAO has previously made recommendations to DHS to strengthen TSA's aviation security programs. DHS generally agreed and has either addressed or has actions underway to address most of them. Consequently, GAO is not making new recommendations in this testimony.