In November 2002, MTSA further required DHS to  issue a maritime worker identification card that uses biometrics to control  access to secure areas of maritime transportation facilities and vessels. Transportation Security Administration. August 2008. Among other things, we recommended that an  evaluation plan and data analysis plan be developed to guide the  remainder of the pilot and to identify how DHS would compensate for  areas where the TWIC reader pilot would not provide the information  needed to report to Congress and implement the TWIC card reader rule. In addition, the Coast  Guard Authorization Act of 2010 required that the findings of the pilot be  included in a report to Congress, and that we assess the reported findings  and recommendations. TWIC Reader Pilot Results Are Not Sufficiently Complete, Accurate, and Reliable for Informing Congress and the TWIC Card Reader Rule

Challenges related to pilot planning, data collection, and reporting affect  the completeness, accuracy, and reliability of the pilot test aimed at  assessing the technology and operational impact of using TSA’s TWIC  with card readers. Moreover, according to our review of the pilot and  TSA’s past efforts to demonstrate the validity and security benefits of the  TWIC program, the program’s premise and effectiveness in enhancing  security are not supported. DHS concurred with these recommendations. While TSA developed a data analysis plan, TSA and USCG reported that  they did not develop an evaluation plan with an evaluation methodology  or performance standards, as we recommended. TSA and the independent test agent did not collect complete data  on malfunctioning TWIC cards. Pilot participants did not document instances of denied access. Issues with DHS’s Congressional Report on the Pilot and the Validity of the TWIC Security Premise Raise Concerns about the Effectiveness of the TWIC Program

			DHS’s Report to Congress Presented Findings and Lessons Learned That Were Not Always Supported by the Collected Data

As required by the SAFE Port Act and the Coast Guard Authorization Act  of 2010, DHS’s report to Congress on the TWIC reader pilot presented  several findings with respect to technical and operational aspects of  implementing TWIC technologies in the maritime environment. However, according to our review, the findings and lessons learned in  DHS’s report to Congress were based on incomplete or unreliable data,  and thus should not be used to inform the development of the future  regulation on the use of TWIC with readers. Transportation Security Administration. Furthermore, one of the driving assumptions in the TWIC cost-benefit  analysis was that the lack of a common credential across the industry  could leave facilities open to a security breach with falsified credentials. Additionally, the TWIC reader pilot report concluded that TWIC cards and  readers provide a critical layer of security at our nation’s ports. The depth and  pervasiveness of the TWIC program’s planning and implementation  challenges require a reassessment of DHS’s efforts to improve maritime  security through the issuance of a U.S. government-sponsored TWIC  card and card readers. Specifically, as discussed in our report, and as  confirmed by the supplemental technical comments provided by DHS, the  pilot test’s results were incomplete, inaccurate, and unreliable for  informing Congress and for developing a regulation about the readers. For example, as discussed in the report:  Installed TWIC readers and access control systems could not collect  required data, including reasons for errors, on TWIC reader use, and  TSA and the independent test agent did not employ effective  compensating data collection measures, such as manually recording  reasons for errors in reading TWICs. TSA and the independent test agent did not record clear baseline data  for comparing operational performance at access points with TWIC  readers. TSA and the independent test agent did not collect complete data on  malfunctioning TWIC cards. To evaluate the extent to which the results from the TWIC reader pilot  were sufficiently complete, accurate, and reliable for informing Congress  and the TWIC card reader rule, we assessed (1) TWIC reader pilot test  planning and preparation activities, (2) pilot implementation and data  collection practices, and (3) the findings reported in the Department of  Homeland Security’s (DHS) February 2012 report to Congress on the  results of the TWIC reader pilot against underlying pilot data. These  techniques provided us with the following summary and comparative  views of collected pilot data, among others, which in part served as  the basis of our data analysis:  compiled data by pilot site;  compiled data on baseline population of users at each pilot site and  reported access points;  comparison of the total population at baseline to total population  reported during the ST&E phase;  view of pilot site access point and reader matches across testing  results (baseline data, Systems Operational Verification Testing  (SOVT) data, EOA data, and ST&E data);  view of tested reader and access control system characteristics;  comparison of baseline throughput times versus EOA and ST&E  throughput times for access points with similar readers used;  comparison of data across the pilot to identify trends, if any, in areas  such as risk level, facility and vessel type, access point type, access  decision location, testing mode throughput and transactions, reader  hardware model and software version, reader types (fixed versus  portable), interface type (contact versus contactless), communication  protocol, whether or not registration was used, the enrollment  process, the source of the biometric reference template, and canceled  card list input frequency by site;  comparison of the total number of access points identified during  baseline data collection versus the total of access points tested during  the EOA and ST&E phases of the pilot;  comparison of the mean, median, and mode based on the ST&E  number of throughput transactions; and assessment of testing duration during EOA and ST&E testing phases  for both throughput and transaction data collection efforts.
Why GAO Did This Study

Within DHS, TSA and USCG manage the TWIC program, which requires maritime workers to complete background checks and obtain biometric identification cards to gain unescorted access to secure areas of Maritime Transportation Security Act (MTSA)-regulated entities. TSA conducted a pilot program to test the use of TWICs with biometric card readers in part to inform the development of a regulation on using TWICs with card readers. As required by law, DHS reported its findings on the pilot to Congress on February 27, 2012. The Coast Guard Authorization Act of 2010 required that GAO assess DHS's reported findings and recommendations. Thus, GAO assessed the extent to which the results from the TWIC pilot were sufficiently complete, accurate, and reliable for informing Congress and the proposed TWIC card reader rule. GAO reviewed pilot test plans, results, and methods used to collect and analyze pilot data since August 2008, compared the pilot data with the pilot report DHS submitted to Congress, and conducted covert tests at four U.S. ports chosen for their geographic locations. The test's results are not generalizable, but provide insights.

What GAO Found

GAO's review of the pilot test aimed at assessing the technology and operational impact of using the Transportation Security Administration's (TSA) Transportation Worker Identification Credential (TWIC) with card readers showed that the test's results were incomplete, inaccurate, and unreliable for informing Congress and for developing a regulation (rule) about the readers. Challenges related to pilot planning, data collection, and reporting affected the completeness, accuracy, and reliability of the results. These issues call into question the program's premise and effectiveness in enhancing security.
Planning. The Department of Homeland Security (DHS) did not correct planning shortfalls that GAO identified in November 2009. GAO determined that these weaknesses presented a challenge in ensuring that the pilot would yield information needed to inform Congress and the regulation aimed at defining how TWICs are to be used with biometric card readers (card reader rule). GAO recommended that DHS components implementing the pilot--TSA and the U.S. Coast Guard (USCG)--develop an evaluation plan to guide the remainder of the pilot and identify how it would compensate for areas where the TWIC reader pilot would not provide the information needed. DHS agreed and took initial steps, but did not develop an evaluation plan, as GAO recommended.
Data collection . Pilot data collection and reporting weaknesses include:
Installed TWIC readers and access control systems could not collect required data, including reasons for errors, on TWIC reader use, and TSA and the independent test agent (responsible for planning, evaluating, and reporting on all test events) did not employ effective compensating data collection measures, such as manually recording reasons for errors in reading TWICs.
TSA and the independent test agent did not record clear baseline data for comparing operational performance at access points with TWIC readers.
TSA and the independent test agent did not collect complete data on malfunctioning TWIC cards.
Pilot participants did not document instances of denied access.
TSA officials said challenges, such as readers incapable of recording needed data, prevented them from collecting complete and consistent pilot data. Thus, TSA could not determine whether operational problems encountered at pilot sites were due to TWIC cards, readers, or users, or a combination of all three.
Issues with DHS's report to Congress and validity of TWIC security premise. DHS's report to Congress documented findings and lessons learned, but its reported findings were not always supported by the pilot data, or were based on incomplete or unreliable data, thus limiting the report's usefulness in informing Congress about the results of the TWIC reader pilot. For example, reported entry times into facilities were not based on data collected at pilot sites as intended. Further, the report concluded that TWIC cards and readers provide a critical layer of port security, but data were not collected to support this conclusion. For example, DHS's assumption that the lack of a common credential could leave facilities open to a security breach with falsified credentials has not been validated. Eleven years after initiation, DHS has not demonstrated how, if at all, TWIC will improve maritime security.

What GAO Recommends

Congress should halt DHSs efforts to promulgate a final regulation until the successful completion of a security assessment of the effectiveness of using TWIC. In addition, GAO revised the report based on the March 22, 2013, issuance of the TWIC card reader notice of proposed rulemaking.