TSA Assesses Air Marshals’ Training Needs by Considering Several Information Sources, but Opportunities Exist to Obtain More Feedback from Air Marshals

		TSA Assesses Its Federal Air Marshal Training Program Using Multiple Information Sources

TSA’s primary method for assessing air marshals’ training needs is by  holding Curriculum Development Conferences (CDC) and Curriculum  Review Conferences (CRC). OTD Could Improve its Collection of Feedback from Air Marshals to Fully Identify and Address Training Needs

			OTD Does Not Systematically Collect Training Feedback from Incumbent Air Marshals on Field-Based Training

OTD conducts surveys to obtain feedback from air marshal candidates  and newly graduated air marshals on the effectiveness of FAMTP  courses they complete at TSATC and the quality of TSATC trainers and  facilities, consistent with Kirkpatrick levels 1 and 3. TSA Has Established Recurrent Training Requirements to Ensure Mission Readiness, but Opportunities for Improvement Exist

FAMS relies on its recurrent training program to help ensure incumbent  air marshals’ mission readiness, but additional actions could strengthen  FAMS’s ability to do so. First, FAMS does not have complete and timely  data on the extent to which air marshals have fulfilled their recurrent  training requirements. In addition, FAMS has established a new  health, fitness, and wellness program as part of its recurrent training  program—in part to address recent concerns with air marshals’ fitness  and injury rates—but it is too early to gauge the program’s effectiveness. Given the number of training records that we found were incomplete or  not entered into FAMIS in a timely manner, as well as the ongoing  challenges that FAMS has faced in ensuring accurate and timely input of  training and exemptions data as described in the OOI findings, policies  that specify who is responsible at the headquarters level for overseeing  these activities could help FAMS ensure its data on air marshals’  recurrent training are consistently accurate and up to date. FAMS Requires Air Marshals to Demonstrate Their Proficiency in Marksmanship, but Does Not Require That They Do So for Other Key Skills

Air marshals must demonstrate their proficiency in marksmanship by  taking the practical pistol course on a quarterly basis and achieving a  minimum score of 255 out of 300—the highest qualification standard for  any federal law enforcement agency, according to FAMS officials. However, for the remainder of air marshals’ required recurrent training  courses, FAMS does not assess air marshals against a similarly identified  level of proficiency, such as by requiring examinations to evaluate air  marshals’ knowledge in classroom-based courses or by using checklists  or other objective tools to evaluate air marshals’ performance during  simulation-based courses, such as mission tactics. Objective and standardized methods of  evaluating incumbent air marshals performance would better enable  FAMS to assess air marshals’ proficiency in key skills and also more  effectively target areas for improvement. Also, by taking additional steps to improve the response rates  for the training surveys it administers to air marshal candidates,  incumbent air marshals, and their supervisors, OTD could be more  reasonably assured that the feedback it receives represents the full  spectrum of views held by its air marshal workforce. Recommendations for Executive Action

To ensure effective evaluation of air marshal training, we recommend that  the TSA Administrator direct OTD to take the following two actions:  implement a mechanism for regularly collecting and incorporating  incumbent air marshals’ feedback on the training they receive from  field office programs, and  take additional steps to improve the response rates of the training  surveys it conducts. To provide reasonable assurance that air marshals are complying with  recurrent training requirements and have the capability to carry out  FAMS’s mission, we recommend the TSA Administrator direct FAMS to  take the following three actions:  specify in policy who at the headquarters level has oversight  responsibility for ensuring that field office SACs or their designees  meet their responsibilities for ensuring that training completion records  are entered in a timely manner,   specify in policy who at the headquarters level is responsible for  ensuring that headquarters personnel enter approved air marshals’  training exemptions into FAMIS, and define the timeframe for doing  so, and develop and implement standardized methods, such as examinations  and checklists, for determining whether incumbent air marshals  continue to be mission ready in key skills. To what extent does the Federal Air Marshal Service (FAMS) ensure  that incumbent air marshals are mission ready? This report is a public version of the prior sensitive report that we  provided to you. We interviewed senior officials responsible for these efforts in  TSA’s Office of Training and Development (OTD). Additionally, we obtained the available response  rates for surveys OTD conducted of FAMTP graduates and their  supervisor’s on the effectiveness of FAMTP courses for calendar years  2009 through 2011—the last three full years that FAMS hired air  marshals. To address the second objective, we assessed FAMS directives that set  forth training requirements for incumbent air marshals, and analyzed air  marshals training data for calendar year 2014, which is the most recent  year for which training data were available, to determine the extent to  which air marshals met these requirements.
Why GAO Did This Study

FAMS, within TSA, is the federal entity responsible for promoting confidence in the nation's aviation system through deploying air marshals to protect U.S. air carriers, airports, passengers, and crews.
GAO was asked to assess FAMS's training program for federal air marshals. This report examines (1) how TSA assesses the training needs of air marshal candidates and incumbent air marshals, and any opportunities that exist to improve this assessment, and (2) the extent to which FAMS ensures that incumbent air marshals are mission ready.
GAO analyzed FAMS training data for calendar year 2014, the last year of available data, reviewed TSA, OTD and DHS guidance and policies on FAMS's air marshal training program, interviewed TSA and FAMS headquarters officials, and visited the TSA Training Center and 7 of FAMS 22 field offices selected based on size and geographic dispersion.

What GAO Found

The Transportation Security Administration's (TSA) Office of Training and Development (OTD) assesses air marshals' training needs using several information sources, but opportunities exist to obtain more feedback from air marshals on whether the training courses they must take met their needs. OTD primarily assesses air marshals' training needs by holding curriculum development and review conferences composed of OTD officials, training instructors, and other subject matter experts. In assessing courses, conference participants use, among other things, the results of surveys that some air marshals complete on the effectiveness of their training. However, while OTD administers these surveys for air marshal candidates and newly graduated air marshals, it does not use them to obtain feedback from incumbent air marshals on the effectiveness of their annual recurrent training courses. Systematically gathering feedback from incumbent air marshals would better position OTD to fully assess whether the training program is meeting air marshals' needs. Additionally, among the training surveys that OTD does currently administer to air marshals, the response rates have been low. For example, among newly hired air marshals and their supervisors from 2009 through 2011—the last three full years in which the Federal Air Marshal Service (FAMS) hired air marshals—the survey response rates ranged from 16 to 38 percent. Until OTD takes steps to achieve sufficient response rates, OTD cannot be reasonably assured that the feedback it receives represents the full spectrum of views held by air marshals.
FAMS relies on its annual recurrent training program to ensure incumbent air marshals' mission readiness, but additional actions could strengthen FAMS's ability to do so. First, FAMS does not have complete and timely data on the extent to which air marshals have completed their recurrent training. For example, nearly one-quarter of all training records for calendar year 2014 had not been entered into FAMS's training database within the required time period. Policies that specify who is responsible at the headquarters level for overseeing these activities could help FAMS ensure its data on air marshals' recurrent training are accurate and up to date. Second, FAMS requires air marshals to demonstrate proficiency in marksmanship by achieving a minimum score of 255 out of 300 on the practical pistol course every quarter. However, for the remaining recurrent training courses FAMS does not assess air marshals' knowledge or performance in these courses against a similarly identified level of proficiency, such as by requiring examinations or by using checklists or other objective tools. More objective and standardized methods of determining incumbent air marshals' mission readiness, as called for by the Department of Homeland Security's (DHS) Learning Evaluation Guide, could help FAMS better and more consistently assess air marshals' skills and target areas for improvement. Additionally, in 2015 FAMS developed a health, fitness, and wellness program to improve air marshals' overall health and wellness, but it is too early to gauge the program's effectiveness. This is a public version of a sensitive report that GAO issued in June 2016. Information that TSA deems “Sensitive Security Information” has been removed.

What GAO Recommends

GAO recommends that OTD implement a mechanism for regularly collecting incumbent air marshals' feedback on their recurrent training, and take steps to improve the response rates of training surveys it conducts. GAO also recommends that FAMS specify in policy who at the headquarters level has oversight responsibility for ensuring that recurrent training records are entered in a timely manner, and develop and implement standardized methods to determine whether incumbent air marshals continue to be mission ready in key skills. DHS concurred with all of the recommendations.