DHS Processes for Evaluating Training Programs Could Be Better Documented and More Reliably Capture Costs

DHS components have documented processes in place for evaluating  their training programs and have used evaluation feedback to improve  their training offerings; however, their documented processes varied on  the extent to which selected attributes of an effective training evaluation  process were included. Further, DHS identified opportunities for  efficiencies and cost savings, but varying approaches for capturing  training costs across the department affect DHS’s ability to reliably  capture and track its training costs department-wide. However,  at DHS headquarters and at the component level, there are  inconsistencies in how training costs are captured across the department  that have made it a challenge to accurately and reliably capture these  costs across DHS. Variations in the  methods used to collect data can greatly affect the analysis of uniform,  high-quality data on the cost and delivery of training and development  programs. DHS Is Implementing a Department-wide Leader Development Framework, but Could Strengthen Its Program Assessment

DHS’s Leader Development Program Office is in the process of  implementing a department-wide, five-tier Leader Development  Framework to build leadership skills across all staff levels. However, the LDP Office could strengthen its performance measurement  efforts by clearly identifying its program goals and better incorporating key  attributes of successful performance measures we have previously  identified. However, the LDP Office has not clearly  identified goals for the program, and the 12 measures that the office has  developed to assess its performance do not consistently exhibit attributes  we have previously identified as key for successful measurement. By updating DHS components’ documented training  evaluation processes to more fully address key attributes for effective  training evaluation, DHS components could have better assurance that  the components have more complete information to guide their efforts in  conducting effective evaluations. Further, given limited budgetary resources, by identifying  existing challenges that prevent DHS from accurately capturing its  training costs department-wide and, to the extent that the benefits exceed  the costs, implementing corrective measures to overcome these  challenges, DHS could improve its awareness about the actual costs of  its training programs, and enhance its ability to consistently and reliably  capture training costs DHS-wide, thereby enhancing its resource  stewardship. As DHS begins to assess the impact of the LDP  program, clearly identifying LDP goals and ensuring that LDP  performance measures possess key attributes, including (1) linkage to the  program’s goals, (2) clarity, and (3) measurable targets by which to  assess the measures could help provide DHS with the actionable  information it needs to identify and make program improvements. Recommendations for Executive Action

To ensure effective evaluation of federal training programs and enhance  DHS’s stewardship of resources for federal training programs, we  recommend that the Secretary of Homeland Security take the following  two actions: direct DHS components to ensure that their documented training  evaluation processes fully address attributes for effective training  evaluation processes as they are drafted, updated, or revised and  identify existing challenges that prevent DHS from accurately capturing  training costs department-wide and, to the extent that the benefits of  addressing those challenges exceed the costs, implement corrective  measures to overcome these challenges. To address the first question, regarding the extent to which DHS has  documented processes to evaluate training and development programs,  and ensure training costs are reliably captured, we reviewed DHS and  component-specific documents and interviewed relevant officials at DHS  OCHCO and each of the components. Appendix II: Presence of Effective Training Attributes in DHS’s Documented Training Evaluation Processes

	Establishes goals about what the training program is supposed to achieve 

Component  U.S. Customs and  Border Protection  (CBP)  U.S. Immigration and  Customs  Enforcement (ICE)  Coast Guard  Transportation  Security  Administration  (TSA)Federal Law  Enforcement  Training Center  (FLETC)
Explains how  the  evaluation  results will be  used     .
Why GAO Did This Study

DHS is the third-largest cabinet-level department in the federal government, with over 230,000 employees doing diverse jobs. To fulfill its complex mission, DHS's workforce must have the necessary skills and expertise. GAO previously reported on DHS's hiring and recruiting efforts. GAO was asked to assess DHS's training practices.
This report addresses (1) the extent to which DHS has documented processes to evaluate training and reliably capture costs and (2) the extent to which DHS measures the performance of its leader development programs. To conduct its work, GAO reviewed documented training evaluation processes, training cost data from fiscal year 2011 through fiscal year 2013, and leadership training programs. GAO also interviewed training officials at the department level and at the five DHS components selected for this review about their varieties of training and development programs. Information from these components cannot be generalized to all of DHS, but provides insights.

What GAO Found

The Department of Homeland Security (DHS) has processes to evaluate training, track resources, and assess leader development. However, various actions could better position the department to maximize the impact of its training efforts.
Training evaluation: All five DHS components in GAO's review—U.S. Customs and Border Protection, U.S. Immigration and Customs Enforcement, the U.S. Coast Guard, the Transportation Security Administration, and the Federal Law Enforcement Training Center—have a documented process to evaluate their training programs. Their documented processes fully included three of six attributes of effective training evaluation processes identifying goals, programs to evaluate, and how results are to be used. However, the documented processes did not consistently include the other three attributes: methodology, timeframes, and roles and responsibilities (see table). By updating documentation to address these attributes, DHS components would have more complete information to guide its efforts in conducting effective evaluations.
Capturing training cost: DHS identified efficiencies and cost savings for delivering a number of training programs. However, different methods are used for capturing training costs across the department, which poses challenges for reliably capturing these costs across DHS. Components capture training costs differently, contributing to inconsistencies in training costs captured across DHS. Variation in methods used to collect data can affect the reliability and quality of DHS-wide training program costs. However, DHS has not identified all challenges that contribute to these inconsistencies. DHS could improve its awareness about the costs of training programs DHS-wide and thereby enhance its resource stewardship by identifying existing challenges that prevent DHS from accurately capturing training costs and implementing corrective measures.
Leader development: DHS's Leader Development Program (LDP) Office is in the process of implementing a department-wide framework to build leadership skills. However, the LDP Office has not clearly identified program goals and the measures it uses to assess program effectiveness do not exhibit some attributes that GAO previously identified as key for successful performance measurement. These include linkage of performance measures to the program's goals, clarity, and establishment of measurable targets to assess the measures. By clearly identifying program goals and incorporating key attributes, the LDP could better ensure actionable information for identifying and making program improvements.

What GAO Recommends

GAO recommends that DHS update its documentation to fully reflect key attributes of an effective evaluation, identify challenges to and corrective measures for capturing training costs department-wide, and clearly identify LDP goals and ensure that LDP performance measures reflect key attributes. DHS concurred and identified actions to address our recommendations.