Information Technology Delays Have Hampered Full Implementation of CSA Interventions and Limited the Quality of Available Information

		FMCSA Has Delayed Implementing Two New CSA Intervention Types Nationwide Until It Develops Information Technology Software

Although FMCSA implemented all four new CSA intervention types in  pilot test states, the agency chose to delay implementing two of the new  intervention types in the remaining states until it develops information  technology (IT) software. However, FMCSA has faced longstanding delays in developing  SENTRI  software as part of its broader IT modernization effort. As part of this effort,  FMCSA plans to complete its development of SENTRI by April 2017. To evaluate the effectiveness of CSA interventions  specifically, FMCSA developed a statistical model intended to annually  evaluate the combined effects of all of its interventions. III  for our complete assessment). This  specific information could help FMCSA identify the circumstances under  which different types of interventions are effective and help managers  optimize their choice of interventions on a day-to-day basis as the agency  implements the program. Although officials told us that FMCSA designed the model to measure the  cumulative effect of FMCSA contact with carriers through CSA  interventions and not to analyze individual  intervention types, FMCSA  used the model to draw conclusions about the safety benefits of warning  letters, one of the most common intervention types used according to  FMCSA estimates. By not  matching the measurement period, FMCSA’s use of a comparison group  was limited in its ability to account for external factors, as intended. The evaluation concluded that cooperative safety plans  and notices of violation had the lowest average estimated costs—ranging  from $95 to $118 respectively. For  example, in April 2013, FMCSA implemented a substantial change to  the way investigators conduct investigations, called EIT, which is  intended to help identify the root cause of motor carriers’ safety  problems. The evaluation’s estimates do not represent costs in all states. However, without cost estimates for CSA interventions that  are current and representative of all states, FMCSA lacks information it  needs to understand the most efficient methods of conducting CSA  interventions in all states. FMCSA Took Some Steps Intended to Improve CSA Intervention Outcomes, but Lacks Measures to Monitor Progress

FMCSA has taken steps intended to improve the effectiveness and  efficiency of CSA interventions—strategic outcomes—principally by  establishing a working group to address these issues and implementing  some of the group’s recommendations. Without establishing  measures for both outcomes, FMCSA is and will be limited in its ability to  monitor program performance and to balance these two priorities, as  needed. FMCSA headquarters officials told us  that effectiveness and efficiency are complementary outcomes that  FMCSA strives to balance. Because FMCSA does not have a complete set of measures that reflects  both its effectiveness and efficiency outcomes for CSA interventions,  FMCSA managers lack benchmarks needed to regularly monitor progress  toward achieving the outcomes. FMCSA aims to reduce such  fatalities by using a data-driven approach that identifies and intervenes  with the highest-risk motor carriers early. In addition, we found that although FMCSA has established some  performance measures for its effectiveness outcomes, the agency has  not established measures to monitor progress toward achieving its  efficiency outcome. To understand the efficiency of CSA interventions the Secretary of  Transportation should direct the FMCSA Administrator to:
Update FMCSA’s cost estimates to determine the resources currently  used to conduct individual intervention types and ensure FMCSA has  cost information that is representative of all states. GAO staff who made major contributions to  this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

The objectives of our report were to examine: (1) the extent to which the  Federal Motor Carrier Safety Administration (FMCSA) has implemented  Compliance, Safety, Accountability (CSA) interventions and how it has  applied them; (2) the extent to which FMCSA has evaluated the  effectiveness and efficiency of CSA interventions; and (3) any steps that  FMCSA has taken to improve and monitor progress toward achieving its  intended outcomes for CSA interventions. In addition, we interviewed FMCSA division officials in eight states that  we selected based upon their participation in FMCSA’s Operational Model  Test of the CSA program, geographic location, and program size, among  other factors. Similarly, we selected two states from  each service center. Specifically, we reviewed:
The University of Michigan Transportation Research Institute,  Evaluation of the CSA 2010 Operational Model Test (August 2011);
FMCSA, FMCSA Safety Program Effectiveness Measurement: Carrier  Intervention Effectiveness Model, Version 1.0: Summary Report for  Fiscal Years 2009, 2010, 2011 (January 2015);
FMCSA, Analysis Brief: Effectiveness of Onsite Focused  Investigations (March 2016); and
FMCSA, Effectiveness of Offsite Investigations: Preliminary Analysis  (March 2016)
We conducted a more detailed assessment of the second report on  FMCSA’s annual effectiveness evaluation model because, according to  FMCSA headquarters officials, it is the primary method FMCSA uses to  evaluate intervention effectiveness and FMCSA intends to use it on a  recurring basis.
Why GAO Did This Study

As part of its mission to reduce crashes and fatalities involving large commercial trucks and buses, FMCSA seeks to use a data-driven approach to identify the highest-risk motor carriers and address safety problems by applying a range of eight CSA program enforcement tools, called interventions, ranging from warning letters to placing carriers out of service.
A provision in a Senate report requires GAO to periodically assess FMCSA's implementation of the CSA program. This report examines the extent to which FMCSA has (1) implemented CSA interventions, (2) evaluated the effectiveness and efficiency of CSA interventions, and (3) monitored progress toward achieving outcomes.
GAO reviewed FMCSA data and documentation on all eight CSA intervention types from fiscal years 2010–2015, including FMCSA's strategic planning documents, guidance, and program evaluations. GAO interviewed industry stakeholders and FMCSA officials in headquarters, in each of FMCSA's service centers, and in eight states selected for their participation in FMCSA's CSA pilot test, location, and program size, among other factors.

What GAO Found

In July 2010, the Federal Motor Carrier Safety Administration (FMCSA) chose to delay nationwide implementation of two of the eight interventions that FMCSA uses to address motor carrier safety concerns under its Compliance, Safety, Accountability (CSA) program. This delay is linked to continuing delays in developing software needed to support the two interventions, offsite investigations and the use of cooperative safety plans. The software under development is intended to help FMCSA overcome some of the information challenges it faces due to its reliance on legacy information systems. FMCSA estimates that the software development project will be completed by April 2017.
FMCSA has conducted evaluations of the effectiveness and efficiency outcomes it established for the CSA program. However, GAO identified several limitations in FMCSA's approaches that impact the usefulness of the evaluations:
Intervention effectiveness: FMCSA has developed a statistical model to annually evaluate the combined effectiveness of interventions. Although the model has some key strengths, such as accounting for a broad range of external factors, GAO identified a number of design and methodology limitations that reduce the usefulness of its results. For example, the model does not include an assessment of individual intervention types. Without this type of specific information, FMCSA is hampered in its ability to identify the circumstances under which different types of interventions are effective. Similarly, these types of limitations affect FMCSA's ability to accurately draw conclusions about intervention effectiveness across all intervention types.
Intervention efficiency: To assess the efficiency of CSA interventions, FMCSA has relied on a study that it sponsored and that was published in 2011. This study estimated the average cost of conducting interventions in four states from October 2008 through May 2009. However, FMCSA has not taken steps to update its cost estimates for interventions since the 2011 evaluation, despite changes since that time in the resources needed to conduct CSA interventions; nor has it taken steps to develop additional information that is representative of the costs in other states. Without current cost estimates that are representative of all states, FMCSA cannot appropriately assess the efficiency of its interventions.
FMCSA has taken some actions to improve the effectiveness and efficiency of CSA interventions, but lacks measures to monitor progress. In April 2014, FMCSA established a working group to assess CSA interventions and make recommendations for improvement. As of April 2016, the group had made 20 recommendations, of which 12 had been implemented. However, GAO found that while FMCSA has established some performance measures for its effectiveness outcome that are appropriate, it has not established similar measures for its efficiency outcome. FMCSA headquarters officials told GAO that effectiveness and efficiency are complementary outcomes that FMCSA strives to balance. Without a complete set of measures for both outcomes, FMCSA lacks benchmarks needed to regularly measure progress to achieve these outcomes.

What GAO Recommends

GAO recommends that FMCSA evaluate the effectiveness of individual intervention types, update cost estimates so that they are current and representative of all states, and establish complete performance measures.
The Department of Transportation concurred with all of GAO's recommendations.