FMCSA’s Role

FMCSA is responsible for overseeing this large and diverse industry. SMS uses data obtained from federal or  state roadside inspections and from crash investigations to identify the  highest risk carriers. Carriers potentially receive an SMS score in  seven categories based on this information. SMS Carrier Performance

SMS, the measurement system component of CSA, uses the data  collected from roadside inspections and crash reports to quantify a  carrier’s safety performance relative to other carriers. CSA Program Increases Carrier Interventions, but FMCSA Faces Challenges in Identifying High Risk Carriers

CSA has been successful in raising the profile of safety in the motor  carrier industry and providing FMCSA with more tools to increase  interventions with carriers. However, FMCSA faces two major challenges  in reliably assessing safety risk for the majority of carriers in the industry  and prioritizing the riskiest carriers for intervention. First, we found that  the majority of regulations used to calculate SMS scores are not violated  often enough to strongly associate them with crash risk for individual  carriers. From fiscal year 2007 to fiscal year 2012, FMCSA increased its number  of annual interventions from about 16,000 to about 44,000, largely by  sending warning letters to carriers deemed to be above the intervention  threshold in one or more BASICs (see table 3). For SMS to be effective in identifying carriers that crash, the  violation information that is used to calculate SMS scores should have a  relationship with crash risk. (For more information, see app. Most Carriers Lack Sufficient Information to Reliably Compare Safety Performance across Carriers

Most carriers lack sufficient safety performance information to ensure that  FMCSA can reliably compare them with other carriers. 4). Strengthened Data Sufficiency Standards Can Improve FMCSA’s Ability to Identify High Risk Carriers

Our analysis shows that FMCSA could improve its ability to identify  carriers at higher risk of crashing by applying a more stringent data  sufficiency standard. FMCSA’s existing SMS method  successfully identified as high risk more than 2,800 carriers whose  vehicles were involved in 12,624 crashes. Prioritizing resources to these carriers  would limit FMCSA’s ability to reduce the number of overall crashes,  resulting in lost opportunities to intervene with the carriers associated with  many crashes. In  setting a data sufficiency standard, FMCSA needs to consider how  precise the scores need to be, and a score’s required precision depends  on the purposes for which the scores are used. According to FMCSA, the  Safety Fitness Determination rulemaking would seek to allow FMCSA to  determine if a motor carrier is not fit to operate based on a carrier’s  performance in five of the BASICs, an investigation, or a combination of  roadside and investigative information. In addition to using SMS for internal purposes, FMCSA has also stated  that SMS provides stakeholders with valuable safety information, which  can “empower motor carriers and other stakeholders…to make safety- based business decisions.” publicly released SMS scores stating that the data are intended for  agency and law enforcement purposes, and readers should not draw  safety conclusions about a carrier’s safety condition based on the SMS  score, but rather the carrier’s official safety rating. GAO continues to believe a data-driven, risk-based approach holds  promise and can help FMCSA effectively identify carriers exhibiting  compliance or safety issues—such as violations or involvement in  crashes. Our analysis shows how  improving the reliability of SMS scores by strengthening data sufficiency  standards could better account for limitations in available safety  performance information and help FMCSA better focus intervention  resources where they can have the greatest impact on reducing crashes. Recommendations for Executive Action

To improve the CSA program, the Secretary of Transportation should  direct the FMCSA Administrator to take the following two actions:  Revise the SMS methodology to better account for limitations in drawing  comparisons of safety performance information across carriers; in doing  so, conduct a formal analysis that specifically identifies:  limitations in the data used to calculate SMS scores including  variability in the carrier population and the quality and quantity of data  available for carrier safety performance assessments, and  limitations in the resulting SMS scores including their precision,  confidence, and reliability for the purposes for which they are used. Appendix I: Scope and Methodology

This report addresses the effectiveness of the Compliance, Safety,  Accountability (CSA) program in assessing safety risk for motor carriers. Given FMCSA’s use of these scores as  quantitative determinations of a carrier’s safety performance, we  assessed the reliability of SMS scores as defined by the precision,  accuracy, and confidence of these scores when calculated for carriers  with varying levels of carrier exposure—measured by FMCSA as either  inspections or an adjusted number of vehicles. Using this  analysis, we designed an illustrative alternative method that incorporates  the following changes:  including only carriers with at least 20 observations in the following  measures of exposure: driver inspections when calculating scores for the Hours-of- Service Compliance, Driver Fitness, and Controlled Substances  BASICs;  vehicle related inspections for the Vehicle Maintenance BASIC;  vehicle related inspections where placardable quantities of  hazardous materials are being transported for Hazardous  Materials BASIC; and average power units for the Unsafe Driving and Crash Indicator assigning an SMS score to any carrier meeting these data sufficiency  standards (e.g., 20 inspections), even if that carrier does not have any  violations, was free of violations for 12 months, or had a clean last  inspection; eliminating safety event groups because of the stricter data sufficiency using only the average number of vehicles as the measure of  exposure for carrier’s assessed in the Unsafe Driving and Crash  Indicator BASICs.
Why GAO Did This Study

From 2009 to 2012, large commercial trucks and buses have averaged about 125,000 crashes per year, with about 78,000 injuries and over 4,100 fatalities. In 2010, FMCSA replaced its tool for identifying the riskiest carriers--SafeStat--with the CSA program. CSA is intended to reduce the number of motor carrier crashes by better targeting the highest risk carriers using information from roadside inspections and crash investigations. CSA includes SMS, a data-driven approach for identifying motor carriers at risk of causing a crash.
GAO was directed by the Consolidated Appropriations Act of 2012 to monitor the implementation of CSA. This report examines the effectiveness of the CSA program in assessing safety risk for motor carriers. GAO spoke with FMCSA officials and stakeholders to understand SMS. Using FMCSA's data, GAO replicated FMCSA's method for calculating SMS scores and assessed the effect of changes--such as stronger data-sufficiency standards--on the scores. GAO also evaluated SMS's ability to predict crashes.

What GAO Found

The Federal Motor Carrier Safety Administration's (FMCSA) Compliance, Safety, Accountability (CSA) program has helped the agency contact or investigate more motor carrier companies that own commercial trucks and buses and has provided a range of safety benefits to safety officials, law enforcement, and the industry than the previous approach, SafeStat. Specifically, from fiscal year 2007 to fiscal year 2012, FMCSA more than doubled its number of annual interventions, largely by sending warning letters to riskier carriers.
A key component of CSA--the Safety Measurement System (SMS)--uses carrier performance data collected from roadside inspections or crash investigations to identify high risk carriers for intervention by analyzing relative safety scores in various categories, including Unsafe Driving and Vehicle Maintenance. FMCSA faces at least two challenges in reliably assessing safety risk for the majority of carriers. First, for SMS to be effective in identifying carriers more likely to crash, the violations that FMCSA uses to calculate SMS scores should have a strong predictive relationship with crashes. However, based on GAO's analysis of available information, most regulations used to calculate SMS scores are not violated often enough to strongly associate them with crash risk for individual carriers. Second, most carriers lack sufficient safety performance data to ensure that FMCSA can reliably compare them with other carriers. To produce an SMS score, FMCSA calculates violation rates for each carrier and then compares these rates to other carriers. Most carriers operate few vehicles and are inspected infrequently, providing insufficient information to produce reliable SMS scores. FMCSA acknowledges that violation rates are less precise for carriers with little information, but its methods do not fully address this limitation. For example, FMCSA requires a minimum level of information for a carrier to receive an SMS score; however, this requirement is not strong enough to produce sufficiently reliable scores. As a result, GAO found that FMCSA identified many carriers as high risk that were not later involved in a crash, potentially causing FMCSA to miss opportunities to intervene with carriers that were involved in crashes.
FMCSA's methodology is limited because of insufficient information, which reduces the precision of SMS scores. GAO found that by scoring only carriers with more information, FMCSA could better identify high risk carriers likely to be involved in crashes. This illustrative approach involves trade-offs; it would assign SMS scores to fewer carriers, but these scores would generally be more reliable and thus more useful in targeting FMCSA's scarce resources.
In addition to using SMS scores to prioritize carriers for intervention, FMCSA reports these scores publicly and is considering using a carrier's performance information to determine its fitness to operate. Given the limitations with safety performance information, determining the appropriate amount of information needed to assess a carrier requires consideration of how reliable and precise the scores need to be for the purposes for which they are used. Ultimately, the mission of FMCSA is to reduce crashes, injuries, and fatalities. GAO continues to believe a data-driven, risk-based approach holds promise; however, revising the SMS methodology would help FMCSA better focus intervention resources where they can have the greatest impact on achieving this goal.

What GAO Recommends

GAO recommends that FMCSA revise the SMS methodology to better account for limitations in drawing comparisons of safety performance information across carriers. In addition, determination of a carrier's fitness to operate should account for limitations in available performance information. In response to comments from the Department of Transportation (USDOT), GAO clarified one of the recommendations. USDOT agreed to consider the recommendations.