FMCSA uses  enforcement as its primary approach for reducing the number of crashes,  fatalities, and injuries involving trucks and buses. In contrast, fewer than 1 percent of  the carriers listed in MCMIS have more than 100 vehicles, and nearly 25  percent of them received a SafeStat rating in categories A through G.

	A Statistical Approach Would Better Identify Carriers That Pose High Crash Risks Than Does FMCSA’s Current Approach

We found that FMCSA could improve SafeStat’s ability to identify carriers  that pose high crash risks if it applied a statistical approach, called a  negative binomial regression model, to the four SafeStat safety evaluation  areas instead of its current approach. Targeting these high-risk carriers would result in FMCSA giving  compliance reviews to carriers that experienced both a higher crash rate  and, in conjunction with the higher crash rate, 9,500 more crashes over an  18-month period than those identified by the SafeStat model. Regression Models Identify Carriers That Pose High Crash Risks Better Than Expert Judgment

Although SafeStat is nearly twice as effective as (83 percent better than)  random selection in identifying carriers that pose high crash risks and,  therefore, has value for improving safety, we found that FMCSA could  improve SafeStat’s ability to identify such carriers by about 9 percent if it  applied a negative binomial regression model approach to its analysis of  motor carrier safety data. FMCSA would choose the  number of carriers to review based on the resources available to it, much  as it currently does. Selecting carriers for roadside inspections. Despite the problems of late-reported crashes and incomplete  and inaccurate data on crashes during the period we studied, we  determined that the data were of sufficient quality for our use, which was  to assess how the application of regression models might improve the  ability to identify high-risk carriers over the current approach—not to  determine absolute measures of crash risk. FMCSA has recently undertaken a number of  efforts to improve crash data quality. In addition, the  percentage of crashes reported by states within 90 days of occurrence has  jumped from 32 percent in fiscal year 2000 to 89 percent in fiscal year  2006. 2.) As a result,  FMCSA could not match these crashes to motor carriers or use them in  SafeStat. To do so would  have required us to gather crash records at the state level—an effort that  was impractical. Inaccurate Data Potentially Limit SafeStat’s Ability to Identify Carriers That Pose High Crash Risks

Inaccurate data, such as reporting a nonqualifying crash to FMCSA,  potentially has a large impact on a motor carrier’s SafeStat score because  SafeStat treats crashes as the most important source of information for  assessing motor carrier crash risk. The University of Michigan Transportation Research Institute’s reports on  crash reporting show that, among the 14 states studied, incorrect reporting  of crash data is widespread. While the quality of crash reporting is sufficient for use in identifying  motor carriers that pose high crash risks and has started to improve,  commercial motor vehicle crash data continue to have some problems  with timeliness, completeness, and accuracy. Using a negative binomial regression model would further FMCSA’s  mission of reducing crashes through the more effective targeting of  compliance reviews to the set of carriers that pose the greatest crash risks. Impact of Data Quality on SafeStat’s Predictive Capability

The 2004 Office of Inspector General report, the 2004 Oak Ridge study, and  reports by the University of Michigan Transportation Research Institute on  state crash reporting all examined the impact of data quality on SafeStat’s  ability to identify carriers that pose high crash risks. We also assessed the extent of late reporting. MCMIS data  change over time because crash data are added, deleted, or changed as  more information about these crashes is obtained. Since  the primary use of SafeStat is to identify and prioritize carriers for  FMCSA and state compliance reviews, the empirical Bayes method did  not identify carriers with the highest safety risk.
Why GAO Did This Study

The Federal Motor Carrier Safety Administration (FMCSA) has the primary federal responsibility for reducing crashes involving large trucks and buses that operate in interstate commerce. FMCSA decides which motor carriers to review for compliance with its safety regulations primarily by using an automated, data-driven analysis model called SafeStat. SafeStat uses data on crashes and other data to assign carriers priorities for compliance reviews. GAO assessed (1) the extent to which changes to the SafeStat model could improve its ability to identify carriers that pose high crash risks and (2) how the quality of the data used affects SafeStat's performance. To carry out its work, GAO analyzed how SafeStat identified high-risk carriers in 2004 and compared these results with crash data through 2005.

What GAO Found

While SafeStat does a better job of identifying motor carriers that pose high crash risks than does a random selection, regression models GAO applied do an even better job. SafeStat works about twice as well as (about 83 percent better than) selecting carriers randomly. SafeStat is built on a number of expert judgments rather than using statistical approaches, such as a regression model. For example, its designers decided to weight more recent motor carrier crashes twice as much as less recent ones on the premise that more recent crashes were stronger indicators of future crashes. GAO estimates that if FMCSA used a negative binomial regression model, FMCSA could increase its ability to identify high-risk carriers by about 9 percent over SafeStat. Carriers identified by the negative binomial regression model as posing a high crash risk experienced 9,500 more crashes than those identified by the SafeStat model over an 18 month follow-up period. The primary use of SafeStat is to identify and prioritize carriers for FMCSA and state compliance reviews. FMCSA measures the ability of SafeStat to perform this role by comparing the crash rate of carriers identified as posing a high crash risk with the crash rate of other carriers. Using a negative binomial regression model would further FMCSA's mission of reducing crashes through the more effective targeting of compliance reviews to the set of carriers that pose the greatest crash risk. Late-reported, incomplete, and inaccurate data reported to FMCSA by states have been a long-standing problem. However, GAO found that late reported data had a small effect on SafeStat's ability to identify carriers that pose high crash risks in 2004. If states had reported all crash data within 90 days after occurrence, as required by FMCSA, a net increase of 299 carriers (or 6 percent) would have been identified as posing high crash risks of the 4,989 that FMCSA identified. Reporting timeliness has improved, from 32 percent of crashes reported on time in fiscal year 2000, to 89 percent in fiscal year 2006. Regarding completeness, GAO found that data for about 21 percent of the crashes (about 39,000 of 184,000) exhibited problems that hampered linking crashes to motor carriers. Having complete information on crashes is important because SafeStat treats crashes as the most important factor for assessing motor carrier crash risk, and crash information is also the crucial factor in the statistical approaches that we employed. Regarding accuracy, a series of studies by the University of Michigan Transportation Research Institute covering 14 states found incorrect reporting of crash data is widespread. GAO was not able to quantify the effect of the incomplete or inaccurate data on SafeStat's ability to identify carriers that pose high crash risks because it would have required gathering crash records at the state level--an effort that was impractical for GAO. FMCSA has acted to improve crash data quality by completing a comprehensive plan for data quality improvement, implementing an approach to correct inaccurate data, and providing grants to states for improving data quality, among other things.