The Field Study Followed Several Accepted Research Standards, but Did Not Report Its Limitations or Make Conclusions That Were Fully Linked to the Results

Based on our prior work evaluating research programs, our internal  expertise in research design, and established guidelines and reports for  conducting research and program evaluations, we identified six generally  accepted research standards that are critical for designing, analyzing, and  reporting the results of scientific research. Without such  guidance, FMCSA research may not include critical elements, as  occurred in the reporting of the field study. FMCSA Used Several Key Assumptions to Estimate the Potential Effects of the Rule

As part of its rule-making process, FMCSA developed assumptions about  the motor carrier industry and its operations to estimate the economic  costs and the safety and health benefits of the 2011 HOS rule. For  example, FMCSA assumed that drivers working more than 65 hours per  week would incur the majority of schedule changes as a result of the  rule. Further, FMCSA assumed that reduced work  time would increase a driver’s opportunity to sleep leading to safety and  health benefits. Analysis of Available Data Provides Some Insight into the Effects of the Rule

Analyzing available data can help provide some insight into the rule’s  potential effects and the extent to which they aligned with FMCSA’s  assumptions and estimates. The results from our analysis of schedule data from 16 for-hire carriers is  consistent with FMCSA’s assumption that drivers working more than 65  hours per work week would be more likely to reduce their work hours. Which drivers were affected: Our analysis of 2012 to 2014 driver  schedule data from 16 for-hire motor carriers is consistent with FMCSA’s  expectation that the percentage of drivers working the longest hours— over 65 hours per 8-day work week—decreased after the rule went into  effect on July 1, 2013. Therefore, our findings cannot be  generalized to the motor carrier industry as a whole. However, the literature is less clear on which  HOS interventions (e.g., rest breaks and limiting driving and on-duty time)  best minimize fatigue and reduce crash risk. First, we used the model to compare hypothetical driver  schedules that complied with two of the 2011 HOS rule provisions—the  168-hour limit and the two-night provision—to similar schedules that did  not comply with the rule but were in compliance with the previous HOS  rule.the rule went into effect would result in lower fatigue scores and,   We found that some schedule changes that would be required after  therefore, a lower risk of driver fatigue.schedules we modelled that had to change after July 1, 2013, in order to  comply with the rule were:
Maximum Allowed-Hours Day and Night schedules: 14-hour shifts  over 5 consecutive days or nights. Lack of Representative Driver Schedule Data Limits Analysis of the Effects of the HOS Rule

The ability of FMCSA and others to assess the effects of the 2011 HOS  rule is impacted by the limited availability and representativeness of driver  schedule data (i.e., records of drivers’ work hours). No organization,  including FMCSA, collects or maintains a centralized database with  representative driver schedule data that can be generalized to the entire  motor-carrier industry. Collecting these types of data has historically been difficult, but a recent  statutory change to how driver schedule data will be collected provides a  potential opportunity to do so. However, FMCSA officials said that they do not  currently plan to collect and use such data for research purposes  because MAP-21 limits their use to the enforcement of laws. There may be ways of mitigating these privacy and cost concerns, but  FMCSA has not examined the costs and benefits of collecting electronic  driver-schedule data on a large-scale and currently does not plan to do  so. For example, to address privacy concerns, information can be “de- identified” for the purposes of analyzing data. While electronically  collected, representative schedule data that will soon be available to  FMCSA would allow it to assess the impact of its rules, MAP-21 places  limitations on the use of this data for purposes other than enforcing laws. Given the potential value of such  data for evaluating the impact of future regulatory rules, Congress may  benefit from information on how the electronic data to be collected in  response to the MAP-21 requirements could be extracted, stored, and  analyzed and how privacy and cost concerns associated with the use of  these data could be addressed. Matter for Congressional Consideration

Congress may wish to consider directing DOT to study and provide a  report to Congress identifying approaches for extracting, storing, and  analyzing electronically collected motor carrier drivers’ schedule data,  including the potential benefits, privacy, and cost concerns, and options  for how such concerns could be mitigated. 2. Appendix III: GAO Identified Assumptions in the Hours of Service Rule’s Regulatory Impact Analysis

As part of our review of the effects of the 2011 hours of service (HOS)  rule, we were asked to identify the assumptions used by the Federal  Motor Carrier Safety Administration (FMCSA) to estimate the costs and  benefits of the rule. These assumptions cover which drivers would be affected by  the HOS rule and how they would be affected.
Why GAO Did This Study

FMCSA—within the Department of Transportation (DOT)—issues rules to address safety concerns of the motor carrier industry, including on truck drivers' HOS. In July 2013, FMCSA began to enforce three new provisions of its HOS rule. GAO was asked to review a 2014 FMCSA study on the rule, as well as the rule's assumptions and effects. This report (1) compares the study to generally accepted research standards, and (2) identifies the assumptions used to estimate the rule's costs and benefits and the rule's driver-operation, economic, safety, and health effects.
GAO identified research standards that professional associations, academics, and GAO's prior work have used. GAO evaluated the 2014 FMCSA study against these standards. GAO also compared FMCSA's assumptions about how drivers would be affected by the HOS rule against actual drivers' schedule data from 16 for-hire carriers that cover the years 2012 through 2014. These data include information on over 15,000 drivers per year, but are not generalizable to the motor carrier industry as a whole.

What GAO Found

GAO found that the January 2014 study issued by the Federal Motor Carrier Safety Administration (FMCSA) to examine the efficacy of its hours of service (HOS) rule—a regulation that governs how many hours truck drivers transporting freight can work—followed most generally accepted research standards. However, FMCSA did not completely meet certain research standards such as reporting limitations and linking the conclusions to the results. For example, by not adhering to these standards, FMCSA's conclusion in the study about the extent to which crash risk is reduced by the HOS rule may be overstated. GAO found that FMCSA has not adopted guidance on the most appropriate methods for designing, analyzing, and reporting the results of scientific research. Without such guidance, FMCSA may be at risk for excluding critical elements in research it undertakes to evaluate the safety of its rules, leaving itself open to criticism.
FMCSA made several assumptions and anticipated certain effects of the HOS rule in the regulatory impact analysis. Specifically, to estimate the economic costs of the rule, FMCSA assumed that some drivers would lose a certain amount of driving and on-duty time and then estimated the amount and cost of the work time lost. Further, FMCSA assumed that reduced work time could increase a driver's opportunity to sleep, leading to safety and health benefits. Assessing the effectiveness of the HOS rule is difficult because of the limited availability of representative driver schedule data (i.e., records of drivers' work hours). Nevertheless, GAO's analysis of a limited sample of available data provides some insight into the rule's effects and the extent to which they aligned with FMCSA's assumptions and estimates. For example, according to GAO's analysis, some drivers at a sample of 16 for-hire carriers who worked the longest hours (over 65 hours per work week) reduced their work hours after the rule went into effect, a finding consistent with FMCSA's assumptions that drivers working over 65 hours were more likely to be affected. However, GAO's analysis found that drivers who worked less than 65 hours per work week also changed their schedules after the rule went into effect, a result not anticipated by FMCSA.
The ability of FMCSA and others to assess the effects of rules, such as the 2011 HOS rule, is impacted by the limited availability of representative driver schedule data. No organization collects or maintains a centralized database with such data that can be generalized to the motor carrier industry as a whole. Collecting schedule data has historically been difficult, but a recent statutory change that requires carriers to electronically record and store these data provides a potential data source for the future. However, before these data can be used for research purposes several challenges would have to be addressed. First, there are statutory limits on the use of these data for purposes other than enforcing motor carrier safety regulations. Additionally, privacy and cost concerns must be resolved before these data could be made available for analysis. According to FMCSA officials, they do not plan to study how to use these data in a way that will address privacy and cost concerns, in part, because of the statutory limits. Given the potential value of these data to future regulatory analysis, it may be important to provide Congress with information on how these data can be extracted, stored, and analyzed while addressing any privacy and cost concerns.

What GAO Recommends

GAO recommends that FMCSA adopt guidance outlining agency research standards. FMCSA agreed with GAO's recommendation. GAO also suggests that Congress consider directing DOT to study and report on how electronically collected driver schedule data can be extracted, stored, and analyzed in a way that addresses cost and privacy concerns.