To analyze the NAOMS air carrier pilot survey’s planning, design, and  implementation (including pretest, interview, and data collection  methods); interviewer training; development of survey questions,  including which safety events to include in the survey; and sampling, we  interviewed officials from NASA, the Federal Aviation Administration  (FAA), and the National Transportation Safety Board (NTSB) and NAOMS  project staff. We asked three external experts to review and assess the NAOMS air  carrier pilot survey’s design and implementation as well as considerations  for analysis of collected data. NAOMS Was Intended to Identify Accident Precursors and Potential Safety Issues

The NAOMS project was conceived and designed in 1997 to provide broad,  long-term measures on trends and to measure the effect of new  technologies and policies on aviation safety. A Web-based version of the air carrier pilot  survey and related information were handed off to ALPA in January 2007. Although ALPA never had access to existing NAOMS  data, this official also expressed uncertainty about what should be done  with the existing data. “demonstrated a survey methodology to quantitatively measure aviation safety, tracked  trends in event rates over time, identified effects of new procedures introduced into the  operating environment, and generated interest and acceptance of NAOMS by some of the  aviation community as described in the Project Plans.”
The OIG report identified several shortcomings of the project, including  that (1) the “contracting officers did not adequately specify project  requirements” or “hold Battelle responsible for completing the NAOMS  Project as designed or proposed”; (2) the “contractor underestimated the  level of effort required to design and implement the NAOMS survey”;   (3) “NASA had no formal agreement in place for the transfer and  permanent service of NAOMS”; and (4) “NAOMS working groups failed to  achieve their objectives of validating the survey data and gaining  consensus among aviation safety stakeholders about what NAOMS survey  data should be released.” An additional deficiency, according to the OIG,  was that, as of February 2008, “NASA had not published an analysis of the  NAOMS data nor adequately publicized the details of the NAOMS Project  and its primary purpose as a contributor to the ASMM Project.”

	NAOMS’s Planning and Design Were Robust, but Implementation Decisions Complicate Data Analysis

We found that, overall, the NAOMS project followed generally accepted  survey design and implementation principles, but decisions made in  developing and executing the air carrier pilot survey complicate data  analysis. The Survey’s Planning and Design

Early documentation of the NAOMS project shows that the project was  planned and developed in accordance with generally accepted principles  of survey planning and design. They devised  mechanisms to protect respondent confidentiality. The NAOMS survey methodologist ran  experiments from 1998 through 1999 to generate and test hypotheses that  could be incorporated into the design of the air carrier pilot survey. A Large-Scale Field Trial Resolved Many Issues, but Not Others

In 1999, following more than 1 year of research, experiments, and  questionnaire development, NAOMS researchers conducted a large-scale  field trial. Potential Problems Related to the Sampling Strategy Require Additional Assessment

While NAOMS researchers designed and selected a sample in accordance  with generally accepted survey research principles, sampling decisions  they made to address complications influenced the nature of the data  collected. To develop NAOMS’s sampling strategy, the team first needed to identify a  target population. NASA envisioned a wide range of participants in the working groups,  including pilots; flight attendants; people familiar with alternative data  systems; and other aviation stakeholders, such as academic researchers  and industry. A New Survey Would Require Detailed Planning and Revisiting Sampling Strategies

A new survey similar to NAOMS would require more coherent planning  and sampling methods linked to specific analytic goals. As a research and  development project, NAOMS was a successful proof of concept. Alternatively, a newly constituted research team might lead operational,  survey, and statistical experts in extensively analyzing existing data to  validate a new survey’s utility for various purposes or to illuminate future  projects of the same type. Agency Comments and Our Evaluation

We provided a draft of this report to the National Aeronautics and Space  Administration and to the Department of Transportation for their review. We agree with NASA’s concern about  pilot identification and have revised the report to highlight NASA’s  concern; however, we also note that other government agencies have  developed mechanisms for releasing, in a controlled manner, extremely  sensitive raw data with high risk for the identification of individuals to  appropriate researchers. Statement on the National Aviation Operations Monitoring  Service.
Why GAO Did This Study

The National Aviation Operations Monitoring Service (NAOMS), begun by the National Aeronautics and Space Administration (NASA) in 1997, aimed to develop a methodology that could be used to survey a wide range of aviation personnel to monitor aviation safety. NASA expected NAOMS surveys to be permanently implemented and to complement existing federal and industry air safety databases by generating ongoing data to track event rates into the future. The project never met these goals and was curtailed in January 2007. GAO was asked to answer these questions: (1) What were the nature and history of NASA's NAOMS project? (2) Was the survey planned, designed, and implemented in accordance with generally accepted survey principles? (3) What steps would make a new survey similar to NAOMS better and more useful? To complete this work, GAO reviewed and analyzed material related to the NAOMS project and interviewed officials from NASA, the Federal Aviation Administration, and the National Transportation Safety Board. GAO also compared the development of the NAOMS survey with guidelines issued from the Office of Management and Budget, and asked external experts to review and assess the survey's design and implementation.

What GAO Found

NAOMS was intended to demonstrate the feasibility of using surveys to identify accident precursors and potential safety issues. The project was conceived and designed to provide broad, long-term measures on trends and to measure the effects of new technologies and aviation safety policies. Researchers planned to interview a range of aviation personnel to collect data in order to generate statistically reliable estimates of risks and trends. After planning and development, a field trial, and eventual implementation of the air carrier pilot survey and the development of a smaller survey of general aviation pilots, the project effectively ended when NASA transmitted a Web-based version of the air carrier pilot survey to the Air Line Pilots Association. NAOMS's air carrier pilot survey was planned and designed in accordance with generally accepted survey principles, including its research and development, consultation with stakeholders, memory experiments to enhance the questionnaire, and a large-scale field trial. The survey's sample design and selection also met generally accepted research principles, but there were some limitations, and the survey data may not adequately represent the target population. Sample frame and design decisions to maintain program independence and pilot privacy complicate analysis of NAOMS data. Certain implementation decisions, including extended methodological experiments and data entry issues, also complicate analytical strategies. Also, working groups of aviation stakeholders were convened as part of NAOMS to assess the validity and utility of the data, but these groups never had access to the raw data and were disbanded before achieving consensus. To date, NAOMS data have not been fully analyzed or benchmarked against other data sources. While NAOMS's limitations are not insurmountable, a new survey would require more coherent planning and sampling methods, a cost-benefit analysis, closer collaboration with potential customers, a detailed analysis plan, a reexamination of the sampling strategy, and a detailed project management plan to accommodate concerns inherent in any survey endeavor. As a research and development project, NAOMS was a successful proof of concept with many strong methodological features, but the air carrier pilot survey could not be reinstated without revisions to address some of its methodological limitations. The designers of a new survey would want to supplement NAOMS where it was self-limiting. Alternatively, a newly constituted research team might lead operational, survey, and statistical experts in extensively analyzing existing data to illuminate future projects. In reviewing a draft of this report, NASA reiterated that NAOMS was a research and development project and provided technical comments, which GAO incorporated as appropriate. NASA also expressed concern about protecting NAOMS respondents' confidentiality, a concern GAO shares. However, GAO noted that other agencies have developed mechanisms for releasing sensitive data to appropriate researchers. The Department of Transportation had no comments.