Governmentwide priorities. Measure produces the same result under similar  conditions. ATO Established Annual Performance Measures to Be Consistent with Federal Guidance and Selects Major Programs for Measuring and Reporting

ATO developed its performance targets to be consistent with targets set in  the Department of Transportation’s strategic plan, OMB guidance, and  the Federal Acquisition Streamlining Act of 1994, which call for other  federal agencies to establish cost and schedule goals for acquisitions and  to achieve at least 90 percent of those goals. To measure schedule performance, at the start  of each fiscal year ATO managers select a minimum of two schedule  milestones from each major acquisition selected for performance  reporting that year. ATO reports the result as the  percentage of major programs that are on schedule. ATO’s Acquisition Performance Measures’ Lack of Certain Successful Attributes and 1-Year Focus Impairs Validity of Performance Reporting

ATO’s acquisition performance measures meet four of eight key attributes  for successful performance measures that we have identified, but the lack  of objective criteria for selecting programs for performance measurement  reduces objectivity, reliability, and assurance that core programs are  included; the clarity of the performance measures could also be improved. Objective performance measures should not allow subjective  considerations or judgments to dominate the outcome of the  measurement. The lack of objective criteria for designating major programs also impairs  the key attribute of reliability and the assurance that the measures include  core program activities. ATO’s On-Budget Performance Measure Does Not Clearly Indicate the Use of Revised Budget Estimates

ATO’s on-budget acquisition performance measure lacks clarity in  reporting because ATO does not indicate that the acquisition performance  of baselined programs is measured using the most recently approved  budget estimates, as reflected in the January CIP. However,  because ATO measures budget performance for an 8-month timeframe  against the most recently approved budget estimate, STARS was  considered on budget for fiscal years 2003 through 2006. Because ATO’s performance measures lack several attributes of successful  performance measures, and are focused on 1-year snapshots of  performance, they may not provide a valid assessment of acquisition  performance over time. ATO’s Performance Measurement Could Mask Budget Increases and Delays in the Transition to NextGen

When measured against original baselines, ATO shows improvement in its  managing of acquisitions, but its performance is lower than indicated in  FAA’s annual Performance and Accountability Report. The absence of original budget and schedule estimates in ATO’s  performance reporting could give the impression to Congress and the  American people that ATO’s acquisitions and the transition to NextGen are  progressing more smoothly than is actually the case. These issues are critical as ATO begins acquiring new  systems with a goal of completing the transition to NextGen by 2025. Recognizing impending budget increases and schedule delays and taking  corrective action will be necessary to keep the overall NextGen effort on  track. Appendix I: Objectives, Scope, and Methodology

We examined (1) how the Air Traffic Organization (ATO) establishes goals  and performance measures for acquiring air traffic control (ATC) systems  and how they are reported; (2) how ATO’s acquisition performance  measures compare with key attributes of successful performance  measures; and (3) the implications of using ATO’s existing performance  measures to assess progress in the transition to the Next Generation Air  Transportation System (NextGen). Although  ATO has other performance measures that it applies to its acquisitions, in  this report we focused only on the two that FAA uses to report its  performance—the percentages of acquisitions on budget and acquisitions  on schedule.
Why GAO Did This Study

Acquiring new systems on budget and on schedule is critically important in transitioning to the Next Generation Air Transportation System (NextGen). However, air traffic control modernization has been on GAO's high-risk list since 1995, in part due to acquisitions exceeding budget and schedule targets. The Federal Aviation Administration's (FAA) Air Traffic Organization (ATO) has responsibility for managing air traffic control acquisitions. GAO was asked to examine (1) ATO's goals, performance measures, and reporting for systems acquisitions; (2) the validity of ATO's performance measures; and (3) the implications of using ATO's performance measures to assess progress in transitioning to NextGen. To address these issues, GAO compared ATO's measures with attributes of successful performance measures, interviewed agency officials, and sought perspectives of aviation experts.

What GAO Found

To be consistent with federal guidance and with targets set in the Department of Transportation's strategic plan, ATO established annual acquisition goals and performance measures that call for a high percentage of its major acquisitions to be within 10 percent of budget and on schedule. ATO identifies major acquisitions and reports performance against its goals using its most recently approved budget and schedule estimates. To measure on-budget performance, ATO calculates budget increases over an 8-month period--between January and August of each year. To measure on-schedule performance, ATO selects a minimum of two annual milestones from its major acquisitions and calculates the percentage of milestones that are on schedule. Because ATO's acquisition performance measures lack objectivity, reliability, coverage of core activities, and clarity, and focus only on the preceding year, they may not provide a valid assessment of performance over time. On the positive side, the measures are aligned with FAA's strategic objectives, are measurable, have no overlap, and address governmentwide priorities. However, the performance measures lack objectivity because ATO has no objective criteria for designating which programs are "major" and should be selected for performance reporting. This makes it possible for subjective considerations to dominate the outcome and leaves the performance measures vulnerable to bias in the selection of programs for reporting. The lack of objective criteria for designating major programs also impairs the reliability of the measures (the ability of the measures to produce the same results each time they are applied under similar conditions) and undermines assurance that ATO managers include all core program activities in performance reporting each year. The performance measures also lack clarity in that they do not indicate that ATO measures the performance of many acquisitions against the most recently approved budget and schedule estimates rather than the original estimates. ATO's acquisition performance measurement and reporting could mask budget increases and schedule delays that could have a negative effect on the transition to NextGen. Although ATO reported performance that exceeded its goals for fiscal years 2004 through 2006 and showed nearly steady improvement, when measured against original baselines, acquisition performance improved but was lower than reported. Going forward, the absence of original budget and schedule information on ATO's acquisitions could give the impression that the transition to NextGen is progressing more smoothly than might actually be the case. It will be important for ATO and Congress to recognize budget increases and schedule delays so that the capacity, efficiency, and safety benefits of NextGen can be realized in a cost-efficient and timely fashion.