Air Force pilots are required to complete various phases of training before they are considered to be mission ready. To assess pilot candidates’ flying aptitude for both traditional (i.e., manned) and remotely-piloted aircraft, the Air Force first requires these candidates to attend initial flight training. Successful candidates for traditional aircraft then attend one of two schools: Euro-NATO Joint Jet Pilot Training, the graduates of which become Specialized Undergraduate Pilot Training, the graduates of which become fighter, bomber, airlift, or tanker aircraft pilots, depending on pilot strengths and Air Force needs. During or following this training, a pilot is assigned to a specific aircraft, and the flight training program proceeds through three stages: Initial Qualification Training. This stage of training qualifies a pilot for basic flying duties associated with the type of aircraft (e.g., an F-16). The pilot accomplishes this stage at a formal training unit before moving to an assigned squadron. Graduates of initial qualification training courses have basic aircraft qualification status. Mission Qualification Training. This stage of training occurs once the basic aircraft qualified pilot is at the assigned unit. The pilot undergoes mission qualification training in the type of aircraft assigned to qualify for the specific missions the unit is required to perform. Continuation Training. This stage of pilot training has two components. Pilots participate in continuation training to (1) in some instances upgrade their qualifications to fill certain positions, such as flight lead, instructor, or forward air controller through specialized continuation training; and (2) in all instances maintain proficiency and improve their capabilities to perform their units’ assigned missions. The Ready Aircrew Program establishes the minimum number of live training events, or “sorties,” and virtual simulator training events, or “simulator missions,” that aircrews of a particular combat aircraft must complete during the annual training cycle to maintain mission readiness. These sorties and simulator missions are aligned with the units’ primary missions, for which the units must maintain “proficiency,” and secondary missions, for which they must maintain “familiarity.” The Air Combat Command, as lead command for the Ready Aircrew Program, with the assistance of other major commands (including the Air Force Global Strike Command, Pacific Air Forces, and U.S. Air Forces in Europe) and associated subordinate organizations (i.e., air wings and squadron commanders), develops tasking memorandums for the Ready Aircrew Program that delineate and specify the annual continuation training requirements for personnel assigned to each of the subordinate combat units. On the basis of our analysis, we found that the RAND report addressed each of the three statutory elements required by Congress in Section 351. First, the RAND report addressed two statutory elements by reviewing and assessing the assumptions underlying annual continuation training requirements for the Ready Aircrew Program and the overall effectiveness of the Ready Aircrew Program in managing aircrew training requirements. These two statutory elements focus on issues raised in our prior report recommendations, which we discuss in more detail later in this report. Table 1 provides detailed information about these statutory elements, our assessment of RAND’s findings, and RAND’s findings associated with each element. The RAND report addressed the third statutory element by making recommendations for the improved management of training requirements. Specifically, the RAND report made nine recommendations, listed in table 2. The RAND report and its underlying analysis is consistent with generally accepted research standards for design, execution, and presentation. Table 3 summarizes our assessment of the extent to which the RAND report conformed with these standards. The RAND study and our previous audit work identified similar deficiencies in the management and operation of the Ready Aircrew Program. Specifically, in September 2016, we reported that the Air Force had used the same underlying assumptions to establish its annual training requirements in the Ready Aircrew Program from 2012 through 2016, which may not reflect current and emerging training needs. We concluded that without fully reassessing the assumptions underlying its training requirements, the Air Force could not be certain that its annual training plans are aligned with its stated goals to ensure a full-spectrum- capable force that can successfully achieve missions across a broad range of current and emerging threats. We recommended that the Air Force comprehensively reassess the assumptions underlying its annual training requirements—and make any appropriate adjustments in future aircrew training plans to ensure that its forces can accomplish a full range of missions. Additionally, in our September 2016 report, we also reported that the Air Force did not systematically evaluate the effectiveness of training performed as part of the Ready Aircrew Program. We recommended that the Air Force establish desired learning objectives and training support elements needed to accomplish the training expectations identified by the Ready Aircrew Program and develop a process to collect data to assess the effectiveness of annual training against these features. We discuss these recommendations and their status in more detail later in this report. Section 351 of the NDAA for Fiscal Year 2017 required the Air Force to report on any actions it plans to take in response to RAND’s recommendations and to estimate the resources required to implement the recommendations. On August 30, 2018, the Air Force provided its report—a one-page transmittal letter from the Secretary of the Air Force with the RAND report incorporated as an enclosure—to congressional committees in fulfillment of the Section 351 requirements. In its report, the Air Force agreed with RAND that more investment is needed in data collection because its current system does not lend itself to analysis that could be used to gain efficiencies. The Air Force also stated that it is addressing the RAND recommendations by working to link readiness to Ready Aircrew Program training requirements. Specifically, the Air Force stated that it was taking the following three actions: building training matrices to help commanders assess their units’ establishing common data architecture through the Air Force’s Chief Data Officer–led effort, and evaluating aspects of the Ready Aircrew Program to increase lethality and improve readiness as the Air Force shifts to executing the mandates of the 2018 National Defense Strategy. However, in its August 2018 report, the Air Force did not provide any additional details to further describe or link these broad actions to the RAND findings and recommendations. Therefore, the extent to which these three actions are responsive to the RAND recommendations is unclear. In October 2019, upon completion of our audit work, Air Force officials provided us with a briefing they described as a corrective action plan that further elaborated on the Air Force’s position with respect to each of RAND’s nine recommendations. Air Force officials conceded that the actions described in that briefing and plan were already underway at the time of the study and were not initiated in response to the study’s recommendations. They stated that, though they generally agreed with the recommendations, the Air Force lacked the manpower, resources, and means to implement them. As such, the Air Force considers each of the recommendations closed and plans no further actions. Further, the Air Force has no plans for future follow-up on implementation of the RAND recommendations. Accordingly, we did not further assess the actions described by the Air Force in relation to the RAND study. Table 4 summarizes the Air Force’s position as provided in its corrective action plan briefing. As described in table 4, the Air Force concurred with three recommendations, partially concurred with five, and did not concur with one. The following summarizes the Air Force position by concurrence category: Concur: In concurring with RAND Recommendations 2 and 8—to invest in data systems for the collection, access, and storage of data to correct deficiencies in current systems and improve analysis and readiness reporting—the Air Force stated that, before the recommendations were made, it had made available $5.15 million in fiscal year 2020 funding to develop an Aircrew Readiness Training Management Module as an upgrade to the Air Force’s Aviation Resource Management System. The Air Force expects that this module will centralize management of the Air Force Ready Aircrew Program Tasking Memorandum at the command level, transfer aircrew training data whenever a member moves to a new station, and improve capability to track the types of flight simulators used for training. Further, because the Air Force does not plan to take additional actions, it estimated no resources are required, beyond the $5.15 million already funded, prior to RAND making these recommendations. In concurring with RAND Recommendation 5—to document training quality to support requests for training resources—the Air Force explained that it is documenting the quality of training prescribed in its annual Air Force Ready Aircrew Program Tasking Memorandum at the squadron level. However, this is not a change based on the RAND recommendation and was being done prior to RAND making this recommendation. The Air Force factors unit training, accomplishments, and readiness inputs into the Air Force Ready Aircrew Program Tasking Memorandum and overall flying hour program. As the flying hour program is the basis for resource training requests and the Air Force factors unit training and accomplishments into its flying hour program, the Air Force explained that it plans to take no additional actions based on this recommendation. Further, because the Air Force does not plan to take additional actions, it estimated no resources were required to implement this recommendation. Partially Concur: In partially concurring with RAND recommendations 1, 3, 4, 6, and 9, Air Force officials explained that the mechanisms reflected in its comments are sufficient to address the intent of the RAND recommendations even though these mechanisms predated RAND’s recommendations. Consequently, according to the Air Force officials who briefed us, the Air Force plans no additional actions based on the recommendations, obviating the need to estimate resources in the case of these five recommendations. Nonconcur: In not concurring with RAND Recommendation 7—to consider changing how Air Force Flying Hour Program requirements affect the flying hour program—the Air Force explained that the flying hour program determination is standard across the Total Air Force, affecting more than combat aircrews alone, and that the centrality of the flying hour program to readiness and combat capability cannot be overemphasized and must be defendable and auditable. Further, based on its not concurring with this recommendation, the Air Force did not estimate resources for this recommendation. Notwithstanding the RAND study, the actions taken by the Air Force may not fully implement our prior recommendations as described previously. Specifically, we recommended that the Air Force comprehensively reassess the assumptions underlying its annual training requirements— including for example the total annual training requirements by aircraft, the criteria for designating aircrews as experienced or inexperienced, and the mix between live and simulator training—and make any appropriate adjustments in future aircrew training plans to ensure that its forces can accomplish a full range of missions. While RAND accomplished such an analysis as part of its review (see table 1, items 1(a) through 1(e) above), RAND concluded in its analysis that the Air Force did not have the objective measures of proficiency needed to determine the minimum and optimum number of sorties and that the combat air forces aviation community lacks consensus on how to define and measure an aircrew member’s proficiency. While the Air Force has defined proficiency in an Air Force manual issued in September 2019, the Air Force has not reassessed its assumptions underlying training requirements and made appropriate adjustments to future training plans per our recommendation. Further, we recommended that the Air Force establish desired learning objectives and training support elements needed to accomplish the training expectations in its annual Ready Aircrew Program tasking memorandums, and develop a process to collect data to assess the effectiveness of annual training. In commenting on RAND’s recommendations 2 and 8, the Air Force stated that, before the recommendations were made, it had made available $5.15 million in fiscal year 2020 funding to develop an Aircrew Readiness Training Management Module as an upgrade to the Air Force’s Aviation Resource Management System. The Air Force’s effort to upgrade this system, while not a result of the RAND recommendations, may meet the intent of our recommendation. For example, the Air Force’s development of the Aviation Resource Management System is expected to centralize management of the Air Force’s Ready Aircrew Program training requirements data at the command level, transfer aircrew training data whenever a member moves to a new station, and improve capability to track the types of flight simulators used for training. When fully implemented, these improvements may ultimately allow the Air Force to assess the effectiveness of annual training, as we recommended. However, it is too early to tell as the actions were under development or had just begun at the end of September 2019, and sufficient data to evaluate the results have not been collected. Fully implementing both of our recommendations would better position the Air Force to ensure that its aircrews receive effective training to achieve a range of missions for current and emerging threats. We provided a draft of this report to DOD for review and comment. In its written comments (reprinted in appendix II), DOD stated that it saw great value in our discussion. DOD added that its position on the two recommendations we made in our 2016 report—with which DOD did not concur—is fundamentally unchanged. However, we continue to believe the recommendations should be implemented by the Air Force, as previously discussed in this report. In its comments, DOD also stated that it is addressing training infrastructure and aircrew proficiency through two initiatives. First, in response to the 2018 National Defense Strategy, the department's Joint Operational Infrastructure Plan is framing the modernization effort of DOD-wide Operational Infrastructure. The Joint Operational Infrastructure Plan specifically addresses areas such as Live Virtual Constructive and aircrew training. Second, the department is pursuing efforts to align training events with the range of current and evolving threats. According to DOD, both efforts will address the underlying assumptions for aircrew training and proficiency with reportable readiness metrics. However, the Joint Operational Infrastructure Plan is a draft and not yet officially issued, according to an official at the Office of the Under Secretary of Defense for Personnel and Readiness. Therefore, details needed to assess this plan are not yet available for us to consider in determining whether the change will help to address the recommendations we made in 2016. Regarding efforts the department is pursuing to better align training events, the DOD comments did not include sufficient details to allow us to state whether the change could be helpful in addressing our recommendations. Nonetheless, to the extent that DOD is successful in completing, issuing, and implementing its new Joint Operational Infrastructure Plan, or takes further actions related to our prior recommendations, we will consider them as we continue to analyze DOD efforts to address our recommendations. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Under Secretary of Defense for Personnel and Readiness, and the Secretary of the Air Force. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-5431 or russellc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. To determine whether the RAND report followed generally accepted research standards, we chose the following criteria, which are based on a review of research literature and Department of Defense (DOD) guidance, from which we identified frequently occurring, generally accepted research standards that are relevant for defense studies, including those related to the presentation of results. These standards have been used in a number of our prior reports, modified as appropriate for each situation. For the purposes of this engagement, we assessed the RAND study against three major elements, listed below, that fall under generally acceptable research standards, as based on our prior work. These generally accepted research standards are consistent with Office of Management and Budget (OMB) Guidelines and DOD guidance on ensuring and maximizing the quality of information disseminated by federal agencies to the public. We discussed these standards with RAND officials, who agreed that they were generally consistent with their own quality standards for research and were applicable to this study. We determined that these standards are still current and relevant for the purposes of this report, based on their consistency with OMB and DOD guidance, discussions with RAND officials, and consideration of prior GAO work applying generally accepted research standards with the assistance of GAO’s Applied Research and Methods Team. The standards include the following: Design—Study is well designed. For the RAND study, we focused on the following elements for design: The study plan, scope, and objectives follow existing guidance. Assumptions and constraints are reasonable and consistent. Execution—Study is well executed. For the RAND study, we focused on the following elements for execution: The methodology is successfully executed. Data used to support study and analyses are validated. Presentation—Results are well presented. For the RAND study, we focused on the following elements for presentation: Timely, complete, accurate, concise, and relevant to stakeholders. Presentation of results supports findings. In addition to the contact named above, Beverly Schladt (Assistant Director), John Strong (Analyst in Charge), John Beauchamp, Vincent Buquicchio, Martin De Alteriis, and Lillian Moyano Yob made key contributions to this report.
In September 2016, GAO reported that annual combat aircrew training requirements delineated in the Air Force's Ready Aircrew Program might not address pilot training needs, and that the Air Force did not systematically evaluate the effectiveness of its training. As a result, Congress included a provision in Section 351 of the NDAA for Fiscal Year 2017 for the Air Force to commission an independent review of its Ready Aircrew Program, report on actions it planned to take in response to any recommendations, and provide an estimate of any resources required. Section 351 also included a provision for GAO to assess the Air Force report. This report examines whether (1) the independent review conducted by the RAND Corporation addressed statutory requirements to review and assess the Ready Aircrew Program, and (2) the Air Force has reported on completed or planned actions to implement the RAND report recommendations. To address these objectives, GAO reviewed the RAND and Air Force reports on the Ready Aircrew Program, assessed the study against generally accepted research standards, and interviewed officials at RAND, Air Force Headquarters, and the Air Combat Command. A July 2018 RAND report—commissioned by the Air Force—addressed the statutory requirements of the National Defense Authorization Act (NDAA) for Fiscal Year 2017 to review and assess the Air Force's Ready Aircrew Program and make recommendations for ways to improve it. The Ready Aircrew Program establishes minimum annual training requirements for combat aircrew. RAND's report, entitled Independent Review and Assessment of the Air Force Ready Aircrew Program , made nine recommendations to improve its management: 1. Leverage internal expertise to implement measures for proficiency. 2. Invest resources to design data collection and storage solutions that facilitate analysis and readiness reporting. 3. Document the Ready Aircrew Program Tasking Memorandum development process in Air Force instruction supplements and ensure that the process incorporates squadron-level input and feedback. 4. Establish a more explicit and formal link between proficiency and Ready Aircrew Program requirements. 5. Document training quality to support requests for training resources. 6. Identify the conditions under which Ready Aircrew Program requirements, including mission types, can be accomplished. 7. Consider changing how Ready Aircrew Program requirements affect the Flying Hour Program. 8. Invest in data systems to correct data collection and assess deficiencies. 9. Leverage the Air Force Research Laboratory's performance data work and invest in added analysis to produce enterprise-wide proficiency metrics. The nine RAND recommendations aligned with two GAO recommendations made in 2016 to comprehensively assess the assumptions underlying the annual aircrew training requirements and develop a process to collect data to assess the effectiveness of the training. The Air Force's August 2018 one-page report to Congress included three broad actions in response to RAND's recommendations. The Air Force planned to build training matrices to help commanders assess their units' effectiveness, establish common data architecture through the Air Force's Chief Data Officer–led effort, and evaluate aspects of the Ready Aircrew Program to increase lethality and improve readiness as the Air Force shifts to executing the mandates of the 2018 National Defense Strategy. The Air Force, however, did not explain how these three efforts would specifically address the nine recommendations. Air Force officials said that, though they generally agreed with RAND's recommendations, the Air Force lacked the resources to fully implement them beyond actions that were underway prior to the RAND report, and considers all recommendations as “closed.” In part due to its not fully implementing RAND's recommendations, the Air Force has not fully addressed GAO's two recommendations. Fully implementing GAO's recommendations would better position the Air Force to ensure its aircrews receive effective training to achieve a range of missions. GAO is not making any new recommendations in this report but restates the need to fully implement GAO's two 2016 recommendations. In comments on a draft of this report, DOD did not concur with the two 2016 recommendations. GAO maintains that the Air Force should implement both recommendations. recommendation. GAO will consider these in its annual recommendations follow-up.