From January through August 2017, the Navy suffered four significant mishaps at sea that resulted in the death of 17 sailors and hundreds of millions of dollars in damage to Navy surface ships (see fig.1). More recently, the Navy experienced two incidents during which Navy surface ships collided. First, on February 5, 2019, a Ticonderoga-class guided missile cruiser—USS Leyte Gulf (CG 55)—collided with a Navy resupply ship—the USNS Robert E. Peary (T-AKE 5)—while conducting an underway replenishment operation off the coast of Florida. Second, on June 21, 2019, a Freedom-class Littoral Combat Ship—the USS Billings (LCS 15)—struck a merchant ship while leaving a pier in Montreal, Canada. According to Navy officials, these recent incidents did not result in serious damage to the ships or injuries to the crews but they demonstrate the need for continued focus and attention on safe ship driving. As of March 2019, the Navy had approximately 8,400 SWOs—Navy officers whose training and primary duties focus on the operation of Navy ships at sea and the management of various shipboard systems. The Navy expects SWOs to progress over the course of their careers from Division Officers driving ships, to Department Heads participating in combat operations, to Executive Officers managing ship crews, and to Commanding Officers overseeing operations. Figure 2 below outlines a SWO’s career progression and associated shipboard duties. The Commander, Naval Surface Forces, in coordination with Office of the Chief of Naval Operations, Surface Warfare Officers School Command, and Navy Personnel Command, manages and provides ship-driving training to SWOs throughout their careers. Initially, the primary focus of a Division Officer is on leading sailors and developing ship-driving competency, ultimately working toward qualification as an Officer of the Deck and Surface Warfare Officer. Therefore, a large part of initial SWO training focuses on leading a division and developing the ship-driving skills needed to qualify and perform as an Officer of the Deck. This training is provided during a SWO’s initial training—in the Basic Division Officer Course—offered in Norfolk, Virginia, and San Diego, California, and is a mix of classroom and simulator-based training. After completing the Basic Division Officer Course, SWO candidates begin their first at-sea assignment as Division Officers. Division Officers have three primary roles aboard a ship: 1. They support ship-driving operations. New Division Officers gain ship- driving experience in pursuit of the qualification to stand as Officers of the Deck. Once qualified as Officers of the Deck, they lead watch teams in driving ships. 2. They support ship department operations under the supervision of Department Heads, and are responsible for a portion of ship equipment and operations. 3. They lead a division of approximately 12 to 50 enlisted personnel within departments, and are responsible for the administrative and supervisory duties for divisions. In addition to their Division Officer responsibilities, new Division Officers are expected to earn qualification as a SWO by completing required education and meeting experience standards, as well as gaining watchstanding experience and demonstrating proficiency in the execution of their duties, according to Navy officials. These qualifications include Officer of the Deck and Combat Information Center Watch Officer, where an officer assists in observing and analyzing information of importance for combat, among others. Navy officials stated that after an officer completes their qualifications, their Commanding Officer reviews the officer’s ability and experiences, and can grant the candidate SWO qualification. In June 2010, we reviewed Navy policies for surface force training— including initial SWO ship-driving training—and found that the Navy had reduced and altered initial SWO training as an efficiency measure, but lacked performance measures and data necessary to evaluate the impact of changes to training programs. We found that in 2003, the Navy had replaced its 6-month Division Officer course consisting of classroom and simulator training with software-based training where new Division Officers were instead expected to learn SWO skills from computer-based education software while onboard their first ship. The Navy said this change saved about $50 million annually, but we found that the Navy lacked outcome-based performance measures to evaluate the effects of these changes to training on officer performance. We recommended that the Navy develop metrics to measure the effects of training on SWO job performance, knowledge, skills, and abilities. The Navy concurred with this recommendation but did not implement the recommendation for the software training or for subsequent training programs. The Navy has enhanced ship-driving training for SWOs at the early stages of their careers following the 2017 collisions at sea, and by 2021 plans to triple the number of ship-driving training hours when compared with the amount of training SWOs were required to receive prior to the collisions. The Navy’s plans to increase ship-driving proficiency hinge on the completion of two new simulator-based training facilities—the Mariner Skills Training Centers—which are planned to be completed in June 2021 (San Diego, California) and in January 2023 (Norfolk, Virginia). Overall, the Navy plans to invest more than $467 million to develop new ship- driving training courses, build simulator facilities, and deliver the training through fiscal year 2025. Prior to the 2017 ship collisions, SWOs were required to complete 174 hours of ship-driving training during their Division Officer assignment by attending the Basic and Advanced Division Officer training courses. Following the collisions, the Navy increased the amount of required ship- driving training in these two courses to 203 hours. In June 2019, the Navy added a 4-week ship-driving course—the Junior Officer of the Deck course—that focused exclusively on building ship-driving skills. This course added 158 hours of required classroom and simulator ship-driving training. In June 2021, the Navy plans to expand the curriculum of the Junior Officer of the Deck course and rename it the Officer of the Deck Phase I course, and add an additional 3-week Officer of the Deck Phase II course. These two courses will add an additional 185 hours of required ship-driving training for Division Officers in preparation for their first and second at-sea assignments. Once these ship-driving training courses are in place, Division Officers will be required to complete a total of 535 hours of training—triple (a threefold increase in) the number of ship- driving training hours SWOs were required to complete prior to the 2017 collisions (see fig. 3). Below are detailed descriptions of the changes completed and planned to enhance ship-driving training. Basic Division Officer Course. From November 2017 through January 2019, the Surface Warfare Officers School Command changed the Basic Division Officer Course—a 9-week course for new SWO candidates—by increasing the required hours of classroom instruction and simulator training by 12 percent, and broadening the course curriculum. Specifically, prior to the 2017 collisions, SWO candidates were required to spend 113 hours (81 hours of classroom instruction and 32 hours in simulators) in this course to develop their ship-driving skills. After January 2019, however, SWO candidates were required to spend 126 hours (89 hours of classroom instruction and 37 hours in simulators) to develop their ship-driving skills. Regarding added course content, the Surface Warfare Officers School Command added subject matter including additional training on the internationally accepted ship-driving standards that govern ship maneuvers; radar navigation; and the tools used to aid ship-driving. Advanced Division Officer Course. From November 2017 through January 2019, the Surface Warfare Officers School Command changed the Advanced Division Officer Course—a 5-week course for SWOs returning from their first at-sea assignment—to improve ship-driving skills by increasing the required hours of simulator training from 24 to 36 hours. Prior to the 2017 collisions, SWOs were required to spend 61 hours (37 hours in the classroom and 24 hours in simulators) refining their ship- driving skills in this course. As of January 2019, SWOs were required to spend 77 hours (41 hours in the classroom and 36 hours in simulators) developing and honing their ship-driving skills. Surface Warfare Officers School Command officials also added subjects to classroom time to build on the subject matter presented in the Basic Division Officer Course, including more complex ship-driving techniques and advanced radar navigation. Surface Warfare Officers School Command plans to reduce the hours of training in this course once the Officer of the Deck Phase II course comes online in 2021. Junior Officer of the Deck course. In June 2019, Surface Warfare Officers School Command provided this new 4-week course for the first time—the course having been developed after the 2017 collisions and focused predominately on building ship-driving skills. The Junior Officer of the Deck course takes place after SWO candidates complete the Basic Division Officer Course and before they begin their first at-sea assignment. SWOs taking this course are required to complete 158 hours of classroom and simulator training designed to increase their ship-driving skills by exposing them to a variety of scenarios involving different maneuvers, and varying sea and weather conditions. The Navy plans to expand this course into a 6 week ship-driving training course (Officer of the Deck Phase I), scheduled to begin in June 2021. Officer of the Deck Phase I course. According to Commander, Naval Surface Forces documentation, the Junior Officer of the Deck course will expand into the Officer of the Deck Phase I course. Officer of the Deck Phase I is under development and will be 6 weeks long (an additional 2 weeks longer than Junior Officer of the Deck), and will take place after SWO candidates complete the Basic Division Officer Course and before they begin their first at-sea assignment. Officer of the Deck Phase I is intended to build on the Junior Officer of the Deck curriculum by increasing the required number of ship-driving training hours from 158 to 241, and expanding the course content to include instruction on more advanced radar navigation techniques. Surface Warfare Officers School Command and Commander, Naval Surface Forces officials expect the Officer of the Deck Phase I course to begin in June 2021. Officer of the Deck Phase II course. According to Navy documentation, the Officer of the Deck Phase II course that is under development will be 3 weeks long, and will take place after SWOs have completed their first at-sea assignment and before they attend the Advanced Division Officer Course. This course is intended to continue the development of ship- driving skills through an additional 102 hours of required classroom and simulator training. Surface Warfare Officers School Command and Commander, Naval Surface Forces officials stated that Officer of the Deck Phase II course could begin as early as June 2021. Mariner Skills Training Centers. According to Commander, Naval Surface Forces and Surface Forces documentation, Surface Warfare Officers School Command will provide the Officer of the Deck Phase I and Phase II courses at the Mariner Skills Training Centers—new simulator-based facilities expedited after the 2017 collisions. Officials from the Office of the Chief of Naval Operations stated that these facilities—including upgraded simulators, the instructors, classrooms, and the curriculum development for the Officer of the Deck Phase I and Phase II courses—will cost approximately $467.5 million through fiscal year 2025. According to Navy officials, construction on the San Diego, California Mariner Skills Training Center will begin in early fiscal year 2020 and will be complete by June 2021 and Norfolk, Virginia Mariner Skills Training Center will begin in fiscal year 2021 and will be complete in January 2023. The Mariner Skills Training Program is based upon the Littoral Combat Ship ship-driving training program, which according to the Navy, provides a balance of classroom, simulation, and shipboard experience. According to Navy officials, since Littoral Combat Ship SWOs serve in rotating crews and have less opportunity to train aboard their ships, the Navy developed the Littoral Combat Ship Training Facility to support SWOs’ training ashore (see fig. 4). The foundation of the Littoral Combat Ship ship-driving program is repetitive training in sophisticated simulators to build ship-driving proficiency. According to the Navy, the effectiveness of this training has been validated over the last 10 years by the superior ship-driving proficiency of Littoral Combat Ship officers during at-sea operations and assessment performance when compared with non- Littoral Combat Ship officers, in many cases. The Navy has relied on a series of added skill checks throughout a SWO’s career to help validate that SWOs have necessary ship-driving and other skills, but has not developed key processes and assessments to evaluate the overall effectiveness of its existing and planned training programs. The Navy is implementing a series of ten skill checks on ship-driving and other mariner tasks at various career points—for example, before a SWO begins leading a ship department and before the SWO takes command of a ship. The Commander, Naval Surface Forces, issued an instruction in September 2018 detailing ten skill checks to be conducted over the course of a SWO’s career to periodically gauge SWOs’ ship-driving skills. These checks, summarized in appendix IV, are to occur at standardized points in a SWO’s career, either during training or at the beginning or conclusion of certain at-sea assignments. Four of the ten checks were already in place at the time the instruction was issued in September 2018, with a preliminary version of a fifth check also in place. According to Navy documentation, three more of the ten checks had also been implemented as of August 2019, and Navy guidance states that the remaining checks are scheduled to be in place by 2021 or earlier. Navy officials stated they were making these checks more rigorous. For example, according to Navy officials, previously Department Heads were allowed to retake the Command Qualification Assessment ship-handling test as many times as they needed to pass the assessment. According to these officials, in 2018 Surface Warfare Officers School Command allowed only three chances to take the test, leading to five of the 256 Department Heads assessed in 2018 to be disqualified from advancing beyond the role of Department Head. Navy officials report that these skill checks are intended to enhance the development and sustainment of ship-driving proficiency across a SWO’s career and to ensure that the changes in training are resulting in competent SWOs at each level of their careers—essentially that SWOs have the skills required to perform their duties. Surface Warfare Officers School Command will administer checks during SWO training on ship- driving to better evaluate individual proficiency and target remediation for those whose performance presents significant concerns. Ship Commanding Officers will also observe and evaluate SWOs on a series of ship-driving scenarios before the completion of their first Division Officer assignment and later as a Department Head to certify that they are prepared for more advanced ship-driving training and responsibilities. While the planned skill checks are designed to help ensure that SWOs have the skills required to perform their duties, senior Navy officials stated that it could take 16 years or more to know if the planned changes to SWO training were effective in increasing Commanding Officer ship- driving proficiency across the fleet. These officials stated that they intend to closely monitor the implementation of changes to the training; however, we found a number of interrelated challenges that limit the Navy’s ability to determine in the near term if the significant investments it is making to expand and enhance SWO ship-driving training are effective. Specifically and described in detail below, in planning an approach for evaluating its efforts, the Navy has not (1) solicited fleet-wide feedback on the quality of the increased ship-driving training, (2) planned to routinely conduct ship- driving competency assessments, (3) provided standard criteria for qualifying Officer of the Deck candidates, and (4) determined how to analyze and use information from logbooks that SWOs are required to complete. The Navy’s Comprehensive Review of Recent Surface Force Incidents—one of the internal reviews completed after the 2017 mishaps—notes the importance of assessing and monitoring performance so that corrective actions can take place. In addition, federal government internal control standards state that management should use quality information and monitoring activities to ensure the agency’s objectives are achieved. Moreover, our prior work on assessing training efforts in the federal government states that an agency should evaluate the effectiveness of its training and development efforts, to include obtaining feedback, assessing competency, and analyzing relevant data. We found that while the Navy collects feedback from certain groups of SWOs, it did not have a formal fleet-wide process to solicit feedback from SWOs on the quality of the increased amount of ship-driving training or to gauge the health of the SWO community. In group discussions we held as part of our review, SWOs identified challenges that Division Officers experience in applying classroom and simulator training to their duties. According to SWOs in 19 of 24 group discussions with Department Heads and Division Officers, Division Officers have challenges in applying the ship-driving training they receive, due to factors such as differences between training curriculum and actual duties, extended lengths of time elapsed between training and application, varying ship-driving opportunities during Division Officer assignments, and difficulty retaining the large volume of course material. SWOs that participated in our discussion groups and interviews identified positive aspects of ship-driving training, as well as concerns about training material. During five of 12 ship group discussions with Division Officers, those Division Officers that had taken the Basic Division Officer Course identified positive aspects of the training such as valuable practical exercises and simulator time. However, SWOs in all 12 Division Officer group discussions also identified challenges related to this training, such as the information covered in training being too broad, and a lack of connection to actual duties on their ship. More experienced Division Officers in four of 12 Division Officer group discussions identified challenges related to the Advanced Division Officer Course, such as insufficient time in ship-driving simulators, and too much time spent covering material that Division Officers were already expected to learn during their first at-sea assignment. Commanding Officers and Executive Officers in seven of 12 interviews, and Department Heads in four of 12 group discussions likewise identified positive aspects of the Basic Division Officer Course, such as improved knowledge of ship operations for Divisions Officers that recently completed the course. However, Commanding Officers and Executive Officers in three of 12 interviews and Department Heads in seven of 12 group discussions identified challenges with the course, including areas where they had to compensate with on-the-job training for skills they felt should have been addressed in initial training, such as ship-driving proficiency in high-traffic environments. Our prior work on assessing training efforts in the federal government states that an agency should evaluate the effectiveness of its training and development efforts, to include obtaining and analyzing feedback. However, the Navy does not currently have a formal fleet-wide method of soliciting feedback from SWOs to obtain input on the quality of their classroom, simulator, and at-sea training on Division Officer performance and evaluate trends in feedback, and instead uses more limited means to assess training. For example: According to Navy officials, Surface Warfare Officers School Command conducts end-of-class surveys at the end of officer training, but no follow-up is conducted by the command after SWOs have assumed their ship duties or to obtain input from the trainees’ superior officers on the value of the training. The Navy had a survey for Division Officers and Department Heads in the past, but this survey gave little helpful feedback on training and, according to Navy officials, the Navy discontinued the survey after 2015. Surface Warfare Officers School Command assembles a board of officers from the fleet each year to review areas of its training curriculum, but Navy officials stated that participants are invited based on their expertise. As a result, only those selected to serve on the board (not officers across the fleet) have the opportunity to provide feedback. The Navy’s current means to assess training do not allow for the full range of junior and senior officers across the fleet to provide feedback on how well training prepares SWOs for their ship duties. Senior Navy officials acknowledged the value of conducting fleet-wide surveys of SWOs to obtain feedback on how to improve SWO training and gauge the health and morale of the SWO community. SWOs’ experiences in the fleet are diverse, therefore fleet-wide data is of particular value as centralized organizations like Naval Surface Forces, Surface Warfare Officers School Command, and the Office of the Chief of Naval Operations consider costly and consequential training investments. Without a method to regularly collect and analyze information from SWOs across the fleet, such as in a survey, regarding the quality of the increased classroom, simulator, and at-sea training on Division Officer performance, and evaluate trends in feedback received, Navy decision makers lack valuable information that could help them to assess the effects of training on SWO performance. Navy Surface Warfare Officer School Command training experts developed a ship-driving proficiency measurement system and used it in fiscal year 2018 to conduct ship-driving competency assessments. Specifically, from January through March 2018 Surface Warfare Officers School Command conducted “spot check” ship-driving competency assessments of 164 SWOs that had recently qualified as Officers of the Deck during their first at-sea assignment. Each assessment was conducted by three Navy inspectors that were independent of the assessed SWOs’ chain of command. The independent Navy inspectors found concerns in the ship-driving competency levels of more than 80 percent of these SWOs (see fig. 5). Specifically, Surface Warfare Officers School Command found that 29 SWOs (18 percent) had significant competency problems and 108 had some concerns (66 percent). According to Surface Warfare Officers School Command officials, those SWOs who experienced significant problems in their assessments likely should not have been qualified as Officer of the Deck at the time of the assessment because they violated fundamental ship-driving rules, among other issues. Navy guidance to the fleet emphasizes that these assessments performed by independent experts are valuable in supporting impartial results and providing quality information for analysis. According to Navy documentation, the Navy also used the 2018 competency assessments to help validate its new Junior Officer of the Deck and Officer of the Deck training curriculum. Specifically, Surface Warfare Officers School Command used the same Officer of the Deck competency assessment criteria to assess six officers in May 2018 and 12 in July 2018 that completed a pilot version of the Junior Officer of the Deck course. Surface Warfare Officer School Command found that the students with no at-sea experience that had completed a pilot of the new Junior Officer of the Deck training course in some cases outperformed qualified Officers of the Deck that had over a year of at-sea experience. According to Navy officials, the ability to compare ship-driving proficiency among populations and with earlier baselines using these competency assessments was valuable to the Navy in identifying the effects of changes to training, and could also be valuable in the future, as well. However, when we visited Surface Warfare Officers School Command in February 2019, officials told us they did not plan to conduct additional competency assessments until 2020. In meetings with Surface Warfare Officers School Command and senior Navy leaders, we noted that delaying additional assessments could limit visibility over ship driving proficiency trends and that small sample sizes could affect the Navy’s ability to make comparisons over time. In response, the Navy accelerated and expanded additional competency assessments. According to Navy officials, in spring 2019, Surface Warfare Officers School Command began to assess a sample of Division Officers using the Officer of the Deck competency assessment at the beginning of each Advanced Division Officer Course to collect and analyze performance data and refine training curriculum. Further, as of July 2019, the Navy had assessed 38 SWOs from three courses and found that the proficiency level of the SWOs assessed had not improved from the proficiency levels seen in the 2018 assessments. Senior Navy officials we met with as part of this review stated that they recognize the value in implementing periodic ship-driving competency assessments by independent inspectors to identify trends in ship-driving proficiency over time. However, we also found that the Navy has not planned to routinely conduct these assessments in the future. Specifically, in July 2019, Navy officials stated that they do not plan to complete these Officer of the Deck competency assessments beyond 2021 and plan to replace them with a different assessment at the end of the planned Officer of the Deck Phase II course. However, our analysis shows that mid-fiscal year 2024 is the first time Officer of the Deck Phase I course graduates will have completed their first at-sea assignment and be available to have their ship-driving training assessed, resulting in a multi-year gap in planned competency assessments. In order to measure the effectiveness of the full complement of Navy’s new and enhanced ship-driving training, the independent Navy inspectors will need to continue administering the Officer of the Deck competency assessments beyond 2021. In addition, an assessment performed at the end of training, such as the planned Officer of the Deck Phase II assessment, indicates the SWOs’ proficiency after additional training and may give a less accurate indication of prior at-sea proficiency. According to federal government internal control standards, management should use quality information and monitoring activities to ensure the entity’s objectives are achieved. Moreover, our prior work on assessing training efforts in the federal government states that an agency should evaluate the effectiveness of its training and development efforts, to include assessing competency and analyzing relevant data. Without routinely conducting Officer of the Deck competency assessments across the fleet using samples of sufficient size and selection methods, the Navy will be hindered in its ability to gauge fleet-wide ship driving proficiency trends and determine the effectiveness of the changes made to training, and the Navy may not know whether additional changes are needed. We found that the Navy has not provided standard criteria to ship Commanding Officers on fleet-wide ship driving proficiency expectations to inform the qualification of Officer of the Deck candidates. Instead, the Navy has determined that ship Commanding Officers should use their individual judgment in granting this qualification based on a set of required officer experiences, which the Navy refers to as Personnel Qualification Standards. Following the 2017 collisions, Surface Warfare Officers School Command developed proficiency standards to measure and test Officer of the Deck ship-driving proficiency to implement the Officer of the Deck competency assessments described above. The proficiency standards require an Officer of the Deck to demonstrate knowledge of navigation systems, rules of the road, and effective bridge resource management and to demonstrate the ability to successfully navigate high-traffic environments. However, the varying at-sea experiences of officers and subjective nature of some requirements have led to different experiences for SWO candidates working to qualify as Officer of the Deck. SWOs must complete a standard series of requirements in ship-driving and other experience before they are eligible to qualify as Officer of the Deck, with Commanding Officers granting qualification after their assessment of the SWO’s performance and fitness. However, when we held group discussions with SWOs on ships in the fleet, SWOs in nine of 12 group discussions with Division Officers, eight of 12 group discussions with Department Heads, and three of 12 interviews with Commanding Officers and Executive Officers identified significant differences in opportunities, experiences, and assessments that Division Officers experience in earning their qualification as Officers of the Deck during their first Division Officer assignment. For example: In one group discussion, Division Officers reported being qualified as Officers of the Deck without ever having stood watch at sea, with the Commanding Officer granting qualifications based on their classroom and simulator experience alone. In five of 12 group discussions with Division Officers, Division Officers stated that SWOs on ships in maintenance had few opportunities to stand watch on the bridge at sea to build proficiency in difficult ship- driving operations, but still received their qualifications. SWOs in 17 of 24 group discussions stated that some Division Officers get more ship driving experience than others before earning their Officer of the Deck qualifications. For example, Division Officers assigned to ships with more time at sea or fewer Division Officers get more experience to practice ship driving than those on ships with little time at sea or that must divide ship-driving opportunities among numerous Division Officers. Commanding Officers in three of 12 interviews reported that they had to temporarily place their Division Officers on other ships to gain qualifying experience, and had to rely on the judgment of the other ships’ Commanding Officers in determining their qualifications as Officers of the Deck. According to Navy officials, the Navy has not provided Officer of the Deck assessment criteria based on the developed proficiency standards to ship Commanding Officers, out of deference to their judgment in interpreting an officer’s preparedness to drive their ship. Navy officials emphasized the importance of allowing ship Commanding Officers to make their own determination of an officer’s preparedness to drive a ship, due to their knowledge of the ship’s operating conditions. Navy officials also stated that they considered the Officer of the Deck assessment standards to be a resource for use by Surface Warfare Officer Schools Command in assessing training curriculum and had not considered using the standards in the fleet for other purposes. However, the Navy’s 2018 and 2019 Officer of the Deck competency assessments identified significant variance in the ship-driving competency levels of recently qualified Officers of the Deck. Since the Navy has developed fleet-wide standards for assessing Officer of the Deck proficiency, the Navy could use these to provide standard Officer of the Deck assessment criteria in guidance to ship Commanding Officers. Federal government internal control standards state that management should internally communicate the necessary quality information to achieve the entity’s objectives and ensure decisions are made based on consistent standards. A SWO’s assigned ship, Commanding Officer, and operating conditions may change during a career, so a standard set of criteria would help Commanding Officers to determine what is expected of Officers of the Deck elsewhere in the fleet as they determine a junior officer’s qualification. Without providing standard Officer of the Deck assessment criteria and incorporating them into surface fleet guidance to Commanding Officers, the Navy risks creating uncertainty in Officer of the Deck qualification expectations— which can contribute to variations in ship-driving proficiency among SWOs that could jeopardize safe operations at sea. In September 2018, the Commander, Naval Surface Forces, U.S. Pacific Fleet and Commander, Naval Surface Forces Atlantic, began requiring SWOs to document their ship-driving and related experience in a handwritten logbook. The logbook—referred to as the Surface Warfare Mariner Skills Logbook (see fig. 6)—captures an officer’s experience gained during each watch aboard a ship, special evolution (e.g., underway replenishment, flight operations, and sea and anchor duty), and simulator training session. During our ship discussion groups, SWOs at the Division Officer and Department Head levels reported that they had begun filling out their logbooks and having them reviewed as required, but some acknowledged that they are inconsistently filling them out or that they were not entering any information in them. Specifically, SWOs in five of 24 discussion groups reported that logbooks are completed with inconsistent quality or not completed at all. Additionally, SWOs in four of 24 discussion groups reported that they are unaware of any plans to use the logbook information to identify any additional training needs and provide opportunities for SWOs to improve their ship-driving proficiency. Navy Personnel Command officials told us that, as of July 2019, they had received 174 summaries of Surface Warfare Mariner Skills Logbook data from Commanding Officers. Navy officials stated that over time, as they gather these data, they intend to examine the link between ship-driving proficiency and SWO experience. However, officials did not have any specific, measurable plans to analyze and use these data or to assess the completeness of these data. Federal internal control standards state that management should obtain relevant data from reliable sources and process those data into quality information to aid decision-making. Furthermore, Naval Surface Forces guidance states that the surface warfare community should analyze and use logbook data to link SWO experience with ship-driving proficiency. Despite this guidance, the Navy does not yet have a plan that includes specific steps to analyze and use logbook information to link SWO experience with ship-driving proficiency. According to senior Navy officials, while the Surface Warfare Mariner Skills Logbook is still relatively new, developing a plan to use the information would be a logical next step. Without a plan for analyzing and using Surface Warfare Mariner Skills Logbook data, the Navy cannot determine the relationship between SWO experience and ship-driving proficiency or use these data to aid decision-making. SWOs play a critical role in Navy surface fleet readiness, as they are responsible for safely driving ships at sea and successfully leading ships in Navy operations across the world. The Navy is making numerous changes and investments to enhance Surface Warfare Officer ship- driving training following the 2017 collisions at sea—with plans to triple initial training hours and spend nearly half a billion dollars to build simulator capacity to deliver this training. The Navy’s oversight of these efforts is centered on a series of added checks throughout SWOs’ careers to ensure that they have basic ship-driving and other skills. These checks are steps in the right direction but may not provide adequate assessment mechanisms in the near term and might lead to missed opportunities going forward. For example, the Navy is expanding its ship- driving training but is not planning to collect fleet-wide feedback on classroom, simulator, and at-sea training received. In addition, the Navy developed standards for conducting spot checks on ship-driving competency but is planning to stop those checks in 2021, missing an opportunity for an outside assessment and to evaluate how well new and updated training is working. Moreover, ship commanders are expected to qualify SWOs on ship driving but have not been provided standard guidance for how to do this, which can contribute to wide variations in SWO competence. Finally, the Navy has developed detailed logbooks for SWOs to track their experiences but the Navy has not developed a specific plan to analyze and use the logbook data. Without actions to address these challenges, the Navy cannot fully assess in the near term if the significant investments it is making to expand and enhance SWO ship-driving training are effective; further adjustments are necessary; and, ultimately, Navy ships are being operated safely at sea. We are making the following four recommendations to the Department of Navy: We recommend that the Secretary of the Navy ensure that the Commander, Naval Surface Forces, in coordination with Surface Warfare Officers School Command, develop a method to regularly collect feedback from SWOs across the fleet, such as in a survey, regarding the quality of their classroom, simulator, and at-sea training on Division Officer performance; and evaluates trends in the feedback received for the purpose of improving SWO training. (Recommendation 1) We recommend that the Secretary of the Navy ensure that the Commander, Naval Surface Forces, routinely conduct regular Officer of the Deck competency assessments using samples of sufficient size and using selection methods to gauge the level of fleet-wide ship-driving proficiency trends following the implementation of the planned ship- driving training programs. (Recommendation 2) We recommend that the Secretary of the Navy ensure that the Commander, Naval Surface Forces, in coordination with Surface Warfare Officers School Command, provide Commanding Officers with standard criteria to inform their evaluation of candidates for their Officer of the Deck qualification and incorporates these criteria into surface fleet guidance. (Recommendation 3) We recommend that the Secretary of the Navy ensure that the Commander, Naval Surface Forces, in coordination with Surface Warfare Officers School Command, develop a plan to analyze and use Mariner Skills Logbook information to inform decision-making. (Recommendation 4) We provided a draft of this report to DOD for review and comment. In written comments provided by the Navy through DOD (reprinted in their entirety in appendix V), the Navy concurred with all four of our recommendations and identified actions it plans to take to evaluate the effectiveness of changes to SWO training. The Navy also provided additional information and context in its comments and provided technical comments, which we incorporated as appropriate. The Navy concurred with our first recommendation that the Commander, Naval Surface Forces, in coordination with Surface Warfare Officers School Command, develop a method to regularly collect feedback from SWOs across the fleet, such as in a survey, regarding the quality of their classroom, simulator, and at-sea training on Division Officer performance; and evaluate trends in the feedback received for the purpose of improving SWO training. The Navy stated that it plans to explore additional means of garnering holistic SWO feedback regarding newly-implemented SWO training and assessments as well as gathering additional targeted feedback. However, the Navy stated that the use of performance data will remain the primary focus of surface force training improvement efforts. While using performance data is valuable, it will be important that the Navy follow through to develop a holistic means of collecting feedback, such as in a survey of SWOs across the fleet, on the effectiveness of Division Officer training on SWO performance to ensure a variety of perspectives are considered. In its comments, the Navy noted that the SWOs who participated in our ship visits and discussion groups had not experienced the changes made or planned to SWO training. We agree that our ship visits did not include officers who had experienced the expanded Division Officer training courses, as they were first introduced to the fleet in June 2019, after we had completed the majority of our work. While our discussion groups pre- date the implementation of new SWO training courses, the discussion groups we conducted with over 200 SWOs reinforced the our finding that the Navy needs to develop a method to regularly collect feedback from SWOs across the fleet. Also, the Navy plans to more than triple initial training, so routinely soliciting and analyzing feedback from SWOs on Division Officer training will be needed to determine the effectiveness of the Navy’s investments in these training programs and inform the Navy’s decisions on whether further adjustments are necessary. The Navy acknowledged that only officers participating in the Surface Warfare Officers School Command’s Board of Visitors provide direct feedback on the training curriculum. The Navy noted, however, that all available Surface Warfare units are invited to participate in Surface Warfare Officer School Board of Visitors events and so could provide feedback then. In addition, the Navy noted that Surface Warfare Officer School Command also solicits feedback through visits to fleet concentration areas and through semiannual symposiums of ship Commanding Officers. While such targeted means of collecting feedback may provide valuable information, officers may not be able to participate due to their deployment status, position on shore duty, timing of events during other personal responsibilities, or other factors. We believe that developing a method to regularly collect feedback from SWOs across the fleet would provide decision makers with valuable information that could help them assess the effects of training on SWO performance. The Navy concurred with our second recommendation that the Commander, Naval Surface Forces, routinely conduct regular Officer of the Deck competency assessments using samples of sufficient size and using selection methods to gauge the level of fleet-wide ship-driving proficiency trends following the implementation of the planned ship- driving training programs. The Navy stated that it plans to routinely collect and analyze standardized mariner skills performance data across an officer’s career path. However, the Navy stated it will use training checks, rather than the current Officer of the Deck competency assessment, to evaluate SWO performance after 2020. This presents two problems in meeting the intent of our recommendation. First, the Navy will need to ensure that the training checks are sufficiently rigorous to assess competency. Second, the Navy will not have valid data to compare the effects of training changes on competency if it changes its assessment approach. In 2018, the Navy used the Officer of the Deck competency assessment to establish a baseline of SWO ship-driving proficiency. We found that the 2018 competency assessments showed significant variation in ship- driving proficiency and the 2019 follow up assessments found that competency had not improved. In currently documented plans, the Officer of the Deck Phase II check after a Division Officer’s first assignment will occur at the end of the course. Even if the Navy changes the Officer of the Deck Phase II check to occur at the beginning of training as stated in its comments, performance data from this check cannot be directly compared with the results of the current competency assessment. Differences in assessment content or difficulty, remediation attempts, and the fact that the new check may have career implications for SWOs as a go/no-go assessment may affect proficiency measurements and pass rates. Due to these factors, we believe that a comparison between the current Officer of the Deck competency assessment and the planned Officer of the Deck Phase II check or another standard should not be considered as valid means for demonstrating changes in ship-driving proficiency over time. That is, adopting a new standard may affect the Navy’s ability to determine the impact of training on ship-driving proficiency compared with the 2018 baseline results. The Navy also stated in its comments that the SWO training and assessment continuum is designed to provide training and evaluation at all career milestone levels. The planned system of additional skills checks will provide the Navy with more insight into SWO proficiency levels over the course of an officer’s career and help the Navy to understand the effects of changes to training. As we stated in the report, we believe these checks are significant steps in the right direction but may not provide adequate assessment mechanisms in the near term. The more robust Officer of the Deck competency assessments are necessary to gauge the level of fleet-wide ship-driving proficiency trends following the implementation of the planned ship-driving training programs. Further, the Navy stated in its comments that while numerous means of assessing SWO mariner skills proficiency at various milestone levels are in place, the ultimate SWO career path goal is to develop the most proficient, experienced, and confident Commanding Officers, which occurs approximately 16 years into the SWO career path. While the quality of ship Commanding Officers is a vital component of Navy readiness and capability, the majority of SWOs do not remain in the Navy long enough to advance beyond the position of Division Officer, according to Navy documentation. Similarly few advance to the position of Commanding Officer during their career as a SWO. Since Division Officers constitute over one third of the SWO workforce and by design of the SWO career path do most of the ship-driving, it is of utmost importance to build and evaluate fundamental ship-driving skills for all Division Officers to support excellence in the ship-driving proficiency across the Navy. Finally, the Navy stated in its comments that our report language implies an absence of any Officer of the Deck assessments from 2021 through 2024. We acknowledge that the Navy will conduct assessments of Officers of the Deck during this time period in line with its planned system of ten checks over a SWO’s career. However, without maintaining the current Officer of the Deck competency assessments through at least 2024, the Navy will be unable to demonstrate any proficiency improvement, compared with the 2018 baseline, resulting from its new training programs. Further, the Navy stated in its comments that it is important to clarify that the SWOs who received competency checks in 2019 had not benefitted from the new and expanded ship-driving training courses. Our ship visits did not include officers who had experienced the expanded Division Officer training courses, as they were first introduced to the fleet in June 2019. Nonetheless, it is concerning that SWO competency had not improved in the 2 years since the 2017 collisions despite the fleet-wide attention to improving ship-driving skills. The Navy concurred with our third recommendation that the Commander, Naval Surface Forces, in coordination with Surface Warfare Officers School Command, provide Commanding Officers with standard criteria to inform their evaluation of candidates for their Officer of the Deck qualification and incorporate these criteria into surface fleet guidance. However, the Navy stated that such criteria are already in place. Specifically, the Navy noted that existing Personnel Qualification Standards provide the standard evaluation criteria for the Officer of the Deck qualification. We agree that the Personnel Qualification Standards are in place, but disagree that Qualification Standards provide standard evaluation criteria. Unless the Navy provides additional guidance for Commanding Officers to measure proficiency in addition to the list of required experiences present in the Personnel Qualification Standards, the actions the Navy identified as addressing our recommendation will not meet the intent of our recommendation. The Navy’s Officer of the Deck Personnel Qualification Standards provide a list of required experiences; however, the 2018 and 2019 competency assessments indicate that these existing criteria have not resulted in high levels of proficiency among Officers of the Deck. The Navy’s Personnel Qualification Standards do not require SWOs to demonstrate a standard level of proficiency, but rather that SWOs participate in a required number of ship-driving experiences at a level determined by his or her Commanding Officer. The absence of a common proficiency standard across the Navy may contribute to inconsistency in ship-driving skills among SWOs. Since the Officer of the Deck competency assessment provides a means to measure proficiency, communicating appropriate standards in line with those used in the current assessments as qualification criteria would help ensure a common understanding of proficiency expectations. In comments, the Navy stated that for junior officers whose ships experience maintenance periods, it is an historic surface force-wide practice for Commanding Officers to temporarily assign those officers to similar ships whose operational schedule better support qualification. This practice is understandable and may contribute to SWO career development, but can lead to significant differences in opportunities, experiences, and assessments that SWOs receive during their first Division Officer assignment. For example, as noted in our report, some Commanding Officers stated because of this temporary assignment, they had to rely on the judgment of the other ships’ Commanding Officers to determine their SWOs’ qualifications as Officers of the Deck. The Navy concurred with our fourth recommendation that Commander, Naval Surface Forces, in coordination with Surface Warfare Officer School Command, develop a plan to analyze and use Mariner Skills Logbook information to inform decision-making. The Navy noted that it would comprehensively evaluate performance data relative to Mariner Skills Logbook data in order to refine mariner skill milestone performance and proficiency criteria. If Navy efforts result in a plan that includes specific and measurable steps for analyzing and using Mariner Skills Logbook data, the efforts will meet the intent of our recommendation. In its comments the Navy stated that during the time we conducted our group discussions (i.e. January through April 2019), Mariner Skills Logbooks were still being introduced to the Fleet and recording practices were still being established. While at the time of our discussion groups the Mariner Skills Logbooks were relatively new, in September 2018, the Navy issued an instruction that established guidance for the implementation and use of the logbooks. In addition, all of the SWOs we met with as part of our review had already received their Mariner Skills Logbooks. We are sending copies of this report to congressional committees, the Secretary of Defense, the Secretary of the Navy, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3489 or pendletonj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. Following the four 2017 mishaps at sea, the Navy completed two internal reviews on surface fleet readiness, ultimately compiling 111 recommendations for improvement. The Navy established a Readiness Reform and Oversight Council under the leadership of the Vice Chief of Naval Operations to oversee implementation of these recommendations. The Readiness Reform and Oversight Council reported in February 2019 that it considered 91 of these recommendations to be implemented. We reviewed the recommendations, identified 12 recommendations related to Surface Warfare Officer (SWO) initial ship-driving training, and requested the implementation status of each of these recommendations from the Commander, Naval Surface Forces. The Navy considers a recommendation to be “implemented” when there is a policy in place or action has been taken to address a recommendation. The Navy considers a recommendation to be “transitioned” when the Readiness Reform and Oversight Council no longer maintains regular oversight of a recommendation and has transitioned oversight to another Navy organization. As of August 2019, the Navy considered all 12 of the recommendations related to ship-driving training as implemented with the final recommendation estimated to transition by September 30, 2019. Table 1 lists the 12 recommendations related to SWO initial ship-driving training, and our summary of the Navy’s explanation for why they are considered to be implemented. The John S. McCain National Defense Authorization Act for Fiscal Year 2019 and Senate Armed Services Committee report 115-262 to accompany a bill for the National Defense Authorization Act for Fiscal Year 2019 contained provisions that we review Surface Warfare Officer (SWO) training and career paths. This report (1) describes the changes the Navy has made or planned to SWO ship-driving training since the 2017 collisions and (2) assesses the extent to which the Navy has taken actions to evaluate the effectiveness of those changes. We plan to issue a separate report on SWO career paths in the future. For objective one, we reviewed Navy documentation from Commander, Naval Surface Forces, Surface Warfare Officers School Command, and the Office of the Chief of Naval Operations on the content, purpose, cost, and status of changes made and further changes planned to ship-driving training since the 2017 collisions. We focused our analysis on changes made to SWOs’ ship-driving training at the junior officer level as the Navy prioritizes ship-driving training and ship-driving experience for junior officers, and the Navy has identified actions it is taking to address recommendations from the Navy’s two 2017 internal reviews to ensure safe operations at sea through improvements to junior officer training. We analyzed planned investments from fiscal year 2018 through fiscal year 2025 for the construction of two ship-driving training facilities and the development of three ship-driving training courses, which includes the cost of purchasing new simulators, hiring new instructors, military construction, and course curriculum development. We discussed implementation plans for the 2017 internal reviews’ recommendations with the Commander, Naval Surface Forces; officials from the Surface Warfare Officers School Command; and officials from the Readiness and Reform Oversight Council, a group within the Office of the Chief of Naval Operations established to monitor the implementation of the internal reviews’ recommendations. For objective two, we reviewed Navy documentation and interviewed Navy officials on how they evaluate SWOs throughout their careers, gather and use feedback from SWOs, assess the effectiveness of SWO ship-driving training, and use available data to inform decisions regarding SWO training. Specifically, we reviewed the implementation of the 10 career milestone checks outlined in Naval Surface Forces Instruction 1412.5 Surface Warfare Officer Milestone Mariner Skills Assessments, Evaluations, and Competency Checks that are to be administered during a SWO’s career; Navy’s efforts to collect feedback from the surface fleet on the quality of SWO ship-driving training and the health of the SWO community; Navy’s 2018 Officer of the Deck competency assessment results, criteria, and plans to continue evaluating SWO ship-driving competency; extent to which the Navy had provided standardized criteria for ships’ Commanding Officers to use when evaluating SWO’s for ship-driving qualification; and format of SWO Mariner Skills Logbooks used to track SWO ship- driving experiences, and Navy policies regarding the logbooks. To do this we compared the Navy’s practices with relevant Navy reviews, instructions, and guidance, Standards for Internal Control in the Federal Government, and our prior work on assessing training efforts in the federal government. We assessed the reliability of the results of the Navy’s 2018 Officer of the Deck competency assessments by examining them for missing values, comparing other sources that provide the same types of data to ensure consistency, and interviewing knowledgeable agency officials regarding the assessments’ accuracy and completeness. In addition, we reviewed the Navy’s internal controls for performing the assessments, such as grading criteria and use of independent inspectors to ensure quality and consistency in the information. We determined that the results of the Navy’s 2018 Officer of the Deck competency assessments were sufficiently reliable for the purposes of reporting on the number and percentage of the graded categories. In addition to meeting with Navy offices, we visited 12 surface ships in the Pacific and Atlantic fleets from January through April 2019, selected according to which ships and crews were available at each of the sites we visited. Aboard the ships we held group discussions and interviews with approximately 225 SWOs to discuss their views on the sufficiency and appropriateness of SWO training. Discussion group sizes ranged from two to 20 SWOs. In conducting these group discussions, we held 24 group discussions, with two separate discussions for each of the 12 ships—one with Department Heads and one with Division Officers; interviewed Commanding and Executive Officers aboard each of the 12 ships, where available; and conducted each group discussion without the group’s supervisors or subordinates present. The ship crews we visited were those the Navy identified as available to hold group discussions with us during site visits, and the results of these group discussions are not generalizable to anyone outside these groups. Due to the timing of our work, the interviews and group discussions did not include SWOs that experienced changes made or planned for SWO training beyond April 2019. We asked each group a standard set of questions to obtain their views on the following topics: the sufficiency and appropriateness of SWO training programs in preparing SWOs for their ship responsibilities, including ship driving; the SWO career path, including the potential benefits and drawbacks of more specialized career paths; and any opportunities to improve the SWO community. We conducted an analysis of the discussion group responses to identify common themes and provide illustrative examples in our report. Specifically, we reviewed the responses received during discussion groups, grouped the responses by themes, and counted how many discussion groups and interviews provided similar feedback to our questions. One GAO analyst conducted this analysis, coding the information and entering it into a record of summary, and a different GAO analyst checked the information for accuracy and agreement on themes. Any initial disagreements in the coding were discussed and reconciled by the analysts. The analysts then tallied the responses to determine the extent to which the certain themes were covered during our discussion groups and interviews. We interviewed officials, or where appropriate, obtained documentation at the organizations listed below: Office of the Chief of Naval Operations Director of Surface Warfare (N96) Surface Warfare (N96) Manpower and Training Readiness Reform and Oversight Council Commander, Naval Surface Forces, U.S. Pacific Fleet Littoral Combat Ship Training Facility Navigation, Seamanship, and Ship-handling Training facility USS Ardent (MCM 12) USS Lake Champlain (CG 57) USS New Orleans (LPD 18) USS Paul Hamilton (DDG 60) USS Tulsa (LCS 16) Commander, Naval Surface Forces, Atlantic Navigation, Seamanship, and Ship-handling Training facility USS Bataan (LHD 5) USS Cole (DDG 67) USS Mahan (DDG 72) USS Mesa Verde (LPD 19) USS Oak Hill (LSD 51) USS San Antonio (LPD 17) USS San Jacinto (CG 56) Surface Warfare Officer School Command Basic Division Officer Course facilities—San Diego, California and Surface Warfare Officer (PERS-41) We conducted this performance audit from November 2018 to November 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Commander, Naval Surface Forces, issued an instruction in September 2018 listing ten ship-driving skill checks to be conducted periodically over the course of a Surface Warfare Officer’s (SWO) career. Failure to pass some of the checks can result in required remediation or disqualification from career advancement. Table 2 lists the ten current and planned checks as of August 2019, as well as information on their timing and content as described in the instruction. In addition to the contact named above, Chris Watson (Assistant Director), Tobin McMurdie (Analyst-in-Charge), David Beardwood, Vincent Buquicchio, Mae Jones, Amie Lesser, Shahrzad Nikoo, Michael Silver, and Brandon Voss made key contributions to this report.
In 2017, the Navy had four mishaps at sea including two collisions that resulted in the loss of 17 sailors' lives and hundreds of millions of dollars in damage to Navy ships. In the wake of those mishaps, the Navy identified deficiencies in SWO ship-driving training and related experience as contributing factors and has undertaken a number of efforts to improve these areas. Senate Report 115-262, accompanying a bill for the Fiscal Year 2019 National Defense Authorization Act, contained a provision that GAO assess SWO training. This report (1) describes the changes the Navy has made to SWO ship-driving training since the 2017 collisions and (2) assesses the extent to which the Navy has taken actions to evaluate the effectiveness of changes made to SWO ship-driving training. GAO reviewed and analyzed changes made to Navy training and assessment practices and related investments; interviewed cognizant officials; and conducted discussions with SWOs aboard 12 ships. Since 2017, the Navy has made numerous changes and plans additional changes to enhance Surface Warfare Officer (SWO) ship-driving training. The Navy plans for these changes to result in a threefold increase in the number of initial ship-driving training hours for SWOs by 2021, compared with the number of training hours prior to the 2017 collisions (see fig.). The Navy added classroom and simulator time to existing training courses to improve ship-driving skills and is developing two additional simulator-based ship-driving courses planned for 2021. These plans hinge on the completion of two new simulator-based training facilities, scheduled for completion in June 2021 and in January 2023. The Navy has relied on added skill checks conducted throughout a SWO's career to ensure that each SWO has basic ship-driving skills, but has not put key processes and assessments in place to evaluate comprehensively the effectiveness of its changes to ship-driving training. Senior Navy officials stated that it could take 16 years or more to know if the planned changes to SWO training were effective in increasing Commanding Officer ship-driving proficiency across the fleet and stated that they intend to closely monitor the implementation of changes to the training. However, GAO found that in planning an approach for evaluating the changes, the Navy has not: (1) identified a method to solicit fleet-wide feedback on the quality of the increased ship-driving training received by SWOs; (2) planned to routinely conduct ship-driving competency “spot checks” that were instituted after the 2017 collisions despite Navy inspectors having found concerns with more than 80 percent of SWOs' ship-driving skills; (3) provided standard criteria to ship Commanding Officers for qualifying SWOs to drive ships, contributing to significant variance in ship-driving experience and competency levels across the fleet; nor (4) developed a specific plan to analyze and use information from logbooks in which SWOs are to document ship-driving and related experience. Without addressing these challenges, the Navy cannot assess in the near term if the significant investments made to expand and enhance SWO ship-driving training are effective; further adjustments are necessary; and Navy ships are being operated safely at sea. GAO is making four recommendations to the Navy to routinely evaluate SWO training, including that the Navy collect and evaluate fleet-wide feedback on the quality of training; routinely conduct ship-driving competency assessments; provide standard criteria for qualifying ship drivers; and develop a plan to analyze and use logbook information. The Navy concurred with GAO's recommendations.