Increasingly, PME schools are using  ADL techniques in their nonresident program offerings. DOD Does Not Have Specific Metrics for Assessing Performance Goals or Learning Outcomes

DOD does not have specific performance goals and metrics to assess the  effectiveness of ADL in PME. Instead, DOD accreditation activities, like  those in the private sector, focus primarily on educational process inputs,  such as quality of facilities and student faculty ratios. We and a private- sector organization have recently established guidelines and frameworks  for assessing the effectiveness of educational programs that stress a focus  on measurable outcomes. ADL is a new and evolving tool for which  systematic evaluation requirements have not been established. The third  agency required schools to evaluate its nonresident programs to ensure  comparability to resident programs. Metrics for Learning Outcomes Are Lacking

Student learning outcomes, as stated by the Council for Higher Education  Accreditation—a national association representing accrediting  organizations—are “properly defined in terms of the knowledge, skills, and  abilities, that a student has attained at the end (or as a result) of his or her  engagement in a particular set of higher education experiences.” PME  schools generally are not assessed for student learning outcomes as a  means of determining program effectiveness. Because we only surveyed nonresident students, we  could not compare the results with those of resident students. However, we found no evidence to indicate that DOD is using  this ability. ADL Conversion Varied by School and by Service Based on Subjective Assessments of Content Suitability

The processes for converting PME courses to ADL varied by school and by  military service, and they feature a mixture of in-house and contractor  approaches. PME schools did not identify any systematic criteria as  the basis for their decisions as to which courses or curricula to convert to  ADL. Cultural, Technological, and Resource Barriers and Challenges Affect ADL Implementation in PME Programs

PME schools identified a number of cultural, technological, and resource  challenges that affect ADL implementation and may affect future  maintenance or expansion of ADL efforts. An emerging funding issue  involves the use of copyrighted material in ADL applications. Without clear  goals and an effective process for evaluating the results of ADL application,  DOD cannot ensure that ADL is achieving appropriate return on  investment, student retention, student access, and other goals in  comparison with prior efforts. Scope and Methodology

We reviewed Department of Defense’s (DOD) implementation of advanced  distributed learning (ADL) in senior- and intermediate-level professional  military education (PME) to determine processes and criteria used for  converting courses, metrics to assess ADL effectiveness and its fulfillment  of learning objectives, and barriers and challenges in ADL implementation. GAO’s  commitment to good government is reflected in its core values of  accountability, integrity, and reliability.
Why GAO Did This Study

As part of its transformation to prepare the armed forces to meet current and future challenges, the Department of Defense (DOD) is expanding its use of advanced distributed learning (ADL) techniques in senior- and intermediate-level officer professional military education (PME). ADL instruction does not require an instructor's presence, and it facilitates the use of varied learning management systems. To date, the application of ADL has been targeted to nonresident students. To determine whether DOD uses a systematic process for evaluating the results of ADL application, GAO was asked to examine DOD's metrics for assessing program effectiveness, to compare DOD's criteria for converting courses to ADL with those of private-sector institutions, and to identify the challenges to ADL implementation.

What GAO Found

DOD does not have specific performance goals and metrics with which to assess ADL effectiveness in PME. Furthermore, although GAO and private-sector organization have established frameworks for assessing the effectiveness of educational programs by focusing on metrics for learning outcomes--that is, the knowledge, skills, and abilities that students attain through learning activities--DOD's oversight focuses instead on educational inputs such as facilities, student to faculty ratios, and student body composition. Since ADL is still a new and evolving tool, systematic evaluative processes have not yet been required. Without clear goals and an effective process for evaluating the results of ADL application, DOD cannot ensure that its program is achieving an appropriate return on investment and other goals. The criteria for converting PME courses and curricula to ADL vary by school and by military service, are based on subjective choices as to which content is suited for online delivery, and are focused solely on nonresident programs. The private sector similarly lacks systematic criteria in its use of ADL. However, DOD's implementation of ADL programs for PME compares favorably with private-sector institutions. Cultural, technological, and resource challenges affect ADL implementation. For example, some military policies reflect a lower estimation of the value of nonresident PME, and many respondents to a survey of ADL students and alumni indicated that its quality and achievement of outcomes did not compare favorably, in their view, with those of resident education programs. The technological challenges of balancing computer access with network security, along with resource challenges of funding and increased burdens on limited administrative staff, are additional concerns.