DOD’s Study Did Not Consistently Follow Relevant Generally Accepted Research Standards

Our analysis of the Study found that it followed some, but not all, of the  relevant generally accepted research standards for research design and  execution. The Study did not sufficiently mitigate limitations concerning its  analysis of the required number of active-duty medical and civilian  medical personnel. The Study included a limited assessment of clinical currency—the  requirements necessary to maintain the skills of active-duty medical  providers. The Study Did Not Fully Mitigate Limitations Concerning Its Analysis of the Required Number of Active-Duty and Civilian Medical Personnel

The Study presented data on the size and composition of the wartime  active-duty medical force, also known as the “operational force.”  However, the data were largely drawn from service models with known  problems and not based on new analysis conducted by the Study team. While the Study clearly outlined the limitations of  its service-oriented approach, the Study did not fully mitigate them, such  as by explaining how these limitations affected the results of the analysis. Absent such efforts to address these limitations,  DOD will not have a full assessment of its medical workforce needs. Having accurate workload information is important  to establishing a sound standard for maintaining the clinical skills of  medical providers. The Study Established Goals for Transferring Health Care into Its Own Network of Hospitals and Clinics and Increasing the Productivity of Providers without a Strategy Explaining How These Goals Would Be Achieved

The Study, in developing its findings and recommendations, assumed  that DOD would be successful in its goals for transferring health care from  patients being treated in DOD’s purchased private-sector network into its  own military treatment facilities, as well as in achieving related goals for  improving provider productivity. Without a strategy that incorporates leading practices, it  remains unclear whether or how DOD can achieve its goals to transfer  health care from the purchased care network into its own network of  facilities. DOD’s Estimate of Cost Savings from Changes to Small Hospitals Did Not Fully Reflect Key Practices

DOD’s estimated cost savings for the recommended changes to 10 small  hospitals and the related achievement of goals related to transfer of  health care to DOD facilities and increased active-duty provider  productivity included a net estimate of savings and costs, but did not fully  reflect key practices for cost and cost savings estimates. First, the Study did not provide an appropriate level of detail concerning  the calculation of the estimated cost savings or the assumptions upon  which they were based. Second, the Study’s predicted net savings estimate did not include all  potentially significant costs. Construction costs, if applicable, at facilities losing or gaining active- duty providers: The Study recommended that a number of inpatient  facilities be closed or converted to ambulatory surgery centers, but  our analysis found that the Study did not identify estimated costs  associated with these changes. The Study’s recommendations for  further analysis and work in a number of areas position DOD, over time,  to explore these areas and take actions to improve the effectiveness and  efficiency of the MHS. To strengthen ongoing efforts to analyze the costs of medical force  readiness and establish clinical currency standards, we recommend that  the Secretary of Defense direct the Assistant Secretary of Defense  (Health Affairs) to take steps to identify and mitigate limitations regarding  the standard for maintaining providers’ clinical skills, including improving  the accuracy of information concerning providers’ workload and  conducting an analytically rigorous calculation of active-duty providers’  time devoted to military-specific responsibilities. To strengthen any future assessments of additional changes to DOD’s  network of military treatment facilities, we recommend that the Secretary  of Defense direct the Assistant Secretary of Defense (Health Affairs) to  take the following two actions: describe steps taken to assess the reliability of data supporting the  assessment, including, at a minimum, the sources of data, data  limitations, and efforts to test data reliability; and  include in any accompanying cost estimates an appropriate level of  detail, all significant costs, and an assessment of the reliability of the  data supporting the cost estimate. Appendix I: Selection of Generally Accepted Research Standards Relevant for the Military Health System Modernization Study

To guide our review of the Department of Defense’s (DOD) Report on  Military Health System Modernization (the Study), we identified relevant  generally accepted research standards for the design, execution, and  presentation of findings that define a sound and complete study. a. Defense Health Care: Applying Key Management Practices Should Help  Achieve Efficiencies within the Military Health System.
Why GAO Did This Study

DOD initiated the Study to address perceived weaknesses within the Military Health System and to leverage advances in civilian business practices. The National Defense Authorization Act for Fiscal Year 2015 included a provision for DOD to submit the Study to the congressional defense committees and for GAO to review the Study. DOD submitted its study in February 2016. This report assesses, among other things, the extent to which the Study followed an approach that is consistent with relevant generally accepted research standards and utilized key practices for estimating cost savings. GAO compared the Study with generally accepted research standards that were developed by reviewing research literature and DOD guidance and with key practices derived from cost-estimating guidance.

What GAO Found

The Department of Defense's (DOD) approach in its Report on Military Health System Modernization (the Study) did not consistently follow relevant generally accepted research standards for research design and execution. While the Study's recommendations position DOD, over time, to take actions to improve the effectiveness and efficiency of the Military Health System, GAO found a number of shortcomings, including the following:
The Study did not fully mitigate limitations identified in its analysis of the required number of active-duty and civilian medical personnel. For example, the Study did not explain how known issues with the military services' workforce models affected the results of its requirements analysis. Without addressing such limitations, DOD will not have a full assessment of its medical workforce needs.
The Study did not sufficiently identify or mitigate limitations concerning its assessment of the requirements necessary to maintain the skills of active-duty medical providers. For example, although there were limitations concerning the accuracy of information on medical providers' workload, the Study did not identify or mitigate these limitations. Having accurate workload information is important to establishing a sound standard for maintaining the clinical skills of medical providers.
The Study established goals for transferring health care from DOD's purchased care network into its own network of hospitals and clinics and for increasing the productivity of active-duty medical providers, but did not develop a strategy explaining how these goals would be achieved. Without such a strategy it remains unclear whether DOD can achieve its goals to transfer health care from the purchased care network into its own network.
DOD's estimated cost savings did not fully utilize key practices for developing such estimates. DOD estimated net annual savings of $366 million from changes to 10 small hospitals and achievement of its goals for recapturing health care and increasing the productivity of active-duty health care providers. However, DOD did not include in its estimate an appropriate level of detail concerning the calculation of estimated savings, all potentially significant costs, or a description of the steps taken by the Study team to assess the reliability of cost data used to develop the estimate. For example, the Study recommended that a number of inpatient facilities be closed, but GAO's analysis found that the Study did not identify estimated costs associated with these changes. As a result, DOD's cost savings estimate did not present a full and accurate picture of possible costs and savings.

What GAO Recommends

GAO is making six recommendations, including that DOD conduct a new analysis of the required number of active-duty and civilian medical personnel that mitigates known limitations; identify and mitigate limitations regarding the standard for maintaining providers' clinical skills; develop a strategy for achieving its goals for transferring health care to DOD facilities and increasing the productivity of active-duty providers; and, when considering proposed changes to facilities, include in any accompanying cost estimates an appropriate level of detail. DOD concurred with each of GAO's recommendations.