Validity of DTS Economic Analysis Questionable

Our analysis of the September 2003 DTS economic analysis found that two  key assumptions used to estimate cost savings were not based on reliable  information. Two primary areas represented the majority of the over  $56 million of estimated annual net savings DTS was expected to realize— personnel savings and reduced CTO fees. Air Force and Navy DTS  program officials stated that they did not anticipate a reduction in the  number of personnel with the full implementation of DTS, but rather the  shifting of staff to other functions. Also, as part of the  Navy’s overall evaluation of the economic analysis, program officials  stated that “the Navy has not identified, and conceivably will not  recommend, any personnel billets for reduction.” Finally, the Naval Cost  Analysis Division (NCAD) October 2003 report on the economic analysis  noted that it could not validate approximately 40 percent of the Navy’s  total costs, including personnel costs, in the DTS life-cycle cost estimates  because credible supporting documentation was lacking. Savings Associated with Reduction of CTO Fees Are Unknown

According to the September 2003 economic analysis, DOD expected to  realize annual net savings of $31 million through reduced fees paid to the  CTOs because the successful implementation of DTS would enable the  majority of airline tickets to be acquired with either no or minimal  intervention by the CTOs. Rather, the  sole support provided by the PMO-DTS was an article in a travel industry  trade publication. The article was not based on information related to  DTS, but rather on the experience of one private sector company. The September 2003 DTS economic analysis did not undertake an  assessment of the effects of the uncertainty inherent in the estimates of  benefits and costs, as required by DOD and OMB guidance. Because an  economic analysis uses estimates and assumptions, it is critical that a  sensitivity analysis be performed to understand the effects of the  imprecision in both underlying data and modeling assumptions. Presently, the reported DTS utilization is based on a DTS Voucher Analysis  Model that was developed in calendar year 2003 using estimated data, but  over the years has not been completely updated with actual data. This lack of accurate and pertinent  utilization data hinders management’s ability to monitor its progress  toward the DOD vision of DTS as the standard travel system, as well as to  provide consistent and accurate data to Congress. Besides the memorandums, DOD is taking other actions to increase DTS  utilization as the following examples illustrate. The following examples highlight the  concerns raised by the military service officials. Air Force DTS program officials stated that DTS releases did  not appear to be well tested prior to implementation. While these actions are a positive step forward, they do not address the  fundamental problem that DTS’s requirements are still ambiguous and  conflicting—a primary cause of the previous problems. Providing Complete Flight Information Has Been a Continuing Problem

In our earlier testimony and report, we noted that DOD did not have  reasonable assurance that the flights displayed met the stated DOD  requirements. For example,  DOD has retained a requirement to display 25 flights for each inquiry. Until DOD  improves DTS requirement management practices, it will not have this  assurance. Furthermore, the shift of  DTS to BTA, which makes DTS an enterprisewide endeavor, should help in  making DTS the standard integrated, end-to-end travel system for business  travel. Equally important, however, will be the department’s ability to resolve the  long-standing difficulties that DTS has encountered with its requirements  management and system testing. DOD’s response indicated that the Defense Travel  Management Office is currently procuring commercial travel services for  DOD worldwide in a manner that will ensure evaluation of cost  effectiveness for all services. Appendix I: Scope and Methodology

To assess the reasonableness of the key assumptions made by DOD to  arrive at the net annual estimated savings of over $56 million shown in the  September 2003 economic analysis addendum, we (1) ascertained if the  economic analysis was prepared in accordance with the prescribed  standards, (2) analyzed two key assumptions that represent the largest  dollar savings for the DTS program, and (3) analyzed the supporting  documentation related to these two assumptions to determine whether the  assumptions were valid.
Why GAO Did This Study

In 1995, the Department of Defense (DOD) began an effort to implement a standard departmentwide travel system. The Defense Travel System (DTS) is envisioned as DOD's standard end-to-end travel system. This report is a follow-up to GAO's January 2006, report which highlighted DTS implementation problems. Because of continued congressional interest in DTS, GAO initiated this follow-up audit under the Comptroller General's statutory authority. GAO determined whether (1) two key assumptions made in the September 2003 economic analysis were reasonable, (2) DOD is taking action to ensure full utilization of DTS and gathering the data needed to monitor DTS utilization, and (3) DOD has resolved the previously identified problems with DTS flight information. To address the above objectives, GAO (1) reviewed the September 2003 DTS economic analysis, (2) analyzed DTS utilization data, and (3) analyzed DTS flight information.

What GAO Found

GAO's analysis of the September 2003 DTS economic analysis found that the two key assumptions used to estimate annual net savings were not based on reliable information. Two cost components represent the majority of the over $56 million in estimated net savings--personnel savings and reduced commercial travel office (CTO) fees. In regard to the personnel savings, GAO's analysis found that the $24.2 million of personnel savings related to the Air Force and the Navy was not supported. Air Force and Navy DTS program officials stated that they did not anticipate a reduction in the number of personnel, but rather the shifting of staff from the travel function to other functions. The Naval Cost Analysis Division stated that the Navy will not realize any tangible personnel cost savings from the implementation of DTS. In regard to the CTO fees, the economic analysis assumed that 70 percent of all DTS airline tickets would either require no intervention or minimal intervention from the CTOs, resulting in an estimated annual net savings of $31 million. However, the sole support provided by the DTS program office was an article in a trade industry publication. The article was not based on information related to DTS, but rather on the experience of one private sector company. Furthermore, the economic analysis was not prepared in accordance with guidance prescribed by OMB and DOD. DOD guidance stated that the life-cycle cost estimates should be verified by an independent party, but this did not occur. The economic analysis did not undertake an assessment of the effects of the uncertainty inherent in the estimates of benefits and costs. Because an economic analysis uses estimates and assumptions, it is critical that the imprecision in both the underlying data and assumptions be understood. Such an assessment is referred to as a sensitivity analysis. DOD acknowledged that DTS is not being used to the fullest extent possible, but lacks comprehensive data to effectively monitor its utilization. DOD's utilization data are based on a model that was developed in calendar year 2003. However, the model has not been completely updated to reflect actual DTS usage. The lack of accurate utilization data hinders management's ability to monitor progress toward the DOD vision of DTS as the standard travel system. GAO also found that the military services have initiated actions that are aimed at increasing the utilization of DTS. Finally, GAO found that DTS still has not addressed the underlying problems associated with weak requirement management and system testing. While DOD has acted to address concerns GAO previously raised, GAO found that DTS's requirements are still ambiguous and conflicting. For example, DTS displaying up to 25 flights for each inquiry is questionable because it is unclear whether this is a valid requirement. Until DOD improves DTS's requirement management practices, the department will not have reasonable assurance that DTS can provide the intended functionality.