Our objective was to determine whether Customs is effectively managing ACE, including whether Customs has adequately justified ACE cost-effectiveness. Customs Is Developing, but Has Not Yet Completed, Its Target Systems Architecture

Customs does not currently have a complete enterprise systems architecture. Because these characteristics were not based on a complete understanding of its enterprisewide functional and information needs, as specified in both best practice and Treasury guidance, we concluded that Customs did not have adequate assurance that its information systems will optimally support its needs across all business areas. Conclusions

Until Customs completes and enforces its enterprise systems architecture, it will not have adequate assurance that ACE and other systems it plans to build and operationally deploy (1) will effectively meet the agency’s business needs and (2) are compatible, efficient, and cost-effective to develop, integrate, and maintain. Customs has not effectively implemented any of these investment management practices on ACE. Specifically, (1) Customs’ investment analysis did not address alternatives to its chosen ACE system solution, (2) Customs’ did not use rigorous cost estimating techniques in preparing its ACE cost estimate and its cost-benefit analysis for ACE omitted substantial costs and inflated benefits, and (3) Customs is not justifying and validating the costs and benefits for each ACE increment. Customs is developing and acquiring ACE in a series of 21 increments. Furthermore, because it is not justifying the return on its investment in each ACE increment, Customs will not be able to demonstrate whether ACE is cost-effective until it has already spent hundreds of millions of dollars to develop/acquire the entire system. Our evaluations found that Customs lacks the capability to either develop or acquire ACE software effectively, and that it lacks a software process improvement program. As a result, Customs’ ACE software development capability is immature. To accomplish this, GAO recommends that the Commissioner of Customs, with the support of Customs’ CIO, ensure that Customs (1) rigorously analyze alternative approaches to building ACE, including ITDS as an alternative to developing ACE entirely within Customs and; (2) make investment decisions incrementally, i.e., for each increment: use disciplined processes to prepare a rigorous life-cycle cost estimate, including an explicit discussion of its inherent uncertainty; prepare realistic and supportable benefit expectations; require a favorable return-on-investment and compliance with Customs’ architecture before making any investment; and validate actual costs and benefits once an increment is piloted, compare these with estimates, use the results in making further decisions on subsequent increments, and report the results to Customs’ House and Senate appropriations and authorizing committees; and (3) strengthen ACE software acquisition management by: establishing an effective process improvement program and correcting the weaknesses in ACE software acquisition processes identified in this report, thereby bringing ACE processes to at least SEI level 2 and requiring at least SEI level 2 processes of all ACE software contractors.
Why GAO Did This Study

Pursuant to a congressional request, GAO reviewed the Customs Service's management of the Automated Commercial Environment (ACE), focusing on whether Customs has adequately justified ACE cost-effectiveness.

What GAO Found

GAO noted that: (1) Customs is not managing ACE effectively, and it does not have a firm basis for concluding that ACE is a cost-effective solution to modernizing its commercial environment; (2) GAO found serious weaknesses relating to architectural definition, investment management, and software development and acquisition that must be corrected before further investment in ACE is justified; (3) Customs is not building ACE within the context of a complete and enforced systems architecture; (4) in May 1998, GAO reported that Customs' architecture was incomplete because it was not based on a complete understanding of its enterprisewide functional and information needs; (5) GAO also reported that Customs had not yet instituted effective procedures for ensuring compliance with the architecture once it is completed; (6) until its architecture is completed and effectively enforced, Customs will not have adequate assurance that information systems like ACE will optimally support its needs across all business areas; (7) further, Customs lacks a reliable estimate of what ACE will cost to build, deploy, and maintain; and Customs has neither adequately justified, nor is it effectively monitoring, ACE's cost-effectiveness; (8) specifically, Customs did not use rigorous cost estimating techniques in preparing its cost estimate, and did not disclose the inherent imprecision of the estimate; (9) additionally, Customs omitted costs and inflated benefits in preparing its cost-benefit analysis; (10) moreover, Customs is not using effective incremental investment management practices; (11) while Customs plans to develop/acquire ACE in 21 increments, these increments are not individually cost-benefit justified, and Customs is not determining what benefits each increment, once operational, actually provides; (12) as a result, Customs will not know if ACE's expected return-on-investment is actually being realized until it has already spent hundreds of millions of dollars developing/acquiring the entire system; and (13) GAO found that Customs has neither the capability to effectively develop nor acquire ACE and that its processes for doing both, according to widely accepted and proven software capability maturity models, are ad hoc, immature, and ineffective.