Navy ERP is one such  system investment. FOC is now planned for fiscal year  2013, and the estimated life-cycle cost is about $2.4 billion (a 31 percent  increase over the original estimate). Prior GAO Reviews of DOD Business System Investments Have Identified IT Management Weaknesses

We have previously reported that DOD has not effectively managed key  aspects of a number of business system investments, including Navy  ERP. We made recommendations to address these areas, including  having the IV&V agent report directly to program oversight bodies, as well  as the program manager. In addition, each  event. Test Results Were Documented and Reported, but Key Information about Changes to the Status of Reported Defects Was Not Always Recorded

According to industry guidance, effective system testing includes  capturing, analyzing, resolving, and disclosing to decision makers the  status of problems found during testing (i.e., test defects). This addition is a positive  step because without an effective information system access audit tool,  the probability of test defect status errors or unauthorized changes is  increased. To its credit, the Navy ERP program has formally documented a change  control process. Further, it has  defined roles and responsibilities and a related decision-making structure  for reviewing and approving system changes. Specifically, our review of a random sample of 60 change requests and  minutes of related board meetings held between May 2006 and April 20 showed that the change requests were captured and tracked using an  automated tool, and they were reviewed and approved by the designated decision-making authorities and boards, in accordance with the program documented process. According to program officials, the cost and  schedule impacts of each change were discussed at control board  meetings. Because  performance of the system development and management role ma contractor potentially unable to render impartial assistance to the  government in performing the IV&V function, the contractor has an  inherent conflict of interest relative to meeting cost and schedule  commitments and disclosing the results of verification and validation  reviews that may affect its ability to do so. As we have previously  reported, the IV&V function should report the issues or weaknesses that  increase the risks associated with the project to program oversight  officials, as well as to program management, to better ensure that the  verification and validation results are objective and that the officials  responsible for making program investment decisions are fully informed . Beyond the IV&V function’s lack of independence, the program office h not ensured that the subcontractor has produced the range of deliv that were contractually required and defined in the IV&V plan. However, the IV&V contractor has largely not delivered these  products. The first planned assessment was delivered to the program in March 2009  he program’s configuration  and provides recommendations for improving t management process, such as using the automated tool to produce certain  reports and enhancing training to understand how the tool is use Further, program officials stated that they have also recently begun  requiring the contractor to provide formal quarterly reports, the first of  which was de of this quarterly report shows that it provides recommendations for  improving the program’s risk management process and organizational  change management strategy. Our review  Notwithstanding the recent steps that the program office has taken,  nevertheless lacks an independent perspective on the program’s products and management processes. The Navy ERP program office has largely implemented a range of effective controls associated with system testing and change control, including  acting quickly to address issues with the audit log for its test manage tool, but more can be done to ensure that the cost and sched proposed changes are explicitly documented and considered when  decisions are reached. Moreover, while the program office has contract for IV&V activities, it has not ensured that the contractor is indepen the products and processes that it is to review and has not held the  contractor accountable for producing the full range of IV&V deliverables  required under the contract. Recommendations for Executive Action

To strengthen the management of Navy ERP’s change control process,  recommend that the Secretary of Defense direct the Secretary of the Navy,  through the appropriate chain of command, to (1) revise the Navy ERP  procedures for controlling system changes to explicitly require that a  proposed change’s life-cycle cost impact be estimated and considered in  making change request decisions and (2) capture the cost and schedule  impacts of each proposed change in the Navy ERP automated change  c ontrol tracking tool. Appendix I: Objectives, Scope, and Methodology

Our objectives were to determine whether (1) system testing is being  effectively managed, (2) system changes are being effectively con and (3) independent verification and validation (IV&V) activities a effectively managed for the Navy Enterprise Resource Planning (ERP)  program.
Why GAO Did This Study

The Department of Defense (DOD) has long been challenged in effectively implementing key acquisition management controls on its thousands of business system investments. For this and other reasons, GAO has designated DOD's business systems modernization efforts as high-risk since 1995. One major business system investment is the Navy's Enterprise Resource Planning (ERP) system. Initiated in 2003, it is to standardize the Navy's business processes, such as acquisition and financial management. It is being delivered in increments, the first of which is to cost about $2.4 billion over its 20-year useful life and be fully deployed by fiscal year 2013. To date, the program has experienced about $570 million in cost overruns and a 2-year schedule delay. GAO was asked to determine whether (1) system testing is being effectively managed, (2) system changes are being effectively controlled, and (3) independent verification and validation (IV&V) activities are being effectively managed. To do this, GAO analyzed relevant program documentation, traced random samples of test defects and change requests, and interviewed cognizant officials.

What GAO Found

The Navy has largely implemented effective controls on Navy ERP associated with system testing and change control. For example, it has established a well-defined structure for managing tests, including providing for a logical sequence of test events, adequately planning key test events, and documenting and reporting test results. In addition, it has documented, and is largely following, its change request review and approval process, which reflects key aspects of relevant guidance, such as having defined roles and responsibilities and a hierarchy of control boards. However, important aspects of test management and change control have not been fully implemented. Specifically, the program's tool for auditing defect management did not always record key data about changes made to the status of identified defects. To its credit, the program office recently took steps to address this, thereby reducing the risk of defect status errors or unauthorized changes. Also, while the program office's change review and approval procedures include important steps, such as considering the impact of a change, and program officials told GAO that cost and schedule impacts of a change are discussed at control board meetings, GAO's analysis of 60 randomly selected change requests showed no evidence that cost and schedule impacts were in fact considered. Without such key information, decision-making authorities lack an adequate basis for making informed investment decisions, which could result in cost overruns and schedule delays. The Navy has not effectively managed its IV&V activities, which are designed to obtain an unbiased position on whether product and process standards are being met. In particular, the Navy has not ensured that the IV&V contractor is independent of the products and processes that it is reviewing. Specifically, the same contractor responsible for performing IV&V of Navy ERP products (e.g., system releases) is also responsible for ensuring that system releases are delivered within cost and schedule constraints. Because performance of this system development and management role makes the contractor potentially unable to render impartial assistance to the government in performing the IV&V function, there is an inherent conflict of interest. In addition, the IV&V agent reports directly and solely to the program manager and not to program oversight officials. As GAO has previously reported, the IV&V agent should report the findings and associated risks to program oversight officials, as well as program management, in order to better ensure that the IV&V results are objective and that the officials responsible for making program investment decisions are fully informed. Furthermore, the contractor has largely not produced the range of IV&V deliverables that were contractually required between 2006 and 2008. To its credit, the program office recently began requiring the contractor to provide assessment reports, as required under the contract, as well as formal quarterly reports; the contractor delivered the results of the first planned assessment in March 2009. Notwithstanding the recent steps that the program office has taken, it nevertheless lacks an independent perspective on the program's products and management processes