According to recognized industry  standards, IV&V can provide management with an objective  assessment of a program’s processes, products, and risks throughout its  life cycle and help ensure conformance to program performance,  schedule, and budget targets. Specifically, the  department does not establish risk-based decision making criteria in  its Acquisition Guidebook for determining whether, or the extent to  which, IT programs should use it and does not require that programs  conduct assessments against such criteria. DHS acquisition policy does not address  the independence of agents. DHS policy does not address IV&V  management or the need to effectively oversee the department’s  investment in this practice. Thus, department officials were unaware whether or the extent to  which IV&V was being used by the largest IT acquisition programs. Furthermore, in the absence of a clearly articulated  risk-based decision framework for undertaking IV&V, applying its results,  and evaluating its effectiveness, DHS’s investments in IV&V efforts are  unlikely to provide optimal value for the department and, in some cases,  may even fail to deliver any significant benefits. DHS Reports Widespread Use of IV&V, but Implementation of Key Elements Is Limited

Many large IT acquisitions from across DHS report using IV&V as part of  their acquisition and/or development process. For example, none of the eight used a structured, risk-based decision  making process when deciding if, when, and how to use IV&V. In part,  these weaknesses can be attributed to the lack of clear departmentwide  policy requiring the application of such elements. However, only one fully defined them. Three programs  did not specify the resources required for their efforts. To realize IV&V’s promise as a tool for reducing the risks  inherent in developing IT systems, DHS needs to promote a common  understanding of effective IV&V across the department, and through its  oversight activities, ensure that component agencies and their large IT  programs conduct efforts that consistently contribute toward meeting IT  acquisition cost, schedule, and mission goals. Recommendations for Executive Action

To help guide consistent and effective execution of IV&V at DHS, we  recommend that the Secretary of Homeland Security direct the  department CIO and CPO to take the following three actions:    Revise DHS acquisition policy such that it establishes  risk-based criteria for (1) determining which major and other high- risk IT acquisition programs should conduct IV&V and (2)  selecting appropriate activities for independent review of these  programs;  requirements for technical, financial, and managerial  independence of agents;  standards and guidance for defining and documenting plans and  products;  controls for planning, managing, and overseeing efforts;    mechanisms to ensure that plans and significant findings inform  DHS acquisition program reviews and decisions, including those  of the ARB; and    mechanisms to monitor and ensure implementation of this policy  on applicable new IT acquisition programs.   Collect and analyze data on IV&V efforts for major IT acquisition  programs to facilitate the development of lessons learned and  evaluation of the effectiveness of DHS’s investments, and establish a  process that uses the results to inform the department’s IT investment  decisions. Appendix I: Objectives, Scope, and Methodology

Our objectives were to determine (1) how the Department of Homeland  Security’s (DHS) independent verification and validation (IV&V) policies  and procedures for information technology (IT) acquisitions compare with  leading practices and (2) the extent to which DHS has implemented IV&V  on its large IT system acquisitions.   Not met.
Why GAO Did This Study

Since its creation in 2003, the Department of Homeland Security (DHS) has been developing new information technology (IT) systems to perform both mission-critical and support functions; however, it has faced challenges in developing these systems. One way to manage the inherent risks of developing and acquiring systems is through independent verification and validation (IV&V)--a process conducted by a party independent of the development effort that provides an objective assessment of a project's processes, products, and risks throughout its life cycle and helps ensure that program performance, schedule, and budget targets are met. GAO was asked to determine (1) how DHS's IV&V policies and procedures for IT acquisitions compare with leading practices and (2) the extent to which DHS has implemented IV&V on its large IT system acquisitions. To do so, GAO assessed DHS's policy against industry standards and leading practice guides, as well as analyzed how eight selected IT programs had implemented IV&V.

What GAO Found

DHS recognizes the importance of IV&V and recommends its use on major IT programs. Nevertheless, its acquisition policy does not address the elements of leading practices for IV&V. Specifically, the department has not established risk-based decision making criteria for determining whether, or the extent to which, programs should utilize IV&V. In addition, department policy does not define the degree of independence required of agents and does not require that programs determine and document the planned scope of their efforts, including the program activities subject to review; the resources required; roles and responsibilities; and how the results will be reported and acted upon. Moreover, the policy does not address overseeing DHS's investment in IV&V. Thus, officials were unaware of the extent to which it was being used on major IT acquisition programs, associated expenditures, or if those expenditures are producing satisfactory results. Absent such policy elements and more effective oversight, the department's investments in IV&V efforts are unlikely to provide optimal value for the department and, in some cases, may even fail to deliver any significant benefits. Many large IT acquisition programs across DHS reported using IV&V as part of their acquisition and/or development processes. Nevertheless, the eight major IT acquisition programs that GAO analyzed did not consistently implement the elements of leading practice. For example, the eight did not fully apply a structured, risk-based decision making process when deciding if, when, and how to utilize IV&V. In part, these weaknesses are due to the lack of clear departmentwide guidance regarding the use of such practices. As a result, the department's IV&V efforts may not consistently contribute toward meeting IT acquisition cost, schedule, and mission goals.

What GAO Recommends

GAO recommends that DHS (1) update its acquisition policy to reflect elements of effective IV&V, (2) monitor and ensure implementation of this policy on applicable new and ongoing IT programs, and (3) collect data on IV&V usage and use it to evaluate the effectiveness of these investments. DHS concurred with GAO's recommendations and described actions planned or under way to address them.