TES Met Some Oversight Requirements for T&E of Acquisition Programs Reviewed; Additional Steps Needed to Ensure That All Requirements Are Met

TES met its oversight requirements when approving test plans and test  reports in accordance with DHS acquisition and T&E directives for the 11  major acquisition programs we selected for review. However, TES did not  consistently document its review and approval of operational test agents  or its review of other required acquisition documentation, which could  provide more assurance that components were meeting T&E directives  when TES reviewed these documents. TES Oversight of Components’ Test Plans and Test Reports

TES is to oversee T&E of major DHS acquisition programs by ensuring  that the requirements set forth in the T&E directive are met and by  working with component program officials to develop T&E  documentation, such as test and evaluation master plans, as required by  DHS’s acquisition directive. Component officials from each of these six programs stated that TES  provided input to the development of the test and evaluation master plans. Additional Steps Needed to Ensure that T&E Requirements Are Met

While TES’s oversight of the test plans and reports for major acquisition  programs selected for review is in accordance with provisions in the T&E  directive, it did not consistently document its review and approval of  certain acquisition documentation or document the extent to which  certain requirements in the T&E directive were met. TES Did Not Consistently Document the Extent to which Criteria Used in Its Approval of Operational Test Agents Were Met

The T&E directive requires that an operational test agent—a government  agency or independent contractor carrying out independent operational  testing for major acquisition programs—is to meet certain requirements to  be qualified and approved by TES, but does not specify how TES’s  approval is to be documented. TES provided documentation, such as  memoranda, of its approval for 3 of these 8 programs. According to  TES officials, they did not have a mechanism in place requiring a  consistent method for documenting their review and approval of  component agents or the extent to which criteria used in reviewing these  agents were met. Without a mechanism for  documenting its review and approval of test agents for major acquisition  programs, it will be difficult for DHS or an independent third party to  validate TES’s decision-making process to ensure that it is effectively  overseeing component testing. TES and Component Officials Cited Challenges in Coordinating and Overseeing T&E across DHS; Efforts Are Underway to Address Some Challenges

TES and component officials reported challenges faced in coordinating  and overseeing T&E across DHS components that fell into four primary  categories: (1) ensuring that a program’s operational requirements—the  key requirements that must be met for a program to achieve its intended  goals—can be effectively tested; (2) working with DHS component  program staff that have limited T&E expertise and experience; (3) using  existing T&E directives and guidance to oversee complex information  technology acquisitions; and (4) ensuring that components allow sufficient  time and resources for T&E while remaining within program cost and  schedule estimates. Both TES and DHS, more broadly, have begun  initiatives to address some of these challenges, but it is too early to  determine their effectiveness. Other components in DHS told us they rely more on TES or contractors  for T&E expertise. Using Existing T&E Directives and Guidance to Oversee Information Technology Acquisitions

Effectively managing IT acquisitions is a governmentwide challenge. Since the efforts DHS is taking to address this challenge have only recently  been initiated, it is too early to tell what impact they will have on the  overall challenges of T&E for IT programs. Since the Homeland Security Act creating DHS was enacted in 2002, S&T  has had the responsibility for overseeing T&E activities across the  department. However, S&T did not have staff or the acquisition and T&E  directives in place to conduct such oversight across DHS components  until May 2009 when DHS issued its T&E directive. Recommendations for Executive Action

To better ensure that testing and evaluation requirements are met, we  recommend that the Secretary of Homeland Security direct the Under  Secretary for Science & Technology to take the following two actions:    Develop a mechanism to ensure that TES documents its approval of  operational test agents and the extent that the test agents meet the  requirements in the T&E directive, and criteria that TES use in  reviewing these test agents for major acquisition programs. For example, some programs, such as Customs and  Border Protection’s acquisition of Border Patrol Facilities would not  involve any T&E activities and therefore would not be subject to the  requirements in the T&E directive or DHS Science and Technology  Directorate’s Test and Evaluation and Standards office (TES) oversight.
Why GAO Did This Study

In recent years, GAO has reported on challenges the Department of Homeland Security (DHS) has faced in effectively managing major acquisitions, including programs which were deployed before appropriate testing and evaluation (T&E) was completed. In 2009 and 2010 respectively, DHS issued new T&E and acquisition directives to address these challenges. Under these directives, DHS Science and Technology Directorate's (S&T) Test & Evaluation and Standards Office (TES) is responsible for overseeing T&E of DHS major acquisition programs--that is, those with over $300 million in life-cycle costs--to ensure that T&E and certain acquisitions requirements are met. GAO was asked to identify (1) the extent to which TES oversees T&E of major acquisitions; and (2) what challenges, if any, TES officials report facing in overseeing T&E across DHS components. GAO reviewed DHS directives and test plans, interviewed DHS officials, and reviewed T&E documentation from a sample of 11 major acquisition programs from each of 11 different DHS components. The results of the sample cannot be generalized to all DHS programs, but provided insights.

What GAO Found

TES met some of its oversight requirements for T&E of acquisition programs GAO reviewed, but additional steps are needed to ensure that all requirements are met. Specifically, since DHS issued the T&E directive in May 2009, TES has reviewed or approved T&E documents and plans for programs undergoing testing, and conducted independent assessments for the programs that completed operational testing during this time period. TES officials told GAO that they also provided input and reviewed other T&E documentation, such as components' documents describing the programs' performance requirements, as required by the T&E directive. DHS senior level officials considered TES's T&E assessments and input in deciding whether programs were ready to proceed to the next acquisition phase. However, TES did not consistently document its review and approval of components' test agents--a government entity or independent contractor carrying out independent operational testing for a major acquisition--or document its review of other component acquisition documents, such as those establishing programs' operational requirements, as required by the T&E directive. For example, 8 of the 11 acquisition programs GAO reviewed had hired test agents, but documentation of TES approval of these agents existed for only 3 of these 8 programs. Approving test agents is important to ensure that they are independent of the program and that they meet requirements of the T&E directive. TES officials agreed that they did not have a mechanism in place requiring a consistent method for documenting their review or approval and the extent to which the review or approval criteria were met. Without mechanisms in place for documenting its review or approval of acquisition documents and T&E requirements, such as approving test agents, it is difficult for DHS or a third party to review and validate TES's decision-making process and ensure that it is overseeing components' T&E efforts in accordance with acquisition and T&E directives and internal control standards for government entities. TES and DHS component officials stated that they face challenges in overseeing T&E across DHS components which fell into 4 categories: (1) ensuring that a program's operational requirements--the key performance requirements that must be met for a program to achieve its intended goals-- can be effectively tested; (2) working with DHS component program staff who have limited T&E expertise and experience; (3) using existing T&E directives and guidance to oversee complex information technology acquisitions; and (4) ensuring that components allow sufficient time for T&E while remaining within program cost and schedule estimates. Both TES and DHS, more broadly, have begun initiatives to address some of these challenges, such as establishing a T&E council to disseminate best practices to component program managers, and developing specific guidance for testing and evaluating information technology acquisitions. In addition, S&T has reorganized to assist components in developing requirements that can be tested, among other things. However, since these efforts have only recently been initiated to address these DHS-wide challenges, it is too soon to determine their effectiveness.

What GAO Recommends

GAO recommends, among other things, that S&T develop mechanisms for TES to document its review or approval of component acquisition documentation and T&E requirements, such as approving operational test agents. DHS agreed with GAO's recommendations.