In addition, the board directed the SBInet System Program  Office (SPO) to deliver a range of program documentation, including  an updated Test and Evaluation Master Plan (TEMP), detailed test  plans, and a detailed schedule for deploying Block 1 to two initial  sites in the Tucson Sector of the southwest border. DHS has not effectively managed key aspects of SBInet testing,  which has in turn increased the risk that the system will not perform  as expected and will take longer and cost more than necessary. While the department’s testing approach appropriately consists of a  series of progressively expansive test events, some of which have  yet to be completed, test plans and test cases for recently executed  test events were not defined in accordance with relevant guidance. For example, none of the plans for tests of system components  addressed testing risks and mitigation strategies. Further, SBInet test procedures were generally not executed as  written. While some of these changes were  relatively minor, others were significant, such as adding  requirements or completely rewriting verification steps. This concern is  underscored by a program office letter to the prime contractor  stating that changes made to system qualification test procedures  appeared to be designed to pass the test instead of being designed to  qualify the system. These limitations are due, among other things, to a lack of detailed  guidance in the TEMP, the program’s aggressive milestones,  schedule, and ambiguities in requirements. SBInet Testing Results Have Identified a Growing Number of System Performance and Quality Problems

The number of new SBInet defects that have been discovered during  testing has increased faster than the number that has been fixed. DHS Science and Technology Directorate Testing Process Is Being Used to Leverage Maturing Technologies for SBInet

The SPO does not have its own process for testing the relevance to  SBInet of technologies that are maturing or otherwise available from  industry or other government entities. GAO Is Making Recommendations to Improve SBInet Test Management and Problem Resolution

To improve the planning and execution of future test events and the  resolution and disclosure of system problems, we are making the  following four recommendations to DHS:  ●  Revise the SBInet Test and Evaluation Master Plan to include  explicit criteria for assessing the quality of test documentation and  for analyzing, prioritizing, and resolving defects. In written comments on a draft of our report, DHS stated that the  report was factually sound, and it agreed with our last three  recommendations and agreed with all but one aspect of the first  one. Given  that major test events remain to be planned and conducted, which in  turn are likely to identify additional system problems, it is important  to correct these testing and problem resolution weaknesses. Our objectives are to determine the extent to which  DHS has (1) defined the scope of its proposed system solution, (2)  developed a reliable schedule for delivering this solution, (3)  demonstrated the cost effectiveness of this solution, (4) acquired  this solution in accordance with key life cycle management  processes, and (5) addressed our recent recommendations. Security Border Initiative Financial Management Controls  Over Contractor Oversight.
Why GAO Did This Study

This testimony is based on our report "Secure Border Initiative: DHS Needs to Address Testing and Performance Limitations That Place Key Technology Program at Risk." In September 2008, we reported to Congress that important aspects of SBInet were ambiguous and in a continuous state of flux, making it unclear and uncertain what technology capabilities were to be delivered when. In addition, the program did not have an approved integrated master schedule to guide the program's execution, and key milestones continued to slip. This schedule-related risk was exacerbated by the continuous change in and the absence of a clear definition of the approach used to define, develop, acquire, test, and deploy SBInet. Furthermore, different levels of SBInet requirements were not properly aligned, and all requirements had not been properly defined and validated. Also, the program office had not tested the individual system components to be deployed to initial locations, even though the contractor had initiated integration testing of these components with other system components and subsystems, and its test management strategy did not contain, among other things, a clear definition of testing roles and responsibilities; or sufficient detail to effectively guide planning for specific test events, such as milestones and metrics. Accordingly, we made recommendations to address these weaknesses which DHS largely agreed to implement. In light of SBInet's important mission, high cost, and risks, you asked us to conduct a series of four SBInet reviews. This statement and report being released today provide the results for the first of these reviews. Specifically, they address (1) the extent to which SBInet testing has been effectively managed, including identifying the types of tests performed and whether they were well planned and executed; (2) what the results of testing show; and (3) what processes are being used to test and incorporate maturing technologies into SBInet.

What GAO Found

SBInet testing has not been adequately managed, as illustrated by poorly defined test plans and numerous and extensive last-minute changes to test procedures. Further, testing that has been performed identified a growing number of system performance and quality problems--a trend that is not indicative of a maturing system that is ready for deployment anytime soon. Further, while some of these problems have been significant, the collective magnitude of the problems is not clear because they have not been prioritized, user reactions to the system continue to raise concerns, and key test events remain to be conducted. Collectively, these limitations increase the risk that the system will ultimately not perform as expected and will take longer and cost more than necessary to implement. For DHS to increase its chances of delivering a version of SBInet for operational use, we are recommending that DHS improve the planning and execution of future test events and the resolution and disclosure of system problems. DHS agreed with our recommendations.