CBP’s Program Schedules and Life- Cycle Cost Estimates Reflect Some but Not All Best Practices

		CBP’s Program Schedules Have Experienced Delays and the Three Highest- Cost Programs Meet Some but Not All Scheduling Best Practices

			The Majority of the Plan’s Programs Experienced Some Delays, and Schedules for the IFT, RVSS, and MSC Programs Are Not Fully Reliable

OTIA has developed a schedule for each of the Plan’s seven programs,  and four programs will not meet their originally planned completion dates. Credible: OTIA’s schedules for the IFT and RVSS programs partially  met the characteristic of being credible; the MSC program schedule  minimally met this characteristic. Further, our analysis  showed that none of the schedules had valid baseline dates for  activities or milestones by which management could track current  performance. In addition, an Integrated Master Schedule that allows managers to  monitor all work activities, how long the activities will take, and how the  activities are related to one another is a critical management tool for  complex systems that involve the incorporation of a number of different  projects, such as the Plan. Because OTIA does not have an Integrated Master  Schedule for the Plan, it is not well positioned to understand how  schedule changes in each individual program could affect implementation  of the overall Plan. For example, we reported that, in terms of being  comprehensive, the estimate included documented technical data. For example, CBP officials had not conducted a sensitivity  analysis and a cost-risk and uncertainty analysis to determine a level of  confidence in the estimate, nor did CBP compare it with an independent  estimate. documented, and credible. However, this approach is not consistent with DHS’s acquisition  guidance, which states that even for commercial-off-the-shelf systems,  operational test and evaluation should occur in the environmental  conditions in which a system will be used before a full production decision  for the system is made and the system is subsequently deployed. For instance, for the 166,976  apprehension events reported by the Border Patrol across the Tucson  sector during fiscal year 2010 through June 2013, an asset assist was not  recorded for 115,517 (or about 69 percent) of these apprehension events. Border Patrol officials did not specify why the agency does not require the  recording and tracking of data on asset assists. CBP Has Identified Mission Benefits for Surveillance Technologies under the Plan but Has Not Yet Developed Performance Metrics

In response to our November 2011 recommendation regarding the  identification of mission benefits and development of key attributes for  performance metrics for the surveillance technologies to be deployed as  part of the Plan, CBP has identified mission benefits expected from the  implementation of the surveillance technologies to be acquired or  deployed as part of the Plan, but has not fully developed key attributes for   In November 2011, we  performance metrics for these technologies.reported that agency officials had not yet defined the mission benefits  expected or quantified metrics to assess the contribution of the selected  approaches in achieving their goal of situational awareness and detection  of border activity using surveillance technology. Specifically, revising the Test and Evaluation  Master Plan to more fully test the IFTs in the various environmental  conditions in which they will be used to determine operational  effectiveness and suitability, in accordance with DHS acquisition  guidance, could help provide CBP with more complete information on  how the IFTs will operate under a variety of conditions before beginning  full production. Conducting analysis of such data, once collected, in combination with  other relevant performance metrics or indicators as appropriate, could  help better position CBP to be able to determine the extent to which its  technology investments have contributed to border security efforts. Recommendations for Executive Action

To improve the acquisition management of the Plan and the reliability of  its cost estimates and schedules, assess the effectiveness of deployed  technologies, and better inform CBP’s deployment decisions, we  recommend that the Commissioner of CBP take the following six actions:
When updating the schedules for the IFT, RVSS, and MSC programs,  ensure that scheduling best practices, as outlined in our schedule  assessment guide, are applied to the three programs’ schedules. DHS did not concur with the other two recommendations in the  report. The programs under the Plan are  intended to provide Border Patrol with a combination of surveillance  capabilities to assist in achieving situational awareness along the Arizona  border with Mexico; and while the programs themselves may be  independent of one another, the Plan’s resources are being shared  among the programs. We continue to believe that developing and maintaining an  Integrated Master Schedule for the Plan could help provide CBP a  comprehensive view of the Plan and help CBP to reliably commit to when  the Plan will be fully implemented and better predict whether estimated  completion dates are realistic to manage programs’ performance, as  noted in the report. However, as noted in this  report, independently verifying the cost estimates is consistent with best  practices and could help provide CBP with more insights into program  costs. With regard to the fifth recommendation, that CBP require data on asset  assists to be recorded and tracked within the Enforcement Integrated  Database, DHS concurred and stated that Border Patrol is changing its  data collection process to allow for improved reporting on asset assists  for apprehensions and seizures and intends to make it mandatory to  record whether an asset assisted in an apprehension or seizure. Appendix I: Objectives, Scope, and Methodology

Our objectives were to determine the extent to which U.S. Customs and  Border Protection (CBP) has (1) developed schedules and Life-cycle Cost  Estimates for the Arizona Border Surveillance Technologies Plan (the  Plan) in accordance with best practices; (2) followed key aspects of the  Department of Homeland Security’s (DHS) acquisition management  framework in managing the Plan’s three highest cost programs; and (3)  assessed the performance of technologies deployed under the Secure  Border Initiative Network (SBInet), identified mission benefits, and  developed performance metrics for surveillance technologies to be  deployed under the Plan. To determine the extent to which CBP has followed best practices in  developing schedules and Life-cycle Cost Estimates for the Plan’s three  highest-cost programs—the Integrated Fixed Tower (IFT), Remote Video  Surveillance System (RVSS), and Mobile Surveillance Capability  (MSC)—we obtained CBP’s Office of Technology Innovation and  Acquisition’s (OTIA) program schedules as of March 2013, which were  current at the time of our review, for the these programs and compared  them against best practices for developing schedules. We characterized whether the schedules met each of the 10 best  practices based on the following scale:
Not met—the program provided no evidence that satisfies any of the  criterion. Minimally met—the program provided evidence that satisfies a small  portion of the criterion. Partially met—the program provided evidence that satisfies about half  of the criterion. Met—the program provided complete evidence that satisfies the entire  criterion. These practices are  summarized into four characteristics of a reliable schedule— comprehensive, well constructed, credible, and controlled. Minimally met—the program provided evidence that satisfies a small  portion of the criterion. Partially met—the program provided evidence that satisfies about half  of the criterion. Substantially met—the program provided evidence that satisfies a  large portion of the criterion.
What GAO Found

The Department of Homeland Security's (DHS) U.S. Customs and Border Protection's (CBP) schedules and Life-cycle Cost Estimates for the Arizona Border Surveillance Technology Plan (the Plan) reflect some, but not all, best practices. Scheduling best practices are summarized into four characteristics of reliable schedules—comprehensive, well constructed, credible, and controlled (i.e., schedules are periodically updated and progress is monitored). GAO assessed CBP's schedules as of March 2013 for the three highest-cost programs that represent 97 percent of the Plan's estimated cost. GAO found that schedules for two of the programs at least partially met each characteristic (i.e., satisfied about half of the criterion), and the schedule for the other program at least minimally met each characteristic (i.e., satisfied a small portion of the criterion), as shown in the table below. For example, the schedule for one of the Plan's programs partially met the characteristic of being credible in that CBP had performed a schedule risk analysis for the program, but the risk analysis was not based on any connection between risks and specific activities. For another program, the schedule minimally met the characteristic of being controlled in that it did not have valid baseline dates for activities or milestones by which CBP could track progress.
Source: GAO analysis of CBP data.
Note: Not met—CBP provided no evidence that satisfies any of the criterion. Minimally met—CBP provided evidence that satisfies a small portion of the criterion. Partially met—CBP provided evidence that satisfies about half of the criterion. Substantially met—CBP provided evidence that satisfies a large portion of the criterion. Met—CBP provided complete evidence that satisfies the entire criterion.
Further, CBP has not developed an Integrated Master Schedule for the Plan in accordance with best practices. Rather, CBP has used the separate schedules for each program to manage implementation of the Plan, as CBP officials stated that the Plan contains individual acquisition programs rather than integrated programs. However, collectively these programs are intended to provide CBP with a combination of surveillance capabilities to be used along the Arizona border with Mexico, and resources are shared among the programs. According to scheduling best practices, an Integrated Master Schedule is a critical management tool for complex systems that involve a number of different projects, such as the Plan, to allow managers to monitor all work activities, how long activities will take, and how the activities are related to one another. Developing and maintaining an Integrated Master Schedule for the Plan could help provide CBP a comprehensive view of the Plan and help CBP better understand how schedule changes in each individual program could affect implementation of the overall Plan.
Moreover, cost-estimating best practices are summarized into four characteristics—well documented, comprehensive, accurate, and credible. GAO's analysis of CBP's estimate for the Plan and estimates completed at the time of GAO's review for the two highest-cost programs showed that these estimates at least partially met three of these characteristics: well documented, comprehensive, and accurate. In terms of being credible, these estimates had not been verified with independent cost estimates in accordance with best practices. Ensuring that scheduling best practices are applied to the three programs' schedules and verifying Life-cycle Cost Estimates with independent estimates could help better ensure the reliability of the schedules and estimates.
CBP did not fully follow key aspects of DHS's acquisition management guidance for the Plan's three highest-cost programs. For example, CBP plans to conduct limited testing of the highest-cost program—the Integrated Fixed Tower (IFT: towers with cameras and radars)—to determine its mission contributions, but not its effectiveness and suitability for the various environmental conditions, such as weather, in which it will be deployed. This testing, as outlined in CBP's test plan, is not consistent with DHS's guidance, which states that testing should occur to determine effectiveness and suitability in the environmental conditions in which a system will be used. Revising the test plan to more fully test the program in the conditions in which it will be used could help provide CBP with more complete information on how the towers will operate once they are fully deployed.
CBP has identified mission benefits for technologies under the Plan, but has not yet developed performance metrics. CBP has identified such mission benefits as improved situational awareness and agent safety. Further, a DHS database enables CBP to collect data on asset assists, defined as instances in which a technology, such as a camera, or other asset, such as a canine team, contributed to an apprehension or seizure, that in combination with other relevant performance metrics or indicators, could be used to better determine the contributions of CBP's surveillance technologies and inform resource allocation decisions. However, CBP is not capturing complete data on asset assists, as Border Patrol agents are not required to record and track such data. For example, from fiscal year 2010 through June 2013, Border Patrol did not record whether an asset assist contributed to an apprehension event for 69 percent of such events in the Tucson sector. Requiring the reporting and tracking of asset assist data could help CBP determine the extent to which its surveillance technologies are contributing to CBP's border security efforts.
This is a public version of a For Official Use Only—Law Enforcement Sensitive report that GAO issued in February 2014. Information DHS deemed as For Official Use Only—Law Enforcement Sensitive has been redacted.

What GAO Recommends

GAO recommends that CBP, among other things, apply scheduling best practices, develop an integrated schedule, verify Life-cycle Cost Estimates, revise the IFT test plan, and require tracking of asset assist data. DHS concurred with four of six GAO recommendations. It did not concur with the need for an integrated schedule or a revised IFT test plan. As discussed in this report, GAO continues to believe in the need for a schedule and a revised test plan.