CBP Uses ATS and Other Tools to Target Maritime Cargo Shipments for National Security Purposes

ATS is the primary system that CBP targeters use to review maritime  cargo shipments for national security purposes, and targeters we spoke  with were generally satisfied with how ATS and its weight set of national  security rules have assisted in their targeting efforts. The risk score,  however, is not the sole factor that determines whether a targeter reviews  the data for a shipment or whether the shipment is selected for a security  examination. In particular, targeters at each of the six ATUs we visited  explained that they use the ATS risk score as a starting point for the  targeting process, but that their decisions are ultimately based on  additional research. On the basis of  the ATS risk score and the research conducted, targeters make a  qualitative assessment of the risk and determine whether to hold a  shipment for examination. Targeters at ATUs are required to review  data in ATS for all medium-risk and high-risk shipments that arrive at their  respective ports. Targeters’ reviews: ATS also indicates whether the shipment data  have been reviewed by a targeter at the targeter’s own location or at  another CBP targeting location, such as a Container Security Initiative  port or NTC-C.
CBP targeters also use tools outside of ATS to conduct research. A targeter could also  determine that an examination is not necessary for a medium-risk  shipment—for example, the weight set may assign a medium-risk score  to a shipment based on the data available, but the targeter could  determine through research that the score is based on a clerical error in  the data provided. CBP’s Efforts to Assess the Weight Set Have Been Limited

		CBP Developed Performance Measures to Assess the Weight Set but Is Continuing to Update the Methodology

We have previously reported that ensuring controls to assess ATS’s  effectiveness in identifying high-risk shipments was important for  providing CBP with the best information to inform its targeting efforts. Currently, CBP assesses the performance of the weight set using the  following performance measures:
True positive rate (TPR) which reflects the percentage of maritime  shipments that ATS assessed as high risk within the population of  shipments in which CBP identified a threat during an examination. False positive rate (FPR) which reflects the percentage of maritime  shipments that ATS assessed as high risk within the population of  shipments in which CBP did not identify a threat during an  examination. The TPR enables CBP to determine the accuracy of the weight  set in identifying high-risk shipments. CBP Does Not Have Reasonable Assurance That the Updated Weight Set Is More Effective than Alternative Versions or the Version It Replaced

Prior to implementing the current version of the weight set in early 2011,  CBP did not conduct an assessment to determine whether the updated  version of the weight set would be more effective than the previous  version of the weight set or other alternatives that were considered during  the update process. Assessing the potential effectiveness of alternative versions of the weight  set prior to selecting one for implementation would provide CBP with  more information to make an informed decision. CBP Has Not Regularly Assessed the Weight Set against Performance Targets to Determine when Updates Are Needed

Since implementing the current version of the weight set in early 2011,  CBP has not regularly assessed the weight set against established  performance targets to monitor its performance and obtain information to  determine when updates to the weight set are necessary. Nevertheless,  given the importance of the weight set to CBP’s process for targeting  cargo containers, regular performance assessments of the weight set that  include evaluating results against established performance targets could  help CBP determine when updates are needed in a timelier manner and  help it better prioritize the resources it needs to complete the updates.Furthermore, CBP officials stated that they intend to continue adjusting  the methodology for calculating the performance measures to mitigate  data limitations and more accurately reflect the performance of the weight  set. Such steps could  help CBP determine when changes may be needed and ensure that its  targeters have the best information available regarding the risk of  maritime cargo container shipments arriving in the United States. Recommendations for Executive Action

To enhance its targeting of maritime cargo containers and better position  CBP to provide reasonable assurance of the effectiveness of ATS, we  recommend that the Commissioner of CBP take the following two actions: ensure that future updates to the weight set are based on results of  assessments that demonstrate that the chosen version of the weight  set is more effective than other alternatives, including the existing  version, and establish targets for CBP’s performance measures and use those  measures to assess the effectiveness of the weight set on a regular  basis to better determine when updates to the weight set are needed. DHS concurred with the two  recommendations. Appendix I: Federal Strategy for Ensuring the Security of Maritime Cargo Container Shipments

This appendix describes the core programs related to U.S. Customs and  Border Protection’s (CBP) strategy for ensuring the security of maritime  cargo container shipments. In this  report, the DHS OIG stated that CBP could improve its process for  changing or deleting targeting rules by, among other things, documenting  (1) rule change decisions and (2) the testing and evaluation of rule  changes.
Why GAO Did This Study

The U.S. economy is dependent on the expeditious flow of millions of tons of cargo each day. Cargo containers are an important instrument of global trade but also can present security concerns. CBP is responsible for administering container security programs, and its strategy for securing maritime cargo containers includes analyzing information to identify shipments that may contain terrorist weapons or other contraband. Because CBP has insufficient resources to examine every container, targeters use ATS to target which container shipments should be examined. GAO was asked to assess CBP's targeting efforts. This report addresses (1) how ATS supports CBP's targeting of maritime cargo container shipments for national security purposes and (2) the extent to which CBP assesses the effectiveness of ATS's national security targeting rules. GAO analyzed fiscal year 2011 CBP data on shipments and containers arriving at U.S. ports and containers scanned at these ports. GAO also visited six CBP units selected on the basis of the percentage of maritime shipments that were scored as high risk or medium risk for national security purposes at these locations in fiscal year 2011, among other factors. GAO also analyzed documents, such as CBP's ATS performance measures.

What GAO Found

U.S. Customs and Border Protection (CBP), within the Department of Homeland Security (DHS), employs a risk-based approach that uses the Automated Targeting System (ATS) and other tools to identify (target) maritime cargo shipments for further examination. ATS is a web-based enforcement and decision support system that includes a set of rules to assess the risk level for each arriving cargo shipment. This set of rules is referred to as the maritime national security weight set (weight set) because each rule in the set has a specific weighted value assigned to it. CBP classifies the risk scores from the weight set as low, medium, or high risk. CBP policy states that a shipment's risk score is to determine, in part, actions taken by CBP officers (targeters) at the ports. Specifically, targeters are generally required to review shipment data for all medium-risk and high-risk shipments and hold high-risk shipments for examination. The risk score, however, is not the sole factor that determines whether a targeter reviews the data for a shipment or whether CBP examines a shipment. In particular, targeters at each of the six ports GAO visited explained that they use the ATS risk score as a starting point for the targeting process but that their decisions regarding which shipments to examine are ultimately based on additional research. Targeters at the six ports GAO visited said they also use tools outside of ATS, such as web searches, to research shipments.
CBP efforts to assess the weight set's effectiveness in identifying the risk of shipments have been limited. CBP has performance measures--represented by the percentage of shipments targeted as high risk that contain a threat and the percentage of shipments targeted as high risk that do not contain a threat--that enable CBP to determine the accuracy of the weight set, given a particular workload or examination rate. However, CBP did not assess the weight set to verify its effectiveness when implementing an updated version in early 2011. Prior to implementing the updated version of the weight set, CBP assessed the potential impact of the update on CBP's workload but did not conduct an assessment to determine whether the updated version of the weight set would be more effective in identifying high-risk shipments than the previous version or other alternatives. Assessing the potential effectiveness of alternative versions of the weight set prior to selecting one for implementation could help CBP make more informed decisions about future updates. Doing so could also provide CBP reasonable assurance that the version it selects is the most effective of the alternatives and is more effective than the previous version it replaces. Furthermore, since implementing the updated version of the weight set in early 2011, CBP has not regularly assessed the weight set to monitor its performance and to help determine when changes are needed. For example, CBP conducted the first assessment of the current version of the weight set, using the performance measures, in the summer of 2012--18 months after the weight set's implementation in early 2011. Regular assessments of the weight set's effectiveness could help CBP determine when updates are needed in a timelier manner and ensure that targeters have the best information available to make targeting decisions. Moreover, CBP has not established targets for the performance measures so that it is not clear whether a particular change in the weight set's performance is significant enough to suggest that changes are needed to improve the effectiveness of the weight set.

What GAO Recommends

GAO recommends that CBP (1) ensure that future updates to the weight set are based on assessments of its performance and (2) establish targets for performance measures and use those measures to regularly assess effectiveness of the weight set. DHS concurred with these recommendations.