The NDAA requires DHS to develop and implement 43 border security metrics in four domains—between POEs, at POEs, the maritime border, and air and marine security in the land environment. Within DHS, CBP and the Coast Guard have primary responsibility for border security within these four domains. CBP and its subcomponents are to secure U.S. borders at and between POEs by preventing inadmissible people and illicit goods from entering the United States, among other responsibilities. Within CBP, the primary offices and components involved in border security are the Office of Field Operations at POEs, Border Patrol between POEs, and Air and Marine Operations for air and marine security in the land and maritime domains. The Coast Guard and CBP’s Air and Marine Operations share responsibility for security of the nation’s maritime borders. Table 1 shows examples of border security metrics by domain and responsible DHS component. According to DHS officials, within DHS, two subcomponents within the Office of Strategy, Policy, and Plans were responsible for coordinating the department’s effort to develop the fiscal year 2017 Border Security Metrics Report. A senior DHS official explained that the report was initially tasked to the Unity of Effort Integration Office, which was part of the Unity of Effort initiative started in 2014 to better understand border security efforts along the southwest border including exploring the development of border security metrics. OIS assumed responsibility for the report in 2017. According to OIS officials, to prepare the report, they obtained data and information related to each NDAA metric from the administrative records of the DHS components with primary responsibilities for border security in the four domains. For example, OIS requested data and information on “turn backs” and “got aways” from Border Patrol—the lead component for the between POE domain—which records sector estimates of turn backs and got aways based on direct and indirect observations. Of the 43 metrics the NDAA listed for inclusion in the Border Security Metrics Report, the majority were counts and rates of border security activities. The remaining metrics were estimates or were not specifically described. For example, the number of apprehensions in each Border Patrol sector is a count metric. In contrast, a rate metric compares one value or number against another. For example, the wait time ratio compares the average wait times to total commercial and private vehicular traffic volumes being processed at a land POE. An estimate is used for metrics of flows or activities that are largely undetected and therefore cannot be measured directly and must be estimated, such as the number of undetected unlawful entries. A few metrics are a combination of counts or rates with an estimate. For example, the metric for total inadmissible travelers at POEs counts known inadmissible travelers that are intercepted at POEs, and also requires an estimate of how many inadmissible travelers may have successfully entered at a POE without being detected, which cannot be directly measured. The NDAA did not specifically describe some metrics. For example, while the NDAA asked for an examination of each of the eight consequences under the Consequence Delivery System, it did not specify how this examination was to be carried out or what it was to include. While many of the metrics required by the NDAA can be addressed with data from DHS’s administrative records, certain metrics that rely on estimates necessitate the use of alternative methodologies and in some cases, specialized technical expertise. For example, DHS contracted with the Institute for Defense Analyses to assist with the development of a statistical model for estimating undetected unlawful entries. In its fiscal year 2017 Border Security Metrics Report, DHS provided information on its methodological approaches, such as how it estimated undetected unlawful entries. In its first Border Security Metrics Report, DHS reported information on 35 of the 43 metrics called for by the NDAA. The metrics DHS provided spanned the four domains outlined in the NDAA and included a mix of counts, rates, estimates, or a combination thereof as shown in figure 1. For 18 of the 35 border security metrics DHS included in its report, we found DHS generally included elements listed in the NDAA. For example, the NDAA asked for the number of detected unlawful entries between POEs, and in its report DHS provided information on the number of detected unlawful entries over a 10-year period. As another example, the NDAA asked for the number of cargo containers at sea ports that were identified to be potentially high-risk. In response, DHS provided information on the number of potentially high-risk containers from fiscal years 2013 through 2016 and also provided contextual information about trends in the volume of such containers over time. See table 2 for more information on these examples as well as other examples of the types of information included in DHS’s fiscal year 2017 Border Security Metrics Report. For some metrics, DHS also provided information in addition to the elements listed in the NDAA. For example, the NDAA described the “AMO apprehensions assisted” metric as a count of the number of apprehensions that were assisted by CBP’s AMO through the use of unmanned aerial systems and manned aircraft. In addition to the counts for such assists, DHS also provided data on the flight hours expended to assist with these apprehensions. For 17 of the 35 reported metrics, we identified differences between the metric as described by the NDAA and as reported by DHS. The differences we identified generally fell into two categories: Metric differed in scope or calculation. Some of the metrics DHS reported on differed in scope or in their calculation from what the NDAA described for reasons such as data availability, among other factors. For example, DHS’s fiscal year 2017 Border Security Metrics Report scoped three metrics on unlawful border crossings between POEs (the “attempted unlawful border crosser apprehension rate,” the “estimated undetected unlawful entries,” and the “probability of detection rate”) to only include data for the southwest border. In these instances, the report noted that a methodology for estimating data on unlawful crossings for the northern border had not yet been completed but that research was underway to do so. As an example of a difference in calculation, DHS presented the interdiction effectiveness rate for each southwest border sector as an alternative to the metric “unlawful border crossing effectiveness rate in each Border Patrol sector.” According to DHS’s report, the department used the interdiction effectiveness rate because it had not yet produced and validated sector-level estimates of unlawful entries required to calculate the unlawful border crossing effectiveness rate. In its report, DHS stated it expects these estimates to be available for the 2019 report. Alternative metric provided. For the situational awareness in the maritime environment metric, DHS stated that it is in a multi-year process to develop a metric that meets the intent of the NDAA. As an alternative, DHS instead provided data on the number of aircraft and vessel operational hours that contributed to maritime domain situational awareness. See appendix I for additional information about any differences we identified for each metric. The eight metrics on which DHS did not provide information spanned all four domains. In its report, DHS explained that the eight omitted metrics were either still in development, under review within the department, or officials were in the process of collecting data for them. Table 3 lists the eight metrics on which DHS did not provide information and the date DHS estimated it will report on each metric. In general, DHS components responsible for collecting the data used in the metrics DHS reported have processes to help ensure the reliability of the data and the quality of the information provided. DHS also identified and disclosed limitations with some of the data elements or methodologies used for the metrics in its report. However, DHS does not have a systematic process for reviewing the reliability of data to identify limitations related to the metrics, and we identified at least one additional limitation for 21 of the 35 metrics on which DHS reported where DHS did not disclose such limitations or could have been more transparent about the limitations or assumptions in its report. Data are considered reliable when they are reasonably free from error and bias. Quality information is derived from relevant and reliable data and is considered to be, among other things, complete, accurate, and timely. The specific processes DHS components use to ensure data reliability vary from metric to metric. Examples of processes DHS or its components have implemented to help ensure the reliability of the data and the quality of information provided include: Issuing guidance and monitoring implementation. In September 2012, Border Patrol headquarters officials issued guidance to help provide a more consistent, standardized approach for the collection and reporting of turn back and got away data by Border Patrol sectors. Each sector is individually responsible for monitoring adherence to the guidance. According to DHS’s report, command staff at Border Patrol stations ensure agents are aware of and utilize proper definitions for apprehensions, got aways, and turn backs at their respective stations and also ensure that the necessary communication takes place between and among sectors and stations to minimize double-counting when subjects cross over multiple areas of responsibility. Supervisory reviews of data entries. With regard to data on AMO vessel and aircraft missions, AMO guidance mandates that supervisors perform a review of all pre- and post-mission data entries to help ensure accurate entry of mission information. AMO officials confirmed that supervisors review the data being entered into the database. Additionally, officials said AMO data teams run monthly validation checks of data entered to check for completeness and accuracy, such as out-of-range values. Using built-in electronic safeguards. CBP’s databases for entering and maintaining data elements—including travelers or passengers seeking admission, known inadmissible aliens at POEs, referrals for secondary examinations, major infractions, and private vehicles processed at a POE—have built-in processes to detect and prevent potential data entry errors. More specifically, as an officer enters a record, the systems check for valid entries into relevant fields and provide an error message to the officer for entries that appear to be invalid (e.g., if an officer leaves a mandatory field blank or enters contradictory information such as charging an individual with a crime while also entering a request for expedited removal). In some cases, the systems will prevent a record from being saved if any required fields are blank. Comparing data against other sources. As part of the Coast Guard’s data reliability processes for data on maritime migrant interdictions used in the “known maritime migrant flow rate” metric, Coast Guard officials said that analysts cross-check the data entered into their database with other Coast Guard reporting documents, such as internal spreadsheets, to ensure accuracy. Independent assessment of performance measure data. Some border security metrics are similar to, or use the same data elements as, performance measures DHS reports annually in response to the Government Performance and Results Act Modernization Act (GPRAMA) of 2010. For those performance measures, DHS annually assesses a subset of measures and their data for completeness and reliability using independent review teams. For example, in May 2017 an independent review team assessed the “migrant interdiction effectiveness in the maritime environment” performance measure, which uses the same data as the border security metric, “known maritime migrant flow rate.” The review team found the measure to be complete and reliable and the data to be of good quality overall, but also recommended that the Coast Guard and DHS continue work on an improved database to enhance the consistency of data collection, among other things. In addition to the components having processes to help ensure the reliability of the data and the quality of the information used in the report, DHS took steps to be transparent in its presentation of the metrics by identifying and disclosing known limitations with some of the data elements or methodologies used for the metrics in its report. Communicating the extent to which such limitations exist and their potential impact is important to help facilitate the appropriate use and understanding of the data and the metrics. DHS identified and disclosed limitations related to the potential for misclassification of observations, the potential for cases not being entered or recorded correctly, and methodological limitations, among other things. For example, one of the key limitations DHS’s report identified for the data on turn backs and got aways is that they are based on potentially subjective observations of agents who have to make a determination on how to classify them based on what they observed or the available evidence (e.g., tracks, sensor activations, interviews with apprehended subjects, camera views, etc.). Further, DHS’s report explained that agents may face challenges in making that determination because some unlawful border crossers may enter the United States to drop off drug loads or to act as decoys to lure agents away from a certain area and then return to Mexico, and therefore may be misidentified as turn backs, for example. As another example, DHS identified limitations due to cases not being entered or recorded correctly. For the “known maritime migrant flow rate” metric, DHS used data on the total number of maritime migrants interdicted. In its report, DHS explained that a potential limitation of this data element is that the Coast Guard relies on international and domestic partners to report their interdictions for compilation in its database. Consequently, the accuracy and completeness of the data depend on whether those reports are made by those partners and the accuracy of their reports. See appendix I for additional information about the limitations identified for each metric. Even as DHS identified and disclosed limitations related to some of its metrics, we identified at least one additional limitation for 21 of the 35 metrics on which DHS reported where DHS did not disclose such limitations or could have been more transparent about the limitations in its report. Examples of such instances include: Potential for cases not being entered or recorded correctly. In our previous work we found that mission data for unmanned aerial systems were inconsistently collected across operation locations. Specifically, in February 2017 we reported that there were instances where no assist information was recorded in AMO’s data system even though such assets participated in investigations and operations. Because AMO’s data may not reflect all asset assists, we recommended that AMO update and maintain guidance for recording mission information in its data collection system and provide training to users of the system. For its fiscal year 2017 Border Security Metrics Report, DHS used asset assists data in metrics such as the “AMO individuals detected,” “AMO apprehensions assisted,” and “illicit drug seizures assisted by AMO,” but did not disclose this limitation in its report. Potential for data to be changed over time. Border Patrol officials told us that data on the apprehension of unaccompanied alien children may change over time because original apprehension records from a shared database have, in some instances, been updated by staff from U.S. Immigration and Customs Enforcement (ICE) Enforcement and Removal Operations (ERO). Officials said that in January 2015 they noticed that ERO staff were inadvertently overwriting Border Patrol’s original data entries about the status of apprehended children when they made updates to those children’s records. For example, if a child was unaccompanied at the time of his or her apprehension and was recorded as such by Border Patrol in the initial record entry, ERO may have changed the “unaccompanied” status in the system after they matched the child with a family member or sponsor. As a result, data may not be reconcilable with initial apprehension counts over time. DHS did not fully disclose limitations for some metrics. We identified instances where DHS could improve transparency about the assumptions or limitations of the data presented in its report. For example, in 2014 Border Patrol implemented a standard, southwest border-wide methodology to improve reporting of turn backs and got aways. While DHS made mention of this change in the text of the report, the data for these metrics are presented in tables without any table notes or disclosures within the table about this change. Further, DHS’s report does not discuss how the change may affect comparability of the data. Consequently, a reader may not be aware that data for before 2014 in a table are not necessarily comparable to the data for 2014 and after in the same table. Without a comprehensive identification of the limitations of the metrics and their associated data, and without an adequate disclosure of those limitations, the value of DHS’s report as a source of information to Congress, policymakers, and the public may be diminished. The metrics in the report were specifically identified and requested by Congress in the NDAA, and provide Congress with important information about the outputs and outcomes of DHS’s border security policies and investments that could be used to inform decision-making. However, those reading the report may not be aware of important contextual information because DHS did not identify and disclose some limitations, thereby creating the potential for the data to be misinterpreted. According to DHS officials who prepared the report, while they took steps to identify methodological limitations of the metrics, no process currently exists to systematically review the reliability of operational data used for public reporting purposes, such as in the metrics report. Specifically, DHS officials within OIS told us that while they were responsible for leading and managing the preparation of the report, they largely relied on the DHS components from which they collected the data to assess the data’s reliability and communicate identified limitations. OIS officials explained that many of the data elements used, such as those from AMO or the Coast Guard, were ones with which they were not familiar or had not worked with previously in their area of immigration statistics. OIS officials also noted that in some cases, the data had previously been used in performance measures or had been collected and tracked for several years, so they trusted the components’ processes for ensuring their reliability and identifying limitations, but reviewed the data provided where possible and consulted with the components as needed. However, OIS officials said that while they included as much information in the report as was known about identified limitations with the existing operational data, no additional effort was made to systematically review the underlying reliability of the data to comprehensively identify limitations that should be acknowledged when publicly reported because no department-wide process exists to do so. Standards for Internal Control in the Federal Government state that management officials should evaluate data sources for reliability and communicate quality information, including relevant data from reliable sources, to achieve an agency’s objectives. The quality information can then be used by agency management and external stakeholders such as policymakers, to make informed decisions and evaluate performance, among other things. Further, DHS’s Management Directive on Information Quality states that data and information disseminated by the department should, among other things, have full, accurate, transparent documentation, and error sources affecting data quality should be identified and disclosed to users. Additionally, our previous work on approaches for verifying and validating performance information found that communicating significant data limitations and their implications allows stakeholders to judge the data’s credibility for their intended use and to use the data in appropriate ways. By developing and implementing a process to systematically review the reliability of the data or consider the results of assessments components have completed, comprehensively identify any limitations, and communicate the data or methodological limitations with the metrics, DHS would improve the quality of the information available to Congress, DHS leadership, and the public. Doing so would also facilitate a better understanding and appropriate interpretation and use of the data in the context of the Border Security Metrics Report, thereby enhancing the report’s value as a source of information for future decision-making. Based upon statistical modelling, DHS developed a Model-based Apprehension Rate to calculate the total number of unlawful border entries between land POEs, including entries both detected by Border Patrol and “estimated undetected unlawful entries.” DHS reported that in fiscal year 2016 there were about 624,000 detected entries (which include apprehensions, turn-backs, and got aways) and estimated that there were about 62,000 undetected unlawful entries. DHS also used the Model- based Apprehension Rate to develop two other metrics in the fiscal year 2017 Border Security Metrics Report: (1) A “probability of detection rate,” which is the estimated proportion of the number of detected unlawful border entries to the total number of unlawful entries between land POEs. DHS estimated that in fiscal year 2016, 91 percent of unlawful border crossers were detected and 9 percent were not detected. (2) The “attempted unlawful border crosser apprehension rate,” which is the estimated proportion of unlawful border entrants apprehended by Border Patrol to the total number of unlawful entrants between land POEs. DHS estimated that in fiscal year 2016, 65 percent of individuals were apprehended by Border Patrol and 35 percent of individuals attempting an unlawful border entry either got away or entered the United States undetected. DHS based its statistical model upon research conducted by the Institute for Defense Analyses that leveraged long-standing research using capture-recapture models. Originally developed and utilized in biological and ecological sciences, capture-recapture models have been applied to other disciplines, including social science. According to the Institute for Defense Analyses, capture-recapture models have been the core approach for academic efforts to model the process of unlawful entry into the United States across land borders for several decades. To develop its statistical model, DHS used a capture-recapture methodology to calculate a probability of apprehension by counting the number of unlawful border crossers that were apprehended multiple times. At a high-level, capture-recapture involves taking an initial sample of the population of interest, in this case individuals attempting to cross the border unlawfully. Then, separately, a second, independent sample of the same population is taken. The samples are then compared to determine the number of individuals who appear in both samples. When the number of individuals who appear in both samples (e.g., individuals who have been apprehended twice) is low, it can be inferred that the overall population of interest (e.g., total unlawful border crossers) is much larger than the total number of individuals in the two samples. On the other hand, if the recapture rate is high, then it can be inferred that the overall population of interest is not much larger than the total number of individuals in the two samples. In the context of unlawful border crossing, when an individual’s first attempt at unlawfully crossing the border is successful, the individual enters the United States and no apprehension is made. However, if an individual is apprehended, Border Patrol records an apprehension of this individual in a DHS data system and the individual is potentially subject to consequences for entering unlawfully, such as administrative enforcement and removal, criminal prosecution, or being barred from legally entering the United States in the future. The individual is then returned to his or her home country, where the individual can then choose whether or not to make another attempt to unlawfully cross the border. During a second attempt to unlawfully cross the border, the individual faces the same possible outcomes (enter the United States unlawfully or apprehension by Border Patrol). Figure 2 provides the framework for DHS’s Model-based Apprehension Rate. DHS modified the traditional capture-recapture methodology by calculating a deterrence rate of 60 percent in fiscal year 2016 to account for individuals who choose not to make another unlawful border crossing attempt. The deterrence rate accounts for an individual being deterred from attempting to unlawfully cross the border again; that is, DHS assumed that some percentage of apprehended individuals, once returned to their country, will remain in their home country. DHS calculated the deterrence rate based upon a survey of Mexican individuals who were apprehended and returned to the border region of Mexico by U.S. immigration authorities. DHS assumed the remaining 40 percent of individuals who were apprehended and removed to their home country in fiscal year 2016 remain undeterred and will attempt to unlawfully cross the border again. Historically, DHS (and its predecessor the Immigration and Naturalization Service) did not use statistical models to calculate an apprehension rate but relied on apprehensions as a proxy measure for all unlawful entries (both observed and unobserved) between POEs. DHS also included in its report information on the apprehension rate using this method. Specifically, DHS also calculated an Observational Apprehension Rate based on direct observations (unlawful border crossers observed by Border Patrol) and indirect observations (residual evidence of a border crosser, i.e., footprints) of attempted unlawful border crossers. Using the observational apprehension rate, DHS calculated that in fiscal year 2016, it apprehended 79 percent of unlawful border crossers. DHS made assumptions about border crossers to develop its statistical model and described these assumptions in its report; however, DHS did not validate some of these assumptions or determine how they potentially could affect the accuracy of the Model-based Apprehension Rate through the use of sensitivity analyses. More specifically, DHS’s model incorporates several assumptions related to border crossers. Among others, these assumptions include: the rate at which individuals will be deterred from crossing again remains the same, regardless of the number of attempts an individual has made; individuals who indicate an intent to stay near the U.S.-Mexico border will attempt re-entry; a single apprehension rate applies to diverse groups of border crossers, regardless of their nationality or the number of attempts an individual has made; and certain individuals will not evade Border Patrol. However, the validity of some of these assumptions—which affect the Model-based Apprehension Rate—is uncertain. For example, DHS’s model estimates the rate at which a diverse group of border crossers attempting to evade detection will be apprehended by Border Patrol. This group includes both Mexicans and non-Mexicans and individuals who attempt to cross again after varying amounts of time. Despite this diversity, the model assumes that all crossers have the same chance of apprehension on each attempt to cross the border. This assumption allows DHS to apply the estimated apprehension rate developed based on a sample of Mexicans re-apprehended within 90 days—the group for whom relevant data exist—to a broader population of individuals regardless of the number of attempts the border crossers have made or their nationality. However, DHS did not make efforts to determine the extent to which an apprehension rate based on Mexican citizens re- attempting entry within 90 days would reflect apprehension rates for non- Mexicans and individuals crossing again after longer periods. Additionally, DHS assumes that the apprehension rate never varies between an individual’s attempts at crossing the border. For example, DHS assumes that an individual making a first attempt at crossing the border faces the same odds of apprehension as an individual making a fourth or fifth attempt at crossing the border. However, DHS has not explored the possibility that, for example, individuals may gain experience and knowledge from border-crossing attempts that could help them better evade Border Patrol on subsequent attempts. Further, DHS’s model assumes that certain individuals unlawfully crossing the border, such as those seeking asylum, will not evade apprehension and will turn themselves in to Border Patrol. Specifically, in addition to individuals who ultimately do seek asylum, DHS also includes within this group and applies this assumption to individuals apprehended as a family unit and unaccompanied minors. Under this assumption, 100 percent of such individuals are apprehended. According to DHS’s fiscal year 2017 Border Security Metrics Report, these individuals have historically been released into the United States with a Notice to Appear in immigration court for legal proceedings on a future date, rather than being subject to immediate DHS enforcement consequences such as voluntary return. Therefore, DHS assumes that 100 percent of these individuals will self-present to Border Patrol because, in doing so, they are able to claim asylum or other protection and potentially remain in the United States. However, representatives from the Institute for Defense Analysis stated that while anecdotally self-presenting rates of these individuals are high, more rigorous analysis is needed to accurately estimate a self- presentation rate. For example, it is possible that not all families crossing the border unlawfully may seek to self-present to Border Patrol; some may attempt to evade capture and enter the United States undetected. In this case, DHS may be underestimating the number of individuals who unlawfully cross the border and enter the United States by assuming 100 percent of these individuals will self-present to Border Patrol agents. Additionally, DHS noted in its fiscal year 2017 Border Security Metrics Report that this assumption does not reflect the actual behavior of all border crossers in this group. OIS officials stated that they based this assumption on interviews with Border Patrol agents but had not done formal or quantitative analysis to support this assumption. Further, OIS officials stated that they did not have a strong alternative assumption to use instead and therefore assumed that 100 percent of individuals within this group are apprehended. DHS described these assumptions in its report but did not provide quantitative information on the extent to which these assumptions affected the Model-based Apprehension Rate through the use of sensitivity analyses. Sensitivity analyses help to convey the extent to which changing the values of variables, assumptions, data, or other input affects statistical estimates. For example, sensitivity analyses could provide information on how different assumptions about unlawful border crossers’ behavior and other inputs to the statistical model could have affected the Model-based Apprehension Rate. OIS officials stated that while they had started to run sensitivity analyses by modifying certain assumptions, they had not completed the analysis and did not include results of the sensitivity analyses in the report. The Office of Management and Budget’s (OMB) statistical standards for federal agencies include providing the results of sensitivity analyses for key methodological assumptions to ensure that these assumptions do not unduly affect the results of the model. By including the results of sensitivity analyses in its Border Security Metrics Report, DHS would allow Congress and the public to better understand the potential limitations associated with its model and make independent assessments on its accuracy. DHS used a statistical model to develop the Model-based Apprehension Rate but did not provide information on the level of uncertainty related to this estimate. Rather, the fiscal year 2017 DHS Border Security Metrics Report provided a single rate that does not fully convey the difficulty and uncertainty of estimating partially unobserved metrics, such as unlawful entries and the probability of detection. Specifically, using the Model- based Apprehension rate, DHS estimated that 65 percent of unlawful border crossers were apprehended in fiscal year 2016, and the remaining 35 percent entered the United States. However, like all statistical models, DHS’s estimate is based upon a limited sample of data and may be affected by random variation, meaning that DHS does not have complete certainty that its rate is accurate. DHS included a discussion of limitations in the report but did not quantify its degree of uncertainty. According to the OMB statistical standards for federal agencies, possible variation in estimates should be noted, such as by reporting the range of each estimate. Measures of statistical uncertainty, such as margins of error or confidence intervals, help to convey the amount by which estimates might vary due to randomness in the data and allows consumers of the estimates to evaluate their accuracy. OIS officials stated that they agree that providing measures of statistical uncertainty would help Congress and the public better understand the Model-based Apprehension Rate to evaluate border security. Officials told us that the office had begun to develop measures of statistical uncertainty but did not complete this effort because the staff member who was working on the analyses recently left the office. Further, OIS officials stated that they were unsure when they would be able to provide measures of statistical uncertainty in future reports. Including measures of statistical uncertainty in future reports would allow Congress, policy makers, and the public to more fully evaluate the extent to which the metrics that use the Model-based Apprehension Rate are valid. Further, while DHS may ultimately adopt a new, simulation-based model in the future, described later in this report, it plans to use the current Model- based Apprehension Rate for estimates in its Border Security Metrics Report for the foreseeable future. Therefore, providing this additional information about the estimates would allow DHS to more accurately convey how limitations in available data and methods could affect the results and provide more useful information about migration and border enforcement. Additionally, to the extent DHS adopts a new estimating metric, that estimate may have some level of uncertainty associated with it. DHS is developing another model because its current statistical model may not sufficiently reflect conditions at the southwest border. Specifically, DHS’s current statistical model does not fully account for the changing population of unlawful border crossers. The capture-recapture methodology, which underlies the Model-based Apprehension Rate, was developed to sample homogenous populations that behave in set, uniform ways. However, those crossing the border have become increasingly diverse in recent years. Our analysis of DHS data used to develop the Model-based Apprehension Rate shows that the number of unlawful border crossers whose characteristics and behavior are best reflected in the statistical model has declined. For example, our analysis illustrated that the population that conforms best to the model’s assumptions—adult Mexicans travelling without dependents who do not plan to claim asylum and who are returned to Mexico in a short amount of time—has fallen from over 60 percent of apprehensions in fiscal year 2000 to less than 25 percent of apprehensions in 2016, as shown in figure 3. Conversely, the number of individuals who are excluded from the statistical model such as non-Mexicans, and individuals whose behavior may not reflect the model’s assumptions, such as asylum-seekers or those who have not departed the United States (e.g., because they are awaiting immigration court proceedings) have increased over time, as shown in figure 4. For example, the percentage of individuals apprehended at the border who are excluded from the model because they await immigration court proceedings increased from 26 percent in fiscal year 2000 to almost 70 percent in fiscal year 2016. DHS acknowledged these trends in its fiscal year 2017 Border Security Metrics Report and noted them as a limitation to the effectiveness of its model. OIS officials further noted that some of these limitations are difficult to address within the bounds of the statistical model. For example, to properly account for non-Mexicans, OIS officials stated that they would need information on the rate at which non-Mexicans are deterred from crossing the border. However, it would be difficult and costly to obtain this information through the use of a survey and real-world data does not already exist, according to OIS officials. To help address limitations of its current statistical model, DHS has invested in another research project to estimate the number of unlawful border crossers between land POEs, including unknown border entries. Border Patrol contracted with Johns Hopkins Applied Physics Laboratory to undertake a project that aims to use a combination of statistical modeling and data from sensors along the border to estimate the total number of unlawful border entries between land POEs, including entries both detected by Border Patrol and those not detected by Border Patrol. According to project documentation we reviewed, the project plans to leverage the CBP Tactical Simulation, an agent-based simulation of tactical border operations. CBP Tactical Simulation incorporates information on terrain at the border based on geographic information systems and sensors along with probability models that reflect how Border Patrol agents and unlawful border crossers behave in given circumstances. Border Patrol and OIS officials told us that this project would be more adaptable to changing border conditions and could help the agency address limitations associated with the Model-based Apprehension rate. Specifically, according to OIS officials, a simulation-based estimate would rely upon fewer assumptions about the types of individuals who unlawfully cross the border as compared to the current Model-based Apprehension rate. However, Border Patrol officials noted that estimates of unobservable phenomena, such as unobserved border entries, always face some limitations in their accuracy and that the new model may still rely upon samples of data that would have associated uncertainty as well as assumptions that would need to be validated. Ultimately, though, Border Patrol officials stated that the simulation-based model may be an improvement upon the current Model-based Apprehension rate. Border Patrol officials stated that the first iteration of the model would be presented to Border Patrol leadership for their review at the end of fiscal year 2019 and if at that time Border Patrol leadership approves the model, the earliest the simulation-based estimate could potentially be incorporated into the DHS Border Security Metrics Report would be for fiscal year 2020. Exploring alternative models is a positive step for DHS, however given that the project is in the early stages, it is too early to tell if it will be able to address the limitations we identified associated with the current model. In addition to the NDAA metrics, we have identified other metrics that DHS could use to help measure the effectiveness of border security. In particular, based on the findings from our previous reviews of border security programs and efforts, we have recommended that DHS use metrics that are relevant to each of the four domains listed in the NDAA— between POEs, at POEs, the maritime border, and for air and marine security in the land domain. For example, Between POEs domain. In February 2017, we reported on the use of border fencing along the southwest border and found that CBP collects data that could help provide insight into how border fencing contributes to border security operations, including the location of illegal entries. For example, we found that CBP collects data it could potentially use to determine the extent to which border fencing diverts illegal entrants into more rural and remote environments, and border fencing’s impact, if any, on apprehension rates over time. However, CBP had not developed metrics that systematically use these data to assess the contributions of border fencing to its mission. To better position CBP to make resource allocation decisions with the best information available to inform competing mission priorities and investments, we recommended that the Chief of the Border Patrol develop metrics to assess the contributions of pedestrian and vehicle fencing to border security along the southwest border using CBP data. DHS agreed with the recommendation and stated that it planned to develop metrics for use in its operational control framework for southwest border security operations. As of October 2018, DHS stated that the department planned to test the metrics and implement them in the framework by September 2019. At POEs domain. In July 2017, we reported on the Importer Security Filing (ISF) program and found that while ISF rule data have improved the program’s ability to identify high-risk cargo shipments, CBP could collect additional performance information to better evaluate program effectiveness. While evaluating the direct impact of using ISF rule data to assess shipment risk is difficult, we identified examples of how CBP could better assess the ISF program’s effectiveness. For example, CBP could track the number of containers not listed on a manifest—which could pose a security risk—it identifies through reviewing vessel stow plans. Collecting this type of additional information would help CBP better assess whether the ISF program is improving its ability to identify high-risk shipments. Therefore, we recommended that CBP identify and collect additional performance information on the impact of the ISF rule data, such as the identification of shipments containing contraband, to better evaluate the effectiveness of the ISF program. DHS agreed with the recommendation and reported that it is working to assess additional performance metrics to evaluate the effectiveness of the ISF program and anticipates completing the assessment by end of December 2019. Maritime border domain. In October 2017, we reported on the Coast Guard’s performance goals and found that although the Coast Guard’s performance goals are generally aligned with its statutory missions, the Coast Guard does not explain why certain aspects of mission performance are measured while others are not. For example, we found that while the Coast Guard’s mission is to interdict all illegal drugs, the agency’s two performance goals related to that mission were for cocaine interdiction only, excluding many other substances. We recommended that the Coast Guard either develop new performance goals to address mission activity gaps, or explain in the Coast Guard’s Annual Performance Report why certain aspects of mission performance are measured while others are not. Developing new goals to address missions, or describing how existing goals sufficiently assess mission performance, could better convey the Coast Guard’s progress in achieving its missions. DHS agreed with the recommendation and in February 2018, the Coast Guard provided us with its updated fiscal year 2017 Annual Performance Report. We found that while the updated report explained why performance goals related to its drug interdiction mission focus solely on cocaine interdiction, for the four other performance goals we previously identified as not fully addressing all related mission activities, the updated report did not include additional goals or explain why certain aspects of mission performance are not measured. We continue to believe that in instances in which performance goals do not fully address all of the respective mission activities, the Coast Guard’s Annual Performance Report should include an explanation. Air and marine security in the land domain. In May 2017, we reviewed DHS’s efforts to address subterranean, aerial, and maritime smuggling of drugs and humans. We found that while DHS established high-level performance measures and collected data on smuggling by ultralight aircraft, it had not assessed its efforts specific to addressing this smuggling method. Additionally, we found that DHS had similarly not assessed smuggling methods such as tunnels, panga boats (a fishing vessel), and recreational vessels. We recommended that DHS direct CBP, ICE, and Coast Guard to establish and monitor performance measures and targets related to ultralight aircraft, cross-border tunnels, panga boats, and recreational vessel smuggling to help provide reasonable assurance that efforts to address these smuggling methods are effective. By establishing measures and monitoring performance against targets, managers could obtain valuable information on successful approaches and areas that could be improved to help ensure that technology investments and operational responses to address these smuggling methods are effective. DHS agreed with the recommendations for measures related to ultralight aircraft and cross-border tunnels. DHS reported that AMO and Border Patrol have drafted a performance measure for ultralight aircraft, however, reviews and approval of the measure will not be completed until November 2019. As of June 2018, DHS reported that ICE was leading the development of measures related to cross-border tunnels. DHS did not agree with the recommendation to establish measures and monitor performance against targets for smuggling by panga boats and recreational vessels because the department believed measures and targets would not provide the most useful strategic assessment of operations to prevent all illicit trafficking, regardless of area of operations or mode of transportation. We continue to believe that the recommendation is valid and recognize the value of high-level strategic performance measures. However, such high-level measures may not provide sufficiently detailed performance information to allow DHS to identify successful approaches to addressing smuggling by panga boats and recreational vessels and areas for improvement. Further, establishing performance measures and targets related to smuggling by panga boats and recreational vessels could, in turn, better position DHS to understand the overall smuggling threat. Appendix II provides additional information on these and other metrics we have previously recommended that DHS could use to help measure the effectiveness of border security in the four domains. Securing U.S. borders is a complex undertaking that spans multiple domains and locations. It is also a key part of DHS’s mission for which DHS has made significant investments over the years. Given the complexity and breadth of border security efforts, having data and information available on the state of border security is important for DHS as well as policymakers and the public to understand the effectiveness of those investments. DHS’s fiscal year 2017 Border Security Metrics Report makes an important contribution in providing such data and information. DHS components generally have processes to help ensure the reliability of the data used in the metrics report and DHS identified and disclosed some data and methodological limitations with the metrics. However, DHS did not systematically review the reliability of data used in all metrics to identify and disclose limitations and their potential implications for the metric. Without complete information about the limitations of the data or the metric methodologies used in the report, Congress, policymakers, and the public may not be aware of important context or information needed to fully and appropriately understand the data being presented. By developing and implementing a process to systematically review the reliability of the data, as well as comprehensively identify limitations and communicate limitations of the metrics, DHS would improve the quality of the data and information provided in the report which would facilitate a better understanding and appropriate interpretation of the data and information provided. To develop three metrics in the report, DHS used a statistical model that incorporated untested assumptions about the behavior of unlawful border crossers that may not reflect real-world conditions. DHS was transparent about the limitations of its model, but providing the results of sensitivity analyses and measures of statistical uncertainty related to the model would allow Congress, policymakers, and the public to better understand its potential limitations and more fully evaluate the validity of DHS’s metrics that use estimates. We are making the following four recommendations to DHS: The Secretary of Homeland Security should develop and implement a process to systematically review the reliability of the data used in its Border Security Metrics Report and comprehensively identify any limitations with the data and methodologies that underlie its metrics. (Recommendation 1) The Secretary of Homeland Security should ensure the communication of the limitations of the metrics identified through the systematic review in the department’s annual Border Security Metrics Report. (Recommendation 2) The Under Secretary for the Office of Strategy, Policy, and Plans should include the results of sensitivity analyses to key assumptions in its statistical models of unlawful entry estimates in its annual Border Security Metrics Report. (Recommendation 3) The Under Secretary for the Office of Strategy, Policy, and Plans should include measures of statistical uncertainty for all metrics based on estimates derived from statistical models in its annual Border Security Metrics Report. (Recommendation 4) We provided a draft of this report to DHS and the Office of National Drug Control Policy for review and comment. DHS provided written comments, which are reproduced in appendix III and discussed below. DHS also provided technical comments, which we incorporated as appropriate. The Office of National Drug Control Policy indicated via e-mail that it did not have any comments on the draft report. In its comments, DHS concurred with our recommendations and stated that it planned to implement 3 of the 4 by October 2020. With respect to our second recommendation, DHS requested that we consider it closed as implemented because the department already detailed some of the limitations in its fiscal year 2017 report, and plans to continue to identify known limitations and the progress made to mitigate previously identified limitations in future reports. As discussed in this report, we agree that DHS identified and disclosed limitations for some metrics in its fiscal year 2017 Border Security Metrics Report; however, we identified at least one additional limitation for 21 of the 35 metrics on which DHS reported that DHS did not disclose or about which it could have been more transparent. To address the intent of this recommendation, once DHS has implemented a process to systematically review the reliability of the data used in its report and comprehensively identified related limitations, it should disclose those limitations in its annual Border Security Metrics Report. We are sending copies of this report to the appropriate congressional committees and the Secretary of the Department of Homeland Security. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. Contacts points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. If you or your staff have any questions about this report, please contact me at (202) 512-8777 or gamblerr@gao.gov. GAO staff who made key contributions to this report are listed in appendix IV. This appendix provides additional information on our analysis of the suitability and validity of the metrics the Department of Homeland Security (DHS) reported in its fiscal year 2017 Border Security Metrics Report for each of the four domains listed in the National Defense Authorization Act for Fiscal Year 2017 (NDAA)—between ports of entry, at ports of entry, the maritime border, and air and marine metrics in the land domain. Specifically, this appendix provides information on the metrics including their status, descriptions, differences between what DHS reported for the metrics and how they were described or defined by the NDAA, limitations, and any additional information or planned actions by DHS, where applicable. Description This metric is a rate comparing apprehensions to the total number of attempted unlawful border crossers. As such, this metric requires an estimate of the number of unlawful entry attempts that are not detected, which is added to the number of detected unlawful border crossers to create the denominator. The Department of Homeland Security (DHS) provided two methods for calculating this rate in its report. The first method, called the Model-based Apprehension Rate, uses a statistical model based on the capture-recapture methodology to estimate the rate.The second method, called the Observational Apprehension Rate, calculates the ratio of apprehensions to the sum of apprehensions and got aways. unlawful entries is limited to the southwest border. According to the report, research is underway on methods to produce estimates for the northern border. Limitations The Observational Apprehension Rate incorporates data on apprehensions, and got aways, while the Model- based Apprehension Rate is based on an estimate for undetected unlawful entries. Consequently, the limitations for those metrics also apply here. For more information on the limitations for those metrics, see the respective sections below. DHS identified: The observational apprehension rate excludes unobserved got aways. Additional information and planned actions by the Department of Homeland Security In its report, DHS noted that it has taken steps to improve situational awareness along the border and mitigate limitations. These steps include investing in technology, refining observational estimates, and developing a methodology to estimate statistical reliability. According to U.S. Border Patrol officials, investments in new technology have enabled U.S. Border Patrol to better detect cross-border activities. For additional information on the data elements used for this metric and DHS’s planned actions, see the respective sections below on apprehensions, got aways, and the estimate for undetected unlawful entries. Description This metric is a count of the total number of attempted unlawful border crossers between land ports of entry who were directly or indirectly observed or detected by U.S. Border Patrol. The Department of Homeland Security (DHS) calculated this metric by adding turn backs, got aways, and apprehensions of unlawful border crossers. Patrol officials, the northern border has different immigration dynamics than the southern border and accounts for a significantly smaller number of turn backs and got aways overall, so northern border data were not included. Limitations Because this metric incorporates data on apprehensions, got aways, and turn backs, the limitations for those metrics also apply here. For more information on the limitations for those metrics, see the respective sections below. Additional information and planned actions by the Department of Homeland Security For additional information on the data elements used for this metric and DHS’s planned actions, see the respective sections below. Description This metric is an estimate of the number of attempted unlawful border crossers that are not directly or indirectly observed or detected by U.S. Border Patrol (Border Patrol). The Department of Homeland Security (DHS) used a statistical model, based on capture-recapture methodology, to estimate total successful unlawful entries, and subtracted detected got aways to calculate the total number of undetected unlawful entries. unlawful entries is limited to the southwest border. According to DHS’s report, research is under way to produce this estimate for the northern border. DHS does not currently have reliable data on the estimated share of migrants who, following an unsuccessful unlawful entry attempt, are deterred from making a subsequent reentry attempt. For its model, DHS used data from a survey of recently removed Mexicans, which asked them about their intentions to re-enter the United States. According to DHS’s report, a shortcoming of the survey is that it does not take account of shifting border enforcement efforts, potential changes in behavior by individuals who have been exposed to consequence programs, or other deterrent factors along the border. Consequently, any resulting undercount in the estimate of the deterred population results in a downward bias. The population that conforms best to the model’s assumptions represents a diminishing share of southwest border apprehensions. Specifically, in its report DHS said that Mexican adults removed to the nearest border accounted for about 95 percent of apprehensions in the 1990s. However, because of recent changes at the border, including changes in the composition of border flows (i.e., rising numbers of Central Americans and asylum seekers) and in Border Patrol’s enforcement strategy, the population best reflected in the model has declined to as few as 20 percent of apprehensions in recent years. Further, DHS noted that some alien populations, such as those seeking asylum and who do not evade detection by Border Patrol agents, are also excluded from the model. However, these populations make up an increasing share of apprehensions in recent years. The model uses restrictive assumptions about which re-apprehensions to include. For example, the model excludes apprehensions occurring at check points and other remote locations and those occurring more than 4 days after an illegal entry. According to DHS, these assumptions result in a downward bias. We identified: DHS described assumptions it made in its report but did not provide quantitative information on the extent to which they affected its estimated undetected unlawful entries through the use of sensitivity analyses. Sensitivity analyses help to convey the extent to which changing the values of variables, assumptions, data, or other input affects statistical estimates. By including the results of sensitivity analyses in its Border Security Metrics Report, DHS would allow Congress and the public to better understand the potential limitations associated with its model and make independent assessments on its accuracy. DHS did not provide information on the statistical level of uncertainty related to this rate, such as margins of error or confidence intervals. This information would help convey how the estimates might vary due to randomness in the data. Instead, DHS provided a single rate that does not fully convey the difficulty and uncertainty of the estimate. This metric incorporated data on apprehensions and got aways. For more information on the limitations associated with those metrics, see the respective sections below. Additional information and planned actions by the Department of Homeland Security According to DHS, officials are continuing to improve the accuracy of the existing statistical model for estimating unlawful border crossers but are also considering alternative methodologies. U.S. Customs and Border Protection has contracted with Johns Hopkins Applied Physics Laboratory to develop a new model for estimating the flow of unlawful border crossers. This model uses a combination of statistical modelling, data from sensors along the border, and probability models that reflect how Border Patrol agents and unlawful border crossers behave in given circumstances. Border Patrol officials estimated that the earliest the simulation-based estimate could potentially be incorporated into the DHS Border Security Metrics Report would be for fiscal year 2020. Description This metric is a count of the number of unlawful border crossers who, after making an unlawful entry into the United States, responded to law enforcement efforts by returning promptly to the country from which they entered. These data came from U.S. Border Patrol (Border Patrol) records. Border Patrol officials, the northern border has different immigration dynamics than the southern border and accounts for a significantly smaller number of turn backs overall, so northern border data were not included. Officials stated that while the current emphasis of reporting is on the southwest border, efforts are underway to identify and find ways to capture data that are important and reflective of the effectiveness in addressing threats specific to the northern border. The estimate aggregates potentially subjective observations from thousands of individual agents. Some unlawful border crossers may enter the United States to drop off drug loads or to act as decoys to lure agents away from a certain area and then return to Mexico, and therefore may be misidentified as turn backs. In our previous work we identified differences in the procedures for reporting and classifying turn backs across sectors, and noted how factors such as terrain and weather may impact agents’ abilities to accurately detect turn backs. According to DHS, since 2014, Border Patrol has implemented a standard, southwest border-wide methodology to improve reporting and mitigate the potential subjectivity of observations by agents. Therefore, data before 2014 are not necessarily comparable to data from 2014 and later. DHS presented the data in a table without explaining that the methodology used to categorize and count turns backs changed in 2014. Additional information and planned actions by the Department of Homeland Security According to DHS’s report, Border Patrol has taken steps to implement a standard, southwest border-wide methodology to improve reporting of potential turn backs. In addition, DHS’s report said that command staff ensure all agents are aware of and utilize proper definitions for apprehensions, got aways, and turn backs at their respective stations. They also ensure necessary communication takes place between and among sectors and stations to minimize double-counting when subjects cross through more than one station. DHS’s report noted that Border Patrol headquarters components validate data integrity. Description This metric is a count of the number of unlawful border crossers who are directly or indirectly observed entering unlawfully, are not apprehended, and are not turn backs. These data came from U.S. Border Patrol (Border Patrol) records. Border Patrol officials, the northern border has different immigration dynamics than the southern border, so northern border data were not included. Officials stated that while the current emphasis of reporting is on the southwest border, efforts are under way to identify and find ways to capture data that are important and reflective of the effectiveness in addressing threats specific to the northern border. Limitations DHS identified: The count aggregates potentially subjective observations from thousands of individual agents. In previous work we identified differences in procedures for reporting and classifying got aways across sectors, and noted how factors such as terrain and weather may impact agents’ abilities to accurately detect got aways. According to DHS, since 2014, Border Patrol has implemented a standard, southwest border-wide methodology to improve reporting and mitigate the potential subjectivity of observations by agents. Therefore, data before 2014 are not necessarily comparable to data from 2014 and later. DHS presented the data in a table without explaining that the methodology used to categorize and count turns backs changed in 2014. For information on limitations with the model-based estimate for undetected unlawful entries, see the section for estimated undetected unlawful entries above. Additional information and planned actions by the Department of Homeland Security According to DHS’s report, Border Patrol has taken steps to implement a standard, southwest border-wide methodology to improve reporting of potential got aways. In addition, DHS’s report said that command staff ensure all agents are aware of and utilize proper definitions for apprehensions, got aways, and turn backs at their respective stations. They also ensure necessary communication takes place between and among sectors and stations to minimize double-counting when subjects cross through more than one station. DHS’s report noted that Border Patrol headquarters components validate data integrity. As a comparison against the counts of documented got aways, DHS also provided an estimate of total successful unlawful entries along the southwest border using a statistical model based on capture-recapture methodology. For more information on the methodology for this estimate, see the section titled “Estimated Undetected Unlawful Entries” in this appendix. Description This metric is a rate comparing the number of apprehensions and turn backs to the number of apprehensions, estimated undetected unlawful entries, turn backs, and got aways in each U.S. Border Patrol sector. rate is not available because sector-level estimates of unlawful entries and attempts have not yet been produced and validated. As an alternative, DHS presented data using the interdiction effectiveness rate. With this rate, the estimated undetected unlawful entries measure is replaced with known got aways. However, DHS does not have an interdiction effectiveness rate for the northern border so it solely provided data for the southwest border. According to DHS’s report, the department has not yet developed a northern border interdiction effectiveness rate because there are only a small number of attempted and successful entries along the northern border. Limitations None identified. Additional information and planned actions by the Department of Homeland Security DHS reported that sector-level estimates of unlawful entries and attempts are projected to be available in its 2019 annual Border Security Metrics Report to Congress. Authorization Act for Fiscal Year 2017 (NDAA) defined this metric as a rate comparing the estimated total undetected unlawful border crossing attempts to the unlawful border crossing effectiveness rate. The Department of Homeland Security (DHS) calculated this metric by dividing the detected unlawful entries by the estimated total unlawful entries. The number of detected unlawful entries is calculated by adding turn backs, got aways, and apprehensions. Estimated total unlawful entries is calculated by adding turn backs, apprehensions and estimated total successful unlawful entries derived from DHS’s statistical model. unlawful entries is limited to the southwest border. Additionally, DHS used detected unlawful entries as the numerator, instead of the estimated total unlawful border crossing attempts not detected as called for in the NDAA. For the denominator DHS used the estimated total unlawful entries instead of the unlawful border crossing effectiveness rate, as called for in the NDAA. Limitations Because this metric incorporates data on apprehensions, got aways, and turn backs, as well as the estimate for undetected unlawful entries, the limitations for those metrics also apply to this metric. For more information on the limitations for those metrics, see the respective sections for those metrics. Additional information and planned actions by the Department of Homeland Security For additional information on apprehensions, got aways, turn backs, and the estimate for undetected unlawful entries, and any planned actions by DHS for those metrics, see the respective sections for those metrics. Description This metric is a count of the number of apprehensions in each U.S. Border Patrol (Border Patrol) sector. Data come from Border Patrol records, and each apprehension of the same unlawful crosser in a fiscal year is counted separately, meaning these data do not represent a count of unique crossers apprehended. Border Patrol officials, the northern border has different immigration dynamics than the southern border, so northern border data were not included. Officials stated that while the current emphasis of reporting is on the southwest border, efforts are under way to identify and find ways to capture data that are important and reflective of the effectiveness in addressing threats specific to the northern border. Limitations DHS identified: In its report, DHS said that apprehensions are not a useful indicator of successful unlawful border crossings over the long-term and across multiple locations because the relationship between apprehensions and successful unlawful entries depends on the apprehension rate, which changes over time and may differ by location. Additional information None. Description This metric is a count of the number of apprehensions of unaccompanied alien children (UAC), and the nationality of such children, in each U.S. Border Patrol (Border Patrol) sector. A UAC is a child under 18 years old with no lawful immigration status, and no parent present and available in the United States to provide care and physical custody. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report The Department of Homeland Security (DHS) only included data for the southwest border. Limitations DHS identified: Agents may not be able to reliably distinguish among older children and young adults or confirm whether children are traveling alone or in family groups. We identified: We previously reported that it can be challenging to obtain accurate information about a child’s country of origin because of absence of documentation, language barriers, and coached responses by smugglers, among other reasons. Border Patrol officials said that the data on UAC may have reliability issues because original data from a shared database had been changed. Specifically, officials said that in January 2015 they noticed that Enforcement and Removal Operations staff were inadvertently overwriting Border Patrol’s original data entries about the status of migrant children apprehended once those children were placed with relatives or a foster family. Additional information and planned actions by the Department of Homeland Security According to Border Patrol officials, agents rely on statements provided by the child to determine the nationality of UACs when verifiable documentation is not available. Verifiable documentation could include biometric checks, birth certificates, state-issued identification cards, and passports. However, officials noted that this list is not all-inclusive and the processing agent determines the validity of any presented documents. Border Patrol officials said that a data integrity team regularly examines data on apprehensions and they conduct biweekly data reliability checks. Additionally, they are working with Enforcement and Removal Operations to modify the data entry process so that updates can be made without overwriting the original apprehension data entered by Border Patrol. Description This metric is a count of the number of apprehensions of family units, and the nationality of such family units, in each U.S. Border Patrol (Border Patrol) sector. A family unit is the number of individuals apprehended with a family member. For example, a mother and child apprehended together are counted as two family units. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report The Department of Homeland Security (DHS) only included data for the southwest border. Limitations DHS identified: DHS noted that the count of apprehensions for family units is considered reliable, but that agents may not be able to reliably identify family units. We identified: According to Border Patrol officials, their data entry system did not have a dedicated field for agents to record apprehensions of persons within a family unit for all of the years presented in the report. In December 2014, Border Patrol added specific data entry fields to its data entry processes for agents to input information about family units. These fields incorporated built in safeguards and edit checks to help ensure that agents make an appropriate family unit classification. Previously, Border Patrol officials said they used proxy data to identify family units. Given the additional safeguards and checks included with the new family unit data entry fields, Border Patrol officials stated that the data after December 2014 may be more reliable overall compared to previous years. Border Patrol officials stated that they have high confidence in the proxy count for data pre-2014, but acknowledged that those data may contain misclassifications of family units. Additional information According to Border Patrol officials, agents are trained in interviewing techniques and the processing agent will consider all available evidence to determine the validity of claims to familial relationships. Border Patrol officials also noted that in order to be categorized as a family unit, at least one member of the family unit must be at least 18 years of age. Consequently, related individuals younger than 18 years of age that are apprehended together would not be categorized as a family unit. Description This metric is a rate comparing the amount and type of illicit drugs seized between ports of entry in any fiscal year to the average of the amount and type of illicit drugs seized between ports of entry in the immediately preceding 5 fiscal years. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations None identified. Additional information None. Description This metric was not specifically defined in the National Defense Authorization Act for Fiscal Year 2017 (NDAA); the NDAA called for an estimate of the impact of the Consequence Delivery System (CDS) on the recidivism rate of unlawful border crossers over multiple fiscal years. The Office of Immigration Statistics (OIS) calculated this metric by providing the average annual recidivism rate for the 3 years prior to fiscal year 2012— when the CDS was implemented—and the average annual recidivism rate for the subsequent 3 years. The annual recidivist rate is calculated by dividing the number of unique crossers apprehended multiple times in a fiscal year by the total number of unique crossers in the fiscal year. to DHS’s report, recidivism data for the northern border were not available due to the small number of attempted illegal entries along the northern border. Noting the findings from our January 2017 review, DHS stated that its current recidivism measure could be strengthened by using the date an unlawful border crosser is removed or returned instead of the date they are apprehended, as well as by counting re-apprehensions within a fixed period of time defined by the crosser’s repatriation date instead of by the fiscal year. In January 2017, we reported that using a crosser’s apprehension history beyond 1 fiscal year, and excluding crossers that have not been previously removed, among other things, produces a significantly different rate compared to how DHS currently calculates it. Consequently, we recommended that DHS calculate recidivism for a period of time longer than 1 fiscal year and that DHS exclude from the recidivism calculation aliens for whom there is no record of removal and who may remain in the United States. As of December 2018, this recommendation remained open. DHS stated that changes in the recidivism rate after 2012 cannot be attributed solely to CDS because enforcement is a complex, dynamic system. We identified: Given that DHS’s methodology is to provide the 3-year average of the recidivism rate before and after CDS was implemented in fiscal year 2012, the data presented will remain static for subsequent annual reports because the periods of comparison for analyzing recidivism are fixed around a specific point in time. According to OIS officials, to help address this issue, in the next report they plan to provide individual rates for each year instead of the 3-year average. Additional information and planned actions by the Department of Homeland Security In its report, DHS noted that future reports will include estimates of the impact of CDS on both the annual recidivism rate and a longer-term recidivism rate. For example, OIS officials said they plan to update the way they calculate recidivism for future issues of the report and are developing a multivariate impact analysis that would take into consideration factors such as crossers’ demographics and immigration history. Description This metric was not specifically defined in the National Defense Authorization Act for Fiscal Year 2017 (NDAA); the NDAA called for an examination of each consequence under the Consequence Delivery System (CDS), including (1) voluntary return, (2) warrant of arrest or notice to appear, (3) expedited removal, (4) reinstatement of removal, (5) alien transfer exit program, (6) criminal consequence program, (7) standard prosecution, and (8) Operation Against Smugglers Initiative on Safety and Security. The Department of Homeland Security (DHS) presented data on the recidivism rates for each consequence between fiscal years 2012 through 2016. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report As noted above for the metric “estimates of the impact of the Consequences Delivery System on recidivism,” DHS only included data for the southwest border because recidivism data for the northern border were not available due to the small number of attempted illegal entries along the northern border. Differences in recidivism rates among the consequences may also reflect differences in the propensity of the targeted populations to attempt to re-enter. As with the metric for estimating the impact of the CDS on recidivism discussed above, DHS noted the limitation that current recidivism data are based on apprehensions within a given fiscal year, and not the date when an individual was repatriated to their country of origin. In January 2017, we reported that some unlawful border crossers were incorrectly classified based on CDS guidance. U.S. Border Patrol (Border Patrol) agents implement CDS by classifying apprehended aliens into one of seven noncriminal or criminal categories and then applying one or more of eight different consequences; therefore, determining the correct classification of the unlawful border crosser is important for identifying and applying the appropriate consequence. Our analysis of Border Patrol apprehension data from fiscal year 2013 through 2015 showed that Border Patrol did not classify 11 percent of apprehensions in accordance with the agency’s guidance. We recommended that Border Patrol provide consistent guidance for classification and take steps to ensure the integrity of classification data. Border Patrol implemented this recommendation as of December 2017, but the issue could potentially have implications for the data DHS used in this metric, which was for fiscal years 2012 through 2016. Additional information and planned actions by the Department of Homeland Security According to its report, DHS is refining its analysis and will seek to specifically address the limitations discussed above in the fiscal year 2018 version of the Border Security Metrics Report. Ports of entry are U.S. government facilities that provide for the controlled entry into or departure from the United States. There are 328 ports of entry in the United States. Specifically, a port of entry is any officially designated location (seaport, airport, or land border location) where U.S. Customs and Border Protection (CBP) officers or employees are assigned to clear passengers, merchandise and other items, collect duties, and enforce customs laws; and where CBP officers inspect persons seeking to enter or depart, or apply for admission into, the United States pursuant to U.S. immigration law and travel controls. CBP’s Office of Field Operations (OFO) is the lead DHS component responsible for carrying out activities at POEs. The 15 metrics in this domain measure the number of travelers attempting to enter the United States at ports of entry, illicit drugs seized at ports of entry, and cargo entering the United States, among other things. DHS included 11 of the 15 metrics called for in the NDAA for this domain in its fiscal year 2017 Border Security Metrics Report, as shown in table 5. DHS reported that the four metrics for which it did not provide information did not yet have a reliable methodology or were under review, and that DHS was in the process of developing methodologies to capture the data needed for the requested metrics. DHS officials said these four metrics would not be ready for inclusion in the next annual report. Description This metric is a count of total inadmissible travelers, and requires an estimate of the number of inadmissible travelers who successfully enter at a port of entry without being detected. The metric is the sum of the number of inadmissible travelers interdicted and the estimated number of inadmissible travelers who successfully enter at a port of entry without being detected. inadmissible travelers who successfully enter at a port of entry without being detected. Therefore, DHS only presented data on known inadmissible travelers. Limitations None identified. Additional information and planned actions by the Department of Homeland Security DHS projected that the department may be able to include estimates on the number of inadmissible travelers who successfully enter at a port of entry in its fiscal year 2019 Border Security Metrics Report to Congress. According to U.S. Customs and Border Protection (CBP) officials, they are in the process of determining whether CBP’s Compliance Measurement Examination (COMPEX) program could be used as a means to reliably measure undetected inadmissible travelers. Description These metrics are rates that require data on travelers seeking admission at a port of entry, interdictions of inadmissible travelers, and an estimate of the number of inadmissible travelers who successfully enter at a port of entry without being detected. The refusal rate is calculated by dividing the number of inadmissible travelers interdicted by all people seeking admission at a port of entry. The interdiction rate is calculated by dividing the number of inadmissible travelers interdicted by the total number of inadmissible travelers who attempt to enter at a port of entry. inadmissible travelers who successfully enter at a port of entry without being detected. Therefore, DHS only presented data on the refusal rate. Limitations None identified. Additional information and planned actions by the Department of Homeland Security DHS projected that the department may be able to include estimates on the number of inadmissible travelers who successfully enter at a port of entry in its next Border Security Metrics Report to Congress. According to U.S. Customs and Border Protection (CBP) officials, they are in the process of reviewing data and program policies for CBP’s Compliance Measurement Examination program to determine if the program could be used as a means to reliably measure undetected inadmissible travelers, which would then be used in calculating the interdiction rate. Description This metric is a count of the amount in kilograms of illicit drugs seized by U.S. Customs and Border Protection officers at ports of entry. In an appendix to the report, the Department of Homeland Security listed out 34 different types of illicit drugs and the amounts seized for each for fiscal years 2007 through 2016. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations None identified. Additional information None. Description This metric is a rate that compares the amount of illicit drugs seized (in kilograms) by Office of Field Operations officials at ports of entry in 1 fiscal year to the average amount seized in the immediately preceding 5 fiscal years. the metric. The Department of Homeland Security provided rates for cocaine, methamphetamine, marijuana, and heroin for fiscal years 2012 through 2016. Limitations None identified. Additional information None. Description This metric is a count of the number of infractions related to travelers and cargo committed by major violators, and an estimate of the number of major infractions not interdicted. The Department of Homeland Security (DHS) calculated an infraction rate by dividing the number of major infractions by the total number of passengers at ports of entry for fiscal years 2007 through 2016. National Defense Authorization Act for Fiscal Year 2017 (NDAA). As an alternative, for the purpose of its report, DHS defined a major infraction as an arrest, including arrests related to terrorism, drugs, criminal aliens, and currency, among other things. DHS reported that it does not have a methodology in place to estimate the number of undetected major infractions. Therefore, only data on known infractions are included. DHS only included data for passenger infractions and not cargo-related infractions. Although not requested by the NDAA, DHS provided an infraction rate by dividing the number of known infractions by the total number of travelers at ports of entry. Limitations We identified: Given that DHS’s alternative approach to this metric involves using arrests as a proxy for major infractions, it is unclear whether there is a one-to-one correspondence between the arrest of a major violator and the number of infractions committed. Additional information According to U.S. Customs and Border Protection (CBP) officials, they plan to use data from CBP’s Compliance Measurement Examination program as a means to report estimated undetected major infractions starting with DHS’s fiscal year 2019 report. Description This metric is a rate that compares the amount of cocaine seized at land ports of entry to the total estimated flow of cocaine. the total flow of cocaine through land ports of entry. The Office of National Drug Control Policy produces annual estimates for total cocaine flow into the United States, but does not have a methodology to estimate the flow of cocaine through land ports of entry alone. Therefore, the estimates the Department of Homeland Security used included cocaine flow through all domains. According to the U.S. Drug Enforcement Administration’s National Drug Threat Assessment, the southwest border remains the key entry point for the majority of the cocaine entering the United States. Limitations None identified. Additional information None. Description This metric is a rate that compares the average wait time for vehicles to pass through a land port of entry to the total number of commercial and private vehicles at each land port of entry. data were not available for every port of entry, such as small ones with negligible wait times. Limitations We identified: We reported in July 2013 that commercial vehicle wait time data were unreliable due to inconsistent data collection processes at ports, and made two recommendations to DHS to improve the reliability of the data. While DHS implemented these recommendations in 2018, older data, including the data for the years presented in the report (fiscal years 2012 through 2016), remain unreliable. Additional information and planned actions by the Department of Homeland Security U.S. Customs and Border Protection (CBP) officials clarified that the wait times shown in the report reflect the average of all hourly recordings for wait times at ports of entry rather than the average passenger or vehicle experience because CBP did not report a volume-weighted measure of wait times. According to the report, CBP’s wait time policy is currently under review and new guidance will be issued in the future to account for improvements in automation and recording. Description This metric is a rate that measures traffic volume at land ports of entry against the physical and staffing capacity at each land port of entry. The Department of Homeland Security (DHS) calculated the average number of vehicles processed per booth, per hour at each land port of entry. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None. Limitations None identified. Additional information In addition to reporting utilization at each port of entry, DHS provided the average utilization rate for all northern border land ports of entry and all southern border land ports of entry. This metric is a rate that measures the frequency of secondary examinations at each land port of entry. The Department of Homeland Security (DHS) calculated the rate by dividing the recorded number of passengers sent for secondary inspection by the total number of recorded passengers at each land port of entry. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report DHS did not include data on secondary examinations of cargo or shipments. Limitations None identified. Additional information None. Description This metric is a count of the number of cargo containers at sea ports that DHS identified as potentially high-risk using National Targeting Center (NTC) security criteria. According to the Department of Homeland Security (DHS), all international cargo shipments coming to the United States are screened to identify potentially high-risk containers, which may then be reviewed, scanned, or physically inspected prior to lading at a port of entry. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations We identified: U.S. Customs and Border Protection (CBP) officials said that the process of defining and identifying “high-risk” shipments can exclude some shipments, such as those in free trade zones. Additional information DHS’s report said that the NTC periodically refines, improves, and revises the security criteria for high-risk shipments, which can affect the number of cargo shipments identified as high-risk. Description This metric is a rate comparing the number of potentially high-risk cargo containers scanned at each sea port of entry during a fiscal year to the total number of high-risk cargo containers that entered the United States at each sea port of entry during the previous fiscal year. separate from cargo containers that were reviewed or assessed; instead, DHS tracks these inspection methods collectively. Therefore, DHS also included data on potentially high-risk cargo containers that were reviewed or assessed as well as those that were scanned in its report. Limitations DHS identified: In its report, DHS noted that ratio data are not available for fiscal year 2014 because U.S. Customs and Border Protection did not collect comparable container-level data (as opposed to shipment-level data) in fiscal year 2013. DHS also noted that the totals across the ports or field offices may include duplicate container counts. We identified: NTC officials said that the definition of “high-risk” shipments excludes some shipments, such as those in free trade zones. NTC officials noted that assessing, reviewing, and scanning containers are different activities and reflect different levels of inspection or review. For example, NTC officials said that while all containers are “assessed” in order to determine their risk level, only higher risk containers may be scanned using radiation detection and nonintrusive inspection equipment. Consequently, when DHS included data on containers that were assessed or reviewed but not scanned, the resulting count was higher. In an appendix to its report, DHS presented a column of data called the “percentage of potentially high-risk containers scanned (same fiscal year)” for each fiscal year. Given DHS’s inability to separate data on the different inspection methods, the data in this column included containers that were reviewed by all inspection methods, not just scanning. In its appendix, DHS did not present data on the number of containers that “entered the United States,” even though it used those data to calculate the ratio and they are specified in the National Defense Authorization Act for Fiscal Year 2017. As a result, it is not possible to verify the accuracy of DHS’s ratio calculations. Additional information None. The U.S. maritime border domain encompasses ports, internal or inland waters, and coastal waters, as well as the territorial sea (waters 12 nautical miles seaward of the U.S. coast), contiguous zone (waters adjacent to and seaward of territorial sea and extending 24 nautical miles from shore), and exclusive economic zone (waters seaward of and adjacent to territorial sea and extending out to 200 nautical miles from shore). U.S. Coast Guard (Coast Guard), Air and Marine Operations, and U.S. Border Patrol share responsibility for patrolling the U.S. maritime borders, and territorial sea. The Coast Guard is a component of DHS and the lead federal maritime law enforcement agency on the high seas (waters beyond 12 nautical miles seaward of the U.S. coast) and all other waters under U.S. jurisdiction. The Coast Guard responds to a variety of maritime border security issues, including trafficking of narcotics, people, illicit goods, unlawful migration, illegal exploitation of natural resources, potential terrorist activities, and the disruption of maritime commerce. The metrics in this domain measure the number of migrants and illicit drugs removed, among other things. DHS included 4 of 6 metrics called for in the NDAA for this domain in its fiscal year 2017 Border Security Metrics Report, as shown in table 6. Description This metric was not specifically defined in the National Defense Authorization Act for Fiscal Year 2017 (NDAA). The NDAA described situational awareness as the knowledge and understanding of current unlawful cross- border activity, including (1) threats and trends concerning illicit trafficking and unlawful crossings, (2) the ability to forecast future shifts in such threats and trends, (3) the ability to evaluate such threats and trends at a level sufficient to create actionable plans, and (4) the operational capability to conduct persistent and integrated surveillance of the international borders of the United States. developing a measure for situational awareness in the maritime domain that meets the intent of the NDAA. While this effort is in process, DHS presented data on U.S. Coast Guard and U.S. Customs and Border Protection (CBP) asset (aircraft and cutter or boat) hours contributing to situational awareness or interdiction support and the number of vessel manifests screened. Limitations None identified. Additional information According to CBP Air and Marine Operations officials, they did not have confidence that the data for years prior to fiscal year 2016 were consistent enough for making comparisons across years. Consequently, only data for fiscal year 2016 were included in DHS’s report for the metrics related to CBP. Description This metric is a count of the total number of undocumented migrants interdicted, identified directly or indirectly but not interdicted, or otherwise believed to have unlawfully entered the United States through the maritime border. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations Department of Homeland Security (DHS) identified: The accuracy of migrant flow counts depends on partners to report interdictions and the ability to detect migrants. According to the DHS report, the U.S. Coast Guard relies on partners to report interdictions for compilation in the U.S. Coast Guard’s database. Interdictions may be double-counted by the U.S. Coast Guard and its partners because they cooperate during operations and some interdictions by partners may not get reported. Further, some migrants may not be apprehended and leave no evidence, and are therefore excluded from the known flow figures. We identified: According to U.S. Coast Guard officials, there is no centralized database for tracking migrant interdictions, and the decentralized nature of the data collection could lead to errors. Additional information According to the U.S. Coast Guard, about 90 percent of the data on migrant interdictions and flow originate from U.S. Coast Guard records. U.S. Coast Guard officials said that as part of a department-wide initiative to standardize illegal immigration statistics, they are in the preliminary stages of building a centralized database to enter and maintain information on migrant interdictions. Additionally, officials said they take steps to ensure the reliability of externally reported data such as communicating with partners and working together to reconcile any errors. Within the U.S. Coast Guard, meetings are held regularly to discuss and vet the accuracy of migrant flow data. Description This metric is a rate comparing the amount and type of illicit drugs removed by the Department of Homeland Security (DHS) maritime security components in any fiscal year, including drugs abandoned at sea, to the average amount removed or abandoned in the immediately preceding 5 fiscal years. by all DHS maritime security components, but DHS only provided data on removals by the U.S. Coast Guard. DHS did not explain in its report why it only included data from the U.S. Coast Guard. DHS officials said that the U.S. Coast Guard is the primary DHS component involved in this activity and was the only component that provided data for this metric, but this was not noted in the report. According to U.S. Coast Guard officials, some of the data for fiscal 2013 was misreported. Specifically, the quantity removed for methamphetamine should be 0 (report shows 17.4) while the value should be 7.9 kilograms for heroin (report shows 0). Additional information None. Description This metric is a rate comparing the amount of cocaine removed by the Department of Homeland Security (DHS) maritime security components inside and outside the maritime transit zone to the total documented cocaine flow rate. DHS used estimates of noncommercial maritime cocaine flow from the Consolidated Counter Drug Database, which are derived from intelligence reporting and case data. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations DHS identified: There is less robust intelligence on noncommercial maritime cocaine flow outside the transit zone than inside the transit zone, so data for outside the transit zone are not considered reliable. Precise cocaine flow estimates through a particular mode or domain can be difficult to obtain. In our prior work, officials with the Office of National Drug Control Policy and other departments and agencies involved in U.S. counternarcotics efforts told us that it is difficult to obtain precise estimates of cocaine flow because of the difficulty in obtaining specific information about the production of cocaine and how it gets to the United States. We have also previously reported that when confronted with threats to their activities, drug-trafficking organizations use a variety of techniques to quickly change their modes of operation, thus avoiding capture of their personnel and seizure of their illegal drugs. For example, when air interdiction efforts have proven successful, traffickers have increased their use of maritime and overland transportation routes. Additional information According to U.S. Coast Guard officials, DHS officials hold quarterly inter-agency meetings to review the reliability of performance data related to cocaine interdiction performance. Air and Marine Operations (AMO) is a federal law enforcement agency within CBP that interdicts unlawful people and cargo approaching U.S. borders, investigates criminal networks, and provides domain awareness in the air and maritime environments, among other things. The metrics in this domain measure AMO’s flight hours, individuals detected, and apprehensions, among other things. DHS included 7 of 8 metrics within this domain called for in the NDAA in its fiscal year 2017 Border Security Metrics Report, as shown in table 7. DHS reported that the “AMO actionable intelligence” metric was under review and estimated that the department would provide information on this metric in its 2019 annual report to Congress. Description This metric is a rate comparing the number of flight hour requirements to the number of flight hours flown by Air and Marine Operations (AMO) in the land domain. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations Department of Homeland Security (DHS) identified: Data prior to fiscal year 2016 were unavailable. According to AMO officials, this is because AMO did not collect these data prior to fiscal year 2016, or because older data were not comparable. We identified: DHS used the terms “funded flight hours,” “unfunded flight hours,” and “unconstrained flight hours” in the report without clearly defining them. AMO officials stated that a definition of these terms will be included in the next report. Additional information AMO officials said they have taken steps to improve how they track flight hour data, such as by adding new data fields to AMO’s system and providing training to staff. Description This metric is a rate comparing the number of funded flight hours appropriated to Air and Marine Operations (AMO) to the number of actual flight hours flown. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations Department of Homeland Security (DHS) identified: Data prior to fiscal year 2016 were unavailable. According to AMO officials, this is because AMO did not collect these data prior to fiscal year 2016, or because older data were not comparable. Additional information AMO officials said they have taken steps to improve how they track flight hour data, such as by adding new data fields to AMO’s system and providing training to staff. Description This metric is a rate comparing the number of aviation missions flown by Air and Marine Operations (AMO) to the number of aviation missions cancelled by AMO due to maintenance, operations, or other causes. the number of missions cancelled due to causes within AMO control, such as maintenance, personnel, and asset availability. However, the Department of Homeland Security (DHS) used the total number of mission requests, which also includes the number of missions flown in addition to the number of missions cancelled for reasons within AMO control. Limitations DHS identified: Data prior to fiscal year 2016 were unavailable. According to AMO officials, this is because AMO did not collect these data prior to fiscal year 2016, or because older data were not comparable. Additional information AMO officials said they have taken steps to improve how they track flight hour data, such as by adding new data fields to AMO’s system and providing training to staff. Description This metric is a rate comparing the number of missions cancelled by Air and Marine Operations (AMO) due to weather compared to the total planned missions. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations Department of Homeland Security (DHS) identified: Data prior to fiscal year 2016 were unavailable. According to AMO officials, this is because AMO did not collect these data prior to fiscal year 2016, or because older data were not comparable. Additional information AMO officials said they have taken steps to improve how they track flight hour data, such as by adding new data fields to AMO’s system and providing training to staff. Description This metric is a count of the number of individuals detected by Air and Marine Operations (AMO) through the use of unmanned aerial systems and manned aircraft. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations Department of Homeland Security (DHS) identified: Data prior to fiscal year 2016 were unavailable. According to AMO officials, this is because AMO did not collect these data prior to fiscal year 2016, or because older data were not comparable. DHS data on detections from manned aircraft were limited to those that led to apprehensions and arrests, and data from unmanned aircraft were limited to the number of Vehicle and Dismount Exploitation Radar (VADER) detections. AMO did not track data from all sensors on unmanned and manned aircraft, and considers this metric to be a work in progress. We identified: In February 2017 we reported that some mission data (such as asset assists) for unmanned aerial systems were collected inconsistently across operation locations, which could affect the accuracy of the counts provided. We recommended that U.S. Customs and Border Protection—of which AMO is a component—update and maintain guidance for recording mission information in its data collection system, and provide training to users of the system. DHS completed implementation of these recommendations in July 2018. Although the recommendations have been implemented, this limitation is relevant because the data presented (for fiscal year 2016) were collected prior to their implementation. Additional information and planned actions by the Department of Homeland Security DHS expects to provide more comprehensive data for this metric in the next annual report. AMO officials said they have taken steps to improve how they track flight hour data, such as by adding new data fields to AMO’s system and providing training to staff. Description This metric is a count of the number of apprehensions assisted by Air and Marine Operations (AMO) through the use of unmanned aerial systems and manned aircraft. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None. Limitations Department of Homeland Security (DHS) identified: Data prior to fiscal year 2016 were unavailable. According to AMO officials, this is because AMO did not collect these data prior to fiscal year 2016, or because older data were not comparable. We identified: In February 2017 we reported that some mission data (such as asset assists) for unmanned aerial systems were collected inconsistently across operation locations, which could affect the accuracy of the counts provided. We recommended that U.S. Customs and Border Protection—of which AMO is a component—update and maintain guidance for recording mission information in its data collection system, and provide training to users of the system. DHS completed implementation of these recommendations in July 2018. Although the recommendations have been implemented, this limitation is relevant because the data presented (for fiscal year 2016) were collected prior to their implementation. Additional information In addition to the number of apprehensions assisted, DHS also provided the number of enforcement flight hours used for the assists. AMO officials said they have taken steps to improve how they track flight hour data, such as by adding new data fields to AMO’s system and providing training to staff. Description This metric is a count of the number and quantity of illicit drug seizures assisted by Air and Marine Operations (AMO) through the use of unmanned aerial systems and manned aircraft. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None. Limitations Department of Homeland Security (DHS) identified: Data prior to fiscal year 2016 were unavailable. According to AMO officials, this is because AMO did not collect these data prior to fiscal year 2016, or because older data were not comparable. We identified: In February 2017 we reported that some mission data (such as asset assists) for unmanned aerial systems were collected inconsistently across operation locations, which could affect the accuracy of the counts provided. We recommended that U.S. Customs and Border Protection—of which AMO is a component—update and maintain guidance for recording mission information in its data collection system, and providing training to users of the system. DHS completed implementation of these recommendations in July 2018. Although the recommendations have been implemented, this limitation is relevant because the data presented (for fiscal year 2016) were collected prior to their implementation. Additional information In addition to the drug seizures assisted (in pounds), DHS also provided the number of enforcement flight hours used for the assists. AMO officials said they have taken steps to improve how they track flight hour data, such as by adding new data fields to AMO’s system and providing training to staff. Based on findings from previous reviews of border security programs and efforts, we have recommended other metrics that the Department of Homeland Security (DHS) could use to help measure the effectiveness of border security. The tables that follow provide information about these recommended metrics in each of the four domains listed in the National Defense Authorization Act for Fiscal Year 2017—between ports of entry, at ports of entry, in the maritime border domain, and the air and marine security in the land domain. The recommendations listed in the tables below remain open; however, implementing them would provide DHS with additional indicators and metrics that could provide important insights into the state of border security. In addition to the contact named above, Taylor Matheson (Assistant Director), David Alexander, Kelsey Burdick, Lilia Chaidez, Kathleen Donovan, Michele Fejfar, Sally Gilley, Christopher Hatscher, Eric Hauswirth, Mikaela Meyer, Sasan J. “Jon” Najmi, Kevin Reeves, and Jeff Tessin made key contributions to this report.
According to DHS, the United States has approximately 6,000 miles of land borders, 95,000 miles of coastline, and more than 300 ports of entry where travelers and cargo are inspected and processed for entry. Securing U.S. border areas is a key part of DHS's mission, and the department's ability to measure its border security efforts is essential for it to manage its responsibilities effectively and efficiently. The NDAA for Fiscal Year 2017 requires DHS to report annually on 43 border security metrics. DHS issued its first report in May 2018. The Act also includes a provision for GAO, within 270 days of receipt and biennially for the following 10 years, to review and report on the data and methodology contained in DHS's report. This report assesses the extent to which DHS: (1) reported metrics as outlined in the NDAA using quality information; and (2) validated assumptions and conveyed statistical uncertainty for unlawful entry metrics, among other objectives. GAO assessed the methodology and data in DHS's report, analyzed DHS's use of statistical models, and interviewed officials from DHS offices and components involved in developing the metrics. The Department of Homeland Security (DHS) reported on 35 of 43 metrics called for by the National Defense Authorization Act (NDAA) for Fiscal Year 2017 (see figure); it generally used quality information, but did not identify some data limitations. GAO found that about half of the 35 metrics generally included elements as called for by the NDAA, while 17 metrics differed, such as in scope or calculation. For example, DHS only provided information on the southwest border for some metrics, such as the estimate of undetected unlawful border crossers for which a methodology for estimating unlawful crossings for the northern border had not yet been completed. DHS components responsible for collecting the metric data generally have processes in place to ensure the reliability of the data and the quality of the information provided. DHS also identified and disclosed limitations for some, but not all, of the data elements and metrics used. For example, GAO found that DHS did not disclose limitations on data related to apprehensions of individuals that were assisted by unmanned aerial systems. By developing and implementing a process to systematically review the reliability of the data and comprehensively identify and communicate limitations, DHS would improve the quality of the information provided. DHS used a statistical model to estimate three metrics on unlawful border entries but did not validate some assumptions the model employs through sensitivity analyses and provide measures of statistical uncertainty in accordance with standards for federal agencies. For example, DHS's model assumes that 100 percent of families unlawfully crossing the border will be apprehended, but DHS did not provide information on the extent to which the assumption affected its metrics. DHS also did not provide information on the level of statistical uncertainty for the metrics, such as margins of error. Providing such information would allow Congress and the public to better understand the potential limitations and accuracy of these metrics of unlawful entry. Additionally, DHS's statistical model, which is based on Mexican adults not seeking asylum, represents a small and declining share of those apprehended at the border and DHS is developing a new model to account for current border conditions. GAO is making four recommendations, including that DHS develop and implement a process to systematically review the reliability of metric data, identify and communicate limitations of the metrics, and include the results of sensitivity analyses and measures of statistical uncertainty for metrics derived from statistical models. DHS concurred with the recommendations.