Reengineered Address Canvassing Was Designed to Substantially Reduce In-Field Canvassing

To achieve cost savings, the Bureau decided in September 2014 that  while it would still verify all the addresses in the country, not all of this  canvassing would take place by going door-to-door (or “in-field”), as it had  in prior decennials. Rather, some areas might only need a review of their  address and map information using computers and what the Bureau  refers to as “in-office” procedures. The Bureau Lacks Information on Implications of Reengineered Design on Cost and Quality of Address Canvassing

		The Bureau Has Not Completed Evaluations of 2016 Tests as Scheduled

The Bureau has not completed evaluations of results from tests and  activities it carried out in 2016 to assess the effectiveness of in-office  address canvassing. In addition to the cancellation of the 2017 Master Address File Coverage  Study, the cancellation of the field work components of the 2017 Census  Test, which the Bureau similarly attributed to budgetary constraints it  faced, represents a lost opportunity to collect data that could have been  used to evaluate the quality of in-office address canvassing. As a result, the Bureau has had limited information to  demonstrate the cost and quality of in-office address canvassing along  the way and has made major design decisions, such as the suspension of  the second phase of in-office address canvassing, without the benefit of  information from testing. We have previously reported on the need for the Bureau to rethink its  approach to, among other things, testing and evaluating the census. The  Bureau agreed, and its early plans for the 2020 Census called for the use  of incremental, small-scale testing throughout the decade. However, as the Bureau begins planning for future census operations in the coming years, it will be important to commit to an  approach of small-scale, more frequent testing that provides real-time,  actionable data to inform decisions. The Bureau’s Measures of In-Office Address Canvassing Do Not Reflect the Effect on Reducing Fieldwork

		Bureau’s Measures Do Not Reflect Total Completion of Workload

The Bureau includes work that has been put on hold in its measure  of completed workload. In measuring the day-to-day process of the in- office address canvassing operation, the Bureau regularly tracks the  number of blocks for which it has conducted an initial round of review. However, the workload  the Bureau identifies as completed as part of this first pass includes  blocks that have been put on hold. For example, this may  occur if imagery was blocked by cloud cover. The Bureau has additional work to update previously-reviewed  blocks. The Bureau  had initially planned on resolving 75 percent of housing units in-office, but  as of March 2017 the blocks that the Bureau had successfully removed  from fieldwork account for 51 percent of all housing units. The Bureau suspended  the second phase of in-office address canvassing without detailed data  on the cost, value, or accuracy of the operation itself, citing budget  uncertainty. Given that the Bureau will need to make consequential  decisions about the scope and nature of its fieldwork for 2020, it will be  important for the Bureau to make use of information on the effectiveness  of its reengineered operation, in terms of both the cost and quality of  constructing the address list, so that it can justify these decisions and  achieve desired outcomes. 2. Early in the next decennial cycle, plan and execute more flexible, and  perhaps smaller, address canvassing test and evaluation activity  needed to support key design decisions having significant effect on  the cost and quality of the census. 3. Use productivity measures that track the progress in completing in- office address canvassing workload and the effectiveness of in-office  address canvassing in reducing fieldwork in order to make informed  decisions on allocating resources to current and future address  canvassing workload so that the operation is completed in a timely  and cost-effective manner. In its  written comments, reproduced in appendix III, the Department of  Commerce had no disagreements with GAO’s findings and  recommendations. In addition, the Secretary noted that the Department  was conducting its own review. Appendix I: Objectives, Scope, and Methodology

You asked us to evaluate the U.S. Census Bureau’s (Bureau)  reengineered approach to 2020 Address Canvassing. We (1) described  the Bureau’s design for its reengineered 2020 Census address  canvassing, (2) evaluated the extent to which the Bureau assessed the  cost and quality implications of its reengineered address canvassing  approach, and (3) assessed the status of the Bureau’s efforts to reduce  the in-field address canvassing workload. For the second objective we interviewed officials in the Bureau’s  Geography and Decennial Census Management Divisions and reviewed  relevant documentation to determine how the Bureau was managing  address canvassing testing, assessing cost and quality implications,  making decisions regarding the design of the reengineered address  canvassing approach, and managing related risks.
Why GAO Did This Study

With a lifecycle cost of $12.3 billion (in 2020 dollars), the 2010 Census was the most expensive in U.S. history. Reengineering how the Bureau builds its address list is one of the ways the Bureau intends to reduce the per–housing unit cost of the 2020 count.
GAO was asked to evaluate the Bureau's reengineered approach for 2020 address canvassing. This report (1) describes the Bureau's design for 2020 address canvassing, (2) evaluates the extent to which the Bureau assessed the cost and quality implications of its reengineered address canvassing approach, and (3) assesses the status of the Bureau's efforts to reduce the in-field address canvassing workload. GAO reviewed relevant design and testing documentation and interviewed cognizant Bureau officials. GAO also reviewed Bureau address canvassing production and payroll data in December 2016 and March 2017.

What GAO Found

To achieve cost savings, the Census Bureau (Bureau) decided in September 2014 to use a reengineered approach for building its address list for the 2020 Census without having to go door-to-door (or “in-field”) across the country, as it has in prior decennial censuses. Rather, some areas (known as “blocks”) might only need a review of their address and map information using computer imagery and third-party data sources – what the Bureau calls “in-office” procedures. The Bureau planned to use a two-phase, in-office address canvassing operation to update 75 percent of all housing units in its master address list, and initially estimated this approach could reduce 2020 Census costs by $900 million.
The Bureau has not completed evaluations of results from tests and activities it carried out in 2016 to assess the effectiveness of in-office address canvassing. Specifically, the Bureau has not completed its evaluations designed to evaluate the accuracy of in-office address canvassing. Similarly, citing budget constraints, the Bureau also cancelled 2017 fieldwork that could have provided additional data to evaluate the reengineered approach. Further, the Bureau has changed the design of the in-office address canvasing operation with limited information on the cost and quality implications of the changes. Citing budget uncertainty, in March 2017 the Bureau suspended its second phase of in-office address canvassing, which was designed to resolve address coverage issues identified in the first phase. Bureau officials indicated that this decision would increase the in-field address canvassing workload from 25 percent to at least 30 percent, but they could not provide details of the cost or quality tradeoffs of this decision. Using its remaining evaluations to determine cost and quality implications would better position the Bureau to justify future decisions for its reengineered approach.
GAO has previously reported on the need for the Bureau to rethink its approach to testing and evaluating the census. The Bureau agreed, and its early plans called for the use of incremental, small-scale testing throughout the decade. As the Bureau begins planning for future census operations, small-scale testing by the Bureau may be less susceptible to budgetary constraints and allow it to better demonstrate the effectiveness of its operational innovations.
The Bureau manages the day-to-day production of in-office address canvassing using measures that do not fully reflect the effectiveness of the approach in reducing fieldwork. For example, the Bureau regularly tracks the number of blocks for which it has conducted an initial round of review. However, the workload the Bureau identifies as completed includes blocks that have been put on hold and that await availability of better information before being reviewed again. Other assumptions about how quickly the workflow proceeds have not been met. In testing the Bureau's productivity assumptions using actual production data through March 2017, GAO found that even small differences in measurement can have significant impacts on estimates of the remaining workload and the time to complete it. Without measuring how much in-office work is truly completed and how much fieldwork has been reduced, the Bureau risks underestimating the time and resources needed to complete the in-office operation, and may require more in-field address canvassing—at greater costs—than the Bureau currently projects.

What GAO Recommends

GAO recommends that the Bureau (1) use its remaining evaluations before 2020 to determine cost and quality implications and to justify future decisions for its reengineered approach; (2) plan and execute smaller, more flexible tests needed to support key address canvassing design decisions in future census operations; and (3) use productivity measures that track the progress of the in-office address canvassing and its effectiveness in reducing fieldwork. The Department of Commerce had no disagreements with GAO's findings and recommendations. In addition, the Secretary noted that the Department was conducting its own review.