Statutory Authority

The program was created under Title IV of the Energy Conservation and Production Act of 1976 ( P.L. Table A -1 shows that the accumulated congressional appropriations for the 32-year period from FY1977 through FY2008 reached a sum of nearly $8.7 billion. For FY2001, Congress approved about $187 million (in 2010 dollars), which was just about $1 million less than was requested by the outgoing Clinton Administration. As with other state and local activities, recession-driven budget shortfalls and staff furloughs delayed many required state training initiatives. In early 2010, the IG found that the nation had not realized the potential economic benefits of the $5 billion in Recovery Act funds allocated to the program. Performance Assessments and Evaluation Studies of Cost-Effectiveness

This part of the report reviews a selection of key studies that examined Weatherization Program management processes, performance, and economic (energy and non-energy) impacts. For a more in-depth examination of program operations, DOE has a history of directing the Oak Ridge National Laboratory (ORNL) to conduct evaluation impact studies. From late 2002 through early 2003, DOE's Office of the Inspector General (IG) conducted a performance audit of the program. The IG observed that the program has a long-established structure for transferring funds to state and local agencies and that improvements over the years had made use of the funds more efficient and effective. A recent OMB program assessment found that DOE had addressed the IG's concern about distorted efficiency (benefit-cost) measures:
Average cost per home employed in the calculation of benefit/cost ratios now reflects total costs (including non-DOE funds) expended per unit in the states whose evaluations were used in the energy savings estimates. 2011 GAO Performance Audit of Recovery Act Funding

The Government Accountability Office (GAO) defines a performance audit broadly:
Performance audits provide objective analysis so that management and those charged with governance and oversight can use the information to improve program performance and operations, reduce costs, facilitate decision making by parties with responsibility to oversee or initiate corrective action, and contribute to public accountability. The Recovery Act directed GAO to conduct bimonthly reviews and reporting on selected states and localities' use of the funds it provided, including the funds for the DOE Weatherization Program. For example, in December 2009, GAO published a performance audit report on Recovery Act funds used by states and localities. GAO made several observations about the status of program implementation and offered some recommendations for program improvement. Selected findings, as of the end of FY2011, included: (1) recipients had spent about $3.46 billion (73%) of the $4.75 billion allocated, (2) recipients reported that implementation challenges were declining, (3) the average cost per home was about $4,900, about 563,000 homes had been weatherized, and DOE projected it would exceed the original target of 607,000 homes by the deadline set for the end of March 2012; and (4) GAO found that the quality of employed data reported by recipients had improved. In particular, a comprehensive evaluation of the PY1989 program was published in 1993. The program perspective compared energy benefits to total program costs. This metaevaluation was based on data from 38 evaluation studies of weatherization programs in 19 states, published between 1993 and 2005. Current Plans for Assessment and Evaluation Studies

Three activities are underway to assess and evaluate DOE's Weatherization program: a DOE (Deloitte) strategic assessment, a DOE (APPRISE) impact study for PY2007 through PY2008, and a DOE (APPRISE) impact evaluation for the Recovery Act period of PY2009 through PY2011. Performance measures will include program costs and benefits. Evaluation Plan for Recovery Act (APPRISE, 2009-2011 Data)

The Recovery Act committed $5 billion over two years to an expanded weatherization program. This large sum greatly heightened interest in the program, the population served, its energy and cost savings, and its cost-effectiveness. First, Reingold argues that an agency's control over evaluation funding allows it to retain a strong influence over evaluation content:
[W]hen purchasing evaluation services via contracts, the purchasing agency has total control over all aspects of the services to be provided ... [thus] ... [i]t is inconsistent to argue that contracted program evaluation can be independent when the agency purchasing the evaluation owns (or controls) the research question and design, the method of data collection, the strategy of analysis, the data, the final report, and the rules governing the dissemination of results.
This report analyzes the Department of Energy's (DOE's) Weatherization Assistance Program. (WAP, the "program"). It provides background—a brief history of funding, program evolution, and program activity—and a review of program assessments and benefit-cost evaluations.
Budget debate over the program is focused on a $5 billion appropriation in the Recovery Act of 2009, a report that state and local governments have yet to commit about $1.5 billion of that total, and concerns about the quality of weatherization projects implemented with Recovery Act funding. During the debate over FY2011 funding, the House Republican Study Committee called for program funding to be eliminated. In April 2011, Congress approved $171 million for the program in the final continuing resolution for FY2011 (P.L. 112-10). For FY2012, DOE requested $320 million, the House approved $30 million, the Senate Appropriations Committee recommended $171 million, and the Conference Committee approved $68 million. The budget debate provides the context for this report, but details of the current debate are beyond the scope of this report, which is focused on evaluations of program cost-effectiveness.
WAP is a formula grant program: funding flows from DOE to state governments and then to local governments and weatherization agencies. Over the 32 years from the program's start-up in FY1977 through FY2008, Congress appropriated about $8.7 billion (in constant FY2010 dollars). The $5 billion provided by the Recovery Act added more than 50% to the previous spending total.
Over the program's history, DOE's Oak Ridge National Laboratory (ORNL) and the Office of Management and Budget have used process and impact evaluation research methods to assess WAP operations and estimate cost-effectiveness. Virtually all the studies conducted through 2005 showed that the program was moderately cost effective. The studies included measures of operational effectiveness, energy savings, and non-energy benefits. The timing of past studies was a bit sporadic, driven mainly by new statutory requirements and program audits. Performance assessments have alternately identified improvements in program operations or identified operational problems that subsequently stimulated program improvements. Only the intensive evaluation study of program year 1989 (published in 1993) was designed to directly draw a national sample to produce empirical data on program cost-effectiveness. Such in-depth evaluations are costly and time-consuming. Most other "metaevaluations" were much less costly, using available state-level evaluation studies as the basis to infer national-level program impacts.
The large infusion of Recovery Act funding—and attendant changes in program structure—heightened interest in conducting a fresh assessment of operations and new scientifically based evaluations of program impacts. Unforeseen recession-driven events delayed use of Recovery Act funding. For example, the DOE Inspector General (DOE IG) found that recession-driven budget shortfalls, state hiring freezes, and state-wide planned furloughs delayed program implementation—and created barriers to meeting spending and home weatherization targets. In December 2011, the Government Accountability Office (GAO) released a performance audit which found that the Recovery Act phase of the program was successfully addressing most goals and the challenges identified by the DOE IG. Recently-launched evaluation studies by DOE aim to determine whether Recovery Act funding was used cost-effectively and whether it fulfilled goals for job creation.
The report concludes by reviewing the debate over the use of "outside" contractors to improve the objectivity and independence of weatherization program evaluations.