Examples of research-related resources include the Regional  Educational Laboratories (REL) and the Doing What Works (DWW) Web  site (http://dww.ed.gov):    Regional Educational Laboratories. One of the goals of the Clearinghouse was to promote  informed education decision making through a Web-based dissemination  system that featured rigorous reviews of studies on the effectiveness of  educational interventions. WWC Reviews Research in Accordance with Accepted Standards and Has Responded to Recommendations and Criticisms

		WWC Follows Accepted Review Standards and Is Improving Its Review Process in Response to a Congressionally Mandated Expert Panel Report

We found that the WWC review process follows generally accepted  practices for reviewing research. For example, in response to the panel’s  recommendation that the WWC include a table of study dispositions (e.g.,  whether studies meet WWC evidence standards) at the front of  intervention reports, the WWC is modifying the report template to include  a summary table along with the existing listing of dispositions in the  reference section. Further, in response to the panel’s concern that  the WWC’s screening process may exclude some eligible studies, the WWC  is undertaking an evaluation of the reliability of its screening process. The WWC  created new standards to include additional study designs. WWC’s Output and Costs Increased; However, IES Has Not Developed Adequate Performance Measures Related to Cost or Product Usefulness

		WWC Increased Output and Introduced New Products

WWC’s report output increased under the current contract, and its scope  expanded to include new products and processes to support production. For example, the current contract calls for  the WWC to increase the number of topic areas and intervention reports. Education Has Three Primary Ways to Disseminate Information about WWC Products, but Awareness and Use Vary among Target Audiences

		Education Has Various Ways to Disseminate Information about WWC Products, but Awareness of the Clearinghouse Is Generally Limited

Education uses the WWC contractor, RELs, and DWW to disseminate  information about the Clearinghouse to its target audience, which includes  state and school district officials, as well as teachers and principals. RELs also disseminate research from WWC when  responding to educator questions or concerns. We found that 33 of the 38 states that responded to our survey reported  that they had heard of the WWC. States and School Districts Generally Used the Clearinghouse to a Small or Moderate Extent to Inform Decisions and Used Specific WWC Products to Varying Extents

Based on our survey, most states and school districts that reported  accessing the WWC Web site used it to inform decisions on effective  education practices—a stated purpose of the WWC—to a small or  moderate extent. An estimated 50 percent of school  districts would likely increase their use of the Clearinghouse if it had  reviews that were more timely (see fig. Improved dissemination of WWC  products could increase awareness and use of the WWC. To better track the costs and usefulness of the WWC, we recommend that  the Secretary of Education direct IES to  incorporate findings from its cost studies to develop performance  measures related to costs, such as identifying a range of acceptable costs  per product and using that information to monitor contractor spending;  and    develop performance measures related to product usefulness and  periodically assess whether WWC products are meeting the needs of target  audiences by gathering information on product usefulness in the proposed  survey or through other means. Agency Comments and Our Evaluation

We provided a draft of this report to the U.S. Department of Education for  review and comment. Education generally agreed with our recommendations. To determine how  performance and costs changed over time, we analyzed the costs and  productivity of the two WWC contractors. To obtain information about the  usefulness of WWC products, we conducted a Web-based survey of all  state education agencies and a nationally representative sample of school  districts. Assessment of WWC Research Review Process

GAO previously assessed the procedures and criteria used by the WWC by  reviewing documents and interviewing IES officials and WWC  contractors. Specifically,  the survey asked officials about (1) their general sources of information on  effective educational practices, (2) the extent to which they use WWC  products to inform curriculum decisions (including questions on specific  intervention reports and practice guides), (3) how useful the officials find  the information in the WWC, (4) the likelihood they would increase their  usage if certain changes were made to the WWC Web site, and (5) the  extent to which the officials use the Doing What Works and Best Evidence  Encyclopedia Web sites to inform curriculum decisions and how useful  the officials find these other information sources to be.
Why GAO Did This Study

In connection with the Omnibus Appropriations Act, 2009, GAO was required to study the What Works Clearinghouse (WWC), a federal source of evidence about effective education practices. Operating through a 5-year contract awarded by the U.S. Department of Education's Institute of Education Sciences (IES), the WWC reviews education research and disseminates its findings. GAO examined: (1) the extent to which the WWC review process meets accepted standards for research evaluation and how the WWC has responded to recommendations and criticism, (2) how WWC output and costs have changed over time and how its performance is measured, and (3) how WWC products are disseminated and how useful educators find them to be. To conduct its work, GAO reviewed WWC-related documents, analyzed cost and performance data, surveyed all states and a nationally representative sample of school districts, and interviewed IES officials, WWC contractors, researchers, and others.

What GAO Found

GAO as well as a congressionallymandated panel of experts, found that the WWC's review process, which includes screening studies to determine if they meet WWC criteria, follows accepted standards for evaluating research on the effectiveness of education interventions. WWC is responding to recommendations made by the expert panel to further improve its review and reporting processes. For example, the panel recommended improvements in the way the WWC presents information to readers on the reasons why studies do not qualify for review. The WWC is revising a report template to include a table summarizing which studies met or did not meet WWC criteria for evaluating research. The WWC has also responded to researchers who have criticized the WWC for presenting limited information because its screening criteria exclude some rigorous research designs that may be appropriate for evaluating certain education programs, such as special education. The WWC responded to this criticism by creating new standards that include two additional study designs and by creating a new product, called a practice guide, which includes a wider range of research. WWC's report output and scope increased under the current contract. For example, the WWC increased its production of various reports, introduced new products, and developed new processes for evaluating research. However, IES had a substantial backlog in its product review process from January 2009 to May 2010. The backlog generally decreased the timeliness of WWC reports, with 20 reports being delayed by up to 6 months. To support the increases in output and scope, WWC's costs doubled from the previous contract to the current one. Both contracts designated about 60 percent of costs to production, while the other 40 percent of costs support other tasks, such as communications, dissemination, and process development. IES' performance goals for the WWC primarily relate to the number of reports produced. However, IES has not developed performance measures related to the cost or usefulness of WWC products. Education uses WWC contractors, Regional Educational Laboratories (RELS) and the Doing What Works (DWW) Web site to disseminate information about WWC products; however, awareness and use of the WWC varies among states, districts, teachers, and principals. WWC contractors disseminate product information in various ways including email alerts and presentations. The RELs host events featuring WWC products for state, district, and school officials and DWW provides resources to educators based on WWC products. Based on our survey, officials from 33 of 38 state education agencies that responded to our survey and an estimated 42 percent of school districts have heard of the WWC. Those states and school districts generally used the WWC to a small or moderate extent to inform decisions on effective practices. Based on our survey, states and school districts reported that they would likely increase their use of the WWC if it included a broader array of information or more timely information.

What GAO Recommends

GAO recommends that IES: develop and implement strategies to avoid backlogs in WWC product reviews; establish performance measures related to costs and usefulness; and improve dissemination efforts to promote awareness and use of the WWC. Education generally agreed with GAO's recommendations.