Resources. 2.) ETA’s Research Areas Generally Reflect Key Issues, but Some Studies Are of Limited Usefulness

		Experts Thought ETA’s 2007 to 2012 Research Plan Reflected Key Areas, but They Also Suggested New Ones for Future Research

Our expert panel generally considered ETA’s research areas to be the right  ones for the period the research plan covered. Of the estimated $96 million that supported the 58  research reports published between January 2008 and March 2010, more  than half—about $56 million—funded research that addressed these two  research areas. Four of these reports were designed to demonstrate what works and for  whom. In several studies we examined that cost $1 million or more, we found  that, for a number of reasons, ETA’s research studies were limited in their  usefulness and in their ability to inform policy and practices. For example,  in a study of the Prisoner Re-entry Initiative, shortcomings in the data  collection phase limited the strength of the findings and, as a result,  limited the study’s opportunity to influence policy directions. Members of our expert panel stressed the importance of ETA  incorporating varied methodological approaches into its future research  proposals to best position the agency to address key employment and  training issues. In 2010, acknowledging the need for  better and more rigorous evaluations to inform its policy, Labor  established the Chief Evaluation Office to oversee the department’s  research and evaluation efforts. Moreover, ETA’s process is missing some critical elements that are needed  to ensure that the current improvements become routine practices. ETA’s process lacks a  formal provision requiring consultation with the newly established  Chief Evaluation Officer at important points in the research process. Beyond ETA’s process for conducting research, current research practices  fall short of ensuring research transparency and accountability—essential  elements of a sound research and evaluation program. In our previous report, we noted that ETA’s research and evaluation  center lacked a specific mechanism to insulate it from undue influence. ETA Has Recently Included More Random Assignment Studies in Its Research Program

ETA has recently begun to include more rigorous studies in its ongoing  research. Some of ETA’s ongoing research studies face challenges in recruiting  sample sizes large enough to meet the studies’ objectives. ETA Has Improved the Availability of Its Research but There Are Opportunities to Improve Its Search Page and Dissemination Methods

		ETA Has Improved the Timeliness of Its Disseminated Research

ETA has recently improved the timeliness with which it disseminates its  research reports. ETA Has Improved Its Research Database but Lacks Plans for Assessing the Usability of Its Search Page

In 2010, ETA updated its online, Web-based search page in order to  improve the usability of its research database—the primary tool for  making ETA research available to policymakers and the general public. Labor officials told us  they have taken other steps, as well, in efforts to improve its Web-based  search page. Although these changes have the potential to improve the usability of  ETA’s database, Labor has not developed a formal plan for assessing the  overall effectiveness of its Web-based search page, including user  satisfaction. Beyond posting reports to  its database, ETA also distributes hard copies of some of its research  reports. In addition to electronic distribution, ETA also organizes various  presentations to disseminate its research findings. Recommendations for Executive Action

To improve ETA’s research program, we recommend that the Secretary of  Labor require ETA to take the following three actions:    Formally incorporate into its research process the routine involvement  of the Chief Evaluation Officer at key milestones, including at the  development of ETA’s annual research agenda and spending priorities,  as well as at the early stages of developing specific research projects. Regarding our recommendation for ETA to develop a mechanism to  enhance the transparency and accountability of its research program,  officials cited several steps they are taking to improve the program,  including involving outside experts in the development of their 5-year  research plan and establishing advisory and peer review groups to  review major evaluations. Labor agrees with this recommendation. Specifically, we answered the following  research questions: (1) To what extent do ETA’s research priorities reflect  key national employment and training issues and how useful were the  studies funded under them? (2) What steps has ETA taken to improve its  research program? (3) How has ETA improved, if at all, the availability of  its research since our last review in January 2010 and what other steps  could ETA take to further ensure its research findings are readily  available? Appendix IV: Characteristics of Research Studies Disseminated between January 2009 and March 2010 That Cost $1 Million or More

Other (comparison  v. treatment group  using propensity  score matching)
Increasing the  labor market  participation of  underutilized  populations  characteristics  collected before  the intervention,  but these were not  used to make  comparisons)
Integration of the  workforce and  regional  economic  development  network analysis)
Integration of the  workforce and  regional  economic  development  network analysis )
Experimental  (randomized  control trials)
Increasing the  labor market  participation of  underutilized  populations  collected on  characteristics  upon entry and  outcome  characteristics  collected after  completion)

Appendix V: Delphi Phase I and Phase II Questionnaires

Appendix VI: Experts Who Agreed to Participate in GAO’s Delphi Panel

Trachtenberg School of Public Policy and Public  Administration, George Washington University  Abt Associates Inc.
Mathematica Policy Research, Inc.
W.E.
Why GAO Did This Study

To help guide the nation's workforce development system, the Department of Labor's (Labor) Employment and Training Administration (ETA) conducts research in areas related to job training and employment. Building upon our earlier work, GAO examined the following: (1) To what extent do ETA's research priorities reflect key national employment and training issues and how useful were the studies funded under them? (2) What steps has ETA taken to improve its research program? (3) How has ETA improved the availability of its research since our last review in January 2010? To answer these questions, GAO reviewed ETA's research reports disseminated between January 2008 and March 2010 costing $1 million or more, as well as ongoing studies costing $2 million or more. GAO also convened a virtual expert panel, interviewed Labor and ETA officials, and reviewed relevant documents.

What GAO Found

ETA's 2007 to 2012 research plan generally addressed key employment and training issues, but some studies were limited in their usefulness. Most experts on our panel reported that the areas in ETA's plan reflected key national employment and training issues at least to a moderate extent. ETA invested most of its research and evaluation resources in the areas of Unemployment Insurance and increasing labor market participation of underutilized groups. Of the $96 million that ETA invested in the 58 research reports we reviewed, more than half--or about $56 million--funded studies in these two areas. The methodological approaches and statistical procedures researchers used in the studies we reviewed were generally consistent with the questions posed, but the studies were not always useful for informing policy and practice. For example, in one study, shortcomings in the data collection phase limited the strength of the findings. Experts suggested that ETA include more varied and rigorous methodologies in its future research projects. They also reported that future research should address additional areas, including a focus on employment and training approaches that work and for whom. Labor and ETA have taken steps to improve the way research is conducted, but additional actions are needed. In acknowledging the need for more rigorous evaluations to inform its policies, Labor recently established the Chief Evaluation Office to oversee departmentwide research and evaluation efforts. In addition, ETA made changes to some of its research practices. For example, ETA has begun involving outside experts in developing its research plan. Despite these improvements, ETA's process lacks critical elements needed to ensure that current improvements become part of its routine practices. For example, ETA's process lacks a formal provision to consult with the newly established Chief Evaluation Officer at important points in the research process. Moreover, ETA's current research practices fall short of ensuring research transparency and accountability--essential elements of a sound research program. For example, its research and evaluation center lacks safeguards to protect it from undue outside influence. ETA has recently begun efforts to increase the rigor of its research designs, but has faced design and implementation challenges. For example, some of ETA's ongoing research studies face challenges in recruiting large enough sample sizes to meet the studies' objectives. ETA has improved the availability of its research findings, but it lacks a plan for assessing the usability of its Web-based search page--the primary tool for making ETA's research publicly available. ETA recently improved the timeliness with which it disseminates its research reports, decreasing the average number of days to release its reports to the public from 804 days in 2008 to 76 days in 2009. ETA has taken steps to update its online, Web-based search page. However, the agency has not developed a formal plan for assessing the overall effectiveness of its Web-based search page, including user satisfaction. In addition to its research database, ETA uses a variety of other methods to disseminate its research, including providing its research reports at conferences and internal briefings. Experts suggested that ETA consider other effective dissemination methods, such as publishing a one-page summary of research findings.

What GAO Recommends

GAO recommends that ETA formally include the Chief Evaluation Officer in its research process, create a mechanism to enhance the transparency and accountability of its research program, and develop a plan to ensure that research reports are accessible through its Web-based search page. Labor agreed with our recommendations and noted its ongoing efforts to improve its research program. While these efforts are important, GAO stresses the need for additional actions to fully address the recommendations.