SSA Has Used Its Demonstration Authority to Initiate Several Projects Since 1998, but Efforts Have Yielded Limited Information

Over the last decade, SSA has initiated 14 demonstration projects to test  policy and program changes, of which SSA has completed 4, cancelled 5,  and had 5 projects in progress as of August 2008. In total, SSA had spent  about $155 million on these demonstration projects as of April 2008, and  officials anticipate spending another $220 million in the coming years on  those projects currently in progress. However, these projects have yielded  limited information for influencing program and policy decisions. We  found that SSA did not conduct impact evaluations for two of its  completed projects and cancelled five projects prior to conducting formal  evaluations; thus, limited information is available. Demonstration Projects Have Yielded Limited Information to Influence Program and Policy Changes

Despite using its demonstration authority to examine various issues, SSA’s  demonstration projects have yielded limited information for influencing  program and policy decisions. Furthermore, SSA intended to evaluate the impacts of policies and  programs being tested in five other projects but could not do so because  the significant challenges those projects faced led SSA to cancel them in  the early stages. While they have the potential  to yield reliable results, it is too early to tell whether they will ultimately  be useful for informing DI and SSI policy and program changes. SSA Has Taken Steps to Improve Its Projects but Continues to Lack Management Controls

SSA has taken steps to improve its demonstration projects, in part by  applying more rigorous methodologies than it did for the projects SSA  initiated prior to 1998; however, it has not fully implemented GAO’s  recommendations from 2004 and does not have written policies and  procedures in place to ensure that projects are routinely reviewed and  effectively managed so that they yield reliable information about their  impacts. On the basis of our assessment, we determined that 11 of  the 12 projects’ designs were strong or reasonable when assessed against  professional research standards (see table 3). To further improve the demonstration projects’ planning and  methodological rigor, SSA has used external research professionals to  work with the agency on the design, implementation, or evaluation of 12 of  the 14 projects. SSA also appointed new program management in 2007. In addition to not fully addressing our prior recommendations, SSA does  not have written policies and procedures governing how it should review  and operate its demonstration program. In addition, we found that SSA does not have written procedures for its  project officers to follow as they design, implement, and evaluate its  demonstration projects. Such procedures could be used to ensure that  standard research practices, such as conducting pilot phases and including  internal and external stakeholders, are applied when planning and  implementing the demonstration projects. GAO staff who made key contributions to this report are  listed in appendix V.

Appendix I: Scope and Methodology

To determine how SSA has used its current demonstration authority, we  reviewed legislation authorizing the Social Security Administration (SSA)  to conduct Disability Insurance (DI) and Supplemental Security Income  (SSI) demonstration projects, prior GAO report and SSA’s Office of the  Inspector General (OIG) reports, and reports to Congress on the  demonstration authority. SSA implemented the project in 2004.
Why GAO Did This Study

Since 1980, Congress has required the Social Security Administration (SSA) to conduct demonstration projects to test the effectiveness of possible changes to its Social Security Disability Insurance (DI) and Supplemental Security Income (SSI) programs that could decrease individuals' dependence on benefits or improve program administration. However, in 2004, GAO reported that SSA had not used its demonstration authority effectively. This follow-up report assesses (1) how SSA has used its demonstration authority to test DI and SSI program changes and what information these efforts have yielded and (2) what steps SSA has taken to improve the planning and management of its demonstration projects. To do this, GAO reviewed documents related to SSA's demonstration project management and the steps it took to implement the recommendations in the 2004 report, as well as the projects' designs, evaluations, and costs. GAO also interviewed officials from SSA, its contractors and project sites, and disability experts.

What GAO Found

Over the last decade, SSA has initiated 14 demonstration projects under its authority to test possible DI and SSI policy and program changes; however, these projects have yielded limited information for influencing program and policy decisions. Of the 14 projects, SSA has completed 4, cancelled 5, and had 5 projects in progress as of June 2008. In total, SSA spent about $155 million on its projects as of April 2008, and officials anticipate spending another $220 million in the coming years on those projects currently under way. Yet, these projects have yielded limited information on the impacts of the program and policy changes they were testing. SSA did not conduct impact evaluations for two of its completed projects, and intended to evaluate five other projects, but could not do so because significant challenges led SSA to cancel them. SSA officials believe the five projects currently under way will yield useful information, but it is too early to tell. SSA has taken steps to improve its demonstration projects but continues to lack management controls to ensure that the projects yield reliable information for making disability policy decisions. SSA has used methodological designs that GAO determined were strong or reasonable when assessed against professional research standards for 11 of its 14 projects. SSA has also used external research professionals to work with the agency on the design, implementation, or evaluation of 12 of the projects, and appointed new program management to oversee its demonstration program. However, as of August 2008, SSA had not fully implemented the recommendations GAO made in 2004 and did not have written policies and procedures governing how it should review and operate its demonstration project program. Specifically, SSA does not have written policies and procedures for its managers and project officers to follow as they design, implement, and evaluate its demonstration projects. Absent such protocols, SSA did not always apply standard research practices, such as conducting pilot phases or obtaining sufficient stakeholder input, which led to data limitations and project cancellations.