HUD Has Not Identified Standard Performance Data and Indicators Needed to Evaluate the Program

MTW agencies provide descriptions of their activities and performance  information in their annual reports to HUD. HUD Lacks a Systematic Process for Identifying Lessons Learned

While HUD has identified some lessons learned on an ad hoc basis, it  does not have a systematic process in place for identifying such lessons. HUD Generally Follows Its Monitoring Policies and Procedures, but Could Strengthen Them

HUD has policies and procedures in place to monitor MTW agencies. HUD’s Monitoring Policies and Procedures Have Several Key Weaknesses

Although HUD follows the policies and procedures that it has in place, it  could do more to ensure that MTW agencies are demonstrating  compliance with statutory requirements and to identify possible risks  relating to activities implemented by each agency, among other things. Without a  process for systematically assessing compliance with statutory  requirements, HUD lacks assurance that agencies are complying with  them. Third, HUD has not performed an annual assessment of program risks. As we noted previously, conclusive information about  the effectiveness of the MTW program is limited in part because HUD  does not have a plan for identifying and analyzing standard performance  data, has not established performance indicators for the program as  whole, and does not have a systematic process for identifying lessons  learned. The report concluded that, given these  limitations, expansion should occur only if newly admitted PHAs structure  their programs for high-quality evaluations that permit lessons learned to  be generalized beyond a single PHA experience. In addition, our own work, some research organizations, and affordable  housing advocates question HUD’s ability to effectively manage an  expanded MTW program. While this flexibility has allowed participating agencies to  implement hundreds of activities, HUD has not done all that it can to  evaluate the program’s effectiveness, identify successful approaches that  could be applied to public housing agencies more broadly, or ensure that  MTW agencies comply with program requirements. Without more complete knowledge of  the program’s effectiveness and the extent to which agencies are  adhering to program requirements, it is difficult for Congress to know  whether an expanded MTW will benefit additional agencies and the  residents they serve. Without more specific guidance on reporting performance  information, HUD cannot be assured of collecting data that reflects the  outcomes of activities. Additionally, HUD has not established performance  indicators specific to MTW. Indicators linked to the statutory purposes of  reducing costs, encouraging self-sufficiency, and increasing housing  choices would help HUD demonstrate that the program has produced  desired results. At the same time, HUD’s monitoring efforts are not as strong as they  could be. Finally, just as HUD does not assess compliance with all three  self-certified requirements, it does not verify the accuracy of key  information that agencies self-report, including information on the impact  of MTW activities. Annual site visits have been used primarily to provide  technical assistance rather than to assess self-reported information. Recommendations for Executive Action

To improve what is known about the effectiveness of the MTW program,  we recommend that the Secretary of the Department of Housing and  Urban Development  improve HUD’s guidance to MTW agencies on providing performance  information in their annual reports by requiring that such information  be quantifiable and outcome-oriented to the extent possible; develop and implement a plan for quantitatively assessing the  effectiveness of similar activities and the program as a whole  including the identification of standard performance data needed; and establish performance indicators for the MTW program as a whole. To improve HUD’s oversight of the MTW program, we recommend that  the Secretary of the Department of Housing and Urban Development  issue guidance that clarifies key program terms, such as the three  statutory purposes of the program and the five statutory requirements  that MTW agencies must meet; develop and implement a systematic process for assessing  compliance with statutory requirements;  conduct an annual risk assessment for the MTW program and  implement risk-based monitoring policies and procedures such as  those currently being considered for site visits; and implement control activities designed to verify the accuracy of a  sample of the performance information that MTW agencies self-report. HUD generally agreed with the three remaining recommendations. For  example, HUD agreed that it should proactively identify lessons learned  and described some of its recent efforts to do so. Appendix I: Scope and Methodology

Our objectives were to examine (1) what is known about the extent to  which the Moving to Work (MTW) demonstration program is addressing  the program’s statutory purposes, (2) the Department of Housing and  Urban Development’s (HUD) monitoring of MTW agencies’ efforts to  address these purposes and meet statutory requirements, and (3)  potential benefits of and concerns about expanding the number of public  housing agencies (PHA) that can participate in the demonstration  program.
Why GAO Did This Study

HUDs MTW demonstration program gives participating PHAs the flexibility to create innovative housing strategies through their fiscal year 2018. MTW agencies must create activities linked to three statutory purposesreducing costs, providing incentives for self-sufficiency, and increasing housing choicesand meet five statutory requirements. Congress is considering expanding MTW and has asked GAO to examine what is known about (1) the programs success in addressing the three purposes, (2) HUDs monitoring efforts, and (3) the potential benefits of and concerns about expansion. GAO analyzed the most current annual reports for 30 MTW agencies; compared HUDs monitoring efforts with internal control standards; and interviewed agency officials, researchers, and industry officials.

What GAO Found

Public housing agencies (PHA) that participate in the Moving to Work (MTW) program report annually on the performance of their activities, which include efforts to reduce administrative costs and encourage residents to work. But this performance information varies, and the Department of Housing and Urban Developments (HUD) guidance does not specify that it be quantifiable and outcome oriented. Further, HUD has not identified the performance data that would be needed to assess the results of similar MTW activities or the program as a whole and has not established performance indicators for the program. The shortage of such analyses and indicators has hindered comprehensive evaluation efforts, although such evaluations are key to determining the success of any demonstration program. Further, while HUD has identified some lessons learned from the program, it has no systematic process to identify them and thus has relied primarily on ad hoc information. The absence of a systematic process for identifying lessons learned limits HUDs ability to promote useful practices that could be more broadly implemented to address the purposes of the program.
HUD generally follows its MTW monitoring policies and procedures, but they could be strengthened. HUD staff review and approve each MTW agencys annual plan to ensure that planned activities are linked to program purposes and visit each MTW agency annually to provide technical assistance. But HUD has not taken key monitoring steps set out in internal control standards, such as issuing guidance that defines program terms or assessing compliance with all of the requirements. Without clarifying key terms and establishing a process for assessing compliance with statutory requirements, HUD lacks assurance that agencies are actually complying with the statute. Additionally, HUD has not done an annual assessment of program risks despite its own requirement to do so and has not developed risk-based monitoring procedures. Without taking these steps, HUD lacks assurance that it has identified all risks to the program. Finally, HUD does not have policies or procedures in place to verify the accuracy of key information that agencies self-report. For example, HUD staff do not verify self-reported performance information during their reviews of annual reports or annual site visits. Without verifying at least some information, HUD cannot be sure that self-reported information is accurate.
Expanding the MTW program may offer benefits but also raises questions. According to HUD, affordable housing advocates, and MTW agencies, expanding MTW to additional PHAs would allow agencies to develop more activities tailored to local conditions and result in more lessons learned. However, data limitations and monitoring weaknesses raise questions about expansion. HUD recently reported that expansion should occur only if newly admitted PHAs structured their programs to permit high-quality evaluations and ensure that lessons learned could be generalized. Until more complete information on the programs effectiveness and the extent to which agencies are adhering to program requirements is available, it will be difficult for Congress to know whether an expanded MTW would benefit additional agencies and the residents they serve. Some researchers and MTW agencies suggested alternatives to expansion, including implementing a program that was more limited in scope.

What GAO Recommends

GAO makes eight recommendations to HUD: that HUD improve its guidance on reporting performance information, develop a plan for identifying and analyzing standard performance data, establish performance indicators, systematically identify lessons learned, clarify key terms, implement a process for assessing compliance with statutory requirements, do annual assessments of program risks, and verify the accuracy of self-reported data. HUD generally or in part agreed with seven of them. HUD disagreed with our recommendation that it create overall performance indicators. GAO believes, however, that they are critical to demonstrating program results and thus maintains its recommendation.