OGAC, CDC, and USAID Evaluated a Broad Range of Programs

OGAC, CDC, and USAID provided 496 evaluations addressing programs  ongoing during fiscal years 2008 to 2010 in all PEPFAR program areas  relating to HIV/AIDS treatment, prevention, and care. Our assessments of judgmental and randomly selected samples of  PEPFAR evaluations indicate that many—particularly those managed by  PEPFAR country and regional teams—contain findings, conclusions, and  recommendations that are not fully supported. Our assessment of seven OGAC- managed PEPFAR PHEs indicates that they all generally adhered to  common evaluation standards, and thus their findings, conclusions, and  recommendations were fully supported. CDC and USAID headquarters-managed evaluations. Based on our analysis of a randomly  selected sample of country and regional team evaluations, we estimate  that findings, conclusions, and recommendations were fully supported in  41 percent of all evaluations provided to us by country and regional  teams, partially supported in 44 percent of these evaluations, and not  supported in 15 percent of these evaluations. As a result, the objectivity and credibility of these  evaluations’ findings, conclusions, and recommendations are in question. PEPFAR Policies and Procedures Do Not Fully Adhere to AEA Evaluation Principles Relating to Planning, Independence, and Dissemination

State, OGAC, CDC, and USAID have developed policies and procedures  that apply to evaluations of PEPFAR programs, as called for in the AEA  Roadmap. Second, State, OGAC, CDC, and USAID guidance does not  specify how to document the independence and competency of  evaluators, and almost half of the evaluations we reviewed did not  provide sufficient information to fully determine whether evaluators were  free of conflicts of interest. Finally, not all evaluation reports are available  online, thus limiting their accessibility and usefulness to PEPFAR decision  makers and other stakeholders.   USAID evaluation policy.   OGAC. For example, OGAC officials noted that  the PEPFAR website contains information on PEPFAR results as well  as monitoring and evaluation guides. CDC policy advises that effort is needed to ensure that  evaluation findings are disseminated appropriately but does not  require online dissemination of evaluation reports. State, CDC, and  USAID have demonstrated a clear commitment to program evaluation by  conducting a wide variety of program evaluations that address at least  one activity in each PEPFAR program area. Global AIDS  Coordinator to take the following four actions in collaboration with CDC  and USAID to enhance PEPFAR evaluations:  1.  develop a strategy to improve PEPFAR implementing agencies’ and  country and regional teams’ adherence to common evaluation  standards;  2.  require implementing agency headquarters and country and regional  teams to include evaluation plans in their annual operational plans;  3.  provide detailed guidance for implementing agencies and country and  regional teams on assessing, ensuring, and documenting the  independence and competence of PEPFAR program evaluators; and  4.  increase the online accessibility of PEPFAR program evaluation  results. Responding jointly with CDC and USAID, State OGAC provided written  comments (see app. Appendix I: Objectives, Scope, and Methodology

This report (1) identifies President’s Emergency Plan for AIDS Relief  (PEPFAR) evaluation activities and examines the extent to which  evaluation results are supported and (2) examines the extent to which  PEPFAR policies and procedures adhere to established principles for the  evaluation of U.S. government programs. To identify PEPFAR program  evaluations and examine the extent to which they generated supported  evaluation results, we collected and analyzed program evaluation  documents provided by Centers for Disease Control and Prevention  (CDC) and U.S. Agency for International Development (USAID) officials in  the 31 PEPFAR countries and 3 regions with PEPFAR country or regional  operational plans in fiscal year 2010, as well as the Department of State’s  (State) Office of the U.S.
Why GAO Did This Study

PEPFAR, reauthorized by Congress in fiscal year 2008, supports HIV/AIDS prevention, treatment, and care overseas. The reauthorizing legislation, as well as other U.S. law and government policy, stresses the importance of evaluation for improving program performance, strengthening accountability, and informing decision making. OGAC leads the PEPFAR effort by providing funding and guidance to implementing agencies, primarily CDC and USAID. Responding to legislative mandates, GAO (1) identified PEPFAR evaluation activities and examined the extent to which evaluation findings, conclusions, and recommendations were supported and (2) examined the extent to which PEPFAR policies and procedures adhere to established general evaluation principles. GAO reviewed these principles as well as agencies policies and guidance; surveyed CDC and USAID officials in 31 PEFAR countries and 3 regions; and analyzed evaluations provided by OGAC, CDC, and USAID.

What GAO Found

The Department of States (State) Office of the U.S. Global AIDS Coordinator (OGAC), the Department of Health and Human Services (HHS) Centers for Disease Control and Prevention (CDC), and the U.S. Agency for International Development (USAID) have evaluated a wide variety of Presidents Emergency Plan for AIDS Relief (PEPFAR) program activities, demonstrating a clear commitment to evaluation. However, GAO found that the findings, conclusions, and recommendations were not fully supported in many PEPFAR evaluations. Agency officials provided nearly 500 evaluations addressing activities ongoing in fiscal years 2008 through 2010 in all program areas relating to HIV/AIDS treatment, prevention, and care. GAOs assessment of a selected sample of seven OGAC-managed evaluations found that they generally adhered to common evaluation standards, as did most of a selected sample of 15 evaluations managed by CDC and USAID headquarters. Based on this assessment, GAO determined that these evaluations generally contained fully supported findings, conclusions, and recommendations. However, based on a similar assessment of a randomly selected sample taken from 436 evaluations provided by PEPFAR country and regional teams, GAO estimated that 41 percent contained fully supported findings, conclusions, and recommendations, while 44 percent contained partial support and 15 percent were not supported.
State, OGAC, CDC, and USAID have established detailed evaluation policies, as recommended by the American Evaluation Association (AEA). However, PEPFAR does not fully adhere to AEA principles relating to evaluation planning, independence and qualifications of evaluators, and public dissemination of evaluation results. Specifically, OGAC does not require country and regional teams to include evaluation plans in their annual operational plans, limiting its ability to ensure that evaluation resources are appropriately targeted. Further, although OGAC, CDC, and USAID evaluation policies and procedures provide some guidance on how to ensure evaluator independence and qualifications, they do not require documentation of these issues. GAO found that most PEPFAR program evaluations did not fully address whether evaluators had conflicts of interest and some did not include detailed information on the identity and makeup of evaluation teams. Finally, although OGAC, CDC, and USAID use a variety of means to share evaluation findings, not all evaluation reports are available online, limiting their accessibility to the public and their usefulness for PEPFAR decision makers, program managers, and other stakeholders.

What GAO Recommends

GAO recommends that State work with CDC and USAID to (1) improve adherence to common evaluation standards, (2) develop PEPFAR evaluation plans, (3) provide guidance for assessing and documenting evaluators independence and qualifications, and (4) increase online accessibility of evaluation results. Commenting jointly with HHSs CDC and USAID, State agreed with these recommendations and noted steps it will take to implement them.