	Introduction

Program evaluations can play an important role in public policy debates and in oversight of government programs, potentially affecting decisions about program design, operation, and funding. One technique that has received significant recent attention in the federal government is the randomized controlled trial (RCT). This report discusses what RCTs are and identifies a number of issues regarding RCTs that might arise when Congress considers making program evaluation policy. Issues regarding RCTs could also arise when actors in the policy process present specific program evaluations to Congress (e.g., in the President's budget proposals) to influence Congress's views and decision making. Are they performing the necessary types of evaluation? In addition, observers and practitioners sometimes disagree about the practical capabilities and limitations of various program evaluation methods, the quality of an individual evaluation, or how a study's findings should be interpreted and used. As discussed later in this report, an RCT attempts to estimate a program's impact on an outcome of interest . From the perspective of internal validity, the methodological rigor of an RCT study can depend on a number of factors, including, but not limited to the following:
how effectively the random assignment of units creates statistically equivalent groups; whether the group of subjects is sufficiently numerous to ensure that an impact large enough to be of interest to stakeholders, if it occurs, will be found statistically significant (the more numerous the units in the group being studied, the better chance there is of detecting a program's potential impact); whether attrition in the treatment and control groups (e.g., subjects dropping out of the study) is comparable with respect to the attributes of subjects; whether factors other than the treatment "contaminate" one group but not the other (e.g., due to problems with delivering the treatment or incomplete environmental controls); whether the behavior of researchers or subjects is affected because they know who is receiving a treatment or not receiving a treatment (ideally, RCTs are double-blind studies, in which neither the subjects nor the researchers know which group gets the treatment, but double-blind studies in social science are uncommon); whether the units being studied comply with the intervention being provided (e.g., did the patient take the medicine being studied? For example, many observers have argued that non-RCT studies, such as in-depth case studies and other observational or qualitative methods, are, among other things, (1) capable of casting doubt on an RCT's findings or causal claims by showing the RCT was contaminated or had poor design or implementation (i.e., revealing poor internal validity); (2) capable of showing a study's inferences are flawed or questionable (e.g., if the measured outcome of interest is judged to not fully reflect the program's goal(s), raising questions of construct validity); (3) essential for establishing the theory and conditions under which an intervention would be expected to make a favorable impact (increasing external validity); and (4) capable of establishing or strongly suggesting causation in certain circumstances (increasing internal validity), even if a study was not intended to estimate an impact. RCT Capabilities

There is wide consensus that, under certain conditions, well-designed and implemented RCTs provide the most valid estimate of an intervention's average impact for a large sample of subjects, as measured on an outcome of interest. However, RCTs are often seen as difficult to design and implement well. RCTs in Context: Program Evaluation and Systematic Review

			Concerns About Single Studies and Study Quality

In a variety of research fields, there appears to be consensus that a single study, no matter how well designed or implemented, is rarely sufficient to reliably support decision making. Moreover, there have been widespread concerns about the quality of individual studies of any design. Recent Attention to Using RCTs in Program Evaluation

The following two subsections briefly illustrate how RCTs have been a subject of attention in two contexts: (1) setting program evaluation policy in one specific policy area (education) and (2) the citation and use of individual studies, or claimed lack thereof, to justify policy and budget proposals to Congress (in this case, as a component of the George W. Bush Administration's Program Assessment Rating Tool). Controversy in Education Policy: A Priority for RCTs? What Types of Evaluations are Necessary? How Should Congress Use Evaluation Information When Considering and Making Policy? Do Agencies Have Capacity and Independence to Properly Conduct, Interpret, and Objectively Present Program Evaluations? Some systematic reviews focus on RCTs (and might include quasi-experiments ), and others include disparate types of studies.
Program evaluations can play an important role in public policy debates and in oversight of government programs, potentially affecting decisions about program design, operation, and funding. One technique that has received significant recent attention is the randomized controlled trial (RCT). There are also many other types of evaluation, including observational and qualitative designs.
An RCT attempts to estimate a program's impact upon an outcome of interest (e.g., crime rate). An RCT randomly assigns subjects to treatment and control groups, administers an intervention to the treatment group, and afterward measures the average difference between the groups. The quality of an RCT is typically assessed by its internal, external, and construct validity. At the federal level, RCTs have been a subject of interest and some controversy in education policy and the George W. Bush Administration's effort to integrate budgeting and performance using the Program Assessment Rating Tool (PART). In addition, in the 109th Congress, pending legislation provides for RCTs (e.g., Sections 3 and 15 of S. 1934; Section 114 of S. 667 (Senate committee-reported bill); and Section 5 of S. 1129).
Views about the practical capabilities and limitations of RCTs, compared to other evaluation designs, have sometimes been contentious. There is wide consensus that, under certain conditions, well-designed and implemented RCTs provide the most valid estimate of an intervention's impact, and can therefore provide useful information on whether, and the extent to which, an intervention causes favorable impacts for a large group of subjects, on average. However, RCTs are also seen as difficult to design and implement well. There also appears to be less consensus about what proportion of evaluations that are intended to estimate impacts should be RCTs and about the conditions under which RCTs are appropriate. Many observers argue that other types of evaluations are necessary complements to RCTs, or sometimes necessary substitutes for them, and can be used to establish causation, help bolster or undermine an RCT's findings, or in some situations validly estimate impacts. There is increasing consensus that a single study of any type is rarely sufficient to reliably support decision making. Many researchers have therefore embraced systematic reviews, which synthesize many similar or disparate studies.
A number of issues regarding RCTs might arise when Congress considers making program evaluation policy or when actors in the policy process present program evaluations to influence Congress. Should Congress focus on RCTs in these situations, a number of issues might be considered, including an RCT's parameters, capabilities, and limitations. In addition, Congress might examine the types of program evaluations that are necessary, question an evaluation's definitions or assumptions, consider how to appropriately use evaluation information in its learning and decision making, evaluate how much confidence to have in a study, and investigate whether agencies have capacity to properly conduct, interpret, and objectively present evaluations. This report will be updated in the 110th Congress.