arXiv:1702.08615v1 [math.ST] 28 Feb 2017

Bridging Finite and Super Population Causal Inference
Peng Ding∗, Xinran Li†, and Luke W. Miratrix‡

Abstract
There are two general views in causal analysis of experimental data: the super population
view that the units are an independent sample from some hypothetical infinite populations, and
the finite population view that the potential outcomes of the experimental units are fixed and the
randomness comes solely from the physical randomization of the treatment assignment. These
two views differs conceptually and mathematically, resulting in different sampling variances of
the usual difference-in-means estimator of the average causal effect. Practically, however, these
two views result in identical variance estimators. By recalling a variance decomposition and
exploiting a completeness-type argument, we establish a connection between these two views in
completely randomized experiments. This alternative formulation could serve as a template for
bridging finite and super population causal inference in other scenarios.
Keywords: Completeness; Finite population correction; Potential outcome; Simple random
sample; Variance of individual causal effects

1.

Introduction

Neyman (1923, 1935) defined causal effects in terms of potential outcomes, and proposed an inferential framework viewing all potential outcomes of a finite population as fixed and the treatment
assignment as the only source of randomness. This finite population view allows for easy interpretation free of any hypothetical data generating process of the outcomes, and is used in a variety
of contexts (e.g., Kempthorne 1952; Hinkelmann and Kempthorne 2008; Copas 1973; Rosenbaum
∗
Department of Statistics, University of California,
Berkeley, California 94720 USA. Email:
pengdingpku@berkeley.edu
†
Department of Statistics, Harvard University, Cambridge, Massachusetts 02138 USA. Email:
xinranli@fas.harvard.edu
‡
Graduate School of Education and Department of Statistics, Harvard University, Cambridge, Massachusetts
02138 USA. Email: luke_miratrix@gse.harvard.edu

1

2002; Imai 2008; Freedman 2008a,b; Rosenbaum 2010; Aronow and Middleton 2013; Aronow et al.
2014; Abadie et al. 2014; Miratrix et al. 2013; Ding 2016; Lin 2013; Middleton and Aronow 2015;
Imbens and Rubin 2015; Chiba 2015; Rigdon and Hudgens 2015; Li and Ding 2016). This approach
is considered desirable because, in particular, it does not assume the data is somehow a representative sample of some larger (usually infinite) population.
Alternative approaches, also using the potential outcomes framework, assume that the potential
outcomes are independent and identical draws from a hypothetical infinite population. Mathematical derivations under this approach are generally simpler, but the approach itself can be criticized
because of this typically untenable sampling assumption. Furthermore, this approach appears to
ignore the treatment assignment mechanism.
That being said, it is well known that the final variance formulae from either approach tend to be
quite similar. For example, while the variance of the difference-in-means estimator for a treatmentcontrol experiment under an infinite population model is different from the one under Neyman
(1923)’s finite population formulation, this difference is easily represented as a function of the variance of the individual causal effects. Furthermore, this difference term is unidentifiable and is often
assumed away under a constant causal effect model (Neyman 1923, 1935; Hodges and Lehmann
1964; Rubin 1990; Reichardt and Gollob 1999; Freedman et al. 2007), or by appeals to the final
estimators being “conservative.”
Strictly speaking, the infinite population variance estimate gives a conservative (overly large)
estimate of the finite population variance for the difference in means. As deriving infinite population variance expressions, relative to finite population variance expressions, tends to be more
mathematically straightforward, we might naturally wonder if we could use infinite population expressions as conservative forms of finite population expressions more generally. In this work we
show that in fact we can assume an infinite population model as an assumption of convenience, and
derive formula from this perspective. This shows that we can thus consider the resulting formula
as focused on the treatment assignment mechanism and not on a hypothetical sampling mechanism, i.e., we show variance derivations under the infinite population framework can be used as
conservative estimators in a finite context.
Mathematically, this result comes from a variance decomposition and a completeness-style argument that characterizes the connection and the difference between these two views. The vari2

ance decomposition we use has previously appeared in Imai (2008), Imbens and Rubin (2015), and
Balzer et al. (2016). The completeness-style argument, which we believe is novel in this domain,
then sharpens the variance decomposition by moving from an expression on an overall average
relationship to one that holds for any specific sample. Our overall goal is simple: we wish to
demonstrate that if one uses variance formula derived from assuming an infinite population sampling model, then the resulting inference one obtains will be correct with regards to the analogous
samples-specific treatment effects (although it could be potentially conservative in that the standard
errors may be overly large) regardless of the existence of any sampling mechanism.

2.

Super Population, Finite Population, and Samples

Assume that random variables (Y (1), Y (0)) represent the pair of potential outcomes of an infinite super population, from which we take an independent and identically distributed (IID) finite
population of size n:
S = {(Yi (1), Yi (0)) : i = 1, . . . , n}.
We first discuss completely randomized experiments, and comment on other experiments in Section
4. For a completely randomized experiment, we randomly assign n1 units to receive treatment,
leaving the remaining n0 = n − n1 units to receive control. Let Z = (Z1 , . . . , Zn ) ∈ {0, 1}n be the
treatment assignment vector, which takes a particular value (z1 , . . . , zn ) ∈ {0, 1}n with probability
P
n1 !n0 !/n!, for any (z1 , . . . , zn ) ∈ {0, 1}n with ni=1 zi = n1 . The observed outcome for unit i is
then Yiobs = Zi Yi (1) + (1 − Zi )Yi (0).

At the super population level, the average potential outcomes are E{Y (1)} and E{Y (0)}, and
the average causal effect is
τ = E{Y (1) − Y (0)}.
The population variances of the potential outcomes and individual causal effect are

V1 = Var{Y (1)},

V0 = Var{Y (0)},

Vτ = Var{Y (1) − Y (0)}.

At the finite population level, i.e., for a fixed sample S, the average potential outcomes and

3

average causal effect are
n

Ȳ1 =

1X
Yi (1),
n

n

Ȳ0 =

i=1

1X
Yi (0),
n

τS = Ȳ1 − Ȳ0 .

i=1

The corresponding finite population variances of the potential outcomes and individual causal
effects are
n

S12 =

1 X
{Yi (1) − Ȳ1 }2 ,
n−1
i=1

n

S02 =

1 X
{Yi (0) − Ȳ0 }2 ,
n−1

n

Sτ2 =

i=1

1 X
(τi − τ )2 ,
n−1
i=1

where we use the divisor n − 1 following the tradition of survey sampling (Cochran 1977). All these
quantities are fully dependent on S. In classical casual inference (Neyman 1923), the potential
outcomes of these n experimental units, S, are treated as fixed numbers. Equivalently, we can
consider such causal inference to be conducted conditional on S (e.g., Copas 1973; Rosenbaum
2002; Imbens and Rubin 2015; Rigdon and Hudgens 2015; Chiba 2015).
Regardless, we have two parameters—the population average treatment effect τ , and the sample
average treatment effect τS . After collecting the data, we would want to draw inference about τ or
τS .
Our primary statistics are the averages of the observed outcomes and the difference-in-means
estimator:
Ȳ1obs =

n
1 X
Zi Yiobs ,
n1

Ȳ0obs =

i=1

n
1 X
(1 − Zi )Yiobs ,
n0
i=1

τb = Ȳ1obs − Ȳ0obs .

We also observe the sample variances of the outcomes under treatment and control using
n

s21

1 X
=
Zi (Yiobs − Ȳ1obs )2 ,
n1 − 1

n

s20

i=1

1 X
=
(1 − Zi )(Yiobs − Ȳ0obs )2 .
n0 − 1
i=1

We do not have the sample analogue of Sτ2 or Vτ because Yi (1) and Yi (0) are never jointly observed
for any unit i in the sample.
We summarize the infinite population, finite population and sample quantities in Table 1.

4

Table 1: Means and variances at the super population, finite population and sample levels.

super population
finite population
sample

3.

treatment
E{Y (1)}
Ȳ1
Ȳ1obs

means
control
effect
E{Y (0)} τ = E{Y (1) − Y (0)}
Ȳ0
τS = Ȳ1 − Ȳ0
Ȳ0obs
τb = Ȳ1obs − Ȳ0obs

variances
treatment control
V1
V0
S12
S02
s21
s20

effect
Vτ
Sτ2
−

Deriving Complete Randomization Results with an Independent Sampling Model

The three levels of quantities in Table 1 are connected via independent sampling and complete
randomization. Neyman (1923), without reference to any infinite population and by using the
assignment mechanism as the only source of randomness, represented the assignment mechanism
via an urn model, and found

Var(b
τ | S) =

S12 S02 Sτ2
+
−
.
n1
n0
n

(1)

He then observed that the final term was unidentifiable but nonnegative, and thus if we dropped
it we would obtain an upper bound of the estimator’s uncertainty.
We next derive this result by assuming a hypothetical sampling mechanism from some assumed
infinite super-population model of convenience. This alternative derivation of the above result,
which can be extended to other assignment mechanisms, shows how we can interpret formulae
based on super-population derivations as conservative formulae for finite-sample inference.

3.1.

Sampling and randomization

To begin, note that IID sampling of S from the super population implies three things: first, the
finite population average causal effect τS satisfies E(τS ) = τ , second
#
n
Vτ
1X
{Yi (1) − Yi (0)} =
,
Var(τS ) = Var
n
n
"

i=1

5

(2)

and third, the sample variances are unbiased for the true variances:
E(S12 ) = V1 ,

E(S02 ) = V0 ,

E(Sτ2 ) = Vτ .

(3)

Conditional on S, randomization of the treatment Z is the only source of randomness. In a
completely randomized experiment, the outcomes in the treatment group form a simple random
sample of size n1 from {Yi (1) : i = 1, . . . , n}, and the outcomes in the control group form a simple
random sample of size n0 from {Yi (0) : i = 1, . . . , n}. Therefore, classical survey sampling theory
(Cochran 1977) for the sample mean and variance gives
E(Ȳ1obs | S) = Ȳ1 ,

E(Ȳ0obs | S) = Ȳ0 ,

E(b
τ | S) = τS ,

E(s21 | S) = S12 ,

E(s20 | S) = S02 .

(4)

We do not use the notation ES (·), because, depending on the contexts, such notation could either
indicate expectation conditional on S or expectation averaged over S. We therefore use conditional
expectations and conditional variances explicitly.
If we do not condition on S, then the independence induced by the assignment mechanism means
the outcomes under treatment are IID samples of Y (1) and the outcomes under control are IID
samples of Y (0), and furthermore these samples are independent of each other. This independence
makes it straightforward to show that τb is unbiased for τ with super population variance
Var(b
τ ) = Var(Ȳ1obs ) + Var(Ȳ0obs ) =

V1
V0
+ .
n1 n0

(5)

This is the classic infinite population variance formula for the two sample difference-in-means
statistic. We could use it to obtain standard errors by plugging in s21 and s20 for the two variances.
This derivation, from (2)–(5), is motivated by the sampling assumption: the assignment mechanism
makes the two samples independent, but it is the IID assumption and classic sampling theory
that gives this result (along with asymptotical normality which allows for associated testing and
confidence interval generation).

6

3.2.

Connecting the finite and infinite population inference with a variance
decomposition

We will now extend the above to indirectly derive the result on the variance of τb in finite popula-

tion inference without explicitly enumerating the potential outcomes. The variance decomposition
formula implies

Var(b
τ ) = E{Var(b
τ | S)} + Var{E(b
τ | S)}

(6)

= E{Var(b
τ | S)} + Var(τS )
= E{Var(b
τ | S)} +

Vτ
,
n

(7)

which further implies that the finite population variance of τb satisfies (using (3))
E {Var(b
τ | S)} = Var(b
τ) −

Vτ
V1
V0 Vτ
=
+
−
=E
n
n1 n0
n



S12 S02 Sτ2
+
−
n1
n0
n



.

(8)

Compare to the classic variance expression (1), which is this without the expectation. Here we
have that on average our classic variance expression holds. Now, because this is true for any
infinite population, as it is purely a consequence of the IID sampling mechanism and complete
randomization, we can close the gap between (1) and (8). Informally speaking, because (8) holds as
an average over many hypothetical super populations, it should also hold for any finite population
at hand, and indeed it does, as we next show using a “completeness” concept from statistics
(Lehmann and Romano 2008).

3.3.

A “Completeness” argument

First, define

f (S) ≡

S12 S02 Sτ2
+
−
− Var(b
τ | S),
n1
n0
n

(9)

a function of a fixed finite sample S, as the difference of the hypothesized finite sample variance
formula and the actual finite sample variance. Formula (8) shows E{f (S)} = 0. Now we are going
to show the stronger result f (S) = 0.
7


For any given sample S, we have fixed sample quantities: U = Y1 (1) − Ȳ1 , . . . , Yn (1) − Ȳ1 ,

and W = Y1 (0) − Ȳ0 , . . . , Yn (0) − Ȳ0 . Some algebra gives
τb − τS =

n
n
1 X
1 X
Zi Ui −
(1 − Zi )Wi .
n1
n0
i=1

(10)

i=1

According to (10), the finite population variance Var(τ̂ | S) = E{(τ̂ − τS )2 | S} must be a quadratic
form of (U, W ). Therefore, from (9), f (S) must be a quadratic form of (U, W ), and therefore must
have the following expression:

f (S) =

n
n X
X

aij Ui Uj +

n
n X
X

bij Wi Wj +

cij Ui Wj .

i=1 j=1

i=1 j=1

i=1 j=1

n
n X
X

for some aij , bij and cij . f (S) is the same regardless of the ordering of units, so by symmetry
the constants aij , bij and cij must be the same regardless of i and j. This gives aii = a11 , aij =
a12 , bii = b11 , bij = b12 , cii = c11 , cij = c12 , for all i 6= j. Thus

f (S) =a11

n
X

Uj2

+ a12

i=1

Because

XX

Ui Uj + b11

i=1 Ui

Ui2

+

i=1

n
X
i=1

Pn

Wi2

i=1 Wi

+

+ b12

XX

Wi Wj + c11

n
X

Ui Wi + c12

i=1

i6=j

XX

Ui W j .

i6=j

= 0, we have
n
X

Similarly, as

Wi2

i=1

i6=j

Pn

n
X

XX

Ui Uj =

n
X

Ui

i=1

i6=j

!2

= 0.

= 0, we have

XX

Wi Wj = 0,

n
X

Ui Wi +

i=1

i6=j

XX

Ui W j =

n
X
i=1

i6=j

Ui

!

n
X
i=1

Wi

!

= 0.

Use the above to replace our cross terms of i 6= j with single index terms to write f (S) as

f (S) =a

n
X
i=1

Ui2 + b

n
X
i=1

Wi2 + c

n
X

Ui Wi ,

i=1

where a = a11 − a12 , b = b11 − b12 , and c = c11 − c12 . Because E{f (S)} = 0, we have aE(U12 ) +

8

aE(W12 ) + cE(U1 W1 ) = 0, where
E(U12 ) =

n−1
n−1
n−1
Var{Y (1)}, E(W12 ) =
Var{Y (0)}, E(U1 W1 )2 =
Cov{Y (1), Y (0)}.
n
n
n

Thus,

aVar{Y (1)} + bVar{Y (0)} + cCov{Y (1), Y (0)} = 0.

(11)

Because (11) holds for any populations regardless of its values of Var{Y (1)}, Var{Y (0)} and Cov{Y (1), Y (0)},
it must be true that a = b = c = 0, which implies f (S) = 0.

4.

Discussion

Equation (8) relies on the assumption that the hypothetical infinite population exists, but Equation (1) does not. However, the completeness-style argument allowed us to make our sampling
assumption only for convenience in order to prove (1) by, in effect, dropping the expectation on
both sides of (8). Similar argument exists in the classical statistics literature; see Efron and Morris
(1973) for the empirical Bayes view of Stein’s estimator. While the final result is, of course, not
new, we offer it as it gives an alternative derivation that does not rely on asymptotics such as a
growing superpopulation or a focus on the properties of the treatment assignment mechanism.
Using Freedman (2008a)’s results, Aronow et al. (2014) considered a super population with N
units, with the finite population being a simple random sample of size n. Letting N → ∞, we
can obtain similar results. We go in the other direction: we use the variance decomposition (6) to
derive the finite population variance from the super population one.
This decomposition approach also holds for other types of experiments. First, for a stratified
experiment, each stratum is essentially a completely randomized experiment. Apply the result to
each stratum, and then average over all strata to obtain results for a stratified experiment. Second, because a matched-pair experiment is a special case of a stratified experiment with two units
within each stratum, we can derive the Neyman-type variance (cf. Imai 2008; Imbens and Rubin
2015) directly from that of a stratified experiment. Third, a cluster-randomized experiment is a
completely randomized experiment on the clusters. If the causal parameters can be expressed as
9

cluster-level outcomes, then the result can be straightforwardly applied (cf. Aronow and Middleton
2013; Middleton and Aronow 2015). Fourth, for general experimental designs, the variance decomposition in (6) still holds, and therefore we can modify the derivation of the finite population
variance according to different forms of (4) and (5).
In a completely randomized experiment, the finite population sampling variance of τb in (1) de-

pends on three terms: the first two can be unbiasedly estimated by s21 /n1 and s20 /n0 , but the third
term Sτ2 /n is unidentifiable from the data. Assuming a constant causal effect model, Sτ2 = 0 and
the variance estimators under both finite and super population inference coincide. However, Robins
(1988), Aronow et al. (2014) and Ding and Dasgupta (2016) demonstrated that the treatment variation term Sτ2 has a sharp lower bound that may be larger than 0, which allows for more precise
variance estimators under the finite population view. This demonstrates that we can indeed make
better inference conditional on the sample we have. On the other hand, in this work, we showed
that assuming an infinite population, while not necessarily giving the tightest variance expressions,
nonetheless gives valid (conservative) variance expressions from a finite-population perspective.
We offer this approach as a possible method of proof that could ease derivations for more complex
designs. More broadly, it is a step towards establishing that infinite population derivations for randomized experiments can be generally thought of as pertaining to their finite population analogs.
Also see Samii and Aronow (2012) and Lin (2013), who provide alternative discussions of super
population regression-based variance estimators under the finite population framework.
Our discussion is based on the frequentists’ repeated sampling evaluations of the difference-inmeans estimator for the average causal effect. In contrast, Fisher (1935) proposed the randomization test against the sharp null hypothesis that Yi (1) = Yi (0) for all units, which is numerically the same as the permutation test for exchangeable units sampled from an infinite population
(Lehmann and Romano 2008). This connection becomes more apparent when only the ranks of the
outcomes are used to construct the test statistics, as discussed extensively by Lehmann (1975).

Acknowledgments
We thank the Associate Editor, Dr. Peter Aronow, and three anonymous reviewers for helpful
comments.
10

References
Abadie, A., Athey, S., Imbens, G. W., and Wooldridge, J. M. (2014). Finite population causal
standard errors. Technical report, National Bureau of Economic Research.
Aronow, P. M., Green, D. P., and Lee, D. K. K. (2014). Sharp bounds on the variance in randomized
experiments. The Annals of Statistics, 42:850–871.
Aronow, P. M. and Middleton, J. A. (2013). A class of unbiased estimators of the average treatment
effect in randomized experiments. Journal of Causal Inference, 1:135–154.
Balzer, L. B., Petersen, M. L., and Laan, M. J. (2016). Targeted estimation and inference for the
sample average treatment effect in trials with and without pair-matching. Statistics in Medicine,
35:3717–3732.
Chiba, Y. (2015). Exact tests for the weak causal null hypothesis on a binary out come in randomized trials. Journal of Biometrics and Biostatistics, 6:doi:10.4172/2155–6180.1000244.
Cochran, W. G. (1977). Sampling Techniques. John Wiley & Sons, 3rd edition.
Copas, J. (1973). Randomization models for the matched and unmatched 2 × 2 tables. Biometrika,
60:467–476.
Ding, P. (2016). A paradox from randomization-based causal inference (with discussion). Statistical
Science, in press.
Ding, P. and Dasgupta, T. (2016). A potential tale of two by two tables from completely randomized
experiments. Journal of American Statistical Association, 111:157–168.
Efron, B. and Morris, C. (1973). Stein’s estimation rule and its competitors—an empirical bayes
approach. Journal of the American Statistical Association, 68:117–130.
Fisher, R. A. (1935). The Design of Experiments. Edinburgh, London: Oliver and Boyd, 1st edition.
Freedman, D., Pisani, R., and Purves, R. (2007). Statistics. W. W. Norton & Company, 4th edition.
Freedman, D. A. (2008a). On regression adjustments in experiments with several treatments. The
Annals of Applied Statistics, 2:176–196.
11

Freedman, D. A. (2008b). Randomization does not justify logistic regression. Statistical Science,
23:237–249.
Hinkelmann, K. and Kempthorne, O. (2008). Design and Analysis of Experiments, Volume 1:
Introduction to Experimental Design. New Jersey: John Wiley & Sons, Inc., 2nd edition.
Hodges, J. L. J. and Lehmann, E. L. (1964). Basic Concepts of Probability and Statistics. San
Francisco: Holden-Day.
Imai, K. (2008). Variance identification and efficiency analysis in randomized experiments under
the matched-pair design. Statistics in Medicine, 27:4857–4873.
Imbens, G. W. and Rubin, D. B. (2015). Causal Inference for Statistics, Social and Biometrical
Sciences: An Introduction. Cambridge: Cambridge University Press.
Kempthorne, O. (1952). The Design and Analysis of Experiments. New York: John Wiley and
Sons.
Lehmann, E. L. (1975). Nonparametrics: Statistical Methods Based on Ranks. San Francisco:
Holden-Day, 1st edition.
Lehmann, E. L. and Romano, J. P. (2008). Testing Statistical Hypotheses. New York : Wiley, 3rd
edition.
Li, X. and Ding, P. (2016). Exact confidence intervals for the average causal effect on a binary
outcome. Statistics in Medicine, 35:957–960.
Lin, W. (2013). Agnostic notes on regression adjustments to experimental data: Reexamining
Freedman’s critique. The Annals of Applied Statistics, 7:295–318.
Middleton, J. A. and Aronow, P. M. (2015). Unbiased estimation of the average treatment effect
in cluster-randomized experiments. Statistics, Politics and Policy, 6:39–75.
Miratrix, L. W., Sekhon, J. S., and Yu, B. (2013). Adjusting treatment effect estimates by poststratification in randomized experiments. Journal of the Royal Statistical Society: Series B
(Statistical Methodology), 75:369–396.
12

Neyman, J. (1923). On the application of probability theory to agricultural experiments. essay on
principles (with discussion). section 9 (translated). reprinted ed. Statistical Science, 5:465–472.
Neyman, J. (1935). Statistical problems in agricultural experimentation (with discussion). Supplement to the Journal of the Royal Statistical Society, 2:107–180.
Reichardt, C. S. and Gollob, H. F. (1999). Justifying the use and increasing the power of a t test
for a randomized experiment with a convenience sample. Psychological Methods, 4:117–128.
Rigdon, J. and Hudgens, M. G. (2015). Randomization inference for treatment effects on a binary
outcome. Statistics in Medicine, 34:924–935.
Robins, J. M. (1988). Confidence intervals for causal parameters. Statistics in Medicine, 7:773–785.
Rosenbaum, P. R. (2002). Observational Studies. New York: Springer, 2nd edition.
Rosenbaum, P. R. (2010). Design of Observational Studies. New York: Springer.
Rubin, D. B. (1990). Comment: Neyman (1923) and causal inference in experiments and observational studies. Statistical Science, 5:472–480.
Samii, C. and Aronow, P. M. (2012). On equivalencies between design-based and regression-based
variance estimators for randomized experiments. Statistics and Probability Letters, 82:365–370.

13

