Sequential identification of nonignorable missing data mechanisms
Mauricio Sadinle and Jerome P. Reiter

arXiv:1701.01395v1 [] 5 Jan 2017

Duke University
January 6, 2017

Abstract
With nonignorable missing data, likelihood-based inference should be based on the joint
distribution of the study variables and their missingness indicators. These joint models cannot
be estimated from the data alone, thus requiring the analyst to impose restrictions that make the
models uniquely obtainable from the distribution of the observed data. We present an approach
for constructing classes of identifiable nonignorable missing data models. The main idea is to use
a sequence of carefully set up identifying assumptions, whereby we specify potentially different
missingness mechanisms for different blocks of variables. We show that the procedure results in
models with the desirable property of being non-parametric saturated.

Key words and phrases: Identification; Non-parametric saturated; Missing not at random;
Partial ignorability; Sensitivity analysis.

1

Introduction

When data are missing not at random (MNAR) (Rubin (1976)), appropriate likelihood-based inference requires explicit models for the full-data distribution, i.e., the joint distribution of the study
variables and their missingness indicators. Because of the missing data, this distribution is not
uniquely identified from the observed data alone (Little and Rubin (2002)). To enable inference,
analysts must impose restrictions on the full-data distribution. Such assumptions generally are
untestable; however, a minimum desideratum is that they result in a unique full-data distribution
for the observed-data distribution at hand, i.e., the distribution that can be identified from the
incomplete data.
We present a strategy for constructing identifiable full-data distributions with nonignorable
missing data. In its most general form, the strategy is to expand the observed-data distribution
sequentially by identifying parts of the full-data distribution associated with blocks of variables, one
block at a time. This partitioning of the variables allows analysts to specify different missingness
mechanisms in the different blocks; for example, use the missing at random (MAR, Rubin (1976))
assumption for some variables and a nonignorable missingness assumption for the rest to obtain a
partially ignorable mechanism (Harel and Schafer (2009)). We ensure that the resulting full-data
1

distributions are non-parametric saturated (NPS, Robins (1997)), that is, their implied observeddata distribution matches the actual observed-data distribution, as detailed in Section 2.2.
Related approaches to partitioning variables with missing data have appeared previously in the
literature. Zhou et al. (2010) proposed to model blocks of study variables and their missingness
indicators in a sequential manner; however, their approach does not guarantee identifiability of the
full-data distribution. Harel and Schafer (2009) mentioned the possibility of treating the missingness
in blocks of variables differently, but they do not provide results on identification. Robins (1997)
proposed the group permutation missingness mechanism, which assumes MAR sequentially for
blocks of variables and results in a NPS model. This is a particular case of our more general
procedure, as we describe in Section 3.4.
The remainder of the article is organized as follows. In Section 2, we describe notation and
provide more details on the NPS property. In Section 3, we introduce our strategy for identifying
a full-data distribution in a sequential manner. In Section 4 we present some examples of how to
use this strategy for the case of two categorical study variables, for the construction of partially
ignorable mechanisms, and for sensitivity analyses. Finally, in Section 5 we discuss possible future
uses of our identifying approach.

2

Notation and Background

2.1

Notation

Let X = (X1 , . . . , Xp ) denote p random variables taking values on a sample space X . Let Mj be
the missingness indicator for variable j, where Mj = 1 when Xj is missing and Mj = 0 when Xj is
observed. Let M = (M1 , . . . , Mp ), which takes values on {0, 1}p . Let µ be a dominating measure
for the distribution of X, and let ν represent the product measure between µ and the counting
measure on {0, 1}p . The full-data distribution is the joint distribution of X and M . We call its
density f with respect to ν the full-data density. In practice, the full-data distribution cannot be
recovered from sampled data, even with an infinite sample size.
An element m = (m1 , . . . , mp ) ∈ {0, 1}p is called a missingness pattern. Given m ∈ {0, 1}p we
define m̄ = 1p − m to be the indicator vector of observed variables, where 1p is a vector of ones of
length p. For each m, we define Xm = (Xj : mj = 1) to be the missing variables and Xm̄ = (Xj :
m̄j = 1) to be the observed variables, which have sample spaces Xm and Xm̄ , respectively. The
observed-data distribution is the distribution involving the observed variables and the missingness
R
indicators, which has density f (Xm̄ = xm̄ , M = m) = Xm f (X = x, M = m)µ(dxm ), where x ∈ X
represents a generic element of the sample space, and we define xm and xm̄ similarly as with the
random vectors.
An alternative way of representing the observed-data distribution is by introducing the mate-

2

rialized variables X ∗ = (X1∗ , . . . , Xp∗ ), where
(
Xj∗ ≡

Xj , if Mj = 0;
∗,

if Mj = 1;

and “∗” is a placeholder for missingness. The sample space Xj∗ of each Xj∗ is the union of {∗}
and the sample space Xj of Xj . The materialized variables contain all the observed information:
if Xj∗ = ∗ then Xj was not observed, and if Xj∗ = xj for any value xj 6= ∗ then Xj was observed
and Xj = xj . Given m ∈ {0, 1}p and xm̄ ∈ Xm̄ , we define x∗ ≡ x∗ (m, xm̄ ), such that x∗m̄ = xm̄
and x∗m = ∗, where ∗ is a vector with the appropriate number of ∗ symbols. For example, if
m = (1, 1, 0) and xm̄ = x3 , then x∗ = (∗, ∗, x3 ). The event X ∗ = x∗ (m, xm̄ ) is equivalent to
M = m and Xm̄ = xm̄ , which implies that the distribution of X ∗ is equivalent to the observed-data
distribution. Therefore, with some abuse of notation, the observed-data density can be written in
terms of X ∗ , that is f {X ∗ = x∗ (m, xm̄ )} ≡ f (Xm̄ = xm̄ , M = m). When there is no need to refer
to the m and xm̄ that define x∗ , we simply write f (X ∗ = x∗ ) to denote the observed-data density
evaluated at an arbitrary point.
In what follows we often write f (X = x, M = m) simply as f (X, M ), f (X ∗ = x∗ ) as f (X ∗ ),
and likewise for other expressions, provided that there is no ambiguity. For the sake of simplicity,
we use “f ” for technically different functions, but their actual interpretations should be clear from
the arguments passed to them. For example, we denote the missingness mechanism as f (M =
m|X = x), or simply f (M |X).

2.2

Non-Parametric Saturated Modeling

Since the true joint distribution of X and M cannot be identified from observed data alone, we
need to work under the assumption that the full-data distribution falls within a class defined by a
set of restrictions.
Definition 1 (Identifiability). Consider a class of full-data distributions FA defined by a set of
restrictions A. We say that the class FA is identifiable if there is a mapping from the set of
observed-data distributions to FA .
If we only require identifiability from a set of full-data distributions, two different observeddata distributions could map to the same full-data distribution. Robins (1997) introduced the
stricter concept of a class of full-data distributions being non-parametric saturated — also called
non-parametric identified (Vansteelandt et al. (2006); Daniels and Hogan (2008)).
Definition 2 (Non-parametric Saturation). Consider a class of full-data distributions FA defined
by a set of restrictions A. We say that the class FA is non-parametric saturated if there is a
one-to-one mapping from the set of observed-data distributions to FA .
The set A of restrictions, or identifiability assumptions, that define a NPS class allow us to build
a full-data distribution, say with density fA (X = x, M = m), from an observed-data distribution
3

with density f (Xm̄ = xm̄ , M = m), so that fA (Xm̄ = xm̄ , M = m) = f (Xm̄ = xm̄ , M = m), where
R
by definition fA (Xm̄ = xm̄ , M = m) = Xm fA (X = x, M = m)µ(dxm ). In terms of X ∗ , the NPS
property is expressed as fA (X ∗ ) = f (X ∗ ).
NPS is a desirable property, particularly for comparing inferences under different approaches
to handling nonignorable missing data. When two missing data models satisfy NPS, we can be
sure that any discrepancies in inferences are due entirely to the different assumptions on the nonidentifiable parts of the full-data distribution. In contrast, without NPS, it can be difficult to
disentangle what parts of the discrepancies are due to the identifying assumptions and what parts
are due to differing constraints on the observed-data distribution. Thus, NPS greatly facilitates
sensitivity analysis (Robins (1997)).
For a given m, we refer to the conditional distribution of the missing study variables given
the observed data as the missing-data distribution—also known as the extrapolation distribution
(Daniels and Hogan (2008))—with density f (Xm = xm |Xm̄ = xm̄ , M = m). These distributions
correspond to the non-identifiable parts of the full-data distribution. A NPS approach is equivalent
to a recipe for building these distributions from the observed-data distribution without imposing
constraints on the latter.
NPS models can be constructed in many ways. For example, in pattern mixture models, one
can use the complete-case missing-variable restriction (Little (1993)), which sets f (Xm = xm |Xm̄ =
xm̄ , M = m) = f (Xm = xm |Xm̄ = xm̄ , M = 0p ), for all m ∈ {0, 1}p . Although Little (1993)
considered parametric models for each f (Xm̄ = xm̄ |M = m), this does not have to be the case, and
therefore pattern mixture models can be NPS. Another example is the permutation missingness
model of Robins (1997), which for a specific ordering of the study variables assumes that the
probability of observing the kth variable depends on the previous study variables and the subsequent
observed variables. The group permutation missingness model of Robins (1997) is an analog of the
latter for groups of variables and is also NPS. Sadinle and Reiter (2017) introduced a missingness
mechanism where each variable and its missingness indicator are conditionally independent given
the remaining variables and missingness indicators, which leads to a NPS model. Tchetgen Tchetgen
et al. (2016) proposed a NPS approach based on discrete choice models. Finally, we note that MAR
models also can be NPS, as shown by Gill et al. (1997).

3

Sequential Identification Strategy

We consider the p variables as divided into K blocks, X = (X1 , . . . , XK ), where Xk = (Xtk−1 +1 , . . .
, Xtk ), which contains tk − tk−1 variables. As our results only concern the identification of full-data
∗)
distributions starting from an observed-data distribution, we assume that f (X ∗ ) = f (X1∗ , . . . , XK

is known. The identification strategy consists of specifying a sequence of assumptions A1 , . . . , AK ,
one for each block of variables, where each Ak allows us to identify the conditional distribution of
∗ ≡ (X ∗ , . . . , X ∗ ), and a carefully chosen subset
Xk and Mk given X<k ≡ (X1 , . . . , Xk−1 ), X>k
K
k+1

of the missingness indicators M<k ≡ (M1 , . . . , Mk−1 ) described below. We first provide a general

4

description of how A1 , . . . , AK allow us to identify parts of the full-data distribution in a sequential
manner, and then in Theorem 1 present the formal identification result.

3.1

Description

We now present the steps needed to implement the identification strategy. A graphical summary
of the procedure is provided in Figure 1.
∗ )f (X ∗ ). Consider an identifiability assumption A on
Step 1. Write f (X ∗ ) = f (X1∗ |X>1
1
>1
∗ that allows us to obtain a distribution with density
the distribution of X1 and M1 given X>1
∗ ) with the NPS property f (X ∗ |X ∗ ) = f (X ∗ |X ∗ ). From this we can define
fA1 (X1 , M1 |X>1
A1
1
>1
1
>1
∗ ) ≡ f (X , M |X ∗ )f (X ∗ ).
fA1 (X1 , M1 , X>1
1
1
A1
>1
>1

Step 2. Suppose we divide the t1 variables in X1 into two sets indexed by R1 and S1 , where R1 ∪
S1 = {1, . . . , t1 } and R1 ∩S1 = ∅; see Remark 1 below for discussion of why one might want to do so.
∗ )=
Let MR1 and MS1 be the corresponding missingness indicators. We can write fA1 (X1 , M1 , X>1
∗ )f (X , M , X ∗ ), where f (X , M , X ∗ ) = f (X ∗ |X , M , X ∗ )f (
fA1 (MS1 |X1 , MR1 , X>1
1
1
1
A1
R1
A1
R1
A1
R1
>1
>1
2
>2 A1
∗ ). Consider an identifiability assumption A on the distribution of X and M given
X1 , MR1 , X>2
2
2
2
∗
∗ that allows us to obtain a distribution with density f
X1 , MR1 and X>2
A≤2 (X2 , M2 |X1 , MR1 , X>2 )
∗ ) = f (X ∗ |X , M , X ∗ ). From this we can define
with the NPS property fA≤2 (X2∗ |X1 , MR1 , X>2
1
A1
R1
2
>2
∗
∗
∗
fA≤2 (X≤2 , MR1 , M2 , X>2
) ≡ fA≤2 (X2 , M2 |X1 , MR1 , X>2
)fA1 (X1 , MR1 , X>2
).

The notation fA≤2 emphasizes that the distribution relies on A1 and A2 .
∗ ). Let R ∪ S =
Step k + 1. At the end of the kth step we have fA≤k (X≤k , MRk−1 , Mk , X>k
k
k

{tk−1 + 1, . . . , tk } ∪ Rk−1 , Rk ∩ Sk = ∅, and MRk and MSk be the corresponding missingness indica∗
∗
∗ )=f
tors. We can write fA≤k (X≤k , MRk−1 , Mk , X>k
A≤k (MSk |X≤k , MRk , X>k )fA≤k (X≤k , MRk , X>k ),
∗
∗
∗
∗ ) = f
where fA≤k (X≤k , MRk , X>k
A≤k (Xk+1 |X≤k , MRk , X>k+1 )fA≤k (X≤k , MRk , X>k+1 ). Now, con-

sider an identifiability assumption Ak+1 on the distribution of Xk+1 and Mk+1 given X≤k , MRk and
∗
∗
that allows to obtain a distribution with density fA≤k+1 (Xk+1 , Mk+1 |X≤k , MRk , X>k+1
) with
X>k+1
∗
∗
∗ |X , M , X ∗
the NPS property fA≤k+1 (Xk+1
Rk
≤k
>k+1 ) = fA≤k (Xk+1 |X≤k , MRk , X>k+1 ). From this we

can define
∗
fA≤k+1 (X≤k+1 , MRk , Mk+1 , X>k+1
)
∗
∗
≡ fA≤k+1 (Xk+1 , Mk+1 |X≤k , MRk , X>k+1
)fA≤k (X≤k , MRk , X>k+1
).

Step K. For the final step, assumption AK is on the distribution of XK and MK given X<K
and MRK−1 . Following the previous generic identifying step, we obtain fA≤K (X, MRK−1 , MK ) ≡
fA≤K (XK , MK |X<K , MRK−1 )fA<K (X<K , MRK−1 ). We can obtain the implied distribution of the
study variables, with density fA≤K (X), from this last equation.
Remark 1. The main characteristic of the Rk subsets is that if an index does not appear in Rk−1 ,
then it cannot appear in Rk , unless it is one of tk−1 + 1, . . . , tk . The flexibility in the choosing of
5

f (X ∗ )
∗ ) }
A1 { f (X1∗ |X>1
∗ )
fA1 (X1 , M1 , X>1
∗ ) }
A2 { fA1 (X2∗ |X1 , MR1 , X>2
∗ )
fA≤2 (X≤2 , MR1 , M2 , X>2
∗ ) }
A3 { fA≤2 (X3∗ |X≤2 , MR2 , X>3

..
.
∗
∗) }
AK−1 { fA≤K−2 (XK−1
|X≤K−2 , MRK−2 , XK
∗)
fA≤K−1 (X≤K−1 , MRK−2 , MK−1 , XK
∗ |X
AK { fA≤K−1 (XK
≤K−1 , MRK−1 ) }

fA≤K (X, MRK−1 , MK )
Figure 1: Sequential identification strategy. We write Ak { fA≤k−1 (Xk∗ | . . . ) } to indicate that
assumption Ak is being used to obtain a conditional full-data density fA≤k (Xk , Mk | . . . ) from
fA≤k−1 (Xk∗ | . . . ).
these subsets gives flexibility in the setting up of the identifiability assumptions: different versions of
our identification approach can be obtained by making assumptions conditioning on different subsets
of the missingness indicators. As long as the Rk subsets satisfy Rk ⊆ {tk−1 + 1, . . . , tk } ∪ Rk−1 ,
Theorem 1 guarantees that the final full-data distribution is NPS.
Remark 2. The sequence for A1 , . . . , AK follows the order of the blocks X1 , . . . , XK . In many cases
these blocks may not have a natural order. Different orderings of the blocks lead to different sets of
assumptions, thereby leading to different final full-data distributions and implied distributions of the
study variables. To clarify this point, suppose that we have three blocks of variables: demographic
variables XD , income variables XI , and health-related variables XH . When XD is first in the order,
∗ ; likewise, when X is first in the
A1 concerns the distribution of XD and MD given XI∗ and XH
I
∗ and X ∗ . Similarly, A and A also
order, A1 concerns the distribution of XI and MI given XD
2
3
H

will change depending on the order of the variables, thereby implying changes in the final full-data
distribution.

6

3.2

Non-Parametric Saturation

The previous presentation makes it clear that the identifying assumptions A1 , . . . , AK allow us
∗ ) for each k < K,
to identify fA≤K (X, MRK−1 , MK ), and furthermore, fA≤k (MSk |X≤k , MRk , X>k

although each of these conditional densities remains unused after step k in the procedure. A
full-data distribution with density f˜A (X, M ) that encodes A1 , . . . , AK can be expressed as
≤K

f˜A≤K (X, M ) = fA≤K (X, MRK−1 , MK )f˜A≤K (MS1 , . . . , MSK−1 |X, MRK−1 , MK ),
where the second factor can be written as

QK−1 ˜
k=1 fA≤K (MSk |X, MS>k , MRK−1 , MK ), with S>k ≡

Sk+1 ∪ · · · ∪ SK−1 , and MS>k ≡ (MSk+1 , . . . , MSK−1 ). From the definition of the sets Sk and
Rk , it is easy to see that S>k ∪ RK−1 = Rk ∪ {tk + 1, . . . , tK−1 }, and therefore we can rewrite
f˜A≤K (MSk |X, MS>k , MRK−1 , MK ) = f˜A≤K (MSk |X, MRk , M>k ).
The sequential identification procedure does not identify any f˜A≤K (MSk |X, MRk , M>k ), but only
∗ ), that is, it identifies the distribution of M
fA≤k (MSk |X≤k , MRk , X>k
Sk given the variables X≤k , the
∗ , but not given the missing variables
missingness indicators MRk , and the materialized variables X>k
among X>k according to M>k . Nevertheless, the full specification of f˜A (MS |X, MR , M>k ) is
≤K

k

k

∗ )
irrelevant given that any such conditional distribution that agrees with fA≤k (MSk |X≤k , MRk , X>k

would lead to the same fA≤K (X). One such distribution is one where
∗
f˜A≤K (MSk |X, MRk , M>k ) = fA≤k (MSk |X≤k , MRk , X>k
), k < K,

(1)

that is, where the conditional distribution of MSk given X, MRk and M>k does not depend on
the missing variables among X>k according to M>k . This guarantees the existence of a full-data
distribution with density
f˜A≤K (X, M ) = fA≤K (X, MRK−1 , MK )

K−1
Y

∗
fA≤k (MSk |X≤k , MRk , X>k
),

(2)

k=1

which encodes the assumptions A1 , . . . , AK . Theorem 1 guarantees that this construction leads to
NPS full-data distributions.
Theorem 1. Let R1 , . . . , RK−1 be a sequence of subsets such that Rk ⊆ {tk−1 + 1, . . . , tk } ∪ Rk−1 .
Let A1 , . . . , AK be a sequence of identifying assumptions, with each Ak being an assumption on the
∗ , such that for a given density
conditional distribution of Xk and Mk given X<k , MRk−1 , and X>k
∗ ), it allows the construction of a density g (X , M |X , M
∗
g(Xk∗ |X<k , MRk−1 , X>k
Ak
Rk−1 , X>k ) with
k
k
<k
∗ ) = g(X ∗ |X , M
∗
the NPS property gAk (Xk∗ |X<k , MRk−1 , X>k
Rk−1 , X>k ). Then, given an observed<k
k
data density f (X ∗ ), there exists a full-data density f˜A≤K (X, M ) that encodes the assumptions
A1 , . . . , AK and satisfies the NPS property f˜A (X ∗ ) = f (X ∗ ).
≤K

Proof. We explained how assumptions A1 , . . . , AK along with the extra assumption in (1) lead to the
full-data density in (2). We now show the NPS property of (2). To start, we integrate (2) over the

7

missing variables in XK according to MK . Since none of the factors in
∗ )
X>k

QK−1
k=1

fA≤k (MSk |X≤k , MRk ,

depend on these missing variables, we obtain
∗
fA≤K−1 (X≤K−1 , MRK−1 , XK
)

K−1
Y

∗
)
fA≤k (MSk |X≤k , MRk , X>k

k=1

=

K−2
Y

∗
fA≤K−1 (X≤K−1 , MRK−1 , MSK−1 , XK
)

∗
= fA≤K−1 (X≤K−1 , MRK−2 , MK−1 , XK
)

k=1
K−2
Y

∗
)
fA≤k (MSk |X≤k , MRk , X>k

∗
).
fA≤k (MSk |X≤k , MRk , X>k

(3)

k=1

Similarly, we now integrate (3) over the missing variables in XK−1 according to MK−1 . Given
Q
∗
that none of the factors in K−2
k=1 fA≤k (MSk |X≤k , MRk , X>k ), depend on these missing variables,
∗ ) is constructed (see generic step k + 1 in
and given the way fA≤K−1 (X≤K−1 , MRK−2 , MK−1 , XK

Section 3.1), we obtain
∗
fA≤K−2 (X≤K−2 , MRK−2 , X>K−2
)

K−2
Y

∗
fA≤k (MSk |X≤k , MRk , X>k
).

k=1

These arguments and process can be repeated, sequentially integrating over the missing variables
in Xk according to Mk , k = K − 2, . . . , 1, finally obtaining the observed-data density f (X ∗ ).

3.3

Special Cases

It is worth describing two special sequential identification schemes that can be derived from our
general presentation. One is obtained when we take all Rk = {tk−1 + 1, . . . , tk } ∪ Rk−1 , Sk = ∅,
and therefore MRk = M≤k . In this case, each Ak+1 is on the distribution of Xk+1 and Mk+1
∗
, that is, the assumption conditions on the whole set of missingness
given X≤k , M≤k and X>k+1

indicators M≤k and not just on a subset of these. The other is obtained when we take all Rk = ∅,
Sk = {tk−1 + 1, . . . , tk }, and therefore MSk = Mk . In this case, each Ak+1 is on the distribution
∗
of Xk+1 and Mk+1 given X≤k and X>k+1
, that is, each assumption conditions on none of the

missingness indicators M≤k .

3.4

Connection with the Mechanisms of Robins (1997)

An important particular case of our sequential identification strategy is obtained when all MSk =
Mk and each Ak is taken to be a conditional MAR assumption, that is, when we assume that
∗ ) = f (M |X
∗
∗
f (Mk |X≤k−1 , Xk , X>k
k
≤k−1 , Xk , X>k ). Along with (1), this leads to the combined

assumption
∗
f (Mk |X, M>k ) = f (Mk |X<k , X≥k
).

8

(4)

The missingness mechanism derived from this approach corresponds to the group permutation
missingness of Robins (1997). When each block contains only one variable, it corresponds to the
permutation missingness mechanism of Robins (1997). If the ordering of the variables or blocks of
variables is regarded as temporal, as in a longitudinal study or a survey that asks questions in a
fixed sequence, Robins (1997) interpreted (4) as follows: the nonresponse propensity at the current
time period depends on the values of study variables in the previous time periods, whether observed
or not, but not on what is missing in the present and future time periods.
If the order of the blocks of variables was reversed, that is, if A1 was on the distribution of XK
∗ , A was on the distribution of X
∗
and MK given X<K
2
K−1 and MK−1 given X<K−1 and XK , and

so on, then we would have the following interpretation: the nonresponse propensity at the current
time period depends on the values of study variables in the future time periods, whether observed
or not, but not on what is missing in the present and past time periods. This interpretation is
arguably easier to explain in the context of respondents answering a questionnaire. The nonresponse
propensity for question t can depend on the respondent’s answers to questions that appear later in
the questionnaire and to questions that she has already answered, but not on the information that
she has not revealed.

4

Applications

4.1

Sequential Identification for Two Categorical Variables

Consider two categorical random variables X1 ∈ X1 = {1, . . . , I} and X2 ∈ X2 = {1, . . . , J}. Let
M1 and M2 be their missingness indicators. Let P denote the joint distribution of (X1 , X2 , M1 , M2 ).
The observed-data distribution corresponds to the probabilities
πij00 ≡ P(X1∗ = i, X2∗ = j) = P(X1 = i, X2 = j, M1 = 0, M2 = 0),
πi+01 ≡ P(X1∗ = i, X2∗ = ∗) = P(X1 = i, M1 = 0, M2 = 1),
π+j10 ≡ P(X1∗ = ∗, X2∗ = j) = P(X2 = j, M1 = 1, M2 = 0),
π++11 ≡ P(X1∗ = ∗, X2∗ = ∗) = P(M1 = 1, M2 = 1),
for i ∈ X1 , j ∈ X2 . We seek to construct a full-data distribution PA≤2 (X1 , X2 , M1 , M2 ) from the
observed-data distribution P(X1∗ , X2∗ ) by imposing some assumptions A1 and A2 . The goal is a
full-data distribution such that PA≤2 (X1∗ , X2∗ ) = P(X1∗ , X2∗ ), that is, we want PA≤2 to be NPS.
To use the general identification strategy presented in Section 3 we define each variable as its
own block. With only two variables, set R1 can be either R1 = {1} or R1 = ∅. We present two
examples below corresponding to these two options.
Example 1. We first consider R1 = {1}, S1 = ∅, and the following identifying assumptions:
A1 : X1 ⊥⊥ M1 |X2∗ ; and A2 : X2 ⊥
⊥ M2 |M1 , X1 .
Under A1 , PA1 (X1 , M1 |X2∗ ) = PA1 (X1 |X2∗ )PA1 (M1 |X2∗ ) = P(X1 |X2∗ , M1 = 0)P(M1 |X2∗ ), where

9

P(X1 |X2∗ , M1 = 0) and P(M1 |X2∗ ) are identified from the observed data distribution. When X2∗ =
j 6= ∗, P(X1 = i|X2∗ = j, M1 = 0) = P(X1 = i|X2 = j, M1 = 0, M2 = 0) = πij00 /π+j00 , and
P(M1 = m1 |X2∗ = j) = P(M1 = m1 |X2 = j, M2 = 0) = π+jm1 0 /π+j+0 . Similarly, when X2∗ = ∗ we
find P(X1 = i|X2∗ = ∗, M1 = 0) = πi+01 /π++01 and P(M1 = m1 |X2∗ = ∗) = π++m1 1 /π+++1 . Since
P(X2∗ ) can be obtained from the observed-data distribution as P(X2 = j, M2 = 0) = π+j+0 when
X2∗ = j 6= ∗, and as P(M2 = 1) = π+++1 when X2∗ = ∗, using PA1 (X1 , M1 |X2∗ ) we obtain a joint distribution for (X1 , M1 , X2∗ ) that relies on A1 , defined as PA1 (X1 , M1 , X2∗ ) ≡ PA1 (X1 , M1 |X2∗ )P(X2∗ ).
Note that PA1 can be written as an explicit function of the observed-data distribution.
We now use PA1 and identifying assumption A2 to obtain PA≤2 (X2 , M2 |X1 , M1 ). From the
definition of X2∗ , PA1 (X1 , M1 , X2∗ ) can be written as PA1 (X1 , M1 , X2 , M2 = 0) when X2∗ 6= ∗ and
PA1 (X1 , M1 , M2 = 1) when X2∗ = ∗. From this we can obtain
PA1 (M2 = 1|X1 , M1 ) =

PA1 (X1 , M1 , M2 = 1)
P
,
PA1 (X1 , M1 , M2 = 1) + x2 ∈X2 PA1 (X1 , M1 , X2 = x2 , M2 = 0)

and

PA1 (X1 , M1 , X2 , M2 = 0)
.
x2 ∈X2 PA1 (X1 , M1 , X2 = x2 , M2 = 0)

PA1 (X2 |X1 , M1 , M2 = 0) = P

We then obtain PA≤2 (X2 , M2 |X1 , M1 ) = PA≤2 (X2 |X1 , M1 )PA≤2 (M2 |X1 , M1 ) = PA1 (X2 |X1 , M1 ,
M2 = 0)PA1 (M2 |X1 , M1 ), which gives us a way to obtain PA≤2 (X2 , M2 |X1 , M1 ) as a function of
the distribution PA1 , which in turn is a function of the observed-data distribution. The final fulldata distribution is obtained as PA≤2 (X1 , M1 , X2 , M2 ) ≡ PA≤2 (X2 , M2 |X1 , M1 )PA1 (X1 , M1 ), where
PA1 (X1 , M1 ) can be obtained from PA1 . After some algebra we find
PA≤2 (X1 = i, X2 = j, M1 = m1 , M2 = m2 ) =

πij00
π+j00 π+jm1 0
P il00
( l ππ+l00
π+lm1 0 )m2



πi+01
π++m1 1
π++01

m2
.

It is easy to see that PA≤2 is NPS, that is PA≤2 (X1∗ , X2∗ ) = P(X1∗ , X2∗ ). From the final distribution
PA≤2 (X1 , X2 , M1 , M2 ) we can now obtain
PA≤2 (X1 = i, X2 = j)
π

ij00
πij00
πij00
πi+01 π+j00 π+j10
P
= πij00 + πi+01
+ π+j10
+ π++11
,
il00
πi+00
π+j00
π++01 l ππ+l00
π+l10

(5)

which is the distribution of inferential interest.
In closing this example, we stress that the final full-data distribution is not invariant to the
order in which the blocks of variables appear in the sequence of assumptions. From expression (5)
it is clear that the final distribution of the study variables would be different had we identified a
distribution for (X1∗ , X2 , M2 ) first. Indeed, if we were to follow the steps in the previous example
but reversing the order of the variables, then we would be assuming that X2 ⊥⊥ M2 |X1∗ and X1 ⊥
⊥
M1 |M2 , X2 , which are different from A1 and A2 in this example.

10

Example 2. We now consider R1 = ∅, S1 = {1}, and the identifying assumptions B1 : X1 ⊥
⊥
M1 |X2∗ , and B2 : X2 ⊥
⊥ M2 |X1 .
Assumption B1 is the same as A1 in Example 1, and so PB1 (X1 , M1 , X2∗ ) = PA1 (X1 , M1 , X2∗ ).
Assumption B2 is made conditioning only on X1 , so we need to marginalize over M1 to obtain
PB1 (X1 , X2∗ ) = PB1 (X1 , M1 = 0, X2∗ ) + PB1 (X1 , M1 = 1, X2∗ ):
PB1 (X1 = i, X2∗ = j) = PB1 (X1 = i, X2 = j, M2 = 0) =
PB1 (X1 = i, X2∗ = ∗) = PB1 (X1 = i, M2 = 1) =

πij00
π+j+0 ,
π+j00

πi+01
π+++1 .
π++01

From this we can obtain
PB1 (M2 = 1|X1 ) =
and

PB1 (X1 , M2 = 1)
P
,
PB1 (X1 , M2 = 1) + x2 ∈X2 PB1 (X1 , X2 = x2 , M2 = 0)
PB1 (X1 , X2 , M2 = 0)
.
x2 ∈X2 PB1 (X1 , X2 = x2 , M2 = 0)

PB1 (X2 |X1 , M2 = 0) = P

Using assumption B2 , we obtain PB≤2 (X2 , M2 |X1 ) = PB≤2 (X2 |X1 )PB≤2 (M2 |X1 ) = PB1 (X2 |X1 ,
M2 = 0)PB1 (M2 |X1 ). From this we obtain PB≤2 (X1 , X2 , M2 ) ≡ PB1 (X1 )PB≤2 (X2 , M2 |X1 ) as
PB≤2 (X1 = i, X2 = j, M2 = m2 ) = P
( l

πij00
π+j00 π+j+0
πil00
m2
π+l00 π+l+0 )



πi+01
π+++1
π++01

m2
.

Marginalizing over M2 , we get
π

ij00
πij00
πi+01 π+j00 π+j+0
P πil00
+ π+++1
.
PB≤2 (X1 = i, X2 = j) = π+j+0
π+j00
π++01 l π+l00 π+l+0

Assumptions B1 and B2 are enough to identify PB≤2 (X1 , X2 , M2 ), and thereby a distribution
of the study variables PB≤2 (X1 , X2 ). Although irrelevant for obtaining the distribution of the study
variables, it is worth noticing that B1 and B2 do not allow us to fully identify PB≤2 (M1 |X1 , X2 , M2 ).
From PB1 (X1 , M1 , X2∗ ) we have PB1 (M1 |X1 , X2 , M2 = 0) = P(M1 |X2 , M2 = 0) and PB1 (M1 |X1 , M2 =
1) = P(M1 |M2 = 1), but PB≤2 (M1 |X1 , X2 , M2 = 1) remains unidentified. A full-data distribution
P̃B≤2 becomes identified under the extra assumption P̃B≤2 (M1 |X1 , X2 , M2 = 1) = P̃B≤2 (M1 |X1 , M2 =
1), which corresponds to the extra assumption in (1).
The set of assumptions that we used in this example can be summarized in terms of the missingness mechanism P̃B≤2 (M1 , M2 |X1 , X2 ) = P̃B≤2 (M1 |X1 , X2 , M2 )P̃B≤2 (M2 |X1 , X2 ), where P̃B≤2 (M1 |
X1 , X2 , M2 = 1) = P(M1 |M2 = 1), P̃B≤2 (M1 |X1 , X2 , M2 = 0) = P(M1 |X2 , M2 = 0), and P̃B≤2 (M2 |
X1 , X2 ) = PB1 (M2 |X1 ). This corresponds to the permutation missingness (PM) mechanism of
Robins (1997).
As in Example 1, the full-data distribution changes when we modify the order in which the

11

blocks of variables appear in the identifying assumptions. Changing the order of the variables in
this example would correspond to making the assumptions X2 ⊥⊥ M2 |X1∗ and X1 ⊥⊥ M1 |X2 .

4.2

Sequential Identification for Partially Ignorable Mechanisms

Harel and Schafer (2009) introduced different notions of the missing data being partially ignorable.
In particular, in some scenarios one may think that the missingness is ignorable for some, but not
for all the variables. For example, consider a survey with two blocks of items XS and XN , which
contain responses to sensitive and non-sensitive questions, respectively. Given the nature of these
variables, one may think that the missingness among the XN variables could be ignored, but not
among XS . Our sequential identification procedure can be used to guarantee identifiability under
such partially ignorable mechanisms. Our goal here is to show that we can identify a NPS full-data
distribution f˜A (XS , XN , MS , MN ) with the property that the missingness mechanism for XN is
≤2

partially MAR given MS (Harel and Schafer (2009)), that is,
∗
f˜A≤2 (MN |XN , XS , MS ) = f˜A≤2 (MN |XN
, XS∗ ),

(6)

while f˜A≤2 (MS |XN , XS ) is determined by some nonignorable assumption. As before, we consider
∗ ) to be known.
f (XS∗ , XN

Following our sequential identification procedure, we first consider an identifying assumption
for the distribution of XN and MN given XS∗ . We use the conditional MAR assumption:
∗
A1 : f (MN |XN , XS∗ ) = f (MN |XN
, XS∗ ).

(7)

This assumption guarantees the existence of a distribution of the variables XN , MN , and XS∗ with
∗ , X ∗ )f (X |X ∗ )f (X ∗ ),
density fA1 (XN , MN , XS∗ ) = fA1 (MN |XN , XS∗ )fA1 (XN , XS∗ ) ≡ f (MN |XN
N
S
S
S A1
∗ |X ∗ ) as described in page 28 of Robins (1997).
where fA1 (XN |XS∗ ) can be obtained from f (XN
S

Taking R1 = ∅ in our identification procedure, we can now consider any identifying assumption,
say A2 , for the distribution of XS and MS given XN that allows us to obtain fA≤2 (XS , MS |XN ) with
the NPS property fA≤2 (XS∗ |XN ) = fA1 (XS∗ |XN ). For example, A2 could come from one of the approaches mentioned in Section 2.2. We then define fA≤2 (XN , XS , MS ) ≡ fA≤2 (XS , MS |XN )fA1 (XN ).
To fully identify a full-data distribution f˜A (XS , XN , MS , MN ) that encodes assumptions A1
≤2

and A2 , we further require the conditional missingness mechanism f˜A≤2 (MN |XN , XS , MS ). Under
the extra assumption
f˜A≤2 (MN |XN , XS , MS ) = f˜A≤2 (MN |XN , XS∗ ),

(8)

∗ , X ∗ )f
and then using A1 we have identified a full-data distribution with density f (MN |XN
S A≤2 (XN ,

XS , MS ). The NPS property of this distribution is guaranteed by Theorem 1.
A possibility for the A2 assumption could come from the itemwise conditionally independent
nonresponse (ICIN) mechanism of Sadinle and Reiter (2017), which is NPS. Denoting XS =

12

(XS1 , . . . , XSpS ), the ICIN assumption for XS and MS given XN can be written as
XSj ⊥
⊥ MSj |XS(−j) , MS(−j) , XN ; j = 1, . . . , pS ;

(9)

where XS(−j) is the vector obtained from removing the jth entry of XS , likewise for MS(−j) . Our
sequential identification procedure guarantees that assumptions in (6) and (9) jointly identify a
NPS full-data distribution.
Example 3. For simplicity, consider XN = X1 and XS = (X2 , X3 ). The observed-data density can
be written as the product of the density of the observed variables given each missingness pattern times
the probability of the missingness pattern, that is f (Xm̄ , M = m) = fm (Xm̄ )πm , which for three
variables is given by f000 (X1 , X2 , X3 )π000 , f100 (X2 , X3 )π100 , f010 (X1 , X3 )π010 , . . . , f011 (X1 )π011 ,
and π111 . Assumption A1 in (7) in this case becomes A1 :

X1 ⊥⊥ M1 |X2∗ , X3∗ , which for all

x2 ∈ X2 and x3 ∈ X3 can be expanded as X1 ⊥⊥ M1 |M2 = 0, M3 = 0, X2 = x2 , X3 = x3 ; X1 ⊥
⊥ M1 |M2 = 1, M3 = 0, X3 = x3 ; X1 ⊥⊥ M1 |M2 = 0, M3 = 1, X2 = x2 ; and X1 ⊥⊥ M1 |M2 =
1, M3 = 1. Using A1 and the observed-data distribution we can obtain fA1 (X1 , M1 |X2∗ , X3∗ ) as
fA1 (X1 , M1 |X2∗ , X3∗ ) = fA1 (X1 |X2∗ , X3∗ )fA1 (M1 |X2∗ , X3∗ ) = f (X1 |M1 = 0, X2∗ , X3∗ )f (M1 |X2∗ , X3∗ ),
where f (X1 |M1 = 0, X2∗ , X3∗ ) is obtained from
f (X1 |M1 = 0, M2 = 0, M3 = 0, X2 , X3 ) = f000 (X1 , X2 , X3 )/f000 (X2 , X3 ),
f (X1 |M1 = 0, M2 = 1, M3 = 0, X3 ) = f010 (X1 , X3 )/f010 (X3 ),
f (X1 |M1 = 0, M2 = 0, M3 = 1, X2 ) = f001 (X1 , X2 )/f001 (X2 ),
f (X1 |M1 = 0, M2 = 1, M3 = 1) = f011 (X1 ); and f (M1 |X2∗ , X3∗ ) from
f (M1 = m1 |M2 = M3 = 0, X2 , X3 ) ∝ [f000 (X2 , X3 )π000 ]I(m1 =0) [f100 (X2 , X3 )π100 ]I(m1 =1) ,
f (M1 = m1 |M2 = 1, M3 = 0, X3 ) ∝ [f010 (X3 )π010 ]I(m1 =0) [f110 (X3 )π110 ]I(m1 =1) ,
f (M1 = m1 |M2 = 0, M3 = 1, X2 ) ∝ [f001 (X2 )π001 ]I(m1 =0) [f101 (X2 )π101 ]I(m1 =1) ,
f (M1 = m1 |M2 = 1, M3 = 1) ∝ [π011 ]I(m1 =0) [π111 ]I(m1 =1) .
From this we can define fA1 (X1 , M1 , X2∗ , X3∗ ) ≡ fA1 (X1 , M1 |X2∗ , X3∗ )f (X2∗ , X3∗ ), where f (X2∗ , X3∗ ) is
obtained from f (M2 = 0, M3 = 0, X2 , X3 ) = f000 (X2 , X3 )π000 + f100 (X2 , X3 )π100 , f (M2 = 1, M3 =
0, X3 ) = f010 (X3 )π010 + f110 (X3 )π110 , f (M2 = 0, M3 = 1, X2 ) = f001 (X2 )π001 + f101 (X2 )π101 , and
f (M2 = 1, M3 = 1) = π011 + π111 .
We now incorporate the ICIN assumption for the distribution of (X2 , X3 , M2 , M3 ) given X1 .
We have A2 : {X2 ⊥
⊥ M2 |X3 , M3 , X1 ; and X3 ⊥⊥ M3 |X2 , M2 , X1 }. The identification results
of Sadinle and Reiter (2017) guarantee that assumption A2 leads to a conditional distribution
fA≤2 (X2 , X3 , M2 , M3 |X1 ) with the NPS property fA≤2 (X2∗ , X3∗ |X1 ) = fA1 (X2∗ , X3∗ |X1 ), where fA1 (X2∗ ,
X3∗ |X1 ) can be obtained easily from fA1 (X1 , M1 , X2∗ , X3∗ ). Section 5.1 of Sadinle and Reiter (2017)
provides explicit formulae for the full-data distribution under the ICIN assumption as a function

13

of the observed-data distribution, in the case of two variables. We can use those formulae here
with fA1 (X2∗ , X3∗ |X1 ) to obtain conditional ICIN full-data distributions that depend on X1 . To simplify the notation below we replace fA1 by g, and fA≤2 by h, and we denote gm2 m3 (X2 , X3 |X1 ) ≡
g(X2 , X3 |X1 , M2 = m2 , M3 = m3 ) and hm2 m3 (X2 , X3 |X1 ) ≡ h(X2 , X3 |X1 , M2 = m2 , M3 = m3 ).
Following the formulae of Sadinle and Reiter (2017) we obtain h00 (X2 , X3 |X1 ) = g00 (X2 , X3 |X1 ),
h01 (X2 , X3 |X1 ) = g00 (X2 , X3 |X1 )g01 (X2 |X1 )/g00 (X2 |X1 ), h10 (X2 , X3 |X1 ) = g00 (X2 , X3 |X1 )g10 (X3 |
X1 )/g00 (X3 |X1 ),
h11 (X2 , X3 |X1 ) ∝

g00 (X2 , X3 |X1 )g01 (X2 |X1 )g10 (X3 |X1 )
,
g00 (X2 |X1 )g00 (X3 |X1 )

and h(M2 = m2 , M3 = m3 |X1 ) = g(M2 = m2 , M3 = m3 |X1 ).
Putting everything together, we obtain
fA≤2 (X1 , X2 , X3 , M2 = m2 , M3 = m3 ) = hm2 m3 (X2 , X3 |X1 )g(M2 = m2 , M3 = m3 , X1 ),
from which we can obtain the distribution of the study variables fA≤2 (X1 , X2 , X3 ). A full-data
density f˜A≤2 (X1 , X2 , X3 , M1 , M2 , M3 ) becomes identified under the extra assumption (8). This
distribution therefore encodes the partial ignorability assumption (6) for the missingness in X1 and
the ICIN assumption (9) for (X2 , X3 , M2 , M3 ) given X1 .

4.3

Usage in Sensitivity Analysis

To illustrate how this approach could be used for sensitivity analysis, we use data related to the
1991 plebiscite where Slovenians voted for independence from Yugoslavia (Rubin et al. (1995)). The
data come from the Slovenian public opinion survey, which contained the questions: XI : are you
in favor of Slovenia’s independence? XS : are you in favor of Slovenia’s secession from Yugoslavia?
XA : will you attend the plebiscite? We call these the Independence, Secession, and Attendance
questions, respectively. The possible responses to each of these were yes, no, and don’t know.
We follow Rubin et al. (1995) in treating don’t know as missing data.
We use the missingness mechanism presented in Example 3, and compare it with an ignorable
approach, a pattern mixture model (PMM) under the complete-case missing-variable restriction
(Little (1993)), and the ICIN approach of Sadinle and Reiter (2017) which here corresponds to
assuming Xj ⊥
⊥ Mj |X−j , M−j ; j = I, S, A. The Attendance question is arguably the less sensitive
of the three questions studied here, so it seems reasonable to consider a partially ignorable mechanism where the nonresponse for XA is ignorable given MI and MS , as in (6), and the nonresponse
for XI and XS satisfy the ICIN assumption conditioning on XA , as in (9) in Example 3. Nothing
prevents us from using this approach exchanging the roles of the variables, so we also consider two
other partially ignorable missingness mechanisms, depending on whether we take the nonresponse
for XI or for XS as ignorable.
To implement these approaches, we first use a Bayesian approach to estimate the observed-

14

data distribution. The observed data can be organized in a three-way contingency table with cells
corresponding to each element of {yes, no, don’t know}3 , as presented in Rubin et al. (1995).
Treating these data as a random sample from a multinomial distribution, we take a conjugate prior
distribution for the cell probabilities: symmetric Dirichlet with parameter 1/27. We take 5,000
draws from the posterior distribution of the observed-data distribution, and for each of these we
apply the formulae presented in Example 3 to obtain posterior draws of the full-data distribution
under each of the three partially ignorable mechanisms. We use a similar approach to obtain
posterior draws of the full-data distribution under ICIN, PMM, and ignorability. For each of the
approaches we then obtain draws of the implied probabilities for the items.
Figure 2 displays 5,000 draws from the joint posterior distribution of P(Independence = yes,
Attendance = yes) and P(Attendance = no) under each of the six missingness mechanisms considered here. Despite the fact that all of these approaches agree in their fit to the observed data,
we obtain quite different inferences under each assumption. When inferences are so sensitive to the
identifying assumptions, perhaps the most honest way to proceed is to report all the results under
all assumptions deemed plausible given the context.
Ignorability

Pattern Mixture

P(Attendance=NO)

0.06

0.04

0.08

0.06

0.04

0.02

0.02
0.82

0.84

0.86

0.88

0.90

0.10

0.84

0.86

0.88

0.90

0.82

P(Attendance=NO)

0.08

0.06

0.04

0.02
0.90

P(Independence=YES, Attendance=YES)

0.86

0.88

0.90

PIM - Secession
0.10

0.08

0.06

0.04

0.08

0.06

0.04

0.02

0.02
0.88

0.84

P(Independence=YES, Attendance=YES)

0.10

0.86

0.04

PIM - Independence

PIM - Attendance

0.84

0.06

P(Independence=YES, Attendance=YES)

P(Independence=YES, Attendance=YES)

0.82

0.08

0.02
0.82

P(Attendance=NO)

P(Attendance=NO)

0.08

0.10

P(Attendance=NO)

0.10

0.10

P(Attendance=NO)

ICIN

0.82

0.84

0.86

0.88

0.90

P(Independence=YES, Attendance=YES)

0.82

0.84

0.86

0.88

0.90

P(Independence=YES, Attendance=YES)

Figure 2: Samples from joint posterior distributions of P(Independence = Yes, Attendance =
Yes) and P(Attendance = No). Pattern mixture model under the complete-case missing-variable
restriction. The three partially ignorable missingness (PIM) mechanisms correspond to which
variable we take as having ignorable missingness. The plebiscite results are represented by .
These are shown to illustrate differences between approaches and not to declare better vs worse
assumptions for these data.
15

5

Discussion

The sequential identification procedure can be set up in many different ways, leading to different
possibilities for constructing nonignorable missingness mechanisms. The main differences among
these possibilities lie in the assumptions about how missingness from any one block of variables
affects missingness in other blocks, as illustrated in the examples of Section 3.3 and Section 4. In
general, the procedure allows for different levels of dependence on missing variables while ensuring
non-parametric saturated models, which provides a useful framework for sensitivity analysis.
Although we presented our identification strategy for arbitrary K blocks of variables, we expect
that in practice most analysts would use K = 2 blocks when the variables do not naturally fall
into ordered blocks of variables. For example, analysts may want to partition variables into one
group that requires careful assessment of sensitivity to various missingness mechanisms, such as
outcome variables in regression modeling with high fractions of missingness, and a second group
that can be treated with generic missingness mechanisms like conditional MAR, such as covariates
with low fractions of missingness. These cases require partially ignorable mechanisms like those in
Section 4.2. Another scenario where two blocks naturally might arise is when analysts have prior
information on how the missingness occurs for a set of variables but not for the rest. Related,
analysts might have auxiliary information on the marginal distribution of a few variables, perhaps
from a census or other surveys, that enable the identification of mechanisms where the probability
of nonresponse for a variable depends explicitly on the variable itself (Hirano et al. (2001); Deng
et al. (2013)).
Our sequential identification procedure provides a constructive way of obtaining estimated fulldata distributions from estimated observed-data distributions while ensuring non-parametric saturated models. To implement these approaches in practice, one needs sufficient numbers of observations for each missing data pattern, so as to allow accurate non-parametric estimation of the
observed-data distribution. This can be challenging in modest-sized samples with large numbers of
variables. Of course, this is the case with most methods for handling missing data, including pattern mixture models. In such cases, one may have to sacrifice non-parametric saturated modeling
of the observed data in favor of parametric models.

Acknowledgements
This research was supported by the grant NSF SES 11-31897.

References
Daniels, M. J. and Hogan, J. W. (2008). Missing Data in Longitudinal Studies: Strategies for Bayesian Modeling and
Sensitivity Analysis. Chapman and Hall/CRC, Boca Raton, Florida.
Deng, Y., Hillygus, D. S., Reiter, J. P., Si, Y. and Zheng, S. (2013). Handling attrition in longitudinal studies: the
case for refreshment samples. Statist. Sci. 28, 238-256.

16

Gill, R. D., van der Laan, M. J. and Robins, J. M. (1997). Coarsening at random: characterizations, conjectures,
counter-examples. In Proceedings of the First Seattle Symposium in Biostatistics: Survival Analysis, pages
255-294. Edited by Lin, D. Y. and Fleming, T. R.”
Harel, O. and Schafer, J. L. (2009). Partial and latent ignorability in missing-data problems. Biometrika 96, 37-50.
Hirano, K., Imbens, G. W., Ridder, G. and Rubin, D. B. (2001). Combining panel data sets with attrition and
refreshment samples. Econometrica 69, 1645-1659.
Little, R. J. A. (1993). Pattern-mixture models for multivariate incomplete data. J. Amer. Statist. Assoc. 88, 125-134.
Little, R. J. A. and Rubin, D. B. (2002). Statistical Analysis with Missing Data. Wiley, Hoboken, New Jersey, second
edition.
Robins, J. M. (1997). Non-response models for the analysis of non-monotone non-ignorable missing data. Statist.
Med. 16, 21-37.
Rubin, D. B. (1976). Inference and missing data. Biometrika 63, 581-592.
Rubin, D. B., Stern, H. S. and Vehovar, V. (1995). Handling “Don’t know” survey responses: The case of the
Slovenian plebiscite J. Amer. Statist. Assoc. 90, 822–828.
Sadinle, M. and Reiter, J. P. (2017). Itemwise conditionally independent nonresponse modeling for incomplete
multivariate data. Biometrika, forthcoming. ArXiv: 1609.00656
Tchetgen Tchetgen, E. J., Wang, L. and Sun, B. (2016). Discrete choice models for nonmonotone nonignorable
missing data: identification and inference. ArXiv: 1607.02631
Vansteelandt, S., Goetghebeur, E., Kenward, M. G. and Molenberghs, G. (2006). Ignorance and uncertainty regions
as inferential tools in a sensitivity analysis. Statist. Sinica 16, 953-979.
Zhou, Y., Little, R. J. A. and Kalbfleisch, J. D. (2010). Block-conditional missing at random models for missing data.
Statist. Sci. 25, 517-532.

17

