--- abstract: 'In studies of discrimination, researchers often seek to estimate a causal effect of race or gender on outcomes. For example, in the criminal justice context, one might ask whether arrested individuals would have been subsequently charged or convicted had they been a different race. It has long been known that such counterfactual questions face measurement challenges related to omitted-variable bias, and conceptual challenges related to the definition of causal estimands for largely immutable characteristics. Another concern, which has been the subject of recent debates, is post-treatment bias: many studies of discrimination condition on apparently intermediate outcomes, like being arrested, that themselves may be the product of discrimination, potentially corrupting statistical estimates. There is, however, reason to be optimistic. By carefully defining the estimand—and by considering the precise timing of events—we show that a primary causal quantity of interest in discrimination studies can be estimated under an ignorability condition that is plausible in many observational settings. We illustrate these ideas by analyzing both simulated data and the charging decisions of a prosecutor's office in a large county in the United States.' author: - | Johann Gaebler\ Stanford University\ - | William Cai\ Stanford University\ - | Guillaume Basse\ Stanford University - | Ravi Shroff\ New York University - | Sharad Goel\ Stanford University\ - | Jennifer Hill\ New York University\ bibliography: - 'refs.bib' title: ' A Causal Framework for Observational Studies of Discrimination[^1]' --- To assess the role of race or gender in decision making, researchers often examine disparities between groups after adjusting for relevant factors. For example, to measure racial discrimination in lending decisions, one might estimate race-specific approval rates after adjusting for creditworthiness, typically via a regression model. This simple statistical strategy—sometimes called benchmark analysis—has been used to study discrimination in a wide variety of domains, including banking [@munnell1996mortgage], employment [@berg2002measuring], education [@baum2005gender], healthcare [@balsa2005testing], housing [@greenberg2016discrimination; @edelman2014digital], and criminal justice [@rehavi2014racial; @fryer2019empirical; @gelman2007analysis; @ayres2002; @macdonald2017analysis]. The results of benchmark analyses are often framed in causal terms (e.g., as an effect of race on outcomes), but it is well understood that such an approach suffers from at least three significant statistical challenges when used to estimate causal quantities. First, at a conceptual level, it is unclear how best to rigorously define causal estimands of interest when the treatment is race, gender, or another largely immutable trait. Second, estimates can be plagued by omitted-variable bias if one does not appropriately adjust for all relevant covariates. Third—and the focus of our paper—there are worries that estimates are corrupted by post-treatment bias when one adjusts for covariates or restricts to samples of individuals determined downstream from the treatment variable. This concern, in particular, has raised doubts about the reliability of the literature on police discrimination, where many studies rely on administrative stop records, and hence implicitly condition on officers stopping an individual, an action that itself is likely discriminatory [@knox-2019; @heckman2020comment]. Here we present a causal framework for conceptualizing and estimating a measure of discrimination that is suitable for many applied problems. Our framing specifically addresses concerns about post-treatment bias. To do so, we first define a causal quantity—the second-stage sample average treatment effect, or $\cdes$—which closely maps to the legal notion of disparate treatment. We then introduce the idea of subset ignorability, which formally justifies the use of benchmark analysis to estimate the $\cdes$, and discuss settings in which this condition is likely to (approximately) hold. We illustrate these ideas by analyzing synthetic data, as well as a detailed dataset of prosecutorial charging decisions for approximately 20,000 felony arrests in a major U.S.county. By developing this statistical foundation, we hope to place discrimination studies on more solid theoretical footing. A Motivating Example {#sec:motex} ==================== Consider the problem of measuring racial discrimination in prosecutorial charging decisions. After an individual has been arrested, prosecutors in the district attorney's office read the arresting officer's incident report and then decide whether or not to press charges. For simplicity, suppose prosecutors only have access to the incident report—and to no other information—when making their decisions. We allow for the possibility that the arrest decision that preceded the charging decision may have suffered from racial discrimination in complex ways that cannot be inferred from the incident reports themselves. Finally, suppose that a researcher has access to these incident reports for arrested individuals, but, importantly, not to any data on individuals that officers considered but ultimately decided against arresting. What, if anything, might one hope to discover about racial discrimination in charging decisions in light of the fact that the people about whom the prosecutor makes charging decisions have been selected—that is, arrested—not randomly, but rather in ways that likely depended on their race? The first challenge is to rigorously define causal estimands of interest. The inherent difficulty is captured by the statistical refrain "no causation without manipulation" [@holland1986statistics], since it is often unclear what it means to alter attributes like race and gender [@sekhonneyman]. One common maneuver is to instead consider the causal effect of *perceived* attributes (e.g., perceived race or perceived gender), which ostensibly can be manipulated—for example, by changing the name listed on an employment application [@bertrand2004emily], or by masking an individual's appearance [@goldin2000orchestrating; @grogger2006testing; @OPP]. In our case, one might imagine a hypothetical experiment in which explicit mentions of race in the incident report are altered (e.g., replacing "white" with "Black"). The causal effect is then, by definition, the difference in charging rates between those cases in which arrested individuals were randomly described (and hence may be perceived) as "Black" and those in which they were randomly described as "white." This conceptualization of discrimination conforms to one common causal understanding of discrimination used, for example, in audit studies. This framing also maps closely to the legal notion of disparate treatment, a form of discrimination in which actions are motivated by animus or otherwise discriminatory intent [@goel2017combatting]. While researchers have carried out such audit studies—including in the case of prosecutorial charging decisions [@chohlaswood2020blind]—it is often infeasible to study important policy questions through randomized experiments. In the absence of a controlled experiment, one can in theory identify this type of causal estimand from purely observational data by comparing charging rates across pairs of cases that are identical in all aspects other than the stated race of the arrested individual.[^2] That strategy, which mimics the key features of the randomized experiment described above, is formally justified when treatment assignment (i.e., description of race on the incident report, and subsequent perception by the prosecutor) is *ignorable* given the observed covariates (i.e., features of the incident report) [@imbens2015causal]. In practice, though, this approach may suffer from omitted-variable bias when the full incident report is not available to researchers, and may suffer from lack of overlap when suitable matches cannot be found for each case—limitations common to many observational studies of causal effects. To address these issues, one can restrict attention to the overlap region and gauge the robustness of estimates to varying forms and degrees of unmeasured confounding [@cornfield1959smoking; @rr; @cinelli2018making], an approach we demonstrate below. Finally, there is the issue of post-treatment bias, especially due to sample selection. @knox-2019 argue that researchers often inadvertently introduce post-treatment bias in observational studies of discrimination by subsetting on apparently intermediate outcomes—such as, in our charging example, being arrested—that themselves may be the product of discrimination. As a result, the authors caution that causal quantities of interest cannot be identified by the data in the absence of implausible assumptions, such as lack of discrimination in the initial arrest decision. In making their argument, @knox-2019 focus on the use of force by police officers in civilian encounters, but they suggest their formal critique applies more broadly, casting doubt on a wide range of observational studies of discrimination. Here we show that such customary subsetting does not pose an insurmountable threat to discrimination research. To understand why, one must precisely define the causal estimand, and carefully consider the timing of events. For instance, in our charging example, there are two relevant treatments, the officer's perception of race, affecting the officer's arrest decision, and the prosecutor's perception of race, affecting the prosecutor's charging decision. The arrest decision is post-treatment relative to the officer's perception of race but, importantly, it is pre-treatment relative to the prosecutor's perception of race. Similarly, the features of the incident report—which we must adjust for in this type of benchmark analysis—are post-treatment relative to the officer's perception of race but pre-treatment relative to the prosecutor's perception of race. In such a two-decider situation, as @greiner2011causal suggest, it is possible to recover estimates of discrimination by the second decider (e.g., in the charging decision) even if there is discrimination by the first decider (e.g., in the arrest decision). A Measure of Discrimination {#sec:post_treat_bias} =========================== We present a simple two-stage model to characterize discriminatory decision making in a variety of real-world situations and define our main causal quantity of interest—the second-stage sample average treatment effect, or $\cdes$—within this general framework. In the context of our motivating example, the $\cdes$ corresponds to the quantity that would be measured in the audit study of prosecutorial decisions described in Section \[sec:motex\]. A central aim of this paper is to formalize technical assumptions that allow one to statistically identify discrimination—more precisely, disparate treatment—in the second stage (e.g., in prosecutorial charging decisions) when data are only available for individuals who made it past the first stage (e.g., those who were arrested). Importantly, our formalization accommodates scenarios in which first-stage decisions may themselves be discriminatory. In the first stage, we assume each individual in some population is subject to a binary decision $M$, such as an offer of employment, admission to college, or law enforcement action. Those who receive a "positive" first-stage decision (e.g., those who are arrested) proceed to a second stage, where another binary decision $Y$ is made. In our running example, the case of each arrested individual is reviewed in the second stage by a prosecutor who may or may not choose to press charges. Those who are not arrested do not have a case that requires review by a prosecutor and, indeed, there may be no administrative record of those individuals. When considering racial discrimination in decisions involving Black and white individuals, our primary quantity of interest is the second-stage sample average treatment effect, $\EE[Y(b) - Y(w)]$, where $Y(z)$ indicates the potential second-stage decision and the expectation is taken over individuals reaching the second stage. Here, we imagine that the perception of race is counterfactually determined after the first-stage decision but before the second-stage decision (e.g., after arrest but before charging, perhaps by altering the description of race on the incident report viewed by a prosecutor). The second-stage sample average treatment effect thus captures discrimination in the second-stage decision among those who made it past the first stage (e.g., discrimination in charging decisions among those who were arrested). This estimand maps onto a common understanding of disparate treatment in second-stage decisions, including in our charging example. A formal model of discrimination {#ssec:discrimination_model} -------------------------------- We now formalize the above discussion to explicitly include decisions made at both the first and second stages. For ease of interpretation, we follow @greiner2011causal and motivate our statistical model by considering settings where there are two deciders (e.g., an officer and a prosecutor) whose perceptions of race—or gender, or another trait—can in theory be independently altered prior to their decisions. There are, however, examples in which one can plausibly intervene twice even when a single decider makes both decisions. For instance, an officer may decide to stop a motorist based in part on a brief impression of the motorist's skin tone as they drive past [@grogger2006testing; @OPP]. This visual impression of race could subsequently be altered if the motorist presents a driver's license bearing a name characteristic of another race group, or speaks a dialect of English at odds with the officer's expectation. It thus may be possible to apply our framework whether one imagines there are two deciders or a single one. We begin by denoting the race of an individual as perceived by the first decider at the first stage by $D \in \{w,b\}$, where, for simplicity, we consider a population consisting of only white and Black individuals. We focus on racial discrimination for concreteness, but similar considerations apply to discrimination based on other attributes, such as gender. Assuming that there is no interference between units [@imbens2015causal], we let the binary variables $M(w)$ and $M(b)$ denote the potential first-stage decisions for an individual (e.g., whether they were arrested), and write $M = M(D)$ for the observed first-stage decision. Next, we let $Z \in \{w,b\}$ denote the race of an individual as perceived by the second decider, at the second stage. In our running example, $Z$ denotes race as perceived by the prosecutor reviewing that person's file, while $D$ denotes race as perceived by the police officer during the encounter. Finally, we define the second-stage potential outcomes as a function of both the first-stage outcome $M$ (e.g., the arrest decision) and the second decider's perception of race $Z$. Thus, assuming once again that there is no interference, the observed second-stage outcome for an individual can be denoted $Y = Y(Z, M)$, where we consider four potential second-stage outcomes for each individual: $Y(z, m)$, where $z\in \{w,b\}$ and $m \in \{0,1\}$. In our example, only those who were arrested can be charged, and so $Y(b, 0) = Y(w, 0) = 0$ for all individuals. We further allow each individual to have an associated vector of (non-race) covariates $X$, representing, for example, their behavior during a police encounter, their recorded criminal history, or both. We imagine these covariates are fixed prior to the second-stage treatment (e.g., prior to the prosecutor's perception of race), since otherwise the key ignorability assumption in Definition \[def:si\] below is unlikely to hold. In practice, $X$ is only observed for a subset of the population (e.g., those who were arrested and hence in the dataset), but we nonetheless define the covariate vector for all individuals in our population of interest. These covariates are not necessary to define our causal estimands of interest, but they play an important role in constructing our statistical estimators. In this model of discrimination, we have taken care to distinguish between the (realized) first- and second-stage perceptions of race, $D$ and $Z$, because this helps to clarify the timing of events and the meaning of causal quantities. Importantly, this makes it clear that we can conceive of $D$ and $Z$ as separately manipulable. At the same time, our focus is observational settings, in which disagreement between $Z$ and $D$ may be realized only rarely, if at all, in the data we observe. For instance, barring manipulation of the incident report, it seems unlikely that an arresting officer's perception of race will frequently differ from a prosecutor's perception. Our simulation in Section \[sec:ex\] thus imposes the further constraint that perceived race is the same at each stage, though this restriction is not necessary in general. With this framing, we now formally describe the primary causal estimand we consider. This quantity, which we call the second-stage sample average treatment effect ($\cdes$) reflects discrimination in the second stage of the decision-making process outlined above, such as discrimination in the prosecutor's charging decision.[^3] \[$\cdes$\] \[def:cdes\] The *second-stage sample average treatment effect*, denoted $\cdes$, is: $$\label{eq:cdes} \cdes = \EE[Y(b, 1) - Y(w, 1) \mid M = 1].$$ The estimand in Eq. compares the potential second-stage decisions under two race perception scenarios. For example, it compares the potential charging decisions when the prosecutor perceives the individual to be either Black or white; importantly, though, the estimand does not explicitly consider the arresting officer's perception of race. Moreover, this estimand restricts to the subset of individuals who had a "positive" first-stage decision (e.g., those who were in reality arrested). Because we condition on $M = 1$ in the definition of the $\cdes$, we may equivalently write Eq. as $$\label{eq:cdes_alt} \cdes = \EE[Y(b, M) - Y(w, M) \mid M = 1].$$ We can further write $$\label{eq:cdes_second_alt} \cdes = \EE[Y(b) - Y(w) \mid M = 1],$$ where we define $Y(z) = Y(z, M)$. Among those who reach the second stage (i.e., individuals with $M=1$), $Y(z) = Y(z, 1)$ denotes the outcome of intervening *only* on the second decider's perception of race. Among those who do not reach the second stage (i.e., individuals with $M=0$), $Y(z) = Y(z, 0) = 0$. Eqs. , , and , as well as the informal estimand introduced at the beginning of Section \[sec:post\_treat\_bias\], are equivalent ways of capturing the same underlying quantity, varying only in the degree to which they are explicit about the staged nature of the process. Estimating the CDE-Ob --------------------- Having defined the $\cdes$, our goal is now to estimate it using only second-stage data. That is, we aim to estimate the $\cdes$ only using observations for those individuals who received a "positive"—and potentially discriminatory—decision in the first stage. For example, we seek to estimate discrimination in charging decisions based only on data describing those who were arrested. As we show now, an ignorability assumption, together with an overlap condition, is sufficient to guarantee the $\cdes$ can be nonparametrically identified by data on the second-stage decisions. \[Subset ignorability\] \[def:si\] We say that $Y(z, 1)$, $Z$, $M$, and $X$ satisfy *subset ignorability* if $$\label{eq:ci} Y(z, 1) \indep Z \mid X, M=1$$ for $z\in\{w,b\}$. In our recurring example, subset ignorability means that among arrested individuals, after conditioning on available covariates, race (as perceived by the prosecutor) is independent of the potential outcomes for the charging decision. As above, we can equivalently write Eq. as $$\label{eq:ci_alt} Y(z) \indep Z \mid X, M=1.$$ This latter expression makes clear that subset ignorability is closely related to the traditional ignorability assumption in causal inference, but where we have explicitly referenced the first-stage outcomes to accommodate a staged model of decision making. Almost all causal analyses implicitly rely on a version of subset ignorability, since researchers rarely make inferences about the full population of interest. For example, analyses are typically limited to the individuals who agreed to participate in the study. Even "gold standard" randomized experiments, while ideal for internal validity, frequently lack external validity because the study participants do not resemble a larger population of interest. Whenever ascribing causal interpretations to non-experimental data, one should never merely assume that ignorability or other assumptions hold. As we discuss in detail in Sections \[sec:ex\] and \[sec:empirical\] below, it is critical to evaluate the credibility of these assumptions in any given application—but we note that the assumptions we rely on are similar to those invoked in nearly every observational study of causal effects. In the traditional, single-stage setting, ignorability is sufficient to obtain consistent estimates of the average treatment effect. Likewise, we now show that in our two-stage model of discrimination, subset ignorability is sufficient to guarantee consistent estimates of the $\cdes$. In practice, if one can adjust for (nearly) all relevant factors affecting second-stage decisions, one can (approximately) satisfy subset ignorability, and in particular, one can estimate the $\cdes$ only using data available at the second stage. In the Appendix, we compare subset ignorability to several alternatives, and show that those variants tend either to be too weak to guarantee identifiability, or unnecessarily demanding for real-world applications. We emphasize that since the first-stage decision, $M$, and the covariates, $X$, can be viewed as pre-treatment relative to the second-stage intervention, concerns about post-treatment bias corrupting estimates of the $\cdes$ are more naturally thought of as familiar concerns about omitted-variable bias. In the following, we assume that $X$ is discrete for simplicity of exposition. \[thm:main\] Suppose $Y(z,1)$, $Z$, $M$, and $X$ satisfy subset ignorability, and that overlap holds, meaning that $\Pr(Z = z \mid X = x, M = 1) > 0$ for all $x$ and $z$. Then, the $\cdes$ equals $$\begin{aligned} \begin{split} & \sum_x \EE[Y \mid Z = b, X=x, M=1] \cdot \Pr(X=x \mid M = 1) \\ & \hspace{1cm} - \sum_x \EE[Y \mid Z = w, X=x, M=1] \cdot \Pr(X=x \mid M = 1). \end{split} \end{aligned}$$ Conditioning on $X$ in Eq. , we have $$\begin{aligned} \begin{split} \label{eq:decomp} \cdes & = \sum_x \EE[Y(b, 1) \mid X = x, M = 1] \cdot \Pr(X = x\mid M = 1)\\ & \hspace{1cm} - \sum_x \EE[Y(w, 1) \mid X = x, M = 1] \cdot \Pr(X = x\mid M = 1). \end{split} \end{aligned}$$ By the subset ignorability assumption, and our assumption of overlap, we can condition the summands in Eq. on $Z = b$ and $Z = w$, respectively, without changing their values, yielding $$\begin{aligned} \begin{split} \cdes & = \sum_x \B E[Y(b, 1) \mid Z = b, X = x, M = 1] \cdot \Pr(X = x\mid M = 1) \\ & \hspace{1cm} - \sum_x \EE[Y(w, 1) \mid Z = w, X = x, M = 1] \cdot \Pr(X = x\mid M = 1). \end{split} \end{aligned}$$ Finally, the statement of the proposition follows by replacing the potential outcomes with their realized values. \[cor:main\] Suppose subset ignorability and overlap hold, and that we have $n$ i.i.d. observations $(X_i, Z_i, Y_i)_{i=1}^n$ with $M_i = 1$. Let $S_{zx} = \{i : Z_i = z \land X_i = x\}$ represent the set of observations with $Z = z$ and $X = x$. Finally, let $n_{zx}$ denote the number of observation in $S_{zx}$, and let $n_x$ denote the total number of observations with $X = x$. Then the stratified difference-in-means estimator, $$\label{eq:estimator} \Delta_n = \sum_x \left[ \frac {1} {n_{bx}} \sum_{i \in S_{bx}} Y_i \right] \frac {n_x} {n} - \sum_x \left[\frac {1} {n_{wx}} \sum_{i \in S_{wx}} Y_i \right]\frac {n_x} {n},$$ yields a consistent estimate of the $\cdes$. Note that $$\begin{aligned} \lim_{n \rightarrow \infty} \frac {1} {n_{zx}} \sum_{i \in S_{zx}} Y_i & \stackrel{\text{a.s.}}{=} \EE[Y \mid Z = z, X=x, M=1], \ \text{and} \\ \lim_{n \rightarrow \infty} \frac{n_x}{n} & \stackrel{\text{a.s.}}{=} \Pr(X = x \mid M = 1). \end{aligned}$$ Consequently, each of the terms in $\Delta_n$ converges to the corresponding terms in the definition of the $\cdes$. A straightforward calculation further shows that the following expression yields a consistent estimate of the standard error of $\Delta_n$: $$\label{eq:sedim} \wh{\se}(\Delta_n) = \sqrt{\sum_x \left( \frac{n_x}{n} \right)^2 \left [ \frac{c_{bx}(1-c_{bx})}{n_{bx}} + \frac{c_{wx}(1-c_{wx})}{n_{wx}} \right] },$$ where $$\begin{aligned} c_{zx} = \frac {1} {n_{zx}} \sum_{i \in S_{zx}} Y_i. \end{aligned}$$ Eq. accordingly allows us to form confidence intervals for $\Delta_n$. The nonparametric stratified difference-in-means estimator $\Delta_n$ is the basis for nearly all applications of benchmark analysis in discrimination studies. In practice, as we discuss further in Section \[sec:ex\], it is common to approximate $\Delta_n$ via a parametric regression model—but the two estimators share the same theoretical underpinnings. As such, our analysis above simply grounds traditional benchmark analysis within a specific causal framework, and demonstrates that a particular ignorability assumption, together with overlap, is sufficient to yield valid estimates. An alternative measure of discrimination {#ssec:causal_comparison} ---------------------------------------- To better understand the $\cdes$, we now contrast it with the total effect ($\te$) [@imai-2010-general], a second estimand considered by discrimination researchers [@knox-2019; @heckman2020comment; @zhao2020note]. The total effect and the $\cdes$ differ in our setting in two ways: (1) the population of individuals about which we make inferences; and (2) the potential outcomes being contrasted. The total effect is not restricted to individuals who had a "positive" first-stage decision (e.g., it is not restricted to those who were arrested). Additionally, we imagine a causal variable that reflects a situation where the perception of race is counterfactually determined *before* the first-stage decision (instead of *after* the first-stage decision, as with the $\cdes$), and is the same at both stages. We note that, in general—as discussed in Section \[sec:motex\] and below—there is no coherent notion of a "total effect" of race, since one cannot intervene on race, *per se*. In our running example, the two treatments (i.e., the officer's perception of race and the prosecutor's perception of race) represent distinct, situation-specific notions of intervening on race. In this restricted context, then, there is a natural estimand that captures the spirit of a "total effect": comparing an individual's potential outcomes had they been perceived as white or Black when *both* the first- and second-stage decisions were made. We formalize this as follows: \[$\te$\] \[def:te\] The *total effect*, denoted $\te$, is given by: $$\label{eq:te} \te = \EE[Y(b, M(b)) - Y(w, M(w))].$$ Unlike the $\cdes$, which only measures discrimination in the second decision, the total effect measures cumulative discrimination across *both* decisions. In our recurring example, the total effect captures the effect of race at the time of arrest on the subsequent charging decision. In particular, if a charged Black individual had instead been perceived as white by an officer, they might never have been arrested, and hence never been at risk of being charged, a possibility encompassed by the definition of the total effect, but not by the $\cdes$. However, in studies of discrimination—particularly racial discrimination—there is often no clear intervention point, and the difference between the $\te$ and the $\cdes$ is largely an artifact of how one defines both the population of interest and the start of the decision-making process. What is the $\te$ in one description of events may be the $\cdes$ in another, equally valid description of the same events, as we describe next. \[t\] \[ minimum size=0.25cm, inner sep=0pt, yscale=0.35 \] (0, 10) node\[anchor=north west, inner sep=5pt\] <span>Spotted</span> rectangle (3, 0); (3, 9) node\[anchor=north west, inner sep=5pt\] <span>Stopped</span> rectangle (6, 1); (6, 8) node\[anchor=north west, inner sep=5pt\] <span>Arrested</span> rectangle (9, 2); (9, 7) node\[anchor=north west, inner sep=5pt\] <span>Charged</span> rectangle (12, 3); at (1.5, 2) (A1) ; at (1.5, 3) (B1) ; at (1.5, 4) (C1) ; at (1.5, 5) (D1) ; at (1.5, 6) (E1) ; at (1.5, 7) (F1) ; at (4.5, 3) (B2) ; at (4.5, 4) (C2) ; at (4.5, 5) (D2) ; at (4.5, 6) (E2) ; at (7.5, 4) (C3) ; at (7.5, 5) (D3) ; at (7.5, 6) (E3) ; at (10.5, 5) (D4) ; (B1) – (B2); (C1) – (C2); (D1) – (D2); (E1) – (E2); (C2) – (C3); (D2) – (D3); (E2) – (E3); (D3) – (D4); (0, 0) – (0, -7); (3, 0) – (3, -7); (6, 1) – (6, -7); (9, 2) – (9, -7); (12, 3) – (12, -7); (3, -1.1) rectangle (6, -1.9); (6, -1.1) rectangle (9, -1.9); (9, -1.1) rectangle (12, -1.9); (3, -1.1) rectangle (12, -1.9); (3, -2) – (12, -2); at (7.5, -3.5) <span>$\cdes = \te$</span>; (0, -5) – (12, -5); (0, -5.1) rectangle (3, -5.9); (3, -5.1) rectangle (6, -5.9); (6, -5.1) rectangle (9, -5.9); (9, -5.1) rectangle (12, -5.9); (0, -5.1) rectangle (12, -5.9); (-1.5, -7) node\[anchor=east\] <span>$\cdots$</span> – (13.5, -7) node\[anchor=west\] <span>$\cdots$</span>; (1.5, -7cm + 4pt) – (1.5, -7cm - 4pt) node\[anchor=north\] <span>$t_{k-1}$</span>; (4.5, -7cm + 4pt) – (4.5, -7cm - 4pt) node\[anchor=north\] <span>$t_{k}$</span>; (7.5, -7cm + 4pt) – (7.5, -7cm - 4pt) node\[anchor=north\] <span>$t_{k+1}$</span>; (10.5, -7cm + 4pt) – (10.5, -7cm - 4pt) node\[anchor=north\] <span>$t_{k+2}$</span>; In our running example, the implicit population of interest consists of those individuals stopped by the police, and the $\te$ reflects a description of events in which the decision-making process starts—and perception of race is counterfactually determined—after the stop decision has occurred but before the arrest decision has been made. We can, however, imagine moving back the clock and starting the process after an officer first spots an individual but before a stop decision has been made, with the population of interest now comprising those individuals spotted by an officer. In this case, the original $\te$ is equivalent to the $\cdes$ on this newly defined population, where the first-stage decision indicates whether an individual was *stopped*. Both the original $\te$ and the new $\cdes$ capture combined discrimination in the arrest and charging decisions, among the subset of individuals who were stopped.[^4] But the moment when an individual is spotted is no more privileged as a starting point than the moment when an officer makes a stop decision. One could similarly measure cumulative discrimination that includes the stop decision itself, either in terms of the $\te$ or the $\cdes$. For the $\te$, as above, we imagine time starting immediately after a potential police encounter, with the first-stage decision indicating whether an individual was stopped (among a population of individuals spotted by the officer). For the $\cdes$, we back up the clock once again and imagine the first-stage decision indicating whether an individual was spotted by an officer, among an even larger population of people walking through the neighborhood where the officer patrols. Figure \[fig:cde-te\] provides a graphical depiction of this interchangeability. Although the $\te$ may appear to avoid conditioning on intermediate outcomes, it simply masks a complex chain of events that came before the nominal start of the process, a chain that itself was likely influenced by discriminatory decisions. For instance, the officer spotting and stopping motorists in our running example could be patrolling the neighborhood in question because of its racial composition. The very idea of "intermediate outcomes"—a concept central to concerns about post-treatment bias—is a slippery notion in the context of discrimination studies, where there is no clear point in time where one can imagine that race is "assigned." Even birth cannot be considered the ultimate starting point since, in theory, one might include, at the least, the race of a child's parents, determined at an earlier stage, when assessing discrimination.[^5] Indeed, such generational counterfactuals may be critical for understanding systemic, institutional discrimination. Assessing Discrimination in a Stylized Scenario {#sec:ex} =============================================== Subset ignorability, in theory, is sufficient to ensure nonparametrically identified estimates of the $\cdes$, even when the first-stage decisions are discriminatory. We illustrate that idea by investigating in detail a hypothetical scenario involving discriminatory arrest decisions in the first stage and discriminatory charging decisions in the second stage. We explore the properties of simple estimators in this setting through a simulation study. We demonstrate that failing to adjust for a factor that directly influences charging decisions can result in biased estimates of discrimination in those decisions, but by accounting for all factors that directly influence charging decisions—and hence satisfying subset ignorability—one can accurately estimate the $\cdes$, even when there is unmeasured confounding in arrest decisions. This example further clarifies the conceptual importance of distinguishing between an officer's perception of race and a prosecutor's perception of race when defining and estimating our quantities of interest. We consider a hypothetical jurisdiction in which police officers observe the behavior and race of individuals who are potentially engaged in specific criminal activity (e.g., a drug transaction) and then decide whether or not to make an arrest. Subsequently, the case files of arrested individuals—consisting of a written copy of the officer's description of the encounter and the arrested individual's criminal history—are brought to a prosecutor who decides whether or not to press charges. We assume the prosecutor only observes the documented race and criminal record of the arrestee, and the arresting officer's written description of the encounter; accordingly, by construction, the charging decision depends only on these three factors. For example, the prosecutor may choose only to charge individuals who have several previous drug convictions and who were reported to be engaging in a drug transaction. Importantly, while the prosecutor has access to an officer's written report, the prosecutor does not directly observe the individual's behavior leading up to the arrest. Our goal is to estimate discrimination in charging decisions, formalized in terms of the $\cdes$. Intuitively, if we can adjust for every arrested individual's criminal history, race, and officer report, then subset ignorability would hold because the prosecutor's charging decision depends only on these factors. Thus, with these three covariates, we could generate valid estimates of discrimination in prosecutorial decisions, even without knowing all of the factors that led to an arrest, a decision that may itself have been discriminatory. However, if any of these three covariates—criminal history, race, or officer report—are unobserved, we will, in general, be unable to accurately assess discrimination in prosecutorial decisions. In both scenarios, with and without unmeasured confounding, our analysis is based on the subpopulation of arrested individuals, where we note that the subsetting (i.e., arrest) is not influenced by the prosecutor's perception of race. In this setting, the primary concern is thus omitted-variable bias, not post-treatment bias. We emphasize that we seek only to estimate discrimination in the second-stage charging decision, not cumulative discrimination stemming from both the arrest and charging decisions. In particular, while officer reports may represent an inaccurate—and discriminatory—account of events, such discrimination is distinct from that in the charging decision itself. Similarly, criminal histories reflect a form of complex, long-term discrimination that we do not aim to measure here. Alternative, and more expansive, notions of discrimination are important to understand, but here we focus on assessing the prosecutor's narrow contribution to inequities at a specific point in the process, a common statistical objective closely tied to policy decisions and legal theories of disparate treatment [@jung2018omitted]. The data-generating process {#sec:simulation} --------------------------- \[xscale = 3.75, yscale = 2, align = center\] (race) at (0, 0) <span>$T$\ <span>"True" Race</span></span>; (behavior) at (4/3, 1.8) <span>$A$\ <span>Behavior</span></span>; (race\_o) at (4/3, 0) <span>$D$\ <span>Officer-Perceived Race</span></span>; (arrest) at (8/3, 0) <span>$M$\ <span>Arrest</span></span>; (history) at (2, -2) <span>$X$\ <span>Criminal History</span></span>; (race\_p) at (8/3, -1) <span>$Z$\ <span>Prosecutor-Perceived Race</span></span>; (report) at (8/3, 1.8) <span>$R$\ <span>Officer Report</span></span>; (charge) at (4, 0) <span>$Y$\ <span>Charge</span></span>; (race) to (behavior); (race) to (race\_o); (race) to (history); (behavior) to (arrest); (behavior) to (report); (race\_o) to (arrest); (race\_o) to (report); (race\_o) to (race\_p); (race\_o) to (report); (race\_o) to (race\_p); (history) to (charge); (race\_p) to (charge); (arrest) to (charge); (report) to (charge); We now formally describe the data-generating process for our stylized example. Under the structural causal model we consider, we can both compute the true $\cdes$ and compute estimates based only on select information available to the prosecutor. In defining the generative process, we closely follow the terminology and conventions of @pearl2016causal.[^6] Our model is defined in terms of the causal directed acyclic graph (DAG) depicted in Figure \[fig:sim-dag\]. In this model, $T \in \{w,b\}$ indicates one's "true" race, and $D$ and $Z$ indicate, respectively, an officer's and a prosecutor's perception of race. Further, $M \in \{0,1\}$ indicates the arrest decision, and $Y \in \{0,1\}$ indicates the charging decision. Finally, $A$ corresponds to an individual's behavior, as observed by an officer, and $X$ and $R$ correspond, respectively, to criminal history and an officer's description of an encounter. For simplicity, in our example these latter three variables are operationalized as being binary—for example, one can imagine that $X$ indicates whether an individual had at least one previous drug conviction, $A$ indicates whether they were seen actively engaging in a drug transaction, and $R$ indicates whether they were reported by the officer to be actively engaging in a drug transaction. Officers observe $D$, $A$, and $R$ for all individuals; prosecutors observe $Z$, $X$, and $R$ only for the subset of arrested individuals. Structural causal models are defined by a set of exogenous random variables and deterministic structural equations specifying the values of all other variables in the DAG. In our example, the independent exogenous variables are: $$\begin{aligned} U_T & \sim \bern(\mu_{T}), \\ U_A,\ U_M,\ U_X,\ U_R,\ U_Y & \sim \unif(0,1), \end{aligned}$$ where $\mu_T$ is an appropriately defined constant and, for ease of exposition, we assume the Bernoulli variable $U_T$ takes values in $\{w,b\}$ rather than $\{0,1\}$. In line with our discussion in Section \[ssec:discrimination\_model\], we set the prosecutor's perception of race ($Z$) equal to the officer's perception of race ($D$), and, for simplicity, we set both equal to one's "true" race ($T$). This choice yields the following structural equations: $$\begin{aligned} f_T(u_T) & = u_T, \\ f_D(t) & = t, \\ f_Z(d) & = d. \end{aligned}$$ Now, for constants $\mu_A$, $\gamma$, $\mu_X$, and $\delta$, the structural equations for behavior ($A$) and criminal history ($X$) are given by: $$\begin{aligned} f_A(t, u_A) & = \B 1(u_A \leq \mu_A + \gamma \cdot \B 1(t = b)), \\ f_X(t, u_X) & = \B 1(u_X \leq \mu_X + \delta \cdot \B 1(t = b)). \end{aligned}$$ This specification allows for the distributions of criminal history and behavior to vary by race. For example, stopped Black individuals may be less likely to be engaged in criminal activity than stopped white individuals, corresponding to $\gamma < 0$. Finally, for constants $\alpha_0$, $\alpha_A$, $\alpha_{\text{black}}$, $\lambda_0$, $\lambda_A$, $\lambda_{\text{black}}$, $\beta_0$, $\beta_X$, $\beta_R$, and $\beta_{\text{black}}$, the structural equations for stop decisions ($M$), police reports ($R$), and charging decisions ($Y$) are given by: $$\begin{aligned} f_M(d,a,u_M) &= \B 1(u_M \leq \alpha_0 + \alpha_A \cdot a + \alpha_{\text{black}} \cdot \B 1(d = b)), \\ f_R(d,a,u_R) &= \B 1(u_R \leq \lambda_0 + \lambda_A \cdot a + \lambda_{\text{black}} \cdot \B 1(d = b)), \\ f_Y(z,m,r,x,u_Y) &= m \cdot \B 1(u_Y \leq \beta_0 + \beta_X \cdot x + \beta_R \cdot r + \beta_{\text{black}} \cdot \B 1(z = b)). \end{aligned}$$ In particular, stop decisions and police reports depend on an officer's perception of race, whereas charging decisions depend on a prosecutor's perception of race. The above structural equations, together with the distributions on the exogenous variables, fully define the joint distribution of realized and potential outcomes. In particular, $$\begin{aligned} T &= f_T(U_T), & D &= f_D(T),\\ Z &= f_Z(D), & A &= f_A(T, U_A),\\ X &= f_X(T, U_X), & M &= f_M(D, A, U_M),\\ R &= f_R(D, A, U_R), & Y &= f_Y(Z, M, R, X, U_Y). \end{aligned}$$ The primary causal quantity we seek to estimate—the $\cdes$—is defined in terms of counterfactuals $Y(z,m)$. As discussed in @pearl2009causality and @pearl2016causal, such counterfactuals require some care to define, as one must appropriately account for the exogenous variables $U$. In particular, for the causal DAG in Figure \[fig:sim-dag\], the bivariate charge potential outcomes, for counterfactual versions of prosecutor-perceived race, are given by $Y(z, m) = f_Y(z, m, R, X, U_Y)$. Further, the arrest potential outcomes—where we consider counterfactual versions of officer-perceived race—are given by $M(d) = f_M(d, A, U_M)$. In general, counterfactuals defined in this way obey the consistency rule, meaning that $M = M(D)$ and $Y = Y(Z, M)$. When $\alpha_\text{black} \geq 0$, anyone who would be arrested if white would also be arrested if Black (i.e., $M(b) \geq M(w)$). When $\alpha_\text{black} > 0$, we say arrest decisions are discriminatory since, all else being equal, an individual is more likely to be arrested if they were Black than if they were white. Likewise, $Y(b, 1) \geq Y(w, 1)$ when $\beta_\text{black} \geq 0$, meaning that an individual who would be charged if arrested and white would also be charged if arrested and Black. We say the charging decision is discriminatory when $\beta_\text{black} > 0$. <span>@ cccccccccccc</span>\ \ $T$ & $D$ & $Z$ & $A$ &$R$ & $X$ & $M(b)$ & $M(w)$ & $Y(b, 1)$ & $Y(w, 1)$ & $M$ & $Y$\ \ $b$ & $b$ & $b$ & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\ $b$ & $b$ & $b$ & 0 & 1 & 1 & 0 & 0 & 1 & 0 & 0 & 0\ $b$ & $b$ & $b$ & 1 & 0 & 1 & 1 & 0 & 1 & 1 & 1 & 1\ $w$ & $w$ & $w$ & 0 & 1 & 0 & 1 & 1 & 0 & 0 & 1 & 0\ $w$ & $w$ & $w$ & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0\ \ #### Features of our data-generating process. Table \[tb:o\] displays a sample of five rows of data generated from our model. From the full set of potential outcomes, we can compute the true $\cdes$ by directly applying Definition 1 to the generated data, taking the average difference between $Y(b, 1)$ and $Y(w, 1)$ among arrested individuals.[^7] However, given the simple linear form of our structural equations, a straightforward calculation also shows that the $\cdes$ is exactly equal to $\beta_{\text{black}}$. Our hypothetical example captures three key features of real-world discrimination studies. First, prosecutorial records do not contain all information that influenced officers' first-stage arrest decisions (i.e., prosecutors only observe $R$, not $A$). Second, our set-up allows for situations where the arrest decisions are themselves discriminatory—those where $\alpha_\text{black} > 0$. Third, the prosecutor's records include the full set of information on which charging decisions are based (i.e., $Z$, $X$, and $R$). Among those who were arrested, the charging potential outcomes depend only on one's criminal history ($X$) and the arrest report ($R$). In particular, they do not depend on one's realized, prosecutor-perceived race ($Z$). Consequently, $Y(z, 1) \indep Z \mid X, R, M = 1$, meaning that the model satisfies subset ignorability relative to $X$ and $R$. As a result, access to $X$ and $R$ guarantees the stratified difference-in-means is a consistent estimator of the $\cdes$, even if one does not have access to $A$. However, in general, $Y(z, 1) \not \indep Z \mid X, M = 1$ (and, likewise, $Y(z, 1) \not \indep Z \mid R, M = 1$), and so if one only has partial information on charging decisions, there is no guarantee the $\cdes$ can be consistently estimated. Indeed, when there is such unmeasured confounding in the prosecutor's decisions, one should expect biased estimates of the $\cdes$. Estimating the CDE-Ob --------------------- Although the data-generating procedure produces the full set of potential outcomes for each individual, the prosecutor only observes a subset of the cells—realized outcomes for arrested individuals, highlighted in gray in Table \[tb:o\]. We explore the performance of two statistical methods for estimating the $\cdes$ based on data observed by the prosecutor: the stratified difference-in-means estimator described in Eq. , and a regression-based estimator. We apply each of these methods to two types of data: the full set of information available to prosecutors (i.e., $Y$, $Z$, $X$ and $R$), and an incomplete dataset comprised only of $Y$, $Z$, and $X$—in which case we view $R$ as an unmeasured confounder. One can compute the stratified difference-in-means estimate in three steps. First, partition arrested individuals into subsets that have the same value of the available control variables (i.e., $X$ and $R$ in the complete data setting, and $X$ alone in the partial data setting). Second, on each resulting subset, compute the average difference in charging rates between Black and white individuals. Third, take a weighted average of these differences, where the weights reflect the proportion of arrested individuals in each subset. In addition, one can apply Eq. to estimate the standard error of this point estimate to generate confidence intervals. The stratified difference-in-means estimator is theoretically appealing in that it is guaranteed to yield consistent estimates of the $\cdes$ when subset ignorability and overlap hold. But the estimator can have high variance when the dimension of the covariate space is high and the sample size is small. Thus, in practice, it is common to model potential outcomes as a function of observed covariates—also known as response surface modeling [@hill2011bayesian]. In particular, on the subset of arrested individuals, one can estimate the $\cdes$ via a parametric model that estimates observed charging decisions as a function of the available information. To demonstrate this latter approach, we use a linear probability model. In the complete data setting, we have: $$\begin{aligned} \label{eq:linear-model} \EE[Y \mid Z, X, R] = \beta_0 + \beta_1 Z + \beta_2 X + \beta_3 R, \end{aligned}$$ where the model is fit on the full set of arrests seen by the prosecutor. Under this model, the $\cdes$ is approximated by the fitted coefficient $\hat{\beta}_1$, since that term captures the difference in charging potential outcomes after adjusting for the observed covariates. For our specific stylized example, the linear regression model in Eq. is in fact perfectly specified—exactly mirroring the prosecutor's charging decisions—and so we are guaranteed to obtain statistically consistent estimates. In the partial data setting, where an analyst only has access to $X$, one must fit a reduced model that excludes $R$: $$\begin{aligned} \label{eq:linear-model-incomplete} \EE[Y \mid Z, X] = \beta_0 + \beta_1 Z + \beta_2 X. \end{aligned}$$ In this case, $\hat{\beta}_1$ in general yields a biased estimate of the $\cdes$, because of the omitted variable $R$. The stratified difference-in-means estimator will in general similarly yield a biased estimate of the $\cdes$ in this omitted-variable setting. Simulation results ------------------ We perform a simulation study to understand the properties of the above estimators, varying our assumptions about discrimination and confounding. We simulate 10,000 datasets of size 100,000 for each of 25 different parameter settings. Each setting is defined as a combination of our two key discrimination parameters, $\alpha_\text{black}$ and $\beta_\text{black}$, where each parameter is allowed to take one of five values: 0.20, 0.25, 0.30, 0.35, and 0.40. Across all simulation settings, we assume the population of individuals encountered by police is $30\%$ Black (i.e., $\mu_T = 0.3$); that $30\%$ of white individuals and $40\%$ of Black individuals have a past drug conviction, indicated by $X$; and that $30\%$ of white individuals and $20\%$ of Black individuals are seen engaging in a drug transaction, indicated by $A$.[^8] These settings allow for a substantial amount of overlap across race groups with regard to the key covariates. ![ *In our hypothetical example of officer and prosecutor behavior, estimates of discrimination in charging decisions are biased when information directly influencing those decisions—in this case, an officer's report—is omitted (left). However, one can obtain accurate estimates of discrimination when accounting for all information directly influencing charging decisions (right). Each plot shows the results of 10,000 simulations for each of 25 different combinations of discrimination in officer and prosecutor decisions, given by $\alpha_\text{black}$ and $\beta_\text{black}$, respectively. The true value of the $\cdes$, indicated by the horizontal colored lines, is computed based on the full set of potential outcomes for each individual, and does not depend on the degree of discrimination in the first stage, as seen by the constant value of the $\cdes$ across different values of $\alpha_\text{black}$. For each parameter choice, we display the mean of the sampling distribution for the stratified difference-in-means estimator (solid circle) and the regression-based estimator (hollow circle), along with the interval spanned by the 2.5th and 97.5th percentiles of the sampling distribution. In the right plot ("unconfounded"), estimates are based on all three factors that directly influence charging decisions: race, criminal history, and officer report; in the left plot ("confounded"), we omit the report. When all variables directly influencing charging decisions are available, both estimators recover the true value of the $\cdes$, even when there is an unknown degree of discrimination in arrest decisions.* <span data-label="fig:cdes_sampling"></span>](figures/millisim.pdf){height="3in"} On each synthetic dataset, we estimate the $\cdes$ using both the stratified difference-in-means estimator and the regression-based estimator, and compare the results to the true population-level $\cdes$ in two scenarios. In the first scenario, we assume the officer's report $R$ is unavailable—meaning there is unmeasured confounding—and therefore only stratify based on $X$ in the difference-in-means estimator, and fit the model in Eq. for the regression-based estimator. In the second scenario, we assume that $R$ is available, and stratify on both $X$ and $R$ in the difference-in-means estimator, and fit the model in Eq. for the regression-based estimator. For each combination of $\alpha_\text{black}$ and $\beta_\text{black}$, the estimates on the 10,000 synthetic datasets yield the approximate sampling distributions for the difference-in-means and regression-based estimators. In Figure \[fig:cdes\_sampling\], we summarize each sampling distribution by its mean, 2.5th percentile, and 97.5th percentile. The solid points correspond to the difference-in-means estimator, and the hollow points to the regression-based estimator. The horizontal lines indicate the true population-level $\cdes$. In the left panel ("confounded") of Figure \[fig:cdes\_sampling\], the points lie below the horizontal lines in all cases, meaning we underestimate discrimination in charging decisions. In this setting, estimates do not account for the officer reports $R$, and so there is unmeasured confounding in the charging decisions. We set $\gamma < 0$ in our simulations, and thus stopped and arrested Black individuals are less likely to be engaging in criminal activity, a pattern (noisily) reflected in the officer reports. Because we assume these arrest reports are not available for analysis, we cannot fully adjust for their direct influence on prosecutor decisions. As a result, by adjusting for $X$ alone, we miss an important, unmeasured difference between arrested white and Black individuals, leading us to underestimate discrimination in prosecutorial decisions. In the right panel ("unconfounded") of Figure \[fig:cdes\_sampling\], the points lie on the horizontal lines in all cases, meaning the estimators are unbiased, and the range between the 2.5th and 97.5th percentiles is relatively narrow, indicating estimates are typically close to the true value. These results hold even when one is unable to assess the degree of discrimination $\alpha_{\text{black}}$ in the arrest decisions. As implied by Theorem \[thm:main\], to accurately estimate the $\cdes$, it is sufficient to measure all covariates that directly influence the prosecutor's decisions. In practice, it is nearly always impossible to do so perfectly, and thus it is important to gauge the sensitivity of estimates to unmeasured confounding in those decisions, as we demonstrate with real-world data in Section \[sec:empirical\] below. The key point is that it is sufficient to adjust for unmeasured confounding in the charging decisions alone; to estimate discrimination in these charging decisions—formalized by the $\cdes$—one need not account for unmeasured confounding in the arrest decisions themselves. Finally, in addition to examining the sampling distributions, we assessed the coverage of our 95% confidence intervals. For the difference-in-means estimator, confidence intervals were constructed
{'source': 'arXiv_out/2006'}