title: Assessing Impact of Unobserved Confounders with Sensitivity Index Probabilities through Pseudo-Experiments
authors: Beilin Jia Donglin Zeng Qing Yang Wei Pan
text: 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  firstpage–lastpage 
  00
  0000
  XXXX
  
  
  
  
  
  00.0000/x.0000-0000.0000.00000.x
  
  
  
  
  
  
  
  
  
  
  
  
  Unobserved confounders are a long-standing issue in causal inference using propensity score methods. This study proposed nonparametric indices to quantify the impact of unobserved confounders through pseudo-experiments with an application to real-world data. The study finding suggests that the proposed indices can reflect the true impact of confounders. It is hoped that this study will lead to further discussion on this important issue and help move the science of causal inference forward. 
  
  
  
  
  
  
  
  
  Causal Inference; Propensity Score Methods; Sensitivity Analysis; Unobserved Confounders.
  
  
  
  
  
  Study of the Curvature of Liquid Surface surrounding a Rotating Spherical Object in Gravity Free Space
      
    Rajdeep Tah^1 rajdeep.tah@niser.ac.in,  Sarbajit Mazumdar^1 sarbajit.mazumdar@niser.ac.in  and Krishna Kant Parida ^1 krishna.parida@niser.ac.in
  
   ^1School of Physical Sciences
  
   ^1 National Institute of Science Education and Research, Bhubaneswar, HBNI, P.O. Jatni, Khurda-752050, Odisha, India.
  
  
      Received XXXX 0000. Revised XXXX 0000.  Accepted XXXX 0000.
  =================================================================================================================================================================================================================================================================================================================
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  § INTRODUCTION
  
  
  
  Causal inference is one of the most important goals of intervention studies in the biosciences. To estimate the causal effect of an intervention (or treatment), it is necessary to sufficiently control for potential confounders. There are many techniques for dealing with confounders so as to obtain unbiased causal estimates, such as restriction, standardization, stratification, matching, regression, and randomization <cit.>. Among them, propensity score methods <cit.> are one of the most popular techniques. In propensity score methods, propensity scores associated with each treatment are first estimated by computing probabilities that subjects receive one particular treatment given all potential confounders. Then, propensity scores can be used to reduce selection bias from the confounders by balancing the distributions of the confounders between the treatment conditions through matching, stratification, weighting, or doubly robust estimation <cit.>. Once subjects are balanced by their propensity scores, causal effects can be unbiasedly estimated by comparing responses of the balanced subjects to outcomes.
  
  Unfortunately, propensity score methods can only deal with overt bias from observed confounders with the assumption that there are no unobserved (or unmeasured) confounders in the data. This assumption is called the strong ignorability in treatment assignment by <cit.>, which is, however, statistically untestable. There have been several attempts to address the sensitivity to unobserved confounders specifically for propensity score methods, such as propensity score-based approach <cit.>, propensity score calibration <cit.>, and high-dimensional propensity score adjustment <cit.> among many other general techniques of sensitivity analysis that can be applied to propensity score methods for unobserved confounders <cit.>.
  
  Both specific and general techniques of sensitivity analysis for propensity score methods can help us understand how sensitive propensity score methods are to unobserved confounders. However, in those techniques, sensitivity is normally assessed based on unknown information or a hypothetical range of the impact of unobserved confounders. Such hypothetical approach to sensitivity analysis only gives us part of the picture about unobserved confounders, and the uncertainty among researchers about their research findings still remains <cit.>. To better understand the impact of unobserved confounders on propensity score methods, it would be desirable to also assess the sensitivity based on known information or the empirical evidence in the data. This empirical approach to sensitivity analysis is on of the two sides of the same coin of sensitivity analysis with the hypothetical approach on the other side (Figure <ref>), but the former can help us understand how robust propensity score methods are to unobserved confounders given the empirical evidence we have in the data so that researchers would have confidence in their research findings. The empirical approach to sensitivity analysis is particularly meaningful with the increased availability of large observational data, such as electronic health records, that possess rich information about almost all potential confounders that are observed either directly (i.e., observed confounders) or indirectly (related to unobserved confounders). 
  
  
  
  Moreover, most of the current techniques of sensitivity analysis focus on the impact of unobserved confounders on causal effects estimated from outcome models with propensity score methods. Alternatively or more logically, as the first and utmost step of propensity score methods, propensity score estimation can also be the focal point. If propensity score estimates are robust to unobserved confounders, so are propensity score methods. As such, <cit.> proposed a novel robustness index of propensity score estimation to unobserved confounders by quantifying the extremity of the propensity score for any given subject. The extremity is calculated as the tail probability of a parametric Pearson distribution <cit.> which has the same first four moments as the empirical distribution. Therefore, the insufficiency in observed confounders leads to high extremity probability. However, one limitation of this technique is that the extremity probability may be sensitive to the parametric distribution assumption. 
  
  In this work, we adopted a similar idea to <cit.> to study the extremity of certain index due to unobserved confounders. Because it is impossible to know unobserved confounders, we assume that the behaviors of unobserved confounders are similar to those of the observed ones distribution-wise. Therefore, we performed pseudo-experiments by utilizing the jackknife (or leave-one-out) technique that treats one of the observed confounders as an unobserved while the rest as observed confounders. We used the latter to obtain the empirical distribution of the influence scores–the change of propensity scores after excluding one particular variable. We then calculated the extremity of the influence score for the experimented unobserved confounder, as compared to the empirical distribution. We defined the influence score as sensitivity index probability (SIP). 
  
  Through pseudo-experiments, we obtained SIPs for all potentially unobserved confounders with respect to the ones among the observed confounders. There are several advantages of the proposed method. First, all SIPs are computed based on ranks, thus robust to parametric distribution assumptions. Second, the derived SIPs are useful for us to understand the impact of unobserved confounders if they look similar or are related to observed ones. Third, our simulated numerical evidence showed that the proposed SIPs are consistent of the true impact of the confounders when there is no unobserved one, but can deviate largely when there is. Motivated by the latter, in our application, we compared the derived SIPs to the ones under the null that the observed confounders are known to be sufficient to identify sufficiency or insufficiency of the confounders for a given subject.
  
  
  
  
  § METHOD
  
  
  
  
   §.§ Sensitivity index probability
  
    
  Consider any subset of confounders, {X_j, j∈ A} where A is the index set. Following <cit.>, we define a propensity score (PS) associated with A as
   
      e_ A(x)=α_ A^*+x_ A^Tβ_ A^*,
   
   where x_ A is the corresponding subset of X=x and  (α_ A^*, β_ A^*) maximizes
  
      l_ A(α_ A, β_ A)≡
      E[Z (α_ A+X_ A^Tβ_ A)-log (1+exp{α_ A+X_ A^Tβ_ A})],
  
  where Z is a binary treatment variable. Essentially, we perform a standard logistic regression by regressing treatment assignments on the variables in A to estimate e_ A(x).
  
  To examine the impact of one particular confounder on the propensity score, we adopt the leave-one-variable-out method. More specifically, for any given confounder in A, say X_j, the influence score from  this confounder is defined as
  
      Δ_j,  A(x)≡ e_ A(x)-e_ A/j(x).
  
  In other words, the influence score for confounder j with respect to set A is the magnitude change of the propensity score after deleting the jth variable from A. Therefore, the larger Δ_j,  A(x) is, the higher impact of confounder j has on the propensity score.
  
  Our main goal is to evaluate whether the whole set of observed confounders, denoted as {X_1,...,X_K},  are complete. In other words, whether there is any unobserved confounder that we should be concerned about. Because we can only rely on the observed data, we assume that any unobserved confounder can be approximated by one of the observed confounders. This assumption has been established in the literature on the similar topics <cit.>. With this assumption, we consider conducting the following K pseudo-experiments. Let N={1,...,K}.  For any given j=1,..,K, we pretend that X_j is an “unobserved" confounder while the rest variables,  {X_k, k∈ N/j}, consist of all observed confounders. With the “observed" confounders, we can obtain their influence scores, Δ_k,  N/j(x), k=1,...,K, k≠ j, which reflects the relative impact of each confounder among the “observed" confounders. Therefore, treating these scores as the benchmark, we can compare the influence score for the “unobserved" X_j,  Δ_j,  N(x), with them to study the completeness of the “observed" confounders with index N/j. In particular,  we define the following sensitivity index probability (SIP) as
  
      SIP_j(x)=1/K-1∑_k=1, k≠ j^K I(|Δ_j,  N(x)|>|Δ_k,  N/j(x)|),
  
  i.e., the proportion of the observed influence scores less extreme than the true influence score for X_j. Clearly, if including X_j  in the “observed" list does not change the propensity score much, then we expect SIP_j(x) to be close to 0. Otherwise, if X_j impacts the propensity score largely, that is, Δ_j,  N(x) is large, SIP_j(x) will be close to 1. Hence, our proposed SIP yields a quantitative measurement of the completeness of the confounders X_k where k∈ N/j. Finally, because any future confounder can be any X_j, an overall SIP for evaluating the completeness of {X_1,...,X_K} is defined as
  
      SIP(x)=1/N∑_j=1^N SIP_j(x).
  
  
  
  
   §.§ Implementation using observed data
  
  
  Suppose that data are collected from n independent subjects whose treatment assignments and covariates are Z_i and  (X_i1,..., X_iK), respectively, for i=1,...,n. We wish to provide an empirical approximation to SIP_j(x) for j=1,...,K and SIP(x). For any subset A⊂{1,...,K}, we first compute (α_ A, β_ A) by maximizing
  
      l_n(α_ A, β_ A)≡
       n^-1∑_i=1^n[Z_i (α_ A+X_i, A^Tβ_ A).
      . -log (1+exp{α_ A+X_i, A^Tβ_ A})],
  
  equivalently, by fitting a logistic regression of Z on X_ A. The resulting propensity score is then
  
      e_ A(x)=α_ A+x^T β_ A.
  
  In particular, we obtain e_ A(x) for  the choice of A as follows:
  
      B={1,...,K},    B_j={1,...,K}/j,   j=1,...,K
  
  and
  
      C_j≡{1,...,K}/j,  C _j,k≡{1,...,K}/{j,k},
             j, k=1,...,K, j≠ k.
  
  Correspondingly, we obtain
  
      Δ_j,  B(x)=e_ B(x)-e_ B_j(x)
  
  and
  
      Δ_k,  C_j=e_ C_j(x)-e_ C_j,k(x).
  
  Therefore, SIP_j(x) can be estimated as
  
      SIP_j(x)=1/K-1∑_k=1, k≠ j^KI(|Δ_j,  B(x)|>| Δ_k,  C_j(x)|)
  
  and the overall SIP(x) is estimated as
  
      SIP(x)=1/K∑_j=1^K SIP_j(x).
  
  Note that this estimation applies to any fixed value x.
  
  To further obtain the confidence band for SIP_j(x), j=1,...,K, and SIP(x), we suggest the following resampling approach. We generate w_1,...,w_n from Exp(1). For each of the above index set, we estimate the coefficient of the propensity score by maximizing
  
      n^-1∑_i=1^nw_i[Z_i (α_ A+X_i, A^Tβ_ A)-log (1+exp{α_ A+X_i, A^Tβ_ A})].
  
  Essentially, we fit a weighted logistic regression. We then proceed the same way as before to obtain SIP^w_j(x) , j=1,...,K, and SIP^w(x). We repeat this process for many simulated w_1,...,w_n. The empirical distribution from this resampling approach can be used to construct the confidence interval or band for SIP_j(x), j=1,...,K, and SIP(x). 
  
  
  
  
  § NUMERICAL STUDY
  
  
  
  
  
   §.§ Simulation study without unobserved confounders
  
  
  We conduct extensive simulation studies to examine the performance of the proposed SIPs. In the first simulation study, we assume that there are no unobserved confounders. More specifically, we simulate 10 observed confounding variables, denoted as (X_1, …, X_10), where (X_1, ⋯, X_4)^T are correlated binary variables with marginal mean (0.2, 0.6, 0.1, 0.3)^T and a correlation matrix
  
      ([  1.00 -0.40  0.00  0.10; -0.40  1.00  0.00 -0.10;  0.00  0.00  1.00  0.10;  0.10 -0.10  0.10  1.00;       ])
  
    and (X_5, ⋯, X_10)^T ∼ N_6(0, Σ) with
  
      Σ = ([ 1.00 0.90 0.30 0.30 0.40 0.30; 0.90 1.00 0.40 0.30 0.50 0.20; 0.30 0.40 1.00 0.20 0.30 0.10; 0.30 0.30 0.20 1.00 0.30 0.00; 0.40 0.50 0.30 0.30 1.00 0.10; 0.30 0.20 0.10 0.00 0.10 1.00;      ])
  
  Treatment variable is generated from a logistic regression model with intercept 1.7 and the coefficients for (X_1,...,X_10) to be
  
      β   =    (β_1, …, β_10)^T 
         =    (0, 1.5, -0.5, -1.2, 4.4, -1.8, -0.3, 0, 0.9, -2)^T.
  
  We generate each dataset with sample size 500. For each dataset, following Section 2.2, we compute SIP_j(x), j=1,...,10 and SIP(x) for x taking values from all observed covariate values. We then report the average of these indices over all x's. The standard errors for the estimated SIP's are obtained from 100 resampled data.
  
  Table <ref> reports the summary statistics of the SIP estimates, where the true value is obtained based on a large dataset with size 100,000,  and the coverage probability is calculated as the proportion of samples for which the true value is contained in the 95% confidence intervals. Additionally, in the table,  the mean rank is the average rank of SIP's among these 10 confounders. From Table <ref>, the average values of the SIP estimates are close to the corresponding true values for most of confounders. The resampling-based standard errors are close to the true standard deviations and the coverage probabilities are around the nominal level. Comparing β's and the mean ranks in Table <ref>, we observe that the mean ranks are generally consistent with the ranks of absolute value of beta, indicating that the SIP can characterize the importance of each variable in the propensity scores. Finally, we compare the simulated SIP estimations with the true SIPs by creating a quantile-quantile (Q-Q) plot. The error bar in the left panel of Figure <ref> illustrates 95% confidence interval for each SIP estimation. Because the points in the left panel of Figure <ref> are around the diagonal line and the error bars are all across the diagonal line, we conclude that this set of confounders are complete for the propensity score calculation. We also note that the estimated SIPs are highly correlated with the true β's for all the confounders.
  
  
  
  
   
  
  
   §.§ Simulation study with one unobserved confounder
  
  
  In this simulation study, we consider the situation that there is an unobserved confounder so as to aim to study the performance of SIPs. We first generate 11 covariates, for which the first 4 covariates are correlated binary variables with marginal mean (0.2, 0.6, 0.1, 0.3)^T and correlation matrix
  
      ([  1.00 -0.40  0.00  0.10; -0.40  1.00  0.00 -0.10;  0.00  0.00  1.00  0.10;  0.10 -0.10  0.10  1.00;       ])
  
  The other 7 covariates follow a truncated multivariate normal distribution from -2 to 2 with mean 0 and
  
      Σ = ([ 1.00 0.90 0.30 0.30 0.40 0.30 0.20; 0.90 1.00 0.40 0.30 0.50 0.20 0.50; 0.30 0.40 1.00 0.20 0.30 0.10 0.00; 0.30 0.30 0.20 1.00 0.30 0.00 0.30; 0.40 0.50 0.30 0.30 1.00 0.10 0.70; 0.30 0.20 0.10 0.00 0.10 1.00 0.10; 0.20 0.50 0.00 0.30 0.70 0.10 1.00;      ])
  
  
  To generate the outcome Z, we use a logistic regression model with intercept 0 and the regression coefficients as
  
      β   =    (β_1, …, β_11)^T 
         =    (0, 0.7, -1.6, -0.8, 2, -0.6, -0.2, 0.8, 1.4, 0.3, 1.6)^T.
  
  The observed data for each subject consist of its treatment outcome and the first 10 covariate values. Thus, the last important covariate, whose coefficient is 1.6, is assumed to be unobserved. In this simulation study, each dataset has size 500 and we repeat 1000 times. We calculate the SIPs using the observed data as before and the results are summarized in Table <ref>. 
  We note that the pattern of the SIPs are inconsistent of the true β's. This is because when there is a significant unobserved confounder, the observed SIPs cannot fully capture the impact of all confounders.
  
  
  
  
  
  
  
  
   §.§ Simulation study based on real-world data
  
  
  We conduct an additional simulation study based on real-world data from a national database of 10,500 at-risk youth in the National Cross-Site Evaluation of High-Risk Youth Demonstration Grant Programs, which was funded by the Substance Abuse and Mental Health Services Administration's Center for Substance Abuse Prevention <cit.>. This multiple-site evaluation study assessed funded prevention programs over 18 months with respect to socio-demographic risk and protective factors. For this simulation study, a sample of 547 youth whose initial uses of substance were prior to entry to the national evaluation is selected. The data set includes 213 youth in the prevention group and 334 in the comparison group. There are 22 confounding variables collected for each subject, including age, gender, race/ethnicity, family composition, family composition, family supervision, school prevention, community protection, neighborhood risk, family bonding, school bonding, self-efficacy, belief in self, self-control, social confidence, parental use attitudes, peer use attitudes, and peer use.
  
  In the simulation study, we first standardize all covariates to obtain 𝐗. Next, the binary outcome Z is simulated in a logistic regression model with the coefficients same as the estimates from fitting a logistic regression model to the real data. We simulate 100 datasets. The average values of SIP_j, j=1,2,…,22 over 547 subjects are given in Table <ref>, where the overall SIP is estimated as the average of SIP_j, j=1,2,…,22. We also report in Table <ref> the SIPs for a specific covariate value same as subject ID = 311031, for example. The summaries of the simulation results in both Tables <ref> and <ref> indicate that the conclusions are similar to what we had before.
  
  
  
  
  
  
  
  
  We use the SIP results from the above simulation study, for which we know there is no unobserved confounder, to compare the SIPs calculated using the real data. The results for the SIPs averaged over all subjects and for the SIPs for one particular covariate value are plotted in the middle and right panel plots of Figure <ref>, respectively. The error bars in both figures indicate 95% confidence interval for each SIP estimation. In the middle panel of Figure <ref>, some points in bottom-left corner are below the diagonal line and error bars are not across the diagonal line, suggesting that these 22 confounders may not be sufficient for the propensity score estimation; however, for the particular covariate value from Subject 311031, the right panel of Figure <ref> shows that observed confounders are sufficient in the propensity score estimation for this particular subject.
  
  
  
  
  
  
  
  § CONCLUSION
  
  
  Unobserved confounding is a long-standing issue in causal inference using propensity score methods. Extended from the prior research on this topic, we have proposed nonparametric indices to quantify the impact of unobserved confounders through pseudo-experiments. The numerical studies suggest that the proposed indices can reflect the true impact of these confounders. The application to the real-world data provides a way to identify sufficiency or insufficiency of the observed confounders.
  
  The pseudo-experiments are based on the assumption that the unobserved confounder is similar to some of the observed ones. This assumption can be restrictive in practice. One possible improvement is to conduct sensitivity analysis by simulating some different confounders to study their impact. It will also be interesting to develop test statistics based on the proposed indices. One possibility is to compare the derived ones vs the ones from the null. Another caveat is that although the proposed indices are robust, they may not be sensitive to detect potential confounders. One possible modification is to build some parametric or semiparametric models of the empirical indices and then evaluate the extremity under those models.
  
  In sum, this study is one step forward to address the unobserved confounding issue in causal inference. Despite a few caveats and limitations, it is hoped that this study will stimulate further discussion on this critical issue and help move the science of causal inference forward.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
    biom 
   
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  